Effectiveness of an Internet-based graduate engineering management course

Effectiveness of an Internet-based graduate engineering management course

evans, Rosemarie M

ABSTRACT

This research provides engineering educators analytical evidence as to the effectiveness of Intemet-based course instruction. The research examined the University of Missouri-Rolla’s Internetbased Advanced Production and Operations Management course, with a focus on determining the effectiveness ofthe Internet-based education tools used. Over 100 students in five Internet-based classes and one traditional, in-class control group were given three sets of surveys, learning style assessments, a course pre-test, and a course final examination. Multiple conclusions were drawn from this study based on analyses ofthe data collected. First, the Internet-based students performed equally as wel as the control group as measured bythe difference between pre-test and post-test scores. Second, the Internet-based students were found to have had exaggerated time requirement expectations for taking a course in the Internet environment. Third, the students rated the effectiveness and satisfaction positively for the Internet classroom format. Initially, the Internet-based students were skeptical of electronic lectures but their experiences were positive.

I.INTRODUCTION

As the global marketplace becomes more competitive, engineers must be able to quickly access, collect, absorb, and utilize technical knowledge from multiple sources. Accordingly, with increasing global interaction, engineers rely more heavily on computers and electronic communication available via the Internet. The use of technology, in particular the World Wide Web, has changed and will continue to change current paradigms regarding the roles of both educators and students at all levels of learning. But to what degree and in what ways higher education will be “revolutionized”remains to be seen.

II. LITERATURE SEARCH

A Internet and Engineering Education

Many definitions of Internet-based, or Web-based, courses exist in literature. Depending on which subset of Internet tools is chosen, Internet-based courses can range from those totally Internet dependent to those which only supplement traditional lectures with complementary Intemet-based material. In the former case, a course is taught entirely across the Internet and does not require any physical interaction between students and professors, as all communications are performed electronically. In the latter case, a professor might complement his or her campus course with an Internet site or ‘Web Page” containing such information as the class syllabus and assignments.

B Distance Education

Leonard defines modern distance education as involving 1) the separation of the teacher and the learners in space and often in time, 2) the shift in volitional control from the teacher to the students, and 3) contiguous interactive communication between teachers and students through the use of electronic media.’

Internet-based tools such as e-mail, chat rooms, and interactive class ‘Web Pages” have become the tools of choice for many distance educators around the world. Unlike “traditional” distance education, the Internet allows students the flexibility to access course material at anytime from anywhere. Now, through Internet-based instruction, an individual student can receive the information at his or her rate of comprehension, allowing for material repetition if necessary.

C Advantages of Distance Education

There are many advantages of distance education stated in current literature. First, technology based distance education provides educational opportunities for non-traditional students, typically over the age of 25. In 1994, 41% of all college students were from this demographic group.’ Second, non-traditional classes allow working individuals of all ages to pursue additional education without interrupting their career or family responsibilities.’ Third, nontraditional courses enable students access to the best professors and guest lecturers, since students can attend the most prestigious schools without leaving their homes.

Some view modem technology as the means of providing high quality education to students more economically without funding expensive campus facilities.’ Traditional classrooms have physical and spatial boundaries, while Internet-based classrooms do not. Internet-based education allows students to have greater control of the rate of learning since material is prerecorded, usually in a digital or hypertext format. Traditional classrooms are synchronous, meaning time-dependent, while Internet-based classrooms are asynchronous, time-independent. Therefore, students are not confined by time or space, resulting in less travel time and greater student flexibility.

Internet technology allows for the interaction and exchange of ideas among students in different parts of the world with diverse educational and professional backgrounds. It provides tools that enable individuals to pursue lifelong learning, a quality necessary to remain competitive within and outside academia.’ In addition, students have access to a large depository of continuously updated knowledge, in contrast to static textbooks or a single professor’s knowledge.4

D. Disadvantages of Distance Education

Common concerns among distance learning students include feelings of isolation, information overload, and a perceived lack of cohesiveness of material since they are able to access materials at their own pace and in the order of their own choosing.5 As the use of technology increases on campus, so do the concerns of current and future instructors. Some educators argue that universities may use technology blindly to solve existing financial woes by teaching more students with fewer instructors.3 Schwartz raises additional concerns involving the accreditation of courses offered by nontraditional means, and granting a degree based on courses from several schools.

Concerns are also raised by students. In regard to the technology being used, students have complained of feeling frustrated due to a lack of knowledge concerning how to use required computer hardware and software.5 Schrum notes frustration levels peak when students are unable to connect to the network@ especially when attempting to connect to unavailable or unreliable computer resources.6 Additional concerns include security and intellectual property rights for both educators and students .3,7

Motivation of students, as well as instructors, can also be problematic in distance learning environments. Moore and Kearsley estimate that as many as 30 to 50% of distance education students drop the course prior to its completion, regardless of educational technology used.8 Thus, the success of Internet-based education depends heavily on a student’s initiative and individual commitment to learn, in addition to an instructor’s ability to provide a motivational learning environment.

E. Learning Styles

Learning styles were defined by Kolb as a systematic method to assess how individuals learn information.’ The experimental learning theory as defined by Kolb provides a model of one’s learning process, consistent with the existing theories of human cognition and the stages of human intellectual development.9 According to the experimental learning model, the learning process consists of four interrelated stages: concrete experience, reflective observation, abstract conceptualization, and active experimentation. The Kolb Learning Style Inventory (LSI) is based on this learning model.9 It has become a popular tool used to categorize four distinct types of learners: accomodators, divergers, assimilators, and convergers, one for each quadrant of the learning model. These learning types are based on combinations of the four learning abilities as depicted in figure 1.

More recently, Felder and Silverman developed another learning styles assessment tool, the Index of Learning Styles (ILS). In contrast to Kolb’s LSI, which was developed to assess all students regardless of academic major, Felder and Silverman’s was developed mainly to assess the learning styles of engineering students. Instead of two dimensions, as in Kolb’s LSI, the Felder and Silverman assessment classifies students along four major dimensions as shown in table 1. Unlike Kolb’s LSI, the ILS is a less established tool, not validated statistically, and is currently under evaluation.10

III. RESEARCH METHODOLOGY

A. Introduction

The University of Missouri-Rolla’s Advanced Production and Operations Management course is a three credit graduate level course which provides graduate engineering management students a broad fundamental knowledge of the multifaceted areas of production and operations. Topics covered in this course include decision theory, forecasting, total quality management (TQM), statistical process control (SPC), acceptance sampling, product design, process selection, facilities layout, location planning, aggregate planning, material requirements planning, scheduling, and project management.

Similar to their traditional classroom counterparts, the students in this Internet-based course were required to purchase a textbook, supplementary casebook, and prepared class notes. The prepared class notes, created by the professor, contained all the visual lecture material in addition to homework answers. On the first day of class, the professor met with the students at an on-campus site to provide an introduction to the course format and to the Internet-based tools used for this course. There were no further meetings of the Internet-based class until the last day of the semester, when the final examination was administered.

The students were required to have access to computers with minimum hardware and software configurations including an IBM compatible computer with either the Windows 95 or Windows NT operating system. Recommended hardware included an Intel Pentium or compatible central processing unit (CPU), at least 100 Mb temporary (Cache) directory, a VGA video card, an audio card with speakers, and a 33.6 kbit or faster modem.

The Advanced Production and Operations Management course utilized an Internet site where its home page resided. The Internetbased course material was designed and created by the course instructor using Microsoft FrontPage software. This home page included links to the class syllabus, assignment lists with instruction, grading policies, the bulletin board (updated daily), the class chat room, class lectures and quizzes.

The lecture, quiz, and homework modules were created by the instructor using Lotus ScreenCam software. These modules were in compressed data format and therefore required the student to have decompression or “unzip ” software on his or her computer. These modules provided the lectures, self-tests, and homework answers for the topics examined in the course. The instructor spent approximately five hours for every one-hour of instruction to convert existing materials in these modules for use on the Internet. The time required to develop a similar completely new course could be five to six times as much-or more.

The modules for this class resembled a Microsoft PowerPoint presentation with associated audio instruction. This presentation software is capable of displaying text, figures, and other graphics. Students were able to stop the lecture or quiz at any time and repeat if necessary. In similar fashion to a video cassette recorder (VCR), the student could stop, pause, speed up or slow down a course module. Students had the option to purchase the lecture and quiz material on CD-ROM, thus avoiding possible delays in downloading compressed or “zipped” formatted modules. Most students chose this option. Visitors to the course’s Internet site at “http://www. umr.edu/-daily/frame1O.htm” can download a sample uncompressed lecture module, not requiring “unzip” software.

Student-instructor and student-student interaction was encouraged via two modes of electronic communication, the chat room and e-mail. As an option for students less familiar with this technology, the instructor was available via phone and fax or, if necessary, at assigned office hours. However, interaction using electronic means was encouraged and used most often by the students.

During the semester, there were two group assignments for which students had the option to communicate with each other either electronically or face-to-face. These assignments were from the case book in which a group of three to four students would collectively examine a case using material reviewed in the course and prepare a report. Students had the option to send their reports by e-mail or send a paper copy depending on their comfort level. In addition, students were required to post at least two summaries of current articles relevant to course material on the chat room for credit.

B. Research Model

Most of the students enrolled in this graduate course were U.S. Army officers assigned to a nearby army post, Fort Leonard Wood (FLW, and pursuing a UMR Engineering Management masters degree. In addition, there was one non-military Internet-based class of students in the Winter 1998 semester which included both on-campus students and remote students obtaining graduate degrees through the National Technological University (NTU).

A total of six classes of students participated in this research. There was one traditional, in residence class serving as a control group and five Internet-based classes. The first class began in August 1997 with the last class concluding in July 1998. More than 100 students were included in this study. Table 2 provides an overview of groups examined in this research.

C Research Design

1) Experimental Design. Each student in the Internet-based class and traditional class completed three sets of surveys and tests; 1) the initial student demographic and computer background, and expectations surveys, 2) the mid-term student evaluation and Kolb’s Learning Style Inventory, and 3) the student final experience survey and Felder-Silverman Index of Learning Styles. In addition, all but Internet-based Class 1 received a pre-test while all classes received a post-test, a three-hour comprehensive final examination. The structure of this examination remained constant to allow for comparison between classes. Care was taken by the instructor to protect the integrity of the exam. The structure of the army program allowed for little interaction between the students of the current class and previous classes, fin-ther preventing any possible advantage for students in the later classes. To accomplish data collection, three sets of data were collected; one each at the beginning, middle and end of the course. A data collection schedule is shown in table 3.

2) Variables. The individual difference variables examined for both the traditional and Intemet-based classes were: learning styles, demographics, and computer familiarity. Furthermore, the Kolb Learning Styles Inventory and Felder-Silverman’s Index of Learning Styles Responses provide two additional individual difference factors. Table 4 summarizes the individual difference factors and possible answer formats for the Initial Student Demographic and Computer Background surveys.

In addition to these individual difference variables, four subjective variables of student expectations were examined; time required for class, overall satisfaction, the effectiveness of Internet-based tools in regards to material comprehension, and the satisfaction associated with each Internet tool. Table 5 summarizes the factors gathered in the expectations survey.

In contrast to the individual difference variables, four subjective variables of student experiences were examined solely for the Internet-based class; time required for class, overall satisfaction, the effectiveness of Internet-based tools in regards to material comprehension, and the satisfaction associated with each Internet tool. Table 6 summarizes these factors and possible answer formats of the mid-term student evaluation and final experience survey.

3) Research Considerations. Prior to statistical evaluation of the survey, individual difference variables such as demographic information, prior computer usage, and competence, learning styles and pre-existing course material knowledge were examined. The same professor taught all classes included in this study, both the control group and the Internet-based. This professor used identical textbooks, case books, prepared notes, lecture material, homework, self quiz problems, and the same final examination, in addition to the same grading policy, for all classes. Therefore, the study did not need to address the additional effects of teaching styles.

All but one of the classes examined, Internet Class 5, were composed of graduate engineering students enrolled through a local Army base. Therefore, most students were of similar age, profession, and gender, and the student population remained fairly consistent throughout the year in which classes were examined. In addition, demographic and computer background surveys coupled with the distributed learning style tests would confirm the similarities in the backgrounds of students involved in this research.

To assess the effectiveness of the Internet-based technology, a Likert scale was used to quantify subjective data.” This provided for qualitative as well as quantitative assessments of effectiveness. For this research, a 5% significance level (alpha) was chosen.

IV. RESULTS

A comparison of student demographic information is shown in table 7. One can conclude that the students in the control group and the five Internet-based groups were not shown to be statistically different, with the exception of differences in pre-test scores between the Internet and control groups. The Internet classes pre-test scores were lower than those of the control group, allowing for a conservative comparison between the two groups. The difference between the pre-test and final examination was used to gage the final course subject knowledge. If one considers these factors, analyzed in combination through a Wilks Lambda multivariate test of significance, the level of significance is 0.057. Thus, one can conclude the control group and Internet-based student populations were not statistically different.

Table 8 demonstrates that students in both groups had comparable learning styles. As with the demographic variables above, if one examines these factors through a Wilks Lambda multivariate test of significance, the level of significance is 0.643. Thus, the learning style preferences for the student populations were not statistically different.

Time expectations, shown in table 9, prove to be the only statistically insignificant variable between the two groups. If one examines all of the factors in table 9 through a Wilks Lambda multivariate test of significance, the level of significance is 0.01. This observation of unequal expectations, in every factor but time expectation, may be attributed to the difference in prior knowledge about an Internetbased course. The Internet class students were provided more information at the beginning of the course about the Internet class environment. The control group students were not provided a similar explanation of the Internet course, as they were to receive their course in a more traditional manner.

Student experiences are depicted in table 10. One can conclude that students in the control group and Internet-based classes had similar experiences regarding time and overall course learning effectiveness. Note, “N/A!’ is for unavailable comparisons. Additional conclusions can be drawn by comparing the Internet-based students’ expectations to their experiences from tables 9 and 10. First, the time requirement experienced by the Internet-based students was actually less than expected. Thus, the students believed the course would take much more time than it actually did. One explanation could be the Internet-based students’ unfamiliarity with the computer applications required in the course. The overall learning effectiveness experienced was rated higher than expected by the students. Electronic lecture effectiveness and satisfaction experiences were also rated higher than expected.

Conclusions can also be drawn about the students’ learning during the course. The mean scores on the pre-test were 56.21% for the control group and 51.14% for the internet-based students. The pretest consisted of multiple choice and true/false questions. The final exam was a problem based comprehensive exam with an average score of 87%. There was no significant difference in the mean final exam grade for the various classes. As shown in figure 2, the Internet students performed slightly better based on the difference in their individual pre-test and final exam scores. Since there was no significant difference in group means of the test score changes, one can conclude from the evidence that there were not statistical differences between the performance of the control group and Intemet-based students. Therefore, the Internet based technology did not prove to hinder the learning process.

V. CONCLUSIONS

The results of this study are based on a single class taught by an individual instructor. The students involved were from a very homogeneous group. Applying specific results in a different situation may not be appropriate. However, there are several general conclusions that can be drawn from this study.

This research made several findings. First, the Internet-based students performed equally as well as the control group students. Using the Internet technologies did not hamper the learning process for the students. Second, students tend to have exaggerated time requirement expectations for Internet-based classes. One possible explanation could be that these students have reservations about the time required to learn the necessary technology in addition to the course material. Third, students tend to have positive overall course effectiveness and satisfaction experiences in an Internet classroom format. Fourth, students tend to be initially skeptical of electronic lectures but experiences prove positive.

Based on this research, there are several recommendations an instructor may follow when pursuing Internet-based instruction. First, it is critical for instructors to explain fully the details of the course at the beginning in a positive manner. This should calm student fears and provide a more effective learning environment. Second, the instructor must create well structured, easy to follow and use electronic lectures. Third, expectations regarding Internetbased tools of communication must be addressed if the course is to be successful for all students. In concluding, an instructor must carefully weigh the advantages and disadvantages of the current communication tools to provide a stimulating Internet-based education environment that is conducive to learning.

REFERENCES

1. Leonard, D.C., “Using the WEB for Graduate Courses and Technical Communication with Distance Learners,” Technical Communications, vol. 43, no. 4, Nov., 1996, pp. 388-401.

2. Gubernick, L., and A. Ebeling, “I Got My Degree through E-mail,” Forbes, June 16, 1997, pp. 84@92.

3. Schwartz, RA, “The Virtual Universe,” ASEE Prism, Dec., 1995, pp. 22-26.

4. Flake, J.L., “The World Wide Web and Education,” Computers in the Schools, vol. 12, no. 1/2,1996, pp. 89-101.

5. Hill, JR, Distance Learning Environments via the World Wide Web, In Khan, B.H. (Ed.), Web-Based Instruction, Educational Technology Publications, Inc., Englewood Cliffs, NJ, 1997.

6. Schrum, L., “Professional Development in the Information Age: an Online Experience,” Educational Technology, vol. 32, no. 12, Dec., 1992, pp. 49-53.

7. Rasmussen, K., Northrup, P., and R. Lee, “Implementing WebBased Instruction,” In Khan, B.H (Ed.), Web-Based Instruction, Educational Technology Publications, Inc., Englewood Cliffs, NJ, 1997.

8. Moore, M.G., and G. Kearsley, “Distance Education: A Systems View,”Wadwoth, Belmont, CA, 1996.

9. Kolb, David A., “Learning Styles Inventory-Technical Manual,” McBer and Company, Boston, MA, 1976.

10. Felder, Richard M., and L.K. Silverman, “Learning and Teacher Styles in Engineering Education,” ASEE Engineering Education, vol. 78, no. 7, April, 1988, pp. 674-681.

11. Likert, R.A, “Technique for the Measurement of Attitudes,” Archives of Psychology, New York, NY, June, 1932.

ROSEMARIE M. EVANS Industrial Engineering Department Wright State University

SUSAN L. MURRAY Engineering Management Department University of Missouri-Rolla

MADISON DAILY Engineering Management Department University of Missouri-Rolla

RICHARD HALL Psychology Department University of Missouri-Rolla

MADISON M.DAILY

Dr. Madison M. Daily was the Koplar Professor in Engineering Management at the University of Missouri-Rolla until his recent retirement. He received his Ph.D. in Engineering Management from the University of Missouri-Rolla. He has been involved with technology in the classroom and distance education for several years, teaching on NTU and the University of Missouri Video Network. ln 1997, he developed and began teaching his first Internetbased course.

Address: Engineering Management Department, University of Missouri-Rolla, Rolla, MO, 65409; telephone: 573-341-4572-, fax: 573-341-6567; e-mail: Daily@umr.edu.

ROSEMARIE MAFFEI EVANS

Dr. Rosemarie Maffei Evans has recently completed her dissertation regarding the effectiveness of Intemet-based technology in engineering education at the University of Missouri-Rolla. She received her M.S. in Industrial Engineering from Texas Tech University and B.E. in Mechanical Engineering from Stevens Institute of Technology. She is currently an adjunct professor at Wright State University, Fairborn, Ohio.

Address: 416 Redbud Lane, Wright-Patterson AFB, OH, 45433; telephone: 937-879-4681; e-mail: Rosemarie.evans@ wright. edu.

RICHARD H. HALL

Dr. Richard H. Hall is an Associate Professor of Psychology at the University of Missouri-Rolla. He received his Ph.D. and M.S. in Experimental Psychology from Texas Christian University and his B.A. in Psychology from North Texas State University. His current research interests include educational psychology, in particular cooperative/collaborative learning, especially guided/scripted interactions; spatial/graphic textual displays (knowledge maps); and instructional technology.

Address: 112H-SS, Psychology Department, University of Missouri-Rolla, Rolla, MO, 65409; telephone: 573-341-4811; fax: 573-341-1110; e-mail: Rhall*umr.edu.

SUSAN L. MURRAY (CONTACT AUTHOR)

Dr. Susan L. Murray is an Assistant Professor of Engineering Management at the University of Missouri-Rolla. She received her Ph.D. and B.S. in Industrial Engineering from Texas A&M University and her M.S. in Industrial Engineering from the University of Texas at Arlington. She is a registered Professional Engineer in Texas. Dr. Murray has over seven years of industrial experience in the aerospace and defense field.

Address. Engineering Management Department, University of Missouri-Rolla, Rolla, MO, 65409; telephone: 573-341-4038; fax. 573-341-6567; e-mail: Murray@umr.edu.

Copyright American Society for Engineering Education Jan 2000

Provided by ProQuest Information and Learning Company. All rights Reserved