How might classroom time be used given WWW-based lectures?

How might classroom time be used given WWW-based lectures?

Wallace, David R

ABSTRACT

This paper investigates how traditional lecture time might be used given the availability of effective web-based courseware for delivering materials typically presented in the classroom. After providing an overview of web-based educational materials, the effectiveness of two teaching approaches is compared using a product design lecture on visual prototyping. One group of students prepared for class using web-based materials and then received a lecture-style second coverage ofthe topic in class (web+class lecture). The other group of students prepared for class using the same web-based materials and then worked with the faculty, applying only a small portion ofthe subject matter, on illustrative examples (web+limited experience). The two groups then completed an assignment based on the subject matter. The average grade performance ofthe web+limited experience group was 10.8% higher than for the web+class lecture group. An achieved significance level of O.001 provides very strong evidence to reject the hypothesis that the two groups performed equally. Analysis of how the two student groups used the web-lecture resource showed that the form of class instruction had a strong influence on student motivation for independent study. The average time spent in web-based preparation by the web+limited experience group was 1.6 times greater than the average preparation time for the web+class lecture group. Sixty percent of the web+class lecture group prepared less than the least prepared student in the web+limited experience group. These findings suggest that, if codified materials are primarily delivered using a medium such as the WWW, traditional classroom time might be liberated for potentiallyhigher value-added activities such as mentoring and experiential activities.

I. INTRODUCTION

The rapid growth of WWW-based teaching materials has awakened interest in their use for local learning on-campus in addition to distance learning applications. The possibility of disseminating appropriately designed teaching materials at-a-distance as effectively as traditional lecture is not a matter of question. In a review of over 230 articles written between 1928 and 1996 (CALOS)1 comparing distance learning with classroom teaching, Russell2 found the performance of distance and local students to be equivalent.

The WWW provides unprecedented flexibility for delivering teaching materials. In a previous experiment, a general structure for the web-based equivalent of a classroom lecture was developed. The premise of this structure was “different students learn differently”. The structure allowed students to: follow lectures as a continuous story or reference subtopics directly; choose the order for receiving motivation, theory, applications and examples; and follow materials in video, audio, or text-and-image formats. After using the structure to implement the equivalent of a 90-minute classroom lecture on visual prototyping, a comparison of students receiving web-only instruction to those receiving classroom-only instruction found evidence that the web-only group performed differently and had a higher average performance.

Does this mean local study and universities will go away? Probably not!4 Bringing students and faculty together in a common location with appropriate facilities is a unique and potentially high valueadded opportunity.5,6 Our interest in further studying the use of the WWW in engineering education is to gain insight into how the quality of education might be improved by maximizing the value of the on-campus experience. Evidence that the WWW can be used to deliver codified information traditionally presented in lectures suggests the possibility that classroom time might be available for different uses.

This paper presents a comparison of the grade performance of students receiving a web-lecture combined with a complete lecturestyle second coverage of the material in class to students receiving a web-lecture combined with limited topical coverage experiential activity with faculty. Additionally, data were gathered to provide insight into how the different teaching approaches motivated the students in independent study.

The paper begins by reviewing our pedagogical motivation for exploring WWW-based educational materials and then provides an overview of web-based teaching initiatives. There is an enormous body of web-based educational material and so we also attempt to provide a classification for these initiatives. The general web-lecture structure and visual prototyping lecture used in both the previous experiment3 and the experiment presented here, is then briefly reviewed. Next, results comparing the grade performance of the two student groups are presented and discussed. Finally, a model for how WWW-based lectures might be used in both distance and on-campus learning contexts is entertained.

II. PEDAGOGICAL MOTIVATION

We believe the WWW may be an enabling technology to provide new opportunities for students to play a more active role in the acquisition of knowledge. This multi-media learning environment has the potential to increase the availability, accessibility, individual control and flexibility of teaching materials.7 If the codified materials traditionally disseminated in the classroom lecture can be effectively communicated using the WWW, then the time allocated for classroom lectures might be used differently- consolidating ideas, mentoring, gaining experience, and motivating independent study. Cognitive psychology emphasizes that effective learning is related to an awareness of and an ability to control one’s thought processes.8 This “metacognition” grows as students become aware of their ability to learn, remember, and solve problems. Ultimately, we want students to learn how to learn.

The lecture is the most commonly used teaching method in university instruction.9 It is the primary knowledge transfer mechanism. Lectures can be very effective but they do not readily accommodate variations in learning styles. Although many new techniques have been introduced to help motivate and challenge students with different learning styles,10 the teacher is still responsible for pacing the lecture and choosing the material order and presentation medium. Thus, the students still play a passive role in many aspects of lecture-based instruction. A constructivist teaching approach asserts that responsibility for learning lies with the student.11 The student should have control over and interact with teaching materials to form and internalize concepts.12 Although this model is logical from a motivational and the learning-to-learn context, it seems impractical for the classroom-based lecture instruction environment.

WWW-based teaching is pertinent to both local and distance education, and literature studying the efficacy of the WWW in distance education is supportive of the premise that WWW-based lectures can be very effective. In an experiment comparing on-line learning with classroom learning at the California State University at Northridge, a professor randomly divided a statistics class in half, teaching one half by lecture and the other half by Web assignments, on-line discussion groups, and e-mail. The students who did not attend lectures scored an average of 20% higher than those who attended classroom lectures.13 The author cites his motivation for the experiment as a desire to provide quantitative experimental evidence comparing web-based teaching to classroom lectures.

The CALOS project provided a comprehensive learning environment for delivery of an Operating Systems course using the WWW.14 An evaluation of the environment revealed that students using web-based lectures performed equally to those attending traditional lectures.1 It is notable that students who both used the web and attended class performed better than those in either of the other two groups. This model, web plus complete in-class coverage, is one of the test cases for the experiment described in this paper.

The trend of these results is consistent with our own findings and support the hypothesis that alternative uses of classroom time might be considered.

III. SURVEY OF WEB-BASED EDUCATIONAL MATERIALS

Use of WWW-based course materials is growing at an incredible rate and these materials have an enormous range in both sophistication and intended use. In this section an overview of web-based teaching materials is provided. We categorized them in the same way labs, recitations, and lectures are categorized in traditional teaching. Our strategy in this section is to provide a few illustrative examples for each category. However, we recognize that the boundaries between the different categories of WWW-based materials will be gray because of the hybrid nature of the environment. The categories used are: Course Administration; Reference Textbooks; Lectures; Laboratory Simulation and Experiments; and Recitations/Assignments/Grading. Ideally, a complete learning environment would include all of these components. Examples are highlighted in the final subsections on Virtual Educational Institutions and Collaborative Learning. In addition to sites highlighted in this section, a multitude of WWW-based teaching materials may be found at the World Lecture Hall.15 This site, developed by the University of Texas, contains links to academic pages throughout the world.

The survey in this section is intended to provide a contextual framework for the experiment presented in this paper, which focuses on the lecture component of education and how WWW- based lecture materials might influence the use of classroom time.

A. Web-Based Course Administration

These sites transfer basic operational course information. They typically provide access to course descriptions, faculty teaching assistant information, course syllabi, reading lists, assignments, lecture and lab times/locations, and, in many cases, links to relevant resources available on the WWW. Basic lecture notes (derived from class notes) may also be available. There are many sites of this type (for example, see reference 16).

Many aspects of course administration are easily implemented, so these sites are a logical entry point to create a visible presence for a class on the WWW. Even so, the design of the site is important to its effectiveness and usefulness,17,18 and many universities have centrally provided course-page templates for faculty or graduate student use. These services are often provided through either computing systems or information technology centers and provide a standardized document structure and layout. The templates also facilitate use of the WWW for those less comfortable with HTML.

The Instructional Web Project at North Dakota State University (NDSU)19 is an example. Templates provide a structure, layout, and links for a syllabus, course schedules, assignments, resources, and a discussion forum. The NDSU Project recently implemented a commercially available web-based discussion forum manager.20 Increasingly, software applications that allow interactive conferencing or manage discussions are incorporated.5

The Virtual Classroom Interface (VCI)21 offers a suite of Web tools developed at the University of Illinois to facilitate communication between faculty and students. The interface “will automate the class web page production process while giving much greater control to the professors.” The University of Illinois at UrbanaChampaign offers faculty access to two additional systems, CyberProf and Mallard. These systems, which are partially funded and enabled by The Sloan Center for Asynchronous Learning Environments (SCALE),22 substantially increase the functionality of the web sites by assisting in the administering and grading of quizzes.

B. Reference TextBooks

These sites emulate the role and layout of the traditional textbook, but they include hypertext links and multi-media to increase functionality. The contents have the advantage of being searchable by keyword and offering audio, video, and interactive modules.23 A key defining characteristic is that they are intended to be a deep archival reference on a subject. An example may be found at The Virtual Hospital (University of Iowa).24 The hypermedia teaching facility in Mechanical Engineering at MIT also provides an on-line fluid mechanics textbook,25 and the MIT Experimental Study Group26 has produced a hypertext Cell Biology book, complete with practice questions and diagrams.

In some cases these books are outgrowths of a series of lectures or text modules for a course. An example is a book, authored by Kenneth Koehler, offering an overview of physics for the nonphysicist.27 Added value is provided by Mathematica software,28 which is used in the application to create interactive exercises. An extensive educational environment in which hypertext books are included is made available by Professor David Stockburger at Southwest Missouri State University.29 This site provides an on-line hypertext book with graphics as well as homework assignments and access to software packages.

C. Lectures

These sites are intended to provide the codified information traditionally delivered in a classroom lecture. We view these as an expert’s guided tour or story about a particular intellectual nugget. Each nugget covers roughly the same amount of material as a single classroom lecture and, in some cases, the web-lectures may be capable of supplanting classroom lectures (e.g., reference 30). Some companies are developing proprietary material of this type. Hewlett Packard uses secure sites on the WWW to make self-paced instructional courses available to its employees via desktop computers.31 There are a number of sites offering modular text-based lectures enhanced by using the hypertext capabilities of the WWW. They offer more than copies of class lecture notes but do not have the same breadth or depth as a hypertext book. These modules are typically linked to sites that also provide services for course administration and on-line quizzes or tutorials for self-testing. The Johns Hopkins School of Medicine has an extensive set of LectureLinks.32 Students access lecture notes, handouts, and sample exams provided in a searchable format. LectureLink’s objectives include allowing students to access relevant materials such as hyper-test books and databases from outside web sites, integrate linked materials into their learning process, and independently locate additional materials.33

Another example of a course with lecture material integrated into its website is at the University of Akron. This site assists students in General, Organic, and Biochemistry Courses. The site contains copies of all lecture slides plus animations, chapter summaries, and interactive practice exams.34

More sophisticated web-lecture sites might offer a mix of text integrated with graphics and video/audio clips. They may also provide access to interactive modeling software. The previously cited CALOS (Computer-Aided Learning for Operating Systems)35 project used the WWW to deliver a course on Operating Systems at the University of British Columbia. WebCT is a commercially available suite of development tools that arose from the CALOS project.36 This tool may be used to create complete on-line courses or supplemental materials to support a traditional course.

D. Laboratory Simulation and Experimentation

These web-based instructional materials allow students to interact with a variety of simulation programs or remote experiments. Applets can be created to provide simulations that allow users to modify/change variables, creating “virtual real lab experiences.” Physics faculty appear to have taken the initial lead in developing simulations for use via the WWW and many demonstrations are available.37,38

Project INTERACT in the UK aims to increase the use of simulations in engineering education by building a system for combining simulations with the hypermedia capabilities of the WWW, creating an interactive exploratory environment.39 Simulations include demonstrations illustrating Fast-Fourier Transforms and other signal processing techniques and ProMech, a package for building and testing mechanisms.40 Another tool from INTERACT is called Footsteps.41 Footsteps is a script for organizing hypertext/simulation pages and creating guided tours. Many distinct tours may be created using some or all of the relevant pages. The software also allows users to digress from and return to a tour.42

The Center for Integrated Electronics and Electronics Manufacturing (CIEEM) at Rensselaer Polytechnic Institute (RPI) is in the process of developing modular multimedia materials “that integrate design and manufacturing knowledge with real world experience.”43 The modules contain animations, case studies, and interactive tutorials.

Boroni et al.44 and Doulai45 discuss issues relevant to the use of animation tools in the delivery of interactive modules on the web. They cite platform independence and the WWWs ability to allow the student to follow his or her own learning path as strong motivational reasons for producing more engaging, interactive learning environments.

E. Recitation,Assignments and Grading

Web-based courseware targeted at recitations include interactive modules to assist students with difficult material or with complex topics. They may also provide reviews of basic materials needed as building blocks for a lecture. The sites are usually highly interactive and emphasize mechanisms for students to evaluate their knowledge of a subject with on-line quizzes. They can be used as a supplement to a traditional lecture/recitation course or as stand-alone modules.

The Center for the New Engineer at George Mason University is “learning how to serve self-paced, proficiency-based, hands-on learning” using hypertext modules.46 A learning module typically contains teaching materials, self-assessment tests, texts, pictures, links, and demonstrations.47 Another example is the Chemistry Hypermedia Project at Virginia Tech.48 It explores new ways to apply computer technology to help students learn chemistry. Tutorials developed include an Introduction to Analytical Chemistry.

Many people believe that using the WWW to deliver and score quizzes and examinations may offer significant time and resource savings.49 Top Class,50 formerly known as WEST,51 is a software application designed to manage the delivery of information using the WWW. One of its major functions is automatically scored quizzing. Quizzes may be delivered in either multiple choice or short-answer format and are scored automatically by the web server, providing students with immediate feedback. Quizzes can be configured to automatically assign extra study units or additional quizzes if a student scores below preset threshold values. Additionally, the instructor can be notified if a student is having difficulties. This software is being used by many institutions, including the State University of New York (SUNY) and the University of Kentucky. The CyberProf and Mallard projects at the University of Illinois at Urbana-Champaign also administer quizzes and assignments.

Web Course in a Box, a company developed by building upon work initiated at Virginia Commonwealth University, also provides the ability to create on-line practice quizzes and tests.52 Barsky and Shafarenko, at the University of Surrey (UK), have developed a system which allows the simultaneous examination of entire classes. This system has been used to assess a first-year discrete mathematics class.53

F. Virtual Educational Institutions

Many different WWW-based instructional materials are brought together in virtual educational institutions. Examples of three different approaches for this concept are: On-line Course Brokers; the Virtual University; and the Virtual University Consortium.

Several On-line Course Brokers are in existence. Some sites compile and broker distance education courses created by both academic institutions and commercial organizations. Features such as on-line bookstores, interactive tutoring, and searchable catalogs are typical. Examples of on-line course brokers include Globewide Network Academy,54 The Internet University,55 and UOL Publishing, Inc.56 The Virtual University Research Project at Simon Fraser University in British Columbia57 is developing a server-based software system that enables customized design, delivery, and enhancement of education and training courses delivered over the WWW. The project does not employ faculty or offer courses but rather provides a platform for the delivery of on-line information. The software may be used by either commercial or academic institutions for providing on-line courses.

Virtual universities employ their own faculty and staff to develop teaching materials.6 The Virtual On-line University, Inc. (VOU), a non-profit corporation with this structure, offers materials for professional development and life-long learning.58 Athena University is an electronic campus administered by VOU. It provides access to classes, a library, and a forum for social interactions and handling administrative matters. Magellan University is also a virtual university59 that is working towards credit courses for degrees in liberal arts and business.

The Virtual University Consortium model involves cooperative efforts between established universities and colleges to provide online access to courses. The motivation is to avoid duplication of materials and defray development costs while sharing expertise. This model is quite popular and is being used regionally, nationally, and internationally.60-62 Commercial entities are also beginning to form their own virtual institutions using this model.62 The Western Governors Virtual University63 was initiated by the Western Governors Association with a mandate to make instruction accessible at the leamer’s convenience via advanced technology.64,65 States involved in the initiative include Washington, Utah, Hawaii, Montana, Arizona, and Colorado. Students will receive certification for their studies and there are plans to confer both certification and degrees.

Additionally, although not a virtual university, NEEDS,66 the National Engineering Education Delivery system, of the Synthesis Coalition67 develops and distributes multimedia courseware to support engineering education in the United States. The NEEDS database of courseware is fairly extensive and the Synthesis Coalition has also created a WWW-based case study file.68

G. Collaborative Learning

Collaborative learning assumes that knowledge is formed as it is shared and reinforced through group discussion and interaction. Communication, listening skills, and participation are key factors in stimulating a good collaborative learning environment.69 This environment may be facilitated by the WWW as it has the benefit of being both time and location independent. Thus collaborative communication may occur at any time between people in many locations. There are a number of sites and initiatives which make use of the WWW in conjunction with classroom-based activities or with video/satellite conferencing.

The Cornell Theory Center (CTC)70 uses a collaborative virtual workshop to train staff, users, and potential users about the principles of parallel and distributed computing. The Virtual Workshop (VW) includes WWW lectures, exercises, communication with the CTC staff, interaction with other participants, and access to the Center’s computer.71

Rensselaer Polytechnic Institute (RPI) has developed studio courses which systematically incorporate technology into an interactive learning environment.72,73 Also at RPI, the Anderson Center for Innovation in Undergraduate Education is involved in a number of projects which focus on fostering a collaborative learning environment through the use of technology. The Virtual Studio Classroom, a project with Northeastern University’s distance learning branch, combines traditional distance-learning satellite broadcasts with a WWW-based interactive environment.74 Instructor interactions can be both synchronous and asynchronous. Students run Lotus’ Learning Space75 from the course-page to submit questions and comments while the real-time interaction is facilitated by LeanLinc-Inet, a product which “provides a virtual classroom experience.”76 The Center is involved with other projects such as the Simulink@distance and projects to evaluate the Effectiveness of Interactive Learning.

The Open University offers on-line interactive training for Professional Development in Educational Technology and assigns each participant a personal tutor and requires student involvement in interactive discussions.77,78

One of the more popular software programs used for facilitation of communication among virtual students is MOO. MOO is a multi-user dimension (MUD) Object-Oriented environment which allows users to construct and manipulate objects, moving freely from virtual room to room.79 MOO software is available through many educational software companies and can also be downloaded from FTP sites.80

IV. PRIOR EXPERIMENT ON WWW-BASED LECTURES

The previous section provides context for the study presented in this paper. We have focused on web-based materials which provide the information typically delivered in a traditional classroom. We assume effective web-based lectures are realizable and consider how time normally devoted to the classroom lectures might be used differently. In order to justify this assumption, the experiment uses a web-based lecture on visual prototyping that has already been demonstrated as an effective substitute for a classroom lecture.3

In this section the general lecture structure designed for the WWW and the specifics of the visual prototyping lecture are briefly outlined. We believe that using an appropriate structure for disseminating materials on the WWW is essential. In an article on the ADEPT Project at Stanford University,81 Harris and DiPaolo have argued that radical pedagogical change is necessary for true success using information technology and the WWW for asynchronous learning environments. Traditional educational structures do not necessarily project well into the effective delivery of distance education.82

The general web lecture structure, implemented for the visual prototyping lecture, is illustrated in figure 1. The structure is designed to accommodate different learning styles without creating confusion. The functionality of the browser’s user interface is divided into three distinct regions dedicated to topic navigation, material type, and material dissemination. To accommodate the different patterns of material access, the web-based lecture provides both sequential and parallel reference access. Although the lecture is a tour through a single topic, it is divided into many detailed subtopics. The visual prototyping web-lecture (equivalent to a 90-minute classroom lecture) is divided into 97 sub-topics. Within a subtopic the content is divided into four types of material: introduction/motivation; theory or rationale; procedural knowledge (how to apply theory to a problem, how to operate a machine); and examples (demonstrations of phenomena or results). By clicking on the buttons, students can choose the order they study the different types of materials. Finally, the material is disseminated in multiple presentation formats. This redundancy allows students to choose their preferred presentation format. In the visual prototyping lecture, students may either follow the material through text-and-images or through short video clips with both audio and subtitles.

The specific lecture implemented for the experiment details how to build visual prototypes. Visual prototypes are non-operational models that appear to be the real product. These models are used to evaluate visual, tactile, and other aesthetic user-interface qualities of product designs. The lecture is equivalent to a 90minute class and serves as a comprehensive introduction to the topic of visual prototyping. The web-lecture implementation indudes 92 QuickTime video clips (typically 5s to 15s in duration) and 142 slides. In total, the slides, video, and text comprise approximately 700 MB of data.

In a previous experiment, the performance of students receiving only the web-based visual prototyping lecture was compared to those receiving only classroom instruction. This study found that the average grade performance of the students receiving web instruction was higher than for those receiving traditional classroom instruction. An achieved significance level of 0.063 provided reasonably strong evidence to reject the hypothesis that the two groups performed equally.

V. COMPLETE LECTURE-STYLE SECOND COVERAGE OF MATERIALS COMPARED TO LIMITED-COVERAGE GUIDED EXPERIENTIAL ACTIVITY

The previous experiment indicates that a carefully designed web-based lecture can be an effective means of communicating the codified materials typically disseminated during classroom lectures. Thus the question arises: given a web-based lecture, can traditional classroom time be better utilized?

There are many possibilities but some distinct alternatives include: having no class at all; walking through the web-lecture inclass; providing a classroom lecture to support the web materials; or to support the web-based materials with a limited-coverage experiential activity guided by faculty. The Handbook of College Teaching83 enumerates many different strategies which could be adopted. The no-class scenario was studied in the previous web-based experiment and also by others. Additionally, presenting the web-based material in the classroom is, in our opinion, not appropriate. It is analogous to putting a lecture structured for the classroom on-line, counter to our premise that effective lectures should be designed/structured differently for different delivery technologies.

We chose to study what we believe are two prominent alternatives. We compared the effectiveness of using classroom time for a lecture-style, second complete coverage of materials (web+class lecture) and a guided experiential activity that covers only a small portion of the materials (web+limited experience). Both of the alternatives provide the students with the opportunity to evaluate and contemplate the information before having personal contact with faculty. This structure is a technique which is advocated as a sound teaching practice.12 Through this comparison, we hope to evaluate the relative effectiveness of two different in-person instruction approaches in combination with web-based materials and, to gain insight on how the approaches influence the study habits of students.

A. Web-Lecture Combined with Complete Topical Coverage in-Class (Web+Class Lecture)

In this model, students are strongly encouraged to prepare by covering the web-based prototyping lecture before the class. In the live classroom a complete coverage of the subject material is provided lecture-style by the instructor. Several demonstrations and examples of tools and materials are included in the class.

The key point of this model is that the live classroom covers the entire visual prototyping lecture content, thereby providing an inperson second coverage or reinforcement of the web-based lecture materials assigned as preparation for the class. Professors David Wallace84 and Woodie Flowers85 co-taught the lecture used in this experiment.

B. Web-Lecture Combined with Limited Topical Coverage Experiential Activity Guided by Faculty (Web+Limited Expeence)

In this model, students are strongly encouraged to prepare by covering the web-based prototyping lecture before the class. In the live classroom students work in the laboratory to each execute two simple projects that draw upon a subset of materials taught in the web-based lecture. During this period the faculty also complete the projects and assist the students as they work on the mini-projects. The time allocated for the exercises is equal to the amount of time normally devoted to the classroom lecture.

In this model, the in-person experience allows the students to apply a subset of the materials covered in the web-lecture under the guidance of the faculty in the prototyping shop environment. Unlike the web+class lecture group, the students do not receive in-person instruction for many of the subtopics in the visual prototyping lecture. Professors David Wallace and Woodie Flowers also cotaught the experiential activity.

C. Description of the comparative experiment

The goal of the experiment was twofold:

to obtain a quantitative comparison of the performance of students receiving the web-lecture with a lecture-style second coverage of materials with students receiving the weblecture with a limited-coverage guided experiential activity, and

to study how the different teaching approaches influence motivation for independent learning given that teaching techniques play a significant role in student motivation.10

1) Grade Performance Comparison: The students were randomly selected and divided into two groups of 15 students. Each student used a password that allowed us to track web-lecture use habits without knowing the identity of the student. A written paragraph with instructions about how to use the web-lecture and how to prepare for the in-person instruction was given to the students, allowing four days for advance web-based preparation. The key differences between the instructions for the two groups are summarized below.

Web+class lecture group:

You have been assigned to the lecture group that will meet at the regular lecture time of 1-2:30 in 3-270. As support for this lecture, you are strongly encouraged to review the complete web-lecture on this subject before class. This web-lecture was used last year and received very positive reviews from students.

Web+limited experience group:

You have been assigned to the lecture group that will meet between 6:30-8 PM in the Pappalardo lab. Do not attend the class during the regular time period. Before attending the lecture, it is essential that you cover the weblecture on this subject. We will assume that you have covered this material and will be working on tasks that require this knowledge. This web-lecture was used last year and received very positive reviews from students.

Following the class, both groups were given a two-week visual prototyping assignment for their wearable computer project. A jury of 11 product designers from industry then assessed the quality of the visual models on a scale from 0 to 10. The jury was unaware of either the experiment or that there were two different groups of students. This jury evaluation was then used to provide a performance comparison.

Additionally, data were gathered to help us check for group bias.

* On the first day of the course, students filled out a survey assessing their own skills in four areas relevant to the course. One of these areas was modeling ability.

The web-lecture was given halfway through the term so that we could calibrate the performance of both groups based upon their class history prior to the experimental lecture.

2) Comparing How the Teaching Methods Influence Motivation and Independent Study: Immediately after the in-person teaching experience (lecture-style and experiential) students filled out an anonymous questionnaire designed to provide feedback about: how the students valued the web and in-person instruction; and the completeness of their independent preparation using the web-lecture before the in-person instruction period. This qualitative preparation data were calibrated against quantitative data from log files. As each student used a unique username and password, we could track individual patterns of use without knowledge of the student’s true identity.

D. Results and Discussion

1) Grade Performance Comparison: The objective of this comparison was to test whether the web+limited experience group performed differently than the web+class lecture group in the prototyping assignment. However, even though the groups were randomly selected, we will first consider how the two groups performed on prior assignments (when both had the same classroom experiences). Figure 2 shows the grade performance of the two groups in assignments prior to the experiment. In these assignments, both groups were undifferentiated and taught in the same classroom. On a scale of 0-10, the difference in the average grade performance (web+limited experience – web+class lecture) was -0.16. Using these data, a probability mass function for the difference in average performance was constructed using 1000 bootstrap samples with replacement86 to assess whether the two groups have the same average performance. An achieved significance level of 0.218 indicates that we must accept the hypothesis that the two groups have the same average performance. These data support our assumption that there was no significant difference between the randomly chosen groups and that it was reasonable to expect the two test groups to have the same average performance on future assignments.

The data for the jury evaluation of the prototypes are presented in figure 3. On a scale of 0-10, the difference in the average performance (web+limited experience – web+class lecture) was 1.08. A probability mass function for the difference in average performance constructed from 1000 bootstrap samples with replacement using these data is shown in figure 4. This distribution is created under the assumption of the null hypothesis that the performance of the two groups was the same. The distribution for the difference in mean was then generated using the grade data and sample sizes corresponding to the two groups. The probability of observing a mean difference of 10.8% or greater if the two groups were the same is 0.001. An achieved significance level of 0.001 provides very strong evidence to reject the hypothesis that the average performance of the two groups was equal. One could argue that due to the small sample size (15 students in each group) that the results are questionable. Although large samples are desirable, the samples are sufficient for the bootstrap simulation technique and also for other statistics such as the student t statistic.

To test for the possibility that the web+limited experience group may have had more prior expertise in visual prototyping than the web+class lecture group, students filled out a survey on the first day of the course in which they rated their skills in four different areas, one of which was prototyping. These data are shown in figure 5.

The Hawthorne effect,87 related to performance improvement due to the subjects’ knowledge that they are being assessed, was not expected to be a factor in these data as the web-lecture use of both groups was being monitored and both groups were aware of and subject to the same evaluation process.

2) Effect on Independent Student Preparation and Motivation: The purpose of this comparison was to see if the teaching techniques had an impact upon student preparation and motivation. The response to the survey question asking students to assess their own level of preparation before class (using the web-lecture) is shown in figure 6a. In addition to this qualitative self assessment, the actual time each student prepared (using the web-lecture) was obtained from the web server log files and is plotted in figure 6b. The qualitative self-assessments from the survey and the quantitative log files show similar trends. The group that felt they were responsible for learning the material and would be applying it with faculty prepared more than the group which felt they were going to have a classroom lecture. The average preparation time for guided experiential activity group was 1.6 times higher than the classroom lecture group. Sixty percent of the Web+class lecture group prepared less than the least prepared student in the experiential group. Forty percent of the web+class lecture group effectively did no preparation whatsoever. The web preparation by the class lecture group matches our intuition about preparation when assigning textbook readings before class.

The data in figure 6 were gathered immediately after the in-person teaching experience. Figure 7 shows the total time spent independently studying the web-lecture two weeks later (after the visual prototyping project was completed). These data clearly illustrate that the web+class lecture group never caught up to the web+limited experience group on independent preparation. Most of the students who did not use the web-lecture before their class lecture never used the web-resource. It appears that the experiential activity was more effective motivation for independent study.

In light of these data there are a number of possible explanations for why the web+limited experience group outperformed the web+class lecture group. Although one might initially conjecture that the difference in grade performance was due to the in-person experiential activity, it might also be attributable to the difference in independent preparation using the web-lecture. Data gathered in this experiment do not allow us to speculate on the relative contributions of these factors.

3) Student Opinion About the Web-Lecture and In-Person Instruction: The survey conducted immediately after the in-person instruction (both the web+class lecture and the web+limited experience) asked questions about the relative value of the web-based and in-person instruction. In particular, one question asked:

If you could only have access to only the web or your class instruction, which would you rather have?

In response to this question, 37% of the web+class lecture group indicated that they would choose the web-only instruction while 18% of the web+limited experience group indicated they would choose the web only. Although these numbers seem different, an achieved significance level of 0.15 for the result (based upon 1000 bootstrap resamples) does not provide evidence to infer the two groups responded differently.

These data included the opinion of the web+class lecture students that did not prepare using the WWW lecture. If the students who did not independently prepare before class are removed from the samples (over half-which leaves a very small sample), 60% of the web+class lecture group indicated they would choose the weblecture only while 18% of the web+limited experience group indicated they would choose the web-only. In this case, an achieved significance level of 0.08 (based upon 1000 bootstrap resamples) and consideration of the small sample size provides `marginal to reasonable’ evidence to reject the hypothesis that the two groups had the same response.

In our opinion, this result suggests that students placed a high value on experiential activity and the opportunity to work with faculty on the exercises. In the lecture-style classroom setting students did not have the same degree of interactivity with the instructors and the web-only instruction option seemed to be more attractive.

VI. CONCLUSIONS

There is substantial activity throughout the world directed toward the development of web-based instructional materials. Initiatives range in their focus from course administration to textbooks, examinations, and collaborative teaching environments. The focus of this paper was the development and use of web-based materials targeting the information traditionally provided in lecture-style classrooms.

There is a growing body of evidence to suggest that web-based lectures can be as effective as classroom lectures in conveying well codified information. Therefore, this work has considered how classroom time might be used differently to better exploit the availability of web-based lectures. Two teaching models were evaluated. One group of students independently prepared for class using webbased materials and then received a lecture-style second coverage of the topic in class. The other group of students independently prepared for class using the same web-based materials and then worked with the faculty, applying a small portion of the subject matter on illustrative examples.

The average grade performance of the group receiving webbased instruction plus limited-coverage experiential activity with faculty was 10.8% higher than for those who received web-based instruction plus a complete classroom lecture from faculty. An achieved significance level of 0.001 provides very strong evidence to reject the hypothesis that the two groups performed equally. There was also evidence indicating the approaches provided different motivational incentives for independent study and preparation. The web+limited experience group used the web-lecture to prepare for class far more than the web+class lecture group. The least prepared student in the experiential activity group studied more than 60 percent of the students in the lecture-style classroom group. Further, over the course of the two-week project, the web+class lecture group never caught up to the web+limited experience group on independent study using the web materials. In a survey, students were asked to compare web-based instruction to their in-person instruction. Our interpretation of these data is that students placed a high-value on the opportunity to gain experience while working with the faculty.

We are motivated to study the use of web-based lecture materials to see if education quality can be improved. The medium appears to offer the flexibility needed to allow students to play a very active role in their education, which may translate into more effective learning-to-learn. It appears that web-based lecture materials may allow classroom time to be spent in mentoring activities, potentially increasing the value of the on-campus experience.

However, there are a number of barriers. The cost of preparing and producing large numbers of high-quality teaching nuggets for use in single classes at single institutions is prohibitive (the resources needed to develop the web-lecture used in this paper were tracked in reference 3). Nuggets will need to be published to a wide audience and designed to facilitate local customization of materials. We envision the creation of quality teaching nugget libraries. Search facilities might allow for the automated design of semi-custom courses using combinations of teaching nuggets, thus tailoring curricula to different schools or to specific learning communities.

ACKNOWLEDGMENTS

We would like to gratefully acknowledge the financial support provided by the Engineering Coalition of Schools for Excellence in Education and Leadership (ECSEL), the MIT Department of Mechanical Engineering, and the MIT Libraries. Additionally, we would like to thank Jun-beom Kim for his assistance in administering the experiment and Carol Robinson for editorial assistance.

REFERENCES

1. M. Goldberg, “CALOS: First Results from an Experiment in Computer-Aided Learning for Operating Systems,” SIGCSE Bulletin, vol. 29, no. l, pp. 48-52,1997.

2. T. Russell (Teleeducation NB), The “No Significant Difference Phenomenon”, URL http://teleeducation.nb.ca/phenom/phenom1.html 3. D. Wallace and P. Mutooni, “A Comparative Evaluation of WorldWide-Web-based and Classroom Teaching,” Journal of Engineering Education, vol. 85, no. 1997.

4. M. L. Dertouzos, What Will Be: Ho w the World of Information Will Change Our Lives. London: Piatkus,1997. 5. J. O’Donnell, “Teaching With Technology,” CSS Journal-Computers in the Social Sciences, vol. 5, no.1,1997. 6. V. Dwyer, “A Crash Course in Reality 101: Generation Y asks Universities to Deliver for their Future,” McLean , vol.109, no. 48, pp. 50,1996. 7. H. Li, “Using Information Technology to Enhance Engineering Education,” proceedings of The 1996 IEEE International Conference on Multi Media Engineering Education, Melbourne, Australia, pp. 325-330,1996.

8. R. Bruning, “The College Classroom from the Perspective of Cognitive Psychology,” in Handbook of College Teaching: Theory and Applications, The Greenwood Educators’ Reference Collection, K. Prichard and R. M. Sawyer, Eds. Westport, CT: Greenwood Press,1994, pp. 3-22. 9. W. Ekeler, “The Lecture Method,” in Handbook of College Teaching: Theory and Applications, The Greenwood Educators’ Reference Collection, K. Prichard and R. M. Sawyer, Eds. Westport, CT: Greenwood Press,1994, pp. 85-98.

10. P. Pintrich, “Student Motivation in the College Classroom,” in Handbook of College Teaching: Theory and Applications, The Green wood Educators’ Reference Collection, K. Prichard and R. M. Sawyer, Eds. Westport, CT: Greenwood Press,1994, pp. 23-44.

11. E. Bilotta, M. Fiorito, D. Iovane, and P. Pantano, “An Educational Environment Using WWW,” proceedings of The Third International World Wide Web Conference, http://www.igd.fhg.de/www/www95/proceedings/papers/97/EduEnv.html, pp.,1995.

12. W. Leonard, “The Laboratory Classroom,” in Handbook of College Teaching: Theory and Applications, The Greenwood Educators’ Reference Collection, K. Prichard and R. M. Sawyer, Eds. Westport, CT: Greenwood Press,1994, pp.155-170.

13. February 21, Chronicle of Higher Education, 1997. 14. M. W. Goldberg, “Using a Web-Based Course Authoring Tool to Develop Sophisticated Web-Based Courses,” in Web-based Instruction, B. H. Khan, Ed.: Educational Technologies Press,1997.

15. WLH (University of Texas – Austin), World Lecture Hall, URL http://www.utexas.edu/world/lecture/

16. 2.009 (Massachusetts Institute of Technology), Product Development Process, URL http://me.mit.eud/2.73

17. J. Lockledge, D. Geister, L. Mitchell, and P. Go, “Internet Based Learning Environment,” proceedings of The 26th Annual Conference on Frontiers in Education (FIE 96), Salt Lake City, UT, vol. 1, pp. 293-296, 1996.

18. R. Owston, “The World Wide Web: A Technology to Enhance Teaching and Learning?,” Educational Researcher, vol. 26, no. 2, pp. 27-33, 1997.

19. NDSU (North Dakota State University), Instructional Web Project at North Dakota State University, URL http://www.ndsu.edu/istruct/ www/home.html

20. BML (Big Mouth Lion), Big Mouth Lion Discussion Manager,

URL http://www.infohiway.com/bigmouthlion/index.html 21. VCI (University of Illinois), The Virtual Classroom Interface, URL http://wsi-www1.cso.uiuc.edu/courses/vci-help/about.html

22. SCALE (University of Illinois Urbana-Champaign), Sloan Center for Asynchronous Learning Environments, URL http://w3.scale.uiuc.edu/scale/

23. G. Pratt, “The Virtual Hospital,” JAMA The Journal of the American Medical Association, vol. 276, no.13, pp.1091-1092,1996. 24. UIowa (University of Iowa), Introduction to the Virtual Hospital, URL http://vh.radiology.uiowa.edu/Welcome/AboutVH.html

25. MIT (Massachusetts Institute of Technology), Mechanical Engineering Hypermedia Project, URL http://hyperweb.mit.edu:8080/ curhyp.html

26. ESGBIO (MIT Experimental Study Group), Hypertext Cell Biology Book, URL http://esg-www.mit.edu:8001/esgbio/7001main.html 27. R. Koehler (Raymond Walters College – University of Cincinnati), College Physics for Students of Biology and Chemistry, URL http://www.rwc.uc.edu/koehler/biophys/text.html

28. Wolfram (Wolfram Research), Mathematica, URL http://www.wolfram.com/look/about.html

29. D. Stockburger (Southwest Missouri State University), Introductory Statistics: Concepts, Models and Applications, URL http://www.psychstat.smsu.edu/

30. MIT (Massachusetts Institute of Technology), Visual Modeling, URL http://me.mit.edu/lectures/productModelling/weblecture 31. S. Caudron, “Your Learning technology primer,” Personnel Journal, vol. 75, no. 6, pp.120,1996.

32. JHMI-Infonet (Johns Hopkins University), Johns Hopkins School

of Medicine Curriculum LectureLinks, URL http://infonet.welch.jhu. edu/~omie/LectureLinks/AboutLL.html

33. J. Freedman and H. Lehmann, “Providing Medical Education Resources over the World Wide Web,”proceedings of WebNet 96, San Fransisco, CA, pp. 542-543,1996.

34. GENOBC (University of Akron), General, Organic and Biochemistry Courses, URL http://ull.chemistry.uakrin.edu/genobc/ 35. M. Goldberg, “CALOS: An Experiment with Computer-aided Learning for Operating Systems,” SIGCSE Bulletin, vol. 28, no. 1, pp. 175179,1996.

36. WebCt (WebCT), World Wide Web Course Tools – WEBCT, URL http://homebrew1.cs.ubc.ca/webct/ 37. Demoroom (Norht Carolina State University), Physics Demo Room, URL http://www2.ncsu.edu/ncsu/pams/physics/EduResources/ demoroom/demoroom.html

38. G. Novak and E. Patterson, “World Wide Web technology as a New Teaching and Learning Environment,” International Journal of Modern Physics C., vol. 8, no. 1, pp. 19-39,1997. 39. Interact (Project Interact), Welcome to Project Interact, URL http://www-interact.eng.cam.ac.uk/

40. Interact (Project Interact), Interact: Simulations for Education, URL http://gorgon.eng.cam.ac.uk/CAL.95/

41. D. Nicol, C. Smeaton, and A. Slater, “Footsteps: Trail-blazing the Web,” Computer Networks & ISDN Systems, vol. 27, no. 1995, pp. 880885,1995.

42. Footsteps (Project Interact), Footsteps: Interact Simulations for Educatin, URL http://www-interact.eng.cam.ac.uk/footsteps/footsteps.html 43. ILM (Rensselaer Polytechnic Institute), Interactive Learning Modules for Manufacturing Engineering Education and Training, URL http://www.deem.rpi.edu/ILM.html

44. C. Boroni, F. Goosey, M. Grinder, R. Ross, and P. Wissenbach, “WebLab! A universal and Interactive Teaching, Learning, and Laboratory Environment for the World Wide Web,” SIGCSE Bulletin, vol. 29, no. 1, pp.199-203,1997.

45. P. Doulai, “Engineering and Science Laboratory Courseware Delivery using World Wide Web Technology,” proceedings of The 1996 IEEE International Conference on Multi Media Engineering Education, Melbourne, Australia, vol., pp. 339-344,1996.

46. CNE (George Mason University), Welcome to the Center for the New Engineer, URL http://www.cne.gmu.edu/ 47. CNE (Center for the New Engineer-George Mason University), CNE Tutorial Modules Central, URL http://www.cne.gmu.edu/modules. html

48. B. M. Tissue (Brian Tissue), The Chemistry Hypermedia Project at Virginia Tech, URL http://www.chem.vt.edu/chem-ed/CHP/ overview.html

49. R Barnett and C. Burton, “Teaching and Assessment via the Internet – Experiences and Directions,” proceedings of The 1996 IEEE International Conference on Multi Media Engineering Education, Melbourne, Australia, pp. 355-360,1996.

50. wbtsystems (WBT Systems), Top Class Software, URL http://www.wbtsystems.com/

51. H. McLoughlin, “WEST: An Internet Based Education Delivery and Support Environment,” Computer Networks &ISDN Systems, vol. 28, no.14, pp.1887-1890,1996.

52. madDuck (madDuck Technologies), Mad Duck Technologies, URL http://www.madduck.com/index.html 53. D. Barsky and A. Shafarenko, “WWW and Java-based Distributed Examination System for Distance Learning Applications,” proceed

ings of The Second AIZU International Symposium on Parallel Algorithms/Architecture Synthesis, Fukushima, Japan, vol. 1, pp. 356-363, 1997.

54. GNA (Globewide Network Academy), The Globewide Network Academy, URL http://www.gnacademy.org:8001/uu-gna/index.html 55. CASO (Cape Software), The Internet University, URL http://www.caso.com/iu.html

56. UOL (UOL Publishing, Inc.), UOL Publishing Inc., URL http://www.uol.com/

57. VirtualUniversity (Simon Fraser University), The Virtual University Research Project, URL http://virtual-u.cs.sfu.ca/vuweb/

58. Athena (Virtual On-line University), Athena University, URL http://www.athena.edu/vou.html

59. Magellan (Magellan University), Magellan University, URL http://www.magellan.edu/

60. ICN (Indiana College Network), Indiana College Network, URL http://www.icn.org/

61. DVP (University of Sheffield), Dearn Valley Higher Education Project, URL http://www2.shef.ac.uk/projects/dvp/

62. L. Hodges, “Goodbye, alma mater?,” in The Independent, 1996, pp. B2.

63. VU (Western Govenors Association), Western Govenors Virtual University, URL http://wga-internet.west.gov.org/smart/vu/vu.html

64. R. M. Epper, “Virtually an Institution,” Trusteeship, Special Issue, pp.30-33,1996.

65. S. Johnstone and B. Krauth, “Balancing Equity and Access: Some Principles of Good Pratice for the Virtual University,” Change, voL 28, no. 2, pp.38-41,1996.

66. NEEDS (NEEDS), National Engineering Education Delivery System of the Synthesis Coalition, URL http://www.needs.org/ 67. Synthesis (Synthesis Coalition), Synthesis Coalition, URL http://www.synthesis.org/

68. MMCS (NEEDS), Synthesis Case Study Library (MultiMedia Case Studies), URL http://www1.needs.org/develop/

69. R Matthews (University of Maryland), Building Bridges Between Cooperative and Collaborative learning, URL http://www2.emc. maricopa.ed /innovation/CCl/building.html

70. CTC (Cornell University), Cornell Theory Center, URL http://www.tc.cornell.edu:80/ctc.html 71. VW (CTC, Cornell UNiversity), Visual Workshops, URL http://www.tc.cornell.edu/Edu/VW/

72. ClUE (Rensselaer Polytechnic Institute), The Rensselaer Studio Courses, URL http://cue.rpi,edu/studio/studio.html 73. C. Holmes and J. Warden, “CIStudio: A Worldwide Web-based, interactive chemical information course,” Journal of Chemical Education, vol. 73, no. 4, pp. 325-331,1996.

74. VSC (Rensselaer Polytechnic Institute), Virtual Studio Classroom, URL http://www.ciue.rpi.edu/projects.html 75. LearningSpace (Lotus), Lotus Learning Space, URL http://198.114.68.60/

76. ILINC Interative Learning International Corporation, URL http://wwsw.ilinc.com/

77. PDET (Open University), Professional Development in Educational Technology, URL http://www-iet.open.ac.ek/PDET/online.html

78. M. Milone, “Virtual Learning Gets Real,” Teachnology &Learning, vol.17, no. 5, pp. 51,1997.

79. K. Fox Moo-Cows FAQ URL http://www.ccs.neu.edu/home/ fox/moo

80. MOO (Xerox Corp.), FTP Site For MOO Software, URL

ftp://parcftp.xerox.com/pub/MOO

81. ADEPT (Stanford University), Asynchronous Distance Education Project (ADEPT)., URL http://adept.stanford.edu/ 82. D. Harris and A. DiPaolo, “Advancing Asynchronous Distance Education Using High-Speed Networks,” IEEE Transactions on Education, vol. 39, no. 3, pp. 444-449,1996. 83. K. Pritchard and R. M. Sawyer, “Handbook of College Teaching: Theory and Applications,” in The Greenwood Educators’ Reference Collection. Westwood, CT: Greenwood Press, 1994. 84. MIT (MIT Department of Mechanical Engineering), David Wallace, URL http://me.mit.edu/people/faculty/DavidWallace.html 85. MIT (MIT Department of Mechanical Engineering), Woodie Flowers, URL http://me.mit.edu/people/faculty/WoodieFlowers.html

86. B. Effron and R. J. Tibshirani, An Introduction to the Bootstrap. New York, NY: Chapman and Hall, 1993.

87. D. I. Cook, The impact ofthe Hawthorn Effect in Experiment Design in Education Research. Columbus, OH: Ohio State University, 1967.

DAVID R. WALLACE

Department of Mechanical Engineering Massachusetts Institute of Technology

SUZANNE T. WEINER

Education Coordinator, Center for Innovation in Product Development Massachusetts Institute of Technology

Copyright American Society for Engineering Education Jul 1998

Provided by ProQuest Information and Learning Company. All rights Reserved