A Model for Studying Multimedia Effectiveness in Education*

Animating to Build Higher Cognitive Understanding: A Model for Studying Multimedia Effectiveness in Education*

Ellis, Timothy


This paper presents a model for designing experiments to test the effectiveness of multimedia and other technological enhancements in education. It is based upon a discussion of an experimental study that examined the impact of one specific multimedia enhancement-animation-on a very specific aspect of learning-the ability to apply knowledge, for a well-defined population-adult, continuing education students, in a highly specified knowledge domain-Boolean algebra.

The paucity of substantial evidence supporting the value of multimedia as an educational enhancement can at least partially be attributed to definitional problems within experimental designs. What specifically does “learning,” and, for that matter, “multimedia”, mean? If these terms are not explicitly defined and intimately incorporated into the experimental controls, there is little hope for creditable results. This study indicates that careful attention to defining these two terms can produce meaningful results.


Although multimedia is being widely hailed as the next great innovation in education, adequate support for its effectiveness in enhancing the educational process is scant. There is considerable anecdotal evidence for the benefits attendant with inclusion of multimedia features in the educational process. Numerous case studies have reported improvements in intangibles such as student attitude, student and teacher enthusiasm, and improved teamwork in test cases in which course materials were augmented by multimedia enhancements [1-3]. There is, however, little quantifiable, measurable data supporting this position. Reviews of numerous studies that purported to establish the value of multimedia in the learning environment have indicated the presence of numerous Type I errors [4]. In more tightly controlled studies evidence is not compelling. For example, Pruisner [5] tested the impact of color-cueing on learning and discovered no significant correlations. In fact, research into the role of any instructional methodology in promoting learning is very disappointing. Numerous meta-studies of educational research [6-9] have indicated that instructional delivery methodology and student achievement are unrelated: “The best current evidence is that media are mere vehicles that deliver instruction but do not influence achievement any more than the truck that delivers our groceries causes changes in nutrition …” [9, p. 445].

If multimedia enhancements or any other improvements to instructional delivery have no real affect on learning, we as professional educators are confronted with some rather uncomfortable questions. Incorporation of multimedia into the educational process entails the commitment of significant financial, personnel and time resources [10-13]. Are these resources being wasted? Perhaps more importantly, if the method of instructional delivery is indeed irrelevant, how could professional educators ever hope for growth or improvement? Every teacher has undoubtedly encountered either individual students or even whole classes she or he could not ‘reach’. If the reason for these lost learning opportunities is nothing more than a mismatch of personalities, the implications for education as a profession are unfortunate. It is incumbent on educators to conduct well-controlled experimental studies to accurately determine if and when multimedia or other instructional delivery enhancements can indeed improve learning.


The potential of multimedia in education does have a theoretical foundation. Bagui [14] and Daniels [15] summarized the theory of multi-channel communication in support of the potential for multimedia. According to this theory, humans have several channels by which data is communicated. If information is presented via two or more of these channels, there will be additional reinforcement, resulting in greater retention and improved learning.

Further support for the potential benefits of multimedia is offered by research in learning styles. Currie [16] reported that posttraining reports from adult learners in a wide variety of training sessions indicated that learning was most likely to take place when the training matched the students’ learning style. McCarthy [17] further explored learning styles and identified four distinct approaches to learning: the feeler, the analyzer, the doer, and the creator. A multimedia approach presents the potential to address these different approaches to learning, as was suggested by the research of Riding and Grimley [18].

Anecdotal evidence and theory lend support to the potential value of multimedia in education. There are, however, counter-theories and cautionary notes being sounded. Cognitive load theory [19-21] and the concepts of redundancy effect and split attention effect suggest that, by communicating to a student through multiple channels instruction is in fact made less efficient. Solomon [12] traced the fates of similar ‘boons to education’-television and the microcomputer-which never fulfilled their initial promise. Since a number of studies-Luna and McKenzie [22] and Wegner and Payne [23], for example-revealed no differences in recall or retention in students using a multimedia-enhanced program versus those using an exclusively text-based system, caution in rushing to adoption of the technology is indeed justified.


Concrete evidence of the value of multimedia in learning is difficult to establish. Four noteworthy obstacles impede this validation.

It is difficult to define and measure learning. Some, for example, distinguish learning on the basis of level of cognitive engagement [24-26], while others focus on the type of activity in which the learner engages [27, 28], and still others on the learner’s preferred approach to the task [11, 18]. Bloom [25], in perhaps the seminal study on human learning, identified six levels of learning, starting with knowledge and progressing through understanding, application, analysis, and synthesis up to evaluation. Any study of the effectiveness of multimedia as a tool to enhance learning must specify learning in a manner that is consistent with accepted learning theory.

Secondly, “learning” is not uniform across student populations. Adult learners, for example, have different needs than children and young adult students. Zemke and Zemke [29] and Boucouvalas and Krupp [30] studied the unique needs of adult students. Among the conclusions presented in these studies was that mature learners need flexibility in time, relevance of material, and control over the educational process. Any study of the effectiveness of multimedia as a tool to enhance learning must specify the student population under consideration.

Learning is, likewise, not uniform across subject matter. Teaching techniques and student activities vary widely based upon the material being learned. A humanities course, for example, is strikingly different in content, assignments, and cognitive processes than a physics course. Technology-intensive courses present unique challenges and require appropriate pedagogical approaches [31]. Any study of the effectiveness of multimedia as a tool to enhance learning must specify the subject matter being learned.

Finally, the term “multimedia” is quite expansive. Tannenbaum [32] offers an excellent working definition of the term: “Multimedia is defined as an interactive computer-mediated presentation that includes at least two of the following elements: text, sound, still graphic images, motion graphics, and animation (p. 4).” This type of definition leads one to conclude that a product that includes text and a picture is the same as a product that includes narration, videos, and animations. A great deal of research in the field, however, strongly suggests the polysemous nature of the term. Taxonomies have been developed to categorize multimedia in terms of media type [33], level of interactivity [34], and media usage [35]. Any study of the effectiveness of multimedia as a tool to enhance learning must specify in the definition of multimedia precisely which media elements are being studied, at what level of interactivity, and toward what end.


A. Purpose of the Study

The purpose of this study was to develop a model for testing the effectiveness of multimedia in promoting learning in order to allow instructors to make informed decisions on how to best manage educational resources and promote an optimal learning environment. The essence of the model tested was to very rigorously and narrowly define the four critical elements: the aspect of learning under consideration, the student population, the subject matter, and the combination of media elements being tested.

B. Research Questions

Two research questions were addressed in this study:

1. Will a study in the effectiveness of multimedia as an enhancement to learning produce meaningful results when learning, the student, the subject, and multimedia are narrowly and specifically defined?

2. Will a tutorial enhanced with animations produce greater learning of Boolean algebra at the application of knowledge level than a text-only tutorial when offered to adult students in a junior college setting?

C. Variables

To address the two research questions underlying the study, one independent and two dependent variables were indicated. The independent variable was method of instruction and it had two states: paper-based and animation-enhanced. The dependent variables were: 1) acquisition of items of information, and 2) ability to apply newly acquired information to solve a problem.

D. Limitations and Assumptions

The following limitations to the results of this investigation are noted:

1. The students participating in the study were adult learners between the ages of 25 and 50. Generalizations to other age groups maybe limited, even in similar conditions.

2. The subjects of the study were all enrolled in college-level, credit courses in the Division of Continuing Education of a private, two-year institution. The participating college has an open enrollment policy, permitting anyone with the equivalent of a high school diploma to take credit classes. Generalizations to other educational and training environments may be limited, even in similar conditions.

3. Only one topic, a one-hour introduction to using the AltaVista search engine to locate information on the World Wide Web, was used to measure the effectiveness of the various educational approaches. Generalizations to other subject areas or course duration may be limited, even in similar conditions.

The following assumptions underlying this study are noted.

1. The students involved in this study were motivated to learn and willing to participate in the tutorial.

2. The two educational products compared in this study, the written tutorial and the computer-based multimedia product, were of equal quality.

E. Sample Population

The participants of this study were selected from students attending one of three classes-Introduction to Computers, Introduction to Keyboarding, and Medical Office Procedures-in the Division of Continuing Education campus of a private, two-year college. These classes were selected for the study because they were all introductory in nature and all required the use of the computer. The students participating in the study, therefore, were at least minimally familiar with using the computer but were not advanced users.

A total of 38 men and women participated in this study. The participants were all adult students ranging in age from 25 to 50. All subjects were employed or recently laid-off and were taking lower-level undergraduate courses to facilitate either a career change or promotion. The subjects were randomly assigned to one of two groups: a control group and an experimental group. The groups were separated, the control group going to a classroom and the experimental group to the computer lab.

F. Tool Development

This study necessitated the development and validation of three instruments: a pretest/post-test; a written tutorial on using Boolean algebra to refine a search on the World Wide Web; and a computer-based, animation-enhanced tutorial on using Boolean algebra to refine a search on the World Wide Web. The same general developmental process was followed for all three instruments: a product was prepared by the researcher, then submitted to a panel of three experienced computer science instructors at the participating college for comment and review. The expert panel reviewed the proposed product, suggested changes, and continued the review-modify process until consensus was reached that the product met stated goals. The final products reflected the consensus of this expert panel that they did in fact accomplish their stated goals. The specifics regarding the development of the three tools necessary for the study follow.

1. Written Tutorial. The first product developed was an eightpage written tutorial of Boolean algebra as a tool to enhance searching on the World Wide Web. The tutorial was reviewed and modified by the expert panel described above to ensure attainment of the following goals: (1) Provide a factual overview of the World Wide Web in general and search engines in specific; (2) Provide descriptions and samples of how to use Boolean expressions to effectively narrow the scope of a search and focus on only the desired information. Figure 1 contains an example of a section from the paper-based tutorial.

2. Animation-Enhanced Tutorial. Based upon the written tutorial described above, a computer-based tutorial enhanced with animations was developed. This tutorial was developed using Microsoft Visual Basic Professional. It was likewise reviewed and modified by the expert panel to ensure attainment of the following goals: (1) The same two educational goals stated for the written tutorial; (2) Use the multimedia enhancement, animation, to provide feedback to student activities during the sections of the tutorial that concerned effectively using Boolean expressions to narrow a search; (3) Be of equal quality to the written tutorial in terms of appearence and content. Figures 2, 3, and 4 illustrate the manner in which the interactive, animated tutorial guided the students through the same information displayed in Figure 1.

3. Pretest/Posttest: The final instrument developed by the researcher and validated by the expert panel was a pretest/posttest. This test consisted of twenty multiple-choice questions organized as follows: ten questions of fact that measured the participant’s acquisition of items of information, and ten questions of application that measured the participant’s ability to analyze a problem and apply information acquired. Figures 5 and 6 display samples of each type of question from the pretest/posttest.

G. Data Collection and Analysis

The control and experimental groups were tested in separate rooms; the control group was assigned to a classroom, and the experimental group to the computer laboratory. For both groups, identical written, multiple-choice pretests were administered, and then collected. The control group was then given the written tutorial while the experimental group was allowed to complete the animation-enhanced tutorial on the computer. After completing the tutorials, each group was given the written, multiple-choice posttest, which they completed without access to the tutorial. Although the posttest contained the same questions as the pretest, the questions were arranged in a different order to reduce the potential for experience threat.

The four sets of tests were graded and two scores were assigned to each test: number of questions of fact answered correctly and number of questions requiring the application of information answered correctly. Since the study analyzed the impact of a single independent variable-method of instruction-on two dependent variables-acquisition of items of information and ability to apply newly acquired information to solve a problem-a multi-variate analysis of variance (MANOVA) was selected as the statistical test.


Table 1 presents the descriptive statistics-means and standard deviations-for the tests used to measure performance on the dependent variables. The multivariatc tests of the MANOVA produced a statistically significant Wilks’ Lambda (f = 3.281, p = 0.023) for the main effect: comparison between the control and experimental groups. The differences in performance on the tests are indeed worthy of further inspection.

Table 2 presents the results of the tests between-subjects effects of the MANOVA. Only one of the tests between subjects in the control and experimental groups-the performance on the posttest measuring the subjects’ ability to apply newly acquired information to solve a problem-recorded a statistically significant difference (f = 6.731, p = .014). The between subjects effects detailed in Table 2 would indicate that the control and experimental groups were indeed drawn from the same population-as indicated by the absence of a statistically significant difference in pretest scores. Furthermore, since there was also not a statistically significant difference in the posttests measuring acquisition of facts, these results would indicate that the difference in performance noted between the control and experimental groups can be attributed to differences in learning at the application of information level.


The results of this study appear to answer both research questions affirmatively. Research into the effectiveness of multimedia as an enhancement to learning can produce meaningful results when “learning”, “student”, “subject”, and “multimedia” are narrowly defined. Animation does appear to foster a greater degree of learning at the application of knowledge level than a text-only tutorial for adult students learning Boolean algebra in a junior college setting.

These results are consistent with the constructs from some of the more current learning theories. Constructivist [27, 28] theory postulates that highly interactive learning environments in which the student has an enhanced degree of control should result in greater, deeper learning. Learning styles theory [17, 18] suggests that educational media that utilizes multiple methods of engaging the student in the learning activity has a greater potential for meeting the individual needs of each student.

The impact of this study could potentially extend well beyond the rather narrow research questions addressed. Although the finding that a tutorial enhanced with animations produced a significant improvement in adult students’ ability to apply knowledge while the text-based counter-part did not offers insight into the effective use of multimedia in a learning environment, the true impact of this study might be to establish parameters under which comparative research in education can be conducted. There is something of an anti-experimental bias in much of educational research, with increasing emphasis being placed on qualitative studies. At least a portion ofthat bias might be attributable to the rather equivocal results of much of educational experimentation that is directly related to the lack of rigorous design that plagues much of the research. This study opens the dialog on how to structure effective experiments in education.

Research in multimedia as an enhancement to the learning environment is made quite difficult because of the multiple meanings associated with both learning and multimedia, coupled with the vast differences among student populations and subject matter content. As this study indicated, research that very narrowly defines learning, student, subject and multimedia is, however, quite capable of producing meaningful results.

Areas for future research might well focus on adding greater granularity to the definitions of multimedia, student, subject, and learning. Some combinations of these four elements that might produce very interesting studies would be:

1. Altering the media element being tested from interactive animation to interactive live video.

2. Adding voice-over narration to the interactive animation to test if engaging an additional communication channel produces a net gain or net loss in learning.

3. More finely defining student by categorizing on the basis of learning style as indicated by Kolb’s Learning Style Inventory [36].

The combinations of media element or elements, learning dimension, student population, and subject matter are virtually limitless and offer the potential for quite interesting research, as long as they are scrupulously specified within the experimental design.

* This article is an expansion of a manuscript that was among those awarded “best paper” in the 2001 Frontiers in Education Conference.


[1] Perry, T. and L.A. Perry, “University Students’ Attitudes Towards Multimedia Presentations.” British Journal of Educational Technology, 29,4, 1998, 375-377.

[2] Peterson, N.K. and B.J. Orde, “Implementing Multimedia In The Middle School Curriculum: Pros, Cons And Lessons Learned.” T.H.E. Journal, February 1995, 70-75.

[3] Wise, M. and P.M. Groom, “The Effects Of Enriching Classroom Learning With The Systematic Employment Of Multimedia.” Education, 117, 1, 1996, 61-69.

[4] Lookatch, R.P. “Multimedia Improved Learning-Apples, Oranges, and the Type I Error.” Contemporary Education, 68, 1997, 110-113.

[5] Pruisner, P. “The Role Of Color In Remembering Graphically Presented Information.” 1995, (ERIC Document Reproduction Service No. ED380073).

[6] Berge, Z.L. and S. Mrozowski, “Review of Research in Distance Education, 1990 to 1999.” The American Journal of Distance Education, 15, 3, 2001, 5-18.

[7] Wang, M.C., G.D. Haertel, and H.J. Walberg, “What Helps Students Learn? Spotlight on Student Success.” Laboratory for Student Success: Philadelphia, PA.:(EPIC Document Reproduction Service No. ED 461 694), 1997.

[8] Russell, T.L. “Television’s Indelible Impact on Distance Education: What We Should Have Learned from Comparative Research.” Research in Distance Education, October, 1992, 2-4.

[9] Clark, R.E. “Reconsidering Research on Learning from Media.” Review of Educational Research, 53, 4, 1983, 445-449.

[10] Mackensie, D.S. and D.G. Jansen, “Impact of Multimedia Computer-Based Instruction on Student Comprehension of Drafting Principles.” Journal of Industrial Teacher Education, 35, 4, 1998, 61-82.

[11] Hall, B. “Easing Into Multimedia.” Training and Development, 50, 4, 1996, 61-62.

[12] Solomon, M.B. “What’s wrong with multimedia in higher education?” T.H.E. Journal, 21, 7, 1994, 81-83.

[13] Ellis, T.J. and M.S. Cohen, “Integrating Multimedia into a Distance Learning Environment: Is the Game Worth the Candle?” British Journal of Educational Technology, 32(4), 2001, 497-500.

[14] Bagui, S. “Reasons For Increased Learning Using Multimedia.” Journal of Educational Multimedia and Hypermedia, 7, 1, 1998, 3-18.

[15] Daniels, L. “Audio Vision: Audio-Visual Interaction In Desktop Multimedia.” In D.G. Beauchamp, R.A. Braden, and R.E. Griffin (Eds.). Imagery And Visual Literacy: Selected Readings From The Annual Conference Of The International Visual Literacy Association (26th, Tempe, Arizona, October 12-16, 1994). (ERIC Document No. ED380056, pp. 57-63).

[16] Currie, G. “Learning Theory and the Design of Training in a Health Authority. “Health Manpower Management, 21, 2, 1995, 13-19.

[17] McCarthy, B. “A Tale Of Four Learners: 4MAT’s Learning Styles.” Educational Leadership, 54, 6, 1997, 46-52.

[18] Riding, R. and M. Grimley, “Cognitive Style, Gender And Learning From Multimedia Materials In 11-Year Old Children.” British Journal of Educational Technology, 30, 1, 1999, 43-56.

[19] Yeung, A. “Cognitive Load and Learner Expertise: Split-Attention and Redundancy Effects in Reading Comprehension Tasks with Vocabulary Definitions.” The Journal of Experimental Education, 67, 1999, 197-217.

[20] Sweller, J., J. van Merrienboer, and F. Pass, “Cognitive Architecture and Instructional Design.” Educational Psychology Review, 10, 1998, 251-296.

[21] Ayres, P. “Cognitive Load Theory.” Mathematics Teaching, 156, 1996, 26-29.

[22] Luna, C.J. and J. McKenzie, “Testing Multimedia in the Community College Classroom.” T.H.E. Journal, 24, 7, 1997, 78-81.

[23] Wenger, M.J. and D.G. Payne, “Effects of a Graphical Browser on Readers’ Efficiency in Reading Hypertext.” Technical Communication, 41, 2, 1994, 224-234.

[24] Bloom, B.S. “Reflections on the Development and Use of the Taxonomy.” Yearbook: National Society for the Study of Education. 92, 2, 1994, 1-8.

[25] Bloom, B.S., M.D. Engelhart, E.J. Furst, W.H. Hill, and D.R. Krathwohl, (Eds.). Taxonomy of Educational Objectives, the Classification of Educational Goals, Handbook I: Cognitive Domain. New York: Longmans, 1956.

[26] Piaget, J. The Construction of Reality in the Child (M. Cook, Trans.). New York: Basic books, 1954.

[27] Smock, C.D. “Constructivism and Educational Practices.” In I.E. Sigel, D.M. Brodzinski & R.M. Golinkoff (Eds.), New Directions in Piagetian Theory and Practice (pp 51-68). Hillsdale, NJ: Erlbaum, 1981.

[28] Zimmerman, B.J. “Social Learning Theory and Cognitive Constructivism. In I.E. Sigel, D.M. Brodzinski & R.M. Golinkoff (Eds.), New Directions m Piagetian Theory and Practice (pp 39-49). Hillsdale, NJ: Erlbaum, 1981.

[29] Zemke, R. and S. Zemke, “Adult Learning: What Do We Know For Sure?” Training. 32, 6, 1995, 31-40.

[30] Boucouvalas, M. and J. Krupp, “Adult Development and Learning.” In S.B. Merriam and P.M. Cunningham (Eds.), Handbook of Adult and Continuing Education, 1989, pp. 183-200. San Francisco: Jossey-Bass Publishers.

[31] Cohen, M.S. and T.J. Ellis, “Teaching Technology in an Online, Distance Education Environment.” Proceedings: Frontiers in Education Conference, 2001 Reno (pp. T1F-1-T1F-6). Piscataway, NJ: IEEE, 2001.

[32] Tannenbaum, R.S. Theoretical Foundations Of Multimedia. 1998, New York: Computer Science Press.

[33] Heller, R.S. and C.D. Martin, “A Media Taxonomy”. IEEE MultiMedia, 2(4), 36-45. Retrieved 5 March 2002, from http://dlib.computer.org/mu/books/mu1995/pdf/u4036.pdf, 1995.

[34] Aleem, T.A. A Taxonomy of Multimedia Interactivity. (Doctoral Dissertation, The Union Institute, 1998). Digital Dissertations, AAT 9919729.

[35] Williams, M. A Taxonomy of Media Usage in Multimedia (T-MUM). Unpublished doctoral dissertation, Nova Southeastern University, Fort Lauderdale, FL. 2003.

[36] Kolb, D.A. Experiential Learning: Experience as the Source of Learning and Development. Prentice-Hall, Inc., Englewood Cliffs, N.J. 1984.


Dr. Timothy Ellis, Associate Professor, Graduate School of Computer and Information Sciences, Nova Southeastern University, 3301 College Ave, Fort Lauderdale, FL 33314. Dr Ellis obtained a B.S. degree in History and Philosophy from Bradley University, a M.A. in Rehabilitation Counseling from Southern Illinois University, a C.A.G.S. in Rehabilitation Administration from Northeastern University, and a Ph.D. in Computing Technology in Education from Nova Southeastern University. His research interests include: multimedia, distance education, adult learning, and databases.

Address: Graduate School of Computer and Information Sciences, Nova Southeastern University, Fort Lauderdale, FL 33314; telephone: (954)262-2029; fax: (954)262-3915; e-mail: ellist@nova.edu; his main website is located at http://www.scis.nova.edu/~ellist.

Copyright American Society for Engineering Education Jan 2004

Provided by ProQuest Information and Learning Company. All rights Reserved