Better Overall Science Concept Learning and Equity Gap Reduction

Middle-School Science Through Design-Based Learning versus Scripted Inquiry: Better Overall Science Concept Learning and Equity Gap Reduction

Mehalik, Matthew M


This paper contrasts performances overall and by gender, ethnicity, and socioeconomic status (SES) for middle school students learning science through traditional scripted inquiry versus a design-based, systems approach. Students designed and built electrical alarm systems to learn electricity concepts over a fourweek period using authentic engineering design methods. The contrast study took place in the eighth grade of an urban, public school district, with the systems approach implemented in 26 science classes (10 teachers and 587 students) and the scripted inquiry approach implemented in inquiry groups of 20 science classes (five teachers and 466 students). The results suggest that a systems design approach for teaching science concepts has superior performance in terms of knowledge gain achievements in core science concepts, engagement, and retention when compared to a scripted inquiry approach. The systems design approach was most helpful to low-achieving African American students.

Keywords: design-based learning, K-12 education, systems design

I. Introduction

What does student performance look like using an authentic design task to teach science concepts? In particular, how does it compare to the performance found with a scripted inquiry approach to do the same? For this project, the authors developed, disseminated, implemented, and evaluated a module for building electrical alarm systems in order to teach students electricity concepts in science classes. The task is described as authentic because students followed the same design process that a systems designer typically uses to propose, investigate, and construct embodied solutions to meet actual needs. The authors chose a design-based approach because of the promising results that learning science has demonstrated in previous research efforts in the areas of Learning by Design (Kolodner et al., 2003), project-based learning (Prince, 2004; Thomas, 2000) and problem-based learning (Barrows, 1985). The current approach differs from these previous efforts in that the authors have adopted a systems design approach (Blanchard and Fabrycky, 1998; Gibson, 1968) as the organizing sequence of design activities. This systems design-based approach adds a unique dimension by having students articulate their own needs for a particular design, in this case alarm systems, and then by having students develop requirements, which become design specifications that guide their design process. In other words, the design process proceeds by having the students articulate their needs for the design first, rather than having the design and its specifications proposed by the teacher or the curriculum. In this way, the design process that the students used was similar to the process that systems designers use when having clients articulate their needs in most types of commercial and governmental designrelated activities. Table 1 summarizes some of the differences between a systems design approach and a scripted inquiry approach. It is the hypothesis of the authors that by having students begin with their own needs, it is possible to increase student motivation and engagement, as the process begins by circumventing one of students’ most pertinent questions in science classes, “Why do I need to learn tiiis?” If students cannot answer this question for themselves or do not get a convincing answer from the teacher or curriculum, the lack of perceived relevance of the instructional task can create a barrier to learning at the very beginning of the entire learning process.

The authors have built upon several approaches that have demonstrated promise in the area of using design as a method to support learning. Kolodner et al. (2003) extensively described an approach called Learning by Design (LBD). This approach consisted of marrying the promising results of models of how learning from experience happens, as suggested by case-based reasoning research (Kolodner, 1993, 2002) and from problem-based learning (Barrows, 1985). Case-based reasoning refers to the ability to use past solutions and experience to guide the solutions to future problems. Problem-based learning is an approach that provides guidance on how classroom activities can be organized from a classroom management perspective. Both of these perspectives embraced the constructivist approach towards learning, as advanced by Harel and Papert (1991) and Kafai (1996). Another key feature of the LBD approach was that learning followed a cognitive apprenticeship method (Roth and Bowen, 1995) for which the key feature was that learning occurred often by observing and learning the language, behavior, and framing of events the way that members of a cultural group do. These combined approaches have been chosen because together they have shown that transferable learning (Klahr and Carver, 1988) is possible using them.

Learning about design and engineering is an intended positive outcome of this approach. It is important for students to learn that engineers design and create devices and systems. It is also important for students to begin to learn that design and discovery are both important thinking processes (International Technology Education Association, 2000). Design tasks create a challenge for students that increase their level of engagement and also serve the added benefit of opening the doors to possible science and engineering careers at a young age (Sadler, Coyle, and Schwartz, 2000).

Other primary objectives of this learning experience are to help students learn fundamental scientific principles about electricity, to improve their skills in reasoning like scientists, and to help students learn the skills for conducting authentic scientific inquiry. Such skills are fundamental for the future construction of expertise, competency, and heuristic style (Von der Weth and Frankenberger, 1995). National Science Education Standards identify as the fundamental skills necessary in order to conduct scientific inquiry as the abilities to:

1. Identify questions that can be answered through scientific investigations;

2. Design and conduct a scientific investigation;

3. Use techniques to gather, analyze, and interpret data;

4. Develop descriptions, explanations, predictions, and models using evidence;

5. Think critically and logically to make the relationships between evidence and explanations;

6. Recognize and analyze alternative explanations and predictions;

7. Communicate scientific procedures and explanations;

8. Use mathematics in all aspects of scientific inquiry (National Research Council, 1996)

Typically, as was the case in the subject school district, electricity (and science in general) is taught using a scripted/guided inquiry approach to learning (Their, 2000). Students are given materials and procedural scaffolding (i.e., instructional guidance that can range from explicit written, verbal, and visual directions to more subtie cues or hints) depending upon the philosophy of the curriculum designers’ views of what they think students need in order to accomplish a learning goal. The materials and scaffolding are intended to help students “discover” properties of electricity and electrical (scientific) principles, such as voltage, resistance, and current in different electronic components using multimeters. By scripted inquiry, we refer to curricula that, in order to allow students to “experience” fundamental concepts in science, provide heavy scaffolding in a cookbook-style, step-by-step approach, that directs the sequencing of how an experiment is put together, run, and is used as a way to gather data (Nagle, Hariani, and Siegel, 2006). In other words, scripted inquiry imposes large limitations on steps 1 and 2 of the scientific inquiry process, the ability for students to identify questions and the ability to design and propose scientific investigations. (Such curricula, on the other hand, typically do support learning for the other stages of the inquiry process.) This may be because of classroom time constraints (staying focused on learning important scientific principles), teacher concerns (the ability to manage a classroom), or other reasons.

The Learning by Design approach provides opportunities for students to have a much stronger experience in terms of the ability to design and propose scientific investigations. However, most design-based approaches begin by posing a design challenge, which limits the ability for students to begin with asking self-motivated questions worthy of scientific investigation.

The systems design based approach begins by having students articulate their own needs for a design which brings students closer to the space of raising their own questions worthy of investigation. We explore how a systems design approach affects student learning when compared/contrasted with the scripted inquiry approach. While it is relatively plausible that systems design-based learning is very engaging to students, it is very much an open question as to how well students will learn science concepts with this approach. In addition, we investigate whether a systems design approach could reduce the commonly found achievement differences associated with gender, edinicity, and socioeconomic status (SES). Is designbased learning (a relatively complex process) only effective for the strongest students, as many teachers would suspect? Or can designbased learning more effectively engage those traditionally unengaged with science? Additionally, can having students begin with the process of articulating their own needs for a design make a difference in helping students achieve at a higher level?

A. Systems Design Approach

Because a systems design approach is uncommon in K-12 science settings, we begin with a description of this approach, with particular emphasis on how it differs from more traditional scripted inquiry approaches to teaching science. The Electrical Alarm System: Design, Construction and Reflection learning module was developed collaboratively with a school district science program officer, the supporting science curriculum team, and with teachers. In addition to meeting state standards involving electricity concepts, the module was designed specifically to address two state and national standards for which the district requested assistance. One standard addresses the issue of design: know and use the technological design process to solve a problem. The other standard (systems thinking): explain the parts of a simple system and their relationships to each other. The module was designed to supplement and partially replace four to five weeks of instruction in the scripted inquiry unit on electricity.

The authors developed a student guide and a teacher guide to provide reading and scaffolding materials. During implementation, the authors provided five, four-hour professional development (PD) workshops for teachers, distributed as one before implementation, three during implementation, and one after implementation. The majority of the teachers attended all of the workshops. The workshops addressed issues of electricity concepts, pedagogy, and implementation details. The workshops had teachers work in groups to test out the unit themselves as learners, share classroom experiences, and reflect on what they and their students were learning. The researchers provided on-site guidance through extensive classroom visits, during which they maintained an ethnographic log account of classroom dynamics.

The system design module was specifically designed to teach several of the same science concepts that are covered in four (of the nine total) of the existing scripted inquiry curriculum modules. These four modules typically took teachers four to five weeks to cover in past years and covered some of the basics of electricity, such as voltage, current, resistance, parallel circuits, series circuits, batteries, lamps, resistors, and conceptual relationships in Ohm’s Law. The systems design module simply replaced those four units in the course of the spring semester for the systems design groups. Once the systems design groups completed the systems design unit, they then moved on to the remaining inquiry unit materials that the systems design unit was not intended to replace (i.e, modules five through nine in the inquiry curriculum) that covered capacitance, charge, power, and other concepts of electricity.

B. Overall Design Process

Students and teachers followed a systems design approach (DeBonno, 1986; Penner, Lehrer, and Schauble, 1998; Wiggins and McTighe, 1998) throughout the course of the four to five weeks of implementation. Based on authentic approaches that systems engineers use in the design and analysis of systems, the design approach consisted of the following major stages, as shown in Figure 1 (Blanchard and Fabrycky, 1998; Gibson, 1968; Gibson, Scherer, and Gibson, 2007):

These seven stages of systems design and analysis were generated from the researchers’ past experiences in systems engineering and design, supplemented by a review of the best practices of empirical studies of design. In a typical system design implementation, the reflect and evaluate stage would involve going back through the entire process iteratively in order to improve and adjust the design specifications as new knowledge and problems are encountered. Although it was not possible to have students repeat a complete second iteration of the entire process, scaffolding for both the students and the teacher encouraged students regularly to go back, review, re-use, and refine what they had produced and documented in earlier stages of this overall process, as is shown by the arrows that point back to earlier stages from later stages in Figure 1.

C. Example Design Process Steps

The researchers provided scaffolding for students to work both individually and in teams for each of the stages of their design process. Figure 2 shows an example of such scaffolding, which illustrates one team’s specification of system inputs and outputs as part of specifying some of the functional capabilities of one of their design ideas during the “Generate Alternatives” stage of the systems design process. Students in this example are being guided to analyze their idea, a device that will alert a user as to when they need to take their medication, according to what materials, energy, and information need to go into its operation in order for it to perform its intended function, “helps the elderly with their health.” The students also are guided to think about what unintended, potentially negative, impacts their device might have.

From this exercise, students are further guided to specify in increasing functional detail, how specific subsystems are coordinated to perform the overall function of their idea. Typically, students discover that their alarm systems must have a “detector” subsystem which can sense when some noteworthy event or activity has occurred, such as the time of day when medication needs to be taken. They also discover that their alarm must have an “indicator” subsystem, which is some form of alert that a noteworthy event has happened. Finally, they discover that they need a “power” subsystem, which makes both the indicator and detector subsystems work. Such activities encourage students to develop skills in analytical thinking, as they break down their ideas into more specific functional details. For example, what will detect when it is time to take medicine, what will supply the power, or what type of alert will notify somebody? They also engage in synthesis thinking as they come up with new ways of linking their new ideas for subsystems and how they inter-relate and work together. In this stage, students eventually begin to think about what specific electronic and mechanical components they might be able to use to build their subsystems: Which electronic components can function as detectors, such as photocells, switches, and heat detectors, among many others? Which function as indicators, such as lights, LED’s, and buzzers, among many others? Science content thinking thus enters the process a little bit at a time, increasingly as student ideas progress in specificity, and always connected to their goal of developing their original ideas.

Figure 3 shows some of the scaffolding in which a student team rated several of their design ideas against a set of performance criteria they constructed, as is associated with the “Making a selection” stage of design. After students have analyzed their needs, developed specific decision requirements from their needs, and have engaged in developing several different ideas, they are guided through a process of choosing a design that best meets their needs. In Figure 3, the students have listed their requirements in the first column. Their requirements specify performance functions, such as “beeps when battery runs out” as well as items they deemed essential for the design, such as “volume contrast” or “rechargeable battery”. Their three design alternatives are listed at the top of the next three columns: a device that required a manual reset and dispensation of medication, a device that automatically reset and dispensed medication, and some combination of both devices. The students next rated how well each design they imagined would perform for meeting each requirement using a five-point scale. The design that received the highest score was deemed the best solution for their requirements. This exercise scaffolded students to think analytically about the performance of each device, synthetically about needs and performance, and in an evaluation mode as they compared ideas against specified performance criteria. Teachers encouraged students to make one more leap in their modes of thinking: reflecting upon the choice that emerged from this process: Does that outcome make sense to you? If not, where does the outcome seem to not fit what you think? Students then go back and examine their whole system of needs, requirements, and alternatives, making changes as their thinking is stimulated by this reflection. Eventually, students make choices that have an extensive basis in thinking that involves articulating clear decision criteria as well as in-depth specification of how different ideas work and function, with science content ideas roped into the process.

D. How Science Concepts Are Explored In Design

Figures 2 and 3 serve as representative examples of what the systems design group students produced in their portfolios. What these examples illustrate is that the design approach provided scaffolding at the various stages of the system design process without specifying exactly what content item each student and each student team chose to pursue. Following this process of making choices about which of their designs to build, next students actually build dieir ideas using kits of electronic components that were assembled from parts from existing scripted inquiry kits, such as wires, springboards, resistors, LED’s, lamps, among other components, supplemented with other electronic components, such as buzzers, photocells, thermistors, among other items. Each electronic component is therefore connected to a specific subsystem and function for each student idea. Science content inquiry thinking is thus connected to the design process. As students attempt to use the various components available to them in order to embody their design plans in working devices, they need to come to understand how each of the components work and how their performance can be improved. This is a process of discovery and inquiry, and it takes place within the context of creative design thinking to which students relate because they are creating from their needs and interests.

E. Contrasts Between Systems Design and Scripted Inquiry

This issue of providing a framework in which students can pursue their own ideas is one of the main differences between a systems design approach when compared to a scripted inquiry approach. Other differences are listed in Table 1. The systems design approach is organized according to modes of different types of thinking. As the discussion about Figures 2 and 3 illustrated, the students were guided to think about needs, requirements, alternatives, and decision criteria, which involve generative and analytical thinking. Thinking about system parameters such as materials, energy, and information inputs and outputs involves analytical thinking and synthesis thinking. Thinking about alternative designs and choosing a design involves evaluative thinking. In the systems design approach, inquiry into scientific materials, methods, and principles becomes another mode of thinking in a scaffolded series of modes of thinking.

The infusion of science concepts through inquiry occurred mainly during the final two stages of the systems design process, and was left rather unscripted except for scaffolding documentation for which students needed to draw their circuit attempts and to explain them during the process of building. The teacher was encouraged to explain a science concept to a student only after that student had made several attempts at exploring the concept and trying different ideas and configurations during circuit construction. This is in major contrast with the scripted inquiry approach, which provides step-bystep instructions for most aspects of an investigation. Scripted inquiry is mainly organized according to the presentation of science concepts in a way chosen by curriculum designers instead of by modes of thinking. Systems design, as an example of project-based learning, was expected to enhance student level of engagement (Hake, 1998) because it begins with a needs analysis that emerges from a student’s interests and world experience.

The foregoing mainly highlights the anticipated positive benefits of the systems design approach: students work from their own needs, develop their own ideas, and engage in a larger range of modes of thinking when compared with the scripted inquiry approach. In addition, students must take responsibility and are accountable for their own learning. Each student was responsible for keeping a portfolio of the scaffolded design process activities, as well as sketches and reflection. They were able to share ideas with their teams; however, they were responsible for turning in their portfolios for a grade at various stages of the module, in addition to a process of teacher and peer evaluations when they presented their final products.

Not all aspects of the systems design approach are clearly superior to scripted inquiry, especially from the perspective of the teacher. First of all, the material in this module was developed as a pilot. This means that the degree of testing and fine-tuning is in a much less refined state than scripted inquiry materials, which have undergone extensive field-testing and refinement, and which also have a strong performance record among many science teachers. Many teachers are familiar with scripted inquiry, so asking teachers to switch to a different organizational scheme required some additional preparation time, involving both professional development training and pre-class activity efforts. The systems design approach also involves turning over a lot of classroom time to student teams, with the teacher acting as a facilitator, instead of director of student attention. This switch can raise teacher anxiety over ensuring that the students are dedicating their time to relevant classroom tasks as they are given more autonomy. Teachers discover that they spend much less time lecturing about science ideas, such as Ohms Law, than what they may have done previously. Spending less time directly on science concepts and instead on design process, a process that is not part of the usual science education training, can also increase teacher anxiety. Finally, as is often the case for middle school science teachers, many science teachers were not trained to teach electricity, so the idea of teaching electricity in itself is a source of anxiety, compounded by having an additional unfamiliar concept, systems design, added to the mix. Therefore, the implementation of the systems design approach involved some nontrivial risks, directly from the perspective of teachers, and indirectly to students.

F. Hypotheses and Research Questions about Systems Design vs. Scripted Inquiry

The researchers hypothesized that the systems design approach would achieve overall increases in student performance. They also hypothesized that the risks involved with the intervention would be offset by the professional development training and implementation support that the researchers would provide. Prior work with design-based learning has shown that it can produce conceptual learning (Barton, 1998; Kolodner, 2002; Kolodner et al., 2003; Penner, Lehrer, and Schauble, 1998). Design-based learning has not previously been contrasted directly against scripted inquiry. This paper presents a study designed to test this contrast.

Finally, a note on the systems design approach and performance and equity gaps. In addition to having generally low performance, K-12 science education in the U.S. generally produces large differences between low and high SES students, as well as between Caucasian or Asian students and Latino and African American Students (Martin et al., 1997, 2000). Moreover, beginning in middle school and growing in high school, there is a significant difference between male and female performance in science (Martin et al., 2000). Permitting students to choose what they design may reduce equity gaps. In addition, students needed to present their ideas to one another, and this was viewed as facilitating a process whereby students take ownership of their ideas, which also is expected to contribute to reducing equity gaps in performance. Another equity gap issue involves equal access to materials for construction. The researchers created the systems design learning module in such a way that all hands-on work occurred in the classroom, using the same sets of materials for all students (this is the same as scripted inquiry). Such a configuration is not always the case for project-based learning. When students are permitted to use materials from home, performance differences can be exacerbated because of unequal access to resources and to available time outside of class.

Additionally, the kits for the systems design module consisted of 85 percent of the same materials that were already a part of the original scripted inquiry curriculum. The system design module included a few extra materials: two buzzers, two battery holders, a thermistor and two photo resistors. The remaining components, such as springboards, lamps, LEDs, and wires all were from the original scripted inquiry curriculum.


The researchers used a paired experimental/contrast design in which 10 teachers and 587 students (26 classes) implemented the system design approach with the Alarm System module (referred to as the design group), and five teachers and 466 students (20 classes) followed the scripted inquiry approach with the scripted inquiry Electricity unit (referred to as the inquiry group). Both groups were recruited from the entire pool of 27 science teachers for 8th grade in an urban school system in the Northeast U.S. The authors offered the entire pool of 8th grade science teachers an opportunity to learn about systems design-based learning during a professional development workshop at the very beginning of the spring 2004 term. This workshop was one option among more than five others available to the teacher pool. Teachers were strongly encouraged to engage in some form of professional development training by the district, although such training was not required. Nineteen teachers attended the workshop, and 12 initially agreed to participate in the design group condition. Two teachers could not participate in the study for various administrative reasons. The result was that 10 teachers eventually participated.

The process for these 10 teachers was largely a self-selection, voluntary process. The teachers were informed that they could discontinue their participation at any time for any reason. There were several factors, however, that must be considered as part of the selection process. First, the district encouraged all teachers to participate in some form of professional development, and so motivation for participation was partially driven by this factor, although teachers did have the opportunity to choose other forms of PD besides the systems design opportunity. Second, the teachers were paid for their time spent in professional development training at the district rate for PD compensation. Although a nominal amount, this did constitute a financial incentive for participating. Finally, the teachers earned continuing education credits for their participation in the systems design PD, as was the case for teachers who participated in other forms of PD.

During the course of this project, the systems design group teachers attended a sequence of five professional development workshops, each lasting four hours in duration. The workshops focused on permitting teachers to experience hands-on, systems design learning. Each teacher completed the systems design module in the same way that their students eventually would. Each workshop emphasized a different stage of the systems design process, including hands-on construction of their ideas using the same electronic component kits that students would use. The teachers presented their ideas and results to one another, and they reflected upon their designs and processes while sharing their ideas with one another. Teachers also exchanged ideas about what worked well with their students during workshops that occurred in the midst of their classroom implementation. The emphasis of the PD workshops was not on content support in terms of lectures on electronics and principles of electricity.

The inquiry group was recruited from the remaining teachers in the district. The authors sent a letter of request (and testing materials) to the remaining teachers to administer pre- and post-tests as part of this research study. The district science program officer also sent a request for participation, although there was not a specified requirement to do so. From the remaining pool, seven teachers returned pre-test results and seven post-test results, although there were different schools in the two pools. There were five teachers common to both groups, and they make up the inquiry group. The inquiry group thus consisted of a more volunteer-oriented composition than the design group because there was less district encouragement, no compensation offered, and no certification credit granted to participate as an inquiry group member. All of the inquiry group teachers had participated within the previous three years in professional development surrounding the implementation of that module. The professional development consisted of at least three four-hour sessions of training on this curriculum. One teacher volunteered out of curiosity to participate in both the design group and inquiry groups as a focal study. This teacher decided to implement the systems design module in one class and to continue to use the scripted inquiry approach in three others.

This selection procedure resulted in some demographic differences between the design group and inquiry group from the schools that reported such data, as Table 2 demonstrates. In terms of overall socioeconomic status (SES) information based on district data for each of the schools, the design group has a higher proportion of students from schools in the low SES range (53 percent of 587 students vs. 32 percent of 466 students). The socioeconomic categories are based on proportion of students considered by the district to be economically disadvantaged, with the low group having schools with more than 66 percent of their students economically disadvantaged, the high group having less than 40 percent, and medium the interim proportion. The four lowest SES schools in the district are in the design group. This difference is offset by a lower proportion of students in the middle SES range (14 percent vs. 38 percent) for the design group. The groups are nearly evenly matched among students in the high SES range.

In terms of gender, the design group had a slightly higher proportion of female students (54 percent vs. 51 percent), a twice-ashigh proportion of African American students (66 percent vs. 33 percent) and a higher proportion of students receiving free and/or reduced lunches (64 percent vs. 58 percent).

Students were given pre- and post-tests to measure changes in student knowledge of electricity concepts. The researchers specifically created a knowledge test that was designed around core concepts in electricity, such as resistance, current, voltage, series and parallel circuits, in order to ensure that both the design group and inquiry groups would be evaluated on content knowledge for which the guided inquiry approach focused. Figure 4 shows a sample question taken from the knowledge test. The full version of the knowledge test is included in Appendix A. The pre-tests were administered immediately (i.e., one day) before the systems design group began the four to five week systems design module and before the scripted inquiry group began their first scripted inquiry module. The post-tests were administered immediately after (i.e., the following day) when the systems design groups completed the four to five week systems design module for the systems design groups and when the scripted inquiry group finished the first four modules of the scripted inquiry curriculum. Both the systems design module and the first four units of the scripted inquiry unit were completed within four to five weeks.

III. Results

A. Overall Science Concept Test Comparisons

Overall, the design group achieved twice the pre-post gains in scores on science knowledge content questions when compared with the inquiry group, as Figure 5 shows. The design group showed a mean gain of 16 percent versus the inquiry group gain of 7 percent (t = 2.02; p

The design group started at an overall lower mean science knowledge pre-test score of 29 percent on the exam than the inquiry group’s score of 38 percent on the exam. The differences in initial test scores may likely be explained by the differences in the socioeconomic status between the design group and inquiry group schools and the teacher selection process. The gap in overall performance decreased, from the initial difference of nine test percentage points to essentially parity in performance, with less than one test percentage point difference, with the design group having a final overall mean test score of 45 percent and the inquiry group with a final overall mean score of 46 percent. In terms of percentage of possible improvements, the design group achieved 22.5 percent of the possible improvement of their post-test score over their pre-test scores. The inquiry group achieved 11.3 percent of the possible improvement in their scores. Additionally, the mean scores on the pre-test and post-test were such that ceiling effects were not of concern.

The systems design approach produces better gains overall. What about its impact on equity gaps? As Figure 6A shows, the science knowledge test gains for African American students in the design group are eight times higher (M = 0.16 vs. 0.02; t = 2.05, p

Figure 6B shows the gains that males and females achieved for both design group and inquiry groups. Gains for females and males in the systems design group were nearly identical (0.14 for females; 0.16 for males), well within the range of measurement uncertainty. This result suggests that the systems design approach does not produce any performance gaps according to gender. Male students using the systems design approach achieved significant gains when compared with their counterparts who followed the scripted inquiry approach (M = 0.16 vs. 0.05; t = 2.04, p

The systems design approach had the biggest impact on low-achieving African American students in terms of science knowledge test performance. Students were grouped into three performance categories according to their scores, with low achievement defined as scores lower than 30 percent, moderate as scores between 30-60 percent, and high as scores over 60 percent correct on science questions. This method of categorizing was applied to both the pre- and post-tests. As Figure 7 shows, the African American systems design group showed upwards shifts for all performance categories relative to their pretest categorization, with a shift of 21 percent of its population out of the low achieving category to higher categories, and with an increase of 15 percent of the entire population into the highest performance category. By comparison, the inquiry group showed a small increase in the proportion of students in the low achieving category. What this finding suggests is that the systems design approach may be particularly effective for reaching this demographic group.

Among non-African American students, there was again a significant shifting of students from low-achieving categories to higher categories; however, the biggest performance inquiry group among this demographic occurs for students who are in the middle-achiever range. There was a 29 percent increase of students into the high-performers as a result of the module, a much higher proportion increase than when compared with the scripted inquiry group. The systems design approach does much better for boosting performance of low-achieving African American students and moderate-achieving non-African American students than scripted inquiry, and it does no worse than scripted inquiry on other performance dimensions in terms of race and achievement level. This finding is particularly significant given the profile of the schools and students in the design group.

As shown in Figure 8, female students showed similar gains profiles in both inquiry group and design group conditions, although with slighdy stronger results across the board in the design group condition. Male students seemed especially engaged by the systems design approach, showing considerably more transitions into the high performance category.

Figures 7 and 8 should be understood as only representing overall changes in overall proportions of the pre- and post-test populations based on where their pre- and post-test scores fell in the ranges low achievement less than 30 percent, moderate between 30-60 percent, and high over 60 percent correct on science questions. The figures do not capture individual student trajectories. In other words, the figures do not show the proportion of specific students whose scores started in the low category and jumped directly to the high category pre-post (or for that matter the proportion of students whose scores started in the high category and jumped to the low category pre-post, or any other specific student trajectory). Instead, the figures show two snapshots-a proportion of the whole DUMMY M’TEXT population at the time the pre test was taken and a proportion when the post test was taken.

Overall, from the perspective of race and gender, the systems design group appears to help low achieving African Americans the most. Other groups that appear to improve disproportionately are moderate-achieving non-African Americans and low and moderate achieving males. Among the other dimensions of race and gender, the systems design approach performs at least as well as the scripted inquiry approach.

B. Summary

What does student performance look like using an authentic design task to teach science concepts when examined concurrently with a scripted inquiry approach to do the same? From an overall performance perspective, the gains in test scores across all performance group categories when compared with the scripted inquiry approach appear to be appreciable: double the gain scores in science content knowledge overall. In terms of race and gender, the systems design group appears to help low achieving African Americans the most. A second group of people that appear to improve disproportionately are moderate-achieving non-African Americans and moderate achieving males. Among the other dimensions of race and gender, the systems design approach performs at least as well as the scripted inquiry approach.


The results of this study suggest that a systems design approach for teaching science concepts has superior performance in terms of knowledge gain achievements, engagement, and retention when compared with a guided inquiry approach. The systems design approach was most helpful to low-achieving African American students, although it was at least as good and typically better than scripted inquiry for all students. The authors have described that what is unique about the systems design approach is that students begin the design process by articulating their own needs for their design, in the same way that a systems analyst and designer would do for any new product or system design. By having students begin the learning process from their own needs, the systems design approach to learning tackles the question that students often articulate and that often serves as a barrier to learning, “Why do I need to know this?” The systems design process permitted students to ask their own questions for investigation in order to design their alarm systems, and it also permitted students to design their own experiments to investigate their ideas. The findings showed that it was possible to achieve higher science concept learning when the scientific inquiry process was integrated into a design setting, motivated by meeting needs that students articulated themselves.


This project was supported by two National Science Foundation programs under awards ERH-0227016 (Sytem-Wide Change for All Learners and Educators) and EEC-0502035 (Research Experience for Teachers).


Barrows, H. S. 1985. How to design a problem-based curriculum for the preclinical years. New York: Springer Verlag.

Barton, A. C. 1998. Examining the social and scientific roles of invention in science education. Research in Science Education 28 (1): 133-51.

Blanchard, B., and W. B Fabrycky. 1998. Systems engineering and analysis. Lebanon, IN: Prentice Hall.

Cooper, J., S. Prescott, L. Cook, L. Smith, R. Mueck, and J. Cuseo. 1990. Cooperative learning and college instruction: Effective use of student learning teams. Long Beach, CA: California State University Foundation.

De Bono, E. 1986. The CoRT thinking program, 2nd ed. Oxford, U.K.: Pergamon Press.

Fiechtner, S. B., and E. A. Davis. 1992. Why some groups fail: A survey of students’ experiences with learning groups. In Collaborative learning: A sourcebook for higher education, ed. A. S. Goodsell, M. R. Maher, and V. Tinto. Syracuse, NY: National Center on Postsecondary Teaching, Learning, and Assessment, Syracuse University.

Gibson, J.E. 1968. Introduction to engineering design. New York: Holt, Rhinehart, and Winston.

Gibson, J. E., W. T Scherer, and W. F. Gibson. 2007. How to do a systems analysis. New York: Wiley Interscience.

Hake, R. 1998. Interactive-engagement vs. traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics 66 (1): 64.

Harel, Idit, and Seymour Papert. 1990. Software design as a learning experiment. Interactive Learning Environments 1 (1): 1-32.

International Technology Education Association (ITEA). 2000. Standards for technological literacy: Content for the study of technology. Reston, VA: ITEA.

Kafai, Y. 1996. Learning design by making games: Children’s development of design strategies in the creation of a complex computational artifact. In Constructionism in practice: Designing, thinking, and learning in a digital world, ed. Y. Kafai and M. Resnick, 71-96. Mahwah, NJ: Lawrence Earlbaum & Associates, Inc.

Klahr, D., and S. M. Carver. 1988. Cognitive objectives in a LOGO debugging curriculum: Instruction, learning and transfer. Cognitive Psychology 20:362-04.

Kolodner, J.L. 1993. Case based reasoning. San Mateo, CA: Morgan Kaufmann.

Kolodner, J. L., D. Crismond, B. B. Fasse, J. T. Gray, J. Holbrook, M. Ryan, and S. Puntambekar. 2003. Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting a learning-by-design curriculum into practice. Journal of the Learning Sciences 12 (4): 495-548.

Kolodner, J. L. 2002. Facilitating the learning of design practices: Lessons learned from inquiry into science education. Journal of Industrial Teacher Education 39 (3):1-31.

Martin, M. O., I. V. S. Mullis, A. E. Beaton, E. J. Gonzalez, T. A. Smith, and D. L. Kelly. 1997. Science achievement in the primary school years: IEA’s third international mathematics and science study. Chestnut Hill, MA: Center for the Study of Testing, Evaluation, and Educational Policy, Boston College.

Martin, M. O., I. V. S. Mullis, E. J. Gonzalez, K. D. Gregory, T. A. Smith, S. Chrostowski, et al. 2000. TIMSS 1999 international science report: Findings from IEA’s repeat of the third international mathematics and science study at the eighth grade. Chestnut Hill, MA: International Study Center, Lynch School of Education, Boston College.

Nagle, B., M. Hariani, and M. Siegel. 2006. Achieving a vision of inquiry Rigorous, engaging curriculum and instruction. In Exemplary Science in Grades 5-8: Standards-Based Success Stories, ed. Rogert E. Yager. Arlington, VA: National Science Teachers Association Press.

National Research Council. 1996. National science education standards. Washington, D.C.: National Academy Press.

Penner, D. E., R. Lehrer, and L. Schauble. 1998. From physical models to biomechanics: A design-based modeling approach. Journal of the Learning Sciences 7 (3-4): 429-49.

Prince, M. 2004. Does active learning work? A review of the research. Journal of Engineering Education 93 (3): 223 – 32.

Rosnow, Ralph L., Robert Rosenthal, Roberta Manner McConochie, and Robert L. Arms. 1969. Volunteer effects on experimental outcomes. Educational and Psychological Measurement 29:825.

Roth, Wolff-Michael, and M. G. Bowen. 1995. Knowing and interacting: A study of culture, practices, and resources in a grade 8 open-inquiry science classroom guided by a cognitive apprenticeship metaphor. Cognition and Instruction 13 (1): 73-128.

Sadler, P. M., H. Coyle, and M. Schwartz. 2000. Engineering competitions in the middle school classroom: Key elements in developing effective design challenges. Journal of the Learning Sciences 9 (3): 259-86.

Slavin, R. E. 1995. Cooperative learning: Theory, research, and practice, 2nd ed. Boston, MA: Allyn & Bacon.

Thier, H. D. 2000. Developing inquiry-based science materials: A guide for educators. New York: Teachers College Press, Columbia University.

Thomas, W. J. 2000. A review of research on project-based learning. San Rafael, CA: The Autodesk Foundation.

Von der Weth, R., and E. Frankenberger. 1995. Strategies, competence, and style-problem solving in engineering design. Learning and Instruction 5:357-83.

Wiggins, G, and J. McTighe. 1998. Understanding by design. Alexandria, VA: Merrill Education/ASCD College Textbook Series.


Industrial Engineering

University of Pittsburgh


Learning Research and Development Center

University of Pittsburgh


Learning Research and Development Center

University of Pittsburgh


Matthew Mehalik is a visiting assistant professor in Industrial Engineering and research associate at the University of Pittsburgh’s Learning Research and Development Center (LRDC). He completed a Ph.D. in Systems Engineering at the University of Virginia in 2001 and held postdoctoral positions at the LRDC and in Science, Technology, and Society at the University of Virginia. His interests involve the intersection of engineering and policy in complex systems, particularly during innovation and design.

Address: 1048 Benedum Hall, Department of Industrial Engineering, University of Pittsburgh, Pittsburgh, PA 15261; telephone: (+1) 412.624.7245; fax: (+1) 412.624.7439; e-mail:

Yaron Doppelt is director of the Science Center for Yemin Orde Youth Village and teaches at Ort Braude Engineering College and the Misgav Mechatronics Department in Israel. He completed a Ph.D. in Education from the Technion, Israel Institute of Technology, Israel in 2001 and completed a post-doctoral position at the Learning Research and Development Center, University of Pittsburgh. His expertise includes developing effective learning schemes within science and technology for disadvantaged youth. He has been a leading force for professional development for science teachers in Israel and the U.S.

Address: Learning Research and Development Center, Room 806, University of Pittsburgh, 3939 O’Hara St., Pittsburgh, PA 15260, USA; telephone: (+1) 412.624.8807; fax: (+1) 412.624.7439; e-mail:

Christian Schunn is associate professor of Psychology, Cognitive Studies in Education, and Intelligent Systems and research scientist at the Learning Research and Development Center at the University of Pittsburgh. Dr. Schunn received his Ph.D. in Psychology from Carnegie Mellon University in 1995. His current research focuses on understanding complex forms of expertise, building models of authentic practice in science and engineering, and applying those models to improve K-20 science and engineering education.

Address: Learning Research and Development Center, Room 821, University of Pittsburgh, 3939 O’Hara St., Pittsburgh, PA 15260, USA; telephone: (+1) 412.624.8807; fax: (+1) 412.624.7439; e-mail:


Provided by ProQuest Information and Learning Company. All rights Reserved