Classroom and school factors affecting mathematics achievement: a comparative study of Australia and the United States using TIMSS

Stephen Lamb

Recent work on differences in mathematics achievement has highlighted the importance of classroom, teacher and school factors. The present study used data from the Third International Mathematics and Science Study (TIMSS) to look at student, classroom and school factors influencing mathematics achievement in Australia and the United States (US). It found that classroom differences account for about one-third of the variation in student achievement in the US and over one-quarter in Australia. Most of the classroom variation in both countries was due to compositional and organisational factors, very little of it due to differences between teachers. This has important implications for policy regarding the improvement of mathematics achievement. It suggests that school systems may gain little by targeting teachers only, and need to give consideration to the role of pupil grouping practices and the effects of tracking and streaming on classroom learning environments.

Introduction

There is widespread interest in improving the levels of mathematics achievement in schools. Apart from the economic benefits that it is argued this would bring, by better preparing young people for the numeracy demands of modern workplaces and raising the overall skill levels of the workforce, there are also social benefits tied to improving access for larger numbers of young people to post-school education and training opportunities and laying stronger foundations to skills for lifelong learning. The interest in raising levels of achievement has led to a focus on identifying the range of factors that shape achievement as well as understanding how these factors operate to limit or enhance the achievement of different groups of students. The impact on different groups of students is important because social differences in mathematics performance persist, despite inequalities in some other areas of school having declined. A study of trends in mathematics achievement over the three decades to 1996, in Australia, shows that substantial social class differences persist (Afrassa & Keeves, 1999). Similar results have been reported in the US for the same period, with differences related to social groups (measured by parental education) remaining strong (National Center for Education Standards, 2000). The evidence is a reminder that at a time when there are weakening social trends on some broad indicators of educational participation, such as school retention rates, social differences in student progress and academic outcomes continue.

This paper examines student, classroom and school factors influencing mathematics achievement in Australia and the US. To do this, it uses data from the Third International Mathematics and Science Study (TIMSS). A recent paper using these data has shown that, in Australia, although student background variables influence differences in achievement in mathematics, classroom and school variables also contribute substantially (Lamb & Fullarton, 2000). How much does this result hold in the US? Are the factors influencing mathematics achievement the same in both contexts? What can the relationships between teachers, classrooms, schools and student achievement in both countries inform us about policies or reforms to improve levels of mathematics achievement for all young people?

School: and classroom effectiveness

The early literature on school effectiveness placed an emphasis on the ability and social backgrounds of students as factors that shape academic performance, and suggested that schools had little direct effect on student achievement. Coleman et al. (1966), for example, in a major study of US schools seemed to cast doubt on the possibility of improving school achievement through reforms to schools. They found that differences in school achievement reflected variations in family background, and the family backgrounds of student peers, and concluded that ‘schools bring little influence to bear on a child’s achievement that is independent of his background and general social context’ (p. 325). A later analysis of the same dataset by Jencks and his colleagues reached the same conclusion: `our research suggests … that the character of a school’s output depends largely on a single input, namely the characteristics of the entering children. Everything else–the school budget, its policies, the characteristics of the teachers–is either secondary or completely irrelevant’ (Jencks et al., 1972, p. 256).

Criticisms of this early work suggested that the modelling procedures employed did not take account of the hierarchical nature of the data, and was not able to separate out accurately school, student and classroom factors (e.g. Raudenbush & Willms, 1991). More recent school effectiveness research has used multi-level modelling techniques to account for the clustering effects of different types of data. The results of such studies show, according to the meta-analysis of school effectiveness research undertaken by Bosker and Witziers (1996), that school effects account for approximately 8 to 10 per cent of the variation in student achievement, and that the effects are greater for mathematics than for language. A number of studies have shown that there are substantial variations between schools (Lamb, 1997; Mortimore et al., 1988; Nuttall et al., 1989; Smith & Tomlinson, 1989).

Several studies have concluded that classrooms as well as schools are important and that teacher and classroom variables account for more variance than school variables (Scheerens, 1993; Scheerens, Vermeulen, & Pelgrum, 1989). Schmidt et al. (1999) in their comparison of achievement across countries using TIMSS data reported that classroom-level differences accounted for a substantial amount of variation in several countries including Australia and the US. Are these differences due more to teachers, to classroom organisation, to pupil management practices or other factors?

Recent work on classroom and school effects has suggested that teacher effects account for a large part of variation in mathematics achievement. In the United Kingdom, a recent study of 80 schools and 170 teachers measured achievement growth over the period of an academic year, when using start-of-year and end-of-year achievement data (Hay Mcber, 2000). Using multi-level modelling techniques, the study modelled the impact teachers had on achievement growth. The report on the work claimed that over 30 per cent of the variance in pupil progress was due to teachers. It concluded that teacher quality and teacher effectiveness, rather than other classroom, school and student factors, are large influences on pupil progress.

Several Australian studies have also pointed to teachers having a major effect on student achievement. In a three-year longitudinal study of educational effectiveness, known as the Victorian Quality Schools Project, Hill and his colleagues (Hill, 1994; Hill et al., 1996; Rowe & Hill, 1994) examined student, class/teacher and school differences in mathematics and English achievement. Using multi-level modelling procedures to study the interrelationships between different factors at each level–student, classroom and school–the authors found in the first phase of the study that, at the primary level, 46 per cent of the variation in mathematics was due to differences between classrooms, whereas at secondary level the rate was almost 39 per cent. Further analyses showed that between-class differences were also important in examining student growth in mathematics achievement, and that differences in achievement progress located at the classroom level ranged from 45 to 57 per cent (Hill et al., 1996; Hill & Rowe, 1998).

In explaining the large classroom-level differences in student achievement in mathematics, Hill and his colleagues highlighted the role of teacher quality and teacher effectiveness. They contended that, although not fully confirmed, they had `evidence of substantial differences between teachers and between schools on teacher attitudes to their work and in particular their morale’ and this supported the view that `it is primarily through the quality of teaching that effective schools make a difference’ (Hill, 1994). In further work that examined the impact of teacher professional development on achievement, they again argued that differences between teachers helped explain much of the variation in mathematics achievement (Hill et al., 1996).

However alternative explanations for the large classroom-level differences were also advanced by Hill and his team. They pointed to the possibility that classroom-level pupil management practices such as streaming and setting could account for the class effects. This was not pursued by the authors who stated that, in all of the schools they surveyed, the classes were of mixed ability (Hill, 1994; Rowe & Hill, 1994). Another possibility was an under-adjustment for initial differences, that is, they did not control adequately for prior achievement differences. A further explanation considered was the possibility of inconsistency in teacher ratings used in the measure of student achievement in mathematics. This possibility was also deemed by Hill and his colleagues as unlikely to have had a major bearing, though its influence was not ruled out. However the authors did not use, or argue for the use of, more objective, independently assessed mathematics tests.

Other studies have shown that contextual, variables such as student body composition and organisational policies play an important role in mathematics achievement. Teacher background attributes such as gender, number of years in teaching and educational qualifications have been shown to be important factors in student achievement (Larkin & Keeves, 1984; Anderson, Ryan, & Shapiro 1989), as have a variety of school effects such as school size (Lee & Smith, 1997) and mean student social composition.

These studies suggest that classrooms and schools matter, as well as student background. A range of studies has examined different effects; however few have been able to use the range of contextual variables available in TIMSS. This paper uses the TIMSS data to investigate the interrelationships among different factors at the student, classroom and school levels in both the US and Australia. A key issue is to investigate whether teacher quality and classroom effectiveness account for classroom-level variation in mathematics achievement or whether there are other factors that are of more importance. To do this, we examine patterns of Grade 8 student achievement by partitioning variance and using multi-level modelling procedures to estimate the amount of variance that can be explained at the student, classroom and school levels. By introducing different classroom and teacher variables, we test the extent to which factors linked to teachers and those linked to classroom organisation and practice influence achievement. If differences in mathematics achievement are heavily influenced by variations in the quality of teachers and teacher effectiveness, as the work of Hill and his colleagues suggests, then there are major policy implications for schools and school systems in terms of changing the provision and quality of teacher training, taking more care in teacher selection practices, re-shaping and investing more heavily in teacher professional development, and reforming the way in which schools deploy teachers and monitor their effectiveness. Alternatively, if other features of classrooms and schools explain more of the variation, then schools and school systems may not obtain the expected benefit in increased nmthematics achievement by targeting teachers only.

Data and methods

TIMSS was sponsored by the International Association for the Evaluation of Educational Achievement (IEA) and was conducted in 1996 (Lokan, Ford, & Greenwood, 1996). It set out to measure, across 45 countries, mathematics and science achievement among students at different ages and grades. In total, over half a million students from more than 30 000 classes in approximately 15 000 schools provided data. Not only were comprehensive mathematics and science tests developed for the study, there were questionnaires developed for students, their teachers and their school principals. Prior to the development of the tests, an extensive analysis of textbooks and curriculum documents was carried out. Mathematics and science curriculum developers from each country also completed questionnaires about the placement of and emphasis on a wide range of mathematics and science topics in their country’s curricula. Together the data provide a unique opportunity to examine an extensive range of contextual variables that influence mathematics and science achievement.

TIMSS investigated mathematics achievement at three stages of schooling with the following target populations:

* Population 1: adjacent grade levels containing the largest proportion of nine-year-old students at the time of testing;

* Population 2: adjacent grade levels containing the largest proportion of thirteen-year-old students at the time of testing; and

* Population 3: the final year of schooling.

This study uses data from the US and Australian samples of Population 2 students. For Population 2, the original TIMSS design specified a minimum of 150 randomly selected schools per population per country, with two classes randomly selected to participate from each of the adjacent grade levels within each selected school. However, due to the cost of collecting such data, most countries were unable to achieve this position, and the US and Australia were two of only three countries which selected and tested more than one class per grade level per school. The importance of the sampling design used in the US and Australia is that it enables differences between schools to be separated from differences between classes within schools. In this way, we are able to analyse school and classroom differences.

For the purposes of comparison, the analysis in the current paper is restricted to Grade 8 students and classes. The final sample numbers are presented in Table 1.

Variables

The main aim of this analysis of the TIMSS data was to compare for the US and for Australia the relationships between student achievement in mathematics and factors at the student, classroom and school levels. Table 2 provides details of the variables that were used in the analysis.

Student background variables. The sex of each student was recorded, as well as the number of people living in the student’s household. A variable representing socioeconomic status (SES) was computed as a weighted composite comprising the mother’s and father’s level of education, the number of books in the home and the number of possessions in the home. Language background was measured as the frequency with which English was spoken at home. Family formation was based on whether or not the student lived with one parent or both.

Student mediating variables A composite variable was derived to represent the student’s enjoyment of mathematics. This variable consisted of positive responses to five attitude prompts: `I usually do well in mathematics’, `I like mathematics’, `I enjoy learning mathematics’, `Mathematics is boring’, and `Mathematics is an easy subject’. A further variable was computed to represent student’s perceptions of the importance of mathematics. This variable was comprised of responses to the items: `Mathematics is important to everyone’s life’, `I would like a job involving mathematics’, `I need to do well in mathematics to get the job I want’, `I need to do well in mathematics to please my parent(s)’, `I need to do well in mathematics to get into the university/post-school course I prefer’, and `I need to do well in mathematics to please myself’. An additional variable was created representing the amount of time spent on mathematics homework. This was based on a scale from 0 to more than 4 hours per night.

Classroom variables A range of classroom variables was collected or derived for this analysis. The stream, track or set of the class was derived if setting was a practice used in the school to organise mathematics classes. Mean SES was derived at the class level. A variable was derived if the classrooms within schools in the data set had the same teacher. The background attributes of teachers–gender, number of years teaching and educational qualifications–were also controlled for. Estimates of the amount of homework teachers set for classes, the extent of their reliance on a prescribed textbook, and the amount of time they spent teaching mathematics were also derived.

School level variables Mean SES was derived for each school to provide a control for the social composition of the school. In addition, a measure of the school size was used, ranging from schools of less than 250 students through to schools of more than 1250 students. Average class sizes, time dedicated to mathematics teaching across a school year, and school climate measured by the levels of absenteeism and behavioural disturbances were also included. Explicit school policies relating to the selection of pupils (open admission from the surrounding area, academic selection of pupils) were also variables included in the analysis.

Method

This study looks at the effects of classrooms, teachers and schools after controlling for student-level factors. An appropriate procedure for doing this is hierarchical linear modelling or HLM (Bryk & Raudenbush, 1992). This procedure allows modelling of outcomes at several levels (e.g. student level, classroom level, school level), partitioning separately the variance at each level while controlling for the variance across levels.

In the present study, the interest is on variability within and between classrooms and schools. Two sets of analyses were undertaken to measure the levels of variation, one for the US and one for Australia. The first set modelled mathematics achievement of Grade 8 students in the United States. In the analyses, several models were tested each adding successively a new group or layer of variables. The first involved fitting a variance-components model to estimate the amount of variance due to the effects of students (level 1), within classrooms (level 2), within schools (level 3) by running the models without any explanatory variables. The second model introduced a group of student background variables comprising sex, socioeconomic status (SES), family size, birthplace of parents, language background, and family formation (single parent or intact family). The third model added a set of mediating variables to the student background variables. The mediating variables included attitudes towards mathematics, views on the importance of mathematics, and time spent on mathematics homework. The fourth model contained a set of classroom composition variables relating to mean SES, stream or track, and whether the classes in Grade 8 had the same teacher or not. The next model added a set of teacher variables including the sex of the teacher, qualifications, years of teaching experience, the amount of homework the teacher sets, the amount of time they spend teaching mathematics, and the amount of time in class they teach using a set textbook. The final model added several school-level factors including the mean SES of the school, school size, average class size, student selection policy (academically selective, open admission), time dedicated to mathematics teaching, and school climate measured by student absenteeism and level of behavioural disturbances.

By examining changes in the size of the variance components estimates after the addition of each group of variables, it was possible to measure the contribution of student, teacher, classroom and school-level factors to mathematics achievement. In this way, it was possible to estimate the extent to which factors linked to teachers rather than classroom composition and organisation shape differences in mathematics achievement and to what extent student-level and school-level factors influence achievement.

The second set of analyses was based on data for Australia. The same sequence of models was applied.

Results

Student, classroom and school variance in mathematics achievement Table 3 presents the results of the HLM analyses for the US and Table 4 presents the results for Australia. The variance components estimates are presented in the second column. The third column presents the percentages of variance (intraclass correlations) in mathematics achievement located at each of the levels–student, classroom and school. The final column contains the percentages of variance explained at each level after controlling for the different groups of variables.

As a first step, a fully unconditional (null) model was tested. This model, the equivalent of a one-way ANOVA with random effects, estimates variances in the outcome variable at the student, classroom and school levels. The results suggest for both the US and Australia considerable variation in mathematics achievement at the classroom and school levels. Over one-half (54.1 per cent) of the estimated variation in mathematics achievement in the US occurs at the student level. However differences between classrooms also account for a substantial amount of variance–33.8 per cent. Differences between schools accounted for the remaining 12.1 per cent of variance. This suggests a moderate though significant level of variation between schools.

The results for Australia show a smaller level of variance at the classroom (27.9 per cent) and school (10.4 per cent)levels, though the results suggest that differences between classrooms and between schools are an important source of variation in mathematics achievement.

The next step in the analysis involved adding the student background predictors (SES, gender, language background, family size, single parent family, birthplace of parents) to the model of mathematics achievement. This allowed differences between classrooms and schools to be adjusted for differences at the individual level. The results presented in column 4 show that differences in the background characteristics of students in the US accounted for 4.7 per cent of the estimated variance at the student level, 15.0 per cent of the variance between classrooms, and 19.5 per cent of the variance at the school level. The Australian results show a higher level of explained variance–7.4, 16.4 and 54.0 per cent, respectively. It suggests that student background factors explain more of the between-school variance in Australia than in the US.

Adding the student mediating variables (time spent on homework, attitudes towards mathematics, and views on the importance of maths) in the next step substantially increased the percentages of explained variance at the student level. When achievement is adjusted for the student background and mediating variables, the amount of variance explained at the student level increased to 12.0 per cent in the US and 19.3 per cent in Australia. At the classroom, level, the amount of variance explained increased only modestly to 15.7 per cent in the US and 27.6 per cent in Australia. The results suggest that, although the mediating variables are important to explaining student level variance, they do not add much to the understanding of classroom and school level variance.

The next step involved the inclusion of the classroom composition, variables–mean SES, high stream or track classroom, low stream or track classroom, non-streamed or tracked classroom, same teacher across classrooms. This further increases the percentage of variance explained at the classroom level. The between-classroom variance explained jumped from 15.7 per cent to 64.6 per cent in the US, and from 27.6 to 74.3 per cent in Australia. It suggests that classroom organisation and composition factors are important in explaining classroom differences in student achievement.

Teacher effects would appear to be quite small, at least based on the changes that occur after adding in the available teacher variables–years of teaching experience, sex of the teacher, qualifications, time spent teaching mathematics, textbook-based teaching methods, and amount of homework set. This group of variables increased the explained variance at the classroom level by only about 3 per cent in both the US and Australia. The school level variables also added little to the explained variances.

The school level variables add more to the explained variance in the US than they do in Australia. The combined effects of the mean SES of the school, school size, average class size, admissions policy, and features of school climate explain roughly 13 per cent of variance between schools in the US (13 per cent) and about 6 per cent in Australia.

Student, classroom and school factors shaping mathematics achievement Table 5 presents the results from the HLM analyses for the United States and Table 6 the results for Australia.

At the first level of analysis, shown in the first column of Table 5, it can be seen that all of the variables, other than family size, have a significant effect on achievement in mathematics for students in the US. As has been found in previous studies, gender has a significant negative effect on mathematics achievement. That is, Grade 8 girls’ achievement levels are still not equal to that of boys. Also, as has been found in previous studies, students from a higher SES background and those from two-parent rather than single-parent families tend to have higher achievement levels in mathematics. Language background is also important. Students from families that more often speak a language other than English at home tend to have lower levels of achievement than those where English is the main language.

For Australia, although Grade 8 girls tend not to do as well as boys in mathematics, the differences are not significant. Similarly there are not significant differences linked to family size or family formation. The most influential variables for Australian students are SES and language background. Students from higher SES origins achieve significantly higher than those from lower SES backgrounds. Students from families that more often speak a language other than English at home do significantly worse in mathematics than those where English is the main language.

The mediating variables–attitudes towards mathematics, perceived importance of mathematics, time spent on mathematics homework–have strong independent effects, at least in Australia (see column 3). They are influential predictors of mathematics achievement. But they not only have independent effects, they also transmit or relay some of the effects of the different student background variables. This is evident from the drop in the sizes of the estimates for SES and family formation when the mediating variables are included in the model.

The results for the mediating variables are weaker for students in the US. The estimates for time spent on mathematics homework and for attitudes towards mathematics are smaller than for Australian students. The estimate for perceived importance of mathematics is positive, though not significant. It suggests that the perceived importance of mathematics is a greater influence on mathematics achievement in Australia than in the US. This is supported by the differential increase in explained variance reported at the base of the tables. The figures show that, whereas the mediating variables increase the level of explained variance in Grade 8 mathematics achievement by approximately 14 per cent in Australia, they increase the level by only 3 per cent in the US.

In summary, the differences between males and females are greater in the US than in Australia. In the US, gender differences, SES and family formation have both a direct effect on achievement and a transmitted effect through their influence on attitudes to mathematics and amount of time spent on homework. These findings reinforce previous studies showing that student background has an effect, both directly and indirectly, on student achievement in mathematics. In Australia, SES and language background are important predictors of mathematics achievement, working independently as well as through their influence on attitudes towards mathematics, perceived importance of mathematics and time spent on homework.

The results presented in the previous section show that, as well as student level factors, classrooms and schools also matter. The next stages of the modelling investigate the effects of classroom variables on achievement.

Tables 5 and 6 show that for the US and for Australia tracking or streaming has a large impact on mathematics achievement. There is a strong positive effect for classes in the top band in schools with streaming or tracking policies. In the US, classes in the top track or stream gain 28 points on average over classes which are in the middle track or band. The advantage in Australia is larger at 38 points. Students in the US in the lowest track or band have significantly lower results than students in the middle track or band. Tracking or streaming clearly benefits those students in the higher band classes, but leads to significantly poorer achievement in lower band classes. The achievement in classes in the lower bands or streams is moderately, though significantly, lower than classes that are not streamed or set in Australia. In the US, however, the result for non-tracked or streamed classes is not much better than that for the bottom track or stream. There are differences in the number of classes that are tracked or streamed between the countries. In Australia, 48 per cent of classes were not streamed or tracked, compared with only about 20 per cent in the US.

Classroom social composition (mean SES) has strong independent effects on student achievement in mathematics, and this applies both in the US and Australia. In both countries, there are achievement advantages to being located in classrooms largely composed of students from higher SES backgrounds. The results show that the higher the mean SES composition of classes, the higher the achievement.

In the US, approximately 30 per cent of the sampled classes were taught by the same teacher in each school. In Australia, the rate was about 10 per cent. The results suggest that having the same teacher does not have any effect on the results for Australia or the US. This does not support the recent research on teacher effects which has suggested that it is teacher effects rather than other classroom factors that are the major influences on mathematics achievement. If this was the case, we might have expected smaller classroom differences where classes have the same teacher.

The classroom composition and organisation variables added substantially to the levels of explained variance in both countries. Addition of the pupil grouping variables and classroom composition factors increased the total variance explained from 13 to 34.7 per cent in the US, and from 24.8 to 39.7 per cent in Australia.

The next step in the analysis was to add the teacher attribute variables to the achievement models. Sex of the teacher and educational qualifications had no significant effect on student achievement. Teacher experience, as measured by years of teaching, had a small but significant positive effect in the US, suggesting that the more experienced teachers achieved better results. This did not apply in Australia.

In both countries, the results suggest that classes where teachers set more homework were associated with higher levels of achievement. In Australia, there was also a positive significant impact in classrooms where the amount of time teachers spent using a prescribed textbook was greater. The results suggest that, in classes where teachers use more traditional textbook-based methods, the results are better. This did not apply in the US where the effect was negative and significant, which suggests that the results were better where teachers used alternative methods. The teacher effect variables in both countries added only marginally to the levels of variance in mathematics achievement.

The addition of the school level factors–mean SES, school size, average class size, admissions policy, and length of time given to mathematics instruction, and school climate–also adds only a small amount to explaining total levels of variance in both countries. However these variables do contribute more to explaining school level variance in the US than in Australia. In the US, school level SES has a positive impact on mathematics achievement, which suggests that students in schools with a higher mean SES do better in mathematics than students in schools with lower levels of SES, other things equal. Social composition of the school influences mathematics achievement.

Discussion

What can we learn from the TIMSS data about differences in mathematics achievement? One thing we learn is that differences between classes and schools matter in both the US and Australia. Early studies examining patterns of student achievement in mathematics had concluded that schools have little impact above and beyond student intake factors. The results from TIMSS show, consistent with current research on school effectiveness, that not only do schools make a difference, but classrooms as well. There are strong classroom effects and modest school effects on mathematics achievement. These effects are linked to particular classroom and school level factors.

The pooling of pupil resources that are associated with the grouping of students–reflected by mean SES and stream or track–heavily influence mathematics achievement. In both the US and Australia, achievement is highest in those classes and schools with higher concentrations of students from middle-class families and students in the highest track or stream. Therefore the effects of residential segregation more broadly and school level pupil management policies more locally (policies such as setting or tracking) shape the contexts within which differences in mathematics learning and achievement develop. The findings support the view that such context setting factors are important influences. School level pupil management practices such as setting or streaming contribute to the classroom effects by shaping classroom composition. Within this context, the effects of teachers are quite modest, in contrast to the claims of other research. This is supported in the current research by the non-significant results in both countries linked to having the same teacher across different classrooms. Having the same teacher did not reduce, significantly, differences between classrooms, suggesting that composition factors and pupil grouping practices are far more influential.

Policies regarding pupil management are critical. Schools which formally group students according to mathematics achievement or ability promote differences in mathematics achievement. The benefits of this practice are large for students who enter higher band or track classes. They receive substantial gains in achievement. The cost is for those students in the lower band or stream classes. They have significantly lower level of achievement compared with their top-streamed peers in the US and also their unstreamed peers in Australia. In Australia, in terms of mathematics achievement, it is better for students to be in a school that does not stream or track mathematics classrooms than in a bottom stream or track in a school where streaming or tracking is policy. It suggests that the different learning environments created through selective pupil grouping may work to inhibit student progress in the bottom streams and accelerate it for those in the top streams.

These findings do not support the view of recent research, which argues that the differences in quality of teachers and teacher effectiveness account for much of the classroom variation in mathematics achievement. Rather they support an alternative explanation, that the types of pupil grouping practices that schools employ shape the classroom learning environments in ways that affect student progress and student achievement, and these kinds of differences more significantly influence classroom effects. By this, it is not suggested that the quality of teachers does not matter or that all teachers have the same effectiveness. Teachers do matter. In the US, more experienced teachers promote higher levels of achievement. The approach they take to homework, measured by the amount of time they set for homework, has a modest but significant effect on achievement, after controlling for other factors. Those more often using less traditional textbook approaches also promote higher levels of achievement. By contrast, in Australia, teachers using more traditional approaches appeared to enhance achievement. Although these teacher effects have an impact, what the TIMSS results suggest is that the organisational and compositional features of classrooms have a more marked impact on mathematics achievement.

Keywords

curriculum policy

international studies

mathematics achievement

school effectiveness

socioeconomic influences

teacher effectiveness

Table 1 Sample sizes

United States Australia

Students 7087 6916

Classrooms 348 309

Schools 183 158

Table 2 Student, classroom and school variables

Variable Description

Student level

Student background variables

Sex Student’s gender

Language background Level of skill in language of test

Family size Number of people living in student’s

home

Socioeconomic status A composite variable representing

family wealth, parents’ education and

number of books in the home

Birthplace of parents Both parents born outside the United

States or Australia

Single parent family Student lives with one parent

Student mediating variables

Time spent on homework Self-reported assessment of length of

time spent doing mathematics home

work

Attitudes towards mathematics A composite variable measuring

attitudes to mathematics.

Perceived importance of A composite variable reflecting the

mathematics perceived importance of mathematics

to the student.

Classroom level

Classroom composition variables

Mean SES Average SES for the class

Grouping practice

High band Highest band or track class

Middle band Middle band or track class

Low band Lowest band or track class

No band Setting, streaming or tracking is

not used

Same teacher or not Same teacher for other class(es)

participating in the survey

Classroom teacher variables

Sex Teacher’s gender

Educational qualifications Teacher’s qualifications

Years teaching Number of years teaching

Teaching practices

Homework set Estimate of amount of homework the

teacher sets

% time teaching in maths Estimate of time spent teaching

mathematics

Amount of time using text- Estimate of amount of teaching time

book focused on prescribed textbook

School level

Mean SES Average SES for the school

School size Number of students enrolled

Class size Average class size in maths

Time on maths Time dedicated to maths teaching

across a school year

Pupil intake policy

Academically selective Intake of students is based on

academic selection

Open admission Intake is not based on academic

selection and is mainly based on

those who live in the local area

Other Selection of intake is based on non-

academic criteria

School climate

Behavioural disturbances Percentage of students who misbehave

in class

Absenteeism Percentage of students who are absent

without an excuse

Table 3 Variance in Grade 8 mathematics achievement explained by

three-level HLM models: United States, population 2, TMSS

Variance

Variance between

levels

%

Variance within classrooms (level 1 variance) 4685.8 54.1

After controlling for:

Student background variables 4466.3

Student mediating variables 4124.1

Variance between classrooms (level 2 variance) 2924.5 33.8

After controlling for:

Student background variables 2485.8

Student mediating variables 2465.0

Classroom composition variables 1035.1

Classroom teacher variables 891.7

Variance between schools (level 3 variance) 1043.1 12.1

After controlling for:

Student background variables 840.1

Student mediating variables 935.4

Classroom composition variables 495.1

Classroom teacher variables 559.7

School-level variables 420.5

Variance

explained

at each

level

%

Variance within classrooms (level 1 variance)

After controlling for:

Student background variables 4.7

Student mediating variables 12.0

Variance between classrooms (level 2 variance)

After controlling for:

Student background variables 15.0

Student mediating variables 15.7

Classroom composition variables 64.6

Classroom teacher variables 69.5

Variance between schools (level 3 variance)

After controlling for:

Student background variables 19.5

Student mediating variables 10.4

Classroom composition variables 52.5

Classroom teacher variables 46.3

School-level variables 59.7

Table 4 Variance in Grade 8 mathematics achievement explained by

three-level HLM models: Australia, population 2, TIMSS

Variance

between

levels

Variance %

Variance w/thin classrooms (level 1 variance) 5415.6 61.7

After controlling for:

Student background variables 5014.2

Student mediating variables 4370.6

Variance between classrooms (level 2 variance) 2446.6 27.9

After controlling for:

Student background variables 2045.7

Student mediating variables 1771.4

Classroom composition variables 627.8

Classroom teacher variables 541.7

Variance between schools (level 3 variance) 908.3 10.4

After controlling for:

Student background variables 417.4

Student mediating variables 451.6

Classroom composition variables 289.0

Classroom teacher variables 258.3

School-level variables 200.9

Variance

explained at

each level

%

Voriance w/thin classrooms (level 1 variance)

After controlling for:

Student background variables 7.4

Student mediating variables 19.3

Variance between classrooms (level 2 variance)

After controlling for:

Student background variables 16.4

Student mediating variables 27.6

Classroom composition variables 74.3

Classroom teacher variables 77.9

Variance between schools (level 3 variance)

After controlling for:

Student background variables 54.0

Student mediating variables 50.3

Classroom composition variables 68.2

Classroom teacher variables 71.6

School-level variables 77.9

Table 5 HLM estimates of Grade 8 mathematics achievement: United

States, population 2, TIMSS

Level 1 Level 1

model model

Student Student

background mediating

variables variables

Intercept 488.3 *** 488.6 ***

Student-level variables

Background variables

Female -10.7 *** -9.2 ***

SES 11.1 *** 9.9 ***

Language -11.2 *** -11.3 ***

Parents not born in United States 6.4 ** 4.8 *

Family size -1.0 * -1.2 *

Single parent family -4.3 ** -3.1 *

Mediating variables

Time spent doing homework -3.7 ***

Positive attitudes towards maths 7.0 ***

Perceived importance of maths 0.4

Classroom level variables

Classroom composition

Mean SES

Top stream or track

Bottom stream or track

No streaming or tracking

Same teacher

Teacher attributes

Sex of the teacher

Educational qualifications

Years in teaching

Amount of homework set

% time teaching maths

Amount of time using textbook

School level variables

SES

School size

Average class size

Academically selective

Open admission

Time dedicated to maths teaching

Behavioural disturbances

Absenteeism

Total variance explained

Level 1 (61.T) 10.0 13.0

Level 2 (27.9)

Level 3 (10.4)

Level 2 Level 2

model model

Classroom Classroom

composition teacher

variables variables

Intercept 489.5 *** 489.4 ***

Student-level variables

Background variables

Female -9.2 *** -9.1 ***

SES 7.8 *** 7.7 ***

Language -10.9 *** -10.7 ***

Parents not born in United States 5.7 ** 5.5 *

Family size -0.8 -0.8

Single parent family -2.9 * -3.0 *

Mediating variables

Time spent doing homework -4.3 *** -4.4 ***

Positive attitudes towards maths 7.0 *** 7.0 ***

Perceived importance of maths 0.4 0.4

Classroom level variables

Classroom composition

Mean SES 23.4 *** 22.7 ***

Top stream or track 28.2 *** 27.7 ***

Bottom stream or track -20.6 *** -22.4 ***

No streaming or tracking -16.8 ** -16.7 **

Same teacher 5.5 4.4

Teacher attributes

Sex of the teacher 4.3

Educational qualifications -2.6

Years in teaching 0.6 **

Amount of homework set 2.3 ***

% time teaching maths 0.0

Amount of time using textbook -2.3 *

School level variables

SES

School size

Average class size

Academically selective

Open admission

Time dedicated to maths teaching

Behavioural disturbances

Absenteeism

Total variance explained

Level 1 (61.T)

Level 2 (27.9) 34.7 35.6

Level 3 (10.4)

Level 3

model

School

variables

Intercept 489.4 ***

Student-level variables

Background variables

Female -9.1 ***

SES 7.8 ***

Language -10.4 ***

Parents not born in United States 6.2 *

Family size -0.8

Single parent family -2.9 *

Mediating variables

Time spent doing homework -4.4 ***

Positive attitudes towards maths 6.9 ***

Perceived importance of maths 0.4

Classroom level variables

Classroom composition

Mean SES 29.5 ***

Top stream or track 29.2 ***

Bottom stream or track -22.7 **

No streaming or tracking -18.5 **

Same teacher 4.6

Teacher attributes

Sex of the teacher 4.3

Educational qualifications -2.5

Years in teaching 0.6 **

Amount of homework set 2.7 ***

% time teaching maths 0.0

Amount of time using textbook -3.7 *

School level variables

SES 10.2 ***

School size 0.0

Average class size -0.9

Academically selective -2.6

Open admission 11.4

Time dedicated to maths teaching 0.0

Behavioural disturbances -0.3

Absenteeism -0.7

Total variance explained

Level 1 (61.T)

Level 2 (27.9)

Level 3 (10.4) 37.2

* Significant at the .10 level;

** Significant at the .05 level;

*** Significant at the .01 level

Table 6 HLM estimates of Grade 8 mathematics achievement: Australia,

population 2, TIMSS

Level 1 Level 1

model model

Student Student

background mediating

variables variables

Intercept 516.6 *** 516.0 ***

Student level variables

Background variables

Female -2.1 1.4

SES 8.7 *** 7.5 ***

Language -14.9 *** -16.7

Parents not born in Australia 2.0 0.7

Family size -1.2 -1.0

Single parent family -1.1 -0.4

Mediating variables

Time spent doing homework -10.3 ***

Positive attitudes towards maths 11.3 ***

Perceived importance of maths 2.4 ***

Classroom level variables

Classroom composition

Mean SES

Top stream

Low stream

No stream

Same teacher

Teacher attributes

Sex of the teacher

Educational qualifications

Years in teaching

Amount of homework set

Time teaching maths

Amount of time using textbook

School level variables

SES

School size

Average class size

Academically selective

Open admission

Time dedicated to maths teaching

Behavioural disturbances

Absenteeism

Total variance explained

Level 1 (61.7) 14.7 24.8

Level 2 (27.9)

Level 3 (10.4)

Level 2 Level 2 Level 3

model model model

Classroom Classroom

composition teacher School

variables variables variables

Intercept 516.4 *** 516.4 *** 516.5 ***

Student level variables

Background variables

Female 0.9 0.9 1.0

SES 6.6 *** 6.6 *** 6.6 ***

Language -16.3 *** -16.3 *** -16.0 ***

Parents not born in Australia 1.2 0.9 1.2

Family size -0.9 -0.8 -0.8

Single parent family -0.8 -0.8 -0.8

Mediating variables

Time spent doing homework -11.7 *** -12.0 *** -11.9 ***

Positive attitudes towards maths 11.2 *** 11.2 *** 11.2 ***

Perceived importance of maths 2.4 *** 2.4 *** 2.4 ***

Classroom level variables

Classroom composition

Mean SES 24.6 *** 21.4 *** 22.5 ***

Top stream 38.6 *** 35.6 *** 34.6 ***

Low stream -45.4 *** -41.1 *** -37.3 ***

No stream 0.2 0.9 0.8

Same teacher -1.5 -1.1 -0.2

Teacher attributes

Sex of the teacher -0.0 -0.0

Educational qualifications 0.4 0.5

Years in teaching 0.3 0.3

Amount of homework set 3.7 *** 3.8 ***

Time teaching maths 0.0 0.0

Amount of time using textbook 3.9 *** 4.1 ***

School level variables

SES 1.2

School size 0.0

Average class size -0.4

Academically selective 3.8

Open admission -0.8

Time dedicated to maths teaching 0.0

Behavioural disturbances -0.5

Absenteeism -0.1

Total variance explained

Level 1 (61.7)

Level 2 (27.9) 39.7 41.0

Level 3 (10.4) 41.7

* Significant at the .10 level;

** Significant at the .05 level;

*** Significant at the .01 level

Acknowledgement

An earlier version of this paper was presented at the annual meeting of the American Educational Research Association, Seattle, April 10-14, 2001.

References

Affrassa, T.H. & Keeves, J.P. (1999), Student-level factors that influence mathematics achievement of Australian students: A path analysis with comparisons over time. Paper presented at the Annual Conference of AARE, December 1999, Melbourne.

Anderson, L. W., Ryan, D. W., & Shapiro, B. J. (1989). The IEA classroom environment study. Oxford: Pergamon Press.

Bosker, R. J. & Witziers, B. (1996). The magnitude of school effects or does it really matter which school a student attends? Paper presented at the Annual Meeting of the American Educational Research Association, New York.

Bryk, A.S. & Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

Coleman, J. S., Campbell, E. Q., Hobson, C. J., Partland, J., Mood, A. M., Weinfeld, F. D., & York, R. L. (1966). Equality of educational opportunity. Washington, DC: US Government Printing Office.

Hay Mcber. (2000). Research into teacher effectiveness: A model of teacher effectiveness. Report commissioned by the Department for Education and Employment. London: Department for Education and Employment.

Hill, P.W. (1994). The contribution teachers make to school effectiveness. In P.W. Hill, P. Holmes-Smith, K. Rowe, & V.J. Russell. (Eds.), Selected reports and papers on findings from the first phase of the Victorian Quality Schools Project. Melbourne: University of Melbourne, Centre for Applied Educational Research.

Hill, P.W. & Rowe, K.J. (1996). Multilevel modelling in school effectiveness research. School Effectiveness and School Improvement, 7, 1-34.

Hill, P.W. & Rowe, K.J. (1998). Modelling student progress in studies of educational effectiveness. School Effectiveness and School Improvement, 9(3), 310-333. Hill, P.W., Rowe, K.J., Holmes-Smith, P., & Russell, V.J. (1996). The Victorian Quality Schools Project: A study of school and teacher effectiveness (Report, Vol. 1). Melbourne: University of Melbourne, Centre for Applied Educational Research.

Jencks, C., Smith, M., Acland, H., Bane, M., Cohen, D., Gintis, H., Heyns, B., & Michelson, S. (1972). Inequality: A reassessment of the effect of family and schooling in America. New York: Basic Books.

Lamb, S.P. (1997). Access to level of mathematics study in high school: Social area and school differences. In People in mathematics education. Conference Proceedings of the twentieth annual meeting of the Mathematics Education Research Group of Australasia (pp. 286-293). Aotearoa, NZ: MERGA.

Lamb, S. & Fullarton, S. (2000). Classroom and teacher effects in mathematics achievement: Results from TIMMS. In Mathematics education beyond 2000. Conference proceedings of the twenty-third annual meeting of the Mathematics Education Research Group of Australasia. Fremantle: MERGA.

Larkin, A. I. & Keeves, J. P. (1984). The class size question: A study at different levels of analysis. Hawthorn, Vic.: Australian Council for Educational Research.

Lee, V. E. & Smith, J. B. (1997). High school size: Which works best and for whom? Education Evaluation and Policy Analysis, 19(3), 205-228.

Lokan, J., Ford, P., & Greenwood, L. (1996). Mathematics & science on the line: Australian Junior Secondary Students’ Performance in the Third International Mathematics and Science Study (TIMSS Australia Monograph, No. 1). Melbourne: ACER.

Mortimore, P., Sammons, P., Stoll, L., Lewis, D., & Ecob, R. (1988). School matters: The junior years. Somerset: Open Books.

National Center for Education Standards. (2000). NAEP 1996: Trends in academic progress. Washington, DC: US Department of Education.

Nuttall, D., Goldstein, H., Prosser, R., & Rasbash, J. (1989). Differential school effectiveness. International Journal of Educational Research, 13(7), 769-776.

Raudenbush, S. W. & Willms, J. D. (Eds.). (1991). Schools, classrooms and pupils: International studies of schooling from a multilevel perspective. New York: Academic Press.

Rowe, K.J. & Hill, P.W. (1994). Multilevel modelling in school effectiveness research: How many levels? In P.W. Hill, P. Holmes-Smith, K. Rowe, & V.J. Russell (Eds.), Selected reports and papers on findings from the first phase of the Victorian Quality Schools Project. Melbourne: University of Melbourne, Centre for Applied Educational Research.

Scheerens, J. (1993). Basic school effectiveness research: Items for a research agenda. School Effectiveness and School Improvement, 4(1), 17-36.

Scheerens, J., Vermeulen, C. J. A. J., & Pelgrum, W. J. (1989). Generalizability of instructional and school effectiveness indicators across nations. International Journal of Educational Research, 13(7), 789-799.

Schmidt, W. H., McKnight, C. C., Cogan, L. S., Jakwerth, P. M., & Houang, R. T. (1999). Facing the consequences: Using TIMSS for a closer look at US mathematics and science education. Dordrecht: Kluwer.

Smith, D. & Tomlinson, S. (1989). The school effect. London: Policy Studies Institute.

Dr Stephen Lamb is a Senior Research Fellow in the Department of Education Policy and Management at the University of Melbourne, Parkville, Victoria 3010. Dr Sue Fullarton is a Senior Research Fellow in the Policy Research Division, Australian Council for Educational Research, Private Bag 55, Camberwell, Victoria 3124.

COPYRIGHT 2002 Australian Council for Educational Research

COPYRIGHT 2008 Gale, Cengage Learning