What Large-Scale,Survey Research Tells Us about Teacher Effects on Student Achievement: Insights from the Prospects Study of Elementary Schools

Abstract

This paper discusses conceptual and methodological issues that arise when educational researchers use data from large-scale, survey research to examine the effects of teachers and teaching on student achievement. Using data from Prospects: The Congressionally Mandated Study of Educational Growth and Opportunity 1991–1994, we show that researchers’ use of different statistical models has led to widely varying interpretations about the overall magnitude of teacher effects on student achievement. However, we conclude that in well-specified models of academic growth, teacher effects on elementary school students’ growth in reading and mathematics achievement are substantial (with d-type effect sizes ranging from .72 to .85). We also conclude that various characteristics of teachers and their teaching account for these effects, including variation among teachers in professional preparation and content knowledge, use of teaching routines, and patterns of content coverage, with effect sizes for variables measuring these characteristics of teachers and their teaching showing d-type effect sizes in the range of .10. The paper concludes with an assessment of the current state of the art in large-scale, survey research on teaching. Here, we conclude that survey researchers must simultaneously improve their measures of instruction while paying careful attention to issues of causal inference.

Get full access to this article

View all access options for this article.

References

Alexander

, & Entwisle

(1992). Summer setback: Race, poverty, school composition, and mathematics achievement in the first two years of school. American Sociological Review, 57(1), 72–84.

Alexander

, & Entwisle

(2001). Keep the faucet flowing: Summer learning and home environment. American Educator, 25(3), 10–15.

Biemer

, Groves

, Lyberg

, Mathiowetz

, & Sudman

(1991). Measurement errors in surveys. New York: John Wiley & Sons.

Boruch

R. F.

(1997). Randomized experiments for planning and evaluation: A practical guide (Applied Social Research Methods Series, Vol. 44) Thousand Oaks, CA: Sage Publications.

Brewer

D. J.

, & Goldhaber

D. D.

(2000). Improving longitudinal data on student achievement: Some lessons from recent research using NELS:88. In Grissmer

D. W.

, & Ross

J. M.

(Eds.), Analytic issues in the assessment of student achievement (pp. 169–188). Washington, DC: U.S. Department of Education.

Brewer

D. J.

, & Stasz

(1996). Enhancing opportunity to learn measures in NCES data. In From data to information: New directions for the National Center for Education Statistics (NCES 96-901, pp. 3-1–3-28). Washington, DC: U.S. Government Printing Office.

Brophy

J. E.

, & Good

(1986). Teacher behavior and student achievement. In Wittrock

M. C.

(Ed.), Handbook of research on teaching (3rd ed., pp. 328–375). New York: Macmillan.

Bryk

A. S.

, & Raudenbush

S. W.

(1992). Hierarchical linear models: Applications and data analysis methods. Newbury Park, CA: Sage.

Bryk

A. S.

, Raudenbush

S. W.

, Cheong

Y. F.

, & Congdon

(2000). HLM 5: Hierarchical linear and nonlinear modeling. Lincolnwood, IL: Scientific Software International.

10.

Burstein

, McDonnell

, Van Winkle

, Ormseth

, Mirocha

, & Guiton

(1995). Validating national curriculum indicators. Santa Monica, CA: RAND.

11.

Calfee

, & Calfee

K. H.

(1976). Beginning teacher evaluation study: Phase II, 1973–74, final report: Volume III.2. Reading and mathematics observation system: Description and analysis of time expenditures. Washington, DC: National Institute of Education (ERIC Document Reproduction Service No. ED127367)

12.

Camburn

, Correnti

, & Taylor

(2000). Using qualitative techniques to assess the validity of teachers’ responses to survey items. Paper presented at the meeting of the American Educational Research Association, New Orleans, LA.

13.

Camburn

, Correnti

, & Taylor

(2001). Examining differences in teachers’ and researchers’ understanding of an instructional log. Paper presented at the meeting of the American Educational Research Association, Seattle, WA.

14.

Chaney

(1994). The accuracy of teachers’ self-reports on their postsecondary education: Teacher transcript study, Schools and staffing survey (NCES 94-04). Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics.

15.

Chiang

F.-S.

(1996). Teacher's ability, motivation and teaching effectiveness. Unpublished doctoral dissertation, University of Michigan, Ann Arbor.

16.

Cohen

D. K.

, Raudenbush

S. W.

, & Ball

D. L.

(in press). Resources, instruction, and research. In Boruch

R. F.

, & Mosteller

F. W.

(Eds.), Evidence matters: Randomized trials in educational research. Washington, DC: Brookings Institution.

17.

Coleman

J. S.

, Campbell

, Hobson

, McPartland

, Mood

, Weinfeld

, & York

(1966). Equality of educational opportunity. Washington, DC: U.S. Government Printing Office.

18.

Cooley

W. W.

, & Leinhardt

(1980). The instructional dimensions study. Educational Evaluation and Policy Analysis, 2(1), 7–25.

19.

Darling-Hammond

, Wise

A. E.

, & Klein

S. P.

(1995). A license to teach: Building a profession for 21st-century schools. San Francisco: Westview Press.

20.

Deng

(1995). Estimating the reliability of the teacher questionnaire used in the Teaching and Learning to Teach (TELT) study (Technical series 95-1) East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED 392 750)

21.

Dunkin

, & Biddle

(1974). The study of teaching. New York: Holt, Rhinehart & Winston.

22.

Elias

P. J.

, Hare

, & Wheeler

(1976). Beginning teacher evaluation study: Phase II, 1973–74, final report: Volume V.5. The reports of teachers about their mathematics and reading instructional activities. Washington, DC: National Institute of Education. (ERIC Document Reproduction Service No. ED127374)

23.

Ferguson

R. F.

, & Brown

(2000). Certification test scores, teacher quality, and student achievement. In Grissmer

D. W.

, & Ross

J. M.

(Eds.), Analytic issues in the assessment of student achievement (pp. 133–156). Washington, DC: U.S. Department of Education.

24.

Fetters

W. B.

, Stowe

P. S.

, & Owings

J. A.

(1984). High school and beyond, a national longitudinal study for the 1980s, quality of responses of high school students to questionnaire items (NCES 84-216). Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics.

25.

Gamoran

, Porter

, Smithson

, & White

(1997). Upgrading high school mathematics instruction: Improving learning opportunities for low-achieving, low-income youth. Educational Evaluation and Policy Analysis, 19(4), 325–338.

26.

Gage

N. L.

, & Needels

M. C.

(1989). Process-product research on teaching: A review of criticisms. Elementary School Journal, 89, 253–300.

27.

Greenwald

, Hedges

L. V.

, & Laine

R. D.

(1996). The effect of school resources on student achievement. Review of Educational Research, 66, 361–396.

28.

Groves

R. M.

(1987). Research on survey data quality. Public Opinion Quarterly, 51, 156–172.

29.

Groves

R. M.

(1989). Survey errors and survey costs. New York: John Wiley & Sons.

30.

Karweit

(1985). Should we lengthen the school term? Educational Researcher, 14, 9–15.

31.

Kennedy

, Ball

, & McDiarmid

(1993). A study package for examining and tracking changes in teachers’ knowledge (Technical series 93-1) East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED 359 170)

32.

Kerckhoff

A. C.

(1983). Diverging pathways: Social structure and career deflections. New York: Cambridge University Press.

33.

Krosnick

J. A.

(1999). Survey research. Annual Review of Psychology, 50, 537–567.

34.

Lambert

N. M.

, & Hartsough

C. S.

(1976). Beginning teacher evaluation study: Phase II, 1973–74, final report: Volume III.1. APPLE observation variables and their relationship to reading and mathematics achievement. Washington, DC: National Institute of Education. (ERIC Document Reproduction Service No. ED127366)

35.

Leighton

, Mullens

, Turnbull

, Weiner

, & Williams

(1995). Measuring instruction, curriculum content, and instructional resources: The status of recent work (NCES Working Paper No. 1995-11). Washington, DC: U.S. Department of Education.

36.

Lipsey

M. W.

, & Wilson

D. B.

(1993). The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist, 48(12), 1181–1209.

37.

Mayer

(1999). Measuring instructional practice: Can policymakers trust survey data? Educational Evaluation and Policy Analysis, 21(1), 29–45.

38.

Monk

D. H.

(1994). Subject area preparation of secondary mathematics and science teachers and student achievement. Economics of Education Review, 13(2), 125–45.

39.

Mullens

(1995). Classroom instructional processes: A review of existing measurement approaches and their applicability for the teacher follow-up survey (NCES Working Paper No. 1995-15). Washington, DC: U.S. Department of Education.

40.

Mullens

, & Gayler

(1999). Measuring classroom instructional processes: Using survey and case study field test results to improve item construction (NCES Working Paper No. 1999-08). Washington, DC: U.S. Department of Education.

41.

Mullens

, & Kasprzyk

(1996). Using qualitative methods to validate quantitative survey instruments. In 1996 Proceedings of the Section on Survey Research Methods (pp. 638–643). Alexandria, VA: American Statistical Association.

42.

Mullens

, & Kasprzyk

(2000). Validating item responses on self-report teacher surveys. In Selected papers on education surveys: Papers presented at the 1998 and 1999 ASA and 1999 AAPOR meetings (NCES Working Paper 2000-04). Washngton, DC: U.S. Department of Education.

43.

Porter

A. C.

, Kirst

, Osthoff

, Smithson

, & Schneider

(1993). Reform up close: An analysis of high school mathematics and science classrooms. Madison: University of Wisconsin, Wisconsin Center for Education Research.

44.

Raudenbush

S. W.

(1995). Hierarchical linear models to study the effects social context on development. In Gottman

J. M.

(Ed.), The analysis of change (pp. 165–202). Mahwah, NJ: Lawrence Earlbaum Associates.

45.

Raudenbush

S. W.

, & Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis (2nd. ed.) Thousand Oaks, CA: Sage.

46.

Raudenbush

, Bryk

, Cheong

, & Congdon

(2000). HLM 5 hierarchical linear and nonlinear modeling. Illinois: Scientific Software International.

47.

Raudenbush

S. W.

, Hong

G. L.

, & Rowan

(2002). Studying the causal effects with application to primary school mathematics (Working Paper) Ann Arbor: University of Michigan, Consortium for Policy Research in Education, Study of Instructional Improvement.

48.

Rogosa

(1995). Myths and methods: Myths about longitudinal research plus supplemental questions. In Gottman

J. M.

(Ed.), The analysis of change (pp. 3–66). Mahwah, NJ: Lawrence Earlbaum Associates.

49.

Rosenthal

(1994). Parametric measures of effect size. In Cooper

, & Hedges

L. V.

(Eds.), The handbook of research synthesis (pp. 231–244). New York: Russell Sage Foundation.

50.

Rowan

(1999). The task characteristics of teaching: Implications for the organizational design of schools. In Bernhardt

, Hedley

C. N.

, Cattaro

, & Svolopoulos

(Eds.), Curriculum leadership for the 21st century. Cresskill, NY: Hampton Press.

51.

Rowan

, Chiang

F.-S.

, & Miller

R. J.

(1997). Using research on employees’ performance to study the effects of teachers on students’ achievement. Sociology of Education, 70, 256–284.

52.

Rowan

, Schilling

, Ball

D. L.

, & Miller

(2001). Measuring teachers’ pedagogical content knowledge in surveys: An exploratory study (Research Note S-2) Ann Arbor: University of Michigan, Consortium for Policy Research in Education, Study of Instructional Improvement.

53.

Salvucci

, Walter

, Conley

, Fink

, & Mehrdad

(1997). Measurement error studies at the National Center for Education Statistics (NCES 97-464). Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics.

54.

Sanders

(1998). Value-added assessment. The School Administrator, 55(11), 24–32.

55.

Sanders

, & Horn

S. P.

(1994). The Tennessee value-added assessment system (TVAAS): Mixed-model methodology in educational assessment. Journal of Personnel Evaluation in Education, 8, 299–311.

56.

Scheerens

, & Bosker

(1997). The foundations of educational effectiveness. New York: Pergamon.

57.

Scherpenzeel

, & Saris

W. E.

(1997). The validity and reliability of survey questions: A meta-analysis of MTMM studies. Sociological Methods & Research, 25(3), 341–383.

58.

Shavelson

R. J.

, & Dempsey-Atwood

(1976). Generalizability of measures of teaching behavior. Review of Educational Research, 46, 553–611.

59.

Shavelson

R. J.

, Webb

N. M.

, & Burstein

(1986). Measurement of teaching. In Wittrock

(Ed.), Handbook of research on teaching (3rd ed., pp. 50–91). New York: MacMillan.

60.

Shulman

L. S.

(1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14.

61.

Smithson

J. L.

, & Porter

A. C.

(1994). Measuring classroom practice: Lessons learned from efforts to describe the enacted curriculum—The reform up close study. Madison: University of Wisconsin, Consortium for Policy Research in Education.

62.

Stedman

L. C.

(1997). International achievement differences: An assessment of a new perspective. Educational Researcher, 26(3), 4–15.

63.

Stodolsky

S. S.

(1988). The subject matters: Classroom activity in math and social studies. Chicago: University of Chicago Press.

64.

Stoolmiller

, & Bank

(1995). Autoregressive effects in structural equation models: We see some problems. In Gottman

J. M.

(Ed.), The analysis of change (pp. 261–276). Mahwah, NJ: Lawrence Earlbaum Associates.

65.

Sudman

, & Bardburn

N. M.

(1982). Asking questions: A practical guide to questionnaire design. San Francisco: Jossey-Bass.

66.

Sudman

, Bradburn

N. M.

, & Schwarz

(1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.

67.

Whittington

(1998). How well do researchers report their measures? An evaluation of measurement in published educational research. Educational and Psychological Measurement, 58(1), 21–37.