This analysis contributes to efforts to improve the use and understanding of survey data in education policy research by asking: How different are student and teacher reports of classroom instruction? Do student, class, or teacher characteristics account for any of the differences? Using National Assessment of Education Progress (NAEP) data, we compare the responses of middle-school students and their teachers to the same questions about mathematics instruction. We found low correlations and small significant mean differences between student and teacher reports; we also found that student reports are sensitive to key student and class variables, most notably to individual and class achievement, student race/ethnicity, income, parent education, and motivation. Implications for using student survey data in policy research are discussed.
Alexander, K.L., Entwisle, D.R., & Olson, L.S. ( 2001). Schools, achievement, and inequality: A seasonal perspective . Educational Evaluation and Policy Analysis, 25(2), 171-191.
2.
Aquilino, W.S. ( 1994). Interview mode effects in drug surveys. Public Opinion Quarterly, 58(2), 210-240.
3.
Aquilino, W.S. ( 1998). Effects of interview mode on measuring depression in younger adults. Journal of Official Statistics, 14(1), 15-30.
4.
Aubrecht, J.D., Hanna, G.S., & Hoyt, D.P. ( 1986). A comparison of high school student ratings of teacher effectiveness with teacher self-ratings: Factor analysis and multitraitmultimethod analysis. Educational and Psychological Measurement, 46, 223-231.
5.
Berends, M. ( 2000). Teacher-reported effects of New American Schools design: Exploring relationships to teacher background and school context. Education Evaluation and Policy Analysis, 22(1), 65-82.
6.
Bidwell, C., Frank, K., & Quiroz, P. ( 1997). Teacher types, workplace controls, and the organization of schools. Sociology of Education, 70, 285-307.
7.
Biemer, P., Groves, R., Lyberg, L., Mathiowetz, N. , & Sudman, S. (Eds.). ( 1991). Measurement errors in surveys. New York: Wiley.
8.
Blackburn, R.T., & Clark, M.J. ( 1975). An assessment of faculty performance: Some correlations between administrator, colleague, student and self-ratings. Sociology of Education, 48(2), 242-256.
9.
Bodilly, S. ( 1996). Lessons from New American Schools Development Corporation’s demonstration phase. Santa Monica, CA: RAND.
10.
Burstein, L., McDonnell, L.M., Van Winkle, J., Ormseth, T., Mirocha, J., & Guitton, G. ( 1995). Validating national curriculum indicators. Santa Monica, CA: RAND.
11.
Carpenter, T.P., Fennema, E., Peterson, P.L., Chiang, C., & Loef, M. ( 1989). Using knowledge of children’s mathematics thinking in classroom teaching: An experimental study. American Educational Research Journal, 26(4), 499-531.
12.
Carroll, J. ( 1963). A model of school learning. Teachers College Record, 64, 722-733.
13.
Clark, K.E., & Keller, R.J. ( 1954). Student ratings of college teaching. In R. Eckert (Ed.), A university looks at its program. Minneapolis: University of Minnesota Press.
14.
Cobb, P., Wood, T., Yackel, E., Nicholls, J., Wheatley, G., Trigatti, B., et al. (1991). Assessment of a problem-centered second-grade mathematics project. Journal for Research in Mathematics Education , 22(1), 3-29.
15.
Cohen, D. ( 1990). A revolution in one classroom: The case of Mrs. Oublier . Educational Evaluation and Policy Analysis, 12(3), 311-329.
16.
Cohen, S.A., & Berger, W.G. ( 1970). Dimensions of students’ ratings of college instructors underlying subsequent achievement on course examinations. Proceedings of the 178th Annual Convention of the American Psychological Association, 5, 605-606.
17.
Cohen, D. K., McLaughlin, M. W., & Talbert, J. E. (Eds.). (1993). Teaching for understanding: Challenges for policy and practice. San Francisco: Jossey-Bass .
18.
Comaford, C. ( 1951). Changes in student rating of instructor. Unpublished research paper, University of Minnesota.
19.
Costin, F. ( 1966). Intercorrelations between students’ and course chairman’s ratings of instructors. Chicago: University of Illinois, Division of General Studies.
20.
Costin, F., Greenough, W.T., & Menges, R. ( 1971). Student ratings of college teaching: Reliability, validity, and usefulness. Review of Educational Research, 41(5), 511-535.
21.
Council of Chief State School Officers (CCSSO), Wisconsin Center for Education Research (WCER), & Eleven-State Collaborative (ESC). ( 2000). Using data on enacted curriculum in mathematics and science: Sample results of a study of classroom practices. Washington, DC: Author.
22.
Darling-Hammond, L., & McLaughlin, M.W. (1995). Policies that support professional development in an era of reform. Phi Delta Kappan, 76(8), 597-604.
23.
Desimone, L. ( 2006). Consider the source: Response differences among teachers, principals and districts on survey questions about their education policy environment. Educational Policy, 20(4), 640-676.
24.
Desimone, L., & LeFloch, K. ( 2004). Probing the "trickle down" effect of standards and assessments: Are we asking the right questions? Educational Evaluation and Policy Analysis, 26(1), 1-22.
25.
Desimone, L.M., Smith, T.S., Hayes, S., & Frisvold, D.E. (2005). Beyond accountability and average math scores: Relating multiple state education policy attributes to changes in student achievement in procedural knowledge, conceptual understanding and problem solving in mathematics. Educational Measurement: Issues and Practice, 24(4), 5-18.
26.
Dillman, D.A., & Tarnai, J. ( 1991). Mode effects of cognitively designed recall questions: A comparison of answers to telephone and mail surveys. In P. N. Beimer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz, & S. Sudman (Eds.), Measurement errors in surveys (pp. 367-393). New York: John Wiley.
27.
Driscoll, A., Peterson, K., Crow, N., & Larson, B. ( 1985). Student reports for primary teacher evaluation. Educational Research Quarterly, 9(3), 43-50.
28.
Elliott, D.H. ( 1950). Characteristics and relationships of various criteria of college and university teaching. Purdue University Studies in Higher Education, 70, 5-61.
29.
Elmore, R.F., & Consortium for Policy Research in Education. (1996). Staff development and instructional improvement in Community District 2. New York: National Commission on Teaching and America’s Future.
30.
Feuerstein, A., & Dietrich, J.A. (2003). State standards in the local context: A survey of school board members and superintendents. Educational Policy, 17(2), 237-256.
31.
Fine, M. ( 1991). Framing dropouts. New York: State University of New York Press.
32.
Fowler, F.J., Jr. ( 1995). Improving survey questions: Design and evaluation (Applied Social Research Methods Series, Vol. 38). Thousand Oaks, CA: Sage.
33.
Fowler, F.J., Jr., Roman, A., & Di, Z. ( 1998). Mode effects in a survey of Medicare prostate surgery patients. Public Opinion Quarterly, 62, 29-46.
34.
Fraser, B.J. ( 1982). Differences between student and teacher perceptions of actual and preferred classroom learning environment. Educational Evaluation and Policy Analysis, 4(4), 511-519.
35.
Gamoran, A., Porter, A.C., Smithson, J., & White, P.A. ( 1997). Upgrading high school mathematics instruction: Improving learning opportunities for low-achieving, low-income youth. Educational Evaluation and Policy Analysis, 19(4), 325-338.
36.
Garet, M., Birman, B., Porter, Yoon, K., & Desimone, L. ( 2002). What makes professional development effective? Analysis of a national sample of teachers. American Education Research Journal, 38(3), 915-945.
37.
Grant, S.G., Peterson, P.L., & Shojgreen-Downer, A. (1996). Learning to teach mathematics in the context of systemic reform. American Educational Research Journal, 33(2), 502-541.
38.
Groves, R.M. ( 1989). Survey errors and survey costs. New York: John Wiley.
39.
Guarino, C.M., Hamilton, L.S., Lockwood, J.R., Rathbun, A.H., & Hausken, E.G. ( 2006.) Teacher qualification, instructional practices, and reading and mathematics gains of kindergartners (Research and Development Report, NCES 2006-031). Washington, DC: U.S. Department of Education.
40.
Guthrie, E.R. ( 1949). The evaluation of teaching. Educational Record, 30, 109-115.
41.
Guthrie, E.R. ( 1954). The evaluation of teaching: A progress report. Seattle: University of Washington Press.
42.
Heilman, J.D., & Armentrout, W.D. (1936). The rating of college teachers on ten traits by their students. Journal of Educational Psychology , 27, 197-216.
43.
Herman, J., Klein, D., & Abedi, J. ( 2000). Assessing students’ opportunity to learn: Teacher and student perspectives. Educational Measurement: Issues and Practice, 19(4), 16-24.
44.
Hill, H. ( 2005). Content across communities: Validating measures of elementary mathematics instruction. Educational Policy, 19(3), 447-475.
45.
Ingersoll, R.M. ( 2001). Teacher turnover and teacher shortages: An organizational analysis. American Educational Research Journal, 39(3), 499-534.
46.
Knapp, M., & Associates. (1995). Teaching for meaning in high-poverty classrooms. New York : Teachers College Press.
47.
Krosnick, J.A. ( 1999). Maximizing questionnaire quality. In J. P. Robinson, P. R. Shaver, & L. S. Wrightsman (Eds.), Measures of political attitudes (pp. 37-57). San Diego, CA: Academic Press.
48.
Ladson-Billings, G., & Tate, B. ( 1995). Toward a critical race theory of education. Teachers College Record, 97(1), 47-67.
49.
Leighton, M., Mullens, J., Turnbull, B., Weiner, L., & Williams, A. ( 1995). Measuring instruction, curriculum content, and instructional resources: The status of recent work (NCES Working Paper 1995-11). Washington, DC: U.S. Department of Education.
50.
Light, R., Singer, J., & Willett, R. ( 1990). By design: Planning research on higher education. Cambridge, MA: Harvard University Press.
51.
Lipman, P. ( 1998). Race, class, and power in school restructuring. Albany: State University of New York Press.
52.
Mann, W.R. ( 1969). Changes in the level of attitude sophistication of college students as a measure of teacher effectiveness. Dissertation Abstracts, 29(8-A), 2443-2444.
53.
Maslow, A.H., & Zimmerman, W. ( 1956). College teaching ability, scholarly activity and personality . Journal of Educational Psychology, 47, 185-189.
54.
Mayer, D.P. ( 1999). Measuring instructional practice: Can policymakers trust survey data? Educational Evaluation and Policy Analysis, 21(1), 29-45.
55.
McCaffrey, D., Hamilton, K., Stecher, B., Klein, S., Bugliari, D., & Robyn, A. ( 2001). Interaction among instructional practices, curriculum, and student achievement: The case of standards-based high school mathematics . Journal for Research in Mathematics Education, 22(5), 493-517.
Mullens, J. ( 1995). Classroom instructional processes: A review of existing measurement approaches and their applicability for the teacher follow-up survey (NCES 95-15). Washington, DC: National Center for Education Statistics.
58.
Mullens, J. ( 1998). Validating teacher responses on instructional practice self-report surveys: Findings from a recent field-test. Paper presented in the Association for Public Policy Analysis and Management Research Conference, October 29-31, 1998, New York City.
59.
Mullens, J.E., & Gayler, K. ( 1999). Measuring classroom instructional processes: Using survey and case study field test results to improve item construction (NCES 1999-08) . Washington, DC: National Center for Education Statistics.
60.
Mullens, J., & Kasprzyk, D. ( 1996). Using qualitative methods to validate quantitative survey instruments. In 1996 Proceedings of the Section on Survey Research Methods (pp. 638-643). Alexandria, VA: American Statistical Association.
61.
Mullens, J., & Kasprzyk, D. ( 1999). Validating item responses on self-report teacher surveys . Washington, DC: U.S. Department of Education.
62.
National Center for Education Statistics (NCES). ( 1999). The NAEP 1996 technical report (NCES 1999-452). Washington, DC: Author.
63.
Porter, A.C. ( 2005). Prospects for school reform and closing the achievement gap. In C. A. Dwyer (Ed.), Measurement and research in the accountability era (pp. 59-95). Mahway , NJ: Lawrence Erlbaum Associates.
64.
Porter, A.C., & Brophy, J. ( 1988). Synthesis of research on good teaching: Insights from the work of the Institute for Research on Teaching. Educational Leadership, 45(8), 74-85.
65.
Porter, A.C., Kirst, M.W., Osthoff, E.J., Smithson, J.L., & Schneider, S.A. (1993). Reform up close: An analysis of high school mathematics and science classrooms. Madison : University of Wisconsin-Madison.
66.
Rayder, N.F. ( 1968). College student ratings of instructors. Journal of Experimental Education, 37, 76-81.
67.
Ross, S., Troutman, A., Horgan, D., Maxwell, S., Laitinen, R., & Lowther, D. ( 1997). The success of schools in implementing eight restructuring designs: A synthesis of first-year evaluation outcomes. School Effectiveness and School Improvement, 8(1), 95-124.
68.
Rowan, B., Correnti, R., & Miller, R. ( 2002). What large-scale survey research tells us about teacher effects on student achievement: Insights from the Prospects study of elementary schools. Philadelphia, PA: Consortium for Policy Research in Education.
69.
Schmidt, W.H., McKnight, C.C., Houang, R.T., Wang, H.C., Wiley, D.E., Cogan, L.S., et al. (2001). Why schools matter: A cross-national comparison of curriculum and learning. San Francisco: Jossey-Bass.
70.
Schmidt, W.H., McKnight, C.C., & Raizen, S.A. ( 1997). A splintered vision: An investigation of U.S. science and mathematics education. Boston: Kluwer Academic.
71.
Shavelson, R.J., & Dempsey-Atwood, N. (1976). Generalizability of measures of teaching behavior. Review of Educational Research, 46(4), 553-611.
72.
Shavelson, R.J., Webb, N.M., & Burstein, L. ( 1986). Measurement of teaching. In M. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 1-36). Washington, DC: American Educational Research Association.
73.
Simon, M.A., & Schifter, D. ( 1993). Toward a constructivist perspective: The impact of a mathematics teacher inservice program on students. Educational Studies in Mathematics, 25(4), 331-340.
74.
Smith, T., Desimone, L., Ueno, K. ( 2005). "Highly qualified" to do what? The relationship between NCLB teacher quality mandates and the use of reform-oriented instruction in middle school math. Educational Evaluation and Policy Analysis , 27(1), 75-109.
75.
Smithson, J.L., & Porter, A.C. ( 1994). Measuring classroom practice: Lessons learned from efforts to describe the enacted curriculum-The Reform Up Close study. (CPRE Research Report Series No. 31). Madison: University of Wisconsin, Consortium for Policy Research in Education.
76.
Spillane, J., & Zeuli, J. ( 1999). Reform and teaching: Exploring patterns of practice in the context of national and state mathematics reforms. Educational Evaluation and Policy Analysis, 21(1), 1-27.
77.
Stecher, B., & Borko, H. ( 2002). Combining surveys and case studies to examine standards-based educational reform (CSE Tech. Rep. No. 565). Los Angeles: University of California, National Center for Research on Evaluation, Standards, and Student Testing .
78.
Steele, J.M., House, E.R., & Kerins, T. ( 1971). An instrument for assessing instructional climate through low-inference student judgments. American Educational Research Journal, 8, 447-466.
79.
Steinberg, L. ( 2005). Adolescence. New York: McGraw-Hill.
Sudman, S., Bradburn, N.M., & Schwarz, N. ( 1996). Thinking about answers: The application of cognitive processes to survey methodology. San Francisco: Jossey-Bass.
82.
Tartwijk, J.V., Brekelmans, M., & Wubbels, T. ( 1998). Students’ perceptions of teacher interpersonal style: The front of the classroom as the teacher’s state. Teaching and Teacher Education, 14(6), 607-617.
83.
Thornburg, D.G. ( 1994). How does my child measure up? Family responses to recent reforms in educational assessment. Educational Policy, 8(2), 224-243.
84.
Turnbull, B., Welsh, M., Heid, C., Davis, W., & Ratnofsky, A.C. (1999). The longitudinal evaluation of school change and performance (LESCP) in Title I schools: Interim report to Congress. Washington, DC: Policy Studies Associates/Westat.
85.
University of Washington Department of Applied Mathematics. (2000). Middle school math comparison for Singapore math, Connected Mathematics Program, and Mathematics in Context (including comparisons with the NCTM Principles and Standards 2000). Report submitted to the National Science Foundation, Arlington, VA.
86.
U.S. Department of Education. (2003). Digest of education statistics, 2002. Washington, DC: National Center for Education Statistics.
87.
U.S. Department of Education, National Center for Educational Statistics. (1998). Toward better teaching: Professional development in 1993-94 (NCES 98-320). Washington, DC: Author.
88.
U.S. Department of Education, National Center for Education Statistics. (1999). The NAEP 1996 technical report (NCES 1999-452). Washington, DC: Author.
89.
U.S. Department of Education, National Center for Education Statistics. (2001). The nation’s report card: Mathematics 2000 (NCES 2001-517). Washington, DC : Author.
90.
Walker, B.D. ( 1969). An investigation of selected variables relative to the manner in which a population of junior college students evaluate their teachers . Dissertation Abstracts, 29(9-B), 3474.
91.
Waxman, H.C., & Eash, M.J. ( 1983). Utilizing students’ perceptions and context variables to analyze effective teaching: A process-product investigation. Journal of Educational Research, 76(6), 321-325.
