Surveys are frequently used to inform consequential decisions about teachers, policies, and programs. Consequently, it is important to understand the validity of these instruments. This study assesses the validity of measures of instruction captured by an annual survey by comparing survey data with those of a validated daily log. The two instruments produced similar rankings of the frequency with which teachers use particular practices but more than three fourths of the teachers in the study were found to overreport their instruction on the annual survey. Multilevel models revealed a number of teacher and school characteristics related to survey reporting error. The study’s implications for users of survey evidence are discussed.
AndersonJ. R. (1983). The architecture of cognition. Cambridge, MA: Harvard University Press.
2.
Bill & Melinda Gates Foundation. (2010). Learning about teaching: Initial findings from the Measures of Effective Teaching project (MET Project Research Paper). Seattle, WA: Author.
3.
BrykA. S.SchneiderB. (2002). Trust in schools: A core resource for improvement. New York, NY: Russell Sage Foundation.
4.
BrykA. S.SebringP. B.AllensworthE.LuppescuS.EastonJ. Q. (2010). Organizing schools for improvement lessons from Chicago. Chicago, IL: University of Chicago Press.
5.
BursteinL.McDonnellL. M.Van WinkleJ.OrmsethT. H.MirochaJ.GuitonG. (1995). Validating national curriculum indicators. Santa Monica, CA: RAND.
6.
CamburnE. M. (2010). Embedded teacher learning opportunities as a site for reflective practice: An exploratory study. American Journal of Education, 116, 463-489.
7.
CamburnE. M.BarnesC. A. (2004). Assessing the validity of a language arts instruction log through triangulation. Elementary School Journal, 105, 49-73.
8.
CamburnE. M.HanS. W. (2009). Investigating connections between distributed leadership and instructional change. Distributed Leadership: Different Perspectives, 7, 25-45. doi:10.1007/978-1-4020-9737-9_3
9.
CamburnE. M.HanS. W. (2011). Two decades of generalizable evidence on U.S. instruction from national surveys. Teachers College Record, 113, 561-610.
10.
CamburnE. M.HuffJ. T.GoldringE. B.MayH. (2010). Assessing the validity of an annual survey for measuring principal leadership practice. Elementary School Journal, 111, 314-335.
11.
CybulskiT. G.HoyW. K.SweetlandS. R. (2005). The roles of collective efficacy of teachers and fiscal efficiency in student achievement. Journal of Educational Administration, 43, 439-461.
12.
DesimoneL. M. (2006). Consider the source: Response differences among teachers, principals, and districts on survey questions about their education policy environment. Educational Policy, 20, 640-676.
13.
DesimoneL. M.Le FlochK. C. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26, 1-22.
14.
DesimoneL. M.SmithT. M.FrisvoldD. E. (2010). Survey measures of classroom instruction: Comparing student and teacher reports. Educational Policy, 24, 267-329.
15.
DillmanD. A. (1991). The Design and Administration of Mail Surveys. Annual Review of Sociology, 17, 225-249.
HeeringaS.LittleR. J. A.RaghunathanT. (2002). Multivariate imputation of coarsened survey data on household wealth. In GrovesR.M.DillmanD.A.EltingeJ.L.LittleR.J.A. (Eds) Survey nonresponse (pp. 1-19). New York, NY: Wiley.
18.
HillH. C. (2005). Content across communities: Validating measures of elementary mathematics instruction. Educational Policy, 19(3), 447-475.
19.
HiltonM. E. (1989). A comparison of a prospective diary and two summary recall techniques for recording alcohol consumption. British Journal of Addiction, 84, 1085-1092.
20.
LemmensP.KnibbeR. A.TanF. (1988). Weekly recall and diary estimates of alcohol consumption in a general population survey. Journal of Studies on Alcohol, 49, 131-135.
21.
LemmensP.TanE. S. (1992). Measuring quantity and frequency of drinking in a general population survey: A comparison of five. Journal of Studies on Alcohol, 53, 476-486.
22.
LintonM. (1988). Real-world memory after six years: An in-vivo study of long-term memory. In GrunebergM. M.MorrisP. E.SykesR. N. (Eds.), Practical aspects of memory: Current research and issues (pp. 69-76). Chichester, UK: Wiley.
23.
LuppescuS.HartH.RosenkranzT.MontgomeryN.SporteS.Bender SebringP.MazzeoC. (2007). CCSR’s 2007 Survey Reports for Chicago Public Schools. Chicago, IL: Consortium on Chicago School Research. Retrieved on 5/12/15 from https://ccsr.uchicago.edu/sites/default/files/publications/rpt7777.pdf.
24.
MayerD. P. (1999). Measuring instructional practice: Can policymakers trust survey data?Educational Evaluation and Policy Analysis, 21, 29-45.
25.
MenonG. (1994). Judgments of behavioral frequencies: Memory search and retrieval strategies. In SchwarzN.SeymourS. (Eds.), Autobiographical memory and the validity of retrospective reports (pp. 161-172). New York, NY: Springer-Verlag.
26.
MilesiC.GamoranA. (2006). Effects of class size and instruction on kindergarten achievement. Educational Evaluation and Policy Analysis, 28(4), 287-313.
27.
MullensJ. E.GaylerK.GoldsteinD.HildrethJ.RubensteinM.SpiggleT.. . . WelshM. (1999). Measuring classroom instructional processes: Using survey and case study fieldtest results to improve item construction (Working paper no. 1999-08). Washington, DC: U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics.
28.
MullisI. V. S.MartinM. O.FoyP.AroraA. (2012). TIMSS 2011 international results in mathematics. Chestnut Hill, MA: Boston College, TIMSS & PIRLS International Study Center.
29.
Organisation for Economic Co-Operation and Development. (2013a). PISA 2012 results: Ready to learn—Students’ engagement, drive and self-beliefs (Vol. III). Paris, France: Program for International Student Assessment, Organisation for Economic Co-Operation and Development.
30.
Organisation for Economic Co-Operation and Development. (2013b). PISA 2012 results: What makes schools successful? Resources, policies and practices. Paris, France: Program for International Student Assessment, Organisation for Economic Co-Operation and Development.
31.
PorterA. C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3-14.
32.
RowanB.CamburnE.CorrentiR. (2004). Using teacher logs to measure the enacted curriculum: A study of literacy teaching in third-grade classrooms. The Elementary School Journal, 105(1), 75-101.
33.
RowanB.MillerR. J. (2007). Organizational strategies for promoting instructional change: Implementation dynamics in schools working with comprehensive school reform providers. American Educational Research Journal, 44(2), 252-297.
34.
SchmidtW. H.McKnightC. C. (1995). Surveying educational opportunity in mathematics and science: An international perspective. Educational Evaluation and Policy Analysis, 17(3), 337-353.
35.
SmithsonJ. L.PorterA. C. (1994). Measuring classroom practice: Lessons learned from efforts to describe the enacted curriculum—The reform up close study (CPRE Research Report Series Report No. 31). New Brunswick, NJ: Consortium for Policy Research in Education.
36.
ThompsonC. P.SkowronskiJ. J.LarsenS. F.BetzA. L. (1996). Autobiographical memory: Remembering what and remembering when. Mahwah, NJ: Lawrence Erlbaum.
37.
TourangeauR.RasinskiK. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological bulletin, 103(3), 299.
38.
TourangeauR.RipsL. J.RasinskiK. A. (2000). The psychology of survey response. Cambridge, UK: Cambridge University Press.
39.
TulvingE. (1983). Elements of episodic memory. Oxford, UK: Oxford University Press.
40.
UngeJ.HanssonG.-A.OhlssonK.NordanderC.AxmonA.WinkelJ.SkerfvingS. (2005). Validity of self-assessed reports of occurrence and duration of occupational tasks. Ergonomics, 48, 12-24.
41.
WagenaarW. A. (1986). My memory: A study of autobiographical memory over six years. Cognitive Psychology, 18, 225-252. doi:10.1016/0010-0285(86)90013-7
42.
WhiteR. T. (1982). Memory for personal events. Human Learning, 1, 171-183.
