Our secondary analysis of Measures of Effective Teaching data contributes to growing evidence that observation ratings, used as part of comprehensive teacher evaluation systems across the nation, may measure factors outside of a teacher’s performance or control. Specifically, men and teachers in classrooms with high concentrations of Black, Hispanic, male, and low-performing students receive significantly lower observation ratings. By using various methodological approaches and a subsample of teachers randomly assigned to classrooms, we demonstrate that these differences are unlikely due to actual differences in teacher quality. These results suggest that policymakers consider the unintended consequences of using observational ratings to evaluate teachers and consider ways to adjust ratings to ensure they are fair.
AllenJ. P.PiantaR. C.GregoryA.MikamiA. Y.LunJ. (2011). An interaction-based approach to enhancing secondary school instruction and student achievement. Science, 333(6045), 1034–1037.
2.
American Educational Research Association. (2015). AERA statement on use of value-added models (VAM) for the evaluation of educators and educator preparation programs. Educational Researcher, 44(8).
3.
BakerE. L.BartonP. E.Darling-HammondL.HaertelE.LaddH. F.LinnR. L.ShepardL. A. (2010). Problems with the use of student test scores to evaluate teachers (EPI Briefing Paper No. 278). Washington, DC: Economic Policy Institute.
4.
CampbellS. L. (2014). Quality teachers wanted: An examination of standards-based evaluation systems and school staffing practices in North Carolina middle schools (Order No. 3633946). Available from ProQuest Dissertations & Theses A&I. (1612601875). Retrieved from https://search.proquest.com/docview/1612601875?accountid=14509
5.
CornellS.HartmannD. (2006). Ethnicity and race: Making identities in a changing world. Thousand Oaks, CA: Sage Publications.
6.
DeeT. S.WyckoffJ. (2015). Incentives, selection, and teacher performance: Evidence from IMPACT. Journal of Policy Analysis and Management, 34(2), 267–297.
FenstermacherG. D.RichardsonV. (2005). On making determinations of quality in teaching. Teachers College Record, 107(1), 186–213.
9.
GarrettR.SteinbergM. (2015). Examining teacher effectiveness using classroom observation scores: Evidence from the randomization of teachers to students. Educational Evaluation and Policy Analysis, 37, 224–242.
10.
GoldhaberD.LaveryL.TheobaldR. (2015). Uneven playing field? Assessing the teacher quality gap between advantaged and disadvantaged students. Educational Researcher, 44(5), 293–307.
11.
GrossmanP.CohenJ.RonfeldtM.BrownL. (2014). The test matters: The relationship between classroom observation scores and teacher value added on multiple types of assessment. Educational Researcher, 43(6), 293–303.
12.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel (Research paper, MET Project). Seattle, WA: Bill and Melinda Gates Foundation.
13.
JacobB. A.WalshE. (2011). What’s in a rating?Economics of Education Review, 30(3), 434–448.
14.
JiangJ. Y.SporteS. E. (2016). Teacher evaluation in Chicago: Differences in observation and value-added scores by teacher, student, and school characteristics (Research report). Chicago, IL: University of Chicago Consortium on School Research.
15.
KalogridesD.LoebS. (2013). Different teachers, different peers the magnitude of student sorting within schools. Educational Researcher, 42(6), 304–316.
16.
KalogridesD.LoebS.BéteilleT. (2013). Systematic sorting teacher characteristics and class assignments. Sociology of Education, 86(2), 103–123.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains (Research paper, MET Project). Seattle, WA: Bill and Melinda Gates Foundation.
19.
KaneT. J.TaylorE. S.TylerJ. H.WootenA. L. (2011). Identifying effective classroom practices using student achievement data. Journal of Human Resources, 46, 587–613.
20.
KraftM. A.PapayJ. P. (2014). Can professional environments in schools promote teacher development? Explaining heterogeneity in returns to teaching experience. Educational Evaluation and Policy Analysis, 36(4), 476–500.
21.
LankfordH.LoebS.WyckoffJ. (2002). Teacher sorting and the plight of urban schools: A descriptive analysis. Education Evaluation and Policy Analysis, 24(1), 37–62.
22.
LauenD. L.GaddisS. M. (2013). Exposure to classroom poverty and test score achievement: contextual effects or selection?American Journal of Sociology, 118(4), 943–979.
23.
LoebS.Darling-HammondL.LuczakJ. (2005). How teaching conditions predict teacher turnover in California schools. Peabody Journal of Education, 80(3), 44–70.
24.
MakelM. C.PluckerJ. A. (2014). Facts are more important than novelty replication in the education sciences. Educational Researcher, 43(6), 304–316.
PapayJ. P.KraftM. A. (2015). Productivity returns to experience in the teacher labor market: Methodological challenges and new evidence on long-term career improvement. Journal of Public Economics, 130, 105–119.
27.
PapayJ.TaylorE. S.TylerJ.LaskiM. (2015). Learning job skills from colleagues at work: Evidence from a field experiment using teacher performance data. Unpublished working paper.
28.
RonfeldtM. (2015). Field placement schools and instructional effectiveness. Journal of Teacher Education, 66(4), 304–320.
29.
SandersW.RiversJ. (1996). Cumulative and residual effects of teachers on future student academic achievement: Research progress report. Knoxville, TN: University of Tennessee Value-Added Research and Assessment Center.
30.
SassT. R.HannawayJ.XuZ.FiglioD. N.FengL. (2012). Value added of teachers in high-poverty schools and lower poverty schools. Journal of Urban Economics, 72(2), 104–122.
31.
SteinbergM. P.GarrettR. (2016). Classroom composition and measured teacher performance: What do teacher observation scores really measure?Educational Evaluation and Policy Analysis, 38(2), 293–317.
32.
WhiteM.RowanB. (2014). User guide to measures of effective teaching longitudinal database (MET LDB). Ann Arbor, MI: Inter-University Consortium for Political and Social Research. University of Michigan.
33.
WhitehurstG. J. R.ChingosM. M.LindquistK. M. (2014). Evaluating teachers with classroom observations. Washington, DC: Brookings Institute.
