Contemporary teacher evaluation systems use multiple measures of performance to construct ratings of teacher quality. While the properties of constituent measures have been studied, little is known about whether composite ratings themselves are sufficiently reliable to support high-stakes decision making. We address this gap by estimating the consistency of composite ratings of teacher quality from New Mexico’s teacher evaluation system from 2015 to 2016. We estimate that roughly 40% of teachers would receive a different composite rating if reevaluated in the same year; 97% of teachers would receive ratings within ±1 level of their original rating. We discuss mechanisms by which policymakers can improve rating consistency, and the implications of those changes to other properties of teacher evaluation systems.
American Educational Research Association. (2015). AERA statement on use of value-added models (VAM) for the evaluation of educators and educator preparation programs. Educational Researcher, 44, 448–452.
2.
Bacher-HicksA.ChinM. J.KaneT. J.StaigerD. O. (2017). An evaluation of bias in three measures of teacher quality: Value-added, classroom observations, and student surveys (Working Paper No. 23478). Cambridge, MA: National Bureau of Economic Research.
3.
BakerE. L.BartonP. E.Darling-HammondL.HaertelE.LaddH. F.LinnR. L.. . . ShepardL. A. (2010). Problems with the use of student test scores to evaluate teachers (EPI Briefing Paper No. 278). Retrieved from http://www.epi.org/files/page/-/pdf/bp278.pdf
BockR. D.PetersenA. C. (1975). A multivariate correction for attenuation. Biometrika, 62, 673–678.
7.
BorkoH. (2004). Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33, 3–15.
8.
BraunH. (2005). Using student progress to evaluate teachers: A primer on value-added models. Princeton, NJ: Educational Testing Service. Retrieved from https://www.ets.org/Media/Research/pdf/PICVAM.pdf
9.
BrennanR. L. (2001). An essay on the history and future of reliability from the perspective of replications. Journal of Educational Measurement, 38, 295–317.
10.
BrennanR. L.WanL. (2004, June). A bootstrap procedure for estimating decision consistency for single-administration complex assessments. Paper presented at the Annual Meeting of the National Council on Measurement in Education, San Diego, CA.
CherngH.-Y. S.HalpinP. F. (2016). The importance of minority teachers: Student perceptions of minority versus white teachers. Educational Researcher, 45, 407–420.
13.
ChesterM. D. (2003). Multiple measures and high-stakes decisions: A framework for combining measures. Educational Measurement: Issues and Practice, 22(2), 32–41.
14.
ChettyR.FriedmanJ. N.RockoffJ. E. (2014). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104, 2633–2679.
15.
Cochran-SmithM. (2003). The unforgiving complexity of teaching: Avoiding simplicity in the age of accountability. Journal of Teacher Education, 54(1), 3–5.
16.
CorcoranS.GoldhaberD. (2013). Value added and its uses: Where you stand depends on where you sit. Education Finance and Policy, 8, 418–434.
17.
CronbachL. J.LinnR. L.BrennanR. L.HaertelE. H. (1997). Generalizability analysis for performance assessments of student achievement or school effectiveness. Educational and Psychological Measurement, 57, 373–399.
18.
CullenJ. B.KoedelC.ParsonsE. (2016). The compositional effect of rigorous teacher evaluation on workforce quality. Cambridge, MA: National Bureau of Economic Research.
19.
DanielsonC. (2007). Enhancing professional practice: A framework for teaching. Alexandria, VA: Association for Supervision and Curriculum Development.
20.
DeeT. S.WyckoffJ. (2015). Incentives, selection, and teacher performance: Evidence from IMPACT. Journal of Policy Analysis and Management, 34, 267–297.
21.
DouglasK. M.MislevyR. J. (2010). Estimating classification accuracy for complex decision rules based on multiple scores. Journal of Educational and Behavioral Statistics, 35, 280–306.
22.
EnglishD.BurniskeJ.MeibaumD.Lachlan-HachéL. (2016). Using student surveys as a measure of teaching effectiveness. Washington, DC: American Institutes for Research.
23.
FergusonR. F. (2010). Student perceptions of teaching effectiveness (Discussion Brief). Cambridge, MA: National Center for Teacher Effectiveness and the Achievement Gap Initiative.
24.
GoeL.HoldheideL. R.MillerT. (2011). A practical guide to designing comprehensive teacher evaluation systems: A tool to assist in the development of teacher evaluation systems. Washington, DC: National Comprehensive Center for Teacher Quality. Retrieved from http://www.lauragoe.com/LauraGoe/practicalGuideEvalSystems.pdf
25.
GoldhaderD. D.BrewerD. J.AndersonD. J. (1999). A three-way error components analysis of educational productivity. Education Economics, 7, 199–208.
26.
GrissomJ. A.LoebS. (2017). Assessing principals’ assessments: Subjective evaluations of teacher effectiveness in low- and high-stakes environments. Education Finance and Policy, 12, 369–395.
27.
GrossmanP.LoebS.CohenJ.WyckoffJ. (2013). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added scores. American Journal of Education, 119, 445–470.
28.
HillH. C.UmlandK. L.LitkeE.KapitulaL. (2012). Teacher quality and quality teaching: Examining the relationship of a teacher assessment to practice. American Journal of Education, 118, 489–519.
KaneT. J.TaylorE. S.TylerJ. H.WootenA. L. (2011). Identifying effective classroom practices using student achievement data. Journal of Human Resources, 46, 587–613.
35.
KoedelC.MihalyK.RockoffJ. E. (2015). Value-added modeling: A review. Economics of Education Review, 47, 180–195.
36.
KraftM. A.GilmourA. F. (2017). Revisiting the widget effect: Teacher evaluation reforms and the distribution of teacher effectiveness. Educational Researcher, 46, 234–249.
37.
KyriakidesL. (2005). Extending the comprehensive model of educational effectiveness by an empirical investigation. School Effectiveness and School Improvement, 16, 103–152.
38.
LeeW. C.HansonB. A.BrennanR. L. (2002). Estimating consistency and accuracy indices for multiple classifications. Applied Psychological Measurement, 26, 412–432.
39.
LeinhardtG.GreenoJ. G. (1986). The cognitive skill of teaching. Journal of Educational Psychology, 78, 75–95.
40.
LivingstonS. A.LewisC. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179–197.
41.
MartínezJ. F.SchweigJ.GoldschmidtP. (2016). Approaches for combining multiple measures of teacher performance: Reliability, validity, and implications for evaluation policy. Educational Evaluation and Policy Analysis, 38, 738–756.
42.
MashburnA. J.MeyerJ. P.AllenJ. P.PiantaR. C. (2014). The effect of observation length and presentation order on the reliability and validity of an observational measure of teaching quality. Educational and Psychological Measurement, 74, 400–422.
43.
McCaffreyD. F.LockwoodJ.KoretzD. M.HamiltonL. S. (2004). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND Corporation.
44.
McCaffreyD. F.SassT. R.LockwoodJ. R.MihalyK. (2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy, 4, 572–606.
45.
MessickS. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 5D, 741–749.
46.
MihalyK.McCaffreyD. F. (2014). Grade level variation in observational measures of teacher effectiveness. In KerrK.PiantaR.KaneT. (Eds.), Designing teacher evaluation systems: New guidance from the Measures of Effective Teaching project (pp. 9–49). San Francisco, CA: Jossey-Bass.
47.
MihalyK.McCaffreyD. F.StaigerD. O.LockwoodJ. R. (2013). A composite estimator of effective teaching. Seattle, WA: Bill & Melinda Gates Foundation.
48.
NyeB.KonstantopoulosS.HedgesL. V. (2004). How large are teacher effects?Educational Evaluation and Policy Analysis, 26, 237–257.
PapayJ. P.KraftM. A. (2015). Productivity returns to experience in the teacher labor market: Methodological challenges and new evidence on long-term career improvement. Journal of Public Economics, 130, 105–119.
51.
PiantaR. C.HamreB. K.HaynesN. J.MintzS.La ParoK. M. (2006). Classroom Assessment Scoring System (CLASS) manual: Middle/secondary version pilot. Charlottesville: Curry School of Education, University of Virginia.
52.
RivkinS. G.HanushekE. A.KainJ. F. (2005). Teachers, schools, and academic achievement. Econometrica, 73, 417–458.
SchweigJ. D. (2018). Pilot to policy: Reconsidering the reliability of observation-based ratings from New Mexico’s statewide teacher evaluation system. Paper presented at the Annual Meeting of the American Educational Research Association, New York, NY.
55.
ShroutP. E.FleissJ. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420–428.
56.
SteeleJ. L.HamiltonL. S.StecherB. M. (2010). Incorporating student performance measures into teacher evaluation systems. Santa Monica, CA: RAND Corporation. Retrieved from https://www.rand.org/pubs/technical_reports/TR917.html
57.
SteinbergM. P.DonaldsonM. L. (2016). The new educational accountability: Understanding the landscape of teacher evaluation in the post-NCLB era. Education Finance and Policy, 11, 340–359.
58.
SteinbergM. P.GarrettR. (2016). Classroom composition and measured teacher performance: What do teacher observation scores really measure?Educational Evaluation and Policy Analysis, 38, 293–317.
59.
SteinbergM. P.KraftM. A. (2017). The sensitivity of teacher performance ratings to the design of teacher evaluation systems. Educational Researcher, 46, 378–396.
60.
StrunkK. O.WeinsteinT. L.MakkonenR. (2014). Sorting out the signal: Do multiple measures of teachers’ effectiveness provide consistent information to teachers and principals?Education Policy Analysis Archives, 22, 1–41.
61.
TaylorE. S.TylerJ. H. (2012). The effect of evaluation on teacher performance. American Economic Review, 102, 3628–3651.
WallaceT. L.KelceyB.RuzekE. (2016). What can student perception surveys tell us about teaching? Empirically testing the underlying structure of the tripod student perception survey. American Educational Research Journal, 53, 1834–1868.
64.
WeisbergD.SextonS.MulhernJ.KeelingD.SchunckJ.PalciscoA.MorganK. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. Retrieved from https://tntp.org/assets/documents/TheWidgetEffect_2nd_ed.pdf
WorrellF. C.KuterbachL. D. (2001). The use of student ratings of teacher behaviors with academically talented high school students. Journal of Secondary Gifted Education, 12, 236–247.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
