This article addresses likely error rates for measuring teacher and school performance in the upper elementary grades using value-added models applied to student test score gain data. Using a realistic performance measurement system scheme based on hypothesis testing, the authors develop error rate formulas based on ordinary least squares and Empirical Bayes estimators. Empirical results suggest that value-added estimates are likely to be noisy using the amount of data that are typically used in practice. Type I and II error rates for comparing a teacher’s performance to the average are likely to be about 25% with 3 years of data and 35% with 1 year of data. Corresponding error rates for overall false positive and negative errors are 10% and 20%, respectively. Lower error rates can be achieved if schools are the performance unit. The results suggest that policymakers must carefully consider likely system error rates when using value-added estimates to make high-stakes decisions regarding educators.
AaronsonD.BarrowL.SanderW. (2007). Teachers and student achievement in the Chicagopublic high schools. Journal of Labor Economics, 25, 95–135.
2.
AmemiyaT. (1985). Advanced econometrics. Cambridge, MA: Harvard University Press.
3.
AndrabiT.DasJ.KhwajaA. I.ZajoncT. (2011). Do value-added estimates add value? Accounting for learning dynamics. American Economic Journal: Applied Economics, 3, 29–54.
4.
BallouD. (2005). Value-added assessment: Lessons from Tennessee. In R. Lissetz (Ed.), Value added models in education: Theory and applications. Maple Grove, MN: JAM Press.
5.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate. A new and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57, 1289–1300.
6.
BergerJ. (1985). Statistical decision theory and Bayesian analysis. New York, NY: Springer-Verlag.
7.
Bill & Melinda Gates Foundation. (2010). Learning about teaching: Initial findings from the measures of effective teaching project. http://www.metproject.org/downloads/Preliminary_Findings-Research_Paper.pdf. Seattle, WA.
8.
BloomH.HillC.BlackA.LipseyM. (2008). Performance trajectories and performance gaps as achievement effect-size benchmarks for educational interventions. New York, NY: MDRC.
9.
BloomH.Richburg-HayesL.BlackA. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29, 30–59.
10.
GelmanA.HillJ.YajimaM. (2009). Why we (usually) don’t have to worry about multiple comparisons. Department of statistics working paper. New York, NY: Columbia University.
11.
GoldhaberD.HansenM. (2008). Is it just a bad class? Assessing the stability of measured teacher performance. CRPE working paper 2008_5. Seattle, WA: Center on Reinventing Public Education.
12.
HanushekE.KainJ.O’BrienD.RivkinS. (2005). The market for teacher quality. NBER working paper 11154. Cambridge, MA: National Bureau of Economic Research.
13.
HanushekE.RivkinS. (2010). Generalizations about using value-added measures of teacher quality. American Economic Review: Papers and Proceedings, 100, 267–271.
14.
HarrisD.SassT. (2006). Value-added models and the measurement of teacher quality. Working paper. Tallahassee: Florida State University.
15.
HedgesL.HedbergE. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29, 60–87.
16.
JacobB.LefgrenL. (2008). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economics, 26, 101–136.
17.
KaneT. (2004). The impact of after-school programs: Interpreting the results of four recent evaluations. Working paper. Los Angeles, CA: University of California.
18.
KaneT.RockoffJ.StaigerD. (2008). What does certification tell us about teacher effectiveness? Evidence from New York City. Economics of Education Review, 27, 615–631.
19.
KaneT.StaigerD. (2002a). The promise and pitfalls of using imprecise school accountability measures. Journal of Economic Perspectives, 16, 91–114.
20.
KaneT.StaigerD. (2002b). Volatility in school test scores: Implications for test-based accountability systems. In RavitchD.,(Ed.) Brookings papers on education policy: 2002. Washington, DC: Brookings Institution Press.
21.
KaneT.StaigerD. (2008). Estimating teacher impacts on student achievement: An experimental evaluation. NBER working paper 14607. Cambridge, MA: National Bureau of Economic Research.
22.
KishL. (1965). Survey sampling. New York, NY: John Wiley.
23.
KoedelC.BettsJ. (2007). Re-examining the role of teacher quality in the educational production function. Working paper. Columbia: University of Missouri.
24.
KoedelC.BettsJ. (2009). Does student sorting invalidate value-added models of teacher effectiveness? An extended analysis of the Rothstein critique. Working paper. Columbia: University of Missouri.
25.
LindleyD.V.SmithA. F. M. (1972). Bayes estimates for the linear model. Journal of the Royal Statistics Society, Series B, 34, 1–41.
26.
LipscombS.TehB.GillB.ChiangH.OwensA. (2010). Teacher and principal value-added: Research findings and implementation practices. Cambridge, MA: Mathematica Policy Research.
27.
McCaffreyD.LockwoodJ. R.KoretzD.HamiltonL. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND.
28.
McCaffreyD.LockwoodJ. R.KoretzD.LouisT.HamiltonL. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29, 67–101.
29.
McCaffreyD.SassT.LockwoodJ. R.MihalyK. (2009). The intertemporal stability of teacher effects. Education Finance and Policy, 4, 572–606.
30.
National Institute for Excellence in Teaching. (2009). TAP: The system for teacher and student advancement. Retrieved October13, 2009, fromhttp://www.tapsystem.org/
31.
NyeB.KonstantopoulosS.HedgesL. (2004). How large are teacher effects?Educational Evaluation and Policy Analysis, 26, 237–257.
32.
RaudenbushS.BrykA. (2002). Hierarchical linear models: Applications and data analysis methods. (2nd ed.). Thousand Oaks, CA: Sage.
33.
RivkinS.HanushekE.KainJ. (2005). Teachers, schools, and academic achievement. Econometrica, 73, 417–458.
34.
RockoffJ. (2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review (AEA Papers and Proceedings), 94, 247–252.
35.
RothsteinJ. (2010). Teacher quality in educational production: Tracking, decay, and student achievement. Quarterly Journal of Economics, 125, 175–214.
36.
SandersW. L.SaxtonA. M.HornS. P. (1997). The Tennessee value-added assessment system: A quantitative, outcome-based approach to educational assessment. In MillmanJ., Grading teachers, grading schools: Is student achievement a valid evaluation measure? (pp. 137–162). Thousand Oaks, CA: Corwin Press.
37.
SchochetP. Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33, 62–87.
38.
SchochetP. Z.ChiangH. S. (2010). Error rates in measuring teacher and school performance based on student test score gains. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
39.
SolmonL.WhiteJ. T.CohenD.WooD. (2007). The effectiveness of the teacher advancement program. Santa Monica, CA: National Institute for Excellence in Teaching.
40.
SpringerM.BallouD.PengA. (2008). Impact of the teacher advancement program on student test score gains: Findings from an independent appraisal. Working paper 2008-19. Nashville, TN: National Center on Performance Incentives.
41.
ToddP.WolpinK. (2003). On the specification and estimation of the production function for cognitive achievement. The Economic Journal, 113, F3–F33.
