We study the relative performance of two policy-relevant value-added models—a one-step fixed effect model and a two-step aggregated residuals model—using a simulated data set well grounded in the value-added literature. A key feature of our data generating process is that student achievement depends on a continuous measure of economic disadvantage. This is a realistic condition that has implications for model performance because researchers typically have access to only a noisy, binary measure of disadvantage. We find that one- and two-step value-added models perform similarly across a wide range of student and teacher sorting conditions, with the two-step model modestly outperforming the one-step model in conditions that best match observed sorting in real data. A reason for the generally superior performance of the two-step model is that it better handles the use of an error-prone, dichotomous proxy for student disadvantage.
AaronsonD.BarrowL.SanderW. (2007). Teachers and student achievement in the Chicago Public High Schools. Journal of Labor Economics, 25, 95–135.
2.
AndersonA.HarrisD. (2013). Bias of public sector worker performance monitoring: A structural model and empirical evidence from secondary school teachers. Unpublished manuscript, Tulane University, New Orleans, LA.
3.
AshenfelterO.KruegerA. (1994). Estimates of the economic return to schooling from a new sample of twins. American Economic Review, 84, 1157–1173.
4.
Bacher-HicksA.KaneT. J.StaigerD. O. (2014). Validating teacher effect estimates using changes in teacher assignments in Los Angeles(NBER Working Paper No. 20657). Cambridge, MA: National Bureau of Economic Research.
5.
BassD. N. (2010). Fraud in the lunchroom?Education Next, 10, 67–71.
6.
BoydD.LankfordH.LoebS.WyckoffJ. (2013). Measuring test measurement error: A general approach. Journal of Educational and Behavioral Statistics, 38, 629–663.
7.
ChettyR.FriedmanJ. N.RockoffJ. E. (2014a). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104, 2593–2632.
8.
ChettyR.FriedmanJ. N.RockoffJ. E. (2014b). Measuring the impacts of teachers II: Teacher value-added and student outcomes in adulthood. American Economic Review, 104, 2633–2679.
9.
ChettyR.FriedmanJ. N.RockoffJ. E. (2017). Measuring the impacts of teachers: Reply. American Economic Review, 107, 1685–1717.
10.
ClotfelterC. T.LaddH. F.VigdorJ. L. (2006). Teacher-student matching and the assessment of teacher effectiveness. Journal of Human Resources, 41, 778–820.
11.
CullenJ. B.KoedelC.ParsonsE. (2016). The compositional effect of rigorous teacher evaluation on workforce quality(NBER Working Paper No. 22805). Cambridge, MA: National Bureau of Economic Research.
12.
EhlertM.KoedelC.ParsonsE.PodgurskyM. (2014). Choosing the right growth measure: Methods should compare similar schools and teachers. Education Next, 14, 66–71.
13.
EhlertM.KoedelC.ParsonsE.PodgurskyM. (2016). Selecting growth measures for use in school evaluation systems: Should proportionality matter?Educational Policy, 30, 465–500.
GoldhaberD.HansenM. (2010). Using performance on the job to inform tenure decisions. American Economic Review, 100, 250–255.
16.
GoldhaberD.QuinceV.TheobaldR. (2018). Has it always been this way? Tracing the evolution of teacher quality gaps in U.S. public schools. American Educational Research Journal, 55, 171–201.
17.
GoldhaberD.WalchJ.GabeleB. (2013). Does the model matter? Exploring the relationship between different student achievement-based teacher assessments. Statistics and Public Policy, 1, 28–39.
18.
GrilichesZ. (1979). Sibling models and data in economics: Beginnings of a survey. Journal of Political Economy, 87, S37–S64.
19.
GuarinoC. M.MaxfieldM.ReckaseM. D.ThompsonP. N.WooldridgeJ. M. (2015). An evaluation of Empirical Bayes’s estimation of value-added teacher performance measures. Journal of Educational and Behavioral Statistics, 40, 190–222.
20.
GuarinoC. M.ReckaseM.WooldridgeJ. (2015). Can value-added measures of teacher performance be trusted?Education Finance and Policy, 10, 117–156.
21.
HanushekE. A.KainJ. F.O’BrienD. M.RivkinS. G. (2005). The market for teacher quality (NBER Working Paper No. 11154). Cambridge, MA: National Bureau of Economic Research.
22.
HarrisD. N.SassT. R. (2014). Skills, productivity and the evaluation of teacher performance. Economics of Education Review, 40, 183–204.
23.
HarwellM.LeBeauB. (2010). Student eligibility for a free lunch as an SES measure in education research. Educational Researcher, 39, 120–131.
24.
HoffmanL. (2012). Free and reduced-price lunch eligibility data in EDFacts: A White Paper on current status and potential changes. Fairfax, VA: U.S. Department of Education.
25.
IsenbergE.MaxJ.GleasonP.JohnsonM.DeutschJ.HansenM.AngeloL. (2016). Do low-income students have equal access to effective teachers? Evidence from 26 districts(NCEE 2017-4007). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
26.
IsenbergE.MaxJ.GleasonP.PotamitesL.SantillanoR.HockH.HansenM.. (2013). Access to effective teaching for disadvantaged students (NCEE 2014-4001). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
KaneT. J.McCaffreyD. F.MillerT.StaigerD. O. (2013). Have we identified effective teachers? Validating measures of effective teaching using random assignment. Seattle, WA: Bill and Melinda Gates Foundation.
33.
KaneT. J.StaigerD. O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation(NBER Working Paper No. 14607). Cambridge, MA: National Bureau of Economic Research.
34.
KaneT. J.TaylorE. S.TylerJ. H.WootenA. L. (2011). Identifying effective classroom practices using student achievement data. Journal of Human Resources, 46, 587–613.
35.
KoedelC.LeathermanR.ParsonsE. (2012). Test measurement error and inference from value-added models. The B.E. Journal of Economic Analysis & Policy, 12.
36.
KoedelC.MihalyK.RockoffJ. E. (2015). Value-added modeling: A review. Economics of Education Review, 47, 180–195.
37.
LockwoodJ. R.McCaffreyD. F. (2014). Correcting for test score measurement in ANCOVA models for estimating treatment effects. Journal of Educational and Behavioral Statistics, 39, 22–52.
38.
LockwoodJ. R.McCaffreyD. F.MarianoL. T.SetodjiC. (2007). Bayesian methods for scalable multivariate value-added assessment. Journal of Educational and Behavioral Statistics, 32, 125–150.
39.
McCaffreyD. F.SassT. R.LockwoodJ. R.MihalyK. (2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy, 4, 572–606.
40.
McCallM.S.HauserC.CroninJ.KingsburyG.G.HouserR. (2006). Achievement gaps: An examination of differences in student achievement and growth (Policy Report). Portland, OR: Northwest Evaluation Association.
41.
MichelmoreK.DynarskiS. (2017). The gap within the gap: Using longitudinal data to understand income differences in educational outcomes. AERA Open, 3, 1–18.
42.
ParsonsE.KoedelC.PodgurskyM.EhlertM.XiangP. B. (2015). Incorporating end-of-course exam timing into educational performance evaluations. Journal of Research on Educational Effectiveness, 8, 130–147.
43.
RothsteinJ. (2010). Teacher quality in educational production: Tracking, decay, and student achievement. Quarterly Journal of Economics, 125, 175–214.
44.
RothsteinJ. (2017). Measuring the impacts of teachers: Comment. American Economic Review, 107, 1656–1684.
45.
SassT.HannawayJ.XuZ.FiglioD.FengL. (2012). Comparison of the value Added of teachers in high-poverty schools and teachers in lower poverty schools. Journal of Urban Economics, 72, 104–122.
46.
StacyB.GuarinoCWooldridgeW. (2018). Does the precision and stability of value-added estimates of teacher performance depend on the types of students they serve?Economics of Education Review, 64, 50–74.
47.
SteinbergM. P.DonaldsonM. L. (2016). The new educational accountability: Understanding the landscape of teacher evaluation in the post-NCLB era. Education Finance and Policy, 11, 340–359.
48.
U.S. Department of Agriculture, Food and Nutrition Services, Office of Research and Analysis. (2007). NSLP/SBP access, participation, eligibility, and certification study—Erroneous payments in the NSLP and SBP: Vol. I. Study findings. Alexandria, VA: U.S. Department of Agriculture.
49.
WintersM. A.CowenJ. M. (2013). Would a value-added system of retention improve the distribution of teacher quality? A simulation of alternative policies. Journal of Policy Analysis and Management, 32, 634–654.
ZamarroG.EngbergJ.SaavedraJ. E.SteeleJ. (2015). Disentangling disadvantage: Can we distinguish good teaching from classroom composition?Journal of Research on Educational Effectiveness, 8, 84–11.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
