Value-added models have become popular in research and pay-for-performance plans. While scholars have focused attention on some aspects of their validity (e.g., scoring procedures), others have received less scrutiny. This article focuses on the extent to which value-added scores correspond to other indicators of teacher and teaching quality. The authors compared 24 middle school mathematics teachers’ value-added scores, derived from a large (N = 222) district data set, to survey- and observation-based indicators of teacher quality, instruction, and student characteristics. This analysis found teachers’ value-added scores correlated not only with their mathematical knowledge and quality of instruction but also with the population of students they teach. Case studies illustrate problems that might arise in using value-added scores in pay-for-performance plans.
American Educational Research Association/American Psychological Association. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
2.
Amrein-BeardsleyA. (2008). Methodological concerns about the education value-added assessment system. Educational Researcher, 37(2), 65–75.
3.
BallD. L.ForzaniF. M. (2007). What makes education research “educational”?Educational Researcher, 36(9), 529–540.
4.
BallD. L.ThamesM. H.PhelpsG. (2008). Content knowledge for teaching: What makes it special?Journal of Teacher Education, 59(5), 389–407.
5.
BallouD.SandersW.WrightP. (2004). Controlling for student background in value-added assessment of teachers. Journal of Educational and Behavioral Statistics, 29(1), 37–65.
6.
BoydD.LankfordR. H.LoebS.WycoffJ. (2003). Understanding teacher labor markets: Implications for educational equity. In PleckiM. L.MonkD. H. (Eds.), School finance and teacher quality: American Education Finance Association 2003 yearbook (pp. 55–84). Larchmont, NY: Eye on Education.
GordonR.KaneT. J.StaigerD. O. (2006). Identifying effective teachers using performance on the job (Discussion paper 2006-01). Washington, DC: Brookings Institution.
18.
GrubbW. N. (2008). Multiple resources, multiple outcomes: Testing the “improved” school finance with NELS88. American Educational Research Journal, 45(1), 104–144.
19.
HanushekE. A. (2007). The single salary schedule and other issues of teacher pay. Peabody Journal of Education, 82(4), 574–586.
20.
HanushekE. A.KainJ. F.O’BrienD. M.RivkinS. G. (2005). The market for teacher quality. Unpublished manuscript.
21.
HillH. C. (2007). Mathematical knowledge of middle school teachers: Implications for the No Child Left Behind Policy initiative. Educational Evaluation and Policy Analysis, 29, 95–114.
22.
HillH. C.BlunkM.CharalambousC.LewisJ.PhelpsG. C.SleepL.BallD. L. (2008). Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study. Cognition and Instruction, 26(4), 430–511.
23.
HillH. C.RowanB.BallD. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371–406.
24.
IshiiJ.RivkinS. G. (2009). Impediments to the estimation of teacher value added. Education Finance and Policy, 4(4), 520–536.
25.
JacobB. A.LefgrenL. (2005). Principals as agents: Subjective performance measurement in education (Faculty Research Working Paper Series RWP05-040). Cambridge, MA: Harvard University, John F. Kennedy School of Government.
26.
KaneM. (2001). Current concerns in validity theory. Journal of Educational Measurement, 38(4), 319–342.
27.
KaneM. (2004). Certification testing as an illustration of argument-based approach validation. Measurement: Interdisciplinary Research and Perspectives, 2(3), 135–170.
28.
KaneM. (2006). Validation. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 18–64). Westport, CT: Praeger.
29.
KaneT. J.RockoffJ. E.StaigerD. O. (2006). What does certification tell us about teacher effectiveness? Evidence from New York City. Unpublished manuscript, Harvard University.
30.
KimballS. M.WhiteB.MilanowskiA. T. (2004). Examining the relationship between teacher evaluation and student assessment results in Washoe County. Peabody Journal of Education, 79(4), 54–78.
KoedelC.BettsJ. R. (in press). Does student sorting invalidate value-added models of teacher-effectiveness? An extended analysis of the Rothstein critique. Education Finance and Policy.
33.
KoretzD. (2008). Limitations in the use of achievement tests. Journal of Human Resources, 27(4), 752–777.
34.
KupermintzH. (2003). Teacher effects and teacher effectiveness: A validity investigation of the Tennessee value added assessment system. Educational Evaluation and Policy Analysis, 25, 287–298.
LinnR. L. (2008). Methodological issues in achieving school accountability. Journal of Curriculum Studies, 40(6), 699–711.
37.
LockwoodJ. R.LouisT. A.McCaffreyD. F. (2002). Uncertainty in rank estimation: Implications for value-added modeling accountability systems. Journal of Educational and Behavioral Statistics, 27(3), 255–270.
38.
LockwoodJ. R.McCaffreyD. F.HamiltonL. S.StecherB.LeV.MartinezJ. F. (2007). The sensitivity of value-added teacher effect estimates to different mathematics achievement measures. Journal of Educational Measurement, 44(1), 47–67.
39.
LockwoodJ. R.McCaffreyD. F.MarianoL. T.SetodjiC. M. (2007). Bayesian methods for scalable multivariate value-added assessment. Journal of Educational and Behavioral Statistics, 32(2), 125–150.
40.
McCaffreyD. F.KoretzD.LockwoodJ. R.HamiltonL. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND.
41.
McCaffreyD. F.LockwoodJ.KoretzD.LouisT. A.HamiltonL. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29(1), 67–101.
42.
McCaffreyD. F.SassT. R.LockwoodJ. R.MihalyK. (2009). The intertemporal variability of teacher effect estimates. Education, Finance and Policy, 4(4), 572–606.
43.
MedleyD. M.CokerH. (1987). How valid are principals’ judgments of teacher effectiveness?Phi Delta Kappan, 69, 138–140.
44.
MessickS. (1988). The once and future issues of validity. Assessing the meaning and consequences of measurement. In WainerH.BraunH. (Eds.), Test validity (pp. 33–45). Hillsdale, NJ: Lawrence Erlbaum.
45.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13–103). New York: American Council on Education and Macmillan.
46.
MilanowskiA. (2004). The relationship between teacher performance evaluation scores and student achievement: Evidence from Cincinnati. Peabody Journal of Education, 79(4), 33–53.
47.
NyeB.KonstantopoulosS.HedgesL. V. (2004). How large are teacher effects?Educational Evaluation and Policy Analysis, 26, 237–257.
48.
PapayJ. P. (in press). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal.
49.
PiantaR. C.BelskyJ.VandergriftN.HoutsR.MorrisonF. J. (2008). Classroom effects on children’s achievement trajectories in elementary school. American Educational Research Journal, 45(2), 365–397.
50.
RockoffJ. E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94(2), 247–252.
51.
RockoffJ. E.StaigerD. O.KaneT. J.TaylorE. S. (2010). Information and employee evaluation: Evidence from a randomized intervention in public schools (Working paper 16240). Cambridge, MA: National Bureau of Economic Research. Retrieved from http://www.nber.org/papers/w16240
52.
RowanB.CorrentiR.MillerR. J. (2002). What large-scale, survey research tells us about teacher effects on student achievement: Insights from the “Prospects” study of elementary schools. Teachers College Record, 104(8), 1525–1567.
53.
SandersW. L. (2006, October). Comparisons among various educational assessment value-added models. Paper presented at the Power of Two—National Value-Added Conference, Columbus, OH.
54.
SandersW. L.RiversJ. C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville: University of Tennessee, Value-Added Research and Assessment Center.
55.
SandersW. L.SaxtonA. M.HornS. P. (1997). The Tennessee value-added assessment system, a quantitative, outcomes-based approach to educational measurement. In MillmanJ. (Ed.), Grading teachers, grading schools: Is student achievement a valid evaluation measure? (pp. 137–162). Thousand Oaks, CA: Corwin Press.
SchacterJ.ThumY. M. (2004). Paying for high and low-quality teaching. Economics of Education Review, 23, 411–430.
58.
TekweC. D.CarterR. L.MaC.AlginaJ.LucasM. E.RothJ.ResnickM. B. (2004). An empirical comparison of statistical models for value-added assessment of school performance. Journal of Educational and Behavioral Statistics, 29(1), 11–35.
59.
WeerasingheD. (2008). How to compute school and classroom effective indices: The value-added model implemented in Dallas Independent School District. Dallas, TX: Office of Institutional Research Dallas Independent School District. Retrieved July 16, 2010, from https://mydata.dallasisd.org/docs/CEI/SEI_CEI_Research.pdf
60.
WeingartenR. (2007, October14). How to fix No Child Left Behind. New York Times, p. WK7.
61.
WeisbergD.SextonS.MulhernJ.KeelingD. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. Brooklyn, NY: New Teacher Project.
WrightS. P.HornS. P.SandersW. L. (1997). Teacher and classroom context effects on student achievement: Implications for teacher evaluation. Journal of Personnel Evaluation in Education, 1(1), 57–67.
