Education agencies are evaluating teachers using student achievement data. However, very little is known about the comparability of test-based or “value-added” metrics across districts and the extent to which they capture variability in classroom practices. Drawing on data from four urban districts, we found that teachers were categorized differently when compared within versus across districts. In addition, analyses of scores from two observation instruments, as well as qualitative viewing of lesson videos, identified stark differences in instructional practices across districts among teachers who received similar within-district value-added rankings. These patterns were not explained by observable background characteristics of teachers, suggesting that factors beyond labor market sorting likely played a key role.
AaronsonD.BarrowL.SanderW. (2007). Teachers and student achievement in the Chicago public high schools. Journal of Labor Economics, 25(1), 95–135.
2.
BellC. A.GitomerD. H.McCaffreyD. F.HamreB. K.PiantaR. C. (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87.
3.
BlazarD. (2015). Effective teaching in elementary mathematics: Identifying classroom practices that support student achievement. Economics of Education Review48, 16–29.
BoydD. J.GrossmanP. L.LankfordH.LoebS.WyckoffJ. (2009). Teacher preparation and student achievement. Educational Evaluation and Policy Analysis, 31(4), 416–440.
6.
BoydD.LankfordH.LoebS.WyckoffJ. (2004). The draw of home: How teachers’ preferences for proximity disadvantage urban schools. Journal of Policy Analysis and Management, 24(1), 113–132.
CharalambousC.HillH. C.McGinnD.ChinM. (2014). Teacher knowledge and student learning: Bringing together two different conceptualizations of teacher knowledge. Paper presented at the American Educational Research Association Annual Meeting, Philadelphia, PA.
9.
ChettyR.FriedmanJ. N.RockoffJ. E. (2014). Measuring the impacts of teachers I: Evaluating bias in teacher value-added estimates. American Economic Review, 104(9), 2593–2632.
10.
ClarkM. A.ChiangH. S.SilvaT.McConnellS.SonnenfeldK.ErbeA.PumaM. (2013). The effectiveness of secondary math teachers from Teach for America and the Teaching Fellows programs. Washington, DC: U.S. Department of Education.
11.
DeckerP. T.MayerD. P.GlazermanS. (2004). The effects of Teach for America on students: Findings from a national evaluation. Princeton, NJ: Mathematica Policy Research.
12.
DiMaggioP.PowellW. (1983). The iron cage revisited: Institutional isomorphism and collective rationality in organizational fields. American Sociological Review, 48(2), 147–160.
ElmoreR.BurneyD. (1999). Investing in teacher learning: Staff development and instructional improvement. In Darling-HammondL.SykesG. (Eds.), Teaching as the learning Profession: Handbook on policy and practice (pp. 263–291). San Francisco, CA: Jossey-Bass.
15.
FirestoneW. A.ManginM. M.MartinezM. C.PolovskyT. (2005). Leading coherent professional development: A comparison of three districts. Educational Administration Quarterly, 41(3), 413–448.
16.
GaretM. S.WayneA. J.StancavageF.TaylorJ.EatonM.WaltersK.Song. . . DoolittleF. (2011). Middle school mathematics professional development impact study: Findings after the second year of implementation (NCEE 2011-4024). Washington, DC: National Center for Education Evaluation and Regional Assistance.
17.
GoldhaberD.GrossB.PlayerD. (2011). Teacher career paths, teacher quality, and persistence in the classroom: Are public schools keeping their best?Journal of Policy Analysis and Management, 30(1), 57–87.
GordonR. J.KaneT. J.StaigerD. (2006). Identifying effective teachers using performance on the job. Washington, DC: Brookings Institution.
20.
GrossmanP.LoebS.CohenJ.WyckoffJ. (2013). Measure for measure: The relationship between measures of instructional practice in middle school English language arts and teachers’ value-added. American Journal of Education, 119(3), 445–470.
21.
GuarinoC.SantibañezL.DaleyG. (2006). Teacher recruitment and retention: A review of the recent empirical literature. Review of Educational Research, 76(2), 173–208.
22.
HanushekE. A. (2009). Teacher deselection. In GoldhaberD.HannawayJ. (Eds.), Creating a new teaching profession (pp. 165–180). Washington, DC: Urban Institute Press.
23.
HanushekE. A.KainJ. F.MarkmanJ. M.RivkinS. G. (2003). Does peer ability affect student achievement?Journal of Applied Econometrics, 18(5), 527–544.
24.
HanushekE. A.KainJ. F.RivkinS. G. (2004). Why public schools lose teachers. Journal of Human Resources, 39(2), 326–354.
25.
HarrisD. N. (2009). Would accountability based on teacher value added be smart policy? An examination of the statistical properties and policy alternatives. Education Finance and Policy, 4(4), 319–350.
26.
HickmanJ. J.FuJ.HillH. C. (2012). Technical report: Creation and dissemination of upper-elementary mathematics assessment modules. Princeton, NJ: Educational Testing Service.
27.
HightowerA. (2002). San Diego’s big boom: Systematic instructional change in the central office and schools. In HightowerA.KnappM.MarshJ.McLaughlinM. (Eds.), School districts and instructional renewal (pp. 61–75). New York, NY: Teachers College Press.
28.
HillH. C. (2007). Mathematical knowledge of middle school teachers: Implications for the No Child Left Behind policy initiative. Educational Evaluation and Policy Analysis, 29(2), 95–114.
29.
HillH. C.BlazarD.LynchK. (2015). Resources for teaching: Examining personal and institutional predictors of high-quality instruction. AERA Open, 1(4), 1–23.
30.
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64.
31.
HillH. C.KapitulaL.UmlandK. (2011). A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal, 48(3), 794–831.
32.
HillH. C.RowanB.BallD. L. (2005). Effects of teachers’ mathematical knowledge for teaching on student achievement. American Educational Research Journal, 42(2), 371–406.
33.
HillH. C.SchillingS. G.BallD. L. (2004). Developing measures of teachers’ mathematics knowledge for teaching. Elementary School Journal, 105(1), 11–30.
34.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel. Seattle, WA: Measures of Effective Teaching Project, Bill and Melinda Gates Foundation.
35.
JacobB. A. (2007). The challenges of staffing urban schools with effective teachers. The Future of Children, 17(1), 129–153.
36.
JohnsonR. B.OnwuegbuzieA. J. (2004). Mixed methods research: A research paradigm whose time has come. Educational Researcher, 33(7), 14–26.
37.
KaneT. J.McCaffreyD. F.MillerT.StaigerD. O. (2013). Have we identified effective teachers? Validating measures of effective teaching using random assignment. Seattle, WA: Measures of Effective Teaching Project, Bill and Melinda Gates Foundation.
38.
KaneT. J.StaigerD. O. (2008). Estimating teacher impacts on student achievement: An experimental evaluation (No. w14607). Cambridge, MA: National Bureau of Economic Research.
39.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Measures of Effective Teaching Project, Bill and Melinda Gates Foundation.
40.
KupermintzH. (2003). Teacher effects and teacher effectiveness: A validity investigation of the Tennessee Value Added Assessment System. Educational Evaluation and Policy Analysis, 25(3), 287–298.
41.
LampertM. (2001). Teaching problems and the problems of teaching. New Haven, CT: Yale University Press.
42.
LankfordH.LoebS.WyckoffJ. (2002). Teacher sorting and the plight of urban schools: A descriptive analysis. Educational Evaluation and Policy Analysis, 24(1), 37–62.
43.
LittleJ. W. (1989). District policy choices and teacher’s professional development opportunities. Educational Evaluation and Policy Analysis, 11(2), 165–179.
44.
LockwoodJ. R.McCaffreyD. F.HamiltonL. S.StecherB.LeV.MartinezJ. F. (2007). The sensitivity of value-added teacher effect estimates to different mathematics achievement measures. Journal of Educational Measurement, 44(1), 47–67.
45.
LynchK.ChinM.BlazarD. (2015). Relationship between observations of elementary teacher mathematics instruction and student achievement: Exploring variability across districts (Working paper). Cambridge, MA: National Center for Teacher Effectiveness, Harvard University.
46.
McCaffreyD. F.LockwoodJ. R.KoretzD. M.HamiltonL. S. (2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND Corporation.
47.
McCaffreyD. F.SassT. R.LockwoodJ. R.MihalyK. (2009). The intertemporal variability of teacher effect estimates. Education Finance and Policy, 4(4), 572–606.
48.
MetzlerJ.WoessmannL. (2012). The impact of teacher subject knowledge on student achievement: Evidence from within-teacher within-student variation. Journal of Development Economics, 99(2), 486–496.
49.
MeyerJ. W.RowanB. (1977). Institutionalized organizations: Formal structure as myth and ceremony. American Journal of Sociology, 83(2), 340–363.
50.
MurnaneR. J.OlsenR. (1990). The effects of salaries and opportunity costs on length of stay in teaching: Evidence from North Carolina. Journal of Human Resources, 25(1), 106–124.
51.
National Council of Teachers of Mathematics. (1989). Curriculum and evaluation standards for school mathematics. Reston, VA: Author.
52.
National Council of Teachers of Mathematics. (1991). Professional standards for teaching mathematics. Reston, VA: Author.
53.
National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: Author.
54.
NewtonX. A.Darling-HammondL.HaertelE.ThomasE. (2010). Value-added modeling of teacher effectiveness: An exploration of stability across models and contexts. Education Policy Analysis Archives, 18(23), 1–27.
55.
PapayJ. P. (2011). Different tests, different answers: The stability of teacher value-added estimates across outcome measures. American Educational Research Journal, 48(1), 163–193.
56.
PapayJ. P. (2012). Refocusing the debate: Assessing the purposes and tools of teacher evaluation. Harvard Educational Review, 82(1), 123–141.
57.
PiantaR. C.HamreB. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38(2), 109–119.
58.
PorterA. C. (2002). Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3–14.
59.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods. Second Edition. Thousand Oaks, CA: Sage Publications.
60.
RothsteinJ. (2010). Teacher quality in educational production: Tracking, decay, and student achievement. The Quarterly Journal of Economics, 125(1) 175–214.
61.
SchochetP. Z.ChiangH. S. (2013). What are error rates for classifying teacher and school performance using value-added models?Journal of Educational and Behavioral Statistics, 38(2), 142–171.
62.
SpillaneJ. P. (2000). Cognition and policy implementation: District policymakers and the reform of mathematics education. Cognition and Instruction, 18(2), 141–179.
63.
StrongeJ. H.WardT. J.GrantL. W. (2011). What makes good teachers good? A cross-case analysis of the connection between teacher effectiveness and student achievement. Journal of Teacher Education, 62(4), 339–355.
64.
ThumY. M.BrykA. S. (1997). Value-added productivity indicators: The Dallas system. In MillmanJason (Ed.) Grading teachers, grading schools: Is student achievement a valid evaluation measure? (pp. 100–119). Thousand Oaks, CA: Corwin.
65.
TurnerJ. C.MeyerD. K. (2000). Studying and understanding the instructional contexts of classrooms: Using our past to forge our future. Educational Psychologist, 35(2), 69–85.
66.
WayneA. J.YoungsP. (2003). Teacher characteristics and student achievement gains: A review. Review of Educational Research, 73(1), 89–122.
67.
WestM. R.ChingosM. M. (2009). Teacher effectiveness, mobility, and attrition in Florida. In SpringerM. G. (Ed.), Performance incentives: Their growing impact on American K–12 education (pp. 251–271): Washington, DC: Brookings Institution Press.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
