Teacher preparation programs are increasingly expected to use data on preservice teacher (PST) skills to drive program improvement and provide targeted supports. Observational ratings are especially vital, but also prone to measurement issues. Scores may be influenced by factors unrelated to PSTs’ instructional skills, including rater standards. Yet we know little about how these measurement challenges play out in the preservice context specifically. Here, we investigate the reliability and sensitivity of two observational measures. We find measures collected during student teaching are especially prone to measurement issues; only 3% to 4% of variation in scores reflects consistent differences between PSTs, while 9% to 17% of variation can be attributed to the mentors with whom they work. When high scores stem not from strong instructional skills, but instead from external circumstances, we cannot use them to make consequential decisions about PSTs’ individual needs or readiness for independent teaching.
AllenJ.GregoryA.MikamiA.LunJ.HamreB.PiantaR. (2013). Observations of effective teacher–student interactions in secondary school classrooms: Predicting student achievement with the classroom assessment scoring system—Secondary. School Psychology Review, 42(1), 76–98.
2.
AllenJ.HafenC. A.GregoryA. C.MikamiA. Y.PiantaR. (2015). Enhancing secondary school instruction and student achievement: Replication and extension of the My Teaching Partner-secondary intervention. Journal of Research on Educational Effectiveness, 8(4), 475–489.
3.
AllenM.CobleC. (2018). Creating a data culture in educator preparation: The role of the states. In MandinachE.GummerE. (Eds.), Data for continuous programmatic improvement (pp. 35–67). Routledge.
AnglinK. L.WongV. C.BoguslavA. (2021). A natural language processing approach to measuring treatment adherence and consistency using semantic similarity. AERA Open, 7, 23328584211028615.
6.
AraujoM. C.CarneiroP.Cruz-AguayoY.SchadyN. (2016). Teacher quality and learning outcomes in kindergarten. The Quarterly Journal of Economics, 131(3), 1415–1453.
7.
AtteberryA.LoebS.WyckoffJ. (2015). Do first impressions matter? Improvements in early career teacher effectiveness. AERA Open Access Journal, 1(4), 1–23.
8.
BartanenB.KwokA. (2020). Pre-service teacher quality and workforce entry (EdWorkingPaper No. 20-223). Annenberg Institute at Brown University.
9.
BartanenB.KwokA. (2021). Examining clinical teaching observation scores as a measure of preservice teacher quality. American Educational Research Journal, 58(5), 887–920.
10.
BassokD.MagouirkP.MarkowitzA. J. (2021). Systemwide quality improvement in early childhood education: Evidence from Louisiana. AERA Open, 7(1), 1–19.
11.
BastianK. C.HenryG. T.PanY.LysD. (2016). Teacher candidate performance assessments: Local scoring and implications for teacher preparation program improvement. Teaching and Teacher Education, 59, 1–12.
12.
BastianK. C.LysD.PanY. (2018). A framework for improvement: Analyzing performance-assessment scores for evidence-based teacher preparation program reforms. Journal of Teacher Education, 69(5), 448–462.
13.
BastianK. C.LysD. B.WhisenantW. R. L. (2023). Does placement predict performance? Associations between student teaching environments and candidates’ performance assessment scores. Journal of Teacher Education, 74(1), 40–54.
14.
BastianK. C.PattersonK. M.CarpenterD. (2022). Placed for success: Which teachers benefit from high-quality student teaching placements?Educational Policy, 36(7), 1583–1611.
15.
BellC. A.DobbelaerM. J.KletteK.VisscherA. (2019). Qualities of classroom observation systems. School Effectiveness and School Improvement, 30(1), 3–29.
16.
BellC. A.GitomerD. H.McCaffreyD. F.HamreB. K.PiantaR. C.QiY. (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87.
17.
BellC. A.QiY.CroftA. J.LeusnerD.McCaffreyD. F.GitomerD. H.PiantaR. C. (2015). Improving observational score quality: Challenges in observer thinking. In KaneT.KerrK.PiantaR. (Eds.), Designing teacher evaluation systems: New guidance from the measures of effective teaching project (pp. 50–97). John Wiley.
18.
BoydD.LankfordH.LoebS.RockoffJ.WyckoffJ. (2008). The narrowing gap in New York City teacher qualifications and its implications for student achievement in high-poverty schools. Journal of Policy Analysis and Management, 27(4), 793–818.
19.
BriggsD. C.AlzenJ. L. (2019). Making inferences about teacher observation scores over time. Educational and Psychological Measurement, 79(4), 636–664.
20.
CampbellS. L.RonfeldtM. (2018). Observational evaluation of teachers: Measuring more than we bargained for?American Educational Research Journal, 55(6), 1233–1267.
21.
CasabiancaJ. M.LockwoodJ. R.McCaffreyD. F. (2015). Trends in classroom observation scores. Educational and Psychological Measurement, 75(2), 311–337.
22.
CastlesA.RastleK.NationK. (2018). Ending the reading wars: Reading acquisition from novice to expert. Psychological Science in the Public Interest, 19(1), 5–51.
23.
CharneyR. S. (1993). Teaching children to care: Management in the responsive classroom. Northeast Foundation for Children.
24.
CohenJ.EricksonS.KrishnamachariA.WongV. (2023). Experimental evidence on the robustness of coaching supports in teacher education. Educational Researcher.
25.
CohenJ.GoldhaberD. (2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45(6), 378–387.
26.
CohenJ.WongV.KrishnamachariA.BerlinR. (2020). Teacher coaching in a simulated environment. Educational Evaluation and Policy Analysis, 43(2), 208–231.
DavisS. C.PeckC. A. (2020). Using data for program improvement in teacher education: A study of promising practices. Teachers College Record, 122(3), 1–48.
29.
DiekerL. A.RodriguezJ. A.Lignugaris/KraftB.HynesM. C.HughesC. E. (2014). The potential of simulated environments in teacher education: Current and future possibilities. Teacher Education and Special Education, 37(1), 21–33.
30.
FeuerM.FlodenR.ChudowskyN.AhnJ. (2013). Evaluation of teacher preparation programs: Purposes, methods, and policy options. National Academy of Education.
31.
ForzaniF. M. (2014). Understanding “core practices” and “practice-based” teacher education: Learning from the past. Journal of Teacher Education, 65(4), 357–368.
32.
GitomerD.BellC.QiY.McCaffreyD.HamreB. K.PiantaR. C. (2014). The instructional challenge in improving teaching quality: Lessons from a classroom observation protocol. Teachers College Record, 116(6), 1–32.
33.
GitomerD. (Ed.). (2009). Measurement issues and assessment for teaching quality. SAGE.
34.
GoldhaberD. (2019). Evidence-based teacher preparation: Policy context and what we know. Journal of Teacher Education, 70(2), 90–101.
35.
GoldhaberD.KriegJ.NaitoN.TheobaldR. (2020). Making the most of student teaching: The importance of mentors and scope for change. Education Finance and Policy, 15(3), 581–591.
36.
GoldhaberD.KriegJ.TheobaldR. (2020). Effective like me? Does having a more productive mentor improve the productivity of mentees?Labour Economics, 63, Article 101792.
37.
GoldhaberD.RonfeldtM.CowanJ.GratzT.BardelliE.TruwitM. (2022). Room for improvement? Mentor teachers and the evolution of teacher preservice clinical evaluations. American Educational Research Journal, 59(5), 1011–1048.
38.
GrossmanP.ComptonC.IgraD.RonfeldtM.ShahanE.WilliamsonP. (2009). Teaching practice: A cross-professional perspective. Teachers College Record, 111(9), 2055–2100.
39.
HafenC. A.HamreB. K.AllenJ. P.BellC. A.GitomerD. H.PiantaR. C. (2015). Teaching through interactions in secondary school classrooms: Revisiting the factor structure and practical application of the classroom assessment scoring system-secondary. The Journal of Early Adolescence, 35(5–6), 651–680.
40.
HenryG. T.CampbellS. L.ThompsonC. L.PatriarcaL. A.LuterbachK. J.LysD. B.CovingtonV. M. (2013). The predictive validity of measures of teacher candidate programs and performance: Toward an evidence-based approach to teacher preparation. Journal of Teacher Education, 64(5), 439–453.
41.
HillH.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64.
42.
HillH.GrossmanP. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371–384.
43.
HillH.MancenidoZ.LoebS. (2020). New research for teacher education (EdWorkingPaper No. 20-252). Annenberg Institute at Brown University.
44.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel. Bill & Melinda Gates Foundation.
45.
JanssenF.GrossmanP.WestbroekH. (2015). Facilitating decomposition and recomposition in practice-based teacher education: The power of modularity. Teaching and Teacher Education, 51, 137–146.
46.
KaneT.StaigerD. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Bill & Melinda Gates Foundation.
47.
KraftM. A.GilmourA. F. (2017). Revisiting the widget effect: Teacher evaluation reforms and the distribution of teacher effectiveness. Educational Researcher, 46(5), 234–249.
48.
LampertM.GrazianiF. (2009). Instructional activities as a tool for teachers’ and teacher educators’ learning. The Elementary School Journal, 109(5), 491–509.
49.
La ParoK. M.PiantaR. C.StuhlmanM. (2004). The classroom assessment scoring system: Findings from the prekindergarten year. The Elementary School Journal, 104(5), 409–426.
MalmbergL.-E.HaggerH.BurnK.MuttonT.CollsH. (2010). Observed classroom quality during teacher education and two years of professional practice. Journal of Educational Psychology, 102(4), 916–932.
52.
MancenidoZ. (2022). Impact evaluations of teacher preparation practices: Challenges and opportunities for more rigorous research (EdWorkingPaper No. 22–534). Annenberg Institute at Brown University.
53.
McDonaldM.KazemiE.KavanaghS. S. (2013). Core practices and pedagogies of teacher education: A call for a common language and collective activity. Journal of Teacher Education, 64(5), 378–386.
54.
PacheoA. (2009). Mapping the terrain of teacher quality. In GitomerD. (Ed.), Measurement issues and assessment for teaching quality (pp. 168–178). SAGE.
55.
PapayJ. (2012). Refocusing the debate: Assessing the purposes and tools of teacher evaluation. Harvard Educational Review, 82(1), 123–141.
56.
ParisD.AlimH. S. (2014). What are we seeking to sustain through culturally sustaining pedagogy? A loving critique forward. Harvard Educational Review, 84(1), 85–100. https://doi.org/10.17763/haer.84.1.982l873k2ht16m77
57.
ParkY. S.ChenJ.HoltzmanS. L. (2015). Evaluating efforts to minimize rater bias in scoring classroom observations. In KaneT.KerrK.PiantaR. (Eds.), Designing teacher evaluation systems: New guidance from the measures of effective teaching project (pp. 381–414). John Wiley.
58.
PeckC. A.McDonaldM. (2013). Creating “cultures of evidence” in teacher education: Context, policy, and practice in three high-data-use programs. The New Educator, 9(1), 12–28.
59.
PiantaR. C.HamreB. K. (2009). Classroom processes and positive youth development: Conceptualizing, measuring, and improving the capacity of interactions between teachers and students. New Directions for Youth Development, 121, 33–46.
ReznitskayaA.KuoL.-J.ClarkA.-M.MillerB.JadallahM.AndersonR. C.Nguyen-JahielK. (2009). Collaborative reasoning: A dialogic approach to group discussions. Cambridge Journal of Education, 39(1), 29–48.
62.
RonfeldtM. (2012). Where should student teachers learn to teach? Effects of field placement school characteristics on teacher retention and effectiveness. Educational Evaluation and Policy Analysis, 34(1), 3–26.
63.
RonfeldtM.BardelliE.TruwitM.MullmanH.SchaafK.BakerJ. C. (2020). Improving preservice teachers’ feelings of preparedness to teach through recruitment of instructionally effective and experienced cooperating teachers: A randomized experiment. Educational Evaluation and Policy Analysis, 42(4), 551–575.
64.
RonfeldtM.BrockmanS. L.CampbellS. L. (2018). Does cooperating teachers’ instructional effectiveness improve preservice teachers’ future performance?Educational Researcher, 47(7), 405–418.
65.
RonfeldtM.MatskoK. K.Greene NolanH.ReiningerM. (2021). Three different measures of graduates’ instructional readiness and the features of preservice preparation that predict them. Journal of Teacher Education, 72(1), 56–71.
66.
Texas State Legislature. Texas Senate Bill 174. (2009).
67.
U.S. Department of Education. (2011). Our future, our teachers: The Obama administration’s plans for teacher education reform and improvement.
68.
van der LansR. M.van de GriftW. J.van VeenK.Fokkens-BruinsmaM. (2016). Once is not enough: Establishing reliability criteria for feedback and evaluation decisions based on classroom observations. Studies in Educational Evaluation, 50, 88–95.
69.
WallaceT. L.ParrA. K.CorrentiR. J. (2020). Assessing teachers’ classroom management competency: A case study of the classroom assessment scoring system–secondary. Journal of Psychoeducational Assessment, 38(4), 475–492.
70.
WeisbergD.SextonS.MulhernJ.KeelingD.SchunckJ.PalciscoA.MorganK. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. New Teacher Project.
71.
WylieE. C. (2020). Observing formative assessment practice: Learning lessons through validation. Educational Assessment, 25(4), 251–258.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
