We examine whether one of the most popular observation systems in teacher evaluation—the Framework for Teaching (FFT)—captures the range of instructional skills teachers need to be effective. We focus on the case of special educators, who are likely to use instructional approaches that, although supported by research, are de-emphasized in common observation systems. Drawing on 206 lessons from 51 teachers, we compare FFT scores to an observation system from special education. We find that FFT’s psychometric properties are consistent with previous studies, but the system is limited in assessing the quality of instructional practices used in special education. We discuss implications of these findings for two practical uses of observations—supporting teacher development and informing human capital decisions.
Al OtaibaS.ConnorC. M.FolsomJ. S.GreulichL.MeadowsJ.LiZ. (2011). Assessment data-informed guidance to individualize kindergarten reading instruction: Findings from a cluster-randomized control field trial. Elementary School Journal, 111(4), 535–560. https://doi.org/10.1086/659031
2.
BakerS. K.GerstenR.HaagerD.DingleM. (2006). Teaching practice and the reading growth of first-grade English learners: Validation of an observation instrument. The Elementary School Journal, 107(2), 199–220.
3.
BaysD. A.CrockettJ. B. (2007). Investigating instructional leadership for special education. Exceptionality, 15(3), 143–161.
4.
BellC. A.DobbelaerM. J.KletteK.VisscherA. (2019). Qualities of classroom observation systems. School Effectiveness and School Improvement, 30(1), 3–29. https://doi.org/10.1080/09243453.2018.1539014
5.
BellC. A.GitomerD. H.McCaffreyD.HamreB.PiantaR.QiY. (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87.
6.
BellC. A.JonesN.LewisJ.QiY.SticklerL.LiuS.McLeodM. (2016). Understanding consequential assessment systems of teaching: Year 1 final report to Los Angeles Unified School District (Research Memorandum No. RM-16-12). Educational Testing Service.
7.
BellC. A.JonesN.QiYLewisJ. (2018). Strategies for assessing classroom teaching: Examining administrator thinking as validity evidence. Educational Assessment, 23(4), 229–249.
8.
BellC. A.QiY.CroftA. J.LeusnerD.McCaffreyD. F.GitomerD. H.PiantaR. C. (2014). Improving observational score quality: Challenges in observer thinking. In KaneT. J.KerrK. A.PiantaR. C. (Eds.), Designing teacher evaluation systems: New guidance from the measures of effective teaching project (pp. 50–97). Wiley.
9.
BettiniE.Mason-WilliamsL.BarberB. (2019). Access to qualified, well-supported principals across alternative educational settings and neighborhood schools. Journal of Special Education, 53(4), 195–205. https://doi.org/10.1177/0022466919831302
10.
BoeE. (2014). Teacher demand, supply, and shortage in special education: A national perspective. In SindelarP.McCrayE.BrownellM. T.Lignugaris-KraftB. (Eds.), Handbook of research on special education teacher preparation (pp. 67–93). Routledge.
11.
BrennanR. L. (2001). Generalizability theory. Springer.
12.
BriggsD. C.DadeyN.KizilR. C. (2014). Comparing student growth and teacher observation to principal judgments in the evaluation of teacher effectiveness. University of Colorado.
13.
BrownellM. T.BishopA. B.GerstenR.KlingnerJ.DiminoJ.HaagerD.MenonS.PenfieldR.SindelarP. T. (2009). The role of domain expertise in beginning special education teacher quality. Exceptional Children, 75(4), 391–411.
14.
CasabiancaJ. M.LockwoodJ. R.McCaffreyD. F. (2015). Trends in classroom observation scores. Educational and Psychological Measurement, 75(2), 311–337. https://doi.org/10.1177/0013164414539163
15.
CashA. H.PiantaR. C. (2014). The role of scheduling in observing teacher–child interactions. School Psychology Review, 43(4), 428–449.
16.
ChudowskyN.ChudowskyV.KeberN. (2009). State test score trends through 2007–08, part 4: Has progress been made in raising achievement for students with disabilities?Center on Education Policy.
17.
ClementsD. H.AgodiniR.HarrisB. (2013). Instructional practices and student achievement: Correlations from a study of math curricula [NCEE evaluation brief]. U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
18.
CohenJ. (2018). Practices that cross disciplines? Revisiting explicit instruction in elementary mathematics and English language arts. Teaching and Teacher Education, 69(1), 324–335. https://doi.org/10.1016/j.tate.2017.10.021
19.
ConnorC. M.JakobsonsL. J.CroweE. C.MeadowsJ. G. (2009). Instruction, student engagement, and reading skill growth in Reading First classrooms. The Elementary School Journal, 109(3), 221–250.
20.
ConnorC. M.MazzoccoM. M.KurzT.CroweE. C.TigheE. L.WoodT. S.MorrisonF. J. (2018). Using assessment to individualize early mathematics instruction. Journal of School Psychology, 66, 97–113. https://doi.org/10.1016/j.jsp.2017.04.005
21.
ConnorC. M.MorrisonF. J. (2016). Individualizing student instruction in reading: Implications for policy and practice. Policy Insights From the Behavioral and Brain Sciences, 3(1), 54–61.
ConnorC. M.MorrisonF. J.PetrellaJ. N. (2004). Effective reading comprehension instruction: Examining child x instruction interactions. Journal of Educational Psychology, 96(4), 682–698.
24.
CroftM.GuffyG.VitaleD. (2018). The shrinking use of growth: Teacher evaluation legislation since ESSA. ACT.
25.
CronbachL. J. (1988). Five perspectives on validation argument. In WainerH.BraunH. (Eds.), Test validity (pp. 3–17). Lawrence Erlbaum.
26.
CronbachL. J.GleserG. C.NandaH.RajaratnamM. (1972). The dependability of behavioral measurement: Theory of generalizability for scores and profiles. John Wiley.
27.
DanielsonC. (1996). Enhancing professional practice: A framework for teaching. Association for Supervision and Curriculum Development.
28.
DanielsonC. (2013). The framework for teaching evaluation instrument, 2013 edition. The Danielson Group.
29.
DeweyJ.SindelarP. T.BettiniE.BoeE. E.RosenbergM. S.LekoC. (2017). Explaining the decline in special education teacher employment from 2005 to 2012. Exceptional Children, 83(3), 315–329. https://doi.org/10.1177/0014402916684620
30.
DoablerC. T.BakerS. K.KostyD. B.SmolkowskiK.ClarkeB.MillerS. J.FienH. (2015). Examining the association between explicit mathematics instruction and student mathematics achievement. The Elementary School Journal, 115(3), 303–333. https://doi.org/10.1086/679969
31.
DonaldsonM. L.WoulfinS. (2018). From tinkering to going “rogue”: How principals use agency when enacting new teacher evaluation systems. Educational Evaluation and Policy Analysis, 40(4), 531–556. https://doi.org/10.3102/0162373718784205
32.
FergusonR. F.DanielsonC. (2014). How framework for teaching and Tripod 7Cs distinguish key components of effective teaching. In KaneT. J.KerrK. A.PiantaR. C. (Eds.), Designing teacher evaluation systems: New guidance from the measures of effective teaching project (pp. 98–143). Wiley.
33.
FuchsL. S.Newman-GoncharR.SchumacherR.DoughertyB.BuckaN.KarpK. S.WoodwardJ.ClarkeB.JordanN. C.GerstenR.JayanthiM.KeatingB.MorganS. (2021). Assisting students struggling with mathematics: Intervention in the elementary grades (WWC 2021006). National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. http://whatworks.ed.gov/.ffuchs
34.
FuchsL. S.PowellS. R.SeethalerP. M.CirinoP. T.FletcherJ. M.FuchsD.HamlettC. L.ZumetaR. O. (2009). Remediating number combination and word problem deficits among students with mathematics difficulties: A randomized control trial. Journal of Educational Psychology, 101(3), 561–576. https://doi.org/10.1037/a0014701
35.
GilmourA. F.JonesN. D. (2020). Policies that define instruction: A systematic review of states’ and districts’ recommendations for evaluating special educators. Educational Researcher, 49(9), 645–655. https://doi.org/10.3102/0013189X20935039
36.
HamreB. K.PiantaR. C.MashburnA. J.DownerJ. T. (2012). Promoting young children’s social competence through the preschool PATHS curriculum and MyTeachingPartner professional development resources. Early Education & Development, 23(6), 809–832. https://doi.org/10.1080/10409289.2011.607360
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64. https://doi.org/10.3102/0013189X12437203
39.
HillH. C.GrossmanP. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371–384.
40.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel [Research paper, MET project]. Bill & Melinda Gates Foundation.
41.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55.
42.
JacobB. A.LefgrenL. (2009). Can principals identify effective teachers? Evidence on subjective performance evaluation in education. Journal of Labor Economics, 26(1), 101–136. https://doi.org/10.1086/522974
43.
JohnsonE. S.CrawfordA. R.ZhengY.MoylanL. A. (2020). Does special educator effectiveness vary depending on the observation instrument used?Educational Measurement: Issues and Practice, 40(1), 36–43.
44.
JohnsonE. S.ZhengY.CrawfordA. R.MoylanL. A. (2021). The relationship of special education teacher performance on observation instruments with student outcomes. Journal of Learning Disabilities, 54(1), 54–65. https://doi.org/10.1177/0022219420908906
45.
JonesN. D.BrownellM. T. (2014). Examining the use of classroom observations in the evaluation of special education teachers. Assessment for Effective Intervention, 39(2), 112–124. https://doi.org/10.1177/1534508413514103
46.
KalyugaS. (2007). Expertise reversal effect and its implications for learner-tailored instruction. Educational Psychology Review, 19(4), 509–539.
47.
KaneM. T. (2006). Validation. In BrennanR. (Ed.), Educational measurement (4th ed., pp. 17–64). American Council on Education/Praeger.
48.
KaneM. T.WoolsS. (2019). Perspectives on the validity of classroom assessments. In BrookhartS. M.McMillanJ. H. (Eds.), Classroom assessment and educational measurement (1st ed., pp. 11–26). Routledge.
49.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Measures of Effective Teaching Project, Bill & Melinda Gates Foundation.
50.
KirschnerP. A.SwellerJ.ClarkR. E. (2006). Why minimal guidance during instruction does not work: An analysis of the failure of constructivist, discovery, problem-based, experiential, and inquiry-based teaching. Educational Psychologist, 41, 75–86. https://doi.org/10.1207/s15326985ep4102_1
51.
KlingnerJ. K.BrownellM.MasonL. H.SindelarP. T.BenedictA. E.GriffinC.ParkY. (2016). Teaching students with special needs in the new millennium. In GitomerD.BellC. (Eds.), Handbook of research on teaching (5th ed., pp. 639–717). American Educational Research Association.
52.
KraftM. A.GilmourA. F. (2017). Revisiting the widget effect: Teacher evaluation reforms and the distribution of teacher effectiveness. Educational Researcher, 46(5), 234–249.
53.
LeV. Y.StecherB. M.LockwoodJ. R.HamiltonL. S.RobynA.WilliamsV. L.RyanG. W.KerrK. A.MartinezJ. F.KleinS. P. (2006). Improving mathematics and science education: A longitudinal investigation of the relationship between reform-oriented instruction and student achievement. Rand Corporation.
54.
LiuS.BellC. A.JonesN. D.McCaffreyD. F. (2019). Classroom observation systems in context: A case for the validation of observation systems. Educational Assessment, Evaluation and Accountability, 31(1), 61–95.
55.
MantzicopoulosP.FrenchB. F.PatrickH.WatsonJ. S.AhnI. (2018). The stability of kindergarten teachers’ effectiveness: A generalizability study comparing the framework for teaching and the classroom assessment scoring system. Educational Assessment, 23(1), 24–46. https://doi.org/10.1080/10627197.2017.1408407
56.
MartinA. J. (2016). Using load reduction instruction (LRI) to boost motivation and engagement. British Psychological Society.
57.
MashburnA. J.DownerJ. T.RiversS. E.BrackettM. A.MartinezA. (2014). Improving the power of an efficacy study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes. Prevention Science, 15(2), 146–155. https://doi.org/10.1007/s11121-012-0357-3
58.
MayerR. E. (2003). Memory and information processes. In ReynoldsW. M.MillerG. E. (Eds.), Handbook of psychology (pp. 47–58). Wiley.
59.
McCaffreyD. F.YuanK.SavitskyT. D.LockwoodJ. R.EdelenM. O. (2015). Uncovering multivariate structure in classroom observations in the presence of rater errors. Educational Measurement: Issues and Practice, 34(2), 34–46.
60.
MercerC. D.LaneH. B.JordanL.AllsoppD. H.EiseleM. R. (1996). Empowering teachers and students with instructional choices in inclusive settings. Remedial and Special Education, 17(4), 226–236.
61.
MeyerJ. P.CashA. H.MashburnA. (2011). Occasions and the reliability of classroom observations: Alternative conceptualizations and methods of analysis. Educational Assessment, 16(4), 227–243. https://doi.org/10.1080/10627197.2011.638884
62.
MorganP. L.FarkasG.MaczugaS. (2015). Which instructional practices most help first-grade students with and without mathematics difficulties?Educational Evaluation and Policy Analysis, 37(2), 184–205. https://doi.org/10.3102/0162373714536608
63.
Morris-MathewsH.StarkK.JonesN. D.BrownellM.BellC. (2020). Danielson’s framework for teaching: Convergence and divergence with conceptions of effectiveness in special education. Journal of Learning Disabilities, 54(1), 66–78. https://doi.org/10.1177/0022219420941804
64.
OliveriM.McCaffreyD.EzzoC.HoltzmanS. (2017). A multilevel factor analysis of third-party evaluations of noncognitive constructs used in admissions decision making. Applied Measurement in Education, 30(4), 297–313. https://doi.org/10.1080/08957347.2017.1353989
65.
PraetoriusA.-K.PauliC.ReusserK.RakoczyK.KliemeE. (2014). One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction, 31, 2–12. https://doi.org/10.1016/j.learninstruc.2013.12.002
SchulteA. C.StevensJ. J. (2015). Once, sometimes, or always in special education: Mathematics growth and achievement gaps. Exceptional Children, 81(3), 370–387. https://doi.org/10.1177/0014402914563695
68.
SchulteA. C.StevensJ. J.ElliottS. N.TindalG.NeseJ. F. T. (2016). Achievement gaps for students with disabilities: Stable, widening, or narrowing on a state-wide reading comprehension test?Journal of Educational Psychology, 108(6), 925–942.
69.
SearleS. R.CasellaG.McCullochC. E. (2009). Variance components (Vol. 391). John Wiley.
70.
SemmelrothC. L.JohnsonE. (2014). Measuring rater reliability on a special education observation tool. Assessment for Effective Intervention, 39(3), 131–145. https://doi.org/10.1177/1534508413511488
71.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer (Vol. 1). SAGE.
72.
SmolkowskiK.GunnB. (2012). Reliability and validity of the Classroom Observations of Student–Teacher Interactions (COSTI) for kindergarten reading instruction. Early Childhood Research Quarterly, 27(2), 316–328.
73.
SnowR. (1989). Aptitude-treatment interaction as a framework for research on individual differences in learning. In AckermanP.SternbergR. J.GlaserR. (Eds.), Learning and individual differences (pp. 13–59). W. H. Freeman.
74.
SonnenscheinS.StapletonL. M.BensonA. (2010). The relation between the type and amount of instruction and growth in children’s reading competencies. American Educational Research Journal, 47(2), 358–389. https://doi.org/10.3102/0002831209349215
75.
SteinbergM. P.DonaldsonM. L. (2016). The new educational accountability: Understanding the landscape of teacher evaluation in the post-NCLB era. Education Finance and Policy, 11(3), 340–359. https://doi.org/10.1162/EDFP_a_00186
76.
SteinbrecherT. D.FixR.MahalS. A.SernaL.McKeownD. (2015). All you need is patience and flexibility: Administrators’ perspectives on special educator knowledge and skills. Journal of Special Education Leadership, 28(2), 89–102.
77.
StelitanoL.RussellJ. L.BrayL. E. (2020). Organizing for meaningful inclusion: Exploring the routines that shape student supports in secondary schools. American Educational Research Journal, 57(2), 535–575. https://doi.org/10.3102/0002831219859307
78.
StockardJ.WoodT. W.CoughlinC.Rasplica KhouryC. (2018). The effectiveness of direct instruction curricula: A meta-analysis of a half century of research. Review of Educational Research, 88(4), 479–507. https://doi.org/10.3102/0034654317751919
79.
SwellerJ. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285.
80.
TaylorE. S. (2018). Skills, job tasks, and productivity in teaching: Evidence from a randomized trial of instruction practices. Journal of Labor Economics, 36(3), 711–742.
81.
VadasyP. F.SandersE. A.PeytonJ. A. (2005). Relative effectiveness of reading practice or word-level instruction in supplemental tutoring: How text matters. Journal of Learning Disabilities, 38(4), 364–380. https://doi.org/10.1177/00222194050380041401
82.
VaughnS.GerstenR.ChardD. (2000). The underlying message in LD intervention research: Findings from research syntheses. Exceptional Children, 67(1), 99–114. https://doi.org/10.1177/001440290006700107
83.
VaughnS.KlingnerJ. K.SwansonE. A.BoardmanA. G.RobertsG.MohammadS. S.Stillman-SpisakS. J. (2011). Efficacy of collaborative strategic reading with middle school students. American Educational Research Association, 48(4), 938–964. https://doi.org/10.3102/0002831211410305
84.
WanzekJ.VaughnS.ScammaccaN. K.MetzK.MurrayC. S.RobertsG.DanielsonL. (2013). Extensive reading interventions for students with reading difficulties after grade 3. Review of Educational Research, 83(2), 163–195. https://doi.org/10.3102/0034654313477212
85.
WeisbergD.SextonS.MulhernJ.KeelingD. (2009). The widget effect: Our national failure to acknowledge and act on differences in teacher effectiveness. The New Teacher Project.
86.
XueY.MeiselsS. J. (2004). Early literacy instruction and learning in kindergarten: Evidence from the Early Childhood Longitudinal Study–Kindergarten Class of 1998-99. American Educational Research Journal, 41(1), 191–229. https://doi.org/10.3102/00028312041001191
