In this study, we developed an Explicit Instruction special education teacher observation rubric that details the elements of explicit instruction and tested its psychometric properties using many-facet Rasch measurement (MFRM). Video observations of classroom instruction from 30 special education teachers across three states were collected. External raters (n = 15) were trained to observe and evaluate instruction using the rubric and assigned scores of “implemented,” “partially implemented,” or “not implemented” for each of the items. Analyses showed that the item, teacher, lesson, and rater facets achieved high psychometric quality for the instrument. Implications for research and practice are discussed.
AdnotM.DeeT.KatzV.WyckoffJ. (2017). Teacher turnover, teacher quality, and student achievement in DCPS. Educational Evaluation and Policy Analysis, 39, 54–76.
2.
AndersonL. M.ButlerA.PalmiterA.ArcairaE. (2016). Study of emerging teacher evaluation systems. Washington, DC: Policy and Program Studies Service, Office of Planning, Evaluation and Policy Development.
3.
BakerS. K.GerstenR.HaagerD.DingleM. (2006). Teaching practice and the reading growth of first-grade English learners: Validation of an observation instrument. The Elementary School Journal, 107, 199–219.
4.
BellC. A.YiQ.CroftA. J.LeusnerD.MccaffreyD. F.GitomerD. H.PiantaR. C. (2015). Improving Observational Score Quality. Designing Teacher Evaluation Systems (2015): 50–97.
5.
BellC.GitomerD.McCaffreyD.HamreB.PiantaR.QiY. (2012). An argument approach to observation protocol validity. Educational Assessment, 17, 62–87.
6.
BiancarosaG.BrykA. S.DexterE. R. (2010). Assessing the value-added effects of literacy collaborative professional development on student learning. The Elementary School Journal, 11, 7–34.
7.
BlazarD.BraslowD.CharalambosY. C.HillH. C. (2017). Attending to general and specific dimensions of teaching: Exploring factors across two observation instruments. Educational Assessment, 22, 71–94. doi:10.1080/10627197.2017.1309274
8.
BondT. G.FoxC. M. (2007). Fundamental measurement in the human sciences. Chicago, IL: Institute for Objective Measurement.
9.
CasabiancaJ. M.LockwoodJ. R.McCaffreyD. F. (2015). Trends in classroom observation scores. Educational and Psychological Measurement, 75, 311–337.
10.
CashA. H.HamreB. K.PiantaR. C.MyersS. S. (2012). Rater calibration when observational assessment occurs at large scale: Degree of calibration and characteristics associated with calibration. Early Childhood Research Quarterly, 27, 529–542.
11.
CiulloS.LembkeE. S.CarlisleA.Newman ThomasC.GoodwinM.JuddL. (2016). Implementation of evidence-based literacy practices in middle school response to intervention: An observation study. Learning Disability Quarterly, 39, 44–57.
12.
DanielsonC. (2011). The framework for teaching evaluation instrument. Princeton, NJ: The Danielson Group.
13.
DoablerC. T.ClarkeB.StoolmillerM.KostyD. B.FienH.SmolkowskiK.BakerS. K. (2017). Explicit instructional interactions: Exploring the black box of a tier 2 mathematics intervention. Remedial and Special Education, 38, 98–110.
14.
EckesT. (2011). Introduction to many-facet Rasch measurement. Frankfurt, Germany: Peter Lang.
15.
EngelhardG.Jr. (1992). The measurement of writing ability with a many-faceted Rasch model. Applied Measurement in Education, 5, 171–191.
16.
GerstenR. M.ChardD.JayanthiM.BakerS. K.MorphyP.FlojoJ. (2009). Mathematics instruction for students with learning disabilities: A meta-analysis of instructional components. Review of Educational Research, 79, 1202–1242.
17.
GitomerD.BellC.QiY.McCaffreyD.HamreB. K.PiantaR. C. (2014). The instructional challenge in improving teaching quality: Lessons from a classroom observation protocol. Teachers College Record, 116(6), 1–32.
18.
GrossmanP.ComptonC.IgraD.RonfeldtM.WilliamsonP. (2009). Teaching practice: A cross-professional perspective. Teachers College Record, 111, 2055–2100.
19.
HerlihyC.KargerE.PollardC.HillH. C.KraftM. A.WilliamsM.HowardS. (2014). State and local efforts to investigate the validity and reliability of scores from teacher evaluation systems. Teachers College Record, 116(1), 1–28.
20.
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41, 56–64. doi:10.3102/0013189X12437203
HoldheideL. (2013). Inclusive design: Building educator evaluation systems that support students with disabilities. Available from www.gtlcenter.org
23.
HudsonP. (2015). Feedback consistencies and inconsistencies: Eight mentors’ observations on one preservice teacher’s lesson. European Journal of Teacher Education, 37, 63–73.
24.
HughesC. A.MorrisJ. R.TherrienW. J.BensonS. K. (2017). Explicit instruction: Historical contemporary contexts. Learning Disabilities Research & Practice, 32, 140–148.
25.
JohnsonE. S.CrawfordA. R.MoylanL. A.ZhengY. (2018). Using evidence-centered design to create a special educator observation system. Educational Measurement: Issues and Practice, 37(2), 35–44. doi:10.1111/emip.12182
26.
JohnsonE. S.SemmelrothC. L. (2014). Special education teacher evaluation: Why it matters, what makes it challenging. Assessment for Effective Intervention, 39, 71–82.
27.
JohnsonE. S.SemmelrothC. L. (2015). Validating an observation protocol to measure special education teacher effectiveness. Journal of the American Academy of Special Education Professionals.
28.
JonesN. D.BrownellM. T. (2014). Examining the use of classroom observations in the evaluation of special education teachers. Assessment for Effective Intervention, 39, 112–124. doi:10.1177/1534508413514103
29.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains (Research paper. MET project). Seattle, WA: Bill & Melinda Gates Foundation.
30.
LeiX.LiH.LerouxA. J. (2018). Does a teacher’s classroom observation rating vary across multiple classrooms?Educational Assessment, Evaluation and Accountability, 30, 27–46.
31.
LinacreJ. M. (2014). A user guide to facets. Chicago, IL: Winsteps.
32.
LockwoodJ. R.SavitskyT. D.McCaffreyD. F. (2015). Inferring constructs of effective teaching from classroom observations: An application of Bayesian exploratory factor analysis without restrictions. The Annals of Applied Statistics, 9, 1484–1509.
33.
McClellanC.DonoghueJ.ParkY. S. (2013). Commonality and uniqueness in teaching practice observation. Available from http://www.clowderconsulting.com
34.
McKennaJ. W.ShinM.CiulloS. (2015). Evaluating reading and mathematics instruction for students with learning disabilities: A synthesis of observation research. Learning Disability Quarterly, 38, 195–207.
35.
McLeskeyJ.BarringerM. D.BillingsleyB.BrownellM.JacksonD.KennedyM.ZieglerD. (2017). High-leverage practices in special education. Arlington, VA: Council for Exceptional Children & CEEDAR Center.
36.
MessickS. (1994). The interplay of evidence and consequences in the validation of performance assessments. Educational Researcher, 23, 13–23.
37.
MislevyR.J.AlmondR.LukasJ.F. (2003). A Brief Introduction to Evidence-Centered Design. ETS Research Report Series, 2003, i-29. doi: 10.1002/j.2333-8504.2003.tb01908.x
38.
MyfordC. M.WolfeE. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4, 386–422.
39.
SmithE. V.Jr.KulikowichJ. M. (2004). An application of generalizability theory and many-facet Rasch measurement using a complex problem-solving skills assessment. Educa-tional and Psychological Measurement, 64, 617–639.
40.
SmolkowskiK.GunnB. (2012). Reliability and validity of the Classroom Observations of Student-Teacher Interactions (COSTI) for kindergarten reading instruction. Early Childhood Research Quarterly, 27, 316–328.
41.
StockardJ.WoodT. W.CoughlinC.Rasplica KhouryC. (2018). The effectiveness of direct instruction curricula: A meta-analysis of a half century of research. Review of Educational Research, 88, 479–507.
42.
SwansonE. A. (2008). Observing reading instruction for students with LD: A synthesis. Learning Disability Quarterly, 31, 115–133. doi:10.1177/0022219411402691
43.
TaylorE. S.TylerJ. H. (2012). Can teacher evaluation improve teaching? Evidence of systematic growth in the effectiveness of teachers. Education Next, 12(4), 78–84.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
