This study describes the development and initial psychometric evaluation of the Recognizing Effective Special Education Teachers (RESET) observation instrument. The study uses generalizability theory to compare two versions of a rubric, one with general descriptors of performance levels and one with item-specific descriptors of performance levels, for evaluating special education teacher implementation of explicit instruction. Eight raters (four for each version of the rubric) viewed and scored videos of explicit instruction in intervention settings. The data from each rubric were analyzed with a four facet, crossed, mixed-model design to estimate the variance components and reliability indices. Results show lower unwanted sources of variance and higher reliability indices with the rubric with item-specific descriptors of performance levels. Contributions to the fields of intervention and teacher evaluation are discussed.
AldersonJ. C. (1991). Bands and scores. In AldersonJ. C.NorthB. (Eds.), Language testing in the 1990’s: The communicative legacy (pp. 71–86). Hemel Hempstead, UK: Modern English Publications.
2.
ArcherA. L.HughesC. A. (2011). Explicit instruction: Effective and efficient teaching. New York, NY: Guilford Press.
BakerS. K.GerstenR.HaagerD.DingleM. (2006). Teaching practice and the reading growth of first-grade English learners: Validation of an observation instrument. Elementary School Journal, 107, 199–220. doi:10.1086/510655
5.
BerkeleyS.ScruggsT. E.MastropieriM. A. (2010). Reading comprehension instruction for students with learning disabilities, 1995–2006: A meta-analysis. Remedial and Special Education, 31, 423–436. doi:10.1177/0741932509355988
6.
BrennanR. L. (2001). Generalizability theory. New York, NY: Springer.
7.
CardinetJ.JohnsonS.PiniG. (2010). Applying generalizability theory using EduG. New York, NY: Routledge.
8.
CashA. H.HamreB. K.PiantaR. C.MyersS. S. (2012). Rater calibration when observational assessment occurs at large scale: Degree of calibration and characteristics of raters associated with calibration. Early Childhood Research Quarterly, 27, 529–542. doi:10.1016/j.ecresq.2011.12.006
9.
CohenJ.GoldhaberD. (2016). Building a more complete understanding of teacher evaluation using classroom observations. Educational Researcher, 45, 378–387. doi:10.3102/0013189X16659442
10.
CronbachL. J.GleserG. C.NandaH.RajaratnamN. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: John Wiley.
11.
DanielsonC. (2007). Enhancing professional practice: A framework for teaching. Alexandria, VA: Association for Supervision and Curriculum Development.
12.
DennisM. S.SharpE.ChovanesJ.ThomasA.BurnsR. M.CusterB.ParkJ. (2016). A meta-analysis of empirical research on teaching students with mathematics learning difficulties. Learning Disabilities Research & Practice, 31, 156–168. doi:10.1111/ldrp.12107
13.
DexterD. D.ParkY. J.HughesC. A. (2011). A meta-review of graphic organizers and science instruction for adolescents with learning disabilities: Implications for the intermediate and secondary science classroom. Learning Disabilities Research & Practice, 26, 204–213. doi:10.1111/j.1540-5826.2011.00341.x
14.
EckesT. (2011). Introduction to many-facet rasch measurement. Frankfurt, Germany: Peter Lang.
15.
ErlichO.ShavelsonR. (1976). Generalizability of measures: A computer program for two- and three-facet designs. Behavior Research Methods and Instrumentation, 8, 407–408.
16.
ErlichO.ShavelsonR. (1978). The search for correlations between measures of teacher behavior and student achievement: Measurement problem, conceptualization problem, or both?Journal of Educational Measurement, 15, 77–89.
17.
FulcherG.DavidsonF.KempJ. (2011). Effective rating scale development for speaking tests: Performance decision trees. Language Testing, 28, 5–29. doi:10.1177/0265532209359514
18.
GerstenR.BeckmannS.ClarkeB.FoegenA.MarshL.StarJ. R.WitzelB. (2009). Assisting students struggling with mathematics: Response to Intervention (RtI) for elementary and middle schools (NCEE 2009-4060). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/rti_math_pg_042109.pdf
19.
GerstenR.ChardD. J.JayanthiM.BakerS. K.MorphyP.FlojoJ. (2009). Mathematics instruction for students with learning disabilities: A meta-analysis of instructional components. Review of Educational Research, 79, 1202–1242. doi:10.3102/0034654309334431
20.
GerstenR.ComptonD.ConnorC. M.DiminoJ.SantoroL.Linan-ThompsonS.TillyW. D. (2008). Assisting students struggling with reading: Response to Intervention and multi-tier intervention for reading in the primary grades: A practice guide (NCEE 2009-4045). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, US Department of Education. Retrieved from https://ies.ed.gov/ncee/wwc/Docs/PracticeGuide/rti_reading_pg_021809.pdf
21.
GillespieA.GrahamS. (2014). A meta-analysis of writing interventions for students with learning disabilities. Exceptional Children, 80, 454–473. doi:10.1177/0014402914527238
HillH. C.CharalambousC. Y.KraftM. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41, 56–64. doi:10.3102/0013189X12437203
25.
HillH. C.GrossmanP. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83, 371–384. doi:10.17763/haer.83.2.d11511403715u376
JitendraA. K.LeinA. E.ImS. H.AlghamdiA. A.HefteS. B.MouanoutouaJ. (2018). Mathematical interventions for secondary students with learning disabilities and mathematics difficulties: A meta-analysis. Exceptional Children, 84, 177–196. doi:10.1177/0014402917737467
28.
JohnsonE. S.CrawfordA. R.MoylanL. A.ZhengY. (2018) Using Evidence-Centered Design to Create a Special Educator Observation System. Educational Measurement: Issues and Practice. https://doi.org/10.1111/emip.12182
29.
JohnsonE. S.SemmelrothC. L. (2014). Special education teacher evaluation: Why it matters, what makes it challenging, and how to address these challenges. Assessment for Effective Intervention, 39, 71–82. doi:10.1177/1534508413513315
30.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Available from http://www.metproject.org
31.
KlingnerJ. K.UrbachJ.GolosD.BrownellM.MenonS. (2010). Teaching reading in the 21st century: A glimpse at how special education teachers promote reading comprehension. Learning Disability Quarterly, 33, 59–74. doi:10.1177/073194871003300201
32.
KnochU. (2009). Diagnostic assessment of writing: A comparison of two rating scales. Language Testing, 26, 275–304. doi:10.1177/0265532208101008
33.
KraftM. A.GilmourA. F. (2017). Revisiting the widget effect: Teacher evaluation reforms and the distribution of teacher effectiveness. Educational Researcher, 46, 234–249. doi:10.3102/0013189X17718797
34.
MashburnA.DownerJ.RiversS.BrackettM.MartinezA. (2014). Improving the power of an efficacy study of a social and emotional learning program: Application of generalizability theory to the measurement of classroom-level outcomes. Prevention Science, 15, 146–155.
35.
McLeskeyJ.BarringerM.-D.BillingsleyB.BrownellM.JacksonD.KennedyM.. . . ZieglerD. (2017). High-leverage practices in special education. Arlington, VA: Council for Exceptional Children & CEEDAR Center.
36.
MislevyR. J.SteinbergL. S.AlmondR. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1, 3–67. doi:10.1207/S15366359MEA0101_02
37.
Nelson-WalkerN. J.FienH.KostyD. B.SmolkowskiK.SmithJ. L. M.BakerS. K. (2013). Evaluating the effects of a systemic intervention on first-grade teachers’ explicit reading instruction. Learning Disability Quarterly, 36, 215–230. doi:10.1177/0731948712472186
38.
NorrisC. E.BorstJ. D. (2007). An examination of the reliabilities of two choral festival adjudication forms. Journal of Research in Music Education, 55, 237–251. doi:10.1177/002242940705500305
39.
NorthB.SchneiderG. (1998). Scaling descriptors for language proficiency scales. Language Testing, 15, 217–262. doi:10.1177/026553229801500204
40.
NunnallyJ. C.BernsteinI. H. (1994). Psychometric theory. New York, NY: McGraw-Hill.
41.
PapageorgiouS.XiX.MorganR.SoY. (2015). Developing and validating band levels and descriptors for reporting overall examinee performance. Language Assessment Quarterly, 12, 153–177. doi:10.1080/15434303.2015.1008480
42.
PiantaR. C.HamreB. K.HaynesN. J.MintzS.La ParoK. M. (2006). CLASS Classroom Assessment Scoring System: Manual middle secondary version pilot. Charlottesville, VA: Teachstone.
43.
PolikoffM. S. (2015). The stability of observational and student survey measures of teaching effectiveness. American Journal of Education, 12, 183–212. doi:10.1086/679390
44.
PollittA.MurrayN. L. (1996). What raters really pay attention to. In MilanovicM.SavilleN. (Eds.), Performance testing, cognition and assessment (pp. 74–91). Cambridge, UK: Cambridge University Press.
45.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Newbury Park, CA: SAGE.
46.
SmolkowskiK.GunnB. (2012). Reliability and validity of the Classroom Observations of Student–Teacher Interactions (COSTI) for kindergarten reading instruction. Early Childhood Research Quarterly, 27, 316–328. doi:10.1016/j.ecresq.2011.09.004
47.
StockardJ.WoodT. W.CoughlinC.Rasplica KhouryC. (2018). The effectiveness of direct instruction curricula: A meta-analysis of a half century of research. Review of Educational Research. Advance online publication. doi:10.3102/0034654317751919
48.
SwansonH. L. (1999). Instructional components that predict treatment outcomes for students with learning disabilities: Support for a combined strategy and direct instruction model. Learning Disabilities Research & Practice, 14, 129–140. doi:10.1207/sldrp1403_1
49.
WongC.OdomS. L.HumeK. A.CoxC. W.FettigA.KurcharczykS.. . . SchultzT. R. (2015). Evidence-based practices for children, youth, and young adults with autism spectrum disorder: A comprehensive review. Journal of Autism and Developmental Disorders, 45, 1951–1966.
