This study investigated the use of brief observations to measure implementation of small group interventions using the Quality of Intervention Delivery and Receipt (QIDR) tool. Videos of 10-min segments representing the beginning, middle, and end of each 30-min intervention lesson were coded for implementation. Results indicated that (a) reliability can be achieved when observing a portion of a lesson; (b) QIDR scores obtained from 10-min segments are significantly correlated with the 30-min observations; and (c) post hoc analysis indicate the tool may be sensitive to changes in implementation quality at the intervention group level across time. Implications for research and practice are discussed, including the utility of shorter observations to increase the monitoring of intervention implementation to potentially improve outcomes.
Al-OtaibaS.WagnerR. K.MillerB. (2014). “Waiting to fail”: Redux understanding inadequate response to intervention. Learning Disability Quarterly, 37, 129–133.
2.
ArcherA. L.HughesC. A. (2011). Explicit instruction: Effective and efficient teaching. New York, NY: The Guilford Press.
3.
BellC. A.QiY.CroftA. J.LeusnerD.McCaffreyD. F.GitomerD. H.PiantaR. C. (2016). Improving observational score quality: Challenges in observer thinking. In KaneT.KerrK.PiantaR. (Eds.), Designing teacher evaluation systems: New guidance from the measures of effective teaching project (pp. 415–443). Hoboken, NJ: John Wiley.
4.
BoardmanA. G.BuckleyP.VaughnS.RobertsG.ScornavaccoK.KlingnerJ. K. (2016). Relationship between implementation of Collaborative Strategic Reading and student outcomes for adolescents with disabilities. Journal of Learning Disabilities, 49, 644–657.
5.
BrophyJ. (1986). Teacher influences on student achievement. American Psychologist, 41, 1069–1077. doi:10.1037//0003-066X.41.10.1069
6.
BrophyJ.GoodT. L. (1986). Teacher behavior and student achievement. In WittrockM. C. (Ed.), Handbook of research on teaching (3rd ed., pp. 328–375). New York, NY: Macmillan Publishing.
7.
CameronC. E.ConnorC. M.MorrisonF. J. (2005). Effects of variation in teacher organization on classroom functioning. Journal of School Psychology, 43, 61–85. doi:10.1016/j.jsp.2004.12.002
8.
Chomat-MooneyL. I.PiantaR. C.HamreB. K.MashburnA. J.LucknerA. E.GrimmK. J.. . .DownerJ. T. (2008). A practical guide for conducting classroom observations: A summary of issues and evidence for researchers. Charlottesville, NC: University of Virginia.
9.
CichettiD. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290.
10.
CohenJ. (1988). Statistical power analyses for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
11.
CookB. G.OdomS. L. (2013). Evidence-based practices and implementation science in special education. Exceptional Children, 79, 135–144.
12.
DanielsonC. (2013). The framework for teaching evaluation instrument, 2013 edition: The newest rubric enhancing the links to the Common Core State Standards, with clarity of language for ease of use and scoring. Alexandria, VA: Association for Supervision and Curriculum Development.
13.
DenoS. L. (2003). Curriculum-based measures: Development and perspectives. Assessment for Effective Intervention, 28(3–4), 3–12.
14.
DurlakJ. A.DuPreE. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0
FoormanB. R.TorgesenJ. (2001). Critical elements of classroom and small-group instruction promote reading success in all children. Learning Disabilities Research & Practice, 16, 203–212. doi:10.1111/0938-8982.00020
20.
Forbes-SpearC. (2014). Examining the relationship between implementation and student outcomes: The application of an implementation measurement framework (Doctoral dissertation). Scholars Bank, University of Oregon, Eugene.
21.
GarganiJ.StrongM. (2014). Can we identify a successful teacher better, faster, and cheaper? Evidence for innovating teacher observation systems. Journal of Teacher Education, 65, 389–401. doi:10.1177/0022487114542519
22.
GerstenR. (2016). Commentary: The tyranny of time and the reality principle. In FoormanB. (Ed.), Challenges to implementing effective reading intervention in schools: New Directions for Child and Adolescent Development, No. 154 (pp. 113–116). Hoboken, NJ: John Wiley.
23.
GerstenR.BakerS. K.HaagerD.GravesA. (2005). Exploring the role of teacher quality in predicting reading outcomes for first-grade English learners. Remedial and Special Education, 26, 197–206.
24.
GerstenR.FuchsL.ComptonD.CoyneM.GreenwoodC.InnocentiM. (2005). Quality indicators for group experimental and quasi-experimental research in special education. Exceptional Children, 71, 149–164.
25.
GoodR. H.KaminskiR. A. (2002). Dynamic Indicators of Basic Early Literacy Skills (6th ed.). Eugene, OR: Institute for the Development of Educational Achievement. Available from http://dibels.uoregon.edu
26.
GoodR. H.KaminskiR. A.ShinnM.BrattenJ.LaimonL.SmithS.FlindtN. (2004). Technical adequacy and decision making utility of DIBELS (Technical Report No. 7). Eugene: University of Oregon.
27.
Hagan-BurkeS.CoyneM. D.KwokO.-M.SimmonsD. C.KimM.SimmonsL. E.. . . McSparran RubyM. (2013). The effects and interactions of student, teacher, and setting variables on reading outcomes for kindergarteners receiving supplemental reading intervention. Journal of Learning Disabilities, 46, 260–277. doi:10.1177/0022219411420571
28.
HallgrenK. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psycholology, 8, 23–34.
29.
HamreB. K.PiantaR. C. (2005). Can instructional and emotional support in the first-grade classroom make a difference for children at risk of school failure?Child Development, 76, 949–967.
30.
HarnB. A.Forbes-SpearC.FritzR.BergT. (2012). Quality of Intervention Delivery and Receipt (QIDR) observation tool. Eugene: University of Oregon.
31.
HarnB. A.Parisi DamicoD.StoolmillerM. (2013). Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools?Exceptional Children, 79, 181–193.
32.
HarnB. A. Parisi DamicoD.StoolmillerM. (2017). Examining the variation of fidelity across an intervention: Implications for measuring and evaluating student learning. Preventing School Failure: Alternative Education for Children and Youth, 61, 289–302.
33.
HarnB. A.SpearC. F.FritzR.BergT. R.BasarabaD. (2014, February). Examining the relation of features of implementation to student outcomes. Conference presentation at the Pacific Coast Research Conference, San Diego, CA.
34.
HillH.GrossmanP. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83, 371–384.
35.
HoA. D.KaneT. J. (2013). The reliability of classroom observations by school personnel (Research paper, MET Project). Seattle, WA: Bill & Melinda Gates Foundation.
36.
HoldheideL.BrowderD.WarrenS.BuzickH.JonesN. (2012). Using student growth to evaluate educators of students with disabilities: Issues, challenges, and next steps. Washington, DC: National Comprehensive Center for Teacher Quality.
37.
HoldheideL. R.GoeL.CroftA.ReschlyD. J. (2010). Challenges in evaluating special education teachers and English language learner specialists (Research & policy brief). Washington, DC: National Comprehensive Center for Teacher Quality.
38.
JohnsonE. S.SemmelrothC. L. (2012). Examining interrater agreement analyses of a pilot special education observation tool. Journal of Special Education Apprenticeship, 1(2), 1–19.
39.
JonesN. D.BuzickH. M.TurkanS. (2013). Including students with disabilities and English learners in measures of educator effectiveness. Educational Researcher, 42, 234–241.
40.
JonesR. R.ReidJ. B.PattersonG. R. (1975). Naturalistic observation in clinical assessment. Advances in Psychological Assessment, 3, 42–95.
41.
JusticeL. M. (2006). Evidence-based practice, response to intervention, and the prevention of reading difficulties. Language, Speech, and Hearing Services in Schools, 37, 284–298.
42.
JusticeL. M.MashburnA. J.HamreB. K.PiantaR. C. (2008). Quality of language and literacy instruction in preschool classrooms serving at-risk pupils. Early Childhood Research Quarterly, 23, 51–68.
43.
KaneT. J.StaigerD. O. (2012). Gathering feedback for teaching: Measures of effective teaching project. Seattle, WA: Bill & Melinda Gates Foundation.
KretlowA. G.BartholomewC. C. (2010). Using coaching to improve the fidelity of evidence-based practices: A review of studies. Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, 33, 279–299. doi:10.1177/0888406410371643
46.
La ParoK.PiantaR. C.StuhlmanM. (2004). Classroom Assessment Scoring System (CLASS): Findings from the pre-k year. The Elementary School Journal, 104, 409–426.
47.
MilesJ.BanyardP. (2007). Understanding and using statistics in psychology. New York, NY: SAGE.
48.
MowbrayC.HolterM.TeagueG.BybeeD. (2003). Fidelity criteria: Development, measurement, and validation. The American Journal of Evaluation, 24, 315–340.
49.
MyersD. M.SimonsenB.SugaiG. (2011). Increasing teachers’ use of praise with a response-to-intervention approach. Education and Treatment of Children, 34, 35–59.
50.
OdomS. L. (2009). The tie that binds: Evidence-based practice, implementation science, and outcomes for children. Topics in Early Childhood Special Education, 29, 53–61. doi:10.1177/0271121408329171
51.
PiantaR. C.CoxM. J.TaylorL.EarlyD. (2013). Kindergarten teachers’ practices related to the transition to school: Results of a national survey. The Elementary School Journal, 100, 71–86.
52.
PiantaR. C.La ParoK.HamreB. (2008). Classroom Assessment Scoring System. Baltimore, MD: Paul H. Brooks.
53.
PiantaR. C.MashburnA. J.DownerJ. T.HamreB. K.JusticeL. (2008). Effects of web-mediated professional development resources on teacher-child interactions in pre-kindergarten classrooms. Early Childhood Research Quarterly, 23, 431–451.
54.
PrattA.LoganJ. (2014). Improving language-focused comprehension instruction in primary-grade classrooms: Impacts of the Let’s Know! Experimental curriculum. Educational Psychology Review, 26, 357–377.
55.
ReppA. C.NieminenG. S.OlingerE.BruscaR. (1988). Direct observation: Factors affecting the accuracy of observers. Exceptional Children, 55, 29–36.
56.
RosenshineB.StevensR. (1986). Teaching functions. In WitrockM. C. (Ed.), Handbook on research and teaching (3rd ed., pp. 376–390). New York, NY: Macmillan Publishing.
57.
SemmelrothC. L.JohnsonE. S. (2014). Measuring rater reliability on a special education observation tool. Assessment for Effective Intervention, 39, 131–145.
58.
SimmonsD. C.Kame’enuiE. J. (2003). Scott Foresman: Early Reading Intervention. Glenview, IL: Scott Foresman.
59.
SteckerP. M.LembkeE. S.FoegenA. (2008). Using progress-monitoring data to improve instructional decision making. Preventing School Failure: Alternative Education for Children and Youth, 52(2), 48–58. doi:10.3200/PSFL.52.2.48-58
60.
TaplinP. S.ReidJ. B. (1973). Effects of instructional set and experimenter influence on observer reliability. Child Development, 44, 547–554.
61.
TorgesenJ. K.WagnerR. K.RashotteC. A.RoseE.LindamoodP.ConwayT.GarvanC. (1999). Preventing reading failure in young children with phonological processing disabilities: Group and individual responses to instruction. Journal of Educational Psychology, 91, 579–593. doi:10.1037//0022-0663.91.4.579
YeatonW. H.SechrestL. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156–167. doi:10.1037//0022-006X.49.2.156
64.
ZigmondN.KlooA. (2011). General and special education are (and should be) different. In KauffmanJ. M. (Ed.), Handbook of special education (pp. 160–172). New York, NY: Routledge.
