Classroom engagement is important to young children’s academic and social development. Accurate methods of capturing this behavior are needed to inform and evaluate intervention efforts. This study compared the accuracy of interval durations (i.e., 5 s, 10 s, 15 s, 20 s, 30 s, and 60 s) of momentary time sampling (MTS) in approximating the duration of classroom engagement as measured by continuous duration recording (CDR). Twenty-four sessions of children in large-group instruction were observed using the Behavioral Observation of Students in School for Early Education (BOSS-EE). In general, shorter intervals produced engagement estimates that more highly correlated with CDR data and had less measurement error relative to estimates from longer intervals. Findings suggest that selection of MTS interval duration may affect the accuracy of estimates for children with low levels of engagement to a greater extent relative to children with high levels of engagement. Implications and future directions are discussed.
AdamsK. A.LawrenceE. M. K. (2014). Research methods, statistics, and applications. New York, NY: SAGE.
2.
AguiarC.McWilliamR. A. (2013). Consistency of toddler engagement across two settings. Early Childhood Research Quarterly, 28, 102–110. doi:10.1016/j.ecresq.2012.04.003
3.
AltmanD. G. (1991). Practical statistics for medical research. London, England: Chapman & Hall.
4.
AlveroA. M.StrussK.RappaportE. (2007). Measuring safety performance: A comparison of whole, partial, and momentary time-sampling recording methods. Journal of Organizational Behavior Management, 27, 1–28. doi:10.1300/J075v27n04_01
5.
BelloneK. M.DufreneB. A.TingstromD. H.OlmiD. J.BarryC. (2014). Relative efficacy of behavioral interventions in preschool children attending Head Start. Journal of Behavioral Education, 23, 378–400. doi:10.1007/s10864-014-9196-6
6.
BrieschA. M.ChafouleasS. M.Riley-TillmanT. C. (2010). Generalizability and dependability of behavior assessment methods to estimate academic engagement: A comparison of systematic direct observation and direct behavior rating. School Psychology Review, 39, 408–421.
7.
BrulleA. R.ReppA. C. (1984). An investigation of the accuracy of momentary time sampling procedures with time series data. British Journal of Psychology, 75, 481–485.
8.
Bulotsky-ShearerR. J.FantuzzoJ. W.McDermottP. A. (2008). An investigation of classroom situational dimensions of emotional and behavioral adjustment and cognitive and social outcomes for Head Start children. Developmental Psychology, 44, 139–154. doi:10.1037/0012-1649.44.1.139
9.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
10.
CooperJ. O.HeronT. E.HewardW. L. (2007). Applied behavior analysis (2nd ed.). Upper Saddle River, NJ: Pearson.
11.
DiCarloC. F.BaumgartnerJ.StephensA.PierceS. H. (2013). Using structured choice to increase child engagement in low-preference centres. Early Child Development and Care, 183, 109–124. doi:10.1080/03004430.2012.657632
12.
DiedenhofenB.MuschJ. (2015). cocor: A comprehensive solution for the statistical comparison of correlations. PLoS ONE, 10, e0121945. doi:10.1371/journal.pone.0121945
13.
Eggum-WilkensN. D.FabesR. A.CastleS.ZhangL.HanishL. D.MartinC. L. (2014). Playing with others: Head Start children’s peer play and relations with kindergarten school competence. Early Childhood Research Quarterly, 29, 345–356. doi:10.1016/j.ecresq.2014.04.008
14.
FitzpatrickC. (2013). Kindergarten classroom engagement skills: The road to academic success in elementary school. Dissertation Abstracts International, 74, NR89016.
15.
FuhsM. W.FarranD. C.NesbittK. T. (2013). Preschool classroom processes as predictors of children’s cognitive self-regulation skills development. School Psychology Quarterly, 28, 347–359. doi:10.1037/spq0000031
16.
GardenierN. C.MacDonaldR.GreenG. (2004). Comparison of direct observational methods for measuring stereotypic behavior in children with autism spectrum disorders. Research in Developmental Disabilities, 25, 99–118. doi:10.1016/j.ridd.2003.05.004
17.
HarropA.DanielsM. (1986). Methods of time sampling: A reappraisal of momentary time sampling and partial interval recording. Journal of Applied Behavior Analysis, 19, 73–77. doi:10.1901/jaba.1986.19-73
18.
HarropA.DanielsM.FoulkesC. (1990). The use of momentary time sampling and partial interval recording in behavioral research. Behavioural Psychotherapy, 18, 121–127. doi:10.1017/S0141347300018231
19.
HintzeJ. M.MatthewsW. J. (2004). The generalizability of system direct observation across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review, 33, 258–270.
20.
HintzeJ. M.VolpeR. J.ShapiroE. S. (2008). Best practices in the systematic direct observation of student behavior. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology V (Vol. 2, pp. 141–158). Washington, DC: National Association of School Psychologists.
21.
HittnerJ. B.MayK.SilverN. C. (2003). A Monte Carlo evaluation of tests for comparing dependent correlations. The Journal of General Psychology, 130, 149–168. doi:10.1080/00221300309601282
22.
HojnoskiR. L.MissallK. N. (2010). Social development in preschool classrooms: Promoting engagement, competence, and school readiness. In ShinnM. R.WalkerH. M. (Eds.), Interventions for achievement and behavior problems in a three-tier model including RTI (pp. 703–728). Bethesda, MD: National Association of School Psychologists.
23.
HojnoskiR. L.WoodB.ShapiroE. S. (2014). The behavioral observation of students in school for early education. Unpublished protocol, Department of Education and Human Services, Lehigh University, Bethlehem, PA.
24.
HuntP.SotoG.MaierJ.LiboironN.BaeS. (2004). Collaborative teaming to support preschoolers with severe disabilities who are placed in general education early childhood programs. Topics in Early Childhood Special Education, 24, 123–142. doi:10.1177/02711214040240030101
25.
KarweitN.SlavinR. E. (1982). Time-on-task: Issues of timing, sampling, and definition. Journal of Educational Psychology, 74, 844–851. doi:10.1037/0022-0663.74.6.844
26.
KazdinA. E. (1982). Single-case research designs: Methods for clinical and applied settings. New York, NY: Oxford University Press.
27.
KearnsK.EdwardsR.TingstromD. H. (1990). Accuracy of long momentary time-sampling intervals: Implications for classroom data collection. Journal of Psychoeducational Assessment, 8, 74–85. doi:10.1177/073428299000800109
28.
KempC.KishidaY.CarterM.SwellerN. (2013). The effect of activity type on the engagement and interaction of young children with disabilities in inclusive childcare settings. Early Childhood Research Quarterly, 28, 134–143. doi:10.1016/jecresq.2012.03.003
29.
KishidaY.KempC. (2006). A measure of engagement for children with intellectual disabilities in early childhood settings: A preliminary study. Journal of Intellectual & Developmental Disability, 31, 101–114. doi:10.1080/13668250600710823
30.
KoltL. D.RappJ. T. (2014). Assessment of therapists’ preferences for discontinuous measurement systems. Behavioral Interventions, 29, 304–314. doi:10.1002/bin.1392
31.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159–174. doi:10.2307/2529310
32.
LaneJ. D.LedfordJ. R. (2014). Using interval-based systems to measure behavior in early childhood special education and early intervention. Topics in Early Childhood Special Education, 34, 1–11. doi:10.1177/0271121414524063
33.
LedfordJ. R.AyresK. M.LaneJ. D.LamM. F. (2015). Identifying issues and concerns with the use of interval-based systems in single case research using a pilot simulation study. The Journal of Special Education, 49, 104–117. doi:10.1177/0022466915568975
34.
LingS. M.BarnettD. W. (2013). Increasing preschool student engagement during group learning activities using a group contingency. Topics in Early Childhood Special Education, 33, 186–196. doi:10.1177/0271121413484595
35.
MannJ.Ten HaveT.PlunkettJ. W.MeiselsS. J. (1991). Time sampling: A methodological critique. Child Development, 62, 227–241. doi:10.2307/1130999
36.
MansellJ. (1985). Time sampling and measurement error: The effect of interval duration and sampling pattern. Journal of Behavior Therapy and Experimental Psychiatry, 16, 245–251. doi:10.1016/0005-7916(85)90070-9
37.
McWilliamR. A.BaileyD. B. (1992). Promoting engagement and mastery. In BaileyD. B.WoleryM. (Eds.), Teaching infants and preschoolers with disabilities (2nd ed., pp. 230–255). New York, NY: Macmillan.
38.
McWilliamR. A.CaseyA. M. (2008). Engagement of every child in the preschool classroom. Baltimore, MD: Brookes.
39.
McWilliamR. A.TrivetteC. M.DunstC. J. (1985). Behavior engagement as a measure of the efficacy of early intervention. Analysis and Intervention in Developmental Disabilities, 5, 33–45. doi:10.1016/S0270-4684(85)80006-9
40.
Meany-DaboulM. G.RoscoeE. M.BourretJ. C.AhearnW. H. (2007). A comparison of momentary time sampling and partial interval recording for evaluating functional relations. Journal of Applied Behavior Analysis, 40, 501–514. doi:10.1901/jaba.2007.40-501
MontroyJ. J.BowlesR. P.SkibbeL. E.FosterT. D. (2014). Social skills and problem behaviors as mediators of the relationship between behavioral self-regulation and academic achievement. Early Childhood Research Quarterly, 29, 298–309. doi:10.1016/j.ecresq.2014.03.002
43.
MudfordO. C.TaylorS. A.MartinN. T. (2009). Continuous recording and interobserver agreement algorithms reported in theJournal of Applied Behavior Analysis (1995–2005). Journal of Applied Behavior Analysis, 40, 501–514. doi:10.1901/jaba.2009.42-165
44.
OdomS. L.BrownW. H.SchwartzI. S.ZercherC.SandallS. R. (2002). Classroom ecology and child participation. In OdomS. L. (Ed.), Widening the circle (pp. 25–45). New York, NY: Teachers College Press.
45.
PowellJ.MartindaleA.KulpS. (1975). An evaluation of time-sample measures of behavior. Journal of Applied Behavior Analysis, 8, 463–469. doi:10.1901/jaba.1975.8-463
46.
PowellJ.MartindaleA.KulpS.MartindaleA.BaumanR. (1977). Taking a closer look: Time sampling and measurement error. Journal of Applied Behavior Analysis, 10, 325–332. doi:10.1901/jaba.1977.10-325
47.
PowellJ.RockinsonR. (1978). On the inability of interval time sampling to reflect frequency of occurrence data. Journal of Applied Behavior Analysis, 11, 531–532. doi:10.1901/jaba.1978.11-531
48.
RappJ. T.ColbyA. M.VollmerT. R.RoaneH. S.LomasJ.BrittonL. N. (2007). Interval recording for duration events: A re-evaluation. Behavioral Interventions, 22, 319–345. doi:10.1002/bin.239
49.
RappJ. T.Colby-DirksenA. M.MichalskiD. N.CarrollR. A.LindenburgA. M. (2008). Detecting changes in simulated events using partial-interval recording and momentary time sampling. Behavioral Interventions, 23, 237–269. doi:10.1002/bin.269
50.
RubleL.McGrewJ. H. (2013). Teacher and child predictors of achieving IEP goals of children with autism. Journal of Autism and Developmental Disorders, 43, 2748–2763. doi:10.1007/s10803-013-1884-x
51.
SalviaJ.YsseldykeJ. E. (2001). Assessment (8th ed.). Princeton, NJ: Houghton Mifflin.
52.
Sanson-FisherR. W.PooleA. D.DunnJ. (1980). An empirical method for determining an appropriate interval duration for recording behavior. Journal of Applied Behavior Analysis, 13, 493–500. doi:10.1901/jaba.1980.13-493
53.
SaudargasR. A.ZanolliK. (1990). Momentary time sampling as an estimate of percentage time: A field validation. Journal of Applied Behavior Analysis, 23, 533–537. doi:10.1901/jaba.1990.23-533
54.
ShapiroE. S. (2004). Academic skills problems workbook (2nd ed.). New York, NY: Guilford Press.
55.
ShapiroE. S. (2014). Behavioral assessment in school psychology. New York, NY: Routledge.
56.
SkinnerC. H.FreelandJ. T.ShapiroE. S. (2003). Procedural issues associated with the behavioral assessment of children. In ReynoldsC. R.KamphausR. W. (Eds.), Handbook of psychological and educational assessment of children: Personality, behavior, and context (2nd., pp. 130–147). New York, NY: Guilford Press.
WilliamsE. J. (1959). The comparison of regression variables. Journal of the Royal Statistical Society: Series B, 21, 396–399. Retrieved from http://www.jstor.org/stable/2983809
59.
WillifordA. P.Vick WhittakerJ. E.VitielloV. E.DownerJ. T. (2013). Children’s engagement within the preschool classroom and their development of self-regulation. Early Education and Development, 24, 162–187. doi:10.1080/10409289.2011.628270
60.
WirthO.SlavenJ.TaylorM. A. (2014). Interval sampling methods and measurement error: A computer simulation. Journal of Applied Behavior Analysis, 47, 83–100. doi:10.1002/jaba.93
61.
WoodB. K.HojnoskiR. L.LaracyS. D.OlsonC. L. (2016). Comparison of observational methods and their relation to ratings of engagement in young children. Topics in Early Childhood Special Education, 35, 211–222. doi:10.1177/0271121414565
