Researchers seeking to make valid conclusions about engagement for young children with autism spectrum disorder (ASD) must first determine the reliability of estimates obtained across the conditions sampled. Working from that premise, we conducted a secondary data analysis of shared book readings between caregivers and their children with ASD, examining the contribution of measurement error on estimates of four states of child engagement. Caregivers read two different book types, each three times, with their children. With book type and occasion as measurement facets, we conducted a generalizability study and a series of decision studies. With the interaction of Persons × Book Type × Occasion contributing the most measurement error for four engagement variables, we only found stable estimates for unengaged behaviors. For the active engagement, visual engagement, and disruptive variables, 4, 5, and more than 10 book types, respectively, were necessary to obtain stable estimates across two occasions.
AtilganH. (2013). Sample size estimation of G and Phi coefficients in generalizability theory. Eurasian Journal of Educational Research, 51, 215–227.
5.
AzurM. J.StuartE. A.FrangakisC.LeafP. J. (2011). Multiple imputation by chained equations: What is it and how does it work?International Journal of Methods in Psychiatric Research, 20(1), 40–49. https://doi.org/10.1002/mpr.329
6.
BerkR. A. (1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability. American Journal of Mental Deficiency, 83(5), 460–472.
7.
Bottema-BeutelK.KimS. Y.CrowleyS.AugustineA.Kecili-KaysiliB.FeldmanJ.WoynaroskiT. (2019). The stability of active engagement states in infant siblings of children with and without ASD: Implications for measurement practices. Autism Research, 12(3), 495–504. https://doi.org/10.1002/aur.2068
8.
Bottema-BeutelK.LloydB.CarterE. W.AsmusJ. M. (2014). Generalizability and decision studies to inform observational and experimental research in classroom settings. American Journal on Intellectual and Developmental Disabilities, 119(6), 589–605. https://doi.org/10.1352/1944-7558-119.6.589
9.
Bottema-BeutelK.MalloyC.LloydB. P.LouickR.Joffe-NelsonL.WatsonL. R.YoderP. J. (2017). Sequential associations between caregiver talk and child play in autism spectrum disorder and typical development. Child Development, 89(3), 157–166. https://doi.org/10.1111/cdev.12848
BrennanR. L. (2001). Generalizability theory. Springer.
12.
BrennanR. L. (2003). Coefficients and indices in generalizability theory (Issue 1). Center for Advanced Studies in Measurement and Assessment.
13.
BrieschA. M.SwaminathanH.WelshM.ChafouleasS. M. (2014). Generalizability theory: A practical guide to study design, implementation, and interpretation. Journal of School Psychology, 52(1), 13–35. https://doi.org/10.1016/j.jsp.2013.11.008
14.
BrownH. M.Oram-CardyJ.JohnsonA. (2013). A meta-analysis of the reading comprehension skills of individuals on the autism spectrum. Journal of Autism and Developmental Disorders, 43(4), 932–955. https://doi.org/10.1007/s10803-012-1638-1
15.
BrucknerC. T.YoderP. J.McWilliamR. A. (2006). Generalizability and decision studies: An example using conversational language samples. Journal of Early Intervention, 28(2), 139–153. https://doi.org/10.1177/105381510602800205
16.
BursteinN. D. (1986). The effects of classroom organization on mainstreamed preschool children. Exceptional Children, 52(5), 425–434. https://doi.org/10.1177/001440298605200504
17.
CardinetJ.JohnsonS.PiniG. (2010). Applying generalizability theory using EduG. Routledge.
18.
CarpenterM.TomaselloM. (2000). Joint attention, cultural learning, and language acquisition: Implications for children with autism. In WetherbyA. M.PrizantB. M. (Eds.), Autism spectrum disorders: A transactional developmental perspective (Vol. 9, pp. 31–54). Paul H. Brookes Publishing.
CronbachL. J.RajaratnamN.GleserG. C. (1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16(2), 137–163. https://doi.org/10.1111/j.2044-8317.1963.tb00206.x
21.
DickinsonD. K.PorcheM. V. (2011). Relation between language experiences in preschool classrooms and children’s kindergarten and fourth-grade language and reading abilities. Child Development, 82(3), 870–886. https://doi.org/10.1111/j.1467-8624.2011.01576.x
22.
FleuryV. P.FordA. L. (2021). Shared reading extratextual talk with children with autism who have spontaneous speech. The Journal of Special Education, 55(1), 23–33. https://doi.org/10.1177/0022466920926165
23.
FleuryV. P.HughM. L. (2018). Exploring engagement in shared reading activities between children with autism spectrum disorder and their caregivers. Journal of Autism and Developmental Disorders, 48(10), 3596–3607. https://doi.org/10.1007/s10803-018-3632-8
24.
FleuryV. P.LeaseE. M. (2018). Early indication of reading difficulty? A descriptive analysis of emergent literacy skills in children with autism spectrum disorder. Topics in Early Childhood Special Education, 38(2), 82–93. https://doi.org/10.1177/0271121417751626
25.
FleuryV. P.SchwartzI. S. (2017). A modified dialogic reading intervention for preschool children with autism spectrum disorder. Topics in Early Childhood Special Education, 37(1), 16–28. https://doi.org/10.1177/0271121416637597
26.
FredricksJ. A.BlumenfeldP. C.ParisA. H. (2004). School engagement: Potential of the concept, state of the evidence. Review of Educational Research, 74(1), 59–109. https://doi.org/10.3102/00346543074001059
27.
GodinJ.FreemanA.RigbyP. (2019). Interventions to promote the playful engagement in social interaction of preschool-aged children with autism spectrum disorder (ASD): A scoping study. Early Child Development and Care, 189(10), 1666–1681. https://doi.org/10.1080/03004430.2017.1404999
28.
GreenwoodC. R. (1991). Longitudinal analysis of time, engagement, and achievement in at-risk versus non-risk students. Exceptional Children, 57(6), 521–535. https://doi.org/10.1177/001440299105700606
29.
HolloA.StaubitzJ. L.ChowJ. C. (2020). Applying generalizability theory to optimize analysis of spontaneous teacher talk in elementary classrooms. Journal of Speech, Language, and Hearing Research, 63(6), 1947–1957. https://doi.org/10.1044/2020_JSLHR-19-00118
30.
HumeK.SteinbrennerJ. R.OdomS. L.MorinK. L.NowellS. W.TomaszewskiB.SzendreyS.McIntyreN. S.Yücesoy-ÖzkanS.SavageM. N. (2021). Evidence-based practices for children, youth, and young adults with autism: Third generation review. Journal of Autism and Developmental Disorders. https://doi.org/10.1007/s10803-020-04844-2
31.
Institute of Education Sciences. (2009). Dialogic reading (What Works Clearinghouse Intervention Report).
KaneM. (2002). Inferences about variance components and reliability-generalizability coefficients in the absence of random sampling. Journal of Educational Measurement, 39(2), 165–181. https://www.jstor.org/stable/1435254
34.
KempC.KishidaY.CarterM.SwellerN. (2013). The effect of activity type on the engagement and interaction of young children with disabilities in inclusive childcare settings. Early Childhood Research Quarterly, 28(1), 134–143. https://doi.org/10.1016/j.ecresq.2012.03.003
35.
KoegelL. K.KoegelR. L.FreaW.Green-HopkinsI. (2003). Priming as a method of coordinating educational services for students with autism. Language, Speech, and Hearing Services in Schools, 34(3), 228–235. https://doi.org/10.1044/0161-1461(2003/019)
36.
KrippendorffK. (2018). Content analysis: An introduction to its methodology. SAGE.
37.
LandisJ. R.KochG. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://www.jstor.org/stable/2529310
LivingstonC. (2017). Types of children’s literature. In EvansR.JoubertI.MeierC. (Eds.), Introducing children’s literature: A guide to South African classroom (pp. 112–128). Van Schaik.
40.
LoebS.DynarskiS.McFarlandD.MorrisP.ReardonS.ReberS. (2017). Descriptive analysis in education: A guide for researchers. U.S. Department of Education, Institute of Education Sciences & National Center for Education Evaluation and Regional Assistance. https://doi.org/10.1094/PDIS.2003.87.5.550
41.
LoniganC. J.WagnerR. K.TorgesonJ. (2007). Test of preschool early literacy. Super Duper Publications.
42.
LordF.NovickM. R. (1968). Statistical theories of mental test scores. Addison-Wesley.
43.
MantzicopoulosP.FrenchB. F.PatrickH.WatsonJ. S.AhnI. (2018). The stability of kindergarten teachers’ effectiveness: A generalizability study comparing the framework for teaching and the classroom assessment scoring system. Educational Assessment, 23(1), 24–46. https://doi.org/10.1080/10627197.2017.1408407
44.
McIntyreN. S.SolariE. J.GonzalesJ. E.SolomonM.LerroL. E.NovotnyS.OswaldT. M.MundyP. C. (2017). The scope and nature of reading comprehension impairments in school-aged children with higher-functioning autism spectrum disorder. Journal of Autism and Developmental Disorders, 47(9), 2838–2860. https://doi.org/10.1007/s10803-017-3209-y
45.
McWilliamR. A.WareW. B. (1994). The reliability of observations of young children’s engagement: An application of generalizability theory. Journal of Early Intervention, 18(1), 34–47. https://doi.org/10.1177/105381519401800104
46.
NationK.ClarkeP.WrightB.WilliamsC. (2006). Patterns of reading ability in children with autism spectrum disorder. Journal of Autism and Developmental Disorders, 36(7), 911–919. https://doi.org/10.1007/s10803-006-0130-1
47.
National Early Literacy Panel. (2008). Developing early literacy: Report of the national early literacy panel.
PetscherY.CabellS. Q.CattsH. W.ComptonD. L.FoormanB. R.HartS. A.LoniganC. J.PhillipsB. M.SchatschneiderC.SteacyL. M.TerryN. P.WagnerR. K. (2020). How the science of reading informs 21st-Century education. Reading Research Quarterly, 55(Suppl. 1), S267–S282. https://doi.org/10.1002/rrq.352
51.
R Core Team. (2015). RStudio: Integrated development for R. R Studio. http://www.rstudio.com/
52.
ReutzelD. R.FawsonP. (2002). Your classroom library: New ways to give it more teaching power. Scholastic.
53.
RichterA.CourageM. L. (2017). Comparing electronic and paper storybooks for preschoolers: Attention, engagement, and recall. Journal of Applied Developmental Psychology, 48, 92–102. https://doi.org/10.1016/j.appdev.2017.01.002
54.
ShavelsonR. J.WebbN. M. (2006). Generalizability theory. In GreenJ. L.CamiliG.ElmoreP. B. (Eds.), Handbook of complementary methods in education research (3rd ed., pp. 309–322). American Educational Research Association.
55.
SigmanM.DijamcoA.GratierM.RozgaA. (2004). Early detection of core deficits in autism. Mental Retardation and Developmental Disabilities Research Reviews, 10(4), 221–233. https://doi.org/10.1002/mrdd.20046
56.
SuenH. K.AryD. (1989). Reliability: The generalizability approach. In. Analyzing quantitative behavioral observation data (pp. 131–156). Lawrence Erlbaum.
TappJ.WehbyJ.EllisD. (1995). A multiple option observation system for experimental studies: MOOSES. Behavior Research Methods, Instruments, & Computers, 27(1), 25–31. https://doi.org/10.3758/BF03203616
59.
TiptonL. A.BlacherJ. B.EisenhowerA. S. (2017). Young children with ASD: Parent strategies for interaction during adapted book reading activity. Remedial and Special Education, 38(3), 171–180. https://doi.org/10.1177/0741932516677831
60.
WasikB. A.BondM. A. (2001). Beyond the pages of a book: Interactive book reading and language development in preschool classrooms. Journal of Educational Psychology, 93(2), 243–250. https://doi.org/10.1037/0022-0663.93.2.243
61.
WesterveldM. F.PaynterJ.WicksR. (2020). Shared book reading behaviors of parents and their verbal preschoolers on the autism spectrum. Journal of Autism and Developmental Disorders, 50(8), 3005–3017. https://doi.org/10.1007/s10803-020-04406-6
62.
WhitehurstG.LoniganC. (1998). Child development and emergent literacy. Child Development, 69(3), 848–872. http://www.jstor.org/stable/1132208
63.
WicksR.PaynterJ.WesterveldM. F. (2020). Looking or talking: Visual attention and active engagement during shared book reading of preschool children on the autism spectrum. Autism, 24(6), 1384–1399. https://doi.org/101177/1362361319900594
64.
WongC.KasariC. (2012). Play and joint attention of children with autism in the preschool special education classroom. Journal of Autism and Developmental Disorders, 42(10), 2152–2161. https://doi.org/10.1007/s10803-012-1467-2
65.
YoderP.LloydB. P.SymonsF. (2018). Observational measurement of behavior. Paul H. Brookes Publishing.
66.
YoungT. A.MossB.CornwellL. (2007). The classroom library: A place for nonfiction, nonfiction in its place. Reading Horizons, 48(1), 1–18.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
