Engagement in early childhood is defined as a child’s level of participation with the environment. Engagement is an important construct in assessment and intervention of social and early learning competence given its link to school achievement. Few tools exist to assess engagement of young children in early education, and there is a need for a systematic direct observation tool that can be applied universally (e.g., with all young children) in these settings. This article describes preliminary evidence of validity and reliability for the Behavioral Observation of Students in Schools–Early Education (BOSS-EE). Specifically, the article describes results from a survey of experts and practitioners in which feedback was solicited on target behaviors and operational definitions, presents reliability data (i.e., interobserver and test–retest), examines correlations with a criterion measure, and describes variability across settings, sites, and methods (i.e., video vs. in vivo). Next steps in measurement development are discussed with attention to the challenges of producing a tool that can be used in a range of early education settings with diverse groups of young children.
BabbieE. (1990). Survey research methods (2nd ed.). Belmont, CA: Wadsworth.
2.
BennettK.ReichowB.WoleryM. (2011). Effects of structured teaching on the behavior of young children with disabilities. Focus on Autism and Other Developmental Disabilities, 26, 143–152.
3.
BevillA. R.GastD. L.MaguireA. M.VailC. O. (2001). Increasing engagement of preschoolers with disabilities through correspondence training and picture cues. Journal of Early Intervention, 24, 129–145.
4.
BrieschA. M.ChafouleasS. M.JohnsonA. (2016). Use of generalizability theory within K–12 school-based assessment: A critical review and analysis of the empirical literature. Applied Measurement in Education, 29, 83–107.
5.
BrieschA. M.DanielsB. (2013). Using self-management interventions to address general education behavioral needs: Assessment of effectiveness and feasibility. Psychology in the Schools, 50, 366–382.
6.
ChafouleasS. M.ChristT. J.Riley-TillmanT. C.BrieschA. M.ChaneseJ. A. M. (2007). Generalizability and dependability of direct behavior ratings to assess social behavior of preschoolers. School Psychology Review, 36, 63–79.
7.
CohenJ. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.
8.
ConeJ. D. (1977). The relevance of reliability and validity for behavioral assessment. Behavior Therapy, 8, 411–426.
9.
CookC. R.CollinsT.DartE.VanceM. J.McIntoshK.GradyE. A.DeCanoP. (2014). Evaluation of the class pass intervention for typically developing students with hypothesized escape-motivated disruptive classroom behavior. Psychology in the Schools, 51, 107–125.
10.
GormleyM. J.DupaulG. J. (2015). Teacher-to-teacher consultation: Facilitating consistent and effective intervention across grade levels for students with ADHD. Psychology in the Schools, 52, 124–138.
11.
GreenwoodC. R.BradfieldT.KaminskiR.LinasM.CartaJ. J.NylanderD. (2011). The response to intervention (RTI) approach in early childhood. Focus on Exceptional Children, 43, 2–22.
12.
GreshamF.ElliottS. N. (2008). Social skills improvement system (SSIS) rating scales. San Antonio, TX: Pearson.
13.
HallgrenK. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8, 23–34.
14.
HemmeterM. L.FoxL.JackS.BroylesL. (2007). A program-wide model of positive behavior support in early childhood settings. Journal of Early Intervention, 29, 337–355.
15.
HintzeJ. M. (2005). Psychometrics of direct observation. School Psychology Review, 34, 507–519.
16.
HintzeJ. M.MatthewsW. J. (2004). The generalizability of systematic direct observations across time and setting: A preliminary investigation of the psychometrics of behavioral observation. School Psychology Review, 33, 258–270.
17.
HintzeJ. M.VolpeR. J.ShapiroE. S. (2008). Best practices in the systematic direct observation of student behavior. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology (5th ed., pp. 319–335). Bethesda, MD: National Association of School Psychologists.
18.
HojnoskiR. L.MissallK. N. (2010). Social development in preschool classrooms: Promoting engagement, competence, and school readiness. In ShinnM. R.WalkerH. M. (Eds). Interventions for achievement and behavior problems in a three-tier model including RTI (pp. 703–728). Bethesda, MD: National Association of School Psychologists.
19.
HojnoskiR. L.WoodB. K.ShapiroE. S. (2014). The behavioral observation of students in school-early education (Unpublished protocol). Department of Education and Human Services, Lehigh University, Bethlehem, PA.
20.
HojnoskiR. L.LaracyS. D.OlsonC. L. (2016). Comparison of observational methods and their relation to ratings of engagement in young children. Topics in Early Childhood Special Education, 35, 211–222.
21.
HostermanS. J.DuPaulG. J.JitendraA. K. (2008). Teacher ratings of ADHD symptoms in ethnic minority students: Bias or behavioral difference?School Psychology Quarterly, 23, 418–435.
22.
KazdinA. E. (1982). Single-case research design: Methods for clinical and applied settings. New York, NY: Oxford University Press.
23.
LeBelT. J.ChafouleasS. M.BritnerP. A.SimonsenB. (2013). Use of a daily report card in an intervention package involving home-school communication to reduce disruptive behavior in preschoolers. Journal of Positive Behavior Interventions, 15, 103–112.
24.
LinH. L.LawrenceF. R.GorrellJ. (2003). Kindergarten teachers’ views of children’s readiness for school. Early Childhood Research Quarterly, 18, 225–237.
25.
MayM. E.HoweA. P. (2013). Evaluating competing reinforcement contingencies on off-task behavior of a preschooler with intellectual disability: A data-based case study. Education and Treatment of Children, 36, 97–109.
26.
McConnellS. R.BradfieldT.Wackerle-HollmanA.RodriguezM. (2012). Individual growth and development indicators of early literacy (2nd ed.). St. Paul, MN: Early Learning Labs.
27.
McFallR. M. (1982). A review and reformulation of the concept of social skills. Behavioral Assessment, 4, 1–33.
MissallK. N. (2002). Reconceptualizing school adjustment: A search for intervening variables (Doctoral Dissertation, University of Minnesota, 2002). Dissertation Abstracts International, 63(5-A), 1712.
30.
MissallK. N.McConnellS.NandaS.XiongE. (2018, March). Developing a prosocial teacher rating scale for universal screening in preschool and kindergarten. Presentation at the Biennial Conference on Research Innovations in Early Intervention, San Diego, CA.
31.
NicholsonH.KehleT. J.BrayM. A.Van HeestJ. V. (2011). The effects of antecedent physical activity on the academic engagement of children with autism spectrum disorder. Psychology in the Schools, 48, 198–213.
PfiffnerL. J.VillodasM.KaiserN.RooneyM.McBurnettK. (2013). Educational outcomes of a collaborative school-home behavioral intervention for ADHD. School Psychology Quarterly, 28, 25–36.
34.
PiotrkowskiC. S.BotskoM.MatthewsE. (2000). Parents’ and teachers’ beliefs about children’s school readiness in a high-need community. Early Childhood Research Quarterly, 15, 537–558.
35.
PonitzC. C.Rimm-KaufmanS. E. (2011). Contexts of reading instruction: Implications for literacy skills and kindergartners’ behavioral engagement. Early Childhood Research Quarterly, 26, 157–168.
36.
PonitzC. C.Rimm-KaufmanS. E.GrimmK. J.CurbyT. W. (2009). Kindergarten classroom quality, behavioral engagement, and reading achievement. School Psychology Review, 38, 102–120.
37.
PowellD. R.BurchinalM. R.FileN.KontosS. (2008). An eco-behavioral analysis of children’s engagement in urban public school preschool classrooms. Early Childhood Research Quarterly, 23, 108–123.
38.
ReaL. M.ParkerR. A. (2005). Designing and conducting survey research: A comprehensive guide (3rd ed.). San Francisco, CA: Jossey-Bass.
39.
SalviaJ.YsseldykeJ. E.BoltS. (2010). Assessment: In special and inclusive education (11th ed.). Boston, MA: Wadsworth.
40.
ShapiroE. S. (2011). Academic skills problems, fourth edition: Direct assessment and intervention. New York, NY: Guilford Press.
41.
ShihT. H.FanX. (2008). Comparing response rates from web and mail surveys: A meta-analysis. Field Methods, 20, 249–271.
42.
SnyderP. A.WixsonC. S.TalapatraD.RoachA. T. (2008). Assessment in early childhood: Instruction focused strategies to support response-to-intervention frameworks. Assessment for Effective Intervention, 34, 25–34.
43.
VieraA. J.GarrettJ. M. (2005). Understanding interobserver agreement: The Kappa statistic. Family Medicine, 37, 360–363.
44.
VolpeR. J.YoungG. I.PianaM. G.ZaslofskyA. F. (2012). Integrating classwide early literacy intervention and behavioral supports: A pilot investigation. Journal of Positive Behavior Interventions, 14, 56–64.
45.
WolfM. M. (1978). Social validity: The case of subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11, 203–214.
46.
WoodB. K.FerroJ. B.UmbreitJ.LiaupsinC. J. (2011). Addressing the challenging behavior of young children through systematic function-based intervention. Topics in Early Childhood Special Education, 30, 221–232.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
