The importance of afterschool hours for youth development is widely acknowledged, and afterschool settings have recently received increasing attention as an important venue for youth interventions, bringing a growing need for reliable and valid measures of afterschool quality. This study examined the extent to which the two observational tools, that is, Caregiver Interaction Scales (CIS) and Promising Practices Rating Scales (PPRS), could serve as reliable and valid tools for assessing the various dimensions of afterschool setting quality. The observation data of 44 afterschool programs were analyzed using both standard psychometric procedures (i.e., internal consistency, interrater reliability, and factor analysis) and generalizability theory. The results show the potential promise of the instruments, on one hand, and suggest future directions for improvement of measurement design and development of the field, on the other hand. In particular, our findings suggest the importance of addressing the effect of day-to-day fluctuations in observed afterschool quality.
ArnettJ. (1989). Caregivers in day-care centers: Does training matter?Journal of Applied Developmental Psychology, 10, 541-552.
3.
BentlerP. M. (1990). Comparative fit indices in structural models. Psychological Bulletin, 107, 238-246.
4.
BlazevskiJ.SmithC. (2007). After-school quality and school-day outcomes in Michigan’s 21st CCLC program. Ypsilanti, MI: High/Scope Educational Research Foundation.
5.
BronfenbrennerU. (1979). The ecology of human development: Experiments by nature and design. Cambridge, MA: Harvard University Press.
6.
BrownT. A. (2006). Confirmatory factor analysis for applied research. New York, NY: Guilford Press.
7.
BrowneM. W.CudeckR. (1992). Alternative ways of assessing model fit. Sociological Methods & Research, 21, 230-258.
8.
CampbellP. H.MilbourneS. A. (2005). Improving the quality of infant-toddler care through professional development. Topics in Early Childhood Special Education, 25, 3-14.
9.
CheungG. W.RensvoldR. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.
10.
DavisH. A. (2006). Exploring the contexts of relationship quality between middle school students and teachers. The Elementary School Journal, 106, 193-223.
11.
de KruifR. E. L.McWilliamR. A.RidleyS. M.WakelyM. B. (2000). Classification of teachers’ interaction behaviors in early childhood classrooms. Early Childhood Research Quarterly, 15, 247-268. doi:10.1016/S0885-2006(00)00051-X
12.
DuBoisD.HollowayB.ValentineJ.CooperH. (2002). Effectiveness of mentoring programs for youth: A meta-analytic review. American Journal of Community Psychology, 30, 157-197. doi:10.1023/a:1014628810714
13.
DurlakJ. A.WeissbergR. P.PachanM. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294-309. doi:10.1007/s10464-010-9300-6
14.
EcclesJ. S.GootmanJ. A. (2002). Community programs to promote youth development. Washington, DC: National Academies Press.
15.
EcclesJ. S.RoeserR. W. (2009). Schools, academic motivation, and stage-environment fit. In LernerR. M.SteinbergL. (Eds.), Handbook of adolescent psychology (3rd ed., pp. 404-434). Hoboken, NJ: John Wiley.
16.
FleissJ. L. (1981). The measurement of interrater agreement. In FleissJ. L. (Ed.), Statistical methods for rates and proportions (pp. 212-236). New York, NY: John Wiley.
17.
FrancoN.LevittM. J. (1998). The social ecology of middle childhood: Family support, friendship quality, and self-esteem. Family Relations, 47, 315-321.
18.
FraserB. J. (1998). Classroom environment instruments: Development, validity and applications. Learning Environments Research, 1, 7-33.
19.
GerstenblithS. A.SouleD. A.GottfredsonD. C.LuS.KelltromM. A.WomerS. C.BrynerS. L. (2005). After-school programs, antisocial behavior, and positive youth development: An exploration of the relationship between program implementation and changes in youth behavior. In MahoneyJ.LarsonR.EcclesJ. (Eds.), Organized activities as contexts of development: Extracurricular activities, after-school and community programs (pp. 457-477). Mahwah, NJ: Lawrence Erlbaum.
20.
GonzalesN. A.CauceA. M.FriedmanR. J.MasonC. A. (1996). Family, peer, and neighborhood influence on academic achievement among African American adolescents: One year prospective effects. American Journal of Community Psychology, 24, 365-387.
21.
GortmakerS. L.LeeR. M.MozaffarianR. S.SobolA. M.NelsonT. F.RothB. A.WiechaJ. L. (2012). Effect of an after-school intervention on increases in children’s physical activity. Medicine & Science in Sports & Exercise, 44, 450-457.
22.
GottfredsonD.GerstenblithS.SouléD.WomerS.LuS. (2004). Do after school programs reduce delinquency?Prevention Science, 5, 253-266.
GrangerR. C.DurlakJ. A.YohalemN.ReisnerE. (2007). Improving after-school program quality. New York, NY: William T. Grant Foundation.
25.
GrossmanJ. B.GoldmithJ.SheldonJ.ArbretonA. J. A. (2009). Assessing after-school settings. New Directions for Youth Development, 121, 89-108. doi:10.1002/yd.298
26.
HalleT.Vick WhittakerJ. E.AndersonR. (2010). Quality in early childhood care and education settings: A compendium of measures. Washington, DC: Child Trends.
27.
HamreB. K.PiantaR. C.MashburnA. J.DownerJ. T. (2007). Building a science classrooms: Application of CLASS framework in over 4,000 U.S. early childhood and elementary classrooms. Retrieved from http://www.icpsr.umich.edu/files/PREK3RD
28.
HarmsT.JacobsE.WhiteD. (1996). School-Age Care Environment Rating Scale. New York, NY: Teachers College Press.
29.
KirtlandK. A.PorterD. E.AddyC. L.NeetM. J.WilliamsJ. E.SharpeP. A.. . .AinsworthB. E. (2003). Environmental measures of physical activity supports: Perception versus reality. American Journal of Preventive Medicine, 24, 323-331.
30.
KupermincG. P.SmithE. P.HenrichC. C. (2013). Introduction to the special issue on social and motivational processes in after-school settings: Bridging gaps between theory, research, and practice. The Journal of Early Adolescence, 33, 5-16.
31.
LarsonR. (2000). Toward a psychology of positive youth development. American Psychologist, 55, 170-183.
32.
MahoneyJ. L.ParenteM. E.LordH. (2007). After-school program engagement: Links to child competence and program quality and content. The Elementary School Journal, 107, 385-404. doi:10.1086/516670
33.
MahoneyJ. L.StattinH. (2000). Leisure activities and adolescent antisocial behavior: The role of structure and social context. Journal of Adolescence, 23, 113-127.
34.
MahoneyJ. L.StattinH.LordH. (2004). Unstructured youth recreation centre participation and antisocial peers. International Journal of Behavioral Development, 28, 553-560.
35.
MahoneyJ. L.ZiglerE. F. (2006). Translating science to policy under the No Child Left Behind Act of 2001: Lessons from the national evaluation of the 21st-Century Community Learning Centers. Journal of Applied Developmental Psychology, 27, 282-294.
36.
McCallG. J. (1984). Systematic field observation. Annual Review of Sociology, 10, 263-282.
37.
McGrawK. O.WongS. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30-46.
38.
McLaughlinM. W.IrbyM. A.LangmanJ. (1994). Urban sanctuaries: Neighborhood organizations in the lives and futures of inner-city youth. San Francisco, CA: Jossey-Bass.
39.
NockM. K.KurtzS. M. S. (2005). Direct behavioral observation in school settings: Bringing science to practice. Cognitive and Behavioral Practice, 12, 359-370.
40.
NunnallyJ.BernsteinI. (1994). Psychometric theory. New York, NY: McGraw-Hill.
41.
OsgoodD. W.AndersonA. L. (2004). Unstructured socializing and rates of delinquency. Criminology, 42, 519-550. doi:10.1111/j.1745-9125.2004.tb00528.x
42.
PerkinsD. F.BordenL. M. (2003). Key elements of community youth development programs. In VillarruelF. A.PerkinsD. F.BordenL. M.KeithJ. G. (Eds.), Community youth development: Practice, policy, and research (pp. 327-340). Thousand Oaks, CA: Sage.
43.
PiantaR. C.AllenJ. P. (2009). Building capacity for positive youth development in secondary school classrooms: Changing teachers’ interactions with students. In ShinnM.YoshikawaH. (Eds.), Toward positive youth development: Transforming schools and community programs (pp. 21-39). New York, NY: Oxford University Press.
44.
PiantaR. C.HamreB. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38, 109-119.
45.
PierceK. M.BoltD.VandellD. B. (2010). Specific features of after-school program quality: Associations with children’s functioning in middle childhood. American Journal of Community Psychology, 45, 381-393.
46.
PierceK. M.HammJ. V.VandellD. L. (1999). Experiences in after-school programs and children’s adjustment in first-grade classrooms. Child Development, 70, 56-767.
RaudenbushS. W.SampsonR. J. (1999). Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observation of neighborhoods. Sociological Methodology, 29, 1-41.
49.
ShavelsonR. J.WebbN. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
50.
ShernoffD. J. (2010). Engagement in after-school programs as a predictor of social competence and academic performance. American Journal of Community Psychology, 45, 325-337.
51.
ShinnM.RapkinB. D. (2000). Cross-level research without cross-ups in community psychology. In RappaportJ.SeidmanE. (Eds.), Handbook of community psychology (pp. 669-695). New York, NY: Springer.
52.
SkinnerC. H.DittmerK. I.HowellL. A. (2000). Direct observation in school settings: Theoretical issues. In ShapiroE. S.KratochwillT. R. (Eds.), Behavioral assessment in schools: Theory, research, and clinical foundations (pp. 19-45). New York, NY: Guilford Press.
53.
SmithC.HohmannC. (2005). Full findings from the youth program quality assessment validation study. Ypsilanti, MI: High/Scope Educational Research Foundation.
54.
SmithE. P.FaulkM.SizerM. A. (2013). Exploring the meso-system: The roles of community, family, and peers in adolescent delinquency and positive youth development. Youth & Society. DOI 10.1177/0044118X13491581
55.
StuhlmanM. W.HamreB. K.DownerJ. T.PiantaR. C. (2010). A practitioner’s guide to conducting classroom observations: What the research tells us about choosing and using observational systems. Charlottesville: University of Virginia.
56.
VandellD. L.ReisnerE. R.BrownB. B.PierceK. M.DadismanK.PechmanE. M. (2004). The study of promising after-school programs: Descriptive report of the promising programs. Retrieved from http://www.wcer.wisc.edu/childcare/statements.html
57.
VandellD. L.ReisnerE. R.PierceK. M. (2007). Outcomes linked to high-quality afterschool programs: Longitudinal findings from the study of promising afterschool programs. Washington, DC: Policy Studies Associates.
58.
VandenbergR. J.LanceC. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3, 4-69.
59.
VéronneauM.-H.DishionT. J. (2011). Middle school friendships and academic achievement in early adolescence: A longitudinal analysis. The Journal of Early Adolescence, 31, 99-124. doi:10.1177/0272431610384485
60.
WalkerK. E.ArbretonA. J. A. (2004). After-school pursuits: An examination of outcomes in the San Francisco beacon initiative. New York, NY: Public/Private Ventures.
61.
YohalemN.Wilson-AhlstromA. (2010). Inside the black box: Assessing and improving quality in youth programs. American Journal of Community Psychology, 45, 350-357.
