This paper reports an investigation into the psychometric properties of the COR-Advantage1.5 (COR-Adv1.5) assessment tool, a criterion-referenced observation-based instrument designed to assess the developmental abilities of children from birth through kindergarten. Using data from 8534 children participating in a state-funded preschool program for disadvantaged 4-year-olds, we employed a multilevel bifactor model to show that CORAdv1.5 has some potential as an unbiased assessment tool for tracking developmental progress in pre-kindergarten children, under a bifactor model parameterization, with fewer dimensions than were proposed by the test developer, HighScope. Examination of factor adequacy indices revealed that, despite the multidimensional structure, CORAdv1.5 items overwhelmingly reflect variance due to the general factors at the child and classroom levels. Evidence for multilevel reliability, response process validity, predictive validity, longitudinal measurement invariance, and generalizability of the items are also reported. The findings of this study have important implications for the use of COR-Adv1.5.
AkaezeH. O.LawrenceF. R.WuJ. H. C. (2023). Resolving dimensionality in a child assessment tool: An application of the multilevel bifactor model. Educational and Psychological Measurement, 83(1), 93–115. https://doi.org/10.1177/00131644221082688
2.
American Educational Research Association (2014). American psychological association. In Standards for educational and psychological testing. National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing.
3.
BarghausK. M.FantuzzoJ. W. (2014). Validation of the preschool child observation record: Does it pass the test for use in Head Start?Early Education and Development, 25(8), 1118–1141. https://doi.org/10.1080/10409289.2014.904646
4.
BialosiewiczS.MurphyK.BerryT. (2013). An introduction to measurement invariance testing: Resource packet for participants. American Evaluation Association, 27(5), 1–37.
BornovalovaM. A.ChoateA. M.FatimahH.PetersenK. J.WiernikB. M. (2020). Appropriate use of bifactor analysis in psychopathology research: Appreciating benefits and limitations. Biological Psychiatry, 88(1), 18–27. https://doi.org/10.1016/j.biopsych.2020.01.013
7.
BrannickM. T. (1995). Critical comments on applying covariance structure modeling. Journal of Organizational Behavior, 16(3), 201–213. https://doi.org/10.1002/job.4030160303
8.
BrownT. A. (2015). Confirmatory factor analysis for applied research. Guilford Publications.
CanivezG. L. (2016). Bifactor modeling in construct validation of multi-factored tests: Implications for understanding multidimensional constructs and test interpretation. In Principles and methods of test construction: Standards and recent advancements. Hogrefe Publishers.
11.
ChenF. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
12.
ChenF. F.ZhangZ. (2018). Bifactor models in psychometric test development. In The wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development. Wiley (pp. 325–345).
13.
CheungG. W.RensvoldR. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/s15328007sem0902_5
14.
ClauserB. E.MazorK. M. (2005). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31–44. https://doi.org/10.1111/j.1745-3992.1998.tb00619.x
15.
DimitrovD. M. (2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43(2), 121–149. https://doi.org/10.1177/0748175610373459
16.
DunnE. C.MasynK. E.JohnstonW. R.SubramanianS. V. (2015). Modeling contextual effects using individual-level data and without aggregation: An illustration of multilevel factor analysis (MLFA) with collective efficacy. Population Health Metrics, 13(1), 12. https://doi.org/10.1186/s12963-015-0045-1
17.
EndersC. K. (2010). Applied missing data analysis. Guilford Press.
18.
FantuzzoJ.HightowerD.GrimS.MontesG. (2002). Generalization of the child observation record: A validity study for diverse samples of urban, low-income preschool children. Early Childhood Research Quarterly, 17(1), 106–125. https://doi.org/10.1016/s0885-2006(02)00131-x
19.
GeldhofG. J.PreacherK. J.ZyphurM. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138
HalleT.ZaslowM.WesselJ.MoodieS.Darling-ChurchillK. (2011). Understanding and choosing assessments and developmental screeners for young children ages 3-5: Profiles of selected measures. In OPRE report# 2011-23. Administration for Children & Families.
22.
HedgesL. V.HedbergE. C. (2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706
HighScope Educational Research Foundation (2010). Preschool child observation record. In User guide (2nd edition). HighScope Press.
25.
HighScope Educational Research Foundation (2013). Technical report of the COR advantage validation study. HighScope Press.
26.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
27.
JakS.OortF. J.DolanC. V. (2010). Measurement bias and multidimensionality; an illustration of bias detection in multidimensional measurement models. AStA Advances in Statistical Analysis, 94(2), 129–137. https://doi.org/10.1007/s10182-010-0128-z
28.
JakS.OortF. J.DolanC. V. (2014). Measurement bias in multilevel data. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 31–39. https://doi.org/10.1080/10705511.2014.856694
29.
KaldarasL.AkaezeH.KrajcikJ. (2021). A methodology for determining and validating latent factor dimensionality of complex multi-factor science constructs measuring knowledge-in-use. Educational Assessment, 26(4), 241–263. https://doi.org/10.1080/10627197.2021.1971966
30.
LawrenceF. R.BlairC. (2003). Factorial invariance in preventive intervention: Modeling the development of intelligence in low birth weight, preterm infants. Prevention Science: The Official Journal of the Society for Prevention Research, 4(4), 249–261. https://doi.org/10.1023/a:1026068115471
31.
LiuY.MillsapR. E.WestS. G.TeinJ. Y.TanakaR.GrimmK. J. (2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological Methods, 22(3), 486–506. http://dx.doi.org/10.1037/met0000075
32.
L MilfontT.FischerR. (2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3(1), 111–130. https://doi.org/10.21500/20112084.857
33.
McDonaldR. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
34.
McKelveyR. D.ZavoinaW. (1975). A statistical model for the analysis of ordinal level dependent variables. The Journal of Mathematical Sociology, 4(1), 103–120. https://doi.org/10.1080/0022250x.1975.9989847
35.
MellenberghG. J. (1982). Contingency table models for assessing item bias. Journal of Educational Statistics, 7(2), 105–118. https://doi.org/10.2307/1164960
MessickS. (1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. https://doi.org/10.3102/0013189x018002005
38.
MillsapR. E.ChamH. (2012). Investigating factorial invariance in longitudinal data. In LaursenB.LittleT. D.CardN. A. (Eds), Handbook of developmental research methods (pp. 109–127). Guilford Press.
PanterJ. E.BrackenB. A. (2013). Preschool assessment. In GeisingerK. F.BrackenB. A.CarlsonJ. F.HansenJ.-I. C.KuncelN. R.ReiseS. P.RodriguezM. C. (Eds), APA handbook of testing and assessment in psychology. Testing and assessment in clinical and counseling psychology (Vol. 3, pp. 21–37). American Psychological Association.
RaudenbushS. W.BrykA. S. (2002) Hierarchical linear models: Applications and data analysis methods. Sage.
46.
RaykovT. (2001). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. The British Journal of Mathematical and Statistical Psychology, 54(Pt 2), 315–323. https://doi.org/10.1348/000711001159582
47.
ReiseS. P. (2012). Invited paper: The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. http://doi.org/10.1080/00273171.2012.715555
48.
ReiseS. P.BonifayW. E.HavilandM. G. (2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140. https://doi.org/10.1080/00223891.2012.725437
49.
ReiseS. P.VenturaJ.NuechterleinK. H.KimK. H. (2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84(2), 126–136. https://doi.org/10.1207/s15327752jpa8402_02
50.
RodriguezA.ReiseS. P.HavilandM. G. (2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045
51.
SchweinhartL. J.McNairS.BarnesH.LarnerA. M. (1993). Observing young children in action to assess their development: The high/scope child observation record study. Educational and Psychological Measurement, 53(2), 445–455. https://doi.org/10.1177/0013164493053002014
52.
SnijdersT. A.BoskerR. J. (2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Sage.
53.
StataCorp (2015). Stata statistical software. StataCorp. (version release 14).
54.
SwaminathanH.RogersH. J. (1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x
55.
WakabayashiT.ClaxtonJ.SmithE. V.Jr (2019). Validation of a revised observation-based assessment tool for children birth through kindergarten: The COR advantage. Journal of Psychoeducational Assessment, 37(1), 69–90. https://doi.org/10.1177/0734282917732491
56.
WilloughbyM. T.BlantonZ. E.Family Life Project Investigators (2015). amp; Family Life Project InvestigatorsReplication and external validation of a bi-factor parameterization of attention deficit/hyperactivity symptomatology. Journal of Clinical child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 44(1), 68–79. https://doi.org/10.1080/15374416.2013.850702
57.
ZinbargR. E.RevelleW.YovelI.LiW. (2005). Cronbach’s α, revelle’s β, and mcdonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. https://doi.org/10.1007/s11336-003-0974-7
58.
ZinbargR. E.YovelI.RevelleW.McDonaldR. P. (2006). Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for ωh. Applied Psychological Measurement, 30(2), 121–144. https://doi.org/10.1177/0146621605278814
59.
ZumboB. D. (1999). A handbook on the theory and methods of differential item functioning (DIF). National Defense Headquarters.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
