Validation of the Child Observation Record Advantage 1.5 Assessment Tool for Preschool Children: A Multilevel Bifactor Modeling Approach

Abstract

This paper reports an investigation into the psychometric properties of the COR-Advantage1.5 (COR-Adv1.5) assessment tool, a criterion-referenced observation-based instrument designed to assess the developmental abilities of children from birth through kindergarten. Using data from 8534 children participating in a state-funded preschool program for disadvantaged 4-year-olds, we employed a multilevel bifactor model to show that CORAdv1.5 has some potential as an unbiased assessment tool for tracking developmental progress in pre-kindergarten children, under a bifactor model parameterization, with fewer dimensions than were proposed by the test developer, HighScope. Examination of factor adequacy indices revealed that, despite the multidimensional structure, CORAdv1.5 items overwhelmingly reflect variance due to the general factors at the child and classroom levels. Evidence for multilevel reliability, response process validity, predictive validity, longitudinal measurement invariance, and generalizability of the items are also reported. The findings of this study have important implications for the use of COR-Adv1.5.

Keywords

child observation record validation preschool children early childhood assessment COR-advantage 1.5 multilevel/hierarchical linear modeling measurement

Get full access to this article

View all access options for this article.

References

Akaeze

H. O.

Lawrence

F. R.

J. H. C.

(2023). Resolving dimensionality in a child assessment tool: An application of the multilevel bifactor model. Educational and Psychological Measurement, 83(1), 93–115. https://doi.org/10.1177/00131644221082688

American Educational Research Association (2014). American psychological association. In Standards for educational and psychological testing. National Council on Measurement in Education, & Joint Committee on Standards for Educational and Psychological Testing.

Barghaus

K. M.

Fantuzzo

J. W.

(2014). Validation of the preschool child observation record: Does it pass the test for use in Head Start? Early Education and Development, 25(8), 1118–1141. https://doi.org/10.1080/10409289.2014.904646

Bialosiewicz

Murphy

Berry

(2013). An introduction to measurement invariance testing: Resource packet for participants. American Evaluation Association, 27(5), 1–37.

Bonifay

Cai

(2017). On the complexity of item response theory models. Multivariate Behavioral Research, 52(4), 465–484. https://doi.org/10.1080/00273171.2017.1309262

Bornovalova

M. A.

Choate

A. M.

Fatimah

Petersen

K. J.

Wiernik

B. M.

(2020). Appropriate use of bifactor analysis in psychopathology research: Appreciating benefits and limitations. Biological Psychiatry, 88(1), 18–27. https://doi.org/10.1016/j.biopsych.2020.01.013

Brannick

M. T.

(1995). Critical comments on applying covariance structure modeling. Journal of Organizational Behavior, 16(3), 201–213. https://doi.org/10.1002/job.4030160303

Brown

T. A.

(2015). Confirmatory factor analysis for applied research. Guilford Publications.

Cahalan

(2015). Assessing the reliability of the COR advantage. Wayne State University Thesis. http://digitalcommons.wayne.edu/oa_theses

10.

Canivez

G. L.

(2016). Bifactor modeling in construct validation of multi-factored tests: Implications for understanding multidimensional constructs and test interpretation. In Principles and methods of test construction: Standards and recent advancements. Hogrefe Publishers.

11.

Chen

F. F.

(2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834

12.

Chen

F. F.

Zhang

(2018). Bifactor models in psychometric test development. In The wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development. Wiley (pp. 325–345).

13.

Cheung

G. W.

Rensvold

R. B.

(2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 9(2), 233–255. https://doi.org/10.1207/s15328007sem0902_5

14.

Clauser

B. E.

Mazor

K. M.

(2005). Using statistical procedures to identify differentially functioning test items. Educational Measurement: Issues and Practice, 17(1), 31–44. https://doi.org/10.1111/j.1745-3992.1998.tb00619.x

15.

Dimitrov

D. M.

(2010). Testing for factorial invariance in the context of construct validation. Measurement and Evaluation in Counseling and Development, 43(2), 121–149. https://doi.org/10.1177/0748175610373459

16.

Dunn

E. C.

Masyn

K. E.

Johnston

W. R.

Subramanian

S. V.

(2015). Modeling contextual effects using individual-level data and without aggregation: An illustration of multilevel factor analysis (MLFA) with collective efficacy. Population Health Metrics, 13(1), 12. https://doi.org/10.1186/s12963-015-0045-1

17.

Enders

C. K.

(2010). Applied missing data analysis. Guilford Press.

18.

Fantuzzo

Hightower

Grim

Montes

(2002). Generalization of the child observation record: A validity study for diverse samples of urban, low-income preschool children. Early Childhood Research Quarterly, 17(1), 106–125. https://doi.org/10.1016/s0885-2006(02)00131-x

19.

Geldhof

G. J.

Preacher

K. J.

Zyphur

M. J.

(2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19(1), 72–91. https://doi.org/10.1037/a0032138

20.

Grice

J. W.

(2001). Computing and evaluating factor scores. Psychological Methods, 6(4), 430–450. https://doi.org/10.1037/1082-989x.6.4.430

21.

Halle

Zaslow

Wessel

Moodie

Darling-Churchill

(2011). Understanding and choosing assessments and developmental screeners for young children ages 3-5: Profiles of selected measures. In OPRE report# 2011-23. Administration for Children & Families.

22.

Hedges

L. V.

Hedberg

E. C.

(2007). Intraclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87. https://doi.org/10.3102/0162373707299706

23.

HighScope (2021). COR advantage: Observation-based child assessment . HighScope. https://highscope.org/cor-advantage

24.

HighScope Educational Research Foundation (2010). Preschool child observation record. In User guide (2nd edition). HighScope Press.

25.

HighScope Educational Research Foundation (2013). Technical report of the COR advantage validation study. HighScope Press.

26.

L. T.

Bentler

P. M.

(1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118

27.

Jak

Oort

F. J.

Dolan

C. V.

(2010). Measurement bias and multidimensionality; an illustration of bias detection in multidimensional measurement models. AStA Advances in Statistical Analysis, 94(2), 129–137. https://doi.org/10.1007/s10182-010-0128-z

28.

Jak

Oort

F. J.

Dolan

C. V.

(2014). Measurement bias in multilevel data. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 31–39. https://doi.org/10.1080/10705511.2014.856694

29.

Kaldaras

Akaeze

Krajcik

(2021). A methodology for determining and validating latent factor dimensionality of complex multi-factor science constructs measuring knowledge-in-use. Educational Assessment, 26(4), 241–263. https://doi.org/10.1080/10627197.2021.1971966

30.

Lawrence

F. R.

Blair

(2003). Factorial invariance in preventive intervention: Modeling the development of intelligence in low birth weight, preterm infants. Prevention Science: The Official Journal of the Society for Prevention Research, 4(4), 249–261. https://doi.org/10.1023/a:1026068115471

31.

Liu

Millsap

R. E.

West

S. G.

Tein

J. Y.

Tanaka

Grimm

K. J.

(2017). Testing measurement invariance in longitudinal data with ordered-categorical measures. Psychological Methods, 22(3), 486–506. http://dx.doi.org/10.1037/met0000075

32.

L Milfont

Fischer

(2010). Testing measurement invariance across groups: Applications in cross-cultural research. International Journal of Psychological Research, 3(1), 111–130. https://doi.org/10.21500/20112084.857

33.

McDonald

R. P.

(1999). Test theory: A unified treatment. Lawrence Erlbaum.

34.

McKelvey

R. D.

Zavoina

(1975). A statistical model for the analysis of ordinal level dependent variables. The Journal of Mathematical Sociology, 4(1), 103–120. https://doi.org/10.1080/0022250x.1975.9989847

35.

Mellenbergh

G. J.

(1982). Contingency table models for assessing item bias. Journal of Educational Statistics, 7(2), 105–118. https://doi.org/10.2307/1164960

36.

Meredith

(1993). Measurement invariance, factor analysis, and factorial invariance. Psychometrika, 58(4), 525–543. https://doi.org/10.1007/bf02294825

37.

Messick

(1989). Meaning and values in test validation: The science and ethics of assessment. Educational Researcher, 18(2), 5–11. https://doi.org/10.3102/0013189x018002005

38.

Millsap

R. E.

Cham

(2012). Investigating factorial invariance in longitudinal data. In Laursen

Little

T. D.

Card

N. A.

(Eds), Handbook of developmental research methods (pp. 109–127). Guilford Press.

39.

Muthén

L. K.

Muthén

B. O.

(1998-2017). Mplus user’s guide (Eighth Edition). Muthén & Muthén.

40.

Neill

(2004). A better way to do preschool assessment: Announcing the Revised Preschool COR. HighScope ReSource, 23(1).

41.

Oort

F. J.

(1992). Using restricted factor analysis to detect item bias. Methodika, 6, 150–166.

42.

Padilla

J. L.

Benítez

(2014). Validity evidence based on response processes. Psicothema, 26(1), 136–144. https://doi.org/10.7334/psicothema2013.259

43.

Panter

J. E.

Bracken

B. A.

(2013). Preschool assessment. In Geisinger

K. F.

Bracken

B. A.

Carlson

J. F.

Hansen

J.-I. C.

Kuncel

N. R.

Reise

S. P.

Rodriguez

M. C.

(Eds), APA handbook of testing and assessment in psychology. Testing and assessment in clinical and counseling psychology (Vol. 3, pp. 21–37). American Psychological Association.

44.

Rabinowitz

Travers

R. M. W.

(1953). Problems of defining and assessing teacher effectiveness. Educational Theory, 3(3), 212–219. https://doi.org/10.1111/j.1741-5446.1953.tb01052.x

45.

Raudenbush

S. W.

Bryk

A. S.

(2002) Hierarchical linear models: Applications and data analysis methods. Sage.

46.

Raykov

(2001). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. The British Journal of Mathematical and Statistical Psychology, 54(Pt 2), 315–323. https://doi.org/10.1348/000711001159582

47.

Reise

S. P.

(2012). Invited paper: The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696. http://doi.org/10.1080/00273171.2012.715555

48.

Reise

S. P.

Bonifay

W. E.

Haviland

M. G.

(2013). Scoring and modeling psychological measures in the presence of multidimensionality. Journal of Personality Assessment, 95(2), 129–140. https://doi.org/10.1080/00223891.2012.725437

49.

Reise

S. P.

Ventura

Nuechterlein

K. H.

Kim

K. H.

(2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84(2), 126–136. https://doi.org/10.1207/s15327752jpa8402_02

50.

Rodriguez

Reise

S. P.

Haviland

M. G.

(2016). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150. https://doi.org/10.1037/met0000045

51.

Schweinhart

L. J.

McNair

Barnes

Larner

A. M.

(1993). Observing young children in action to assess their development: The high/scope child observation record study. Educational and Psychological Measurement, 53(2), 445–455. https://doi.org/10.1177/0013164493053002014

52.

Snijders

T. A.

Bosker

R. J.

(2011). Multilevel analysis: An introduction to basic and advanced multilevel modeling. Sage.

53.

StataCorp (2015). Stata statistical software. StataCorp. (version release 14).

54.

Swaminathan

Rogers

H. J.

(1990). Detecting differential item functioning using logistic regression procedures. Journal of Educational Measurement, 27(4), 361–370. https://doi.org/10.1111/j.1745-3984.1990.tb00754.x

55.

Wakabayashi

Claxton

Smith

E. V.

Jr (2019). Validation of a revised observation-based assessment tool for children birth through kindergarten: The COR advantage. Journal of Psychoeducational Assessment, 37(1), 69–90. https://doi.org/10.1177/0734282917732491

56.

Willoughby

M. T.

Blanton

Z. E.

Family Life Project Investigators (2015). amp; Family Life Project InvestigatorsReplication and external validation of a bi-factor parameterization of attention deficit/hyperactivity symptomatology. Journal of Clinical child and Adolescent Psychology: The Official Journal for the Society of Clinical Child and Adolescent Psychology, American Psychological Association, Division 53, 44(1), 68–79. https://doi.org/10.1080/15374416.2013.850702

57.

Zinbarg

R. E.

Revelle

Yovel

(2005). Cronbach’s α, revelle’s β, and mcdonald’s ωH: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1), 123–133. https://doi.org/10.1007/s11336-003-0974-7

58.

Zinbarg

R. E.

Yovel

Revelle

McDonald

R. P.

(2006). Estimating generalizability to a latent variable common to all of a scale’s indicators: A comparison of estimators for ω_h. Applied Psychological Measurement, 30(2), 121–144. https://doi.org/10.1177/0146621605278814

59.

Zumbo

B. D.

(1999). A handbook on the theory and methods of differential item functioning (DIF). National Defense Headquarters.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB