Multidimensionality and hierarchical data structure are common in assessment data. These design features, if not accounted for, can threaten the validity of the results and inferences generated from factor analysis, a method frequently employed to assess test dimensionality. In this article, we describe and demonstrate the application of the multilevel bifactor model to address these features in examining test dimensionality. The tool for this exposition is the Child Observation Record Advantage 1.5 (COR-Adv1.5), a child assessment instrument widely used in Head Start programs. Previous studies on this assessment tool reported highly correlated factors and did not account for the nesting of children in classrooms. Results from this study show how the flexibility of the multilevel bifactor model, together with useful model-based statistics, can be harnessed to judge the dimensionality of a test instrument and inform the interpretability of the associated factor scores.
AhmadS.ZulkurnainN. N. A.KhairushalimiF. I. (2016). Assessing the fitness of a measurement model using confirmatory factor analysis (CFA). International Journal of Innovation and Applied Studies, 17(1), 159–168.
BarghausK. M.FantuzzoJ. W. (2014). Validation of the preschool Child Observation Record: Does it pass the test for use in Head Start?Early Education and Development, 25(8), 1118–1141.
4.
BollenK. A. (1989). Structural equations with latent variables. Wiley.
5.
BollenK. A.LennoxR. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110(2), 305–314.
6.
BonifayW.CaiL. (2017). On the complexity of item response theory models. Multivariate Behavioral Research, 52(4), 465–484.
7.
BonifayW.LaneS. P.ReiseS. P. (2017). Three concerns with applying a bifactor model as a structure of psychopathology. Clinical Psychological Science, 5(1), 184–186.
8.
BornovalovaM. A.ChoateA. M.FatimahH.PetersenK. J.WiernikB. M. (2020). Appropriate use of bifactor analysis in psychopathology research: Appreciating benefits and limitations. Biological Psychiatry, 88(1), 18–27.
9.
BrownT. A. (2015). Confirmatory factor analysis for applied research. Guilford Press.
10.
BrownT. A.MooreM. T. (2012). Confirmatory factor analysis. In HoyleR. H. (Ed.), Handbook of structural equation modeling (pp. 361–379). Guilford Press.
11.
BrowneM. W. (1972a). Oblique rotation to a partially specified target. British Journal of Mathematical and Statistical Psychology, 25, 207–212.
12.
BrowneM. W. (1972b). Orthogonal rotation to a partially specified target. British Journal of Mathematical and Statistical Psychology, 25, 115–120.
13.
BrowneM. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36(1), 111–150.
14.
BrunnerM.NagyG.WilhelmO. (2012). A tutorial on hierarchically structured constructs. Journal of Personality, 80(4), 796–846.
15.
CanivezG. L. (2016). Bifactor modeling in construct validation of multifactored tests: Implications for understanding multidimensional constructs and test interpretation. In SchweizerK.DiStefanoC. (Eds.), Principles and methods of test construction: Standards and recent advancements (pp. 247–271). Hogrefe.
16.
ChenF. F.WestS. G.SousaK. H. (2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41(2), 189–225.
17.
ChenF. F.ZhangZ. (2018). Bifactor models in psychometric test development. In IrwingP.BoothT.HughesD. J. (Eds.), The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development (pp. 325–345). Wiley Blackwell.
18.
ChoS. J.CohenA. S.KimS. H. (2014). A mixture group bifactor model for binary responses. Structural Equation Modeling, 21(3), 375–395.
19.
DueberD. M. (2017). Bifactor indices calculator: A Microsoft Excel-based tool to calculate various indices relevant to bifactor CFA models. https://doi.org/10.13023/edp.tool.01
20.
DyerN. G.HangesP. J.HallR. J. (2005). Applying multilevel confirmatory factor analysis techniques to the study of leadership. The Leadership Quarterly, 16(1), 149–167.
21.
EmbretsonS. E.ReiseS. P. (2000). Item response theory for psychologists. Lawrence Erlbaum.
22.
EpsteinA. S.SchweinhartL. J.DeBruin-PareckiA.RobinK. B. (2004). Preschool assessment: A guide to developing a balanced approach. Preschool Policy Matters, 7, 1–2.
23.
GeldhofG. J.PreacherK. J.ZyphurM. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72–91.
24.
GerbingD. W.AndersonJ. C. (1988). An updated paradigm for scale development incorporating unidimensionality and its assessment. Journal of Marketing Research, 25, 186–192.
25.
GignacG. E. (2006). The WAIS-III as a nested factors model: A useful alternative to the more conventional oblique and higher-order models. Journal of Individual Differences, 27, 73–86.
26.
GoldsteinH. (1995). Hierarchical data modeling in the social sciences. Journal of Educational and Behavioral Statistics, 20(2), 201–204.
27.
GorsuchR. L. (1983). Factor analysis (2nd ed.). Lawrence Erlbaum.
HammerJ. H. (2016, October). Percent of uncontaminated correlations (PUC) calculator: A Microsoft Excel-based tool to calculate the PUC statistic. https://doi.org/10.13023/edp.tool.01
30.
HarrellF. E. (2001). Regression modeling strategies: With applications to linear models, logistic regression, and survival analysis. Springer.
31.
HattieJ. (1985). Methodology review: Assessing unidimensionality of tests and itenls. Applied Psychological Measurement, 9(2), 139–164.
HolzingerK. J.SwinefordR. (1937). The bifactor method. Psychometrika, 2, 41–54.
34.
HuL. T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6(1), 1–55.
35.
HumphreysL. G. (1985). General intelligence: An integration of factor, test, and simplex theory. In WolmanB. B. (Ed.), Handbook of intelligence: Theories, measurements, and applications (pp. 15–35). Wiley.
36.
JennrichR. I.BentlerP. M. (2012). Exploratory bifactor analysis: The oblique case. Psychometrika, 77(3), 442–454.
37.
JiaoH.KamataA.WangS.JinY. (2012). A multilevel testlet model for dual local dependence. Journal of Educational Measurement, 49(1), 82–100.
38.
JöreskogK. G. (1971). Statistical analysis of sets of congeneric test. Psychometrika, 36, 109–133.
39.
MansolfM.ReiseS. P. (2016). Exploratory bifactor analysis: The Schmid-Leiman orthogonalization and Jennrich-Bentler analytic rotations. Multivariate Behavioral Research, 51(5), 698–717.
40.
McDonaldR. P. (1999). Test theory: A unified treatment. Lawrence Erlbaum.
41.
MurrayA. L.JohnsonW. (2013). The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence, 41(5), 407–422.
42.
MuthénB. O.KaplanD.HollisM. (1987). On structural equation modeling with data that are not missing completely at random. Psychometrika, 52(3), 431–462.
43.
MuthénB. O. (1991). Multilevel factor analysis of class and student achievement. Journal of Educational Measurement, 28, 338–354.
MuthénL. K.MuthénB. O. (1998–2017). Mplus user’s guide (8th ed.).
46.
PanterJ. E.BrackenB. A. (2013). Preschool assessment. In GeisingerK. F.BrackenB. A.CarlsonJ. F.HansenJ.-I. C.KuncelN. R.ReiseS. P.RodriguezM. C. (Eds.), APA handbook of testing and assessment in psychology, Vol. 3: Testing and assessment in school psychology and education (pp. 21–37). American Psychological Association.
47.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models: Applications and data analysis methods. SAGE.
48.
RavandH. (2015). Assessing testlet effect, impact, differential testlet, and item functioning using cross-classified multilevel measurement modeling. SAGE Open, 5(2), 2158244015585607.
49.
RaykovT. (2001). Estimation of congeneric scale reliability using covariance structure analysis with nonlinear constraints. British Journal of Mathematical and Statistical Psychology, 54(2), 315–323.
50.
ReckaseM. D. (2009). Multidimensional item response theory. Springer.
51.
ReiseS. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667–696.
52.
ReiseS. P.BonifayW. E.HavilandM. G. (2018). Bifactor modeling and the evaluation of scale scores. In IrwingP.BoothT.HughesD. J. (Eds.), The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development (Vol. 2, pp. 677–708). Wiley.
53.
ReiseS. P.HavilandM. G. (2005). Item response theory and the measurement of clinical change. Journal of Personality Assessment, 84, 228–238.
54.
ReiseS. P.KimD. S.MansolfM.WidamanK. F. (2016). Is the bifactor model a better model or is it just better at modeling implausible responses? Application of iteratively reweighted least squares to the Rosenberg Self-Esteem Scale. Multivariate Behavioral Research, 51(6), 818–838.
55.
ReiseS. P.MooreT. M.HavilandM. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544–559.
56.
ReiseS. P.MooreT. M.Maydeu-OlivaresA. (2011). Targeted bifactor rotations and assessing the impact of model violations on the parameters of unidimensional and bifactor models. Educational and Psychological Measurement, 71, 684–711.
57.
ReiseS. P.MorizotJ.HaysR. D. (2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research, 16, 19–31.
58.
ReiseS. P.ScheinesR.WidamanK. F.HavilandM. G. (2013). Multidimensionality and structural coefficient bias in structural equation modeling: A bifactor perspective. Educational and Psychological Measurement, 73(1), 5–26.
59.
ReiseS. P.VenturaJ.NuechterleinK. H.KimK. H. (2005). An illustration of multilevel factor analysis. Journal of Personality Assessment, 84(2), 126–136.
60.
RindskopfD.RoseT. (1988). Some theory and applications of confirmatory second-order factor analysis. Multivariate Behavioral Research, 23(1), 51–67.
61.
RodriguezA.ReiseS. P.HavilandM. G. (2016a). Applying bifactor statistical indices in the evaluation of psychological measures. Journal of Personality Assessment, 98(3), 223–237.
62.
RodriguezA.ReiseS. P.HavilandM. G. (2016b). Evaluating bifactor models: Calculating and interpreting statistical indices. Psychological Methods, 21(2), 137–150.
63.
SchererR.GustafssonJ. E. (2015). Student assessment of teaching as a source of information about aspects of teaching quality in multiple subject domains: An application of multilevel bifactor structural equation modeling. Frontiers in Psychology, 6, Article 1550.
64.
SchmidJ.LeimanJ. M. (1957). The development of hierarchical factor solutions. Psychometrika, 22, 53–61.
65.
SchweinhartL. J.McNairS.BarnesH.LarnerM. (1993). Observing young children in action to assess their development: The High/Scope Child Observation Record study. Educational and Psychological Measurement, 53, 445–455.
66.
SellbomM.TellegenA. (2019). Factor analysis in psychological assessment research: Common pitfalls and recommendations. Psychological Assessment, 31(12), 1428–1441.
67.
SmithL. L.ReiseS. P. (1998). Gender differences on negative affectivity: An IRT study of differential item functioning on the multidimensional personality questionnaire stress reaction scale. Journal of Personality and Social Psychology, 75, 1350–1362.
68.
SnijdersT. A.BoskerR. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. SAGE.
69.
StuckyB. D.EdelenM. O. (2015). Using hierarchical IRT models to create unidimensional measures from multidimensional data. In ReiseS. P.RevickiD. A. (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 183–206). Routledge.
70.
TabachnickB. G.FidellL. S. (2013). Using multivariate statistics. Pearson Education.
71.
WangY.KimE. S.DedrickR. F.FerronJ. M.TanT. (2018). A multilevel bifactor approach to construct validation of mixed-format scales. Educational and Psychological Measurement, 78(2), 253–271.
72.
YungY. F.ThissenD.McLeodL. D. (1999). On the relationship between the higher-order factor model and the hierarchical factor model. Psychometrika, 64, 113–128.
73.
ZinbargR.YovelI.RevelleW.McDonaldR. (2006). Estimating generalizability to a universe of indicators that all have one attribute in common: A comparison of estimators for omega. Applied Psychological Measurement, 30, 121–124.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
