The current study provides a methodological review of studies supporting a general factor of intelligence as the primary model for contemporary measures of cognitive abilities. A further evaluation is provided by an empirical evaluation that compares statistical estimates using different approaches in a large sample of children (ages 9–13 years, N = 780) administered a comprehensive battery of cognitive measures. Results from this study demonstrate the ramifications of using the bifactor and Schmid–Leiman (BF/SL) technique and suggest that using BF/SL methods limit interpretation of cognitive abilities to only a general factor. The inadvertent use of BF/SL methods is demonstrated to impact both model dimensionality and variance estimates for specific measures. As demonstrated in this study, conclusions from both exploratory and confirmatory studies using BF/SL methods are significantly questioned, especially for studies with a questionable theoretical basis. Guidelines for the interpretation of cognitive test scores in applied practice are discussed.
American Psychological Association. (2017). Ethical principles of psychologists and code of conduct (2002, Amended June 1, 2010 and January 1, 2017). Retrieved fromhttp://www.apa.org/ethics/code/index.aspx
2.
BartlettM. S. (1950). Tests of significance in factor analysis. British Journal of Statistical Psychology, 3(2), 77-85.
3.
BeaujeanA.BensonN.McGillR.DombrowskiS. (2018). A misuse of IQ scores: Using the dual discrepancy/consistency model for identifying specific learning disabilities. Journal of Intelligence, 6(3), 36. doi:10.3390/jintelligence6030036
4.
BergeronR.FloydR. G. (2013). Individual part score profiles of children with intellectual disability: A descriptive analysis across three intelligence tests. School Psychology Review, 42(1), 22-38.
5.
BonifacyW.CaiL. (2017). On the complexity of item response theory models. Multivariate Behavioral Research, 52, 465-484.
6.
BorsboomD.MellenberghG. J.van HeerdenJ. (2004). The concept of validity. Psychological Review, 111(4), 1061-1071. doi:10.1037/0033-295X.111.4.1061
7.
BrowneM. W. (1968). A note on lower bounds for the number of common factors. Psychometrika, 33(2), 233-236.
8.
CanivezG. L. (2014). Construct validity of the WISC-IV with a referred sample: Direct versus indirect hierarchical structures. School Psychology Quarterly, 29(1), 38-51. doi:10.1037/spq0000032
9.
CanivezG. L.McGillR. J. (2016). Factor structure of the differential ability scales-second edition: Exploratory and hierarchical factor analyses with the core subtests. Psychological Assessment, 28(11), 1475-1488. doi:10.1037/pas0000279
10.
CanivezG. L.WatkinsM. W. (2010a). Exploratory and higher-order factor analyses of the Wechsler adult intelligence scale-fourth edition (WAIS-IV) adolescent subsample. School Psychology Quarterly, 25(4), 223-235. doi:10.1037/a0022046
11.
CanivezG. L.WatkinsM. W. (2010b). Investigation of the factor structure of the Wechsler adult intelligence scale-fourth edition (WAIS-IV): Exploratory and higher order factor analyses. Psychological Assessment, 22(4), 827-836. doi:10.1037/a0020429
12.
CanivezG. L.WatkinsM. W.DombrowskiS. C. (2016). Factor structure of the Wechsler intelligence scale for children-fifth edition: Exploratory factor analyses with the 16 primary and secondary subtests. Psychological Assessment, 28(8), 975-986. doi:10.1037/pas0000238
13.
CanivezG. L.WatkinsM. W.DombrowskiS. C. (2017). Structural validity of the Wechsler intelligence scale for children-fifth edition: Confirmatory factor analyses with the 16 primary and secondary subtests. Psychological Assessment, 29(4), 458-472. doi:10.1037/pas0000358
14.
CattellR. B. (1941). Some theoretical issues in adult intelligence testing. Psychological Bulletin, 38, 592.
15.
CattellR. B. (1971). Abilities: Their structure, growth, and action.Boston: Houghton Mifflin.
16.
ChenF. F.WestS.SousaK. (2006). A comparison of bifactor and second-order models of quality of life. Multivariate Behavioral Research, 41(2), 189-225.
17.
DavisJ. T.ParrG.LanW. (1997). Differences between learning disability subtypes classified using the revised Woodcock-Johnson psycho-educational battery. Journal of Learning Disabilities, 30(3), 346-352.
18.
DeckerS. L. (2004). Incremental validity of tactile measures in predicting letter-word identification skills. Archives of Clinical Neuropsychology, 19(7), 907-908.
19.
DeckerS. L. (2008). School neuropsychology consultation in neurodevelopmental disorders. Psychology in the Schools, 45(9), 799-811.
20.
DeckerS. (2012). Dimensional integration of assessment outcomes with intervention services for children with specific learning disabilities. Journal of Applied School Psychology, 28(2), 175-199.
21.
DeckerS. L.EnglundJ. A.RobertsA. M. (2014). Higher-order factor structures for the WISC-IV: Implications for neuropsychological test interpretation. Applied Neuropsychology: Child, 3(2), 135-144. doi:10.1080/21622965.2012.737760
22.
DeckerS. L.HaleJ. B.FlanaganD. P. (2013a). Professional practice issues in the assessment of cognitive functioning for educational applications. Psychology in the Schools, 50(3), 300-313.
23.
DeckerS. L.RobertsA. M.EnglundJ. A. (2013b). Cognitive predictors of rapid picture naming. Learning and Individual Differences, 25, 141-149.
24.
DombrowskiS. C. (2013). Investigating the structure of the WJ-III cognitive at school age. School Psychology Quarterly, 28(2), 154-169.
25.
DombrowksiS. C.CanivezG. L.WatkinsM. W.BeaujeanA. A. (2015). Exploratory bifactor analysis of the Wechsler intelligence scale for children-fifth edition with the 16 primary and secondary subtests. Intelligence, 53, 194-201. doi:10.1016/j.intell.2015.10.009
26.
DombrowskiS. C.McGillR. J.CanivezG. L. (2017). Exploratory and hierarchical factor analysis of the WJ-IV cognitive at school age. Psychological Assessment, 29(4), 394-407. doi:10.1037/pas0000350
27.
DombrowskiS. C.McGillR. J.CanivezG. L. (2018). An alternative conceptualization of the theoretical structure of the Woodcock-Johnson IV tests of cognitive abilities at school age: A confirmatory factor analytic investigation. Archives of Scientific Psychology, 6(1), 1–13. doi:10.1037/arc0000039
28.
DombrowskiS. C.McGillR. J.CanivezG. L.PetersonC. H. (2019). Investigating the theoretical structure of the differential ability scales-second edition through hierarchical exploratory factor analysis. Journal of Psychoeducational Assessment, 37(1), 91-104.
29.
DombrowskiS. C.WatkinsM. W. (2013). Exploratory and higher order factor analysis of the WJ-III full test battery: A school-aged analysis. Psychological Assessment, 25(2), 442-455. doi:10.1037/a0031335
30.
ElliottC. D. (2007). Differential ability scales—second edition (DAS-II). San Antonio, TX: Harcourt Assessment.
31.
FabrigarL. R.WegenerD. T.MacCallumR. C.StrahanE. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4(3), 272-299.
32.
FiorelloC. A.HaleJ. B.HoldnackJ. A.KavanaghJ. A.TerrellJ.LongL. (2007). Interpreting intelligence test results for children with disabilities: Is global intelligence relevant?Applied Neuropsychology, 14(1), 2-12.
33.
FlanaganD. P.OrtizO. S. (2002). Best practices in nondiscriminatory assessment. In ThomasA.GrimesJ. (Eds.), Best practices in school psychology IV (Vol. II, pp. 1351-1372). Bethesda, MD: National Association of School Psychology.
34.
FloydR. G.McGrewK. S.BarryA.RafaelF.RogersJ. G. (2009). General and specific effects on Cattell-Horn-Carroll broad ability composites: Analysis of the Woodcock-Johnson III normative update Cattell-Horn-Carroll factor cluster across development. School Psychology Review, 38, 249-265.
35.
FrazierT. W.YoungstromE. A. (2007). Historical increase in the number of factors measured by commercial tests of cognitive ability: Are we overfactoring?Intelligence, 35, 169-182.
36.
FrebergM. E.VandiverB. J.WatkinsM. W.CanivezG. L. (2008). Significant factor score variability and the validity of the WISC-III full scale IQ in predicting later academic achievement. Applied Neuropsychology, 15(2), 131-139. doi:10.1080/09084280802084010
37.
FuchsD.YoungC. L. (2006). On the irrelevance of intelligence in predicting responsiveness to reading instruction. Exceptional Children, 73(1), 8-30.
38.
GignacG. E. (2007). Multi-factor modeling in individual differences research: Some recommendations and suggestions. Personality and Individual Differences, 42(1), 37-48.
39.
GignacG. E. (2016). The higher-order model imposes a proportionality constraint: That is why the bifactor model tends to fit better. Intelligence, 55, 57-68.
40.
GignacG. E.WatkinsM. W. (2013). Bifactor modeling and the estimation of model-based reliability in the WAIS-IV. Multivariate Behavioral Research, 48(5), 639-662. doi:10.1080/00273171.2013.804398
41.
GoldA. B.Ewing-CobbsL.CirinoP.FuchsL. S.StuebingK. K.FletcherJ. M. (2013). Cognitive and behavioral attention in children with math difficulties. Child Neuropsychology, 19(4), 420-437. doi:10.1080/09297049.2012.690371
42.
GorsuchR. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.
43.
HornJ. L.NollJ. (1997). Human cognitive capabilities. Gf-Gc theory. In FlanaganD. P.GenshaftJ. L.HarrisonP. L. (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 53-91). New York: Guilford Press.
44.
HornJ. L.BlanksonA. N. (2012). Foundations for better understanding of cognitive abilities. In FlanaganD. P.HarrisonP. L. (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (pp. 73-98). New York: Guilford Press.
45.
HuntE. B. (2011). Human intelligence. Cambridge: Cambridge University Press.
46.
IBM Corp. (2013). IBM SPSS statistics for windows, version 22.0. New York: Armonk.
47.
IBM Corp. Released. (2012). IBM SPSS statistics for windows, version 21.0. Armonk, NY: IBM Corp.
48.
JensenA. R. (1998). The g factor. Westport: Praeger.
49.
KaiserH. F. (1974). An index of factorial simplicity. Psychometrika, 39(1), 31-36.
50.
KaufmanA. S.KaufmanN. L. (2004). KABC-II: Kaufman assessment battery for children. Circle Pines, MN: American Guidance Service.
51.
KavaleK. A.FornessS. R. (1994). Learning disabilities and intelligence: An uneasy alliance. In ScruggsT. E.MastropieriM. A. (Eds.), Advances in learning and behavioral disabilities (Vol. 8, pp. 1-63). Greenwich, CT: JAI Press.
52.
KavaleK. A.HoldnackJ. A.MostertM. P. (2006). Responsiveness to intervention and the identification of specific learning disability: A critique and alternative proposal. Learning Disability Quarterly, 29, 113-127.
53.
KeithT. Z.ReynoldsM. R. (2018). Using confirmatory factor analysis to aid in understanding the constructs measured by intelligence tests. In FlanaganD. P.HarrisonP. L. (Eds.), Contemporary intellectual assessment: Theories, tests, and issues (4th ed., pp. 853-900). New York: Guilford.
54.
KlineR. B. (1998). Principles and practice of structural equation modeling. New York: The Guilford Press.
55.
MansolfM.ReiseS. P. (2017). When and why the second-order and bifactor models are distinguishable. Intelligence, 61, 120-129.
56.
MarkonK. E. (2019). Bifactor and hierarchical models: Specification, inference, and interpretation. Annual Review of Clinical Psychology, 15, 51-69.
57.
Maydeu-OlivaresA.CoffmanD. L. (2006). Random intercept item factor analysis. Psychological methods, 11(4), 344-362.
58.
McGillR. J. (2015). Incremental criterion validity of the WJ-III COG clinical clusters. Canadian Journal of School Psychology, 30(1), 51-63.
59.
McGillR. J.CanivezG. L. (2016). Orthogonal higher order structure of the WISC-IV Spanish using hierarchical exploratory factor analytic procedures. Journal of Psychoeducational Assessment, 34(6), 600–606. doi:10.1177/0734282915624293
60.
McGillR. J.DombrowskiS. C. (2018). Factor structure of the CHC model for the KABC-II: Exploratory factor analyses with the 16 core and supplementary subtests. Contemporary School Psychology, 22(3), 1-15. doi:10.1007/s40688-017-0152-z
61.
McGillR. J.DombrowskiS. C.CanivezG. L. (2018). Cognitive profile analysis in school psychology: History, issues, and continued concerns. Journal of School Psychology, 71, 108-121.
62.
McGrewK. S.KeithT. Z.FlanaganD. P.VanderwoodM. (1997). Beyond g: The impact of Gf-Gc specific cognitive abilities research on the future use and interpretation of intelligence tests in the schools. School Psychology Review, 26(2), 189-210.
63.
McGrewK. S.SchrankF. A.WoodcockR. (2007). Technical manual: Woodcock-Johnson III normative update. Rolling Meadows, IL: Riverside Publishing.
64.
MiciakJ.FletcherJ. M.StuebingK. K.VaughnS.TolarT. D. (2014). Patterns of cognitive strengths and weaknesses: Identification rates, agreement, and validity for learning disabilities identification. School Psychology Quarterly, 29(1), 21-37. doi:10.1037/spq0000037
65.
MolenaarD. (2016). On the distortion of model fit in comparing the bifactor model and the higher-order factor model. Intelligence, 57, 60-63. doi:10.1016/j.intell.2016.03.007
66.
MorganG.HodgeK.WellsK.WatkinsM. (2015). Are fit indices biased in favor of bi-factor models in cognitive ability research?: A comparison of fit in correlated factors, higher-order, and bi-factor models via Monte Carlo simulations. Journal of Intelligence, 3(1), 2-20. doi:10.3390/jintelligence3010002
67.
MulaikS. A.QuartettiD. A. (1997). First order or higher order general factor?Structural Equation Modeling: A Multidisciplinary Journal, 4(3), 193-211.
68.
MurrayA. L.JohnsonW. (2013). The limitations of model fit in comparing the bi-factor versus higher-order models of human cognitive ability structure. Intelligence, 41(5), 407-422.
69.
NathansL. L.OswaldF. L.NimonK. (2012). Intepreting multiple linear regression: A guidebook of variable importance. Practical Assessment, Research, & Evaluation, 17(9), 1-19.
70.
NelsonJ. M.CanivezG. L.WatkinsM. W. (2013). Structural and incremental validity of the Wechsler adult intelligence scale-fourth edition with a clinical sample. Psychological Assessment, 25(2), 618-630. doi:10.1037/a0032086
71.
PengP.BarnesM.WangC.WangW.LiS.SwansonH. L., … TaoS. (2018). A meta-analysis on the relation between reading and working memory. Psychological Bulletin, 144(1), 48-76. doi:10.1037/bul0000124
72.
ReiseS. P. (2012). The rediscovery of bifactor measurement models. Multivariate Behavioral Research, 47(5), 667-696. doi:10.1080/00273171.2012.715555
73.
ReiseS. P.BonifayW. E.HavilandM. G. (2018). Bifactor modeling and the evaluation of scale scores. In IrwingP.BoothT.HughesD. J. (Eds.), The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development (Vol. 2, pp. 677-708). Hoboken, NJ: Wiley.
74.
ReiseS. P.KimD. S.MansolfM.WidamanK. F. (2016). Is the bifactor model a better model or is it just better at modeling implausible repsonses? Application of iteratively reweighted least squares to the Rosenberg self-esteem scale. Multivariate Behavioral Research, 51, 818-838.
75.
ReiseS. P.MooreT. M.HavilandM. G. (2010). Bifactor models and rotations: Exploring the extent to which multidimensional data yield univocal scale scores. Journal of Personality Assessment, 92(6), 544-559.
76.
ReynoldsM. R.KeithT. Z. (2013). Measurement and statistical issues in child assessment research. In SoklofskeD. H.ReynoldsC. R.SchweanV. L. (Eds.), The Oxford handbook of child psychological assessment (pp. 48-83). New York: Oxford University Press.
77.
SchmidJ. (1957). The comparability of the bi-factor and second-order factor patterns. The Journal of Experimental Education, 25, 249-253.
78.
SchmidJ.LeimanJ. M. (1957). The development of hierarchical factor solutions. Psychometrika, 22, 53-61.
79.
SchmiedekF.LiS.-C. (2004). Toward an alternative representation for disentangling age-associated differences in general and specific cognitive abilities. Psychology and Aging, 19(1), 40-56.
80.
SchneiderW. J.KaufmanA. S. (2017). Let’s not do away with comprehensive cognitive assessments just yet. Archives of Clinical Neuropsychology, 32(1), 8-20. doi:10.1093/arclin/acw104
81.
SchneiderW. J.McGrewK. S. (2012). The Cattell-Horn-Carroll model of intelligence. In D. P. Flanagan & P. L. Harrison (Eds.), Contemporary intellectual assessment: Theories, tests, and issues., 3rd ed. (pp. 99-144). New York: Guilford Press.
82.
SchrankF. A.McGrewK. S.MatherN.WendlingB. J.LaForteE. M. (2014). Woodcock-Johnson IV tests of cognitive abilities. Rolling Meadows, IL: Riverside Publishing.
83.
SchulzeR. (2005). Modeling structures of intelligence. In WilhelmO.EngleR. W. (Eds.), Handbook of understanding and measuring intelligence (pp. 241-263). Thousand Oaks: Sage Publications Inc.
84.
Sotelo‐DynegaM.DixonS. G. (2014). Cognitive assessment practices: A survey of school psychologists. Psychology in the Schools, 51(10), 1031-1045.
85.
SpearmanC. (1915). The measurement of intelligence. The Eugenics Review, 6(4), 312-313.
86.
SpearmanC. (1928). The abilities of man. Science, 68(1750), 38. doi:10.1126/science.68.1750.38-a
87.
SpearmanC. (1939). Intelligence tests. The Eugenics Review, 30(4), 249-254.
88.
SpearmanC. (1973). The nature of “intelligence” and the principles of cognition. New York: Arno Press.
89.
SternbergR. J.KaufmanJ. C. (1998). Human abilities. Annual Review of Psychology, 49, 479-502. doi:10.1146/annurev.psych.49.1.479
90.
TaubG. E.KeithT. Z.FloydR. G.McGrewK. S. (2008). Effects of general and broad cognitive abilities on mathematics achievement. School Psychology Quarterly, 23(2), 187-198. doi:10.1037/1045-3830.23.2.187
91.
WatkinsM. W. (2003). IQ subtest analysis: Clinical acumen or clinical illusion?The Scientific Review of Mental Health Practice, 2(2), 118-141.
92.
WatkinsM. W. (2010). Structure of the Wechsler intelligence scale for children-fourth edition among a national sample of referred students. Psychological Assessment, 22(4), 782-787.
93.
WatkinsM. W.BeaujeanA. A. (2014). Bifactor structure of the Wechsler preschool and primary scale of intelligence-fourth edition. School Psychology Quarterly, 29(1), 52-63. doi:10.1037/spq0000038
94.
WatkinsM. W.GluttingJ. J.LeiP.-W. (2007). Validity of the full-scale IQ when there is significant variability among WISC-III and WISC-IV factor scores. Applied Neuropsychology, 14(1), 13-20.
95.
WechslerD. (2008). Wechsler adult intelligence scale–fourth edition (WAIS–IV) (Vol. 22, p. 498). San Antonio, TX: NCS Pearson.
96.
WechslerD. (2012). Technical and interpretative manual: WPPSI-IV. New York: Pearson Inc.
97.
WeschlerD. (2014). Wechsler intelligence scale for children (WISC-V). Bloomington. MN: Pearson.
98.
WolffH.-G.PreisingK. (2005). Exploring item and higher order factor structure with the Schmid–Leiman solution: Syntax codes for SPSS and SAS. Behavior Research Methods, 37, 48-58.
99.
WoodcockR. W.McGrewK. S.MatherN. (2001). Woodcock-johnson III. Itasca: Riverside Publishing.
100.
YungY.-F.ThissenD.McLeodL. D. (1999). On the relationship between the higher-order factor model and the hierarchical factor model. Psychometrika, 64(2), 113-128.
