The Cognitive Diagnostic Model Analysis of the Relationship Between Verbal and Quantitative Skills With Background Characteristics: Evidence From the Swedish Scholastic Assessment Test 2023

Abstract

For decades, researchers have sought analytic methods that yield diagnostic information about test-takers while ensuring that high-stakes tests remain free of construct-irrelevant bias. This study applies a general cognitive-diagnostic-model (CDM) framework to analyze the quantitative and verbal skills (QVS) assessed by the Swedish Scholastic Aptitude Test (SweSAT). A three-step latent-class logistic-regression approach was used to investigate the relationship between test-takers’ background characteristics and performance in each domain subskill. The analysis was conducted using representative data from 41,451 test-takers from the 2023 administration of the SweSAT, focusing on performance across four quantitative and four verbal subskills, and examining test characteristics. The results showed that the CDM method was appropriate for analyzing QVS, with evidence of measurement invariance across sex, age, and educational levels in the subskills. Additionally, the findings revealed distinct associations between sex, educational level, and age with performance in each QVS subskill. Implications for equitable selection in higher education are discussed.

Keywords

quantitative and verbal skills background characteristics cognitive diagnosis models three-step latent analysis differential item functioning logistic regression Swedish Scholastic Aptitude Test

Get full access to this article

View all access options for this article.

References

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705

Bastedo

M. N.

Glasener

K. M.

Deane

K. C.

Bowman

N. A.

(2022). Contextualizing the SAT: Experimental evidence on college admission recommendations for low-SES applicants. Educational Policy, 36(2), 282–311. https://doi.org/10.1177/0895904819874752

Bolt

Chen

DiBello

Hartz

Henson

Roussos

Stout

Templin

(2008). The Arpeggio Suite: Software for cognitive skills diagnostic assessment [Computer software and manual]. Assessment Systems.

Chen

de la Torre

Zhang

(2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123–140. https://doi.org/10.1111/j.1745-3984.2012.00185.x

Cho

S.-J.

Goodwin

A. P.

Naveiras

Salas

(2024). Differential and functional response time item analysis: An application to understanding paper versus digital reading processes. Journal of Educational Measurement, 61(2), 219–251. https://doi.org/10.1111/jedm.12389

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge.

de la Torre

(2009). DINA model and parameter estimation: A didactic. Journal of Educational and Behavioral Statistics, 34(1), 115–130. https://doi.org/10.3102/1076998607309474

de la Torre

(2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199. https://doi.org/10.1007/s11336-011-9207-7

de la Torre

Minchen

(2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicologia Educativa, 20(2), 89–97. https://doi.org/10.1016/j.pse.2014.11.001

10.

de la Torre

van der Ark

L. A.

Rossi

(2018). Analysis of clinical data from a cognitive diagnosis modeling framework. Measurement and Evaluation in Counseling and Development, 51(4), 281–296. https://doi.org/10.1080/07481756.2017.1327286

11.

Douglas

H. E.

Rubin

Scevak

Southgate

Macqueen

Richardson

J. T. E.

(2020). Older women, deeper learning: age and gender interact to predict learning approach and academic achievement at university. Frontiers in Education, 5, Article 158. https://doi.org/10.3389/feduc.2020.00158

12.

Gebremeskel

T. A.

Bachore

M. M.

Bushisho

E. W.

(2024). The effects of multiple intelligence-based reading tasks on EFL students reading skills achievements: The case of university students in Ethiopia. Heliyon, 10(13), e33591. https://doi.org/10.1016/j.heliyon.2024.e33591

13.

Graetz

Karimi

(2022). Gender gap variation across assessment types: Explanations and implications. Economics of Education Review, 91, 102313. https://doi.org/10.1016/j.econedurev.2022.102313

14.

Hannon

McNaughton-Cassill

(2011). SAT performance: Understanding the contributions of cognitive/learning and social/personality factors. Applied Cognitive Psychology, 25(4), 528–535. https://doi.org/10.1002/acp.1725

15.

Henson

Templin

Willse

(2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. https://doi.org/10.1007/s11336-008-9089-5

16.

Hou

de la Torre

Nandakumar

(2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the Wald test to investigate DIF in the DINA model. Journal of Educational Measurement, 51(1), 98–125. https://doi.org/10.1111/jedm.12036

17.

Huebner

Wang

(2011). A note on comparing examinee classification methods for cognitive diagnosis models. Educational and Psychological Measurement, 71(2), 407–419. https://doi.org/10.1177/0013164410388832

18.

Iaconangelo

de la Torre

. (2016). Three-step estimation of cognitive diagnosis models with covariate. In Paper presented at the international meeting of the psychometric society, Asheville, NC, USA.

19.

Jeffreys

(1961). Theory of probability (3rd ed.). Oxford University Press.

20.

Junker

B. W.

Sijtsma

(2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. https://doi.org/10.1177/01466210122032064

21.

Koponen

Eklund

K. M.

Heikkilä

Salminen

Fuchs

L. S.

Fuchs

Aro

(2020). Cognitive correlates of the covariance in reading and arithmetic fluency: Importance of serial retrieval fluency. Child Development, 91(4), 1063–1080. https://doi.org/10.1111/cdev.13287

22.

Kreitchmann

R. S.

de la Torre

Sorrel

M. A.

Najera

Abad

F. J.

(2022). Improving reliability estimation in cognitive diagnosis modeling. Behavior Research Methods, 55, 3446–3460. https://doi.org/10.3758/s13428-022-01967-5

23.

Lee

S.-J.

Chae

Y.-G.

(2007). Children’s internet use in a family context: Influence on family relationships and parental mediation. CyberPsychology and Behavior, 10(5), 640–644. https://doi.org/10.1089/cpb.2007.9975

24.

Lee

Y. S.

Park

Y. S.

Taylan

(2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the US national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144–177. https://doi.org/10.1080/15305058.2010.534571

25.

Liang

de la Torre

Law

(2021). Do background characteristics matter in children’s mastery of digital literacy? A cognitive diagnosis model analysis. Computers in Human Behavior, 122, 106850. https://doi.org/10.1016/j.chb.2021.106850

26.

de la Torre

. (2020). Gdina: The generalized DINA model framework. Comprehensive R Archive Network (CRAN). http://CRAN.R-project.org/package=GDINA

27.

Terzi

de la Torre

(2021). Detecting differential item functioning using multiple-group cognitive diagnosis models. Applied Psychological Measurement, 45(1), 37–53. https://doi.org/10.1177/0146621620965745

28.

Magis

Beland

Tuerlinckx

de Boeck

(2010). A general framework and an R package for the detection of dichotomous differential item functioning. Behavior Research Methods, 42(3), 847–862. https://doi.org/10.3758/BRM.42.3.847

29.

Maris

(1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187–212. https://doi.org/10.1007/BF0229453

30.

American Educational Research Association , American Psychological Association , & National Council on Measurement in Education . (2014). Standards for educational and psychological testing. American Educational Research Association.

31.

Rothstein

J. M.

(2004). College performance predictions and the SAT. Journal of Econometrics, 121, 297–317. https://doi.org/10.1016/j.jeconom.2003.10.003

32.

Rupp

A. A.

Templin

J. L.

Henson

R. A.

(2010). Diagnostic measurement: Theory, methods, and applications. The Guilford Press.

33.

Sackett

P. R.

Kuncel

N. R.

Beatty

A. S.

Rigdon

J. L.

Shen

Kiger

T. B.

(2012). The role of socioeconomic status in sat-grade relationships and in college admissions decisions. Psychological Science, 23(9), 1000–1007. https://doi.org/10.1177/0956797612438732

34.

Salthouse

T. A.

(2019). Trajectories of normal cognitive aging. Psychology and Aging, 34(1), 17–24. https://doi.org/10.1037/pag0000288

35.

Schmitt

Oswald

F. L.

Kim

B. H.

Imus

Merritt

Friede

Shivpuri

(2007). The use of background and ability profiles to predict college student outcomes. Journal of Applied Psychology, 92(1), 165–179. https://doi.org/10.1037/0021-9010.92.1.165

36.

Stage

C. W.

Ögren

(2004). The Swedish Scholastic Assessment Test (SweSAT): Development, results and experiences. Umea University.

37.

Sun

Wang

Guo

Xin

Song

(2023). Using a generalized logistic regression method to detect differential item functioning with multiple groups in cognitive diagnostic tests. Applied Psychological Measurement, 47(4), 328–346. https://doi.org/10.1177/01466216231174559

38.

Tatsuoka

K. K.

(1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. https://doi.org/10.1111/j.1745-3984.1983.tb00212.x

39.

Templin

J. L.

Henson

R. A.

(2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287. https://doi.org/10.1037/1082-989X.11.3.287

40.

Toprak-Yildiz

T. E.

(2021). An international comparison using cognitive diagnostic assessment: Fourth graders’ diagnostic profile of reading skills on PIRLS 2016. Studies in Educational Evaluation, 70, 101057. https://doi.org/10.1016/j.stueduc.2021.101057

41.

Vermunt

J. K.

(2010). Latent class modeling with covariates: Two improved three-step approaches. Political Analysis, 18(4), 450–469. https://doi.org/10.1093/pan/mpq025

42.

von Davier

(2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61(2), 287–307. https://doi.org/10.1348/000711007X193957

43.

von Davier

Lee

Y-S

. (2019). Handbook of diagnostic classification models (1st ed.). Springer, https://doi.org/10.1007/978-3-030-05584-4

44.

Wang

Duan

. (2024). Determining the number of attributes in the GDINA model. British Journal of Mathematical and Statistical Psychology, 78(1), 1–28. https://doi.org/10.1111/bmsp.12349

45.

Wedman

(2017). Reasons for gender-related differential item functioning in a college admissions test. Scandinavian Journal of Educational Research, 62(6), 959–970. https://doi.org/10.1080/00313831.2017.1402365

46.

Chang

H. H.

Kong

Zhang

(2020). International comparative study on PISA mathematics achievement test based on cognitive diagnostic models. Frontiers in Psychology, 11, 2230. https://doi.org/10.3389/fpsyg.2020.02230

47.

Sun

Kong

(2023). Cognitive diagnostic analysis of students’ mathematical competency based on the DINA model. Psychology in the Schools, 20(9), 3135–3150. https://doi.org/10.1002/pits.22916

48.

Zwick

Green

J. G.

(2007). New perspectives on the correlation of SAT scores, high school grades, and socioeconomic factors. Journal of Educational Measurement, 44(1), 23–45. https://doi.org/10.1111/j.1745-3984.2007.00025.x