For results from large-scale surveys to inform policy and practice appropriately, all participants must interpret and respond to items similarly. While organizers of surveys assessing student outcomes often ensure this for achievement measures, doing so for psychological questionnaires is also critical. We demonstrate this by examining the dimensionality of reading self-concept—a crucial psychological construct for several outcomes—across reading achievement levels. We use Programme for International Student Assessment 2018 data (N = 529,966) and local structural equation models (LSEMs) to do so. Results reveal that reading self-concept dimensions (assessed through reading competence and difficulty) vary across reading achievement levels. Students with low reading achievement show differentiated responses to the two item sets (high competence–high difficulty). In contrast, students with high reading achievement have reconciled responses (high competence–low difficulty). Our results highlight the value of LSEMs in examining factor structure generalizability of constructs in large-scale surveys and call for greater cognitive testing during item development.
BauerD. J. (2017). A more general model for testing measurement invariance and differential item functioning. Psychological Methods, 22(3), 507–526. https://doi.org/10.1037/met0000077
4.
BorgersN.SikkelD.HoxJ. (2004). Response effects in surveys on children and adolescents: The effect of number of response options, negative wording, and neutral mid-point. Quality and Quantity, 38(1), 17–33.
5.
ChapmanJ. W.TunmerW. E. (1995). Development of young children’s reading self-concepts: An examination of emerging subcomponents and their relationship with reading achievement. Journal of Educational Psychology, 87(1), 154–167. https://doi.org/10.1037/0022-0663.87.1.154
6.
CheungG. W.RensvoldR. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9(2), 233–255. https://doi.org/10.1207/S15328007SEM0902_5
EndersC. K.BandalosD. L. (2001). The relative performance of full information maximum likelihood estimation for missing data in structural equation models. Structural Equation Modeling, 8(3), 430–457. https://doi.org/10.1207/S15328007SEM0803_5
9.
FloydR. G.BergeronR.AlfonsoV. C. (2006). Cattell–Horn–Carroll cognitive ability profiles of poor comprehenders. Reading and Writing, 19(5), 427–456. https://doi.org/10.1007/s11145-006-9002-5
GnambsT.SchroedersU. (2020). Cognitive abilities explain wording effects in the Rosenberg Self-Esteem Scale. Assessment, 27(2), 404–418. https://doi.org/10.1177/1073191117746503
12.
GuoJ.MarshH. W.ParkerP. D.DickeT.Van ZandenB. (2019). Countries, parental occupation, and girls’ interest in science. The Lancet, 393(10171), e6–e8. https://doi.org/10.1016/S0140-6736(19)30210-7
13.
GuoJ.MarshH. W.ParkerP. D.MorinA. J.YeungA. S. (2015). Expectancy-value in mathematics, gender and socioeconomic background as predictors of achievement and aspirations: A multi-cohort study. Learning and Individual Differences, 37, 161–168. https://doi.org/10.1016/j.lindif.2015.01.008
14.
GuoJ.ParkerP. D.MarshH. W.MorinA. J. (2015). Achievement, motivation, and educational choices: A longitudinal study of expectancy and value using a multiplicative perspective. Developmental Psychology, 51(8), 1163–1176. https://doi.org/10.1037/a0039440
15.
HildebrandtA.LüdtkeO.RobitzschA.SommerC.WilhelmO. (2016). Exploring factor model parameters across continuous variables with local structural equation models. Multivariate Behavioral Research, 51(2–3), 257–258. https://doi.org/10.1080/00273171.2016.1142856
16.
HildebrandtA.WilhelmO.RobitzschA. (2009). Complementary and competing factor analytic approaches for the investigation of measurement invariance. Review of Psychology, 16(2), 87–102. https://hrcak.srce.hr/70641
17.
HülürG.WilhelmO.RobitzschA. (2011). Intelligence differentiation in early childhood. Journal of Individual Differences, 32, 170–179. https://doi.org/10.1027/1614-0001/a000049
18.
JerrimJ.ParkerP.ChoiA.ChmielewskiA. K.SälzerC.ShureN. (2018). How robust are cross-country comparisons of PISA scores to the scaling model used?Educational Measurement: Issues and Practice, 37(4), 28–39. https://doi.org/10.1111/emip.12211
LafontaineD.DupontV.JaegersD.SchillingsP. (2019). Self-concept in reading: Factor structure, cross-cultural invariance and relationships with reading achievement in an international context (PIRLS 2011). Studies in Educational Evaluation, 60, 78–89. https://doi.org/10.1016/j.stueduc.2018.11.005
21.
MarshH. W. (1986). Negative item bias in ratings scales for preadolescent children: A cognitive-developmental phenomenon. Developmental Psychology, 22(1), 37–49. https://doi.org/10.1037/0012-1649.22.1.37
22.
MarshH. W. (1996). Positive and negative global self-esteem: A substantively meaningful distinction or artifactors?Journal of Personality and Social Psychology, 70(4), 810–819. https://doi.org/10.1037/0022-3514.70.4.810
23.
MarshH. W.CravenR. G.DebusR. (1991). Self-concepts of young children 5 to 8 years of age: Measurement and multidimensional structure. Journal of Educational Psychology, 83(3), 377–392. https://doi.org/10.1037/0022-0663.83.3.377
24.
MarshH. W.ParkerP. D.GuoJ.PekrunR.BasarkodG. (2020). Psychological comparison processes and self-concept in relation to five distinct frame-of-reference effects: Pan-human cross-cultural generalizability over 68 countries. European Journal of Personality, 34(2), 180–202. https://doi.org/10.1002/per.2232
MarshH. W.TrautweinU.LüdtkeO.KöllerO.BaumertJ. (2005). Academic self-concept, interest, grades, and standardized test scores: Reciprocal effects models of causal ordering. Child Development, 76(2), 397–416. https://doi.org/10.1111/j.1467-8624.2005.00853.x
27.
MarshH. W.Van ZandenB.ParkerP. D.GuoJ.ConigraveJ.SeatonM. (2019). Young women face disadvantage to enrollment in university STEM coursework regardless of prior achievement and attitudes. American Educational Research Journal, 56(5), 1629–1680. https://doi.org/10.3102/0002831218824111
28.
McNeishD. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
NagengastB.MarshH. W. (2012). Big fish in little ponds aspire more: Mediation and cross-cultural generalizability of school-average ability effects on self-concept and career aspirations in science. Journal of Educational Psychology, 104(4), 1033–1053. https://doi.org/10.1037/a0027697
OlaruG.RobitzschA.HildebrandtA.SchroedersU. (2020). Local structural equation modeling for longitudinal data. Psyarxiv. https://doi.org/10.31234/osf.io/q79c5
33.
OlaruG.SchroedersU.HartungJ.WilhelmO.WrzusC. (2019). Ant colony optimization and local weighted structural equation modeling: A tutorial on novel item and person sampling procedures for personality research. European Journal of Personality, 33(3), 400–419. https://doi.org/10.1002/per.2195
34.
ParkerP. D.MarshH. W.CiarrochiJ.MarshallS.AbduljabbarA. S. (2014). Juxtaposing math self-efficacy and self-concept as predictors of long-term achievement outcomes. Educational Psychology, 34(1), 29–48. https://doi.org/10.1080/01443410.2013.797339
35.
RadM. S.MartinganoA. J.GingesJ. (2018). Toward a psychology of Homo sapiens: Making psychological science more representative of the human population. Proceedings of the National Academy of Sciences, 115(45), 11401–11405. https://doi.org/10.1073/pnas.1721165115
36.
R Core Team. (2015). R: A language and environment for statistical computing. R Foundation for Statistical Computing.
37.
RetelsdorfJ.KöllerO.MöllerJ. (2011). On the effects of motivation on reading performance growth in secondary school. Learning and Instruction, 21(4), 550–559. https://doi.org/10.1016/j.learninstruc.2010.11.001
RosseelY. (2012). Lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
40.
RubinD. B. (1987). Multiple imputation for nonresponse in surveys. Wiley.
41.
SchmalzX.MarinusE.ColtheartM.CastlesA. (2015). Getting to the bottom of orthographic depth. Psychonomic Bulletin & Review, 22(6), 1614–1629. https://doi.org/10.3758/s13423-015-0835-2
42.
ShavelsonR. J.HubnerJ. J.StantonG. C. (1976). Self-concept: Validation of construct interpretations. Review of Educational Research, 46(3), 407–441. https://doi.org/10.3102/00346543046003407
43.
SongS.PerryL. B.McConneyA. (2014). Explaining the achievement gap between Indigenous and non-Indigenous students: An analysis of PISA 2009 results for Australia and New Zealand. Educational Research and Evaluation, 20(3), 178–198. https://doi.org/10.1080/13803611.2014.892432
44.
TunmerW. E.PrattC.HerrimanM. L.BoweyJ. (1984). Metalinguistic awareness in children: Theory, research, and implications. Springer. https://doi.org/10.1007/978-3-642-69113-3
45.
WagemakerH. (2013). International large-scale assessments: From research to policy. In RutkowskiL.von DavierM.RutkowskiD. (Eds.), Handbook of international large-scale assessment: Background, technical issues, and methods of data analysis (pp. 11–36). CRC Press.
46.
WelshJ. A.NixR. L.BlairC.BiermanK. L.NelsonK. E. (2010). The development of cognitive skills and gains in academic school readiness for children from low-income families. Journal of Educational Psychology, 102(1), 43–53. https://doi.org/10.1037/a0016738
47.
WigfieldA.GuthrieJ. T. (1997). Relations of children’s motivation for reading to the amount and breadth or their reading. Journal of Educational Psychology, 89(3), 420–432. https://doi.org/10.1037/0022-0663.89.3.420
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
