The present study explores the sources of validity evidence of the mathematics self-concept construct in Programme for International Student Assessment (PISA) 2022 with a focus on concurrent, discriminant, and predictive evidence. Mathematics self-concept was provided with a single index variable in PISA 2022. Using only the U.S. students who participated in PISA 2022, we included 4552 students. While we grounded our analyses in the Standards (2014), we aimed to evaluate whether the mathematics self-concept is related to mathematics motivation, distinct from mathematics self-efficacy measures and predicts mathematics achievement. Results revealed a positive association between mathematics self-concept and motivation (χ2 = 86008.92, p < .001, φ = 0.153), suggesting concurrent evidence. Significant but negligible correlations with two separate mathematics self-efficacy measures (rpb = −0.078 and −0.073, p < .001) supported discriminant evidence. Multilevel modeling analysis demonstrated that mathematics self-concept significantly predicted mathematics performance (β = 20.20, p < .001), supporting predictive validity. Validity evidence for the mathematics self-concept scale in PISA 2022 could serve as a basis for research focused on enhancing student outcomes. Implications, limitations, and future directions are discussed.
American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME]. (2014). Standards for educational and psychological testing. American Educational Research Association. https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf
3.
AssorA.ConnellJ. P. (1992). The validity of students' self-reports as measures of performance affecting self-appraisals. In SchunkD. H.MeeceJ. L. (Eds.), Student perceptions in the classroom (pp. 25–47). Lawrence Erlbaum Associates, Inc. https://doi.org/10.4324/9780203052532
4.
BongM. (1998). Tests of the internal/external frames of reference model with subject-specific academic self-efficacy and frame-specific academic self-concepts. Journal of Educational Psychology, 90(1), 102–110. https://doi.org/10.1037/0022-0663.90.1.102
5.
BongM.ClarkR. E. (1999). Comparison between self-concept and self-efficacy in academic motivation research. Educational Psychologist, 34(3), 139–153. https://doi.org/10.1207/s15326985ep3403_1
6.
BongM.SkaalvikE. M. (2003). Academic self-concept and self-efficacy: How different are they really?Educational Psychology Review, 15(1), 1–40. https://doi.org/10.1023/A:1021302408382
7.
CalsynR. J.KennyD. A. (1977). Self-concept of ability and perceived evaluation of others: Cause or effect of academic achievement?Journal of Educational Psychology, 69(2), 136–145. https://doi.org/10.1037/0022-0663.69.2.136
8.
ClarkL. A.WatsonD. (2019). Constructing validity: New developments in creating objective measuring instruments. Psychological Assessment, 31(12), 1412–1427. https://doi.org/10.1037/pas0000626
9.
CookD. A.BrydgesR.GinsburgS.HatalaR. (2015). A contemporary approach to validity arguments: A practical guide to Kane's framework. Medical Education, 49(6), 560–575. https://doi.org/10.1111/medu.12678
10.
CooperK. M.KriegA.BrownellS. E. (2018). Who perceives they are smarter? Exploring the influence of student characteristics on student academic self-concept in physiology. Advances in Physiology Education, 42(2), 200–208. https://doi.org/10.1152/advan.00085.2017
11.
DaiS.HaoT.ArdashevaY.RamazanO.DanielsonR. W.AustinB. (2023). PISA reading achievement: Identifying predictors and examining model generalizability for multilingual students. Reading and Writing, 36(10), 2763–2795. https://doi.org/10.1007/s11145-022-10357-4
12.
DanielsonR. W.SinatraG. M.DaiS.SeyranianV.HeddyB. C.MarshJ. A.PolikoffM. S. (2023). The development and validation of the elementary activity interest measure. The Journal of Experimental Education, 93(2), 1–23. https://doi.org/10.1080/00220973.2023.2276933
13.
DingY.Yang HansenK.KlappA. (2022). Testing measurement invariance of mathematics self-concept and self-efficacy in PISA using MGCFA and the alignment method. European Journal of Psychology of Education, 38(2), 709–732. https://doi.org/10.1007/s10212-022-00623-y
14.
EndersC. K.KellerB. T.LevyR. (2018). A fully conditional specification approach to multilevel imputation of categorical and continuous variables. Psychological Methods, 23(2), 298–317. https://doi.org/10.1037/met0000148
15.
GorgesJ.HollmannJ. (2019). The structure of academic self-concept when facing novel learning content: Multidimensionality, hierarchy, and change. Europe's Journal of Psychology, 15(3), 491–508. https://doi.org/10.5964/ejop.v15i3.1716
16.
GustafssonJ. (2018). International large scale assessments: Current status and ways forward. Scandinavian Journal of Educational Research, 62(3), 328–332. https://doi.org/10.1080/00313831.2018.1443573
17.
HammoudiM. M. (2019). Measurement of students’ mathematics motivation and self-concept at institutions of higher education: Evidence of reliability and validity. International Journal of Mathematical Education in Science & Technology, 51(1), 63–86. https://doi.org/10.1080/0020739x.2019.1670369
18.
Hernández-TorranoD.CourtneyM. G. (2021). Modern international large-scale assessment in education: An integrative review and mapping of the literature. Large-scale Assessments in Education, 9(1), 17. https://doi.org/10.1186/s40536-021-00109-1
19.
HolensteinM.BruckmaierG.GrobA. (2021). How do self‐efficacy and self‐concept impact mathematical achievement? The case of mathematical modelling. British Journal of Educational Psychology, 92(1), 155–174. https://doi.org/10.1111/bjep.12443
20.
HopfenbeckT. N.LenkeitJ.El MasriY.CantrellK.RyanJ.BairdJ. (2017). Lessons learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research, 62(3), 333–353. https://doi.org/10.1080/00313831.2016.1258726
21.
HudsonN. W.AnusicI.LucasR. E.DonnellanM. B. (2017). Comparing the reliability and validity of global self-report measures of subjective well-being with experiential day reconstruction measures. Assessment, 27(1), 102–116. https://doi.org/10.1177/1073191117744660
22.
IBM. (2017). IBM SPSS statistics for Windows[Version 25; Statistical Software]. https://www.ibm.com/spss
23.
IEA. (2022). The international Database analyzer (IDB)[Version 5; Statistical Software]. https://www.iea.nl/
24.
JansenM.BodaZ.LorenzG. (2021). Social comparison effects on academic self-concepts – which peers matter most?Developmental Psychology, 58(8), 1541–1556. https://doi.org/10.1037/dev0001368
25.
MessickS. (1989). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13–103). Macmillan Publishing Co, Inc; American Council on Education. https://psycnet.apa.org/record/1989-97348-000
26.
MessickS. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037//0003-066x.50.9.741
27.
NiemivirtaM.TapolaA.TuominenH.ViljarantaJ. (2023). Developmental trajectories of school-beginners’ ability self-concept, intrinsic value and performance in mathematics. British Journal of Educational Psychology, 94(2), 441–459. https://doi.org/10.1111/bjep.12655
PassiatoreY.CostaS.GrossiG.CarrusG.PirchioS. (2023). Mathematics self-concept moderates the relation between cognitive functions and mathematical skills in primary school children. Social Psychology of Education, 27(3), 1143–1159. https://doi.org/10.1007/s11218-023-09854-3
32.
PerinelliE.PisanuF.ChecchiD.ScalasL. F.FraccaroliF. (2022). Academic self-concept change in junior high school students and relationships with academic achievement. Contemporary Educational Psychology, 69(102071), Article 102071. https://doi.org/10.1016/j.cedpsych.2022.102071
33.
RamazanO.DaiS.DanielsonR. W.ArdashevaY.HaoT.AustinB. W. (2023). Students’ 2018 PISA reading self-concept: Identifying predictors and examining model generalizability for emergent bilinguals. Journal of School Psychology, 101(1), Article 101254. https://doi.org/10.1016/j.jsp.2023.101254
34.
RamazanO.DanielsonR. W.RougeeA.ArdashevaY.AustinB. W. (2023). Effects of classroom and school climate on language minority students’ PISA mathematics self-concept and achievement scores. Large-scale Assessments in Education, 11(1), 11. https://doi.org/10.1186/s40536-023-00156-w
35.
RutkowskiL.GonzalezE.JoncasM.von DavierM. (2010). International large-scale assessment data: Issues in secondary analysis and reporting. Educational Researcher, 39(2), 142–151. https://doi.org/10.3102/0013189X10363170
36.
SewasewD.SchroedersU.SchieferI. M.WeirichS.ArteltC. (2018). Development of sex differences in math achievement, self-concept, and interest from grade 5 to 7. Contemporary Educational Psychology, 54(1), 55–65. https://doi.org/10.1016/j.cedpsych.2018.05.003
37.
SkaalvikE. M.ValåsH. (1999). Relations among achievement, self-concept, and motivation in mathematics and language arts: A longitudinal study. The Journal of Experimental Education, 67(2), 135–149. https://doi.org/10.1080/00220979909598349
38.
SongJ.HoweE.OltmannsJ. R.FisherA. J. (2023). Examining the concurrent and predictive validity of single items in ecological momentary assessments. Assessment, 30(5), 1662–1671. https://doi.org/10.1177/10731911221113563
39.
SteinbergO.KulakowS.RaufelderD. (2024). Academic self-concept, achievement, and goal orientations in different learning environments. European Journal of Psychology of Education, 39(4), 3893–3917. https://doi.org/10.1007/s10212-024-00825-6
ValdiviaD. S.LiP. F. (2021). Psychometric properties and internal validity of the multidimensional existential meaning scale. Measurement and Evaluation in Counseling and Development, 55(3), 166–186. https://doi.org/10.1080/07481756.2021.1998779
