This study employed item response theory (IRT), exploratory factor analysis (EFA), and confirmatory factor analysis (CFA) to examine the measurement invariance of TIMSS science self-concept (SSC) scales for grades 4 and 8 across genders, cultures, and languages in six countries and regions. IRT, and EFA and CFA were utilized to examine the reliability and validity of the SSC scales, respectively. The IRT results revealed substantial variation in each item and test information function values across the six societies. The EFA and CFA results identified two dimensions of SSC scales, that is, positive perception of SSC (SSCp) and negative perception of SSC (SSCn), with satisfactory fit indices. Measurement invariance was confirmed across genders, and cultural and linguistic groups. Recommendations are put forward for TIMSS program developers and researchers regarding the revision of the SSC scales.
Abu-HilalM. M.AbdelfattahF. A.AlshumraniS. A.AbduljabbarA. S.MarshH. W. (2013). Construct validity of self-concept in TIMSS’s student background questionnaire: A test of separation and conflation of cognitive and affective dimensions of self-concept among Saudi eighth graders. European Journal of Psychology of Education, 28(4), 1201–1220. https://doi.org/10.1007/s10212-012-0162-1
2.
American Educational Research Association (AERA). (2014). American psychological association (APA), and national council on measurement in education (NCME). In Standards for educational and psychological testing. American Psychological Association.
3.
ArensA. K.JansenM.PreckelF.SchmidtI.BrunnerM. (2021). The structure of academic self-concept: A methodological review and empirical illustration of central models. Review of Educational Research, 91(1), 34–72. https://doi.org/10.3102/0034654320972186
4.
BentlerP. M.BonettD. G. (1980). Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin, 88(3), 588–606. https://doi.org/10.1037/0033-2909.88.3.588
5.
BergerN.MackenzieE.HolmesK. (2020). Positive attitudes towards mathematics and science are mutually beneficial for student achievement: A latent profile analysis of TIMSS 2015. The Australian Educational Researcher, 47(3), 409–444. https://doi.org/10.1007/s13384-020-00379-8
6.
BochnerS. (1994). Cross-cultural differences in the self concept a test of Hofstede’s individualism collectivism distinction. Journal of Cross-Cultural Psychology, 25(2), 273–283. https://doi.org/10.1177/0022022194252
7.
BoyleG. J. (1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales?Personality and Individual Differences, 12(3), 291–294. https://doi.org/10.1016/0191-8869(91)90115-r
ChenF. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling: A Multidisciplinary Journal, 14(3), 464–504. https://doi.org/10.1080/10705510701301834
10.
ChiuM.-S. (2008). Achievements and self-concepts in a comparison of math and science: Exploring the internal/external frame of reference model across 28 countries. Educational Research and Evaluation, 14(3), 235–254. https://doi.org/10.1080/13803610802048858
11.
DongY.HuangJ. (2024). Validating the Chinese version of short‐form foreign language classroom anxiety scale. Psychology in the Schools, 61(8), 23216–23283. https://doi.org/10.1002/pits.23216
12.
EdelenM. O.ReeveB. B. (2007). Applying item response theory (IRT) modeling to questionnaire development, evaluation, and refinement. Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation, 16(Suppl 1), 5–18. https://doi.org/10.1007/s11136-007-9198-0
13.
EmbretsonS. E.ReiseS. P. (2013). Item response theory for psychologists. Psychology Press.
14.
GroblerA.MathafenaR. (2021). Assessing the dimensionality of three LMX instruments within a diverse cultural and linguistic context. Psihologija, 54(1), 33–48. https://doi.org/10.2298/PSI191011019G
15.
GuayF.RatelleC. F.RoyA.LitalienD. (2010). Academic self-concept, autonomous academic motivation, and academic achievement: Mediating and additive effects. Learning and Individual Differences, 20(6), 644–653. https://doi.org/10.1016/j.lindif.2010.08.001
16.
HairJ. F.BlackW. C.BabinB. J.AndersonR. E. (2019). Multivariate data analysis (Eighth edition). Cengage.
17.
HofstedeG. (2001). Culture’s consequences: Comparing values, behaviors, institutions, and organizations across nations (2nd ed.). Sage.
18.
HuL.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
19.
HuangJ.DongY.HanC.WangX. (2023a). Evaluating the language- and culture-related construct-irrelevant variance and reliability of the sense of school belonging scale: Suggestions for revision. Journal of Psychoeducational Assessment, 41(8), 07342829231189319–07342829231189871. https://doi.org/10.1177/07342829231189319
20.
HuangJ.ShuT.DongY.ZhuD. (2023). Constructing and validating a self-assessment scale for Chinese college English-major students’ feedback knowledge repertoire in EFL academic writing: Item response theory and factor analysis approaches. Assessing Writing, 56(13), Article 100716. https://doi.org/10.1016/j.asw.2023.100716
21.
KangJ. (2023). Moderating role of science self-concept in eliciting state curiosity when confronting a violation outcome. Journal of Baltic Science Education, 22(6), 1025–1037. https://doi.org/10.33225/jbse/23.22.1025
22.
KimM. H.SidneyP. G. (2024). Do teacher instructional practices shape children’s academic self‐concept and interest in mathematics and science? Evidence from TIMSS 2015. Infant and Child Development, 33(2), Article e2429. https://doi.org/10.1002/icd.2429
23.
KlineR. B. (2023). Principles and practice of structural equation modeling. Guilford Publications.
24.
LiouP.-Y.LinJ. J. H. (2021). Comparisons of science motivational beliefs of adolescents in Taiwan, Australia, and the United States: Assessing the measurement invariance across countries and genders. Frontiers in Psychology, 12, Article 674902. https://doi.org/10.3389/fpsyg.2021.674902
25.
LiouP.-Y.LinY.-M.HuangS.-C.ChenS. (2023). Gender differences in science motivational beliefs and their relations with achievement over grades 4 and 8: A multinational perspective. International Journal of Science and Mathematics Education, 21(1), 233–249. https://doi.org/10.1007/s10763-021-10243-5
26.
LiuS.MengL. (2010). Re-examining factor structure of the attitudinal items from TIMSS 2003 in cross-cultural study of mathematics self-concept. Educational Psychology, 30(6), 699–712. https://doi.org/10.1080/01443410.2010.501102
27.
MaY. (2022). Profiles of student science attitudes and its associations with gender and science achievement. International Journal of Science Education, 44(11), 1876–1895. https://doi.org/10.1080/09500693.2022.2101705
28.
MacCallumR. C.BrowneM. W.SugawaraH. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1(2), 130–149. https://doi.org/10.1037//1082-989x.1.2.130
29.
MarshH. W. (1987). The hierarchical structure of self-concept and the application of hierarchical confirmatory factor analysis. Journal of Educational Measurement, 24(1), 17–39. https://doi.org/10.1111/j.1745-3984.1987.tb00259.x
30.
MarshH. W. (1990). The structure of academic self-concept: The Marsh/Shavelson model. Journal of Educational Psychology, 82(4), 623–636. https://doi.org/10.1037/0022-0663.82.4.623
31.
MarshH. W.CraverR. G. (2006). Reciprocal effects of self-concept and performance from a multidimensional perspective: Beyond seductive pleasure and unidimensional perspectives. Perspectives on Psychological Science, 1(2), 133–163. https://doi.org/10.1111/j.1745-6916.2006.00010.x
32.
MarshH. W.PekrunR.MurayamaK.ArensA. K.ParkerP. D.GuoJ.DickeT. (2018). An integrated model of academic self-concept development: Academic self-concept, grades, test scores, and tracking over 6 years. Developmental Psychology, 54(2), 263–280. https://doi.org/10.1037/dev0000393
33.
MartinM. O.von DavierM.MullisI. V. S. (2020). Methods and procedures: TIMSS 2019 technical report. TIMSS & PIRLS International Study Center.
34.
McNeishD. (2018). Thanks coefficient alpha, we’ll take it from here. Psychological Methods, 23(3), 412–433. https://doi.org/10.1037/met0000144
35.
MöllerJ.RetelsdorfJ.KöllerO.MarshH. W. (2011). The reciprocal internal/external frame of reference model: An integration of models of relations between academic achievement and self-concept. American Educational Research Journal, 48(6), 1315–1346. https://doi.org/10.3102/0002831211419649
36.
MoronyS.KleitmanS.LeeY. P.StankovL. (2013). Predicting achievement: Confidence vs self-efficacy, anxiety, and self-concept in Confucian and European countries. International Journal of Educational Research, 58, 79–96. https://doi.org/10.1016/j.ijer.2012.11.002
37.
MullisI. V. S.MartinM. O.FoyP.KellyD. L.FishbeinB. (2020). TIMSS 2019 international results in mathematics and science. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://timssandpirls.bc.edu/timss2019/international-results/
38.
PinxtenM.WoutersS.PreckelF.NiepelC.De FraineB.VerschuerenK. (2015). The formation of academic self-concept in elementary education: A unifying model for external and internal comparisons. Contemporary Educational Psychology, 41, 124–132. https://doi.org/10.1016/j.cedpsych.2014.12.003
39.
PutnickD. L.BornsteinM. H. (2016). Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Developmental Review: DR, 41, 71–90. https://doi.org/10.1016/j.dr.2016.06.004
40.
ReiseS. P.RevickiD. A. (2014). Handbook of item response theory modeling. Taylor & Francis.
41.
ReynoldsK.KhorramdelL.von DavierM. (2022). Can students’ attitudes towards mathematics and science be compared across countries? Evidence from measurement invariance modeling inTIMSS2019. Studies in Educational Evaluation, 74, Article 101169. https://doi.org/10.1016/j.stueduc.2022.101169
42.
RutkowskiL.SvetinaD. (2014). Assessing the hypothesis of measurement invariance in the context of large-scale international surveys. Educational and Psychological Measurement, 74(1), 31–57. https://doi.org/10.1177/0013164413498257
43.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34(4, Pt. 2), 100–197. https://doi.org/10.1007/bf03372160
44.
SchroedersU.JansenM. (2022). Science self-concept - More than the sum of its parts?The Journal of Experimental Education, 90(2), 435–451. https://doi.org/10.1080/00220973.2020.1740967
45.
SchütteK. (2015). Science self-concept and valuing science: A cross-cultural analysis of their relation among students from Western and East Asian countries. Soc Psychol Educ, 18, 635–652. https://doi.org/10.1007/s11218-015-9311-0
46.
ShavelsonR. J.HubnerJ. J.StantonG. C. (1976). Self-concept: Validation of construct interpretations. Review of Educational Research, 46(3), 407–441. https://doi.org/10.3102/00346543046003407
47.
ShowersC. (1992). Compartmentalization of positive and negative self-knowledge: Keeping bad apples out of the bunch. Journal of Personality and Social Psychology, 62(6), 1036–1049. https://doi.org/10.1037//0022-3514.62.6.1036
48.
SkaalvikE. M.SkaalvikS. (2002). Internal and external frames of reference for academic self-concept. Educational Psychologist, 37(4), 233–244. https://doi.org/10.1207/S15326985EP3704_3
49.
TaskinenP. H.SchütteK.PrenzelM. (2013). Adolescents’ motivation to select an academic science-related career: The role of school factors, individual interest, and science self-concept. Educational Research and Evaluation, 19(8), 717–733. https://doi.org/10.1080/13803611.2013.853620
50.
TeoT.HuangF.HeJ. (2024). Measurement invariance and latent mean differences of the digital native assessment scale across Chinese mainland, Macau, and Taiwan: An exploratory structural equation modeling approach. Interactive Learning Environments, 32(5), 2064–2076. https://doi.org/10.1080/10494820.2022.2137528
51.
ThissenD. (1991). MULTILOG: Multiple category item analysis and test scoring using item response theory. [Computer software].Scientific Software International.
52.
WangM.-T.DegolJ. L. (2017). Gender gap in science, technology, engineering, and mathematics (STEM): Current knowledge, implications for practice, policy, and future directions. Educational Psychology Review, 29(1), 119–140. https://doi.org/10.1007/s10648-015-9355-x
53.
WilkinsJ. L. M. (2004). Mathematics and science self-concept: An international investigation. The Journal of Experimental Education, 72(4), 331–346. https://doi.org/10.3200/jexe.72.4.331-346
54.
WuX.WuQ. (2022). Can students be more engaged and confident? A multiple membership multilevel analysis of science engagement and confidence and their effects on science achievement in East Asia. Studies in Educational Evaluation, 73, Article 101147. https://doi.org/10.1016/j.stueduc.2022.101147
55.
YoshinoA. (2012). The relationship between self-concept and achievement in TIMSS 2007: A comparison between American and Japanese students. International Review of Education, 58(2), 199–219. https://doi.org/10.1007/s11159-012-9283-7
56.
ZhangF.BaeC. L.BrodaM. (2022). Science self-concept, relatedness, and teaching quality: A multilevel approach to examining factors that predict science achievement. International Journal of Science and Mathematics Education, 20(3), 503–529. https://doi.org/10.1007/s10763-021-10165-2
57.
ZhangL.LiuX.FengH. (2023). Development and validation of an instrument for assessing scientific literacy from junior to senior high school. Disciplinary and Interdisciplinary Science Education Research, 5(21), 21. https://doi.org/10.1186/s43031-023-00093-2
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
