Combining Statistical and Machine Learning Methods to Identify Predictors of Brazilian Students’ Proficiency in PISA 2018

Abstract

Brazil’s education system lags behind international standards, with two-fifths of students scoring below the minimum level of proficiency in mathematics, science, and reading. Thus, this study combined machine learning with traditional statistics to identify the most important predictors and to interpret their effects on proficiency in the PISA 2018 mathematics, science, and reading tests. Predictors encompassed a wide range of variables, sociodemographic characteristics, teaching and learning processes, and non-cognitive skills. The outcome of the present study was proficiency in mathematics, science, and reading. PISA proficiency levels were grouped into “low proficiency” and “proficient” categories, using a classification system commonly employed in PISA reports. Using random forest analysis, a machine learning method, I compared the importance of predictors for proficiency in mathematics, science, and reading. I then adjusted multilevel logistic regression analyses to investigate the relationship between the top predictors and the outcomes. Among the top predictors for the three outcomes identified, annual household income, parents’ highest occupational status, and early childhood education and care were positively associated with proficiency in mathematics, science, and reading, while grade repetition and additional instruction were negatively associated with these outcomes. These findings urge Brazilian policymakers and educators to prioritize initiatives that strengthen early childhood programs, minimize grade repetition, and promote effective learning strategies.

Keywords

programme for international student assessment international student assessment adolescence student achievement

Get full access to this article

View all access options for this article.

References

Afshartous

de Leeuw

(2005). Prediction in multilevel models. Journal of Educational and Behavioral Statistics, 30(2), 109–139. A Quarterly Publication Sponsored by the American Educational Research Association and the American Statistical Association. https://doi.org/10.3102/10769986030002109

Aksu

Güzeller

C. O.

(2016). Classification of PISA 2012 mathematical literacy scores using decision-tree method: Turkey sampling. Dialysis, Transplantation & Burn: Medical Journal of the Turkish Transplantation and Burn Foundation, 41(185), 101–121. https://doi.org/10.15390/eb.2016.4766

Areepattamannil

Santos

I. M.

(2019). Adolescent students’ perceived information and communication technology (ICT) competence and autonomy: Examining links to dispositions toward science in 42 countries. Computers in Human Behavior, 98(5), 50–58. https://doi.org/10.1016/j.chb.2019.04.005

Aronson

J. Z.

Zimmerman

Carlos

(1999). Improving student achievement by extending school: Is it just a matter of time? WestEd.

Barrios Fernández

(2023). Instruction time and educational outcomes. IZA World Labor. https://doi.org/10.15185/izawol.509

Becker

(2007). Early Childhood Education in Brazil: The obstacles to a successful experience. Revista de La Sede Academica de Mexico de La Facultad Latinoamericana de Ciencias Sociales, 5(2), 515–537.

Belsky

Vandell

D. L.

Burchinal

Clarke-Stewart

K. A.

McCartney

Owen

M. T.

NICHD Early Child Care Research Network . (2007). Are there long-term effects of early child care? Child Development, 78(2), 681–701. https://doi.org/10.1111/j.1467-8624.2007.01021.x

Breiman

(2001a). Random forests. Machine Learning, 45(1), 5–32. https://doi.org/10.1023/A:1010933404324

Breiman

(2001b). The Two Cultures (with comments and a rejoinder by the author). Statistical Science, 16(3), 199–231. https://doi.org/10.1214/ss/1009213726

10.

Bzdok

Altman

Krzywinski

(2018). Statistics versus machine learning. Nature Methods, 15(4), 233–234. https://doi.org/10.1038/nmeth.4642

11.

Cheung

K.-C.

Sit

P.-S.

Soh

K.-C.

Ieong

M.-K.

Mak

S.-K.

(2014). Predicting academic resilience with reading engagement and demographic variables: Comparing Shanghai, Hong Kong, Korea, and Singapore from the PISA perspective. The Asia-Pacific Education Researcher, 23(4), 895–909. https://doi.org/10.1007/s40299-013-0143-4

12.

Couronné

Probst

Boulesteix

A.-L.

(2018). Random forest versus logistic regression: A large-scale benchmark experiment. BMC Bioinformatics, 19(1), 270. https://doi.org/10.1186/s12859-018-2264-5

13.

Courtney

Karakus

Ersozlu

Nurumov

(2022). The influence of ICT use and related attitudes on students’ math and science performance: Multilevel analyses of the last decade’s PISA surveys. Large-Scale Assessments in Education, 10(1), 1–26. https://doi.org/10.1186/s40536-022-00128-6

14.

Fischbach

Keller

Preckel

Brunner

(2013). PISA proficiency scores predict educational outcomes. Learning and Individual Differences, 24(6), 63–72. https://doi.org/10.1016/j.lindif.2012.10.012

15.

Friedman

J. H.

(1997). Data mining and statistics What’s the connection? Computing science and statistics: Mining and modeling massive data sets in science, engineering, and business [Symposium]. In 29th Symposium on the Interface.

16.

Gamazo

Martínez-Abad

(2020). An exploration of factors linked to academic performance in PISA 2018 through data mining techniques. Frontiers in Psychology, 11(1), 575167. https://doi.org/10.3389/fpsyg.2020.575167

17.

Gamazo

Martínez-Abad

Olmos-Migueláñez

Jose Rodriguez-Conde

(2018). Assessment of factors related to school effectiveness in PISA 2015. A multilevel analysis. Revista de La Sociedad Española de Farmacia Hospitalaria, 379(3), 56–78. https://doi.org/10.4438/1988-592X-RE-2017-379-369

18.

Gómez-Fernández

Mediavilla

(2021). Exploring the relationship between information and communication technologies (ICT) and academic performance: A multilevel analysis for Spain. Socio-Economic Planning Sciences, 77(5), 101009. https://doi.org/10.1016/j.seps.2021.101009

19.

Goos

Pipa

Peixoto

(2021). Effectiveness of grade retention: A systematic review and meta-analysis. Educational Research Review, 34(3), 100401. https://doi.org/10.1016/j.edurev.2021.100401

20.

Gordon

M. S.

Cui

(2012). The effect of school-specific parenting processes on academic achievement in adolescence and young adulthood. Family Relations, 61(5), 728–741. https://doi.org/10.1111/j.1741-3729.2012.00733.x

21.

Gottfredson

D. C.

Fink

C. M.

Graham

(1994). Grade retention and problem behavior. American Educational Research Journal, 31(4), 761–784. https://doi.org/10.3102/00028312031004761

22.

Guo

Zhang

Shi

Yang

(2015). Predicting students performance in educational data mining. In 2015 international symposium on educational Technology (ISET) (pp. 125–128). https://doi.org/10.1109/ISET.2015.33

23.

Herawati

Nisa

Setiawan

Nusyirwan

Tiryono

(2018). Regularized multiple regression methods to deal with severe multicollinearity. International Journal of Statistics and Applications, 8(4), 167–172. https://doi.org/10.5923/j.statistics.20180804.02

24.

Hindman

(2015). Building better models: Prediction, replication, and machine learning in the social sciences. The Annals of the American Academy of Political and Social Science, 659(1), 48–62. https://doi.org/10.1177/0002716215570279

25.

IBGE . (2022). Synthesis of social indicators: An analysis of the living conditions of the Brazilian population. IBGE.

26.

Immekus

J. C.

Jeong

T.-S.

Yoo

J. E.

(2022). Machine learning procedures for predictor variable selection for schoolwork-related anxiety: Evidence from PISA 2015 mathematics, reading, and science assessments. Large-Scale Assessments in Education, 10(1), 30. https://doi.org/10.1186/s40536-022-00150-8

27.

Janitza

Hornung

(2018). On the overestimation of random forest’s out-of-bag error. PLoS One, 13(8), Article e0201904. https://doi.org/10.1371/journal.pone.0201904

28.

Karakolidis

Pitsia

Emvalotis

(2016). Examining students’ achievement in mathematics: A multilevel analysis of the Programme for international student assessment (PISA) 2012 data for Greece. International Journal of Educational Research, 79(5), 106–115. https://doi.org/10.1016/j.ijer.2016.05.013

29.

Kell

(2014). Global testing: PISA, TIMSS and PIRLS. In Kell

Kell

(Eds.), Literacy and language in East Asia: Shifting meanings, values and approaches (pp. 33–49): Springer. https://doi.org/10.1007/978-981-4451-30-7_5

30.

Kidron

Lindsay

(2014). The effects of increased learning time on student academic and nonacademic outcomes: Findings from a meta-analytic review. Institute of Education Sciences.

31.

Kim

Sherraden

Clancy

(2013). Do mothers’ educational expectations differ by race and ethnicity, or socioeconomic status? Economics of Education Review, 33(2), 82–94. https://doi.org/10.1016/j.econedurev.2012.09.007

32.

Knighton

(2006). Educational outcomes at age 19 associated with reading ability at age 15. Statistics Canada.

33.

Kuger

Klieme

Jude

Kaplan

(2016). Assessing contexts of learning: An international perspective. Springer International Publishing. https://doi.org/10.1007/978-3-319-45357-6

34.

Lazarevic

L. B.

Orlic

(2018). PISA 2012 mathematics literacy in Serbia: A multilevel analysis of students and schools. Psihologija, 51(4), 413–432. https://doi.org/10.2298/psi170817017l

35.

Lépine

(2015). School reputation and school choice in Brazil: A regression discontinuity design (No. 38). Department of Economics- FEA/USP.

36.

Lezhnina

Kismihók

(2022). Combining statistical and machine learning methods to explore German students’ attitudes towards ICT in PISA. International Journal of Research and Method in Education, 45(2), 180–199. https://doi.org/10.1080/1743727X.2021.1963226

37.

Mangino

A. A.

Finch

W. H.

(2021). Prediction with mixed effects models: A Monte Carlo simulation study. Educational and Psychological Measurement, 81(6), 1118–1142. https://doi.org/10.1177/0013164421992818

38.

Martínez-Abad

Gamazo

Rodríguez-Conde

M.-J.

(2020). Educational Data Mining: Identification of factors associated with school effectiveness in PISA assessment. Studies In Educational Evaluation, 66(3), 100875. https://doi.org/10.1016/j.stueduc.2020.100875

39.

Martínez García

J. S.

Oinonen

Merino

Perosa

(2021). Education and inequality in Finland, Spain and Brazil. In Towards a comparative analysis of social inequalities between Europe and Latin America. https://doi.org/10.1007/978-3-030-48442-2_4

40.

Mostafa

Silva

(2007). Brazil’s single registry experience: A tool for pro-poor social policies. International Labour Organization.

41.

Nasteski

(2017). An overview of the supervised machine learning methods. Horizons B, 4, 51–62. https://doi.org/10.20544/HORIZONS.B.04.1.17.P05

42.

OECD . (2007). PISA 2006 science competencies for tomorrow’s world: Volume 1: Analysis: Science competencies for tomorrow's world: Volume 1: Analysis. OECD Publishing. https://doi.org/10.1787/9789264040014-en

43.

OECD . (2019a). PISA 2018 assessment and analytical framework. OECD Publishing. https://doi.org/10.1787/b25efab8-en

44.

OECD . (2019b). PISA 2018 results (Volume I) what students know and can do. OECD Publishing. https://doi.org/10.1787/5f07c754-en

45.

OECD . (2019c). PISA 2018 results (Volume II) where all students can succeed: Where all students can succeed. OECD Publishing. https://doi.org/10.1787/b5fd1b8f-en

46.

OECD . (2020). PISA 2018 results (Volume V) effective policies, successful schools. OECD Publishing. https://doi.org/10.1787/ca768d40-en

47.

Oliveira

A. S.

(2016). Progressão continuada e outros dispositivos escolares: êxito e fracasso escolar nos anos iniciais do ensino fundamental. [Universidade de São Paulo]. https://doi.org/10.11606/t.48.2014.tde-17122014-094918

48.

Pedregosa

Varoquaux

Gramfort

Michel

Thirion

Grisel

Blondel

Müller

Nothman

Louppe

Prettenhofer

Weiss

Dubourg

Vanderplas

Passos

Cournapeau

Brucher

Perrot

Duchesnay

É.

(2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85), 2825–2830. arXiv. http://arxiv.org/abs/1201.0490

49.

Peng

Wang

(2023). Examining ICT attitudes, use and support in blended learning settings for students’ reading performance: Approaches of artificial intelligence and multilevel model. Computers & Education, 203(12), 104846. https://doi.org/10.1016/j.compedu.2023.104846

50.

Perosa

G. S.

Dantas

A. S. R.

(2017). The choice of private schools in families from low-income groups. Educação e Pesquisa, 43(4). https://doi.org/10.1590/S1517-9702201704177976

51.

Pholphirul

(2017). Pre-primary education and long-term education performance: Evidence from Programme for international student assessment (PISA) Thailand. Journal of Early Childhood Research, 15(4), 410–432. https://doi.org/10.1177/1476718X15616834

52.

Romero

Ventura

(2020). Educational data mining and learning analytics: An updated survey. Wiley Interdisciplinary Reviews. Data Mining and Knowledge Discovery, 10(3). https://doi.org/10.1002/widm.1355

53.

Royal Society . (2017). Machine learning: The power and Promise of computers that learn by example. Royal Society.

54.

Ryan

R. M.

Deci

E. L.

(2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford Publications.

55.

Ryo

Rillig

M. C.

(2017). Statistically reinforced machine learning for nonlinear patterns and variable interactions. Ecosphere, 8(11), Article e01976. https://doi.org/10.1002/ecs2.1976

56.

Schleicher

(2007). Can competencies assessed by PISA be considered the fundamental school knowledge 15-year-olds should possess? Journal of Educational Change, 8(4), 349–357. https://doi.org/10.1007/s10833-007-9042-x

57.

She

H.-C.

Lin

H.-S.

Huang

L.-Y.

(2019). Reflections on and implications of the Programme for International Student Assessment 2015 (PISA 2015) performance of students in Taiwan: The role of epistemic beliefs about science in scientific literacy. Journal of Research in Science Teaching, 56(10), 1309–1340. https://doi.org/10.1002/tea.21553

58.

Strobl

Boulesteix

A.-L.

Zeileis

Hothorn

(2007). Bias in random forest variable importance measures: Illustrations, sources and a solution. BMC Bioinformatics, 8(1), 25. https://doi.org/10.1186/1471-2105-8-25

59.

Strobl

Malley

Tutz

(2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14(4), 323–348. https://doi.org/10.1037/a0016973

60.

Takayama

(2008). The politics of international league tables: PISA in Japan’s achievement crisis debate. Comparative Education Review, 44(4), 387–407. https://doi.org/10.1080/03050060802481413

61.

Touron

Lopez-Gonzalez

Lizasoain Hernandez

Garcia San Pedro

M. J.

Navarro Asencio

(2018). Spanish high and low achievers in science in PISA 2015: Impact analysis of some contextual variables. Revista de La Sociedad Española de Farmacia Hospitalaria, 380(2), 156–184. https://doi.org/10.4438/1988-592X-RE-2017-380-376

62.

Unesco . (1997). International standard classification of education: Isced 1997. Unesco.

63.

Wagner

Merlo

(2014). Discriminatory accuracy of a random effect in multilevel logistic regression. International Journal of Epidemiology, 44(5). https://doi.org/10.1093/IJE/DYV097.190

64.

Wang

X. S.

Perry

L. B.

Malpique

Ide

(2023). Factors predicting mathematics achievement in PISA: A systematic review. Large-Scale Assessments in Education, 11(1), 24. https://doi.org/10.1186/s40536-023-00174-8

65.

Williamson

B. D.

Gilbert

P. B.

Carone

Simon

(2021). Nonparametric variable importance assessment using machine learning techniques. Biometrics, 77(1), 9–22. https://doi.org/10.1111/biom.13392

66.

Wright

M. N.

König

I. R.

(2019). Splitting on categorical predictors in random forests. PeerJ, 7(1), Article e6339. https://doi.org/10.7717/peerj.6339

67.

Yarkoni

Westfall

(2017). Choosing prediction over explanation in psychology: Lessons from machine learning. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 12(6), 1100–1122. https://doi.org/10.1177/1745691617693393

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.25 MB