Fixed Effects or Mixed Effects Classifiers? Evidence From Simulated and Archival Data

Abstract

This study seeks to compare fixed and mixed effects models for the purposes of predictive classification in the presence of multilevel data. The first part of the study utilizes a Monte Carlo simulation to compare fixed and mixed effects logistic regression and random forests. An applied examination of the prediction of student retention in the public-use U.S. PISA data set was considered to verify the simulation findings. Results of this study indicate fixed effects models performed comparably with mixed effects models across both the simulation and PISA examinations. Results broadly suggest that researchers should be cognizant of the type of predictors and data structure being used, as these factors carried more weight than did the model type.

Keywords

predictive classification mixed effects models Monte Carlo simulation Program for International Student Assessment

Get full access to this article

View all access options for this article.

References

Abu-Nimeh

Nappa

Wang

Nair

(2007). A comparison of machine learning techniques for phishing detection. In Proceedings of the Anti-Phishing Working Groups 2nd Annual eCrime Researchers Summit (eCrime ‘07) (pp. 60–69). Association for Computing Machinery. https://doi-org.proxy.bsu.edu/10.1145/1299015.1299021

Bauer

Kohavi

(1999). An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning, 36(1–2), 105–139.

Beleites

Neugebauer

Bocklitz

Krafft

Popp

(2012). Sample size planning for classification models. Analytica Chimica Acta, 760, 25–33. https://doi.org/10.1016/j.aca.2012.11.007

Bolin

Finch

(2014). Supervised classification in the presence of misclassified training data: A Monte Carlo simulation study in the three group case. Frontiers in Psychology, 5, Article 118. https://www.doi.org/10.3389/fpsyg.2014.00118

Breiman

(2001). Random forests. Machine Learning, 45(1), 5–32.

Capitaine

Genuer

Thiebaut

(2021). Random forests for high-dimensional longitudinal data. Statistical Methods in Medical Research, 30(1), 166–184. https://doi.org/10.1177/0962280220946080

Choi

Gil

Mediavilla

Valbuena

(2018). Predictors and effects of grade repetition. Revista De Economía Mundial, 48, 21–42.

Corman

(2003). The effects of state policies, individual characteristics, family characteristics, and neighbourhood characteristics on grade repetition in the United States. Economics of Education Review, 22(4), 409–420. https://doi.org/10.1016/S0272-7757(02)00070-5

Crane-Droesch

(2017). Semiparametric panel data models using neural networks. arXiv preprint arXiv, 1702.06512, https://arxiv.org/pdf/1702.06512.pdf

10.

Dietterich

T. G.

(2000). An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization. Machine Learning, 40(2), 139–157. https://doi.org/10.1023/A:1007607513941

11.

Downes

Carlin

J. B.

(2020). Multilevel regression and poststratification as a modeling approach for estimating population quantities in large population health studies: A simulation study. Biometrical Journal, 62(2), 479–491. https://doi.org/10.1002/bimj.201900023

12.

Eisemon

T. O

., United Nations Educational, Scientific, and Cultural Organization, Paris (France), & International Institute for Educational Planning. (1997). Reducing repetition: Issues and strategies (Fundamentals of Educational Planning Series, Number 55). International Institute for Educational Planning.

13.

Fawcett

(2006). An introduction to ROC analysis. Pattern Recognition Letters, 27(8), 861–874. https://doi.org/10.1016/j.patrec.2005.10.010

14.

Figueroa

R. L.

Zeng-Treitler

Kandula

Ngo

L. H.

(2012). Predicting sample size required for classification performance. BMC Medical Informatics and Decision Making, 12(1), 8. https://doi.org/10.1186/1472-6947-12-8

15.

Glick

Sahn

D. E.

(2010). Early academic performance, grade repetition, and school attainment in senegal: A panel data analysis. The World Bank Economic Review, 24(1), 93–120. https://doi.org/10.1093/wber/lhp023

16.

Hajjem

Bellavance

Larocque

(2014). Mixed-effects random forest for clustered data. Journal of Statistical Computation and Simulation, 84(6), 1313–1328. https://doi.org/10.1080/00949655.2012.741599

17.

Harrison

R. L.

(2010, January). Introduction to Monte Carlo simulation. AIP Conference Proceedings, 1204(1), 17–21.

18.

Hastie

Tibshirani

Friedman

(2009). The elements of statistical learning: Data mining, inference, and prediction. Springer.

19.

Ikeda

García

(2014). Grade repetition: A comparative study of academic and non-academic consequences. OECD Journal: Economic Studies, 2013(1), 269–315. https://doi.org/10.1787/eco_studies-2013-5k3w65mx3hnx

20.

James

Witten

Hastie

Tibshirani

(2013). An introduction to statistical learning (Vol. 112, p. 18). Springer.

21.

Karpievitch

Y. V.

Hill

E. G.

Leclerc

A. P.

Dabney

A. R.

Almeida

J. S.

(2009). An introspective comparison of random forest-based classifiers for the analysis of cluster-correlated data by way of RF. PLOS ONE, 4(9), Article e7087. https://doi.org/10.1371/journal.pone.0007087

22.

Kilham

Hartebrodt

Kändler

(2019). Generating tree-level harvest predictions from forest inventories with random forests. Forests, 10(1), 20. https://doi.org/10.3390/f10010020

23.

Kreft

I. G.

(1996). Are multilevel techniques necessary? An overview, including simulation studies [Unpublished manuscript]. California State University, Los Angeles.

24.

Lavery

M. R.

Acharya

Sivo

S. A.

(2019). Number of predictors and multicollinearity: What are their effects on error and bias in regression? Communications in Statistics-Simulation and Computation, 48(1), 27–38. https://doi.org/10.1080/03610918.2017.1371750

25.

Lee

Stankov

(2018). Non-cognitive predictors of academic achievement: Evidence from TIMSS and PISA. Learning and Individual Differences, 65, 50–64. https://doi.org/10.1016/j.lindif.2018.05.009

26.

Lei

P. W.

Koehly

L. M.

(2003). Linear discriminant analysis versus logistic regression: A comparison of classification errors in the two-group case. The Journal of Experimental Education, 72(1), 25–49. https://doi.org/10.1080/00220970309600878

27.

Little

R. J.

(2013). In praise of simplicity not mathematistry! Ten simple powerful ideas for the statistical scientist. Journal of the American Statistical Association, 108(502), 359–369. https://doi.org/10.1080/01621459.2013.787932

28.

Luke

D. A.

(2019). Multilevel modeling (Vol. 143). SAGE.

29.

Maas

C. J. M.

Hox

J. J.

(2005). Sufficient sample sizes for multilevel modeling. Methodology, 1(3), 86–92. https://doi.org/10.1027/1614-2241.1.3.86

30.

Mangino

A. A.

Finch

W. H.

(2021). Prediction with mixed effects models: A Monte Carlo simulation study. Educational and Psychological Measurement, 81(6), 1118–1142. https://doi.org/10.1177/0013164421992818

31.

Maroco

Silva

Rodrigues

Guerreiro

Santana

de Mendonça

(2011). Data mining methods in the prediction of dementia: A real-data comparison of the accuracy, sensitivity and specificity of linear discriminant analysis, logistic regression, neural networks, support vector machines, classification trees and random forests. BMC Research Notes, 4(1), 299–313. https://doi.org/10.1186/1756-0500-4-299

32.

McMahon

S. D.

Parnes

A. L.

Keys

C. B.

Viola

J. J.

(2008). School belonging among low-income urban youth with disabilities: Testing a theoretical model. Psychology in the Schools, 45(5), 387–401. https://doi.org/10.1002/pits.20304

33.

McNeish

Kelley

(2019). Fixed effects models versus mixed effects models for clustered data: Reviewing the approaches, disentangling the differences, and making recommendations. Psychological Methods, 24(1), 20–35. https://doi.org/10.1037/met0000182

34.

McNeish

Stapleton

L. M.

(2016). Modeling clustered data with very few clusters. Multivariate Behavioral Research, 51(4), 495–518. https://doi.org/10.1080/00273171.2016.1167008

35.

McNeish

Stapleton

L. M.

Silverman

R. D.

(2017). On the unnecessary ubiquity of hierarchical linear modeling. Psychological Methods, 22(1), 114–140. https://doi.org/10.1037/met0000078

36.

Muchlinski

Siroky

Kocher

(2016). Comparing random forest with logistic regression for predicting class-imbalanced civil war onset data. Political Analysis, 24(1), 87–103. https://doi.org/10.1093/pan/mpv024

37.

Murtaugh

P. A.

(2007). Simplicity and complexity in ecological data analysis. Ecology, 88(1), 56–62. https://doi.org/10.1890/0012-9658(2007)88[56:SACIED]2.0.CO;2

38.

Ngufor

(2019). Vira: Virtual Intelligent Robot Assistant (R Package Version 0.1). https://rdrr.io/github/nguforche/Vira/

39.

Ngufor

Van Houten

Caffo

B. S.

Shah

N. D.

McCoy

R. G.

(2019). Mixed effect machine learning: A framework for predicting longitudinal change in hemoglobin A1c. Journal of Biomedical Informatics, 89, 56–67. https://doi.org/10.1016/j.jbi.2018.09.001

40.

Paccagnella

(2011). Sample size and accuracy of estimates in multilevel models: New simulation results. Methodology, 7(3), 111–120. https://doi.org/10.1027/1614-2241/a000029

41.

Palvanov

Cho

Y. I.

(2018). Comparisons of deep learning algorithms for MNIST in real-time environment. International Journal of Fuzzy Logic and Intelligent Systems, 18(2), 126–134. https://doi.org/10.5391/IJFIS.2018.18.2.126

42.

Ramos

Franco-Pedroso

Lozano-Diez

Gonzalez-Rodriguez

Ramos

Gonzalez-Rodriguez

Franco-Pedroso

Lozano-Diez

(2018). Deconstructing cross-entropy for probabilistic binary classifiers. Entropy, 20(3), 208. https://doi.org/10.3390/e20030208

43.

Raudenbush

S. W.

Bryk

A. S.

(2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). SAGE.

44.

Raudys

S. J.

Jain

A. K.

(1991). Small sample size effects in statistical pattern recognition: Recommendations for practitioners. IEEE Transactions on Pattern Analysis and Machine Intelligence, 13(3), 252–264. https://doi.org/10.1109/34.75512

45.

R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org/

46.

Shah

A. D.

Bartlett

J. W.

Carpenter

Nicholas

Hemingway

(2014). Comparison of random forest and parametric imputation models for imputing missing data using MICE: A CALIBER study. American Journal of Epidemiology, 179(6), 764–774. https://doi.org/10.1093/aje/kwt312

47.

Speiser

J. L.

Wolf

B. J.

Chung

Karvellas

C. J.

Koch

D. G.

Durkalski

V. L.

(2019). BiMM forest: A random forest method for modeling clustered and longitudinal binary outcomes. Chemometrics and Intelligent Laboratory Systems, 185, 122–134. https://doi.org/10.1016/j.chemolab.2019.01.002

48.

Speiser

J. L.

Wolf

B. J.

Chung

Karvellas

C. J.

Koch

D. G.

Durkalski

V. L.

(2020). BiMM tree: A decision tree method for modeling clustered and longitudinal binary outcomes. Communications in Statistics-Simulation and Computation, 49(4), 1004–1023. https://doi.org/10.1080/03610918.2018.1490429

49.

Steyerberg

E. W.

(2019). Clinical prediction models. Springer.

50.

Strobl

Malley

Tutz

(2009). An introduction to recursive partitioning: Rationale, application, and characteristics of classification and regression trees, bagging, and random forests. Psychological Methods, 14(4), 323–348. https://doi.org/10.1037/a0016973

51.

VanDerHeyden

A. M.

(2013). Universal screening may not be for everyone: Using a threshold model as a smarter way to determine risk. School Psychology Review, 42(4), 402–414. https://www.doi.org/10.1080/02796015.2013.12087462

52.

Waljee

A. K.

Mukherjee

Singal

A. G.

Zhang

Warren

Balis

Marrero

Zhu

Higgins

P. D.

(2013). Comparison of imputation methods for missing laboratory data in medicine. BMJ Open, 3(8), Article e002847. https://doi.org/10.1136/bmjopen-2013-002847

53.

Westreich

Lessler

Funk

M. J.

(2010). Propensity score estimation: Machine learning and classification methods as alternatives to logistic regression. Journal of Clinical Epidemiology, 63(8), 826–833. https://doi.org/10.1016/j.jclinepi.2009.11.020

54.

Wößmann

(2003). Schooling resources, educational institutions and student performance: The international evidence. Oxford Bulletin of Economics and Statistics, 65(2), 117–170. https://doi.org/10.1111/1468-0084.00045

55.

Zhang

(2010). Handwritten digit classification using the MNIST data set (Course Project CSE802: Pattern Classification & Analysis). https://www.researchgate.net/profile/Ming-Wu-45/publication/228685853_Handwritten_Digit_Classification_using_the_MNIST_Data_Set/links/5409c76b0cf2f2b29a2c57b5/Handwritten-Digit-Classification-using-the-MNIST-Data-Set.pdf

56.

Yan

(2019). Anomaly detection in categorical data with interpretable machine learning: A random forest approach to classify imbalanced data [Thesis]. http://liu.diva-portal.org/smash/record.jsf?pid=diva2%3A1330907&dswid=6631

57.

Zellner

Keuzenkamp

H. A.

McAleer

(2001). Simplicity, inference and modelling: Keeping it sophisticatedly simple. Cambridge University Press.

58.

Zhang

J. L.

Haerdle

W. K.

(2010). The Bayesian additive classification tree applied to credit risk modelling. Computational Statistics & Data Analysis, 54(5), 1197–1205. https://doi.org/10.1016/j.csda.2009.11.022

59.

Zigler

Phillips

(1961). Psychiatric diagnosis: A critique. Journal of Abnormal and Social Psychology, 63(3), 607–618. https://doi.org/10.1037/h0040556