Challenging Conventional Wisdom for Multivariate Statistical Models With Small Samples

Abstract

In education research, small samples are common because of financial limitations, logistical challenges, or exploratory studies. With small samples, statistical principles on which researchers rely do not hold, leading to trust issues with model estimates and possible replication issues when scaling up. Researchers are generally aware of such pitfalls and often attempt to tailor their analyses to accommodate small samples. Despite well-intentioned efforts, conventional statistical axioms do not always translate to best practice with small samples, and common recommendations can, counterintuitively, exacerbate small sample problems. In this article, we overview the landscape of small sample techniques and note why conventionally recommended approaches can fail with small samples while also suggesting lesser known alternatives that tend to perform better in statistical research but are not widely adopted in education research. Topics include bootstrapping, latent variable model fit, and Bayesian methods for multilevel, latent variable, and growth models. Simulated and real-data examples are interspersed throughout.