Making Meaning out of MANOVA: The Need for Multivariate Post Hoc Testing in Gifted Education Research

Abstract

Multivariate analysis of variance (MANOVA) is a statistical method used to examine group differences on multiple outcomes. This article reports results of a review of MANOVA in gifted education journals between 2011 and 2017 (N = 56). Findings suggest a number of conceptual and procedural misunderstandings about the nature of MANOVA and its application, including pervasive use of univariate post hoc tests to interpret MANOVA results. Accordingly, this article aims to make MANOVA more accessible to gifted education scholars by clarifying its purpose and introducing descriptive discriminant analysis as a more appropriate post hoc technique. A heuristic data set is used to demonstrate the procedures for running a descriptive discriminant analysis, both in place of a one-way MANOVA and as a post hoc analysis to a factorial design. SPSS and R syntax are provided.

Keywords

MANOVA discriminant analysis multivariate analysis

Get full access to this article

View all access options for this article.

References

American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.

Appelbaum

Cooper

Kline

R. B.

Mayo-Wilson

Nezu

A. M.

Rao

S. M.

(2018). Journal article reporting standards for quantitative research in psychology: The APA Publications and Communications Board task force report. American Psychologist, 73, 3-25. doi:10.1037/amp0000191

Barton

Yeatts

P. E.

Henson

R. K.

Martin

S. B.

(2016). Moving beyond univariate post-hoc testing in exercise science: A primer on descriptive discriminant analysis. Research Quarterly for Exercise and Sport, 87, 365-375. doi:1080/02701367.2016.1213352

Bathke

A. C.

Friedrich

Pauly

Konietschke

Staffen

Strobl

Höller

(2018). Testing mean differences among groups: Multivariate and repeated measures analysis with minimal assumptions. Multivariate Behavioral Research, 53, 348-359. doi:10.1080/00273171.2018.1446320

Benjamini

Hochberg

(1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B Methodological, 57, 289-300. doi:10.1111/j.2517-6161.1995.tb02031.x

Boedeker

Kearns

N. T.

(2019). Linear discriminant analysis for prediction of group membership: A user-friendly primer. Advances in Methods and Practices in Psychological Science, 2, 250-263.

Cai

T. T.

Liang

Zhou

H. H.

(2015). Law of log determinant of sample covariance matrix and optimal estimation of differential entropy for high-dimensional Gaussian distributions. Journal of Multivariate Analysis, 137, 161-172. doi:10.1016/j.jmva.2015.02.003

Chae

Gentry

(2011). Gifted and general high school students’ perceptions of learning and motivational constructs in Korea and the United States. High Ability Studies, 22, 103-118. doi:10.1080/13598139.2011.577275

Cohen

(1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-443. doi:10.1037/h0026714

10.

Conger

A. J.

(1974). A revised definition for suppressor variables: A guide to their identification and interpretation. Educational and Psychological Measurement, 34, 35-46.

11.

Courville

Thompson

(2001). Use of structure coefficients in published multiple regression articles: B is not enough. Educational and Psychological Measurement, 61, 229-248. doi: 10.1177/0013164401612006

12.

Enders

C. K.

(2003). Performing multivariate group comparisons following a statistically significant MANOVA. Measurement and Evaluation in Counseling and Development, 36, 40-56.

13.

Field

(2013). Discovering statistics using IBM SPSS Statistics (4th ed.). Thousand Oaks, CA: Sage.

14.

Finch

French

(2013). A Monte Carlo comparison of robust MANOVA test statistics. Journal of Modern Applied Statistical Methods, 12, 35-81. doi:10.22237/jmasm/1383278580

15.

Fish

L. J.

(1988). Why multivariate methods are usually vital. Measurement and Evaluation in Counseling and Development, 21, 130-137. doi:10.1080/07481756.1988.12022895

16.

Friedrich

Pauly

(2018). MATS: Inference for potentially singular and heteroscedastic MANOVA. Journal of Multivariate Analysis, 165, 166-179. doi:10.1016/j.jmva.2017.12.008

17.

Friendly

Sigal

(2017). Graphical methods for multivariate linear models in psychological research: An R tutorial. Quantitative Methods for Psychology, 13, 20-45. doi:10.20982/tqmp.13.1.p020

18.

Friendly

Sigal

(2018). Visualizing tests for equality of covariance matrices. The American Statistician. Advance online publication. doi:10.1080/00031305.2018.1497537

19.

Graham

J. M.

(2008). The general linear model as structural equation modeling. Journal of Educational and Behavioral Statistics, 33, 485-506.

20.

Grice

J. W.

Iwasaki

(2007). A truly multivariate approach to MANOVA. Applied Multivariate Research, 1, 199-226. Retrieved from https://amr.uwindsor.ca/ojs/leddy/index.php/AMR

21.

Henson

R. K.

(2002, April). The logic and interpretation of structure coefficients in multivariate general linear model analyses. Paper presented at the annual meeting of the American Educational Research Association, New Orleans. (ERIC Document Reproduction Service No. ED 467381)

22.

Henson

R. K.

(2006). Effect-size measures and meta-analytic thinking in counseling psychology research. The Counseling Psychologist, 34, 601-629. doi:10.1177/0011000005283558

23.

Huang

F. L.

(2018). Multilevel modeling and ordinary least squares regression: How comparable are they? Journal of Experimental Education, 86, 265-281. doi:10.1080/00220973.2016.1277339

24.

Huang

F. L.

(2020). MANOVA: A procedure whose time has passed? Gifted Child Quarterly, 64, 56-60.

25.

Huberty

C. J.

Hussein

M. H.

(2003). Some problems in reporting use of discriminant analyses. Journal of Experimental Education, 71, 177-192. doi:10.1080/00220970309602062

26.

Huberty

C. J.

Morris

J. D.

(1989). Multivariate analysis versus multiple univariate analyses. Psychological Bulletin, 105, 302-308. doi:10.1037/0033-2909.105.2.302

27.

Huberty

C. J.

Olejnik

(2006). Applied MANOVA and discriminant analysis. Hoboken, NJ: Wiley.

28.

Keselman

H. J.

Huberty

C. J.

Lix

L. M.

Olejnik

Cribbie

R. A.

Donahue

. . .Levin

J. R.

(1998). Statistical practices of educational researchers: An analysis of their ANOVA, MANOVA, and ANCOVA analyses. Review of Educational Research, 68, 350-386. Retrieved from http://www.jstor.org/stable/1170601

29.

Kieffer

K. M.

Reese

R. J.

Thompson

(2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: A methodological review. Journal of Experimental Education, 69, 280-309. doi:10.1080/00220970109599489

30.

Kim

(2019). The causal structure of suppressor variables. Journal of Educational and Behavioral Statistics, 44, 367-389. doi:10.3102/1076998619825679

31.

Knapp

T. R.

(1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410-416. doi:10.1037/0033-2909.85.2.410

32.

Konietschke

Bathke

A. C.

Harrar

S. W.

Pauly

(2015). Parametric and nonparametric bootstrap methods for general MANOVA. Journal of Multivariate Analysis, 140, 291-301. doi:10.1016/j.jmva.2015.05.001

33.

Kraha

Turner

Nimon

Zientek

L. R.

Henson

R. K.

(2012). Tools to support interpreting multiple regression in the face of multicollinearity. Frontiers in Psychology, 3, 44. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3303138/

34.

Leys

Delacre

Mora

Y. L.

Läkens

Ley

(2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1), 5. doi:10.5334/irsp.289

35.

Leys

Klein

Dominicy

Ley

(2018). Detecting multivariate outliers: Use a robust variant of the Mahalanobis distance. Journal of Experimental Social Psychology, 74, 150-156. doi:10.1016/j.jesp.2017.09.011

36.

Maxwell

(1992). Recent developments in MANOVA applications. In Thompson

(Ed.), Advances in social science methodology (Vol. 2, pp. 137-168). Greenwich, CT: JAI Press.

37.

McCoach

D. B.

(2010). Dealing with dependence (Part II): A gentle introduction to hierarchical linear modeling. Gifted Child Quarterly, 54, 252-256. doi:10.1177/0016986210373475

38.

McCoach

D. B.

Adelson

J. L.

(2010). Dealing with dependence (Part I): Understanding the effects of clustered data. Gifted Child Quarterly, 54, 152-155. doi:10.1177/0016986210363076

39.

McNeish

Stapleton

L. M.

Silverman

R. D.

(2017). On the unnecessary ubiquity of hierarchical linear modeling. Psychological Methods, 22, 114-140. doi:10.1037/met0000078

40.

Meehl

P. E.

(1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244. doi:10.2466/pr0.1990.66.1.195

41.

Nimon

K. F.

(2012). Statistical assumptions of substantive analyses across the general linear model: A mini-review. Frontiers in Psychology, 3. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3428812/

42.

Nimon

K. F.

(2015). Secondary data analyses from published descriptive statistics: Implications for human resource development theory, research, and practice. Advances in Developing Human Resources, 17, 26-39. doi:10.1177/1523422314559805

43.

Pyryt

M. C.

(2004). Pegnato revisited: Using discriminant analysis to identify gifted children. Psychology Science, 46(3), 342-347.

44.

Puryear

J. S.

Kettler

(2017). Rural gifted education and the effect of proximity. Gifted Child Quarterly, 61, 143-152. doi:10.1177/0016986217690229

45.

R Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org

46.

Sherry

(2006). Discriminant analysis in counseling psychology research. The Counseling Psychologist, 34, 661-683. doi:101177/0011000006287103

47.

Sherry

Henson

R. K.

(2005). Conducting and interpreting canonical correlation analysis in personality research: A user-friendly primer. Journal of Personality Assessment, 84, 37-48. doi:10.1207/s15327752jpa8401_09

48.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. doi:10.1177/0956797611417632

49.

Tabachnick

B. G.

Fidell

L. S.

(2019). Using multivariate statistics (7th ed.). Boston, MA: Pearson.

50.

Thompson

(1999, April). Common methodology mistakes in educational research, revisited, along with a primer on both effect sizes and the bootstrap. Paper presented at the annual meeting of the American Educational Research Association, Montreal, Canada. (ERIC Document Reproduction Service No. ED 429 110)

51.

Thompson

(2006). Foundations of behavioral statistics: An insight-based approach. New York, NY: Guilford Press.

52.

Thompson

(2015). The case for using the general linear model as a unifying conceptual framework for teaching statistics and psychometric theory. Journal of Methods and Measurement in the Social Sciences, 6(2), 30-41. doi:10.2458/v6i2.18801

53.

Tonidandel

LeBreton

J. M.

(2013). Beyond step-down analysis: A new test for decomposing the importance of dependent variables in MANOVA. Journal of Applied Psychology, 98, 469-477. doi:10.1037/a0032001

54.

Warne

R. T.

(2014). A primer on multivariate analysis of variance (MANOVA) for behavioral scientists. Practical Assessment, Research & Evaluation, 19(17), 1-10.

55.

Warne

R. T.

Lazo

Ramos

Ritter

(2012). Statistical methods used in gifted education journals, 2006-2010. Gifted Child Quarterly, 56, 134-149. doi:10.1177/0016986212444122

56.

Vallejo

Ato

(2012). Robust tests for multivariate factorial designs under heteroscedasticity. Behavior Research Methods, 44, 471-489. doi:10.22606/jas.2016.12006

57.

Zhang

J. T.

Zhou

Guo

Liu

(2016). A modified Bartlett test for heteroscedastic two-way MANOVA. Journal of Advanced Statistics, 1(2), 94-108. doi:10.22606/jas.2016.12006

58.

Zientek

L. R.

Thompson

(2009). Matrix summaries improve research reports: Secondary analyses using published literature. Educational Researcher, 38, 343-352. doi:10.3102/0013189X09339056

59.

Zimmermann

Pauly

Bathke

A. C.

(2019). Multivariate analysis of covariance when standard assumptions are violated. Retrieved from https://arxiv.org/abs/1902.10195v1