Sage Journals: Discover world-class research

Abstract

This article supplements a review covering all (quasi)experimental studies of second language (L2) learning that appeared in Language Teaching Research (LTR) during the 19-year period 1997–2015. The dual focus of that review is practices in inferential statistics and the reporting of results. The present article sets out guidelines and recommendations reflecting findings reported in that review; and it includes additional discussion of topics of especial relevance to the analysis of data from L2 (quasi)experimental studies; for example, statistical power, effect sizes and confidence intervals for effect sizes, t-tests, the Wilcoxon–Mann–Whitney test, anova, contrast analysis in the context of anova, and control of the familywise Type I error rate. Discussed as well are two modern robust approaches to data analysis (i.e. randomization tests and bootstraps); ceiling and floor effects in pretest–posttest studies; and the ‘language-as-fixed-effect fallacy’.

Keywords

ceiling and floor effects confidence intervals L2 quantitative research language-as-fixed-effect fallacy reporting statistics robust statistical procedures statistical power Wilcoxon–Mann–Whitney test

Get full access to this article

View all access options for this article.

References

Abdi

Williams

(2010). Contrast analysis. In: Salkind

(Ed.), Encyclopedia of research design. Thousand Oaks, CA: Sage.

Abelson

(1995). Statistics as principled argument. New York: Psychology Press.

Allison

(1999). Multiple regression: A primer. Thousand Oaks, CA: Pine Forge Press.

American Psychological Association (APA). (2010). Publication manual of the American Psychological Association. Washington, DC: American Psychological Association.

Baayen

(2008). Analyzing linguistic data. Cambridge: Cambridge University Press.

Baguley

(2010). When correlations go bad. The Psychologist, 23, 122–123. Available at: https://thepsychologist.bps.org.uk/volume-23/edition-2/methods-when-correlations-go-bad (accessed May 2016).

Baguley

(2012). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave Macmillan.

Bakker

Wicherts

(2014). Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods, 19, 409–427.

Bird

Hadzi-Pavlovic

(2014). Controlling the maximum familywise Type I error rate in analyses of multivariate experiments. Psychological Methods, 19, 265–280.

10.

Bonate

(2000). Analysis of pretest–posttest designs. Boca Raton, FL: Chapman and Hall/CRC.

11.

Clark

(1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335–359.

12.

Cohen

(1992). A power primer. Psychological Bulletin, 112, 155–159.

13.

Cohen

(1994). The Earth is round, p < .05. American Psychologist, 49, 997–1003. Available at: http://en.wikipedia.org/wiki/Jacob_Cohen_(statistician) (accessed May 2016).

14.

Cribbie

Jamieson

(2004). Decreases in posttest variance and the measurement of change. Methods of Psychological Research Online, 9, 37–55. Available at: http://www.dgps.de/fachgruppen/methoden/mpr-online (accessed May 2016).

15.

Cumming

(2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Hove: Routledge.

16.

Cumming

(2014). The new statistics: Why and how. Psychological Science, 25, 7–29. Available at: http://pss.sagepub.com/content/25/1/7.full.pdf+html (accessed May 2016).

17.

de Winter

(2013). Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research & Evaluation, 18. Available online: http://pareonline.net/getvn.asp?v=18&n=10 (accessed May 2016).

18.

Dunlap

Cortina

Vaslow

Burke

(1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170−177.

19.

Ellis

P.D.

(2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge: Cambridge University Press.

20.

Erceg-Hurn

Mirosevich

(2008). American Psychologist, 63, 591–601. Available at: http://www.unt.edu/rss/class/mike/5700/articles/robustAmerPsyc.pdf (accessed May 2016).

21.

Fagerland

Sandvik

(2009).The Wilcoxon–Mann–Whitney Test under scrutiny. Statistics in Medicine, 28, 1487–1497.

22.

Faul

Erdfelder

Lang

A.-G.

Buchner

(2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.

23.

Fay

Proschan

(2010). Wilcoxon–Mann–Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Stat Survey, 4, 1–39. Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2857732 (accessed May 2016).

24.

Field

(2005). Intraclass correlation. In: Everitt

Howell

(Eds), Encyclopedia of statistics in behavioral science: Volume 2 (pp. 948–954). Chichester: Wiley.

25.

Field

Miles

Field

(2012). Discovering statistics using R. Thousand Oaks, CA: Sage.

26.

Field

Wright

(2006). PsyPAG Quarterly, 58, 9–23. Available at: http://www.statisticshell.com/docs/psypageffectsize.pdf (accessed May 2016).

27.

Games

(1971). Multiple comparisons of means. American Educational Research Journal, 8, 1–565.

28.

Grissom

Kim

(2012). Effect sizes for research: Univariate and multivariate applications. 2nd edition. Hove: Routledge.

29.

Hart

(2001). Mann–Whitney test is not just a test of medians: differences in spread can be important. British Medical Journal, 323, 391–393. Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1120984/pdf/391.pdf (accessed May 2016).

30.

Hesterberg

Moore

Monaghan

Clipson

Epstein

(2014). Bootstrap methods and permutation tests. In: Moore

McCabe

Craig

(Eds) Introduction to the practice of statistics (pp.16–1 to 16–57). 7th edition. Chapter 16. New York: WH Freeman. Available at: http://www.timhesterberg.net/bootstrap (accessed May 2016).

31.

Howell

(2010). Statistical methods for psychology. 7th edition. Belmont, CA: Cengage Wadsworth.

32.

Howell

(2015). Randomization test on two medians. Unpublished paper. Available at: https://www.uvm.edu/~dhowell/StatPages (accessed May 2016).

33.

Hsu

(1996). Multiple comparisons: Theory and methods. Boca Raton, FL: Chapman and Hall/CRC.

34.

Hudson

Llosa

(2015). Design issues and inference in experimental L2 research. Language Learning, 65, Supplement 1, 76–96.

35.

Jaccard

(1998). Interaction effects in factorial analysis of variance. Thousand Oaks, CA: Sage.

36.

Kelly

Preacher

(2012). On effect size. Psychological Science, 17, 137–152. Available at: http://www.quantpsy.org/pubs/kelley_preacher_2012.pdf (accessed May 2016).

37.

Keselman

Algina

Kowalchuk

(2001). The analysis of repeated measures designs: A review. British Journal of Mathematical and Statistical Psychology, 54, 1–20.

38.

Keselman

Cribbie

Holland

(2004). Pairwise multiple comparison test procedures: An update for clinical child and adolescent psychologists. Journal of Clinical Child and Adolescent Psychology, 33, 623–645. Available at: http://home.cc.umanitoba.ca/~kesel/jccap-4.pdf (accessed May 2016).

39.

Keselman

Othman

Wilcox

(2013). Preliminary testing for normality: Is this a good practice? Journal of Modern Applied Statistical Methods, 12, 2–19. Available at: http://digitalcommons.wayne.edu/cgi/viewcontent.cgi?article=1770&context=jmasm (accessed May 2016).

40.

Keselman

Othman

Wilcox

(2014). Testing for normality in the multi-group problem: Is this a good practice? Clinical Dermatology, 2, 29–43.

41.

Keselman

Othman

Wilcox

(2015). Assessing normality: Applications in multi-group designs. Malaysian Journal of Mathematical Sciences, 9, 53–65. Available at: http://einspem.upm.edu.my/journal/fullpaper/vol9/4.%20abdulrahman%20othman.pdf (accessed May 2016).

42.

Kimmel

(1957). Three criteria for the use of a one-tailed test. Psychological Bulletin, 54, 351–353.

43.

Lakens

(2015) Always use Welch’s t-test instead of Student’s t-test. The 20% Statistician [Weblog]. Available at: http://daniellakens.blogspot.co.uk/2015/01/always-use-welchs-t-test-instead-of.html (accessed May 2016).

44.

Larson-Hall

Plonsky

(2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65, 127–159.

45.

Lindstromberg

(2016). Inferential statistics in Language Teaching Research: A review and ways forward. Language Teaching Research. DOI: 10.1177/1362168816649979.

46.

Linck

Cunnings

(2015). The utility and application of mixed-effects models in second language research. Language Learning, 65, 185–207.

47.

Maxwell

Delaney

(2004). Designing experiments and analyzing data: A model comparison perspective. New York: Psychology Press.

48.

Micceri

(1989). The unicorn, the normal curve, and other improbable creatures. Psychological Bulletin, 105, 156–166. Available at: http://isites.harvard.edu/fs/docs/icb.topic988008.files/micceri89.pdf (accessed May 2016).

49.

Miller

Chapman

(2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110, 40–48.

50.

Morris

DeShon

(2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105–125.

51.

Norris

(2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65, 97–126.

52.

O’Keefe

(2007). Post hoc power, observed power, a priori power, retrospective power, achieved power: Sorting out appropriate uses of statistical power analyses. Communication Methods and Measures, 1, 291–299. Available at: http://www.dokeefe.net/pub/okeefe07cmm-posthoc.pdf (accessed May 2016).

53.

Plonsky

(2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687.

54.

Plonsky

Gass

(2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325–366.

55.

R Core Team. (2016). R: A language and environment for statistical computing [computer software]. Vienna: R Foundation for Statistical Computing. Available at: http://www.R-project.org (accessed May 2016).

56.

Rosenthal

Rosnow

(1985). Contrast analysis: Focused comparisons in the analysis of variance. Cambridge: Cambridge University Press.

57.

Ruxton

(2006). The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behavioral Ecology, 17, 688–690. Available at: http://beheco.oxfordjournals.org/content/17/4/688.full.pdf+html (accessed May 2016).

58.

Schaffer

(1995). Multiple hypothesis testing. Annual Review of Psychology, 46, 561–584.

59.

Wilcox

(1995). ANOVA: A paradigm for low power and misleading measures of effect size? Review of Educational Research, 65, 51–77.

60.

Wilcox

(1998). How many discoveries have been lost by ignoring modern statistical methods? American Psychologist, 53, 300–314. Available at: http://psych.colorado.edu/~willcutt/pdfs/Wilcox_1998.pdf (accessed May 2016).

61.

Wilcox

(2005). Introduction to robust estimation and hypothesis testing. 2nd edition. Amsterdam: Elsevier.

62.

Wilcox

(2010). Fundamentals of modern statistical methods. 2nd edition. Heidelberg: Springer.

63.

Wilcox

(2012). Modern statistics for the social and behavioral sciences: A practical introduction. Boca Raton, FL: CRC.

64.

Wilkinson

& Task Force on Statistical Inference. (1999.) Statistical methods in psychological journals: Guidelines and explanations. American Psychologist, 54, 594–604.

65.

Williams

Zimmerman

(1996). Are simple gain scores obsolete? Applied Psychological Measurement, 20, 59–69.

66.

Wright

(2006). Comparing groups in a before-after design: When t test and ANCOVA produce different results. British Journal of Educational Psychology, 76, 663–675.

67.

Wuensch

(2009). An overview of power analysis. Greenville, NC: East Carolina University. Available at: http://core.ecu.edu/psyc/wuenschk/StatHelp/PowerAnalysis_Overview.pdf (accessed May 2016).

68.

Zimmerman

(1998). Invalidation of parametric and nonparametric statistical tests by concurrent violation of two assumptions. Journal of Experimental Education, 67, 55–68.

69.

Zimmerman

(2003). A warning about the large-sample Wilcoxon–Mann–Whitney test. Understanding Statistics, 2, 267–280.

70.

Zimmerman

(2004a). Inflation of Type I error rates by unequal variances associated with parametric, nonparametric, and rank-transformation tests. Psicológica, 25, 103–133. Available at: http://www.uv.es/revispsi/articulos1.04/6-zimmerman.pdf (accessed May 2016).

71.

Zimmerman

(2004b). A note on preliminary tests of equality of variances. British Journal of Mathematical and Statistical Psychology, 57, 173–181.

72.

Zimmerman

D.W.

(2011). A simple and effective decision rule for choosing a significance test to protect against non-normality. British Journal of Mathematical and Statistical Psychology, 64, 388–409.

73.

Zimmerman

Zumbo

(2009). Hazards in choosing between pooled and separate-variances t tests. Psicológica, 30, 371–390.

Guidelines,recommendations,and supplementary discussion

Abstract

Keywords

Get full access to this article

References