This article supplements a review covering all (quasi)experimental studies of second language (L2) learning that appeared in Language Teaching Research (LTR) during the 19-year period 1997–2015. The dual focus of that review is practices in inferential statistics and the reporting of results. The present article sets out guidelines and recommendations reflecting findings reported in that review; and it includes additional discussion of topics of especial relevance to the analysis of data from L2 (quasi)experimental studies; for example, statistical power, effect sizes and confidence intervals for effect sizes, t-tests, the Wilcoxon–Mann–Whitney test, anova, contrast analysis in the context of anova, and control of the familywise Type I error rate. Discussed as well are two modern robust approaches to data analysis (i.e. randomization tests and bootstraps); ceiling and floor effects in pretest–posttest studies; and the ‘language-as-fixed-effect fallacy’.
AbdiH.WilliamsL. (2010). Contrast analysis. In: SalkindN. (Ed.), Encyclopedia of research design. Thousand Oaks, CA: Sage.
2.
AbelsonR. (1995). Statistics as principled argument. New York: Psychology Press.
3.
AllisonP. (1999). Multiple regression: A primer. Thousand Oaks, CA: Pine Forge Press.
4.
American Psychological Association (APA). (2010). Publication manual of the American Psychological Association. Washington, DC: American Psychological Association.
5.
BaayenH. (2008). Analyzing linguistic data. Cambridge: Cambridge University Press.
BaguleyT. (2012). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave Macmillan.
8.
BakkerM.WichertsJ. (2014). Outlier removal, sum scores, and the inflation of the type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods, 19, 409–427.
9.
BirdK.Hadzi-PavlovicD. (2014). Controlling the maximum familywise Type I error rate in analyses of multivariate experiments. Psychological Methods, 19, 265–280.
10.
BonateP. (2000). Analysis of pretest–posttest designs. Boca Raton, FL: Chapman and Hall/CRC.
11.
ClarkH. (1973). The language-as-fixed-effect fallacy: A critique of language statistics in psychological research. Journal of Verbal Learning and Verbal Behavior, 12, 335–359.
12.
CohenJ. (1992). A power primer. Psychological Bulletin, 112, 155–159.
CribbieR.JamiesonJ. (2004). Decreases in posttest variance and the measurement of change. Methods of Psychological Research Online, 9, 37–55. Available at: http://www.dgps.de/fachgruppen/methoden/mpr-online (accessed May 2016).
15.
CummingG. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Hove: Routledge.
de WinterJ. (2013). Using the Student’s t-test with extremely small sample sizes. Practical Assessment, Research & Evaluation, 18. Available online: http://pareonline.net/getvn.asp?v=18&n=10 (accessed May 2016).
18.
DunlapW.CortinaJ.VaslowJ.BurkeM. (1996). Meta-analysis of experiments with matched groups or repeated measures designs. Psychological Methods, 1, 170−177.
19.
EllisP.D. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge: Cambridge University Press.
FagerlandM.SandvikL. (2009).The Wilcoxon–Mann–Whitney Test under scrutiny. Statistics in Medicine, 28, 1487–1497.
22.
FaulF.ErdfelderE.LangA.-G.BuchnerA. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
23.
FayM.ProschanM. (2010). Wilcoxon–Mann–Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Stat Survey, 4, 1–39. Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2857732 (accessed May 2016).
24.
FieldA. (2005). Intraclass correlation. In: EverittB.HowellD. (Eds), Encyclopedia of statistics in behavioral science: Volume 2 (pp. 948–954). Chichester: Wiley.
25.
FieldA.MilesJ.FieldZ. (2012). Discovering statistics using R. Thousand Oaks, CA: Sage.
HesterbergT.MooreD.MonaghanS.ClipsonA.EpsteinR. (2014). Bootstrap methods and permutation tests. In: MooreD.McCabeG.CraigB. (Eds) Introduction to the practice of statistics (pp.16–1 to 16–57). 7th edition. Chapter 16. New York: WH Freeman. Available at: http://www.timhesterberg.net/bootstrap (accessed May 2016).
KeselmanH.AlginaJ.KowalchukR. (2001). The analysis of repeated measures designs: A review. British Journal of Mathematical and Statistical Psychology, 54, 1–20.
38.
KeselmanH.CribbieR.HollandB. (2004). Pairwise multiple comparison test procedures: An update for clinical child and adolescent psychologists. Journal of Clinical Child and Adolescent Psychology, 33, 623–645. Available at: http://home.cc.umanitoba.ca/~kesel/jccap-4.pdf (accessed May 2016).
Larson-HallJ.PlonskyL. (2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65, 127–159.
45.
LindstrombergS. (2016). Inferential statistics in Language Teaching Research: A review and ways forward. Language Teaching Research. DOI: 10.1177/1362168816649979.
46.
LinckJ.CunningsJ. (2015). The utility and application of mixed-effects models in second language research. Language Learning, 65, 185–207.
47.
MaxwellS.DelaneyH. (2004). Designing experiments and analyzing data: A model comparison perspective. New York: Psychology Press.
MillerG.ChapmanJ. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110, 40–48.
50.
MorrisS.DeShonR. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105–125.
51.
NorrisJ. (2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65, 97–126.
52.
O’KeefeD. (2007). Post hoc power, observed power, a priori power, retrospective power, achieved power: Sorting out appropriate uses of statistical power analyses. Communication Methods and Measures, 1, 291–299. Available at: http://www.dokeefe.net/pub/okeefe07cmm-posthoc.pdf (accessed May 2016).
53.
PlonskyL. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687.
54.
PlonskyL.GassS. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325–366.
55.
R Core Team. (2016). R: A language and environment for statistical computing [computer software]. Vienna: R Foundation for Statistical Computing. Available at: http://www.R-project.org (accessed May 2016).
56.
RosenthalR.RosnowR. (1985). Contrast analysis: Focused comparisons in the analysis of variance. Cambridge: Cambridge University Press.
57.
RuxtonG. (2006). The unequal variance t-test is an underused alternative to Student’s t-test and the Mann–Whitney U test. Behavioral Ecology, 17, 688–690. Available at: http://beheco.oxfordjournals.org/content/17/4/688.full.pdf+html (accessed May 2016).
WilcoxR. (2005). Introduction to robust estimation and hypothesis testing. 2nd edition.Amsterdam: Elsevier.
62.
WilcoxR. (2010). Fundamentals of modern statistical methods. 2nd edition.Heidelberg: Springer.
63.
WilcoxR. (2012). Modern statistics for the social and behavioral sciences: A practical introduction. Boca Raton, FL: CRC.
64.
WilkinsonL. & Task Force on Statistical Inference. (1999.) Statistical methods in psychological journals: Guidelines and explanations. American Psychologist, 54, 594–604.
65.
WilliamsR.ZimmermanD. (1996). Are simple gain scores obsolete?Applied Psychological Measurement, 20, 59–69.
66.
WrightD. (2006). Comparing groups in a before-after design: When t test and ANCOVA produce different results. British Journal of Educational Psychology, 76, 663–675.
ZimmermanD. (1998). Invalidation of parametric and nonparametric statistical tests by concurrent violation of two assumptions. Journal of Experimental Education, 67, 55–68.
69.
ZimmermanD. (2003). A warning about the large-sample Wilcoxon–Mann–Whitney test. Understanding Statistics, 2, 267–280.
70.
ZimmermanD. (2004a). Inflation of Type I error rates by unequal variances associated with parametric, nonparametric, and rank-transformation tests. Psicológica, 25, 103–133. Available at: http://www.uv.es/revispsi/articulos1.04/6-zimmerman.pdf (accessed May 2016).
71.
ZimmermanD. (2004b). A note on preliminary tests of equality of variances. British Journal of Mathematical and Statistical Psychology, 57, 173–181.
72.
ZimmermanD.W. (2011). A simple and effective decision rule for choosing a significance test to protect against non-normality. British Journal of Mathematical and Statistical Psychology, 64, 388–409.
73.
ZimmermanD.ZumboB. (2009). Hazards in choosing between pooled and separate-variances t tests. Psicológica, 30, 371–390.
