This article reviews all (quasi)experimental studies appearing in the first 19 volumes (1997–2015) of Language Teaching Research (LTR). Specifically, it provides an overview of how statistical analyses were conducted in these studies and of how the analyses were reported. The overall conclusion is that there has been a tight adherence to traditional methods and practices, some of which are suboptimal. Accordingly, a number of improvements are recommended. Topics covered include the implications of small average sample sizes, the unsuitability of p values as indicators of replicability, statistical power and implications of low power, the non-robustness of the most commonly used significance tests, the benefits of reporting standardized effect sizes such as Cohen’s d, options regarding control of the familywise Type I error rate, analytic options in pretest–posttest designs, ‘meta-analytic thinking’ and its benefits, and the mistaken use of a significance test to show that treatment groups are equivalent at pretest. An online companion article elaborates on some of these topics plus a few additional ones and offers guidelines, recommendations, and additional background discussion for researchers intending to submit to LTR an article reporting a (quasi)experimental study.
American Psychological Association (APA). (2010). Publication manual of the American Psychological Association. Washington, DC: American Psychological Association.
5.
AnQ.XuD.BrooksG. (2013). Type I error rates and power of multiple hypothesis testing procedures in factorial ANOVA. Multiple Linear Regression Viewpoints, 39, 1–16.
6.
AsendorpfJ.ConnerM.De FruytF.. (2013). Recommendations for increasing replicability in psychology. (Target article.)European Journal of Personality, 27, 108–119. See also ‘Authors’ response’, pp. 138–144.
BaguleyT. (2012). Serious stats: A guide to advanced statistics for the behavioral sciences. Basingstoke: Palgrave Macmillan.
9.
BakkerM.WichertsJ. (2014). Outlier removal, sum scores, and the inflation of the Type I error rate in independent samples t tests: The power of alternatives and recommendations. Psychological Methods, 19, 409–427.
10.
BenjaminiY.HochbergY. (2000). On the adaptive control of the false discovery rate in multiple testing with independent statistics. Journal of Educational and Behavioral Statistics, 25, 60–83.
11.
BonateP. (2000). Analysis of pretest–posttest designs. Boca Raton, FL: Chapman and Hall/CRC.
12.
BorensteinM.HedgesL.HigginsJ.RothsteinH. (2009). Introduction to meta-analysis. Oxford: Wiley.
ChaudronC. (2001). Progress in language classroom research: Evidence from The Modern Language Journal, 1916–2000. The Modern Language Journal, 85, 57–76.
CribbieR.JamiesonJ. (2004). Decreases in posttest variance and the measurement of change. Methods of Psychological Research Online, 9, 37–55. Available at: http://www.dgps.de/fachgruppen/methoden/mpr-online (accessed May 2016).
18.
CummingG. (2008). Replication and p intervals: P values predict the future only vaguely; but confidence intervals do much better. Perspectives on Psychological Science, 3, 286–300.
19.
CummingG. (2012a). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. Hove: Routledge.
20.
CummingG. (2012b). Researchers underestimate the variability of p values over replication. Methodology, 8, 61–62.
CummingG.WilliamsJ.FidlerF. (2004). Replication, and researchers’ understanding of confidence intervals and standard error bars. Understanding Statistics, 3, 299–311.
23.
CurranP.ObeidatK.LosardoD. (2010). Twelve frequently asked questions about growth curve modeling. Journal of Cognition and Development, 11, 121–136.
24.
EllisN. (2015). Forward. Language Learning, 65, Supplement 1, v–vi.
25.
EllisN.SagarraN. (2011). Learned attention in adult language acquisition: A replication and generalization study and meta-analysis. Studies in Second Language Acquisition, 33, 589–624.
26.
EllisP. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge: Cambridge University Press.
FieldA. (1999). A bluffer’s guide to meta-analysis. Newsletter of the mathematical, statistical and computing section of the British Psychological Society, 7, 16–25. Available at: http://users.sussex.ac.uk/~andyf/meta.pdf (accessed May 2016).
29.
FieldA.MilesJ.FieldZ. (2012). Discovering statistics using R. Thousand Oaks, CA: Sage.
30.
FitzmauriceG. (2001). A conundrum in the analysis of change. Nutrition, 17, 360 –361.
31.
GarcíaL. (2004). Escaping the Bonferroni iron claw in ecological studies. OIKOS, 105, 557–663.
32.
GassS. (2009). A historical survey of SLA research. In RitchieW.BhatiaT. (Eds.), Handbook of second language acquisition (pp. 3–28). Bingley: Emerald.
33.
GrissomR.KimJ. (2012). Effect sizes for research: Univariate and multivariate applications, 2nd edition.Hove: Routledge.
34.
HesterbergT. (2008). It’s time to retire the ‘n > = 30’ rule. Proceedings of the American Statistical Association, statistical computing section [CD-ROM]. Available at: http://www.timhesterberg.net/articles/JSM08-n30.pdf (accessed May 2016).
35.
HoenigJ.HeiseyD. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55, 1–6. Available at: http://www.tc.umn.edu/~alonso/Hoenig_AmericanStat_2001.pdf (accessed May 2016).
36.
HorstM.CobbT.MearaP. (1998). Beyond ‘A Clockwork Orange’: Acquiring second language vocabulary through reading. Reading in a Foreign Language, 11, 207–223.
HoxJ.StoelR. (2014). Multilevel and SEM approaches to growth curve modeling. Wiley StatsRef: Statistics Reference Online. Originally published online in 2005 in Encyclopedia of Statistics in Behavioral Science. Oxford: Wiley.
39.
HudsonT.LlosaL. (2015). Design issues and inference in experimental L2 research. Language Learning, 65, 76–96.
KeselmanH.CribbieR.HollandB. (2004). Pairwise multiple comparison test procedures: An update for clinical child and adolescent psychologists. Journal of Clinical Child and Adolescent Psychology, 33, 623–645. Available at: http://home.cc.umanitoba.ca/~kesel/jccap-4.pdf (accessed May 2016).
42.
KeselmanH.OthmanA.WilcoxR. (2013). Preliminary testing for normality: Is this a good practice?Journal of Modern Applied Statistical Methods, 12, Article 2. Available at: http://digitalcommons.wayne.edu/jmasm/vol12/iss2/2 (accessed May 2016).
43.
KeselmanH.OthmanA.WilcoxR. (2014). Testing for normality in the multi-group problem: Is this a good practice?Clinical Dermatology, 2, 29–43.
44.
KeselmanH.HubertyC.LixL., et al. (1998). Statistical practices of educational researchers: An analysis of Their ANOVA, MANOVA and ANCOVA analyses. Review of Educational Research, 68, 350–86. Available at: http://home.cc.umanitoba.ca/~kesel/rer1998.pdf (accessed May 2016).
45.
KlineR. (2013). Beyond significance testing: Statistics reform in the behavioral sciences. 2nd edition.Washington, DC: American Psychological Association.
46.
KnappT.SchaferW. (2009). From gain score t to ANCOVA F (and vice versa). Practical Assessment, Research & Evaluation, 14. Available at: http://pareonline.net/getvn.asp?v=14&n=6 (accessed May 2016).
47.
LaiJ.FidlerF.CummingG. (2009). Subjective p intervals: Researchers underestimate the variability of p values over replication. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 8, 51–62.
48.
Larson-HallJ.HerringtonR. (2009). Improving data analysis in second language acquisition by utilizing modern developments in applied statistics. Applied Linguistics, 31, 368–390.
49.
Larson-HallJ.PlonskyL.2015. Reporting and interpreting quantitative research Findings: What gets reported and recommendations for the field. Language Learning, 65, 127–159.
50.
LenthR. (2000). Two sample-size practices that I don’t recommend: Comments from the panel discussion at the 2000 Joint Statistical Meetings. Unpublished article, presented at the 2000 Joint Statistical Meetings, Indianapolis, IN, USA. Available at: http://homepage.stat.uiowa.edu/~rlenth/Power/2badHabits.pdf (accessed May 2016).
51.
LinckJ.CunningsJ. (2015). The utility and application of mixed-effects models in second language research. Language Learning, 65, 185–207.
52.
LindstrombergS.EyckmansJ. (2014). How big is the positive effect of assonance on the near-term recall of L2 collocations?ITL International Journal of Applied Linguistics, 165, 19–45.
53.
LindstrombergS. (2016). Guidelines, recommendations, and supplementary discussion: For researchers planning to submit a report of an intervention study toLanguage Teaching Research. Language Teaching Research. DOI: 10.1177/1362168816651895.
MackeyA.GassS. (2005). Language research: Methodology and design. Mahwah, NJ: Lawrence Erlbaum.
56.
MaxwellS. (2004). The persistence of underpowered studies in psychological research: Causes, consequences, and remedies. Psychological Methods, 9, 147–163.
57.
McGrawK.WongS. (1992). A common language effect size statistic. Psychological Bulletin, 111, 361–365.
MillerG.ChapmanJ. (2001). Misunderstanding analysis of covariance. Journal of Abnormal Psychology, 110, 40–48.
60.
NakataT. (2015). Effects of feedback timing on second language vocabulary learning: Does delaying feedback increase learning?Language Teaching Research, 19, 416–434.
61.
NassajiH. (2012). Statistical significance tests and result generalizability: Issues, misconceptions, and a case for replication. In PorteG. (Ed.), Replication research in applied linguistics, (pp. 92–115). Cambridge: Cambridge University Press.
NorrisJ. (2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65, 97–126.
64.
NorrisJ.OrtegaL. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417–528.
65.
NorrisJ.OrtegaL. (2006). The value and practice of research synthesis for language learning and teaching. In NorrisJ.OrtegaL. (eds), Synthesizing research on language learning and teaching (pp. 3–50). Amsterdam: John Benjamins.
66.
NorrisJ.RossS.SchoonenR. (2015). Improving second language quantitative research. Language Learning, 65, 1–8.
O’KeefeD. (2007). Post hoc power, observed power, a priori power, retrospective power, achieved power: Sorting out appropriate uses of statistical power analyses. Communication Methods and Measures, 1, 291–299. Available at: http://www.dokeefe.net/pub/okeefe07cmm-posthoc.pdf (accessed May 2016).
69.
OsborneJ. (2013). Is data cleaning and the testing of assumptions relevant in the 21st century?Frontiers in Psychology, 4, 4–7.
70.
OswaldF.PlonskyL. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85–110.
71.
PlonskyL. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687.
72.
PlonskyL.GassS. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325–366.
73.
PlonskyL.OswaldF. (2014). How big is ‘big’? Interpreting effect sizes in L2 research. Language Learning, 64, 878–912.
74.
PlonskyL.EgbertJ.LaflairG. (2014). Bootstrapping in applied linguistics: Assessing its potential using shared data. Applied Linguistics, 36, 591–610.
75.
PolioC. (2012). Replication in published applied linguistics research: A historical perspective. In PorteG. (Ed.), Replication research in applied linguistics (pp. 47–91). Cambridge: Cambridge University Press.
76.
PorteG. (2012). Introduction. In PorteG. (Ed.), Replication research in applied linguistics (pp. 1–17). Cambridge: Cambridge University Press.
77.
R Core Team. (2016). R: A language and environment for statistical computing [computer software]. Vienna: R Foundation for Statistical Computing. Available at: http://www.R-project.org (accessed May 2016).
78.
RogosaD. (1988). Myths about longitudinal research. In SchaieK.CampbellR.MeredithW.RawlingsS. (Eds), Methodological issues in aging research (pp. 171–209). New York: Springer.
TverskyA.KahnemanD. (1971). Belief in the law of small numbers. Psychological Bulletin, 76, 105–110.
83.
ValentineJ.PigottT.RothsteinH. (2010). How many studies do you need? A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 34, 215–247.
84.
VarghaA.DelaneyH. (2000). A critique and improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25, 101–132.
85.
VickersA. (2001). The use of percentage change from baseline as an outcome in a controlled trial is statistically inefficient: A simulation study. BMC Medical Research Methodology, 1, 6. Available at: http://www.biomedcentral.com/content/pdf/1471-2288-1-6.pdf (accessed May 2016).
86.
VickersA.AltmanD. (2001). Analysing controlled trials with baseline and follow up measurements. British Medical Journal, 323, 1123–1124. Available at: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1121605 (accessed May 2016).
87.
WestfallP.YoungS. (1993). Resampling-based multiple testing: Examples and methods for p-value adjustment. New York: Wiley.
WilcoxR. (2010). Fundamentals of modern statistical methods. 2nd edition.Heidelberg: Springer.
90.
WilcoxR. (2012a). Introduction to robust estimation and hypothesis testing. 3rd edition.Waltham, MA: Academic Press.
91.
WilcoxR. (2012b). Modern statistics for the social and behavioral sciences: A practical introduction. Boca Raton, FL: CRC Press.
92.
WilkinsonL., & Task Force on Statistical Inference. (1999.) Statistical methods in psychological journals: Guidelines and explanations. American Psychologist, 54, 594–604.
93.
WilliamsR.ZimmermanD. (1996). Are simple gain scores obsolete?Applied Psychological Measurement, 20, 59–69. Available at: http://conservancy.umn.edu/handle/11299//119063 (accessed May 2016).
ZimmermanD. (1998). Invalidation of parametric and nonparametric statistical tests by concurrent violation of two assumptions. Journal of Experimental Education, 67, 55–68.
97.
ZimmermanD.W. (2003). A warning about the large-sample Wilcoxon–Mann–Whitney test. Understanding Statistics, 2, 267–280.
98.
ZimmermanD. (2004a). A note on preliminary tests of equality of variances. British Journal of Mathematical and Statistical Psychology, 57, 173–181.
99.
ZimmermanD. (2004b). A note on the influence of outliers on parametric and nonparametric tests. Journal of General Psychology, 121, 391–401.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
