This report presents a review of the statistical practices of 30 journals representative of the second language field. A review of 150 articles showed a number of prevalent statistical violations including incomplete reporting of reliability, validity, non-significant results, effect sizes, and assumption checks as well as making inferences from descriptive statistics and failing to correct for multiple comparisons. Scopus citation analysis metrics and whether a journal is SSCI-indexed were predictors of journal statistical quality. No clear evidence was obtained to favor the newly introduced CiteScore over SNIP or SJR. Implications of the results are discussed.
BensonP.ChikA.GaoX.HuangJ.WangW. (2009). Qualitative research in language teaching and learning journals, 1997–2006. The Modern Language Journal, 93, 79–90.
2.
BolstadW.M. (2007). Introduction to Bayesian Statistics. 2nd edition.Hoboken, NJ: Wiley.
3.
BrownJ.D. (1990). The use of multiple t tests in language research. TESOL Quarterly, 24, 770–773.
4.
BrownJ.D. (2004). Resources on quantitative/statistical research for applied linguists. Second Language Research, 20, 372–393.
5.
BrumbackR.A. (2009). Impact factor wars: Episode V: The empire strikes back. Journal of Child Neurology, 24, 260–262.
6.
ChadeganiA.A.SalehiH.YunusM.M., et al. (2013). A comparison between two main academic literature collections: Web of science and scopus databases. Asian Social Science, 9, 18–26.
7.
CohenJ. (1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.
8.
CohenJ.CohenJ.WestS.G.AikenL.S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences. 3rd edition.Mahwah, NJ: Lawrence Erlbaum.
9.
ColledgeL.de Moya-AnegónF.Guerrero-BoteV., et al. (2010). SJR and SNIP: Two new journal metrics in Elsevier’s Scopus. Serials, 23, 215–221.
10.
da SilvaJ.A.T.MemonA.R. (2017). CiteScore: A cite for sore eyes, or a valuable, transparent metric?Scientometrics, 111, 553–556.
11.
DienesZ. (2014). Using Bayes to get the most out of non-significant results. Frontiers in Psychology, 5. Available online at http://doi.org/10.3389/fpsyg.2014.00781 (assessed March 2018).
12.
EgbertJ.O.Y. (2007). Quality analysis of journals in TESOL and applied linguistics. TESOL Quarterly, 41, 157–171.
13.
GarfieldE. (2006). The history and meaning of the journal impact factor. JAMA, 295, 90–93.
14.
GelmanA.SternH.S. (2006). The difference between ‘significant’ and ‘not significant’ is not itself statistically significant. The American Statistician, 60, 328–331.
15.
GreenS.B.LissitzR.W.MulaikS.A. (1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827–838.
16.
Guerrero-BoteV.P.Moya-AnegónF. (2012). A further step forward in measuring journals’ scientific prestige: The SJR2 indicator. Journal of Informetrics, 6, 674–688.
17.
GuzA.N.RushchitskyJ.J. (2009). Scopus: A system for the evaluation of scientific journals. International Applied Mechanics, 45, 351–362.
JungU.O.H. (2004). Paris in London revisited or the foreign language teacher’s top-most journals. System, 32, 357–361.
22.
LakensD. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. Available online at http://doi.org/10.3389/fpsyg.2013.00863 (assessed March 2018).
23.
Larson-HallJ. (2016). A guide to doing statistics in second language research using SPSS and R. 2nd edition.New York: Routledge.
24.
Larson-HallJ. (2017). Moving beyond the bar plot and the line graph to create informative and attractive graphics. The Modern Language Journal, 101, 244–270.
25.
Larson-HallJ.PlonskyL. (2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65, 127–159.
26.
LeydesdorffL.OpthofT. (2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61, 2365–2369.
27.
LindstrombergS. (2016). Inferential statistics in Language Teaching Research: A review and ways forward. Language Teaching Research, 20, 741–768.
28.
LoewenS.GassS. (2009). The use of statistics in L2 acquisition research. Language Teaching, 42, 181–196.
29.
LoewenS.CrowtherD.IsbellD.LimJ.MaloneyJ.TigchelaarM. (2017). The statistical literacy of applied linguistics researchers. Unpublished paper presented at the American Association for Applied Linguistics (AAAL), Portland, Oregon, USA.
30.
LoewenS.LavoletteE.SpinoL.A., et al. (2014). Statistical literacy among applied linguists and second language acquisition researchers. TESOL Quarterly, 48, 360–388.
31.
LudbrookJ. (1998). Multiple comparison procedures updated. Clinical and Experimental Pharmacology and Physiology, 25, 1032–1037.
32.
MoreyR.D.HoekstraR.RouderJ.N.LeeM.D.WagenmakersE.-J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23, 103–123.
33.
NassajiH. (2012). Statistical significance tests and result generalizability: Issues, misconceptions, and a case for replication. In PorteG.K. (Ed.), Replication research in applied linguistics (pp. 92–115). Cambridge: Cambridge University Press.
34.
NeymanJ. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical transactions of the Royal Society of London, Series A: Mathematical and Physical Sciences, 236, 333–380.
35.
NorrisJ.M. (2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65, 97–126.
36.
NorrisJ.M.PlonskyL.RossS.J.SchoonenR. (2015). Guidelines for reporting quantitative methods and results in primary research. Language Learning, 65, 470–476.
37.
OxfordR.L. (1990). Language learning strategies: What every teacher should know. New York: Newbury.
38.
PlonskyL. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687.
39.
PlonskyL. (2014). Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform. The Modern Language Journal, 98, 450–470.
40.
Plonsky (Ed.) (2015). Advancing quantitative methods in second language research. New York: Routledge.
41.
PlonskyL.GassS. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325–366.
42.
PlonskyL.OswaldF.L. (2014). How big Is ‘big’? Interpreting effect sizes in L2 research. Language Learning, 64, 878–912.
43.
RadwanA.A. (2011). Effects of L2 proficiency and gender on choice of language learning strategies by university students majoring in English. The Asian EFL Journal, 13, 115–163.
44.
RaysonP.BerridgeD.FrancisB. (2004). Extending the Cochran rule for the comparison of word frequencies between corpora. Unpublished paper presented at the 7th International Conference on Statistical analysis of textual data (JADT 2004), Louvain-la-Neuve, Belgium.
45.
RothsteinH.R.SuttonA.J.BorensteinM. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester: Wiley.
46.
SchmittN. (1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350–353.
47.
SijtsmaK. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120.
48.
StevensJ. (2009). Applied multivariate statistics for the social sciences. 5th edition.New York: Routledge.
49.
TabachnickB.G.FidellL.S. (2013). Using multivariate statistics. 6th edition.Boston, MA: Pearson.
50.
TsengW.-T.DörnyeiZ.SchmittN. (2006). A new approach to assessing strategic learning: The case of self-regulation in vocabulary Acquisition. Applied Linguistics, 27, 78–102.
51.
VittaJ.P.Al-HoorieA.H. (2017). Scopus- and SSCI-indexed L2 journals: A list for the Asia TEFL community. The Journal of Asia TEFL, 14, 784-792.
52.
WeinerG. (2001). The academic journal: Has it a future?Education Policy Analysis Archives, 9. Available online at http://doi.org/10.14507/epaa.v9n9.2001 (assessed March 2018).
