The seven sins of L2 research: A review of 30 journals’ statistical quality and their CiteScore,SJR,SNIP,JCR Impact Factors

Abstract

This report presents a review of the statistical practices of 30 journals representative of the second language field. A review of 150 articles showed a number of prevalent statistical violations including incomplete reporting of reliability, validity, non-significant results, effect sizes, and assumption checks as well as making inferences from descriptive statistics and failing to correct for multiple comparisons. Scopus citation analysis metrics and whether a journal is SSCI-indexed were predictors of journal statistical quality. No clear evidence was obtained to favor the newly introduced CiteScore over SNIP or SJR. Implications of the results are discussed.

Keywords

citation analysis metrics CiteScore JCR Impact Factor journal quality quantitative research second language SJR SNIP

Get full access to this article

View all access options for this article.

References

Benson

Chik

Gao

Huang

Wang

(2009). Qualitative research in language teaching and learning journals, 1997–2006. The Modern Language Journal, 93, 79–90.

Bolstad

W.M.

(2007). Introduction to Bayesian Statistics. 2nd edition. Hoboken, NJ: Wiley.

Brown

J.D.

(1990). The use of multiple t tests in language research. TESOL Quarterly, 24, 770–773.

Brown

J.D.

(2004). Resources on quantitative/statistical research for applied linguists. Second Language Research, 20, 372–393.

Brumback

R.A.

(2009). Impact factor wars: Episode V: The empire strikes back. Journal of Child Neurology, 24, 260–262.

Chadegani

A.A.

Salehi

Yunus

M.M.

, et al. (2013). A comparison between two main academic literature collections: Web of science and scopus databases. Asian Social Science, 9, 18–26.

Cohen

(1994). The earth is round (p < .05). American Psychologist, 49, 997–1003.

Cohen

West

S.G.

Aiken

L.S.

(2003). Applied multiple regression/correlation analysis for the behavioral sciences. 3rd edition. Mahwah, NJ: Lawrence Erlbaum.

Colledge

de Moya-Anegón

Guerrero-Bote

, et al. (2010). SJR and SNIP: Two new journal metrics in Elsevier’s Scopus. Serials, 23, 215–221.

10.

da Silva

J.A.T.

Memon

A.R

. (2017). CiteScore: A cite for sore eyes, or a valuable, transparent metric? Scientometrics, 111, 553–556.

11.

Dienes

(2014). Using Bayes to get the most out of non-significant results. Frontiers in Psychology, 5. Available online at http://doi.org/10.3389/fpsyg.2014.00781 (assessed March 2018).

12.

Egbert

J.O.Y.

(2007). Quality analysis of journals in TESOL and applied linguistics. TESOL Quarterly, 41, 157–171.

13.

Garfield

(2006). The history and meaning of the journal impact factor. JAMA, 295, 90–93.

14.

Gelman

Stern

H.S.

(2006). The difference between ‘significant’ and ‘not significant’ is not itself statistically significant. The American Statistician, 60, 328–331.

15.

Green

S.B.

Lissitz

R.W.

Mulaik

S.A.

(1977). Limitations of coefficient alpha as an index of test unidimensionality. Educational and Psychological Measurement, 37, 827–838.

16.

Guerrero-Bote

V.P.

Moya-Anegón

(2012). A further step forward in measuring journals’ scientific prestige: The SJR2 indicator. Journal of Informetrics, 6, 674–688.

17.

Guz

A.N.

Rushchitsky

J.J.

(2009). Scopus: A system for the evaluation of scientific journals. International Applied Mechanics, 45, 351–362.

18.

Hoekstra

Morey

R.D.

Rouder

J.N.

Wagenmakers

E.-J.

(2014). Robust misinterpretation of confidence intervals. Psychonomic Bulletin & Review, 21, 1157–1164.

19.

Hong-Nam

Leavell

A.G.

(2006). Language learning strategy use of ESL students in an intensive English learning context. System, 34, 399–415.

20.

Hox

J.J.

(2010). Multilevel analysis: Techniques and applications. 2nd edition. New York: Routledge.

21.

Jung

U.O.H.

(2004). Paris in London revisited or the foreign language teacher’s top-most journals. System, 32, 357–361.

22.

Lakens

(2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4. Available online at http://doi.org/10.3389/fpsyg.2013.00863 (assessed March 2018).

23.

Larson-Hall

(2016). A guide to doing statistics in second language research using SPSS and R. 2nd edition. New York: Routledge.

24.

Larson-Hall

(2017). Moving beyond the bar plot and the line graph to create informative and attractive graphics. The Modern Language Journal, 101, 244–270.

25.

Larson-Hall

Plonsky

(2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65, 127–159.

26.

Leydesdorff

Opthof

(2010). Scopus’s source normalized impact per paper (SNIP) versus a journal impact factor based on fractional counting of citations. Journal of the American Society for Information Science and Technology, 61, 2365–2369.

27.

Lindstromberg

(2016). Inferential statistics in Language Teaching Research: A review and ways forward. Language Teaching Research, 20, 741–768.

28.

Loewen

Gass

(2009). The use of statistics in L2 acquisition research. Language Teaching, 42, 181–196.

29.

Loewen

Crowther

Isbell

Lim

Maloney

Tigchelaar

(2017). The statistical literacy of applied linguistics researchers. Unpublished paper presented at the American Association for Applied Linguistics (AAAL), Portland, Oregon, USA.

30.

Loewen

Lavolette

Spino

L.A.

, et al. (2014). Statistical literacy among applied linguists and second language acquisition researchers. TESOL Quarterly, 48, 360–388.

31.

Ludbrook

(1998). Multiple comparison procedures updated. Clinical and Experimental Pharmacology and Physiology, 25, 1032–1037.

32.

Morey

R.D.

Hoekstra

Rouder

J.N.

Lee

M.D.

Wagenmakers

E.-J.

(2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23, 103–123.

33.

Nassaji

(2012). Statistical significance tests and result generalizability: Issues, misconceptions, and a case for replication. In Porte

G.K.

(Ed.), Replication research in applied linguistics (pp. 92–115). Cambridge: Cambridge University Press.

34.

Neyman

(1937). Outline of a theory of statistical estimation based on the classical theory of probability. Philosophical transactions of the Royal Society of London, Series A: Mathematical and Physical Sciences, 236, 333–380.

35.

Norris

J.M.

(2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65, 97–126.

36.

Norris

J.M.

Plonsky

Ross

S.J.

Schoonen

(2015). Guidelines for reporting quantitative methods and results in primary research. Language Learning, 65, 470–476.

37.

Oxford

R.L.

(1990). Language learning strategies: What every teacher should know. New York: Newbury.

38.

Plonsky

(2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687.

39.

Plonsky

(2014). Study quality in quantitative L2 research (1990–2010): A methodological synthesis and call for reform. The Modern Language Journal, 98, 450–470.

40.

Plonsky (Ed.) (2015). Advancing quantitative methods in second language research. New York: Routledge.

41.

Plonsky

Gass

(2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325–366.

42.

Plonsky

Oswald

F.L.

(2014). How big Is ‘big’? Interpreting effect sizes in L2 research. Language Learning, 64, 878–912.

43.

Radwan

A.A.

(2011). Effects of L2 proficiency and gender on choice of language learning strategies by university students majoring in English. The Asian EFL Journal, 13, 115–163.

44.

Rayson

Berridge

Francis

(2004). Extending the Cochran rule for the comparison of word frequencies between corpora. Unpublished paper presented at the 7th International Conference on Statistical analysis of textual data (JADT 2004), Louvain-la-Neuve, Belgium.

45.

Rothstein

H.R.

Sutton

A.J.

Borenstein

(Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment and adjustments. Chichester: Wiley.

46.

Schmitt

(1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350–353.

47.

Sijtsma

(2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120.

48.

Stevens

(2009). Applied multivariate statistics for the social sciences. 5th edition. New York: Routledge.

49.

Tabachnick

B.G.

Fidell

L.S.

(2013). Using multivariate statistics. 6th edition. Boston, MA: Pearson.

50.

Tseng

W.-T.

Dörnyei

Schmitt

(2006). A new approach to assessing strategic learning: The case of self-regulation in vocabulary Acquisition. Applied Linguistics, 27, 78–102.

51.

Vitta

J.P.

Al-Hoorie

A.H.

(2017). Scopus- and SSCI-indexed L2 journals: A list for the Asia TEFL community. The Journal of Asia TEFL, 14, 784-792.

52.

Weiner

(2001). The academic journal: Has it a future? Education Policy Analysis Archives, 9. Available online at http://doi.org/10.14507/epaa.v9n9.2001 (assessed March 2018).