A Case Study for the Interrater Reliability of Journal Referees

Abstract

Purpose: Current research indicates problems are emerging in our current system of anonymous journal reviews. We find a proliferation of predatory journals and difficulty in identifying them and explaining the dangers of publishing them to scholars. Predatory journals claim to offer anonymous reviews, but often they do not. Methods: We employ data from The International Journal of Social Work Values and Ethics as a model for transparency (n = 246). Interrater reliability correlations and chi square was used to contrast data from our literature review. Results: Although the interrater correlations were deemed moderate, our results were better than those in the baseline found in the review of literature. This type of data addressing journals is not available to the scientific community but should be. Conclusion: Based on the synthesis of the review of literature and our case study, eight recommendations are offered to improve the anonymous review process for academic journals.

Keywords

interrater reliability case study journals predatory journals transparency

Get full access to this article

View all access options for this article.

References

Adair

R. K.

(1982). A physics editor comments on Peters and Ceci’s peer-review study. Behavioral and Brain Sciences, 5(2), 196.

Anonymous (2014). Double-blind peer review. Nature Nanotechnology, 9(11), 869.

Anonymous (2015a). A double-blind option for peer review. Nature Medicine, 21(3), 199.

Anonymous (2015b). Double-blind peer review. Nature Chemical Biology, 11(4), 237.

Anonymous (2015c). Introducing double-blind peer review. Nature Cell Biology, 17(3), 195.

Arumugam

Mehta

Baxter

G. D.

(2020). Double-blind peer review of manuscripts: Opportunities, challenges, and way forward. Physical Therapy Reviews, 25(1), 1-6.

Bailar

J. C.

(1991). Reliability, fairness, objectivity, and other inappropriate goals in peer review. Behavioral and Brain Sciences, 14(1), 137-138.

Berger

Rittman

(2016). Double blind peer-review in games. Games, 7(1), 1.

Binder

(1984). Restrictions on statistics imposed by method of measurement: Some reality, some myth. Journal of Criminal Justice, 12, 467-481.

10.

Bornmann

(2008). An analysis of the peer review process from the perspective of sociology of science theories. Human Architecture: Journal of The Sociology of Self-Knowledge, 6(2), 23-38.

11.

Bornmann

Mutz

Daniel

H-D

(2010). A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE, 5(12), e14331. https://doi.org/10.1371/journal.pone.0014331.

12.

Caputo (2019). Peer review: A vital gatekeeping function and obligation of professional scholarly practice. Families in Society: The Journal of Contemporary Human Services, 100(1), 6-16. https://doi.org/10.1177/1044389418808155.

13.

Cole

J. R.

(2000). The role of journals in the growth of scientific knowledge. In B.

Cronin

H. B.

Atkins

(Eds.), The web of knowledge. A festschrift in honor of Eugene Garfield (pp. 109-142). Information Today.

14.

Cureton

(1978). Psychometrics. In W.

Kruskal

Tanur

(Eds.), International encyclopedia of statistics. Free Press.

15.

de Kemp

Ware

(2008). Peer review in scholarly journals: Perspective of the scholarly community – Results from an international study. Information Services & Use, 28(2), 109-112.

16.

de Winter

J. C. F.

Gosling

S. D.

Potter

(2016). Comparing the pearson and spearman correlation coefficients across distributions and sample sizes: A tutorial using simulations and empirical data. Psychological Methods, 21(3), 273-290. https://doi-org.proxy181.nclive.org/10.1037/met0000079.supp.

17.

Ellwood

(1916). Objectivism in Sociology. American Journal of Sociology, 22(3), 289-305. Retrieved September 12, 2021, from http://www.jstor.org/stable/2763724.

18.

Elmore

S. A.

Weston

E. H.

(2020). Predatory journals: What they are and how to avoid them. Toxicologic Pathology, 48(4), 607-610.

19.

Gibbons

J. D.

(1993). Nonparametric measures of association. New York, NY: SAGE Publications.

20.

Gignac

(2011a). Kandall’s tau vs Spearman rank correlation (part 1). Retrieved from https://www.youtube.com/watch?v=D56dvoVrBBE.

21.

Gignac

(2011b). Kandall’s tau vs Spearman rank correlation (part 2). Retrieved from https://www.youtube.com/watch?v=7i5aU2pBcMY&list=RDCMUCr3OHuCSrwAO2KYP2CJB6zg&index=1.

22.

Gottfried

G. E.

(1991). Introduction to statistics. New York, NY: Springer-Verlag.

23.

Guba

(2013). How journals select articles: The manuscript review process in American sociology. Laboratorium: Russian Review of Social Research, 5(3), 66-89.

24.

Hillman

A. J.

Rynes

S. L.

(2007). The future of double-blind review in management. Journal of Management Studies, 44(4), 622-627.

25.

R. C.-M.

Mak

K.-K.

Tao

Day

J. R.

Pan

(2013). Views on the peer review system of biomedical journals: An online survey of academics from high-ranking universities. BMC Medical Research Methodology, 13(1), 74. https://doi.org/10.1186/1471-2288-13-74.

26.

Jenson

(2005). Structural factors and the quality of publication in social work journals. Social Work Research, 29(2), 67-71. https://doi-org.proxy181.nclive.org/10.1093/swr/29.2.67.

27.

Kim

J. O.

(1975). Multivariate analysis of ordinal variables. American Journal of Sociology, 81, 261-298.

28.

Kiresuk

T. J.

Smith

, & Cardillo

J. E.

(1994). Goal attainment scaling: Applications theory, and measurement. Beverly Hills, CA: Lawrence Erlbaum Associates.

29.

Kirk

S. A.

Franke

T. M.

(1997). Research note. Agreeing to disagree: A study of the reliability of manuscript reviews. Social Work Research, 21(2), 121-126. https://doi-org.proxy181.nclive.org/10.1093/swr/21.2.121.

30.

Labovitz

(1967). Some observations on measurement and statistics. Social Forces, 46, 151-160.

31.

Labovitz

(1970). The assignment of numbers to rank order categories. American Sociological Review, 35, 515-524.

32.

LeBreton

J. M.

Senter

J. L.

(2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11(4), 815-852. https://doi.org/10.1177/1094428106296642.

33.

Lee

C. J.

Sugimoto

C. R.

Zhang

Cronin

(2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2-17.

34.

Leopold

S. S.

(2016). Editorial: CORR1 will change to double-blind peer review—What took us so long to get there?. Clinical Orthopaedics and Related Research, 475(2), 297-299.

35.

Leung

S.-O.

(2011). A comparison of psychometric properties and normality in 4-, 5-, 6-, and 11-point Likert scales. Journal of Social Service Research, 37(4), 412-421.

36.

Liebetrau

A. M.

(1983). Measures of association. Chicago, IL: SAGE Publications.

37.

Marson

S. M.

Finn

(2004). Editorial comment: The birth of The Journal of Social Work Values and Ethics. The Journal of Social Work Values and Ethics, 1(1), 2. Retrieved from https://jswve.org/download/2004-1/JSWVE-Fall-2004-Complete.pdf.

38.

Martin

(2016). How double-blind peer review works and what it takes to be a good referee. Current Sociology, 64(5), 691-698.

39.

Merton

R. K.

(1973). The sociology of science: Theoretical and empirical investigations. Thousand Oaks, CA: University of Chicago Press.

40.

Mulligan

Hall

Raphael

(2013). Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the American Society for Information Science & Technology, 64(1), 132-161. https://doi.org/10.1002/asi.22798.

41.

Osborne

J. W.

(2007). Best practices in quantitative methods. SAGE Publications.

42.

Pritchard

S. M.

(2016). Double-blind review: A commitment to fair editorial practices. Libraries and the Academy, 12(2), 117-119.

43.

Resnik

Elmore

(2016). Ensuring the quality, fairness, and integrity of journal peer review: A possible role of editors. Science & Engineering Ethics, 22(1), 169-188.

44.

Rittman

Kaufman

P. I.

(2016). Double blind peer-review in Religions. Religions, 7(1), 1.

45.

Robson

Pitt

West

D. C.

(2015). Navigating the peer-review process: reviewers’ suggestions for a manuscript. Journal of Advertising Research, 55(1), 9-17.

46.

Wicherts

J. M.

(2016). Peer review quality and transparency of the peer-review process in open access and subscription journals. PLoS ONE, 11(1), e0147913. https://10.1371/journal.pone.0147913.

47.

Yaffe

(2017). From the editor—On peer review: Improving the credibility of social work scholarship. Journal of Social Work Education, 53(4), 577-579. https://doi-org.proxy181.nclive.org/10.1080/10437797.2017.1376953.

48.

Zuckerman

Merton

R.K.

(1971). Patterns of evaluation in science: Institutionalisation, structure and functions of the referee system. Minerva, 9, 66-100.

49.

Zumbo

B. D.

Zimmerman

D. W.

(1993). Is the selection of statistical methods governed by level of measurement?. Canadian Psychology, 34, 390-399.