What’s in Your File Drawer? The Case of the Missing Null in Criminology and Criminal Justice

Abstract

Analysis of scholarship in the physical, biological, and social sciences has discovered that peer-reviewed journals publish a much larger proportion of articles with statistically significant findings compared to articles with null results. Publication bias in criminology and criminal justice (CCJ) has received very little attention, however. The current study is an exploratory analysis of research in leading CCJ journals across 2 years to determine the current state of null findings in contemporary CCJ scholarship. Our findings are consistent with studies in other disciplines; null results are rare in leading CCJ journals. We explore the context of our findings, outline the importance of examining publication bias to improve CCJ research and better inform policy, and discuss the limitations of our approach.

Keywords

publication bias null results criminology criminal justice research findings

Get full access to this article

View all access options for this article.

References

Ahlin

E. M.

(2020). Broadening the productivity lens in criminology and criminal justice: An exploratory study of the research contributions of master’s degree program faculty, 2014–2018. Journal of Criminal Justice Education, 31(1), 100–123.

Ariel

Farrar

W. A.

Sutherland

(2015). The effect of police body-worn cameras on use of force and citizens’ complaints against the police: A randomized controlled trial. Journal of Quantitative Criminology, 31(3), 509–535.

Banks

G. C.

Kepes

McDaniel

M. A.

(2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20(2), 182–196.

Barnes

J. C.

TenEyck

M. F.

Pratt

T. C.

Cullen

F. T.

(2020). How powerful is the evidence in criminology? On whether we should fear a coming crisis of confidence. Justice Quarterly, 37(3), 383–409. https://doi.org/10.1080/07418825.2018.1495252

Barranco

R. E.

Jennings

W. G.

May

D. C.

Wells

M. J.

(2016). What journals are the most cited journals in criminology and criminal justice’s “big three” journals? Journal of Criminal Justice Education, 27(1), 19–34. https://doi.org/10.1080/10511253.2015.1065337

Bartels

(2017). Swift, certain and fair: Does Project HOPE provide a therapeutic paradigm for managing offenders? MacMillan.

Begg

C. B.

(1994). Publication bias. In Cooper

Hedges

L. V.

(Eds.), The handbook of research synthesis (pp. 399–409). Russell SAGE Foundation.

Bonner

DeLisi

Jones-Johnson

Caudill

J. W.

Trulson

(2020). Chaotic homes, adverse childhood experiences, and serious delinquency: Differential effects by race and ethnicity. Justice Quarterly, 37(4), 697–714. https://doi.org/10.1080/07418825.2019.1688852

Bozarth

J. D.

Roberts

R. R.

(1972). Signifying significant significance. American Psychologist, 27(8), 774–775.

10.

Cairo

A. H.

Green

J. D.

Forsyth

D. R.

Behler

A. M. C.

Raldiris

T. L.

(2020). Gray (literature) matters: Evidence of selective hypothesis reporting in social psychological research. Personality and Social Psychology Bulletin, 46(9), 1344–1362. https://doi.org/10.1177/0146167220903896

11.

Cao

(2020). Criminology qua criminal justice as an open discipline: On the relationship between sociology and criminology in the USA. Journal of Criminal Justice Education, 31(4), 509–522.

12.

Carnegie Classification of Institutions of Higher Education. (n.d.). About the Carnegie classification. http://carnegieclassifications.iu.edu/

13.

Chalmers

Dickersin

(2013). Biased under-reporting of research reflects biased under-submission more than biased editorial rejection. F1000Research, 2(1), 1–6.

14.

Chan

A.-W.

Altman

D. G.

(2005). Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. BMJ, 330, 753. https://doi.org/10.1136/bmj.38356.424606.8f

15.

Clear

T. R.

(2001). Has academic criminal justice come of age? Justice Quarterly, 18(4), 709–726.

16.

Coburn

K. M.

Vevea

J. L.

(2015). Publication bias as a function of study characteristics. Psychological Methods, 20(3), 310–330.

17.

Cohn

E. G.

Farrington

D. P.

Iratzoqui

(2020). Changes in scholarly influence in major American criminology and criminal justice journals between 1986 and 2015. Journal of Criminal Justice Education, 31(4), 580–608. https://doi.org/10.1080/10511253.2020.1830140

18.

Cook

P. J.

(2016). Behavioral science critique of HOPE. Criminology & Public Policy, 15(4), 1155–1161.

19.

Copes

Beaton

Ayeni

Dabney

Tewksbury

(2020). A content analysis of qualitative research published in top criminology and criminal justice journals from 2010 to 2019. American Journal of Criminal Justice, 45(6), 1060–1079.

20.

Crow

M. S.

Smykla

J. O.

(2013). A mixed methods analysis of methodological orientation in national and regional criminology and criminal justice journals. Journal of Criminal Justice Education, 24(4), 536–555. https://doi.org/10.1080/10511253.2013.827229

21.

Crow

M. S.

Smykla

J. O.

(2015). An examination of author characteristics in national and regional criminology and criminal justice journals, 2008–2010: Are female scholars changing the nature of publishing in criminology and criminal justice? American Journal of Criminal Justice, 40(2), 441–455. https://doi.org/10.1007/s12103-014-9250-x

22.

Cullen

F. T.

Pratt

T. C.

Turanovic

J. J.

(2016). It’s hopeless: Beyond zero-tolerance supervision. Criminology & Public Policy, 15(4), 1215–1227.

23.

Cullen

F. T.

Smith

Lowenkamp

C. T.

Latessa

E. J.

(2009). Nothing works revisited: Deconstructing Farabee’s rethinking rehabilitation. Victims & Offenders, 4(2), 101–123.

24.

DeJong

St. George

(2018). Measuring journal prestige in criminal justice and criminology. Journal of Criminal Justice Education, 29(2), 290–309. https://doi.org/10.1080/10511253.2017.1398344

25.

Dickersin

(2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In Rothstein

Sutton

A. J.

Borenstein

(Eds.), Publication bias in meta analysis: Prevention, assessment and adjustments (pp. 11–34). Wiley.

26.

Dickersin

Min

Y. I.

(1993). Publication bias: The problem that won’t go away. Annals of the New York Academy of Sciences, 703(1), 135–148. https://doi.org/10.1111/j.1749-6632.1993.tb26343.x

27.

Dowdy

(1994). Federal funding and its effect on criminological research: Emphasizing individualistic explanations for criminal behavior. The American Sociologist, 25(4), 77–89. https://doi.org/10.1007/bf02691991

28.

Dwan

Gamble

Williamson

P. R.

Kirkham

J. J.

(2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—An updated review. PLoS One, 8(7), e66844. https://doi.org/10.1371/journal.pone.0066844

29.

Easterbrook

P. J.

Gopalan

Berlin

J. A.

Matthews

D. R.

(1991). Publication bias in clinical research. Lancet, 337(8746), 867–872. https://doi.org/10.1016/0140-6736(91)90201-y

30.

Editor. (2019). Look out for potential bias in chemical data sets. Nature, 573, 164.

31.

Eichelsheim

V. I.

Blokland

Meeus

W. H. J.

Branje

S. J. T.

(2019). Adolescent delinquency and family conflict as precursors of romantic relationships in early adulthood. Crime and Delinquency, 65(10), 1371–1401. https://doi.org/10.1177/0011128718781313

32.

Fanelli

(2010a). Do pressures to publish increase scientists’ bias? An empirical support from US states data. PLoS One, 5(4), e10271. https://doi.org/10.1371/journal.pone.0010271

33.

Fanelli

(2010b). “Positive” results increase down the hierarchy of the sciences. PLoS One, 5(4), e10068. https://doi.org/10.1371/journal.pone.0010068

34.

Fanelli

(2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90(3), 891–904. https://doi.org/10.1007/s11192-011-0494-7

35.

Finckenauer

J. O.

Laufer

W. S.

(1996). The second generation of American criminal justice. Journal of Research in Crime and Delinquency, 33(1), 5–15.

36.

Franco

Malhotra

Simonovits

(2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484

37.

Frost

N. A.

Clear

T. R.

(2007). Doctoral education in criminology and criminal justice. Journal of Criminal Justice Education, 18(1), 35–52.

38.

Gerber

A. S.

Malhotra

(2008a). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3(3), 313–326. https://doi.org/10.1561/100.00008024

39.

Gerber

A. S.

Malhotra

(2008b). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results? Sociological Methods & Research, 37(1), 3–30. https://doi.org/10.1177/0049124108318973

40.

Ginsel

Aggarwal

Xuan

Harris

(2015). The distribution of probability values in medical abstracts: An observational study. BMC Research Notes, 8(1), 721. https://doi.org/10.1186/s13104-015-1691-x

41.

Gottfredson

M. R.

Hirschi

(1990). A general theory of crime. Stanford University Press.

42.

Greenwald

A. G.

(1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20.

43.

Gulbrandsen

Smeby

J. C.

(2005). Industry funding and university professors’ research performance. Research Policy, 34(6), 932–950.

44.

Hawken

Kleiman

M. A. R.

(2009). Managing drug involved probationers with swift and certain sanctions: Evaluating Hawaii’s HOPE. U.S. Department of Justice, National Institute of Justice.

45.

Helfgott

J. B.

Parkin

W. S.

Fisher

Diaz

(2020). Misdemeanor arrests and community perceptions of fear of crime in Seattle. Criminal Justice Journal, 69, 1–19. https://doi.org/10.1016/j.jcrimjus.2020.101695

46.

Holmes

M. D.

Taggart

W. A.

(1990). A comparative analysis of research methods in criminology and criminal justice journals. Justice Quarterly, 7(2), 421–437.

47.

Jacob

B. A.

Lefgren

(2011). The impact of research grant funding on scientific productivity. Journal of Public Economics, 95(9–10), 1168–1177. https://doi.org/10.1016/j.jpubeco.2011.05.005

48.

Kaiser

K. A.

Pratt

T. C.

(2016). Crystal ball or moneyball: Does publishing success during graduate school predict career publication productivity? Journal of Crime and Justice, 39(3), 438–454. https://doi.org/10.1080/0735648x.2014.992801

49.

Katz

C. M.

Choate

D. E.

Ready

J. R.

Nuño

(2014). Evaluating the impact of officer worn body cameras in the Phoenix police department. Center for Violence Prevention & Community Safety, Arizona State University.

50.

Kim

(2022). Publication bias: A “bird’s-eye view” of meta-analytic practice in criminology and criminal justice. Criminal Justice Journal, 78, 1–13.

51.

Kleck

Barnes

J. C.

(2011). Article productivity among the faculty of criminology and criminal justice doctoral programs (2005–2009). Journal of Criminal Justice Education, 22(1), 43–66. https://doi.org/10.1080/10511253.2010.517650

52.

Kosar

Scott

D. W.

(2018). Examining the Carnegie Classification methodology for research universities. Statistics and Public Policy, 5(1), 1–12.

53.

Krimsky

(2013). Do financial conflicts of interest bias research? An inquiry into the “funding effect” hypothesis. Science Technology & Human Values, 38(4), 566–587. https://doi.org/10.1177/0162243912456271

54.

Laub

J. H.

(2004). The life course of criminology in the United States: The American Society of Criminology 2003 presidential address. Criminology, 42(1), 1–26.

55.

Lilienfeld

S. O.

Waldman

I. D.

(2017). Psychological science under scrutiny: Recent challenges and proposed solutions. Wiley.

56.

Lortie

C. J.

Dyer

A. R.

(1999). Over-interpretation: Avoiding the stigma of non-significant results. Oikos, 87(1), 183. https://doi.org/10.2307/3547010

57.

Lum

Koper

C. S.

Wilson

D. B.

Stoltz

Goodier

Eggins

Higginson

Mazerolle

(2020). Body-worn cameras’ effects on police officers and citizen behavior: A systematic review. Campbell Systematic Reviews, 16, e1112. https://doi.org/10.1002/cl2.1112

58.

Marks-Anglin

Chen

(2020). A historical review of publication bias. Research Synthesis Methods, 11(6), 725–742.

59.

May

(2021). Bias in science: Natural and social. Synthese, 199, 3345–3366. https://doi.org/10.1007/s11229-020-02937-0

60.

McCord

(2003). Cures that harm: Unanticipated outcomes of crime prevention programs. The Annals of the American Academy of Political and Social Science, 587(1), 16–30. https://doi.org/10.1177/0002716202250781

61.

Miller

Toliver

Forum

P. E. R.

(2014). Implementing a body-worn camera program: Recommendations and lessons learned. Office of Community Oriented Policing Services.

62.

Mowen

T. J.

Boman

J. H.

(2019). Do we have it all wrong? The protective roles of peers and criminogenic risks from family during prison reentry. Crime and Delinquency, 65(5), 681–704. https://doi.org/10.1177/00111287188002

63.

Nagin

D. S.

(2016). Project HOPE: Does it work? Criminology & Public Policy, 15(4), 1005–1007.

64.

Neuendorf

K. A.

(2017). The content analysis guidebook (2nd ed.). SAGE.

65.

Nuijten

M. B.

Hartgerink

C. H.

van Assen

M. A.

Epskamp

Wicherts

J. M.

(2016). The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods, 48(4), 1205–1226. https://doi.org/10.3758/s13428-015-0664-2

66.

ODS Consulting. (2011). Body worn video projects in paisley and Aberdeen: Self-evaluation. Author.

67.

Owens

Mann

McKenna

(2014). The Essex body worn video trial: The impact of body worn video on criminal justice outcomes of domestic abuse incidents. College of Policing.

68.

Pautasso

(2010). Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics, 85(1), 193–202. https://doi.org/10.1007/s11192-010-0233-5

69.

Perneger

T. V.

Combescure

(2017). The distribution of P-values in medical research articles suggested selective reporting associated with statistical significance. Journal of Clinical Epidemiology, 87, 70–77. https://doi.org/10.1016/j.jclinepi.2017.04.003

70.

Police and Crime Standards Directorate. (2007). Guidance for the police use of body-worn video devices. U.K. Home Office.

71.

Pridemore

W. A.

Makel

M. C.

Plucker

J. A.

(2018). Replication in criminology and the social sciences. Annual Review of Criminology, 1(1), 19–38. https://doi.org/10.1146/annurev-criminol-032317-091849

72.

Rankin

(2013). End of program evaluation and recommendations: On-officer body camera system. Mesa Police Department.

73.

Rosenthal

(1979). The ‘file drawer problem’ and tolerance for null results. Psychological Bulletin, 86(3), 638–641.

74.

Rothstein

H. R.

(2008). Publication bias as a threat to the validity of meta-analytic results. Journal of Experimental Criminology, 4(1), 61–81. https://doi.org/10.1007/s11292-007-9046-9

75.

Royal Society of Chemistry. (2019). Is publishing in the chemical sciences gender biased? Thomas Graham House.

76.

Savelsberg

J. J.

Cleveland

L. L.

King

R. D.

(2004). Institutional environments and scholarly work: American criminology, 1951-1993. Social Forces, 82(4), 1275–1302.

77.

Savelsberg

J. J.

Flood

S. M.

(2004). Criminological knowledge: Period and cohort effects in scholarship. Criminology, 42(4), 1009–1042.

78.

Scheel

A. M.

Schijen

M. R. M. J.

Lakens

(2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 1–12. https://doi.org/10.1177/25152459211007467

79.

Scherer

R. W.

Dickersin

Langenberg

(1994). Full publication of results initially presented in abstracts: A meta-analysis. The Journal of the American Medical Association, 272(2), 158–162. https://doi.org/10.1001/jama.1994.03520020084025

80.

Senn

(2012). Misunderstanding publication bias: Editors are not blameless after all. F1000Research, 1(59), 1–5.

81.

Smart

R. G.

(1964). The importance of negative results in psychological research. Canadian Psychologist, 5(4), 225–232.

82.

Sterling

T. D.

(1959). Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. Journal of the American Statistical Association, 54, 30–34.

83.

West

M. P.

Rorie

Cohen

M. A.

(2020). The “pliability” of criminological analyses: Assessing bias in regression estimates using Monte Carlo simulations. Journal of Quantitative Criminology, 36(2), 371–394. https://doi.org/10.1007/s10940-018-9398-5

84.

Wooditch

Sloas

L. B.

Key

(2020). Outcome reporting bias in randomized experiments on substance use disorders. Journal of Quantitative Criminology, 36(2), 273–293. https://doi.org/10.1007/s10940-018-9388-7

85.

Woodward

V. H.

Webb

M. E.

Griffin

O. H.

Copes

(2016). The current state of criminological research in the United States: An examination of research methodologies in criminology and criminal justice journals. Journal of Criminal Justice Education, 27(3), 340–361. https://doi.org/10.1080/10511253.2015.1131312

86.

Young

(2014). Implementation of a randomized controlled trial in Ventura, California: A body-worn video camera experiment [Unpublished master’s thesis]. University of Cambridge.