Analysis of scholarship in the physical, biological, and social sciences has discovered that peer-reviewed journals publish a much larger proportion of articles with statistically significant findings compared to articles with null results. Publication bias in criminology and criminal justice (CCJ) has received very little attention, however. The current study is an exploratory analysis of research in leading CCJ journals across 2 years to determine the current state of null findings in contemporary CCJ scholarship. Our findings are consistent with studies in other disciplines; null results are rare in leading CCJ journals. We explore the context of our findings, outline the importance of examining publication bias to improve CCJ research and better inform policy, and discuss the limitations of our approach.
AhlinE. M. (2020). Broadening the productivity lens in criminology and criminal justice: An exploratory study of the research contributions of master’s degree program faculty, 2014–2018. Journal of Criminal Justice Education, 31(1), 100–123.
2.
ArielB.FarrarW. A.SutherlandA. (2015). The effect of police body-worn cameras on use of force and citizens’ complaints against the police: A randomized controlled trial. Journal of Quantitative Criminology, 31(3), 509–535.
3.
BanksG. C.KepesS.McDanielM. A. (2012). Publication bias: A call for improved meta-analytic practice in the organizational sciences. International Journal of Selection and Assessment, 20(2), 182–196.
4.
BarnesJ. C.TenEyckM. F.PrattT. C.CullenF. T. (2020). How powerful is the evidence in criminology? On whether we should fear a coming crisis of confidence. Justice Quarterly, 37(3), 383–409. https://doi.org/10.1080/07418825.2018.1495252
5.
BarrancoR. E.JenningsW. G.MayD. C.WellsM. J. (2016). What journals are the most cited journals in criminology and criminal justice’s “big three” journals?Journal of Criminal Justice Education, 27(1), 19–34. https://doi.org/10.1080/10511253.2015.1065337
6.
BartelsL. (2017). Swift, certain and fair: Does Project HOPE provide a therapeutic paradigm for managing offenders?MacMillan.
7.
BeggC. B. (1994). Publication bias. In CooperH.HedgesL. V. (Eds.), The handbook of research synthesis (pp. 399–409). Russell SAGE Foundation.
8.
BonnerT.DeLisiM.Jones-JohnsonG.CaudillJ. W.TrulsonC. (2020). Chaotic homes, adverse childhood experiences, and serious delinquency: Differential effects by race and ethnicity. Justice Quarterly, 37(4), 697–714. https://doi.org/10.1080/07418825.2019.1688852
9.
BozarthJ. D.RobertsR. R. (1972). Signifying significant significance. American Psychologist, 27(8), 774–775.
10.
CairoA. H.GreenJ. D.ForsythD. R.BehlerA. M. C.RaldirisT. L. (2020). Gray (literature) matters: Evidence of selective hypothesis reporting in social psychological research. Personality and Social Psychology Bulletin, 46(9), 1344–1362. https://doi.org/10.1177/0146167220903896
11.
CaoL. (2020). Criminology qua criminal justice as an open discipline: On the relationship between sociology and criminology in the USA. Journal of Criminal Justice Education, 31(4), 509–522.
ChalmersI.DickersinK. (2013). Biased under-reporting of research reflects biased under-submission more than biased editorial rejection. F1000Research, 2(1), 1–6.
14.
ChanA.-W.AltmanD. G. (2005). Identifying outcome reporting bias in randomised trials on PubMed: Review of publications and survey of authors. BMJ, 330, 753. https://doi.org/10.1136/bmj.38356.424606.8f
15.
ClearT. R. (2001). Has academic criminal justice come of age?Justice Quarterly, 18(4), 709–726.
16.
CoburnK. M.VeveaJ. L. (2015). Publication bias as a function of study characteristics. Psychological Methods, 20(3), 310–330.
17.
CohnE. G.FarringtonD. P.IratzoquiA. (2020). Changes in scholarly influence in major American criminology and criminal justice journals between 1986 and 2015. Journal of Criminal Justice Education, 31(4), 580–608. https://doi.org/10.1080/10511253.2020.1830140
18.
CookP. J. (2016). Behavioral science critique of HOPE. Criminology & Public Policy, 15(4), 1155–1161.
19.
CopesH.BeatonB.AyeniD.DabneyD.TewksburyR. (2020). A content analysis of qualitative research published in top criminology and criminal justice journals from 2010 to 2019. American Journal of Criminal Justice, 45(6), 1060–1079.
20.
CrowM. S.SmyklaJ. O. (2013). A mixed methods analysis of methodological orientation in national and regional criminology and criminal justice journals. Journal of Criminal Justice Education, 24(4), 536–555. https://doi.org/10.1080/10511253.2013.827229
21.
CrowM. S.SmyklaJ. O. (2015). An examination of author characteristics in national and regional criminology and criminal justice journals, 2008–2010: Are female scholars changing the nature of publishing in criminology and criminal justice?American Journal of Criminal Justice, 40(2), 441–455. https://doi.org/10.1007/s12103-014-9250-x
22.
CullenF. T.PrattT. C.TuranovicJ. J. (2016). It’s hopeless: Beyond zero-tolerance supervision. Criminology & Public Policy, 15(4), 1215–1227.
23.
CullenF. T.SmithP.LowenkampC. T.LatessaE. J. (2009). Nothing works revisited: Deconstructing Farabee’s rethinking rehabilitation. Victims & Offenders, 4(2), 101–123.
24.
DeJongC.St. GeorgeS. (2018). Measuring journal prestige in criminal justice and criminology. Journal of Criminal Justice Education, 29(2), 290–309. https://doi.org/10.1080/10511253.2017.1398344
25.
DickersinK. (2005). Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm. In RothsteinH.SuttonA. J.BorensteinM. (Eds.), Publication bias in meta analysis: Prevention, assessment and adjustments (pp. 11–34). Wiley.
DowdyE. (1994). Federal funding and its effect on criminological research: Emphasizing individualistic explanations for criminal behavior. The American Sociologist, 25(4), 77–89. https://doi.org/10.1007/bf02691991
28.
DwanK.GambleC.WilliamsonP. R.KirkhamJ. J. (2013). Systematic review of the empirical evidence of study publication bias and outcome reporting bias—An updated review. PLoS One, 8(7), e66844. https://doi.org/10.1371/journal.pone.0066844
Editor. (2019). Look out for potential bias in chemical data sets. Nature, 573, 164.
31.
EichelsheimV. I.BloklandA.MeeusW. H. J.BranjeS. J. T. (2019). Adolescent delinquency and family conflict as precursors of romantic relationships in early adulthood. Crime and Delinquency, 65(10), 1371–1401. https://doi.org/10.1177/0011128718781313
32.
FanelliD. (2010a). Do pressures to publish increase scientists’ bias? An empirical support from US states data. PLoS One, 5(4), e10271. https://doi.org/10.1371/journal.pone.0010271
FinckenauerJ. O.LauferW. S. (1996). The second generation of American criminal justice. Journal of Research in Crime and Delinquency, 33(1), 5–15.
36.
FrancoA.MalhotraN.SimonovitsG. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. https://doi.org/10.1126/science.1255484
37.
FrostN. A.ClearT. R. (2007). Doctoral education in criminology and criminal justice. Journal of Criminal Justice Education, 18(1), 35–52.
38.
GerberA. S.MalhotraN. (2008a). Do statistical reporting standards affect what is published? Publication bias in two leading political science journals. Quarterly Journal of Political Science, 3(3), 313–326. https://doi.org/10.1561/100.00008024
39.
GerberA. S.MalhotraN. (2008b). Publication bias in empirical sociological research: Do arbitrary significance levels distort published results?Sociological Methods & Research, 37(1), 3–30. https://doi.org/10.1177/0049124108318973
40.
GinselB.AggarwalA.XuanW.HarrisI. (2015). The distribution of probability values in medical abstracts: An observational study. BMC Research Notes, 8(1), 721. https://doi.org/10.1186/s13104-015-1691-x
41.
GottfredsonM. R.HirschiT. (1990). A general theory of crime. Stanford University Press.
42.
GreenwaldA. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82(1), 1–20.
43.
GulbrandsenM.SmebyJ. C. (2005). Industry funding and university professors’ research performance. Research Policy, 34(6), 932–950.
44.
HawkenA.KleimanM. A. R. (2009). Managing drug involved probationers with swift and certain sanctions: Evaluating Hawaii’s HOPE. U.S. Department of Justice, National Institute of Justice.
45.
HelfgottJ. B.ParkinW. S.FisherC.DiazA. (2020). Misdemeanor arrests and community perceptions of fear of crime in Seattle. Criminal Justice Journal, 69, 1–19. https://doi.org/10.1016/j.jcrimjus.2020.101695
46.
HolmesM. D.TaggartW. A. (1990). A comparative analysis of research methods in criminology and criminal justice journals. Justice Quarterly, 7(2), 421–437.
47.
JacobB. A.LefgrenL. (2011). The impact of research grant funding on scientific productivity. Journal of Public Economics, 95(9–10), 1168–1177. https://doi.org/10.1016/j.jpubeco.2011.05.005
48.
KaiserK. A.PrattT. C. (2016). Crystal ball or moneyball: Does publishing success during graduate school predict career publication productivity?Journal of Crime and Justice, 39(3), 438–454. https://doi.org/10.1080/0735648x.2014.992801
49.
KatzC. M.ChoateD. E.ReadyJ. R.NuñoL. (2014). Evaluating the impact of officer worn body cameras in the Phoenix police department. Center for Violence Prevention & Community Safety, Arizona State University.
50.
KimB. (2022). Publication bias: A “bird’s-eye view” of meta-analytic practice in criminology and criminal justice. Criminal Justice Journal, 78, 1–13.
51.
KleckG.BarnesJ. C. (2011). Article productivity among the faculty of criminology and criminal justice doctoral programs (2005–2009). Journal of Criminal Justice Education, 22(1), 43–66. https://doi.org/10.1080/10511253.2010.517650
52.
KosarR.ScottD. W. (2018). Examining the Carnegie Classification methodology for research universities. Statistics and Public Policy, 5(1), 1–12.
53.
KrimskyS. (2013). Do financial conflicts of interest bias research? An inquiry into the “funding effect” hypothesis. Science Technology & Human Values, 38(4), 566–587. https://doi.org/10.1177/0162243912456271
54.
LaubJ. H. (2004). The life course of criminology in the United States: The American Society of Criminology 2003 presidential address. Criminology, 42(1), 1–26.
55.
LilienfeldS. O.WaldmanI. D. (2017). Psychological science under scrutiny: Recent challenges and proposed solutions. Wiley.
56.
LortieC. J.DyerA. R. (1999). Over-interpretation: Avoiding the stigma of non-significant results. Oikos, 87(1), 183. https://doi.org/10.2307/3547010
57.
LumC.KoperC. S.WilsonD. B.StoltzM.GoodierM.EgginsE.HigginsonA.MazerolleL. (2020). Body-worn cameras’ effects on police officers and citizen behavior: A systematic review. Campbell Systematic Reviews, 16, e1112. https://doi.org/10.1002/cl2.1112
58.
Marks-AnglinA.ChenY. (2020). A historical review of publication bias. Research Synthesis Methods, 11(6), 725–742.
McCordJ. (2003). Cures that harm: Unanticipated outcomes of crime prevention programs. The Annals of the American Academy of Political and Social Science, 587(1), 16–30. https://doi.org/10.1177/0002716202250781
61.
MillerL.ToliverJ.ForumP. E. R. (2014). Implementing a body-worn camera program: Recommendations and lessons learned. Office of Community Oriented Policing Services.
62.
MowenT. J.BomanJ. H. (2019). Do we have it all wrong? The protective roles of peers and criminogenic risks from family during prison reentry. Crime and Delinquency, 65(5), 681–704. https://doi.org/10.1177/00111287188002
63.
NaginD. S. (2016). Project HOPE: Does it work?Criminology & Public Policy, 15(4), 1005–1007.
64.
NeuendorfK. A. (2017). The content analysis guidebook (2nd ed.). SAGE.
65.
NuijtenM. B.HartgerinkC. H.van AssenM. A.EpskampS.WichertsJ. M. (2016). The prevalence of statistical reporting errors in psychology (1985-2013). Behavior Research Methods, 48(4), 1205–1226. https://doi.org/10.3758/s13428-015-0664-2
66.
ODS Consulting. (2011). Body worn video projects in paisley and Aberdeen: Self-evaluation. Author.
67.
OwensC.MannD.McKennaR. (2014). The Essex body worn video trial: The impact of body worn video on criminal justice outcomes of domestic abuse incidents. College of Policing.
68.
PautassoM. (2010). Worsening file-drawer problem in the abstracts of natural, medical and social science databases. Scientometrics, 85(1), 193–202. https://doi.org/10.1007/s11192-010-0233-5
69.
PernegerT. V.CombescureC. (2017). The distribution of P-values in medical research articles suggested selective reporting associated with statistical significance. Journal of Clinical Epidemiology, 87, 70–77. https://doi.org/10.1016/j.jclinepi.2017.04.003
70.
Police and Crime Standards Directorate. (2007). Guidance for the police use of body-worn video devices. U.K. Home Office.
RankinH. (2013). End of program evaluation and recommendations: On-officer body camera system. Mesa Police Department.
73.
RosenthalR. (1979). The ‘file drawer problem’ and tolerance for null results. Psychological Bulletin, 86(3), 638–641.
74.
RothsteinH. R. (2008). Publication bias as a threat to the validity of meta-analytic results. Journal of Experimental Criminology, 4(1), 61–81. https://doi.org/10.1007/s11292-007-9046-9
75.
Royal Society of Chemistry. (2019). Is publishing in the chemical sciences gender biased?Thomas Graham House.
76.
SavelsbergJ. J.ClevelandL. L.KingR. D. (2004). Institutional environments and scholarly work: American criminology, 1951-1993. Social Forces, 82(4), 1275–1302.
77.
SavelsbergJ. J.FloodS. M. (2004). Criminological knowledge: Period and cohort effects in scholarship. Criminology, 42(4), 1009–1042.
78.
ScheelA. M.SchijenM. R. M. J.LakensD. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 1–12. https://doi.org/10.1177/25152459211007467
79.
SchererR. W.DickersinK.LangenbergP. (1994). Full publication of results initially presented in abstracts: A meta-analysis. The Journal of the American Medical Association, 272(2), 158–162. https://doi.org/10.1001/jama.1994.03520020084025
80.
SennS. (2012). Misunderstanding publication bias: Editors are not blameless after all. F1000Research, 1(59), 1–5.
81.
SmartR. G. (1964). The importance of negative results in psychological research. Canadian Psychologist, 5(4), 225–232.
82.
SterlingT. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. Journal of the American Statistical Association, 54, 30–34.
83.
WestM. P.RorieM.CohenM. A. (2020). The “pliability” of criminological analyses: Assessing bias in regression estimates using Monte Carlo simulations. Journal of Quantitative Criminology, 36(2), 371–394. https://doi.org/10.1007/s10940-018-9398-5
84.
WooditchA.SloasL. B.WuX.KeyA. (2020). Outcome reporting bias in randomized experiments on substance use disorders. Journal of Quantitative Criminology, 36(2), 273–293. https://doi.org/10.1007/s10940-018-9388-7
85.
WoodwardV. H.WebbM. E.GriffinO. H.CopesH. (2016). The current state of criminological research in the United States: An examination of research methodologies in criminology and criminal justice journals. Journal of Criminal Justice Education, 27(3), 340–361. https://doi.org/10.1080/10511253.2015.1131312
86.
YoungJ. (2014). Implementation of a randomized controlled trial in Ventura, California: A body-worn video camera experiment [Unpublished master’s thesis]. University of Cambridge.
