Objective: The aim of this systematic review is to estimate: (i) the overall effect of blinding models on bias; (ii) the effect of each blinding model; and (iii) the effect of un-blinding on reviewer's accountability in biomedical research proposals. Methods: Systematic review of prospective or retrospective comparative studies that evaluated two or more peer review blinding models for biomedical research proposals/funding applications and reported outcomes related to peer review efficiency. Results: Three studies that met the inclusion criteria were included in this review and assessed using the QualSyst tool by two authors. Conclusion: Our systematic review is the first to assess peer review blinding models in the context of funding. While only three studies were included, this highlighted the dire need for further RCTs that generate validated evidence. We also discussed multiple aspects of peer review, such as peer review in manuscripts vs proposals and peer review in other fields.
AlbertsB.KirschnerM. W.TilghmanS.VarmusH. (2014). Rescuing US biomedical research from its systemic flaws. Proceedings of the National Academy of Sciences of the United States of America, 111(16), 5773–5777. https://doi.org/10.1073/pnas.1404402111
2.
BaggsJ. G.BroomeM. E.DoughertyM. C.FredaM. C.KearneyM. H. (2008). Blinding in peer review: The preferences of reviewers for nursing journals. Journal of Advanced Nursing, 64(2), 131–138. https://doi.org/10.1111/j.1365-2648.2008.04816.x
3.
BenosD. J.BashariE.ChavesJ. M.GaggarA.KapoorN.LaFranceM.MansR.MayhewD.McGowanS.PolterA.QadriY.SarfareS.SchultzK.SplittgerberR.StephensonJ.TowerC.WaltonR. G.ZotovA. (2007). The ups and downs of peer review. Advances in Physiology Education, 31(2), 145–152. https://doi.org/10.1152/advan.00104.2006
4.
BlankR. M. (1991). The effects of double-blind versus single-blind reviewing: Experimental evidence from the American economic review. American Economic Review, 81(5), 1041–1067. http://www.jstor.org/stable/2006906
5.
BruceR.ChauvinA.TrinquartL.RavaudP.BoutronI. (2016). Impact of interventions to improve the quality of peer review of biomedical journals: A systematic review and meta-analysis. BMC Medicine, 14(1), 85. https://doi.org/10.1186/s12916-016-0631-5
6.
BuddenA. E.TregenzaT.AarssenL. W.KorichevaJ.LeimuR.LortieC. J. (2008). Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution, 23(1), 4–6. https://doi.org/10.1016/j.tree.2007.07.008
7.
Canadian Institutes of Health Research (2018). Peer review manual - foundation. Canadian Institutes of Health Research. https://cihr-irsc.gc.ca/e/48486.html
8.
CheccoA.BraccialeL.LoretiP.PinfieldS.BianchiG. (2021). AI-assisted peer review. Humanities and Social Sciences Communications, 8(1). https://doi.org/10.1057/s41599-020-00703-8
CicchettiD. V. (1991). The reliability of peer review for manuscript and grant submissions: A cross-disciplinary investigation. Behavioral and Brain Sciences, 14(1), 119–135. https://doi.org/10.1017/S0140525X00065675
DemicheliV.Di PietrantonjC. (2007). Peer review for improving the quality of grant applications. The Cochrane Database of Systematic Reviews, 2007(2), Mr000003. https://doi.org/10.1002/14651858.MR000003.pub2
HorbachS. P.HalffmanW. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review, 3(1), 1–15. https://doi.org/10.1186/s41073-018-0051-5
20.
HorrobinD. F. (1974). Referees and research administrators: Barriers to scientific research?British Medical Journal, 2(5912), 216–218. https://doi.org/10.1136/bmj.2.5912.216
HrenD.PinaD. G.NormanC. R.MarušićA. (2022). What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports. Journal of Informetrics, 16(2), 101289. https://doi.org/10.1016/j.joi.2022.101289
24.
HugS. E.HołowieckiM.MaL.AeschbachM.OchsnerM. (2020). Practices of peer review in the SSH I: A systematic review of peer review criteria. In OchsnerM.Kancewicz-HoffmanN.HołowieckiM.,, & HolmJ. (Eds.), Overview of peer review practices in the SSH: ENRESSH report (pp. 61–66). European Network for Research Evaluation in the Social Sciences and Humanities. https://serval.unil.ch/resource/serval:BIB_A5B1EBD0B51E.P001/REF.pdf
25.
JadadA. R.MooreR. A.CarrollD.JenkinsonC.ReynoldsD. J.GavaghanD. J.McQuayH. J. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary?Controlled Clinical Trials, 17(1), 1–12. https://doi.org/10.1016/0197-2456(95)00134-4
26.
JagsiR.BennettK. E.GriffithK. A.DeCastroR.GraceC.HollidayE.ZietmanA. L. (2014). Attitudes toward blinding of peer review and perceptions of efficacy within a small biomedical specialty. International Journal of Radiation Oncology, Biology, Physics, 89(5), 940–946. https://doi.org/10.1016/j.ijrobp.2014.04.021
KarhulahtiV. M.BackeH. J. (2021). Transparency of peer review: A semi-structured interview study with chief editors from social sciences and humanities. Research Integrity and Peer Review, 6(1), 13. https://doi.org/10.1186/s41073-021-00116-4
29.
KatzD. S.ProtoA. V.OlmstedW. W. (2002). Incidence and nature of unblinding by authors: Our experience at two radiology journals with double-blinded peer review policies. American Journal of Roentgenology, 179(6), 1415–1417. https://doi.org/10.2214/ajr.179.6.1791415
LeeC. J.SugimotoC. R.ZhangG.CroninB. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1), 2–17. https://doi.org/10.1002/asi.22784
33.
LeeM.OmK.KohJ. (2000). The bias of sighted reviewers in research proposal evaluation: A comparative analysis of blind and open review in Korea. Scientometrics, 48(1), 99–116. https://doi.org/10.1023/A:1005636503358
34.
MayoN. E.BrophyJ.GoldbergM. S.KleinM. B.MillerS.PlattR. W.RitchieJ. (2006). Peering at peer review revealed high degree of chance associated with funding of grant applications. Journal of Clinical Epidemiology, 59(8), 842–848. https://doi.org/10.1016/j.jclinepi.2005.12.007
35.
McGowanJ.SampsonM.SalzwedelD. M.CogoE.FoersterV.LefebvreC. (2016). PRESS peer review of electronic search strategies: 2015 guideline explanation and elaboration (PRESS E&E). Canadian Agency for Drugs and Technologies in Health. https://www.cadth.ca/press-peer-review-electronic-search-strategies-0
36.
McNuttR. A.EvansA. T.FletcherR. H.FletcherS. W. (1990). The effects of blinding on the quality of peer review. A randomized trial. JAMA, 263(10), 1371–1376. https://doi.org/10.1001/jama.1990.03440100079012
MeirmansS.ButlinR. K.CharmantierA.EngelstädterJ.GrootA. T.KingK. C.KokkoH.ReidJ. M.NeimanM. (2019). Science policies: How should science funding be allocated? An evolutionary biologists’ perspective. Journal of Evolutionary Biology, 32(8), 754–768. https://doi.org/10.1111/jeb.13497
39.
NakamuraR. K.MannL. S.LindnerM. D.BraithwaiteJ.ChenM. C.VanceaA.ByrnesN.DurrantV.ReedB. (2021). An experimental test of the effects of redacting grant applicant identifiers on peer review outcomes. eLife, 10, https://doi.org/10.7554/eLife.71368
OkikeK.HugK. T.KocherM. S.LeopoldS. S. (2016). Single-blind vs double-blind peer review in the setting of author prestige. JAMA, 316(12), 1315–1316. https://doi.org/10.1001/jama.2016.11014
42.
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., McGuinness, L. A., Stewart, L. A., Thomas, J., Tricco, A. C., Welch, V. A., Whiting, P., & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ (Clinical Research ed.), 372(8284), n71. https://doi.org/10.1136/bmj.n71
43.
Perez VelazquezJ. L. (2019). The tragicomedy of peer review—the publication game and the lottery of grants. In The rise of the scientist-bureaucrat (pp. 67–83). Springer. https://doi.org/10.1007/978-3-030-12326-0_5
44.
PeruginelliG.Sanz-CasadoE.StojanovskiJ. (2020). Practices of peer review in the SSH III: Peer review in the legal domain. Three parallel case studies in Italy, Spain and Croatia. In OchsnerM.Kancewicz-HoffmanN.HołowieckiM.HolmJ. (Eds.), Overview of peer review practices in the SSH: ENRESSH report (pp. 71–77). European Network for Research Evaluation in the Social Sciences and Humanities. https://serval.unil.ch/resource/serval:BIB_A5B1EBD0B51E.P001/REF.pdf
45.
PinaD. G.HrenD.MarušićA. (2015). Peer review evaluation process of Marie Curie actions under EU's seventh framework programme for research. PloS One, 10(6), e0130753. https://doi.org/10.1371/journal.pone.0130753
46.
Recio-SaucedoA.CraneK.MeadmoreK.FackrellK.ChurchH.FraserS.Blatch-JonesA. (2022). What works for peer review and decision-making in research funding: A realist synthesis. Research Integrity and Peer Review, 7(1), 1–28. https://doi.org/10.1186/s41073-022-00120-2
47.
SantosA.MorrisD. S.RattanR.ZakrisonT. (2021). Double-blinded manuscript review: Avoiding peer review bias. The Journal of Trauma and Acute Care Surgery, 91(1), e39–e42. https://doi.org/10.1097/ta.0000000000003260
48.
Serrano VelardeK. (2018). The way we ask for money… the emergence and institutionalization of grant writing practices in academia. Minerva, 56(1), 85–107. https://doi.org/10.1007/s11024-018-9346-4
ShepherdJ.FramptonG. K.PickettK.WyattJ. C. (2018). Peer review of health research funding proposals: A systematic map and systematic review of innovations for effectiveness and efficiency. PloS One, 13(5), e0196914. https://doi.org/10.1371/journal.pone.0196914
51.
SmithR. (2006). Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99(4), 178–182. https://doi.org/10.1258/jrsm.99.4.178
52.
SnodgrassR. (2006). Single- versus double-blind reviewing: An analysis of the literature. SIGMOD Record, 35(3), 8–21. https://doi.org/10.1145/1168092.1168094
53.
Solans-DomènechM.GuillamónI.RiberaA.Ferreira-GonzálezI.CarrionC.Permanyer-MiraldaG.PonsJ. M. V. (2017). Blinding applicants in a first-stage peer-review process of biomedical research grants: An observational study. Research Evaluation, 26(3), 181–189. https://doi.org/10.1093/reseval/rvx021
54.
TomkinsA.ZhangM.HeavlinW. D. (2017). Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences of the United States of America, 114(48), 12708–12713. https://doi.org/10.1073/pnas.1707323114
55.
VaesenK.KatzavJ. (2017). How much would each researcher receive if competitive government research funding were distributed equally among researchers?PloS One, 12(9), e0183967. https://doi.org/10.1371/journal.pone.0183967
van RooyenS.GodleeF.EvansS.SmithR.BlackN. (1998). Effect of blinding and unmasking on the quality of peer review: A randomized trial. JAMA, 280(3), 234–237. https://doi.org/10.1001/jama.280.3.234
58.
VisscherP. M.YengoL. (2023). The effect of the scale of grant scoring on ranking accuracy [version 2; peer review: 2 approved]. F1000Research, 11, 1197. https://doi.org/10.12688/f1000research.125400.2
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
