Three Changes Public Health Scientists Can Make to Help Build a Culture of Reproducible Research

In July 2018, the New England Journal of Medicine identified problems with randomization of study participants in the retraction of a foundational study associating the Mediterranean diet ¹ with prevention of cardiovascular disease. ² Although news of this retraction was widespread, the article was just 1 of the 500-600 articles retracted each year from peer-reviewed journals. ³ Sustaining high-quality research in fields such as public health is important for protecting and improving human health and well-being, but scientific progress and public trust have weakened with the growing rate of article retractions ⁴ and reports of research containing errors, omissions, or fraud. ^{5 -7}

Researchers in psychology and biomedicine have been the most active in examining research quality. Studies of psychology researchers found that most admit to using at least 1 questionable research practice. A questionable research practice is a choice made during data management, analysis, or reporting that appears incorrect or dishonest and results in more favorable research findings such as statistical significance. ^8,9 Use of questionable research practices may have contributed to successful replication of only 40%-60% of 100 articles published during 2008 in psychology. ^10,11 Likewise, a 2011 study found that just 14 of 67 (21%) drug studies were successfully replicated. ⁷ Although there is little direct examination of the quality of public health research, a study published in 2018 by the lead author (J.K.H.) attempted to reproduce results from 6 published articles and found the use of questionable research practices and numerous errors in reporting. ⁵ Low-quality research has serious consequences for human health; 1 project found 400 000 human participants were enrolled and 70 501 were treated in medical studies that were later retracted. ¹² Adopting reproducible research practices can help improve research quality.

Replication—or conducting a study again to verify its results—provides the strongest means to ensure research quality and to validate findings. Reproducing a study—or analyzing an existing data source to produce the same results—is an alternative when replication is not feasible. Reproducing research does not ensure original analyses were correct, but the process can improve research quality by verifying findings and identifying weaknesses, questionable research practices, and errors in in data management, analyses, and reporting.

Research is reproducible when data are shared and either (1) the statistical code is shared or (2) detailed, clear, and complete instructions for analysis are accessible. ¹³ Using reproducible research practices can increase the pace of scientific discovery, promote greater exchange of ideas and development of new collaborations among scientists, and improve the visibility of scientific contributions. ^14,15 Research articles with shared data or shared code are cited more often than articles without shared data or code, and articles with shared data have fewer errors than articles without shared data. ¹⁵ Data sources that are shared are used in more publications than data sources that are not shared. ¹⁶

Publications with the necessary elements for reproducibility appear rare in public health. ^5,14 Widespread use of reproducible research practices in public health will require a sustained and deliberate effort from stakeholders across the field. Numerous researchers and organizations, many from psychology, have suggested guidelines for improving research quality. ^{17 -20} For researchers, trying to adopt all these recommended reproducible research practices may feel like trying to eat an elephant. That is, the task seems impossibly large because of barriers such as data privacy policies, journal word limits, and time constraints. Add to this the pressure to publish or secure external funding for promotion and tenure, and the task may seem even more overwhelming. We suggest researchers take small bites and start with 3 changes to their research practices ²¹ : (1) use existing coding guidelines (eg, the Google R Style Guide, ²² Best Practice Programming Techniques for SAS Users ²³ ) to write clear, well-documented code; (2) publicly share the statistical code—or include adequate details about methodology in the main text or supplemental materials—for each published article; and (3) publicly share original, de-identified, or simulated data whenever possible.

Although our focus here is on quantitative research, researchers are increasingly making parallel calls for enhancing transparency and reproducibility in qualitative and mixed-methods research. ²⁴ The 3 suggestions apply to qualitative work with small modifications. Specifically, sharing de-identified data and detailed documentation could increase reproducibility for qualitative research. ²⁵ However, given the more subjective and iterative nature of qualitative data collection, analysis, and interpretation, tasks such as data de-identification and development of sufficiently detailed documentation are not as straightforward as for quantitative research and can require excessive time and effort. ^20,26 Given the role of qualitative and mixed-methods research in improving our understanding of public health problems and their solutions, ²⁶ additional work on development of strategies for reproducible qualitative research is needed to identify more feasible strategies.

Along with researchers, journals and funders can contribute to the culture shift by rewarding ²⁷ or requiring the use of reproducible research practices, such as shared data and code, as a condition for publishing or funding. For journals publishing qualitative and mixed-methods research, modifications of journal data-sharing policies should be considered to make them more appropriate for facilitating sharing for all data types. ²⁵ Employers might consider acknowledging, incentivizing, or rewarding researchers who adopt reproducible research practices. Public health training programs and program accreditors can build reproducible research practices into the curriculum so that the next generation formats and shares its data and code in reproducible ways by default.

Improving the quality of reported research through the use of reproducible research practices could hasten scientific progress, increase public trust in public health evidence, and increase the positive impact of this evidence on human health. Three small steps for researchers could contribute to 1 giant leap for public health.