Abstract
More than a decade of advocacy and policy reforms have attempted to increase the uptake of transparent research practices in the field of psychology; however, their collective impact is unclear. We estimated the prevalence of transparent research practices in (a) all psychology journals (i.e., field-wide), and (b) prominent psychology journals, by manually examining two random samples of 200 empirical articles (N = 400) published in 2022. Most articles had an open-access version (field-wide: 74%, 95% confidence interval [CI] = [67%, 79%]; prominent: 71% [64%, 77%]) and included a funding statement (field-wide: 76% [70%, 82%]; prominent: 76% [70%, 82%]) or conflict-of-interest statement (field-wide: 76% [70%, 82%]; prominent: 73% [67%, 79%]). Relatively few articles had a preregistration (field-wide: 7% [2.5%, 12%]; prominent: 14% [8.5%, 19%]), materials (field-wide: 16% [9%, 24%]; prominent: 19% [12%, 27%]), raw/primary data (field-wide: 14% [7%, 21%]; prominent: 16% [9.5%, 24%]), or analysis scripts (field-wide: 8.5% [4.5%, 13%]; prominent: 14% [9.5%, 19%]) that were immediately accessible without contacting authors or third parties. In conjunction with prior research, our results suggest transparency increased moderately from 2017 to 2022. Overall, despite considerable infrastructure improvements, bottom-up advocacy, and top-down policy initiatives, research transparency continues to be widely neglected in psychology.
Transparency is a core feature of an efficient, reproducible, and self-correcting scientific ecosystem (Ioannidis, 2012; Munafò et al., 2017; Vazire & Holcombe, 2022). However, in practice, transparent research practices are widely neglected across scientific disciplines (Hamilton et al., 2023; Hardwicke et al., 2020, 2022; Minocher et al., 2021; Serghiou et al., 2021; Towse et al., 2020). More than a decade of reform initiatives have attempted to increase the uptake of transparent research practices in psychology (Nelson et al., 2018), but their collective impact is unclear. The purpose of the present study was to obtain contemporary prevalence estimates for the adoption of transparent research practices in the field of psychology. These data will provide an empirical measure of progress and help to motivate and strategically design further efforts to improve the transparency of research in psychology.
In a previous study, we estimated the field-wide prevalence of seven transparent research practices by manually examining a random sample of psychology articles published between 2014 and 2017 (Hardwicke et al., 2022). Most articles had an open-access version (65%, 95% confidence interval [CI] = [59%, 71%]), and inclusion of funding (62% [56%, 69%]) and conflict-of-interest (39% [32%, 45%]) statements was quite common. However, we rarely observed use of preregistration (2% [1%, 4%]), sharing of materials (10% [6%, 15%]), sharing of raw/primary data (2% [0%, 3%]), or sharing of analysis scripts (1% [0%, 1%]). 1
There are reasons to think that transparency has improved since 2017, but it is unclear by how much. Repositories, such as OSF and AsPredicted, have reported a growing number of users sharing and preregistering aspects of their research (Nosek et al., 2022). In addition, some funders, journals, and other stakeholders have signaled that they are prepared to enact policies that encourage or require transparency (Nosek et al., 2015). A few psychology journals, for example, have adopted mandatory data-sharing policies, which appear to have been effective (Hardwicke et al., 2018; Nuijten et al., 2017). Nevertheless, these journals are in the minority; an assessment of 50 top-ranked and 40 randomly selected psychology journals found that the majority had no explicit policies related to a variety of transparent research practices (Nosek et al., 2022). It therefore remains unclear to what extent transparency has improved since we last estimated field-wide prevalence in psychology between 2014 and 2017 (Hardwicke et al., 2022).
In the present study, we obtained contemporary prevalence estimates for the adoption of transparent research practices in psychology. In line with previous research (Hardwicke et al., 2022), we focused on seven specific practices considered to be particularly important (e.g., Munafò et al., 2017; Nosek et al., 2015), including open access (to articles), disclosure of funding and conflicts of interest, preregistration, and sharing of materials, raw/primary data, and analysis scripts. We identified the use of these practices through manual examination of random samples of empirical articles published in psychology journals in general and in prominent psychology journals (top-ranked by Impact Factor 2 ) specifically. We included a sample of prominent journals because prior research has indicated that they tend to have more stringent transparency policies than journals on average, although the difference is small (Nosek et al., 2022).
When articles in our sample stated that core research artifacts (preregistration, materials, data, analysis scripts) were available, we also checked if they were functionally available, meaning that we could access, download, and view them without having to contact authors or third parties (which we did not do). This study, combined with the baseline estimates established by Hardwicke et al. (2022), provides an empirical measure of progress for the various infrastructure improvements, bottom-up advocacy, and top-down policy initiatives intended to increase transparency in psychology over the last decade or so (Morey et al., 2016; Nosek et al., 2015).
The goal of the study was description and estimation; we did not test any hypotheses. Our specific objectives were to (a) estimate the field-wide prevalence of transparent research practices in empirical articles published in all psychology journals in 2022, (b) estimate the prevalence of transparent research practices in empirical articles published in prominent psychology journals in 2022, and (c) describe whether reportedly available preregistrations, data, materials, and analysis scripts were functionally available and the method of sharing.
Research Transparency Statement
We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study. The research aims, methods, and analysis plan were preregistered (https://osf.io/xh6mg) on October 4, 2023, before data collection, which began on the same day. Ten articles were coded for piloting purposes before the protocol was preregistered; these articles are included in the final results. There were four minor deviations from the preregistration (see Supplementary Information A in the Supplemental Material available online).
All study materials are publicly available (https://osf.io/zk97j/files/osfstorage). All raw data are publicly available (https://osf.io/zk97j/files/osfstorage). All analysis scripts are publicly available (https://osf.io/zk97j/files/osfstorage). A reproducible version of this article interleaving regular prose and analysis code is available in a Code Ocean container (https://doi.org/10.24433/CO.5081898.v2).
This article adheres to the STROBE (Elm et al., 2007) and PRISMA-S (Rethlefsen et al., 2021) reporting guidelines. No artificial-intelligence-assisted technologies were used in this research or the creation of this article. This study did not require ethics approval.
Method
The methods were adapted from Hardwicke et al. (2022), preregistered (https://osf.io/xh6mg), and outlined in entirety in this article and its supplementary information.
Design
The study had a cross-sectional design. In two samples of articles, we measured the following transparency indicators: (a) open access (to articles), (b) preregistration, (c) sharing of research materials, (d) sharing of raw/primary data, 3 (e) sharing of analysis scripts, (f) disclosure statements on funding, and (g) disclosure statements on conflicts of interest. We also checked whether research artifacts (preregistration, sharing of research materials, sharing of raw/primary data, and sharing of analysis scripts) were functionally available (without contacting authors or third parties) and the method of sharing (e.g., which repository was used). For each transparency indicator, we did not check if all possible information had been made available, so it was sufficient for only some relevant information to be shared (e.g., one type of research material). For details about the measures, including operational definitions, see Table B1 in the Supplemental Material.
Sample
Target populations
There were two target populations: (a) field-wide population—all English-language empirical articles published in psychology journals in 2022—and (b) prominent journals population—all English-language empirical articles published in the top 50 psychology journals ranked by Journal Impact Factor publishing empirical research in 2022. We were interested in only legitimate, peer-reviewed, academic journals.
Sampling units
The sampling units were individual articles.
Sample size and justification
We obtained our target sample size of 200 eligible articles for each population (i.e., 400 eligible articles in total). The sample size target was informed by precision analyses (see Supplementary Information C in the Supplemental Material).
Data sources
On October 2, 2023, we downloaded from the Web of Science Core Collection all 75,683 bibliographic records pertaining to English-language articles published in 2022 in psychology journals. The search string was “WC=psychology and PY=2022 and DT=article and LA=english.” After de-duplication, 75,657 records remained. These articles represent the entire field-wide population of interest.
For the prominent population, we selected the subset of 5,802 records from the field-wide population that were published in the most prominent psychology journals. To identify prominent journals, we first used Clarivate Journal Citation Reports to obtain a list of psychology journals ranked by 2022 Journal Impact Factor and then checked whether these journals published empirical research until we had identified the 50 top-ranked eligible journals. For more details on obtaining the sampled articles, see Supplementary Information D in the Supplemental Material.
Eligibility criteria
To be eligible, articles had to be (a) accessible to both assigned coders (e.g., through their university library), (b) classified as empirical, and (c) written in English.
Procedure
First, to ensure random selection of articles from the populations and random assignment of articles to coders (to minimize coder drift), we first randomly shuffled the list of 75,657 articles (field-wide population) and the list of 5,802 articles (prominent population). We then selected the first 400 rows from each list. We selected more than the target sample size to allow for replacement of noneligible articles. One article appears in both lists. An R script documenting the list-creation process is available at https://osf.io/yhnsv.
Second, the transparency indicators (Supplementary Table B1 in the Supplemental Material) were extracted and classified (coded) using a Google Form (https://osf.io/hr68n). Coders first attempted to identify indicators by searching each article’s full text for keywords (noted in Supplementary Table B1 in the Supplemental Material). If the keyword search was unsuccessful, coders also manually examined parts of articles in which the indicators are often found (front matter, method section, immediately after the discussion section, and the acknowledgments section).
Third, coders also checked if reportedly available research artifacts were functionally available: When articles contained links to preregistrations, materials, data, or analysis scripts, coders attempted to access, download, and open any linked files and briefly inspected them to ensure they met our operational definitions (e.g., what we considered to be raw/primary data; for definitions, see Supplementary Table B1 in the Supplemental Material).
Finally, each article was coded independently by two coders. Coding differences were resolved by one coder (T. E. Hardwicke).
Data analysis
For each measured variable (Supplementary Table B1 in the Supplemental Material), we report raw counts and percentages. For variables related to availability, we also report 95% CIs in square brackets calculated with the Wilson method for binomial variables and the Sison-Glaz method for multinomial variables (Newcombe, 1998).
Results and Discussion
The prevalence of the seven transparency indicators is shown in Figure 1 for the field-wide sample and the prominent sample. For comparison, the field-wide sample of articles published between 2014 and 2017 from Hardwicke et al. (2022) is also included in Figure 1. Figure 1 shows the percentage of articles for which research artifacts were functionally available, meaning that we could access, download, and view them without having to contact authors or third parties (which we did not do). Interrater agreement is reported in Supplementary Information E in the Supplemental Material. More detailed results are reported in the text and tables below.
Prevalence of transparent research practices in empirical psychology articles. The field-wide and prominent samples (2022) from the present study each have sample sizes of 200 articles. The 2014–2017 sample is from a prior study (Hardwicke et al., 2022)— the sample sizes were open access (n = 237), conflict of interest and funding (n = 228), materials (n = 183), and all other practices (n = 188). Error bars represent 95% confidence intervals. Functional availability means that the research artifact could be accessed, downloaded, and viewed without contacting authors or third parties. Conflict-of-interest statements and funding statements include articles that disclosed a potential conflict or funding source and those that stated there was no potential conflict or no funding (see main text for details).
Included articles
Field-wide sample
In total, we examined 224 articles from the field-wide list before reaching the target sample size of 200 articles. Five articles were excluded because at least one coder could not access any version of them. Nineteen additional articles were excluded because they were not empirical. Two hundred eligible articles remained. Research designs were classified as nonexperimental (n = 141), experimental (n = 57), experimental and nonexperimental (n = 1), and meta-analysis (n = 1).
Prominent sample
In total, we examined 232 articles from the prominent list before reaching the target sample size of 200 articles. All articles could be accessed. Thirty-two articles were excluded because they were not empirical. Two hundred eligible articles remained. Research designs were classified as nonexperimental (n = 145), experimental (n = 44), experimental and nonexperimental (n = 4), and meta-analysis (n = 7).
Open access (to articles)
Open (free) access to academic articles increases the accessibility of research to researchers, policymakers, practitioners, and the general public. Psychologists can make their research open access by publishing in open-access journals, publishing open access in paywalled journals (if that is an option), or uploading manuscripts to institutional repositories or free preprint servers, such as PsyArXiv (https://psyarxiv.com/; Moshontz et al., 2021).
Field-wide sample
We identified an open-access version for 147 of 200 articles (74% [67%, 79%]; Fig. 1). The remaining 53 of 200 articles (26% [21%, 33%]) were accessible only through a paywall.
Prominent sample
We identified an open-access version for 142 of 200 articles (71% [64%, 77%]; Fig. 1). The remaining 58 of 200 articles (29% [23%, 36%]) were accessible only through a paywall.
Preregistration
Preregistration involves declaring a study plan in an online registry, typically before data are collected. Although different disciplines have different norms, in psychology, the goal of preregistration is typically to reduce bias arising from data-dependent research decisions and increase transparency, enabling readers to assess the risk of bias and calibrate their confidence in the research claims (Hardwicke & Wagenmakers, 2023). Psychologists can preregister their research in registries such as OSF (https://osf.io/registries) or AsPredicted (https://aspredicted.org/), and a range of templates is available for specific research designs (Bosnjak et al., 2022; Crüwell & Evans, 2021; van den Akker et al., 2021).
Our results (Table 1) indicate that the vast majority of empirical psychology research is not preregistered. Preregistration is almost twice as common in prominent journals compared with the average journal. A few preregistrations were not functionally available because of broken links and unclear access instructions.
Prevalence of Preregistration
Note: Percentages are based on a denominator of N = 200 (i.e., all articles in a given sample) and rounded to two significant digits. CI = confidence interval.
The values for the number of preregistrations on each platform do not sum to 14 because one article had a preregistration on two platforms, OSF and AsPredicted.
Materials availability
Access to research materials (e.g., survey instruments, software code unrelated to analyses, stimuli) facilitates comprehensive evaluation of research (Vazire, 2017) and the conduct of high-fidelity replications (Simons, 2014). It also reduces waste and increases efficiency because researchers can reuse materials rather than recreate them. Research materials can be shared in online repositories such as OSF; for guidance, see Klein et al. (2018).
Our results (Table 2) indicate that the majority of empirical psychology articles do not share research materials. Materials sharing may be slightly more common in prominent journals. In some cases, reportedly available materials were not functionally available because of broken links, unclear access instructions, or only being “available on request.” Only a few articles provided justification for a lack of materials availability.
Prevalence of Materials Availability
Note: Percentages are based on a denominator of N = 200 (i.e., all articles in a given sample) and rounded to two significant digits. CI = confidence interval.
No articles provided a reason for limiting access.
In our view, materials sharing was applicable because the research involved materials.
Data availability
Sharing raw/primary research data facilitates error detection, independent verification of computational reproducibility (Hardwicke et al., 2021), fraud detection (Simonsohn, 2013), analytic robustness checks (Steegen et al., 2016), enhanced evidence synthesis (Tierney et al., 2015), and novel discovery through reanalysis (Voytek, 2016). Psychologists can share data publicly in repositories such as OSF or Zenodo (for guidance, see Klein et al., 2018). Sharing data can sometimes be complicated by legal or ethical concerns, and researchers should always share responsibly (Wicherts et al., 2022); however, such concerns have to be balanced with the ethical prerogative to share data (Meyer, 2018). Sometimes, a middle ground can be reached; for example, repositories such as the UK Data Service or ICPSR enable controlled access to sensitive data. A useful maxim is that data should be as open as possible and as closed as necessary.
Our results (Table 3) indicate that the majority of empirical psychology articles do not share raw or primary data. Data sharing may be slightly more common in prominent journals. In some cases, reportedly available data were not functionally available because of broken links, unclear access instructions, or only being “available on request.” Only a few articles provided justification for a lack of data availability.
Prevalence of Data Availability
Note: Percentages are based on a denominator of N = 200 (i.e., all articles in a given sample) and rounded to two significant digits. CI = confidence interval.
Stated reasons for limiting access were privacy concerns (n = 5) and public sharing not permitted by third party (n = 1). The other 51 articles did not provide a reason.
Stated reasons for limiting access were privacy concerns (n = 6) or participants had not consented to public sharing (n = 2). The other 29 articles did not provide a reason.
In our view, data sharing was applicable because the research involved empirical data.
Analysis-script availability
Analysis scripts (computer code/syntax or step-by-step instructions in the case of point-and-click software) explicitly document the data-analysis procedures, including processing, summarizing, modeling, and visualizing. Analysis scripts can be reused by other researchers, reducing waste and increasing efficiency. Sharing scripts also facilitates error detection and computational reproducibility (Hardwicke et al., 2018). Psychologists can share analysis scripts in repositories such as OSF or Zenodo or share the scripts along with the computational environment in which they successfully run using tools such as Docker and platforms such as Code Ocean (Wiebels & Moreau, 2021).
Our results (Table 4) indicate that the vast majority of empirical psychology articles do not share analysis scripts. Analysis-script sharing may be slightly more common in prominent journals. In some cases, reportedly available analysis scripts were not functionally available because of broken links or only being “available on request.” Only a few articles provided justification for a lack of analysis-script availability.
Prevalence of Analysis Script Availability
Note: Percentages are based on a denominator of N = 200 (i.e., all articles in a given sample) and rounded to two significant digits. CI = confidence interval.
No articles provided a reason for limiting access.
In our view, analysis-script sharing was applicable because the research involved data analysis.
As noted by a reviewer, sharing either data or analysis scripts can have independent benefits, but sharing both data and analysis scripts will be maximally beneficial because it may enable computational reproducibility (Hardwicke et al., 2018). Both data and analysis scripts were shared in 14 (7% [4.2%, 11%]) articles in the field-wide sample and 22 (11% [7.4%, 16%]) articles in the prominent-journals sample.
Funding and conflict-of-interest disclosures
Disclosing funding sources and potential conflicts of interest is important because it helps readers to understand the risk of bias (Chivers, 2019; Cristea & Ioannidis, 2018). It is also important to explicitly disclose a lack of conflicts or funding sources because the absence of a statement is ambiguous.
Our results (Tables 5 and 6) indicate that the majority of empirical psychology articles include a funding disclosure statement and conflict-of-interest disclosure statement. Prevalence was similar in the field-wide and prominent samples. Most articles disclosed funding sources, and some said there was no funding. Disclosure of a conflict of interest was fairly uncommon; most articles stated that there was no conflict of interest.
Prevalence of Funding Disclosure Statements
Note: Percentages are based on a denominator of N = 200 (i.e., all articles in a given sample) and rounded to two significant digits. CI = confidence interval.
One article stated that funding sources had been disclosed but did not identify any; the other article had a censored funding statement, presumably retained from masking the authors’ identity during peer review.
Prevalence of Conflict-of-Interest Disclosure Statements
Note: Percentages are based on a denominator of N = 200 (i.e., all articles in a given sample) and rounded to two significant digits. CI = confidence interval.
Both articles stated that the funders did not influence various aspects of the study but did not refer to the authors’ potential conflicts of interest (or lack thereof).
The article stated that the funders did not influence various aspects of the study but did not refer to the authors’ potential conflicts of interest (or lack thereof).
General Discussion
The principle of transparency is as old as science itself (Wootton, 2016) and deeply embedded in its ethos, as symbolized by the motto of the Royal Society (Nullius in Verba [take nobody’s word for it]) and the common classroom maxim, “Show your work.” The endeavor to increase transparency gained renewed urgency in recent years, driven from the bottom-up by groups such as the Society for the Improvement of Psychological Science and the Peer Reviewers’ Openness Initiative (Morey et al., 2016) and receiving top-down endorsement from major funders, publishers, journals, and governmental organizations (Nosek et al., 2015; UNESCO, 2021). A plethora of new online repositories and registries, tools, and workflows have emerged (Borghi & Van Gulick, 2021; Klein et al., 2018). Working scientists appear to accept that research is often undermined by reproducibility problems (Baker, 2016; Cobey et al., 2023) and that transparency is a desirable goal (Anderson et al., 2010; Ferguson et al., 2023; Houtkoop et al., 2018). After more than a decade of improvements in infrastructure, bottom-up advocacy, and top-down policy initiatives, to what extent are psychologists actually using transparent research practices? Our results show that transparency has improved moderately since 2014–2017 (Hardwicke et al., 2022), and some practices (open access, disclosure of funding and conflicts of interest) are quite common. However, the availability of core research artifacts (preregistration, materials, data, analysis scripts) remains low. This widespread neglect of transparency may be undermining the efficiency, reproducibility, and self-correcting ideal of scientific inquiry (Ioannidis, 2012; Munafò et al., 2017; Vazire & Holcombe, 2022).
Full transparency is not always possible or straightforward, especially when there are competing ethical considerations, such as participant privacy (Meyer, 2018). However, when we encountered a lack of transparency, it was rarely explicitly justified. When a justification was provided, it tended to be brief, nonspecific, and unverifiable (e.g., “Data not available due to ethical concerns”). A substantial number of articles stated that research artifacts, especially data, were “available on request,” and very few provided a reason why. We did not consider research artifacts “available on request” to be functionally available, nor did we contact authors in an attempt to access them. If we had contacted authors, it is likely that some additional research artifacts would have been made available; however, meta-research studies have repeatedly demonstrated that such requests are often rejected or ignored, with the likelihood of obtaining data progressively declining after publication (Hardwicke & Ioannidis, 2018; Minocher et al., 2021; Tedersoo et al., 2021; Vanpaemel et al., 2015; Vines et al., 2014).
Another recurrent barrier to functional availability appears to be the ambiguities of third-party ownership/stewardship, especially for materials. We frequently encountered articles that cited the source of materials (e.g., the creators of a survey instrument) but did not make clear whether the materials were available or explain how to access them. Sometimes, articles did say that research artifacts were available from a third party and provided a link, but this often led to labyrinthian websites, and it was unclear how to obtain the exact version or subset of an artifact used in the research we were examining.
An important caveat to our study is that we performed only minimal quality checks. Specifically, we checked if links to research artifacts actually worked and briefly checked any shared files to see if they met our operational definitions (e.g., raw/primary data rather than summary data). However, it is likely that if we had probed further, we would have found additional problems. Prior research has found that shared data are often incomplete or lack clear documentation (Hardwicke et al., 2018; Towse et al., 2020). For example, Hardwicke et al. (2018) found that only 108 of 204 (53%) data sets shared with articles published in Cognition were actually independently reusable. Shared data also do not always enable computational reproducibility (Crüwell et al., 2023; Hardwicke et al., 2021). For example, Hardwicke et al. (2021) found that of 25 Psychological Science articles awarded open-data badges between 2014 and 2015, 16 (64%) contained at least one numerical value that could not be independently reproduced. Preregistrations also vary in quality; Bakker et al. (2020) reported that preregistrations on OSF often failed to mention important researcher degrees of freedom, thus undermining their ability to reduce bias. Researchers also frequently deviate from preregistrations without disclosure (Claesen et al., 2021; TARG Meta-Research Group & Collaborators, 2023). For example, Claesen et al. (2021) found that of 27 Psychological Science articles awarded preregistration badges between 2015 and 2017, 25 (93%) contained deviations from the preregistration plan, and 24 of them failed to disclose all of the deviations. In sum, although our study provides an indication of the functional availability of core research artifacts, it does not speak to the substantive quality, reusability, and comprehensiveness of those artifacts.
Our study estimates the prevalence of transparent research practices during a short time window (2022) and in two broad domains (all of psychology and prominent psychology journals). This limited scope is necessitated by the amount of work required to manually extract and classify information from scientific articles. We are currently working on automated methods that will help to expand the scope of this work and provide deeper insights into how adoption of transparent research practices varies across different domains (e.g., subfields of psychology) and contexts (e.g., different journals or types of research) and over time (Hardwicke et al., 2023).
In conclusion, our results indicate that overall, the adoption of transparent research practices has increased since 2017. Some transparent research practices, such as open access and disclosure of funding and conflicts of interest, are relatively common. However, after more than a decade of improvements in infrastructure, bottom-up advocacy, and top-down policy initiatives, the transparency of core research artifacts continues to be widely neglected in psychology.
Supplemental Material
sj-docx-1-amp-10.1177_25152459241283477 – Supplemental material for Prevalence of Transparent Research Practices in Psychology: A Cross-Sectional Study of Empirical Articles Published in 2022
Supplemental material, sj-docx-1-amp-10.1177_25152459241283477 for Prevalence of Transparent Research Practices in Psychology: A Cross-Sectional Study of Empirical Articles Published in 2022 by Tom E. Hardwicke, Robert T. Thibault, Beth Clarke, Nicholas Moodie, Sophia Crüwell, Sarah R. Schiavone, Sarah A. Handcock, Khanh An Nghiem, Fallon Mody, Tuomas Eerola and Simine Vazire in Advances in Methods and Practices in Psychological Science
Footnotes
Transparency
Action Editor: David A. Sbarra
Editor: David A. Sbarra
Author Contributions
