Abstract
Ensuring the integrity and trustworthiness of science relies on reproducibility, a cornerstone of the scientific process. The availability of raw data in a FAIR manner (Findable, Accessible, Interoperable, and Reusable) is a fundamental prerequisite for this pursuit (see https://www.go-fair.org/fair-principles/). In this note, we share our experience of obtaining the raw data and successfully reproducing the results published by Agarwal et al. [1] to both provide valuable information about a specific study and to illustrate the value and challenges of reproducibility and verification investigations.
We reproduced and verified the results and conclusions published in Agarwal et al.’s paper [1]. The process took us almost a year, with the direct involvement of 4 investigators, led by the senior investigators and biostatisticians in our team. There were also administrative tasks involved related to the data use agreement (DUA), which were managed by our Institution. We requested the raw data in August 2022, and after establishing a DUA and resolving initial issues, we could access the final analytical dataset six months later. In reproducing the analyses, we developed the code in R (R version 4.3.0) independently based on our understanding from the published methods, rather than relying on the code produced by the original investigators. Thus, we reproduced (not repeated) the statistical analyses, and the chances of replicating any error present in the original code was minimal. We reproduced the majority of the analyses, with minor typographical errors in one model (Model2A) presented in Table 2 of the original paper. We provide the reproduced Table 2 here. It is important to note that our work confirms the reproducibility of the reported outcomes within the scope of the provided data and methods. That is, successfully reproducing the numerical values does not inherently demonstrate the definitive correctness of the conclusions drawn. Alternatively, in the event that the numerical values could not be reproduced, it would not necessarily imply incorrectness of the study’s fundamental scientific conclusions, but it would suggest that the conclusions could not be considered to be substantiated by the study’s data.
Association of strawberry and pelargonidin intake with global AD pathology, amyloid-β load, and phosphorylated tau tangles, among deceased participants of the Rush Memory and Aging Project
We note that our work aligns with the definition of “reproducing” the analyses by National Academies of Sciences, Engineering, and Medicine [3], as we used the same input data and computational steps. We did not “replicate” the analyses (see definition of replication in [3]) because this would require obtaining our own data, which we did not do. With regard to verifiability 1 , focusing only on the correctness of the statistical methods chosen and the interpretation of results from those analytic methods, we believe the methods used by the authors are verifiable. That is, the quantitative methods used were established methods appropriate to answer whether “pelargonidin or berry intake is associated with Alzheimer’s disease (AD) neuropathology in human brains.”
Of course, there can be other issues to any study beyond the statistical methods chosen and the interpretations of the results they produce. One such issue which is applicable to all studies lacking an explicitly prespecified or pre-registered analysis plan pertains to the process by which the statistical analyses were selected from among the universe of analyses that might be reproducible and verifiable. If analyses are selected by any process that permitted the data collected to influence the choice, this can introduce biases into the estimation of population values due to various selection practices, including informal data visualization before formal analysis, covariate selection, and reporting bias [2]. These practices potentially inflate type 1 error rate, produce biased estimates, and compromise replicability (distinguished from reproducibility) [3].
We commend Agarwal et al. for addressing an interesting research question. The study ostensibly found that strawberry and pelargonidin intake are associated with less AD pathology, including reduced phosphorylated tau tangles. The findings are consistent with previous studies purporting that berry consumption affords protection against neurodegenerative diseases. If prevention of oxidative stress is important in terms of protection against AD, it is surprising that berry intake was not found to impact AD pathology. The anthocyanin composition of strawberries is unique when compared with other berries. Strawberries are rich in pelargonidin, whereas other berries are rich in different anthocyanins (delphinidin, malvidin, petunidin, and peonidin), especially cyanidin.
Greater numbers of hydroxyl groups on the B-rings of anthocyanins are associated with greater antioxidant activity. Delphinidin which contains three hydroxyl groups on its B-ring, and cyanidin, which contains two hydroxyl groups, possess higher antioxidant activity than pelargonidin, which contains only one hydroxyl group. Hence, if prevention of oxidative stress is important in terms of protection against AD, it is surprising that berry intake was not found to be significant. Another important factor to consider is the postulated role of gut microbiota, which can hydroxylate pelargonidin to cyanidin, which would presumably boost antioxidant activity. Additionally, gut microbiota can catabolize anthocyanins to simple phenolic acids, which may confer protection against oxidative stress and inflammation. More research is needed to confirm whether and, if so, elucidate the mechanism through which pelargonidin reduces phosphorylated tau tangles as well as other biomarkers associated with AD.
We sincerely appreciate Agarwal et al.’s commitment to transparent and reproducible science. By successfully reproducing their analyses, we demonstrated the value of and ability to conduct research in a manner that is open, transparent, and accountable, allowing other researchers to independently verify, and build on the findings. We encourage fellow investigators to use our team’s evolving checklist on ‘how’, ‘when’ and ‘where’ to share raw data and statistical methods to enhance reproducibility of their published findings (see https://osf.io/t83w2). As we continue refining this tool, we welcome insights and input from the research community to further strengthen its effectiveness and comprehensiveness.
AUTHOR CONTRIBUTIONS
Yasaman Jamshidi-Naeini (Conceptualization; Investigation; Writing – original draft; Writing – review & editing); Nicolas Escobar Velasquez (Conceptualization; Formal analysis; Investigation; Methodology; Writing – review & editing); Lilian Golzarri-Arroyo (Conceptualization; Formal analysis; Investigation; Methodology; Writing – review & editing); Sumayyah Ali (Conceptualization; Formal analysis; Investigation; Methodology; Writing – review & editing); Luke R. Howard (Conceptualization; Investigation; Writing – review & editing); Stephanie Dickinson (Conceptualization; Investigation; Supervision; Writing – review & editing); David B. Allison (Conceptualization; Funding acquisition; Investigation; Supervision; Writing – review & editing).
Footnotes
ACKNOWLEDGMENTS
The authors have no acknowledgments to report.
FUNDING
The reproducibility analyses reported in this correspondence are supported in part by NIH grants R25DK099080, R25HL124208, U24AG056053, P30AG050886, and the Gordon and Betty Moore Foundation. The opinions expressed are those of the authors and do not necessarily represent those of the NIH or any other organization.
CONFLICT OF INTEREST
Dr. Allison and his institution (Indiana University) have received payments for consultation, grants, contracts, in-kind donations, and contributions from multiple for-profit and not-for-profit entities interested in statistical design and analysis of experiments.
A study is said to have been verified, when: (a) the study is reproducible, and the results have been reproduced (by the definition of reproducibility in (National Academies of Sciences, Engineering, and Medicine, 2019)); and (b) a determination is made that the methods used to generate the results reproduced are valid methods and that the interpretations validly and logically follow from the obtained results.