Restricted accessResearch articleFirst published online 2021-7
The Quality Assurance and Quality Control Protocol for Neuropsychological Data Collection and Curation in the Ontario Neurodegenerative Disease Research Initiative (ONDRI) Study
As large research initiatives designed to generate big data on clinical cohorts become more common, there is an increasing need to establish standard quality assurance (QA; preventing errors) and quality control (QC; identifying and correcting errors) procedures for critical outcome measures. The present article describes the QA and QC approach developed and implemented for the neuropsychology data collected as part of the Ontario Neurodegenerative Disease Research Initiative study. We report on the efficacy of our approach and provide data quality metrics. Our findings demonstrate that even with a comprehensive QA protocol, the proportion of data errors still can be high. Additionally, we show that several widely used neuropsychological measures are particularly susceptible to error. These findings highlight the need for large research programs to put into place active, comprehensive, and separate QA and QC procedures before, during, and after protocol deployment. Detailed recommendations and considerations for future studies are provided.
BarchardK. A.PaceL. A. (2011). Preventing human error: The impact of data entry methods on data accuracy and statistical results. Computers in Human Behavior, 27(5), 1834-1839. https://doi.org/10.1016/j.chb.2011.04.004
BeatonD.SunderlandK. M.LevineB.MandziaJ.MasellisM.SwartzR. H.TroyerA. K., ONDRI Investigators, BinnsM. A.AbdiH.StrotherS. C. (2018). Generalization of the minimum covariance determinant algorithm for categorical and mixed data types. bioRxiv Preprint. https://doi.org/10.1101/333005
4.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B: Methodological, 57(1), 289-300.
FarhanS. M. K.BarthaR.BlackS. E.CorbettD.FingerE.FreedmanM.GreenbergB.GrimesD. A.HegeleR. A.HudsonC.KleinstiverP. W.LangA. E.MasellisM.McIlroyW. E.McLaughlinP. M.Montero-OdassoM.MunozD. G.MunozD. P.StrotherS.. . . StrongM. J. (2016). The Ontario Neurodegenerative Disease Research Initiative (ONDRI). Canadian Journal of Neurological Sciences, 44(2), 196-202. https://doi.org/10.1017/cjn.2016.415
9.
FarhanS. M. K.DilliottA. A.GhaniM.SatoC.LiangE.ZhangM.McIntyreA. D.CaoH.RacachoL.RobinsonJ. F.StrongM. J.MasellisM.St George-HyslopP.BulmanD. E.RogaevaE.HegeleR. A. (2016). The ONDRISeq panel: Custom-designed next-generation sequencing of genes related to neurodegeneration. Genomic Medicine, 1, Article 16032. https://doi.org/10.1038/npjgenmed.2016.32
10.
GarthwaiteP. H.KochI. (2016). Evaluation the contributions of individual variables to a quadratic form. Australian & New Zealand Journal of Statistics, 58(1), 99-119. http://doi.org/10.1111/anzs.12144
11.
HarrisP. A.TaylorR.ThielkeR.PayneJ.GonzalezN.CondeJ. G. (2009). Research Electronic Data Capture (REDCap): A metadata-driven methodology and workflow process for providing translational research informatics support. Journal of Biomedical Informatics, 42(2), 377-381. https://doi.org/10.1016/j.jbi.2008.08.010
HasherL.GoldsteinD.MayC. P. (2005). It’s about time: Circadian rhythms, memory, and aging. In ChizukoI.NobuoO. (Eds.), Human learning and memory: Advances in theory and application (pp. 199-217). Lawrence Erlbaum.
14.
HoustonM. L.YuA. P.MartinD. A.ProbstD. Y. (2018). Defining and developing a generic framework for monitoring data quality in clinical research. AMIA: Annual Symposium Proceedings: AMIA Symposium, 2018, 1300-1309.
International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use. (2014). ICH-E6: Good Clinical Practice Guideline. http://www.ichgcp.net
17.
KahnM. G.BrownJ. S.ChunA. T.DavidsonB. N.MeekerD.RyanP. B.SchillingL. M.WeiskopfN. G.WilliamsA. E.ZozusM. N. (2015). Transparent reporting of data quality in distributed data networks. eGEMs: Journal for Electronic Health Data and Methods, 3(1), Article 1052. https://doi.org/10.13063/2327-9214.1052
18.
KawadoM.HinotsuS.MatsuyamaY.YamaguchiT.HashimotoS.OhashiY. (2003). A comparison of error detection rates between the reading aloud method and the double data entry method. Controlled Clinical Trials, 24(5), 560-569. https://doi.org/10.1016/S0197-2456(03)00089-8
19.
KeefeR. S. E.HarveyP. D. (2008). Implementation considerations for multisite clinical trials with cognitive neuroscience tasks. Schizophrenia Bulletin, 34(4), 656-663. https://doi.org/10.1093/schbul/sbn042
20.
KnatterudG. L.RockholdF. W.GeorgeS. L.BartonF. B.DavisC. E.FairweatherW. R.HonohanT.MoweryR.O’NeillR. (1998). Guidelines for quality assurance in multicenter trials: A position paper. Controlled Clinical Trials, 19(5), 477-493. https://doi.org/10.1016/S0197-2456(98)00033-6
21.
KozoraE.KongsS.HamptonM.ZhangL. (2008). Effects of examiner error on neuropsychological test results in a multi-site study. The Clinical Neuropsychologist, 22(6), 977-988. https://doi.org/10.1080/13854040701679025
22.
MesserlyJ.MarceauxJ. C. (2019). Examination of the reliability and validity of the NAB Naming Test in a diverse clinical sample, The Clinical Neuropsychologist, 34(2), 406-422. https://doi.org/10.1080/13854046.2019.1635647
23.
MitchelJ. T.KimY. J.ChoiJ.ParkG.CappiS.HornD.KistM.D’AgostinoR. B.Jr. (2011). Evaluation of data entry errors and data changes to an electronic data capture clinical trial database. Drug Information Journal, 45(4), 421-430. https://doi.org/10.1177/009286151104500404
24.
Montero-OdassoM.Pieruccini-FariaF.BarthaR.BlackS.FingerE.FreedmanM.GreenbergB.GrimesD. A.HegeleR. A.HudsonC.KleinstiverP. W.LangA. E.MasellisM.McLaughlinP. M.MunozD. P.StrotherS.SwartzR. H.SymonsS.TartagliaM. C.. . . McllroyW (2017). Motor phenotype in neurodegenerative disorders: Gait and balance platform study design protocol for the Ontario Neurodegenerative Research Initiative (ONDRI). Journal of Alzheimer’s Disease, 59(2), 707-721. https://doi.org/10.3233/JAD-170149
25.
R Core Team. (2017). R: The R Project for statistical computing. www.R-project.org/
SadlerM. E.YamamotoR. T.KhuranaL.DallabridaS. M. (2017). The impact of rater training on clinical outcomes assessment data: A literature review. International Journal of Clinical Trials, 4(3), 101-110. https://doi.org/10.18203/2349-3259.ijct2017313
28.
SchoolerJ. W. (2014). Metascience could rescue the “replication crisis.”Nature, 515(7525), Article 9. https://doi.org/10.1038/515009a
StraussE.ShermanE. M. S.SpreenO. (2006). A compendium of neuropsychological tests: Administration, norms, and commentary (3rd ed.). Oxford University Press.
31.
SunderlandK. M.BeatonD.FraserJ.KwanD.McLaughlinP. M.Montero-OdassoM.PeltschA. J.McLaughlinP. M.Montero-OdassoM.PeltschA. J.Pieruccini-FariaF.SahlasD. J.SwartzR. H., ONDRI Investigators, StrotherS. C.BinnsM. A. (2019). The utility of multivariate outlier detection techniques for data quality evaluation in large studies: An application within the ONDRI project. BMC Medical Research Methodology, 19, Article 102. https://doi.org/10.1186/s12874-019-0737-5
32.
ThriemerK.LeyB.AmeS. M.PuriM. K.HashimR.ChangN. Y.SalimL. A.OchiaiR. L.WierzbaT. F.ClemensJ. D.von SeidleinL.DeenJ. L.AliS. M.AliM. (2012). Replacing paper data collection forms with electronic data entry in the field: Findings from a study of community-acquired bloodstream infections in Pemba, Zanzibar. BMC Research Notes, 5, Article 113. https://doi.org/10.1186/1756-0500-5-113
33.
VaccarinoA. L.DharseeM.StrotherS.AldridgeD.ArnottS. R.BehanB.DafnasC.DongF.EdgecombeK.El-BadrawiR.El-EmamK.GeeT.EvansS. G.JavadiM.JeansonF.LefaivreS.LutzK.MacPheeF. C.MikkelsenJ.. . . EvansK. R. (2018). Brain-CODE: A secure neuroinformatics platform for management, federation, sharing and analysis of multi-dimensional neuroscience data. Frontiers in Neuroinformatics, 12, Article 28. https://doi.org/10.3389/fninf.2018.00028
34.
Verde ArregoitiaL. D.CooperN.D’ElíaG. (2018). Good practices for sharing analysis-ready data in mammalogy and biodiversity research. Hystrix: The Italian Journal of Mammalogy, 29(2), 155-161. https://doi.org/10.4404/hystrix-00133-2018
35.
WesterveltH. J.BernierR. A.FaustM.GoverM.BockholtH. J.ZschiegnerR.LongJ. D.PaulsenJ. S.PREDICT-HD Investigators, & Coordinators of the Huntington Study Group. (2017). Data quality assurance and control in cognitive research: Lessons learned from the PREDICT-HD study. International Journal of Methods in Psychiatric Research, 26(3), Article e1534. https://doi.org/10.1002/mpr.1534
WilkinsonM. D.DumontierM.Jan AalbersbergI. J.AppletonG.AxtonM.BaakA.BlombergN.BoitenJ.-W.da Silva SantosL. B.BourneP. E.BouwmanJ.BrookesA. J.ClarkT.CrosasM.DilloI.DumonO.EdmundsS.EveloC. T.FinkersR.. . . MonsB. (2016). The FAIR guiding principles for scientific data management and stewardship. Scientific Data, 3, Article 160018. https://doi.org/10.1038/sdata.2016.18
38.
YochimB. P.KaneK. D.MuellerA. E. (2009). Naming test of the Neuropsychological Assessment Battery: Convergent and discriminant validity. Archives of Clinical Neuropsychology, 24(6), 575-583. https://doi.org/10.1093/arclin/acp053
39.
ZozusM. N.KahnM. G.WeiskopfN. G. (2019). Data quality in clinical research. In RichessonR. L.AndrewsJ. E. (Eds.), Clinical research informatics (pp. 213-248). Springer.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
