Progression from Prodromal Alzheimer’s Disease to Mild Alzheimer’s Disease Dementia in the Verubecestat APECS Study: Adjudicating Diagnostic Transitions

Abstract

Background:

Delay of progression from prodromal Alzheimer’s disease (AD) to dementia is an important outcome in AD trials. Centralized adjudication is intended to improve the consistency of dementia diagnosis but has not been scrutinized.

Objective:

To evaluate centralized adjudication for determining progression to dementia compared with Site Investigator opinion or change in Clinical Dementia Rating (CDR).

Methods:

We used data from the 2-year APECS trial of verubecestat versus placebo in 1,451 prodromal AD participants. Cases were triggered for central adjudication if: 1) the Site Investigator judged the participant had progressed to dementia, or 2) the participant’s CDR sum-of-boxes score increased ≥2 points from baseline. Post-hoc analyses were performed on pooled treatment-group data to compare methods of assessing progression.

Results:

581/1,451 (40%) participants had changes triggering adjudication and most (83%) were confirmed as progression to dementia. Only 66% of those who met CDR criteria (regardless of whether they also met Site Investigator criteria) were adjudicated to have progressed to dementia and just 15% of those who met only CDR criteria were adjudicated to have progressed, representing 5% of progressors. In contrast, 99% of those who met Site Investigator criteria (regardless of whether they also met CDR criteria) were adjudicated to have progressed, and the same was true for those who met only Site Investigator criteria.

Conclusion:

A positive Site Investigator opinion is an excellent predictor for a positive adjudication decision regarding onset of dementia. Conversely, sole use of CDR sum-of-boxes change ≥2 is inadequate. The benefit of centralized adjudication appears doubtful.

Keywords

Alzheimer’s disease APECS Clinical Dementia Rating diagnosis mild cognitive impairment randomized controlled trial

Trial Registration:ClinicalTrials.gov NCT01953601

INTRODUCTION

A key goal of disease-modifying therapeutics in Alzheimer’s disease (AD) is to delay the progression from preclinical or prodromal states such as mild cognitive impairment (MCI) to dementia [1]. While the time to progression from MCI to dementia is a clinically relevant endpoint with high face validity, in practice the determination is complex, subjective, and sensitive to factors unrelated to the potential therapeutic under investigation [2, 3]. One potential solution adopted in clinical trials is the use of centralized, independent adjudication intended to improve the consistency of evaluation across multiple sites, countries, and cultures. However, the adjudication process is complicated and resource-intensive (see Methods) and has not been subjected to scientific scrutiny.

The aim of this post hoc investigation was to evaluate the relative performance of centralized adjudication versus alternate methods of assessing progression based on Site Investigator determination or the change in Clinical Dementia Rating (CDR) sum-of-boxes (SB) score [4] using data from APECS, a large phase III clinical trial of the BACE1 inhibitor verubecestat in prodromal AD [5]. If the adjudication approach largely confirms CDR-based or Site Investigator determination of progression, then this suggests a means of simplifying future trials in which progression from MCI to AD dementia is an outcome.

METHODS

Trial design

Full details of the trial design and methods are provided in Egan et al. [5]. The trial consisted of a randomized, double-blind, placebo-controlled, parallel group, multi-center, 104-week evaluation of placebo and two verubecestat doses (12 mg, 40 mg) in 1,451 people with biomarker-confirmed prodromal AD/MCI due to AD. The trial was performed at 238 sites in 22 countries from 2013 to 2018, with the major regions (percentage of total participants recruited) being North America (47%), Europe, Australia and New Zealand (33%), and Japan (12%). The trial was conducted in accordance with principles of Good Clinical Practice and protocol was approved by local institutional review boards. Written informed consent was provided by the participant or their legal representative.

Participants

Participants between 50–85 years of age were eligible if they did not meet criteria for dementia [6, 7] and had: 1) subjective memory decline for ≥1 year corroborated by an informant; 2) score ≥1 standard deviation below the appropriate population mean on the Repeatable Battery for the Assessment of Neuropsychological Status Delayed Memory Index [8]; 3) brain amyloid positivity by positron emission tomography (PET) imaging visual read. All participants underwent medical and neurological evaluations, including magnetic resonance imaging (or computed tomography if magnetic resonance imaging was contraindicated). Other entry criteria included a score of 24–30 on the Mini-Mental State Examination (MMSE) [9]. The diagnosis of prodromal AD/MCI due to AD was confirmed by independent review.

Assessments

Clinic visits were scheduled at screening, baseline/randomization and then at weeks 2, 6, 13, 26, 39,52, 65, 78, 91, and 104. The CDR and MMSE were scheduled to be routinely performed at baseline and weeks 13, 26, 52, 78, and 104, and additionally at other clinic visits if progression to dementia was suspected (see below). The AD Assessment Scale-Cognitive Subscale (ADAS-Cog) [10], a battery of other cognitive tests [5], and the AD Cooperative Study –Activities of Daily Living for Mild Cognitive Impairment (ADCS-ADL_MCI) scale [11] were scheduled to be performed at screening, baseline/randomization, and weeks 13, 26, 39, 52, 65, 78, 91, and 104, and additionally at other clinic visits if progression to dementia was suspected (see below). The CDR was administered and scored by a qualified, trained rater according to instructions in a standard operating procedures manual. The primary endpoint for the trial was change-from-baseline in the CDR-SB score at 104 weeks. There were 7 secondary endpoints, one of which was time to progression to probable AD, with the determination of probable AD being subject to adjudication as described below.

Adjudication process

The adjudication process for determining progression to dementia is summarized in Fig. 1 and was detailed in a 50-page Charter. An assessment as to whether a participant may have progressed to dementia was made by the Site Investigator (who was blinded to treatment allocation) at all post randomization clinic visits using a standard Diagnostic Classification Form. The Site Investigator was instructed to consider a participant as having potentially progressed to dementia if either: 1) in the investigator’s own expert judgment, they thought the participant might have progressed, or 2) the participant’s CDR-SB score was ≥2 points higher than it was at baseline. For visits where CDR was not scheduled, progression could be suspected based on clinical worsening as assessed via the ADCS-ADL_MCI scale and/or the ADAS-Cog. Once potential progression was identified, the investigator was instructed to perform a complete clinical assessment of the participant, including administration of the CDR, ADAS-Cog, Cognitive Battery, ADCS-ADL_MCI scale, and MMSE, if not already performed at that visit.

Fig. 1

General process of the adjudication of progression to dementia in the verubecestat prodromal AD study. BARDS, Biostatistics and Research Decision Sciences (MSD’s statistics group); CDR-SB, Clinical Dementia Rating Sum-of-Boxes; CPM, Clinical Project Manager; DCF, Diagnostic Classification Form; MA, MedAvante (Contract Research Organization involved in the adjudication process); PI, Primary (Site) Investigator; QCAT, Quality Control Assessment Tool; MRL, MSD Research Laboratories (the study sponsor); QC, Quality Control; TMF, Trial Master File

For all potential cases of progression, the Diagnostic Classification Form and a diagnostic narrative summary were prepared according to instructions in a 12-page Guidance Document for Assessment of Progression to Dementia and submitted with any relevant clinical or laboratory reports for review by two members of an adjudication committee, which consisted of a panel of five MD or PhD specialists (neurologists, psychiatrists, or clinical neuropsychologists) who were experts in the diagnosis and treatment of AD dementia as well as the conduct of clinical trials. The adjudicators were blinded to treatment allocation. Each of the two adjudicators was tasked with determining whether the suspected case had progressed to dementia, and if so the probable cause (AD or non-AD), as defined by National Institute of Neurological Diseases and Stroke –Alzheimer’s Disease and Related Disorders Association [7], and Diagnostic and Statistical Manual of Mental Disorder, 4th Edition, Text Revision criteria [6]. If the two adjudication members did not agree on the participant’s diagnosis, then a 3rd adjudication committee member was assigned to review the case and the procedure outlined in Fig. 1 was followed to arrive at a final adjudication decision. If the committee’s final decision was that progression to dementia had not occurred, the participant was reassessed at the next scheduled clinic visit; if the participant met at least one of the criteria for possible progression at the next visit, then an updated adjudication package was re-submitted to the adjudication committee. Thus, the same participant could be adjudicated more than once.

Analyses

It is important to note that the trial was not designed to address the questions that we attempt to investigate here and there are limitations in the data and their interpretation (described in the Discussion). All analyses were performed on a post hoc basis using pooled data across the treatment groups (verubecestat and placebo) and were descriptive in nature (i.e., there was no formal hypothesis testing and no p-values were calculated for any comparisons). Summary information (numbers and percentages, or means and standard deviations) is provided regarding the following: 1) number of participants sent for adjudication and the number of adjudications performed; 2) adjudication decision at the participant’s first adjudication (information was also derived for the participant’s last adjudication but the results were similar to the first adjudication and are not shown); 3) triggers associated with decision to adjudicate - Site Investigator assessment, CDR-SB change ≥2, or both; 4) determination of progression based on Site Investigator review versus CDR-SB change ≥2 as the trigger for adjudication; 5) adjudication committee agreement with triggers for adjudication - Site Investigator review versus CDR-SB change ≥2; 6) Kaplan-Meier plot of confirmed cases of progression to dementia due to AD; 7) descriptors of progressors at time of progression to dementia - comparison of mean changes across different CDR domains/boxes in progressors versus non-progressors. In addition, as part of routine study monitoring, agreement on diagnosis amongst the five expert adjudication committee members was assessed over time/number of cases using a kappa analysis [12]. Statistical analyses to support items 3–6 and the kappa analysis were performed using SAS Versions 9.3 and 9.4 (SAS Institute, Cary, NC, USA).

RESULTS

Number of participants adjudicated, and number of times adjudicated

Of the 1,451 participants in the trial, 581 (40.0%) were sent for adjudication at least once. Of the 581 sent for adjudication, 504 (86.7%) were adjudicated once only; 77 (13.3%) were adjudicated more than once but few were adjudicated more than twice (15/581, 2.6%). A total of 678 adjudications were performed over the course of the trial, including repeat adjudications.

Adjudication decision

At the first adjudication, of the 581 adjudicated participants 436 (75.0%) were adjudicated to have progressed to dementia while 144 (24.8%) were adjudicated to have not progressed. Most cases of progression due to dementia were adjudicated to be dementia due to AD, with progression to non-AD dementia (e.g., Lewy body dementia, vascular dementia) being rare (4/581 [0.7%] cases). There was 1 “unable to adjudicate” decision.

Adjudication trigger

The number of participants available for this analysis was 555 which is less than the total 581 adjudicated (see preceding section). The reduced sample size was due to missing data (e.g., instances where the participant met criteria per the Site Investigator but the site did not conduct a CDR). At the first assessment, of 555 adjudication cases included in the analysis, 245 (44.1%) were triggered by both the Site Investigator and CDR, while 149 (26.8%) were triggered by the Site Investigator only, and 161 (29.0%) were triggered by CDR only.

Proportion of cases adjudicated as dementia due to AD by adjudication trigger

Table 1 shows the proportion of cases adjudicated as dementia due to AD (i.e., excluding the 4 non-AD dementia cases) by adjudication trigger. At the first assessment, 66.1% of those who met CDR criteria (regardless of whether they also met Site Investigator criteria) were adjudicated as AD dementia versus 99.2% for those who met Site Investigator criteria (regardless of whether they also met CDR criteria). For “CDR only” cases, 14.5% were adjudicated as AD dementia versus 98.7% for “Site Investigator only” cases. For cases that met “Both CDR and Site Investigator” criteria, 99.6% were adjudicated as AD dementia.

Table 1

Proportion of cases adjudicated as dementia due to AD by adjudication trigger

Adjudication triggered by	First Assessment
	n/m (%)
CDR (independent of Site Investigator)	267/404 (66.1%)
Site Investigator (independent of CDR)	391/394 (99.2%)
CDR only	23/159 (14.5%)
Site Investigator only	147/149 (98.7%)
CDR and Site Investigator	244/245 (99.6%)

n/m, Number adjudicated to have dementia due to AD/Number with the given adjudication trigger. Corresponding CDR scores were missing for some of the Site Investigator submitted cases.

Kaplan-Meier plot of confirmed cases of progression to dementia due to AD

The Kaplan-Meier plot of confirmed cases of progression to dementia due to AD over time suggested that cases tended to rise after those visits where the CDR was scheduled to be performed (Fig. 2).

Fig. 2

Kaplan-Meier plot of time to progression to dementia due to AD.

CDR profile of progressors

Mean changes from baseline across different CDR domains/boxes and subscales in progressors (those adjudicated to have progressed to dementia) and non-progressors (those adjudicated to have remained at the MCI stage) at the time of an individual’s final adjudication are shown in Table 2. There was no difference in CDR-SB between groups (score of 2.3 in each group) and generally only small differences were observed on individual domains or subscales. However, there was a trend toward greater change (worsening) in the composite functional scale for progressors versus non-progressors (1.2 versus 0.9) whereas there tended to be greater change (worsening) in the composite cognitive scale for non-progressors versus progressors (1.3 versus 1.1). An important caveat in interpreting these data is that due to difficulties in matching databases and missing data, the sample sizes for these analyses were reduced compared to the sample sizes for some of the other analyses; data for only 36 non-progressors were available.

Table 2

CDR mean (SD) change from baseline at the time of the final adjudication result

Final Adjudication Result
CDR Measure	MCI due to AD	Dementia due to AD
	(N = 36)	(N = 380)
Domain (box) scores
1. Memory	0.5 (0.3)	0.3 (0.4)
2. Orientation	0.5 (0.3)	0.4 (0.4)
3. Judgment and Problem Solving	0.4 (0.3)	0.3 (0.4)
4. Community Affairs	0.4 (0.3)	0.4 (0.4)
5. Home and Hobbies	0.4 (0.3)	0.5 (0.4)
6. Personal Care	0.2 (0.4)	0.3 (0.5)
Composite scores
Cognition Subscale (domains 1–3)	1.3 (0.5)	1.1 (0.8)
Functional Subscale (domains 4–6)	0.9 (0.6)	1.2 (0.9)
Total Score	2.3 (0.4)	2.3 (1.3)

If a CDR score was not available at the time of final adjudication or if no adjudication was performed, then the last available CDR score was used. The sample sizes for this analysis are smaller than those for other analyses reported in this paper due to missing data and difficulties in matching up separate databases.

Agreement amongst adjudication committee members

Results of the kappa analyses of agreement on diagnosis among the five adjudication committee members are shown in Table 3. Interpretation of kappa scores is typically classified as: ≤0.20 = no/slight agreement, 0.21 to 0.40 = fair agreement, 0.41 to 0.60 = moderate agreement, 0.61 to 0.80 = substantial agreement, and 0.81 to 1.00 = almost perfect agreement [13]. Agreement was moderate initially (kappa = 0.46) improving to substantial (kappa = 0.79) over the course of the study.

Table 3

Agreement among adjudication committee members (N = 5)

Number of Cases	Kappa	95% CI
31	0.46	0.17–0.75
50	0.54	0.32–0.76
79	0.69	0.53–0.86
104	0.76	0.63–0.89
115	0.79	0.67–0.90

DISCUSSION

The current analyses provide useful insights regarding the adjudication process for determining progression from MCI/prodromal AD to dementia in a large phase III clinical trial. In APECS, all participants had prodromal AD at baseline (MCI and amyloid positive on a PET scan), unlike most other recent trials that included MCI and mild AD dementia [14]. Thus, the sample size for MCI/prodromal AD was substantially larger than previous studies. Over the 2-year course of APECS, there were 581 participants who had data sent for adjudication and 678 adjudications were performed. Given the complexity of the adjudication process, this represents a considerable investment in time and resources.

An important question we sought to address is whether the adjudication process appeared to improve the determination of progression to dementia compared to simply using CDR criteria (CDR-SB change ≥2) or Site Investigator opinion. The results suggest that CDR thresholds alone are insufficient for determining the onset of dementia. For example, at the first assessment only 66% of those who met CDR criteria (regardless of whether they also met Site Investigator criteria) were adjudicated to have progressed to dementia and just 15% of those who met only CDR criteria (i.e., did not also meet Site Investigator criteria for progression) were adjudicated as dementia.

The results were different when looking at Site Investigator criteria as the trigger for adjudication. At the first assessment, 99% of those who met Site Investigator criteria (regardless of whether they also met CDR criteria) were adjudicated to have progressed to dementia, and the same was true for those who met only Site Investigator criteria (i.e., did not also meet CDR criteria for progression). At face value, the finding that the adjudication committee nearly always agreed with the Site Investigator diagnosis suggests that, in future trials, the adjudication process could be abandoned in favor of relying on Site Investigator opinion. However, this conclusion may be over-simplified. It is possible that Site Investigator awareness that their diagnosis was being evaluated by experts had an impact on their assessment. The results may also have been influenced by the expertise of the Site Investigators selected for the study, as well as the study-specific training and guidance they received. Furthermore, the data suggest that approximately 5% (N = 23, Table 1) of progressions would be missed if the CDR criterion were omitted. An alternative would be to adjudicate, or simply review, cases meeting only the CDR criterion and accepting Site Investigator judgment of progression without review.

Several other aspects of the results merit discussion. First, the finding that the number of confirmed cases of progression to dementia rose after each visit where the CDR was scheduled to be routinely performed (Fig. 2) suggests that the CDR results may have influenced the Site Investigator in arriving at their diagnosis. This would argue for administering the CDR at every clinic visit where the Site Investigator is asked to make a diagnostic classification. Second, the analysis of changes in individual CDR domains for progressors versus non-progressors suggests that determination of progression was driven by functional worsening rather than cognitive worsening, as would be expected given that dementia is defined as cognitive decline sufficient to interfere with independence in everyday activities. Finally, our kappa analysis showing moderate initial agreement amongst adjudication committee members exemplifies the point that determining progression to dementia is challenging, even for experienced clinicians. Agreement improved to substantial over the course of the study, which may have been in part due to regular discussion of discordant cases. Case review of challenging situations may be a useful training method even for trials where an adjudication committee is not used.

While the generally large sample size is a strength of the present analyses, several important limitations should be acknowledged. The trial was not designed to address the questions that we attempt to investigate here, and all analyses were post hoc. The adjudicators saw only a selected subset of all those in the trial (i.e., those identified as potentially having progressed to dementia). A more informative approach would be based on a random sample of all participants. Due to its unplanned nature, there were challenges in matching different databases resulting in differing sample sizes, notably a reduced sample size for the analysis of CDR domain scores in progressors versus non-progressors. Finally, we were concerned here only with the merits of different approaches to determining progression and our analysis does not inform on the relative pros and cons of using change in CDR score versus ordinal measures of progression as endpoints for evaluating treatment differences. In the APECS trial, analyses based on change in CDR-SB score (the primary endpoint) and time to progression to probable AD (a secondary endpoint) yielded similar results regarding treatment differences [5].

Clinical trials in AD are resource-intensive; increasing the speed of trials and decreasing their cost will allow more potential therapies to be evaluated [15, 16]. Prodromal AD and early AD are among the most common populations included in AD trials and progression from MCI to dementia is often a primary or secondary outcome [17]. Identifying the progression from prodromal AD to mild AD dementia is a somewhat artificial distinction dependent on determining when functional impairment becomes substantial enough to support the diagnosis of dementia [18]. Use of adjudication committees is a common strategy to reach consensus and provide confidence in the progression from one clinical state to another. Our study suggests that the Site Investigator’s diagnosis has comparable validity to the diagnosis provided by the adjudication committee and captures most cases. This suggests that the Site Investigator diagnosis of progression to dementia alone could be sufficient. Alternatively, to capture the small percent of missed cases, a central review of individuals with decline on the CDR, particularly functional decline, could be included. The value of a formal central adjudication committee appears doubtful.

Footnotes

ACKNOWLEDGMENTS

The authors thank Sheila Erespe of Merck & Co., Inc., Rahway, NJ, USA for assistance in submitting the manuscript.

FUNDING

The study was funded by Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA (MSD).

CONFLICT OF INTEREST

TV, JK, SPM, CF, CL, and MFE are current or former full-time employees and shareholders of Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA (MSD).

JC has provided consultation to Acadia, Alkahest, AlphaCognition, AriBio, Avanir, Axsome, Behren Therapeutics, Biogen, Biohaven, Cassava, Cerecin, Cortexyme, Diadem, EIP Pharma, Eisai, GemVax, Genentech, Green Valley, Grifols, Janssen, LSP, MSD, NervGen, Novo Nordisk, Oligomerix, Ono, Otsuka, PRODEO, Prothena, ReMYND, Renew, Resverlogix, Roche, Signant Health, Suven, United Neuroscience, and Unlearn AI pharmaceutical, assessment, and investment companies. JC is supported by NIGMS grant P20GM109025; NINDS grant U01NS093334; NIA grant R01AG053798; NIA grant P20AG068053; NIA grant P30AG072959; NIA grant R35AG71476; Alzheimer’s Disease Drug Discovery Foundation (ADDF); Ted and Maria Quirk Endowment for the Pam Quirk Brain Health and Biomarker Laboratory; and the Joy Chambers-Grundy Endowment.

CR is the author of the RBANS and receives royalties from the copyright holder, Pearson. He had served as a consultant for MSD.

DATA AVAILABILITY

The data sharing policy, including restrictions, of Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA is available at

References

Jelic

, Kivipelto

, Winblad

(2006) Clinical trials in mild cognitive impairment: Lessons for the future. J Neurol Neurosurg Psychiatry 77,429–438. Erratum in: J Neurol Neurosurg Psychiatry 77, 892.

Burns

, Alexander

, Welsh-Bohmer

, Culp

, Chiang

, O’Neil

, Evans

, Harrigan

, Plassman

, Burke

, Wu

, Lutz

, Haneline

, Schwarz

, Schneider

, Yaffe

, Saunders

, Ratti

TOMMORROW study investigators (2021) Safety and efficacy of pioglita zone for the delay of cognitive impairment in people at risk of Alzheimer’s disease (TOMMORROW): A prognostic biomarker study and a phase 3, randomised, double-blind, placebo-controlled trial.Lancet Neurol 20,537–547.

Feldman

, Ferris

, Winblad

, Sfikas

, Mancione

, He

, Tekin

, Burns

, Cummings

, del Ser

, Inzitari

, Orgogozo

, Sauer

, Scheltens

, Scarpini

, Herrmann

, Farlow

, Potkin

, Charles

, Fox

, Lane

(2007) Effect of rivastigmine on delay to diagnosis of Alzheimer’s disease from mild cognitive impairment: The InDDEx study. Lancet Neurol 6, 501–512. Erratum in: Lancet Neurol 6, 849.

Morris

(1993) The Clinical Dementia Rating (CDR): Current version and scoring rules.. Neurology 43, 2412–2414.

Egan

, Kost

, Voss

, Mukai

, Aisen

, Cummings

, Tariot

, Vellas

, van Dyck

, Boada

, Zhang

, Li

, Furtek

, Mahoney

, Harper Mozley

, Mo

, Sur

, Michelson

(2019) Randomized trial of verubecestat for prodromal Alzheimer’s disease. N Engl J Med 380, 1408–1420.

American Psychiatric Association (2000) Diagnostic and statistical manual of mental disorders, 4th ed., text revision: DSM-IV-TR. Washington, DC.

McKhann

, Drachman

, Folstein

, Katzman

, Price

, Stadlan

(1984) Clinical diagnosis of Alzheimer’s disease: Report of the NINCDS-ADRDA Work Group under the auspices of Department of Health and Human Services Task Force on Alzheimer’s Disease. Neurology 34, 939–944.

Holden

, Milano

, Horner

(2020) Five-factor structure of the RBANS is supported in an Alzheimer’s disease sample: Implications for validation of neuropsychological assessment instruments. Appl Neuropsychol Adult 27, 232–242.

Folstein

, Folstein

, McHugh

(1975) “Mini-mental state". A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 12, 189–198.

10.

Mohs

, Knopman

, Petersen

, Ferris

, Ernesto

, Grundman

, Sano

, Bieliauskas

, Geldmacher

, Clark

, Thal

(1997) Development of cognitive instruments for use in clinical trials of antidementia drugs: Additions to the Alzheimer’s Disease Assessment Scale that broaden its scope. The Alzheimer’s Disease Cooperative Study Alzheimer Dis Assoc Disord 11(Suppl 2), S13–21.

11.

Galasko

, Bennett

, Sano

, Ernesto

, Thomas

, Grundman

, Ferris

(1997) An inventory to assess activities of daily living for clinical trials in Alzheimer’s disease. The Alzheimer’s Disease Cooperative Study. Alzheimer Dis Assoc Disord 11, 33–S39.

12.

Fleiss

(1971) Measuring nominal scale agreement among many raters. Psychol Bull 76, 378–382.

13.

Cohen

(1960) A coefficient of agreement for nominal scales. Educ Psychol Meas 20, 37–46.

14.

Cummings

, Lee

, Zhong

, Fonseca

, Taghya

(2021) Alzheimer’s disease drug development pipeline. Alzheimers Dement (N Y) 7, e12179.

15.

Aisen

, Touchon

, Andrieu

, Boada

, Doody

, Nosheny

, Langbaum

, Schneider

, Hendrix

, Wilcock

, Molinuevo

, Ritchie

, Ousset

, Cummings

, Sperling

, DeKosky

, Lovestone

, Hampel

, Petersen

, Legrand

, Egan

, Randolph

, Salloway

, Weiner

, Vellas

(2016) Registries and cohorts to accelerate early phase Alzheimer’s trials. A report from the E.U./U.S. Clinical Trials in Alzheimer’s Disease Task Force. J Prev Alzheimers Dis 3, 68–74.

16.

Cummings

, Aisen

, Barton

, Bork

, Doody

, Dwyer

, Egan

, Feldman

, Lappin

, Truyen

, Salloway

, Sperling

, Vradenburg

(2016) Re-engineering Alzheimer clinical trials: Global Alzheimer’s Platform Network. J Prev Alzheimers Dis 3, 114–120.

17.

Petersen

, Thomas

, Grundman

, Bennett

, Doody

, Ferris

, Galasko

, Jin

, Kaye

, Levey

, Pfeiffer

, Sano

, van Dyck

, Thal

, Alzheimer’s Disease Cooperative Study Group (2005) Vitamin E and donepezil for the treatment of mild cognitive impairment. N Engl J Med 352, 2379–2388.

18.

Grundman

, Petersen

, Ferris

, Thomas

, Aisen

, Bennett

, Foster

, Jack

Jr , Galasko

, Doody

, Kaye

, Sano

, Mohs

, Gauthier

, Kim

, Jin

, Schultz

, Schafer

, Mulnard

, van Dyck

, Mintzer

, Zamrini

, Cahn-Weiner

, Thal

;Alzheimer’s Disease Cooperative Study (2004) Mild cognitive impairment can be distinguished from Alzheimer disease and normal aging for clinical trials. Arch Neurol 61, 59–66.