Audio review as a measure of quality assurance and control of cognitive assessments in Alzheimer's disease studies: An experience from Japanese Trial-Ready Cohort Study

Abstract

Background

Clinical trials targeting preclinical Alzheimer's disease (AD) require accurate cognitive assessments to detect subtle changes over time. Audio review of assessment sessions has been proposed as a quality assurance (QA) and control (QC) measure, yet evidence regarding its effectiveness remains limited.

Objective

We aim to investigate how audio review contributes to the QA/QC process.

Methods

In the Japanese Trial-Ready Cohort (J-TRC) onsite study, 194 Preclinical Alzheimer's Cognitive Composite (PACC) sessions were audio-recorded and independently reviewed. Examiners were certified psychometrists who had undergone pre-training. Audio reviewers issued “queries” for scoring deviations warranting score amendments and “comments” for procedural deviations or recommended improvements. Mixed-effects models analyzed associations between the number of queries/comments and (1) cumulative PACC examination experience and (2) cumulative received feedback for individual examiners.

Results

Of 194 reviewed sessions, 63.4% contained at least one query, and 84.5% included one or more comments. A higher cumulative number of PACC sessions by each examiner was associated with fewer queries and smaller score changes over time. Notably, the cumulative number of feedback reports was significantly associated with a lower number of comments, suggesting targeted feedback helped refine examiners’ testing manner. However, cumulative feedback was not clearly linked to the frequency of scoring deviations (queries).

Conclusions

These findings indicate that audio review can serve as an effective QA/QC measure by reducing non-adherence to assessment standards.

Introduction

Along with the approval of disease-modifying therapy (DMT) drugs for early Alzheimer's disease (AD), ¹ the development of DMTs for preclinical AD, an earlier phase characterized by evidence of amyloid burden but without significant clinical symptoms, ² is also anticipated as the next target to prevent the development of AD.^2,3 Since individuals at this stage have no or minimal cognitive impairment, minimizing any controllable bias that could affect the accuracy of cognitive assessments is critical to detecting subtle cognitive changes seen in this population. The capability of individual examiners to conduct cognitive tests accurately, particularly in adhering to standard test administration protocols, is one of the major keys in reducing potential sources of bias. ⁴

While pre-training and certification of examiners before initiating cognitive assessments in clinical studies or trials may serve as effective measures to ensure their basic skills and control their initial variability in adhering to the standard administration of these tests, it remains challenging to verify the degree of adherence to the procedures once they are involved in projects. This arises because the presence of an observer is reported to alter the test performance, ⁵ and consequently cognitive tests are typically conducted one-on-one in private environments. In essence, it is a matter of quality assurance (QA) and control (QC): QA has been defined as “the process set up to ensure the quality of a product and to reduce the likelihood of errors”, and QC is defined as “the identification and correction of errors within a product” in earlier literature. ⁴

One of the QA/QC measures for cognitive assessments is the introduction of “audio review” in ongoing research projects, where the audio-recorded (e.g., audiotaping or videotaping) cognitive test sessions are periodically reviewed by reviewers, ⁶ with their reports then returned to individual examiners for feedback, and amendments to scores are made if needed. This approach has actually been incorporated into clinical trials including the A4 Study, a phase-3 clinical trial examining solanezumab for preclinical AD individuals in the U.S and Japan, ⁷ and has also been implemented in our ongoing Japanese Trial-Ready Cohort (J-TRC) onsite study, which targets individuals with preclinical AD or mild cognitive impairment (MCI). ⁸ To date, however, there is limited evidence on how audio review contributes to the QA/QC process, which we aim to investigate using data from the J-TRC study.

Methods

Study design

This is a retrospective study based on the J-TRC onsite study, ⁸ and informed consent for audio recording and data analysis was obtained from participants upon registration. The J-TRC onsite study is an ongoing clinical study initiated in 2020, aiming to recruit preclinical or MCI individuals with evidence of AD pathology to facilitate future clinical trials. ⁹ In this study, participants are invited from web ¹⁰ or from local registries to study facilities where detailed in-person cognitive assessments, amyloid PET, and other evaluations are conducted. Those who are amyloid PET-positive and have a Clinical Dementia Rating Global Score (CDR-GS) of 0 or 0.5 are registered to form the “trial-ready cohort”. Further details of the J-TRC onsite study can be found in our previous reports.^10,11

Cognitive assessments and reviewing

In the J-TRC onsite study, as is the case with some clinical trials for AD (e.g., A4 Study investigating solanezumab ⁷ and AHEAD 3–45 Study investigating lecanemab for preclinical AD ¹² ), cognitive assessments using several tests are performed by examiners, most of whom are certified psychometrists in Japan. All examiners receive training before participating in study assessments. The audio-reviewed tests in the J-TRC onsite study include Mini-Mental State Examination (MMSE), Wechsler Memory Scale-Revised (WMS-R) Logical Memory Immediate Recall (LM-I), WMS-R Logical Memory Delayed Recall (LM-D), Digit Symbol Substitution Test (DSST), and Free and Cued Selective Reminding Test (FCSRT) scores, from which the Preclinical Alzheimer's Cognitive Composite (PACC) score is calculated. ¹³

The workflow of one PACC session, from examination to feedback on its review report, is outlined in Figure 1. In the J-TRC onsite study, a single PACC measurement comprising four subdomain tests (i.e., MMSE, LM, DSST, and FCSRT) is conducted for a participant by an examiner (Figure 1A), without fixed pairing rules for participants and examiners. For each examiner, PACC sessions are routinely audio recorded at a predetermined schedule (i.e., first, second, third, and every tenth session thereafter), and the recording data is uploaded to the Electronic Data Capture (EDC) system (Figure 1B).

Figure 1.

Flowchart of audio-review of cognitive assessment in the J-TRC onsite study. The workflow of one PACC session, from examination to feedback on its review report, is outlined. In the J-TRC onsite study, a single PACC measurement comprising four subdomain tests (i.e., MMSE, LM, DSST, and FCSRT) is conducted for a participant by an examiner (A), without fixed pairing rules for participants and examiners. For each examiner, PACC sessions are routinely audio recorded at a predetermined schedule (i.e., first, second, third, and every tenth session thereafter), and the recording data is uploaded to the Electronic Data Capture (EDC) system (B). The audio records are then reviewed by one of the external psychometrist reviewers randomly assigned (C) based on a predetermined checklist (Supplemental Table 1). If the examiner's scoring procedure deviates from the scoring standard and requires amendments for accurate scoring, a query is issued regarding that point. If the testing manner deviates from the assessment standard and is deemed inappropriate or in need of improvement, a comment is included in the report. The reports, which may include queries and/or comments, are uploaded to the EDC system (D). The reports are subsequently disclosed to individual examiners on the computer, providing feedback on their PACC assessments (E). If the reports include queries, appropriate corrections are to be made to the original document (F), and these corrections are also reflected in the scoring data on the EDC system with timestamp (G). Each procedure–audio recording, data uploading, reviewing, report uploading, and disclosure to examiners–is conducted sequentially and not in batch. PACC: Preclinical Alzheimer Cognitive Composite; EDC: Electronic Data Capture; J-TRC: Japanese Trial-Ready Cohort; CRC: Clinical Research Coordinator.

The audio records are then reviewed by one of four external psychometrist reviewers randomly assigned (Figure 1C) based on a predetermined checklist (Supplemental Table 1). If the examiner's scoring procedure deviates from the scoring standard and requires amendments for accurate scoring, a query is issued regarding that point. If the testing manner deviates from the assessment standard and is deemed inappropriate or in need of improvement, a comment is included in the report. The reports, which may include queries and/or comments, are uploaded to the EDC system (Figure 1D). The reports are subsequently disclosed to individual examiners on the computer, providing feedback on their PACC assessments (Figure 1E).

If the reports include queries, appropriate corrections are to be made to the original document (Figure 1F), and these corrections are also reflected in the scoring data on the EDC system with timestamp (Figure 1G). Each procedure–audio recording, data uploading, reviewing, report uploading, and disclosure to examiners–is conducted sequentially and not in batch.

Statistical analyses

All data processing and analyses were conducted using R. For each PACC session reviewed and reported, we collected the following variables: study site ID, examiner ID, reviewer ID, the number of queries included in its report (abbreviated as “# Queries”), the number of comments included in its report (abbreviated as “# Comments”), the degree of change in each score following correction (Δ score, e.g., Δ PACC, Δ MMSE, Δ LMD, Δ DSST, Δ FCSRT; Δ score = 0 if no action was taken to the query or no queries were issued) calculated from the original score based on the queries included, the cumulative number of all PACC sessions conducted by individual examiners up to that session (abbreviated as “cPACC”), the cumulative number of feedback (FB) disclosed to the examiner up to that session (abbreviated as “cFB”), the number of days elapsed since the previous session for the examiner (abbreviated as “Days”), and whether the session was the examiner's first (abbreviated as “First”). PACC score was calculated based on the following formula, using the data of J-TRC onsite participants who have a CDR-GS score of 0 as a reference for data normalization:

$$PACC=\left\{{\displaystyle \frac{LMD}{SD(LM{D}_{CDRGS0})}+{\displaystyle \frac{DSST}{SD(DSS{T}_{CDRGS0})}+{\displaystyle \frac{MMSE}{SD(MMS{E}_{CDRGS0})}+{\displaystyle \frac{FCSRT}{SD(FCSR{T}_{CDRGS0})}}}}}\right\}/4$$

We examined mixed-effects models based on the above variables using the following equations:

$$\ln (\mathrm{\#}Queries)={\beta }_0+{\beta }_{1}cFB/10+{\beta }_{2}cPACC/100+{\beta }_{3}Days/1000+{\beta }_{4}First+{\gamma }_1+{\gamma }_2+{\gamma }_3$$

$$\ln (\mathrm{\#}Comments)={\beta }_0+{\beta }_{1}cFB/10+{\beta }_{2}cPACC/100+{\beta }_{3}Days/1000+{\beta }_{4}First+{\gamma }_1+{\gamma }_2+{\gamma }_3$$

$$\ln (Oddsofabs(\mathrm{\Delta }score)>0)={\beta }_0+{\beta }_{1}cFB/10+{\beta }_{2}cPACC/100+{\beta }_{3}Days/1000+{\beta }_{4}First+{\gamma }_1+{\gamma }_2+{\gamma }_3$$

Here, γ₁, γ₂, and γ₃ represent random intercepts corresponding to site ID, examiner ID, and reviewer ID, respectively. The cFB is the primary variable of interest regarding its association with the targeted outcomes, and we simultaneously included cPACC and cFB because repeated experience in PACC assessments–even without feedback–may contribute to enhance examiners’ ability to conduct assessments more accurately. We used Poisson regression for # Queries (equation I) and # Comments (equation II). Logistic regression was used for the equation (III) to analyze the odds of the absolute value of Δ score exceeding a threshold of 0, thereby examining the likelihood of report queries changing the original score.

In order to determine the cumulative number of feedback instances, cFB, in the above equations, we needed to identify when examiners actually confirmed their reports. However, no deadline was set for examiners to confirm their feedback reports, and the timing of when they opened the feedback report file on the computer was not timestamped in the EDC system. Although the exact timing of feedback confirmation likely varied between individual examiners, we attempted to estimate it based on the available log of uploaded timings. We assumed three possible timing when examiners may have confirmed their reports (Figure 1, assumptions [a] ∼[c]): (a) the timing when the review report, with or without queries and comments, was uploaded to the EDC system and became available to the examiners, (b) the timing when the review report, including queries or comments, was uploaded to the EDC system and became available to the examiners, and (c) the timing when corrections to scoring were uploaded to the EDC system in response to issued queries. Assumptions (a) and (b) represent the earliest possible timings, while assumption (c) represents the latest possible timing.

Ethics

The J-TRC study has been approved by the University of Tokyo Graduate School of Medicine Institutional Ethics Committee (ID:2019132NI-(3)).

Results

In the J-TRC onsite study from August 2020 to February 2024, a total of 852 PACC sessions were conducted for 685 unique participants by 27 unique examiners across seven facilities. During this period, audio review was conducted for 194 sessions (22.5%) involving 182 unique participants (Figure 1), among which 164 (84.5%) session reports included comments and 123 (63.4%) included queries.

Demographics of the J-TRC participants whose PACC sessions were reviewed were as follows: median age of 73 years (IQR: 67.3 ∼ 76.0), 41.2% female, a CDR-GS of 0.5 in 31.5% and 0 in the remaining 68.5%, APOE ε4 positive in 22.2%, and amyloid PET positivity (visual reads) in 25.9%. While a particular disparity in the number of reviews per reviewer was observed (see Supplemental Figure 1 for details), no significant deviations were identified in the allocation of reviewers to individual examiners (p = 0.053, Chi-squared test) or to individual facilities (p = 0.090, Chi-squared test).

Frequency of queries and comments by test is summarized in Supplemental Table 2: LM and FCSRT showed a relatively high rate to have queries issued (29.4% and 25.3%, respectively), and more than half of the FCSRT and MMSE sessions included some form of comment. The direction of change in test scores from the initial score following queries (e.g., whether the score became larger or smaller) varied by test, and the calculated PACC score showed slight average increase of 0.022.

The relationship between examiners’ cumulative experience and the number of queries/comments included in each report is illustrated in Figure 2. The examiner-level plots (Figure 2A-D), where each point corresponds to an individual examiner, indicate that the cumulative number of queries or comments per examiner tended of decline as PACC experience increased, although no significant association was observed.

Figure 2.

Review results by the cumulative experience of PACC assessments. The Examiner-level plots (A-D), where each point corresponds to an individual examiner, indicate that the cumulative number of queries or comments per examiner tended of decline as PACC experience increased, although no significant association was observed. The session-level plots (E-H), where each point corresponds to an individual session, indicate a significant negative correlation between the cumulative number of queries or comments and the cumulative number of PACC experiences of individual examiners (E, G). The degree of change in PACC scores in response to the queries showed a gradual process, with the magnitude of the absolute change decreasing (F), and the absolute change in PACC scores actually showed a significant negative correlation with cumulative PACC experience (H). PACC: Preclinical Alzheimer Cognitive Composite.

Meanwhile, the session-level plots (Figure 2E-H), where each point corresponds to an individual session, indicate a significant negative correlation between the cumulative number of queries or comments and the cumulative number of PACC experiences of individual examiners (Figure 2E, G). The degree of change in PACC scores in response to the queries showed a gradual process, with the magnitude of the absolute change decreasing (Figure 2F), and the absolute change in PACC scores actually showed a significant negative correlation with cumulative PACC experience (Figure 2H). The number of comments per report had a median of 2 (IQR: 1 ∼ 5), and the number of queries per report had a median of 1 (IQR: 1 ∼ 2), with a significant difference observed (paired t-test, p < 0.001).

Next, mixed-effects models were applied to analyze the session-level longitudinal data (Figure 2E, G), and the results are displayed as forest plots in Figure 3. Figure 3A presents the log-transformed coefficients and their 95% confidence intervals (CIs) from models with the following target variables: # Queries, # Comments, and the binary indicator of whether the absolute change in Δ PACC due to queries is > 0. Variables with significant change (i.e., a 95% CI not spanning 0) are marked in red. Except for the intercept, the cumulative number of feedback provided to individual examiners was the only variable significantly associated with a lower number of comments included in the reports, regardless of the definitions of report disclosure timing (a ∼ c). Meanwhile, the cumulative times of PACC experiences was significantly associated with a lower likelihood of Δ PACC changes, but only when the report disclosure timing definition was (c). First time PACC session for individual examiners was the only variable significantly associated with a lower number of queries included in the reports, regardless of the definitions of report disclosure timing (a ∼ c).

Figure 3.

Mixed-Effects model results. Mixed-effects models were applied to analyze the session-level longitudinal data (Figure 2E, G), and the results are displayed as forest plots. The log-transformed coefficients and their 95% confidence intervals (CIs) from models with the following target variables: # Queries, # Comments, and the binary indicator of whether the absolute change in Δ PACC due to queries is > 0 are shown in (A). Variables with significant change (i.e., a 95% CI not spanning 0) are marked in red. For target variables # Queries or # Comments, Poisson regression was used, and logistic regression was used for the target binary variables such as whether Abs(Δ score) > 0. Except for the intercept, the cumulative number of feedback provided to individual examiners was the only variable significantly associated with a lower number of comments included in the reports, regardless of the definitions of report disclosure timing (a ∼ c). Meanwhile, the cumulative times of PACC experiences was significantly associated with a lower likelihood of Δ PACC changes, but only when the report disclosure timing definition was (c). First time PACC session for individual examiners was the only variable significantly associated with a lower number of queries included in the reports, regardless of the definitions of report disclosure timing (a ∼ c). Additionally, the log-transformed coefficients and their 95% CIs from models targeting binary outcomes indicating whether the absolute changes in Δ MMSE, Δ LMD, Δ DSST, and Δ FCSRT following queries are > 0 are demonstrated in (B). No variables other than the intercept were associated with the absolute change in the subtests (i.e., MMSE, LM, DSST, and FCSRT). PACC: Preclinical Alzheimer Cognitive Composite; FB: feedback; MMSE: Mini-Mental State Examination; LMD: Logical Memory Delayed Recall; DSST: Digit Symbol Substitution Test; FCSRT: Free and Cued Selective Reminding Test; Def: definition; CI: confidence interval.

Additionally, Figure 3B demonstrates the log-transformed coefficients and their 95% CIs from models targeting binary outcomes indicating whether the absolute changes in Δ MMSE, Δ LMD, Δ DSST, and Δ FCSRT following queries are > 0. No variables other than the intercept were associated with the absolute change in the subtests (i.e., MMSE, LM, DSST, and FCSRT) (Figure 3B).

Discussion

In this study, we investigated the effectiveness of an audio review process as a QA/QC measure in cognitive assessments within the J-TRC onsite study. ⁸ Our analysis focused on how the feedback from audio review related to deviations from standard testing procedures, as quantified by the number of queries and comments, as well as changes in actual cognitive test scores. As a result, we found that the cumulative number of feedback instances provided to individual examiners was associated with a significant reduction in the number of comments included in reports. This demonstrates that audio review can serve as an effective QA/QC measure, highlighting the potential value of systematic QA/QC processes in clinical studies or trials involving individuals with preclinical AD or MCI.

The significant association between the cumulative number of feedback (‘cFB’) and a reduction in the number of comments in session-level data (Figure 3A, definitions [b, c]) was noted. It is reasonable to assume that feedback contributes to refining an examiner's testing manner, which typically relates to aspects of the testing process beyond mere scoring errors. This finding emphasizes the importance of not only initial examiner certification and training but also the ongoing implementation of targeted feedback to maintain and improve assessment quality over time. In contrast, it remains uncertain why a similar association was not observed between queries–which reflect scoring errors–and the cumulative feedback times (Figure 3A). The finding that the first session/feedback for examiners was associated with a higher risk of issuing queries is understandable, and the effect of initial feedback identifying more comments may have canceled out the effect of subsequent cumulative feedback in reducing the number of comments.

The decline in the number of queries and comments with increasing cumulative PACC sessions was observed, apart from the effect of feedback (Figure 2, Figure 3A). This finding is consistent with earlier literature on the QA/QC of cognitive assessments ⁴ and supports the simple assumption that examiner performance improves with experience. As examiners become more familiar with the standardized administration procedures, they may be likely to commit fewer deviations.

The lack of precise timestamps for when examiners actually checked their feedback, combined with the retrospective nature of this study, introduced potential variability into our analysis (by definitions [a] ∼ [c]). The significant association between cumulative PACC experiences and lower odds of score changes was observed only under the latest timing assumption (definition [c], when corrections were uploaded), suggesting that the effect of feedback may only be realized when examiners incorporate corrections into their scoring practices. It may be better for future studies to track feedback confirmation timing more accurately.

The absolute change in PACC scores or subdomain test scores following queries was not associated with any of the imputed variables except for the intercept (Figure 3A,B). This implies that factors related to audio reviews may not have a significant effect on revising the original scores to accurate scores, on average. This can be considered good news, as it suggests that reviewing all PACC sessions for future participants may not be necessary, nor does it imply that previous unreviewed PACC session scores were significantly inaccurate.

Taken together, our findings have important implications for clinical studies and trials–particularly those targeting preclinical AD or MCI, where early detection of subtle cognitive changes is critical–highlighting the value of incorporating audio reviews into study protocols by default as a standard QA/QC measure. Such measures not only help identify and correct deviations in real time but also appear to facilitate examiner learning and performance improvement over time.

Our study includes several limitations. First, the retrospective study design prevents us from sufficiently examining causal relationships between examiner experience, feedback, and improvements in cognitive assessment quality. Second, the absence of precise timestamps for feedback confirmation introduces uncertainty into the estimation of cumulative feedback effects. Third, while we identified audio review may be effective for QA/QC of PACC assessments, its efficacy remains uncertain for other frequently used cognitive and clinical assessments such as the Montreal Cognitive Assessment (MoCA), Alzheimer's Disease Assessment Scale-Cognitive Subscale (ADAS-Cog), and Clinical Dementia Rating (CDR). Lastly, the study's findings may not be generalizable beyond the specific context of preclinical AD and MCI assessments in Japan, and replication in other settings is needed.

In conclusion, this study provides evidence that systematic implementation of audio review and feedback as a QA/QC measure can reduce deviations in test administration, thereby enhancing the quality of cognitive assessments in clinical trials and studies targeting individuals with preclinical AD and MCI.

Supplemental Material

sj-docx-1-alr-10.1177_25424823251349188 - Supplemental material for Audio review as a measure of quality assurance and control of cognitive assessments in Alzheimer's disease studies: An experience from Japanese Trial-Ready Cohort Study

Supplemental material, sj-docx-1-alr-10.1177_25424823251349188 for Audio review as a measure of quality assurance and control of cognitive assessments in Alzheimer's disease studies: An experience from Japanese Trial-Ready Cohort Study by Kenichiro Sato, Ryoko Ihara, Yoshiki Niimi, Kazushi Suzuki, Atsushi Iwata and Takeshi Iwatsubo in Journal of Alzheimer's Disease Reports