Abstract
This study addresses the need for screening tests that can discriminate between dementia, mild cognitive impairment (MCI), and normal age-related memory functioning in understudied populations. One hundred sixty-four Romanian patients with dementia, MCI, and community members were assessed with the Memory of Objects and Digits and Examination of Memory Malingering (MODEMM), the MMSE-2 standard version (MMSE-2-SV), and quick mild cognitive impairment (QMCI) screen to determine each instrument’s ability to distinguish between diagnostic groups and controls. The integral version of the MODEMM (MODEMM-I) classified diagnostic groups with outstanding accuracies (area under the curve [AUC] = .91–.99, p < .001), similar to QMCI (AUCs = .92–.98, p < .001) and the MMSE-2-SV (AUCs = .89–.99, p < .001). Cutoffs were adjusted for each diagnostic condition according to levels of education. Despite high-accuracy values, the MODEMM subscales were less sensitive to MCI than the integral version. Results support the MODEMM-I as an accurate screening tool for cognitive impairment in the understudied Romanian population.
Introduction
The estimated number of people living with dementia worldwide was 46.8 million in 2015 (Prince et al., 2015). As of 2019, Romania had an estimated 341,195 (1.7%) people with dementia, which was estimated to increase by 69% by 2050, largely due to population aging (GBD 2019 Dementia Forecasting Collaborators, 2022). Some studies have found that the prevalence and incidence of dementia have remained stable or even declined in some countries (Wu et al., 2017). However, there is no clear consensus on the observed trends in prevalence and incidence, and, presently, no single risk or protective factor has been identified to fully explain these observations. In this framework, extensive research has been conducted in the last decade to establish clinical and biological diagnostic criteria for mild cognitive impairment (MCI) and dementia (e.g., Alzheimer’s dementia; Dubois et al., 2010, 2014; Jack et al., 2011, 2018, 2024; Sperling et al., 2011). Numerous studies by different research groups point to three stages in the evolution of Alzheimer’s disease, including (a) an asymptomatic or preclinical stage, (b) a prodromal stage where MCI is evident, and (c) a final stage of dementia (Jack et al., 2024; Jessen et al., 2014; McKhann et al., 2011).
MCI and mild neurocognitive disorder (mild NCD; American Psychiatric Association [APA], 2013) are closely related conditions that often overlap in clinical presentations and only appear to differ with respect to the age of onset, seemingly due to sampling procedures: while MCI was studied primarily on the elderly population (aged ≥ 50; Roberts & Knopman, 2013), mild NCD includes all age groups (Stokin et al., 2015). After years of controversy since MCI was first introduced as a term for the prodromal stage of dementia (Reisberg et al., 1988), a current consensus has been reached on the criteria and the amnestic and non-amnestic subtypes of MCI (Winblad et al., 2004).
While some studies have found that the amnestic type is the most common (Petersen et al., 2010), the existence of a pure amnestic type of MCI is still a controversial topic (Golomb et al., 2004). Among the cognitive domains affected by MCI, memory is described by the Diagnostic and Statistical Manual of Mental Disorders, fifth edition (DSM-5) as the most complex, encompassing multiple subdomains: memory processing (comprising encoding, storage, and retrieval), procedural or motor memory, prospective memory, and working memory. Both working memory (due to its reliance on intact receptive functions and attention) and prospective memory (due to its planning component) are included within the broad term of executive functioning (Harvey, 2019).
There is evidence suggesting that all MCI subtypes are at risk of developing dementia. For example, deficits in immediate word recall may develop before deficits in delayed paragraph recall (Rountree et al., 2007). Routine screening is currently not recommended due to the lack of evidence of its effectiveness (Ismail et al., 2020; Patnode et al., 2020). However, there is overwhelming evidence suggesting that subjective complaints of memory difficulties or other cognitive dysfunctions, as noticed by family/friends/caregivers in any of the cognitive domains, should be taken seriously into clinical account (Bruscoli & Lovestone, 2004; Chen et al., 2017).
Subjective cognitive decline is the term coined to represent the preclinical stage before the decline can be objectively corroborated (i.e., biologically and psychometrically; Jessen et al., 2014). Despite being difficult to determine objectively, subjective cognitive decline has been established as a criterion for MCI (Albert et al., 2011; Dubois et al., 2007) and a risk factor for developing dementia (An et al., 2024; Pike et al., 2022; Wen et al., 2021). As such, the DSM-5 criteria include either subjective concerns or clinical practitioners’ concerns regarding cognitive decline, together with an objective test result supporting the decline (Gifford et al., 2014).
Despite the pervasive pattern of dysfunctions occurring in dementia, it is memory deficits that are mostly associated with this disorder by the public, largely due to its portrayal in various popular media. Most people would recognize the basic features of poor memory, such as forgetfulness or losing small objects, and report these symptoms first as part of their subjective cognitive decline during clinical assessment. On the contrary, poor memory can also be a common symptom often endorsed by examinees who intend to fabricate or exaggerate their impairment to obtain external benefits (van Oorsouw & Merckelbach, 2010). Whenever memory deficits are reported by patients, employing performance validity tests (PVTs) to verify the credibility of complaints is required for a comprehensive assessment in clinical settings (Sweet et al., 2021). However, PVT performance can be affected by the presence of cognitive impairment, requiring the adjustment of traditional cutoffs in samples with MCI or dementia to maintain adequate specificity levels (Dandachi-FitzGerald et al., 2020; Martin et al., 2020). In addition, dementia is often a rule-out condition in the selection of clinical participants for PVT studies due to the increased risk of false-positive errors, which affects the classification accuracy of the instruments (Bortnik & Dean, 2021; Corriveau-Lecavalier et al., 2022; Gaudet et al., 2022; Grewal et al., 2024). To the best of our knowledge, only a few memory tests have been validated as embedded PVTs in samples with diagnosed MCI or dementia (i.e., digit span-based indicators; (Dean et al., 2009; Martin et al., 2022). Therefore, there is a need for instruments that can assess both cognitive impairment and performance validity, especially in populations with MCI and dementia, both worldwide and in Romania.
In Romania, the Mini-Mental State Examination (MMSE; Folstein et al., 1975) and Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005) are the most used instruments in the evaluation of MCI and dementia, despite their limited diagnostic utility. In addition, repeated testing with the same instruments often produces practice effects that impact their accuracy as screeners for cognitive impairment. Other particularities stand out in clinical evaluations, such as low levels of education and poor health literacy, which are frequently encountered in many elderly patients (Oneţiu et al., 2016). Such patients often report having received eight or fewer years of formal schooling, partly due to restrictions imposed by the Communist regime, which lasted until 1989 (Tatomirescu et al., 2025), along with limited access to later education and health care services. In this framework, late detection of dementia is not uncommon in Romania: factors, such as the scarcity of epidemiological studies on the prevalence of dementia and the low national awareness of the disorder, its symptoms, and consequences prevent patients and their caregivers from seeking early investigation (Oneţiu et al., 2016). In addition, the high likelihood of MCI conversion into dementia, as documented in Romanian samples (i.e., 43.8%; Roman-Filip et al., 2013), calls for the implementation of effective detection instruments in this population.
This study aims to provide supplemental knowledge within this framework and offer clinical practitioners a novel screening and diagnostic tool to measure memory impairment and detect invalid performance, specifically the Memory of Objects and Digits and Examination of Memory Malingering test (MODEMM). The PVT subform of the MODEMM has recently been validated in experimental (Crişan et al., 2022) and criterion-group studies with heterogeneous clinical samples, including patients with MCI (Crișan & Sava, 2023). Nevertheless, empirical evidence is needed on its effectiveness as a screener for MCI and dementia in Romanian patients. This study was conducted as the first step in a longitudinal design attempting to evaluate the MODEMM’s ability to detect memory-related changes over time in both clinical and non-clinical participants.
The objectives of this diagnostic accuracy study were to (a) determine the effectiveness of the MODEMM as a screening tool for dementia and MCI and establish cutoff scores for both conditions and (b) establish the accuracy of the MODEMM in discriminating between dementia and MCI. We hypothesized that (a) the classification accuracies of the MODEMM would be comparable with other cognitive screening instruments and (b) differential cutoffs on each instrument would be needed for identifying MCI and dementia, respectively.
Method
Participants
We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study. Prior to data collection, a power analysis was conducted in the MedCalc (2023) software to determine the sample size for accuracy statistics, indicating 26 participants per group, with a total of 78 participants in three groups for a minimum area under the curve (AUC) value of .80. Rigorous inclusion and exclusion criteria were established to avoid heterogeneity among the groups and ease replicability. Full inclusion and exclusion criteria are appended as Supplemental Material.
The total sample included 164 participants (82 male; MAge = 65.45, SDAge = 7.65; MEducation = 12.28, SDEducation = 3.69; range = 8–30). Of the sample, 98.7% identified as Romanian and 1.21% as Romani. Self-reported native languages were Romanian for 93.2%, Hungarian for 4.87%, Romani for 1.21%, and German for 0.6% of the sample. All participants were fluent in Romanian. General inclusion criteria were an age between 55 and 79, at least 8 years of formal education, and no severe levels of depression at the time of assessment (a score ≤ 12 on the 15-item Geriatric Depression Scale [GDS-15]; Sheikh & Yesavage, 1986). Reasons for selecting age 79 as the upper limit included (a) controlling for possible confounds of cognitive decline associated with aging, (b) balancing the between-group age difference, and (c) ease of test administration in examinees under 80. At least 8 years of formal schooling were required to ensure participants’ literacy and balance between-group educational differences. General exclusion criteria were uncorrected sensorial deficiencies, difficulties communicating in Romanian, loss of writing abilities, and severe levels of depression (score ≥ 12 on the GDS-15).
Participants were divided into three diagnostic groups: dementia (n = 56), MCI (n = 55), and control (n = 53). To be included in the dementia group, participants had to have a formal diagnosis of Alzheimer’s dementia, vascular, or mixed dementia and score ≥ 11 on the Mini-Mental State Examination-2 Standard Version (MMSE-2-SV). In Romania, a formal diagnosis of dementia is made by physicians (i.e., neurologists or psychiatrists) and implies clinical evaluation and neuroimaging results showing cognitive impairment (i.e., computed tomography [CT] or magnetic resonance imaging [MRI] results). Patients with severe levels of deterioration (a score ≤ 10 on MMSE-2 standard version [MMSE-2-SV]) and a formal diagnosis of Parkinson’s disease, Lewy body disease, or delirium were excluded from the study. Clinical reasons for exclusion involved psychotic symptoms often encountered in early stages of Lewy body disease, confusion in acute delirium, and less cognitive dysfunctions reported in early stages of Parkinson’s disease. Patients with frontotemporal dementia were also excluded based on the high occurrence of aphasia and speech-related dysfunctions in this disorder. In addition, the higher prevalence of Alzheimer’s and vascular dementia documented worldwide (Johnson et al., 2024) and in Eastern Europe (Cenko et al., 2021), including Romania (Roman-Filip et al., 2013), facilitated the recruitment of patients with these diagnoses.
Participants were included in the MCI group if they had either a formal diagnosis of MCI or mild NCD or met the DSM-5 criteria for subjective cognitive decline corroborated by a caregiver or a clinician (Albert et al., 2011; Dubois et al., 2007; Gifford et al., 2014), presented an objective cognitive deficit (i.e., scores on the story memory and processing speed items from MMSE-2-EV below the mean age and educational level), and minimal general loss of function (stage 2–3 on the Global Deterioration Scale [GDS]). Exclusion criteria for MCI were any formal diagnostic of any type of dementia. Finally, participants were included in the control group if they reported no formal neurological or psychiatric diagnosis, as well as no cognitive deficit or memory complaints other than those of normal aging, at the time of data collection.
Measures
MODEMM (Sava & Crișan, 2022) is a new instrument developed to evaluate memory performance and performance validity, normed, and validated on the Romanian population. It contains two types of stimuli: visual items (12 pictures of common objects) and verbal items (nine double strings of digits of increasing lengths, similar to the digit span subtests of the Wechsler scales) that are presented to the participant as memory tasks. The list of objects to be memorized is presented visually, not verbally, to avoid overfocusing on phonology, and the digit strings are presented verbally. The integral test (MODEMM-I) comprised nine tasks administered in the following sequence: (a) for the visual items: two immediate recall trials, one verbal cued recall trial, two forced-choice recognition trials (visual and verbal), and one delayed recall trial administered at the end and (b) for the verbal items: one direct, one reversed, and one sequential digit span trial. The administration of the integral MODEMM requires approximately 15 to 20 minutes.
MODEMM has four administration forms: integral (I), with its three subforms: objects (O), digits (D), and malingering (M), the latter being used as a PVT. Administering the MODEMM-I entails administering all three subforms, so that clinicians can collect all the information the instrument can provide within a single assessment. Table 1 and Figure 1 illustrate the structure, administration forms, and scoring procedures of the test.
Structure, Administration Forms, and Types of Scores of the MODEMM.
Note. Trials are listed in the order of administration; MODEMM-I: integral administration of the MODEMM; MODEMM-O: MODEMM-Objects; MODEMM-D: MODEMM-Digits; MODEMM-M: MODEMM-Malingering; Adm = administered trials; scores: MODEMM-I = A1 + A2 + A3 + D1 + D2 + D3; MODEMM-O: A1 + A2 + A3; MODEMM-D: D1 + D2 + D3; MODEMM-M: inclusion false-positives computed from I1; mean recognition and inconsistent recognition computed from R1 + R2.
Structure and Administration Forms of the MODEMM.
Multiple scoring options are available, depending on the administration form. For the present study, we used the total scores for three forms—MODEMM-O, MODEMM-D, and MODEMM-I—calculated as follows: the MODEMM-O total score is obtained by adding the scores from the three object recall trials (i.e., two immediate and one delayed). The MODEMM-D total score is computed by summing the scores from the three digit span trials. The MODEMM-I total score represents the sum of the MODEMM-O and MODEMM-D scores. Although MODEMM-M and MODEMM-O involve largely the same tasks (except for the delayed recall task included only in MODEMM-O), the scoring of MODEMM-M is based on different indicators, which are not computed for the MODEMM-O score. Specifically, it includes three PVT indicators, namely, inclusion false-positives encountered in the cued recall task, recognition inconsistencies, and mean recognition computed from the recognition trials (Crișan & Sava, 2023; Crişan et al., 2022). Given that this study focused on evaluating MODEMM’s accuracy as a screener for cognitive decline, we analyzed only the scores from the I, O, and D forms.
The quick mild cognitive impairment (QMCI; O’Caoimh, 2015) screen is a short cognitive screener that can be administered and scored in 5 to 10 minutes. Evidence suggests that it is sensitive in differentiating NCD from MCI and dementia (O’Caoimh et al., 2012), as well as between dementia subtypes (O’Caoimh & Molloy, 2019). QMCI contains six subsets: orientation, registration, clock drawing, delayed recall, verbal fluency, and logical memory. An analysis of its subtests suggests they all contribute to outlining the diagnosis, with registration being the weakest and logical memory and verbal fluency the strongest (O’Caoimh et al., 2013). The recommended cutoff scores are ≤ 75 for MCI and ≤ 50 for dementia (O’Caoimh et al., 2016).
The Mini-Mental State Examination Second Edition (MMSE-2; Folstein et al., 1995) was developed as an update to the MMSE (Folstein et al., 1975) to provide finer discrimination between MCI and NCD. The MMSE-2-SV was used in this study, which includes the same five areas of cognitive functioning as the original MMSE: orientation, immediate memory, attention/concentration, delayed recall, language, and praxis. It comprises the following subsets: registration, time and special orientation, recall, attention and calculation, language, and drawing. In addition, the extended version (MMSE-2-EV) includes the story memory and processing speed subtests. The recommended cutoff scores for the standard version are ≤ 28 for MCI and ≤ 26 for dementia (Sleutjes et al., 2020). The instrument can be administered and scored in 5 to 10 minutes.
The Geriatric Depression Scale Short form (GDS-15; Sheikh & Yesavage, 1986) was administered to measure the severity of depression symptomatology. Scores range from 0 to 15, with scores of 5 to 8 indicating mild depression, 9 to 11 moderate depression, and 12 to 15 severe depression (Sheikh & Yesavage, 1986).
The GDS (Reisberg et al., 1982) is commonly used in Romanian clinical settings for assessing degenerative dementia. It describes seven stages: stage 1 with no cognitive decline, stage 2 with very mild cognitive decline (age-associated memory impairment), stage 3 with mild cognitive decline (MCI), stage 4 with moderate cognitive decline (mild dementia), stage 5 with moderately severe cognitive decline (moderate dementia), stage 6 with severe cognitive decline (moderately severe dementia), and stage 7 with very severe cognitive decline (severe dementia). In this study, the GDS was used to determine whether the level of functioning aligns with the gold standard criteria for MCI, which are represented by the second and third stages of the condition.
Procedure
Prior to data collection, the original version of the QMCI screen was translated into Romanian by two clinical psychologists fluent in both Romanian and English. Afterward, a different clinical psychologist, also fluent in both languages and blind to the study’s purpose, performed the back-translation.
Participants were recruited from psychiatric hospitals, elderly care facilities, neurology clinics, and general practitioners’ offices within two counties in Western Romania. The clinical personnel who agreed to aid in this process were informed of the inclusion and exclusion criteria. Staff members then referred potential participants to clinicians conducting the assessments, who were blind to participants’ diagnoses prior to the evaluation.
Before the assessment, all volunteering participants were informed about the study, its purpose, and length. If they agreed to participate, they were given a consent form to sign. All selected participants consented to the inclusion of their anonymized test data in the analyses. When patients required aid, their caregivers were also asked to sign the consent form (dementia group, GDS stage 4 or 5). When caregivers met the general criteria, they were also invited to join. We also included self-referred participants who were interested in learning more about their cognitive performance. All consenting participants were evaluated; however, if they did not meet the criteria for any group, their results were not included in the analysis. Data were collected in face-to-face individual sessions by clinical psychologists trained in the use of the instruments mentioned above. The administration order was the same for all participants: MODEMM, QMCI, and MMSE-2SV. Assessment sessions took about 30 to 40 minutes per participant.
Before testing, participants were asked to assess their current memory functioning on a Likert-type scale (very good, good, satisfactory, poor, and very poor). In their initial subjective assessment, participants in the dementia group (n = 56) reported functioning as very good (7.1%), good (30.4%), satisfactory (23.2%), poor (32.1%), and very poor (7.1%). The MCI group (n = 55) reported very good (5.5%), good (27.3%), satisfactory (30.9%), poor (30.9%), and very poor (5.5%) levels of functioning. Control participants (n = 53) described their memory functioning as very good (7.5%), good (62.3%), and satisfactory (30.2%). Participants were also asked to provide their contact information if they agreed to participate in the follow-up study. All participants agreed to be contacted for a follow-up evaluation 12 months after the initial assessment.
Data Analysis
Data were analyzed using SPSS (IBM SPSS 26.0, IBM Corporation, 2019) and R (version 4.4.1, R Core Team, 2024) via RStudio (Posit Team, 2024) with the statix package (Kassambara, 2023). Descriptive statistics were reported for the three diagnostic groups. Normality tests were performed to determine the distribution of scores within each group. A significant departure from normality was shown within the control group for both QMCI and MMSE-2-SV, indicating the use of nonparametric tests. The Kruskal–Wallis test was used to compare between-group results. Post hoc pairwise comparisons between the diagnostic groups were calculated using Dunn’s test with Bonferroni correction and Mann–Whitney U tests. Receiver-operating characteristic (ROC) curves were performed to determine the discrimination accuracy of the three instruments between the three diagnostic groups by calculating sensitivity, specificity, positive, and negative predictive values (PPV and NPV), and likelihood ratios (LR+ and LR–). Since the instruments were screening tools, higher sensitivity was prioritized over specificity (Fabricant, 2024). For each predictor, cutoff points were selected based on the highest sensitivity at specificity ≥ .80, where possible or otherwise ≥ .75, while also considering the best combination of PPV, NPV, LR+, and LR–. Finally, classification accuracies were compared between instruments using DeLong’s formula (DeLong et al., 1988).
Results
Participants’ demographic and diagnostic data are listed in Table 2. There was no significant difference in gender between the three groups, χ2 (2) = 3.51, p = .17. However, age and education differed significantly between the diagnostic groups, with control participants being the youngest and most educated (see Table 2). Levels of education (8–11 vs. ≥ 12 years) were unequally distributed across diagnostic groups, particularly in the dementia and control conditions: 73.2% of patients with dementia (n = 41) reported 8 to 11 years of education, while 94.3% of controls (n = 50) reported ≥ 12 years of education. In the MCI group, 38.2% (n = 21) had less than 12 years, and 61.8% (n = 34) reported at least 12 years of formal schooling.
Participant Characteristics.
Note. MCI = mild cognitive impairment.
Age showed strong negative correlations with all three instruments, r(162) = –.41, p < .001 for MODEMM-I; r(162) = –.42, p < .001 for QMCI; r(162) = –.32, p < .001 for MMSE-2 (Table 3). Education showed strong positive correlations with all three instruments, r(162) = .56, p < .001 for MODEMM; r(162) = .50, p < .001 for QMCI; r(162) = .51, p < .001 for MMSE-2. The three instruments were also strongly inter-correlated, r(162) = .83, p < .001 for MODEMM with QMCI; r(162) = .84, p < .001 for MODEMM with MMSE-2; and r(162)= .80, p < .001 for QMCI with MMSE-2.
Correlation Matrix for the Variables.
Note. N = 164. QMCI = quick mild cognitive impairment.
Correlation is significant at the .01 level (two-tailed).
Considering these associations and accounting for the potential effects of age and education on test performance, we performed analyses of covariance (ANCOVAs) for each instrument to determine its discrimination accuracy between diagnostic groups, with age and education as covariates. On one hand, there were no significant interactions of age, F(1, 2) = .632, p = .428, or education, F(1, 2) = 2.79, p = .097, with the ability of the QMCI to discriminate between the diagnostic groups, F(2) = 138.13, p < .001. On the other hand, unlike age, F(1, 2) = .447, p = .505, education, F(1, 2) = 5.94, p = .016, was significantly related to the MMSE-2-SV’s discrimination ability, F(2) = 96.63, p < .001. Similarly, the MODEMM-I’s discrimination ability, F(2) = 130.57, p < .001, was associated with education, F(1, 2) = 12.03, p = .001, but not with age, F(1, 2) = .162, p = .688. Following recommendations against correcting tests for age and education due to the risk of diminishing their discrimination performance (O’Connell & Tuokko, 2010; Piccininni et al., 2023), we conducted the first set of analyses without educational adjustments.
First, we compared performances between the three groups using the Kruskal–Wallis H test. Significant differences were observed on all three tests, H(2) = 122.17, p < .001, η2 = 0.74 for MODEMM, H(2) = 120.78, p < .001, η2 = 0.73 for QMCI, and H(2) = 117.36, p < .001, η2 = 0.71 for MMSE-2. Post hoc pairwise comparisons using Dunn’s test with Bonferroni correction indicated significant differences between all diagnostic groups for each predictor: MODEMM-I, QMCI, and MMSE-2 (ps < .00l) (Table 4). The comparison of group performances showed the same pattern across tests, as patients with dementia scored lower than patients with MCI, who in turn scored lower than controls. Dunn’s test plots and tables are presented in the Supplemental Material.
Between-Groups Comparisons on MODEMM-I, QMCI, and MMSE-2-SV.
Note. MODEMM-I: Memory of Objects and Digits and Examination of Memory Malingering Test—Integral (Sava & Crișan, 2022); QMCI = quick mild cognitive impairment (O’Caoimh, 2015); MMSE-2-SV = Mini-Mental State Examination-2 Standard Version (Folstein et al., 1995).
Second, ROC analyses were used to determine the accuracy of each instrument in distinguishing between the criterion groups, which were considered as binary variables (i.e., dementia vs. MCI, dementia vs. control, and MCI vs. control; Table 5). Cutoffs were selected based on the best achieved balance between sensitivity and specificity. Graphics of the ROC curves are presented in Figure 2.
Classification Accuracy of Predictors Against Diagnostic-Group Comparisons.
Note. MODEMM-I: Memory of Objects and Digits and Examination of Memory Malingering Test—Integral (Sava & Crișan, 2022); QMCI = quick mild cognitive impairment(O’Caoimh, 2015); MMSE-2-SV = Mini-Mental State Examination-2 Standard Version (Folstein et al., 1995); SENS = sensitivity; SPEC = specificity; PPV = positive predictive value; NPV = negative predictive value; LR+ = positive-likelihood ratio; LR– = negative-likelihood ratio.
Roc Curve Graphs: Dementia vs. MCI, MCI vs. Control, Dementia vs. Control.
All three instruments were able to discriminate between criterion groups with excellent to outstanding accuracies (AUCs = .89–.99, ps < .001). MODEMM-I and QMCI showed virtually similar AUCs (AUCs =.91–.92; .93; .99–.98) across all diagnostic criteria. All instruments reached almost perfect accuracy in differentiating dementia from control participants. Nevertheless, MODEMM-I and QMCI proved slightly more accurate than the MMSE-2-SV in classifying MCI and dementia and MCI and control, respectively.
Differential cutoff points for distinguishing between conditions were established for each screener. On MODEMM-I, a cutoff ≤ 27 correctly classified MCI and dementia groups with .87 sensitivity at .83 specificity, a PPV of .84, and NPV of .86. A cutoff of ≤ 40 demonstrated .94 sensitivity at .83 specificity, .85 PPV, and .93 NPV in discriminating patients with MCI from controls, whereas a cutoff of ≤ 39 proved most sensitive (1.00) and highly specific (.86) to the dementia vs. control condition while maintaining high PPV (.88) and NPV (1.00). Lowering the cutoff to ≤ 38 resulted in a specificity of .90 at the cost of slightly reduced sensitivity (.98).
On the QMCI, a cutoff of ≤49 discriminated MCI from dementia with similar accuracies (Sn = .87, Sp = .81, PPV = .83, NPV = .86) to the MODEMM-I ≤27. Although the lower cutoff of ≤48 produced similar accuracies (Sn = Sp = .85), the higher score was preferred due to its higher sensitivity. In the MCI vs. control condition, a score of ≤ 69 produced .89 sensitivity at .83 specificity, .84 PPV, and .88 NPV. Conversely, a score of ≤ 65 proved most accurate (Sn = .98, Sp = .90, PPV = .91, NPV = .97) in distinguishing between dementia and control participants.
Finally, a cutoff of ≤ 23 on the MMSE-2-SV was the only score with a sensitivity and specificity ≤ 80 in distinguishing between MCI and dementia (PPV = .81, NPV = .80). For the MCI vs. control condition, a cutoff of ≤ 28 provided the best combination of sensitivity (.89) and specificity (.83), PPV (.84), and NPV (.88), whereas a score of ≤ 26 was needed to best classify dementia from control participants (Sn = .98, Sp = .94, PPV = .94, NPV = .98).
Given the significant interaction between education and performance on the MODEMM-I and MMSE-2-SV, as well as the between-group differences in education, we computed covariate-adjusted ROC curves for all three instruments (Janes & Pepe, 2009), using the probabilities obtained from logistic regression models with education as a binary covariate (< 12 vs. ≥ 12 years; O’Caoimh et al., 2016). While all three screeners produced excellent AUC values across diagnostic groups, AUCs = .91–.99, 95% confidence intervals (CIs) (.85, 1.00), we noted a variation in cutoffs as a function of education, mainly on MODEMM-I and MMSE-2-SV (see Tables 2 and 3 in the Supplemental Material). Generally, higher cutoffs were needed for the more educated group. In the dementia vs. MCI group, a MODEMM-I ≤ 26 cutoff for patients with less than 12 years (Sn = .83, Sp = .76) and a ≤ 28 cutoff for patients with more than 12 years of education (Sn = .87, Sp = .82) achieved a good combination of sensitivity and specificity. In the MCI vs. control group, a ≤ 40 cutoff rendered similar sensitivity (.94–.95) but lower specificity for the more educated group (.82) compared to the less educated group (1.00). Finally, the cutoff for the dementia vs. control condition needed to be increased with 10 points, from ≤ 28 for less educated (Sn = .93, Sp = 1.00) to ≤ 38 for more educated participants (Sn = 1.00, Sp = .90). MMSE-2-SV cutoffs also needed to be raised for participants with higher education, from ≤ 23 to ≤ 24 in dementia vs. MCI, from ≤ 26 to ≤ 28 in MCI vs. control, and from ≤ 24 to ≤ 25 in dementia vs. control comparisons. Finally, the QMCI proved more robust to educational levels, with a cutoff of ≤49 proving acceptable accuracies in the dementia vs. MCI group (Sn = .87–.88, Sp = .81–.82), and a cutoff of ≤ 54 achieving excellent accuracies (Sn = .93–.95, Sp = .96–1.00) in dementia vs. control comparisons. Nevertheless, in the MCI vs. control group, QMCI cutoffs needed to be increased from ≤ 66 for less educated participants (Sn = .95, Sp = 1.00) to ≤ 70 for more educated participants (Sn = .85, Sp = .82).
Third, another series of ROC analyses were performed for the two subscales of MODEMM-Objects (MODEMM-O) and MODEMM-Digits (MODEMM-D) (see Table 6) to investigate if using one part of the test would be accurate enough in situations where using the integral version is not appropriate (e.g., time constraints and examinees with perceptual deficits). Graphics of the secondary set of ROC curves are presented in Figure 3.
Classification Accuracy of MODEMM Subscales Against Diagnostic Group Comparisons.
Note. MODEMM-O: Memory of Objects and Digits and Examination of Memory Malingering Test-Objects (Sava & Crișan, 2022); MODEMM-D = Memory of Objects and Digits and Examination of Memory Malingering Test-Digits (Sava & Crișan, 2022); SENS = sensitivity; SPEC = specificity; PPV = positive predictive value; NPV = negative predictive value; LR+ = positive-likelihood ratio; LR– = negative-likelihood ratio.
Roc Curve Graphs: Dementia vs. MCI, MCI vs. Control, Dementia vs. Control.
Both MODEMM-O and MODEMM-D demonstrated excellent to outstanding accuracies as predictors of criterion groups (AUCs = .86–.98, ps < .001), thereby allowing for the investigation of differential cutoffs to discriminate between diagnostic conditions. Nevertheless, on both subscales, cutoffs failed to achieve specificities of ≥ .80 when discriminating between dementia and MCI, and MCI and controls. On MODEMM-O, a cutoff of ≤ 15 produced the best combination of sensitivity (.80) and specificity (.78), PPV (.78), and NPV (.79) in discriminating between MCI and dementia, whereas a cutoff of ≤ 22 appeared appropriate for distinguishing between MCI and control participants (Sn = .87, Sp = .77, PPV = .80, NPV = .85). On MODEMM-D, a score of ≤ 11 reached .78 sensitivity at .72 specificity (PPV = .74, NPV = .76) against dementia vs. MCI, with a score of ≤ 17 achieving .83 sensitivity at .77 specificity (PPV = .79, NPV = .82) against MCI vs control. In contrast, classification accuracies for both scales increased when comparing patients with dementia to control participants. A MODEMM-O cut score of ≤ 20 reached excellent sensitivity (.96) with high specificity (.88), PPV (.90), and NPV (.95). A MODEMM-D score of ≤ 14 produced an excellent combination of sensitivity (.94), specificity (.92), PPV (.92), and NPV (.94).
Comparisons between individual classification accuracies of instruments across diagnostic groups revealed no significant differences between the AUCs of the three main instruments (zs = .13–1.21, ps = .228–.747, Table 7). Nevertheless, MODEMM-I outperformed MODEMM-D in discriminating patients with dementia from patients with MCI (z = 2.08, p = .038), as well as both MODEMM-O (z = 2.87, p = .004) and MODEMM-D (z = 1.91, p = .056) in discriminating MCI from control participants, suggesting that the integral form of MODEMM is preferable for these criteria. No significant differences were noted in the dementia vs. control comparison group.
Classification Accuracy Comparisons Between Instruments.
Note. MODEMM-I: Memory of Objects and Digits and Examination of Memory Malingering Test—Integral (Sava & Crișan, 2022); QMCI = quick mild cognitive impairment(O’Caoimh, 2015); MMSE-2-SV = Mini-Mental State Examination-2 Standard Version (Folstein et al., 1995); z = DeLong z statistic (DeLong et al., 1988).
Discussion
The purpose of this diagnostic accuracy study was to investigate the ability of MODEMM as a screening tool for dementia and MCI, to compare its accuracy with QMCI and MMSE-2-SV in detecting and discriminating between these conditions, and to explore potential cutoff scores for MCI and dementia in a heterogeneous Romanian sample. We hypothesized that (a) MODEMM will classify participants with similar accuracies to the QMCI and MMSE-2-SV and (2) differential cutoffs on each instrument will be needed for identifying MCI and dementia, respectively.
Results broadly confirmed our hypotheses. MODEMM-I proved virtually similar classification accuracies to the QMCI and MMSE-2-SV in predicting criterion groups, suggesting it can be used as a valid screening tool for dementia and MCI. Among the three instruments, MODEMM-I and QMCI obtained the highest accuracy values against all three criteria, followed by the MMSE-2-SV, which also displayed a ceiling effect and a narrower distribution of scores among the dementia group, visible in the Dunn plots. Meanwhile, both MODEMM and QMCI showed more robust distributions across each group. These findings provide additional support for the effectiveness of the MODEMM-I as a screening tool for cognitive impairment in the Romanian population.
As expected, cutoffs needed to be adjusted according to diagnostic comparisons: a MODEMM-I cutoff score of ≤ 40 best distinguished MCI from controls, lowering it to ≤ 39 proved most sensitive to differentiating dementia from controls, and further lowering it to ≤ 27 proved most sensitive and specific to discriminate MCI from dementia. In other words, for an examinee within the present sample, obtaining a MODEMM-I score of ≤ 40 likely suggests MCI, a score of ≤ 39 may suggest a potential onset of dementia, and a score of ≤ 27 may indicate that dementia is more likely than MCI when memory deficits are present. Further adjustments, as a function of education, indicated lower cut scores for participants with less than 12 years of education and higher cut scores for participants with more than 12 years of education on the MODEMM-I and MMSE-2-SV. The greatest shift in cutoffs was observed in the dementia vs. control group, driven by significant education differences between patients with dementia and control participants. Nevertheless, in the MCI vs. control condition, a MODEMM-I score of ≤40 demonstrated consistent discrimination ability across levels of education, matching the cutoff determined for the total sample. This cut score, therefore, appears to be most useful in distinguishing between MCI and age-related dysfunction, seemingly regardless of educational level. Further research is needed to replicate our findings on larger samples with a more representative distribution of education before drawing any firm conclusions about its clinical utility.
On the contrary, QMCI cutoffs proved stable in dementia vs. MCI and dementia vs. control conditions across educational levels, unlike initial reports by O’Caoimh et al. (2012, 2016). A possible explanation for these findings includes the lack of interaction between education and the QMCI’s discrimination ability observed in our sample. Interestingly, in the MCI vs. control comparison groups, the same trend of increasing the cutoff in the more educated group was observed, therefore suggesting that, within our sample, the MODEMM-I might be more effective across educational levels in distinguishing MCI from normal age-related cognitive complaints. Nevertheless, such findings call for future replication studies on larger, more representative clinical and community samples.
In the total sample, the selected cutoff points for the QMCI and MMSE-2 partially matched previously reported cut scores. On the QMCI, a score of ≤ 69 suggested the possibility of MCI, a score of ≤ 65 indicated a potential onset of dementia, and a score of ≤ 49 distinguished dementia from MCI. In contrast, the original cutoffs were ≤ 60 for MCI, ≤ 50 for dementia, and ≤ 44 for distinguishing between MCI and dementia, with further adjustments depending on the subject’s age and years of education (O’Caoimh et al., 2012, 2016). As for MMSE-2-SV, an optimal cutoff for MCI was ≤ 28, a cutoff of ≤ 26 suggested potential dementia onset, and a cutoff of ≤ 23 distinguished MCI from dementia, supporting previous reports that established cutoffs of ≤ 28 for MCI and ≤ 26 for dementia with excellent accuracy (Sleutjes et al., 2020). These results provide proof of concept for the variability of cutoffs according to the reference population and the need to establish them empirically in every new population, rather than inferring them a priori (Crişan & Erdodi, 2022; Crișan & Sava, 2023).
Due to the amount of time required to administer the MODEMM-I, an AUC comparison was conducted to verify whether any of its subscales could be used as an independent screener. The integral form proved the MODEMM-I to be more accurate than both its subscales in discriminating MCI from control participants, and more accurate than the MODEMM-D in differentiating MCI from dementia. Therefore, in such clinical cases, the integral version of the test is recommended. Both subscales demonstrated similar accuracy to the MODEMM-I in distinguishing dementia from control participants, with a score of ≤ 20 on the MODEMM-O and a score of ≤ 14 on the MODEMM-D indicating a potential onset of dementia in a non-diagnosed examinee. However, due to the difference in education observed in this comparison group, results on either subscale must be interpreted with caution in clinical cases. Therefore, at this research stage, we recommend the full administration of MODEMM-I for distinguishing potential memory deficits from normal (age-related) issues, considering education-appropriate cutoffs for each examinee.
Although MODEMM-I requires a longer time of administration than MMSE-2-SV and QMCI, it includes the MODEMM-M subform, which allows the possibility of computing performance validity indicators (i.e., inclusion false-positives, inconsistent recognitions, mean recognition; Crișan & Sava, 2023; Crişan et al., 2022) derived from the cued recall and recognition tasks. In other words, once MODEMM-I is administered and task responses are recorded, clinicians can readily calculate MODEMM-M validity indicators from the examinee’s performance in the cued recall and recognition tasks and compare them with available clinical cutoffs (Crișan & Sava, 2023). Therefore, MODEMM-I offers practitioners an effective method for assessing both cognitive decline and performance validity, functioning as a cognitive screener which includes a PVT subform. Given its dual role, it presents a viable assessment alternative to classic cognitive screeners, especially in cases where subjective cognitive decline is primarily based on memory deficits and malingering is suspected (Bush et al., 2020; Chafetz et al., 2015; Sweet et al., 2021). Future studies should replicate and extend previously reported findings on the accuracy of MODEMM-M in patients with subjective cognitive decline (Crișan & Sava, 2023) and determine optimal cutoffs for patients with MCI and dementia.
The present study has several noteworthy limitations. First, the small sample size and the difficulty of accessing patients with dementia diagnoses who are not committed to any care facility restrain the generalizability of results to the general population. Romania trends highly toward at-home care for patients with a dementia diagnosis, and as such, patients found in care facilities are not entirely representative of the Romanian population with dementia. Second, the fact that patients’ diagnoses were provided by various physicians and the lack of a centralized diagnostic system in Romania might have introduced an important source of variance, especially affecting the validity of formal MCI diagnoses where no neuroimaging data were required.
Another important limitation concerns the severity of cognitive impairment in the MCI and dementia groups, which might have inflated classification accuracies. Potential explanations may be found within the Romanian dementia diagnostic framework. Since a large proportion of patients access medical services rather late after the onset of MCI (Oneţiu et al., 2016), the chances increase of encountering patients formally diagnosed with MCI but presenting greater cognitive impairment. In addition, in contrast with the diagnostic procedure of dementia, which includes neuroimaging confirmation of cognitive deterioration, the formal diagnostic procedure of MCI in Romania only requires medical assessment (i.e., psychiatric or neurological), whereas psychological evaluations are recommended but not mandatory. Therefore, a proportion of the sample formally diagnosed with MCI might not have undergone neuroimaging or psychological assessment at the point of receiving the medical diagnosis, limiting diagnostic decisions to physicians’ reliance on patients’ subjective memory complaints and self-reported areas of dysfunction. To compensate for this shortcoming, we relied on the logical memory and processing speed scores of the MMSE-2-EV in gauging the cognitive deficit of participants before including them in the MCI group.
Additional sample-specific limitations include the significant differences in educational levels across groups (especially within dementia and control conditions). While the low mean levels of education in the dementia group have been empirically accounted for and can be regarded as a cultural aspect in Romania (Oneţiu et al., 2016; Tatomirescu et al., 2025), establishing no upper limit of education as an inclusion criterion may have introduced selection bias in our sampling procedure, resulting in a large proportion of examinees with high levels of education in the control group. Future studies are encouraged to include larger samples, stratified according to age and education. In addition, the community sample may have been contaminated by the occurrence of unreported and undiagnosed cognitive deficits, which could have affected their test performance. Finally, the limited geographical representativeness of the sample, as the data were collected from the Western part of Romania, requires extending the sampling area as a direction for future research.
Conclusion
While cognitive screening is not advised for asymptomatic people (Ismail et al., 2020), either self-reported or observed symptoms should be taken seriously (Bruscoli & Lovestone, 2004; Chen et al., 2017). Presently, there is no known cure for dementia, but there is evidence of modifiable factors helping in its prevention or delay once MCI is diagnosed. Therefore, the use of effective screening tools normed and validated on the examinee’s reference population is key to the early diagnosis of MCI. The results of our study support the use of MODEMM-I as an instrument for screening memory dysfunctions, as it is effective in discriminating between MCI and dementia, as well as age-related cognitive issues, and in indicating a possible onset of dementia. Given the significant interaction between education and MODEMM-I performance, differential cutoffs are recommended based on levels of education, particularly for distinguishing between dementia and MCI, as well as between dementia and normal aging. Further research is clearly needed to replicate the present results on a larger, more representative Romanian clinical sample and to investigate the MODEMM’s cross-cultural applicability to non-Romanian-speaking populations.
Supplemental Material
sj-docx-1-asm-10.1177_10731911251389197 – Supplemental material for A New Instrument for Assessing Cognitive Decline and Dementia: Results on the Classification Accuracy of the MODEMM in a Romanian Clinical and Community Sample
Supplemental material, sj-docx-1-asm-10.1177_10731911251389197 for A New Instrument for Assessing Cognitive Decline and Dementia: Results on the Classification Accuracy of the MODEMM in a Romanian Clinical and Community Sample by Cătălina Şandru, Iulia Crișan, Daniela Reisz and Florin Alin Sava in Assessment
Footnotes
Acknowledgements
None to declare.
Author Contribution Statement
Conceptualization: F.A. Sava and C. Şandru; data curation: C. Şandru and I. Crișan; formal analysis: C. Şandru and I. Crișan; funding acquisition: NA; investigation: C. Şandru; methodology: F.A. Sava, C. Şandru, I. Crișan, and D. Reisz; project administration: F.A. Sava, C. Şandru, I. Crișan, and D. Reisz; resources: D. Reisz, C. Şandru, and I. Crișan; software: C. Şandru, F.A. Sava, and I. Crișan; supervision: F.A. Sava and I. Crișan; validation: C. Şandru, F.A. Sava, I. Crișan, and D. Reisz; visualization: C. Şandru and I. Crișan; writing—original draft: C. Şandru; writing—review and editing: I. Crișan, F.A. Sava, and C. Şandru.
Declaration of Conflicting Interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: The second and fourth authors hold intellectual property rights associated with the MODEMM and receive royalties from its sales.
Funding
The authors received no financial support for the research, authorship, and/or publication of this article.
Data Availability Statement
The raw de-identified data and analytic syntax, as well as the instruments used are available to qualified researchers upon reasonable request from the corresponding author.
Ethical Approval
The study has been approved by the University’s Ethics committee (Nr. 81400/12.12.2022) and was not preregistered. All procedures contributing to this work are in accordance with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.
Consent to Participate
Written informed consent was obtained from all participants.
Consent for Publication
Not applicable.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
