Self-administered computerized neuropsychological assessments (CNAs) provide lower cost, more accessible alternatives to traditional in-person assessments but lack critical information on psychometrics and subjective experience of older adults in remote testing environments.
Objective:
We used an online brief battery of computerized tasks selected from the Cogstate Brief Battery (CBB) and Cambridge Brain Sciences (CBS) to 1) determine test-retest reliability in an unsupervised setting; 2) examine convergent validity with a comprehensive ‘gold standard’ paper-and-pencil neuropsychological test battery administered in-person; and 3) explore user-experience of remote computerized testing and individual tests.
Methods:
Fifty-two participants (mean age 65.8±5.7 years) completed CBB and CBS tests on their own computer, unsupervised from home, on three occasions, and visited a research center for an in-person paper-and-pencil assessment. They also completed a user-experience questionnaire.
Results:
Test-retest reliabilities varied for individual measures (ICCs = 0.20 to 0.83). Global cognition composites showed excellent reliability (ICCs > 0.8 over 1-month follow-up). A strong relationship between a combination of CNA measures and paper-and-pencil battery was found (canonical correlation R = 0.87, p = 0.04). Most tests were rated as enjoyable with easy-to-understand instructions. Ratings of general experience with online testing were mostly favorable; few had difficulty concentrating (17%) or using the computer for tasks (10%), although over one-third experienced performance anxiety (38%).
Conclusion:
A combined brief online battery selected from two CNAs demonstrated robust psychometric standards for reliability (global composite), and convergent validity with a gold standard battery, and mostly good usability and acceptability in the remote testing environment.
BilderRM, PostalKS, BarisaM, AaseDM, CullumCM, GillaspySR, HarderL, KanterG, LancaM, LechugaDM, MorganJM, MostR, PuenteAE, SalinasCM, WoodhouseJ (2020) InterOrganizational practice committee recommendations/guidance for teleneuropsychology (TeleNP) in response to the COVID-19 pandemic. Clin Neuropsychol34, 1314–1334.
2.
GatesNJ, KochanNA (2015) Computerized and on-line neuropsychological testing for late-life cognition and neurocognitive disorders: Are we there yet? Curr Opin Psychiatry28, 165–172.
3.
DarbyDG, FredricksonJ, PietrzakRH, MaruffP, WoodwardM, BrodtmannA (2014) Reliability and usability of an internet-based computerized cognitive testing battery in community-dwelling older people. Comput Human Behav30, 199–205.
4.
MielkeMM, MachuldaMM, HagenCE, EdwardsKK, RobertsRO, PankratzVS, KnopmanDS, JackCRJr., PetersenRC (2015) Performance of the CogState computerized battery in the Mayo Clinic Study on Aging. Alzheimers Dement11, 1367–1376.
5.
RentzDM, DekhtyarM, ShermanJ, BurnhamS, BlackerD, AghjayanSL, PappKV, AmariglioRE, SchembriA, ChenhallT, MaruffP, AisenP, HymanBT, SperlingRA (2016) The feasibility of at-home iPad cognitive testing for use in clinical trials. J Prev Alzheimers Dis3, 8–12.
6.
TroyerAK, RoweG, MurphyKJ, LevineB, LeachL, HasherL (2014) Development and evaluation of a self-administered on-line test of memory and attention for middle-aged and older adults. Front Aging Neurosci6, 335.
7.
BuckleyRF, SparksKP, PappKV, DekhtyarM, MartinC, BurnhamS, SperlingRA, RentzDM (2017) Computerized cognitive testing for use in clinical trials: A comparison of the NIH Toolbox and Cogstate C3 Batteries. J Prev Alzheimers Dis4, 3–11.
8.
ValdesEG, SadeqNA, Harrison BushAL, MorganD, AndelR (2016) Regular cognitive self-monitoring in community-dwelling older adults using an internet-based tool. J Clin Exp Neuropsychol38, 1026–1037.
9.
EdgarCJ, SiemersE, MaruffP, PetersenRC, AisenPS, WeinerMW, AlbalaB (2021) Pilot evaluation of the unsupervised, at-home Cogstate Brief Battery in ADNI-2. J Alzheimers Dis83, 915–925.
10.
de JagerCA, SchrijnemaekersA-CMC, HoneyTEM, BudgeMM (2009) Detection of MCI in the clinic: Evaluation of the sensitivity and specificity of a computerised test battery, the Hopkins Verbal Learning Test and the MMSE. Age Ageing38, 455–460.
11.
HammersD, SpurgeonE, RyanK, PersadC, BarbasN, HeidebrinkJ, DarbyD, GiordaniB (2012) Validity of a brief computerized cognitive screening test in dementia. J Geriatr Psychiatry Neurol25, 89–99.
12.
BrenkelM, ShulmanK, HazanE, HerrmannN, OwenAM (2017) Assessing capacity in the elderly: Comparing the MoCA with a novel computerized battery of executive function. Dement Geriatr Cogn Dis Extra7, 249–256.
13.
HackettK, KrikorianR, GiovannettiT, Melendez-CabreroJ, RahmanA, CaesarEE, ChenJL, HristovH, SeifanA, MosconiL, IsaacsonRS (2018) Utility of the NIH Toolbox for assessment of prodromal Alzheimer’s disease and dementia. Alzheimers Dement (Amst)10, 764–772.
14.
MaruffP, LimYY, DarbyD, EllisKA, PietrzakRH, SnyderPJ, BushAI, SzoekeC, SchembriA, AmesD, MastersCL (2013) Clinical utility of the Cogstate Brief Battery in identifying cognitive impairment in mild cognitive impairment and Alzheimer’s disease. BMC Psychol1, 30.
15.
LimYY, JaegerJ, HarringtonK, AshwoodT, EllisKA, StöfflerA, SzoekeC, LachovitzkiR, MartinsRN, VillemagneVL, BushA, MastersCL, RoweCC, AmesD, DarbyD, MaruffP (2013) Three-month stability of the CogState brief battery in healthy older adults, mild cognitive impairment, and Alzheimer’s disease: Results from the Australian Imaging, Biomarkers, and Lifestyle-rate of change substudy (AIBL-ROCS). Arch Clin Neuropsychol28, 320–330.
16.
FredricksonJ, MaruffP, WoodwardM, MooreL, FredricksonA, SachJ, DarbyD (2010) Evaluation of the usability of a brief computerized cognitive screening test in older people for epidemiological studies. Neuroepidemiology34, 65–75.
RacineAM, ClarkLR, BermanSE, KoscikRL, MuellerKD, NortonD, NicholasCR, BlennowK, ZetterbergH, JedynakB, BilgelM, CarlssonCM, ChristianBT, AsthanaS, JohnsonSC (2016) Associations between performance on an Abbreviated CogState Battery, other measures of cognitive function, and biomarkers in people at risk for Alzheimer’s disease. J Alzheimers Dis54, 1395–1408.
19.
MackinRS, InselPS, TruranD, FinleyS, FlennikenD, NoshenyR, UlbrightA, ComachoM, BickfordD, HarelB, MaruffP, WeinerMW (2018) Unsupervised online neuropsychological test performance for individuals with mild cognitive impairment and dementia: Results from the Brain Health Registry. Alzheimers Demen (Amst)10, 573–582.
20.
KoyamaAK, HaganKA, OkerekeOI, WeisskopfMG, RosnerB, GrodsteinF (2015) Evaluation of a self-administered computerized cognitive battery in an older population. Neuroepidemiology45, 264–272.
21.
PerinS, BuckleyRF, PaseMP, YassiN, LavaleA, WilsonPH, SchembriA, MaruffP, LimYY (2020) Unsupervised assessment of cognition in the Healthy Brain Project: Implications for web-based registries of individuals at risk for Alzheimer’s disease. Alzheimers Dement (N Y)6, e12043.
22.
SanoM, EgelkoS, FerrisS, KayeJ, HayesTL, MundtJC, DonohueM, WalterS, SunS, Sauceda-CerdaL (2010) Pilot study to show the feasibility of a multicenter trial of home-based assessment of people over 75 years old. Alzheimer Dis Assoc Disord24, 256–263.
23.
OwenAM, HampshireA, GrahnJA, StentonR, DajaniS, BurnsAS, HowardRJ, BallardCG (2010) Putting brain training to the test. Nature465, 775–778.
24.
HuntleyJ, CorbettA, WesnesK, BrookerH, StentonR, HampshireA, BallardC (2018) Online assessment of risk factors for dementia and cognitive function in healthy adults. Int J Geriatr Psychiatry33, e286–e293.
25.
HonarmandK, MalikS, WildC, Gonzalez-LaraLE, McIntyreCW, OwenAM, SlessarevM (2019) Feasibility of a web-based neurocognitive battery for assessing cognitive function in critical illness survivors. PLoS One14, e0215203.
26.
HampshireA, HighfieldRR, ParkinBL, OwenAM (2012) Fractionating human intelligence. Neuron76, 1225–1237.
27.
American Educational Research Association
(2014) Standards for Educational and Psychological Testing. American Educational Research Associations, Washington, DC.
28.
CollieA, MaruffP, DarbyDG, McStephenM (2003) The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test-retest intervals. J Int Neuropsychol Soc9, 419–428.
29.
BanksE, RedmanS, JormL, ArmstrongB, BaumanA, BeardJ, BeralV, BylesJ, CorbettS, CummingR, HarrisM, SitasF, SmithW, TaylorL, WutzkeS, LujicS (2008) Cohort profile: The 45 and up study. Int J Epidemiol37, 941–947.
30.
NganduT, LehtisaloJ, SolomonA, LevälahtiE, AhtiluotoS, AntikainenR, BäckmanL, HänninenT, JulaA, LaatikainenT, LindströmJ, MangialascheF, PaajanenT, PajalaS, PeltonenM, RauramaaR, Stigsdotter-NeelyA, StrandbergT, TuomilehtoJ, SoininenH, KivipeltoM (2015) A 2 year multidomain intervention of diet, exercise, cognitive training, and vascular risk monitoring versus control to prevent cognitive decline in at-risk elderly people (FINGER): A randomised controlled trial. Lancet385, 2255–2263.
31.
HeffernanM, AndrewsG, Fiatarone SinghMA, ValenzuelaM, AnsteyKJ, MaederAJ, McNeilJ, JormL, LautenschlagerNT, SachdevPS, GinigeJA, HobbsMJ, BoulamatsisC, ChauT, CobiacL, CoxKL, DanielK, FloodVM, GuerreroY, GunnJ, JainN, KochanNA, LampitA, MavrosY, MeiklejohnJ, NobleY, O’LearyF, Radd-VagenasS, WaltonCC, BrodatyH (2019) Maintain Your Brain: Protocol of a 3-year randomized controlled trial of a per-sonalized multi-modal digital health intervention to prevent cognitive decline among community dwelling 55 to 77 year olds. J Alzheimers Dis70, S221–s237.
32.
HarringtonKD, LimYY, AmesD, HassenstabJ, Rainey-SmithS, RobertsonJ, SalvadoO, MastersCL, MaruffP (2017) Using robust normative data to investigate the neuropsychology of cognitive aging. Arch Clin Neuropsychol32, 142–154.
33.
HampshireA, HighfieldRR, ParkinBL, OwenAM (2012) Fractionating human intelligence. Neuron76, 1225–1237.
34.
AlbertMS, DeKoskyST, DicksonD, DuboisB, FeldmanHH, FoxNC, GamstA, HoltzmanDM, JagustWJ, PetersenRC, SnyderPJ, CarrilloMC, ThiesB, PhelpsCH (2011) The diagnosis of mild cognitive impairment due to Alzheimer’s disease: Recommendations from the National Institute on Aging-Alzheimer’s Association workgroups on diagnostic guidelines for Alzheimer’s disease. Alzheimers Dement7, 270–279.
35.
WeintraubS, SalmonD, MercaldoN, FerrisS, Graff-RadfordNR, ChuiH, CummingsJ, DeCarliC, FosterNL, GalaskoD, PeskindE, DietrichW, BeeklyDL, KukullWA, MorrisJC (2009) The Alzheimer’s Disease Centers’ Uniform Data Set (UDS): The neuropsychologic test battery. Alzheimer Dis Assoc Disord23, 91–101.
36.
RabinLA, PaolilloE, BarrWB (2016) Stability in test-usage practices of clinical neuropsychologists in the United States and Canada over a 10-year period: A follow-up survey of INS and NAN members. Arch Clin Neuropsychol31, 206–230.
37.
StraussE, ShermanE, SpreenO (2006), A Compendium of Neuropsychological Tests: Administration, Norms, and Commentary, Oxford University Press, New York.
38.
CollertonJ, CollertonD, AraiY, BarrassK, EcclesM, JaggerC, McKeithI, SaxbyBK, KirkwoodT (2007) A comparison of computerized and pencil-and-paper tasks in assessing cognitive function in community-dwelling older people in the Newcastle 85+ Pilot Study. J Am Geriatr Soc55, 1630–1635.
39.
FillitHM, SimonES, DonigerGM, CummingsJL (2008) Practicality of a computerized system for cognitive assessment in the elderly. Alzheimers Dement4, 14–21.
40.
KooTK, LiMY (2016) A guideline of selecting and reporting intraclass correlation coefficients for reliability research. J Chiropr Med15, 155–163.
41.
CohenJ (1988) Statistical Power Analysis for the Behavioural Sciences,Lawrence Erlbaum Associates, USA.
42.
DavisAS, PiersonEE, FinchWH (2011) A canonical correlation analysis of intelligence and executive functioning. Appl Neuropsychol18, 61–68.
43.
NunnallyJC (1978), Psychometric Theory, McGraw-Hill, New York.
44.
BarchDM, CarterCS (2007) measurement issues in the use of cognitive neuroscience tasks in drug development for impaired cognition in schizophrenia: A report of the Second Consensus Building Conference of the CNTRICS Initiative. Schizophr Bull34, 613–618.
45.
Fawns-RitchieC, DearyIJ (2020) Reliability and validity of the UK Biobank cognitive tests. PLoS One15, e0231627.
46.
CalamiaM, MarkonK, TranelD (2013) The robust reliability of neuropsychological measures: Meta-analyses of test-retest correlations. Clin Neuropsychol27, 1077–1105.
StrickerNH, LundtES, AldenEC, AlbertsonSM, MachuldaMM, KremersWK, KnopmanDS, PetersenRC, MielkeMM (2020) Longitudinal comparison of in clinic and at home administration of the Cogstate Brief Battery and demonstrated practice effects in the Mayo Clinic Study of Aging. J Prev Alzheimers Dis7, 21–28.
49.
FeinkohlI, BorchersF, BurkhardtS, KrampeH, KraftA, SpeidelS, KantIMJ, van MontfortSJT, AartsE, KruppaJ, SlooterA, WintererG, PischonT, SpiesC (2020) Stability of neuropsychological test performance in older adults serving as normative controls for a study on postoperative cognitive dysfunction. BMC Res Notes13, 55.
50.
LimYY, EllisKA, HarringtonK, AmesD, MartinsRN, MastersCL, RoweC, SavageG, SzoekeC, DarbyD, MaruffP, The Aibl Research Group (2012) Use of the CogState Brief Battery in the assessment of Alzheimer’s disease related cognitive impairment in the Australian Imaging, Biomarkers and Lifestyle (AIBL) study. J Clin Exp Neuropsychol34, 345–358.
51.
AdlerK, AppleS, FriedlanderA, ChengF, CohenC, LevineSR, LucchesiM, MaruffP, ReinhardtM, SalmieriM, StevensD, WeingastS, GustafsonD (2019) Computerized cognitive performance assessments in the Brooklyn Cognitive Impairments in Health Disparities Pilot Study. Alzheimers Dement15, 1420–1426.
52.
DorenkampMA, VikP (2018) Neuropsychological assessment anxiety: A systematic review. Pract Innov (Wash D C)3, 192–211.
53.
GamaldoAA, TanSC, SardinaAL, HenziC, GuestR, RossLA, WillinghamK, ZondermanAB, AndelRA (2018) Older Black adults’ satisfaction and anxiety levels after completing alternative versus traditional cognitive batteries. J Gerontol B Psychol Sci Soc Sci75, 1462–1474.
54.
IrvineAB, GelattVA, SeeleyJR, MacfarlaneP, GauJM (2013) Web-based Intervention to promote physical activity by sedentary older adults: Randomized controlled trial. J Med Internet Res15, e19.
55.
PeelR, RenS, HureA, EvansT-J, D’EsteCA, AbhayaratnaWP, TonkinAM, HopperI, ThriftAG, LeviCR, SturmJ, DurrheimD, HungJ, BriffaTG, ChewDP, AndersonP, MoonL, McEvoyM, HansbroPM, NewbyDA, AttiaJR (2019) Evaluating recruitment strategies for AUSPICE, a large Australian community-based randomised controlled trial. Med J Aust210, 409–415.
56.
García-CasalJA, Franco-MartínM, Perea-BartoloméMV, Toribio-GuzmánJM, García-MojaC, Goñi-ImizcozM, CsipkeE (2017) Electronic devices for cognitive impairment screening: A systematic literature review. Int J Technol Assess Health Care33, 654–673.
57.
KochanNA (2019) Australian and New Zealand Clinical Trials Registry (ANZCTR).
58.
ScottEP, SorrellA, BenitezA (2019) Psychometric properties of the NIH Toolbox Cognition Battery in healthy older adults: Reliability, validity, and agreement with standard neuropsychological tests. J Int Neuropsychol Soc25, 857–867.
59.
SkirrowC, CashdollarN, GrangerK, JenningsS, BakerE, BarnettJ, CormackF (2021) Test-retest reliability on the Cambridge Neuropsychological Test Automated Battery: Comment on Karlsen et al. (2022). Applied Neuropsychology Adult29, 889–892.
60.
NordlundA, RolstadS, GöthlinM, EdmanA, HansenS, WallinA (2010) Cognitive profiles of incipient dementia in the Goteborg MCI study. Dement Geriatr Cogn Disord30, 403–410.
