Abstract
Objective:
This study evaluated a modified working memory training program, Cogmed, for ADHD youth.
Method:
Forty youth were randomized to modified Cogmed training (MCT) or treatment as usual (CON). MCT was delivered in an outpatient mental health clinic in 3 weekly 35-minute sessions with a dedicated coach for 10 weeks. Participants completed assessments at baseline, after the intervention, and again 3 months later.
Results:
After controlling for baseline, groups were comparable on working memory, academics, and ADHD symptoms. The MCT group was rated by parents and teachers as having fewer executive function challenges and youth endorsed better self-concept compared with the CON group.
Conclusions:
MCT was associated with some improvement, which could not be attributed to increased working memory capacity and may reflect other facets of the program. Results question the usefulness of Cogmed but highlight considerations for optimizing adherence, engagement, and the therapeutic alliance in interventions for ADHD youth.
Keywords
Introduction
ADHD is a childhood-onset neurodevelopmental disorder defined by atypically high and impairing levels of inattention, hyperactivity, and impulsive behavior (American Psychiatric Association, 2013). The prevalence of ADHD is estimated to be 5% to 7% of school-aged children (Polanczyk et al., 2014)—most of whom will experience impairing traits that persist into adolescence and adulthood (Faraone et al., 2006). In North America, first-line treatment of ADHD entails stimulant medication, which is often effective for short-term symptom management (Wolraich et al., 2019). Importantly, however, medication benefits do not necessarily generalize to common correlates of ADHD—such as deficits in executive skills, academic proficiency, and social-emotional function (Coghill et al., 2014; Kofler et al., 2018, Kortekaas-Rijlaarsdam et al., 2019)—and approximately 20% to 35% of individuals with ADHD are medication non-responders (Childress & Sallee, 2014).
One major impetus for ADHD research has been evaluating whether non-medication treatments can reduce core symptoms of ADHD whilst simultaneously improving function in other domains. The marketing phrase “brain-training” has become synonymous with interventions that seek to improve relatively circumscribed cognitive skills through repeated exposure to computerized games of increasing challenge (Simons et al., 2016). One of the most predominant examples is the working memory (WM) training program Cogmed, first developed by Klingberg et al. (2002). Preliminary research with ADHD youth identified enduring improvements in working memory coupled with training gains that transferred to other cognitive skills, academics, and ADHD symptoms (Klingberg et al., 2002, 2005). Cogmed is now a commercially available product used for ADHD treatment planning (Pearson Canada Assessment, 2021).
In the interim period of almost 20 years, a formidable body of research has painted a more equivocal picture of working memory training interventions like Cogmed. On the one hand, neuroimaging work has identified discernable changes in brain function following completion of Cogmed (Olesen et al., 2004; Stevens et al., 2016)—consistent with the view that this adaptive training program gives rise to underlying changes at the level of the brain. Importantly, however, meta-analytic reviews have yielded mixed findings regarding behavioral outcomes in heterogenous samples as well as samples specific to ADHD. Whilst there is compelling evidence that working memory training programs like Cogmed yield improvement in performance on closely related working memory tasks, the duration of this improvement is unclear and benefits do not appear to generalize to other domains of function (e.g., Aksayli et al., 2019; Cortese et al., 2015; Melby-Lervåg et al., 2016; Rapport et al., 2013; Sala & Gobet, 2020). One notable exception is real-world inattention, which was shown to improve following completion of Cogmed in one meta-analysis of children and adults with and without ADHD (Spencer-Smith & Klingberg, 2015).
Whilst the forgoing suggests that there may be a lack of empirical support to recommend Cogmed as an ADHD intervention, we suggest that there are three methodological limitations to this body of work that should be considered. The standard Cogmed protocol for youth is home-based and is typically offered in 30- to 45-minute duration active training sessions five times weekly for five consecutive weeks with regular check-ins from qualified coaches (Roche & Johnson, 2014). 1 First, the intensive nature of this protocol necessitates a commitment from children and caretakers that may disincline many from seeking out this kind of intervention, resulting in samples in published studies that are unlikely to be representative of the families with whom clinicians typically work. Second, the consistent application of attention and effort required for cognitively challenging tasks that are up to 45 or 50 minutes in duration may be too taxing for ADHD youth whose symptoms typically include inattentive and/or restless behaviors, thus raising the possibility that they mentally disengage from the program before reaching the end of each training session. Indeed, a study of college students with ADHD found that Cogmed sessions as short as 15-minutes had the potential to be clinically beneficial (Mawjee et al., 2015). Lastly, a month of intervention, even though intensive, may be of insufficient duration to enable ADHD youth to consolidate their training into enduring and meaningful change, giving rise to some of the null findings regarding transfer effects that have been documented in this literature.
This study was undertaken to explore the potential effectiveness of Cogmed in ADHD youth using a protocol that was modified to address these methodological concerns. To do so, our modified Cogmed training (MCT) was delivered in a hospital clinic by a dedicated coach in relatively short sessions distributed over a more protracted period (i.e., 3 weekly 35-minute sessions for 10 consecutive weeks). We believed that doing so would enable us to recruit a more representative sample of participants compared with previous studies, reduce the likelihood of fatigue and/or mental disengagement by youth in the training program, and increase the likelihood that the benefits of improved working memory would be consolidated into enduring change that transferred to other domains. Compared with ADHD youth receiving standard treatment-as-usual, we hypothesized that ADHD youth randomized to MCT would evidence improved working memory, better executive skills, higher achievement in math and reading, a reduction in ADHD symptoms, and improved self-concept.
Method
Participants
Participants were 40 youth (39 male) ages 8 to 16 years (M = 11.59 years, SD = 2.17). Thirty-nine identified as Caucasian (one Aboriginal). Parental education ranged from less than high school (mothers: n = 16; fathers: n = 18), to completion of high school (mothers: n = 8; fathers: n = 6), to a college or university degree (mothers: n = 16; fathers: n = 14). The distribution of gross (pre-tax) household income was bimodal, with most families reporting an income of less than $60,000 (n = 14) or $100,000 and above (n = 16). Most youth resided in the same household as both of their parents (n = 23). All youth had a primary diagnosis of ADHD based on DSM-V diagnostic criteria (inattentive: n = 1; combined: n = 39) and most had at least one co-morbid diagnosis (n = 27), with the most common being ODD (n = 10) and anxiety disorders (n = 11). Learning problems were common (n = 17) though relatively few youth had been formally diagnosed with a learning disability (n = 5). Most youth had used medication for management of ADHD prior to the study (n = 29) and were taking medication at the time of study enrollment (n = 24).
Procedure
A schematic of the study is presented in Figure 1. Families were physician-referred to the CMH Outpatient Mental Health Clinic due to having a child between 8 and 16 years of age with known or suspected ADHD coupled with an interest in learning about the hospital-based intervention. Forty-two parents/guardians were initially screened over the phone by the senior clinician (SS), a registered clinical psychologist, using the CADDRA ADHD checklist (Canadian Attention Deficit Hyperactivity Disorder Resource Alliance [CADDRA], 2014). Per parent/guardian report, all youth had at least six of nine traits of inattention or hyperactivity-impulsivity at a clinically significant threshold (i.e., the trait was rated by the parent/guardian as pretty much or very much evident). The senior clinician provided parents/guardians with more information about the intervention and invited interested families to the hospital for a baseline visit. One family withdrew at this time due to the time commitment required for the study.
Research procedure flowchart.
The baseline visit was attended by 41 families and began with a detailed informed consent procedure with the senior clinician. Families then participated in a diagnostic assessment with a child and adolescent psychiatrist to confirm youths’ ADHD diagnostic status, assess for potential comorbid conditions, and review current and past treatment history. Youth were then administered a self-report questionnaire and standardized cognitive tests by the senior clinician or trained personnel working under the clinician’s supervision. Parents/guardians completed questionnaires regarding their child’s social-emotional and cognitive function, with parallel forms completed by teachers which were subsequently returned to the senior clinician. Exclusion criteria, based on information gleaned from the baseline visit, were not having a primary diagnosis of ADHD, having an estimated IQ below 80 or reading below a grade 2 level (n = 1), experiencing a physical limitation that would make computer use difficult, or intending to change the type or dose of youths’ medication during the study unless instructed to do so by their physician.
Using a random number generator, youth were randomly assigned either to a control group in which they received treatment-as-usual under the care of their referring physician with the option of receiving Cogmed at the conclusion of the study (CON; n = 20) or to an experimental group in which they received modified Cogmed training (MCT; n = 20). As described elsewhere (Klingberg et al., 2005), Cogmed RM is an adaptive training program consisting of 12 visual or auditory WM exercises with task difficulty individually calibrated so that each participant is constantly working at his/her optimal level of challenge. Our modified version of this program required youth to complete three training sessions per week for a 10-week period, thereby increasing the total number of sessions and delivering them over a longer consolidation period. All 30 sessions also were carried out with a clinical psychologist or trained research assistant (i.e., designated coach) in a quiet room at CMH, as opposed to the standard at-home administration. Participants completed 8 of 12 exercises at each training session. Each session lasted approximately 35 minutes, during which time the coach remained in the room with youth to answer any questions and provide motivational support. Coaches also had access to progress reports provided by Cogmed, tracking trial-by-trial performance, and overall weekly performance. These reports were used to provide individualized feedback to youth, in addition to feedback built-in to the computer program as participants completed exercises.
Following the 10-week intervention period (post 1) and again 3 months later (post 2), all baseline measures other than a measure of intellectual function were re-administered to youth, parents/guardians, and teachers (with the same teacher providing ratings of youth at each time point). Approximately 2 weeks after the last follow-up assessment, youth and their families had a final feedback session with the senior clinician at which time a subset (n = 10) were interviewed about their subjective experiences with the MCT intervention (Sadeghi et al., 2020). All youth and families were then provided with individualized feedback regarding youths’ performance on the assessment measures. Demographic, clinical, and cognitive-behavioral characteristics of the sample as a function of condition are presented in Table 1.
Descriptive Information for ADHD Youth Randomized to Modified Cogmed Treatment (MCT) or Treatment-as-Usual (CON).
All participants received compensation for transportation/parking, a two-dollar gift card for each session attended at CMH, and a $100 iTunes gift card at final feedback. Youth also were allowed to keep the mini-iPad that was used to administer modified Cogmed training. This study was jointly approved by the Office of Research Ethics at the University of Waterloo and the Tri-Hospital Research Ethics Board for Cambridge Memorial Hospital (CMH) in Waterloo Region. It is registered as a clinical trial at clinicaltrials.gov, ID number NCT02610244.
Measures
Achenbach System of Empirically Based Assessment (ASEBA): Using the Child Behavior Checklist and Teacher Report Form (Achenbach & Rescorla, 2001), respectively, parents and teachers provided ratings of youths' behaviors during a 6-month period on a 3-point Likert scale ranging from 0 (Not True) to 2 (Very True or Often True). Items for the subscale measuring ADHD symptomatology were summed and converted to a norm-referenced T-score.
Automated Working Memory Assessment (AWMA): Youth were administered four subtests of the AWMA (Alloway, 2007). Our investigation was interested in complex span measures, which required that participants hold information in mind whilst simultaneously engaging in other cognitive processing. Auditory complex span (Listening Recall) was assessed by requiring that youth listen to short sentences, decide whether each was true or false, and then report the last word of each sentence in reverse order. Visual-spatial complex span (Spatial Recall) was assessed by requiring that youth compare two shapes, decide if the shapes were identical or mirror opposites, then recall the location of red dots in the reverse order they were presented. Age-normed standard scores on each complex span measure were entered into analyses.
Beck-Youth Inventories—Second Edition (BYI-II): Twenty items from the self-concept inventory (Beck et al., 2005) were administered to youth, assessing perceptions of their competency and self-worth. Items were rated on a scale ranging from 0 (Never) to 3 (Always) and aggregated into a T-score using sex and age norms from the standardization sample.
Behavior Rating Inventory of Executive Function—Second Edition (BRIEF-2): This 63-item questionnaire (Gioia et al., 2015) assessed perceptions of youths’ executive functions during the preceding 1-month period as evidenced at home (per parent/guardian report) and school (per teacher report). Items were rated on a 3-point Likert scale ranging from 1 (Never) to 3 (Often) and aggregated to comprise the General Executive Composite, reflecting youths’ overall level of executive dysfunction. Raw scores for parent and teacher forms were converted into T-scores using sex and age norms from the standardization sample.
Test of Variables of Attention (TOVA): Scores on the TOVA (Greenberg, 1990) were invalidated for a large proportion of participants due to technical problems and are not reported further.
Wechsler Abbreviated Scale of Intelligence—Second Edition (WASI-II): Youth were administered all four subtests from the WASI-II (Wechsler, 2011), which required them to assemble blocks to match models (Block Design), complete abstract patterns (Matrix Reasoning), define words (Vocabulary), and explain how word pairs were conceptually related (Similarities). On each subtest, items increased in difficulty until the discontinue criterion was met. Using the manual, raw scores were aggregated into a Full Scale IQ using age norms from the standardization sample.
Wechsler Individual Achievement Test—Third Edition (WIAT-III): Youth were administered select reading and math subtests from the WIAT-III (Wechsler, 2009). A reading composite was derived from subtests that required youth to read words aloud (Word Reading) and answer questions about written passages (Reading Comprehension). A math composite was derived from subtests that required youth to solve mathematical computations (Numerical Operations) and questions involving math concepts (Math Problem Solving). For both composites, raw scores were converted into age-normed standard scores as described in the WIAT-III manual.
Results
No data were missing on any of the analyzed measures. GLMs with group as a between-subjects factor and time as a within-subjects factor were used to determine whether the MCT and CON groups were comparable on measures aligning with core hypotheses following the 10-week intervention after taking potential baseline differences into account (Table 2). All analyses were undertaken with SPSS version 26.
General Linear Models Examining Change in Outcome Measures as a Function of Group, Time, and the Interaction of Group and Time.
There were no significant group × time interactions on any of the performance-based measures; however, there were significant main effects of time on measures of visual-spatial working memory and reading achievement—indicating that, overall, youths’ performance on these measures improved across assessments irrespective of group.
There also were no significant group × time interactions for ratings of ADHD symptoms, although there was a significant main effect of time for ratings provided by parents (though not teachers)—indicating that, overall, youth in both groups were perceived by their parents as showing fewer ADHD traits across assessments.
There were significant group × time interactions for parent and teacher ratings of executive dysfunction (Figures 2 and 3). Follow-up t-tests revealed that parents rated youth in each group comparably at baseline, t(33) = −0.49, p = .63, and at post-1 follow-up, t(33) = 0.84, p = .41, but tended to perceive youth in the MCT group as having fewer EF challenges compared with youth in the CON group at post-2 follow-up, t(33) = 1.95, p = .06. There were no significant group differences in teacher ratings at baseline, t(34) = 0.85, p = .40, but teachers rated MCT youth as having fewer EF challenges compared with CON youth at post-1 follow-up, t(34) = 1.83, p = .07, and post-2 follow-up, t(34) = 3.57, p = .001.
Interaction of group and assessment timepoint on parental perceptions of youth’s executive function challenges.
Interaction of group and assessment timepoint on teacher perceptions of youth’s executive function challenges.
For youths’ self-concept (Figure 4), there also was a significant group x time interaction: ratings of self-concept were comparable across groups at baseline, t(38) = 0.03, p = .98, but MCT youth perceived themselves significantly more positively than CON youth at post-1 follow-up, t(38) = −2.30, p = .03, and at post-2 follow-up, t(38) = −3.40, p = .002.
Interaction of group and assessment timepoint on youth’s reported self-concept.
Amongst youth who were randomized to receive modified Cogmed training, there was 100% adherence to the intervention protocol (i.e., all 30 sessions were attended) and evidence of improvement on the trained tasks, as reflected in a significant difference between the Max Index and Start Index, t(19) = −19.19, p < .001. The difference between these indices reflects the mean of the three best successful trials on days 2 and 3 of training versus the two best training days, and is captured in the Improvement Index. An Improvement Index of 17 has been identified as indicative of substantive improvement in the intervention, which was met or exceeded by 19 of 20 youth in our study (Gray et al., 2012). To determine whether intervention-related changes on parent and teacher ratings of EF and youths’ ratings of self-concept were attributable to youths’ engagement with the modified Cogmed protocol, we next examined partial correlations between post-2 follow-up ratings and the Improvement Index after controlling for ratings at baseline. Inspection of partial correlations revealed that the Improvement Index was significantly associated with parent ratings of youths’ executive dysfunction (r = −.66, p = .002), such that youth who improved more on trained Cogmed tasks were perceived by their parents as evidencing fewer EF challenges following the intervention. However, there was no significant association between the Improvement Index and teacher ratings of EF (r = .19, p = .48) or youth’s ratings of self-concept (r = .03, p = .98).
Discussion
This study was undertaken to address potential methodological factors that have characterized research pertaining to the clinical utility of Cogmed for youth with ADHD. Our modified protocol included a fewer number of weekly sessions that were shorter in duration and distributed across a longer time period (i.e., 10 vs. 5 weeks). The sessions were delivered within a hospital setting with a dedicated coach present at each session. Parking and transportation fees were covered for youth and their families, and they were also provided with meaningful positive rewards after each session and at the conclusion of the study. Adherence to our modified protocol was exceptionally high, with all youth attending all Cogmed sessions and post-intervention assessments. Our procedure was consistent with Shinaver & Entwistle’s (2015) review of Cogmed training which noted that “compliance with treatment is a key to achieving benefit” (p. 81).
In spite of our modifications to the standard Cogmed protocol, which we believe resulted in a more representative sample of ADHD youth who were highly committed to and engaged with the intervention, we found no evidence that training improved performance on any of the objective performance-based measures. Contrary to our results, prior reviews of Cogmed have identified a benefit of working memory training on highly similar working memory tasks—though, consistent with our null results, this benefit has not been found to generalize to other cognitive abilities or academic proficiency (e.g., Aksayli et al., 2019; Melby-Lervåg et al., 2016; Shipstead et al., 2012). As noted by others, the lack of far transfer to domains that are known to be influenced by working memory capacity poses formidable challenges for the rationale of the Cogmed intervention (e.g., Sala & Gobet, 2020).
Although early work linked Cogmed to clinical improvement in ADHD (Klingberg et al., 2002), we also did not find evidence that modified Cogmed training improved ADHD symptoms in our sample. Prior reviews on this topic have yielded similar lackluster findings. For example, Spencer-Smith and Klingberg (2015) conducted a meta-analysis of 15 Cogmed studies in which they were specifically interested in modeling change in ratings of inattention. They found that participation in Cogmed was associated with a moderate reduction in inattentiveness that persisted over time across a broad range of participant groups (e.g., children and adults with and without ADHD). In a subsequently published correction to this article, however, effect sizes for children and youth as well as individuals with ADHD were nonsignificant (Spencer-Smith & Klingberg, 2016). Cortese et al. (2015) conducted a meta-analytic review of 15 randomized control trials that administered cognitive training (including Cogmed) to youth with ADHD specifically. They also reported an improvement in total ADHD symptoms and symptoms of inattention, but these effects were no longer significant when analyses were limited to interventions that focused solely on training of working memory. Since these reviews, other studies have explored the potential for Cogmed to treat ADHD. Whilst some have reported improvement in core clinical features of the disorder (Bigorra et al., 2016), others have identified improvement only for youth taking medication (Ackermann et al., 2018), and still others have reported null findings similar to ours (Dentz et al., 2016; Rivard et al., 2020). We believe that our study adds to this literature by suggesting that, even with methodological changes to the standard protocol, Cogmed is likely not an effective treatment for the core clinical features of ADHD given the totality of evidence.
Although several of our core hypotheses were not supported, there are two notable findings to emerge from our work. One significant finding is that parent- and teacher-ratings of youths’ EF challenges were considerably lower post-intervention for ADHD youth who participated in modified Cogmed training compared with treatment-as-usual. Whilst parents were not blinded with regards to youths’ group membership, we believe that most teachers were unaware of the condition to which their student was assigned. As such, it is unlikely that these positive findings are solely attributable to demand characteristics (e.g., the expectation that youth who participated in Cogmed would improve). Our findings are consistent with those of Cortese et al. (2015), who reported a small though significant benefit of cognitive training on parent-report of executive dysfunction. A limitation of this work is that findings were pooled across different kinds of cognitive training regimens and comparable information was not available from teachers. Our work thus expands upon that of Cortese et al. (2015) by highlighting a potential benefit of Cogmed specifically on ADHD youths’ everyday EF challenges per the perspective of multiple informants. We also note that ADHD youth who had a larger Improvement Index had parents who endorsed greater perceived improvement in their youth’s real-world EF. To our knowledge, this finding is novel insofar as implicating the extent of ADHD youths’ improvement in EF to the magnitude of their improvement in Cogmed training. Although our results do not indicate that Cogmed increased ADHD youths’ working memory capacity per se, there may have been other aspects to the training regimen that better-enabled ADHD youth to apply their executive skills in the real-world contexts of home and school—such as more explicit goal-setting, greater mental effort, and increased tolerance for challenge (e.g., Hsu et al., 2018; In de Braek et al., 2017). These ideas are speculative, however, and require further study.
Another significant finding is that self-concept improved in ADHD youth who participated in modified Cogmed training compared with those who received standard treatment. Unlike with their peers, youth who completed working memory training felt more competent and had greater self-esteem after the intervention period and this effect was sustained over time. We are not aware of any other studies that have elicited feedback from ADHD youth regarding the ways in which working memory training has the potential to impact their sense of self-worth. However, given that ADHD youth are at increased risk for experiencing mental health concerns like depression (Daviss, 2008; Mayer et al., 2021), we believe that this information is important to consider when evaluating the clinical utility of cognitive training regimens like Cogmed. As reported in Sadeghi et al. (2020), most of the youth in our study enjoyed their experience with modified Cogmed training. Further, although not reflected in their objective test performance, they also identified myriad ways in which this training was helpful (e.g., improved focus, memory, and ability to achieve goals). In addition to potential benefits derived from the Cogmed intervention itself, we also note that consideration was given to fostering a strong therapeutic alliance between ADHD youth and research staff. For example, the senior clinician and coach both greeted youth and their parent/guardian in the waiting room at the outset of each session prior to the coach taking the youth to a designated office for training. Furthermore, every youth was assigned a dedicated coach such that they worked with the same person at each session. We believe this procedure positively affected the therapeutic relationship with ADHD youth, which is a well-established common factor in psychotherapy outcome research, with corollary benefits on youths’ self-regard (Wampold, 2015).
Limitations of our study include having an insufficient sample size to explore medication as a potential moderator of findings, having relatively few performance-based measures of cognition, and only one measure of ADHD that did not permit a more refined analysis of symptomatology. Nevertheless, we believe that our work adds to research regarding the clinical utility of Cogmed for youth with ADHD in two important ways. First, some of our findings confirm previously reported null results regarding the potential for Cogmed to increase working memory capacity and generalize to other domains of function that are theoretically and empirically linked to this cognitive skill (e.g., reading and math). We have done so, however, with a representative sample of ADHD youth who have completed a modified version of Cogmed in an ideal context, which increases our confidence in the null effects being true. Second, we have identified methodological changes that may have contributed to significant findings involving domains that are frequently impaired in this clinical population (e.g., executive dysfunction, low self-concept). These changes have included alterations to the standard Cogmed protocol to maximize adherence and engagement, placing the therapeutic relationship with ADHD youth at the forefront of the intervention, and including information from multiple informants (i.e., parents, teachers, and youth themselves). Although the balance of available evidence indicates that Cogmed should not be recommended as either a primary or adjunctive treatment for ADHD, our work highlights considerations for other non-pharmacological interventions that may contribute to positive improvements in ADHD youth.
Footnotes
Author Note
Salek (Sol) Sandberg has retired and is now in Toronto, Ontario, Canada.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a private grant awarded to S.S. Cogmed and Cogmed Working Memory Training are trademarks, in the U.S. and/or other countries, of Pearson Education, Inc. or its affiliate(s). Pearson Education did not play a role in the design or conduct of the study; analysis or interpretation of the data; or preparation or review of the manuscript. Many thanks to Martyn Gabel and Meagan Koufis for their work as Cogmed coaches.
