Abstract
Background:
Interventions targeting cognition in people with schizophrenia have shown moderate effects on improving functioning. Recent cognitive remediation (CR) approaches have begun to target metacognition to improve functioning outcomes. This study aims to develop a novel measure of metacognition and assess whether metacognitive-based CR (mCR) can improve metacognition.
Method:
We use data from a single-blinded randomized controlled trial comparing mCR plus treatment as usual (TAU) to TAU alone in people with schizophrenia. Participants were assessed with measures of cognition, functioning, and a new measure of metacognition at three time points: Week 0 (baseline), Week 12, and Week 24.
Results:
The evaluation of the novel metacognition assessment suggests that it is a reliable and valid measure. The measure positively correlates with cognitive measures, in particular with executive function and IQ but also with a measure of functioning. We found that a course of mCR was able to increase metacognition compared to TAU by Week 24.
Conclusions:
Metacognition may be an important mechanism to explain how CR affects functioning outcomes in people with psychosis. The systematic assessment of metacognition as part of CR studies may help to evaluate more clearly its role and relevance to functioning.
Introduction
Cognitive difficulties are a landmark feature of schizophrenia and recognized to contribute to functioning difficulties (Green & Harvey, 2014; Keefe & Harvey, 2012; Wykes, 1994). Cognitive remediation (CR) was developed in an attempt to reduce the impact of cognitive difficulties on functional outcome. CR is a psychological therapy that uses learning principles and behavioral techniques (e.g., massed practice and errorless learning) to improve cognition and ultimately to reduce functional difficulties. There are two main CR approaches, one based on drill and practice and the other using drill and practice and strategy (Cella, Huddy, Reeder, & Wykes, 2012). While drill and practice programs rely solely on task practice to reduce cognitive difficulties, drill and practice plus strategy programs actively teach strategy use to overcome cognitive difficulties and achieve functioning milestones. Evidence suggests that both these approaches are effective in improving cognition but drill and practice plus strategy is more effective in bringing about functioning improvements (Wykes, Huddy, Cellard, McGurk, & Czobor, 2011).
A key unanswered question for CR programs is how to maximize functioning improvements. While strategies have been shown to contribute to functioning improvements, increasing this therapy ingredient on its own may not be enough to deliver large changes. For example, if a CR program teaches strategies that people forget to use or are not able to use efficiently then introducing strategies alone may be unlikely to influence functioning. A well-learned strategy in a particular context may also suffer from difficulties in being generalized and adapted to new situations. This may restrict the overall utility and applicability of strategies in CR.
This is the context that prompted clinical researchers in this field to look for a process that can maximize the applicability and usefulness of strategies so that they can have a lasting and more significant effect on functioning difficulties. Research in people with psychosis has shown that metacognitive impairments are associated with difficulties in everyday life functioning including employment and social relationships (Koren, Seidman, Goldsmith, & Harvey, 2006; Lysaker et al., 2011; Stratta, Daneluzzo, Riccardi, Bustini, & Rossi, 2009). Metacognition has also been suggested as a process that can aid strategy use and may extend their usefulness.
Although several therapeutic approaches have attempted to target metacognition, they have used different definitions. While all definitions largely revolve around Flavell’s original conceptualization of metacognition as the cognitive processes that relate to “thinking about thinking” (Flavell, 1979), there are many subtle and sometime key differences in the way this term is used. Some proponents consider metacognition as the ability to make sense of one’s own and other people’s experiences (Lysaker & Dimaggio, 2014). Others consider it a process responsible for illness insight and symptom awareness (Corcoran & Frith, 2003; David, Bedford, Wiffen, & Gilleen, 2012), and still others focus on narrower aspects of metacognition such as introspective accuracy, or the ability to accurately judge one’s own skills and abilities (Pinkham, Klein, Hardaway, Kemp, & Harvey, 2018). In the context of cognitive behavioral therapy for people with psychosis, this concept has been associated with the ability to recognize and regulate thinking biases (Moritz & Woodward, 2007; Morrison et al., 2014).
We have proposed that metacognition is particularly suited to CR as it regulates learning and behavior and links the task specific learning to effective behavior in key functioning domains such as employment, education, and social life (Cella, Reeder, & Wykes, 2015, 2016; Wykes & Reeder, 2005). We have proposed that two metacognitive functions are vital for translating cognitive to functional improvement. The first is metacognitive knowledge, which refers to knowledge of cognitive processes, such as someone’s beliefs about their own cognitive strengths and weaknesses. The second is metacognitive regulation, which is responsible for planning, monitoring, and improving cognitive processes and behaviors through evaluation. Metacognitive regulation and knowledge can both support effective strategy use. For example, by planning what strategies best apply to a journey, evaluating different transport options, taking enough money, and anticipating possible problems.
Over the last 10 years, we have developed a CR program targeting metacognitive knowledge and regulation alongside cognitive difficulties. The resulting therapy, called Computerized Interactive Remediation of Cognition–Interactive Training for Schizophrenia, or CIRCuiTS, is acceptable and feasible (Reeder et al., 2015). Recent research showed promising results for this therapy to improve cognitive difficulties and functioning (Reeder et al., 2017). However, before we can investigate whether metacognitive CR effectively targets metacognitive processes we need a measure that can capture metacognitive regulation and knowledge. To this end, the current study aims to: (i) develop a metacognition measure based on task performance and strategy use and (ii) evaluate whether metacognition-based CR can improve metacognition.
Method
Design
This study used data from a single blind randomized controlled trial comparing a novel metacognitive CR approach, called CIRCuiTS, plus treatment as usual (TAU) to TAU alone. A detailed description of this study methods can be found in Reeder et al. (2017). Outcomes were measured at three time points: baseline (Week 0), post-treatment (Week 12), and follow-up (Week 24).
Participants
Participants for this study were recruited via Community Mental Health Teams and Rehabilitation services from the South London and Maudsley Mental Health National Health Service Foundation Trust. Participant inclusion criteria were (i) Diagnostic and Statistical Manual of Mental Disorders, 4th Edition diagnosis of schizophrenia or schizoaffective disorder assessed with structured interview by a trained clinician, (ii) at least 1 year’s contact with mental health services, (iii) 17–65 years, and (iv) performance more than 1 SD below the normative mean in working memory, as assessed by the digit span (Wechsler, 2008) and/or cognitive flexibility, as assessed by the Wisconsin Card Sorting Test (WCST) (Heaton & Psychological Assessment Resources Inc., 1993) and/or executive function, as assessed by the Hayling Sentence Completion Test (Burgess & Shallice, 1997). We chose to focus our inclusion criteria on these cognitive domains as they are strongly associated with functioning difficulties (Cella & Wykes, 2017; Wykes et al., 2012). Exclusion criteria were (i) plans to change medication during the study, (ii) substance dependence, or (iii) evidence of an organic cause to cognitive difficulties.
Interventions
Treatment as usual
This is routine psychiatric care offered by the service including individualized multidisciplinary contacts for medication review and symptom and well-being monitoring. The professionals involved in these contacts were mostly psychiatrists and mental health nurses. Less frequently, psychologists, occupational therapists, and residential support services were also offering relevant support.
Metacognitive CR
CR was delivered by a therapist using CIRCuiTS, a Web-based computerized software. Sessions with therapist were supplemented with independent sessions. In addition to providing massed practice of basic cognitive functions (e.g., memory, attention, executive function), the program targets metacognition and encourages strategy use. The therapist’s role is centered around supporting motivation, strategy development and generalization, and teaching, as well as using metacognitive regulations skills to promote an understanding of cognitive and noncognitive strengths and difficulties (i.e., metacognitive knowledge). The therapist also supports learning transfer to maximize participants’ real-world goals. Therapists provide additional scaffolding by simplifying task demands where needed and supporting clients’ motivation to ensure consistent successful performance. The software targets memory, executive functioning, and attention. Tasks increase in difficulty gradually in line with participants performance to maintain high levels of success allowing for errorless learning. Participants also develop a set of personalized strategies to improve their cognitive performance and achieve their goals.
At the beginning of the therapy, real-world goals are set collaboratively, and progress is reviewed regularly. Therapy sessions are offered at least three times a week (for maximum 12 weeks), up to 40 sessions each lasting approximately 1 hr. Where possible, according to participants’ ability and choice, therapists encourage carrying out additional independent sessions.
CIRCuiTS tasks are either “abstract” (neutral content, such as numbers, and designed to target specific cognitive functions) or “exercises” (cognitively complex and ecologically valid) associated with work, social situations, cooking, shopping, and traveling. Therapists encourage participants to apply the skills learnt in daily life and to practice in vivo to achieve their real-world goals. Thus, functional outcomes are directly targeted by the therapy.
Measures
Cognition
Verbal working memory was assessed using Digit Span (Wechsler Adult Intelligence Scale–Fourth Edition [WAIS-IV]-UK) (Wechsler, 2008), and we used the total raw score output with high scores being indicative of good performance.
Visual memory was assessed with the Rey–Osterrieth complex figure (ROCF) (Meyers, Bayless, & Meyers, 1996), and we used the immediate recall raw score with high scores being indicative of good performance.
Verbal executive function was assessed with the Hayling Sentence Completion test (Burgess & Shallice, 1997), and we used the total scaled score with high scores being indicative of good performance.
Performance executive function was assessed with the WCST (Heaton & Psychological Assessment Resources Inc., 1993), testing abstraction, and cognitive flexibility. We used the percentage of errors as this measure outcome with high scores being indicative of poor performance.
Current IQ was estimated using the Vocabulary and Block Design tests from the Wechsler Adult Intelligence Scale–Third UK Edition (Christensen, Girard, & Bagby, 2007).
Functioning
Functioning was assessed using the Time Use Survey (UK 2000 Time Use Survey–Short, 2006). This is a semi-structured interview recording how participants use their time which has been adapted for use with people with psychosis (Cella, Edwards, & Wykes, 2016). Key outcomes of this measure are expressed in total hours per week over the past month spent in structured activities, including employment, education, voluntary employment, structured leisure activities, housework and chores, childcare, sports, and hobbies.
Symptoms
Symptoms were assessed using the Positive and Negative Syndrome Scale (Kay, Fiszbein, & Opler, 1987). This is a 30-item structured clinical interview assessing symptom severity. For this study, we considered three symptoms dimensions: positive, negative, and general symptoms.
Metacognition
This measure assessed how participants approach two cognitive tasks: the ROCF and the Hayling Sentence Completion test. Performance on both these tasks benefit from the implementation of strategy. We assessed four metacognitive components:
Understanding, monitoring, and evaluating: Assess the participant’s ability to acknowledge having mental functions, distinguish between mental operations using cognitive words (i.e., memorizing, realizing, and imaging), and evaluate their task performance.
Strategy implementation: Evaluates whether the participant knew how they completed the task, if they acknowledged using a strategy which was either effective or ineffective, could label and describe the strategy used, as well as evaluating their chosen method of implementation.
Objective strategy: Assesses whether a clear and objective strategy was used to complete the task.
Strategy congruence: Evaluates whether the subjective strategy stated is congruent with the cognitive demands of the task and to what degree the strategy was implemented (e.g., only at the beginning and then dropped; during the whole task). This is evaluated by the interviewer and together with objective strategy represents an index of metacognition independent from the participants’ ability to describe strategy use, which may be indirectly supported by CR.
For each of these items participants answers or performance, is scored on a 0–3 scale. Raters were provided with a scoring manual with each score level anchoring description and examples. Lower scores indicate lack of or difficulties in the application and use of that metacognitive function, while high scores are evidence of mastery. The total scale score is the sum of the four items scores and it ranges from 0 to 12. Each interview was scored independently by two raters, and each score discrepancy that could not be resolved was discussed with a third senior author and a final score agreed.
The measure was administered as a structured interview after the administration of the text was completed. All interviews were audio recorded and transcribed. To ensure the reliability of each metacognition transcript, every transcript was assessed for interviewer adherence to the script, using a rating scale ranging from 0 (poor adherence) to 3 (highly adherent). Poor adherence reflects the interviewer inadvertently suggesting strategies or using words that could prompt the participants to use reflective language (e.g., what were you thinking while doing the task). Any transcript rated as 0 or 1 was excluded from the analysis. Scoring on Understanding, monitoring, and evaluating and Strategy implementation was performed on interview transcripts while scoring on the Objective strategy and Strategy congruence was performed by the rater on the basis of the observed behavior or the relevance of the strategy use to the task.
Analysis
We investigated the psychometric properties of our novel metacognitive measure. As two (i.e., metacognitive understanding and subjective strategy use) of the four interview items rely on raters’ scores, we calculated inter-rater reliability from two independent raters using a Cohen’s κ. Interpretation of κ levels was based on Viera and Garrett (2005).
We evaluated the distribution properties of each measure considered by exploring distributional parameters (e.g., skewness) and using Shapiro–Wilks test to assess normality. We assessed correlation using Pearson correlation coefficients.
Scale reliability was measured by Cronbach’s α coefficient. Conventionally, α values above .70 are considered satisfactory (Bland & Altman, 1997). We evaluated the new metacognitive measure’s construct validity using correlations with cognitive domains, functioning levels, and symptom domains using Pearson correlation coefficients.
To explore whether metacognitive CR can improve scores on our novel metacognitive measure, we conducted a mixed-effect analysis of covariance (ANCOVA). The model considered the baseline score (Week 0) of the metacognition measure as a covariate. The model considered time at post-therapy (Week 12) and follow-up (Week 24) assessments, treatment group (i.e., CR and TAU alone), and the interaction between time and group as fixed categorical variables. Partial η2 are reported as proxy for effect size.
Results
Of the 93 participants recruited in the study, 76 completed the metacognitive interview. Reasons for non-completion were not providing consent for the interview recording (11), recorder not available (3), and poor audio quality or partial recording (3). Of these 76 interviews, 9 were rated as unreliable due to excessive interviewer influence or interviewer not following the script. Percentage of missing data from the 67 participants who took part was 6% at baseline assessment, 12% at 12-week assessment, and 19% at 24-week assessment.
Participants were mostly men (61%) and had an average age of 39.6 years (SD = 10.7). They all received antipsychotic medications with an average chlorpromazine level of 421.6 (SD = 323.9).
Metacognitive measure
Inter-rater agreement on interview scoring across the three assessment points for Understanding, monitoring, and evaluating and Strategy implementation was between .38 and .51. This is indicative of a fair-to-moderate inter-rater agreement. All disagreements were discussed and resolved with the help of a third author.
The scale reliability for baseline assessments was evaluated using Cronbach’s α and was .78.
The scale items intercorrelation ranged between .82 (p < .001) for the correlation between Strategy congruence and Objective strategy use to .29 (p < .001) for the correlation between Understanding, monitoring, and evaluating and Objective strategy use.
Correlations between the metacognitive measure total score and the cognitive and symptoms measures at Week 0 are reported in Table 1.
Correlation between the total metacognitive measure score, cognitive performance, and symptoms.
Note. WCST: Wisconsin Card Sorting Test; PANSS: Positive and Negative Syndrome Scale; WATR: Wechsler Test of Adult Reading; SA: structured activities.
*p < .05; **p < .001.
The measure has moderate levels of correlation with executive function measures and modest relations with functioning and negative symptoms.
Is metacognitive CR improving metacognition?
Of the 67 participants considered, 34 were randomized to CR and 33 to TAU.
Figure 1 shows the metacognitive mean scores for the CR and the TAU groups. ANCOVA analysis showed no group by time interaction at Week 12, F(1, 56) = 1.9, p = .17, η2 = .03 (mean difference = .75; SE = .17), but did show a significant group by time interaction at Week 24, F(1, 51) = 6.5, p = .01, η2 = .15 (mean difference = 1.38; SE = .51), suggesting that metacognition scores improve significantly by Week 24 in the CR group compared to TAU.

Mean metacognition scores at Week 0, Week 12, and Week 24 for the CR and TAU groups. CR: cognitive remediation; TAU: treatment as usual.
Discussion
Metacognition is a key element of an effective cognitive system allowing individuals to expand knowledge and effectively understand, control, and manipulate their own cognitive processes (Flavell, 1979). Metacognitive skills also allow individuals to be more flexible and able to cope with a rapidly changing environment. A wealth of research demonstrates that metacognition is important for learning and acquiring new skills and that individuals who have higher levels of metacognition tend to perform better in studies and work (Hong, Vadivelu, Daniel, & Sim, 2015; Kentridge & Heywood, 2000). Similarly, research conducted in people with psychosis supports the notion that difficulties in metacognition are associated with higher levels of functioning difficulties. This is the rationale that prompted us to develop a CR program targeting metacognition to maximize its effects on functioning.
The results of this study showed that the new measure is feasible, and most trial participants were able to complete it. One aspect to consider in future studies is interviewer training as we lost 10% of participants’ results because of a lack of adherence to the script. This lack of adherence occurred mostly at the beginning of the session, and it was soon rectified in supervision sessions. So future studies using this measure should maintain close monitoring on how the measure is administered. The psychometric proprieties of the novel metacognitive measure were found to be acceptable. As with all interviewer-based assessments, rater agreement is crucial to the sensitivity of the measure and appropriate training for raters is necessary.
The correlations between the metacognition items and the internal consistency values suggest that the domains considered appear to be part of the same construct. This is important as it provides reassurance that the items representing the theoretical constructs are related to each other. While it is plausible to link effective strategy use to good metacognitive skills, it is reassuring that our measure shows links between effective strategy deployment and good metacognitive regulation. We consider this an important aspect that differentiates our program from compensatory CR programs where strategy may be provided, but limited consideration is given to how these are regulated and implemented.
The correlation results, presented in Table 1, suggest that the new metacognitive measure is related more to executive function than other cognitive domains. This is not a new finding and is in line with previous research suggesting that metacognition and executive functions may be partly overlapping concepts (Follmer & Sperling, 2016).
It is not clear from the literature, however, whether metacognitive gains, after therapy, are independent from executive function. Part of our concept of metacognition, metacognitive regulation, is similar to self-regulation and considers cognitive skills necessary to plan, monitor, and evaluate behaviors, including strategy use. The effect of metacognitive CR on executive function, as reported in the study by Reeder et al. (2017), suggests that a course of this therapy can improve performance on executive function and this effect may not be independent from the effect on metacognition. It may be that metacognition exerts an effect on functioning via executive functions but also directly via metacognitive knowledge. Planning a visit to a family member should be positively affected by planning skills, but realizing that the plan may be poorly executed on a low mood day may equally be important and prevent failure.
Our measure is based largely on how participants approach a task and deal with its cognitive demands and apply and regulate strategies, and it is different to other conceptualizations (e.g., Moritz & Lysaker, 2018). As such our measure is more akin to performance-based assessments such as neuropsychological tests, however, we consider this concept still within the family of metacognition on the basis this is an essential process regulating behavior. Most of the research to date has measured metacognition using self-assessment measures (e.g., Barbato et al., 2014), and this may be problematic, particularly in people with psychosis. Research suggests, in fact, that there is a discrepancy between self-assessed, interviewer-based, and performance-based measures (Cella, Swan, Medin, Reeder, & Wykes, 2014; Medalia & Thysen, 2010). It is therefore likely that a performance-based measure of metacognition may better capture how metacognitive skills are deployed in everyday life and how they may influence functioning. The correlation between our metacognitive and functioning measure, as presented in Table 1, suggests that the amount of time people spend in structured activities in their everyday life is associated with their metacognitive competencies and strategy implementation skills.
In this study, we showed, for the first time, that our CR approach was effective in improving metacognitive scores compared to TAU. This is important as it shows that the measure is sensitive to change and that the therapy program targets the intended competencies. What remains to be established is whether and how metacognitive improvement transfers to functional gains. Result of this study, reported in the study by Reeder at al. (Reeder et al., 2017), found that individuals in the CR group improved their functioning at post-therapy, but these gains were not maintained at follow-up. This effect was likely due to the lack of opportunities and support participants were experiencing after therapy and this might have resulted in limited opportunities to expand functioning levels. Despite the lack of functioning improvement at follow-up, in this study, we showed that metacognitive levels are different between the groups and significantly higher in the metacognitive CR group. This suggests that improving metacognition alone may not be sufficient for people who experience high levels of social and functioning difficulties and that providing opportunities to deploy new strategies may be a necessary precondition for programs to have a lasting effect on functioning.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
