Abstract
Objective:
Self-reports or patient-reported outcome measures are seldom used in psychosis due to concerns about the ability of patients to accurately report their symptomatology, particularly in cases of low awareness of illness. The aim of this study was to assess the effect of insight on the accuracy of self-reported psychotic symptoms using a computerized adaptive testing tool (CAT-Psychosis).
Methods:
A secondary analysis of data drawn from the CAT-Psychosis development and validation study was performed. The Brief Psychiatric Rating Scale and the Scale of Unawareness of Mental Disorders were administered by clinicians. Patients completed the self-reported version of the CAT-Psychosis. Patients were median-split regarding their insight level to compare the correlation between the two psychosis severity measures. A subgroup sensitivity analysis was performed only on patients with schizophrenia spectrum disorders.
Results:
A total of 159 patients with a psychotic disorder who completed both CAT-Psychosis and Scale of Unawareness of Mental Disorders were included. For the whole sample, CAT-Psychosis scores showed convergent validity with Brief Psychiatric Rating Scale ratings (r = 0.517, 95% confidence interval = [0.392, 0.622], p < 0.001). Insight was found to moderate this correlation (β = –0.511, p = 0.005), yet agreement between both measures remained statistically significant for both high (r = 0.621, 95% confidence interval = [0.476, 0.733], p < 0.001) and low insight patients (r = 0.408, 95% confidence interval = [0.187, 0.589], p < 0.001), while psychosis severity was comparable between these groups (for Brief Psychiatric Rating Scale: U = 3057, z = –0.129, p = 0.897; disorganization: U = 2986.5, z = –0.274, p = 0.784 and for CAT-Psychosis: U = 2800.5, z = –1.022, p = 0.307). Subgroup of patients with schizophrenia spectrum disorders showed very similar results.
Conclusions:
Insight moderates the correlation between self-reported and clinician-rated severity of psychosis, yet CAT-Psychosis remains valid in patients with both high and low awareness of illness.
Introduction
Interest in self-reports and patient-reported outcome measures (PROMs) is growing in psychiatry (Coles et al., 2019). Compared with observer-rated measures, PROMs are easier to administer, do not require specific training and are less burdensome and time-consuming for both clinicians and patients. Some authors have observed that implementing PROMs in routine medical practice can improve treatment outcomes and facilitate decision-making (Chua et al., 2020; Coles et al., 2019; Guo et al., 2015; Hong et al., 2021).
Self-reported symptom scales have shown validity and reliability compared with their rater-administered counterparts (Chakhssi et al., 2019; Hershenberg et al., 2020; Hiranyatheb et al., 2015; Zimmerman et al., 2019). However, a valid, widely used self-administered psychopathology scale has been missing in schizophrenia research and in clinical practice, partly because of lack of trust in the ability of patients to accurately self-report psychotic symptoms.
In fact, it is still debated whether patients with psychosis can accurately report their symptomatology (Takeuchi et al., 2016). On one hand, correlations of poor insight with greater positive and negative symptoms could indirectly suggest that patients with psychosis may not be a reliable source of information (McCabe et al., 2007; Mingrone et al., 2013; Sevy et al., 2004). On the other hand, some authors have found that patients with schizophrenia were able to accurately report positive (Statham et al., 2019), negative (Lincoln et al., 2017) and non-psychotic (Smith et al., 2019) symptoms, as well as personality traits (Bell et al., 2007).
Apart from insight, agreement between observer- and self-rated psychotic measures appears to be related to other variables such as age (De Hert et al., 2009), gender (Liraud et al., 2004; Mingrone et al., 2013), educational level (Lincoln et al., 2010), cognition (Saeedi et al., 2007), disorganization (Takeuchi et al., 2016) and duration of illness (Liraud et al., 2004).
In recent years, numerous PROMs and/or self-reported assessments of psychotic symptoms have been developed and validated in clinical populations (Barrera et al., 2008; Dollfus et al., 2016; Doyle et al., 1999). Commonly, these measures tend to focus on specific symptoms, such as hallucinations (Bell et al., 2006), delusions (Doyle et al.,1999; Green et al., 2008; Nederlof et al., 2011), formal thought disorder (Barrera et al., 2008; Waring et al., 2003), apathy (Faerden et al., 2008), blunted affect (Larsen, 1984) or anhedonia (Chapman et al., 1976; Doyle et al.,1999). Some scales measuring negative symptoms can be found (Dollfus et al., 2016; Park et al., 2012; Selten et al., 1993), yet valid PROM and self-reported measures that take into account the broad symptomatology of psychosis are not widely used, and their relationship with illness awareness remains to be elucidated (Maß et al., 2000) or fails to show external validity and follow classic test theory, therefore taking significant time to administer (Iancu et al., 2005).
More recently, new tools based on computerized adaptive testing (CAT) have emerged, in which a person’s initial item responses are used to determine a provisional estimate of his or her standing on a given trait, which is then used for the adaptive selection of subsequent questions targeted to the person’s specific impairment level, allowing a precise measurement while only requiring the administration of a small subset of questions (Gibbons et al., 2012, 2014, 2017). Taking advantage of this technology, a new instrument was developed to allow valid and reliable self-evaluations of psychotic symptoms in a significantly shorter administration time (Guinart et al., 2021), further stressing the need to elucidate the effect of insight on the accuracy of self-reported psychotic symptoms.
Methods
This study involves a secondary analysis of data drawn from the development and validation study of a CAT tool for discrimination and measurement of psychotic symptoms (CAT-Psychosis) (Guinart et al., 2021). This investigation was carried out in accordance with the Declaration of Helsinki (World Medical Association, 2013) and all participants provided written informed consent after review of the procedures as approved by the local institutional review board (IRB#180626).
Study population
Included were English-speaking inpatients and outpatients, 18–80 years of age, with a chart diagnosis of schizophrenia, schizoaffective disorder, schizophreniform disorder, delusional disorder or brief psychotic disorder, as well as bipolar disorder, and major depressive disorder, drawn from a previous study (Guinart et al., 2021). Participants were recruited from inpatient and outpatient units based on their medical record and/or referred from other research studies. All study visits were conducted at the Zucker Hillside Hospital, division of Psychiatry Research, a semi-urban, tertiary care facility that draws from inner-city populations in Brooklyn and Queens, as well as from more middle-class and affluent regions on Long Island, providing a representative racial/ethnic and sociodemographic mix of eligible patients. Participants who did not understand English or were unable to sign informed consent and patients with an active substance use disorder, structural brain disease or any serious, acute or chronic medical illnesses that could pose a danger to self and others and could interfere with the patient’s ability to comply with the study procedures were not included.
Procedures
Basic demographic information, family history, current treatment and diagnoses assigned by the current clinical treatment team were obtained from the patient and from the medical record. Psychosis severity was rated by clinicians using the Brief Psychiatric Rating Scale (BPRS) (Woerner et al., 1988), a rater-administered 18-item scale ranging from 1 (absent) to 7 (very severe). Patients rated themselves with the CAT-Psychosis, which was administered using tablet computers with touch screens. Assessment order (BPRS first vs CAT-Psychosis first) was assigned in an alternating manner. Research staff were available to assist the participants if they had difficulty answering the questions or manipulating the tablet. Briefly, the CAT-Psychosis is a self-administered tool to assess the severity of psychotic symptoms that adaptively draws questions from an item bank following an item-response-theory-based full-information item bifactor model (Gibbons et al., 2007; Gibbons and Hedeker, 1992). CAT-Psychosis item bank questions are reworded from commonly used clinician-rated scales, including the Schedule for Affective Disorders and Schizophrenia (SADS), the Scale for the Assessment of Positive Symptoms (SAPS), the Brief Psychiatry Rating Scale (BPRS) and the Scale for the Assessment of Negative Symptoms (SANS) (Guinart et al., 2021). Beginning with an item in the middle of the severity distribution, CAT-Psychosis estimates the person’s severity based on their initial response and subsequently selects the next maximally informative item, learning the severity of the person as the test adaptively administers items, thus increasing measurement precision and reducing administration time. With a median length of assessment of 12 self-reported questions lasting less than 2 minutes, CAT-Psychosis elicits a valid severity rating, maintaining high correlation with well-established symptom severity assessments, as well as high sensitivity and specificity for discriminating patients from healthy individuals (Guinart et al., 2021). Awareness of illness was evaluated by clinicians using the three general items (G1: ‘awareness of mental disorder’, G2: ‘awareness of the achieved effect of medication’ and G3: ‘awareness of the social consequences of mental disorder’) of the Scale of Unawareness of Mental Disorders (SUMD) (Amador et al., 1993).
Statistics
A descriptive analysis of the data was performed and comparison between groups was explored with chi-square and Mann–Whitney tests. Pearson product-moment correlation coefficients were used to examine associations between continuous measures. To determine whether the relationship between self-reported CAT-Psychosis severity scores and clinician-rated BPRS scores depended on patient’s insight, a moderated regression analysis was performed. Patients were median-split in two groups regarding their insight level to compare the correlation between the two psychosis severity measures among these groups. Regression analyses were performed to test the influence of age, gender and educational level on insight. A moderated regression analysis was performed to study the effect of medication and disorganization (measured by the ‘conceptual disorganization’ item on BPRS) in the correlation among the study variables. In addition, a subgroup sensitivity analysis was performed on patients with schizophrenia spectrum disorders to test the validity of the results in this group. Statistical analyses were performed using SPSS version 23.0 software package (IBM Corp., Armonk, NY, USA). The p-values lower than 0.05 were considered statistically significant.
Results
A total of 200 participants were enrolled in the original study and completed the assessment (Guinart et al., 2021). Of those, 159 completed both the self-administered version of CAT-Psychosis and the clinician-rated SUMD and thus were included in this analysis. Participants were diagnosed with both affective and non-affective psychoses (n = 76 schizophrenia, n = 35 schizoaffective disorder, n = 26 bipolar disorder, n = 10 major depressive disorder, n = 7 psychosis not otherwise specified and n = 5 schizophreniform disorder). Consolidated Standards of Reporting Trials (CONSORT) (Schulz et al., 2010) flow diagram is available in Figure 1. Sociodemographic characteristics of the sample are described in Table 1.
CONSORT diagram.
Sociodemographic characteristics of the included sample.
Q1: first quartile; Q3: third quartile.
Reported p-values refer to the result of χ2 or Mann–Whitney tests comparing the two groups.
Self-administered CAT-Psychosis showed convergent validity with total BPRS scores (r = 0.690, 95% confidence interval [CI] = [0.609, 0.756], p < 0.00001) for the whole sample (N = 200). For patients affected with psychosis who received the SUMD (n = 159), self-administered CAT-Psychosis scores also showed convergent validity with BPRS ratings (r = 0.517, 95% CI = [0.392, 0.622], p < 0.001). SUMD scores were found to moderate this correlation (β = –0.511, p = 0.005), with lower levels of insight associated with less agreement between clinician-rated and self-reported scores (r = 0.621, 95% CI = [0.476, 0.733], p < 0.001 for high insight patients, and r = 0.408, 95% CI = [0.187, 0.589], p < 0.001 for low insight patients) (Figure 2). Correlations between individual BPRS items and CAT-Psychosis scores are shown in Supplementary Table 1.
Association between BPRS scores and CAT self-reported scores across high and low insight patients.
Disorganization was not found to moderate the correlation between clinician-rated and self-reported psychosis scores (β = 0.223, p = 0.638). No statistically significant differences were found in BPRS (U = 3057, z = –0.129, p = 0.897), disorganization (U = 2986.5, z = –0.274, p = 0.784) and CAT-Psychosis (U = 2800.5, z = –1.022, p = 0.307) scores between high and low insight patients. No correlation was found between SUMD and BPRS scores (r = 0.105, 95% CI = [–0.052, 0.256], p = 0.189), and regression analyses did not find any correlations between age (r = –0.025, p = 0.383), gender (r = –0.057, p = 0.254) or years of education (r = –0.020, p = 0.408) with SUMD scores. No moderation effect was found for the use of antipsychotics or benzodiazepines and the correlation between self-reported and clinician-rated psychosis scores (β = –2.701, p = 0.324, and β = –0.883, p = 0.592, respectively).
An additional subgroup sensitivity analysis including only schizophrenia spectrum disorders (n = 123; n = 76 schizophrenia, n = 35 schizoaffective disorder, n = 7 psychosis not otherwise specified and n = 5 schizophreniform disorder) showed a statistically significant correlation between CAT-Psychosis and BPRS total scores (r = 0.521, 95% CI = [0.379, 0.639], p < 0.001) that was moderated by the level of insight (β = –0.494, p = 0.034). Correlation between self-reported and clinician-rated severity remained statistically significant for patients with high (r = 0.658, 95% CI = [0.476, 0.786], p < 0.001) and low insight (r = 0.450, 95% CI = [0.238, 0.622], p < 0.001).
Discussion
We found that insight partially influences the ability of patients to measure the severity of their psychotic symptoms. In our sample, patients with lower insight show slightly less agreement between self-reported and clinician-rated psychotic symptoms than patients with higher insight. However, psychosis severity scores measured by the BPRS were not significantly different between high and low insight groups. This suggests that insight, and not severity of illness, better explains the disagreement between self-reported and clinician-rated scores. Importantly, patients with lower awareness of illness were still able to accurately report their psychotic symptoms, as score correlations remain statistically significant in both groups. Thus, in our study, patients with psychotic symptoms and poor illness awareness were a reliable source of information, also when including only schizophrenia spectrum disorders.
Our findings are highly relevant because, unlike in other areas of mental health, self-report measures in psychosis are still infrequently used (Rabinowitz et al., 2008), partly due to concerns of minimization and/or denial of symptoms due to stigma and/or impaired awareness of illness (Doyle et al., 1999). Yet, we show that while the impact of insight on self-reported assessment of psychosis indeed exists, the amount of such impact does not challenge the validity of self-reports of symptom severity. In line with our results, some studies (Lincoln et al., 2010) show that self-reported assessments of delusions, such as Peters et al. Delusions Inventory (Doyle et al., 1999) and the Paranoia Checklist (Doyle et al., 1999), were significantly correlated to delusion scores as rated by clinicians with the Positive and Negative Syndrome Scale (Kay et al., 1987) in patients with schizophrenia, and the level of insight modified this correlation, with lower insight patients reporting less severity of delusions. However, it remained unclear whether the assessments remained valid and reliable in patients with low insight. In our study, even patients with low insight showed capacity to accurately rate themselves. Furthermore, we expand these findings to not just delusions but to a general score of psychosis severity, and on a larger sample of patients.
Other studies (Iancu et al., 2005; Liraud et al., 2004) did not find an effect of insight on the correlation between self-reported and clinician-rated psychotic symptoms. Beyond differences in sample size, some considerations may explain our discrepant findings. On one hand, this discrepancy could be explained by methodological differences, since Liraud et al. (2004) administered the self-report after the clinician interview, which could have made patients more aware of their symptoms, thus inducing bias. On the other hand, Iancu et al. (2005) report a measure with low external validity, especially for negative symptoms (r = –0.086, p > 0.1), although patients rated themselves higher in this domain. Thus, patient’s responses may reflect a different construct than the one measured by clinicians, and therefore level of insight fails to influence severity scores. Unlike previous studies (Takeuchi et al., 2016), data from this study do not show an influence of disorganization in moderating the correlation between clinician-rated and self-reported psychosis scores. Differences in sample size and study methodology, including assessment tools, could explain such discrepancies, exposing a potential additional advantage of adaptive testing versus traditional, fixed-length assessments.
In a context of a progressive incorporation of self-reports and PROMs into routine clinical practice to improve quality of care and decision-making (Guo et al., 2015), instruments such as CAT-Psychosis represent a useful tool to expand evidence-based medicine beyond research facilities. Self-reports are less costly, as they do not require highly trained professionals for administration and are generally shorter than clinician-administered tests. Furthermore, a major advantage over fixed-length, identical tests is that these cannot be administered in short time spans due to risk of response bias. CAT-Psychosis, however, produces a tailored yet adaptive set of questions at each administration, shortening assessments significantly, minimizing patient burden and allowing for more frequent evaluations, even in patients with low insight.
This study comes with some limitations. First, psychosis severity in the CAT-Psychosis battery is calculated as an overall severity score, including all symptom domains, so it remains unclear whether awareness of illness may influence self-rating of specific psychosis domains differently. Second, duration of illness and cognitive dysfunction, which are known mediators of insight, could not be evaluated as these were not assessed in our study. Finally, in this study, we used the SUMD in its short form, which, despite being valid and commonly used (Birindelli et al., 2014; Cobo et al., 2020; Dumas et al., 2013; Michel et al., 2013), does not tap into awareness of specific symptoms, nor assesses symptom attribution, unlike longer versions.
Conclusion
Insight moderates the agreement between self-reported and clinician-rated severity of psychosis, yet CAT-Psychosis remains a valid tool to measure severity of psychotic symptoms in patients with both high and low awareness of illness.
Supplemental Material
sj-docx-1-anp-10.1177_00048674211057480 – Supplemental material for Awareness of illness moderates self-assessment of psychotic symptoms
Supplemental material, sj-docx-1-anp-10.1177_00048674211057480 for Awareness of illness moderates self-assessment of psychotic symptoms by Luis Martinez Agulleiro, Renato de Filippis, Stella Rosson, Bhagyashree Patil, Lara Prizgint, Nahal Talasazan, Herbert Y Meltzer, John M Kane, Robert D Gibbons and Daniel Guinart in Australian & New Zealand Journal of Psychiatry
Footnotes
Funding
The author(s) received no financial support for the research, authorship and/or publication of this article.
Declaration of Conflicting Interests
L.M.A., R.d.F., S.R., B.P., N.T. and L.P. have nothing to disclose. H.Y.M. has been a consultant and/or advisor for or has received honoraria from Eli Lily, Gerson Lehrman Group, Guidepoint Global, LB Pharmaceuticals, Quincy Bioscience and Sumitomo Dainippon Pharma and is a board member and shareholder in Acadia Pharmaceuticals. J.M.K. has been a consultant and/or advisor for or has received honoraria from Alkermes, Allergan, LB Pharmaceuticals, H. Lundbeck, Intracellular Therapies, Janssen Pharmaceuticals, Johnson & Johnson, Merck, Minerva, Neurocrine, Newron, Otsuka, Pierre Fabre, Reviva, Roche, Sumitomo Dainippon, Sunovion, Takeda, Teva and UpToDate and is a shareholder in LB Pharmaceuticals and Vanguard Research Group. R.D.G. has been an expert witness for the US Department of Justice, Merck, GlaxoSmithKline, Pfizer and Wyeth and is a founder of Adaptive Testing Technologies, which distributes the CAT-MH™ battery of adaptive tests in which CAT-Psychosis is included. D.G. has been a consultant for and/or has received speaker honoraria from Otsuka America Pharmaceutical, Inc. and Janssen Pharmaceuticals.
Supplemental Material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
