Look Before You Leap: Representativeness of Those Completing Self-Reports in Early Psychosis Research

A randomized controlled trial (RCT) was conducted to compare the efficacy of early intervention (EI) for psychosis extended for 5 years with 2 years of EI followed by 3 years of regular care (details in prior publications ^1,2 ). As secondary hypotheses, we postulated that the extended EI group (n = 110) would have higher levels of alliance with their treatment providers than the control group (n = 110) and that differences in medication adherence and retention in treatment between the two groups would be predicted by working alliance. Alliance was to be measured every 6 months after randomization (i.e., Months 30, 36, 42, 48, 54, and 60) with the Working Alliance Inventory (WAI), a self-report instrument. ³

The average working alliance in the extended EI and regular care groups was 63.53 (SD = 12.24, range: 24 to 84, N = 85) and 59.35 (SD = 12.30, range: 30 to 82, N = 46), respectively, t(129) = 1.862, P = 0.065. The minimum score on the WAI is 12 and the maximum 84, indicating that in both arms, individuals reported moderate to high levels of alliance.

Our results should not be interpreted as indicative of a true lack of difference in therapeutic alliance between persons receiving extended EI and regular care, given that individuals in the EI group were significantly likelier to have filled out the WAI at least once during follow-up than those receiving regular care (83% vs.44%, respectively; χ² = 36.129, P < 0.001). We therefore chose not to conduct any additional analyses of the impact of alliance on group differences in medication adherence and retention.

Willingness to fill out self-reports may in itself serve as an indicator of alliance/engagement with mental health systems/teams. Earlier, we reported that individuals receiving extended EI were likelier to remain engaged in their follow-up and have more contacts with their doctors and other treatment providers. ² That they were also likelier to complete the WAI may be an additional indicator of better “engagement” in the EI group. Assessments were conducted in both arms by the same research assistant. Nonetheless, some factors may have facilitated completion of measures in the EI group. Self-reports and structured interviews to assess symptoms were completed during in-person appointments with the research assistant who was in the same institution as the EI program, albeit in a separate pavilion. When individuals could not come in person, the research assistant completed the structured symptom interviews on the telephone. Self-reports, however, could not be done telephonically. This may have contributed to a higher rate of completion of symptom assessments (our primary outcome ² ). If individuals consented, self-report measures were mailed to them, but very few mailed measures were returned. In the context of an RCT, it was important for the research assistant to be blind to treatment condition. This impeded us from adopting methods that prior research including in psychosis ⁴ suggests may have facilitated completion of self-reports, for example, asking people to complete self-reports during their clinical appointments, so that completed measures could inform treatment decision-making in real time.

Because a sizable number of individuals did not ever fill out the WAI, we examined the differences between individuals who completed the WAI during at least one of the six time points (n = 139) versus those who never completed it at any time point (n = 81), irrespective of treatment condition. Although 139 individuals filled out the measure, 8 left out individual items, resulting in a total of 131 individuals for whom average WAI could be calculated. There was no significant difference between those who completed the WAI and those who did not in terms of duration of untreated psychosis, gender, education, diagnosis of nonaffective versus affective psychosis, and presence of a substance use diagnosis. Compared to those who did not fill out the WAI, those who completed it had longer periods of symptom remission during the 3-year extension period, M = 40.08 weeks, SD = 37.95 vs. M = 70.14 weeks, SD = 42.59; t(152) = 4.014, P < 0.001. Completers were also likelier to have remained engaged through the 5-year study period (76% vs. 50.0%, χ² = 12.943, P = 0.001) and to have received a higher number of interventions from a physician, M = 16.96, SD = 13.99 versus M = 10.74, SD = 10.65; t(187) = 3.138, P = 0.002, and another health-care professional, M = 53.37, SD = 56.51 versus M = 25.51, SD = 31.19; t(186) = 4.366, P < 0.001, during the 3-year extension period. They had also been more adherent to their medications over their follow-up (94% vs. 83%; χ² = 5.012, P < 0.05). We then evaluated whether our results were particular to the WAI and found the exact same pattern of differences between those who completed and those who did not complete select sections of the client version of the Wisconsin Quality of Life Questionnaire, ⁵ the other self-report in this RCT. Our results highlight the need for research based on self-reports to carefully examine sample representativeness. ⁶ In our study, those who filled out self-reports were likelier to stay engaged in treatment, be medication-adherent, and receive more treatment from physicians and other health-care professionals. Thus, completing self-reports may itself be an indication of higher alliance.

When self-reports are available only for a subsample, conclusions about the links between constructs assessed by self-reports (e.g., alliance) and other key variables such as outcomes, adherence, and so on, may be erroneous as the sample with data may have been self-selected for high levels of service engagement.

Interestingly, the sample that was likelier to fill out self-reports also spent more time in symptom remission, the trial’s primary outcome. Those who are symptomatically better or have the potential to be symptomatically better may be more willing to participate in research, particularly the completion of self-reports. It is also possible that filling out self-reports may itself be an indicator of better alliance and that higher levels of alliance contribute to better adherence and longer periods of symptom remission. This would be in keeping with previous evidence for an association between working alliance and clinical outcomes in early psychosis. ⁷

Few publications detail the challenges of implementing longitudinal services research projects. Our study highlights the difficulty of engaging patients with psychosis in filling out self-reports at repeated intervals over a 3-year period and the need for innovative strategies to mitigate this. One strategy could be e-administration, which would allow individuals to complete self-reports at their preferred time and place and can be a user-friendly and effective method of gathering clinical information. ^8,9 Perhaps, even more importantly, higher rates of completion may be facilitated by allowing individuals to have access to their data and its interpretation and by creating specific mechanisms for data to inform treatment decision-making. Such an approach would be consistent with not only measurement-based care ^10,11 but also a recovery orientation. ¹²