Abstract
Schizophrenia is a complex mental disorder characterized by serious deficits in thinking, affect, perception, and social skills (Rus-Calafell et al., 2014). For some people, schizophrenia is a particularly disabling condition, and impairment in self-care skills and other areas of occupational performance can lead to substantial burdens for caregivers and society in general. Occupational therapy practitioners provide a variety of interventions to enhance occupational performance and improve occupational participation for this population (Lipskaya-Velikovsky et al., 2016). An observational measure is crucial to enable clinicians and researchers to measure clients’ behaviors as they participate in activities and then frame and modify treatment plans.
The Comprehensive Occupational Therapy Evaluation Scale (COTES; Chiu et al., 2019) is a measure designed to assess behaviors that affect occupational performance; it is used in many countries, including Taiwan (Chen et al., 2006; Liang & Yai, 2017; Liu et al., 2018). The COTES was developed on the basis of occupational therapy theories (Azima & Azima, 1959; Fidler & Fidler, 1963; Mosey, 1971) and Brayman et al.’s (1976) Comprehensive Occupational Therapy Evaluation Scale, as well as consideration of psychiatric symptoms and behaviors of people with mental illness. It has been culturally adapted from Brayman et al.’s version for use in Asian countries. The COTES contains 20 items in three subscales: General Behavior, Social Behavior, and Work Behavior (Kau et al., 1981). The COTES is an observational measure that can be administered during a group activity in a hospital or community setting (Chen et al., 2006; Li et al., 2012; Liu et al., 2018). The COTES includes multidimensional facets of behaviors affecting occupational performance, which make it suitable for comprehensively determining clients’ strengths and weaknesses.
Test–retest reliability indicates the degree to which a measure yields stable results (i.e., free from measurement error) over repeated measurements (Hobart et al., 2004). Random measurement error needs to be considered when interpreting score changes. Minimal detectable change (MDC) is the minimum change measurable beyond random measurement error at certain confidence levels (e.g., 95%) in consecutive measurements (Haley & Fragala-Pinkham, 2006). The MDC helps explain whether a score change indicates a real change (improvement or deterioration) for the client.
Test–retest reliability of the COTES has not been evaluated or the MDC calculated for people with schizophrenia, limiting the application of this tool and explanation of its results. The purpose of this study was to examine the test–retest reliability of the COTES and to estimate its MDC in people with schizophrenia.
Method
Participants
This retrospective study used total population sampling. The data were gathered from occupational therapy records at a psychiatric center in northern Taiwan. We examined COTES data gathered from April 2006 to June 2014 from people with schizophrenia who were eligible to receive occupational therapy intervention twice a day in the acute psychiatric setting. People with schizophrenia diagnosed according to the criteria of the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision (American Psychiatric Association, 2000) were included in the study. Exclusion criteria were diagnosis of substance abuse or intellectual disability and history of brain injury. This study was approved by the institutional review board of the local hospital.
Procedure
After participants’ symptoms were stable, they were assessed with the COTES and began occupational therapy intervention. The COTES was administered by occupational therapists as part of participants’ weekly assessments. The number of COTES measurements was different for each participant. Most received at least two measurements, so we used the initial and second measurements to determine test–retest reliability. We expected that participants with stable symptoms would show steady occupational performance over the short period between the two measurements.
Two research assistants reviewed the occupational therapy records and selected eligible participants. COTES and demographic data were collected by one research assistant and proofread by the other.
Instrument
The COTES is administered during observation of clients’ behaviors as they participate in activities (e.g., activities of daily living, leisure, work). The General Behavior subscale contains seven items: attendance, appearance, activity level, emotional disturbance, nonproductive behavior, physiological needs, and impairment of speech. The Social Behavior subscale consists of five items: sociability, impression of others, acceptance of opinions, role in groups, and self-assertion. The Work Behavior subscale includes eight items: motivation, duration, responsibility, frustration tolerance, self-expectation, comprehension, technique, and fine motor. The COTES items are rated from 1 (indicating inappropriate behavior) to 5 (indicating appropriate behavior). Most items (client behaviors) are rated on a 5-point scale ranging from 1 to 5. Four items (nonproductive behavior, motivation, comprehension, and technique) are rated on a 9-point scale ranging from 1 to 5 with intervals of 0.5 between the integers (e.g., 1, 1.5, 2). One item, appearance, is rated on an 11-point scale ranging from 0 to 5 with intervals of 0.5 between the integers (e.g., 0, 0.5, 1). Subscale score ranges are 7–35 for General Behavior, 5–25 for Social Behavior, and 8–40 for Work Behavior. A higher score on an individual subscale indicates better subscale-specific behavior. The total score of the three subscales ranges from 20 to 100 and provides an overall measure of behavior affecting occupational performance (Kau et al., 1981).
Data Analysis
Test–retest reliability was examined using the intraclass correlation coefficient (ICC2,1) based on a two-way random-effects model with an absolute agreement type. The criteria for ICC values were >.90, excellent reliability; .75–.90, good reliability; .50–.75, moderate reliability; and <.50, poor reliability (Koo & Li, 2016). The ICC values were ≥.70 for group comparisons and ≥.90 for individual comparisons (Aaronson et al., 2002). The MDC values for the overall scale and subscales were computed as follows (Haley & Fragala-Pinkham, 2006):
where SEM is the standard error of measurement, SD 1 is the standard deviation of the first measurement, and z scorelevel of confidence is the confidence interval in a normal distribution. In this study, the confidence interval used was 95%, so the z score used was 1.96.
The MDC% was calculated by dividing the MDC by the mean score of all participants and then multiplying by 100 (Lexell & Downham, 2005). The MDC% can be applied to compare the relative quantities of random measurement error among different measures. MDC% values of <10% are considered excellent and <30%, acceptable for random measurement errors (Smidt et al., 2002).
Bland–Altman plots with limits of agreement (LOAs) can represent reproducibility of results in two repeated measurements (Bland & Altman, 1986). In a Bland–Altman plot, the difference between scores in two measurements is plotted against the mean values of the two measurements. The LOA was calculated as the mean difference ± (1.96 × SD of the difference). The plot was used to demonstrate whether there was a tendency to heteroscedasticity—for example, whether the differences between the two measurements generally increased when the mean values of the two measurements increased (Bland & Altman, 1999). We evaluated heteroscedasticity with Pearson’s r between differences in the scores and mean values of the two measurements, with r > .30 indicating heteroscedasticity (Atkinson & Nevill, 1998).
Systematic bias was evaluated using paired t tests and effect sizes. A paired t test (two tailed, α = .05) was conducted to verify whether a statistically significant difference appeared between the two measurements. Effect size (Cohen’s d) was used to determine the size of systematic bias. The criteria for Cohen’s d values were as follows: 0.20–0.49, small effect size; 0.50–0.79, moderate effect size; and ≥0.80, large effect size (Chiu et al., 2016).
Results
The data of 118 people with schizophrenia were collected. About two-thirds of the participants (63.6%) were male. Their mean age was 39.0 yr, and the mean age at onset was 23.8 yr. Participants received a mean of 8.7 (SD = 1.6) occupational therapy interventions between the initial and second measurements. Table 1 presents the participants’ characteristics.
Participant Characteristics (N = 118)
Note. M = mean; SD = standard deviation.
The ICC value of the overall COTES was .95 (Table 2), and the ICC values of the three subscales were .91–.97. The MDC values were 6.5 for the overall scale and 2.3–3.4 for the three subscales. The MDC% values for the overall scale and subscales were 10.5%–15.2%.
Reliability Results for the COTES
Note. CI = confidence interval; COTES = Comprehensive Occupational Therapy Evaluation Scale; ICC = intraclass correlation coefficient; M = mean; MDC = minimal detectable change; SD = standard deviation; SEM = standard error of measurement; Time 1 = initial measurement; Time 2 = second measurement.
Figure 1 shows the Bland–Altman plots for the overall scale and subscales of the COTES. LOA ranges were −4.4 to 7.4 for the overall scale, −2.7 to 3.8 for the General Behavior subscale, −1.6 to 2.6 for the Social Behavior subscale, and −1.8 to 2.7 for the Work Behavior subscale. The correlations (r) between the differences in the scores and mean values of the two measurements for the overall scale and subscales were .02–.10 (Table 2). Regarding systematic bias, the effect sizes (d) of the overall scale and subscales were 0.09–0.19 (Table 2). Differences between the two measurements were significant (p < .05) for the overall scale and subscales.
Bland–Altman plots: (A) overall COTES, (B) General Behavior subscale, (C) Social Behavior subscale, (D) Work Behavior subscale.
Discussion
The COTES has been used as an outcome measure to assess behaviors that affect occupational performance in people with schizophrenia (Chen et al., 2006). To the best of our knowledge, this study is the first to examine its test–retest reliability and estimate its MDC with a large sample (N = 118) of people with schizophrenia. Our results showed good test–retest reliability and yielded MDC values that may be generalizable to other samples of people with schizophrenia. Clinicians and researchers can use our results to determine real changes in behavior for people with schizophrenia.
ICCs for the overall scale and subscales of the COTES exceeded the criterion of .90 for individual comparisons. A previous study examining the interrater reliability of the overall scale with a small sample (N = 23) found an ICC of .91 (Li et al., 2012), similar to our finding. For the overall scale and subscales, an MDC% of <30% indicates acceptable random measurement error. Regarding systematic bias, no systematic trends were found for the overall scale or subscales in the Bland–Altman plots (i.e., differences in scores were not associated with the mean values of the scores). In the Bland–Altman plots, the mean scores of measurements were distributed within the ranges of the overall scale and subscales, indicating that the participants displayed a wide range of difficulties in behaviors affecting occupational performance. The Cohen’s ds of the overall scale and subscales were <0.20, indicating that the systematic bias was trivial. The paired t test results for the overall scale and subscales showed significant differences between the two measurements. Improvements in occupational performance may have occurred during the first week of occupational therapy, so the results might reflect real changes. Thus, the systematic bias of the COTES is negligible, and the MDC values may have been overestimated.
Regarding the clinical application of this study, clinicians and researchers can use the MDC values to explain whether a person with schizophrenia shows a real change between two measurements. For instance, a posttest score on the overall COTES that is 6.5 points higher than the pretest score demonstrates a real improvement with 95% certainty. The MDC values can also be perceived as thresholds to verify a statistically significant change for clients with schizophrenia (Haley & Fragala-Pinkham, 2006). A score change between repeated measurements higher than the MDC can be treated as a significant change. The MDC can thus be used in clinical and research settings to identify whether a client has achieved a significant improvement.
The MDC at the individual level can be adjusted to the MDC at the group level for research purposes. The MDC at the group level is the MDC at the individual level divided by the square root of the sample size (de Vet et al., 2001). The MDC at the group level would seldom be used if the sample size is sufficient. For instance, if the MDC at the individual level of the General Behavior subscale is 3.4 points with a sample size of 30, then the MDC at the group level would be 0.6 (i.e., 3.4/√30), which is too small to be of concern. We recommend that researchers use the MDC at the individual level as the threshold to report the proportion of participants in a study group who show real change (Haley & Fragala-Pinkham, 2006). In clinical trial studies, researchers estimate the mean differences within the study group to represent the significant mean changes. However, the significant mean changes do not imply that all participants showed significant improvement. Even if the mean changes within the study group were statistically significant, individual participants might not show changes higher than the MDC at the individual level. Therefore, the proportion of participants with scores higher than the MDC at the individual level can assist researchers in interpreting research findings under clinical conditions and improve the applicability of research results.
Limitations
This study has two major limitations. First, secondary data were used from occupational therapy records, and thus the data quality might be unstable. Further studies are warranted to examine test–retest reliability in a prospective study of people with schizophrenia to cross-validate our results. Second, we computed the MDC values of the COTES for real change (beyond random measurement error) showing statistical significance. We did not examine the minimal clinically important difference (MCID), which indicates whether a change is meaningful, for people with schizophrenia or clinical users. Further studies examining MCID with this population are needed to provide better estimates for decision making in clinical settings.
Implications for Occupational Therapy Practice
The results of this study have the following implications for occupational therapy practice:
Behaviors that affect occupational performance are a crucial indicator for participation in occupations in daily life. Occupational therapists can use the overall scale and three subscales of the COTES to measure the progress of people with schizophrenia.
Occupational therapy practitioners can apply the MDC values to COTES results to determine whether clients with schizophrenia make real improvements in behavior over time.
Conclusion
Our results show that the test–retest reliability of the COTES was good, random measurement error was acceptable, and systematic bias was not observed. The COTES can be used to monitor clients’ behaviors affecting occupational performance over time. In addition, the MDC values we calculated for the overall scale and subscales for people with schizophrenia can help clinicians and researchers explain clients’ score changes.
Footnotes
Acknowledgments
We are grateful to the participants for their involvement. This study was supported by a research grant from the Taipei City government (10701-62-065) and the Ministry of Science and Technology (106-2314-B-532-002).