Abstract
Aims
This study was to test the reliability and validity of the Assessment of Positive Occupational 15 (APO-15) for individuals experiencing mental illness.
Methods
A sample of 408 people experiencing mental illness living in communities or admitted to hospitalized was recruited. The sample has completed demographic information, the Assessment of Positive Occupation 15 (APO-15), the Japanese version of the Self-identified Stage of Recovery Part-B (SISR-B), the Japanese version of the Recovery Assessment Scale (RAS), the General Health Questionnaire 12 (GHQ-12). APO-15 is a measure of how engaged one is in occupations that promote well-being. The final version of the APO-15 was developed by assessing the validity and reliability by mainly using confirmatory factor analysis (CFA), item response theory (IRT).
Results
This study indicated satisfactory the validity and reliability of APO-15 in a group of individuals experiencing mental illness. CFA showed acceptable values for all indices of fit, namely comparative fit index (CFI), Tucker-Lewis index (TLI) (i.e., greater than .90), and the value of root mean square error of approximation (RMSEA) was .087, which was acceptable. The IRT showed satisfactory responses for the item slope parameter (α) and item difficulty parameter (β) in APO-15.
Discussion
APO-15 was demonstrated good psychometric properties in measuring involvement in the occupation to promote well-being in individuals experiencing mental illness. In conclusion, the APO-15 is an important tool to enable occupational therapists to assess clients who are not engaged in well-being promoting occupations and thus enable them to participate in such occupations.
Introduction
In recent years, an increase in suicides that continue to increase year after year in mental health problems has been warning the social issues, such as economic loss (World Health Organization (WHO), 2013). Therefore, trends to promote the well-being of individuals with experiences of mental illness have been observed in the field of mental health (Corrigan & Phelan, 2004; Slade, 2009; WHO, 2013). These trends suggest that the promotion of well-being should be considered in the rehabilitation of clients with experiences of mental illness. Psychosocial occupational therapy is a client-centered practice concerned with promoting the well-being of individuals through occupation (Giroux Bruce & Borg, 2002). Occupation is defined as a center of the human experience; such as work, leisure, play, self-care, social interaction (Wilcock, 2006). Well-being is defined as the perceived state of harmony in all aspects of one’s life (Low et al., 1998). Occupational well-being is defined as a perceived state of satisfaction and pleasure from everyday experience (Charles & Townsend, 2013; Schultz, 2015). This study refers to occupations that promote human health and well-being as positive occupation and defines them as human experiences that constitute a state of physical, mental, and social well-being. From this, at the core of occupational therapy is a belief about the engagement between occupation and well-being (Canadian Association of Occupational Therapists, 1997; Low et al., 1998; Wilcock, 2006). Therefore, psychosocial occupational therapy needs to be able to assess involvement in the occupation to promote well-being.
At present, the relevant assessments used include the Canadian Occupational Performance Measure (COPM), the Occupational Self-Assessment (OSA), the Model of Human Occupation Screening Tool (MOHOST) has assessments that have been focused on the relationship between occupation and well-being (Carswell et al., 2004; Kobayashi et al., 2010; Parkinson et al., 2008). In other words, they focus on the clients' occupation, which is closely related to their life experiences and daily life, and they are designed to enable the clients' occupation. On the other hand, previous research in occupational therapy, occupational science, and positive psychology has identified occupations with evidence for the promotion of health and well-being. For example, regarding research on flow, improvements in wellbeing, self-esteem, and subjective well-being levels have been reported (Larson & von Eye, 2010; Nakamura & Csikszentmihalyi, 2014; Rebeiro & Polgar, 1999; Wright et al., 2007). Meaningful occupation participation has been reported to promote occupational performance and contribute to health and well-being (Christiansen et al., 2005; Crepeau et al., 2009; Eklund et al., 2017). Previous studies on adults, students, and people with mental illnesses reported that people with gratitude habits are more optimistic and have higher levels of happiness and life satisfaction (Algoe, 2012; McCullough et al., 2002; Post, 2005; Wood et al., 2010). However, in the field of occupational therapy, there is no measure of the extent to which clients are engaged in positive occupations that promote health and well-being described above (Noguchi et al., 2016).
Therefore, we developed a measurement tool called the Assessment of Positive Occupation 15 (APO-15) (Noguchi et al., 2016). The theoretical background of the APO-15 is the PERMA model in positive psychology. The PERMA model is a theory that attempts to understand the state of well-being in individuals according to the five domains of P(positive emotion), E(engagement), R(positive relationship), M(meaning), and A(achievement) (Seligman, 2011). The APO-15 is unique in that it employs positive occupation with promoting health and well-being as scale items, as described above. That is, APO-15 is a scale that measures the extent to which clients are engaged in a positive occupation. The APO-15 prototype was studied in 110 people with experience of mental illness living in the community. The APO-15 was examined for scale characteristics based on statistical evidence, including exploratory factor analysis (EFA), confirmatory factor analysis (CFA), and item response theory (IRT). The results of the EFA converged on 15 items of the four factors [Positive Relationships, Achievement, Meaning, and Engagement]. CFA was performed on the APO-15 scale structure, result in a good fit that was confirmed. IRT showed that APO-15 showed good discrimination and difficulty in all items. Also, confirmation of the sensitivity measure in the item characteristic curve (ICC) showed that APO-15 was highly sensitive to clients who were less likely to engage in positive occupation. Thus, the overall assessment properties of the APO-15 prototype were very good.
On the other hand, although the APO-15 prototype was developed for people experiencing mental illness living in the community, to date there have been no reports confirming its use in hospitalized patients. Psychosocial occupational therapy supports individuals with experiences of mental illness living in the community and those admitted to the hospitals. Therefore, the purpose of this study was to investigate the reliability and validity of the APO-15 final version with the aforementioned individuals.
Methods
Ethics statement
The research protocol was approved by the Ethics Committee of Kibi International University (No. 14-32) and the Research Ethics Committee of Zikei Hospital (No. 103[27-2]). All participants provided both written and verbal informed consent before participation. Participation was voluntary, and participants had the right to withdraw from the research at any time without providing any reason. This study was conducted according to the Declaration of Helsinki.
Participants
Sample size was determined with reference to the COnsensus-based Standards for Selection of Health Measurement INstruments (COSMIN) (Mokkink et al., 2019). COSMIN recommends a minimum sample size of 100 participants. To ensure an enough sample size, this study decided to collect a sample of more than 400 participants. Participants were included among those diagnosed with a mental illness based on the DSM-5 who were using psychiatric hospitals, and day hospitals. The selection criteria were: (1) persons with a diagnosis of mental illness, (2) persons aged 15 to 75 years of age, and (3) persons agreeing to this study. Exclusion criteria were (1) medical condition was unstable and those who were judged by the doctor or facility manager as impossible to cooperate, (2) persons aged under the 15, or over the 75, (3) those who did not agree with this study. The recruitment of the subject posted posters showing the outline of this study were displayed in the hall and other easily visible places and asked for participation under the subject's free will. Also, subjects who expressed interest in the study were individually presented with a guide describing the study's content to review with them and obtain their consent. To ensure a large number of subjects, this study also sought the cooperation of adolescents (i.e., 15-18 years old) clients with their own and their families' consent.
Measures
The study asked subjects to respond to the measures listed below. In this study, we also used a scale related to recovery, because recovery has become an important trend in the psychiatric field in recent years (Corrigan & Phelan, 2004; Slade, 2009; WHO, 2013).
Demographic information
We examined age, gender, diagnosis, and sense of happiness. Happiness was scored on a five-point Likert scale, ranging from 1 (not at all happy) to 5 (very happy).
APO-15
The APO-15 is a self-administered scale that measures the extent to which people with experiences of mental illness can engage in positive occupations. APO-15 is a 15-item measure of engagement in a positive occupation based on four factors: Positive relationship (5 items), Achievement (4 items), Meaning (3 items), and Engagement (3 items). A positive relationship is defined as derive happiness and satisfaction from human relationships. Achievement is defined as an attempt to complete a target in life. Meaning is defined as significance found in activities and life. Engagement is defined as the flow experienced and the process leading to it. APO-15 evaluates 15 items on a 4-point Likert scale (1 = disagree, 4 = agree). High total scores are related to a higher degree of well-being through positive occupation (Noguchi et al., 2016). APO-15 is paper-based and can be easily completed in 5 to 10 minutes.
The Japanese version of the Self-identified Stage of Recovery Part-B (SISR-B)
SISR-B measures the process of recovery based on four factors: hope (1 item), identity (1 item), meaning (1 item), and responsibility (1 item). SISR-B is assessed using a 6-point Likert scale from 1 (strongly disagree) to 6 (strongly agree). Higher total scores of SISR-B indicate a higher recovery level (Chiba, Kawakami, et al., 2010).
The Japanese version of the Recovery Assessment Scale (RAS)
We selected RAS, which is comprised of 24 items, to measure the perceptions of recovery in five factors: personal confidence and hope (9 items), willingness to ask for help (3 items), goals and success orientation (5 items), reliance on others (4 items), and no domination by the symptoms (3 items). RAS is evaluated using a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree). A high RAS total score indicates a higher recovery level (Chiba, Miyamoto, et al., 2010).
The General Health Questionnaire-12 (GHQ-12)
We used GHQ-12, which is comprised of 12 items, each evaluates the severity of a mental problem over the past few weeks. GHQ-12 had two factors; it includes previous studies on depressive anxiety (6 items) and disability (6 items). Each item is assessed on a 4-point Likert scale from 1 (can have) to 4 (could not have at all) (Lesage et al., 2011). The two most commonly used scoring types are the bi-modal (0-0-1-1) and Likert scoring methods (0-1-2-3) (Liang et al., 2016). We used a standard 0-0-1-1 scoring system for interpretation of results (0 = codes 1 and 2, 1 = codes 3 and 4).
Statistical analysis
This study followed the standards established by the COSMIN (Mokkink et al., 2019). COSMIN is alarmed by the fact that despite the development of many scales, the characteristics that need to be elucidated scientifically have not been adequately tested. Therefore, COSMIN has been developed to solve the above problems, and it defines the scale characteristics to be revealed in many aspects. The elucidation of scale properties proposed by COMIN has four aspects: reliability, validity, responsiveness, and interpretability. This study used polyserial correlation coefficient, structural validity, hypothesis testing, and concurrent validity were used to investigate the validity of APO-15 with reference to COSMIN. Entropy, internal consistency reliability, and item response were used to investigate reliability. Also, the cut-off point was calculated for interpretability. SPSS Statistics (https://www.ibm.com/jp-ja/products/spss-statistics) was used for descriptive statistics, internal consistency reliability, and concurrent validity. HAD (http://norimune.net/had) was used for the normality test. Exametrika (http://antlers.rd.dnc.ac.jp/∼shojima/exmk/index.htm) was used for considering the validity of the items. Mplus 7.3 (http://www.statmodel.com) was used for CFA, hypothesis testing (convergence and discriminant validity), and IRT analysis.
Sample characteristics
The demographic data were summarized using descriptive statistics.
Item validity
We assessed the item validity using polyserial correlation coefficients with critical values above 0.2 and entropy with critical values above 0.5 (Toyota, 2009), and the normality test used was the Jarque-Bera test (p < 0.05).
Structural validity
The factor structure of APO-15 was determined by performing CFA using a weighted least squares estimation with mean and variance (WLSMV), with missing data. WLSMV is suitable for the analysis of categorical data. We used three indices to assess the model fits of CFA based on APO-15 factor structures. The first index was the root mean square error of approximation (RAMSE), with critical values of 0.08–0.10, indicating a mediocre fit, and those of <0.08 indicated a good fit (MacCallum et al., 1996). The second and third indices were the comparative fit index (CFI) and the Tucker-Lewis index (TLI), with critical values above 0.95 (Kline, 2011).
Hypothesis testing (convergent validity and discriminant validity)
In hypothesis testing, we used the multitrait scaling analysis to examine convergent and discriminant validity (Hays & Hayashi, 1990). Specifically, evaluated using the square of the correlation between the factors and average variance extracted (AVE) based on the factor structure of APO-15 supported by CFA. Convergent validity indicates whether each factor assumed to be a component of the scale can properly measure the construct. Discriminative validity indicates whether each factor assumed to be a component of the scale can measure the construct from a different aspect. Therefore, this study confirms whether the four factors of the APO-15 measure “engaging in a positive occupation” with convergent validity. Also, discriminant validity confirmed whether the four elements of the APO-15 (positive relationship, accomplishment, meaning, and engagement) were able to measure the construct from different aspects of the construct. Discriminant validity was assessed by the comparison of the squared correlation between each pair of constructs against the average of AVE. Convergent validity was assessed to investigate whether the square root of each AVE value belonging to each latent construct was >0.5 (Fornell & Larcker, 1981).
Internal consistency reliability
Internal consistency reliability was evaluated using Cronbach’s α coefficient. Cronbach alpha values of 0.7 or higher indicate acceptable internal consistency (Gliem & Gliem, 2003).
Concurrent validity
Concurrent validity was determined using Spearman’s rank correlation coefficient to measure the association between each item of APO-15, the sensation of happiness, SISR-B, and RAS. The criteria for the correlation coefficient are r < 0.3: none or very weak, 0.3< r < 0.5: weak, 0.5< r < 0.7: moderate, 0.7 < r < 0.9: strong.
Item response
Item response was assessed by performing graded IRT using maximum likelihood robust (MLR). The IRT estimated the item slope parameters and item difficulty parameters, total information curve (TIC) in APO-15. (Kosugi & Shimizu, 2014). The item discrimination allows for determining how well items identify clients at different levels of the latent trait. The critical value of 0.5 to 2.0 for item discrimination and an absolute value of –4.0 to 4.0 for item difficulty is typical ranges. (Fayers et al., 2005; Toyota, 2009; Yang & Kao, 2014). The IRT was employed to estimate Akaike’s information criterion (AIC) and Bayesian information criterion (BIC).
Cut-off point
The purpose of using cut-off point in the examination and measurement is to identify a pathological condition, and it is a value that delimits the range that is considered normal based on the reference range. In other words, the reason for using the cut-off point in the APO-15 is to distinguish between clients who can engage in positive occupations and those who are not. The cut-off point for APO-15 was assessed against GHQ-12 as the gold standard by calculating the receiver operating characteristic (ROC) curves. ROC curve is a graph of sensitivity and 1-specificity. The area under the ROC curve of >0.70 was chosen as the critical value to identify good prediction (Mandrekar, 2010).
Results
Sample characteristics
Table 1 shows that there were a total of 408 participants (mean age was 52.4 ± 13.05 years): 273 (67%) were males and 135 (33%) were females. Also, the diagnosis for 302 participants (74%) was schizophrenia: 53 participants (12.99%) were having mood disorders: and others for 53 participants (12.99%). The participants' details are presented in Table 1.
Participant characteristics (n = 408).
SD: standard deviation.
The majority of the participants in the cooperating institutions had experienced mental illness such as schizophrenia and mood disorders.
Item validity
Table 2 shows the values of the Jarque-Bera test, polyserial correlation coefficient, and entropy for each item of APO-15. Normal distribution was shown in item 14. polyserial correlation coefficients indicated values (.550 to .747), and entropy indicated values (1.661 to 1.837). In other words, APO-15 had only one item that could be checked for normality. On the other hand, the APO-15 showed the association between each item and the total score, indicating that the amount of information obtained from each item is sufficient.
APO-15 item analysis.
SD: standard deviation; PCC: polyserial correlation coefficient; JB: Jarque-Bera test.
Structural validity
Table 3 shows the results of the CFA. CFA of APO-15 was a good estimate of the model fit (RMSEA = .087; CFI = .946; TLI = .932). In other words, the four-factor structure of the model assumed in APO-15 was found to be valid.
Structural validity and internal consistency reliability of APO-15.
CI: confidence interval, Factor 1: achievement; Factor 2: meaning; Factor 3: positive relationship; Factor 4: engagement, α = Cronbach’s alpha coefficient.
Hypothesis testing (convergent and discriminant validity)
APO-15 demonstrated good convergent (AVE = .446 to .641) and discriminant validity (squared correlation coefficient = .396 to .583). From this, although the positive relationship factor was somewhat smaller than the criteria, the four APO-15 factors as a whole were generally able to measure the construct of being “engaged in positive occupation”. The four factors also indicated that it was possible to measure the components from different aspects.
Internal consistency reliability
Table 3 shows the results related to internal consistency. The internal consistency of APO-15 (total score and all subscales) had a good and acceptable (APO-15 items = .893, Positive relationship = .741, Achievement = .797, Meaning = .782, Engagement = .787). This indicates that the APO-15 items were able to measure the same constructs as a whole, and the items corresponding to each factor were able to adequately measure that factor as well.
Concurrent validity
The concurrent validity was confirmed by the correlations between APO-15, the sensation of happiness, SISR-B, RAS, and GHQ-12. APO-15 showed a positive correlation with participant’s happiness for each factor score (r = .128 to .317, p < .01). APO-15 showed a positive correlation with SISR-B and RAS for each factor score (r = .256 to .660, p < .01). Moreover, APO-15 showed a negative correlation with the 2-factor score of GHQ-12 (r = −.206 to −.476, p < .01). In other words, APO-15 was shown to be moderately associated with recovery issues, although it had a low association with client well-being.
Item response
Table 4 and Figure 1 show the results of item slope parameters (α) and item difficulty parameters (β). Overall, items on APO-15 demonstrated satisfactory item response, with item slopes (.602 to 1.300). The item difficulty parameter ranges from APO-15 demonstrated satisfactory item response in providing the appropriate discrimination and difficulty indices (–3.352 to 1.813). Therefore, the APO-15 is particularly good at finding participants who are unable to engage in positive occupation and suggests that the 4-point Likert scale of the APO-15 is functioning adequately.
Item response of APO-15.
α: item slope parameters; β: difficulty parameters; AIC: Akaike’s information criterion; BIC: Bayesian information criterion; Factor 1: positive relationship; Factor 2: achievement; Factor 3: meaning, Factor 4: engagement.
TIC of APO-15. θ = capability value, 0 = standard capacity. APO-15 is its focus on the level of participation in the occupation to promote well-being. 0 more of the clients having a positive value is determined to be able to participate in the occupation promoting wellbeing. On the other hand, less than or equal to 0 clients who have negative values are judged to be constraints on participating in well-being promoting occupations. APO-15 has a high measurement sensitivity to clients with participating constrained in occupation promoting well-being.
Cut-off point
Figure 2 shows the cut-off point of APO-15. The cut-off point became a 43-point sensitivity (.769) and demonstrated generally good results with 1– specificity (.441). The area under the ROC curve (.664) was lower than the standard value but demonstrated generally good results. Therefore, the APO-15 scored more than 43 points for the state of being able to engage in a positive occupation, indicating that it would be possible to distinguish between client states based on those points.
ROC curve of APO-15. The cut-off point became a 43-point sensitivity (.769) and demonstrated results with 1– specificity (.441). Area under the ROC curve (.664) was lower than standard value demonstrated result.
Discussion
Psychometric properties of APO-15
The APO-15 final version was validated as a self-administered scale for assessing people living in the community and the hospital, who are experiencing mental illness. To the best of our knowledge, this is the first study on the development of the assessment of occupational participation to promote well-being. In conclusion, APO-15 showed good results in terms of validity, reliability, and interpretability.
Firstly, in terms of validity, APO-15 had an overall good model fit. The structural validity of APO-15 was assessed by CFA; it indicated a good model fit (Table 3). This means that the APO-15's four-factor scale structure can be applied not only to clients living in the community who are experiencing mental illness, group homes, and inpatient facilities. For each item score of the polyserial correlation coefficient of APO-15, the reference value was confirmed as being higher (Table 2). The hypothesis testing was demonstrated good value for the convergent and discriminant validity of APO-15. On the other hand, the positive relationship obtained a rather small value in discriminant validity and may need to be reviewed in the future. However, the only factor that showed small values was the above-mentioned part, which overall does not seem to compromise the discriminant validity of APO-15. Also, the concurrent validity of APO-15 was tested by Happiness, SISR-B, RAS, and GHQ-12 which showed a weak correlation with Happiness, but a moderate correlation with SISR-B, RAS, and GHQ-12. An almost modest negative correlation among APO-15, GHQ-12 was observed. This finding was consistent with the purpose of measuring occupation participation to promote well-being with APO-15. The degree of positive occupational engagement as measured by the APO-15 might indicate that it is associated with recovery and mental health for individuals experiencing mental illness.
Secondly, the reliability of APO-15 was assessed by Cronbach's α coefficient, internal consistency was acceptable. Also, IRT was used to assess the individual item characteristics of APO-15. APO-15 had modest item slope parameters (.602 to 1.300). The difficulty parameter scores for APO-15 were very wide (−3.352 to 1.813) (Table 4 and Figure 1). These results show that the APO-15 strongly supports good item responses and that the item design using the 4-point Likert scale also works correctly. As mentioned above, we believe that the evidence is sufficient to demonstrate the validity and reliability of APO-15.
Third, the cut-off point of APO-15 became a 43-point sensitivity (.769) and 1– specificity (.441) with generally good results (Figure 2). Consequently, it can provide useful information for occupational therapists in the selection of clients to be supported through the cut-off value (43 points) of APO-15.
Clinical application of APO-15
The APO-15 can be used for people living in the community and inpatients experiencing mental illness within the practice of psychosocial occupational therapy. Specifically, the APO-15 can be used to assess a client's well-being status or changes in well-being based on engagement in positive occupation in the course of psychosocial occupational therapy. Also, the APO-15 can be used by occupational therapists to design occupational therapy interventions to promote the client’s well-being based on the results of the assessment, because all items on the scale consist of occupations that are likely to promote well-being. For example, clients who can engage well in a positive relationship occupation may be able to use the strengths of that occupation to further promote well-being. Also, clients with mental illness who could not find any meaning in their lives may be supported to seek meaning in their lives again by providing positive occupations related to other factors that promote well-being (positive relationships, achievement, and engagement). In conclusion, the APO-15 has the potential to contribute to clinical reasoning for evaluation and interventions in engaging in positive occupations. Therefore, APO-15 may be helpful in individuals with mental disabilities and may serve as a means of promoting the outcomes for psychosocial occupational therapy.
Limitations
This study design has several limitations. First, we had not investigated test-retest reliability. Secondly, the survey was conducted at only 20 sites, and the majority of the sample was individuals who had experienced schizophrenia. Third, all participants were Japanese only. Fourth, the APO-15 may need to go through continuous investigation for the cut-off point because the area under the ROC curve was slightly smaller than the standard value. Fifth, the cross-cultural validity of the APO-15 had not been tested yet. Despite these limitations, APO-15 as a measure for estimating occupation participation to promote the well-being of the client, is still a valid and reliable tool. In the future, there will be a need to translate the APO-15 to make it available to the English-speaking population.
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
