Validity of a smartphone application for self-monitoring psychiatric symptoms in patients with schizophrenia

Abstract

Objective

Despite increasing research on digital technologies for psychiatric disorders, studies specifically examining self-monitoring of symptoms with smartphone applications by patients with schizophrenia remain limited. This study aims to evaluate the validity and reliability of self-monitoring psychiatric symptoms using a smartphone application among patients with schizophrenia at Mindlink, a community-based early intervention center.

Methods

Fifty-three young patients with schizophrenia spectrum disorders participated. They rated their psychiatric symptoms across five domains—delusions, hallucinations, anxiety, depression, and perceived stress—using an 11-point Likert scale at baseline, 1 week, 8 weeks, and 16 weeks. Test–retest reliability was assessed using intraclass correlation coefficients (ICCs) between baseline and 1-week ratings. Concurrent validity was determined by correlating app-based ratings with established self-report and clinician-administered scales, including the Eppendorf Schizophrenia Inventory, Hamilton Program for Schizophrenia Voices Questionnaire, Beck Depression Inventory, Generalized Anxiety Disorder-7, and Perceived Stress Scale. The accuracy of the app's depression rating was assessed using receiver operating characteristic (ROC) analysis.

Results

ICCs for test–retest reliability were high across all symptom domains, ranging from 0.741 to 0.876 (p < 0.001). Significant correlations were observed between app-based ratings and formal assessments at all time points. ROC analysis for single-item self-ratings using the app yielded an area under the curve of 0.829 (p = 0.002), indicating good accuracy.

Conclusion

This study demonstrates that self-monitoring of key symptoms and stress using a smartphone application is valid and reliable for patients with schizophrenia. These findings support the app's potential to enhance symptom management and enable early detection of relapse in this population.

Introduction

Schizophrenia spectrum disorders are chronic mental health conditions characterized by symptoms such as delusions and hallucinations, often comorbid with depression and anxiety. ¹ Perceived stress is closely linked to relapse and symptom exacerbation, making effective management crucial. ² Given the recurrent nature of schizophrenia, where relapses often follow periods of stability, early detection of symptom exacerbation is vital to prevent full-blown relapses, which can severely impact long-term prognosis. ³ However, traditional methods of monitoring symptoms in patients with schizophrenia rely heavily on periodic clinical assessments, which may not capture the dynamic nature of symptom changes occurring between appointments.

In recent years, the application of digital technology in healthcare has expanded significantly, particularly in identifying and managing symptoms and illnesses. There is a growing interest in utilizing mobile technology, such as smartphone applications (apps), to facilitate continuous self-monitoring of psychiatric symptoms, offer psychoeducation, and provide therapeutic platforms and content.^4–8 These tools can provide a convenient and accessible way for patients to track their mental health in real-time, potentially enabling more timely interventions. In addition, several potential benefits of these apps include improved engagement with intervention services, greater convenience in assessing healthcare, and enhanced data collection.^6–9 However, although many apps are available on app marketplaces, only a few have been rigorously tested and validated in research settings specifically for patients with schizophrenia and other serious mental illnesses. ⁷ Some remain outdated on both the Apple App Store and Google Play App Store or are not clinically appropriate. ⁷ Furthermore, evidence on the reliability and validity of smartphone apps for self-monitoring in patients with schizophrenia is insufficient, particularly for tracking key symptoms.^10–12 This gap between research and publicly available apps highlights the need for continued study and development in this area.

Schizophrenia often manifests in the late teens and early twenties, with treatment and rehabilitation typically beginning in young adulthood. Case management should align with the young person's desires and needs, emphasizing active participation in their care. Mindlink, the first community-based early intervention center in Korea, developed a smartphone application (app) for case management of young clients in 2013. ¹³ The feasibility of this app (Heal Your Mind, HYM) has been previously reported. ¹⁴ Clients engaged in the Mindlink service were recommended to use the app as part of routine care to facilitate symptom monitoring and communication with their case manager. Our previous study demonstrated that participants found the app helpful for these purposes and reported high satisfaction levels. ¹⁴ A key module, “symptom record,” allows clients to rate subjective psychiatric symptoms across five domains: distressed thoughts (delusions), voices (hallucinations), anxiety, depression, and perceived stress, using an 11-point Likert scale. These ratings are stored graphically, representing changes in symptom severity over time. These data are shared with case managers to help monitor clients’ symptoms and signs of relapse (Figure 1).

Figure 1.

Examples of the symptom record screen using an 11-point Likert scale with corresponding description. Ratings are stored in both graphical and tabular forms. Image copyright owned by the author.

Despite increasing research on digital technologies in psychiatric disorders, studies specifically focusing on the self-use of applications by patients with schizophrenia remain limited. ¹² This study evaluates the validity and reliability of self-monitoring psychiatric symptoms using the smartphone application for patients with schizophrenia at Mindlink. We seek to determine the feasibility of incorporating such digital tools into routine clinical practice to improve symptom management, patient engagement, and early intervention in schizophrenia care.

Methods

Study procedure and subjects

This prospective observational study was conducted at Mindlink, a community mental health center for young people, between June 2017 and March 2022. A subset of eligible participants who had used the application during the recruitment period were included. Subjects were instructed to record their symptoms using the symptom record module at baseline, 1 week, 8 weeks, and 16 weeks at home. To measure convergent validity, a validated self-reported scale and/or an objective scale for the five symptom domains were administered three times every 8 weeks. Trained mental health clinicians, blinded to the client's app ratings, and conducted objective assessments. Self-ratings in the app during the first week were used to measure test–retest reliability. Participants were not actively prompted by the study team to respond. However, if ratings were missed, the app and/or study team provided automated reminders for symptom ratings.

Inclusion criteria included individuals aged 18–40 years who used the Mindlink smartphone app, met the criteria for schizophrenia spectrum and other psychotic disorders according to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), ¹⁵ and were able to understand and respond to a survey. Exclusion criteria included intellectual impairment with an IQ of 70 or lower, severe impairment in reality testing and behavioral symptoms, and severe physical illness. This study was approved by the Institutional Review Board (IRB) of Chonnam National University Hospital (IRB No.: CNUH-2017-070). Written informed consent was obtained from all participants prior to study initiation.

Measures

Five formal self-rated scales were administered corresponding to the five symptom domains of the app: the “idea of reference” subscale from the Eppendorf Schizophrenia Inventory (seven items)^16,17 for delusion, the Hamilton Program for Schizophrenia Voices Questionnaire (nine items)^18,19 for hallucinations, the Beck Depression Inventory (BDI) (21 items)^20,21 for depression, the Generalized Anxiety Disorder 7-item scale^22,23 for anxiety, and the Perceived Stress Scale (10 items)^24,25 for stress.

Objective measures included the Clinician-Rated Dimensions of Psychosis Symptom Severity (CRDPSS) scale,^12,26 which rated eight dimensions of schizophrenia on a 5-point scale. The CRDPSS subscales for delusions, hallucinations, and depression were matched with the respective self-ratings via the app. Additionally, the anxiety item (G2) from the Positive and Negative Syndrome Scale was administered to investigate its correlation with anxiety self-rating via the app.^27,28

Sociodemographic information (age, sex) and clinical information, including duration of illness, duration of untreated psychosis (DUP), and histories of psychiatric hospitalization and suicide attempts, were collected.

Statistical analysis

Test–retest reliability was calculated using the intraclass correlation coefficient (ICC) between baseline and 1-week app ratings. Spearman correlation coefficients were calculated between app ratings and corresponding formal scales at each time point to assess concurrent validity. For test–retest reliability, studies have demonstrated that with a sample of at least 50 participants, the confidence intervals for ICC estimates are stable, providing robust reliability assessments. ²⁹ To detect a moderate correlation (ρ ≈ 0.4–0.5) with a statistical power of 0.80 and an alpha level of 0.05, statistical calculations suggest a sample size of approximately 47–59 participants (calculated via G*Power).

Receiver operating characteristic (ROC) analyses were conducted to explore the validity of single-item self-ratings via the app, specifically for diagnosing depression, based on the BDI scores (≥16) indicative of clinically significant depression. ¹⁷ The area under the curve (AUC) was used to assess the accuracy of the app-based depression rating. Missing data were excluded from the analysis. However, if observed data were available at subsequent follow-up visits, they were included in the analysis. Data were analyzed using SPSS for Windows (ver. 23.0; IBM Corp., Armonk, NY, USA), with p-values <0.05 considered statistically significant.

Results

We enrolled 53 patients (26 women, 49.1%) with a schizophrenia spectrum disorder. The mean age and years of education were 23.9 ± 4.1 and 13.4 ± 2.0 years, respectively. The median (interquartile range) duration of illness and DUP were 1.4 (0.5–4.6) years and 2.2 (1.0–11.9) months, respectively. The most frequent diagnosis was schizophrenia (n = 47, 88.7%). Follow-up assessments were conducted in 51 patients (96.2%) at week 1, 48 (90.6%) at week 8, and 46 (86.8%) at week 16. Additionally, 39 patients (73.6%) had a history of psychiatric hospitalization, and 18 patients (34.0%) had a history of suicide attempts. Twelve patients (23.5%) had clinically significant depression based on BDI scores.

ICC values for each domain between baseline and 1-week assessments ranged from 0.741 to 0.876, all statistically significant (p < 0.001) (Table 1). Correlations between self-measures via the app and formal assessments at three time points were statistically significant. Correlation coefficients were highest for anxiety (0.646–0.770), followed by hallucinations (0.529–0.670), depression (0.479–0.610), perceived stress (0.426–0.469), and distressed thoughts (0.311–0.508). Correlation coefficients with objective measures were relatively slightly lower than those with subjective scales, while they were all statistically significant (Table 2).

Table 1.

The intraclass correlation coefficients between baseline and 1-week app ratings.

	Thoughts	Hallucination	Depression	Anxiety	Stress
ICC	0.741	0.876	0.740	0.802	0.823
95% CI	0.547–0.852	0.782–0.929	0.544–0.851	0.653–0.887	0.690–0.899
p	<0.001	<0.001	<0.001	<0.001	<0.001

ICC: intraclass correlation coefficients; CI: confidence interval.

Table 2.

Spearman correlation coefficients between app ratings and corresponding formal scales at each time point.

Formal scales: app ratings	Baseline	8 weeks	16 weeks
ESI: distressed thoughts	0.418**	0.508***	0.311*
CRDPSS, delusion: distressed thoughts	0.394**	0.422**	0.329*
HPSVQ: hallucination	0.529***	0.670***	0.629***
CRDPSS, hallucination: hallucination	0.593***	0.593***	0.588***
Beck Depression Inventory: depression	0.546***	0.479**	0.610***
CRDPSS, depression: depression	0.423**	0.361*	0.654***
GAD-7: anxiety	0.770***	0.679***	0.646***
PANSS G2: anxiety	0.484***	0.383**	0.489***
Perceived Stress Scale: stress	0.426**	0.437**	0.469**

ESI: Eppendorf Schizophrenia Inventory; CRDPSS: Clinician-Rated Dimensions of Psychosis Symptom Severity; HPSVQ: Hamilton Program for Schizophrenia Voices Questionnaire; GAD-7: Generalized Anxiety Disorder 7-item scale; PANSS: Positive and Negative Syndrome Scale.

*p < 0.05, **p < 0.01. ***p < 0.001.

ROC analysis for app-based depression ratings yielded an AUC of 0.829 (p = 0.002) with an optimal cut-off score of 5. This threshold provided optimal sensitivity (69.2%) and specificity (85.0%), along with favorable negative (89.5%) and positive (60.0%) predictive values (Figure 2). No distress related to using the app or serious adverse events were recorded during the 16-week study period.

Figure 2.

Receiver operating characteristic (ROC) curve analysis of the app-based depression rating. Area under the curve (AUC) = 0.829 (p = 0.002).

Discussion

Validity and reliability

In patients with schizophrenia, the clinical use of digital devices remains underutilized despite their potential importance. This is mainly due to challenges such as difficulties in using these devices and concerns about reliability, which stem from the symptoms and functional impairments associated with the disorder. This study evaluated the validity and utility of a smartphone app developed for self-monitoring psychiatric symptoms in young patients with schizophrenia spectrum disorders. Follow-up and adherence to rating protocols among our participants were high. Given the strong correlation between app-based ratings and established measures of psychiatric symptoms, the app demonstrated high concurrent validity. The findings provide encouraging evidence that self-monitoring via a smartphone app can be reliable and valid for tracking perceived stress and key symptoms.

The high ICCs between baseline and 1-week follow-up ratings demonstrate strong test–retest reliability across all symptom domains, indicating that patients can consistently self-report their symptoms using the app. This is particularly crucial in schizophrenia management, where monitoring symptom fluctuations is vital for preventing relapse. ³⁰

Clinical implications and early detection of relapse

Regular symptom monitoring enables patients and clinicians to identify subtle changes that may signal the onset of a relapse, such as increasing anxiety, intensifying hallucinations, or emerging delusional thoughts.^31,32 This study suggests the app's potential to detect early signs of symptom recurrence or relapse accurately. Early detection of these signs can prompt timely interventions, such as medication adjustments or intensified case management, which are crucial for preventing full-blown relapses and reducing the duration and severity of exacerbations.^33,34

Integrating artificial intelligence (AI) techniques could further enhance this capability by analyzing patterns in symptom data over time and identifying subtle changes that may not be immediately apparent to patients or clinicians. ³⁵ AI algorithms can be trained to recognize patterns linked to impending relapses, providing timely alerts to patients and healthcare providers.^36–38 This proactive approach could facilitate even earlier interventions. Furthermore, incorporating more sophisticated features like real-time data analytics and personalized feedback could enhance the app's utility and impact. Our study establishes a foundational basis for further development and research into integrating AI techniques with self-monitoring via smartphone apps in daily life.

Single-item self-rating

The study also highlights the clinical utility of a single-item self-rating via the smartphone app. The ROC curve analysis demonstrated a statistically significant AUC, indicating good overall accuracy in predicting BDI-based diagnoses of depression. This single-item tool could be an effective initial screening method. ³⁹ The convenience and accessibility of using a smartphone app for self-assessment align with the growing trend of digital health solutions. By minimizing the burden of repeated assessments and enhancing the continuity of care, self-rating with a single item via an app presents a solution that can be easily integrated into routine clinical practice. ⁴⁰

Correlations between the app-based ratings and the objective measures were slightly lower than those with the formal subjective scales. This difference may reflect the discrepancies between self-reported symptoms and clinician-assessed symptom severity. Subjective distress is also an essential area in the management of schizophrenia. It underscores the importance of integrating self-reports with clinician assessments to understand a patient's condition comprehensively. ⁴¹

Limitations

Despite the promising results, this study has several limitations that should be acknowledged. First, the sample size, while adequate for the study's purposes, was relatively small and limited to young adults with schizophrenia spectrum disorders. This restricts the generalizability of the findings to broader populations, including older adults and those with more chronic forms of schizophrenia. Future research should replicate these findings in more extensive and diverse populations, including those with different psychiatric diagnoses. Second, the 16-week follow-up period may not be sufficient to investigate long-term changes or exacerbations of psychiatric symptoms in patients with schizophrenia spectrum disorders. Future research should incorporate longer follow-up durations to better assess the advantages of symptom monitoring during the clinical course of schizophrenia. Third, adherence to treatment was not directly monitored in this study. Future studies should incorporate mechanisms to monitor treatment adherence alongside symptom monitoring. Finally, digital interventions that target or involve the family members of patients with schizophrenia should be explored to improve clinical outcomes.^42,43

Conclusions

This study provides compelling evidence that a smartphone app can be a valid and reliable tool for self-monitoring psychiatric symptoms in young patients with schizophrenia spectrum disorders. The ability to track key symptoms and detect early signs of relapse using this app module could play a critical role in preventing more severe exacerbations, ultimately improving the long-term prognosis for individuals with schizophrenia.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076251317556 - Supplemental material for Validity of a smartphone application for self-monitoring psychiatric symptoms in patients with schizophrenia

Supplemental material, sj-docx-1-dhj-10.1177_20552076251317556 for Validity of a smartphone application for self-monitoring psychiatric symptoms in patients with schizophrenia by Sung-Wan Kim, Jae-Kyeong Kim, Min Jhon, Ju-Wan Kim, Seunghyong Ryu, Ju-Yeon Lee and Jae-Min Kim in DIGITAL HEALTH