Sage Journals: Discover world-class research

Abstract

Background

Suicide rates have significantly increased in South Korea, yet many individuals lack adequate support. Barriers such as reluctance to seek mental health help and fear of social stigma contribute to this gap. A mobile app focused on suicide risk awareness could provide accessible support, though none are currently available for public use in South Korea. This study conducted a usability test on a newly developed suicide risk awareness app using a mixed methods approach.

Methods

Thirty-eight students from a large university in South Korea participated in the study, with 19 in a high-risk suicide group and 19 in a nonrisk suicide group. After using the app for 2 weeks, all participants completed an online usability survey, and 19 students took part in individual interviews.

Results

Independent samples t-tests showed that participants, regardless of risk group, rated the app positively for ease of use, accessibility, design, perceived learning, and satisfaction. Regression analysis identified perceived learning as the strongest predictor of satisfaction, followed by ease of use. The qualitative analysis highlighted areas for improvement, including providing direct and guided feedback on suicide risk.

Conclusion

The study demonstrated the potential of a mobile app to enhance suicide risk awareness among young adults in South Korea. Moreover, user engagement with the app can be improved by ensuring confidentiality and fostering perceived learning.

Keywords

Mobile health app suicide risk awareness app self-awareness usability user experience

Introduction

Suicide has been identified as the primary cause of death in young adults in South Korea since 2007.¹ In fact, the rate of suicide attempts among South Koreans in their 20s has increased every year since 2016.² By 2020, the number of emergency room visits resulting from suicide attempts by those in this age group was 10,007 (28.7%), the highest number among all age groups. South Korea's suicide rate in 2020 was 25.7 deaths per 100,000, the highest among the Organization for Economic Cooperation and Development countries.³ For young adults in their 20s, the rate was 23.5 per 100,000 people in 2021, contributing to the rising suicide rate in South Korea.⁴

Despite the high suicide rate and attempts among those in their 20s, many individuals in this age group do not seek professional help for suicidal thoughts or suicide concerns.⁵ This reluctance stems from various factors, including the strong stigma surrounding suicide issues in Korean society.⁶ For example, many young Koreans fear the consequences of disclosing suicidal ideation, such as involuntary hospitalization or legal repercussions.⁷ Additionally, there is a widespread perception that suicide-related struggles are personal weaknesses rather than treatable conditions, leading to further isolation of those at risk.^6,8 Given these complex barriers to help-seeking and the critical need for improved suicide risk literacy, accessible tools are essential in helping individuals recognize their suicide risk status and seek assistance.^9,10 However, at the time of this writing, no mobile suicide risk awareness apps are available to the general public in South Korea.

A mobile app has considerable potential to help individuals develop awareness of their suicide risk status.^11–13 Because using a mobile app is a common part of everyday life (e.g. on the subway or in a restaurant), suicide risk awareness apps allow young adults to discreetly access support without drawing attention. This privacy helps avoid the stigma and shame often associated with seeking help for suicidal thoughts, even when used in public settings. Thus, mobile apps allow young adults to use mental health apps anywhere and anytime as long as they carry an internet-connected smartphone with them.

Furthermore, anonymity is guaranteed as one accesses mental health assessment tools, and contact can be made using a mobile device without going to a psychiatric office or mental health center in person.¹⁴ According to Torous et al.,¹⁵ 67% of the patients with anxiety and depression expressed interest in and willingness to try mobile apps designed to monitor their mental health. These findings suggest that smartphones and mobile apps provide an immediate opportunity for users to develop suicide risk awareness at the right time and place. In addition, Chung et al.¹⁶ compared the accuracy of depression risk among young adults using two methods in Korea—mobile app-based depression screening and standard pencil-and-paper test tools—and found that a mobile app allowed both users and practitioners to detect depressive symptoms and their severity more quickly and accurately compared to pencil-and-paper tests.

Mobile apps also allow users to track changes in their self-views and negative thoughts over time. Individuals with mental health issues often hesitate to recognize or confront their condition, opting instead to alleviate psychological discomfort through alternative coping strategies. However, monitoring changes in self-views and mental state is crucial for better understanding one's emotions and thoughts, thus facilitating positive change.¹⁷ This is especially relevant in cases of mild suicidal ideation, where quickly identifying negative thought patterns or distorted self-perceptions can aid in recognizing early warning signs of suicide risk.¹⁸ Consequently, timely intervention and therapeutic support can be sought before the situation deteriorates, further underscoring the importance of early detection and intervention.

Despite the potential benefits, mental health apps have drawn criticism for the lack of attention to usability.¹⁹ Usability, which refers to the extent to which a specific user group achieves a particular task efficiently, effectively, and satisfactorily,^20,21 is critical for helping users engage in mobile apps for mental health. Alqahtani and Orji²² have found that because of the lack of consideration of usability in mental health apps, many apps have high attrition rates and are short-lived, which is likely to prevent users from experiencing the benefits of their use. User experience with a mobile app for suicide risk awareness is critical because high engagement helps users enhance their awareness of their suicide risk status, essentially developing a perception of their emotional and psychological health.^19,23,24 In this study, we aimed to examine the usability and user experience of a mobile app for suicide risk awareness for its further development.

Importance of awareness in the early stage of suicidal ideation

According to mental health researchers,²⁵ suicidal behavior usually occurs in three stages: in suicidal ideation, one thinks about suicide; in the stage of suicide planning, one plans the possible method and timing of suicide; and in the suicide attempt, one performs actions that are fatal to one's life to reach death. Suicidal ideation is an important predictor of suicide attempts and eventual suicide²⁶; thus, detecting it as early as possible and intervening in the development of subsequent suicide stages are necessary. Suicide prevention programs that self-assess risk through suicide screening have the potential to significantly reduce suicide attempts among young adults.²⁷ Although further studies are required to determine which aspects of the program are the most effective, self-assessment of suicidal ideation may have a short-term positive effect on suicide attempt rates; moreover, self-assessing the level of risk in the early stages of suicidal ideation, prior to an actual suicide attempt, can increase the possibility of help-seeking behaviors.²⁸ Knowledge and understanding of one's mental health are components of mental health literacy; furthermore, individual attitudes and decisions to seek professional help for mental health problems are influenced by mental health literacy.²⁹ Consequently, self-awareness of suicide risks in the early stage of suicidal ideation is particularly important to prevent subsequent suicidal actions.^30,31

Importance of suicide apps related to awareness

A critical function of mental health apps is to provide users with an opportunity to develop awareness of the risks associated with the symptoms from which they suffer.³² Suicide awareness is the first step toward seeking help. Awareness of mental health, emotions, behaviors, and self-limitations is a prerequisite for professional assistance,³³ and mobile apps have the clear potential to help develop that awareness. For example, the Application for Suicide Risk and Depression Assessment (ASRaDA), a mobile mental health app, was developed to rapidly screen depression and suicide risk among individuals living in Fiji and Pacific provincial counties.³⁴ The ASRaDA has been shown to be useful for identifying individuals at risk of depression and suicide, directing treatment, and managing health early. Moreover, Suicide Risk Idea Identification for Teenager, a mobile application developed by Wahyuningrum et al.,³⁵ was helpful to identify suicide risk in adolescents. Adolescents can use this tool to detect risk factors for suicidal ideation independently, and professionals can use it to develop new management strategies for the prevention of risk for suicidal ideation.

Research purpose

The purpose of this study was to conduct a usability study with a mobile app for suicide risk awareness in South Korea. To support adults in their 20s (and all age groups in the later stages of the project), we (a) developed a mobile app equipped with a suicide risk awareness tool and information about available human services and resources and (b) used mixed methods for the study. More specifically, the usability test was conducted quantitatively by using a survey, and individual interviews with 19 volunteers were conducted to learn about the user experience in detail.

Methods

Participants

Participants were recruited through a listserv associated with a large university Trauma Stress Center in South Korea, which includes individuals actively using the center's services as well as those seeking information and updates about available resources. The email explained the study's purpose, activities involved, research ethics, and compensation details, specifying $30 for mobile app use and online survey completion and $20 for a one-time interview. Students interested in participating were asked to complete a brief online survey, providing their contact information, names, and responses to two screening questions regarding their suicide risk status (e.g. “Have you recently thought about committing suicide?” and “Have you recently attempted suicide?”). Based on the initial screening, students were categorized into a high-risk group (n = 23) and a nonrisk group (n = 20). During the course of the research, five students (four from the high-risk group and one from the nonrisk group) withdrew from the study, resulting in 19 students remaining in each group.

A power analysis was conducted using G*Power software version 3.1.9.7³⁶ to determine the necessary sample size, ensuring the study had sufficient power to detect group differences with a reasonable level of confidence. For an independent t-test with a medium effect size (Cohen's d = 0.5), an alpha level of 0.05, and a desired power of 0.80, the analysis indicated that 51 participants per group were required for a one-tailed test. However, the final sample size in this study was smaller than the recommended 51 participants per group. Given the constraints of the project timeline and available resources, we proceeded with the research despite the smaller sample size, acknowledging this as a limitation.

Table 1 presents detailed demographic information. A scale for suicidal ideation, originally developed by Beck et al.,³⁷ and validated in the Korean context,³⁸ was administered to all participants to determine whether they were assigned to the correct group: high risk or nonrisk for suicide. Cronbach's alpha for the current participants was 0.88. A significant difference in suicidal ideation was observed, t (25.05) = 6.36, p < 0.001, Cohen's d = 2.07, between the groups at high risk (M = 12.37, SD = 6.73) and nonrisk for suicide (M = 1.58, SD = 3.04). The result indicates that the allocation of participants into the two groups was appropriate. Additionally, the large effect size (Cohen's d = 2.07) suggests that the analysis was sufficiently powered to detect a significant difference.

Table 1.

Demographic information.

Demographic variable	High-risk group (n = 19)	Nonrisk group (n = 19)
Gender
Female	14	19
Male	5	0
Age
<20	4	0
20–29 years	15	19
Marital status
Married	0	0
Single	19	19
Education
Undergraduate	17	8
Graduate	2	11

Suicide risk awareness app

A suicide risk awareness app was developed to enable users to self-assess their suicide risk and monitor changes over time. The Suicide Screening Questionnaire‒Self-Rating (SSQ‒SR), recently developed and validated by a Korean research team,³⁹ was programed into a mobile application as a self-assessment tool. The SSQ–SR may better capture suicidal ideation in young adults and Koreans than other screening questionnaires developed with Western cultures because items are developed according to the Korean context. Cronbach's alpha for the participants was 0.92.

Users downloaded and installed the app on their mobile phones, then signed up and received a randomized ID number to protect their privacy and ensure confidentiality. They could then self-assess their risk for suicidal ideation. Additionally, the app provided contact information for mental health support centers, including phone numbers, locations, and email addresses. See screenshots in Figure 1.

Figure 1.

Screenshots of the mobile app for suicide risk awareness.

This study was conducted in collaboration with psychiatric medical professionals at a general hospital in Seoul, South Korea. In addition to providing helpline numbers, high-risk individuals identified in the study were actively connected to the hospital for appropriate diagnosis and treatment, ensuring access to necessary medical support.

Measures

Two instruments were used in this study. First, a usability survey was adopted from existing studies,^40,41 then slightly modified after consultations involving the researchers and counselors. The final usability survey consisted of 16 questions, representing five dimensions of user experience with mobile apps: (a) ease of use, (b) accessibility, (c) design, (d) perceived learning, and (e) satisfaction. We used a 5-point Likert scale, ranging from 1 (strongly disagree) to and 5 (strongly agree). Cronbach's alpha for the participants was 0.90. Second, structured interview questions, developed and reviewed by the research team, were used to uncover further information about user experience with the mobile app (see Appendix 1).

Procedures

This study was approved by the Institutional Review Board on the campus where the study was conducted (IRB No. SKKU-2019-12-001-006). All participants engaged in an orientation on how to download and use the developed mobile app for suicide risk awareness. They were given two weeks to use the app. After this period, all the participants were invited to complete an online survey, which they accessed by agreeing to an informed digital consent form. In addition, emails inviting participants to individual interviews were sent to all the online survey participants. Nineteen students (10 from the high-risk and 9 from the nonrisk group) responded and participated in individual interviews lasting 25‒30 min to explore user experiences in detail. Before each interview, participants were informed of the study's purpose, ethics, and procedures. Once they signed the informed interview consent form, the interview began and was audio recorded.

Results

Usability with the Mobile app for suicide risk awareness

Overall, the participants evaluated the suicide risk awareness app highly and positively, using a 5-point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree; see Table 2). They provided positive ratings on each usability dimension: ease of use (high risk: M = 4.36, SD = 0.77; nonrisk: M = 4.29, SD = 0.89), accessibility (high risk: M = 4.42, SD = 0.61; nonrisk: M = 4.56, SD = 0.80), design (high risk: M = 4.39, SD = 0.63; nonrisk: M = 4.65, SD = 0.53), perceived learning (high risk: M = 3.97, SD = 0.77; nonrisk: M = 3.55, SD = 0.97), and satisfaction (high risk: M = 4.09, SD = 0.67; nonrisk: M = 4.03, SD = 0.98).

Table 2.

Usability of the mobile app for suicide risk awareness.

Dimension of evaluation		High risk (n = 19)		Nonrisk (n = 19)		t	p
Dimension of evaluation		M	SD	M	SD	t	p
Ease of use		4.36	0.77	4.29	0.89	0.25	0.79
1.	The app was easy to use.	4.37	1.01	4.47	0.84	−.35	0.72
2.	Figuring out how to use the app did not take long.	4.26	0.93	4.26	0.99	.00	1.00
3.	The layout made using the app easy.	4.47	0.69	4.16	1.17	1.01	0.32
Accessibility		4.42	0.61	4.56	0.80	−0.60	0.54
4.	No technical problems arose in accessing the app.	4.32	0.88	4.53	0.96	−.70	0.48
5.	Installing the app on my cell phone was easy.	4.42	0.77	4.47	1.07	−.17	0.86
6.	No time delay occurred when using the app.	4.53	0.61	4.68	0.67	−.75	0.45
Design		4.39	0.63	4.65	0.53	−1.38	0.17
7.	The menu was appropriate in helping me understand the overall use of the app.	4.32	0.94	4.58	0.69	−.97	0.33
8.	The text was readable.	4.47	0.84	4.79	0.53	−1.38	0.17
9.	The icons matched well with the content.	4.37	0.83	4.63	0.59	−.1.12	0.27
10.	The colors used made reading the text comfortable.	4.42	0.83	4.63	0.68	−.84	0.40
Learning		3.97	0.77	3.55	0.97	1.48	0.14
11.	Using the app, I learned information about help centers, where I can receive immediate help.	3.95	1.02	3.05	1.39	2.25	0.03*
12.	Using the app, I understood the status of my mind better.	4.00	0.81	4.05	1.17	−.16	0.87
Satisfaction		4.09	0.67	4.03	0.98	0.19	0.84
13.	I would recommend this app to others who might need it.	3.84	0.83	4.00	1.10	−.49	0.62
14.	Overall, I am satisfied with the app.	4.11	0.80	4.11	0.93	0.00	1.00
15.	The app was helpful in evaluating status for self-harm.	4.16	0.76	4.11	1.10	.17	0.86
16.	The app is useful.	4.26	0.73	3.95	1.12	1.02	0.31

In addition, we found no significant differences between the two groups in each usability dimension: ease of use, t (36) = 0.25, p > .05; accessibility, t (36) = -0.60, p > .05; design, t (36) = -1.38, p > .05; understanding, t (36) = 1.48, p > .05; and satisfaction, t (36) = 0.19, p < .05, indicating that both groups had equal values for each usability dimension (see Table 2). The only significant difference between the groups at high risk (M = 3.95, SD = 1.02) and not at risk for suicide groups (M = 3.05, SD = 1.39) occurred in Item 11 of the usability questions, t (36) = 2.25, p < 0.05. The participants in the high-risk suicide group valued information about help centers more than those in the group not at risk for suicide. Perhaps participants in the high-risk suicide group, who developed self-awareness about their mental health through the mobile app, may have been more focused on information about help centers, where they could receive immediate assistance, compared to those in the nonrisk group. Readers should interpret the results with caution due to the small sample size in each group. A power analysis was conducted using G*Power software version 3.1.9.7³⁶ to determine the necessary sample size, ensuring the study had sufficient power to address group differences in usability dimensions with a desired level of confidence. For the independent t-test with a medium effect size (Cohen's d = 0.5), an alpha level of 0.05, and a desired power of 0.80, the analysis indicated that 51 participants per group were required for a one-tailed test.

Furthermore, a multiple regression analysis was conducted to examine the best linear combination of usability dimensions—ease of use, accessibility, design, and perceived learning—to predict participant satisfaction with the suicide risk awareness app. Satisfaction is critical in predicting user acceptance of a mobile app⁴² and user engagement with apps. The combination of usability dimensions significantly predicted user satisfaction with the app, F (4, 33) = 14.52, p < .001, with the two usability dimensions significantly contributing to the prediction. The beta weights suggest that perceived learning (beta weight = 0.38, p < 0.001) has contributed the most to predicting user satisfaction with the app, followed by ease of use (beta weight = 0.32, p < 0.05). The adjusted R² value was 0.594, indicating that 59.4% of the variance in user satisfaction with the app was explained by the model.

User feedback about suicide risk awareness app

To further examine the participants’ experiences with the mobile app, individual interviews were conducted with 19 volunteers, each lasting 25‒30 min. After each interview was transcribed, personal information was removed. The two researchers individually coded 40% of the interview data using Excel. After the initial coding, they reviewed each other's coding results, gave feedback to each other, discussed, and created a coding book with the agreement. Then, each researcher coded 60% of the interviews (40% of the initial interviews and an additional 20% of the interviews) individually with the agreed coding book. Then, they reconvened, checked the agreement level, which was 85%, discussed, and made minor changes in the coding book. Last, they individually coded the entire interview data with 92% agreement. The remaining discrepancies were discussed with one another until a 100% agreement was reached. Since no new codes emerged, the research team concluded that data saturation had been reached. Consequently, the research team decided not to recruit additional interview participants. The six themes and 14 subthemes that emerged appear in Table 3. Explanations of each theme and subtheme appear below.

Table 3.

Experiences with the mobile app for suicide risk self-awareness app.

No.	Themes	subthemes	Definitions	No of users
1	Overall experiences	Self-awareness	Individual's ability to recognize and understand their thoughts, emotions, and behaviors associated with suicidal ideation or risk	13
		Accessibility	Opportunity to access and utilize the mobile app for suicide risk awareness	8
		Trustworthiness	Users’ trust in the self-assessment questionnaire as a valid and reliable source	3
2	Readability	Wording	Appropriateness of words and phrases in the self-assessment questionnaire	5
2	Readability	Legibility	Clarity and readability of the information presented on the mobile app	5
3	App interface	Color scheme	Overall perceived level of comfort with the mobile app due to color choice	4
		Navigation	Ease of exploring and finding menu and functions	10
		Easy-to-answer questions	Simplicity of answering the questions in the self-assessment questionnaire presented on the app	5
4	App operations	Technical glitch	Short-lived fault or malfunctions while using the app	2
4	App operations	Confidentiality	Concerns about privacy when using the app	4
5	Functional requirements	Clear direction	User guidance and explanation for the app's functions, menus, or self-assessment questionnaire	8
5	Functional requirements	Feedback	Interpretations of results and suggested actions or resources for further help.	14
6	Customized questionnaire	Questionnaire length	Appropriateness of the number of self-assessment questions	5
6	Customized questionnaire	Need for comprehensive questionnaire	Desire for a self-assessment questionnaire that includes both suicide risk and other emotional states	2

Overall experience

Overall, participants had positive experiences with the mobile app for suicide risk awareness. They reported three positive experiences with the mobile app: self-awareness, accessibility, and trustworthiness.

Self-awareness

The mobile app was perceived as an aid to develop their own self-awareness of suicidal risk in two ways. First, it facilitated the users’ thoughts about their risk for suicide, which they may not have consciously realized in their daily lives. One participant stated:

Usually, I don’t spend much time reflecting on myself in my daily life. My focus tends to be on the immediate situations at hand, and I don’t delve deeply into introspection; however, being prompted with such questions in the app has allowed me to develop my awareness of my suicide risk status. It has allowed me to evaluate my thoughts and behaviors in the situations as each question is stated.

Second, using the app, users can better elaborate on their suicide risk status and communicate with others to seek help. Another participant said:

By answering the questions in the app, I’ve been able to define my state more concretely. There are times when I need to accurately explain my state to others. I feel that I would be able to communicate my needs more effectively when I find myself in a situation where I need assistance and have to ask for help.

Accessibility

Users thought that the app provided high accessibility to assess their risk for suicide. Specifically, the app allowed them to assess their psychological state with less paperwork. One participant said:

Typically, to undergo an assessment, I have to make an appointment, go to the counseling office, and receive in-person help from a professional; however, using the app, I can complete the self-assessment questionnaire at my own pace and get the results immediately.

Also, the app was beneficial for many young adults who carry smartphones because they can download and use it for free as long as they are connected to the internet. This also enhances accessibility. Another participant commented:

I believe that the app benefits everyone who carries a smartphone because they can download it anywhere as long as they have internet access. This is a very cost-effective way because obtaining an assessment at a counseling center could pose financial burdens for many young adults like me.

Trustworthiness

The participants perceived the app as a reliable self-assessment tool because they were unsure about the accuracy and reliability of the other available mental health apps resulting from unknown sources of the apps. One participant said:

When searching the App store or Google Play, you often come across depression tests, suicide risk assessments, and similar tools; however, there is uncertainty about the accuracy and reliability of these. But I believe that if this app is further developed with experts and is available to the public, it would help many people in need.

Other users appreciated that the source of information was one they could trust, and they did not have to look at or read commercial advertisements on the app. Another participant said:

When searching on the internet, especially on platforms like Naver (a web browser in S. Korea), you often come across advertisements for hospitals urging you to visit them. Blogs, on the other hand, tend to be more like personal diaries rather than reliable sources of information. In reality, it can be challenging to find the necessary and reliable information in such places.

Readability

Users suggested changes in wording and expressions as well as changes in font size to enhance the readability of the information.

Wording

Some users were uncomfortable because the words used in the self-assessment questionnaire directly mentioned the terms “suicide” or “suicide ideation,” which are prohibited in daily life under Korean social norms. One participant stated, “I was a bit surprised because some questions directly mentioned suicide or suicide ideation, which was uncommon for me. I feel the words should be more neutralized.”

Incorporating this consideration could compromise the integrity of the questionnaire. One way to mitigate stigma while maintaining the scale's validity is to use alternative phrasing, such as “thoughts of self-harm” or “ending one's life,” when referring to socially sensitive terms like “suicide” or “suicidal ideation.” Another approach is to emphasize that all responses are strictly confidential, securely stored, and collected anonymously to reassure participants and reduce discomfort when using the suicide risk awareness mobile app

Legibility

Some users requested more space between words to enhance legibility. One participant commented: “I feel uncomfortable reading the text because of very little space between words and between lines.”

Other users were concerned that font size might cause a readability issue when elderly people use this app. Another participant said: “It seems like the font size might be a bit small, especially for older individuals.”

App interface

For the app interface, three subthemes emerged: color scheme, navigation, and easy-to-answer questions. Notably, two very different opinions emerged among users regarding the color scheme and navigation; however, users were satisfied with the questionnaire page because the interface on the self-assessment questionnaire was easy to answer.

Color scheme

In terms of the color scheme, two opinions emerged. On one hand, some participants wanted to have warmer color than blue, which can make users feel cold. One participant stated, “Changing the color from blue to warmer will be good because the app's content is sensitive.”

On the other hand, some participants viewed the color scheme as comfortable to read because they didn’t experience eye fatigue. Another participant mentioned:

When I checked the first screen in the app, I saw that the color was soothing to the eyes, making it comfortable for anyone to use without feeling eye fatigue. The color choice in the current app would make everyone feel comfortable using it.

Navigation

Users had two opinions on the navigation. Some suggested the improvement of the navigation because the instructions were unclear to them. One user said, “The process of entering the questionnaire page was somewhat confusing. I wasn’t sure which questionnaire I needed to take first.”

Other users indicated that they were comfortable using the app because the app structure and design were straightforward and intuitive. One said:

The navigation system was user-friendly. It clearly indicated what to click for certain purposes. It was easy to understand. In addition, the quick access to the questionnaire through direct clicks and a straightforward consent process make me comfortable navigating the app.

Easy-to-answer questions

The participants mentioned that the questionnaire screen was efficiently designed. The app users easily identified how to respond to each question item. One user said, “Understanding how to check each question item and submitting the checked answers were easy.”

Another said, “No discomfort occurred in reading and checking each item.”

App operation

A technical glitch and confidentiality issue emerged in the app operation.

Technical glitch

Two users who reported a technical glitch experienced buffering when they moved to subsequent pages. One noted, “The screen stuttered a bit when transitioning to the next page.”

Confidentiality

Some app users were concerned about providing their personal information on the app. One user mentioned, “Because the societal stigma about mental health conditions exists, individuals like me may hesitate to provide their private information on the app.”

Another participant said, “Many people want to keep their identities confidential, so anonymity should be ensured.”

Functional requirements

Users wanted to see clearer directions on how to access the suicide risk awareness questionnaire and see the recommended follow-up actions beyond the self-assessment results.

Clear direction

Some users mentioned the need for step-by-step instruction. One participant stated, “The step-by-step instructions for certain actions will be very helpful. For example, I wanted to check my suicide risk awareness, but it was somewhat confusing to access the questionnaire.”

Others emphasized the clear explanation on each page, especially the menu pages. One said, “It would be helpful to have clearer explanations about each menu and each page, especially for first-time users. It was unclear to me.”

Feedback

Once users completed the questionnaire to indicate their suicidal thoughts and risk levels, they received a brief report detailing their suicide risk evaluation. This report included the assessment date and the indicated level of risk. The risk score ranged from 0 to 116, helping users understand the severity of their condition based on their score. The report also displayed the number of recent evaluations and the average scores over time. For example, a user might receive feedback such as: “Based on your most recent self-assessment using the suicide risk questionnaire in the app, your suicide risk severity is evaluated as mild, with a score of 45 out of 116. This score is lower than your average score of 52.5 from the two previous assessments. Furthermore, your current suicide risk severity is below the moderate level identified in the two earlier evaluations.”

Some users wanted a more thorough explanation and specific guidance to further address the suicide risk issues after the initial self-assessment results. One said:

The self-assessment results only allow me to develop a basic understanding of my mental status. In addition to the information about the level of my mental status, other detailed explanations about one's status and more tangible guidance on how to address it are needed.

In addition, one user expected that the app would be a more proactive intervention based on the self-assessment results, stating, “It would be even more beneficial if the process didn't end just with the self-assessment results but included a system where someone from relevant institutions or support services could call them directly to check their suicide risk status.”

Customized questionnaire

Some app users had divergent opinions about the number of questionnaires delivered on the app. Others provided suggestions about working on the questionnaire.

Questionnaire length

Regarding questionnaire length three different opinions emerged. One app user thought the number of items in the questionnaire was appropriate. Other users suggested that the questionnaire needed to have more items. One said, “I felt like it ended too quickly, almost disappointingly so. I wish the questionnaire could be longer to check various aspects of mental health for the participants.”

Another user, however, suggested a smaller number of self-assessment questions: “When I responded to the questionnaire through the app, the content was lengthy. A concise version of the questionnaire would be better.”

Need for having a comprehensive questionnaire

Two participants suggested a more comprehensive questionnaire covering diverse levels of mental health status beyond suicide risk awareness. One participant stated:

The questions focused on individuals whose suicide risk is high. I felt the questions were limited in understanding detailed insights. I would suggest having more comprehensive question items that cover both high- and low-risk participants, allowing a broader understanding of one's mental health state.

Another user stated a desire to see the app contain diverse types of emotions along with suicide risk questions. Yet another participant said:

Some questions in the survey asked about suicide-related situations, such as whether the participant attempted suicide or experienced violence. However, we experience very diverse emotions in our daily lives other than suicide-related. I want to see the questionnaire cover questions both directly related to suicide and everyday life-related emotions.

Discussion

The aim of this study was to assess the user experiences of a mobile app for suicide risk awareness by young adults in South Korea. We employed a mixed methods design to comprehensively examine user experience with the app both quantitatively and qualitatively. Overall, no significant differences between high-risk and nonrisk suicide groups emerged in user evaluation in each dimension of usability. Users rated each dimension of usability highly, which may lead to user engagement. In addition, qualitative data analyses revealed that participants believed that the app could enhance their suicide risk awareness and would be helpful when seeking reliable services and resources.

One of the important findings in the study is that users’ perceived learning plays a critical role in their satisfaction with the app, which is related to user engagement with mental health apps.⁴³ Our results revealed that perceived learning most significantly predicted user satisfaction. In our study, however, perceived learning was rated lower than the other dimensions of usability—ease of use, accessibility, and design—indicating the need to improve the perceived learning dimension of the usability of our app. Qualitative data analysis revealed that improving the perceived learning and satisfaction dimensions of usability requires customizing self-assessment results with more detailed explanations and guidance for further actions, along with relevant resources.^44,45 Some users commented on the need for more comprehensive information, including clear explanations and a range of possible actions and resources to help them obtain appropriate support at the right time. Additionally, several users suggested proactive interventions from relevant and reliable authorities, such as direct calls or texts to check their suicide risk status. In future app development, we will consider incorporating customized information and features based on these user suggestions.

Another finding is that some participants may have anticipated more from the current version of the mobile app, which is primarily designed to assess suicide risk and provide help-seeking resources. In our interviews, some users expressed a desire for more comprehensive information, such as more detailed coping strategies to enhance their suicide risk literacy. However, incorporating such in-depth information could alter the core purpose of the app. An alternative approach might involve setting users’ expectations at the outset by clearly communicating the app's primary intention as a suicide risk awareness tool. Providing this clarity could help align users’ expectations with the app's objectives, fostering a more mutually agreed understanding of its scope and purpose.

In addition, the results of the qualitative data analysis suggest ways to improve each dimension of usability. For example, to improve ease of use, accessibility, and design dimension of usability, we will provide clearer directions on each page about what information is presented and what actions are expected from users. In addition, we will conduct a formative evaluation with general audiences who have no knowledge of suicide risk to determine whether the wording on the app is easy to understand during their first read and the font size is comfortable enough to read the text. Also, users’ diverse opinions on the current color scheme, navigation, questionnaire length, and stated items will be reviewed with experts and another group of participants.

Furthermore, we found that the app effectively enhanced suicide risk awareness among the Korean public.^12,13 Our participants appreciated the opportunities that the app provided: the self-assessment tool to check their suicide risk status and the information about resources to obtain urgent care. They trusted the sources of information they saw on the app. Our study shows that the mobile app has the potential to significantly enhance the awareness of suicide risk status among members of the Korean public, and the mobile app can serve as a resource for users at high risk for suicide and those not at risk. Overall, the results are promising for the wider use of a mobile app for suicide risk awareness by the Korean public.

Our study is significant in several ways. First, it responds to the call for more usability studies with mental health apps. Researchers on mental health apps have criticized the lack of a usability test before a mental health app is released to the public²² because usability is related to engagement with the app and its continued use. Second, the study demonstrates the significant role a usability study plays in the development of mental health apps. The study results reveal the specific dimensions of usability that require improvement. Qualitative results provide detailed information about how to improve usability. Finally, our study can serve as an example of a usability study. To the best of our knowledge, no usability studies were conducted in S. Korea before mental health apps were released to the public. We adopted both quantitative and qualitative research designs to better understand the user experience with the app we developed. We hope our approach can be beneficial to researchers who plan to conduct a usability test with other mobile health apps.

Limitations and future research

This study has several limitations. First, a limitation of this study lies in the potential bias of the sample. Participants were recruited from individuals engaged with the Trauma Stress Center or actively seeking mental health-related information, indicating they were already inclined to acknowledge and address mental health concerns. Consequently, the findings may not fully represent the perspectives of individuals less likely to seek help or mental health information. Second, the interpretation of the study needs to be limited to Korean adults in their 20s enrolled in higher education. Future researchers may recruit users from diverse backgrounds, including those not enrolled in higher education. In addition, most participants in the study were women. Efforts to reach gender balance among participants are necessary. Third, the mobile app for suicide risk awareness was designed primarily to promote awareness of suicide risk and support help-seeking efforts by providing relevant information. To further explore the app's potential contributions, additional interactive features—such as synchronous text-based chatting or embedded log files that connect users to 24/7 mental health service helplines—should be integrated to provide guided feedback.^46,47 Incorporating these features would enable direct measurement of users’ app usage and their help-seeking behaviors. Fourth, while a power analysis indicated that 51 participants per group were required for adequate statistical power, recruitment constraints resulted in a smaller final sample. This limitation should be considered when interpreting the findings, as a larger sample may have yielded more robust conclusions. Fifth, although the screening questions were necessary for participant selection, their direct nature may have influenced participant responses or comfort levels. Future research should explore alternative screening approaches that balance sensitivity with methodological rigor. Six, the feedback provided by the app was standardized and may not have fully accounted for individual user experiences or cultural nuances. Future research should explore ways to make the feedback more personalized and contextually relevant to enhance user acceptance and engagement. Finally, in their suggestions for improving the mobile app for suicide risk awareness, interviewees emphasized the need for stronger measures to protect confidentiality. Our literature review highlights various strategies to achieve this. For example, some mental health apps encrypt metadata and unique IDs, delete unnecessary personal data from the service provider's storage, collaborate with mobile security apps, block data linking to third-party apps, and require user permission before accessing smartphone functions (e.g. contacts, camera, biometric data, location).^48,49 We will reflect upon these suggestions in another round of app development.

Conclusion

This study highlights the significant potential of mental health mobile apps, particularly in promoting suicide risk awareness among young adults in South Korea. The findings underscore the importance of user-centered design in developing effective tools for mental health support, as user feedback revealed both the strengths and areas for improvement in the app's design and functionality. While the app demonstrated promise in enhancing suicide risk awareness, especially in a Korean context, further refinements are necessary to improve user experience, particularly regarding perceived learning and guided feedback on suicide risk. Addressing concerns around confidentiality and providing clearer navigation are also critical for fostering user trust and engagement. These findings contribute to the development of mental health apps within South Korea and provide preliminary insights that may inform future research on digital interventions for suicide risk awareness in similar cultural contexts. Future research should expand this work by assessing similar apps across diverse populations and evaluating their long-term impact, ensuring that mental health apps continue to evolve as vital tools for early intervention and support.

Footnotes

Authors’ contributions

MHC and DHL contributed to the research conception, design, and process. They are responsible for the survey design and development, data collection, statistical analysis consultation, and research funding. The first draft of the manuscript was written by SWH and YJK, and the draft has been thoroughly revised by MHC, HJJ, and KEP. All authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Ethical approval

The Institutional Review Board (IRB) from SungKyunKwan University approved the study. All methods throughout the study were performed in accordance with the relevant guidelines and regulations. Informed consent was obtained from all participants prior to their enrollment in the study.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea (NRF2021S1A3A2A02089682), the Korean Mental Health R&D Project, funded by the Ministry of Health & Welfare, Republic of Korea (HL19C0001), and a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (No. HR21C0885).

ORCID iDs

Moon-Heum Cho

Seokwon Hwang

Dong Hun Lee

Appendix 1: Semi-structured interview questions

What motivated you to use the app, and how frequently, when, and where did you use it?

How easy was the mobile app to use? Were there any parts that were confusing or difficult to navigate?

How was your overall experience with accessing and navigating the app's content?

What did you think of the app's design, including its color, font, and text?

Did you feel that using the app helped you learn about your suicide risk status? How did it make you feel about your mental health?

How satisfied were you with the mobile app, and did it meet your expectations?

Which features or content did you find the most helpful, and why?

Were there any features or content you found irrelevant, unhelpful, or concerning?

Is there anything else you would like to share with the research team about your experience with the mobile app?

References

Statistics Korea. Annual report on the cause of death statistics in 2020. http://kostat.go.kr/portal/eng/pressReleases/8/10/index.board, 2020.

Ministry of Health and Welfare & Korea Foundation for Suicide Prevention. White Paper on Suicide Prevention. https://www.korea.kr/archive/expDocView.do?docId=40007, 2022.

World Health Organization. Live life: An implementation guide for suicide prevention in countries. https://www.who.int/publications/i/item/9789240026629, 2021.

Korea Foundation for Suicide Prevention. Suicide status by individual characteristics. https://kfsp-datazoom.org/korea02.do, 2023.

Kim

. Mediating effect of stigma on the relationship between mental health literacy and help-seeking attitudes among university students in South Korea. Int J Ment Health 2023; 52: 163–178.

Lee

, et al. Social stigma of suicide in South Korea: a cultural perspective. Death Stud 2023; 47: 259–267.

Park

Kim

Cho

, et al. Factors affecting stigma toward suicide and depression: a Korean nationwide study. Int J Soc Psychiatry 2015; 61: 811–817.

Park

Ratcliff

. Rising youth suicide and the changing cultural context in South Korea. Göttingen, Germany: Hogrefe Publishing, 2014.

McManama O’Brien

LeCloux

Ross

, et al. A pilot study of the acceptability and usability of a smartphone application intervention for suicidal adolescents and their parents. Arch Suicide Res 2017; 21: 254–264.

10.

Sarubbi

Rogante

Erbuto

, et al. The effectiveness of mobile apps for monitoring and management of suicide crisis: a systematic review of the literature. J Clin Med 2022; 11: 5616.

11.

Byrne

Lind

Horn

, et al. Using mobile sensing data to assess stress: associations with perceived and lifetime stress, mental health, sleep, and inflammation. Digital Health 2021; 7: 20552076211037227.

12.

Greeson

Treglia

Morones

, et al. Youth matters: philly (YMP): development, usability, usefulness, & accessibility of a mobile web-based app for homeless and unstably housed youth. Child Youth Serv Rev 2020; 108: 104586.

13.

Iyengar

Chang

Florez-Arango

, et al. Development and usability of a mobile tool for identification of depression and suicide risk in Fiji. Technol Health Care 2021; 29: 143–153.

14.

Denecke

Schmid

Nüssli

. Implementation of cognitive behavioral therapy in e–mental health apps: literature review. J Med Internet Res 2022; 24: e27791.

15.

Torous

Friedman

Keshavan

. Smartphone ownership and interest in mobile applications to monitor symptoms of mental health conditions. JMIR Mhealth Uhealth 2014; 2: e2994.

16.

Chung

Park

Joung

, et al. Response time as an implicit self-schema indicator for depression among undergraduate students: preliminary findings from a mobile app-based depression assessment. JMIR Mhealth Uhealth 2019; 7: e14657.

17.

Yao

Wang

, et al. Mental health literacy and suicidal ideation among Chinese college students: the mediating role of depressive symptoms and anxiety symptoms. J Affect Disord 2023; 339: 293–301.

18.

Pittman

Lindamer

Afari

, et al. Implementing eScreening for suicide prevention in VA post-9/11 transition programs using a stepped-wedge, mixed-method, hybrid type-II implementation trial: a study protocol. Implementation Sci Commun 2021; 2: 46.

19.

Cho

Chang

Lee

, et al. Development of mental health application: a formative evaluation. SAGE Open 2022; 12: 1–14.

20.

Arthurs

Tully

O’Malley

, et al. Usability and engagement testing of mHealth apps in pediatric obesity: a narrative review of current literature. Int J Environ Res Public Health 2022; 19: 1453–1465.

21.

Hajesmaeel-Gohari

Khordastan

Fatehi

, et al. The most used questionnaires for evaluating satisfaction, usability, acceptance, and quality outcomes of mobile health. BMC Med Inform Decis Mak 2022; 22: 22–31.

22.

Alqahtani

Orji

. Insights from user reviews to improve mental health apps. Health Informatics J 2020; 26: 2042–2066.

23.

Frías

Palma

Salvador

, et al. B· RIGHT: usability and satisfaction with a mobile app for self-managing emotional crises in patients with borderline personality disorder. Australas Psychiatry 2021; 29: 294–298.

24.

Firth

Minen

, et al. User engagement in mental health apps: a review of measurement, reporting, and validity. Psychiatr Serv 2019; 70: 538–544.

25.

Klonsky

May

. The three-step theory (3ST): a new theory of suicide rooted in the “ideation-to-action” framework. Int J Cogn Ther 2015; 8: 114–129.

26.

Klonsky

May

Saffer

. Suicide, suicide attempts, and suicidal ideation. Annu Rev Clin Psychol 2016; 12: 307–330.

27.

Garraza

Walrath

Goldston

, et al. Effect of the Garrett Lee Smith memorial suicide prevention program on suicide attempts among youths. JAMA Psychiatry 2015; 72: 1143–1149.

28.

Emqi

Hartini

. Pathways to get help: help-seeking on college students with non-suicidal self-injury. Edumaspul: Jurnal Pendidikan 2022; 6: 2136–2141.

29.

Rowe

French

Henderson

, et al. Help-seeking behaviour and adolescent self-harm: a systematic review. Aust N Z J Psychiatry 2014; 48: 1083–1095.

30.

Lotito

Cook

. A review of suicide risk assessment instruments and approaches. Mental Health Clin 2015; 5: 216–223.

31.

Singer

Erbacher

Rosen

. School-based suicide prevention: a framework for evidence-based practice. School Ment Health 2019; 11: 54–71.

32.

Chandrashekar

. Do mental health mobile apps work: evidence and recommendations for designing high-efficacy mental health mobile apps. mHealth 2018; 4: 6.

33.

Martínez-Hernáez

DiGiacomo

Carceller-Maicas

, et al. Non-professional-help-seeking among young people with depression: a qualitative study. BMC Psychiatry 2014; 14: 1–11.

34.

Chang

Patel

Iyengar

, et al. Impact of a mobile-based (mHealth) tool to support community health nurses in early identification of depression and suicide risk in Pacific Island countries. Australas Psychiatry 2021; 29: 200–203.

35.

Wahyuningrum

Fitriana

Wardhana

, et al. Developing suicide risk idea identification for teenager (SERIINA) mobile apps prototype using extended rapid application development. In: 2021 9th International Conference on Information and Communication Technology, pp.92‒97. IEEE. https://doi.org/10.1109/ICoICT52021.2021.9527508, 2021, August.

36.

Faul

Erdfelder

Lang

, et al. G*Power 3: a flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behav Res Methods 2007; 39: 175–191.

37.

Beck

Kovacs

Weissman

. Assessment of suicidal intention: the scale for suicide ideation. J Consult Clin Psychol 1979; 47: 343–352.

38.

Choi

Lee

Hwang

, et al. Reliability and validity of the Beck scale for suicide ideation (BSS) in Korean adult participants. Korean J Clin Psychol 2020; 39: 111–123.

39.

Lee

Jeon

, et al. Development and validation of the self-rating suicide risk screening questionnaire. OMEGA: J Death Dying 2022; 90: 1–29.

40.

Cho

. Online student orientation in higher education: a developmental study. Educ Technol Res Dev 2012; 60: 1051–1069.

41.

Zhou

Bao

Setiawan

IMA

, et al. The mHealth app usability questionnaire (MAUQ): development and validation study. JMIR Mhealth Uhealth 2019; 7: e11500.

42.

Teles

Paúl

Lima

, et al. User feedback and usability testing of an online training and support program for dementia carers. Internet Interv 2021; 25: 100412.

43.

Cho

Chi

Chiu

. Understanding sustained usage of health and fitness apps: incorporating the technology acceptance model with the investment model. Technol Soc 2020; 63: 101429.

44.

Chang

Alon

Torous

. An exploratory analysis of the effect size of the mobile mental health application, mindLAMP. Digital Health 2023; 9: 20552076231187244.

45.

Nagar

Quirk

Anderson

. User experiences of college students using mental health applications to improve self-care: implications for improving engagement. Internet Interv 2023; 34: 100676.

46.

Hung

Chen

, et al. Smartphone-based ecological momentary assessment for Chinese patients with depression: an exploratory study in Taiwan. Asian J Psychiatr 2016; 23: 131–136.

47.

O’Grady

Melia

Bogue

, et al. A mobile health approach for improving outcomes in suicide prevention (SafePlan). J Med Internet Res 2020; 22: e17481.

48.

Iwaya

Babar

Rashid

, et al. On the privacy of mental health apps: an empirical investigation and its implications for app development. Empir Softw Eng 2023; 28: 2.

49.

Surani

Bawaked

Wheeler

, et al. Security and privacy of digital mental health: an analysis of web services and mobile applications. In: Atluri

Ferrara

(eds) Data and applications security and privacy XXXVII. DBSec 2023. Lecture Notes in Computer Science, vol 13942. Cham: Springer, 2023. https://doi.org/10.1007/978-3-031-37586-6_19 .

Usability of a mobile app for suicide risk awareness in South Korea

Abstract

Background

Methods

Results

Conclusion

Keywords

Introduction

Importance of awareness in the early stage of suicidal ideation

Importance of suicide apps related to awareness

Research purpose

Methods

Participants

Suicide risk awareness app

Measures

Procedures

Results

Usability with the Mobile app for suicide risk awareness

User feedback about suicide risk awareness app

Overall experience

Self-awareness

Accessibility

Trustworthiness

Readability

Wording

Legibility

App interface

Color scheme

Navigation

Easy-to-answer questions

App operation

Technical glitch

Confidentiality

Functional requirements

Clear direction

Feedback

Customized questionnaire

Questionnaire length

Need for having a comprehensive questionnaire

Discussion

Limitations and future research

Conclusion

Footnotes

Authors’ contributions

Ethical approval

Declaration of conflicting interests

Funding

ORCID iDs

Appendix 1: Semi-structured interview questions

References