Sage Journals: Discover world-class research

Abstract

Introduction

Internet-based self-help interventions have the potential to help people address their emotional needs at relatively low costs. However, if the system does not offer optimal functions, it could reduce end-user adherence and satisfaction with treatment and compromise the effectiveness of the program. This study evaluated the usability of an Internet-based self-help intervention for emotional self-management among the general population of Italy during the COVID-19 pandemic.

Methods

A balanced sex-age sample of 10 individuals who met the inclusion criteria were consecutively recruited online. The think-aloud testing method, the system usability scale and an ad hoc semi-structured interview were used to determine the overall system usability.

Quantitative data were analyzed using descriptive statistics and qualitative data were analyzed using thematic analysis.

Results

The participants were mostly satisfied with the usability of the program. However, older users (<45 years) encountered some problems, which took longer, made more mistakes, and needed more help in performing the tasks than their younger counterparts. The analysis of the interviews revealed three central themes: general thoughts about the platform, weaknesses of the platform and difficulties encountered while navigating and completing tasks, and strengths of the platform.

Discussion

Based on the results of this study, important improvements will be made before the RinasciMENTE program is tested under real-world conditions. Conducting usability testing is a crucial step at an early stage of the development process of an Internet-based self-help intervention to identify potential usability problems with the system.

Keywords

Internet usability testing mental health self-care COVID-19

Introduction

Internet-delivered self-help interventions and COVID-19

Internet-delivered interventions, also known as online interventions or e-Health interventions, have been extensively studied since their emergence in the late 1990s. Research has consistently shown that these interventions can be equally effective as face-to-face¹ in addressing a wide range of mental health disorders,^2,3 not only in controlled trial settings but also in the context of routine psychiatric care.^4,5

Internet-delivered interventions offer several advantages, including increased accessibility, flexibility in delivery, scalability, cost-effectiveness, and the ability to reach people who may not have access to traditional face-to-face interventions due to geographical, logistical, or stigma-related barriers.⁶

Research has also documented the impact of the degree of involvement of the therapist, ranging from “pure” to “guided” self-help, on treatment outcomes. Self-help interventions involve individuals advancing independently through a diverse range of therapeutic material delivered via the Internet at their own convenience and speed.^7–10 These interventions utilize various modalities, including AI chatbots,¹¹ and interventions featuring automated feedback websites.^6,12 As this means that no human professionals are involved, Internet-delivered self-help interventions ensure discreet access to therapy, and it is easier and cheaper to widely implement.¹³

This approach can be an effective treatment option for various psychological problems, including anxiety^14,15 and depressive symptoms,¹⁶ eating disorders,¹⁷ and adjustment disorders.¹⁸

Internet-based self-help interventions have gained increasing relevance and popularity in light of the COVID-19 pandemic, and research suggests that interest and utilization of such solutions continue to remain substantial.¹⁹ With the increasing demand for mental health support, Internet-based self-help interventions offer a potential solution to these challenges.^20–23 Their accessibility and effectiveness make self-help interventions valuable tools for providing support to individuals facing unprecedented difficulties, especially while adhering to social distancing measures.^24–26

Furthermore, previous studies have consistently highlighted that Internet-based self-help interventions commonly rely on evidence-based techniques to help people cope with mental health disorders. Well-established methods such as traditional cognitive behavioral therapy (CBT) and more recent variations such as third-wave CBT, which includes mindfulness, acceptance, and commitment therapy, have received strong support as evidence-based techniques to provide Internet-based self-help interventions targeting mental health issues like anxiety and depression.^27–29

Despite these advantages, compared to Internet-based interventions delivered through some form of guidance, self-guided programs exhibit low levels of adherence^20,30 and higher dropout rates.³¹

Since adherence is related to the outcome, it is important to consider all factors that could improve the completion of self-guided Internet-based interventions.³² This starts with designing and evaluating the usability of the system during the development process.^33,34

The need for usability assessment

Usability is defined as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (p. 3).³⁵ Usability testing is a critical step in the development of Internet interventions as it solicits end-user feedback about what works, what does not, and where gaps in information or functionality exist to improve the website interface and content.^36–40

Problems with the usability of technology, such as ineffective system design, lack of ease of use and convenience of access, and a mismatch between the system features and the needs, expectations, and characteristics of users.^41,42 They are also likely to result in frustration and reduced user satisfaction when interacting with systems,⁴³ a higher probability of committing errors, and suboptimal clinical outcomes or refusal of the intervention.⁴⁴ This is particularly important for healthcare technologies, as their usability can affect the quality and effectiveness of treatment.⁴⁵ Even if other Internet-based self-management programs have conducted usability studies to help refine their systems,^46,47 the number of Internet-based programs that publish their usability evaluation results remains scarce.³⁶ To identify issues related to system design and functionalities, this study evaluates the usability of the RinasciMENTE Internet-based self-help intervention.

Overview of the RinasciMENTE e-platform to promote emotional self-management among the community during the COVID-19 pandemic

This program has previously been described.⁴⁸ In summary, it consists of an Internet-based self-help intervention, based on CBT principles and techniques, specifically developed to address immediate distress and prevent the long-term psychological consequences of the COVID-19 pandemic among Italians living within the country and abroad. The content is organized into eight weekly e-sessions over a period of two months. After logging into the e-platform, users are asked to complete self-report measures, and a brief online meeting with a mental health professional is then scheduled. Subsequently, users can access contents that promote: (1) information about possible psychological consequences of COVID-19 and how CBT works, (2) understanding the connection between feelings and behaviors, (3) understanding how negative thoughts can influence mood and the use of alternative thinking strategies to manage cognitive traps, (4) acceptance, (5) stress management, (6) problem resolution, and (7) a plan of completion and maintenance. In addition, an eighth module is tailored to the specific needs of the users: (a) emotional education, (b) anxiety and exposition, (c) enduring anxiety, (d) social anxiety, (e) panic attack, (f) sleep quality, (g) perfectionism, (h) relaxation, and (i) management of default memories.

The e-platform proactively sends out scheduled emails to inform users that new content is available, and a weekly reminder containing a brief encouraging message is emailed to those who do not access the material.

Objective

This study aims to assess the usability of a self-guided Internet-provided intervention for emotional self-management in terms of user performance and satisfaction with the content, interface, and functionality of the program. The main goals were to: (1) collect qualitative and quantitative data on usability; and (2) identify usability problems.

Methods

Study design

The study was carried out between January 2021 and March 2021. It employed a mixed methods sequential explanatory design approach, comprising the following end-user testing: (1) collection and analysis of performance measures employing the think-aloud method,⁴⁹ (2) collection and analysis of quantitative data through the use of a System Usability Scale (SUS),^50,51 and (3) collection and analysis of qualitative data through the use of an ad hoc semistructured interview to further explain the results obtained in the earlier stages. The study adhered to the Good Reporting of A Mixed Methods Study in health services research criteria.⁵² Supplemental File 1 reports the COnsolidated criteria for REporting Qualitative research Checklist.

Ethical statement

The study was reviewed and approved by the Ethics Committee of the Catholic University of Milan (ID: 32–22). All procedures were carried out following the ethical standards of the institutional and/or national research committee and with the Helsinki Declaration and its subsequent amendments or comparable ethical standards. Participants gave their informed written consent to participate in this study.

Recruitment

Ten individuals who showed interest in participating in the RinasciMENTE project by writing to the email address behind the official website (https://www.iterapi.se/sites/rinascimente/) were consecutively asked to participate in the usability test based on inclusion criteria. The study was carried out using the platform that was specifically developed for the project. Participants were stratified according to sex and age to help ensure that the study sample was representative of the larger population studied (demographics are reported in Table 1). No participants declined to participate or withdrew from the study.

Table 1.

Sociodemographic characteristics of participants.

Characteristic	Value
Age (years), M (SD); range
	40.7 (15.36)
	20–70
Sex, n (%)
Male	5 (50%)
Female	5 (50%)
Education, n (%)
Elementary school	1 (10%)
Middle school	1 (10%)
High school	2 (20%)
University	6 (60%)
n (%)
Self-reported good computer skills	6 (60%)
Self-reported good Internet skills	7 (70%)
Prior usability test, n (%)
Yes	3 (30%)
No	7 (70%)

To be eligible, participants had to (1) be ≥18 years, (2) be able to read and speak Italian, (3) have basic computer skills and Internet access, (4) own a computer or compatible device, and (5) provide informed consent to participate. Participants were excluded if they presented self-reported visual, hearing, or cognitive impairments that prevented them from following instructions.

Procedures

As a result of the COVID-19 pandemic, all procedures were performed online (completion time: 45 minutes, on average). First, each participant received an email with (a) the link to the online meeting, including the date and time, and (b) the link and access credentials of the RinasciMENTE platform. Before starting the usability test, the researcher provided detailed information about the test procedures and described the purpose of the computer-based self-management system. The participants were then asked to give their informed consent, provide demographic information, and report on their level of comfort with computers and the Internet.

They were also directly asked to self-report their computer and Internet skills using an ad hoc question and to tell if they had already taken a usability test before.

All participants were assigned a unique ID not related to their identity. Personal data was linked only to this unique ID and stored on a secure online platform accessible only by the research team and protected by a password.

No relationship between the researcher and the participants was established prior to the start of the study. The participants’ understanding of the researcher was limited to the reasons behind conducting the research.

Think-aloud and tasks competition

Participants were instructed to log into the system; and then perform a series of tasks on the platform while providing continuous commentary following the think-aloud method.⁴⁹ This is a common approach to usability testing that allows for the evaluation of the ease with which a system is learned. ⁵³ Unlike a formal qualitative coding process that involves a thematic analysis to discover the main findings,⁵⁴ the think-aloud process was used to gather feedback on the prototype of the platform.

A facilitator guided each participant through three goal-oriented tasks (see Textbox 1) by presenting them and reminding (if needed) the participants to provide reactions to the features of the platform, navigation, and perceived utility by continuously speaking their thoughts and actions out loud.⁵⁵ Only in the case of failure to verbalize their thoughts aloud, the facilitator reminds the user, as studies have shown that active solicitation during the think-aloud process can interfere and have a strong impact on the reliability and validity of the results.^56,57 Also, if the participant asked a question, the facilitator remained neutral and replied, “What do you think?” or “What would you do?” This allowed participants to explain their thoughts as they tried to work through the intervention and obtain real-time feedback and emotional responses to the intervention.

Textbox 1.

Tasks.

Task 1	Task 2	Task 3
Go to the “troubleshooting” treatment module, select “exercise 1” and verbalize what you are doing, thinking, and feeling while completing the task.	Go to the “dysfunctional thoughts” treatment module, select “exercise 1” and verbalize what you are doing, thinking, and feeling while completing the task.	Go to the “dysfunctional thoughts” treatment module, select “exercise 2” and verbalize what you are doing, thinking, and feeling while completing the task.

Several performance measures were collected, including task completion rate, task completion time, error frequency, and frequency of help. The completion rate was defined as the percentage of participants who completed the task. The task completion time was the average time it took to complete the task. The amount of time that participants had to complete the tasks was not limited, but they were instructed to try their best to perform the tasks. They were also asked to report to the researcher/facilitator if they were unable to complete the tasks. The frequency of error was defined as the total number of errors made on the task by all of the participants who went through the task (errors included choosing a wrong button, being unable to find and interpret the information correctly, etc.). The participants were corrected and asked to try again when they made an error. The frequency of help was defined as the total number of times that all participants needed help with the task. The facilitator video-recorded the interviews and took systematic notes on a Word document while the participant provided feedback.

Before starting the observation with the test participants, a pilot test was conducted to verify any issues in the execution of tasks or other problems that could be resolved before involving the participants.

Questionnaire administration

At the end of the session, the Italian version SUS^50,51,58,59 was administered to assess the perceived usability of the platform by users, with a particular focus on their satisfaction with the system (see Supplemental File 2). The SUS is a 10-item questionnaire with response options ranging from 1 (strongly disagree) to 5 (strongly agree). The total score varies between 0 and 100, with higher overall scores indicating a higher quality of usability and an overall usability score below 68 showing poor usability of the evaluated system.⁶⁰ Based on different studies in the literature, the SUS is an easy-to-use and reliable tool for distinguishing between usable and non-usable systems, even with small sample sizes.⁶¹

User perceptions (semistructured interviews)

Lastly, ad hoc semistructured interviews with seven questions (see Textbox 2) were conducted by author MS to further explore the experiences of the participants with the platform—including their general opinions about the system, any difficulties encountered while navigating and completing tasks, and what they believed to be the strengths and weaknesses of the platform. The interview lasted about 15 minutes. The researcher recorded the interviews on video and made field notes during the interview. Transcripts were not returned to participants for comment, corrections, or feedback on the findings.

Textbox 2.

Semistructured interviews.

1. Is the platform easy to navigate?

2. Is the platform design appealing?

3. Is the content presented in an engaging, interesting, and simple way?

4. Is the information provided clear and appropriate?

5. Does the platform provide relevant and interesting activities?

6. What do you like most about the platform?

7. What do you like least about the platform?

Data analyses

The data was analyzed by two researchers (authors MS and GM) using both quantitative and qualitative methods. Quantitative data were analyzed using SPSS Version 24.0.⁶² To summarize and describe the main characteristics of the sample, descriptive statistics, including means and standard deviations, were first used. Following, data from the performance measures (think-aloud) were extracted from the videos, and the means/frequencies were examined. The semistructured interviews were then recorded and transcribed verbatim. After transcription, researchers likely engaged in the thematic analysis of the interview content,^63,64 thus identifying key themes, concepts, or patterns that emerged from participants’ responses to gain insight into their experiences, opinions, or behaviors. More in detail, the thematic analysis process began with data familiarization. Once researchers became familiar with the interview data by reading and rereading transcripts, they started the coding procedure by systematically identifying and labeling interesting patterns within the data. This involved highlighting text segments that were relevant to the research question or objectives. The codes were grouped based on similarities or relationships, forming potential themes. The researchers systematically searched for patterns, connections, and meanings within the coded data. Once potential themes have been identified, the researchers reviewed and refined them. Subsequently, they defined and clearly described each theme based on the patterns observed in the data, also providing illustrative examples. Each theme was given a descriptive and meaningful name.

Sample size determination

The sample size was based on considerations by Nielsen⁶⁵ and Virzi⁶⁶ according to which involving the first five users in usability testing is generally sufficient to identify a significant portion (95%) of usability problems. Then, these problems are expected to be confirmed by subsequent users, suggesting that the initial five users are representative enough to capture most of the issues. Furthermore, considering various factors such as the complexity of the system being tested, the diversity of the user population, and the specific goals of the testing process, a larger number of participants were included in the study to identify most of the usability problems.⁶⁷ The researchers reached a consensus on data saturation following their discussion.

Results

Characteristics of the participants

The participants were evenly distributed by sex, with a mean age of 40.7 years (SD = 15.36). They had a high level of education and most of the respondents self-reported good computer and Internet skills.

Performance measures

Half of the users (n = 5) did not experience difficulties in identifying the icon to access the intervention, while the remaining five users were confused in distinguishing between the command “Welcome” and “Login.” Further negative judgments were associated with the platform home pages, as follows: “I am confused, I don't understand the difference between the home page and the beginning of the intervention”; or “The beginning is not very clear, as an introductory part seems to be missing” (Figure 1). In contrast, positive judgments were made about the presentation of the treatment modules: “The page that presents the treatment modules is clear and intuitive” (Figure 2).

Figure 1.

The RinasciMENTE welcome page.

Figure 2.

The RinasciMENTE modules.

Task completion rate

The completion rate reflects the achievement of all points related to each exercise. The results reveal that only three participants fully completed all navigation tasks. Specifically, 30% of the users did not complete Task 1; 100% of the participants completed Task 2; and 90% of the sample completed Task 3. On average, a success rate equal to 73% was achieved in completing all three activities.

Task completion time

The average completion time of the three tasks was about 20 minutes, and the results showed that the ones that took the longest were users aged ≥45 years.

Frequency of error

Concerning the insertion of more answer options in a single line or the erroneous compilation of the tables required by the tasks, these errors were made by only three users (30% of the sample).

Frequency of help

Similarly, to the frequency of error, the finding revealed that only Tasks 1 and 3 produced a call for help (n = 1), which was made uniquely by users older than 45 years. Performance measures are reported in Table 2 below:

Table 2.

Performance measures as assessed via three tasks.

Task	N	Task completion rate (%)	Mean task completion time (min)	Frequency of error n_error (n)^a	Frequency of help n_help (n)^b
1	10	70%	7.3	1 (2)	1 (1)
2	10	100%	4.7	0	0
3	10	90%	6.4	2 (1)	1 (1)

n_error(n) represents the number of times an error was made and n represents the number of people who made the error.

n_help(n) represents the number of times help was given and n represents the number of people who needed help.

Perceived usability outcomes

The results of perceived usability (as measured by the SUS) are presented in Table 3. Data analysis showed that the RinasciMENTE platform achieved an average score of 68.25, meaning that the platform does not present serious usability problems that need to be addressed.⁶⁰ The highest score was 77.5, reported by user #3, while the lowest score of 42.5 was achieved by user #9.

Table 3.

Result of the System Usability Scale.

Item	P1^a	P2	P3	P4	P5	P6	P7	P8	P9	P10
1	4	3	3	4	1	3	4	3	4	3
2	2	3	1	3	1	1	2	2	3	2
3	4	4	4	2	4	3	3	4	2	4
4	3	3	1	1	1	1	4	4	4	3
5	5	3	3	2	1	1	5	4	4	4
6	2	2	2	1	3	2	1	2	3	2
7	5	4	5	3	5	5	4	4	4	3
8	3	1	3	3	1	1	2	2	4	1
9	3	4	4	4	3	3	4	3	1	5
10	1	1	1	2	1	1	1	2	4	1
Odd score	21	18	19	15	14	15	20	18	15	19
#X	16	13	14	10	9	10	15	13	10	14
Even score	11	10	8	10	7	6	10	12	18	9
Y	14	15	17	15	18	19	15	13	7	16
X + Y = Z	30	28	31	25	27	29	30	26	17	30
Z × 2.5	75	70	77.5	62.5	67.5	72.5	75	65	42.5	75

P1–10: participant 1–10; Item: 1. I think that I would like to use this intervention; 2. I found the intervention unnecessarily complex; 3. I found the intervention easy to use; 4. I think I would need support from a technical person to use this intervention; 5. I found the various functions in this intervention were well integrated; 6. I thought there was too much inconsistency in this intervention; 7. I would imagine that most people would learn to use this intervention quickly; 8. I found the intervention very cumbersome to use; 9. I felt very confident using the intervention; 10. I needed to learn a lot of things before I could start using this intervention.

According to the classification of Bangor et al.’s,⁶⁸ 60% of the respondents scored higher than 70, thus rating the platform as “good”; 30% of the users achieved an SUS total score ranging from 62.5 to 67.5, suggesting that usability improvements are needed; and only 10% of the respondents (n = 1) classified the platform as “poor” (SUS score ≤ 51). Those who achieved the higher scores had a mean age of 37.5 (range 22–66) years and were equally distributed between males and females. The lower score was, instead, shown by a female respondent aged 61 years.

Of the 10 participants, seven indicated the RinasciMENTE platform as “easy to use,” and eight respondents stated that “most people would learn to use the intervention quickly” (see Figure 1).

Regarding the need for technical support to use the platform, the results showed good usability of the system, as seven out of the 10 respondents disagreed or strongly disagreed with the need for help; and six users indicated that the system was not complex or cumbersome. Still, two respondents did not “feel very confident using the intervention.” Remarkably, none of the participants was “strongly in agreement” with the negatively worded questions.

Responses to the semistructured interview

The qualitative analysis of the semistructured interviews was conducted, and three main themes were derived from the data: general thoughts about the platform; weaknesses of the platform and difficulties encountered while navigating and completing tasks, and strengths of the platform.

Theme 1—General thought about the platform

In general, the platform was perceived as clear by users; however, certain aspects revealed areas for improvement. Excerpts of dialog below illustrate instances where users identified limitations of the platform. Doubts were mostly related to the amount of information, which appeared excessively noisy for each module, but scant in the introductory part, where participants expected to receive more details.

The platform is clear and seems easy to use; just some points are not perfectly comprehensible, but overall—I find it a useful program.

The first page is not very clear, there is no introductory part that I expected, but overall, the platform is not difficult to use. I find it rich in information, but I would only remember the main information. Anyway, I find it to be functional (the program) for its purpose if used consistently.

It seems incomplete; I was expecting more information mainly on the homepage in the ‘Welcome’ section.

Theme 2: Weaknesses of the platform and difficulties encountered while navigating and completing tasks

The main difficulties concerned the first web page, for which doubts emerged with the login and the homepage.

I found it difficult to navigate, I did not understand the difference between ‘Go to the module’ and ‘Homepage’ commands after reading the instructions.

At the beginning, I couldn't find the ‘login’ command, then I had trouble understanding what the ‘Selected’ entry meant. Therefore, I reread the instructions but I did not understand them, and I asked for help. Furthermore, I did not know how to complete the exercise because I did not understand the example.

Theme 3—Strengths of the platform

From the interviews, it emerged how important the graphic part is, including colors and images. In terms of content, the presence of examples also helped to make the platform more usable.

It is a useful platform for its functionalities; I find the setup of the treatment modules clear, especially for the use of images; in general, I think the platform is very intuitive. It is easy to understand how to use it.

I appreciated the clarity of the content and the presence of examples that help to understand how to complete the exercises.

In my opinion, the positive aspects of the platform are its colors, and graphics that are simple and clean, so they can help people focus on appropriate information and not irrelevant details.

Discussion

Principal results

This study investigated the usability of the RinasciMENTE platform by evaluating the interaction between users and the system through a set of quantitative–qualitative tools.⁶⁹ According to Nielsen's assumption,⁶⁵ the main problems with the system were encountered by the first five users, while subsequent users only confirmed them, and no further difficulty was detected.

Overall, the results show that most of the participants were able to perform the tasks without additional time and resource expenditures. This reflects the “practice of simplicity” prerequisite of an Internet-based intervention, which is to be as simple as possible, allowing users to find what they are looking for quickly and logically.⁶⁵

However, older users (45 years) experienced more difficulties, as they took longer, made more mistakes and needed help in carrying out the tasks compared to younger people. These findings are consistent with previous studies investigating the influence of age on usability testing.^70–72 Van Deursen's investigations^73,74 showed that older people display lower operational and navigational skills more often than their younger counterparts. This discrepancy could be attributed to the lack of practical digital skills of older participants rather than to the design of the platform and could be overcome by offering them introductory sessions on the platform to help them through the program for the first time.⁷²

In this study, the participants were very positive about the aesthetics and layout. This is an important outcome, as studies claimed that visual elements, including graphics, are the first things that grab users’ attention by creating their first impression.^75,76 However, some areas that could be improved to further enhance the system were identified. First, the homepage appeared unclear and confusing to some respondents. The participants also indicated that they felt overwhelmed by the amount of information reported on the homepage. In fact, text-heavy web pages can make content difficult to process, acting as a barrier to intervention adherence,⁷⁷ and comprehensibility of website design should be considered to “reduce the cognitive effort required to effectively use eHealth applications.”⁷⁸ Ultimately, the SUS revealed that the program was rated “good,” indicating that the intervention was considered useful.

These findings are similar to those of other usability studies of digital interventions that employed CBT techniques to address stress due to COVID-19.^79–82 Specifically, a study aimed to evaluate the feasibility, acceptance, and usability of a resilience app on a sample of 75 students revealed a satisfactory degree of usability.⁸³ Analogous results were obtained by a previous test of a digital mental health tool for essential workers and those unemployed due to COVID-19.⁸¹ Moreover, the results of a recent meta-analysis of 16 studies indicate that digital mental health treatments could be feasible and beneficial for various groups, including the general public, those at risk, and individuals with existing mental health conditions in both high- and middle-income nations.⁸⁴ Still, studies evaluating the usability of digital tools differ in the technologies employed—such as mobile apps, web-based programs, virtual reality environments, or wearable devices—the methods used to assess their usability, including user surveys, interviews, usability testing, or qualitative analysis of user feedback, and outcome measures (e.g. efficiency, effectiveness, learnability, memorability, errors, and subjective perceptions of usability). Other sources of variability are represented by the sample size and characteristics, such as the clinical or nonclinical population, age group, levels of digital literacy, technological preferences, access to digital devices or Internet connectivity, or different aspects of the user experience being investigated (e.g. ease of navigation, clarity of instructions, visual design, interactivity, responsiveness, and overall satisfaction).

Given this variability, the findings may lead to different recommendations to improve the usability of digital interventions, such as redesign of the interface, customization of content, technical support, or integration with existing healthcare services.

Multi-method usability testing is, therefore, essential to ensure the relevance of the content and discover intervention improvement needs (e.g. more clear buttons) and for content inadequacy (e.g. inconsistencies, lengthy text).

Participants saw value in the RinasciMENTE intervention and confirmed their interest in its use to self-treat symptoms once it was available.

Improvements to the usability of the internet-based intervention will be made before going to the full trial.

Strengths and limitations

A strength of this study is the use of a mixed-methods design to identify potential usability problems more reliably.⁸⁵ Furthermore, for the present usability study, the methodology was carefully set based on systematic usability test guidelines and empirical research.^65,86 Therefore, this study provides an effective method to identify elements that require modification for solutions delivered through the Internet. These results, indeed, further highlight the importance of performing a usability test during a system development process to avoid drawing incorrect conclusions about the effectiveness of a given intervention. Internet-based self-management programs have the potential to improve the quality of care at a relatively low cost, but, if their content and/or design are perceived as unpleasant by users, this might negatively impact health outcomes regardless of the quality of the intervention provided. Despite the small sample, the number of participants meets the feasibility study recommendations for feasibility studies,⁶⁵ and it can be deemed representative of the population of potential end users of the RinasciMENTE program.

It is also important to acknowledge the limitations of this study. First, only participants from the general population were included in the study. In addition to testing the platform with a community sample, providing valuable insights from a range of perspectives, which can help identify usability issues that may not be apparent when testing with a clinical population alone, the inclusion of both clinical and nonclinical populations would have enhanced the validity, generalizability, and inclusivity of findings, ultimately improving the effectiveness of the online platform for mental health support across diverse user groups. Another limitation of this study is that, due to its nature, it did not allow the inclusion of subjects who did not have computer and Internet access at home, which could affect the representativeness of the sample. Furthermore, since usability testing was done online, it is important to take into account the inability to account for the environmental factors that vary between testing locations. ⁸⁷ User characteristics (e.g. age, sex, and computer experience) may also have affected measurement outcomes.

Future research and implications

The feasibility and effectiveness of the RinasciMENTE program will be evaluated in a randomized controlled trial design.⁴⁸ A community sample that experienced mild to moderate psychological problems during the COVID-19 pandemic will be selected for inclusion based on the selected criteria. Individuals will be randomly assigned to a group receiving the CBT intervention or placed on a waiting list control. The effect of the RinasciMENTE program on selected primary and secondary psychological outcomes will be tested at the end of the intervention (two months) and at six- and 12-month follow-ups.

Conclusions

Internet-based technologies can play an important role in helping people better manage their psychological symptoms. However, to increase end-user satisfaction with the interventions and prevent them from committing errors or having difficulties using the system, the program must be tailored to the characteristics, needs, and preferences of the end-users. The methods used in the present usability study were effective in identifying elements that needed modification, and significant improvements will be made before the RinasciMENTE program is deemed ready for testing under real-world conditions to go full trial.

Supplemental Material

sj-pdf-1-dhj-10.1177_20552076241258419 - Supplemental material for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing

Supplemental material, sj-pdf-1-dhj-10.1177_20552076241258419 for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing by Michelle Semonella, Gloria Marchesi, Gianluca Castelnuovo, Gerhard Andersson and Giada Pietrabissa in DIGITAL HEALTH

Supplemental Material

sj-pdf-2-dhj-10.1177_20552076241258419 - Supplemental material for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing

Supplemental material, sj-pdf-2-dhj-10.1177_20552076241258419 for Internet-delivered emotional self-management program for the general population during the COVID-19 pandemic: Usability testing by Michelle Semonella, Gloria Marchesi, Gianluca Castelnuovo, Gerhard Andersson and Giada Pietrabissa in DIGITAL HEALTH

Footnotes

Acknowledgments

We thank all of those with whom I have had the pleasure to work during this and other related projects.

Authors contribution

MS and GP conceptualized and designed. MS coordinated the study's recruitment and conducted interviews. GM and MS coded and analyzed the interviews under the supervision of GP. MS and GP wrote the manuscript. All authors participated in the critical review of the manuscript, and read and approved its final version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationship that could be construed as a potential conflict of interest.

Data availability statement

The dataset for this study is available upon request to interested researchers.

Ethical approval

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by Italian Ministry of Health - Ricerca Corrente.

Guarantor

Giada Pietrabissa.

Supplemental material

Supplemental material for this article is available online.

ORCID iDs

Michelle Semonella

Giada Pietrabissa

References

Hedman-Lagerlöf

Carlbring

Svärdman

, et al. Therapist-supported internet-based cognitive behaviour therapy yields similar effects as face-to-face therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. World Psychiatry 2023; 22: 305–314.

Andersson

Titov

Dear

, et al. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry 2019; 18: 20–28.

Folker

Mathiasen

Lauridsen

, et al. Implementing internet-delivered cognitive behavior therapy for common mental health disorders: a comparative case study of implementation challenges perceived by therapists and managers in five European internet services. Internet Interv 2018; 11: 60–70.

Hedman

Ljotsson

Kaldo

, et al. Effectiveness of internet-based cognitive behaviour therapy for depression in routine psychiatric care. J Affect Disord 2014; 155: 49–58.

Mathiasen

Riper

Andersen

, et al. Guided internet-based cognitive behavioral therapy for adult depression and anxiety in routine secondary care: observational study. J Med Internet Res 2018; 20: e10927.

Wang

Zhang

. A systematic review and meta-analysis of internet-based self-help interventions for mental health among adolescents and college students. Internet Interv 2023; 34: 100690.

Soklaridis

Lin

Lalani

, et al. Mental health interventions and supports during COVID-19 and other medical pandemics: a rapid systematic review of the evidence. Gen Hosp Psychiatry 2020; 66: 133–146.

Wang

Pan

Wan

, et al. Immediate psychological responses and associated factors during the initial stage of the 2019 coronavirus disease (COVID-19) epidemic among the general population in China. Int J Environ Res Public Health 2020; 17.: 1-25

Wind

Rijkeboer

Andersson

, et al. The COVID-19 pandemic: the ‘black swan’ for mental health care and a turning point for e-health. Internet Interv 2020; 20: 100317.

10.

Andersson

Titov

. Advantages and limitations of internet-based interventions for common mental disorders. World Psychiatry 2014; 13: 4–11.

11.

Liu

Peng

Song

, et al. Using AI chatbots to provide self-help depression interventions for university students: a randomized trial of effectiveness. Internet Interv 2022; 27: 100495.

12.

Kim

Choi

. A practical development protocol for evidence-based digital integrative arts therapy content in public mental health services: digital transformation of mandala art therapy. Front Public Health 2023; 11: 1175093.: 1-15

13.

Barak

Klein

Proudfoot

. Defining internet-supported therapeutic interventions. Ann Behav Med 2009; 38: 4–17.

14.

van Ballegooijen

Riper

Klein

, et al. An internet-based guided self-help intervention for panic symptoms: randomized controlled trial. J Med Internet Res 2013; 15: e154.

15.

Kahlke

Berger

Schulz

, et al. Efficacy of an unguided internet-based self-help intervention for social anxiety disorder in university students: a randomized controlled trial. Int J Methods Psychiatr Res 2019; 28: e1766.

16.

Cuijpers

Donker

Johansson

, et al. Self-guided psychological treatment for depressive symptoms: a meta-analysis. PLoS One 2011; 6: e21274.

17.

Kass

Trockel

Safer

, et al. Internet-based preventive intervention for reducing eating disorder risk: a randomized controlled trial comparing guided with unguided self-help. Behav Res Ther 2014; 63: 90–98.

18.

Eimontas

Rimsaite

Gegieckaite

, et al. Internet-based self-help intervention for ICD-11 adjustment disorder: preliminary findings. Psychiatr Q 2018; 89: 451–460.

19.

Dominiak

Gedek

Antosik

, et al. Prevalence, attitudes and concerns toward telepsychiatry and mobile health self-management tools among patients with mental disorders during and after the COVID-19 pandemic: a nationwide survey in Poland from 2020 to 2023. Front Psychiatry 2023; 14: 1322695.

20.

Newman

Szkodny

Llera

, et al.

A review of technology-assisted self-help and minimal contact therapies for anxiety and depression: is human contact necessary for therapeutic efficacy?

Clin Psychol Rev 2011; 31: 89–103.

21.

Clarke

Kelleher

Hornbrook

, et al. Randomized effectiveness trial of an internet, pure self-help, cognitive behavioral intervention for depressive symptoms in young adults. Cogn Behav Ther 2009; 38: 222–234.

22.

Pietrabissa

Simpson

. Psychological consequences of social isolation during COVID-19 outbreak. Front Psychol 2020; 11: 2201.

23.

Giusti

Pedroli

D'Aniello

, et al. The psychological impact of the COVID-19 outbreak on health professionals: a cross-sectional study. Front Psychol 2020; 11: 1684.

24.

Semonella

Andersson

Dekel

, et al. Making a virtue out of necessity: COVID-19 as a catalyst for applying internet-based psychological interventions for informal caregivers. Front Psychol 2022; 13: 856016.

25.

Castelnuovo

Pietrabissa

Manzoni

, et al. Fighting the COVID-19 pandemic using the technology-based second-line in Italy and Lombardy: the urgent need of home-based remote monitoring systems to avoid the collapse of the hospital-centred first line. J Glob Health 2020; 10: 010371.

26.

Simpson

Richardson

Pietrabissa

, et al. Videotherapy and therapeutic alliance in the age of COVID-19. Clin Psychol Psychother 2021; 28: 409–421.

27.

Waters

Algoe

Dutton

, et al. Positive psychology in a pandemic: buffering, bolstering, and building mental health. J Posit Psychol 2022; 17: 303–323.

28.

Heber

Ebert

Lehr

, et al. The benefit of web- and computer-based interventions for stress: a systematic review and meta-analysis. J Med Internet Res 2017; 19: e32.

29.

Harty

Enrique

Akkol-Solakoglu

, et al. Implementing digital mental health interventions at scale: one-year evaluation of a national digital CBT service in Ireland. Int J Ment Health Syst 2023; 17: 29.

30.

Rodrigues TR

Reijnders

Breeman

, et al. Use intention and user expectations of human-supported and self-help eHealth interventions: internet-based randomized controlled trial. JMIR Form Res 2024; 8: e38803.

31.

Karyotaki

Kleiboer

Smit

, et al. Predictors of treatment dropout in self-guided web-based interventions for depression: an ‘individual patient data’ meta-analysis. Psychol Med 2015; 45: 2717–2726.

32.

Kazlauskas

Eimontas

Olff

, et al. Adherence predictors in internet-delivered self-help intervention for life stressors-related adjustment disorder. Front Psychiatry 2020; 11: 137.

33.

Tindale

Dimitri

. Medtech innovation across the life course - the importance of users and usability. J Med Eng Technol 2022; 46: 427–432.

34.

Stoddard

Augustson

Mabry

. The importance of usability testing in the development of an internet-based smoking cessation treatment resource. Nicotine Tob Res 2006; 8: S87–S93.

35.

ISO 9241-11. Ergonomics of Human-System Interaction - Part 11: Usability: Definitions and Concepts. International Organization for Standardization. 2018. https://www.iso.org/standard/63500.html [accessed 2023-01-10].

36.

Maramba

Chatterjee

Newman

. Methods of usability testing in the development of eHealth applications: a scoping review. Int J Med Inform 2019; 126: 95–104.

37.

Chang

. Evaluation framework for telemedicine using the logical framework approach and a fishbone diagram. Healthc Inform Res 2015; 21: 230–238.

38.

Sadegh

Khakshour Saadat

Sepehri

, et al. A framework for m-health service development and success evaluation. Int J Med Inform 2018; 112: 123–130.

39.

Harte

Glynn

Rodríguez-Molinero

, et al. A human-centered design methodology to enhance the usability, human factors, and user experience of connected health systems: a three-phase methodology. JMIR human Factors 2017; 4: e8.

40.

National Institute of Standards and Technology. NIST guide to the processes approach for improving the usability of electronic health records. CreateSpace Independent Publishing Platform, 2010, pp. 17–27.

41.

Galavi

Montazeri

Ahmadian

. Barriers and challenges of using health information technology in home care: a systematic review. Int J Health Plann Manage 2022; 37: 2542–2568.

42.

Wallin

Norlund

Olsson

EMG

, et al. Treatment activity, user satisfaction, and experienced usability of internet-based cognitive behavioral therapy for adults with depression and anxiety after a myocardial infarction: mixed-methods study. J Med Internet Res 2018; 20: e87.

43.

Bevan

Kirakowski

Maissel

. What is usability? In:Proceedings of the 4th International Conference on HCI; Stuttgart, Germany: Proceedings of the 4th International Conference on HCI, 1991.

44.

Jacko

Sears

Sorensen

. Framework for usability: healthcare professionals and the internet. Ergonomics 2001; 44: 989–1007.

45.

Wilks

Yin

Zuromski

. User experience affects dropout from internet-delivered dialectical behavior therapy. Telemed J E Health 2020; 26: 794–797.

46.

Kenter

RMF

Schonning

Inal

. Internet-delivered self-help for adults with ADHD (MyADHD): usability study. JMIR Form Res 2022; 6: e37137.

47.

Nitsch

Dimopoulos

Flaschberger

, et al. A guided online and mobile self-help program for individuals with eating disorders: an iterative engagement and usability study. J Med Internet Res 2016; 18: e7.

48.

Bertuzzi

Semonella

Andersson

, et al. Study protocol for a randomized controlled trial evaluating the effectiveness of an internet-based self-help intervention to cope with psychological distress due to COVID-19 in the Italian general population: the RinasciMENTE project. Trials 2022; 23: 801.

49.

Someren

Barnard

Sandberg

. The think aloud method - a practical guide to modelling cognitive processes. London, England: Academic Press, 1994.

50.

Brooke

. SUS-A quick and dirty usability scale. In Usability evaluation in industry. London, England: CRC Press, 1996; 189: 4–7.

51.

Mobilio

. La sperimentazione di piattaforme. Innovazione 2006; 10: 7–8.

52.

O'Cathain

Murphy

Nicholl

. The quality of mixed methods studies in health services research. J Health Serv Res Policy 2008; 13: 92–98.

53.

Brinck

Gergle

Wood

. Usability for the web: designing web sites that work. San Francisco, CA: Elsevier, 2001.

54.

Nielsen Norman Group. Thematic analysis of qualitative user research data. https://www.nngroup.com/videos/thematic-analysis-qualitative-user-research-data/ [accessed 2023-01-04]

55.

Fan

Lin

Chung

, et al. Concurrent think-aloud verbalizations and usability problems. ACM Trans Comput-Hum Interact 2019; 26: 1–35.

56.

Boren

Ramey

. Thinking aloud: reconciling theory and practice. IEEE Trans Profess Commun 2000; 43: 261–278.

57.

Nisbett

Wilson

. Telling more than we can know: verbal reports on mental processes. Psychol Rev 1977; 84: 231–259.

58.

Gruppo di Lavoro per l'Usabilità. eGLU 1.0 - Protocollo per l'esplorazione dei siti web delle Pubbliche Amministrazioni 2013.

59.

Federici

Borsci

Meloni

. Le misure dell'usabilità: sstudio sulle caratteristichepsicometriche del QUIS e del SUMI nella versioneitaliana. Giornale di Psicologia 2009; 3: 233–253.

60.

Bangor

Kortum

Miller

. An empirical evaluation of the system usability scale. Int J Hum-Comput Interact 2008; 24: 574–594.

61.

System Usability Scale. Usability.Gov. https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html [accessed 2024-01-04].

62.

IBM Corp. IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp., 2016.

63.

Braun

Clarke

. Using thematic analysis in psychology. Qual Res Psychol 2006; 3: 77–101.

64.

Braun

Clarke

. Successful qualitative research: a practical guide for beginners. London, England: SAGE Publications Ltd 2013: 1-400.

65.

Nielsen

. Usability engineering. U.S.: Morgan Kaufmann, 1994.

66.

Virzi

. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 1992; 34: 457–468.

67.

Faulkner

. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35: 379–383.

68.

Bangor

Kortum

Miller

. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4: 114–123.

69.

Church

Fish

Haines-Young

, et al. UK National Ecosystem Assessment Follow-On. Work Package Report 5: Cultural ecosystem services and indicators; Cambridge, UK 2014.

70.

Sonderegger

Schmutz

Sauer

. The influence of age in usability testing. Appl Ergon 2016; 52: 291–300.

71.

Sauer

Sonderegger

Schmutz

. Usability, user experience and accessibility: towards an integrative model. Ergonomics 2020; 63: 1207–1220.

72.

van der Vaart

van Driel

Pronk

, et al. The role of age, education, and digital health literacy in the usability of internet-based cognitive behavioral therapy for chronic pain: mixed methods study. JMIR Form Res 2019; 3: e12883.

73.

van Deursen

van Dijk

. Internet skills performance tests: are people ready for eHealth? J Med Internet Res 2011; 13: e35.

74.

van Deursen

AJAM

van Dijk

JAGM

. Internet skill levels increase, but gaps widen: a longitudinal cross-sectional analysis (2010–2013) among the Dutch population. Information. Commun Soc 2015; 18: 782–797.

75.

Liu

Chiu

Hsieh

, et al. Optimizing the usability of mobile phones for individuals who are deaf. Assist Technol 2010; 22: 115–127.

76.

Gronier

. Measuring the first impression: testing the validity of the 5 second test. J Usability Stud 2016; 12: 8-25

77.

Eccles

Nannarone

Lashewicz

, et al. Barriers to the use of web-based mental health programs for preventing depression: qualitative study. JMIR Form Res 2021; 5: e16949.

78.

Rotondi

Spring

Hanusa

, et al. Designing eHealth applications to reduce cognitive effort for persons with severe mental illness: page complexity, navigation simplicity, and comprehensibility. JMIR Hum Factors 2017; 4: e1.

79.

Semonella

Marchesi

Andersson

, et al. Usability study of SOSteniamoci: an internet-based intervention platform to support informal caregivers in Italy. Digit Health 2024; 10: 20552076231225082.

80.

Newton

Bagnell

Rosychuk

, et al. A mobile phone-based app for use during cognitive behavioral therapy for adolescents with anxiety (MindClimb): user-centered design and usability study. JMIR Mhealth Uhealth 2020; 8: e18439.

81.

Mata-Greve

Johnson

Pullmann

, et al. Mental health and the perceived usability of digital mental health tools among essential workers and people unemployed due to COVID-19: cross-sectional survey study. JMIR Ment Health 2021; 8: e28360.

82.

Minian

Gayapersad

Saiva

, et al. An e-mental health resource for COVID-19-associated stress reduction: mixed methods study of reach, usability, and user perceptions. JMIR Ment Health 2022; 9: e39885.

83.

Nicolaidou

Aristeidis

Lambrinos

. A gamified app for supporting undergraduate students’ mental health: a feasibility and usability study. Digit Health 2022; 8: 20552076221109059.

84.

Zhong

Yang

Pan

, et al. The usability, feasibility, acceptability, and efficacy of digital mental health services in the COVID-19 pandemic: scoping review, systematic review, and meta-analysis. JMIR Public Health Surveill 2023; 9: e43730.

85.

Jaspers

. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009; 78: 340–353.

86.

Yen

Bakken

. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012; 19: 413–422.

87.

Tuckson

Edmunds

Hodgkins

. Telehealth. N Engl J Med 2017; 377: 1585–1592.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.41 MB

0.08 MB