Sage Journals: Discover world-class research

Abstract

Background

Providing informal care can be experienced as stressful and lead to caregiver burden. Internet-based interventions, a specific form of eHealth, have proven to be a good option to support informal caregivers. SOSteniamoci, an internet-based intervention already tested in Lithuania, was translated and adapted for Italian caregivers.

Objective

As many novel eHealth solutions have been rejected by end-users due to usability problems, we aimed to evaluate the usability of the adapted platform, using a computer-based prototype.

Methods

The following methods and metrics were applied: 1. task analysis, using audio and video recordings that included three usability metrics: task completion rate, frequency of errors, and frequency of help requests; 2. the system usability scale (SUS); and 3. a semi-structured interview to collect additional data about the system's design and overall satisfaction.

Results

Ten informal caregivers (60% female; age M = 47.8, SD = 15.21) provided insights and suggestions for increasing the usability of the platform. The platform was considered satisfactory, with a mean score on the SUS of 75 (SD = 13.07) out of 100. The task analysis measurements highlighted difficulties in how to log in to the platform, understanding what the intervention is about, and texting the therapist. The same difficulties were also mentioned during the post-experience interview. Thus, improvements were subsequently made to enhance users’ experience when navigating the platform. Finally, the platform overall was found to be intuitive and friendly, and the contents were appreciated.

Conclusion

To maintain participants’ engagement and prevent drop-out, it is crucial to test the usability of internet-based interventions. Even though the platform proved to be user-friendly, intuitive and easy to use, several enhancements were implemented based on participants’ feedback. Thus, the usability of internet-based interventions should be tested, and end-users must be involved in the development process of such solutions.

Keywords

Internet-based intervention informal caregiver usability test computer prototype caregiver burden self-help

Introduction

Informal care is generally defined as unpaid care provided to a relative or friend who is coping with a disability, chronic illness, or other care needs.¹ This phenomenon is growing as the number of adults living with chronic conditions is increasing.² Indeed, it is estimated that 80% of all care in Europe is provided by informal caregivers, and the number of informal caregivers ranges from 10% up to 25% of the total European population.³ Thus, informal caregivers can be considered the backbone of long-term care in Europe. Nevertheless, it is well-known that providing informal care may lead to several health consequences, lack of time, financial strain, and thus to caregiver burden.^4–7

Recognizing the negative health impacts on informal caregivers has led to the development of different psychological support interventions and solutions, which have been found to be beneficial.⁸ Lately, and mostly due to the COVID-19 pandemic, using technology to achieve positive outcomes has received interest, in the informal care context as well as in the formal care context.^9–11

The use of eHealth interventions is growing, mostly because the use of overall internet-based technology in people's daily lives is growing and its affordability is increasing. In the field of clinical psychology, delivering online psychological interventions may reduce geographical and financial barriers that might otherwise prevent potential participants from engaging in such interventions, and thus increase their scalability.¹² In this regard, internet-based interventions, a specific form of eHealth interventions in which therapeutic tasks based on cognitive behavioral principles are delivered via computerized systems,¹³ have been found to be as effective as face-to-face interventions.^14,15 Hence, such interventions provide a solution that might be complementary, or even an alternative, to face-to-face treatment.¹⁶

Despite the well-proven beneficial aspects of eHealth solutions and interventions, many innovations are rejected by stakeholders mostly due to usability problems such as incomplete content, ineffective system design, and lack of ease of use, causing a mismatch between the systems and users’ needs, characteristics, and expectations.^17–19 Frustrations and irritations may emerge as a result of poor usability, generating inefficiency and a higher likelihood of committing errors²⁰ and dropping out. To avoid these negative aspects, testing the system's usability in advance is highly recommended.²¹

The current study is part of a larger project titled SOSteniamoci, which consists of the assessment of an internet-based intervention, based on cognitive-behavioral therapy principles,²² for supporting informal caregivers in Italy.²³ As every internet-based cognitive behavioral therapy, what is usually referred in face-to-face treatment as “session”, it is called “module” in internet treatment. The duration of the treatment is defined beforehand, and it typically may last between 5 and 15 weeks (SOSteniamoci counts of 8-weeks modules), and it starts with psychoeducational material and ends with relapse prevention.²⁴ The aim of this psychological intervention is to reduce caregiver burden, depression, and anxiety while improving quality of life and relationship satisfaction.²³ The intervention platform allows users to access the platform, read weekly modules, complete the evaluation at the end of each module, complete weekly assignments, and be in contact with a therapist. The assessment records are saved in the platform and can be retrieved for review. This intervention has already been tested in Lithuania²⁵; as such, SOSteniamoci is an implementation and adaptation of an existing intervention. Although the intervention has already proven to be efficient as well as beneficial for informal caregivers in Lithuania, it is very important to tailor and adapt its contents and materials to a specific population with a different cultural background.

Hence, our aim in the current study was to test the usability of the adapted Italian version of the platform among Italian caregivers, and as such, to broaden our understanding and knowledge about the usability and cultural adaptation of internet-based solutions overall.

Overview of the SOSteniamoci platform to support informal caregivers in Italy

SOSteniamoci is a translated and adapted internet-based intervention that aims to support informal caregivers in Italy (See Figure 1).²³ It consists of eight weekly modules delivered over a period of two months, with each module dedicated to a different theme (See Figure 2). The themes are as follows: 1. introduction, 2. thoughts, 3. stress and relaxation, 4. problem-solving, 5. communication, 6. anxiety, 7. behavioral activation, and 8. maintenance. Each module consists of psychoeducational material, examples, and a few exercises. Contents, examples, and exercises have been translated and adapted from Lithuanian to Italian (See Figure 3). Even though this intervention is meant to be conducted on a self-help fashion, participants can contact a therapist throughout the intervention (8 weeks). Communication between participants and therapists takes place over messaging via the SOSteniamoci platform.

Figure 1.

SOSteniamoci welcome page. On the left side users can navigate the introduction menu.

Figure 2.

The eight intervention modules.

Figure 3.

Sample of the intervention's content (module 1). On the left side, users can move among the chapters of the module.

Objective

In this study, we assessed the usability of a self-guided internet-based intervention to support informal caregivers in Italy. Thus, the main goals were (1) to collect both qualitative and quantitative data on users’ performance and satisfaction with the content, interface, and functionality of the SOStenimaoci platform, and (2) to identify usability problems.

Methods

Study design

Given that computerized interactive prototypes detect more usability issues and thus are able to elicit more comments than paper prototypes,²⁶ a computer prototype was adopted to test the platform. Multiple usability evaluation methods were employed to gain a comprehensive understanding of the usability of the SOSteniamoci platform.²⁷ Methods included: 1. task analysis; 2. system usability scale (SUS); and 3. semi-structured interviews. The Consolidated Checklist for Reporting Qualitative Research (COREQ) has been used.²⁸ (See Appendix 1).

Ethical statement

The study was approved by the Istituto Auxologico Italiano in Milan, Italy (2021_12_14_03). All procedures performed in the study were run in accordance with the ethical standards of the institutional and/or national research committee and with the Helsinki Declaration and its later amendments or comparable ethical standards.

Recruitment

Ten Italian informal caregivers were recruited to complete the usability test. Inclusion and exclusion criteria are listed below.

Inclusion criteria were: 1) being 18 years old or over; 2) being an informal caregiver who spent at least three months providing informal care; 3) living in Italy, with the ability to read and type in Italian; 4) having access to a computer, internet, or any other compatible device; 5) being able to complete a videocall interview and use emails; 6) provide informed consent to participate in the study. Exclusion criteria were: 1) reporting any major medical, cognitive, psychiatric, or neurological disorders which would prevent one from adequately participating in this study.

Materials

Sociodemographic characteristics

The sociodemographic characteristics of participants were collected via an online questionnaire. Information was divided into four main areas: a. sociodemographic; b. context of care; c. use of the computer; d. support intervention.

Task analysis

The participants were asked to complete tasks of navigating and using the platform, based on a usability study conducted among older adults with chronic diseases.²⁹ The tasks included a set of navigation tasks, and a set of information search, and simple cognitive tasks (e.g., “enter therapeutic module number one and go to the third paragraph” or “text your therapist using ‘the message and communicate area’”) (“enter module 3 and complete the task filling in the table”). Different performance measures were applied, including task completion time, which consists of the mean time it takes one to complete the task compared to the task completion time of an expert; frequency of error , the total number of errors made on each task (e.g., choosing the wrong button, unable to find and interpret the information provided correctly); and frequency of help requests , which is the total number of times participants asked for assistance while completing the task.

Textbox 1

Tasks.

Please sign-up for the SOSteniamoci platform using the user account name and password I sent you via mail.

Please open module number 1, go to chapter 7, and complete the exercise.

Please go to the evaluation session of module number 1 and complete the evaluation.

Please go to module number 7 and complete the evaluation of module number 7.

Please go to module number 6 and press play to the introduction video.

Please access the “messaging area” and send an email to the therapist.

Please log out of the platform.

System usability scale (SUS)

The SUS, developed by Brooke³⁰ is a common usability scale, comprising 10 items (e.g., “I think I would like to use the system frequently”; “I found the system very cumbersome to use”) that aims to evaluate a complex system (such as a website) and is considered one of the gold standard evaluation measures within the usability field. This scale has been validated in Italian by Borsci and colleagues.³¹ Included items are both positive items and reversed items and are measured using a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The final score items were converted from 0 to 4, in which a higher score reflected higher perceived usability; reversed items were converted, and scores were summed. The final score was later multiplied by 2.5, giving a perceived usability score of 0 to 100. A SUS score above 68 represents above-average usability.³²

Semi-structured interview

Finally, participants took part in a semi-structured interview, which lasted about fifteen-twenty minutes, aimed to assess additional usability and design problems. The interviews were conducted using Zoom, and were audio and video recorded. Three main areas were explored: 1. overall appreciation of the platform; 2. difficulties encountered while navigating the platform and completing the tasks; and 3. positive aspects of the platform.

Textbox 2

Semi-structured interviews questions.

Is the platform easy to navigate?

Is the platform design appealing?

Is the content presented in an engaging, interesting, and simple way?

Is the information provided clear and appropriate?

Does the platform provide relevant and interesting activities?

What do you like most about the platform?

What do you like least about the platform?

Data analysis

Data were analyzed by MS and GM using descriptive statistics and thematic analysis.^33,34 Data from the task navigation performance were extracted from the video recordings, and the means/frequencies were analyzed.

Sample size calculation

Sample size calculation was based on Nielsen's³⁵ and Virzi's³⁶ recommendations, according to which the first five users are sufficient for carrying out usability tests, identifying 95% of the problems that will then be confirmed by subsequent users.³⁷

Procedure

Participants were recruited through direct advertisement and were informed of the aim of the study. Two researchers (MS and AM) scheduled a Zoom meeting with ten informal caregivers. A Zoom link was sent via email. Participants were asked to attend the meeting alone, thus the experiments counted only the participant and the researcher. When opening the meeting, participants were asked to close their cameras and to name their meeting “Subject X”. Before starting the task navigation session, participants were provided with the link to the SOSteniamoci platform, a demo username, and a password. Participants were not allowed to log in before the first navigation task (logging in to the platform). The first part of the meeting consisted of completing seven navigation tasks, during which different measures were taken by the researcher: a. time to complete the task; b. frequency of errors: and c. frequency of help requests. Participants were then allowed to freely navigate the platform for 1 min and 30 s. Subsequently, participants took part in a semi-structured interview that aimed to collect additional feedback regarding further usability problems and design difficulties. All interviews were conducted by two female researchers, MS, a PhD student in clinical psychology and AM, a psychologist and psychotherapist student. Lastly, another link was sent that consisted of questionnaires aiming to collect sociodemographic information and the SUS questionnaire.

Results

Participants

Ten informal caregivers provided feedback on the SOSteniamoci platform. See Tables 1 to 4 for informal caregivers’ sociodemographic information (Table 1); context of care characteristics, which counted for care receiver (CR) demographic characteristics, type of relationship between the informal caregiver and the care receiver, and informal care information (i.e., duration and weekly commitment) (Table 2); computer context, perceived ability using a computer and weekly hours spent using a computer (Table 3), and support context, perceived need of support and experience with online support (Table 4).

Table 1.

Informal caregivers’ sociodemographic characteristics.

Sociodemographic	Value
Gender, n (%)
Female	6 (60%)
Male	4 (40%)
Age (years), M (SD)	47.8 (15.21)
Education (years), M (SD)	14.5 (3.47)
Middle school	1 (10%)
High school	5 (50%)
Bachelor's degree	1 (10%)
Master's degree	2 (20%)
Master II level	1 (10%)

Note. The Italian education system counts two types of Master: 1. First Level Master, which requires a bachelor's degree; and 2. Second Level Master, which requires a master's degree.

Table 2.

Context of care characteristics.

Context of care	Value
CR gender, n (%)
Male	8 (80%)
Female	2 (20%)
CR age (years), M (SD)	78.7 (18.98)
Relationship with the CR, n (%)
Spouse	1 (10%)
Child	7 (70%)
Grandchild	2 (20%)
Hours caring (per week)
1 h—5 h	3 (30%)
5 h—10 h	5 (50%)
10 h—20 h	1 (10%)
> 20 h	1 (10%)
Caregiving duration (years)
< 1 year	4 (40%)
1–5 years	2 (20%)
5–10 years	3 (30%)
> 10 years	1 (10%)
CR disorder
Cardiac illness	2 (20%)
Disability	1 (10%)
Cancer	2 (20%)
Mental disorder	1 (10%)
Dementia	4 (40%)

Note. CR: Care Recipient OR Care Receiver.

Table 3.

Computer context.

Computer context	Value
Hours using a computer (per week)
<10 h	1 (10%)
10 h—20 h	1 (10%)
20 h—30 h	4 (40%)
> 30h	4 (40%)
Familiarity with a computer (Likert 1 to 10)
7	3
8	5
9	2

Table 4.

Support intervention.

Support interventions	Value
Need for support (Yes), N (%)	7 (70%)
Ever experienced online support (Yes), N (%)	0 (0%)

System usability scale

The SUS was administered to evaluate the usability of the SOSteniamoci platform. The platform received a satisfactory usability mean score of 75 (SD = 13.07), ranging from 52.5 to 92.5. Following Bangor and colleagues,³⁸ 20% of participants gave the highest rating possible (had a mean score above or equal to 90), 50% rated it as good, and 30% rated it as OK (see Table 5). The mean score per item ranges from 2.8 to 3.4 of 5. (See Table 6).

Table 5.

From Bangor et al.³⁸

Adjective	SUS cutoff score	Percentage
Worst imaginable	12.5	0%
Awful	20.3	0%
Poor	35.7	0%
OK	50.9	30%
Good	71.4	50%
Excellent	85.5	0%
Best imaginable	90.0	20%

Table 6.

Results of the system usability scale.

Item	P1^a	P2	P3	P4	P5	P6	P7	P8	P9	P10
1	2	3	5	4	5	4	2	5	4	4
2	2	2	3	1	1	2	2	2	1	2
3	4	4	2	4	5	4	3	4	5	4
4	2	2	5	1	2	3	3	4	1	2
5	3	3	5	4	5	4	4	4	4	4
6	4	1	3	2	1	1	1	2	1	1
7	4	4	5	4	5	4	3	4	4	4
8	4	1	3	1	1	1	2	2	1	1
9	3	4	3	4	4	4	4	3	4	4
10	3	2	3	4	2	1	2	1	1	1
Odd Score	26	18	20	20	24	20	16	24	21	20
X	11	13	15	15	19	15	11	19	16	15
Even Score	15	8	17	9	7	8	10	13	5	7
Y	10	17	8	16	18	17	15	12	20	18
X + Y = Z	21	30	23	31	37	32	26	31	36	33
Z*2,5	52.5	75	57,5	77,5	92,5	80	65	77,5	90	82,5

P1-10: participant 1-10.

Item: 1. I think that I would like to use this intervention; 2. I found the intervention unnecessarily complex; 3. I found the intervention easy to use; 4. I think I would need support from a technical person to use this intervention; 5. I found the various functions in this intervention to be well integrated; 6. I thought there was too much inconsistency in this intervention; 7. I would imagine that most people would learn to use this intervention quickly; 8. I found the intervention very cumbersome to use; 9. I felt very confident using the intervention; 10. I needed to learn a lot of things before I could start using this intervention.

Task analysis

Participants were asked to complete seven navigation tasks. Frequency of errors, frequency of helps requests, and time to complete each task were measured. Table 7 presents data for each navigation task (see in bold text from 1 to 7) including the time taken to complete the task (Time), the number of errors made during task completion (e.g., clicking the wrong button or opening the wrong module) (Errors), and the number of requests for assistance made to complete the task (e.g., asking for suggestions or hints). The mean time to complete each navigation task (Time) was compared to the time taken by an expert researcher familiar with the platform (Time Exp).Task 2 and Task 6 registered a higher frequency of errors (14, 7). Task 2, Task 3, Task 4, and Task 6 registered a higher frequency of g help requests (15, 10, 12, 15). Accordingly, the time to complete the task , in Task 2, Task 3, Task 4, and Task 6 showed a wider gap between participants’ mean time and an expert's time to complete the task, compared to other tasks. (See Table 7).

Table 7.

Task analysis scores and results.

	CG. 1	CG. 2	CG. 3	CG. 4	CG. 5	CG.6	CG. 7	CG. 8	CG. 9	CG. 10	Total	Time Exp.
	1. Please sign-up for the SOSteniamoci platform using the user account name and password I sent you via mail.
Task 1
Time	00:35	00:45	00:10.31	00:19	00:52	00:49	00:22	00:45	00:14	00:19	00:31	00:17
Errors	0	1	0	0	0	0	0	0	0	0	1
Help	0	0	0	0	1	0	1	0	0	0	2
	2. Please open module number 1, go to chapter 7, and complete the exercise.
Task 2
Time	02:58	02:19	03:02	03:00	01:01	01:05	00:40	02:06	1:19	01:31	01:55	00:36
Errors	1	1	2	2	1	1	1	2	1	2	14
Help	4	2	2	4	0	0	0	1	1	1	15
	3. Please go to the evaluation session of module number 1 and complete the evaluation.
Task 3
Time	02:54	01:03	01:38	02:52	01:01	01:00	00:43	01:05	00:29	00:35	01:20	00:23
Errors	0	0	0	0	0	0	0	0	1	0	1
Help	2	1	1	5	0	0	0	0	1	0	10
	4. Please go to module number 7 and complete the evaluation of module number 7.
Task 4
Time	01:24	01:08	01:09	02:56	01:10	01:11	00:26	00:32	01:05	00:57	01:11	00:35
Errors	0	0	0	2	0	0	0	0	2	0	4
Help	3	0	2	3	1	1	0	0	1	1	12
	5. Please go to module number 6 and press play to the introduction video.
Task 5
Time	00:15	00:43	00:13	00:12	00:04	00:05	00:05	00:07	00:04	00:10	00:11	00:08
Errors	0	0	0	0	0	0	0	0	0	0	0
Help	0	1	0	0	0	0	0	0	0	0	1
	6. Please access the “messaging area” and send an email to the therapist.
Task 6
Time	02:02	00:58	02:52	01:24	00:54	00:58	00:24	01:14	00:32	00:35	01:11	00:22
Errors	1	0	1	1	0	0	0	1	1	2	7
Help	6	0	1	1	2	2	0	2	1	0	15
	7. Please log out of the platform.
Task 7
Time	00:19	00:05	00:03	00:03	00:02	00:03	00:03	00:03	00:09	00:03	00:05	00:02
Errors	0	0	0	0	0	0	0	0	0	0	0
Help	0	0	0	0	0	0	0	0	0	0	0

Responses to the semi-structured interview using thematic analysis

Thematic analysis was employed to summarize recurring patterns across participants.^33,34 We used open coding, and seventeen codes emerged. Codes that were identified among more than one participant were clustered into themes. Codes and themes were discussed and agreed upon by two researchers (MS and GM).

As shown in Table 8, three main themes emerged from the analysis: the quality of the platform, the desire to change the platform, and difficulties in using the platform. (Codes for each theme are presented in Table 8).

Table 8.

Themes and codes.

Theme	Code
General thoughts about and strengths of the platform	Interesting platform Simple platform Clear platform Nice platform Pleasant platform Light platform Useful platform Effective platform Usable platform Platform rich in content
Suggestions for improvements	Suggestion for enlarging and clarifying the introduction part Suggestion for facilitating the contact with the therapist Suggestion for reducing the length of the modules Suggestion for changing the color of the platform for something more engaging
Difficulties that emerged when navigating the platform	Difficulties in understanding the intervention at the beginning Difficulties in accessing the platform Difficulties in finding the chat area

Theme 1 – general thoughts about and strengths of the platform

Users generally appreciated the platform, focusing on its usability and intuitiveness. Contents, videos, and pictures were found particularly engaging and interesting.

“I really appreciate the fact that there is a lot of multimedia content such as videos and pictures. It makes the platform more engaging and intuitive.” (Participant 4)

“I believe this platform is very intuitive and easy to use. Even though I did not have the chance to go through all the modules and contents, I can say that it seems very useful, clear, and helpful for informal caregivers. I particularly like the fact that you can log in and complete exercises whenever you want.” (Participant 3)

“The contents seem very interesting and clear, just upon taking a look at the module titles, and I like the fact that there are a lot of examples.” (Participant 8)

Theme 2 – suggestions for improvement

A few suggestions for improving the platform emerged during the interview. Suggestions were mainly related to the difficulties encountered.

“I’d like an introduction or something that would better explain at the beginning what this intervention is about, and what to expect from this intervention. Particularly, I would add instructions about how to use the platform itself.” (Participant 4)

“I don’t know, maybe you should add a bigger button with a text icon or explain how to contact your therapist at the beginning. Let's make it more intuitive.” (Participant 10)

Theme 3 – difficulties that emerged when navigating the platform

The main difficulties that emerged when navigating the platform pertained to the introduction part of the platform and the use of the chat area. Particularly, participants had doubts when logging in and contacting the therapist.

“It was difficult to understand how to log in at the beginning. Especially because after the login I found myself I bit lost.” (Participant 1)

“The option of sending a message to the therapist, maybe, is not immediately visible. Indeed, it was a bit frustrating not to find it immediately and to keep on navigating.” (Participant 5)

Discussion

Main findings

In this study we aimed to assess end-users’ performance and satisfaction with the translated and adapted internet-based intervention platform “SOSteniamoci” to support informal caregivers in Italy. Digital or eHealth solutions are usually designed by researchers, and it is rare that inputs from end-users are collected during the development process. Poor usability of such solutions or interventions may result in user frustration and irritation, leading to limited acceptability among the end-user population.³⁹ Findings from the current study strengthen the importance of testing the usability of digital solutions in the realm of psychology before delivering them. In alignment with prior research, our findings reinforced the significance of crafting precise and succinct content tailored to the target population, incorporating media elements such as images and videos, and adopting a straightforward design. These factors emerged as pivotal and indispensable considerations in the development or adaptation of internet-based interventions.⁴⁰ Although the current internet-based intervention had previously demonstrated efficacy in a study supporting informal caregivers in Lithuania,²⁵ a few minor adjustments were identified and incorporated based on participants’ feedback and reviews. This underscores the importance of cultural adaptation. Past research emphasizes the necessity of conducting usability tests when translating websites into different languages, as literal translations can lead to misunderstandings and analogies may not effectively carry over.^41,42 Despite efforts to ensure translation accuracy, some analogies developed in Lithuanian lost their intended meaning in Italian and required adaptation. Additionally, as suggested by end-users in this study, adjustments to content, not just language, may be necessary. Indeed, data collected during our usability test indicated a need for a few changes unrelated to translation accuracy.

For example, the introduction section was enlarged to better explain what internet-based interventions are and how they work, as suggested during the interviews. Unlike the Lithuanian internet-based intervention, the SOSteniamoci platform counts of more media contents (such as images and videos) to increase Italian participants’ engagement and adherence. To keep a website engaging, as suggested by other studies, it is important to incorporate multiple methods to deliver information.^43,44 Moreover, our findings encourage the use of media (such as video or pictures) to better engage participants in the intervention and allow for a pleasant interaction with the platform. In conclusion, the usability test of the SOSteniamoci platform revealed positive feedback from participants, indicating that the platform is user-friendly, relevant in terms of content, and engaging. However, these findings underscore the importance of considering cultural differences in the adaptation of internet-based interventions. This consideration is crucial to ensure participants’ sustained engagement and adherence to treatment while preventing potential frustrations.

Future directions

Following the usability testing and the changes implemented on the basis of the findings in this study, the SOSteniamoci platform will be studied for feasibility and efficacy. A pilot study will be conducted with informal caregivers in Italy, randomizing participants into two groups: a. a study group using the SOSteniamoci platform and completing the 8 modules; and b. a control group (i.e., a wait-list group). Comparisons of caregiver burden, anxiety, depression, stress, quality of life, quality of the relationship, and inclusion of the illnesses in the self will be assessed between the two groups at the beginning and the end of the study.

Limitations

A few limitations need to be considered. First, the controlled testing environment of the study might have influenced participants’ responses. The presence of the researcher throughout the testing session may have led to overly positive feedback and lower criticism toward the SOSteniamoci platform. Second, the lab-based setting resulted in testing the platform in front of a researcher, even though it was conducted online. Indeed, even if it was demonstrated that it is possible to rely on investigating usability in a lab setting,⁴⁵ this method may not represent a real-world setting. Thus, we may have missed important information about the use of the platform in a more ecological setting – that is, a setting in which users are really using the platform, such as in their apartments.

Conclusion

To enhance participants’ engagement, adherence to treatment, and prevent drop-out, it is necessary to test the usability of the intervention platform. Data from our study demonstrated that the SOSteniamoci platform is user-friendly, intuitive, and easy to use. Important areas of improvement were provided by users, and the platform was modified accordingly. Even though results may be relatively limited in terms of their generalizability to the broader informal caregiver population, preliminary findings from our study might help in the development of future internet-based interventions and their usability tests, and may potentially influence the design of existing interventions.

Footnotes

Abbreviations

Acknowledgements

MS would like to thank AM for her contribution to the study and the informal caregivers who invested their time participating in this study.

Contributorship

MS and NV conceptualized and designed the study. MS and AM coordinated the study recruitment and conducted the interviews. MS and GM coded and analyzed the interviews under the supervision of NV. All authors participated in the critical revision of the manuscript and approved the final manuscript.

Declaration of conflicting interests

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Funding

This study was undertaken as part of a PhD, thesis. The PhD was funded by EC-funded Marie Sklodowska-Curie Innovative Training Network (H2020-MSCA-ITN-2018), Grant Agreement No. 814072. The funder has not had any role in the preparation of the manuscript.

Guarantor

MS.

ORCID iDs

Michelle Semonella

Giada Pietrabissa

References

Revenson

Griva

Luszczynska

, et al. Caregiving as a dyadic process. In Caregiving in the illness context, 2016, pp. 25–37. London: Springer.

Organization

. Preventing noncommunicable diseases (NCDs) by reducing environmental risk factors . 2017.

Hassan

AYI

Lamura

Hagedoorn

. Predictors of digital support services use by informal caregivers: a cross-sectional comparative survey. BMJ open 2022; 12: e059897.

Lam

WWY

Nielsen

Sprigg

, et al. The demands and resources of working informal caregivers of older people: a systematic review. Work & Stress 2022; 36: 105–127.

Zhang

, et al. The effectiveness of e-health interventions on caregiver burden, depression, and quality of life in informal caregivers of patients with cancer: a systematic review and meta-analysis of randomized controlled trials. Int J Nurs Stud 2022; 127: 104179.

Sun

Leng

, et al. Comparative efficacy of 11 non-pharmacological interventions on depression, anxiety, quality of life, and caregiver burden for informal caregivers of people with dementia: a systematic review and network meta-analysis. Int J Nurs Stud 2022; 129: 104204.

Zwar

König

H-H

Hajek

. Gender differences in mental health, quality of life, and caregiver burden among informal caregivers during the second wave of the COVID-19 pandemic in Germany: a representative, population-based study. Gerontology 2022; 69: 149–162.

Corry

While

Neenan

, et al. A systematic review of systematic reviews on interventions for caregivers of people with chronic conditions. J Adv Nurs 2015; 71: 718–734.

Davies

Brini

Hirani

, et al. The impact of assistive technology on burden and psychological well-being in informal caregivers of people with dementia (ATTILA study). Alzheimer's & Dementia: Translational Res Clin Interv 2020; 6: e12064.

10.

Eysenbach

Patt

Houston

, et al. Impact Factor (2018): 4.945-ranked# 1 medical informatics journal by Impact Factor Volume 5 (2003), Issue 2 ISSN: 1438-8871 Editor in Chief: Gunther Eysenbach, MD, MPH. Journal of Medical Internet Research, 2003; 5(2).

11.

Hassan

AYI

. Challenges and recommendations for the deployment of information and communication technology solutions for informal caregivers: scoping review. JMIR Aging 2020; 3: e20310.

12.

Vlaescu

Alasjö

Miloff

, et al. Features and functionality of the iterapi platform for internet-based psychological treatment. Internet Interv 2016; 6: 107–114.

13.

Andersson

Titov

. Advantages and limitations of internet-based interventions for common mental disorders. World Psychiatry 2014; 13: 4–11.

14.

Andersson

Carlbring

Ljótsson

, et al. Guided internet-based CBT for common mental disorders. J Contemp Psychother 2013; 43: 223–233.

15.

Hedman-Lagerlöf

Carlbring

Svärdman

, et al. Therapist-supported Internet-based cognitive behaviour therapy yields similar effects as face-to-face therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. World Psychiatry 2023; 22(2): 305–314..

16.

Andersson

. Internet interventions: past, present and future. 2018.

17.

Jimison

Gorman

Woods

, et al. Barriers and drivers of health information technology use for the elderly, chronically ill, and underserved. Evid Rep Technol Assess (Full Rep) 2008; 175: 1–1422.

18.

Karsh

B-T

. A systematic review of patient acceptance of consumer health information technology. J Am Med Inform Assoc 2009; 16: 550–560.

19.

Wilks

Yin

Zuromski

. User experience affects dropout from internet-delivered dialectical behavior therapy. Telemedicine and E-Health 2020; 26: 794–797.

20.

Chen

Slattery

Tomasino

, et al. Challenges and benefits of an internet-based intervention with a peer support component for older adults with depression: qualitative analysis of textual data. J Med Internet Res 2020; 22: e17586–e17586.

21.

Henshall

Davey

Jacelon

, et al. A usability study to test the effectiveness, efficiency and simplicity of a newly developed internet-based exercise-focused health app for lung cancer survivors (iEXHALE): protocol paper. Health Informatics J 2020; 26: 1431–1442.

22.

Andersson

Carlbring

Berger

, et al.

What makes internet therapy work?

Cogn Behav Ther 2009b; 38: 55–60.

23.

Semonella

Vilchinsky

Dakel

, et al. SOSteniamoci: An internet-based intervention to support informal caregivers. PSYCHOBIT, 2020. September 28-29, Naples, IT.

24.

Andersson

Carlbring

Berger

, et al.

What makes internet therapy work?

Cogn Behav Ther 2009a; 38: 55–60.

25.

Biliunaite

Kazlauskas

Sanderman

, et al. Internet-based cognitive behavioral therapy for informal caregivers: randomized controlled pilot trial. J Med Internet Res 2021; 23: e21466.

26.

Liu

Khooshabeh

. Paper or interactive? A study of prototyping techniques for ubiquitous computing environments. CHI'03 extended abstracts on Human factors in computing systems, 2003.

27.

Davis

Gardner

Schnall

. A review of usability evaluation methods and their use for testing eHealth HIV interventions. Curr HIV/AIDS Rep 2020; 17: 203–218.

28.

Tong

Sainsbury

Craig

. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007; 19: 349–357.

29.

Tao

. Usability study of a computer-based self-management system for older adults with chronic diseases. JMIR Res Protoc 2012; 1: e2184.

30.

Brooke

. Usability evaluation in industry, chap. SUS: a “quick and dirty” usability scale. London, UK: Taylor and Francis, 1996.

31.

Borsci

Federici

Lauriola

. On the dimensionality of the system usability scale: a test of alternative measurement models. Cogn Process 2009; 10: 193–197.

32.

Brooke

. SUS: A quick and dirty usability scale. Usability Eval. Ind. 1996 189(3), 189-194.

33.

Boyatzis

. Transforming qualitative information: Thematic analysis and code development, 1998. Thousand Oaks, CA: Sage.

34.

Marks

Yardley

. Research methods for clinical and health psychology, 2004. London: Sage.

35.

Nielsen

. Usability Engineering, 1994. San Francisco, CA: Morgan Kaufmann.

36.

Virzi

. Refining the test phase of usability evaluation: how many subjects is enough? Hum Factors 1992; 34: 457–468.

37.

Faulkner

. Beyond the five-user assumption: benefits of increased sample sizes in usability testing. Behav Res Methods Instrum Comput 2003; 35: 379–383.

38.

Bangor

Kortum

Miller

. Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 2009; 4: 114–123.

39.

Wichansky

. Usability testing in 2000 and beyond. Ergonomics 2000; 43: 998–1006.

40.

Breakey

Warias

Ignas

, et al. The value of usability testing for internet-based adolescent self-management interventions: “managing hemophilia online”. BMC Med Inform Decis Mak 2013; 13: 113.

41.

Salamanca-Sanabria

Richards

Timulak

. Adapting an internet-delivered intervention for depression for a Colombian college student population: an illustration of an integrative empirical approach. Internet Interv 2019; 15: 76–86.

42.

Verkleij

Georgiopoulos

Friedman

. Development and evaluation of an internet-based cognitive behavioral therapy intervention for anxiety and depression in adults with cystic fibrosis (eHealth CF-CBT): an international collaboration. Internet Interv 2021; 24: 100372.

43.

Baltierra

Muessig

Pike

, et al. More than just tracking time: complex measures of user engagement with an internet-based health promotion intervention. J Biomed Inform 2016; 59: 299–307.

44.

Blasi

Gobbo

Sedita

. Smart cities and citizen engagement: evidence from twitter data analysis on Italian municipalities. Journal of Urban Management 2022; 11: 153–165.

45.

Jake-Schoffman

Silfee

Waring

, et al. Methods for evaluating the content, usability, and efficacy of commercial mobile health apps. JMIR Mhealth Uhealth 2017; 5: e8758.

Usability study of SOSteniamoci: An internet-based intervention platform to support informal caregivers in Italy

Abstract

Background

Objective

Methods

Results

Conclusion

Keywords

Introduction

Overview of the SOSteniamoci platform to support informal caregivers in Italy

Objective

Methods

Study design

Ethical statement

Recruitment

Materials

Sociodemographic characteristics

Task analysis

System usability scale (SUS)

Semi-structured interview

Data analysis

Sample size calculation

Procedure

Results

Participants

System usability scale

Task analysis

Responses to the semi-structured interview using thematic analysis

Theme 1 – general thoughts about and strengths of the platform

Theme 2 – suggestions for improvement

Theme 3 – difficulties that emerged when navigating the platform

Discussion

Main findings

Future directions

Limitations

Conclusion

Footnotes

Abbreviations

Acknowledgements

Contributorship

Declaration of conflicting interests

Funding

Guarantor

ORCID iDs

References