Abstract
Background
Providing informal care can be experienced as stressful and lead to caregiver burden. Internet-based interventions, a specific form of eHealth, have proven to be a good option to support informal caregivers. SOSteniamoci, an internet-based intervention already tested in Lithuania, was translated and adapted for Italian caregivers.
Objective
As many novel eHealth solutions have been rejected by end-users due to usability problems, we aimed to evaluate the usability of the adapted platform, using a computer-based prototype.
Methods
The following methods and metrics were applied: 1. task analysis, using audio and video recordings that included three usability metrics: task completion rate, frequency of errors, and frequency of help requests; 2. the system usability scale (SUS); and 3. a semi-structured interview to collect additional data about the system's design and overall satisfaction.
Results
Ten informal caregivers (60% female; age M = 47.8, SD = 15.21) provided insights and suggestions for increasing the usability of the platform. The platform was considered satisfactory, with a mean score on the SUS of 75 (SD = 13.07) out of 100. The task analysis measurements highlighted difficulties in how to log in to the platform, understanding what the intervention is about, and texting the therapist. The same difficulties were also mentioned during the post-experience interview. Thus, improvements were subsequently made to enhance users’ experience when navigating the platform. Finally, the platform overall was found to be intuitive and friendly, and the contents were appreciated.
Conclusion
To maintain participants’ engagement and prevent drop-out, it is crucial to test the usability of internet-based interventions. Even though the platform proved to be user-friendly, intuitive and easy to use, several enhancements were implemented based on participants’ feedback. Thus, the usability of internet-based interventions should be tested, and end-users must be involved in the development process of such solutions
Keywords
Introduction
Informal care is generally defined as unpaid care provided to a relative or friend who is coping with a disability, chronic illness, or other care needs. 1 This phenomenon is growing as the number of adults living with chronic conditions is increasing. 2 Indeed, it is estimated that 80% of all care in Europe is provided by informal caregivers, and the number of informal caregivers ranges from 10% up to 25% of the total European population. 3 Thus, informal caregivers can be considered the backbone of long-term care in Europe. Nevertheless, it is well-known that providing informal care may lead to several health consequences, lack of time, financial strain, and thus to caregiver burden.4–7
Recognizing the negative health impacts on informal caregivers has led to the development of different psychological support interventions and solutions, which have been found to be beneficial. 8 Lately, and mostly due to the COVID-19 pandemic, using technology to achieve positive outcomes has received interest, in the informal care context as well as in the formal care context.9–11
The use of eHealth interventions is growing, mostly because the use of overall internet-based technology in people's daily lives is growing and its affordability is increasing. In the field of clinical psychology, delivering online psychological interventions may reduce geographical and financial barriers that might otherwise prevent potential participants from engaging in such interventions, and thus increase their scalability. 12 In this regard, internet-based interventions, a specific form of eHealth interventions in which therapeutic tasks based on cognitive behavioral principles are delivered via computerized systems, 13 have been found to be as effective as face-to-face interventions.14,15 Hence, such interventions provide a solution that might be complementary, or even an alternative, to face-to-face treatment. 16
Despite the well-proven beneficial aspects of eHealth solutions and interventions, many innovations are rejected by stakeholders mostly due to usability problems such as incomplete content, ineffective system design, and lack of ease of use, causing a mismatch between the systems and users’ needs, characteristics, and expectations.17–19 Frustrations and irritations may emerge as a result of poor usability, generating inefficiency and a higher likelihood of committing errors 20 and dropping out. To avoid these negative aspects, testing the system's usability in advance is highly recommended. 21
The current study is part of a larger project titled SOSteniamoci, which consists of the assessment of an internet-based intervention, based on cognitive-behavioral therapy principles, 22 for supporting informal caregivers in Italy. 23 As every internet-based cognitive behavioral therapy, what is usually referred in face-to-face treatment as “session”, it is called “module” in internet treatment. The duration of the treatment is defined beforehand, and it typically may last between 5 and 15 weeks (SOSteniamoci counts of 8-weeks modules), and it starts with psychoeducational material and ends with relapse prevention. 24 The aim of this psychological intervention is to reduce caregiver burden, depression, and anxiety while improving quality of life and relationship satisfaction. 23 The intervention platform allows users to access the platform, read weekly modules, complete the evaluation at the end of each module, complete weekly assignments, and be in contact with a therapist. The assessment records are saved in the platform and can be retrieved for review. This intervention has already been tested in Lithuania 25 ; as such, SOSteniamoci is an implementation and adaptation of an existing intervention. Although the intervention has already proven to be efficient as well as beneficial for informal caregivers in Lithuania, it is very important to tailor and adapt its contents and materials to a specific population with a different cultural background.
Hence, our aim in the current study was to test the usability of the adapted Italian version of the platform among Italian caregivers, and as such, to broaden our understanding and knowledge about the usability and cultural adaptation of internet-based solutions overall.
Overview of the SOSteniamoci platform to support informal caregivers in Italy
SOSteniamoci is a translated and adapted internet-based intervention that aims to support informal caregivers in Italy (See Figure 1). 23 It consists of eight weekly modules delivered over a period of two months, with each module dedicated to a different theme (See Figure 2). The themes are as follows: 1. introduction, 2. thoughts, 3. stress and relaxation, 4. problem-solving, 5. communication, 6. anxiety, 7. behavioral activation, and 8. maintenance. Each module consists of psychoeducational material, examples, and a few exercises. Contents, examples, and exercises have been translated and adapted from Lithuanian to Italian (See Figure 3). Even though this intervention is meant to be conducted on a self-help fashion, participants can contact a therapist throughout the intervention (8 weeks). Communication between participants and therapists takes place over messaging via the SOSteniamoci platform.
SOSteniamoci welcome page. On the left side users can navigate the introduction menu.
The eight intervention modules.
Sample of the intervention's content (module 1). On the left side, users can move among the chapters of the module.
Objective
In this study, we assessed the usability of a self-guided internet-based intervention to support informal caregivers in Italy. Thus, the main goals were (1) to collect both qualitative and quantitative data on users’ performance and satisfaction with the content, interface, and functionality of the SOStenimaoci platform, and (2) to identify usability problems.
Methods
Study design
Given that computerized interactive prototypes detect more usability issues and thus are able to elicit more comments than paper prototypes, 26 a computer prototype was adopted to test the platform. Multiple usability evaluation methods were employed to gain a comprehensive understanding of the usability of the SOSteniamoci platform. 27 Methods included: 1. task analysis; 2. system usability scale (SUS); and 3. semi-structured interviews. The Consolidated Checklist for Reporting Qualitative Research (COREQ) has been used. 28 (See Appendix 1).
Ethical statement
The study was approved by the Istituto Auxologico Italiano in Milan, Italy (2021_12_14_03). All procedures performed in the study were run in accordance with the ethical standards of the institutional and/or national research committee and with the Helsinki Declaration and its later amendments or comparable ethical standards.
Recruitment
Ten Italian informal caregivers were recruited to complete the usability test. Inclusion and exclusion criteria are listed below.
Inclusion criteria were: 1) being 18 years old or over; 2) being an informal caregiver who spent at least three months providing informal care; 3) living in Italy, with the ability to read and type in Italian; 4) having access to a computer, internet, or any other compatible device; 5) being able to complete a videocall interview and use emails; 6) provide informed consent to participate in the study. Exclusion criteria were: 1) reporting any major medical, cognitive, psychiatric, or neurological disorders which would prevent one from adequately participating in this study.
Materials
Sociodemographic characteristics
The sociodemographic characteristics of participants were collected via an online questionnaire. Information was divided into four main areas: a. sociodemographic; b. context of care; c. use of the computer; d. support intervention.
Task analysis
The participants were asked to complete tasks of navigating and using the platform, based on a usability study conducted among older adults with chronic diseases.
29
The tasks included a set of navigation tasks, and a set of information search, and simple cognitive tasks (e.g., “enter therapeutic module number one and go to the third paragraph” or “text your therapist using ‘the message and communicate area’”) (“enter module 3 and complete the task filling in the table”). Different performance measures were applied, including
Tasks.
Please sign-up for the SOSteniamoci platform using the user account name and password I sent you via mail. Please open module number 1, go to chapter 7, and complete the exercise. Please go to the evaluation session of module number 1 and complete the evaluation. Please go to module number 7 and complete the evaluation of module number 7. Please go to module number 6 and press play to the introduction video. Please access the “messaging area” and send an email to the therapist. Please log out of the platform.
System usability scale (SUS)
The SUS, developed by Brooke 30 is a common usability scale, comprising 10 items (e.g., “I think I would like to use the system frequently”; “I found the system very cumbersome to use”) that aims to evaluate a complex system (such as a website) and is considered one of the gold standard evaluation measures within the usability field. This scale has been validated in Italian by Borsci and colleagues. 31 Included items are both positive items and reversed items and are measured using a Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). The final score items were converted from 0 to 4, in which a higher score reflected higher perceived usability; reversed items were converted, and scores were summed. The final score was later multiplied by 2.5, giving a perceived usability score of 0 to 100. A SUS score above 68 represents above-average usability. 32
Semi-structured interview
Finally, participants took part in a semi-structured interview, which lasted about fifteen-twenty minutes, aimed to assess additional usability and design problems. The interviews were conducted using Zoom, and were audio and video recorded. Three main areas were explored: 1. overall appreciation of the platform; 2. difficulties encountered while navigating the platform and completing the tasks; and 3. positive aspects of the platform.
Semi-structured interviews questions.
Is the platform easy to navigate? Is the platform design appealing? Is the content presented in an engaging, interesting, and simple way? Is the information provided clear and appropriate? Does the platform provide relevant and interesting activities? What do you like most about the platform? What do you like least about the platform?
Data analysis
Data were analyzed by MS and GM using descriptive statistics and thematic analysis.33,34 Data from the task navigation performance were extracted from the video recordings, and the means/frequencies were analyzed.
Sample size calculation
Sample size calculation was based on Nielsen's 35 and Virzi's 36 recommendations, according to which the first five users are sufficient for carrying out usability tests, identifying 95% of the problems that will then be confirmed by subsequent users. 37
Procedure
Participants were recruited through direct advertisement and were informed of the aim of the study. Two researchers (MS and AM) scheduled a Zoom meeting with ten informal caregivers. A Zoom link was sent via email. Participants were asked to attend the meeting alone, thus the experiments counted only the participant and the researcher. When opening the meeting, participants were asked to close their cameras and to name their meeting “Subject X”. Before starting the task navigation session, participants were provided with the link to the SOSteniamoci platform, a demo username, and a password. Participants were not allowed to log in before the first navigation task (logging in to the platform). The first part of the meeting consisted of completing seven navigation tasks, during which different measures were taken by the researcher: a. time to complete the task; b. frequency of errors: and c. frequency of help requests. Participants were then allowed to freely navigate the platform for 1 min and 30 s. Subsequently, participants took part in a semi-structured interview that aimed to collect additional feedback regarding further usability problems and design difficulties. All interviews were conducted by two female researchers, MS, a PhD student in clinical psychology and AM, a psychologist and psychotherapist student. Lastly, another link was sent that consisted of questionnaires aiming to collect sociodemographic information and the SUS questionnaire.
Results
Participants
Ten informal caregivers provided feedback on the SOSteniamoci platform. See Tables 1 to 4 for informal caregivers’ sociodemographic information (Table 1); context of care characteristics, which counted for care receiver (CR) demographic characteristics, type of relationship between the informal caregiver and the care receiver, and informal care information (i.e., duration and weekly commitment) (Table 2); computer context, perceived ability using a computer and weekly hours spent using a computer (Table 3), and support context, perceived need of support and experience with online support (Table 4).
Informal caregivers’ sociodemographic characteristics.
Note. The Italian education system counts two types of Master: 1. First Level Master, which requires a bachelor's degree; and 2. Second Level Master, which requires a master's degree.
Context of care characteristics.
Note. CR: Care Recipient OR Care Receiver.
Computer context.
Support intervention.
System usability scale
The SUS was administered to evaluate the usability of the SOSteniamoci platform. The platform received a satisfactory usability mean score of 75 (SD = 13.07), ranging from 52.5 to 92.5. Following Bangor and colleagues, 38 20% of participants gave the highest rating possible (had a mean score above or equal to 90), 50% rated it as good, and 30% rated it as OK (see Table 5). The mean score per item ranges from 2.8 to 3.4 of 5. (See Table 6).
From Bangor et al. 38
Results of the system usability scale.
P1-10: participant 1-10.
Item: 1. I think that I would like to use this intervention; 2. I found the intervention unnecessarily complex; 3. I found the intervention easy to use; 4. I think I would need support from a technical person to use this intervention; 5. I found the various functions in this intervention to be well integrated; 6. I thought there was too much inconsistency in this intervention; 7. I would imagine that most people would learn to use this intervention quickly; 8. I found the intervention very cumbersome to use; 9. I felt very confident using the intervention; 10. I needed to learn a lot of things before I could start using this intervention.
Task analysis
Participants were asked to complete seven navigation tasks. Frequency of errors, frequency of helps requests, and time to complete each task were measured. Table 7 presents data for each navigation task (see in bold text from 1 to 7) including the time taken to complete the task (Time), the number of errors made during task completion (e.g., clicking the wrong button or opening the wrong module) (Errors), and the number of requests for assistance made to complete the task (e.g., asking for suggestions or hints). The mean time to complete each navigation task (Time) was compared to the time taken by an expert researcher familiar with the platform (Time Exp).Task 2 and Task 6 registered a higher
Task analysis scores and results.
Responses to the semi-structured interview using thematic analysis
Thematic analysis was employed to summarize recurring patterns across participants.33,34 We used open coding, and seventeen codes emerged. Codes that were identified among more than one participant were clustered into themes. Codes and themes were discussed and agreed upon by two researchers (MS and GM).
As shown in Table 8, three main themes emerged from the analysis: the quality of the platform, the desire to change the platform, and difficulties in using the platform. (Codes for each theme are presented in Table 8).
Themes and codes.
Theme 1 – general thoughts about and strengths of the platform
Users generally appreciated the platform, focusing on its usability and intuitiveness. Contents, videos, and pictures were found particularly engaging and interesting.
“I really appreciate the fact that there is a lot of multimedia content such as videos and pictures. It makes the platform more engaging and intuitive.” (Participant 4)
“I believe this platform is very intuitive and easy to use. Even though I did not have the chance to go through all the modules and contents, I can say that it seems very useful, clear, and helpful for informal caregivers. I particularly like the fact that you can log in and complete exercises whenever you want.” (Participant 3)
“The contents seem very interesting and clear, just upon taking a look at the module titles, and I like the fact that there are a lot of examples.” (Participant 8)
Theme 2 – suggestions for improvement
A few suggestions for improving the platform emerged during the interview. Suggestions were mainly related to the difficulties encountered.
“I’d like an introduction or something that would better explain at the beginning what this intervention is about, and what to expect from this intervention. Particularly, I would add instructions about how to use the platform itself.” (Participant 4)
“I don’t know, maybe you should add a bigger button with a text icon or explain how to contact your therapist at the beginning. Let's make it more intuitive.” (Participant 10)
Theme 3 – difficulties that emerged when navigating the platform
The main difficulties that emerged when navigating the platform pertained to the introduction part of the platform and the use of the chat area. Particularly, participants had doubts when logging in and contacting the therapist.
“It was difficult to understand how to log in at the beginning. Especially because after the login I found myself I bit lost.” (Participant 1)
“The option of sending a message to the therapist, maybe, is not immediately visible. Indeed, it was a bit frustrating not to find it immediately and to keep on navigating.” (Participant 5)
Discussion
Main findings
In this study we aimed to assess end-users’ performance and satisfaction with the translated and adapted internet-based intervention platform “SOSteniamoci” to support informal caregivers in Italy. Digital or eHealth solutions are usually designed by researchers, and it is rare that inputs from end-users are collected during the development process. Poor usability of such solutions or interventions may result in user frustration and irritation, leading to limited acceptability among the end-user population. 39 Findings from the current study strengthen the importance of testing the usability of digital solutions in the realm of psychology before delivering them. In alignment with prior research, our findings reinforced the significance of crafting precise and succinct content tailored to the target population, incorporating media elements such as images and videos, and adopting a straightforward design. These factors emerged as pivotal and indispensable considerations in the development or adaptation of internet-based interventions. 40 Although the current internet-based intervention had previously demonstrated efficacy in a study supporting informal caregivers in Lithuania, 25 a few minor adjustments were identified and incorporated based on participants’ feedback and reviews. This underscores the importance of cultural adaptation. Past research emphasizes the necessity of conducting usability tests when translating websites into different languages, as literal translations can lead to misunderstandings and analogies may not effectively carry over.41,42 Despite efforts to ensure translation accuracy, some analogies developed in Lithuanian lost their intended meaning in Italian and required adaptation. Additionally, as suggested by end-users in this study, adjustments to content, not just language, may be necessary. Indeed, data collected during our usability test indicated a need for a few changes unrelated to translation accuracy.
For example, the introduction section was enlarged to better explain what internet-based interventions are and how they work, as suggested during the interviews. Unlike the Lithuanian internet-based intervention, the SOSteniamoci platform counts of more media contents (such as images and videos) to increase Italian participants’ engagement and adherence. To keep a website engaging, as suggested by other studies, it is important to incorporate multiple methods to deliver information.43,44 Moreover, our findings encourage the use of media (such as video or pictures) to better engage participants in the intervention and allow for a pleasant interaction with the platform. In conclusion, the usability test of the SOSteniamoci platform revealed positive feedback from participants, indicating that the platform is user-friendly, relevant in terms of content, and engaging. However, these findings underscore the importance of considering cultural differences in the adaptation of internet-based interventions. This consideration is crucial to ensure participants’ sustained engagement and adherence to treatment while preventing potential frustrations.
Future directions
Following the usability testing and the changes implemented on the basis of the findings in this study, the SOSteniamoci platform will be studied for feasibility and efficacy. A pilot study will be conducted with informal caregivers in Italy, randomizing participants into two groups: a. a study group using the SOSteniamoci platform and completing the 8 modules; and b. a control group (i.e., a wait-list group). Comparisons of caregiver burden, anxiety, depression, stress, quality of life, quality of the relationship, and inclusion of the illnesses in the self will be assessed between the two groups at the beginning and the end of the study.
Limitations
A few limitations need to be considered. First, the controlled testing environment of the study might have influenced participants’ responses. The presence of the researcher throughout the testing session may have led to overly positive feedback and lower criticism toward the SOSteniamoci platform. Second, the lab-based setting resulted in testing the platform in front of a researcher, even though it was conducted online. Indeed, even if it was demonstrated that it is possible to rely on investigating usability in a lab setting, 45 this method may not represent a real-world setting. Thus, we may have missed important information about the use of the platform in a more ecological setting – that is, a setting in which users are really using the platform, such as in their apartments.
Conclusion
To enhance participants’ engagement, adherence to treatment, and prevent drop-out, it is necessary to test the usability of the intervention platform. Data from our study demonstrated that the SOSteniamoci platform is user-friendly, intuitive, and easy to use. Important areas of improvement were provided by users, and the platform was modified accordingly. Even though results may be relatively limited in terms of their generalizability to the broader informal caregiver population, preliminary findings from our study might help in the development of future internet-based interventions and their usability tests, and may potentially influence the design of existing interventions.
Footnotes
Abbreviations
Acknowledgements
MS would like to thank AM for her contribution to the study and the informal caregivers who invested their time participating in this study.
Contributorship
MS and NV conceptualized and designed the study. MS and AM coordinated the study recruitment and conducted the interviews. MS and GM coded and analyzed the interviews under the supervision of NV. All authors participated in the critical revision of the manuscript and approved the final manuscript.
Declaration of conflicting interests
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Funding
This study was undertaken as part of a PhD, thesis. The PhD was funded by EC-funded Marie Sklodowska-Curie Innovative Training Network (H2020-MSCA-ITN-2018), Grant Agreement No. 814072. The funder has not had any role in the preparation of the manuscript.
Guarantor
MS.
