Mixed methods,single case design,feasibility trial of a motivational conversational agent for rehabilitation for adults with traumatic brain injury

Abstract

Objective

Rehabilitation for adults with traumatic brain injury (TBI) incorporates client-centred goal-setting and motivational support to achieve goals. However, face-to-face rehabilitation is time-limited. New therapy approaches which leverage care are warranted. Conversational agents (CAs) offer a human–computer interface with which a person can converse. This study tested the feasibility, usability and acceptability of using a novel CA – RehabChat – alongside brain injury rehabilitation.

Design

Mixed methods, single case design, feasibility pilot trial.

Setting

Ambulatory and community brain injury rehabilitation.

Participants

Adults with TBI receiving brain injury rehabilitation and clinicians providing this care.

Intervention

Following 1:1 training, client–clinician dyads used RehabChat for two weeks alongside usual care.

Main measures

Pre-post clinical measures (Motivation for Traumatic Brain Injury Rehabilitation Questionnaire, Rehabilitation Therapy Engagement Scale, Brain Injury Rehabilitation Trust Motivation Questionnaire-Relative, Brain Injury Rehabilitation Trust Motivation Questionnaire-Self) repeated measures (Hospital Anxiety and Depression Scale, researcher-developed wellbeing screening questions); and post-intervention (System Usability Scale (SUS), semi-structured 1:1 interview).

Results

Six participants (two clients and four clinicians) completed training. Two client–clinician dyads completed the intervention. Two other clinicians used RehabChat in a mock client–clinician session. SUS scores indicated good usability. Client well-being did not deteriorate. No adverse events were experienced. Interviews indicated RehabChat was feasible, acceptable and easy to use; and supported motivation, goal-setting and completing practice activities.

Conclusions

RehabChat was feasible and acceptable to use alongside usual ambulatory and community brain injury rehabilitation, had good usability and supported client needs. Further testing of RehabChat with a larger cohort for longer duration is warranted.

Keywords

Rehabilitation conversational agent traumatic brain injury motivation feasibility

Introduction

The worldwide incidence of traumatic brain injury (TBI) is at least 27 million cases¹(p^.47⁾. Adults with TBI can experience challenges with cognition and physical skills impacting on their functional independence.² Recovery following TBI is possible due to the physiological mechanism of neuroplasticity – in which the brain is able to form new neuronal connections in response to rehabilitative training.³ Recovery is optimised through multi-disciplinary rehabilitation which supports motivation,⁴ and enables client-centred goals to be set and also pursued.⁵ However, access to face-to-face rehabilitation resources may be time-limited. Novel care approaches which are motivational and also leverage service provision regarding goal-setting and achievement, are needed.

Conversational agents (CAs) offer a responsive human–computer interface with which a client can converse.⁶ A scoping review investigating the design and use of CAs for rehabilitation of clients with brain-related conditions identified that within this early field of research, CAs have been designed and tested to an early stage for use on a range of computing devices, and for varied clinical purposes including improving conversation participation, providing health education, and supporting wellbeing.⁷ When testing a CA, it is necessary to assess not only health outcomes, but also non-health outcomes including feasibility and usability as these impact upon how much the end-user engages with the CA, and therefore the interaction they have with the CA's health-focussed content.⁸ This study assessed the feasibility, usability and acceptability of RehabChat, a goal setting and pursuit tool for clients with moderate-severe TBI. This is the first time that a goal-setting CA has been tested in the clinical setting with clients who have TBI.

RehabChat is a CA designed to support goal-setting and goal-pursuit in brain injury rehabilitation. RehabChat incorporates a humanoid avatar and text-based interaction (Figure 1). It has been developed using the Virtual Human software platform (Clevertar Pty Ltd⁹), with controlled content managed utilising constrained language. The overall purpose of RehabChat is to provide a conversational platform which supports the client with TBI in-between scheduled therapy appointments to engage in their rehabilitation goals and prescribed practice activities. The clinician oversees their client using RehabChat at weekly appointments, including the content being entered to ensure that this is relevant and safe.

Figure 1.

Design of RehabChat: showing the rehabilitation module; examples of conversation & features of user interface. UI: user interface capabilities; Conv: conversation content; MCQ: multiple choice question.

RehabChat was developed using an iterative, co-design approach in keeping with international expert recommendations for developing new behavioural change digital technologies¹⁰ – to optimise its design meeting clinical needs. RehabChat has been developed in-house through alpha and beta testing with academic participants as previously reported.¹¹ The resultant prototype was further developed through a series of four co-design workshops with clients with TBI and clinicians providing brain injury rehabilitation.¹² The final stable model of RehabChat used in this current study incorporated an initial training module and a rehabilitation module to be completed by client–clinician dyads in the clinical setting (Figure 2). The rehabilitation module comprised goal-setting, practice activities, symptom management and weekly progress reviews.

Figure 2.

Using RehabChat alongside usual ambulatory brain injury rehabilitation care. Text in italics: completed prior to intervention, 1:1 session with researcher, outside of clinic time; white boxes: completed during intervention, at appointments; grey boxes: home-based use.

The aim of this study was to conduct an early evaluation of the feasibility, acceptability and usability, of using RehabChat alongside usual care in the ambulatory care brain injury rehabilitation setting. This is the first time that a goal-setting CA has been tested in the clinical setting with clients who have TBI.

Methods

This mixed methods feasibility study implemented a short, two-week intervention period of using RehabChat alongside usual care, so as to minimise impact upon usual care whilst also enabling participants to experience using RehabChat as intended. This approach aligned with literature recommendation to conduct thorough contextual feasibility testing of a novel digital health intervention prior to clinical testing.¹³ Each client-participant used RehabChat on a clinic-owned iPad. Technical issues relating to software functionality and loading RehabChat onto the clinic iPads were managed promptly to enable each client to complete the two-week intervention without interruption. All data entered into RehabChat was stored by the software company Clevertar Pty Ltd according to their privacy policy (see https://clevertar.com/privacy-policy/).

This study was undertaken in suburban South Australia at two ambulatory and community rehabilitation settings of a state-wide brain injury rehabilitation service. Both clients and clinicians were recruited for this study as the intended end-users of RehabChat; and feedback from both cohorts was treated with equal importance. All participants completed an informed consent process. This project was approved by the Central Adelaide Local Health Network Human Research Ethics Committee (project number 14079). Eligible clinician-participants were any clinician providing direct rehabilitation care. Each recruited clinician received 1:1 training to learn to use RehabChat, screen their clients for eligibility, and to support their client using RehabChat. Clinicians were also offered the opportunity to provide feedback on RehabChat at this time.

The eligibility screening process for client-participants was based on the CONSORT guidelines for reporting screening and recruitment¹⁴ and adapted for this single-arm study (Figure 3). Clinicians screened their clients for the set inclusion criteria: have received care from the recruited clinician for at least three appointments (this ensured the clinician understood the client's rehabilitation needs); diagnosis of TBI; able to use an iPad; and had mental capacity to provide their own consent (this was determined by the screening clinician). The clinician also screened each client for other reasons which may mean the client is not suitable to participate in the project (any factor which might preclude being able to participate comfortably, for example mental health concerns). The clinician invited their eligible client to participate, and interested clients contacted the researcher. After a recruited client had completed their 1:1 training to use RehabChat, the clinician-client dyad commenced using RehabChat (Figure 2).

Figure 3.

Details for screening and recruitment (based on CONSORT guidelines¹⁴).

Each client–clinician dyad used the rehabilitation module for two weeks alongside usual care (Figure 2). RehabChat was used for approximately 10–15 min at weekly appointments. In-between appointments, the client used RehabChat independently for support to complete their practice activities and check for symptoms, and be reminded about their goals (Figure 2). Participants were informed that RehabChat did not provide clinical care, and clients should discuss their clinical concerns with their clinician. For any clinician-participant for whom no clients could be recruited, the clinician participated in a mock client–clinician RehabChat session lasting approximately 30 min and completed a semi-structured interview.

Clients’ safety and wellbeing was supported by both the supervising clinician providing usual clinical oversight of their client, and the researcher making twice-weekly support phone calls to the client during which any concerns or questions about the project could be addressed. Quantitative wellbeing measures were also completed during these phone calls according to a modified A-B-A ‘within-subjects’ single case design¹⁵: the A-phases (one-week prior and one-week following the intervention) were shorter than the B-phase (two-week intervention phase) so as to minimise participant burden. The wellbeing measures were the Hospital Anxiety and Depression Scale (HADS)¹⁶ administered weekly, and four researcher-developed VAS wellbeing screening questions for anxiety, depression, motivation and energy¹² asked in randomised order twice weekly. If any wellbeing issues arose, the researcher would facilitate referral to the clinic therapist.

Pre-post Likert-scale clinical measures were completed at the start and finish of the intervention, either by clients (Motivation for Traumatic Brain Injury Rehabilitation Questionnaire (MOT-Q)⁴), and the (Brain Injury Rehabilitation Trust Motivation Questionnaire-Self (BMQ-S)¹⁷), or by supervising clinicians (Brain Injury Rehabilitation Trust Motivation Questionnaire-Relative (BMQ-R)¹⁷), and the Rehabilitation Therapy Engagement Scale (RTES)¹⁸). Finally, following completion of the intervention all participants completed the System Usability Scale (SUS)¹⁹ and a semi-structured 1:1 interview which focused on the feasibility, usability and acceptability of RehabChat. Interview questions were based on the Unified Theory of Acceptance and Use of Technology,²⁰ and the Web Content Accessibility Guidelines.²¹ Clinicians completing the mock session participated in the interview. Each interview was recorded digitally. If the recording process failed, the researcher completed a detailed written summary of the interview, and the participant reviewed this for accuracy and made corrections as required. Audio data was deidentified, transcribed and checked for errors.

Participant feedback from the semi-structured interviews was analysed using a qualitative Framework Analysis approach²² which focussed on extracting data relevant to identifying participants' experiences and opinions of using RehabChat, clients’ clinical needs as they related to RehabChat, and recommendations for changes to RehabChat. Quantitative analyses included descriptive statistics for the pre-post clinical measures, and single-case line graphing for the repeated well-being measures. Results from these mixed methods analyses helped determine the changes needed for RehabChat and its intended mode of use prior to testing in an intended larger feasibility trial.

Results

Recruitment was conducted from July to December 2021. During this time, participating clinic sites experienced increased administrative demands due to the Covid-19 pandemic, service accreditation review, and internal service re-structuring. Four clinicians (all females) were recruited. They held a range of professions – Speech Pathologist (n = 2), Occupational Therapist (n = 1), Education Tutor (n = 1) – and had each worked in their profession for 10–33 (mean 23.25) years, much of which was at their current rehabilitation service (range 9–30 (mean 20.25) years). Clinicians’ comfort in using a CA on a scale from 0–4 (4 indicating high comfort) varied, with three clinicians scoring 3/4, and one scoring 0/4.

Table 2.

Description of finalised design of RehabChat.

Content development; implementation
Clinical rehab purpose of CA*	Support client-centred goal-setting and pursuit, using a motivational approach
Context (health domain; setting): intended, actual*	Intended and actual: Brain injury rehabilitation services (ambulatory, community, or tele-rehabilitation care)
Participants: intended, actual*	Intended and actual: Clients with TBI, receiving rehabilitation (ambulatory, community, or tele-rehabilitation care)
Content of conversation:	Client-centred goal-setting, prescribed practice activities, symptom checks, weekly progress reviews.
Results:	Very good usability (SUS and qualitative feedback). Participants’ feedback also indicated good understandability, and able to be integrated alongside usual care; as well as multiple recommendations for further refinement of RehabChat to improve clinical integration and breadth of application.
Safety:	No adverse events. Potential risks identified: confidentiality breach; client feeling overwhelmed; client over-exertion with practice activities; potentially stressful for client if the set rehabilitation goal too difficult to achieve; mild frustration when technical issues occurred.
Use barriers:	Client has other technology tools to manage, financial limitations, or decreased memory.
Use facilitators:	Has very good usability; is easy to understand; mode of use alongside usual care is well-organised, requires minimal technical support by supervising clinician, integrates well into usual care because it can incorporate a goal already being considered in rehabilitation care, and it provides cues for client which are similar in style to clinician's cues.
Technology description
Task-orientated:	Yes
Hardware:	Clients used iPad; clinicians used iPad or Personal Computer
Software:	Clevertar Virtual Human software platform; Google Chrome web browser
Dialogue management:	Finite
Dialogue initiative:	Mixed: system asks for required information; user input determines direction of the conversation (e.g. for practice activities, progress review etc.)
Input modality:	Text; click button responses
Output modality:	Speech, text, visuals (humanoid character, text bubbles)
Appearance:	User interface (UI): has minimise button in top right corner, mute button in lower left corner, and two sections divided longitudinally. Left side (one-third of UI) displays animated avatar (female, long dark hair; speaks with Australian accent). Right side (two-thirds of UI) displays dialogue text boxes, and in lower fifth is section for freeform text entry.

*A category based on Macedo et al.²⁶. Italics: content based directly on recommendations by Laranjo et al.²⁵.

Clinicians screened a total of 23 clients, of which 21 clients were excluded due to: being ineligible (n = 10); being eligible but declining to participate (n = 6); or other reasons identified by the clinician relating to the client's suitability to participate, such as psychosocial stress (n = 5) (Figure 3).

Two clients were recruited (1M, 1F) each of whom had differing times both of when they sustained their TBI (6 weeks and 22 weeks respectively (mean 14 weeks)) and of how long they had attended inpatient and/or ambulatory care rehabilitation (86 weeks (due to an additional prior TBI) and 16 weeks respectively (mean 51 weeks)). Both client-participants reported good comfort using a CA (scores 3/4) despite minimal prior use of CAs (between 1–6 times of prior use).

Two clinicians were unable to identify any eligible clients, and thus participated in a mock client–clinician session using RehabChat with the researcher. Three sub-cohorts were thus included in this study – clients (n = 2); clinicians supervising a client (n = 2); and clinicians with mock client (n = 2).

The full two-week intervention was completed by both client–clinician dyads. No adverse events occurred. The 1:1 semi-structured interviews lasted 25–60 min and were conducted via tele-conferencing or phone.

Quantitative results

The results for SUS scores indicated that RehabChat had very good usability (mean 76.25 (range 67.5, 82.5)). The lowest score just below the average threshold score for determining good usability of 68²³ was from one client-participant experiencing initial technical issues when using RehabChat prior to then having no technical issues during the two-week intervention period. Results for the repeated wellbeing measures – twice weekly wellbeing screening questions, and weekly HADS measures – and pre-post clinical questionnaire measures are presented below.

Repeated measures results for client A showed stable HADS scores, and for the wellbeing screening questions, the Anxiety and Depression ratings overall gradually reduced, and the Motivation and Energy scores varied and did not align to the other (Figure 4). Client B's repeated measures results showed a slight improvement in HADS scores, whereas the wellbeing screening scores remained stable (Figure 5).

Figure 4.

Client A repeated measures scores.

Figure 5.

Client B repeated measures scores.

Pre-post measures for both clients are shown in Table 1. Client A remained stable overall for all four outcome measures: scores for the BMQ-S and BMQ-R indicated a small amount of deterioration in extrinsic motivation, however, the baseline self-report BMQ-S for this client had missing responses, which may have skewed the comparative result. In contrast, the MOT-Q score of 42 indicates good intrinsic motivation, from a possible range of scores of −62 to +62.⁴ The clinician's score of Client A's therapy engagement in the RTES was very good and remained stable at pre = 39 and post = 38, out of a possible total score of 45.²⁴ Client B showed consistency in pre-post measures for the BMQ-S and MOT-Q, whilst the clinician scores for the BMQ-R and RTES showed some improvement. Client B's MOT-Q score indicated reasonable motivation (pre = 32, post = 31), and similarly the clinician report for therapy engagement in the RTES indicated reasonable engagement (pre = 24, post = 32).

Table 1.

Pre-post measures for Client A and Client B.

Measure		Pre score	Post score	Change – total score
BMQ-S	Client A	81	89	+8 minimal deterioration
BMQ-S	Client B	80	78	−2; negligible improvement
BMQ-R	Client A	58	64	+6
BMQ-R	Client B	91	68	−23; small trend for improvement
MOT-Q	Client A	43	42	−1 (equivocal)
MOT-Q	Client B	32	31	−1 (equivocal)
RTES	Client A	39	38	−1 (equivocal)
RTES	Client B	24	32	+8; some improvement

Italics font: unable to obtain definitive score due to missing five responses in baseline BMQ-S.

MOT-Q : Motivation for Traumatic Brain Injury Rehabilitation Questionnaire⁴: items rated on 5-point scale −2 (strongly disagree) to +2 (strongly agree), 0 being undecided; score range −62 to 62; higher scores suggest higher motivation⁴.

BMQ-S: Brain Injury Rehabilitation Trust Motivation Questionnaire-Self¹⁷: score range 34 to 136, higher scores suggest more challenges.

BMQ-R: Brain Injury Rehabilitation Trust Motivation Questionnaire-Relative¹⁷: score range 34 to 136, higher scores suggest more challenges.

RTES: Rehabilitation Therapy engagement Scale¹⁸: items rated 0 to 3; score range 0 to 45; lower scores indicate lower engagement.

Qualitative results

Qualitative data from the interviews was categorized into five main themes: ‘What went well’, ‘Motivation and engagement’, ‘Barriers and concerns’, ‘Use in clinical setting’, and ‘Recommendations’; with sub-themes for each. Key results are presented below, with participant quotes highlighted in italics; and sub-cohorts indicated as C = client, CWC = clinician with client, and CNC = clinician with no client.

Theme 1 – what went well

Participant feedback indicated that RehabChat was highly usable – ‘I found it pretty straightforward, pretty easy’ (C); – and not effortful – ‘I didn’t feel that there was a lot of effort needed’ (C). The dialogue text was spaced appropriately – ‘the spacing .. of the writing was good … it was like little clouds all the time and they had spaces’ (CWC) and easy to follow–‘I didn’t have any issues understanding the language’ (C).

Using RehabChat alongside usual care worked smoothly – ‘… the way it was organised was very well done’ (C); ‘I didn't really need to support a lot’ (CWC). Current rehabilitation priorities could easily be entered into RehabChat – ‘Planning the [the goal] has been something we were talking about, so it was good to incorporate that and have something else to work with’ (C) –, through utilising – ‘a simple, clear framework, which incorporates the SMART framework, and … the standardised requirements of what should be done’ (CNC).

A benefit of using RehabChat was to provide ‘opportunities for people to have check-ins and making sure that they're on target with their goals (C); which helps ‘people to stay on track or encourages them to stay on track with their goals’ (CWC).

Theme 2 – motivation and engagement

The RehabChat avatar – Jo – supported motivation – ‘She's been my motivation to good health, … I felt more motivated with things’ (C). RehabChat helped the client ‘see that you [are] actually able to achieve an outcome [it] felt motivating in itself’ (CWC); and supported self-accountability ‘… just using the RehabChat made him more accountable to the goal that he set’ (CWC).

RehabChat was able to reinforce and motivate towards goal-pursuit – ‘something … to reinforce and motivate me to keep working on things’ (C); ‘Even when I couldn't do it, … reminded me that I really did want to do it’ (C); ‘it really reminds me that I am working towards a bigger goal’ (C). RehabChat helped with practical planning – ‘it kept me on schedule, made me get up early' (C); ‘she could see … the steps, what she had to be ready for’ (CWC) and checking of the key symptom – ‘it's reminding me what I should be looking for … it made me more aware’ (C); and then decide when to have a rest and so avoid overexertion.

Personal connection with RehabChat was facilitated by incorporating the client's name regularly during the conversation – ‘Liked that the user's name is used regularly in the conversation – to help it more feel more personalised ‘ (CNC), and being ‘sort of like a personal trainer yes but … not shouting with me’ (CWC).

Theme 3 – barriers and concerns

Barriers to use which were identified included: decreased memory – ‘the biggest thing with my mind is that I can't remember’ (C) –, meaning that the client may ‘forget… what goals they have set’ (CWC); low mood – ‘you do get down’ (C) –; fatigue; and auditory processing for which the client may need ‘more time to process auditory information’ (CWC). Additionally, if the client was already managing other technology tools home, this may inhibit uptake – ‘when you’ve got lots of other things on your table, … an extra thing might be … challenging’ (CWC)).

Participants also expressed concerns related to potential risks including: breach of confidentiality regarding the details entered, but also that this could be mitigated by clinician oversight– ‘but having worked with the therapist to start out, I don't think it’d be a problem’ (C); and a client over-exerting themselves doing their practice activities as ‘self-awareness can be very poor …’ (CWC). Feedback also highlighted that using RehabChat could be potentially stressful if an unrealistic goal was entered; and that daily use of RehabChat was too much – ‘seven days is too long’ (CWC); and one client ‘missed two out of the five days’ (CWC).

Theme 4 – Use in clinical setting

Clinician feedback indicated that due to COVID-19 requirements there was ‘not … as much face to face work with clients’ (CWC) during usual care, and also less reiteration for the client about their rehabilitation goals – ‘not having as many people around to help facilitate that’ (CWC), which indicated the usefulness of RehabChat for increasing reiterative input.

The three-way interaction between the client–clinician dyad and RehabChat was seen as important for clarifying goals and weekly reviews – ‘it was sitting and thinking about how can I make it better and then [therapist] discussing my symptoms with me and then refining what I was doing’ (C) –; noting the clinician supporting identifying the ‘… goal and maybe edit it’ (CWC). Wording of the client's goals should be from the client – ‘people's goals can be the words that they've got’ (CNC) because ‘ if it ends up being … watered down, they'll lose their sense of – that fits my personal values – and then they'll lose their motivation for it’ (CNC).

Theme 5 – Recommendations

Participants suggested a range of recommendations for extending the way RehabChat could be used, and ways to improve its functionality. Clinicians described the clinical process of ‘a rehab break which means maybe two weeks, maybe in the middle of their program, that we might set them up to see how they go at being independent in and carrying things over just to test it’ (CWC), and that RehabChat could potentially augment this process – ‘It's a good interim step … between a real person and … the avatar prompting you, which … is a good step down from us and reliance on us, so I see it as a good transitional tool’ (CWC).

A reminder system was suggested to ‘have some sort of remind ability to turn on and use ‘cause … I reckon I chooffed off three times without turning it on’ (C) which could be to use whatever is the client's current preferred reminder system – ‘people will be having their own external memory and organizational aids’ (CNC) – such as an online calendar system.

An alternative style of goal setting added could be added into RehabChat which focussed on ‘more subjective type goals’ (CWC) for cognitive training and communication skills, for example relating – ‘more to people or … confidence to talk to people’ (CWC). Similarly, RehabChat could be developed to provide a supportive cognitive framework for clients to help address challenges of ‘executive dysfunction’ (CNC), which enabled the client ‘to monitor for their own fatigue or overload’ (CWC) with RehabChat ‘ just prompting those thoughts … to … keep someone on track with that evaluation cycle’ (CWC).

Increased support for clients with auditory processing needs was identified. This could be achieved through the clinician assisting ‘the client directly over multiple sessions’ (CNC) to become accustomed to how RehabChat worked; the content of RehabChat offering ‘examples … [and/or provide] repetition’ (CWC) and having a predictable dialogue and conversation structure to facilitate clients’ ability to ‘develop an ability to learn to use it themselves if there's predictability’ (CWC); and for the content of what the avatar is saying to be displayed ‘in-sync as she says it’ (CNC) rather than afterwards as was presented in the trial.

Suggestions were also made for additional conversation content to be included: if technical issues arose, advice to ‘turn it off and on again’ (C); cues for how to write a free-text response – ‘have a bit more explanation in terms of the wording that needs to be used’ (C); and prompts to only enter a response when the user has understood the dialogue as the conversation then progresses, such as ‘can you click the box below if you understand and you're ready for me to keep going?’ (CNC).

Suggestions for improving flexibility in how RehabChat could be used alongside usual care included: using it for as long or short as needed – ‘We try stuff in rehab and if it looks bad it quickly gets side swiped’ (CWC); to initiate use of RehabChat more responsively to the client's needs – ‘where these things that just emerge in a conversation, you want to be able to move with that traction’ (CWC); and have easy access to RehabChat loaded on the iPad – ‘if I had it as a tool just next to me like I said, do you [to] want try this?’ (CNC).

Recommendations for RehabChat's UI included for key points of the conversation to be visually highlighted to facilitate recap, because currently ‘the text bubble content is all in same font, and so the key points do not visually stand out’ (CWC). Additionally, the UI needed ‘more space to write more text in the free-text fields, because … at times there was not enough space' (CWC). However, enabling a shortened version of the goal to be entered was also seen as important for being used in later reiterative dialogues about it – ‘Later on, when we're recapping how you're going, is there a short way I can talk about it?’ (CNC).

Description of finalised design of RehabChat

The finalised design of RehabChat is presented below (Table 2) using a reporting framework (7) informed by previous recommendations for reporting on CA development.^25,26 Utilising these recommendations facilitates research transparency and comparability between studies.

Discussion

This study evaluated the feasibility, acceptability and usability, of using a novel CA – RehabChat – for brain injury rehabilitation alongside usual care: both RehabChat as a CA entity, and also its intended mode of use alongside usual care. The mixed methods results from six participants identified client- and clinician-participants’ experiences of and perspectives about using RehabChat, and foci for design changes and future research.

This study builds upon earlier step-wise development of RehabChat¹¹ which incorporated co-design with clients and clinicians.¹² This concurs with recent recommendations from a thematic literature analysis of the use of CAs for clients with dementia: that the CA be co-designed from its early stages of development; to carefully plan how efficacy of the CA will be appraised through gathering qualitative feedback on facilitators and barriers to using it, and also about how it impacts upon, and can be used within, the person's life.²⁷ In our study, efficacy appraisal was achieved through implementing appropriate outcome tools and qualitative interviews which explored the applicability of RehabChat to the person's life regarding both the content (rehabilitation goals) and use of RehabChat at clinic appointments and in-between appointments. Notably, comparable factors have been reported in a four-week mixed methods usability and acceptability trial of a multi-modal CA by people with dementia living at home,²⁸ which focussed on the user's ability to achieve independent use of the CA, and the ability of the CA to support memory and enjoyable activities.²⁸ With our current study, we also appraised the ability of clients to use RehabChat independently in-between appointments, and for RehabChat to support engagement in rehabilitation activities and overall motivation; and feedback indicated that these factors were achieved. Participants in our study reported RehabChat was easy to use and motivational for clients. The ability to input client goals and practice activities into RehabChat was useful and aligned well with usual care. This process provided the basis for supporting clients to complete their practice activities and review their progress. Research investigating the design and use of CAs to support rehabilitation for adults with brain-related conditions including brain injury and dementia is an emerging field⁷: it shows high relevance for further enquiry given the clinical needs relating to client memory, therapy engagement and leveraging of therapeutic input.

Feedback identified that RehabChat supported client motivation and engagement by enabling the client to make well-thought-through decisions about their rehabilitation in consultation with their clinician. Quantitative scores for motivation, wellbeing and therapy engagement showed consistent results for these factors which aligned with the positive qualitative feedback; however, due to the short duration of this study, the quantitative results cannot be extrapolated broadly.

Interview feedback indicated that RehabChat was feasible to use alongside usual care because it reflected usual rehabilitation care; it was easy for clinicians to supervise their client using RehabChat; RehabChat had very good usability; and no safety issues were experienced. Participants also envisaged future uses of RehabChat including during therapy breaks and to support transitioning from receiving care to independence. Quantitative results concurred with these qualitative findings: high SUS scores showed very good usability; and the repeated and pre-post measures results showed minimal variability and no indication of increased stress from using RehabChat. Participants indicated their acceptance of RehabChat: overall the content was pitched at an appropriate level of complexity and duration; and participants would have been happy to continue using RehabChat if given the opportunity.

Findings from this study identified future research priorities regarding confidentiality, and RehabChat's design and mode of clinic integration. These included further clarifying the clinician's role for ensuring confidentiality and achievability of content entered into RehabChat by the client, and enabling RehabChat to be more flexibly used within clinical care. Modifications to RehabChat's content were also suggested including: providing brief prompts for solving technical issues if they arose; increased character limits for free text entry; and including examples to illustrate how free text answers should be structured. Future development of RehabChat could investigate the use of Natural Language Processing of spoken user input, to reduce reliance on dexterity for typed entries; and the usefulness of different UI options such as displaying an avatar and/or text dialogue boxes, and enabling of responsive avatar facial expressions and directive gaze.

The limitations of this project were the low sample size and short intervention duration. The sample size for this study although low (n = 6) showed heterogeneity: three sub-cohorts, each having one participant without prior experience using RehabChat; and varied clinicians’ professions (SP, OT and ET). Additionally, comprehensive feedback was obtained from participants, with all quantitative assessments and qualitative interviews completed. Nonetheless, the small sample size meant that this study's findings cannot be widely extrapolated. Despite a six-month recruitment phase and screening of a large number of clients, only two clients were recruited. The low number of recruited participants was likely due in part to Covid-19 and increased clinic administrative demands. Additionally, some eligible clients were excluded due to suitability reasons determined by the screening clinician. This suitability aspect of screening was important to ensure client wellbeing, and aligned with previously reported safety considerations to exclude clients based on the risk of stress-related reactions in a comparable study of CA use for participants with dementia.²⁸ Challenges recruiting clients with TBI²⁹ and stroke³⁰ for clinical research have been previously reported. Similarly, a scoping review of studies of CAs for participants with brain-related conditions revealed low sample sizes: for example, a CA study for severe TBI had one participant,³¹ and research of CAs for dementia have had one participant,³² five participants³³ and 28 participants.³⁴ A larger sample size is required for further feasibility testing of RehabChat to enable eventual progression to a clinical effectiveness trial. Recruitment should be optimised by recruiting over a longer period and to more rehabilitation services; and expanding eligibility criteria to include clients with any type of acquired brain injury. The two-week intervention duration, although appropriate for initial feasibility testing, was too short in comparison to usual clinical care planning. Future research should allow longer use of RehabChat, for example up to six weeks.

This study demonstrated how a newly developed CA can be carefully tested within the complex setting of brain injury rehabilitation to provide results to form the basis of further feasibility and effectiveness testing. This stepwise research approach allows for the needs and preferences of clients and clinicians to be identified and integrated at each stage of the development of RehabChat. Participants’ feedback indicated that with further development, RehabChat could effectively augment usual rehabilitation care and support improved clinical outcomes for clients with TBI.

Clinical messages

A co-designed CA for brain injury rehabilitation – RehabChat – was implemented in a feasibility pilot trial for two weeks alongside usual rehabilitation care.

Early results indicate that RehabChat has good feasibility, usability and acceptability.

Larger scale feasibility testing of RehabChat is warranted.

Footnotes

Acknowledgements

The authors thank Dr Martin Luerssen, Chief Technical Officer of Clevertar Pty Ltd, for providing pro bono use of the Clevertar Virtual Human software platform and ongoing technical support; Dr Maggie Killington for providing expert comment on recruitment and integration of the study into the clinical setting; Mr Damen Lange for configuring RehabChat onto clinic iPads and addressing any related technical concerns; Mr James Light for configuring RehabChat onto University iPads.

Author contributions

JH initiated and completed this project: the topic, its aims and methodology, and ethics processes. JH also managed the CA software, provided training of and ongoing support for project participants, undertook data gathering and analysis, and worked in consultation with other co-authors to complete the manuscript ready for submission. BL contributed to refining the project aims and methodology and reviewing the qualitative analyses, assisted with the presentation of quantitative results, and advised on write-up of the manuscript. AM and LL contributed to refining the project aims and methodology and conclusions, and advised on write-up of the manuscript. DP and LL contributed to refining the project aims and methodology and conclusions, advised on write-up of the manuscript, and assisted with software management. BD approved and oversaw implementation of the project in the clinic settings, and facilitated provision of technical and computing resources necessary for the project.

Declaration of conflicting interests

The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Ownership of and rights to the RehabChat IP are as per the Flinders University IP Policy, stipulating ownership rests with the University. There are no plans to commercialise this IP yet. The Flinders University commercialisation team is conducting due diligence to determine whether a commercial pathway is viable or if an open-source option would be best. No industry partners have been engaged to date for commercialisation. The Virtual Human software platform is owned by Clevertar Pty Ltd. This study was undertaken as part of JH's PhD. The results of this study did not influence JH's PhD outcome.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: JH was a PhD scholarship recipient from the Australian Research Council Industrial Transformation Research Hub for Digital Enhanced Living (IH170100013). The funders had no input into the study.

ORCID iDs

Judith Hocking

Anthony Maeder

References

World Health organization. Package of interventions for rehabilitation: module 3: neurological conditions. [Internet]. 2023 [cited 2023 Aug 13]. Available from: https://www.who.int/publications/i/item/9789240071131.

Iverson

Lange

. Moderate and severe traumatic brain injury. In: Schoenberg

Scott

(eds) The little black book of neuropsychology: a syndrome-based approach. Boston, MA: Springer US, 2011, pp. 663–696.

Crosson

Hampstead

Krishnamurthy

, et al. Advances in neurocognitive rehabilitation research from 1992 to 2017: the ascension of neural plasticity. Neuropsychol 2017; 31: 900–920.

Boosman

van Heugten

Winkens

, et al. Further validation of the motivation for traumatic brain injury rehabilitation questionnaire (MOT-Q) in patients with acquired brain injury. Neuropsychol Rehabil 2016; 26: 87–102.

Turner-Stokes

Pick

Nair

, et al. Multi-disciplinary rehabilitation for acquired brain injury in adults of working age. Cochrane Database Syst Rev [Internet]. 2015. Article ID: CD004170 [cited 2022 Mar 11].

Fadhil

Wang

Reiterer

. Assistive conversational agent for health coaching: a validation study. Methods Inf Med 2019; 58: 9–23.

Hocking

Oster

Maeder

, et al. Design, development, and use of conversational agents in rehabilitation for adults with brain-related neurological conditions: a scoping review. JBI Evid Synth 2023; 21: 326–372.

Cole-Lewis

Ezeanochie

Turgiss

. Understanding health behavior technology engagement: pathway to measuring digital behavior change interventions. JMIR Form Res [Internet] 2019; 3: e14052. [cited 2022 March 11] 10 pages. Available from https://formative.jmir.org/2019/4/e14052.

Clevertar. Clevertar – Virtual Human software [Internet]. Adelaide, Australia [cited 2020 Feb 17]. Available from: https://www.clevertar.com/.

10.

Michie

Yardley

West

, et al. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res 2017; 19: e232.

11.

Hocking

Maeder

. Motivational embodied conversational agent for brain injury rehabilitation. In: Maeder

Higa

van den Berg

, et al. (eds) Ebook: Telehealth Innovations in Remote Healthcare Services Delivery. Global TeleHealth. Amsterdam: IOS Press. 2020 [Internet], 277, pp. 37–46. 2021 [cited 2022 Mar 11].

12.

Hocking

. Design and development of a motivational conversational agent for brain injury rehabilitation. PhD Thesis, Flinders University, Aus, 2022.

13.

Kowatsch

Otto

Harperink

, et al. A design and evaluation framework for digital health interventions. IT - Inf Technol 2019; 61: 253–263.

14.

Eldridge

Chan

Campbell

, et al. CONSORT 2010 Statement: extension to randomised pilot and feasibility trials. Br Med J 2016, 355: i5239–i5239.

15.

Lobo

Moeyaert

Baraldi Cunha

, et al. Single-case design, analysis, and quality assessment for intervention research. J Neurol Phys Ther 2017; 41: 187–197.

16.

Zigmond

Snaith

. The hospital anxiety and depression scale. Acta Psychiatr Scand 1983; 67: 361–370.

17.

Oddy

Cattran

Wood

. The development of a measure of motivational changes following acquired brain injury. J Clin Exp Neuropsychol 2008; 30: 568–575.

18.

Lequerica

Rapport

Whitman

, et al. Psychometric properties of the rehabilitation therapy engagement scale when used among individuals with acquired brain injury. Rehabil Psychol 2006; 51: 331–337.

19.

Brooke

. SUS – a quick and dirty usability scale. In: Jordan

Thomas

Weerdmeester

McClelland

. (eds) Usability evaluation in industry. London, UK: Taylor and Francis; 1996, pp. 189–195.

20.

Venkatesh

Thong

JYL

. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology (Report). MIS Q 2012; 36: 57.

21.

World Wide Web Consortium Web Accessibility Initiative (W3C WAI). Cognitive accessibility at W3C. [Internet]. 2021 Apr [cited 2021 Mar 13]. Available from: https://www.w3.org/WAI/cognitive/#cognitive-accessibility-in-w3c-standards.

22.

Furber

. Framework analysis: a method for analysing qualitative data. Afr J Midwifery Women's Health 2010; 4: 97–100.

23.

Sauro

. Measuring usability with the system usability scale (SUS). 2011 [cited 8-1-22] Available at https://measuringu.com/sus/.

24.

Lequerica

Rapport

Loeher

, et al. Agitation in acquired brain injury: impact on acute rehabilitation therapies. J Head Trauma Rehabil 2007; 22: 177–183.

25.

Laranjo

Dunn

Tong

, et al. Conversational agents in healthcare: a systematic review. J Am Med Inform Assoc 2018; 25: 1248–1258.

26.

Macedo

Pereira

Mota

, et al. Conversational agent in mhealth to empower people managing Parkinson's disease. Procedia Comput Sci 2019 Mar; 160: 402–408.

27.

Rampioni

Stara

Felici

, et al. Embodied conversational agents for patients with dementia: thematic literature analysis. JMIR Mhealth Uhealth 2021; 9: e25381.

28.

Stara

Vera

Bolliger

, et al. Usability and acceptance of the embodied conversational agent Anne by people with dementia and their caregivers: exploratory study in home environment settings. JMIR Mhealth Uhealth 2021; 9: e25891.

29.

Joshi

Dunbar

Taylor

, et al. Streamlining participant recruitment for TBI and PTSD research studies. Mil Med 2017; 182: 124–127.

30.

Ferreira

Pinto

Saleh Velez

, et al. Recruitment challenges in stroke neurorecovery clinical trials. Contemp Clin Trials Commun [Internet] 2019; 15: 100404. [cited 2022 Mar 11] 10 pages. Available from https://formative.jmir.org/2019/4/e14052.

31.

Wilks

Jasiewicz

Catizone

, et al. CALONIS: an artificial companion within a smart home for the care of cognitively impaired patients. Lect Notes Comput Sci (Incl Lect Notes Artif Intell; Lect Notes Bioinform) 2015; 8456: 255–260.

32.

Huang

Matsushita

Kawagoe

, et al. Toward a memory assistant companion for the individuals with mild memory impairment. In: Proceedings of the 11th IEEE international conference on cognitive informatics and cognitive computing, ICCI*CC 2012. 2012, pp. 295–299.

33.

Nakatani S, Saiki S, Nakamura M, et al. Generating personalized virtual agent in speech dialogue system for people with dementia. Lecture notes in computer science (Including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics). Springer Nature. [Internet]. Cham: Springer International Publishing AG, part of Springer Nature Location, 2018, pp. 326–337. [cited 2022 Mar 12] Available from https://link.springer.com/chapter/10.1007%2F978-3-319-91397-1_27.

34.

Saito

Okada

Nitta

, et al. Estimating user's attitude in multimodal conversational system for elderly people with dementia. In: AAAI Spring Symposium – Technical Report. 2015, vol. SS-15-07, pp. 100–103.