Usability evaluation of a self-management mobile application for individuals with a mild traumatic brain injury

Abstract

Objective

Mild traumatic brain injuries (mTBIs) are common and may result in persisting symptoms. Mobile health (mHealth) applications enhance treatment access and rehabilitation. However, there is limited evidence to support mHealth applications for individuals with an mTBI. The primary purpose of this study was to evaluate user experiences and perceptions of the Parkwood Pacing and Planning™ application, an mHealth application developed to help individuals manage their symptoms following an mTBI. The secondary purpose of this study was to identify strategies to improve the application. This study was conducted as part of the development process for this application.

Methods

A mixed methods co-design encompassing an interactive focus group and a follow-up survey was conducted with patient and clinician-participants (n = 8, four per group). Each group participated in a focus group consisting of an interactive scenario-based review of the application. Additionally, participants completed the Internet Evaluation and Utility Questionnaire (UQ). Qualitative analysis on the interactive focus group recordings and notes was performed using phenomenological reflection through thematic analyses. Quantitative analysis included descriptive statistics of demographic information and UQ responses.

Results

On average, clinician and patient-participants positively rated the application on the UQ (4.0 ± .3, 3.8 ± .2, respectively). User experiences and recommendations for improving the application were categorized into four themes: simplicity, adaptability, conciseness, and familiarity.

Conclusion

Preliminary analyses indicates patients and clinicians have a positive experience when using the Parkwood Pacing and Planning™ application. However, modifications that improve simplicity, adaptability, conciseness, and familiarity may further improve the user's experience.

Keywords

Mild traumatic brain injury mTBI mobile health mHealth self-management

Introduction

According to the Canadian Community Health Survey, brain injury rates have increased 250% between 2005 and 2014.¹ Mild traumatic brain injuries (mTBIs) constitute 70–90% of all traumatic brain injuries (TBIs),² and may result in time lost at work,^3,4 or at school.^5,6 Additional time may be lost when people experience persistent symptoms, which occurs in approximately 15% of individuals.^7,8 Individuals experiencing persisting symptoms following an mTBI require tertiary care, but may experience long wait times in areas with government-funded healthcare.⁹ Long waitlists are problematic because treatment delays may result in continued or exacerbated symptoms. Innovative rehabilitation techniques that are quickly and easily accessible are necessary to treat this population and reduce health service costs.

Following an mTBI, early interventions, including education on common symptoms, length of symptom presentation, and coping strategies, are recommended to reduce the number of symptoms at three months post-injury.¹⁰ Additionally, the Third Edition of the Guideline for Concussion/Mild Traumatic Brain Injury & Prolonged Symptoms recommends individualized treatment based on symptoms and physical examination findings.^11,12 Following an acute rest phase (24–48 hours post-injury), ‘activity as tolerated’ (physical and cognitive activities that do not exacerbate symptoms) is recommended.¹² This means that individuals with an mTBI should gradually return to normal activity while avoiding heavy exertion that exacerbates symptoms for two weeks following their injury. People with an mTBI should also avoid activities with an increased risk of head impacts for at least one week following their injury, and resume supervised physical activity within a month of injury.¹³

However, patients may find it difficult to gauge an appropriate level of activity, resulting in attempts to return to too many pre-injury activities and ultimately exacerbate symptoms. Others may, due to fear of symptom exacerbation, excessively reduce their level of activity so much so that their symptoms do not improve.¹³ Innovative rehabilitation techniques for individuals with an mTBI should involve individualized support that can help patients gauge an appropriate activity level according to their tolerance.¹²

Within the Acquired Brain Injury Outpatient Program at Parkwood Institute in London, Ontario, a ‘pacing and planning’ points program was created to help individuals manage an appropriate level of daily activities. Each activity is allotted a point value, depending on each individual's perceived difficulty and symptom exacerbation. All activities in a day are summed and are not to exceed a daily maximum that is determined by the clinician. This idea proved very beneficial anecdotally. However, implementing the pacing and planning program in a pen and paper format was problematic because patients often misplaced or forgot their daily activity plan, resulting in decreased use and accuracy in activity recall. Furthermore, the program's benefits were limited to patients in the Acquired Brain Injury Outpatient Program at Parkwood Institute when using a pen and paper format. An alternative method for administering the ‘pacing and planning’ program that can facilitate uptake and promote patient use is through digital platforms, such as a mobile application.

Mobile health (mHealth) is a medical and public health practice supported by mobile devices, which may include mobile phones, patient monitoring devices, or other wireless devices.¹⁴ mHealth applications improve healthcare by reducing health service costs, promoting access, and improving individualized treatment.¹⁵ mHealth applications are also becoming more popular and may benefit individuals with a TBI.¹⁶ Although individuals with a TBI state that electronic devices are acceptable and desirable compensatory aids,¹⁷ barriers to application use include accessibility and usability, symptom exacerbation, stigma, perceived usefulness,¹⁸ forgetting to use the app, and poor vision.¹⁹ Additionally, mHealth applications are not formally regulated²⁰ and may have limited evidence to support their efficacy.¹⁶ Accordingly, more research is needed on mHealth applications for individuals with a TBI to ensure validity and foster maximum application uptake and use.

St Joseph's Healthcare has created a mobile version of the ‘pacing and planning’ program, called the Parkwood Pacing and Planning™ application. This study represents part of an overall program of research to develop a systematic approach to the ‘pacing and planning’ program. The program is investigating the effectiveness of this intervention in assisting individuals with an mTBI in managing their symptoms. The primary purpose of this study was to evaluate the experiences and perceptions of individuals with symptoms following an mTBI and clinicians serving these individuals, when using the Parkwood Pacing and Planning™ application. The secondary purpose of this study was to identify strategies to improve the application.

Methods

The Parkwood Planning and Pacing™ application design

The Parkwood Pacing and Planning™ application was created using a co-design approach with a team of stakeholders, including five adults with prolonged symptoms post-mTBI, three clinicians with expertise in assessing and treating adults with persistent symptoms post-concussion/mTBI, six researchers with expertise in implementation science, five donors from corporate and community groups, and three mobile application developers. Stakeholders suggested a calendar format for adding and tracking activities, where the total point value is listed at the bottom of the calendar day (Figure 1). Additionally, stakeholders recommended using colored point values to indicate if the application user can continue to spend points in a day (green), or if they have used more than their allotted daily point value (red). Stakeholders wanted the application to function as similarly as possible to common calendar applications on mobile devices, where users can input the name, date, and time of their activities (Figure 2).

Figure 1.

Screenshot of the main menu.

Figure 2.

Screenshot of the calendar with added activities.

Specific to this application, stakeholders requested the ability to add any symptoms application users experienced during or after the activity, along with intensity and duration of the symptoms (Figure 3). The list of symptoms available for patients to choose was based on the Rivermead Post-Concussion Symptom Questionnaire (RPQ).²¹ The RPQ is a 16-item, 5-point Likert scale symptom rating assessment, where a higher total score indicates greater symptomatology. Stakeholders also requested a separate section in the application to view cumulative symptom exacerbation (Figure 4), where users can see which symptoms they have experienced over the past week. Lastly, stakeholders requested a section for education (e.g., strategies for grocery shopping and attending social events, and strategies for reducing anxiety). Stakeholders also determined the name, logo, and color scheme of the application.

Figure 3.

Screenshot of adding an activity.

Figure 4.

Screenshot of ‘My Data’ screen.

When implementing the ‘pacing and planning’ program in a pen and paper format, clinicians created a bank of 60 common activities and their average point values. This bank of activities was added to the application. However, stakeholders requested the ability to add new activities to the application. Accordingly, an algorithm was developed by the clinicians to determine the point value of any activity. Application users rate their symptom aggravation and difficulty completing the activity on a scale of 0 (not difficult at all) to 10 (the most difficult activity they have ever completed). The two values are summed and then divided by 4, providing their new activity's point value. Additionally, the clinicians determined that everyone beginning the ‘pacing and planning’ program should begin with 15 points per day. Clinical expertise was used to determine when to change the daily point value for individuals using the ‘pacing and planning’ program in a pen and paper format. In order to replicate clinical expertise, the clinicians created an algorithm for modifying daily point allotment within the application based on the user's response to a bi-weekly check in (Appendix A).

After first-time users register and log into the application, they complete the RPQ.²¹ The initial total score is recorded as their baseline assessment, and re-assessed every 30 days upon logging into the application. After completing the RPQ, the user is directed to the home page of the application, where they can access the Calendar (Figure 2), My Data (Figure 4), Resources (Tips and Tricks), and Settings. An outline of the application's components and key features are listed in Table 1. Perceived change in symptoms was evaluated every 14 days using a bi-weekly check in, when the application prompts the user to provide a rating of their overall symptoms using the options: much better, somewhat better, no change, somewhat worse, and much worse.

Table 1.

Application overview.

Component	Key functions
1. My Calendar	• Plan daily activities (includes a default point value) • Customize activity-specific points based on difficulty and symptom aggravation • View planned activities • View number of points used, and maximum points given per day • Record activity-specific symptoms
2. My Data	Track pacing and planning skills: • View frequency of staying within a recommended total point range daily • Assess the proportion of high and low point-specific activities planned according to number of associated total points
3. Tips & Tricks	• A list of common strategies for effective pacing and planning and preventing the onset of symptoms

Note. Points range from 1 to 7; low-point activity = 1–3; high-point activity = 4–7; users commence with 15 daily points.

Task difficulty and symptom aggravation scales ranges from 0 to 10.

Study design

This study was performed as part of an iterative co-design process using a convergent design, whereby qualitative and quantitative data were simultaneously collected and analyzed.²² A mixed methods evaluation was conducted to capture multiple user experiences and perspectives on navigating and improving the application.²³ Combining the results of both the qualitative and quantitative analyses demonstrated triangulation, which enhanced the credibility of this study.²⁴

Participants

Purposive-sampling was used to recruit four patient-participants and four clinician-participants for this study. A sample size of eight participants (n_patient = 4, and n_clinician = 4) was required to detect a minimum of 80% of potential usability problems.²⁵ All participants were referred to the research coordinator by an occupational therapist in the Acquired Brain Injury Outpatient Program. Inclusion criteria for all participants included: (a) minimum age of 18 years old; (b) access to an Android® mobile phone or tablet; (c) able to complete a paper-based survey independently; and (d) able to read, write, and speak English fluently. Patient-participants were included if they had an mTBI diagnosed by a physician, and were patients in the Acquired Brain Injury Outpatient Program. Patient-participants also must have received the outpatient education package from the Acquired Brain Injury Outpatient Program, which included information on pacing and planning.

Inclusion criteria for clinician-participants included experience providing treatment to patients with an mTBI. Clinicians were included in this study because they understood the purpose of the application, and how it was clinically implemented in a pen and paper format. Clinicians were also included because future application iterations will investigate the clinical applicability of the Parkwood Pacing and Planning™ application and how the application can support clinical care. Demographic information collected from participants to describe the sample included participant age, sex, discipline and years of clinical practice (clinician-participants), and mechanism of injury, but not number of previous head injuries (patient-participants). This study received institutional research ethics board approval from Western University's Health Science Research Ethics Board (108298) and Lawson Health Research Institute (R16282).

Protocol

All participants provided written informed consent prior to beginning this study. Participants were introduced to the Parkwood Pacing and Planning™ application in a 90-minute interactive group session at Parkwood Institute; one for clinician-participants and one for patient-participants. To test user performance of critical tasks, participants in each group followed the same specific task scenarios directed by the moderator (SJ). Prior to beginning the group sessions, the moderator informed participants of her biases, assumptions, and reasoning for completing this study. Her biases included helping develop the application and being a paid research coordinator on this project. Her assumptions included that the app will help individuals with a brain injury in the future. There was no relationship between participants and the moderator prior to this study. Tasks included registering their account, adding multiple events to the calendar, adding items on the ‘To-Do List’, recording symptoms, finding information on reducing anxiety, and locating cumulative activity and symptom data. Participants were encouraged to discuss the tasks they were completing and their thoughts as they completed the tasks, similar to the individual think aloud protocol.²⁶ The group session was audio-recorded, and a research assistant transcribed field notes. No other individuals were present during the focus groups.

Following the group session, participants were given one week to continue to use the app at their leisure, as if they were using the app to complement their mTBI rehabilitation and/or to pace and plan their daily activities. No further direction on what resources (Tips and Tricks) or number of activities to record was offered to participants. Participants were instructed to complete and return the Internet Evaluation and Utility Questionnaire (UQ)^27–29 to the research team by the end of the week. The UQ is a 15-item questionnaire that assesses 13 constructs: ease of use, convenience, engagement, enjoyment, layout, privacy, satisfaction, acceptability, comprehension, credibility, likelihood of returning, mode of delivery, and helpfulness. The 13 items are rated on a 5-point Likert scale ranging from 1 (not at all) to 5 (very). The final two items are open-ended questions assessing the most and least helpful components of the app. The UQ questions were modified to apply to the Parkwood Pacing and Planning^TM application (Table 2).

Table 2.

UQ responses according to study group on a 5-point Likert scale (1 = not at all, 5 = extremely).

Participant	How easy was the app to use?	How convenient was the app?	Did the app keep your interest and attention?	Did you like the mobile app?	Did you like the way the app looked?	Were you concerned about your privacy when using the app?	How satisfied were you with the app?	How acceptable was the app?	How useful was the app's information	How easily could you understand the app's information?	How trustworthy was the app's information?	How likely are you to use the app if you had persistent symptoms?	How well did the app convey pacing and planning information?
Clinician 1	4	4	3	3	4	1	3	4	3	3	3	3	3
Clinician 2	4	4	4	4	4	1	4	4	4	4	4	4	4
Clinician 3	4	4	4	4	5	2	4	3	3	4	4	3	4
Clinician 4	4	4	5	4	4	1	5	5	4	5	5	5	5
Clinician Average	4	4	4	3.75	4.25	1.25	4	4	3.5	4	4.25	3.75	4
Patient 1	4	4	4	4	1	4	4	4	4	4	4	5	4
Patient 2	5	3	4	4	5	1	4	4	3	4	5	4	4
Patient 3	3	3	3	4	4	1	3	3	2	3	4	4	1
Patient 4	4	4	4	3	4	1	4	4	2	3	3	3	2
Patient Average	4	3.5	3.75	3.75	4.25	1	3.75	3.75	2.75	3.5	4	4	2.75

The rating scale for the question ‘How concerned were you about your privacy in using the mobile app’ was reversed (1 = extremely, 5 = not at all).

Data analyses

Qualitative analyses of the focus group transcripts were performed using van Manen's phenomenological approach^30,31 to understand the meaning of participant's lived experiences using the application. Two researchers (SJ and SM) performed an existential investigation to generate data by recording and transcribing what participants said while completing their pre-specified tasks during the focus groups. Two additional researchers (MB and LG) completed thematic analyses on the focus group data using phenomenological reflection. Each transcript from the clinician-participant and patient-participant focus groups was independently analyzed by MB. During transcript analyses, similar quotes were grouped and categorized. Afterwards, categories were clustered into themes. Data analyses were then reviewed by LG, a senior qualitative researcher, to ensure saturation of the data. Transcripts and themes were not reviewed by participants for additional feedback.

Descriptive statistics were used to characterize age, sex, mechanism of injury (patients), and clinical background (clinicians). The question on privacy in the UQ was negatively worded; therefore, the reverse scale for this question was used (i.e., a higher total score reflected a more positive user experience) to allow a total UQ score.³² Both qualitative and quantitative analyses were performed with post-positivist assumptions.

Results

Four patient-participants (45 ± 12.7 years; three females) and four clinician-participants (40 ± 4.7; four females) completed this study, and no individuals refused or dropped out. All participants had some form of post-secondary education, either a college diploma or university degree. Three of the clinician-participants were occupational therapists, and one was a speech and language pathologist. All clinician-participants had more than eight months experience working with patients with an mTBI. Patient-participants sustained their mTBI from falls (three participants) and a bicycle accident (one participant).

Clinician and patient-participants positively rated the application (4.0 ± .3, 3.8 ± .2; respectively). Clinician responses were generally higher than patient responses (Table 2). ‘How good of a method was the mobile app for delivering this planning and pacing information’ and ‘How did you find the information in the mobile app?’ were the two questions with the biggest discrepencies between clinican-participant and patient-participant ratings. On average, clinicians rated ‘How good of a method was the mobile app for delivering this planning and pacing information’ 1.25 points higher than patient-participants, and ‘How did you find the information in the mobile app?’ 0.75 points higher than patient-participants. Clinician and patient-participants both indicated the app was easy to use, with both groups averaging a rating of four (out of five).

The phenomenological reflection yielded four themes demonstrating application qualities that were important to participants (Table 3). These themes represented strengths or areas for consideration of application improvement or both. For simplicity, the positive version of the selected noun/adjective was used to describe the category (sub-theme) and theme. The first theme identified was simplicity. For example, ‘I find the clock easier now that I know how the clock works’ (Patient 4). However, participants indicated the clock and scroll bars were difficult to navigate, including ‘I find the slider to be a little tricky… It's not super smooth’ (Patient 1). Secondly, the application's adaptability was identified as a theme, including data personalization. ‘It has good options…for how you hurt your head’ (Clinician 1). The importance of this theme was further promoted when a participant said, ‘You need to have flexibility there [when repeating an event] …’ (Patient 2).

The third theme that emerged was conciseness, with suggestions to improve clarity including, ‘I almost wonder if there could be a definition to describe how those [difficulty and symptom aggravation] are different…’ (Clinician 1). Furthermore, the placement of some buttons (e.g., save) could be in a more intuitive place, but ‘at least it prompted you if you didn’t do it [click save]’ (Clinician 3). The fourth and final theme to emerge was familiarity, including the benefits of similarities between this application and others. Many comparisons were made, including ‘I’m used to in my calendar in tapping a date and doing that…’ (Clinician 4) and ‘… There are better clocks out there…’ (Patient 2). The complete list of themes, categories, and exemplar quotes are listed in Table 3.

Table 3.

Themes, categories, and exemplar quotes from focus group transcripts highlighting application qualities important to participants.

Theme	Category	Exemplar Quotes
Simplicity: Something that is plain, natural, and/or easy to understand or do	Ease of use	‘I find the slider to be a little tricky to be honest. It's not super smooth.’—Patient 1 ‘I find the clock easier now that I know how the clock works.’—Patient 4
	Intuitive	‘I know that [falling asleep] goes with sleeping and calming yourself, because if I didn’t take the program I wouldn’t know what mindfulness was.’—Patient 3 ‘I… thought the calendar went Sunday to Sunday, but it starts on Monday.’—Clinician 1
	Consistent	‘I don’t like how the scales aren’t the same all the way through.’—Clinician 2
Adaptability: The ability to adjust to new conditions	Personalization	‘It had good options… for how you hurt your head.’—Clinician 1 ‘[When editing the point value for grocery shopping] That's still good because you can still customize it afterwards…’—Clinician 3
Adaptability: The ability to adjust to new conditions	Flexible	‘I want to be able to change the date but I can’t.’—Patient 2 ‘You need to have flexibility there [when repeating an event]. It might be alternate days, it might be each week, it might be the first Monday of the month.’—Patient 2
Conciseness: Excluding extra or unnecessary information	Organization	‘Not a big deal but the save button on the bottom, I would worry that some of my patients would miss it.’—Clinician 3 ‘At least it prompted you if you didn’t do it [click save].’—Clinician 3
Conciseness: Excluding extra or unnecessary information	Clarity	‘I found the “how did you hurt your head” [question] vague.’—Patient 1 ‘I almost wonder if there could be a definition to describe how those [difficulty and symptom aggravation] are different…’—Clinician 1
Familiarity: A quality of being well-known	Similarity to other devices/applications	‘In my texting app I can enlarge my font by using my volume button on my device. I thought I’d be able to do that here.’— Clinician 3 ‘… There's better clocks out there…’—Patient 2 ‘I’m used to in my calendar in tapping a date and doing that so it's just a process that I’ve developed.’—Clinician 4

Discussion

Preliminary analyses indicated individuals with an mTBI and clinicians working with individuals with an mTBI had a positive experience while using the Parkwood Pacing and Planning™ application. However, modifications that improve simplicity, adaptability, conciseness, and familiarity may further improve the user's experience. On average, clinicians and patients positively rated the usability, likability, usefulness, understandability, and convenience of the application via the UQ. Participants stated the application included features that promoted each theme; however, they also suggested ways to further improve the aforementioned themes. Accordingly, the Parkwood Pacing and Planning™ application should undergo modifications informed by these findings to further improve the application, which may promote application uptake and long-term use.

Successful mHealth applications must be acceptable to end-users, which often includes both patients and healthcare providers.³³ Although clinicians generally rated the Parkwood Pacing and Planning™ application higher than patients, the largest difference between clinician and patient ratings was for the method of information delivery. Clinicians may have highly rated the method of delivery because mobile applications increase service delivery, and access to care greatly influences clinicians’ adoption of mHealth tools.³⁴ Previous research also shows electronic devices are acceptable and desirable compensatory aids for individuals with a moderate to severe TBI.¹⁷ However, patients in this study may have rated the application lower than clinicians because they prefer to receive this information in-person from a healthcare provider. Therefore, these results support previous recommendations to include mobile technology as a supplement to rehabilitation programs, and not replace in-person treatment.¹⁶

Clinicians also indicated the information in the app was more useful than patient-participants. This discrepancy may be attributed to the patient-participants already completing mTBI rehabilitation, and therefore having pre-exisiting knowledge of pacing and planning. As this information was not new to them, they may not have found it useful. Conversely, clinicians may have rated the information useful because it contained education they consistently provide to their patients. Accordingly, there may have been differences in clinician and patient-participant's perception of the application's ability to enhance users’ rehabilitation.³⁵ Furthermore, differences in ease of accessing information³⁶ may have influenced participant's ratings.

mHealth applications should supplement, and not replace, in-person treatment because more than half of the individuals with a TBI have difficulty using their smartphone following their injury.¹⁹ mHealth applications that are simple and familiar to users may improve uptake and utilization because they require less learning, which can be negatively influenced by cognitive impairments.^19,37,38 Positive responses to the questions ‘How easy was the information to understand in the mobile app’ and ‘How easy was the mobile app to use’ indicate the application was simple and familiar. However, participants stated the Parkwood Pacing and Planning™ application could be improved by using the same scales for all questionnaires, which has previously been identified in a similar study with individuals with an mTBI.³⁹ Making all of the scales consistent in the application will also improve compliance and data quality,³⁹ which may improve future studies on the Parkwood Pacing and Planning™ application and ultimately further our knowledge on mTBI symptomology and rehabilitation.

Additional suggestions for improving the Parkwood Pacing and Planning™ application included the ability to change the date, moving the ‘Save’ button to the top of the screen when adding events to the calendar, clarifying the definitions of difficulty and symptom aggravation, and changing the calendar layout to start on Sundays. The suggestion to move the ‘Save’ button and changing the calendar to start on Sundays highlight the importance of familiarity and organization. Vision deficits contribute to increased difficulty navigating a smartphone following a TBI.¹⁹ Accordingly, easily viewed displays are one of the most important features of an application for individuals with a TBI.¹⁹ In improving the organization of the application, participants will require less mental exertion and less screen-time; both of which contribute to headaches following an mTBI.⁴⁰ With fewer symptoms, long-term mHealth application uptake and use may be improved.¹⁸ Therefore, evaluation in change of symptoms while using the application will be a critical component of the next phase of our systematic development program.

This study was not performed without limitations. First, study participants only included individuals who were familiar with pacing and planning, as inclusion criteria required participants who were referred to the Acquired Brain Injury Outpatient Program. Therefore, these findings cannot be extrapolated to individuals with an mTBI who do not already know how to pace and plan their daily activities. Secondly, this study did not evaluate participant's long-term application use, as the goal of these focus groups was to provide immediate feedback for application development. Although participants positively rated the application, it is unclear if these positive ratings will remain over time and result in high usage. Third, all participants were highly educated, making them more likely to use mHealth applications.⁴¹ Experience with mHealth applications may have helped these participants intuitively navigate and understand the Parkwood Pacing and Planning™ application, which may have inflated responses to questions like ‘How easy was the app to use’ from the UQ. Lastly, cognitive impairment was not evaluated in patient-participants. Cognitive impairment may negatively influence a user's ability to learn and use the application;¹⁶ therefore, these results may not apply to all individuals with an mTBI. Future studies with individuals that are not familiar with pacing and planning, have a breadth of education levels, and evaluate long-term use (adherence and effectiveness) and cognitive impairment (impact of patient characteristics on outcomes) should be performed.

Conclusion

Preliminary analyses indicated individuals with an mTBI and clinicians working with individuals with an mTBI rated the Parkwood Pacing and Planning™ application positively. Analyses of participants’ experiences provided insight on areas to focus future application modifications to improve user experience: simplicity, adaptability, conciseness, and familiarity. The Parkwood Pacing and Planning^TM application could be improved by implementing changes that support the specific needs of individuals with an mTBI, such as consistent measurement scales, organized screen layouts, and clear and concise content. Additional evaluation is planned to explore these suggested changes, level of cognitive impairment, and long-term application usage patterns, with individuals new to pacing and planning.

Footnotes

Acknowledgements

We would like to thank Jonathan Binder and Dr Robert Kent at Windsor University for developing the prototypes of the Parkwood Pacing and Planning™ application.

Declaration of Conflicting Interests

This mobile application was created to support a government-funded brain injury rehabilitation program. Although Laura Graham and Becky Moran are employed as clinicians at Parkwood Institute, the authors have no financial interest in the results of this study.

Funding

This project was funded by Cowan Insurance Group, St Joseph's Healthcare Foundation (grant number 039-1516), and the Brain and Mind Matters Fund through the Stratford Perth Community Foundation.

ORCID iD

Marquise M. Bonn

Appendix A

References

Gordon

Kuhle

. Reported concussion’ time trends within two national health surveys over two decades. Brain Inj 2018; 32: 843–849. 2018/04/14.

Cassidy

Carroll

Peloso

, et al. Incidence, risk factors and prevention of mild traumatic brain injury: results of the WHO collaborating centre task force on mild traumatic brain injury. J Rehabil Med 2004; 36: 28–60.

Silverberg

Panenka

Iverson

. Work productivity loss after mild traumatic brain injury. Arch Phys Med Rehabil 2018; 99: 250–256. 2017/08/02.

Bloom

Thomas

Ahrensberg

, et al. A systematic review and meta-analysis of return to work after mild traumatic brain injury. Brain Inj 2018; 32: 1623–1636. 2018/10/12.

Purcell

Davis

Gioia

. What factors must be considered in ‘return to school’ following concussion and what strategies or accommodations should be followed? A systematic review. Br J Sports Med 2019; 53: 250–250.

Rozbacher

Selci

Leiter

, et al. The effect of concussion or mild traumatic brain injury on school grades, national examination scores, and school attendance: a systematic review. J Neurotrauma 2017; 34: 2195–2203. 2017/01/14.

Cassidy

Cancelliere

Carroll

, et al. Systematic review of self-reported prognosis in adults after mild traumatic brain injury: results of the international collaboration on mild traumatic brain injury prognosis. Arch Phys Med Rehabil 2014; 95: S132–S151. 2014/03/04.

Cancelliere

Verville

Stubbs

, et al. Post-concussion symptoms and disability in adults with mild traumatic brain injury: a systematic review and meta-analysis. J Neurotrauma 2023. 2022/12/07. DOI: 10.1089/neu.2022.0185.

Langer

Bayley

Levy

. Evaluating access to appropriate concussion care in Ontario. Arch Phys Med Rehabil 2013: 94. DOI: 10.1016/j.apmr.2013.08.202

10.

Ponsford

Willmott

Rothwell

, et al. Impact of early intervention on outcome following mild head injury in adults. Journal of Neurology, Neurosurgery and Psychiatry 2002; 73: 330–332. 2002/08/20.

11.

Silverberg

Gardner

Brubacher

, et al. Systematic review of multivariable prognostic models for mild traumatic brain injury. J Neurotrauma 2015; 32: 517–526. 2014/09/16.

12.

Ontario Neurotrauma Foundation. Guidelines for concussion/mild traumatic brain injury & persistent symptoms: for adults (18 + years of age). Ontario Neurotrauma Foundation, 2018.

13.

Silverberg

Iverson

. Is rest after concussion “the best medicine?”: recommendations for activity resumption following concussion in athletes, civilians, and military service members. Journal of Head Trauma Rehabilitation 2013; 28: 250–259. 2012/06/13.

14.

World Health Organization. mHealth: new horizons for health through mobile technologies: second global survey on eHealth. Healthc Inform Res 2011; 64: 66–71.

15.

Steinhubl

Muse

Topol

. Can mobile health technologies transform health care? JAMA 2013; 310: 2395–2396. 2013/10/26.

16.

Juengst

Hart

Sander

, et al. Mobile health interventions for traumatic brain injuries. Curr Phys Med Rehabil Rep 2019; 7: 341–356.

17.

Hart

Buchhofer

Vaccaro

. Portable electronic devices as memory and organizational aids after traumatic brain injury: a consumer survey study. J Head Trauma Rehabil 2004; 19: 351–365.

18.

Bradley

Haun

Powell-Cope

, et al. Qualitative assessment of the use of a smart phone application to manage post-concussion symptoms in Veterans with traumatic brain injury. Brain Inj 2020: 1–8. 2020/06/05. DOI: 10.1080/02699052.2020.1771770

19.

Wong

Sinclair

Seabrook

, et al. Smartphones as assistive technology following traumatic brain injury: a preliminary study of what helps and what hinders. Disabil Rehabil 2017; 39: 2387–2394. 2016/10/18.

20.

Lee

Sullivan

Schneiders

, et al. Smartphone and tablet apps for concussion road warriors (team clinicians): a systematic review for practical users. Br J Sports Med 2015; 49: 499–505. 2014/03/29.

21.

King

Crawford

Wenden

, et al. The Rivermead Post Concussion Symptoms Questionnaire: a measure of symptoms commonly experienced after head injury and its reliability. J Neurol 1995; 242: 587–592.

22.

Fetters

Curry

Creswell

. Achieving integration in mixed methods designs-principles and practices. Health Serv Res 2013; 48: 2134–2156. 2013/11/28.

23.

Creswell

Klassen

Plano Clark

, et al. Best practices for mixed methods research in the health sciences. Bethesda (Maryland): National Institutes of Health, 2011.

24.

Tracy

. Qualitative quality: eight “big-tent” criteria for excellent qualitative research. Qual Inq 2010; 16: 837–851.

25.

Hwang

Salvendy

. Number of people required for usability evaluation: the 10 ± 2 rule. Commun ACM 2010; 53: 130–133.

26.

Fonteyn

Kuipers

Grobe

. A description of think aloud method and protocol analysis. Qual Health Res 1993; 3: 430–441.

27.

Schubart

. An e-learning program to prevent pressure ulcers in adults with spinal cord injury: a pre-and post-pilot test among rehabilitation patients following discharge to home. Ostomy Wound Manage 2012; 58: 38–49.

28.

Thorndike

Saylor

Bailey

, et al. Development and perceived utility and impact of an internet intervention for insomnia. E-Journal of Applied Psychology: Clinical and Social Issues 2008; 4: 32.

29.

Ritterband

Ardalan

Thorndike

, et al. Real world use of an internet intervention for pediatric encopresis. Journal of Medical Internet Resarch 2008; 10: e16. 2008/07/26.

30.

van Manen

. Phenomenology of practice: Meaning-giving methods in phenomenological research and writing. New York: Routledge, 2016.

31.

van Manen

. Practicing phenomenological writing. Phenomenology+ Pedagogy 1984; 2(1): 36–69.

32.

Warmbrod

. Reporting and interpreting scores derived from Likert-type scales. J Agric Educ 2014; 55: 30–47.

33.

Norris

Stockdale

Sharma

. A strategic approach to m-health. Health Informatics J 2009; 15: 244–253. 2009/08/29.

34.

Jacob

Sanchez-Vazquez

Ivory

. Social, organizational, and technological factors impacting clinicians’ adoption of mobile health tools: systematic literature review. JMIR Mhealth Uhealth 2020; 8: e15935. 2020/03/05.

35.

Davis

. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989; 13: 319–340.

36.

Yilmaz

Verma

Craswell

, et al. Relevance and effort: an analysis of document utility. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, 2014, pp.91–100.

37.

Groussard

Pigot

Giroux

. From conception to evaluation of mobile services for people with head injury: a participatory design perspective. Neuropsychol Rehabil 2018; 28: 667–688. 2015/12/19.

38.

Powell

Wild

Glang

, et al. The development and evaluation of a web-based programme to support problem-solving skills following brain injury. Disabililty and Rehabilitation: Assistive Technology 2019; 14: 21–32. 2017/10/25.

39.

Juengst

Graham

Pulantara

, et al. Pilot feasibility of an mHealth system for conducting ecological momentary assessment of mood-related symptoms following traumatic brain injury. Brain Inj 2015; 29: 1351–1361. 2015/08/20.

40.

Silverberg

Panenka

. Headache trigger sensitivity and avoidance after mild traumatic brain injury. J Neurotrauma 2019; 36: 1544–1550. 2018/11/30.

41.

Bol

Helberger

Weert

JCM

. Differences in mobile health app use: a source of new digital inequalities? The Information Society 2018; 34: 183–193.