Sage Journals: Discover world-class research

Abstract

A promising approach to meeting the need in many jurisdictions for timely, in-depth qualitative health systems experience data, is to elicit feedback through smartphone and web applications (apps). Apps offer an appealing tool to elicit data from patients and family members who may feel stigma when receiving some services and a power imbalance when providing feedback to health-care providers. In this article, we examine the effectiveness of a suite of smartphone and web apps called myExperience (myEXP) that were created to gather care experiences of youth, family members, and service providers as part of an experience-based co-design (EBCD) study in Ontario involving youth with mental disorders. We analyzed data from 12 triads of youth (aged 16–24), family members, and service providers gathered between August 2015 and December 2016. We used qualitative content analysis to understand participant feedback on the myEXP apps and identify thematic categories that emerged from experience data elicited through the myEXP apps. We found overall that the myEXP apps were more effective at eliciting experience data from youth compared with family members and service providers. Rich experience data were gathered from youth about treatment plans in real time through the apps. The apps also showed important promise as reflective tools for all participants. They may offer advantages in research that seeks to improve responsiveness in service delivery and build mutual understanding. The apps also offer choice in how data are elicited, encourage more candid feedback and help to overcome stigma, which are important considerations for some vulnerable populations. For service redesign research using approaches such as EBCD, apps offer real-time data gathering that can complement and enhance traditional approaches such as retrospective interviews and observation.

Keywords

smartphone apps web apps mobile apps eliciting experience data qualitative data collection health-care experiences youth with mental disorders

Introduction

Health policy makers in many jurisdictions are increasingly calling for patient and family experience as a basis for improving the quality of health services and for system planning (Amann, Zanini, & Rubinelli, 2016; Government of Ontario, 2015; National Health Service, 2014), but there are challenges to doing this in practice. The literature suggests that traditional approaches such as satisfaction surveys (Shale, 2013; Wolf, 2014; Wolf, Niederhauser, Marshburn, & LaVela, 2014) may not capture the depth of patient experiences and values, with the result that patient perspectives are not appropriately represented in the planning and evaluation of health care (Avis, Bond, & Arthur, 1995).

One potential approach to meeting the increasing need for in-depth and timely qualitative experience data is to elicit feedback through smartphone and web apps. Apps have been used in EMA to collect data on feelings and experiences of children and adults to assess psychological issues over an extended period of time within the context that these experiences are occurring (Khor, Gray, Reid, & Melvin, 2014; Shiffman, Stone, & Hufford, 2007; Smith et al., 2012). This approach offers important advantages when addressing experiences of health system users, such as the ability to elicit responses in real time, reducing issues with recall (Garcia, Welford, & Smith, 2016; Haag Granello & Wheaton, 2004; Shiffman, 2009); over one or more episodes of care (Khor et al., 2014) or system transition points; across multiple services involved in a person’s care; and at a time that is convenient for users (Matthews, Doherty, Coyle, & Sharry, 2008). Furthermore, when it comes to designing system improvements, apps can be used to gather experience data across multiple perspectives, including patient, family, and service providers, which are essential in good service design (Bate & Robert, 2006, 2007a, 2007b). Apps can also offer a way to elicit experiences pertaining to stigmatized health conditions such as mental disorders (Arboleda-Florez, 2003; BinDhim et al., 2015; Corrigan, Watson, & Miller, 2006; Goodyear-Smith et al., 2017; Stuart, Arboleda-Florez, & Sartorius, 2012) or for vulnerable and disadvantaged populations where power imbalances are pronounced (e.g., youth with mental disorders; DeRoche & Lahman, 2008). In such contexts, apps may encourage more open sharing than during a face-to-face interview. These opportunities are particularly relevant in the mental health domain, where efforts are increasingly being made to involve system users as important partners in the process of system design and provision of services (Government of Ontario, 2011; National Health Service England, 2016).

A growing body of literature supports the potential for use of mobile apps to elicit experience data for mental health service use in Canada. First, mobile phone use is increasing across all Canadian demographics (Statistics Canada, 2016), and online health apps are well received by patients and families, with more than a third of Canadian smartphone owners having at least one health app on their phone (Catalyst, 2017; Statistics Canada, 2016). Apps are used in a multitude of ways by patients and families with mental disorders and are increasingly being used in mental health-care delivery (Christensen, Griffiths, & Evans, 2002; Reid et al., 2012).

While the potential for use of apps in eliciting health care experiences is clear, there is little known about their effectiveness in gathering data across multiple services and participant perspectives. In this article, we present findings from a suite of apps, called myEXP used to gather experiences in research to co-design improved youth mental health services in Ontario. We present participant feedback on the myEXP apps and discuss their effectiveness in eliciting experience data from youth, family members (or other lay support person identified by the youth), and service providers on an ongoing basis. We discuss implications for qualitative research and health service improvement.

Method

Our overarching research question was: How effective are the myEXP apps at eliciting experience data from youth, family members, and service providers as youth with mental health problems journey through services? We were interested in (a) understanding to what extent the myEXP apps were used, (b) obtaining user feedback about their experiences using the apps, and (c) identifying key themes that emerged from experience data elicited from the apps.

Overview of Data Collection Through myEXP Apps

As part of an EBCD research project (the myCo-design study) to improve coordination of care and transitions to adult care for youth with mental disorders in Ontario, our team developed a suite of award-winning (Davey Award 2015) smartphone (for youth) and web apps (for family members and service providers) called myEXP. EBCD is an approach to health service redesign rooted in the participatory research tradition (Bate & Robert, 2006, 2007a, 2007b) that aligns with policy objectives of experience-driven health-care improvement. A key step in the EBCD process is to identify touchpoints—or key moments in the service experience where there is a strong emotive response (positive or negative) that powerfully shapes the overall experience. The apps were designed to elicit the experiences of young people, their family member, and their primary service provider as youth journey through mental health services.

Most applications of EBCD have been within a single service, with experience data gathered retrospectively using interviews or observation (Donetto, Tsianakas, & Robert, 2014; Larkin, Boeden, & Newton, 2015). The use of the myEXP apps offers potential to elicit a more fulsome understanding of the touchpoints over the course of the mental health journey across multiple services as they occur. In developing the myEXP questionnaires, we drew from principles of effective care coordination in youth mental health services that include quality of relationship with the provider, information sharing, collaboration across providers, patient involvement in care planning, and family involvement in services (Tobon, 2013). We also drew upon experience mapping theory by considering experiences during attraction to, entry into, engagement in, exit from, and extension following each service encounter (Schauer et al., 2013).

Youth enter any mental health–related appointment into the myEXP smartphone app as it is booked or based on a weekly reminder prompt sent via e-mail on Sunday evenings. One hour after the scheduled appointment time, the youth receives a prompt to complete the myEXP app post-appointment questionnaire. This includes two types of questions: (a) questions with yes or no response options (e.g., Did you meet with the scheduled provider?) and (b) questions about appointment experiences pertaining to care coordination and transitions that have an emotion-face response scale and an optional box for open-ended comments (e.g., How did you feel when you arrived at your appointment?; How did you feel about: (a) the provider?, (b) information sharing?, (c) family involvement, and (d) your treatment plan(s)?; How did you feel after leaving the appointment?; and How did you feel about your overall experience?). Questions are also asked to identify the stage of the journey, for example, whether this was a first, last, or ongoing appointment with the provider, and feelings about any upcoming changes or transitions across providers. The questionnaire also asks about whether appointments took place at the scheduled time and with the intended provider and associated feelings. Figure 1 provides a graphic representation of the myEXP youth questionnaire including the questions posed.

Figure 1.

myEXP Smartphone App Questionnaire for Youth.

In addition to the post-appointment questionnaire, youth can provide open-ended spontaneous feedback about their experiences through the myComments section of myEXP at any point in their journey. This is particularly important in the event of unplanned service encounters, or when the youth is waiting for or between services. The youth version of myEXP also provides links to online mental health resources. Youth also complete a brief intake survey when the app is installed on their phone. The intake survey asks basic demographic information and prompts youth to enter the name and e-mail addresses of the people they identify to represent the roles of family member and service provider as part of the study.

The myEXP apps for family members and service providers are web-based. An online platform was selected for service providers because they were unlikely to want to install it on a personal phone or to carry two smartphones. Similarly, not all family members were expected to own a smartphone in 2014, at the time myEXP was being developed. An e-mail link to the appropriate online post-appointment questionnaire is sent to the family member and service provider the evening following any appointment that the youth has entered into the smartphone app. Although immediate post-appointment feedback would be preferred, we anticipated that evening might be a better option for service providers who may be too busy to complete the questionnaire if other clients were waiting to be seen and for family members who may not have attended the appointment with the youth. The service provider and family member post-appointment questionnaires ask about the same principles of care coordination as the youth questionnaire, with refinements for each perspective. For example, it was important to know what role the family member played in the appointment (if any) and how they felt about their level of involvement in the youth’s mental health care. It was also important to understand whether the service provider was commenting on their own appointment with the youth, or their extent of knowledge of an appointment the youth had with a different service provider.

Recruitment

We used purposive sampling to identify 12 triads of youth¹ aged 16–24 receiving mental health services because of a mental disorder or behavioral difficulty, the family member they identified, and their current primary service provider. The triads enabled us to gather data on experiences from three perspectives about each service encounter. Note that youth had to own a smartphone or tablet to participate in the study. Youth participants were recruited through service provider organizations in Hamilton, Brampton, and Niagara regions of Ontario. Upon recruitment by service providers, a member of the research team (first, second and third authors) met with youth to discuss participation in the study, complete the informed consent process, conduct an intake interview, download the myEXP app, and provide training.The intake screens were completed by the youth during this initial meeting with the researchers. At that meeting, youth identified the family member they wished to participate with them as well as the current primary service provider (if this individual differed from the provider who recruited them into the study). A member of the research team called the youth two weeks following the initial meeting to check in about the smartphone app functionality and respond to any questions that the youth had about using the app. The research team checked in with youth via text message or telephone monthly for the duration of the youth’s time in the study.

Data Collection and Analysis

Data were collected using the myEXP apps and through individual interviews at intake, 6 and 12 months over the period August 2015 and December 2016, but depending on when youth were recruited into the study, they may have used the apps for longer. The interview guides had two sections: one pertaining to experience using the myEXP apps and the other pertaining to experiences of mental health service use since the previous interview. Sample questions about experiences using the apps were: One of the goals we had in making the app was to see if you could use it to record your experience. Tell me what it has been like for you using the myEXP app; Another thing we tried was to have you let us know when you would be having appointments. How is that going?; and We wanted to provide you with the opportunity in the myComments section to enter anything about your experience at any point in time rather than after an appointment. How is that working for you? The interviews were audio recorded and transcribed verbatim using a transcription service.

To analyze the performance of the myEXP apps in eliciting experience data for this article, we considered several dimensions. First, we summarized data on frequency of myEXP app usage. Second, the lead author employed qualitative content analysis (Sandelowski, 2000) to analyze interview feedback about participant experiences using the myEXP apps. Third, we purposively selected data from one triad of users to illustrate the three perspectives on service experiences. This triad was selected because (i) all members had entered qualitative data into the myEXP apps, (ii) they had used the app for a longer than average period (9 months), (iii) they had close to average usage at twice a month, and (iv) the youth had completed a questionnaire following every appointment. Fourth, we analyzed the experiences of care coordination elicited by the apps. We first categorized the open-ended data using continuity of care principles in youth mental health services (Tobon, 2013) as an “a priori” coding framework. The first and second authors then independently reviewed the content associated with each principle to identify emergent themes pertaining to touchpoints and met to discuss any discrepancies until consensus was reached. All qualitative data were managed using NVivo 10 qualitative software.

The project was approved by the Hamilton Integrated Ethics Board under project number 15-059. Ethical concerns focused on confidentiality and security of data. All data are stored on a secure server at McMaster University; no data are stored on the smartphone. A secure URL for the web apps are used, and all transactions are through a secure protocol. Two firewalls are used for network level security. All data from the smartphone app are encrypted, and only research team members have access to the data. We use a code to present quotes attributable by participant type (youth: Y, family member: FM, and service provider: SP) and an identifying triad number. For example, Y4 refers to the youth in triad 4.

Findings

Overview of myEXP App Usage

Table 1 provides a summary of the myEXP app usage. For youth, average usage (2.8 entries per month) slightly exceeded the number of appointments (2.4 appointments per month) over the time period, reflecting usage of the intake survey, the post-appointment questionnaires, and myComments response options.

Table 1.

myEXP App Data Summary.

Triad	Months of App Usage	Number of Appointments	Monthly Average		Total Number of Entries by Participant Type
Triad	Months of App Usage	Number of Appointments	Youth Appointments	Youth myEXP Entries	Youth myEXP Post-appointment	Youth myComment	Family Member	Service Provider
1	2	17	9	11	17	5	0	0
2^a	17	34	2	2	34	3	22	24
3	9	18	2	2	18	1	6	13
4	1	1	1	1	1	0	0	0
5	1	3	3	4	3	1	1	1
6	4	1	0	1	1	1	0	0
7	2	2	1	1	2	0	0	0
8	6	12	2	2	12	1	3	0
9	2	4	2	2	4	0	0	0
10	4	5	1	2	5	3	0	0
11	6	19	3	2	19	0	6	3
12	6	20	3	4	20	2	6	0
Minimum	1	1	0	1	1	0	0	0
Maximum	17	34	9	11	34	5	22	24
Mean	5	11.3	2.4	2.8	11.3	1.4	3.7	3.4

^a This triad joined the study early in the recruitment process and continued to use myEXP throughout the data collection period.

The highest usage was by a triad that joined the study at the beginning of the recruitment phase and continued using the app throughout the entire data collection phase. This youth had 34 appointments over a 17-month period, completed 34 post-appointment questionnaires, and made three myComments entries. The family member and service provider completed 22 and 24 post-appointment questionnaires, respectively. In five other cases, there was high usage by the youth but typically lower usage by family members and service providers. In two cases, the youth only completed the intake survey, and so there was no input from family members or service providers. In one of those cases, the youth discontinued app use following referral to another service. In other cases, the youth may not have entered or may have incorrectly entered the e-mail addresses, as discussed in the interview feedback below.

Feedback From myEXP Users

Overall, most users had positive feedback about the myEXP apps as tools for eliciting experiences (Table 2). Participants described the myEXP apps as user-friendly, simple to use, and helpful in providing reminders about upcoming appointments. The questionnaire structure with emotion faces and optional open-ended comments was well received. Youth liked reflecting on their experiences, and service providers felt that completing the questionnaires encouraged them to be more reflective about their practice, not only with the participating youth but also with all their clients.

Table 2.

Summary of User Feedback About the myEXP Apps.

Feedback Theme	Youth	Family Members	Service Providers
Positive features of the myEXP apps
Overall impression	“It’s been good. It’s very easy to use…. I could probably put in a little bit more detail, but it’s something I’d do after my appointment or on the bus real quick.” Y6	“I think it’s fine…because you don’t want to make it too complicated.” FM3	“It’s pretty straightforward.” SP4
User-friendly	“The layout was very simplistic…straightforward, and I didn’t get confused with the calendar…it was very user-friendly.” Y4	“I liked the faces better…[they] capture everything…and it was simple and fast.” FM3	“It’s been really easy to complete, it’s not a burden in my day.” SP3
Questionnaire structure	“There was really happy, and really sad, and…there were a couple of middle ones, so I like that. I liked how it was before and after, because…sometimes I was going into appointments and felt completely different than I felt after.” Y1	“I felt that it made it easier and quicker, like a user-friendly kind of thing because you know how you’re feeling…. And then of course the box. If you had anything to add then you had the opportunity to do that…that’s probably the best.” FM1
Use as a reflection tool	“It [enables] me to share what I’ve been through, or what’s helped in the past. It’s also good to recognize my emotions about how I feel about the appointment, or after the appointment.” Y2“I like that you can reflect on your experience…that’s kind of cool.” Y6	“That [open text space] would probably be cathartic for me.…I probably would have used that to get the information out, because it was so frustrating for me at that time.”^a FM1	“That [questionnaire] gives [an opportunity to] reflect on myself [in my practice].” SP2
Technical issues
Operating system upgrades	“Other than the brief time of not working, it’s been working pretty good.” Y6	“I didn’t get any [e-mails].” FM2	“For some time, the application was not working.” SP2
Notifications	“I noticed that there wasn’t any notifications on my phone which made it completely slip out of my mind.” Y4	“I did receive the questionnaires…no problems there. Then…the app was down for a bit, and after that I never received another questionnaire.” FM4
Lost information	“I’d be on the app 1 week. The next week it would be like, ‘Oh, you have to re-enter your provider, your name, your e-mail.’” Y2“There was a lot of things that I tried to enter into the myEXP app, but a lot of it was not saved.” Y4
Suggested improvements for the myEXP apps
Notifications to enter experience data	“A text would be better, because I’m not always on my e-mail.” Y2“I have a school e-mail…I check that one. When I started checking that one, I stopped checking my personal one. I think a text prompt would be a lot more useful” Y6	“[Notifications are] a good reminder to make sure you do it.” FM3“I knew when they were coming, and I expected [them] because I knew when the appointments would be. But then I don’t know if…it’s maybe a glitch or something because I wasn’t getting any e-mails, and then I thought maybe she had cut me off.” FM1^a	“You can’t really reach out to them [youth] and be like, ‘Hey, I just saw an e-mail pop up here; what’s going on?’ because you’re…no longer involved.” SP1“These follow-up [calls] you are missing that. That telephone call is basically to find out [how] he was doing, does he need an urgent support, an immediate meeting.” SP2
Refining the questionnaire	“I just didn’t know what to write…. I would’ve used it [open text space] more than what I actually did…they [the faces] were just right there.” Y2	“One weakness I saw in the questionnaire. There was no way to deal with that [cancelled appointment]. I got the thing and it asked me still, ‘How did it go?’” FM4	“I think there should be one more option [about family attending the appointment]…I’m trying to make him more independent…. Because I talk with them [family members] on phone, but when he’s coming, I want him to come alone.” SP2
Details of appointment in notification	“…syncing it up with my calendar in my phone. That’d be cool if it could do that. I get with security and stuff, I do get that, but that would be cool.” Y1		“It would be helpful if the questionnaire told me which appointment that was referring to…and then also just to prompt me look at my calendar to deduce it.” SP3^b

^a This family member did not receive any e-mail notifications and so did not actually use the questionnaire. ^bThe name of the youth is not shared in the e-mail prompt to protect privacy concerns.

At the same time, there were some technical challenges and frustrations that arose when iOS and Android operating system were updated, leaving the myEXP app software nonfunctional for several months. One youth indicated that some intake survey data were not saved and had to be reentered.

There were also several suggestions to improve the notification systems. Family members appreciated the e-mail notifications; however, youth would have preferred that notifications be sent by text rather being a numerical notification appearing on the app icon. One service provider was frustrated that they continued to receive notifications for a youth who had left the practice. Some service providers and family members were frustrated that they did not receive notifications with questionnaires links, likely because there was an error in the e-mail address entered into the smartphone app when the youth intake survey was completed. Some family member and service provider participants suggested linking notifications to appointments entered in their own calendars directly. Youth suggested that the myEXP app calendar be synchronized automatically with the calendar on their phone to avoid duplicate appointment entry.

Usefulness of myEXP Apps in Gathering Experience Data

Table 3 provides an example of data elicited from one triad of participants in order to illustrate the kind of qualitative data that can be gathered using a brief smartphone app. Several points are notable from this example. First, the themes discussed differed by participant type. The youth commented about feelings prior to and leaving the appointments. For example, the youth expressed apprehension about not completing homework prior to an upcoming appointment and mixed feelings about leaving the provider and moving on to the next stage of their journey. The youth’s open-ended myComments entry reflected optimism with respect to transitioning to adult services. The family member commented mostly about not feeling as involved in different aspects of the youth’s care as they would like to be. The family member wanted more information about the treatment plan but was not as concerned about attending the appointment with the youth. The service provider data reflected a more professional stance and were more descriptive and less emotive about experiences than the youth and family data. In this case, the youth was discharged from the service provider’s care part way through the study, so there was less to comment on.

Table 3.

Illustrative Real-Time Open-Ended Comments About Experiences From a Single Triad of myEXP Users.

Domain of Experience	Youth	Family Member	Service Provider
Questionnaire following appointment
Feelings about upcoming appointment	“Didn’t have time to completely do the homework [appointment prep]. [I’m] a little worried they’re [service providers] gonna be disappointed” Y3“Just have to finish my post treatment plan and I’ll be done.” Y3
Feelings after leaving the appointment	“It was a really good and supportive group and I’m happy I participated.” Y3
Feelings about level of involvement in your family member/friend’s care		“I'm there to support but I’m not as involved as I’d like to be” FM3
Feelings about your family member/friend’s treatment plan		“Didn’t know anything about the plan.” FM3
Feelings about not attending your family member/friend’s appointment		“I don’t really care if I’m there or not.” FM3
Feelings about your role in the youth’s appointment			“I scheduled the appointment; I did not attend the appointment.” SP3
How did the youth get in contact with this provider?			“I am occasionally a co-facilitator for this intervention. [Youth] is enrolled in it through me originally.” SP3
Anything else to share at this point about your experience providing service to this youth			“I have changed positions so I am no longer a part of [Youth’s] circle of care. I can only guess at what appointment she has based on what I had organized. It is possible that this appointment was arranged through a different provider, and I do not have a way of knowing that unless I ask directly.” SP3
myComments
Open-ended comment	“I’m feeling good I’ve had a very good transition into adult services so far and I’m very lucky and thankful for that. And I just finished one step of my plan and now I’m on to another so I’m feeling pretty optimistic.” Y3

Themes From Youth Experience Data Gathered by myEXP Apps

As discussed above, key questions that were posed in the myEXP apps pertained to the feelings about relationship with providers, information sharing, family involvement, treatment plan, transition to new providers, and overall experience. Table 4 presents the themes that emerged and illustrative quotes from youth participants. We focus on youth data, given the smaller number of family and service provider entries.

Table 4.

Themes Emerging From myEXP App Data Entered by Youth and Illustrative Quotes.

Themes	myEXP App Quote
Interactions with providers
Having a good relationship with providers	“I feel like I have a good connection with her.” [Y1]“I feel my counsellor has better understanding than my doctor because both understand about research and healthcare.” [Y2]
Communication and support from providers	“…last time she didn’t seem to care but, whatever.” [Y8]“…the doctor is very kind and knows what he’s doing.” [Y1]
Provider insensitivity	“…he’s been problematic in the past especially when it comes to mental health.” [Y3]“…he wasn’t very helpful and didn’t listen to me at all and is very, very problematic when it comes to dealing with mental health and I am currently seeking a new doctor to replace him.” [Y3]
Information sharing
Not enough or inappropriate information being shared	“I feel very self-conscious about information that is shared about me. It makes me very uncomfortable and puts me in an awkward position and makes me feel upset.” [Y12]
Violations of privacy	“I have been in a constant battle with [service] in regards to sharing information with people I don’t want to have involved in my case or life. They consistently disrespect my privacy and confidentiality.” [Y12]
Family involvement
Reflections about the youth’s support network	“My family is extremely supportive.’” [Y1]“Family is not involved for my well-being.” [Y12]
Treatment plan
Medication: Effects and management	“Hospital appointments are very tough, medications are too much.” [Y2]
Not satisfied with treatment plan	“I feel as though [service] doesn’t think about what is best for me as their client and chooses what is best for them as an organization instead. As it stands right now I am not in a good position in terms of my treatment plan. The only plan for me when I leave [youth housing service] is for me to live and bounce from shelter to shelter. Which is going to deteriorate my wellbeing drastically.” [Y12]
Feeling equipped with strategies for coping	“…we talked about a new way to deal with my negative thoughts that doesn’t involve worksheets (annoying!!!)” [Y1]
Having a plan for services	“I’ll be going back to day treatment which makes me feel happy.” [Y1]
Transition to new providers
Reactions to waiting for services	“…this is the first appointment I’ll be having since about 2 months ago, I’m pretty hyped because a lot of new things have happened.” [Y5]
Overall experience
Experiences along the journey in real time	“I’ve been getting worse. I’m going back to the hospital right now because I still want to kill myself.” [Y1]^a
Reflections about self-discovery that occurred during service journey	“At first I was doing really terrible and then things started to get much better and then I stopped going which just made everything hell.” [Y5]^a

^a myComments.

When discussing interactions with providers, youth spoke about having good relationships. For example, one youth described that their “ doctor is very kind and knows what he is doing” and building strong connections [Y1] with some providers. There were also experiences of extreme insensitivity in particular interactions “he didn’t listen to me at all” [Y3], and statements about how clearer communication could help to address some issues. When it came to information sharing, violations of privacy and inappropriate or insufficient information sharing were reported as highly negative touch points by one youth, such as feeling “consistently disrespected,” “very self-conscious,” “uncomfortable,” and being put “into an awkward position” due to information sharing [Y12].

Youth had mixed reactions about family involvement. Youth indicated either that their family was a great source of support, explaining, “my family is very supportive” [Y1] or conversely, unhelpful in the journey through mental health services. Another youth explained that “family is not involved for my well-being” [Y12]. In addition, youth had strong feelings about their treatment plans. One youth spoke about feeling her mental health status was at risk because there was an inadequate plan in place for her to transition out of a shelter.

The only plan for me when I leave [youth housing service] is for me to live and bounce from shelter to shelter. Which is going to deteriorate my wellbeing drastically. [Y12]

Another youth was very unhappy about medication and hospital treatment, which they found to be very difficult. Youth also shared positive touchpoints. One youth was very happy about returning to a day treatment program, and another was happy about finding a care plan that did not involve filling out worksheets, which were described as “very annoying!!!” [Y1]. One youth who had been waiting for 2 months to transition to a new provider described being “pretty hyped” about an upcoming appointment [Y5].

When it came to statements about the overall experience, the app data provided youth with the opportunity for reflection and catharsis, in a similar way as was reported previously (Table 2) by family members and service providers. One youth noted in myComments, how much he had improved when using services, and now realized that he deteriorated after stopping services.

Discussion

Our analysis suggests that apps offer a potentially important and innovative elicitation method by which to identify touchpoints during health service journeys (Miatello, Mulvale, & Roussakis, 2017). As demonstrated with myEXP pilot data, apps offer the ability to gather repeated observations over the course of a service journey (Hackett, Mulvale, & Miatello, 2018; Schauer et al., 2013) that may be particularly helpful in understanding experiences of those with episodic or chronic conditions such as cancer care or mental health care (Donetto et al., 2014).

Apps may be a particularly important elicitation approach for youth, who often feel more comfortable sharing feelings and opinions on apps (Ambresin, Bennett, Patton, Sanci, & Sawyer, 2013; Rickwood, Deane, & Wilson, 2007), than in face-to-face interviews with adults. This is especially the case for youth receiving mental health services, where the balance of power during an interview may resemble a mental health visit (Heflinger & Hinshaw, 2010; Matthews et al., 2008). Apps also offer insight into changing experiences as they occur. Youth in our study were comfortable sharing strong feelings about aspects of their care plans as they evolved over time, that they might not have recalled during a retrospective interview. For vulnerable populations, it is important that apps be simple, intuitive, and offer choice, as a way to empower users (Bonney & Stickley, 2008). In the case of myEXP, there is choice about how users share their experience data, for example, through a structured questionnaire scale, open-ended response options, spontaneous entry (Russ et al., 2013), or a combination, and we found that different options were used by different participants. Future iterations of myEXP could offer choice with respect to the login identity or personal representation (e.g., selection of an avatar) on the device (Boyd, 2014; Gardner & Davis, 2013), which may be particularly appealing for stigmatized groups (Arboleda-Florez, 2003; BinDhim et al., 2015; Goodyear-Smith et al., 2017; Laugharne & Priebe, 2006). Other choice options could be personalization of the colors used or the background wallpaper on the app, selection of frequency and timing of notifications, or the type of honorarium or gamification reward (Helf & Hlavacs, 2016) offered.

Other myEXP features, including the use of visual cues such as a face-based feelings scale, may be particularly helpful in EBCD studies to identify emotional touchpoints. This approach was particularly well received by our participants who uniformly found it quick, intuitive, and easy to use. Research suggests that people may find a face-based scale easier to process than a written scale, which may be an advantage for youth with mental disorders or those with cognitive challenges (Stange, Barry, Smyth, & Olson, 2016). At the same time, this approach may have limitations for some participant groups where a face-based feelings scale may not be culturally relevant (Lee, Jones, Mineyama, & Zhang, 2002), as suggested by one service provider in our study. The relevance of myEXP as an elicitation tool for different cultural groups is an important area for future research.

At the same time, there are potential disadvantages to using apps to elicit qualitative data. For example, we found that structured questionnaires require careful design because of the limited ability to ask probing follow-up questions (Teachman & Gibson, 2013) that may make it difficult to reach saturation in qualitative inquiry (Morse, 1995). At the same time, we were surprised by the depth of detail provided in the open-ended boxes and the spontaneous entries made using the myComments feature by some participants. Further potential exists to use apps to gather more complex data by enabling users to upload photos, videos, or voice recordings. This could also provide greater choice and sense of empowerment for app users. A further disadvantage is the reliance on participants to continue to enter data, which can be challenging for those experiencing severe depression, anxiety, or psychosis. In conversations with youth and family members, we heard several reasons why particular youth stopped using myEXP, such as being admitted to a treatment program that did not allow use of technology (e.g., upon admission to a hospital psychiatric unit), choosing to reduce or discontinue smartphone use more generally, or losing or no longer having a smartphone for other reasons. A lesson learned from our study design was that data entry by one participant should not depend on another participant continuing to stay involved or accurately entering information. Such a dependency prohibited some service providers and family members from receiving questionnaire links.

There were also some unanticipated effects of using apps for data collection. For example, participants from all three perspectives commented on using myEXP as a tool for reflection either about their service journey (youth and families) or their service delivery (service providers; Lapuman, Yau, Church, Ruttonsha, & David, 2015). These unanticipated consequences may be disadvantageous for those researchers hoping to elicit data without changing the phenomenon being studied. On the other hand, researchers seeking reflective insights could enable participants to view their previous entries to witness and reflect on their changing experiences over time. During pretesting of myEXP, youth suggested that future versions allow them to see the data they have previously entered to monitor their own progress. From a research perspective, an unexpected benefit was how ongoing interactions between participants and the researchers about app usage helped to build rapport and strengthen relationships with the research team over time, which is critical in EBCD research (Hackett et al., 2018).

Overall, we found that apps may be an important tool in the qualitative researcher’s elicitation tool kit that can be particularly helpful in certain contexts, such as for youth with mental disorders (Reid et al., 2012; Statistics Canada, 2016) and for other vulnerable and stigmatized populations. The ability to elicit rich and timely experience data for the purposes of qualitative research or health service improvement in the participant’s naturalistic setting is consistent with the essence of qualitative inquiry (Rolfe, 2006). It can help to reduce issues with recall of experiences (Depp et al., 2010; Shiffman, 2009) and enhance mutual understanding over the course of a service journey (Khor et al., 2014; Shiffman et al., 2007). Triangulating experience data gathered in real time through apps with other data sources such as interviews, observations, or documentary analysis will ultimately enhance the credibility of the qualitative study (Patton, 1999). In EBCD practice, it may be beneficial to use the app data to identify challenges with service delivery and then probe more deeply about them at a follow-up interview.

Our analysis suggests that myEXP holds great promise in gathering rich data from youth to improve care coordination. In future work, we hope to work closely with youth, family members, and service providers to codesign refinements to the app to address some of the challenges experienced in this study. In particular it is important to improve app functionality for increased data elicitation from family members and service providers and to codesign ways to incorporate additional data gathering while respecting privacy concerns. Examples of other potential data sources include photos, videos, or voice recording. We will also test the apps across a larger sample and broader range of services to enable a more fulsome comparison of experience data elicited by participant perspectives, across cultural groups and across the different app features (e.g., questionnaires vs. myComments).

Strengths and Limitations

To our knowledge, this is the first study to examine the use of apps as an elicitation tool for ongoing data gathering about experiences from multiple perspectives over the course of a service journey. We assess effectiveness based on analysis of app usage, participant feedback, comparison across perspectives, and identification of key touchpoints that emerged. These findings contribute new insights about how to operationalize real-time data gathering through apps as an additional elicitation option for qualitative researchers. Apps may be particularly applicable for certain vulnerable populations, for example, youth and stigmatized groups, and in contexts where real-time feedback is important, for example, in ongoing quality improvement.

At the same time, this research is subject to a number of limitations. Analysis of the feedback related to the myEXP apps is based on a small number of triads recruited into a pilot study, and in some instances, the triads were incomplete due to attrition (e.g., family member or service provider had dropped out, or the youth was discharged from the service provider’s care). In instances where youth chose not to use the smartphone app, we had no way to continue gathering family member or service provider data through the web apps. Similarly, experiences associated with unplanned service interactions (e.g., hospital or crisis services) were only elicited if the youth chose to use myComments without prompting. Also, due to study time constraints, some triads had the opportunity to use the myEXP for a year or more, while others only had 6 months. At this time, we do not know whether usage patterns and the nature of the experience data elicited changes over time. Finally, the approach is conditional on youth having a smartphone, or access to a tablet. Given these limitations, we cannot assess based on these findings whether apps could be viewed as a substitute for in-depth interviews and observations, which are key methodologies in EBCD (Bate & Robert, 2007a, 2007b; Mulvale, Milatello, Hackett, & Mulvale, 2016) and recommend that they be used as a complementary elicitation approach to gather feelings. They are most helpful in eliciting experience data over the course of a service journey in a timely fashion, across multiple perspectives and sites of service delivery, and may be particularly helpful when seeking to understand experiences of vulnerable populations.

Conclusion

Qualitative researchers interested in understanding service user experiences are witnessing a new era wherein policy makers and health-care organizations are increasingly demanding experience data be incorporated in system and service planning. At the same time, technology is offering new possibilities with respect to how to elicit this essential information. Apps offer the opportunity to elicit experiences over a period of time, in the naturalistic setting, across multiple services in the health system and from different perspectives. Our findings suggest that apps can be an effective way to collect experience data for methods such as EBCD in the mental health domain. Apps may be important for other groups who may experience stigma and may find technology more comfortable. At the same time, researchers choosing any elicitation technique must consider potential impacts on the phenomenon of study beyond the intended use for research purposes. In the case of youth mental health discussed here, the elicitation tool itself has the potential for greater reflection by all users that can result in a mental health system that is more responsive to service user needs as issues arise, consistent with the aims of EBCD.

Footnotes

Authors’ Note

The views expressed in this article are the views of the authors and should not be taken to represent the views of the Government of Ontario.

Acknowledgments

The authors would like to acknowledge Christina Roussakis for her excellent research coordination and support throughout the project. The authors would also like to acknowledge and thank the youth, family members, and service providers who participated in the myCo-design study. Sincere thanks to the Ontario Ministry of Research, Innovation, and Science. Additional funding support provided by the DeGroote School of Business at McMaster University is gratefully acknowledged.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article: This research is funded through an Ontario Early Researcher Award.

ORCID iD

Gillian Mulvale

Note

References

Amann

Zanini

Rubinelli

(2016). What online user innovation communities can teach us about capturing the experiences of patients living with chronic health conditions. A scoping review. PLoS ONE, 11, e0156175. doi:10.1371/journal.pone.0156175

Ambresin

A. E.

Bennett

Patton

G. C.

Sanci

L. A.

Sawyer

S. M.

(2013). Assessment of youth-friendly health care: A systematic review of indicators drawn from young people’s perspectives. Journal of Adolescent Health, 52, 670–681. doi:10.1016/j.jadohealth.2012.12.014

Arboleda-Florez

(2003). Considerations on the stigma of mental illness. The Canadian Journal of Psychiatry, 48, 645–650.

Avis

Bond

Arthur

(1995). Satisfying solutions? A review of some unresolved issues in the measurement of patient satisfaction. Journal of Advanced Nuring, 22, 316–322. doi:10.1046/j.1365-2648.1995.22020316.x

Bate

Robert

(2007a). Bringing user experience to healthcare improvement: The concepts, methods and practices of experience-based design. Oxford, England: Radcliffe.

Bate

Robert

(2007b). Toward more user-centric OD: Lessons from the field of experience-based design and a case study. The Journal of Applied Behavioral Science, 43, 1. doi:10.1177/0021886306297014

Bate

Robert

(2006). Experience-based design: From redesigning the system around the patient to co-designing services with the patient. Quality and Safety in Health Care, 15, 307–310. doi:10.1136/qshc.2005.016527

BinDhim

N. F.

Shaman

A. M.

Trevena

Basyouni

M. H.

Pont

L. G.

Alhawassi

T. M.

(2015). Depression screening via smartphone app: Cross-country user characteristics and feasibility. Journal of the American Medical Informatics Association, 22, 29–34.

Bonney

Stickley

(2008). Recovery and mental health: A review of British literature. Journal of Psychiatric and Mental Health Nursing, 15, 140–153.

10.

Boyd

(2014). It’s complicated: The social lives of networked teens. New Haven, CT: Yale University Press.

11.

Catalyst. (2017). Canadian smartphone behaviour in 2017: Continued shifts to viral spaces. Retrieved from http://catalyst.ca/2017-canadian-smartphone-behaviour/

12.

Christensen

Griffiths

K. M.

Evans

(2002). e-Mental health in Australia: Implications of the internet and related technologies for policy. Retrieved from http://unpan1.un.org/intradoc/groups/public/documents/apcity/unpan046316.pdf

13.

Corrigan

Watson

A. C.

Miller

F. E.

(2006). Shame, blame, and contamination: A review of the impact of mental illness stigma on family members. Journal of Family Psychology, 20, 239–246. doi:10.1037/0893-3200.20.2.239

14.

Depp

C. A.

Mausbach

Granholm

Cardenas

Ben-Zeev

Patterson

T. L.

… Jeste

D. V.

(2010). Mobile interventions for severe mental illness: Design and preliminary data from three approaches. Journal of Nervious and Mental Disorders, 198, 715–721.

15.

DeRoche

K. K.

Lahman

M. K. E.

(2008). Methodological considerations for conducting qualitative interviews with youth receiving mental health services. Forum: Qualitative Social Research, 9, 17.

16.

Donetto

Tsianakas

Robert

(2014). Using experience-based co-design (EBCD) to improve the quality of healthcare: Mapping where we are now and establishing future directions. London, England: National Nursing Research Unit. Retrieved from http://www.kcl.ac.uk/nursing/research/nnru/publications/reports/ebcd-where-are-we-now-report.pdf

17.

Garcia

Welford

Smith

(2016). Using a smartphone app in qualitative research: The good, the bad and the ugly. Qualitative Research, 15, 508–525. doi:10.1177/1468794115593335

18.

Gardner

Davis

(2013). The app generation: How today’s youth navigate identity, intimacy, and imagination in a digital world. New Haven, CT: Yale University Press.

19.

Goodyear-Smith

Martel

Darragh

Warren

Thabrew

Clark

T. C.

(2017). Screening for risky behaviour and mental health in young people: The YouthCHAT programme. Public Health Reviews, 38, 20. doi:10.1186/s40985-017-0068-1

20.

Government of Ontario. (2015). Patients first: Action plan for health care. Retrieved from http://www.health.gov.on.ca/en/ms/ecfa/healthy_change/docs/rep_patientsfirst.pdf

21.

Government of Ontario. (2011). Open minds, healthy minds: Ontario’s comprehensive mental health and addictions strategy. Retrieved from http://www.health.gov.on.ca/en/common/ministry/publications/reports/mental_health2011/mentalhealth_rep2011.pdf

22.

Haag Granello

Wheaton

J. E.

(2004). Online data collection: Strategies for research. Journal of Counselling and Development, 82, 387–393.

23.

Hackett

Mulvale

Miatello

(2018). Co-designing for quality: Creating a user-driven tool to improve quality in youth mental health services. Health Expectations, 1–11. doi:10.1111/hex.12694

24.

Heflinger

C. A.

Hinshaw

S. P.

(2010). Stigma in child and adolescent mental health services research: Understanding professional and institutional stigmatization of youth with mental health problems and their families. Administration and Policy in Mental Health, 37, 61–70. doi:10.1007/s10488-010-0294-z

25.

Helf

Hlavacs

(2016). Apps for life change: Critical review and solution directions. Entertainment Computing, 14, 17–22. doi:10.1016/j.entcom.2015.07.001

26.

Khor

A. S.

Gray

K. M.

Reid

S. C.

Melvin

G. A.

(2014). Feasibility and validity of ecological momentary assessment in adolescents with high-functioning autism and Asperger’s disorder. Journal of Adolescence, 37, 37–46. doi:10.1016/j.adolescence.2013.10.005

27.

Larkin

Boeden

Newton

(2015). On the brink of genuinely collaborative care: Experience-based co-design in mental health. Qualitative Health Research, 25, 1463–1476. doi:10.1177/1049732315576494

28.

Laugharne

Priebe

(2006). Trust, choice and power in mental health: A literature review. Social Psychiatry and Psychiatric Epidemiology, 41, 843–852.

29.

Lapuman

Yau

Church

Ruttonsha

David

A. M.

(2015). Un-earthing emotions through art: Facilitating reflective practice with poetry and photographic imagery. Journal of Medical Humanities, 36, 171–176.

30.

Lee

J. W.

Jones

P. S.

Mineyama

Zhang

X. E.

(2002). Cultural differences in responses to a Likert scale. Research in Nursing and Health, 25, 295–306.

31.

Matthews

Doherty

Coyle

Sharry

(2008). Designing mobile applications to support mental health interventions. In Lumsden

(Ed.), Handbook of research on user interface design and evaluation for mobile technology (pp. 635–656). Hershey, PA: IGI Global. doi:10.4018/978-1-59904-871-0.ch038

32.

Miatello

Mulvale

Roussakis

(2017). Using experiences to improve transitions from youth to adult mental health services: Understanding the perspectives and values of youth, family members and service providers for improving care processes. The Journal of Mental Health Policy and Economics, 20, s25.

33.

Morse

J. M.

(1995). The significance of saturation. Qualitative Health Research, 5, 147–149.

34.

Mulvale

Miatello

Hackett

Mulvale

(2016). Applying experience-based co-design with vulnerable populations: Lessons from a systematic review of methods to involve patients, families and service providers in child and youth mental health service improvement. Patient Experience Journal, 3, 117–129.

35.

National Health Service. (2014). The NHS five year forward view. Retrieved from http://www.england.nhs.uk/wp-content/uploads/2014/10/5yfv-web.pdf

36.

National Health Service England. (2016). Children and youth people’s mental health local transformation plans—A summary of key themes. Retrieved from https://www.england.nhs.uk/mentalhealth/wp-content/uploads/sites/29/2016/08/cyp-mh-ltp.pdf

37.

Patton

M. Q.

(1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34, 1089–1208.

38.

Reid

S. C.

Kauer

S. D.

Khor

A. S.

Hearps

S. J. C.

Sanci

L. A.

Kennedy

A. D.

Patton

G. C.

(2012). Using a mobile phone application in youth mental health. Australian Family Physician, 41, 711–714.

39.

Rickwood

Deane

F. P.

Wilson

C. J.

(2007). When and how do young people seek professional help for mental health problems? Medical Journal of Australia, 187, 35–39.

40.

Rolfe

(2006). Validity, trustworthiness and rigour: Quality and the idea of qualitative research. Journal of Advanced Nursing, 53, 301–310. doi:10.1111/j.1365-2648.2006.03727.x

41.

Russ

L. R.

Phillips

Brzozowicz

Chafetz

L. A.

Plsek

P. E.

Blackmore

C. C.

Kaplan

G. S.

(2013). Experience-based design for integrating the patient care experience into healthcare improvement: Identifying a set of reliable emotion words. Healthcare, 1, 91–99. doi:10.1016/j.hjdsi.2013.07.004

42.

Sandelowski

(2000). What ever happened to qualitative description. Research in Nursing & Health, 23, 334–340.

43.

Schauer

Quattlebaum

Risdon

Bowman

Daghlian

Brune

… Hui

(2013). Adaptive path’s guide to experience mapping. Retrieved from http://mappingexperiences.com/#thankyoufordownloading

44.

Shale

(2013). Patient experience as an indicator of clinical quality in emergency care. Clinical Governance: An International Journal, 18, 285–292. doi:10.1108/CGIJ-03-2012-0008

45.

Shiffman

(2009). Ecological Momentary Assessment (EMA) in studies of substance use. Psychological Assessment, 21, 486–497. doi:10.1037/a0017074

46.

Shiffman

Stone

Hufford

(2007). Ecological momentary assessment. Annual Review of Clinical Psychology, 4, 1–32. doi:10.1037/a0017074

47.

Smith

Harms

W. D.

Burres

Korda

Rosen

Davis

(2012). Enhancing behavioral health treatment and crisis management through mobile ecological momentary assessment and SMS messaging. Health Informatics Journal, 18, 294–308. doi:10.1177/1460458212445349

48.

Stange

Barry

Smyth

Olson

K. M.

(2016). Effects of smiley face scales on visual processing of satisfaction questions in web surveys. Sociology Department, Faculty Publications, 337. Retrieved from http://digitalcommons.unl.edu/sociologyfacpub/337

49.

Statistics Canada. (2016). Communications monitoring report 2016: Canada’s communication system: An overview for Canadians. Retrieved from https://www.crtc.gc.ca/eng/publications/reports/policymonitoring/2016/cmr2.htm

50.

Stuart

Arboleda-Florez

Sartorius

(2012). Paradigms lost: Fighting stigma and the lessons learned. New York, NY: Oxford University Press.

51.

Teachman

Gibson

B. E.

(2013). Children and youth with disabilities: Innovative methods for single qualitative interviews. Qualitative Health Research, 23, 264–274.

52.

Tobon

J. I.

(2013). Continuity of care in children’s mental health: Development of a measure (Unpublished doctoral dissertation). London, England: The University of Western Ontario.

53.

Wolf

J. A.

(2014). Patient experience, satisfaction not one in the same. Hospital Impact. Retrieved from https://www.fiercehealthcare.com/hospitals/patient-experience-satisfaction-not-one-and-same

54.

Wolf

J. A.

Niederhauser

Marshburn

LaVela

S. L.

(2014). Defining patient experience. Patient Experience Journal, 1, 7–19.

Data Elicited Through Apps for Health Systems Improvement

Abstract

Keywords

Introduction

Method

Overview of Data Collection Through myEXP Apps

Recruitment

Data Collection and Analysis

Findings

Overview of myEXP App Usage

Feedback From myEXP Users

Usefulness of myEXP Apps in Gathering Experience Data

Themes From Youth Experience Data Gathered by myEXP Apps

Discussion

Strengths and Limitations

Conclusion

Footnotes

Authors’ Note

Acknowledgments

Declaration of Conflicting Interests

Funding

ORCID iD

Note

References