Abstract
The objective was to describe the utility of mixed methods to inform the development of an automated telephone intervention to improve patients’ compliance with asthma medication. As part of intervention development for a larger trial, we conducted 15 focus groups (n = 53) to design and develop calls, and to identify factors influencing intervention acceptability and usefulness. We piloted four call types and interviewed 64 participants to further improve call content and receptivity to the intervention. Feedback led to several changes to the intervention scripts and eventual calls, and an initial pilot led us to drop one of the calls. During the pilot, we reached 43 percent of target participants; 74 percent of those stayed on the call until it ended. This process provided key insights about automated calls, and may have broader applicability for the development of automated interventions designed to help patients manage a variety of chronic conditions.
Keywords
Introduction
Automated telephone messaging systems (commonly called interactive voice recognition or IVR systems) have been used in multiple studies to collect health data, monitor patients, provide health education, and remind patients about appointments or health screening activities. 1 IVR systems can be used to provide health education, to increase compliance with childhood and influenza immunizations or with medication regimens, and to assist in management of cardiovascular disease and diabetes. 2–9 In a systematic review of IVR-mediated applications, Krishna et al. concluded that automated telephone interventions can have a significant impact on both behavioral and clinical outcomes. 10
IVR systems use voice-processing technology to link individuals to their health information stored in a database. Pre-recorded voice files generally prompt the caller to press telephone buttons to answer questions or request information. The IVR system can then access the host computer’s database, and give information back to the patient. 11,12 Recent applications of IVR technology utilize advanced speech recognition software that allows patients to respond to recorded queries with natural-language answers, thus potentially enhancing the acceptance and effectiveness of this telephone-based interaction. 13,14 Overall, IVR systems have the advantage of delivering tailored health information and scripts to large populations at low cost, as well as efficiently collecting data. 12,15 Thus, IVR systems may be a promising strategy to enhance chronic-disease management in managed care settings. 15,16
Finding ways to maximize the acceptability, usability, and usefulness of IVR interventions to patients may be key to their success. Yet, although a variety of healthcare settings seem to be increasingly utilizing IVR systems to improve patient care, 17 little is known about exactly how patients receiving these types of interventions respond to them. In particular, review of the literature suggests that few, if any, studies have explored participants’ reaction to, and use of, IVR and other automated phone technology during design and implementation stages of the intervention. 11,18,19
We report our findings from a mixed-methods approach used to develop an IVR intervention using speech recognition software. This article describes the mixed-methods approach we used and summarizes the lessons it provided to us as part of our intervention development. The findings should have broader applicability for the development of automated interventions designed to help patients manage a variety of chronic conditions.
Methods
Study overview
This study is based on the intervention development phase for a larger randomized clinical trial, ‘Phone Calls to Promote Adherence with Inhaled Corticosteroids’ (PEANUT), testing the effectiveness of telephone outreach that uses speech recognition software to improve compliance with inhaled corticosteroids among patients with asthma.
The objective of the developmental evaluation reported here was to enhance the acceptability and usefulness to asthma patients of an IVR intervention to improve compliance with medication. We used a mixed-methods approach relying on three components that built upon each other: (a) member focus groups; (b) pilot testing of IVR calls; and (c) qualitative debrief phone interviews in a subset of those who received pilot calls. Specifically, we wanted to obtain participants’ feedback about getting automated calls, including reasons for continuing or hanging up, and to evaluate areas where we could improve the intervention during the development phase of the study. The pilot test of calls provided a gauge of how many participants the intervention might reach, how long participants stayed on the call, how many accepted a transfer to the refill line or pharmacist, and points in the script where people commonly hung up.
Research setting and participant recruitment
The study setting was Kaiser Permanente Northwest (KPNW) and Kaiser Permanente Hawaii (KPH). Kaiser Permanente (KP) is a group-model health maintenance organization (HMO) that provides comprehensive, prepaid healthcare services to about 450,000 members in KPNW and 230,000 members in KPH. KPNW serves a population that is largely Caucasian (~91%), while the KPH population includes about 25 percent Caucasians, 47 percent Asians, native Hawaiians, or Pacific Islanders, and 29 percent mixed heritage. Kaiser Permanente’s electronic medical record (EMR), which provides both administrative and clinical data, was utilized to identify potential study participants for recruitment.
We recruited focus group and pilot participants from a group of KPNW and KPH members who met the eligibility criteria for the larger PEANUT trial. Criteria included being 18 years or older, being diagnosed with asthma, having had at least one dispensing of an anti-asthma medication, and having health maintenance organization membership for the past year (Figure 1).
Patient recruitment for the three components of the developmental evaluation
Qualitative methods
We utilized qualitative methods, including focus groups and interviews, because they elicit participants’ perspectives, 20–22 and are therefore useful in defining the range and variability of beliefs, behaviors, and experiences of study populations, as well as the natural language used to discuss these issues. 20–24
Focus group methods
First, we conducted a series of 15 focus groups (KPNW = 7; KPH = 8) to aid in call design and content development, and in creating an acceptable IVR intervention. 25 We chose a two-phase approach so that changes to the call scripts, based on feedback from the first round of focus groups, could be ‘member checked’ with participants in the second round. 20–24 Approximately 400 potential focus group participants received letters of invitation, followed by a phone call from study staff. Participants attending focus groups were provided with a meal and gift card. Participants attended two consecutive focus groups, approximately 2 months apart, each of which lasted approximately 2 hours. The first round of focus groups explored barriers participants experienced in both taking and filling asthma medications as prescribed. Participants also listened and reacted to sample telephone calls that used IVR technology. Data from the initial focus groups were used to create sample intervention telephone call scripts, and scripts were recorded to mirror what the intervention telephone calls would sound like in real time. Approximately 2 months later, participants were invited back, listened to the sample telephone calls, and provided feedback, which was used to further refine the content and delivery of the scripts for the larger clinical trial.
IVR pilot call methods
We used focus group data to assist in programming the phone intervention script content and branching logic. We tested four unique call types (welcome call, new-user call, simple refill reminder call, and tardy refill call) with 500 patients (KPNW = 250; KPH = 250). We selected equal numbers of participants by site and call-type eligibility. Pilot participants received an invitational letter and brochure, including an opt-out postcard. Those not opting out received an IVR pilot call in month 1 of the pilot, and may or may not have received further calls in the subsequent 3 months of the pilot, according to their eligibility as determined by monthly queries of dispensing data from the electronic medical record.
The four call types tested in the pilot included the following features, depending on call type. The average call lasted less than 2 minutes. Figure 2 shows a sample call flow and branching logic.
Sample call flow and branching logic
Welcome call: orientation to refill reminder service and active consent (included in all call types if the participant was interacting with the IVR intervention for the first time).
Simple refill reminder: brief refill reminder and offer to transfer to refill line.
Tardy refill reminder: refill reminder, assess asthma control, provide medication education, offer suggestions for overcoming barriers to compliance with medication, and offer refill line and/or pharmacist transfer.
New-user/restart call: first fill reminder, assess asthma control, provide medication education, offer suggestions for overcoming barriers to compliance with medication, and offer pharmacist transfer.
Qualitative phone interview methods
We contacted 64 of the pilot participants for telephone debrief interviews. All pilot participants who had received at least one of the four call types were eligible to be contacted for a debrief phone interview. We purposely recruited individuals equally from among those who had received the different call types, and we over-sampled participants who hung up compared with those that completed a call in order to gain better understanding of reasons for hanging up. 22–24,26 Participants were contacted approximately 5–7 days after receiving an IVR call, and telephone interviews lasted approximately 20 minutes. Participants who fully completed an IVR call were asked about their overall reaction to the call content and IVR technology, the perceived usefulness of the call, and their likelihood of taking future calls. Participants who hung up early in the call or declined participation were asked their reasons for discontinuing the call, as well as about their overall reaction to the call content and IVR technology, the perceived usefulness of the call, and the likelihood of taking future calls.
Analysis methods
Focus groups 25 and telephone interviews 27 were designed, conducted, and analyzed by the first-named author (JS), a researcher trained in qualitative methodology, and by other study staff also trained in qualitative methods. A common interview guide was developed for both rounds of focus groups, and for telephone debrief interviews, to increase consistency across interview sessions while also allowing individual participants’ experiences to emerge. 24–27 Focus groups were audio taped and professionally transcribed for analysis. Telephone interviews were manually recorded in detail. We used a thematic content analysis approach, guided by a specific set of standard qualitative data reduction and reconstitution techniques that included coding passages of text. 28–31 For example, we developed a coding dictionary based on the interview guides and review of transcribed interviews. Transcribed interviews were then coded by marking passages of text with phrases indicating the content of the discussion. We utilized a qualitative research software package, ATLAS.ti 5.0 (Scientific Software Development, 1997) to electronically code the transcripts and generate summary reports of coded text. Using the report and query functions of ATLAS.ti, coded text was further reviewed through an iterative process, resulting in refined themes.
Pilot call data were analyzed by placing them into mutually exclusive groups based on the best call result of any call attempt made and how much call content was received by the participant during the call. The two primary, mutually exclusive groups identified were: confirmed target reached; and confirmed target not reached. To help assess level of participant engagement in the call attempt, the confirmed target reached group was further divided into the following subcategories: willingness to continue with the call after confirming identity; intervention began; normal wrap-up/medication refill transfer option offered; answering machine message left; and declined taking a message for confirmed target. For the confirmed target not reached group, subcategories of data included: hang-up; wrong number; and unreachable (busy or no answer).
Results
Focus groups
A total of 53 individuals (KPNW = 31; KPH = 22) participated in the focus groups (Tables 1 and 2). The first round primarily focused on identifying common barriers to taking and filling asthma medications. The three most commonly cited barriers were: (1) forgetfulness/too busy; (2) lack of routine; and (3) not understanding the reason and purpose for the asthma medication. Regarding refill behavior, the primary barrier related to a sense of feeling good or being symptom free, thus causing patients to choose not to refill or to forget to refill. These participant-identified barriers to taking and filling asthma medication were incorporated into the IVR scripts, providing guidance on call content, the logic/structure of the call, and the natural language used (see Table 1, column 4). Participants also briefly listened to some sample IVR calls and provided feedback, particularly focusing on what could be improved to keep the participant engaged in the phone call.
Examples of how focus group results guided development of IVR script design and content
During the second round of focus groups, participants listened and reacted to the newly designed and recorded IVR intervention calls. Analysis of participants’ reactions and feedback revealed seven key factors to improving intervention acceptability and usability: (1) immediate and clear identification of call source and purpose; (2) personalization and relevance of message; (3) acknowledgement of automated technology; (4) delivery style of call; (5) voice personality; (6) duration of call; and (7) accompanying written information. These key factors and corresponding patient recommendations (Table 2) were incorporated into the IVR telephone scripts and intervention recruitment materials.
Key factors to increasing acceptability and usability of IVR calls as identified by focus group participants
IVR pilot calls
We invited 500 eligible members to receive reminder phone calls for their asthma medications, and made 992 calls to them over a 4 month test period (Table 3). Of attempted calls, 47 percent resulted in reaching the target health plan member and keeping him/her on the telephone long enough to hear ‘Hello, this is the Kaiser Permanente Breathe Easy Medication Reminder Program …’. We successfully left a message in an additional 36 percent of attempted calls. The telephone was not answered or was busy in 17 percent of attempted calls. In 15 percent of attempted calls, the member interacted with the intervention to the fullest extent and had a normal wrap-up. An additional 5 percent interacted with the intervention to some degree, but hung up before the call concluded.
Summary of pilot call outcomes
Of those reached, 57 percent hung up before the intervention portion of the call began (49% hung up and 8% declined participation in the program during the consent portion of their first call). Twenty-six percent were offered a transfer to either the automated refill line or a pharmacist, and of those who were offered, 49 percent accepted the transfer (this is 13% of the 495 calls resulting in reaching the target member and 6% of the total number of attempted calls).
Debrief qualitative phone interviews
Of the 64 telephone interviews conducted, 22 were with participants who had fully completed one of the IVR phone calls (Table 4). Of these, 91 percent had a positive overall reaction to the call; 95 percent found the purpose of the call clear; 82 percent found the call timely and useful; 91 percent reacted positively to the automated technology; 23 percent chose to use the transfer option; and 86 percent indicated a willingness to receive future IVR calls.
Participant reactions to the IVR calls: synopsis of key findings from qualitative debrief phone interviews
We also interviewed 42 participants who either hung up before the call conclusion, or declined participation shortly after the call began (Table 4). Of these, 31 percent were not interested in the IVR call due to already having a good medication reminder system in place; 19 percent believed they did not receive the call and assumed another member of their household took the call on their behalf; 17 percent indicated the call came when they were too busy or rushed to take it; 12 percent were not currently living at the designated phone number; 12 percent were confused by or struggled with the automated technology (mostly older participants); and 7 percent cited language barriers.
Regardless of whether participants completed a call, 63 percent (40/64) of participants interviewed indicated they found the purpose of the IVR calls clear and the automated technology acceptable. Furthermore, 52 percent (33/64) of participants interviewed indicated they would take future calls from the intervention even if they initially did not complete their first call experience.
In general, reaction to the calls was similar across all four call types. However, participants who received only the welcome call were more likely to be uninterested in the service than participants who received other call types. Additionally, approximately 10 percent of participants interviewed indicated that the name of the program (‘Medication Reminder Program’) was potentially confusing by giving the impression that calls might also be about non-steroidal breathing medications or about other medications entirely. This feedback led us to change the program name on all materials to ‘Breathe Easy Medication Reminder Program’. Additionally, feedback from the debrief calls, in combination with pilot call data, led us to discontinue the welcome call, as participants did not perceive it as useful or meaningful.
Discussion
This three-stage developmental intervention process led to numerous valuable findings that helped to shape the final intervention used in the main randomized clinical trial. The findings from the focus groups, pilot test, and qualitative phone interviews provided understanding about how participants may interact and engage with IVR calls relying on speech recognition technology for the purposes of improving compliance with asthma medication.
To our knowledge, previous health studies utilizing qualitative methods to assess participants’ use of, and reaction to, automated phone calls have not reported relying on developmental focus groups. 11,18,19 The qualitative data we gathered in the developmental focus groups helped identify important areas of script design and content that would increase call acceptability. These data led us to modifications we would not have otherwise considered, such as immediately identifying the call source, making calls more regionally/culturally appropriate, determining which type of voice and call style would be most effective, identifying the call as automated, and including additional written information (e.g. descriptive letter/brochure sent prior to call receipt).
Studies 11,18,19 using qualitative methods to assess participants’ reactions to IVR calls have found participants react negatively to the call when: content is repeated or lacks new information; they feel talked down to; interactions are inflexible; they are not understood by the technology; they are too busy; and the content and tone of the call make them feel guilty. Our qualitative data identified similar issues. However, because we conducted formative focus groups, pilot tested the calls, and conducted debrief interviews prior to finalizing the intervention, we were able to clearly identify and modify upfront words or phrases that were repetitious or condescending, content areas that felt redundant, and points of branching logic in the call that created confusion. The two-phased focus group approach allowed us to identify these issues early in the design phase, make adjustments, and then recheck the altered calls in the second round of focus groups. We were than able to further refine the calls based on issues raised in the qualitative debrief telephone interviews and patterns observed during the pilot test.
Two of the key findings from our focus groups as reported in the results section included the importance of identifying the call as automated and exploring the delivery style and tone of the call voice. In designing calls, we were uncertain as to whether stating upfront that the calls were automated or computerized would increase initial call hang-up. Focus group data informed us that participants want the call to be identified as automated because the call voice and aspects of the interaction sound human, yet the yes/no branching logic of the calls creates a formality in speech that does not necessarily replicate human interaction. We addressed this concern by including descriptive language in our one-time ‘opt-out’ portion of the call that made it clear the call was automated, and by providing a more detailed explanation and reasoning for utilizing automated calls (versus a live person) in the written recruitment/informational brochure.
Additionally, the recorded voice and delivery style of the call elicited the most negative reactions during the focus groups. The voice used on the call was particularly important in call acceptability for Hawaiian participants. Participants in Hawaii generally reacted more negatively to the recorded voice, perceiving it as unfriendly and condescending. Participants indicated they would likely hang up immediately if the current recorded voice was used because the voice did not sound like them, and lacked familiarity, friendliness, and credibility. Using a local voice in the phone recordings for Hawaii, along with customary local greetings, was an important design feature that increased call acceptability and meaningfulness for this population. We perhaps would not have considered recording the Hawaii-based calls in a voice reflective of the local culture had this not been identified as an issue during the focus groups.
We believe our qualitative data from the telephone debrief interviews with pilot participants help demonstrate that participants’ behaviors during the calls (hanging up before a call conclusion) had less to do with call content or issues with the automated technology, and more to do with other issues. The IVR technology itself does not appear to be a primary reason for discontinuing a call; participants who indicated they already had a good reminder system in place for filling and taking their ICS prescription were the most likely to indicate they might decline future intervention calls. However, participants who were too busy to take the call, or did not believe they received the call due to someone else in the household taking it on their behalf, were likely to indicate a willingness and desire for future IVR calls. Every participant we interviewed who reacted negatively to the computerized technology (seven out of the 64 interviewed) was older and cited hearing difficulties that made interacting with IVR phone technology challenging.
Quantitative pilot data informed us that shorter calls allowed for our key messages to reach more people, and that shortening time spent in the call introduction increases the rate of intervention delivery. In the pilot, 57 percent of those reached by the automated reminder call prematurely hung up (before the intervention content began) after replying ‘Yes’ to ‘May I continue?’ This larger-than-expected proportion of ‘early hang-ups’ led us to take a close look at the importance and relevance of content in the ‘first call module’. These pilot data, along with the interview debrief data, led us to shorten the description of the program, remove the opportunity for active consent on the call, and remove the welcome call from the program, since its main purpose was to obtain consent. Initial results from the 5 months of calling rounds during the full PEANUT trial using these modified scripts show that the early hang-up rate dropped 37 percent (from 57% of reached members to 36%) and the ‘normal wrap-up’ rate increased 73 percent (from 15% in the pilot to 26% in the trial). (Full outcome data from the PEANUT trial will be reported in a future article.)
Limitations
We believe that many of the patient recommendations and learning from this developmental process will be useful to organizations relying on speech-recognition-based interventions to manage chronic conditions, facilitate prevention, and improve patient care. However, there are a few limitations to consider. For example, the formative focus group data capture patient reaction to IVR calls within a formal research setting rather than a natural setting such as the participant’s home, which may have shaped participants’ reactions to calls. We attempted to address this issue by guiding and encouraging patients to respond honestly to the played-back IVR phone calls, but we cannot determine how well this strategy worked. Participants, however, were quite forthcoming in their critique of the calls during the focus groups. We also acknowledge the possibility participants may have felt uncomfortable during the focus groups in sharing all of the reasons they may not take or refill their asthma medication regularly. Additionally, participants’ responses to our focus group questions may have been shaped to please the interviewer. Thus, we may not have captured additional medication compliance barriers that could have been potentially addressed in the call content and structure. Overall, we conducted a large enough number of focus groups, with a range of participant ages, gender, and disease experience, to generate sufficient thick and rich description to meet qualitative standards of data credibility and transferability. 20,21
Regarding the pilot calls, we chose to conduct debrief phone interviews with participants rather than in-person interviews; as such, phone interviews are less likely to generate in-depth explorations of topics than face-to-face interviews. Additionally, some of the positive views of the calls expressed by participants who did not complete a call or hung up early may be due to self-selection (being willing to talk with us) or a desire to be polite to the interviewer.
Two factors may have contributed to incomplete recall by participants about their experience with IVR calls. First, because uploading call data is time consuming, and we made multiple attempts to reach participants at home for interview availability, interviews did not occur until approximately 5 to 7 days after the participant received an IVR call. Given this, participants might not have thoroughly recalled the call content and/or their reactions to the call. Second, the interview was a one-time event conducted after a participant had received one of their first IVR calls. While we asked during the debrief phone interviews if participants would be willing to take future calls from the program, we do not know how participants might respond to multiple IVR calls over time. However, our formative focus group data provide some insight into this question. Participants indicated that, given familiarity with the call purpose and content, they might start to hang up on the call more frequently, or earlier in the call, but that this would not necessarily indicate that the IVR calls were not useful; a familiar IVR call could still be a reminder to take action (e.g. refill medication) without the person listening to the entire call. Focus group participants advised that changing the IVR call content, even slightly, as the program continued, may help to retain people’s interest in listening to most, if not the entirety, of a call.
Overall, researchers wishing to explore participant behavior and reactions to IVR calls may want to conduct debrief interviews closer to the time of receiving the call, and should consider conducting face-to-face interviews to elicit more detailed and accurate data. Future explorations should also obtain feedback on how participants respond and react to IVR calls over time. Additionally, future research identifying ways that IVR interventions can be made more practical, acceptable, and user-friendly to older populations would be extremely useful.
Conclusion
Formative and developmental processes such as focus groups, pilot testing, and qualitative debrief interviews – and modifications to an intervention based on what is learned in these evaluations – appear to improve the acceptability and usefulness of an IVR intervention utilizing speech recognition technology. From a public health perspective, there is considerable potential value in identifying large-scale, low-intensity interventions that achieve even small improvements in patients’ compliance with medications on a population basis. Automated phone interventions can serve such a purpose. Understanding how patients receive and interact with these types of interventions, and improving their acceptance and usefulness, can help increase their benefit.
Footnotes
Acknowledgements
The project described was supported by award R01HL0834333 from the National Heart, Lung, and Blood Institute. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Heart, Lung, and Blood Institute or the National Institutes of Health. The authors wish to express their appreciation to the focus group, telephone debrief, and pilot participants. We are grateful for their time and commitment. We also wish to acknowledge Vicki Axford, Cheryl Johnson, Allen Gervacio, Kathy Mau, Aleli Vinoya, Barbara Bachman, Suzanne Gillespie, Judy Wick, Jan Whitehead, Chalinya Bruce, Leslie Bienen, and Jill Pope for their valuable assistance with this developmental study.