Abstract
This article reports redesign strategies identified to create a Web-based user-interface for the Self-management TO Prevent (STOP) Stroke Tool. Members of a Stroke Quality Improvement Network (N = 12) viewed a visualization video of a proposed prototype and provided feedback on implementation barriers/facilitators. Stroke-care providers (N = 10) tested the Web-based prototype in think-aloud sessions of simulated clinic visits. Participants’ dialogues were coded into themes. Access to comprehensive information and the automated features/systematized processes were the primary accessibility and usability facilitator themes. The need for training, time to complete the tool, and computer-centric care were identified as possible usability barriers. Patient accountability, reminders for best practice, goal-focused care, and communication/counseling themes indicate that the STOP Stroke Tool supports the paradigm of patient-centered care. The STOP Stroke Tool was found to prompt clinicians on secondary stroke-prevention clinical-practice guidelines, facilitate comprehensive documentation of evidence-based care, and support clinicians in providing patient-centered care through the shared decision-making process that occurred while using the action-planning/goal-setting feature of the tool.
Keywords
Introduction
This article reports redesign strategies identified to create a Web-based user-interface for an evidence-adaptive clinical decision support (CDS) tool that targets secondary stroke prevention. The Self-management TO Prevent (STOP) Stroke Tool was designed to improve the management of stroke risk factors, particularly in high-risk veterans with a history of ischemic stroke or transient ischemic attack (TIA). Ischemic stroke is associated with multiple risk factors. Although some stroke risk factors cannot be modified, such as age, male gender, race, family history, and personal history of prior stroke or TIA, many stroke risk factors are modifiable with lifestyle interventions and/or medications. The American Heart Association/American Stroke Association and the Department of Veterans Affairs (VA) have developed specific clinical-practice guidelines for secondary stroke prevention. These guidelines also provide a systematic approach to reduce practice variability and improve patient outcomes after stroke. Despite the importance of evidence-based care, the recommendations contained in clinical-practice guidelines are applied only about 50 percent of the time. 1
CDS tools link clinical knowledge and patient-specific information to augment patient care decisions. 2 Automated and on-demand CDS tools that integrate with the electronic health record (EHR) offer an effective strategy to implement best practices and improve the quality of patient care.2–4 When a CDS tool applies evidence-based recommendations for patient-specific problems, the system is classified as evidence adaptive. 5 Therefore, an automated and on-demand evidence-adaptive CDS tool that is integrated into the EHR may be an effective strategy to improve the implementation of clinical-practice guidelines for secondary stroke prevention.6,7
The evidence-adaptive STOP Stroke Tool was initially developed as a platform-dependent, template-based tool within the Computerized Patient Record System (CPRS)—the VA’s EHR. The template-based tool prompts providers on clinical-practice guidelines for secondary stroke prevention, while providing access to printable patient-education materials and instruction for patient self-management action plans. Although the overall usability of the template-based tool was acceptable, multiple barriers related to the inherent limitations of reminder dialog applications were identified with a think-aloud technique. 8 Specifically, reminder dialog templates within CPRS do not allow simultaneous access to other information in the EHR. Moreover, the use of checkbox and drop-down screens within the reminder dialog template was found to be burdensome, particularly for patients with multiple stroke risk factors. Lack of clarity on how to access embedded guidelines, patient-education materials, and patient action plans within the template was also noted as a usability barrier (Figure 1).
Barriers identified in original STOP Stroke Tool.
The template-based tool provided clinicians automated and accessible stroke clinical-practice guidelines, printable patient-education materials, and self-management action plans within CPRS. However, multiple usability barriers restricted its usefulness in busy practice settings such as primary care. Thus, we sought to identify interface-redesign strategies to create a Web-based user-interface of the STOP Stroke Tool that would enhance accessibility, usability, and usefulness of the STOP Stroke Tool and to also incorporate design strategies that encouraged collaboration on stroke risk-factor management between providers and patients. At the time of development, testing, and publication of these findings, no other Web-based CDS tool for stroke risk management was known to exist.
Objectives
Our immediate objective for this study was to apply the results obtained during alpha-testing to improve the design of the STOP Stroke Tool with the overall objective of producing a beta-product for implementation and testing in clinical practice. Specifically, we sought to answer two research questions.
What implementation facilitators and barriers are identified among multidisciplinary clinicians interacting with a Web-based prototype of the STOP Stroke Tool?
What barriers and facilitators to access, usability, and usefulness of the Web-based application are identified among a sample of multidisciplinary stroke-care providers using the prototype in a simulation lab?
Methods
Design and setting
A pre-experimental design was applied using descriptive and qualitative methods. A framework based on usability engineering 9 guided the design and development of the Web-based STOP Stroke Tool prototype (Figure 2), which was to be used by clinicians in the presence of patients to deliver recommendations for stroke risk reduction. Usability testing of the Web-based prototype was completed in two phases (Figure 3). The first phase was conducted with members of Stroke Quality Improvement Network (SQUINT) from sites throughout the VA system. The second phase was among multidisciplinary clinicians at two clinical sites, the Michael E. DeBakey VA Medical Center (MEDVAMC) in Houston, Texas, and the Richard L. Roudebush VA Medical Center (RVAMC) in Indianapolis, Indiana. The MEDVAMC has a large, active stroke program with more than 300 stroke and TIA admissions annually. The RVAMC is a moderate-sized stroke-care facility with approximately 120 stroke and TIA admissions each year. In both facilities, veterans with a history of stroke or TIA routinely receive follow-up care from clinicians in primary care and in specialty clinics in neurology and rehabilitation medicine.
Solutions in the Web-based STOP Stroke Tool to address previously identified barriers.
Purpose, materials and evaluators, and outcomes of the phases of evaluation.
Phase 1 procedures
Members of the SQUINT (N = 12) viewed a demonstration video of the Web-based STOP Stroke Tool prototype (Figure 2) and provided feedback on perceived barriers and facilitators to implementing a Web-based tool in their respective areas of practice. The mission of SQUINT is to serve as a communication forum for sharing and discussion of stroke quality–improvement initiatives and programs among VA stroke-care providers in the field. Complete anonymity was guaranteed to SQUINT participants for participating in the evaluation and, therefore, no demographic information was obtained from the sample.
The Principal Investigator presented a demonstration video to SQUINT members of how the STOP Stroke Tool would function as a Web-based platform during a Live Meeting conference. The video demonstrated how a provider might use the tool during a visit with a patient. Specifically, the video demonstrated the format (i.e. MS Windows–based, larger font, functionality with CPRS) and content (i.e. patient history, medication recommendations, patient education, and goal setting) of the Web-based STOP Stroke Tool. Members were invited to visit the Stroke Quality Enhancement Research Initiative (QUERI) Web site (http://www.queri.research.va.gov/tools/stop_stroke/) to view the STOP Stroke Tool demonstration video and provide feedback on implementation barriers and facilitators. After Institutional Review Board (IRB) approval was obtained, SQUINT members were informed that completing the evaluation form constituted their consent to participate in an evaluation study. The evaluation form included five open-ended questions that elicited the participant’s perceived barriers and facilitators to implementing the Web-based tool in his or her respective practice setting. Feedback obtained from the SQUINT members (N = 12) was applied to develop a Web-based prototype for usability testing among a sample of stroke-care providers.
Phase 2 procedures
A convenience sample of stroke-care providers completed usability testing of the Web-based prototype during simulated clinic visits. A total of 10 participants (5 from each site) were recruited. There were 4 female and 6 male participants. Since providers in the specialty areas of primary care, neurology, and rehabilitation medicine generally deliver follow-up care to veterans with a history of stroke or TIA, providers in these specialty areas were targeted for recruitment. In all, 5 participants had specialty training in primary care, 4 participants had specialty training in neurology, and 1 provider specialized in rehabilitation medicine.
Usability testing—think-aloud protocol and structured interview
A think-aloud technique was employed to test the usability of the Web-based prototype (Figure 2). The think-aloud technique is an efficient and effective methodology for gathering rich qualitative data about end-user cognitive processes during evaluation of an interface design; it is a common usability testing methodology that provides a verbal record of the participants’ decision-making process and any erroneous assumptions they may make about the system. 10 The think-aloud technique for this project was developed by usability engineers in the Human–Computer Interaction Lab at the Center of Excellence on Implementing Evidence-Based Practices at the Indianapolis VA. Study teams at both sites underwent training before think-aloud sessions were initiated. After we obtained IRB approval and signed consent, providers (N = 10) that deliver follow-up care to veterans with stroke or TIA completed usability testing of the STOP Stroke Tool Web-based prototype.
Each participant completed the same 1-hour think-aloud session with researchers that included the following: (a) a brief description of the Web-based STOP Stroke Tool prototype, (b) instructions on the think-aloud technique, and (c) a scripted test-case scenario administered by a trained patient actor. The test-case scenario was structured using a single patient actor playing the role of a patient with multiple stroke risk factors, recently discharged from the hospital after experiencing a stroke. To simulate a clinic visit, participants were instructed to interact with the patient actor and the STOP Stroke Tool prototype as if they were in their clinic setting. The patient actor memorized standard responses to commonly asked questions that a stroke care-provider would ask, and these standard responses were matched to preloaded patient data in the STOP Stroke Tool prototype. For example, providers routinely ask patients about their reason for their visit or their “chief complaint.” The patient actor’s standard response to this question was “I’m doing well, but I have had trouble managing my blood pressure.” When the participant clicked on the blood pressure tab in the STOP Stroke Tool, the last three blood pressure values for the patient were displayed and showed consistent hypertension. Physician participants then continued to ask the patient actor questions about their stroke as they would do in a typical clinic encounter and interacted with the STOP Stroke Tool as the patient actor responded to the physician participant’s questions.
Participants used a desktop computer to access the STOP Stroke Tool prototype. The patient actor was stationed next to the participant during the simulation encounter. As participants interacted with the patient actor and a functional STOP Stroke Tool prototype, they were asked to verbalize their thoughts, feelings, and opinions. 11 When necessary, the investigators prompted participants to clarify statements or elicited further input from the participants. Because the STOP Stroke Tool was designed to be used by providers interacting with their patients during a clinic visit, the participant’s interaction with a patient actor was key to test usability in this context. Only verbal data directly relevant to the participant’s interactions with the STOP Stroke Tool were used for analysis.
Upon completion of the think-aloud technique, the investigator conducted a structured interview with each participant using five open-ended questions. The questions were developed by the investigators to elicit providers’ perceptions on the barriers and facilitators of the STOP Stroke Tool for usefulness in the provider’s specific practice setting and for provider-to-patient communication. All think-aloud and structured interview sessions were audiotaped, and field notes of their observations were taken by investigators.
Data analysis—phase 1
The investigators grouped and tabulated SQUINT members’ responses into a priori categories of barriers or facilitators of accessibility, usability, and usefulness for clinical practice and for provider-to-patient communication. Direct-approach content analysis employs a systematic process of coding text based on operationally defined a priori categories. 12 Participants were provided the following definitions: Accessibility was defined as a measure of the ease of access, usability was defined as a measure of the ease of use, and usefulness was defined as a measure of perceived benefits.
Data analysis—phase 2
Think-aloud audiotapes were transcribed verbatim and investigator field notes were incorporated into the transcripts. The investigators applied a directed approach for content analysis 13 to segment the data and code participants’ dialogue into word and word phrases indicative of either barriers or facilitators to the a priori categories. Two investigators coded the transcripts independently. Discussion and review occurred intermittently until 100 percent consensus was achieved. Word and word phrases were coded based on frequency, and irrelevant information was disregarded.
Results
Phase 1—demonstration video evaluation
SQUINT members identified implementation facilitators and barriers of the Web-based design for clinical practice (see Table 1). Facilitators of the Web-based design were that it was systematic, comprehensive, risk-factor specific, and automated. Identified barriers included lack of time to complete the tool, lack of interoperability, and lack of access to a printer.
SQUINT member comments regarding facilitators and barriers of the Web-based design to provider-to-patient communication.
Facilitators and barriers of provider-to-patient communication were identified among key design features of the Web-based interface (Table 1). Respondents noted that links to patient-education materials and the action-planning/goal-setting option would facilitate provider-to-patient communication. Barriers for provider-to-patient communication were less related to the Web-based design than to constraints of technology. Feedback obtained from the SQUINT members on the barriers was applied to Web-based prototype that was then tested for usability among a sample of stroke-care providers.
Phase 2—usability test findings
Results from the think-aloud protocol showed that participants used positive words more frequently than negative words to describe the Web-based prototype. Participants’ responses were overwhelmingly positive (see Figure 4). For example, the word “good” was used in 28 word phrases, “this is a good thing,” “this is good to know,” and “good for practice.” The tool was also described as helpful 17 times in phrases like the tool is “helpful for educating patients about their risk factors,” “helpful to remember what to do,” and “helpful to teach residents.”
Frequency of positive and negative words or word phrases used by participants.
Themes for facilitators and barriers to accessibility, usability, and usefulness were identified based on participants’ responses during the structured interview (Table 2). Facilitators of the Web-based design included “comprehensive information,” “color coding,” and “systematic structure.” Participants also found the Web-based tool to be useful for “patient accountability,” “reminder of best practice,” “goal-focused care,” and “enhanced communication.” Barriers were “lack of interoperability,” the “time” it would take to complete, the need for “training,” and concerns about “computer-centric care.” Participants also questioned who would be the “targeted end-user” and thought the tool might have “limited options” in addressing care issues other than stroke risk-factor management.
Participant comments regarding facilitators and barriers to accessibility, usability, and usefulness of STOP Stroke Tool, a Web-based clinical support tool.
SQUINT: Stroke Quality Improvement Network; STOP: Self-management TO Prevent; CPRS: computerized patient record system.
Discussion
Results from this project provide valuable feedback as we move forward in development of the STOP Stroke Tool. Specifications for the Web-based design, along with end-users’ input, will be incorporated to produce a beta-product of the STOP Stroke Tool for testing in the clinical-practice setting. The investigators will incorporate the identified usability facilitators of systematic format, comprehensive information, and automated features into the Web-based STOP Stroke Tool. The most effective CDS tools are incorporated into clinicians’ routine workflow, providing actionable recommendations at the time and place of decision making. 14 Clinical decision making occurs while obtaining patient information in the EHR and during the documentation of care. Incorporating these usability facilitators in the STOP Stroke Tool will not only promote the implementation of clinical-practice guidelines for stroke risk-factor management but also have the potential to ultimately produce positive patient outcomes and enhance provider-to-patient communication.
An important finding from this study is that the STOP Stroke Tool was perceived as helpful for provider-to-patient communication about stroke risk-factor management and as providing insight on design strategies to foster provider-to-patient interaction for stroke risk-factor management. Participants overwhelmingly identified the patient-education materials, collaborative action–planning/goal-setting, and clinical-practice guideline options as design features that would encourage providers to educate their patients about stroke risk factors and engage patients in self-management actions to reduce stroke risk. Additionally, several participants noted the Web-based prototype would help providers communicate with patients. Provider-to-patient communication has been shown to have a positive influence on patient self-management behaviors.14–16
Another important finding was that the action-planning/goal-setting option was perceived by providers as a facilitator for patient accountability and behavior change. All providers made comments describing the tool as helpful for keeping patients accountable in managing their stroke risk factors and for helping them to be more involved in their care. Multiple participants noted that the action-plan feature would provide documentation of what the patient agreed to work on to reduce his or her stroke risk and that this would help providers support patients in reaching their risk-factor management goals. This theme of patient accountability is also related to patient behavior change. Accountability implies the patient is taking responsibility for his or her own care, with the expectation that the patient will give an account of his or her healthcare behaviors. These findings may indicate that the STOP Stroke Tool supports the paradigm of patient-centered care—where the patient is involved in the decision-making process about his or her care. The action-planning/goal-setting option engages the patient to participate in decision making with his or her provider about how to reduce his or her stroke risk. The healthcare provider works with the patient, as a consultant, supporting the patient in the decision-making process. These two experts, the patient and the provider, work in unison to achieve the behavioral changes necessary for stroke risk-factor management. 17
The primary barrier to usability of the STOP Stroke Tool prototype was related to time required to use the tool. The average time a provider has to spend with a patient during a clinic visit in the VA is about 15–20 min. In this short period of time, the provider must complete a focused history and physical examination, diagnose health problems, prescribe treatment options, and then document the entire process of care he or she completed. This was participants’ first interaction with the STOP Stroke Tool, and therefore, the time barrier to complete the tool was not unexpected. Multiple participants commented that with repeated use, interactions with the STOP Stroke Tool would become faster. The most usable and useful CDS tools include time-saving features for all aspects of care delivery and documentation. The STOP Stroke Tool includes automated features to facilitate time saving for the provider, including automated guideline prompting, order sets, consultations, patient-education and self-management materials, and documentation of care processes. Although time needed to document while using the unfamiliar STOP Stroke Tool was an identified barrier, the quality of the note produced by the tool was markedly superior as compared to the usual documentation available in CPRS. The note produced by the STOP Stroke Tool was much more comprehensive and reflective of thorough evidence-based care that could be communicated to other providers.
Despite the positive feedback on the accessibility, usability, and usefulness of the STOP Stroke Tool, a few participants expressed concern that using a computer during the patient encounter may be distracting and impersonal for the patient. It was noted that the provider may increase his or her focus on following the computer prompts and decrease his or her focus on addressing the patient’s needs. This concern highlights the ongoing challenge of implementing a CDS tool in the context of a typical outpatient setting. Providers need immediate access to patient information, evidence-based guidelines, and patient-education materials to effectively and efficiently address patient-specific needs during the clinic visit, including accessing printed materials and interfacing seamlessly with CPRS. The most effective types of interventions to facilitate providers’ adoption of the CDS tool are unclear. Additional research is needed to elucidate the best tools and implementation strategies to mediate the adoption of CDS tools is in an outpatient setting.
This study was focused on gaining insight related to usability issues experienced by providers interacting with the STOP Stroke Tool. During this early stage of design, we only included the provider perspective to better focus on provider usability issues. However, we are now testing patients’ perspectives in the clinical-practice setting using the revised STOP Stroke Tool in real time with providers and patients. Specifically, we will evaluate patients’ perspectives based on their experience with their provider using the STOP Stroke Tool, including the possible barrier of “computer-centric care.”
Limitations
Our usability assessment included a small number of users. However, a small sample of users is often sufficient to detect the majority of usability issues.18,19 Results from formative usability testing are important to guide further development and are not intended for generalization to the larger population. The think-aloud technique was completed in a controlled setting using a patient actor in a simulated clinic visit. Therefore, our findings may not have identified all usability issues. Some functionality and usability barriers can be identified only with clinicians using the system in real-time clinical practice.
Conclusion
The involvement of end-users in the development and testing of CDS tools is critical for successful implementation in practice. 1 This project contributes to a growing body of knowledge on use of health information technology to implement and monitor quality care within the VA. Moreover, findings from this project support integrating CDS tools with the EHR to foster provider-to-patient communication and patient-centered care. This project enhances knowledge regarding the interface between information technology, self-care management, and the human dimension of medical encounters. However, more research is needed to determine whether using the STOP Stroke Tool during patient encounters improves provider-to-patient communication, patient self-management, shared decision making, and collaborative goal setting for managing stroke risk factors.
Health outcomes are improved when providers engage patients in discussions about health behaviors and emphasize the patient’s central role in taking responsibility for his or her own health. Moreover, patient behavior change may be more effective when a partnership approach is applied through shared decision making and collaborative goal setting, with the provider and patient working together to identify the patient’s specific risk profile and review evidence-based treatment options to determine an action plan the patient can follow to manage his or her risk factors. This project identified redesign strategies and key usability barriers and facilitators of a Web-based STOP Stroke Tool prototype. The findings will be applied to further develop the STOP Stroke Tool and to facilitate its implementation in clinical practice. To date, prototype development of the STOP Stroke Tool has produced an evidence-adaptive CDS tool that is effective in prompting clinicians on secondary stroke-prevention clinical-practice guidelines, facilitating comprehensive documentation of evidence-based care, and encouraging shared decision making and collaborative goal setting. We are encouraged by findings from this project that the STOP Stroke Tool is perceived as helpful in aiding provider-to-patient communication and in implementing patient-centered care.
Footnotes
Acknowledgements
The authors wish to acknowledge the Human–Computer Interaction (HCI) & Simulation Laboratory at Roudebush VA Medical Center in Indianapolis and Adam Neddo for their work on this project.
Declaration of conflicting interests
This article presents the findings and conclusions of the author(s); it does not necessarily reflect the views of the Department of Veterans Affairs (VA), the VA Health Services Research and Development, the US government, or Baylor College of Medicine.
Funding
This work was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service (RRP 09-147). This work was partly supported by the Health Services Research & Development Houston VA Center of Excellence (HFP90-020) and by the National Research Service Award (T32 HP10031).