Abstract
Telemedicine, the use of audiovisual technology to provide health care from a remote location, is increasingly used in intensive care units (ICUs). However, studies evaluating the impact of ICU telemedicine show mixed results, with some studies demonstrating improved patient outcomes, while others show limited benefit or even harm. Little is known about the mechanisms that influence variation in ICU telemedicine effectiveness, leaving providers without guidance on how to best use this potentially transformative technology. The Contributors to Effective Critical Care Telemedicine (ConnECCT) study aims to fill this knowledge gap by identifying the clinical and organizational factors associated with variation in ICU telemedicine effectiveness, as well as exploring the clinical contexts and provider perceptions of ICU telemedicine use and its impact on patient outcomes, using a range of qualitative methods. In this report, we describe the study protocol, data collection methods, and planned future analyses of the ConnECCT study. Over the course of 1 year, the study team visited purposefully sampled health systems across the United States that have adopted telemedicine. Data collection methods included direct observations, interviews, focus groups, and artifact collection. Data were collected at the ICUs that provide in-person critical care as well as at the supporting telemedicine units. Iterative thematic content analysis will be used to identify and define key constructs related to telemedicine effectiveness and describe the relationship between them. Ultimately, the study results will provide a framework for more effective implementation of ICU telemedicine, leading to improved clinical outcomes for critically ill patients.
What Is Already Known?
Telemedicine is used as a quality improvement strategy in intensive care units (ICUs) to provide access to critical care clinicians who remotely care for patients. Previous quantitative research on the clinical impact of ICU telemedicine has demonstrated mixed results. Therefore, more exploratory work is needed to better understand the conditions that promote effective ICU telemedicine. Qualitative methods are particularly well suited to developing a nuanced, in-depth understanding of the contextual factors that underlie telemedicine effectiveness.
What This Paper Adds?
This paper describes a qualitative multi-site study designed to identify mechanisms that contribute to ICU telemedicine effectiveness. Using direct observations, interviews, focus groups, and examination of artifacts, we will compare practices and perceptions across telemedicine programs, clinicians in the ICUs and telemedicine units, and professions. This research will yield a conceptual model for ICU telemedicine effectiveness that can be used to improve current practices and guide future implementation.
Background and Study Justification
The ICU is a specialized area in the hospital providing dedicated acute care services to critically ill patients at high risk for death. Between 4 and 7 million Americans are admitted to an ICU each year (Halpern, 2010). Mortality in the ICU is extraordinarily high—nearly 1,500 people die after admission to an ICU each day and nearly one in five Americans will die in an ICU (Angus et al., 2004). Because the ICU is extremely resource intensive, it is among the costliest places in the hospital to deliver care. Critical care represents about 15% of all hospital costs; and in the United States, the total cost of critical care services exceeds US$50 billion annually (Halpern, 2010).
ICU telemedicine is one proposed strategy to improve outcomes and lower costs. Telemedicine is the remote provision of medical care through audiovisual technology and advanced patient monitoring systems. Using ICU telemedicine, trained intensivist clinicians (including physicians, nurses, and pharmacists) located at high-capability hospitals can remotely monitor critically ill patients at hospitals with fewer resources and less availability of adequately trained staff, providing targeted instruction to bedside clinicians when necessary (Breslow, 2007). Telemedicine also allows clinicians to continuously monitor large numbers of patients around the clock, quickly recognizing deterioration even in the vulnerable nighttime period (Breslow, 2007). In addition to improving quality, telemedicine might lower costs by preventing costly complications such as health-care-acquired infections through improved monitoring or by creating economies of scale by which fewer clinicians can care for greater numbers of patients.
Indeed, early empirical evaluations of telemedicine suggested a large clinical and financial benefit (Breslow et al., 2004). As a result of this early evidence, ICU telemedicine rapidly expanded in the United States. As of 2010, over 200 U.S. hospitals, equating to just under 5,800 or about 8% of ICU beds, had adopted ICU telemedicine (Kahn, Cicero, Wallace, & Iwashyna, 2014). Yet as telemedicine expanded, subsequent studies evaluating the clinical impact of ICU telemedicine show mixed results. For example, one study found that the introduction of ICU telemedicine was associated with lower in-hospital mortality and reduced ICU lengths of stay, while another study found no impact on these same indicators (Lilly et al., 2014; Thomas, Lucke, Wueste, Weavind, & Patel, 2009). A recent systematic review of 11 studies found an encouraging overall mortality benefit (relative risk for inhospital mortality: 0.83, 95% confidence interval [0.73, 0.94]) but also found wide heterogeneity among the studies (I 2 = 0.72%), with many showing large benefit and others showing no benefit or even harm (Wilcox & Adhikari, 2012).
In light of this evidence, clinicians and hospital administrators have little guidance on whether, how, and where to use this potentially transformative technology (Berenson, Grossman, & November, 2009). High-quality effectiveness research is needed to fill these knowledge gaps by determining the strategies through which ICU telemedicine is most likely to reduce mortality for the critically ill (Kahn et al., 2011). To fill this knowledge gap, we designed the ConnECCT study, a qualitative study designed to identify the clinical and organizational factors that are associated with both more effective and less effective ICU telemedicine programs (Kahn et al., 2016). The overall goal of this study is to provide a more nuanced understanding of the variation in ICU telemedicine program effectiveness in order to discern which aspects of ICU telemedicine are beneficial and in what contexts.
Explanation and Justification of the Methods
Overview of Methodological Approach
ConnECCT is a qualitative study designed to identify barriers and facilitators to effective ICU telemedicine use as well as identify the potential mechanisms by which ICU telemedicine impacts patient care. Data collection included site visits at dyads composed of an individual ICU and the telemedicine unit that provided the ICU with remote care. Each site visit lasted approximately 4 days. During the site visits, we collected four types of data: direct observations of in-person and remote patient care, semistructured interviews with care providers and other stakeholders, focus groups with care providers, and telemedicine-related artifacts (Table 1). In total, the research team conducted site visits at 10 ICU/telemedicine unit pairs within the continental United States. The overall research framework was guided by the 32-item Consolidated Criteria for Reporting Qualitative Research, which is provided as Appendix A (Tong, Sainsbury, & Craig, 2007).
Data Collection Rationale.
Note. ICU = intensive care unit.
Justification of Methodological Approach
An ethnographic approach was taken because such studies are better suited for obtaining a more in-depth, nuanced understanding of contexts and provide a detailed description and analyses of the culture, processes, and practices related to the population or the situations studied (Genzuk, 2003; Giacomini & Cook, 2000; Patton, 2002). Investigating the determinants of effective critical care telemedicine necessitates understanding the complex perceptions and experiences of those involved in the daily practice (Moeckli, Cram, Cunningham, & Reisinger, 2013; Sinuff, Cook, & Giacomini, 2007). Moreover, qualitative methods are ideally suited to conduct initial explorations, develop new theories, and generate hypotheses. This is particularly true in organizational research—in their landmark paper “Methodological Fit in Management Field Research,” Edmondson and McManus (2007) describe observation and interviews followed by content analysis as building the so-called Nascent Theory Research, in which the goals of the study are to explore new hypotheses about a novel phenomenon such as ICU telemedicine.
A transdisciplinary research team was assembled to include experts in critical care medicine, health services research, nursing, health law and policy, public health, and qualitative research. As outlined by Ciesielski, Aldrich, Marsit, Hiatt, and Williams (2017), for complex multicausal research questions, the incorporation of cross-discipline perspectives can result in more comprehensive hypotheses, broaden the scope and type of data collection, enrich analysis, and develop appropriate next steps for translation and intervention. Based on prior experience, two investigators (J.K., D.A.) developed the initial conceptual model that guided preliminary hypotheses (Kahn et al., 2011). However, taking a qualitative and comparative approach to data collection and analysis allowed for emergent themes to be incorporated and revisions to be made to initial hypotheses. This was facilitated by standardized team debriefings mid-site visit to discuss data that appeared to be consistent across sites, findings potentially novel to a site, logistical challenges, and avenues for follow-up. Interview and focus group guides were revised to reflect ongoing interplay between data generation and developing analysis.
Sampling, Recruitment, and Data Collection
Site Selection
We based our site selection strategy off prior work which quantified the clinical impact of adopting telemedicine at 133 individual ICU programs in the United States using matched hospital pairs and a difference-in-difference quantitative analysis (Kahn et al., 2016). We purposefully sampled dyads (ICU and corresponding telemedicine unit) using an extreme or deviant sampling approach based on the change in risk-adjusted mortality among ICU patients after introduction of ICU telemedicine. An extreme or deviant sample focuses on outliers of a given phenomenon to better understand factors that set a case apart from what is typical (Bradley et al., 2009).
The unit of analysis was a site, which we defined as the paired ICU and telemedicine unit. This definition allowed us to account for structural and organizational features unique to each individual care unit as well as the interactions that occur between the two. In addition to the use of effectiveness data, we augmented our sampling strategy by selecting dyads of varying hospital size, community size, academic status, and geographic location in order to ensure maximum variation in the study. Participation was restricted to general medical ICUs rather than specialty ICUs (e.g., cardiac ICUs or neurological ICUs) in order to avoid making invalid comparisons across specialized ICUs and their corresponding specialized patient populations.
At initiation of the study, 18 sites were contacted to participate: 8 “high effectiveness sites” (i.e., introduction of ICU telemedicine led to a statistically significant decrease in risk-adjusted mortality) and 10 “low effectiveness sites” (i.e., introduction of ICU telemedicine led to a statistically significant increase in mortality). Of these 18, 8 dyads agreed to participate: 5 high effectiveness and 3 low effectiveness. During the course of the site visits and preliminary data review, it was decided to add an additional “null effectiveness” site (i.e., introduction of ICU telemedicine led to no appreciable change in mortality) in order to explore an ICU telemedicine program that neither helped nor harmed patient care. Additionally, we added a dyad with an ICU that had recently adopted telemedicine but for which no effectiveness data were available to learn about the decision to adopt and the onboarding process. Thus, the final sample included 10 dyads (Table 2). The 10 pairs equate to six individual sites, as some telemedicine units covered two selected ICU (4 telemedicine units for 8 ICUs).
General Characteristics of Participating Hospitals.a
Note. ICU = intensive care unit.
aFrom Health-Care Cost Report Information System (HCRIS) through 2010.
Initial invitations for selected dyads were sent via an introductory e-mail followed by a comprehensive information packet sent by mail. These mailings were directed to an identified representative of the telemedicine unit obtained from the predominant commercial telemedicine application provider. If, based on review of this information, a telemedicine unit was interested in learning more about participation, a telephone conference call was arranged between the study team, representatives of the telemedicine unit, and others in the health system to provide key details about the expectations and burden of site visits.
If a telemedicine unit continued to express interest in participation after the initial conference call, the study team and representatives from the telemedicine unit held further conversations with the corresponding ICU leadership teams. When both the telemedicine unit and the ICU agreed to participate, site visit planning commenced.
Development of Data Collection Tools
Data collection was based on a preliminary conceptual model for telemedicine effectiveness developed during a multistakeholder, federally funded consensus conference to define a research agenda in ICU telemedicine (Kahn et al., 2011). This conference brought together 29 experts in critical care, nursing, health economics, health services research, rural health, information technology, and organizational science. Participants posited that telemedicine effectiveness might be influenced by three domains: (1) target ICU characteristics, including staffing, rounding patterns, and protocols; (2) telemedicine unit characteristics, including staffing, hours, and quality improvement activities; and (3) the interaction between the two, including implementation of telemedicine, services provided by the telemedicine unit, and relationships between practitioners in the two units. This preliminary conceptual model was used to develop three types of data collection tools.
Interview guides
An interview guide was developed based on the above framework. Key interview domains included the implementation of telemedicine, services provided, interactions between the telemedicine unit and ICU, impact on outcomes, and suggested changes in the telemedicine program. This guide was then tailored for use in either the telemedicine unit or the ICU, as well as for use with either administrators or bedside clinicians. This process resulted in a total of four interview guides: ICU clinician, ICU administrator, telemedicine unit clinician, and telemedicine unit administrator. Interview guides are presented in Appendix B.
Focus group guides
We also developed a focus group guide. It began with a general overview question, followed by questions about how telemedicine impacts patient outcomes, when telemedicine works well or not, relationships between the ICU and the telemedicine unit, and integration of telemedicine services. This guide was then tailored for use in either the telemedicine unit or the ICU, resulting in a total of two focus group guides. Focus group guides are presented in Appendix C.
Unit and provider characteristics surveys
Presurveys, a survey distributed for completion before a site visit, in order to inform the site visit, were developed for both the ICUs and the telemedicine units to collect information about staffing, telemedicine use, and services provided. In addition, for the ICU, questions were included to ascertain hospital and ICU characteristics (e.g., size and census). For the telemedicine units, additional questions were added related to program characteristics such as number of hospitals, ICUs, and beds covered. We also developed a survey for interview and focus group participants. The 21-item survey asked about profession, education, employment, and demographics.
Pilot Site Visit
Two members of the study team visited an established telemedicine unit that covers multiple hospitals in order to refine the site visit procedures and pilot test the data collection tools. This telemedicine unit was not included in the sample of sites eligible for participation in ConnECCT. The study team spent 2 days touring the telemedicine unit and the ICUs to which it provided services to informally observe care delivery, after which interviews were conducted with clinicians and administrators. The data collection procedures and interview/focus group guides were revised based upon the feedback received from the pilot visit.
Site Visit Process
Site visits were conducted between February 2015 and January 2016. The site visit team consisted of between 3 and 5 researchers (K.R., C.K., L.E.A., A.B., and M.H.). Site characteristics of the 10 ICUs and 6 telemedicine units are presented in Table 3.
Staffing and Coverage of Participating Units.
Note. ICU = intensive care unit.
To conduct the direct observations, the site visit team was split into groups, so that two researchers would be in the ICU, while one or two researchers were in the telemedicine unit, allowing for concurrent observations of telemedicine interactions from both sides. During site visits, researchers rotated shifts between the telemedicine unit and the ICU, such that a minimum of two distinct researchers collected data from each unit. Duration of site visits ranged depending on whether we were visiting one ICU or two ICUs, 4 days versus 8 days, respectively. Hours of observation included both evenings and weekends in order to ensure that we were able to observe a range of interactions. Observations also targeted periods of scheduled interactions or activities between the telemedicine unit and ICU, such as shift reports, patient admissions, and telemedicine video rounding. Observations were recorded as field notes. In addition to field notes, each researcher kept a daily journal reflecting on that day’s events, items, or questions for follow-up and analytic insights.
In order to include a broad range of professions and roles in the interviews and focus groups, we used a stratified purposive sampling frame to maximize representation (Table 4). Participation for interviews and focus groups was facilitated through announcement flyers, e-mails to staff, previsit invitations and scheduling, and on-site recruitment. If a participant was not available to take part in an interview during the site visit, the interview was completed by telephone at a later date. Focus groups were structured to be profession-specific and limited to care providers. Interview and focus group participants were compensated US$40 and US$50, respectively. Interviews and focus groups were conducted at times and private locations convenient for participants and were digitally recorded.
Target Sampling Frame.
Note. ICU = intensive care unit.
Artifacts collected included workflows, marketing/informational materials, orientation guidelines, quality reporting, and communication documents used between the ICU and the telemedicine unit.
Data Handling and Analysis
Preliminary Analysis and Interim Member Checking
Following an initial analysis of data from each site, the study team developed a written summary reflecting the most common themes related to ICU telemedicine, as well as barriers and recommendations as expressed by participants from that dyad. These reports were sent to each site after data collection at their site was complete for assessment regarding credibility. The study team arranged separate telephone calls with each telemedicine unit and each ICU in order to discuss the content of the reports. The study team specifically requested feedback on which themes were consistent with the units’ understanding, which themes were novel, and which themes merited additional elaboration. Additionally, the study team solicited feedback about sites’ experiences participating in the study and any changes they would suggest to the overall study process.
Data Management
During each site visit, debrief phone calls were held between the on-site research team, the principal investigator (J.K.), and coinvestigators (A.B., M.H.). These discussions focused on emergent and reoccurring themes, site-specific logistical factors impacting data collection, and investigator guidance on how to refine and focus inquiry for the remaining time on-site. Additionally, site researchers, the principal investigator, and coinvestigators met regularly during data collection to review progress and discuss preliminary findings.
Following each site visit, all field notes were typed, audio recordings of interviews and focus groups were transcribed, and all drawings and artifacts were digitally rendered. Data files were de-identified and imported into NVivo 11 qualitative data analysis software (QSR International, Australia). Development of a thematic codebook was initiated after the second site visit and included both predetermined and emergent codes (Guest & MacQueen, 2008). Two members of the site visit team (K.R., C.K.) along with an outside qualitative researcher (P.M.), who was unaware of the effectiveness status of the dyads, developed the codebook. Codebook development took place from the fall of 2015 and through spring of 2016. We ended codebook development when no changes were made to the codebook and the percentage agreement across coders reached 90% per code, with a corresponding κ score of .89 (McHugh, 2012). In total, 27 source documents, representing 7% of the overall data set, were coded in order to achieve the target agreement among coders.
Future Coding and Analysis
Three researchers will independently code the rest of the data. To ensure understanding of the contextual background of each site and variation across participants and methods, coding will be done by site and assigned to coders in predetermined batches that include a mixture of interviews, field notes, and focus groups. Intercoder reliability will be evaluated after the completion of coding for each site. This will assess for continued consistency in code use over time and determine whether there are any potential changes needed to the codebook. Following completion of primary coding, key themes will be audited for quality assurance and to assess for any nuances, or subthemes, within the larger theme. Interpretation of themes and subthemes, as well as the relationships between themes impacting the effectiveness of ICU telemedicine, will be developed through ongoing team discussion and regular meetings. The main findings will be presented as a conceptual model of organizational features and service delivery processes that contribute to ICU telemedicine effectiveness.
Ethics
Regulatory Review
This research was approved by the University of Pittsburgh Human Research Protection Office (PRO14050448). The project was considered minimal risk and received expedited review. While all participating sites were covered by the University of Pittsburgh Human Research Protection Office approval, several sites requested additional local ethics reviews and approvals.
Ethical Considerations for Interview and Focus Group Participants
All interview and focus group subjects were provided informed consent forms detailing the purpose of the study and the potential risks of participation. The primary risk was psychological discomfort associated with discussing aspects of their employment. Another risk is that subjects might feel pressured to participate by their employer, thus potentially jeopardizing their professional relationships if they refuse. Participant confidentiality was addressed by assigning each participant a unique study ID number, with all data collected from those subjects indexed to that number. In the field, the study team did not record any personal information other than demographic descriptions of the cohort, and thus, interview and focus group participants are not identifiable in the data. All voice recordings were stored on a secure, password-protected server, and all transcripts de-identified by a trained transcriptionist.
Ethical Considerations for Staff Present During Direct Observations
Before the site visit, e-mails from unit directors informed all staff employed in the telemedicine unit and corresponding ICU of the upcoming visit. During each site visit, the study team introduced themselves and the study to staff they encountered and provided them with the opportunity to decline to participate in observations. In field notes, the study team identified staff by role and not by name. A waiver of signed consent from [redacted] covered individuals who took part in direct observations. The waiver was granted under the stipulation that the research involved no more than minimal risk, that the research could not be practically carried out without the waiver, and that the waiver did not adversely affect the rights and welfare of study participants.
Ethical Considerations for Patients During Periods of Direct Observation
We understood that we would encounter patients and patient information during the course of the site visits, such as when the telemedicine team was providing remote instruction to clinicians in a patient’s room. It could also occur indirectly, such as when a care team in the ICU was rounding outside of a patient’s room but used their name or other identifying information. Although we did not consider patients to be research participants, we nonetheless carefully considered strategies to protect their rights. First, we avoided collecting any data about patients, including either protected health information or other clinical information. Second, we introduced ourselves and the study to patients and their families when applicable.
Rigor
We took several steps throughout the study design, data collection, and data analysis to ensure the scientific rigor of our approach and the trustworthiness of our results (Shenton, 2004).
We addressed the issue of researcher bias in several ways. Investigators with extensive experience related to ICU telemedicine did not conduct primary data collection, so that their preconceived ideas of contextual and organizational influences on effectiveness did not preclude the identification of emergent factors. Furthermore, while the site visit team was aware of sites’ effectiveness classification, data collection was undertaken using an organizational learning approach that recognized ICU telemedicine as a process rather than a static phenomenon. We looked at ways telemedicine programs evolved over time: what problems were identified, ways problems were addressed, and areas for future improvement. Lastly, we developed the thematic codebook with the help of a PhD trained qualitative researcher [PM] who was unaware of site selection status.
We also employed multiple methods of triangulation. First, we employed triangulation of methods via direct observations, semistructured interviews, focus groups, and artifact collection. This allowed for multiple and distinctive ways to assess for consistent or supplemental aspects of reported phenomenon. Second, we employed triangulation of researchers, such that the focus of data collected, aspects of inquiry, and preliminary identification of relevant constructs were not confined to a single frame of reference or set of biases. Third, we collected data from a wide range of users and providers of ICU telemedicine to include multiple perspectives and variation based on participants’ position and experiences. Fourth, we collected data at multiple sites (environmental triangulation) of varying characteristics, allowing for both comparisons of common versus unique features and opportunity to further elaborate on emergent domains. Fifth, we performed member checking by way of site-specific reports sent to each dyad.
Conclusions
We describe the first comprehensive qualitative examination of ICU telemedicine effectiveness. The planned analysis will yield novel insights into the key clinical and organizational determinants of telemedicine and aid in the development of a tool kit to guide hospitals, health systems, and health professionals, as they seek to implement ICU telemedicine most effectively and efficiently.
Footnotes
Appendix A
Appendix B
Appendix C
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding: United States National Institutes of Health (R01HL120980).