Sage Journals: Discover world-class research

Abstract

French

Background:

As the burden of chronic kidney disease (CKD) continues to increase, many geographically dispersed Canadians have limited access to specialist nephrology care, which tends to be centralized in major urban areas. As a result, many rural/remote-dwellers in Canada experience poor quality of care and related adverse outcomes. It is imperative to develop alternative care delivery mechanisms to ensure optimal health outcomes for all Canadians.

Objective:

To investigate the feasibility and effectiveness of electronic consultation (eConsult) as a new model for interactions between specialists and primary care providers (PCPs) to improve access to care for patients with CKD.

Design:

This is a sequential, mixed methods study that will be conducted in 3 phases.

Setting:

The study will be conducted across the entire province of Alberta, supported by Alberta Kidney Care (formerly, Northern and Southern Alberta Renal Programs [NARP/SARP]).

Patients:

Patients suffering from CKD will be included in the study.

Measurements:

We will assess the barriers and enablers of implementation and adoption of an e-consultation protocol to facilitate access to care for patients with CKD in Alberta with a focus on rural/remote-dwellers with CKD. We will also evaluate the impact of the eConsult system (eg, improved access to specialist care, reduction in care gaps), assess the feasibility of province-wide implementation, and compare eConsult with practice facilitation versus eConsult alone in terms of access to specialist care, quality of care, and related outcomes.

Methods:

The study will be conducted in 3 phases. In phase 1, we will assess the perceptions of stakeholders (ie, PCPs, nephrologists, patients, policymakers, and other care providers) to improve CKD care delivery, quality, and outcomes in Alberta with focus groups and semistructured interviews. Phase 2 will engage specific family physicians for their input on key factors and logistical issues affecting the feasibility of implementing eConsult for the care of patients with CKD. Phase 3 will provide academic detailing including practice facilitation to clinics in Alberta to assess how eConsult with practice facilitation compares with eConsult alone in terms of access to specialist care, quality of care, and related outcomes.

Results:

We will assess stakeholder perceptions about potential barriers to and enablers of a new eConsult and decision support system strategy, focusing on elements that are most important for the design of a feasible and implementable intervention. We will develop, pilot test, and assess the impact of the eConsult model in improving access to specialist nephrology care and the feasibility of province-wide implementation. The final phase of the project will address key challenges for optimal care for patients with CKD living in rural, remote, and underserved areas of Alberta, particularly timely referral and disease management as well as the cost-effective benefits of eConsult.

Limitations:

Lack of high-speed Internet in many rural and remote areas of Alberta may lead to more time spent in completing the eConsult request online versus faxing a referral the traditional way. Allied health care staff (referral coordinators, administrative staff) require training to the eConsult system, and physicians at many remote sites do not have adequate staff to handle eConsult as an added task.

Conclusions:

Implementation of eConsult can favorably influence referral patterns, access to care, care quality, patient outcomes, and health care costs for people with CKD. Results of this study will inform the optimization of care for rural/remote-dwellers with CKD and will facilitate future partnerships with policymakers and provincial renal programs in Alberta to ensure optimal kidney health for all residents.

Trial registration:

Not required.

Keywords

chronic kidney disease (CKD)quality of care rural remote adverse outcomes e-consultation decision support system (DSS)

What was known before

Previous work demonstrated that the existing system for arranging specialist nephrology care is suboptimal. The use of electronic consultation (eConsult) to facilitate access to specialist care is becoming a standard clinical practice in many countries. Nevertheless, eConsult is not widely adopted in Canada.

What this adds

By developing an eConsult protocol for chronic kidney disease (CKD) and combining it with practice facilitation within Alberta, we hypothesize that this model of care facilitates timely identification of patients with CKD at high risk for end-stage kidney disease (ESKD) and cardiovascular disease (CVD) and improves adherence to evidence-based, guideline-concordant care to optimize kidney and cardiovascular outcomes.

Introduction

Specialist nephrology care is critical for some (but not all) chronic kidney disease (CKD) patients. Over the last decade, the number of referrals to nephrologists for management of CKD has increased dramatically in Alberta.^1-6 Our previous work demonstrated that the existing system for arranging specialist nephrology care is suboptimal: Interactions between nephrologists and primary care providers (PCPs) are limited; PCPs do not work to their full scope of practice; and the system is inefficient, leading to many unnecessary referrals and late referrals of patients with severe CKD.^3,4 Over 30% of incident dialysis patients in Alberta do not receive specialist kidney care prior to initiating dialysis treatment^3,4; this is associated with increased emergency room visits, hospitalization, and health care use. This issue is compounded by the geography of Alberta, which leads to disparities in the distribution of health care resources, the health workforce, and access to care.⁴ There is an obvious need for an alternate CKD care delivery model that can facilitate efficient, effective, cost-saving, convenient, and timely care for patients living with CKD, particularly in rural/remote Alberta.

The use of electronic consultation (eConsult) to facilitate access to specialist care is becoming a standard clinical practice in many countries. Nevertheless, eConsult is not widely adopted in Canada.^7-9 Alberta Netcare,¹⁰ the provincial electronic health record system (available to 80% of Alberta’s PCPs) is an ideal framework to support this model of care. However, it is crucial to establish the feasibility, acceptability, and ideal format for such a system prior to province-wide implementation. Once the format has been selected, it will be critical to document the new system’s impact on outcomes and costs to ensure its sustainability.

We have developed an eConsult protocol for CKD supported with practice facilitation within the Alberta Netcare system to address previously identified barriers, thereby facilitating better care, improving the referral experience for patients and providers, and conserving scarce resources. This model of care involves direct communication between referring PCPs and nephrologists via a Netcare portal to provide context-sensitive decision support at the time and location of decision-making, thereby limiting face-to-face visits between patients and nephrologists to situations where such visits are truly required.

Hypothesis

We hypothesize that this model of care facilitates timely identification of patients with CKD at high risk for end-stage kidney disease (ESKD) and cardiovascular disease (CVD) and improves adherence to evidence-based, guideline-concordant care to optimize kidney and cardiovascular outcomes. Furthermore, we predict that this model will help build capacity among PCPs for CKD management, eliminates unnecessary travel for patients to see nephrologists, reduces wait times for necessary specialist care, and consequently reduces health care costs.

Objectives

To determine the barriers and enablers of implementation and adoption of an e-consultation protocol to facilitate access to care for patients with CKD in Alberta with a focus on rural/remote-dwellers with CKD (phase 1).

To design, refine, and pilot test the eConsult system; evaluate its impacts (eg, improved access to specialist care, reduction in care gaps); and assess the feasibility of province-wide implementation (phase 2).

To compare eConsult with practice facilitation versus eConsult alone in terms of access to specialist care, quality of care, and related outcomes (phase 3).

Methods/Design

Setting

The study will be conducted across the entire province of Alberta, supported by Alberta Kidney Care (formerly, Northern and Southern Alberta Renal Programs [NARP/SARP]). These are the largest individual renal programs in Canada, providing care to ~4 million people residing in Western and Northern Canada. The programs receive referrals from PCPs in all regions of Alberta, as well as parts of British Columbia and Saskatchewan. The study’s host organization is the Alberta Kidney Disease Network (AKDN; http://www.akdn.info),¹¹ a Canadian not-for-profit organization and a joint initiative of researchers from the Universities of Alberta and Calgary. AKDN has broad methodological expertise and a proven track record in clinical and health services research. AKDN also has in-house expertise in epidemiology, health economics, health policy, health law, and knowledge translation (KT). Drawing on our prior experience in health services research, we will deliver high-quality evidence with important practice and policy implications.

Theoretical Construct/Framework

We have considered the multiple theoretical frameworks that underpin the development, evaluation, implementation, and sustainability process in testing health care delivery interventions, including the realist framework, logic models, causal modeling, intervention mapping, and the normalization process model (NPM).¹² The aforementioned frameworks have been tested in health care projects, but do not cover the critical components necessary for understanding the complexities of interventions in entirety. An example is the realist framework, which while providing linkage between the intervention and outcomes surfaced during project implementation does not adequately inform about the actual efficacy of the intervention.¹³ Similarly, logic models are also fraught with limitations, such as failure to capture the interactions between interventions and lack of flexibility and modifications during implementation.¹⁴ In health care studies, NPM is widely adopted because it is a theoretical framework that allows deep insight about the integration and evaluation of an intervention. It provides in-depth knowledge about factors that facilitate or are barriers to an implementation, as well assesses how feasible and probable it is to incorporate the invention into clinical practice.¹² Compared to other theoretical frameworks, NPM¹⁵ and its derivatives, the Normalization Process Theory (NPT)¹⁶ and the Extended Normalization Process Theory (ENPT)^17,18 have proven to provide a very useful framework in evaluating the various factors associated with a complex intervention, from development to process testing and outcome assessment¹⁶ and have been implemented successfully.¹⁹ The NPT provides much needed information about context and about how changes can be made for adaptation of an intervention and factors affecting users interactions with the intervention.^17,20 Thus, as process evaluation is an important component of our project, we have used NPM and NPT as the framework for our project. Our model of care fits with the defined characteristics of a complex intervention, as it leverages 2 levels of care (primary and specialist) and providers (PCPs and nephrologists), and involves the implementation of a new system (eConsult) across multiple organizations for the purpose of process evaluation and development. We have used the NPM as a theoretical framework to develop our hypotheses and define our objectives for this study.

Intervention

The eConsult model of care will be used by both PCPs and nephrologists. Using the existing CKD clinical pathway (www.ckdpathway.ca),²¹ PCPs will identify high-risk patients with CKD based on their recent laboratory results (estimated glomerular filtration rate [eGFR] <30 mL/min/1.73 m² and/or albumin to creatinine ratio [ACR] >60 mg/mmol and/or sustained hematuria unexplained by a urinary tract source with ACR 3-60 mg/mmol irrespective of eGFR, or eGFR <60 mL/min/1.73 m²) and obtain care recommendations from nephrologists via the eConsult system within Netcare. PCPs will provide minimal demographic and clinical data (ie, eGFR, proteinuria, blood pressure, CVD risk factors, and medications) to initiate the eConsult process. The attending nephrologist will review these data and complete 3 guideline-concordant decision support steps as per the CKD management guidelines as provided on the Alberta CKD Pathway (www.CKDpathway.ca) and KDIGO²² to (1) determine the etiology of kidney disease, (2) assess the risk for progression/ESKD, and (3) assess relevant comorbidities, specifically CVD risk and target organ damage. These steps will be used to generate 1 of 2 automated responses to the PCP: (1) patient needs a face-to-face visit with a nephrologist, and additional investigations are requested or (2) patient does not need to be seen by a nephrologist, and follow-up care recommendations, thresholds for renewed referral, and educational material for patient support (eg, self-management, blood pressure, diet, lifestyle) are provided. This advice will be sent electronically to the PCP within 5 working days of receiving the required baseline data. If required, the on-call nephrologist will arrange the face-to-face visit directly with an available colleague, accounting for zone, location, expected wait time, and urgency of need.

Design and Research Plan

This integrated, sequential, mixed methods study will be conducted in 3 phases, with findings combined using a triangulation procedure according to standard frameworks. Based on the NPM theoretical framework, we will follow the approaches described below to achieve the outlined objectives for each phase.

Phase 1: Preimplementation

Research Question 1: What are stakeholders’ perceptions about the use of eConsult to improve access to specialist nephrology care?

In this phase, we will assess the perceptions of stakeholders (ie, PCPs, nephrologists, patients, policymakers, and other care providers) about our model to improve CKD care delivery, quality, and outcomes in Alberta. During this preimplementation phase, our goal will be to assess perceptions, readiness, and key barriers and enablers affecting the implementation and widespread adoption of eConsult in Alberta.

Design and Study Population

This phase of the study was qualitative based on grounded theory²³ as we want to understand the views of stakeholders about eConsult usage. We will conduct semistructured qualitative interviews and/or focus groups with participants recruited via purposive sampling and perform thematic analysis on the collected data.^24,25 We chose this design because it is the most appropriate for studies aimed at exploring program feasibility and stakeholder opinions about implementation, especially when little is known about the topic.²⁶ Purposive sampling is a strategy leveraging a nonprobability sample selected based on characteristics of a population and/or the objective of the study. This sampling approach is widely used in health care research as the sample set consists of experts in the specific area of research.²⁷ Purposive sampling is particularly useful in implementation science where the aim is to get in-depth views of experts, compared to random sampling where the focus is to minimize the impact of selection bias on results.²⁸ Purposive sampling is appropriate because statistical power and generalization are not the aim and it will ensure that our study captures the views of the different stakeholder groups identified by the research team in consultation with the relevant policymakers responsible for Alberta Health and Alberta Health Services (AHS). Following recommendations published in the literature and solicited from many experts in the field, we will use criterion/quota sampling techniques; specifically, we will collect data on relevant themes from a minimum number of participants from each stakeholder group to guarantee reasonable theoretical saturation. It has been estimated that using this analytical framework, at least 50 interviews will be required to reach theoretical saturation. We therefore propose to conduct 50 interviews and 8 to 12 focus groups; if necessary, we will conduct additional interviews and focus groups until saturation is reached. Theoretical saturation is the point at which no additional information can be found in a research study and the data cannot be further categorized. Essentially, at the point of theoretical saturation, no additional or different information is found and collection of additional data is repetitive of previous findings.²⁹ Theoretical saturation is determined when data cannot be subcategorized further and new information is not found.³⁰

Data Collection and Analysis

Members of each stakeholder group will be invited to participate in an interview or focus group (face-to-face, telephone, or Skype), which will be conducted using predetermined and semistructured interview questions focused on barriers and enablers associated with implementing and adopting the new model of care. We selected the semistructured modality of interviews and focus groups based on their reported efficacy and validity in this type of research. The open-ended nature of the questions provides opportunities for more extensive exploration of the issues at stake through the use of cues and prompts to elicit more relevant and explicit answers to questions. To develop questions for the patient stakeholder group, we will utilize the Picker Institute (http://www.picker.org) model,³¹ which highlights 8 dimensions of patient perspectives on care provision. There have been other models of integrative health care, but with inadequate and nonstreamlined focus on patient-centered care, such as The University of Gothenburg Center for Person-Centered Care (GPCC).³² Similarly, it has been shown that while other models did cover some aspects of patient-centered care,^33-35 The Picker Institute covers all important aspects of patient-centered care in detail.³⁶

Interviews and focus groups will be recorded and transcribed verbatim. After identifying information is removed, the transcripts will be entered into a software application designed to analyze qualitative data (eg, NVivo 10, ATLAS Ti 5.6). We will apply a conceptual framework to thematically analyze the collated data by coding and identifying key concepts and their relationships. The analysis will occur in 2 stages: basic descriptive documentation (manifest analysis) and thematic interpretation (latent analysis).³⁷ Using the predetermined thematic structure of our interview questions, we will divide the transcript data into small meaningful units (ie, sentence, phrase, paragraph) and attach a descriptor code to each. Subsequently, we will group defined codes into categories, which will enable us to map themes representing the content of each focus group. We will examine data across the various stakeholder groups to identify common categories and themes and assess whether data saturation has been reached. When interpreting our findings, we will focus not only on common themes but also on diverse perspectives that emerge. The results of the interviews and the thematic analysis will be used to inform the design of the eConsult portal and its mode of implementation.

Phase 2: Pilot Implementation

Research Question 2: What are the key factors and logistical issues affecting the feasibility of implementing eConsult for the care of patients with CKD?

In this phase, we will investigate the feasibility of implementation and associated logistical issues and potential impacts of eConsult on improving access to care (ie, reduced wait times, less travel for patients, easy access to specialists, better clinical surrogate outcomes, and cost savings for the health care system). We will pilot our model of care using the existing AHS e-Health infrastructure and secure web-based environment (Netcare). Drawing on existing frameworks from other specialties (ophthalmology, dermatology) and CKD frameworks from other countries,^38-41 we will engage in a year-long collaboration with AHS to refine and consolidate the system to ensure high-quality service design and continuous quality improvement by defining quality standards, operational policies, and sustainability pathways.

To pilot test the system, we will select providers from several primary care practices located in different rural and urban communities across Alberta and 4 kidney specialists from the study team. After appropriate training, participating PCPs will be granted access to the e-portal and will use the eConsult system to obtain recommendations from nephrologists about appropriate care for patients with CKD. The objective of this phase is to refine the system to increase its acceptability, feasibility, and impacts to facilitate the scaled adoption of the system across the province. After considering different evaluation methods, including multiphase optimization strategy (MOST), we decided to use the RE-AIM framework, which has been used in multiple settings to evaluate novel health care interventions.^42-48 The RE-AIM framework incorporates 5 essential elements: reach (scope of intervention), efficacy (a measure of usefulness), adoption (willingness to accept and use the intervention), implementation (extent of application), and maintenance (sustainability).⁴⁹ At the end of this phase, an operational eConsult system will be available for wider implementation, and its effectiveness, costs, and impacts on the health care system will be able to be evaluated. The RE-AIM framework will inform about the feasibility of engaging PCPs in the project, to what extent and how willing they are to adopt the system, how effective the adoption of the system is, and what is the outlook for routine, long-term implementation of the intervention in their practice. Detailed field notes will be compiled describing the effect of each element of the RE-AIM framework by the research team.

Phase 3: Implementation and Evaluation

Research Question 3: How does eConsult with practice facilitation compare with eConsult alone in terms of access to specialist care, quality of care, and related outcomes?

Standardized KT interventions have been used to minimize inappropriate care practices. We have 3 specific objectives for phase 3:

To determine the effectiveness of eConsult implementation plus practice facilitation compared to eConsult alone in enhancing guideline-concordant CKD care in underserved areas of Alberta.

To conduct a process evaluation to identify system, provider, and patient-level factors that affect implementation, as well as barriers and enablers that may affect wider adoption of the eConsult system.

To conduct a cost-effectiveness analysis to compare the effects of eConsult plus practice facilitation versus eConsult alone on cost savings to the health system from a societal perspective.

To address our objectives, we will leverage the knowledge-to-action cycle framework¹⁵ to (1) identify and learn about the current issue, (2) adapt knowledge to the current problem, (3) assess barriers to and enablers of knowledge use, (4) select and implement required interventions, (5) monitor knowledge use, (6) evaluate outcomes, and (7) sustain knowledge use (Supplemental Appendix 1).

Intervention

The full intervention will comprise a variety of KT strategies and tools to both passively and actively disseminate information and promote usage of the CKD Pathway (www.ckdpathway.ca) and the eConsult system within Netcare for nephrology referrals. The target audience will be PCPs (family physicians, nurse practitioners) and support staff (nurses, clinic managers, referral coordinators) in selected underserved communities, as described below. We will passively disseminate knowledge to PCPs across Alberta by distributing infographics and conducting Continuing Medical Education (CME) accredited workshops to teach practitioners how to use the CKD Pathway and the eConsult system. We will actively disseminate knowledge through practice facilitation by engaging PCPs and support staff to foster usage of the eConsult system as per the established CKD criteria. Practice facilitation is a widely implemented tool in clinical practice that supports primary care practices to help overcome challenges and improve patient care and health outcomes.¹⁸

In this study, practice facilitation will entail academic detailing visits by the University of Alberta’s Nephrology Team and the AHS Access Care Team to educate PCPs and support staff about CKD diagnostic and management criteria and utilization of the Netcare eConsult system for nephrology referrals. Practice facilitation will be applied at the level of the clinic (or a cluster of clinics in a single location) across the intervention sites (ie, selected health care centers in rural/remote communities in northern Alberta, inner-city Edmonton, and indigenous communities.) A local champion (opinion leader) will be identified at each site to serve as the primary contact for the study team; this person will help PCPs adhere to the protocol and use the CKD Pathway and eConsult system in their everyday practice. The focus of practice facilitation will be to enhance guideline-concordant CKD care at the clinic level. The strategies and tools we plan to use are summarized in Tables 1 and 2.

Table 1.

Knowledge Translation Strategies to Be Employed in the Intervention Package.

Intervention	Contextual definition	Evidence for efficacy
Local champion	A practice leader (physician, clinic manager, nurse, or referral coordinator) in a clinic will serve as a liaison to the detailing specialist and will help practice colleagues understand and apply the knowledge and skills delivered by the practice facilitators.	Local champions have positive impacts on outcomes of clinical projects in multiple areas of health and disease prevention worldwide.^41,50
Academic detailing	During visits to participating clinics, nephrologists will engage in practice facilitation to educate primary care providers and staff on chronic kidney disease management and referral practices and teach them how to use eConsult.	Academic detailing is highly recommended in evidence-based practice to improve patient care.^26,51 In many studies, academic detailing to local champions resulted in successful achievement of goals.^52,53
Audit and feedback	Through practice facilitation, ongoing feedback is provided from baseline to follow-up on performance and achievement of quality metric targets.	The implementation and positive impact of audit and feedback in health care has been widely explored. Reviews show that audit and feedback strategies are effective in health care settings when robust feedback is provided, health care professionals actively participate, and more importantly, when feedback is delivered periodically in both verbal and written forms.⁵⁴

Table 2.

Knowledge Translation Tools.

Intervention	Application	Description
CKD Pathway	Guidelines and criteria for CKD identification, management, and specialist referral	http://www.ckdpathway.ca/referral
Infographics	Documents describing the impact of CKD on patients and health care, and how to use the CKD pathway and eConsult system
Presentations	In-person presentations and video conferences designed to educate stakeholders affiliated with primary care networks, clinics and health care centers about the implementation of the eConsult system, and hands-on coaching on how to use eConsult functions	The presentations will focus on CKD diagnostic and management criteria, and utilization of eConsult through Netcare.

Note. CKD = chronic kidney disease.

Design and Study Population

We have considered various study designs (Supplemental Appendix 2) to determine what would work best to achieve our objectives. To fulfill objective 1, we will conduct a parallel-group, pre/post longitudinal quasi-experimental design to compare outcomes for primary care networks (PCNs) exposed to the eConsult with practice facilitation versus eConsult alone.

We will select intervention clinics and control clinics that provide services to underserved populations in Alberta, and in urban centers where the homeless population experience significant health issues and limited access to health services (Supplemental Appendix 3). The study plan and data analysis are outlined in Supplemental Appendix 3. We will match control sites (as a group, and defined by the PCN) with intervention sites based on the following variables: population size, number of practicing physicians, and relative prevalence (observed-to-expected ratios) of CKD and associated risk factors (eg, diabetes mellitus, hypertension).

Statistical Analyses

All statistical analyses will be performed using Stata 11.2. Outpatient nephrology visit rates across intervention and control sites during the baseline and study periods will be calculated by dividing the total number of visits for each PCN factored by population size. To evaluate the impact of the intervention within and between PCNs, we will leverage a mixed-effects model to control for potential clustering (random effects) of clinicians by site. Mixed-effects models are the appropriate statistical test for population-based intervention studies when it is not feasible to randomize individuals (or physicians) to intervention and control groups, yet patient-level observations remain the unit of analysis.⁵² For the within-PCN analyses, monthly eConsult rates, patient demographics, and practice type will be included as fixed effects. For between PCN analyses, we will also add site as a fixed effect.

Study Outcomes

The primary outcome will be the impact of the intervention on the rate of inappropriate face-to-face nephrology consults.²⁹ Inappropriate face-to-face nephrology consults are those patients who did not need to be seen by a nephrologist and could have been managed by their PCPs in their own communities.⁵⁵

Secondary outcomes will be as follows:

Impact on utilization (net usage of eConsult calculated as increase/decrease in usage of the eConsult system after implementation of the intervention)¹⁰

Impact on total number of appropriate face-to-face consults (patient needs to be seen by specialist)¹⁰

Appropriate prescriptions (eg, statins, ACEi/ARBs) as per the CKD Pathway guidelines⁵⁶

Time required for specialist to complete the eConsult request, reported as days (mean ± SEM)¹⁰

Improved risk factor control (ie, proteinuria, glycemia)⁵⁶

Based on findings from previous studies related to electronic referral and consultation systems, we expect the intervention to result in at least a 10% improvement in process of care parameters.^57,58 Thus, we expect to see at least a 10% reduction in the number of inappropriate nephrology visits after the intervention. To examine the use of appropriate medications (a secondary outcome), we will analyze the monthly rates of angiotensin-converting enzyme inhibitor (ACEi)/angiotensin receptor blocker (ARB) use preintervention and postintervention. Drug use will be classified as new use (no prescriptions for an ACEi or ARB in the prior year) or any use (irrespective of prior use). We will conduct the same analysis for cholesterol-lowering agents (both statin and nonstatin agents).

Process Evaluation

We will perform a process evaluation, a method commonly used in health care studies to determine whether intended program objectives were accomplished and resulted in defined outputs^59,60 and to determine the efficacy and robustness of methods used in a study.⁶¹ To fulfill objective 2 for this phase, we will conduct a process evaluation of our intervention using the framework described by Linnan and Steckler,⁶¹ which has been adopted in other public health studies.^62,63 The components of the framework include reach (ie, proportion of participants to whom the intervention was delivered), dose delivered (ie, extent of intervention delivered), dose received (ie, extent to which participants responded, engaged with, and utilized the intervention), fidelity (ie, extent to which the intervention was delivered as planned), and barriers (ie, issues and problems that occurred in implementing intervention components).⁶¹

Our process evaluation will reveal barriers and enablers at the provider and patient levels that influence adoption of this new model of care. Specifically, we will focus on how the intervention may be fully utilized and integrated into routine practices. To achieve this objective, we will use formative evaluation, which enables researchers to focus on key elements of implementation in complex settings and reveals ways to answer questions about context, adaptations, and responses to change.⁶⁴ Data sources include the interviews and focus groups with patients, clinicians, and policymakers (PCN directors, practice managers) from phase 1 of the study and postimplementation, as well as the outcome evaluation (provider satisfaction and use of health services) following the intervention.

We will track satisfaction with and acceptability of the eConsult system longitudinally (every 3 months) after the study with the Provider Satisfaction Survey. PCPs will be asked to complete the Provider Satisfaction Survey each time they see a patient with CKD and return it to the study team via mail, fax, or e-mail; all relevant contact information will be provided on the survey sheet. This short survey was adapted from a previous study evaluating PCPs’ perspectives on the use of eConsult.²⁵ The survey was tested by 5 PCPs and 3 nephrologists in Alberta for satisfaction, validity, and comments as well. When surveys are received, we will record responses in password-protected datasheets. PCP satisfaction will be assessed with a Likert score, where values of 4 to 5 will be considered high. We will analyze open-ended responses with NVivo 10 and perform a thematic analysis.

Cost-Effectiveness Analysis

To fulfill objective 3 for this phase of the project, we will conduct a detailed cost analysis of eConsult with practice facilitation versus eConsult alone. We will leverage previously published frameworks⁶⁵ to create a decision analytic model to determine the likely impacts of eConsult with practice facilitation on short- and long-term health outcomes and health care costs, versus eConsult alone. We will apply simulation models such as agent-based modeling (ABM), discrete event simulation (DES), and system dynamics (SD)⁶⁶ and use sensitivity analysis techniques to explore the impact of the new model on resource utilization across several scenarios to determine the ideal model that minimizes costs (ie, in-patient, ambulatory care, physician billing, medications) and maximizes effectiveness (eg, reduction in unnecessary specialist visits; reduction in late referral and urgent dialysis starts). Using simulation models, we will calculate the proportion of patients who were able to avoid unnecessary travel and assess impacts on nonmedical costs. Moreover, we will evaluate potential benefits such as reduced wait times, increased convenience for patients, time savings for PCPs and specialists, and user satisfaction. The data for the cost analysis will be obtained from administrative records and will be compared for the effects of eConsult plus practice facilitation versus eConsult alone on cost savings to the health system. The analysis will be conducted during the 5 years following the study. The cost analysis will inform about the benefits incurred to the health care system and the patients due to the intervention and will inform future decisions about health care resources allocations and changes in referral systems and standard of care practices.

Discussion

In this proposal, we have described a novel initiative to evaluate a model of care aimed at improving the process and quality of care for patients with CKD. Employing a robust mixed methods design, we will investigate stakeholder perceptions about potential barriers to and enablers of a new eConsult and decision support system strategy, focusing on elements that are most important for the design of a feasible, acceptable, implementable intervention. We will develop, pilot test, and assess the impact of the model in improving access to specialist nephrology care and apply the well-validated RE-AIM framework to evaluate the feasibility of province-wide implementation. Phase 3 of our project will address key challenges to obtaining optimal care for patients with CKD living in rural, remote, and underserved areas of Alberta, particularly timely referral and disease management. We have proposed to implement a KT intervention in selected PCNs and health care centers in Alberta to maximize the benefits of the new model of care. We will engage in practice facilitation to train PCPs and staff how to deliver evidence-based, guideline-concordant CKD care (www.ckdpathway.ca) and how to use the eConsult system within Netcare to obtain guidance from specialists. We expect our findings to reveal that eConsult combined with practice facilitation results in the best possible care for patients with CKD, especially those in rural/remote areas.

This project is designed to close evidence gaps on the effectiveness of population-level strategies for improving health outcomes among patients with CKD who live in rural, remote, and underserved communities. Specifically, this program will enhance efficiency by reducing rates of inappropriate referrals, build capacity for CKD care, and serve as a proof-of-concept and model for developing relevant policies and KT strategies for other chronic disease states and settings. This work focuses on elements that are most important for building an optimal care delivery system and a strategy to transform the management of CKD and associated risk factors in primary care by making it more effective, efficient, accessible, and timely, with a focus on rural, remote, and underserved communities.

Anticipated benefits include the following:

Improvement in the care process by eliminating unnecessary service utilization (ie, unnecessary specialist referrals, and hospitalizations or emergency room visits due to late referrals). We will accomplish this by leveraging existing well-developed guidelines and frameworks to standardize the referral system. Identifying high-risk populations and facilitating collaboration between PCPs and specialists will ensure that those who need specialist care receive it in an efficient and timely fashion, thereby reducing wait times, and improving the process of care and related outcomes.

Better health outcomes for high-risk patients in rural, remote, and underserved communities by reducing urgent visits to hospital emergency rooms and dialysis units. Our new model will facilitate identification and appropriate management of patients with CKD at high risk of progressing to ESKD and has the potential to significantly reduce the number of patients with CKD who initiate dialysis without having received predialysis care.

Development of an exemplary KT strategy to enhance optimal CKD care in underserved communities. This program will provide a proof of concept for a new model of care delivery that can be used to develop relevant policies and KT strategies to enhance its adoption. Once benefits are demonstrated for patients with CKD, our model can be applied to other chronic disease domains.

In order to determine the success of the eConsult system, it is essential to evaluate the feasibility of its adoption in routine clinical practice and to determine the challenges impeding implementation, as well as satisfaction with it.⁶⁷ The process evaluation of the intervention will reveal provider-level factors that influence implementation. Identifying barriers and enablers will help facilitate wider adoption of the intervention.

This proposed study has several strengths. First, the use of a mixed methods design will significantly improve the robustness of our findings. Many previous studies of e-Health interventions are based on observational designs (before-after and/or time series) in which outcomes are compared before and after implementation. Such an approach is weakened by an inability to control for secular trends involving health system changes and provider practices. Second, integrating the model with an existing province-wide and secure electronic health record system (Netcare) with an automated interface for consultation and patient data extraction will facilitate wider practice adoption and implementation. Third, our study has strong policy implications and potential for impact, as it will enable us to partner with providers and policymakers in renal programs across the province to improve care by implementing the new model for specialist–PCP interactions. Our expertise in policy-relevant research, evidence synthesis, and KT will ensure that high-quality evidence is generated to facilitate implementation of this new technology to improve CKD care, not only in Alberta but around the globe. Fourth, our proposed study could provide a foundation for more in-depth studies aimed at generating evidence on the relevance and feasibility of the model to improve care for patients with CKD. Our model may substantially change the way care is delivered by making it more effective, efficient, accessible, and timely and ensuring that care is of the highest quality. The proposed model has some limitations as well. First, high-speed Internet is not available in many rural and remote areas of Alberta, leading to more time spent in completing the eConsult request online versus faxing a referral the traditional way. Second, allied health care staff (referral coordinators, administrative staff) have to be trained on using the eConsult system and physicians at many remote sites do not have adequate staff to handle eConsult as an added task.

Finally, our model could favorably influence referral patterns, access to care, care quality, patient outcomes, and health care costs for people with CKD, which is a common and expensive condition. Once benefits for patients with CKD are demonstrated, our model can be applied to other chronic diseases.

Supplemental Material

Appendix_1_2_supp – Supplemental material for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease

Supplemental material, Appendix_1_2_supp for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease by Aminu Bello, Deenaz Zaidi, Branko Braam, Mark Courtney, Jodi Glassford, Kailash Jindal, Scott Klarenbach, Julia Kurzawa, Mohammed Osman, Nairne Scott-Douglas, Sue Szigety, Stephanie Thompson, Braden Manns, Brenda Hemmelgarn and Marcello Tonelli in Canadian Journal of Kidney Health and Disease

Supplemental Material

Appendix_2_4_supp – Supplemental material for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease

Supplemental material, Appendix_2_4_supp for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease by Aminu Bello, Deenaz Zaidi, Branko Braam, Mark Courtney, Jodi Glassford, Kailash Jindal, Scott Klarenbach, Julia Kurzawa, Mohammed Osman, Nairne Scott-Douglas, Sue Szigety, Stephanie Thompson, Braden Manns, Brenda Hemmelgarn and Marcello Tonelli in Canadian Journal of Kidney Health and Disease

Supplemental Material

Appendix_3_3_supp – Supplemental material for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease

Supplemental material, Appendix_3_3_supp for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease by Aminu Bello, Deenaz Zaidi, Branko Braam, Mark Courtney, Jodi Glassford, Kailash Jindal, Scott Klarenbach, Julia Kurzawa, Mohammed Osman, Nairne Scott-Douglas, Sue Szigety, Stephanie Thompson, Braden Manns, Brenda Hemmelgarn and Marcello Tonelli in Canadian Journal of Kidney Health and Disease

Supplemental Material

Reference-Semi-structured_Interview_Template_1 – Supplemental material for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease

Supplemental material, Reference-Semi-structured_Interview_Template_1 for Protocol: Improving Access to Specialist Nephrology Care Among Rural/Remote Dwellers of Alberta: The Role of Electronic Consultation in Improving Care for Patients With Chronic Kidney Disease by Aminu Bello, Deenaz Zaidi, Branko Braam, Mark Courtney, Jodi Glassford, Kailash Jindal, Scott Klarenbach, Julia Kurzawa, Mohammed Osman, Nairne Scott-Douglas, Sue Szigety, Stephanie Thompson, Braden Manns, Brenda Hemmelgarn and Marcello Tonelli in Canadian Journal of Kidney Health and Disease

Footnotes

Ethics Approval and Consent to Participate

Prior to commencement of the study, ethics approval was obtained from the University of Alberta Research Ethics Board. All participants (including primary care providers and patients) will be consented before participating in the study.

Consent for Publication

All authors have given their consent for publication of this article.

Availability of Data and Materials

The data for the study will remain with Dr. Aminu Bello and is available on request.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for the program is provided through the Northern Alberta Renal Program (NARP), the Interdisciplinary Chronic Disease Collaboration (ICDC) funded through an Alberta Innovates Collaborative Research & Innovation Opportunity Team Grant.

ORCID iDs

Deenaz Zaidi

Braden Manns

Supplemental Material

Supplemental material for this article is available online.

References

James

Hemmelgarn

Tonelli

Early recognition and prevention of chronic kidney disease. Lancet. 2010;375(9722):1296-1309. doi:10.1016/S0140-6736(09)62004-3.

El Nahas

Bello

AK.

Chronic kidney disease: the global challenge. Lancet. 2005;365(9456):331-340.

Bello

Wiebe

Hemmelgarn

, et al. A population-based study on care and clinical outcomes in remote dwellers with heavy proteinuria. Kidney Int Suppl (2011). 2013;3(2):254-258. doi:10.1038/kisup.2013.26.

Bello

Hemmelgarn

Lin

, et al. Impact of remote location on quality care delivery and relationships to adverse health outcomes in patients with diabetes and chronic kidney disease. Nephrol Dial Transplant. 2012;27(10):3849-3855. doi:10.1093/ndt/gfs267.

Bello

Hemmelgarn

Lloyd

, et al. Associations among estimated glomerular filtration rate, proteinuria, and adverse cardiovascular outcomes. Clin J Am Soc Nephrol. 2011;6(6):1418-1426. doi:10.2215/CJN.09741110.

Hemmelgarn

Zhang

Manns

, et al. Nephrology visits and health care resource use before and after reporting estimated glomerular filtration rate. JAMA. 2010;303(12):1151-1158. doi:10.1001/jama.2010.303.

Liddy

Rowan

Afkham

Maranger

Keely

Building access to specialist care through e-consultation. Open Med. 2013;7(1):e1-e8.

Scherpbier-deHaan

van Gelder

Van Weel

Vervoort

Wetzels

de Grauw

WJ.

Initial implementation of a web-based consultation process for patients with chronic kidney disease. Ann Fam Med. 2013;11(2):151-156. doi:10.1370/afm.1494.

Stoves

Connolly

Cheung

, et al. Electronic consultation as an alternative to hospital referral for patients with chronic kidney disease: a novel application for networked electronic health records to improve the accessibility and efficiency of healthcare. Qual Saf Health Care. 2010;19(5):e54. doi:10.1136/qshc.2009.038984.

10.

Alberta Netcare. http://www.albertanetcare.ca/. Accessed September 12, 2019.

11.

Hemmelgarn

Clement

Manns

, et al. Overview of the Alberta kidney disease network. BMC Nephrol. 2009;10:30. doi:10.1186/1471-2369-10-30.

12.

May

Finch

Mair

, et al. Understanding the implementation of complex interventions in health care: the normalization process mode. BMC Health Serv Res. 2007;7:148.

13.

Ogrinc

Batalden

Realist evaluation as a framework for the assessment of teaching about the improvement of care. J Nurs Educ. 2009;48(12):661-667.

14.

Agency for Healthcare, Reasearch and Quality. The Logic Model: The Foundation to Implement, Study, and Refine Patient-Centered Medical Home Models. https://pcmh.ahrq.gov/sites/default/files/attachments/LogicModel_032513comp.pdf. Accessed September 12, 2019.

15.

May

Mair

Dowrick

Finch

TL.

Process evaluation for complex interventions in primary care: understanding trials using the normalization process model. BMC Fam Pract. 2007;8:42. doi:10.1186/1471-2296-8-42.

16.

Mann

Shaw

Guthrie

, et al. Protocol for a process evaluation of a cluster randomised controlled trial to improve management of multimorbidity in general practice: the 3D study. BMJ Open. 2016;6(5):e011260. doi:10.1136/bmjopen-2016-011260.

17.

May

Johnson

Finch

Implementation, context and complexity. Implement Sci. 2016;11(1):141.

18.

May

Towards a general theory of implementation. Implement Sci. 2013;8:18. doi:10.1186/1748-5908-8-18.

19.

Sutton

Herbert

Burden

, et al. Using the Normalization Process Theory to qualitatively explore sense-making in implementation of the Enhanced Recovery After Surgery programme: “it’s not rocket science.” PLoS One. 2018;13(4):e0195890.

20.

Hooker

Small

Humphreys

Hegarty

Taft

Applying normalization process theory to understand implementation of a family violence screening and care model in maternal and child health nursing practice: a mixed method process evaluation of a randomised controlled trial. Implement Sci. 2015;10:39. doi:10.1186/s13012-015-0230-4.

21.

Donald

McBrien

Jackson

, et al. Development and implementation of an online clinical pathway for adult chronic kidney disease in primary care: a mixed methods study. BMC Med Inform Decis Mak. 2016;16:109. doi:10.1186/s12911-016-0350-z.

22.

Levin

Stevens

Bilous

, et al. Kidney Disease: Improving Global Outcomes (KDIGO) CKD Work Group. KDIGO 2012 clinical practice guideline for the evaluation and management of chronic kidney disease. Kidney Int Suppl. 2013;3(1):1-150.

23.

Foley

Timonen

Using grounded theory method to capture and analyze health care experiences. Health Serv Res. 2015;50(4):1195-1210. doi:10.1111/1475-6773.12275.

24.

Martinez-Mesa

Gonzalez-Chica

Duquia

Bonamigo

Bastos

JL.

Sampling: how to select participants in my research study. An Bras Dermatol. 2016;91(3):326-330. doi:10.1590/abd1806-4841.20165254.

25.

Liddy

Afkham

Drosinis

Joschko

Keely

Impact of and satisfaction with a new eConsult service: a mixed methods study of primary care providers. J Am Board Fam Med. 2015;28(3):394-403. doi:10.3122/jabfm.2015.03.140255.

26.

Allen

Ferrier

O’Connor

Fleming

Family physicians’ perceptions of academic detailing: a quantitative and qualitative study. BMC Med Educ. 2007;7:36. doi:10.1186/1472-6920-7-36.

27.

Cresswell

Clark

VL.

Designing and Conducting Mixed Method Research. 2nd ed. Thousand. Oaks, CA: Sage; 2011.

28.

Bernard

Research Methods in Anthropology: Qualitative and Quantitative Approaches. 3rd ed. Walnut Creek, CA: AltaMira Press; 2002.

29.

Glaser

Strauss

AL.

The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine; 1967.

30.

Guest

Bunce

Johnson

How many interviews are enough? an experiment with data saturation and variability. Field Meth. 2006;18(1):59-82.

31.

The Picker Institute. http://www.picker.org. Accessed September 12, 2019.

32.

Britten

Moore

Lydah

l D

Naldemirci

Elam

Wolf

Elaboration of the Gothenburg model of person-centred care. Health Expect. 2016;20(3):407-418.

33.

Ouwens

Hermens

Hulscher

, et al. Development of indicators for patient-centred cancer care. Support Care Cancer. 2010;18(1):121-130. doi:10.1007/s00520-009-0638-y.

34.

Stewart

MA.

Effective physician-patient communication and health outcomes: a review. CMAJ. 1995;152(9):1423-1433.

35.

Mead

Bower

Patient-centredness: a conceptual framework and review of the empirical literature. Soc Sci Med. 2000;51(7):1087-1110. doi:10.1016/s0277-9536(00)00098-8.

36.

Gerteis

Edgman-Levitan

Daley

Delbanco

, eds. Through the Patient’s Eyes: Understanding and Promoting Patient-Centered Care. Jossey-Bass; 2002.

37.

Graham

Logan

Harrison

, et al. Lost in knowledge translation: time for a map. J Contin Educ Health Prof. 2006;26(1):13-24. doi: 10.1002/chp.47.

38.

Campbell

O’Neill

Gibson

Thurston

WE.

Primary healthcare needs and barriers to care among Calgary’s homeless populations. BMC Fam Pract. 2015;16:139. doi:10.1186/s12875-015-0361-3.

39.

Vollman

Anderson

McFarlane

JM.

Canadian Community as Partner: Theory & Multidisciplinary Practice. Philadelphia, PA: Lippincott Williams & Wilkins; 2007.

40.

Laferriere

Liddy

Nash

Hogg

Navigating change: how outreach facilitators can help clinicians improve patient outcomes. J Am Board Fam Med. 2012;25(2):232-237. doi:10.3122/jabfm.2012.02.110192.

41.

Damschroder

Banaszak-Holl

Kowalski

Forman

Saint

Krein

SL.

The role of the champion in infection prevention: results from a multisite qualitative study. Qual Saf Health Care. 2009;18(6):434-440. doi:10.1136/qshc.2009.034199.

42.

Kessler

Purcell

Glasgow

Klesges

Benkeser

Peek

CJ.

What does it mean to “employ” the RE-AIM model?

Eval Health Prof. 2013;36(1):44-66.

43.

Carlfjord

Andersson

Bendtsen

Nilsen

Lindberg

Applying the RE-AIM framework to evaluate two implementation strategies used to introduce a tool for lifestyle intervention in Swedish primary health care. Health Promot Int. 2011;27(2):167-176.

44.

Glasgow

Askew

Purcell

, et al. Use of RE-AIM to Address Health Inequities: application in a low-income community health center based weight loss and hypertension self-management program. Transl Behav Med. 2013;3(2):200-210. doi:10.1007/s13142-013-0201-8.

45.

Glasgow

Dickinson

Fisher

, et al. Use of RE-AIM to develop a multi-media facilitation tool for the patient-centered medical home. Implement Sci. 2011;6:118. doi:10.1186/1748-5908-6-118.

46.

Glasgow

Phillips

Sanchez

MA.

Implementation science approaches for integrating eHealth research into practice and policy. Int J Med Inform. 2014;83(7):e1-11. doi:10.1016/j.ijmedinf.2013.07.002.

47.

Altpeter

Gwyther

Kennedy

Patterson

Derence

From evidence to practice: using the RE-AIM framework to adapt the REACHII caregiver intervention to the community. Dementia (London). 2015;14(1):104-113. doi:10.1177/1471301213494499.

48.

Huye

Connell

Crook

Yadrick

Zoellner

Using the RE-AIM Framework in formative evaluation and program planning for a nutrition intervention in the Lower Mississippi Delta. J Nutr Educ Behav. 2014;46(1):34-42. doi:10.1016/j.jneb.2013.09.006.

49.

Gaglio

Shoup

Glasgow

RE.

The RE-AIM framework: a systematic review of use over time. Am J Public Health. 2013;103(6):e38-e46. doi:10.2105/AJPH.2013.301299.

50.

Gallagher

Nutting

Nease

Jr. , et al. It takes two: using coleaders to champion improvements in small primary care practices. J Am Board Fam Med. 2010;23(5):632-639.

51.

VanHoof

Harrison

Miller

Pappas

Fischer

MA.

Characteristics of academic detailing: results of a literature review. Am Health Drug Benefits. 2015;8(8):414-422.

52.

Shaw

Howard

West

, et al. The role of the champion in primary care change efforts: from the State Networks of Colorado Ambulatory Practices and Partners (SNOCAP). J Am Board Fam Med. 2012;25(5):676-685.

53.

Taylor

Machta

Meyers

Genevro

Peikes

DN.

Enhancing the primary care team to provide redesigned care: the roles of practice facilitators and care managers. Ann Fam Med. 2013;11(1):80-83. doi:10.1370/afm.1462.

54.

Ivers

Jamtvedt

Flottorp

, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012(6):Cd000259.

55.

Keely

Magner

Afkham

Liddy

Nephrology eConsults for primary care providers: original investigation. Can J Kidney Health Dis. 2018;5:1-6. doi:10.1177/2054358117753619.

56.

Alberta Health Services. CKD Pathway. http://www.ckdpathway.ca/referral. Accessed September 12, 2019.

57.

Keely

Liddy

Afkham

Utilization, benefits, and impact of an e-consultation service across diverse specialties and primary care providers. Telemed J E Health. 2013;19(10):733-738. doi:10.1089/tmj.2013.0007.

58.

Olayiwola

Anderson

Jepeal

, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Ann Fam Med. 2016;14(2):133-140. doi:10.1370/afm.1869.

59.

Center for Disease Control. Types of Evaluation. https://www.cdc.gov/std/program/pupestd/types%20of%20evaluation.pdf. Published 2014. Accessed September 12, 2019.

60.

Moore

Audrey

Barker

, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. doi:10.1136/bmj.h1258.

61.

Steckler

Linnan

Israel

Process Evaluation for Public Health Interventions and Research. San Francisco, CA: Jossey-Bass; 2002.

62.

Hall

Zeveloff

Steckler

, et al. Process evaluation results from the HEALTHY physical education intervention. Health Educ Res. 2012;27(2):307-318. doi:10.1093/her/cyr107.

63.

Nakimuli-Mpungu

Musisi

Wamala

, et al. The effect of group support psychotherapy delivered by trained lay health workers for depression treatment among people with HIV in Uganda: protocol of a pragmatic, cluster randomized trial. JMIR Res Protoc. 2017;6(12):e250. doi:10.2196/resprot.8925.

64.

Stetler

Legro

Wallace

, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006;21(suppl 2). doi:10.1111/j.1525-1497.2006.00355.x.

65.

Liddy

Drosinis

DeriArmstrong

McKellips

Afkham

Keely

What are the cost savings associated with providing access to specialist care through the Champlain BASE eConsult service? a costing evaluation. BMJ Open. 2016;6(6):e010920. doi:10.1136/bmjopen-2015-010920.

66.

Marshall

Burgos-Liz

IJzerman

, et al. Applying dynamic simulation modeling methods in health care delivery research-the SIMULATE checklist: report of the ISPOR simulation modeling emerging good practices task force. Value Health. 2015;18(1):5-16. doi:10.1016/j.jval.2014.12.001.

67.

Fox

Vest

Kahn

, et al. Improving evidence-based primary care for chronic kidney disease: study protocol for a cluster randomized control trial for translating evidence into practice (TRANSLATE CKD). Implement Sci. 2013;8:88. doi:10.1186/1748-5908-8-88.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.11 MB

0.02 MB

0.09 MB

0.30 MB