Abstract
Context
The Clinical Practice Improvement (CPI) approach is a methodological and quality improvement approach that has emerged and is gaining in popularity. However, there is no systematic description of its use or the determinants of its practice in rehabilitation settings.
Method
We performed a scoping review of the use of CPI methodology in rehabilitation settings.
Results
A total of 103 articles were reviewed. We found evidence of 13 initiatives involving CPI with six different populations. A total of 335 citations of determinants were found, with 68.7% related to CPI itself. Little information was found about what type of external and internal environment, individual characteristics and implementation process might facilitate or hinder the use of CPI.
Conclusion
Given the growing popularity of this methodological approach, CPI initiatives would gain from increasing knowledge of the determinants of its success and incorporating them in future implementation.
Introduction
In the current era of evidence-based practice, clinicians, researchers, and policy makers are under increasing pressure to find and analyze the evidence of the efficacy and effectiveness of rehabilitation interventions. Preferably, this knowledge is gained through the use of randomized clinical trials (RCTs). 1 However, this design faces pragmatic limitations that make it difficult to use in rehabilitation settings. In RCTs, the interventions studied are controlled rigorously 2 and delivered in a precise, given setting. 2 RCTs are sensitive to the issues of participant homogeneity.1,3 Thus, while RCTs may provide arguments for the efficacy of interventions in controlled settings, they do not demonstrate the interventions’ effectiveness in real settings where various clinicians provide individualized treatment to patients with complex and varied conditions.2,3 Moreover, ethical issues often limit the use of RCT in actual rehabilitation programs. 4 Finally, RCTs are considered costly to conduct,2,5 which makes this design less attractive for clinical settings.
New methodological avenues have been explored to produce evidence that rehabilitation treatments lead to effective outcomes. Among others, the clinical practice improvement (CPI) approach, also known as practice-based evidence (PBE) methodology, has emerged in the last few decades and is gaining in popularity. 6 CPI is an observational and mostly prospective methodological approach that was first described by Susan Horn. The most salient characteristic of CPI is that it takes place in actual (natural) health care settings during the course of natural episodes of care and with real patients. CPI combines (1) an accurate description of the individual characteristics of the client (patient factors) with (2) a detailed quantification of the interventions actually provided to the patient (process factors) in order to (3) better explain the results of these interventions on various characteristics of the patient (outcomes).2,7,8 Since CPI considers the natural process of care and takes place in a given setting, it requires data to be collected from several patients (individual data and outcomes), from several professionals (process factors), and ideally during entire episodes of care in order to be able to establish the relationships between the three types of factors and examine which interventions are more effective on patients with specific conditions. Thus, when using CPI methodology, the process of care is not altered and almost every patient is included, which has thus far allowed for the inclusion of large samples of patients (eg,>1,500) in recently published studies. 1 One important characteristic of the CPI is that the clinicians themselves must collect the process factor data on a patient chart or using a point-of-care (POC) tool, which describes the arrays of interventions they can provide to a client for each therapy session and outlines specific interventions and durations. POC tools are often created by therapists to describe the interventions they provide in their own setting. They are considered to provide an accurate description of rehabilitation services, which is essential when attempting to explain the actual contributions of interventions to rehabilitation outcomes and thus open the black box of rehabilitation interventions. According to CPI methodology, patient factors and outcomes are collected in the course of the usual care but can also be supplemented if this information is deemed insufficient. Data are gathered in large databases, where sophisticated analyses can be performed, typically to account for various confounders such as disability severity and age. The internal validity of the CPI methodology is considered moderate to high, but the external validity is deemed high due to its naturalistic character, which is one of its salient strength. A detailed description of CPI methodology can be found elsewhere.4,6,9–11
Attempts at using CPI have varied especially in nature and complexity. An example of a large complex study is one by DeJong et al named JOINTS I12–15 in which the researchers studied the care offered to people who underwent a joint replacement (hip or knee). Using paper POC and chart review, occupational therapists and physical therapists were able to obtain patient and process information for 59,982 therapy sessions conducted with 2,158 patients in 11 US-based facilities. The JOINTS I study is the most comprehensive description to date of services received by patients who underwent joint replacement. In a smaller study, Khan et al 5 used CPI methodology to study the care and outcome of 24 patients with multiple sclerosis who were admitted in a tertiary hospital in Australia. The care offered by occupational therapists, physical therapists, speech language therapists, and social workers were documented in an attempt to determine their effects on functional status and length of stay.
In addition to being a research methodology, CPI is considered a practical, bottom-up approach for quality improvement 16 that can be used by clinicians on a day-to-day basis to document, test, and improve their professional practices. The active involvement of clinicians in every CPI step is another distinctive characteristic of the method and a good example of a participatory initiative. CPI requires a strong commitment by clinicians in developing protocols, collecting relevant data, and interpreting results while, in return, offering them very practical opportunities for evidence-based improvement of care in their own settings. Indeed, close clinician implication is deemed as a facilitator to knowledge translation and use of results. Thus, the CPI methodology's compatibility with natural rehabilitation regimen,1,17 relatively low cost,1,12,18 and potential direct impact on practices 4 have made it an increasingly interesting methodology for both researchers and clinicians.
CPI development and related clinical outcomes have been extensively described in various impairment-specific settings,1,2,4,7–63 but to date, there is no systematic description of CPI initiatives in rehabilitation. Moreover, the factors that facilitate or hinder the implementation and use of CPI are not systematically described in the literature. This is of critical importance since the characteristics of the method (naturalistic approach, high external validity, etc.) make it relevant and potentially very appealing for rehabilitation settings, especially those in physical rehabilitation. Thus, the clinicians and researchers might consider its use without being aware of the conditions that are likely to influence the success or failure of their efforts. Since the methodology is growing in popularity and there are more and more calls for it to be implemented in a variety of settings, it is critical to increase the knowledge base regarding the determinants of its use. Thus, the goals of this paper are to (1) describe CPI approaches in physical rehabilitation settings and (2) examine under what conditions CPI is likely to be used in physical rehabilitation settings.
Methods
We performed a scoping review of the use of CPI methodology in rehabilitation settings. Scoping reviews aim to systematically map the literature pertaining to a particular matter and identify the key concepts, theories, and gaps in the research. 64 They do not systematically assess the quality of the selected studies or lead to an analytical synthesis (meta-analysis) of the results of the studies under review. 65 We used the enhanced version of Arksey and O'Malley's methodological framework 64 to structure our review.
Identification of the research question: This scoping review aimed to answer the following two broad research questions: “(1) What are the existing CPI approaches in physical rehabilitation settings and (2) under what conditions CPI is more likely to be used in physical rehabilitation settings?”
Identification and selection of the relevant studies: We performed a systematic literature search on the topic of CPI use in rehabilitation. 66 In order to capture the largest possible number of studies while keeping a narrow focus on studies that use or discuss CPI or PBE (in rehabilitation), only two exact key terms were used in the literature search: “Clinical Practice Improvement” and “Practice-Based Evidence.” These terms had to be included in the title, abstract, or keywords in order for a paper to be considered for inclusion. We did not limit our search by publication year or language. We performed our search in the main medical/rehabilitation databases (MEDLINE, CINAHL, Web of Knowledge, PEDro, AMED, and EMBASE) in June 2015. We also chose to include additional studies from a journal supplement published in August 2015, which was found through other sources (database automated alerts), after the search in the databases was completed. All articles that describe CPI methodology and/or results related to physical rehabilitation, ie, studies that were relevant to any rehabilitation clinician working with clients having physical and sensorial disabilities, in any setting (acute care, rehabilitation, community-based settings), were considered for inclusion. Thus, only theoretical papers about the CPI methodology (without specific references to context) and papers that describe physical rehabilitation and CPI experiences and results were included in the review.
Two researchers (M-EL, CG) independently screened the list of potential articles along with the abstracts to select the articles to be included in the scoping review. After comparison of the two lists, a third researcher (LN) resolved discrepancies in the study selection. We performed a secondary search by reviewing all the references of the articles resulting from the primary search and by screening the articles citing the selected articles from the primary search, particularly the seminal papers by Horn. We observed that typically, one CPI study leads to the publication of many papers, which explains the relatively large number of papers found through the secondary search.
Data extraction
Two researchers (MEL, ASA) designed and pilot tested an extraction chart based on the consolidated framework for the advancement of implementation science (CFAI) 67 (Appendix 1). This framework combines the constructs included in 19 different models and theories of implementation and translation of research findings into practice, primarily in the health care sector. The consolidated framework offers an overarching typology of 39 constructs that are grouped into five major domains: (1) intervention characteristics (8 constructs), (2) outer setting (4 constructs), (3) inner setting (14 constructs), (4) characteristics of individuals (5 constructs), and (5) processes (8 constructs).
The comprehensiveness of the CFAI framework suited the intent of this scoping review. From each article, we extracted the determinants pertaining to the constructs in the CFAI framework when available.
First, two researchers (M-EL and A-SA) independently extracted the data from 10 articles. Since the extractions were virtually identical, the validation was deemed unnecessary and the other articles were extracted by a single researcher.
Results
Selection of articles
The primary search in the databases generated 626 records (Fig. 1), and 56 additional records were identified through other sources. After removing duplicates and screening of the records, 146 full-text articles were assessed for eligibility and 103 were included in the review.
Flow chart of the scoping review.
Scope and breadth of the use of CPI methodology in rehabilitation
We found evidence of 13 initiatives involving CPI in rehabilitation settings. Characteristics of each CPI initiatives are described in Table 1. CPI was used in five studies involving stroke patients and three studies involving spinal cord injury patients. Other populations studied using CPI included individuals with joint replacements, outpatient physical therapy users, children with developmental disabilities, individuals with multiple sclerosis, and individuals with brain injury. The majority of the CPI initiatives were multisite (2-11 sites; mean of 4.7 sites). Three of the initiatives were located in Australia, four in the United States, two in the United Kingdom, and one in the Netherlands. Two of the initiatives were international efforts involving two countries (United States and New Zealand and two studies involving the United States and Canada).
Characteristics of CPI initiatives in rehabilitation.
Nine studies included assessments of all three types of factors of the CPI simultaneously (patient factors, process factors, and outcomes), while two were essentially concerned with a description of data collection for the process factors and patient factors. The POC tools used to gather detailed information on the therapy sessions were often paper-and-pencil tool and designed for a specific CPI use, but at least two POC tools were adapted from existing tools. Other types of POC tools included optical recognition forms (forms whose answers could be electronically managed) and electronic forms. Most of the therapy interventions were recorded using 5- or 15-minute intervals, but at least two used dichotomic (present/absent) recording. A mixed method (retrospective chart extraction/ prospective therapy intervention recording) was most often used, but a few studies only used retrospective data collection methods. All the CPI initiatives analyzed involved physical therapists, while a number of initiatives involved (one or several of the following) occupational therapists (n = 6), nurses (n = 3), speech language therapists (n = 4), psychologists (n = 3), recreational therapists (n = 3), vocational rehabilitation counselors (n = 1), and/or respiratory therapists (n = 1).
It is interesting to note that five of the CPI experiences reported did not use outcome measures and thus only use component of the CPI approach to describe the care provided to clients.
Determinants of CPI implementation
In the 66 articles analyzed, as many as 335 mentions of determinants of CPI implementations were identified (mean: 5.0/article; range: 1-26). The majority of these mentions were related to the intervention itself (CPI), while other determinant domains each represented less than 10% of the determinants reported (Table 2). In the following text, the CFAI domains are inscribed in bold while the dimensions are underlined.
Domains and dimensions of CFAI.
In the
Several determinants related to the
Many determinants pertaining to the
No CPI initiatives empirically investigated the
Finally, a few authors conducted a detailed examination of the influence of the
Discussion
The goal of this scoping review was to examine the use and determinants of the use of CPI methodology in rehabilitation settings.
We found that CPI has been used mostly to examine multidisciplinary rehabilitation interventions offered to individuals with neurological issues (stroke or spinal cord injury) in large, multi-site initiatives. We also found many mentions of determinants of CPI implementation. The majority of the mentions pertained to characteristics of the CPI and a few related to other domains of the CFAI framework, such as the internal or external environment of rehabilitation settings, personal characteristics of the clinicians involved, and the implementation process.
With regard to the use of CPI in rehabilitation settings, we observed that the majority of CPI rehabilitation projects were large initiatives involving many facilities and professions. This choice of settings might have been necessary because of the relatively limited number of users typically seen in specialized rehabilitation settings. However, the size and complexity of CPI initiatives in rehabilitation settings contrast with the bottom-up and participatory aspects of CPI. Indeed, the conclusions of CPI analyses are derived from large datasets built across several facilities. The analysis used for the determination of statistical associations is typically complex and usually carried out by experienced researchers. 6 Consequently, it may be difficult for clinicians to be involved in the management of CPI data as much as they can or might want to. Moreover, there is little information about how the evidences emerging from CPI experiences have been used to influence practice. From the experience gained, it remains unclear how clinicians who gathered the data actually used them to improve their practice.
It is also surprising that CPI has been used most often in the study of rehabilitation services offered to individuals with neurological conditions. Depending on the personal characteristics of the injured individual and the type, location, and severity of the neurological injury, the resulting clinical portrait varies greatly from one individual to another. As a result, neurological injuries call for a highly individualized mix of services, thus creating the so-called black box of rehabilitation services.20,23,78 However, neurological conditions are not very different from other conditions such as amputations or orthopedic injuries. The concentration of CPI initiatives in the neurorehabilitation field may be explained by the importance of the problem in terms of incidence (stroke) or long-term costs (spinal cord injury) of the conditions studied; the presence of researchers, managers, and clinicians willing to participate in CPI initiatives; and the existence of specialized programs specific to the population studied. These elements refer, respectively, to the problem proneness, volume, and operational feasibility discussed by Horn. 7 However, it will not be surprising if the method is applied to other rehabilitation populations in the near future. Moreover, despite the fact that intervention quantification by social workers, psychologists, and physicians has been more difficult, 18 these professionals seem to have ultimately benefited from the experience. CPI could grow in popularity within professions providing systemic interventions and help them to identify their unique contribution to the rehabilitation outcomes observed in their clients.
This scoping review also enabled us to document many determinants of CPI implementation. In fact, the large number of determinants related to the intervention characteristics and especially to the relative advantage of CPI suggests that the implementation of the methodology might have been facilitated by CPI's advantageous reputation. Indeed, Greenhalgh et al suggest that if users perceive a clear, unambiguous advantage in one intervention over another in terms of effectiveness or efficiency, it is more likely that they will adopt it. 79 Thus, the fact that CPI is deemed to confer a pragmatic advantage in contrast to RCTs might have facilitated its adoption by researchers and within clinical settings. The adaptability of the CPI also seems to facilitate its implementation in different rehabilitation settings. In spite of this, the relative complexity of the method should not been minimized. CPI requires detailed information on patients, interventions, and outcomes, which might not be readily available. Before adopting this methodological approach to quality improvement, it must be recognized that important and continuous data collection by many rehabilitation professionals with numerous clients might be required before one is able to draw valid associations and answer precise clinical questions. Data collection, management, and analysis on a large scale can turn out to be costly and burdensome,5,50 especially considering that systematic data collection and large databases are less frequently found in rehabilitation than in other medical settings. One might also consider the opportunity costs, that is, the cost of not providing rehabilitation interventions while designing, collecting, and analyzing CPI data in calculating the overall cost of using the methodology. Opportunity costs are especially important to bear in mind when considering the use of a participatory method such as CPI. The immediate added value of the approach on the quality or evidence-based nature of the rehabilitation interventions should thus be closely examined in order to balance the cost of clinician participation and the expected clinical benefits when considering the approach.
Given the scale of the involvement of clinicians and clinical settings in CPI initiatives, it is somewhat surprising that individual characteristics and internal setting characteristics, which promote or facilitate the use of CPI, have not been studied in more detail. Many theories suggest that personal characteristics, such as the attitude of a person toward a given intervention 80 or his/her state of change, 81 will influence the adoption of this intervention. The results of our scoping review suggest that in order to succeed, CPI requires knowledgeable, engaged, and enthusiastic clinicians. However, little is known about the readiness to change, maturity, or knowledge and beliefs about the interventions that might facilitate or hinder the implementation of the approach. It is very important to assess these personal characteristics if clinicians in a given setting are likely to engage in a CPI initiative. Similarly, it is increasingly recognized that the environment in which an intervention takes place greatly influences the outcome and eventual success of its implementation. 82 This review shows that CPI can be carried out in a variety of rehabilitation settings. However, it remains unclear whether CPI functions best in larger or smaller organizations, if it benefits from a distinctive form of management or organizational arrangement or if a specific implementation climate is required for the approach to be implemented successfully. We believe that CPI implementation benefits from the presence of a dedicated project coordinator, which might not be possible in smaller settings. Given the importance of CPI implementation and its potential impact on rehabilitation clinical practice, it is likely that some internal environment characteristics could be influenced prior to CPI implementation.
This study has some limitations that should be acknowledged. First, we chose to limit the review to theoretical papers and to studies explicitly referring to CPI (or PBE) methodology. This could have led to the exclusion of some studies that are based on CPI methodology but do not explicitly acknowledge it. Second, only CPI initiatives involving rehabilitation professionals were considered. This means that other determinants in the numerous CPI initiatives used in other health care settings (medical settings in particular) were not included in this review. Third, only one research extracted the data. Having two researchers extracting the data would have enhanced the robustness of the study. Fourth, the CPI experiences were often reported in an incomplete or inconsistent way within the papers under study, making their comparison and analysis into a common framework difficult. Fifth, the facilitators and barriers to CPI implementation were seldom explicitly labeled as such by authors, although such elements clearly appear to be determinants of successful CPI implementation. Thus, it was impossible to empirically determine if the determinants under review actually facilitated or hindered the intervention. Sixth, publication bias may have affected the results: we are personally aware of CPI initiatives that were not successful but were not the subject of published papers. Thus, these initiatives, their outcomes, and their determinants could not include them in the scoping review.
Conclusion
CPI methodology is an approach that is increasingly being used has allowed researchers and clinicians to open and analyze the black box of rehabilitation interventions for many rehabilitation populations. However, behind its intuitive and participatory features, it is also a complex and challenging approach, and its success depends on numerous determinants, many of which have not yet been empirically studied. Researchers and clinicians who wish to adopt this method to study rehabilitation processes and outcomes should secure an adequate amount of time and resources in order to gather sufficient data enabling valid conclusions. In particular, the design of POC tools and the resources necessary to perform fastidious data collection should be considered before the approach is adopted. However, when performed with sufficient resources and time, CPI studies (have been shown to) lead to important results, which answer concerns of clinicians and managers alike. Given the growing popularity of this methodological approach, CPI initiatives would gain from increasing knowledge of the determinants of its success and incorporating them in future implementation projects.
Author Contributions
Conceived and designed the experiments: MEL. Analyzed the data: MEL, CG, and ASA. Wrote the first draft of the manuscript: MEL. Contributed to the writing of the manuscript: MEL, CG, and LN. Agreed with manuscript results and conclusions: MEL, CG, ASA, and LN. Jointly developed the structure and arguments for the paper: MEL, CG, and LN. Made critical revisions and approved the final version: MEL, CG, and LN. All authors reviewed and approved the final manuscript.