Abstract
Background:
Systematic measure reviews can facilitate advances in implementation research and practice by locating reliable, valid, pragmatic measures; identifying promising measures needing refinement and testing; and highlighting measurement gaps. This review identifies and evaluates the psychometric and pragmatic properties of measures of readiness for implementation and its sub-constructs as delineated in the Consolidated Framework for Implementation Research: leadership engagement, available resources, and access to knowledge and information.
Methods:
The systematic review methodology is described fully elsewhere. The review, which focused on measures used in mental or behavioral health, proceeded in three phases. Phase I, data collection, involved search string generation, title and abstract screening, full text review, construct assignment, and cited citation searches. Phase II, data extraction, involved coding relevant psychometric and pragmatic information. Phase III, data analysis, involved two trained specialists independently rating each measure using Psychometric and Pragmatic Evidence Rating Scales (PAPERS). Frequencies and central tendencies summarized information availability and PAPERS ratings.
Results:
Searches identified 9 measures of readiness for implementation, 24 measures of leadership engagement, 17 measures of available resources, and 6 measures of access to knowledge and information. Information about internal consistency was available for most measures. Information about other psychometric properties was often not available. Ratings for internal consistency were “adequate” or “good.” Ratings for other psychometric properties were less than “adequate.” Information on pragmatic properties was most often available regarding cost, language readability, and brevity. Information was less often available regarding training burden and interpretation burden. Cost and language readability generally exhibited “good” or “excellent” ratings, interpretation burden generally exhibiting “minimal” ratings, and training burden and brevity exhibiting mixed ratings across measures.
Conclusion:
Measures of readiness for implementation and its sub-constructs used in mental health and behavioral health care are unevenly distributed, exhibit unknown or low psychometric quality, and demonstrate mixed pragmatic properties. This review identified a few promising measures, but targeted efforts are needed to systematically develop and test measures that are useful for both research and practice.
Plain language abstract:
Successful implementation of effective mental health or behavioral health treatments in service delivery settings depends in part on the readiness of the service providers and administrators to implement the treatment; the engagement of organizational leaders in the implementation effort; the resources available to support implementation, such as time, money, space, and training; and the accessibility of knowledge and information among service providers about the treatment and how it works. It is important that the methods for measuring these factors are dependable, accurate, and practical; otherwise, we cannot assess their presence or strength with confidence or know whether efforts to increase their presence or strength have worked. This systematic review of published studies sought to identify and evaluate the quality of questionnaires (referred to as measures) that assess readiness for implementation, leadership engagement, available resources, and access to knowledge and information. We identified 56 measures of these factors and rated their quality in terms of how dependable, accurate, and practical they are. Our findings indicate there is much work to be done to improve the quality of available measures; we offer several recommendations for doing so.
Introduction
Like many new scientific fields, implementation science faces a host of measurement issues, including reliance on single-use, adapted measures with unknown reliability and validity; skewed distribution of measures across theoretically important constructs; and inattention to practical measurement considerations (Martinez et al., 2014). Systematic measure reviews help new fields like implementation science address instrumentation issues by locating reliable, valid, pragmatic measures; identifying promising measures needing further refinement and evaluation, and highlighting measurement gaps for key constructs (Lewis et al., 2018).
Two systematic measure reviews (Gagnon et al., 2014; Weiner et al., 2008) have focused on organizational readiness for change, a construct in the inner setting domain of the Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009). Readiness for implementation, defined as “tangible and immediate indicators of organizational commitment to its decision to implement an intervention,” is considered a critical precursor to successful implementation (Damschroder et al., 2009; Gleeson, 2017; Kotter, 1996; Weiner, 2009). Although both reviews identified measures with some evidence of reliability and validity, their utility for guiding measure selection is limited in three respects. First, both only considered evidence available in the source article reporting the development or first use of a measure, neglecting evidence available about that measure’s psychometric properties in subsequent uses. Since a measure’s reliability and validity is to some extent sample-dependent, assessing all available evidence provides a fuller picture of the measure’s psychometric properties. Second, both employed a checklist appraisal approach that indicated simply whether or not evidence was available for specific psychometric properties. Although it is helpful to know evidence exists about the reliability and validity of a measure, it is more helpful to know the extent to which the measure is reliable or valid. Finally, neither review appraised the pragmatic quality of measures, that is, the extent to which they are relevant to practice or policy stakeholders and feasible to use in real-world settings (Glasgow, 2013; Glasgow & Riley, 2013; Stanick et al., 2019).
The aim of this review is to assess the reliability, validity, and practicality of measures of readiness for implementation and its CFIR-delineated sub-constructs: leadership engagement, available resources, and knowledge and information. The present review contributes to the literature by rating both the psychometric and pragmatic quality of measures using measure-focused evidence rating criteria (Lewis et al., 2018) and information from all published uses of a measure in mental health or behavioral health, not just the original or first use. Moreover, it facilitates head-to-head comparisons of measures of readiness for implementation and its sub-constructs, enabling researchers and practitioners to readily select measures with the most desirable profile of psychometric and pragmatic features.
Methods
Design overview
The general study protocol for this review is published elsewhere (Lewis et al., 2018). The review had three phases: data collection (Phase I), data extraction (Phase II), and data analysis (Phase III).
Phase I: data collection
Literature searches were conducted in PubMed and Embase bibliographic databases using search strings curated in consultation from PubMed support specialists and a library scientist. Consistent with our funding source and aim to identify and assess implementation-related measures in mental health and behavioral health, our search was built on four core levels: terms for implementation, terms for measurement, terms for evidence-based practice, and terms for behavioral health (Lewis et al., 2018). For the current study, we included a fifth level for each of the following CFIR constructs (Damschroder et al., 2009): (1) Readiness for Implementation, (2) Leadership Engagement, (3) Available Resources, and (4) Access to Knowledge and Information (Table 1). Searches were completed from April to May 2017; searches were updated through June 2019 to identify additional articles citing the original measure development studies. Two authors (C.D. and K.M.) independently screened titles, abstracts, and full-texts of articles using pre-specified criteria. Only empirical studies published in English in peer-reviewed journals containing one or more quantitative measures of these four constructs and used the measure in an evaluation of an implementation effort in a behavioral health context were included. Disagreement in screening decisions were resolved through discussion, with adjudication by a third author (B.W.) as needed.
Database search terms.
Screened studies progressed to the fourth step: construct assignment. Measures often have scales or items that map to multiple CFIR constructs; therefore, a two-pronged approach was employed (Lewis et al., 2018). First, two authors (C.D. and K.M.) assigned measures to constructs based on the measure authors’ indications of the construct, or constructs, that the measure purportedly measures. However, descriptions (or labels) that authors provide for their measures often fail to capture the range of item content reflected in those measures. For example, an author could describe a measure as an assessment of readiness for implementation, yet the measures’ item content might also reflect CFIR constructs such as implementation climate or CFIR sub-constructs like available resources, or leadership engagement. To account for these additional content areas, an item-analysis approach was employed (Chaudoir et al., 2013). Two authors (C.D. and K.M.) reviewed each measure’s items and assigned the measure’s scales or items to a CFIR construct or sub-construct only if two or more items in the measure reflect that construct or sub-construct. These authors discussed discrepancies in construct assignments and engaged the principal investigator (C.C.L.) in those cases when they could not reach agreement.
In the fifth and final step, articles citing the original measure development studies in PubMed and Embase were hand-searched to identify all empirical articles that used each measure in implementation research in mental or behavioral health care. Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagrams for each construct can be found in Appendix 1 (Moher et al., 2009).
Phase II: data extraction
Once all relevant literature was retrieved, articles were compiled into “measure packets,” which included the measure itself (if available), the measure development article (or article with the first empirical use in mental or behavioral health care), and all additional empirical uses of the measure in mental or behavioral health care. Three authors (C.D., K.M., and E.A.N.) reviewed each article and extracted information relevant to 14 psychometric and pragmatic rating criteria referred to hereafter as the Psychometric and Pragmatic Evidence Rating Scales (PAPERS). The full rating system and criteria for PAPERS is published elsewhere (Lewis et al., 2018; Stanick et al., 2018, 2019). Data were extracted for full measures and their constituent scales.
Anticipating the peer-reviewed literature would contain little information about the pragmatic properties of measures, we conducted a supplemental Google search. To mitigate the influence of algorithmically personalized search results, at least two authors (E.A.N., H.H., or K.M.) completed a Google search using the following steps: (1) the author entered the measure name in quotations in the search field of Google.com (if no formal measure name was provided, then the Google search could not be completed); (2) the search was timed to last no longer than 2 min to reflect the amount of time an average person would spend looking for basic information online (Baker, 2017; Singer et al., 2012)—we tested different lengths of time searching and observed diminishing returns; and (3) only the first page of results was scanned for promising results to reflect the average person’s willingness to search online search engines for information (Diakopoulos, 2015; Hassan Awadallah et al., 2014). If a website relating to the measure was located, it was mined for novel pragmatic data.
PAPERS includes nine criteria for assessing psychometric properties: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. In addition, PAPERS includes five criteria for assessing pragmatic properties of measures: cost, language accessibility, assessor burden (training), assessor burden (interpretation), and length (Stanick et al., 2019).
Each PAPERS property was rated by two authors independently (C.D., E.A.N., or K.M.) by applying the following scale to extracted data: “poor” (–1), “none” (0), “minimal/emerging” (1), “adequate” (2), “good” (3), or “excellent” (4) (Stanick et al., 2019). Final ratings were determined from either a single score or “rolled up” median approach. If a measure was unidimensional or the measure packet contained only one rating for a PAPERS property (e.g., internal consistency), then this value was used as the final rating for that property. If a measure had multiple scales, the ratings for those scales for a given PAPERS property were “rolled up” by computing and assigning the median rating for that PAPERS property to the measure as its final rating. If a measure had five scales, each scale was rated for internal consistency and the median of those ratings was assigned to the whole measure as its final rating for internal consistency. Likewise, if a measure had multiple ratings for a PAPERS property across articles in the packet, those ratings were “rolled up” by computing and assigning the median rating to the measure as its final rating for that PAPERS property. If a measure was used in four different studies, the measure’s internal consistency was rated for each article and the median of those ratings was assigned to the measure as its final rating for internal consistency. If the computed median resulted in a non-integer rating, the non-integer rating was rounded down (e.g., a median rating of 2.5 was rounded down to a 2). This decision results in a conservative rating. The “rolled up” median approach—intended to offer potential users of a measure a single rating for each PAPERS property—offers a less negative assessment of a PAPERS property than the “worst counts” approach wherein a PAPERS property is assigned the lowest rating for that property when there are multiple ratings (Lewis et al., 2015; Terwee et al., 2012). To give potential users a sense of the variability of ratings that were “rolled up” into a median, the range of ratings (high and low) is reported for PAPERS properties where there are multiple ratings. Once these steps were applied, a total score was calculated for each measure by summing the final ratings. The maximum possible rating is 36; the minimum is −9.
Descriptive data were also extracted (Table 2). These variables included construct definition, number of scales in the measure, number of items in the measure, number of uses of the measure in mental or behavioral health care, country of origin, target setting, target respondent, target problem, and implementation outcome assessed (Proctor et al., 2011). Data were also collected on the number of articles contained in each packet, providing bibliometric data for each measure.
Description of measures and subscales.
Some measures did not report total number of items.
Some measures were used in multiple settings, levels, populations, and outcomes.
Phase III: data analysis
Tables 3 and 4 report the presence or absence of psychometric and pragmatic information for each measure, by focal construct (i.e., readiness for implementation and its sub-constructs, as delineated in CFIR). Table 5 summarizes the availability of psychometric and pragmatic information for each PAPERS property for the measures of each construct. Table 6 displays for each focal construct the median and range of ratings for each PAPERS property, summarizing the psychometric and pragmatic strength of the measures of each construct. Tables 7 and 8 report the median and range of ratings for each PAPERS property for each measure, by construct. Figures 1 to 4 contain bar charts to display visual head-to-head comparisons across all measures, by construct.
Availability of psychometric information for measures, by focal construct.
OCTE, Offices, Staffing, Training, Computer Access, and E-Communications Scales.
Availability of pragmatic information for measures, by focal construct.
Psychometric and pragmatic information availability.
Summary statistics for instrument ratings. a
Median, excluding zeros where psychometric or pragmatic information not available.
Psychometric ratings for measures (median and range), by focal construct.
Pragmatic ratings for measures, by focal construct.
Head-to-head comparison of measures of readiness.
Head-to-head comparison of measures of leadership engagement.
Head-to-head comparison of measures of resource availability.
Head-to-head comparison of measures of access to knowledge and information.
Results
Search results
Searches of electronic bibliographic databases yielded 9 measures of readiness for implementation used in mental or behavioral health care; 24 measures of leadership engagement, of which 10 were scales of measures of readiness for implementation or scales of measures of other constructs; 17 measures of available resources, of which all but 2 were scales of measures of readiness for implementation or scales of measures of other constructs; and 6 measures of access to knowledge, of which 2 were scales of measures of other constructs.
Table 2 describes measures used to assess readiness for implementation, leadership engagement, available resources, and access to knowledge and information in mental or behavioral health care. Most measures of readiness for implementation defined the construct of readiness (89%), possessed six or more scales (89%), and included 11 or more items (78%). Many were used more than once (56%). All originated in the United States (100%). Most were employed in the outpatient community setting (78%), with providers (78%), directors (67%), or supervisors (67%) as respondents. These measures were used most commonly to assess readiness to implement evidence-based practice (EBPs) in substance use disorder (67%) or general mental health (44%). Two measures were used to predict implementation outcomes: the Texas Christian University Organizational Readiness for Change, or TCU-ORC (adoption) (Henggeler et al., 2008), (fidelity) (Beidas et al., 2014; Henggeler et al., 2008; Lundgren et al., 2013), (penetration) (Beidas et al., 2012, 2014; Guerrero et al., 2014), and (sustainability) (Hunter et al., 2017), and the Texas Christian University Survey of Organizational Functioning, or TCU-SOF (adoption) (Becan et al., 2012).
Measures of leadership engagement, available resources, and access to knowledge and information in mental or behavioral health care reported descriptive patterns similar to those for measures of readiness for implementation. The exception to the pattern was that many to most measures of leadership engagement and access to knowledge and information were only used once (63% and 83%, respectively). Four measures of leadership engagement were used to predict implementation outcomes: the Cook Implementation Measure managerial relations scale and leadership and vision scale (adoption) (Cook et al., 2015); the TCU-SOF leadership scale (adoption) (Becan et al., 2012), the State Health Authority Yardstick (fidelity and penetration) (Finnerty et al., 2009); and the Implementation Leadership Scale (sustainability) (Hunter et al., 2017). Several measures of available resources were used to predict implementation outcomes: the TCU-SOF funding, physical plant, staffing, training development, resources, and internal support scales (adoption) (Henderson et al., 2008); the Clinical Practice Organization Survey financial insufficiency, space insufficiency, clinical provider insufficiency scales (adoption) (Chang et al., 2013); and the TCU-ORC training resources and resources superscale (adoption) (Henggeler et al., 2008; Herbeck et al., 2008), (fidelity) (Beidas et al., 2014), and (penetration) (Beidas et al., 2014). The Chou Measure of Guidelines Information (Chou et al., 2011) is the only access to knowledge and information measure that was used to predict an implementation outcome (fidelity).
Measure ratings
Readiness for implementation
Damschroder et al. (2009) define readiness for implementation as “tangible and immediate indicators of organizational commitment to its decision to implement an intervention.” Nine measures were used in mental or behavioral health care. Table 5 summarizes the psychometric and pragmatic evidence available for these measures. Evidence about internal consistency was available for eight measures, convergent validity for one measure, discriminant validity for no measures, known-groups validity for four measures, predictive validity for three measures, concurrent validity for no measures, structural validity for one measure, responsiveness for one measures, and norms for nine measures. Information for the pragmatic property of cost was available for six measures, language readability for six measures, assessor burden (training) for no measures, assessor burden (interpretation) for four measures, and brevity for nine measures.
Table 6 describes the median ratings and range of ratings for psychometric and pragmatic properties for those measures for which evidence or information was available (i.e., those with non-zero ratings on PAPERS criteria). For measures of readiness for implementation used in mental or behavioral health care, the median rating for internal consistency was “2—adequate,” for convergent validity “2—adequate,” for known-groups validity “1—minimal,” for predictive validity “1—minimal,” for structural validity “–1—poor,” for responsiveness “2—adequate,” and for norms “2—adequate.” Note the median rating of “–1—poor” for structural validity was based on the rating of just one measure: the Survey of Organizational Functioning (Broome et al., 2007).
The TCU-ORC (Lehman et al., 2002) had the highest overall psychometric rating among measures of readiness for implementation used in mental or behavioral health care (psychometric total score = 12; maximum possible score = 36), with ratings of “3—good” for responsiveness and “2—adequate” for internal consistency, convergent validity, known-groups validity, and norms. However, the TCU-ORC was not assessed for structural validity using either exploratory or confirmatory factor analysis; consequently, its internal structure remains untested beyond a principal components analysis despite its use in more than 89 studies in mental and behavioral health care. Its predictive validity rating is “1—minimal” despite 87 tests of association between individual scales and a variety of outcomes.
For the nine measures of readiness for implementation used in mental or behavioral health care, the median rating for the pragmatic property of cost was “4—excellent,” for language “3—good,” for assessor burden (interpretation) “1—minimal,” and for brevity “2—adequate.” All nine measures of readiness for implementation used in mental or behavioral health care are free, hence the high pragmatic rating for cost. The TCU-ORC, the measure with the highest overall psychometric rating, scored a 9 out of 20 possible points for pragmatic criteria, with a “4—excellent” for cost, a “3—good” for language readability, and a “1—minimal” for assessor burden (interpretation) and brevity. However, no information was available for assessor burden (training). The four superscale version of the TCU-ORC scores were somewhat higher with 10 out of 20 possible points, the additional point resulting from a rating of “4—excellent” for language readability.
Leadership engagement
Damschroder et al. (2009) regard leadership engagement as a sub-construct of readiness for implementation and define it as “commitment, involvement, and accountability of leaders and managers with the implementation.” Twenty-four measures were used in mental or behavioral health care. Table 5 summarizes the psychometric and pragmatic evidence available for these measures. Evidence about internal consistency was available for 17 measures, convergent validity for 3 measures, discriminant validity for 1 measure, known-groups validity for 5 measures, predictive validity for 6 measures, concurrent validity for 2 measures, structural validity for 6 measures, responsiveness for no measures, and norms for 19 measures. Information for the pragmatic property of cost was available for 12 measures, language readability for 13 measures, assessor burden (training) for 4 measures, assessor burden (interpretation) for 6 measures, and brevity for 20 measures.
For measures of leadership engagement used in mental or behavioral health care (Table 6), the median rating for internal consistency was “3—good,” for convergent validity “4—excellent,” for discriminant validity “2—adequate,” for known-groups validity “1—minimal,” for predictive validity “1—minimal,” for concurrent validity “1—minimal,” for structural validity “2—adequate,” and for norms “2—adequate.” Note the median rating of “2—adequate” for discriminant validity was based on the rating of just one measure: the Implementation Leadership Scale (Aarons et al., 2014).
The Implementation Leadership Scale (Aarons et al., 2014) had the highest overall rating among measures of leadership engagement used in mental and behavioral health care (psychometric total score = 14; maximum possible score = 36), with ratings of “4—excellent” for internal consistency and convergent validity, “3—good” for structural validity, and “2—adequate” for discriminant validity and norms. However, its predictive validity rating was “–1—poor” based on a single study in which implementation leadership was not significantly associated with the extent of sustainment of an evidence-based treatment to address adolescent substance use following the end of federally funded implementation support grants to 78 community-based organizations (Hunter et al., 2017).
For the 24 measures of leadership engagement used in mental or behavioral health care, the median rating for the pragmatic property of cost was “4—excellent,” for language readability “3—good,” for assessor burden (training) “3—good,” for assessor burden (interpretation) “1—minimal,” and for brevity “3—good.” The Implementation Leadership Scale, the measure with the highest overall psychometric rating, scored a 10 out of a possible 20 points for pragmatic criteria, with a “4—excellent” for cost, “3—good” for brevity, and a “2—adequate” for language readability. However, no information was available for assessor burden (training) and assessor burden (interpretation) was rated “1—minimal.” The Team Leadership Scale of the Team Assessment Questionnaire (Mahoney et al., 2012) received the highest score on pragmatic criteria (15 points out of a possible 20), with ratings of “4—excellent” for cost, assessor burden (training), and brevity, and “3—good” for language readability. No information was available on assessor burden (interpretation).
Available resources
Damschroder et al. (2009) regard available resources as a sub-construct of readiness for implementation and define it as “the level of resources for implementation and on-going operations including money, training, education, physical space, and time.” Nineteen measures were identified in mental and behavioral health care. Table 5 summarizes the psychometric and pragmatic evidence available for these measures. Evidence about internal consistency was available for 14 measures, convergent validity for 3 measures, discriminant validity for no measures, known-groups validity for 6 measures, predictive validity for 8 measures, concurrent validity for no measures, structural validity for 1 measure, responsiveness for 1 measure, and norms for 15 measures. Information for the pragmatic property of cost was available for 14 measures, language readability for 13 measures, assessor burden (training) for 1 measure, assessor burden (interpretation) for 6 measures, and brevity for 15 measures.
For measures of available resources used in mental or behavioral health care (Table 6), the median rating for internal consistency was “2—adequate,” for convergent validity “1—minimal,” for known-groups validity “–1—poor,” for predictive validity “1—minimal,” for structural validity “2—adequate,” for responsiveness “4—excellent,” and for norms “2—adequate.” Note the median rating of “1—minimal” for structural validity was based on the rating of just one measure: the Educational Support scale of the Implementation Climate Scale (Ehrhart et al., 2014). Likewise, the median rating of “4—excellent” for responsiveness was based on the rating of just one measure: the Resources superscale of the TCU-ORC (Lehman et al., 2002).
The Texas Christian University Program Training Needs Survey (Simpson, 2002) had the highest overall rating among measures of available resources used in mental or behavioral health care (psychometric total score = 12; maximum possible score = 36), with ratings of “4—excellent” for norms, “3—good” for known-groups and predictive validity, and “2—adequate” for internal consistency. However, the TCU-ORC Program Training Needs Survey has not been assessed for structural validity using either exploratory or confirmatory factor analysis; consequently, its internal structure remains untested.
For the 19 measures of available resources used in mental or behavioral health care, the median rating for the pragmatic property of cost was “4—excellent,” for language readability “3—good,” for assessor burden (training) “2—adequate,” for assessor burden (interpretation) “1—minimal,” and for brevity “4—excellent.” Note the median rating of “2—adequate” for assessor burden (training) was based on the rating of just one measure: the Work Environment Scale (Simpson, 2002). The TCU-ORC Program Training Needs Survey, the measure with the highest overall psychometric rating, scored a 9 out of a possible 20 points for pragmatic criteria. However, no information was available for assessor burden (training) or assessor burden (interpretation). Measures that received higher scores on PAPERS pragmatic criteria include the General Resources scale of the TCU-ORC and the Director Resources scale of the TCU-ORC Director Version, both of which scored 12 out of 20 possible points for pragmatic criteria.
Access to knowledge and information
Damschroder et al. (2009) regard access to knowledge and information as a sub-construct of readiness for implementation and define it as “easy access to digestible information and knowledge about the intervention and how to incorporate it into work tasks.” Six measures were identified in mental and behavioral health care. Table 5 describes the psychometric and pragmatic evidence available for these measures. Evidence about internal consistency was available for three measures, convergent validity for two measures, discriminant validity for one measure, known-groups validity for one measures, predictive validity for one measure, concurrent validity for no measures, structural validity for one measure, responsiveness for no measures, and norms for two measures. Information for the pragmatic property of cost was available for two measures, language readability for two measures, assessor burden (training) for no measures, assessor burden (interpretation) for no measures, and brevity for two measures.
For measures of access to knowledge and information used in mental or behavioral health care (Table 6), the median rating for internal consistency was “2—adequate,” for convergent validity “1—minimal,” for discriminant validity “2—adequate,” for known-groups validity “–1—poor,” for predictive validity “–1—poor,” for structural validity “1—poor,” and for norms “2—adequate.” Note that the median ratings for discriminant validity, known-groups validity, predictive validity, and structural validity are based on the ratings of a single measure, although not the same measure in all instances.
The Structured Interview of Evidence Use (Palinkas et al., 2016) had the highest overall rating among measures of access to knowledge and information used in mental or behavioral health care (psychometric total score = 6; maximum possible score = 36), with ratings of “2—adequate” for internal consistency, discriminant validity, and norms; and “1—minimal” for convergent validity. However, the measure was rated “–1—poor” for structural validity, indicating more work is needed to ascertain its internal structure.
Of the six measures of access to knowledge and information used in mental or behavioral health care, only two had information available on the PAPERS pragmatic criteria. The Structured Interview of Evidence Use, the measure with the highest overall psychometric rating, scored a 7 out of 20 possible points for pragmatic criteria, with a “3—good” for language readability and a “2—adequate” for cost and brevity. The Visibility Scale of the Moore and Benbasat Adoption of Information Technology Innovation measure scored somewhat higher with 9 out of 20 possible points, with a “4—excellent” for language readability, “3—good” for brevity, and a “2—adequate” for cost.
Discussion
The findings of this review indicate that measurement of organizational readiness for change in mental and behavioral health care, much like measurement in implementation science generally, is poor. Although the review identified fewer adapted, single-use measures than a comparable review of implementation outcomes (Lewis et al., 2015), a similarly uneven distribution of measures across readiness for change and it sub-constructs is apparent. For example, although many measures of leadership engagement have been developed or used in mental and behavioral health care, few measures of access to knowledge have been developed or used.
Consistent with previous reviews (Gagnon et al., 2014; Weiner et al., 2008), limited psychometric evidence was available for measures of readiness for implementation and its sub-constructs. Although evidence of internal consistency was available for most measures, evidence for other psychometric properties was often not available. Remarkably, evidence of structural validity was available for only 9 of the 58 measures included in the review. Good measurement practice dictates checking a measure’s dimensionality before checking its internal consistency (DeVellis, 2012). This apparently occurs infrequently. Also striking is the lack of evidence for two psychometric properties of importance to implementation scientists: predictive validity and responsiveness. Fewer than half of the measures have been tested for predictive validity; of these, few were tested for associations with implementation outcomes. Thus, the question of whether readiness for implementation matters remains open. Responsiveness, or sensitivity to change, is important for evaluating effectiveness of efforts to increase readiness for implementation or its sub-constructs. Yet, the responsiveness of measures developed or used in mental and behavioral health care is largely unknown; only 3 of 58 measures included in this review were tested for this psychometric property.
It is possible that poor reporting contributed to the limited and uneven availability of evidence regarding measures’ psychometric properties. Although many peer-reviewed journals require reporting of internal consistency, few require reporting of other psychometric properties. Inconsistent reporting was evident in this review, often making it difficult to rate the psychometric strength of measures even when the psychometric property was assessed. Although it is possible that researchers are assessing multiple psychometric properties of the measures they use and are simply not reporting these results, it is perhaps more probable that they are not employing rigorous, systematic measure development and testing processes.
Generally speaking, measures included in this review exhibited low psychometric quality. Although median ratings for internal consistency were “adequate” or “good,” median ratings for all other psychometric properties were less than “adequate.” The measures with the highest overall psychometric rating were the TCU-ORC (Lehman et al., 2002) for readiness for implementation, the Implementation Leadership Scale (Aarons et al., 2014) for leadership engagement, the Texas Christian University Program Training Needs Survey (Simpson, 2002) for available resources, and the Structured Interview of Evidence Use (Palinkas et al., 2016) for access to knowledge. None of these measures scored higher than 14 out of a maximum possible score of 36. The Implementation Leadership Scale is a relatively new measure that has not been used much in mental health or behavioral health care, yet exhibits “good” to “excellent” ratings for several psychometric properties and “adequate” ratings for others. With additional testing of its psychometric properties, and additional use in empirical studies, its overall psychometric rating would likely increase. Conversely, the TCU-ORC was used 89 times in mental health and behavioral health care yet has not improved in its psychometric properties with repeated use. Although some TCU-ORC scales consistently demonstrate “good” internal consistency (e.g., program needs, training needs, cohesion, and communication), other scales consistently demonstrate “minimal” or “poor” internal consistencies (e.g., training, equipment/computer access, internet/e-communication, growth, adaptability, and autonomy). In addition, some TCU-ORC scales demonstrate variable levels of internal consistency across studies (e.g., offices, efficacy, and change) for reasons that have not been explored. Moreover, the TCU-ORC has not been tested for structural validity with exploratory or confirmatory factor analysis, perhaps because its length (118 items across 18 scales) imposes daunting sample size needs for such validation. Although the TCU-ORC has exhibited statistically significant associations with implementation outcomes in some studies, it has not in others and no discernible pattern emerges across multiple studies. Despite 55 tests of association between individual TCU-ORC scales and various outcomes, the predictive validity rating of the measure remains “minimal.” Although the TCU-ORC exhibited the highest overall psychometric rating among the measures of readiness for implementation, more work is needed to improve the measure’s psychometric properties before further use of the measure.
This review is the first to systematically assess the pragmatic properties of readiness for implementation measures. Although information was generally available regarding measures’ cost, language readability, and brevity, information was less available regarding measures’ training burden and interpretation burden. Moreover, information was less available for measures of access to knowledge than for measures of readiness for implementation and its other sub-constructs. Median ratings varied considerably across pragmatic properties, with cost and language readability generally exhibiting “good” or “excellent” ratings, interpretation burden generally exhibiting “minimal” ratings, and training burden and brevity exhibiting mixed ratings across measures of readiness for implementation and its sub-constructs. Although the measures with the highest overall psychometric ratings were not the measures with the highest overall pragmatic ratings, it is possible for measures to be both psychometrically strong and pragmatic. For example, the Implementation Leadership Scale could increase its overall pragmatic rating by easing assessor burden through the development of a manual for self-trained administration or, better yet, automated administration, score calculation, and interpretation. Likewise, the actionability of the Implementation Leadership Scale could be enhanced by empirically establishing cut-off scores to guide decision-making (e.g., whether to move forward with implementation, or pause and take action to increase leadership engagement) (Zieky & Perie, 2006). The incorporation of pragmatic ratings into systematic measure reviews such as this one could lead to the salutary effect of prompting measure developers to attend to practical measurement considerations. This, in turn, would close the research-to-practice gap in measurement in implementation science.
Recommendations
The results of this review reinforce recent calls for action to improve the state of measurement in implementation science. Improvement is urgently needed given the speed with which the field is growing. A shaky measurement foundation undermines the robustness or trustworthiness of the scientific knowledge that implementation science produces. Strengthening measurement will require significant investment of resources, effort, and time. The call for measure development and testing in recent funding announcements (National Institutes of Health [NIH], 2017; NIH & National Cancer Institute, 2016, 2018) and the provision of resources to support such efforts in recent funding opportunity announcements will help close the measurement gap and improve the quality of measurement in implementation science. What is needed is a federally funded, measurement-focused “moonshot” initiative to support a programmatic effort to develop, refine, and test measures of implementation constructs across the board; otherwise, measure development and testing will proceed haphazardly and slowly. Greater collaboration with psychometricians would augment implementation science’s general lack of training in measure development and testing. Although developing and testing new measures can take years, rapid-cycle approaches that combine “laboratory” and field methods can reduce the time to systematically develop and test new measures to as little as 18 months.
Implementation scientists currently looking for measures of readiness for implementation and its sub-constructs should consider the measures (mentioned above) that have the highest total score for psychometric strength. Although these measures could use additional testing and refinement, they represent the psychometrically strongest measures used in mental and behavioral health care. Potential measure users should bear in mind that a measure’s psychometric properties are to some extent sample-dependent, especially when samples are relatively small and non-representative of the target population. Hence, caution is warranted. The psychometric evidence for the measures reviewed here is context-specific; when measures are used in new or different samples, results may vary.
In addition to total score, potential measure users should also consider the specific type of psychometric evidence supporting the intended use of the measure. If the intention is to classify participants or settings in term of levels of readiness, for example, evidence of known-groups validity is essential. If classified participants or settings are to be assigned to receive differing readiness-enhancing strategies, evidence of predictive validity for targeted implementation outcomes—evidence that readiness matters—is also needed. The bar charts included here allow for visual determination of the overall psychometric strength, as indicated by the total length of the bars. The bars’ shading allows for within-criterion comparison of the strength of each measure with respect to specific criteria, allowing potential measure users to select a measure rated highly for specific psychometric properties of interest.
Two specific recommendations follow for increasing the quality and utility of measures of readiness for implementation and its sub-constructs. First, improve construct validity by (1) conceptually defining readiness for implementation, (2) situating the readiness construct in a theoretical framework specifying its causal relations with other constructs in the inner setting domain of CFIR, and (3) assessing both content validity and discriminant content validity to verify that the measure is measuring the readiness construct and not another construct. Our review indicates that many readiness measures include items or scales that measure other constructs like implementation climate, networks and communications, and organizational culture. A just-published review of organizational readiness for change measures used in health care reported similar findings (Miake-Lye et al., 2020). As a result, it is not at all clear if we are measuring what we think we are measuring. When developing a new measure, or refining an existing one, it is essential to clearly define what readiness is, what it is not, and how it relates to other constructs influencing implementation. Likewise, when writing or revising items, it is important to assess the fit between items and the construct the items purportedly measure.
Second, develop item banks and use item response theory to facilitate tailoring of readiness measures to specific implementation contexts, improving the actionability of readiness measures for making “go/no-go” decisions. Researchers often adapt existing measures or create new ones because they want measures that are specific to the evidence-based intervention of interest or the context in which implementation will occur (Martinez et al., 2014). The result is a proliferation of measures used only once with unknown psychometric properties, and importantly, are not comparable. In addition, existing measures are not useful for making decisions about whether to proceed with implementation or engage first in additional efforts to increase readiness, because they lack norm-referenced or criterion-referenced guidelines for interpreting measure scores. A measurement-focused “moonshot” initiative could support efforts to develop item banks using item response theory. When calibrated, researchers and implementers could tailor readiness measures yet still compare scores to norm- and criterion-reference guidelines and make decisions based on empirically sound interpretations of readiness measure scores.
Limitations
Two limitations are noteworthy. First, the review focused on measures of readiness for implementation and its sub-constructs developed or used in mental health or behavioral health care. It is possible that reliable, valid, and practical measures of these constructs exist that have not yet been used in this context. With respect to readiness for implementation, previously published reviews identified no psychometrically strong, pragmatic measures in health care, education, or business (Gagnon et al., 2014; Weiner et al., 2008); although, the first author’s Organizational Readiness for Implementation Change (ORIC) measure exhibits several desirable psychometric and pragmatic properties (Shea et al., 2014). Organization and management science has generated many measures of leadership over the past 70 years, some of which undoubtedly possess desirable psychometric and pragmatic properties. Whether these measures could be adapted for use in mental health and behavioral health care is unknown.
It is also possible that some of the measures included in this review have been used outside the context of mental health or behavioral health care; in that case, the psychometric and pragmatic ratings described above could change positively or negatively with additional evidence from such studies. Again, previously published measure reviews do not indicate this is likely for measures of readiness for implementation (Gagnon et al., 2014; Weiner et al., 2008). However, this could occur for measures like the Multifactor Leadership Questionnaire and other measures of leadership engagement that are used frequently in organization and management science.
Second, poor reporting practices in published articles not only limited evidence available about measures’ psychometric and pragmatic properties, but also the completeness of that evidence for psychometric and pragmatic rating. For example, some authors reported that structural validity was assessed through exploratory or confirmatory factor analysis yet did not report the variance explained by the factors, or neglected to report key model fit statistics. Likewise, authors sometimes stated that all scales exhibited internal consistency above a certain threshold (e.g., α > .70) rather than stating the exact values. As is the case in all systematic reviews, the consistency and quality of reporting of measurement properties in included studies influenced the extent to which measures’ psychometric and pragmatic properties could be rated and the level of uncertainty of those ratings.
Conclusion
This systematic measure review highlights the need for additional measure development and testing both to correct the maldistribution of measures of readiness for implementation and its sub-constructs and to enhance the psychometric and pragmatic properties of existing measures. Although some measures included in this review seem promising and merit further refinement and evaluation, most measures did not demonstrate sufficient reliability, validity, or practicality to recommend further use. High-quality, practical measures are critical for advancing implementation science. A concerted, coordinated effort to develop such measures is needed for implementation science to close the research–practice gap.
Footnotes
Appendix 1
Declaration of conflicting interest
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Funding for this study came from the National Institute of Mental Health, awarded to Dr Cara C. Lewis as principal investigator. Dr Lewis is both an author of this manuscript and an editor of the journal, Implementation Research and Practice. Due to this conflict, Dr. Lewis was not involved in the editorial or review process for this manuscript.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the National Institute of Mental Health (NIMH) “Advancing implementation science through measure development and evaluation” (1R01MH106510), awarded to Dr Cara C. Lewis as Principal Investigator.
