Sage Journals: Discover world-class research

Abstract

Background

Mental health is a critical component of wellness. Public policies present an opportunity for large-scale mental health impact, but policy implementation is complex and can vary significantly across contexts, making it crucial to evaluate implementation. The objective of this study was to (1) identify quantitative measurement tools used to evaluate the implementation of public mental health policies; (2) describe implementation determinants and outcomes assessed in the measures; and (3) assess the pragmatic and psychometric quality of identified measures.

Method

Guided by the Consolidated Framework for Implementation Research, Policy Implementation Determinants Framework, and Implementation Outcomes Framework, we conducted a systematic review of peer-reviewed journal articles published in 1995–2020. Data extracted included study characteristics, measure development and testing, implementation determinants and outcomes, and measure quality using the Psychometric and Pragmatic Evidence Rating Scale.

Results

We identified 34 tools from 25 articles, which were designed for mental health policies or used to evaluate constructs that impact implementation. Many measures lacked information regarding measurement development and testing. The most assessed implementation determinants were readiness for implementation, which encompassed training (n = 20, 57%) and other resources (n = 12, 34%), actor relationships/networks (n = 15, 43%), and organizational culture and climate (n = 11, 31%). Fidelity was the most prevalent implementation outcome (n = 9, 26%), followed by penetration (n = 8, 23%) and acceptability (n = 7, 20%). Apart from internal consistency and sample norms, psychometric properties were frequently unreported. Most measures were accessible and brief, though minimal information was provided regarding interpreting scores, handling missing data, or training needed to administer tools.

Conclusions

This work contributes to the nascent field of policy-focused implementation science by providing an overview of existing measurement tools used to evaluate mental health policy implementation and recommendations for measure development and refinement. To advance this field, more valid, reliable, and pragmatic measures are needed to evaluate policy implementation and close the policy-to-practice gap.

Plain Language Summary

Mental health is a critical component of wellness, and public policies present an opportunity to improve mental health on a large scale. Policy implementation is complex because it involves action by multiple entities at several levels of society. Policy implementation is also challenging because it can be impacted by many factors, such as political will, stakeholder relationships, and resources available for implementation. Because of these factors, implementation can vary between locations, such as states or countries. It is crucial to evaluate policy implementation, thus we conducted a systematic review to identify and evaluate the quality of measurement tools used in mental health policy implementation studies. Our search and screening procedures resulted in 34 measurement tools. We rated their quality to determine if these tools were practical to use and would yield consistent (i.e., reliable) and accurate (i.e., valid) data. These tools most frequently assessed whether implementing organizations complied with policy mandates and whether organizations had the training and other resources required to implement a policy. Though many were relatively brief and available at little-to-no cost, these findings highlight that more reliable, valid, and practical measurement tools are needed to assess and inform mental health policy implementation. Findings from this review can guide future efforts to select or develop policy implementation measures.

Keywords

implementation science public policy mental health service system measurement systematic review health policy policy implementation

Introduction

Mental health is a critical component of wellness at both the individual- and population-level. A significant proportion of the population is affected by mental illness, including an estimated 20%–40% of the United States (Kessler et al., 2005; Substance Abuse and Mental Health Services Administration, 2018) and international populations (Chisholm et al., 2007; Wittchen et al., 2011). The prevalence of mental illness should also be considered in light of global current events, such as the COVID-19 pandemic (Auerbach & Miller, 2020; Czeisler et al., 2020; Holmes et al., 2020; United Nations, 2020), splintered political systems (Nayak et al., 2021; Yan et al., 2021), and civil unrest (Labott, 2021; Ni et al., 2020; Strohecker, 2021). Unfortunately, access to mental health services is limited for many, due to barriers such as provider shortages, distance from services, and insufficient funding (Kilbourne et al., 2018; Saxena et al., 2007; Stirman et al., 2016; Williams, 2016). These challenges are often exacerbated in developing countries, where resources and mental health infrastructure (e.g., psychiatric hospitals or residential treatment facilities) are even more limited (Ngui et al., 2010). As a result, policy interventions are needed to improve mental health on a population-level.

Public policies present an opportunity to impact mental health education and services on a large scale, reaching, for example, communities, organizations, families, and individuals simultaneously. These “Big P” policies include “laws, regulatory measures, courses of action, and funding priorities concerning a given topic by a governmental entity or its representatives” (Eyler & Brownson, 2016). Within this study, we define “Big P” mental health policies as mandates, guidelines, or recommendations established by governmental entities or legislative bodies for certain practices or services that affect mental health education and/or the provision of mental health services. In the United States public mental health policies have increased access to evidence-based mental health interventions, encouraged changes to mental health service provision using incentives or disincentives, and created preferred treatment lists (Goldman & Azrin, 2003; Raghavan et al., 2008; Wensing et al., 2020). Public policies also have the potential to improve mental health equity by addressing the social determinants of health (SDOH; i.e., economic stability, built environment, health and healthcare, social context, and education) (Dawes, 2016; Rust et al., 2015), concepts such as a stigma and discrimination for those with mental illness, as well as healthcare infrastructure, housing, transportation, and education (Alegría et al., 2003, 2018; Castillo et al., 2018; Hinshaw, 2006; Powell et al., 2016; Purtle et al., 2020).

Despite the potential of public policies to improve health, developing and successfully passing a policy does not guarantee its implementation or sustainment. Policy implementation refers to the process of carrying out a policy mandate (Natesan and Marathe, 2015) and remains understudied compared to earlier stages in the policy process (Bullock et al., 2021). There is an opportunity to apply principles and methods from implementation science, which seeks “to understand not only what is and is not working, but also how and why implementation is going right or wrong, and testing approaches to improve it” (Peters et al., 2013). Despite the potential impact on population health, there is a dearth of implementation science literature directed at the policy sphere, particularly related to mental health, leading to calls for a greater emphasis on policy research (Emmons & Chambers, 2021; Hoagwood et al., 2020; Oh et al., 2021; Purtle et al., 2017, 2016).

Policy implementation is studied by multiple fields, including—but not limited to—implementation science and political science; however, the terminology differs between disciplines. Though both fields emphasize context and its role in helping or hindering implementation, implementation science often distinguishes between internal and external context (commonly called inner setting and outer setting; Damschroder et al., 2009; Nilsen et al., 2013). Additionally, implementation science researchers measure implementation outcomes, frequently citing the Implementation Outcomes Framework (IOF) (Proctor et al., 2011), which encompasses the following: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability. Political science, on the other hand, refers to implementation outcomes as outputs, while implementation outcomes refer to the policy's overall impact on the identified problem (e.g., changes in health outcomes after implementing a nutrition policy; DeGroff & Cargo, 2009). Finally, fidelity to a policy (i.e., the degree to which it was implemented as intended; Proctor et al., 2011) is conceptualized as compliance within political science (Natesan and Marathe, 2015). This review applies an implementation science lens to policy implementation, thus using the following terms throughout: outer setting (see Table 1 for constructs and definitions) and implementation outcomes (see Table 2 for constructs and definitions), which includes fidelity/compliance.

Table 1.

Mental Health Policy Implementation Determinants Assessed in Included Measures (n = 34)^a.

Domain	Included measuresn (%)	Definition
Policy characteristics
Adaptability	5 (14%)	Degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs^c
Complexity	4 (11%)	Perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, and intricacy and number of steps required to implement^c
Inner setting
Presence of champions	3 (9%)	Field or practice leaders, people who can facilitate and support practice change among professionals^c,d
Organizational culture and climate	11 (31%)	Culture: Norms, values, and basic assumptions of a given organization; or Climate: Absorptive capacity for change, extent policy compliance will be rewarded, supported, and expected within their organization^c
Policy implementation climate	10 (29%)	Extent to which compliance with the policy mandate will be rewarded, supported, and expected within their organization^c
a. Goals and feedback	3 (9%)	Degree to which [the policy-mandate] goals are clearly communicated, acted upon, and fed back to staff and alignment of that feedback with goals^c
b. Relative priority	6 (17%)	Individuals’ shared perception of importance of the [policy] implementation within the organization, competing priorities^c
Opinion leaders	1 (3%)	Individuals in an organization who have formal or informal influence on attitudes and beliefs of their colleagues with respect to implementing the policy^c
Readiness for implementation	27 (77%)	Tangible and immediate indicators of organizational preparations to implement a policy intervention^c
a. Communication of policy^b	4 (11%)	Actions taken to disseminate policy requirements and guidelines to implementers
b. Policy awareness or knowledge^b	6 (17%)	Implementing staff/provider awareness the policy mandate exists, or knowledge of policy content
c. Leadership for implementation	5 (14%)	Commitment, involvement, and accountability of leaders and managers with the implementation^c,d
d. Training^b	20 (57%)	Training of staff/providers on how to implement the policy-mandated practices
e. Non-training resources^b	12 (34%)	Level of resources dedicated for implementation and on-going operations including money…physical space, and time other than training resources^c
Structure of organization	6 (17%)	The social architecture, age, maturity, and size of an organization^c
Outer setting
Actor relationships and networks	15 (43%)	Presence and characteristics of relationships between parallel organizations that must collaborate for policy implementation to be effective^d
Visibility of policy role/actors to others	2 (6%)	Perceived presence and importance of different actors pertinent to implementation of the policy^d
Political will for policy implementation	1 (3%)	Societal desire and commitment to generate resources to carry out policies^d
Target population characteristics	4 (11%)	Demographics, norms, neighborhood environments of population groups that affect implementation^d

Implementation determinants: “Factors believed or empirically shown to influence implementation outcomes” (also called barriers, obstacles, facilitators, etc.) (Nilsen & Bernhardsson, 2019).

These constructs were derived from Damschroder et al., 2009 but were sub-divided based on findings during full-text screening.

Definition derived from Damschroder et al., 2009. ^d Definition derived from Bullock et al., 2021.

Table 2.

Mental health policy implementation outcomes assessed in included measures (n = 34)^a.

Domain	Included measures n (%)	Definition
Acceptability	7 (20%)	Perceptions by staff in organizations mandated to implement the policy, or perceptions of other stakeholders, that the policy mandate is agreeable, palatable, or satisfactory (Proctor et al., 2011)
Adoption	4 (11%)	Intention and initial actions of mandated organizations to revise their organizational policies to address policy mandates. (Not policy development or passage of bills into law.) (Proctor et al., 2011)
Appropriateness	7 (20%)	Perceived ﬁt, relevance, or compatibility of the [policy] for a given practice setting, provider, or consumer; and/or perceived ﬁt of the [policy] to address a particular issue or problem; context fit (Proctor et al., 2011)
Cost	5 (14%)	Cost impact of an implementation effort (Proctor et al., 2011)
Feasibility	6 (17%)	Extent to which a new [policy] can be successfully used or carried out within a given agency or settingLevel of administration required to implement a policy, often called policy automaticity (Proctor et al., 2011)
Fidelity/compliance	9 (26%)	Degree to which a [policy] was implemented as it was prescribed [mandated] (Proctor et al., 2011)
Penetration	8 (23%)	Integration of a [policy] within a service setting and its subsystems (Proctor et al., 2011)
Sustainability	0 (0%)	Extent [new policy] is maintained or institutionalized within a service setting's ongoing, stable operations (Proctor et al., 2011)
Health equity	6 (17%)	Measure contains items related to health equity or assessing the social determinants of health as they relate to the implementing organization or target population (U.S. Department of Health and Human Services, 2021)

Implementation outcomes: “Effects of deliberate and purposeful action to implement new treatments, practices” or policies, which can serve as indicators of the implementation process and overall success (Proctor et al., 2011).

Current Work and Gaps in the Literature

Though policies may be developed and passed by government entities at multiple levels, the burden of implementing policies and sustaining related practices often falls on actors beyond the legislative body (e.g., healthcare settings, community organizations). Additionally, many actors or organizations face additional barriers during policy implementation, including limited financial resources, personnel, or infrastructure (Wright, 2017). To enable comparisons of policy implementation across studies and settings, generalize findings, and ultimately address these barriers, a better understanding of policy-focused implementation constructs is needed, along with high-quality measures of these constructs.

Unfortunately, myriad measurement challenges exist within implementation science. First, many measures fail to operationalize the constructs being assessed (Bruns et al., 2019; McHugh et al., 2020; Watson et al., 2018). There is also an overreliance on single-use measures, which often lack testing and/or reporting of psychometrics, and splicing items from different measures, leading to limited measurement consistency and replicability of findings (Lewis, Weiner, et al., 2015; Martinez et al., 2014; Proctor et al., 2011). Additionally, there is a limited emphasis on pragmatic measures, which are developed to minimize burden and are typically brief, inexpensive, and easy to score and interpret (Glasgow & Riley, 2013; Martinez et al., 2014; Rabin et al., 2015). Despite the importance of psychometric and pragmatic properties, researchers rarely analyze both when developing and testing a new measure (Martinez et al., 2014; Stanick et al., 2018).

Implementation researchers acknowledge the dearth of measures assessing outer setting constructs (Damschroder et al., 2009), which encompass fiscal and policy factors, as well as external networks that can affect implementation (McHugh et al., 2020). These constructs are often difficult to disentangle and assess, which may explain why they remain understudied compared to inner setting constructs (McHugh et al., 2020). Inner setting constructs include factors within an implementing organization that influence implementation (e.g., organizational structure, implementation climate, leadership support) (Damschroder et al., 2009). Policy implementation can be affected by multi-level contextual factors, including outer and inner setting characteristics, and characteristics of the policy itself (Bullock et al., 2021; Eaton et al., 2011; Goldman & Azrin, 2003; Malekinejad et al., 2018). Also known as barriers, obstacles, enablers, or facilitators (Nilsen & Bernhardsson, 2019), we refer to these multi-level constructs as “policy determinants.” Given the limited emphasis on measurement of outer setting constructs and the effect these determinants can have on policy implementation, a better understanding of these constructs is needed.

Additionally, health equity is receiving increased attention within implementation science, underscoring the impact of the SDOH on mental health outcomes (Allen et al., 2014; Compton and Shim, 2015; Shim & Compton, 2020; World Health Organization & Calouste Gulbenkian Foundation, 2014). There are growing calls for improved integration of equity-related constructs into implementation methods, models, and measures (Baumann & Long, 2021; Brownson et al., 2021; Chinman et al., 2017; Douglas et al., 2019; Emmons & Chambers, 2021; Exworthy, 2008; Loper et al., 2021; Metz et al., 2021; Oh et al., 2021; Woodward et al., 2021). However, little is known about the current state of equity-related constructs in mental health policy implementation.

Lastly, while systematic reviews have examined quantitative measures of implementation across public health topics (Chaudoir et al., 2013; Chor et al., 2015; Clinton-McHarg et al., 2016; Emmons et al., 2011; Kaplan et al., 2010; Weiner et al., 2008), less scholarship has been done related to measures of policy implementation. Recent work has examined generalizable measures of health policy implementation (Allen et al., 2020), as well as health policy measures targeting school settings (McLoughlin et al., 2021), healthy food environments (Phulkerd et al., 2016), and chronic disease (Walsh-Bailey et al., 2020). To our knowledge, no such study has been conducted for mental health policies. Measures to evaluate mental health policy implementation are particularly important, as mental health stigma is pervasive and can impact policy development and implementation, as well as mental health outcomes (Conley, 2021; Purtle et al., 2018).

Study Aims

To fill these gaps in the literature, this study sought to (1) identify quantitative measurement tools (i.e., measures, questionnaires, surveys, scales) used to evaluate the implementation of public mental health policies; (2) describe the implementation determinants and outcomes assessed in these measures; and (3) assess the pragmatic and psychometric quality of the identified measures.

Methods

This review builds from a previous review of generalizable implementation measures (i.e., measures transferable across topics and settings) of public policies passed at federal, state/regional, or local levels (Allen et al., 2020). The Allen et al., study identified 70 generalizable measures of health policy implementation and excluded 597 measures, which were policy- or setting-specific (e.g., specific to school settings or mental health policies) (Allen et al., 2020). The current review examined the previously excluded or policy-specific quantitative measurement tools used to evaluate the implementation of governmental mental health policies. We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Moher et al., 2009) (see Figure 1 and Appendix A).

Figure 1.

Quantitative studies: PRISMA flow diagram.

Search Strategy

Our search strategy was guided by three implementation science frameworks—the Consolidated Framework for Implementation Research (CFIR) (Damschroder et al., 2009), the IOF (Proctor et al., 2011), and a newer, policy-specific framework—the Policy Implementation Determinants Framework (Bullock et al., 2021). The Policy Implementation Determinants Framework and CFIR provide a set of determinants known to impact implementation and were used to create a more thorough representation of policy implementation determinants. While the CFIR is applicable to a variety of implementation levels and settings, the Policy Implementation Determinants Framework contains constructs tailored to policy and the outer setting. The study team operationalized construct definitions from the frameworks to make them applicable to policy (e.g., changed the referent). For example, the IOF describes acceptability of a treatment, service, or innovation; we expanded the definition to include policy or policy-mandated practice. These frameworks guided the identification and selection of implementation determinants and outcomes. Both the CFIR and Policy Implementation Determinants Framework were used to identify determinants of policy implementation, while the IOF was used to identify outcomes of policy implementation.

Allen et al., (2020) conducted the original search, encompassing health policy broadly, in April 2019 in CINAHL Plus, Medline, PsycINFO, ERIC, Worldwide Political, and PAIS databases. The search terms and strategy were constructed with guidance from a research librarian, based on policy and implementation science literature (Bullock et al., 2021; Damschroder et al., 2009; Lewis et al., 2018; Proctor et al., 2011; Rabin et al., 2015; Sabatier & Mazmanian, 1980; Sabatier, 1988; Watson et al., 2018) and expert input. Studies excluded due to policy- or setting-specific wording (e.g., applicable only to mental health policies) were marked in Covidence systematic review software (Covidence Systematic Review Software, 2021).

In the current review, we retrieved studies excluded during full-text screening by Allen et al., due to policy- or setting-specific measures (n = 597). We then updated the original search strategy, with guidance from a research librarian, to include mental health-specific terms; a similar approach was used in previous reviews (McLoughlin et al., 2021; Walsh-Bailey et al., 2020). We ran the updated search in January 2021, using search strings related to mental health, public policy, implementation, and measurement (see Appendix B; full search strategy available in Appendices C and D).

Inclusion and Exclusion Criteria

The inclusion criteria were: (1) empirical study (including study protocols) of the implementation of public mental health policies (i.e., “Big P” policies) already passed or approved; (2) use of a quantitative measurement tool (i.e., measure, questionnaire, survey, scale); (3) inclusion of at least one implementation determinant or outcome; and (4) published in a peer-reviewed journal between January 1995 through December 2020 in English language. Given faint line between policy enactment and stages of policy implementation, we included studies that did not explicitly name the policy as the independent variable.

To distinguish between “little p” (i.e., organizational-level mandates or guidelines) and “Big P” policies, empirical authors’ descriptions of governing or legislative bodies and policy decision levels (e.g., local, state, national) were used. When information was limited in articles, one reviewer (MP) conducted hand searches to determine the governing body and level responsible for mandating or recommending the policy. Because our coding required copies of each measurement tool, January 1995 was selected as the beginning of the inclusion period as web-based surveying began around that time (Dillman et al., 2009). The onset of web-based data collection was thought to increase the likelihood that instruments had been maintained and were accessible, as compared to paper-based data collection.

The team conducted hand searches to locate measure development articles, technical documents, etc. that contained verbatim items and described the initial development and psychometric testing of the quantitative instruments used in studies that met all inclusion criteria (Lewis et al., 2018). Measure development articles were not included as part of the review, but these were reviewed by the primary (MP) and secondary (EJ) reviewers for relevant information about the measurement tool properties.

We applied the following exclusion criteria: (1) Articles focused solely on agenda setting, policy development, policy content analysis, or policy evaluation were excluded, as this review centered on the implementation of policies once adopted. (2) Empirical studies reported in theses and books were also excluded, as this study included only peer-reviewed journal articles. (3) Because this review aimed to assess implementation outcomes related to policy, studies which assessed only individual-level outcomes (e.g., health behaviors)—also referred to as client outcomes—or service outcomes (e.g., efficiency, treatment effectiveness) were excluded (Proctor et al., 2011). (4) Other health policies implemented in mental health settings or with mental health service users as the target populations (e.g., chronic disease screenings among individuals in residential treatment facilities; housing policies for adults with serious mental illness) were excluded, as this review focused specifically on policies with direct influence on practices or services that affect mental health education and the provision of mental health services (e.g., Affordable Care Act and Medicaid expansion increasing access to mental health care; state-level mandate for implementation of evidence-based mental health interventions). (5) Studies were excluded if full or partial measurement tools could not be located.

Screening Procedures

De-duplication was conducted using Endnote version X9 and Covidence (Covidence Systematic Review Software, 2021), which was also used to manage references and screen studies. We first reviewed all records excluded by Allen et al., (2020) during full-text screening (n = 597) for policy- and setting-specific measures, then reviewed studies published between April 2019 and December 2020 from the updated search. Two independent reviewers (MP, EJ) screened titles and abstracts to evaluate eligibility. They then independently screened full-text articles to determine inclusion for extraction and resolved disagreements by discussion. When needed, one reviewer (MP) contacted study authors for additional materials or information (e.g., copies of data collection instruments) to clarify eligibility. Three attempts were made to contact each study author for additional materials; those that remained unavailable were excluded.

Data Extraction

Data were extracted from study articles and accompanying materials (e.g., measure development articles, survey instruments) using Microsoft Excel. One reviewer (MP) extracted data for each measure, and a second reviewer (EJ) checked each entry and noted disagreements. The reviewers then resolved disagreements by discussion. A codebook was iteratively developed to aid in extracting and deductively coding study- and measure-related information. For studies using multiple measures, each measure was extracted separately (i.e., reported in separate rows).

Data extraction included information about: (1) general study and policy information; (2) methods used; (3) measure development and testing; (4) implementation determinants and outcomes (see Tables 1 and 2); (5) measure pragmatic and psychometric properties; and (6) policy impact on health equity in included measures. Health equity could be conceptualized as either a policy determinant or outcome, and include references to health disparities, health equity, and the SDOH (U.S. Department of Health and Human Services, 2021). Information about measure development/testing or a measure's pragmatic and psychometric properties was extracted from included empirical articles or, when applicable, from measure development articles.

The Psychometric and Pragmatic Evidence Rating Scales (PAPERS) were used to assess pragmatic and psychometric properties of quantitative measures (see Appendix E) (Lewis et al., 2018, 2021; Powell et al., 2017; Stanick et al., 2019). Psychometric criteria consist of nine categories, each scored on a −1 to 4 scale, with possible total scores of −9 to 36. There are five pragmatic criteria, with possible total scores of −5 to 20. Higher values indicate stronger psychometric properties or more pragmatic characteristics. The psychometric properties evaluated include: (1) internal consistency; (2) convergent validity; (3) discriminant validity; (4) known groups validity; (5) predictive validity; (6) concurrent validity; (7) structural validity; (8) responsiveness; and (9) norms. The pragmatic scale includes five domains: (1) brevity; (2) simplicity of language; (3) cost; (4) training burden; and (5) analysis burden.

Results

General Characteristics of Included Studies

As outlined in Figure 1, the search located 16,015 articles. After removing 4,790 duplicates, we reviewed 11,225 records during title and abstract screening. Two reviewers then screened 105 full-text articles; ultimately, 80 were excluded, leaving 25 articles for inclusion in this review. Within the 25 included studies, there were 34 unique quantitative measures (see Table 3). Many studies administered multiple measures, and two measures—the Evidence-Based Practice Attitude Scale (Aarons, 2004) and the Organizational Social Context Measurement System (Glisson et al., 2008)—were used in two studies.

Table 3.

Quantitative measures identified in studies of mental health policy implementation (n = 34 unique measures in 25 studies).

Tool name	Number of items	Empirical useAuthor, year	Country	Description of mental health policy
World Health Organization's Mental Health Policy Checklist	88	(Awenva et al., 2010)	Ghana	12 mental health objectives: Includes de-centralizing mental health services; providing training for mental health professionals; providing transportation for community mental health workers; raising mental health awareness for the family and community; and providing community rehabilitation of the mentally ill
World Health Organization's Mental Health Plan Checklist	159	(Awenva et al., 2010)	Ghana
Organizational Social Context Measurement System	105	(Beidas et al., 2013)	United States	Policy promoting evidence-based treatments in community centers through Philadelphia's Department of Behavioral Health and Intellectual DisAbility Services (DBHIDS)
Evidence-Based Practice Attitude Scale	15	(Beidas et al., 2013)	United States
Infrastructure, Policies and Practices in Child and Adolescents’ Mental Health Questionnaire	160	(Bielsa et al., 2010)	Multiple: 15 European countries	Policies related to mental health promotion and mental disorder prevention for children and adolescents
Higher Education Institution Survey	43	(Callaghan et al., 2012)	England	17 recommendations from the Chief Nursing Officer for England: Includes providing evidence-based care; strengthening relationships with service users and carers; supporting continued professional development for mental health nurses; and improving recruitment and retention for mental health nurses
Mental Health Trust Survey	96	(Callaghan et al., 2012)	England
Mental Health Nursing Survey	128	(Cusack et al., 2017)	Ireland	Policy proposes a holistic view of mental health difficulties, recommends an integrated multidisciplinary approach to mental healthcare, and emphasizes an active partnership between service users, their carers, and family members at every level of service provision
Health Policy Questionnaire	246	(Dlouhy, 2014)	Multiple: 7 European countries	Any national mental health policy initiative across Eastern Europe
Health Education Experts Survey	27	(Eisman et al., 2020)	United States	Michigan Model for Health: A comprehensive, evidence-based K-12 health curriculum, which includes core content areas focused on social and emotional health, nutrition and physical activity, safety, substance use, and personal health and wellness
Primary Care Integration Survey	9	(Gerber, 2018)	South Africa	Requirements for a revised mental health system with supports for the protection of human rights through the Department of Health of South Africa
School Health Policies and Practices Survey (SHPPS)	93	(Guerra et al., 2019)	United States	Recommendations from the Institute of Medicine's 2009 report: Individual states and communities coordinate to apply resources to the promotion and prevention of mental, emotional, and behavior disordersEvery Student Succeeds Act: Promoting evidence-based interventions and student supports within the school system
School Health Profiles Survey for Principals	180	(Guerra et al., 2019)	United States
School Health Profiles Survey for Teachers	207	(Guerra et al., 2019)	United States
Cognitive Behavioral Therapy Survey	10	(Kenicer et al., 2012)	Scotland	Recommendations from the National Institute of Clinical Excellence and the Scottish Intercollegiate Guidelines Network: Promoting guided self-help and computerized cognitive behavioral therapy for the treatment of depression, anxiety, and bulimia nervosa
Perceived Characteristics of Intervention Scale	8	(Lau & Brookman-Frazee, 2016)	United States	Los Angeles County Department of Mental Health, Prevention and Early Intervention Program: Providing training and implementation support for six evidence-based/informed practices to address a range of prevalent youth mental health problems
Knowledge and Confidence Survey/Therapists’ Perceptions of EBP Self-Efficacy	2	(Lau & Brookman-Frazee, 2016)	United States
Practice-specific Implementation Support	24	(Lau & Brookman-Frazee, 2016)	United States
Organizational Climate Measure	82	(Lau & Brookman-Frazee, 2016)	United States
Organizational Readiness to Change	97	(Lau & Brookman-Frazee, 2016)	United States
Adaptations of Evidence-based Practices	9	(Lau & Brookman-Frazee, 2016)	United States
Evidence-Based Practice Attitude Scale	15	(Lau & Brookman-Frazee, 2016)	United States
Organizational Social Context Measurement System	105	(Lau & Brookman-Frazee, 2016)	United States
Crisis Resolution Team Survey	91	(Lloyd-Evans et al., 2018a)	England	National Health Service Plan of 2000: National mandate for crisis resolution teams (i.e., multidisciplinary, specialist mental health services that offer brief intensive home treatment to people experiencing a mental health crisis)
Crisis Resolution Team Practice Survey	137	(Lloyd-Evans et al., 2018b)	England
World Health Organization's Assessment Instrument for Mental Health Systems Version 2.2 (WHO-AIMS)	156	(Lund et al., 2010)	South Africa	Mental Health Care Act of 2002: Sets in place mechanisms for decentralization of services, integration of mental health into general health care and the development of community-based care
Implementation Leader Survey	277	(McGinty et al., 2018)	United States	Affordable Care Act: Enables creation of state-level Medicaid health home programs
Illness Management and Recovery Survey	37	(McGuire et al., 2013)	United States	Uniform Mental Health Services Handbook: Mandates the inclusion of “illness management and recovery groups” within Psychosocial Rehabilitation and Recovery Centers
Suicide Prevention Monitor Survey	10	(Mokkenstorm et al., 2018)	Netherlands	National Suicide Prevention Strategy 2014–2017 from the Dutch Ministry of Health, Welfare and Sports: Includes 17 goals across 4 domains—healthcare, education, media and the social-economic sector
Social Network Analysis Questionnaire	N/A	(Park et al., 2014)	Canada	Changing Directions, Changing Lives: The Mental Health Strategy for Canada: Promotes recovery-oriented services in mental health settings
School Mental Health Policy Survey	26	(Patalay et al., 2016)	Multiple: 10 European countries	European Joint Action on Mental Health and Wellbeing (2013–2016): Provides a framework for action in mental health policy and addresses the promotion of mental health in schools
Veteran's Health Administration All Employee Survey	14	(Smith et al., 2017)	United States	Veteran's Health Administration policy, Re-Engage Program: Requires all VA sites to implement a new brief care management program for veterans with serious mental illness
Diagnosis Treatment Combinations Survey	23	(Tummers, 2011)	Netherlands	Health Insurance Law, Diagnosis Related Groups Policy: Stipulates a standard cost for each mental health disorder, thus determining the level of financial exchange for mental health care provision
Diagnosis Treatment Combinations Survey II	23	(Tummers, 2012)	Netherlands
Diagnosis Related Groups Survey	21	Tummers & Van de Walle, 2012)	Netherlands
Diagnosis Related Groups Survey II	15	(Tummers & Bekkers, 2014	Netherlands

The studies were published between 2009 and 2020 and were conducted primarily in the United States (n = 8), the Netherlands (n = 5), multiple European countries (n = 3), England (n = 3), and South Africa (n = 2). A single study was also conducted in the following countries: Scotland, Ireland, Ghana, and Canada. A variety of mental health policies were represented. Some policies impacted mental health support services for children and adolescents, while others addressed health insurance or the associated costs of implementation for both consumers and organizations/systems. While some measurement tools were developed specifically to evaluate the implementation of mental health policies (e.g., the World Health Organization's Mental Health Plan/Policy Checklists), others were used to evaluate constructs that impact implementation (e.g., Evidence-Based Practice Attitude Scale, Organizational Readiness to Change Assessment).

While the application of theories, models, and frameworks varied across studies, implementation frameworks such as the Exploration, Preparation, Implementation, Sustainment framework (Moullin et al., 2015), and the CFIR were used in multiple studies. Additional study characteristics, including study design, quantitative methods, setting, and policy level, are highlighted in Table 4, while Table 5 provides details about the measures used in included studies.

Table 4.

Characteristics of included studies with quantitative measures (n = 25).

Study characteristic	n
Study design
Cross-sectional	18
Protocol	3
Case study	2
Longitudinal	2
Quantitative method used
Survey	24
Secondary data	1
Use of theories, models, or frameworks reported
No	14
Yes	11
Setting
Healthcare	14
Health insurance	4
Broad/Multiple	4
Schools	3
Policy level
National	20
State	1
Local	4

Table 5.

Characteristics of measures utilized in studies of mental health policy implementation (n = 34 unique measures in 25 studies).

Tool name	DevelopmentAuthor, year	Empirical useAuthor, year	Implementation determinants assessed	Implementation outcomes assessed	Pragmatic PAPERS score	Psychometric PAPERS score	Psychometric properties assessed
World Health Organization's Mental Health Policy Checklist	(World Health Organization, 2007)	Awenva et al., 2010	RI: Non-training resources; RI: Training; Actor relationships/networks; Visibility of policy role and actors	Feasibility; Fidelity/compliance; Cost; Health equity	11	−1	Norms
World Health Organization's Mental Health Plan Checklist	(World Health Organization, 2007)	Awenva et al., 2010	Readiness to implement; RI: Non-training resources; RI: Training; Visibility of policy role and actors	Fidelity/compliance; Cost; Health equity	10	−1	Norms
Organizational Social Context Measurement System	(Glisson et al., 2008)	Beidas et al., 2013	Organizational culture and climate; RI: Communication of policy;	N/A	0	6	Internal consistency; Structural validity; Norms
Evidence-Based Practice Attitude Scale	(Aarons, 2004)	Beidas et al., 2013	RI: Training	Acceptability; Adoption; Appropriateness	13	8	Internal consistency; Structural validity; Norms
Infrastructure, Policies and Practices in Child and Adolescents’ Mental Health Questionnaire	(Braddick et al., 2009)	Bielsa et al., 2010	Readiness to implement; RI: Communication of policy; RI: Non-training resources; RI: Training; Actor relationships/networks; Target population characteristics	Fidelity/compliance; Penetration; Health equity	9	2	Norms
Higher Education Institution Survey	(Callaghan et al., 2012)	Callaghan et al., 2012	IC: Relative priority; Actor relationships/networks	Penetration	12	−1	Norms
Mental Health Trust Survey	(Callaghan et al., 2012)	Callaghan et al., 2012	Champions; IC: Relative priority; RI: Training; Actor relationships/networks	Penetration; Health equity	10	−1	Norms
Mental Health Nursing Survey	(Cusack et al., 2017)	Cusack et al., 2017	RI: Training; Actor relationships/networks	Penetration	9	4	Norms
Health Policy Questionnaire	Dlouhy, 2014	Dlouhy, 2014	Readiness to implement; RI: Policy awareness/knowledge; RI: Non-training resources; Structure of organization	Cost	7	−1	Norms
Health Education Experts Survey	(Eisman et al., 2020)	Eisman et al., 2020	Organizational culture and climate; Policy implementation climate; IC: Relative priority; RI: Non-training resources; RI: Training	Appropriateness; Fidelity/compliance	12	−1	Norms
Primary Care Integration Survey	(Gerber, 2018)	Gerber, 2018	RI: Policy awareness/knowledge; RI: Leadership for implementation; RI: Training	Acceptability	12	1	Norms
School Health Policies and Practices Survey (SHPPS)	(Centers for Disease Control and Prevention, 2019a)	Guerra et al., 2019	Readiness to implement; RI: Communication of policy; RI: Non-training resources; RI: Training; Actor relationships/networks	Acceptability; Fidelity/compliance	14	−1	Norms
School Health Profiles Survey for Principals	(Centers for Disease Control and Prevention, 2019b)	Guerra et al., 2019	Readiness to implement; RI: Communication of policy; RI: Training	Fidelity/compliance; Penetration	9	4	Norms
School Health Profiles Survey for Teachers	(Centers for Disease Control and Prevention, 2019b)	Guerra et al., 2019	Readiness to implement; RI: Training; Actor relationships/networks	N/A	7	4	Norms
Cognitive Behavioral Therapy Survey	(Kenicer et al., 2012)	Kenicer et al., 2012	Readiness to implement; RI: Non-training resources	N/A	13	−1	Norms
Perceived Characteristics of Intervention Scale	(Cook et al., 2014)	Lau & Brookman-Frazee, 2016	Adaptability; Complexity	Appropriateness; Feasibility	14	−1	Internal consistency; Structural validity; Norms
Knowledge and Confidence Survey/Therapists’ Perceptions of EBP Self-Efficacy	(Lau & Brookman-Frazee, 2016)	Lau & Brookman-Frazee, 2016	RI: Training	N/A	13	5	Internal consistency; Norms
Practice-specific Implementation Support	(Lau & Brookman-Frazee, 2016)	Lau & Brookman-Frazee, 2016	RI: Non-training resources; RI: Training; Actor relationships/networks	N/A	11	0	Not reported
Organizational Climate Measure	(Patterson et al., 2005)	Lau & Brookman-Frazee, 2016	Organizational culture and climate; IC: Goals and feedback; RI: Training	N/A	12	4	Internal consistency; Norms
Organizational Readiness to Change	(Lehman et al., 2002)	Lau & Brookman-Frazee, 2016	Organizational culture and climate; IC: Goals and feedback; RI: Non-training resources; RI: Training	N/A	16	3	Internal consistency; Norms
Adaptations of Evidence-based Practices	(Stirman et al., 2013)	Lau & Brookman-Frazee, 2016	Adaptability	Fidelity/compliance	12	0	Not reported
Evidence-Based Practice Attitude Scale	(Aarons, 2004)	Lau & Brookman-Frazee, 2016	RI: Training	Acceptability; Adoption; Appropriateness	13	8	Internal consistency; Structural validity; Norms
Organizational Social Context Measurement System	(Glisson et al., 2008)	Lau, 2016	Organizational culture and climate; RI: Communication of policy	N/A	0	6	Internal consistency; Structural validity; Norms
Crisis Resolution Team Survey	(Lloyd-Evans et al., 2018a)	Lloyd-Evans, 2018a	Structure of organization; Actor relationships/networks	Penetration; Cost	10	2	Norms
Crisis Resolution Team Practice Survey	(Lloyd-Evans et al., 2018b)	Lloyd-Evans, 2018b	Readiness to implement; RI: Non-training resources; RI: Training; Structure of organization; Actor relationships/networks; Target population characteristics	Fidelity/compliance; Penetration	10	2	Norms
World Health Organization's Assessment Instrument for Mental Health Systems Version 2.2	(Saxena et al., 2007)	Lund et al, 2010	Readiness to implement; RI: Non-training resources; RI: Training; Structure of organization; Actor relationships/networks; Target population characteristics	Cost; Health equity	13	−1	Norms
Implementation Leader Survey	(McGinty et al., 2018)	McGinty et al., 2018	Organizational culture and climate; Policy implementation climate; RI: Policy awareness/knowledge; RI: Leadership for implementation; RI: Non-training resources; RI: Training; Structure of organization; Actor relationships/networks; Target population characteristics	Health equity	8	−1	Norms
Illness Management and Recovery Survey	(McGuire et al., 2013)	McGuire et al., 2013	IC: Goals and feedback; RI: Policy awareness/knowledge; RI: Training	Adoption; Fidelity/compliance; Penetration	12	2	Norms
Suicide Prevention Monitor Survey	(Mokkenstorm et al., 2018)	Mokkenstorm et al., 2018	Readiness to implement; RI: Policy awareness/knowledge; RI: Training; Actor relationships/networks	N/A	13	−1	Norms
Social Network Analysis Questionnaire	(Park et al., 2014)	Park et al., 2014	Opinion leaders	N/A	0	0	Not applicable
School Mental Health Policy Survey	Patalay et al., 2016	Patalay et al., 2016	Organizational culture and climate; Policy implementation climate; IC: Relative priority; Readiness to implement; RI: Policy awareness/knowledge; Structure of organization; Actor relationships/networks	N/A	11	4	Norms
Veterans Health Administration All Employee Survey	(U.S. Department of Veteran Affairs, 2021)	Smith et al., 2017	Organizational culture and climate	N/A	12	4	Internal consistency; Structural validity; Norms
Diagnosis Treatment Combinations Survey	(Tummers, 2009)	Tummers, 2011	Adaptability; Complexity; Organizational culture and climate	Acceptability; Appropriateness; Feasibility	12	9	Internal consistency; Convergent validity; Discriminant validity; Structural validity; Norms
Diagnosis Treatment Combinations Survey II	(Tummers, 2012)	Tummers, 2012	Adaptability; Complexity; Organizational culture and climate; Readiness to implement; RI: Leadership for implementation	Acceptability; Appropriateness; Feasibility	12	9	Internal consistency; Convergent validity; Discriminant validity; Structural validity; Norms
Diagnosis Related Groups Survey	(Tummers & Van de Walle, 2012)	Tummers, 2012	Champions; Organizational culture and climate; Policy implementation climate; IC: Relative priority; Readiness to implement; RI: Leadership for implementation	Acceptability; Adoption; Appropriateness; Feasibility	12	4	Internal consistency; Norms
Diagnosis Related Groups Survey II	(Tummers & Bekkers, 2014)	Tummers & Bekkers, 2014	Adaptability; Complexity; Champions; Organizational culture and climate; Policy implementation climate; IC: Relative priority; Readiness to implement; RI: Leadership for implementation	Acceptability; Adoption; Appropriateness; Feasibility	13	8	Internal consistency; Structural validity; Norms

Note. IC = implementation climate; RI = readiness to implement. PAPERS pragmatic score range: −5 to 20; PAPERS psychometric score range: −9 to 36.

Measure Development and Testing

Of the 34 unique measurement tools, separate measure development articles were available for 10. The remaining quantitative instruments consisted of a mix of “home-grown” questionnaires, which were developed for use within a single study, and multi-use tools (e.g., The World Health Organization's Mental Health Plan Checklist). Overall, information regarding measure development and testing was limited. Less than half of the measures (n = 17, 45%) defined the constructs they assessed (i.e., provided definitions of constructs within the empirical use article, development article or other supporting documents, or the instrument itself). Only five measures (13%) reported that items were generated by experts in the field. Finally, there was limited information regarding pilot testing of measures. Six measures (16%) reported pilot testing with a representative sample. Only three measures (8%) reported validity or reliability tests based on pilot testing, and only two (5%) reported that the measure was refined based on pilot testing.

Implementation Determinants and Outcomes

As outlined in Table 1, only five measures (14%) assessed policy characteristics, such as the complexity or adaptability of a policy. Inner setting characteristics were far more common, with 32 measures (91%) assessing at least one organization-level construct. Outer setting characteristics, on the other hand, were assessed in less than half of the included measures (n = 15, 43%).

The most assessed implementation determinant was readiness for implementation (n = 27, 77%), which included the following subcomponents: communication efforts related to the policy, policy awareness and knowledge, leadership support for implementation, training, and non-training resources (see Table 1 for definitions). Each of these subcomponents were assessed in multiple measures, with particular emphasis on training (n = 20, 57%) related to policy implementation and non-training resources (n = 12, 34%). Other commonly measured determinants included actor relationships/networks (n = 15, 43%), organizational culture and climate (n = 11, 31%), and the policy implementation climate (n = 10, 29%).

The most commonly assessed implementation outcomes from the 34 measures were fidelity/compliance to the policy (n = 9, 26%) and penetration of the policy within the designated setting (n = 8, 23%) (see Table 2 for definitions). The acceptability and appropriateness of the policy were each assessed by seven measures (20%). The feasibility of implementing a policy was assessed by six measures (17%), while the cost of implementation was assessed by five measures (14%). Only four measures assessed adoption (11%), and no measures addressed sustainability.

Health Equity

Finally, six measures (17%) included at least one item assessing health equity or SDOH. Some measures inquired about the integration or coordination of a mental health policy with existing policy efforts to target SDOH, such as housing, poverty, or antidiscrimination. Other measures assessed factors affecting access to mental health services, such as geographic proximity to providers or availability of transportation.

Psychometric and Pragmatic Evidence Rating Scale (PAPERS) Criteria

As described in Table 5, higher scores on the PAPERS criteria are preferable, indicating that the measures possess better psychometric properties (psychometric score) or are easier to use (pragmatic score). For the 34 unique measures, limited information was available regarding psychometric properties. Possible psychometric scores range from −9 to 36 across nine categories of reliability and validity. PAPERS psychometric scores for included measures ranged from −1 to 17, with a median score of 5. The majority of studies reported only norms, an indication of sample size and distribution (n = 33, 97%), with a median PAPERS score of 3 (good). Eleven studies reported internal consistency, with Cronbach's alpha scores ranging from 0.49 to 0.97 (median PAPERS score = 3, good). Seven measures reported structural validity with a median PAPERS score of 3 (good). Lastly, convergent and discriminant validity were reported for two measures, with median PAPERS scores of 4 (excellent) and 2 (adequate), respectively. No information was found regarding known-groups construct validity, predictive validity, concurrent criterion validity, or responsiveness within the 34 included measures.

All available measures but one—the Organizational Social Context Measurement System—were free and accessible to the public either through a web search or contacting the study team (median score = 4). While the measures varied considerably in length, ranging from 2 to 277 items, 11 measures contained 11 to 50 items (median score = 3). Most measures were accessible at an 8th–12th grade reading level (median score = 3). Instructions regarding scoring and interpreting the measures were lacking overall (median score = 2); minimal information was included to aid in interpreting score ranges, identifying clear cut-off scores, and handling missing data. Similarly, information on training requirements or availability of self-training manuals for those administering the measure was often not reported (median score = 0, Not reported). Overall, the included measures received a median total score of 12, out of a possible 20, indicating that certain pragmatic criteria were underreported in this sample.

Discussion

This study examined quantitative measurement tools used to evaluate the implementation of mental health policies affecting mental health education and services and assessed measure development, testing (i.e., the extent to which measures are psychometrically tested), and application (i.e., the pragmatic nature of measures). Though there are best practices when developing and testing measures (Boateng et al., 2018), many measures lacked information regarding gold-standard criteria for measurement development and testing. Additionally, there was a number of “home-grown,” single-use measures identified in this sample, which has been reported in previous measures reviews (Dorsey et al., 2021; Mettert et al., 2020; Proctor et al., 2011; Stanick et al., 2021). Unfortunately, the measure development process and psychometric validation of such surveys were rarely reported, presenting a measurement challenge within implementation science and an area for further improvement (Lewis, Weiner, et al., 2015; Martinez et al., 2014; Proctor et al., 2011).

Our findings also highlight the need for a more critical examination of policy implementation determinants and outcomes. Within the included measures, some constructs—such as readiness for implementation, acceptability, or fidelity/compliance—were represented repeatedly. Others were underrepresented (e.g., political will for policy implementation) or absent (e.g., sustainability) from measures, despite the known significance within implementation science (Glasgow & Chambers, 2012). Outer setting implementation determinants were assessed less frequently than inner setting determinants, which has been reported in previous studies (Chaudoir et al., 2013; Chor et al., 2015; Clinton-McHarg et al., 2016; McHugh et al., 2020; Powell et al., 2021; Weiner et al., 2020). The limited attention to outer setting implementation determinants is problematic, as factors like political will can impact mental health outcomes through policy development and resource allocation (Corrigan & Watson, 2003; Shera & Ramon, 2013; Weinberg et al., 2012). Additionally, the overemphasis on compliance as a means of studying implementation has been a critique of policy studies in the past (Thomann & Zhelyazkova, 2017).

The imbalance of constructs within quantitative measures may stem from the length of time that these constructs have received attention. Historically, a large emphasis has been placed on acceptability, while sustainability may appear less often due to its relative “newness” within the implementation literature (Lewis, Fischer, et al., 2015). The absence of sustainability in these studies is perhaps unsurprising, as researchers report similar findings with other assessment models, such as the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) Framework. For example, a 2013 review assessing the use and reporting of RE-AIM in 71 articles found that, of all five RE-AIM dimensions, maintenance (sustainment) was the least frequently reported—at both the individual-level and setting-level (Gaglio et al., 2013). Many of these constructs are theoretically grounded, though, and have a significant impact on policy implementation. Assessing these understudied constructs in future studies may provide researchers with a more complete picture of the processes, contexts, and outcomes of implementation.

Notably, health equity was assessed in few quantitative measures of mental health policy implementation, which has been reported in previous studies of policy measures (Allen et al., 2020, 2021). Moving forward, it will be vital to engage policymakers in this pursuit of equity, as they have considerable potential to impact health, such as allocating resources to and developing appropriate policies (Douglas et al., 2019). Connecting with community members and other stakeholders to identify and address health equity issues during implementation might also prove useful (Douglas et al., 2019). Additional work is needed to bridge health equity research and implementation science, both theoretically (e.g., within frameworks) and practically (e.g., pragmatic equity measures) (Brownson et al., 2021; Emmons & Chambers, 2021; Pollack Porter et al., 2018; Woodward et al., 2021). This presents many opportunities for further evaluation of the presence of equity within policy implementation.

Psychometric and pragmatic qualities of measures were also assessed in this study. With the exception of internal consistency and sample norms, psychometric properties were frequently unreported, as found in previous implementation measures reviews (Allen et al., 2020; Clinton-McHarg et al., 2016; McHugh et al., 2020; Powell et al., 2021; Weiner et al., 2020). For example, information about structural validity was available for seven measures, and convergent and discriminant validity were reported for two measures. However, none of the measures in this sample contained information about known-groups validity, predictive validity, concurrent validity, or responsiveness. These findings suggest that there is significant room for improvement in the development or refining of existing measures to make them more psychometrically sound.

Overall, the observed pragmatic qualities aligned with previous findings (Allen et al., 2020; McLoughlin et al., 2021; Powell et al., 2021; Weiner et al., 2020). Most included measures were freely available (i.e., accessible through supplemental materials with the article or by contacting authors directly), though one was proprietary. Additionally, though some measures were quite lengthy, most were relatively brief (fewer than 50 items). There is substantial need for pragmatic measures to assess implementation (Glasgow & Riley, 2013; Powell et al., 2017; Stanick et al., 2018, 2019). The findings of this work indicate that while practical, accessible measures used to evaluate policy implementation are being developed and utilized, there is still considerable room for improving the availability of brief, user-friendly measures of policy implementation. The development and use of such measures would help organizations to evaluate implementation without placing undue burden on practitioners.

Policy implementation research remains an understudied—but growing—area of implementation science (Emmons & Chambers, 2021; Hoagwood et al., 2020; McGinty et al., 2022; Oh et al., 2021; Purtle et al., 2017, 2016). Policy implementation is complex and can vary significantly across contexts (Berenson et al., 2017), making it crucial to evaluate implementation determinants, processes, and outcomes. Policy implementation also spans multiple levels of government and geopolitical units of epidemiological analyses, furthering the complexity in studying policy implementation (Schnake-Mahl et al., 2022). There are particular challenges in measuring policy implementation across levels of government. For example, policy surveillance systems are often more developed at the national level compared with the local level. The nexus between political science and implementation science offers benefits for measurement efforts. In particular, political science has a rich history of measuring the outer context of policy implementation (the policy and political environments) (Emmons et al., 2021). Valid, reliable, and pragmatic measures are needed to evaluate policy implementation and close the policy-to-practice gap (Martin et al., 2019; Oh et al., 2021). As a subset of the need for valid and reliable measures, there is growing attention on the need for metrics to track indicators of equity (e.g., SDOH, structural racism) (Brownson et al., 2022). Ultimately, this work contributes to the nascent study of policy-focused implementation science by providing an overview of existing measures used to assess mental health policy implementation, as well as guidance and recommendations for future measure development and refinement.

Limitations

While this study fills gaps in the literature, there are several limitations. First, there were likely relevant measures not captured in this study based on our search strategy and inclusion criteria. Our search strategy included only studies and accompanying measures that were published in English language peer-reviewed journals, so measures found exclusively in grey literature, books, or theses were not included. Second, because we used verbatim instrument items in data extraction (e.g., implementation determinants and outcomes), studies were excluded if we could not obtain the measures. Though we reviewed articles and supplemental materials, hand searched peer-reviewed and grey literature, and contacted authors directly, we ultimately excluded 18 studies because measures were inaccessible, introducing the possibility of sampling bias. The missing tools were commonly from older articles (1995–2010) that did not report rigorous measure development and psychometric testing. Third, though we examined the prevalence of implementation determinants and outcomes in the included measures, it is possible that these constructs were assessed qualitatively in the included studies. Finally, based on the study's definition of “Big P” policies, the included studies encompassed both mandated changes in practices or services, as well as the voluntary adoption and implementation of practices or services based on recommendations or guidelines from governmental entities. Despite these limitations, this review contributes to the policy implementation literature by assessing the state of measure development and quality in mental health policy studies.

Future Directions

Based on the findings of this study, there are several areas for further study. First, there is significant room for improvement regarding measure development or refinement of existing measures to make them more pragmatic and psychometrically sound, as well increasing reporting of pragmatic and psychometric properties. Second, there is need for greater transparency and availability of measurement instruments; we recommend that authors make measures available as supplemental materials to publications or add these to web-based measures repositories. Third, this study echoes previous calls for a greater focus on health equity as both a determinant and outcome of policy implementation (Brownson et al., 2021; Emmons & Chambers, 2021; Woodward et al., 2021). Finally, while this project focused on the implementation of mental health policies once adopted, additional work is needed to explore factors involved with policy enactment and the decision-making processes during adoption and implementation.

Supplemental Material

sj-docx-1-irp-10.1177_26334895221141116 - Supplemental material for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Supplemental material, sj-docx-1-irp-10.1177_26334895221141116 for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review by Meagan Pilar, Eliot Jost, Callie Walsh-Bailey, Byron J. Powell, Stephanie Mazzucca, Amy Eyler, Jonathan Purtle, Peg Allen and Ross C. Brownson in Implementation Research and Practice

Supplemental Material

sj-docx-2-irp-10.1177_26334895221141116 - Supplemental material for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Supplemental material, sj-docx-2-irp-10.1177_26334895221141116 for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review by Meagan Pilar, Eliot Jost, Callie Walsh-Bailey, Byron J. Powell, Stephanie Mazzucca, Amy Eyler, Jonathan Purtle, Peg Allen and Ross C. Brownson in Implementation Research and Practice

Supplemental Material

sj-docx-3-irp-10.1177_26334895221141116 - Supplemental material for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Supplemental material, sj-docx-3-irp-10.1177_26334895221141116 for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review by Meagan Pilar, Eliot Jost, Callie Walsh-Bailey, Byron J. Powell, Stephanie Mazzucca, Amy Eyler, Jonathan Purtle, Peg Allen and Ross C. Brownson in Implementation Research and Practice

Supplemental Material

sj-docx-4-irp-10.1177_26334895221141116 - Supplemental material for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Supplemental material, sj-docx-4-irp-10.1177_26334895221141116 for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review by Meagan Pilar, Eliot Jost, Callie Walsh-Bailey, Byron J. Powell, Stephanie Mazzucca, Amy Eyler, Jonathan Purtle, Peg Allen and Ross C. Brownson in Implementation Research and Practice

Supplemental Material

sj-docx-5-irp-10.1177_26334895221141116 - Supplemental material for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review

Supplemental material, sj-docx-5-irp-10.1177_26334895221141116 for Quantitative measures used in empirical evaluations of mental health policy implementation: A systematic review by Meagan Pilar, Eliot Jost, Callie Walsh-Bailey, Byron J. Powell, Stephanie Mazzucca, Amy Eyler, Jonathan Purtle, Peg Allen and Ross C. Brownson in Implementation Research and Practice

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported in part by the National Institute of Diabetes and Digestive and Kidney Diseases (number P30DK092950), the National Institute of Allergy and Infectious Diseases (number K24AI134413), the National Cancer Institute (number P50CA244431), the National Institute of Mental Health (number K01MH113806; R21MH125261; P50MH11366), the National Center for Advancing Translational Sciences (number UL1 TR002345), the Centers for Disease Control and Prevention (number U48DP006395), and the Foundation for Barnes-Jewish Hospital. The findings and conclusions in this paper are those of the authors and do not necessarily represent the official positions of the National Institutes of Health or the Centers for Disease Control and Prevention.

ORCID iDs

Meagan Pilar

Callie Walsh-Bailey

Byron J. Powell

Supplemental Material

Supplemental material for this article is available online.

References

Aarons

G. A.

(2004). Mental health provider attitudes toward adoption of evidence-based practice: The Evidence-Based Practice Attitude Scale (EBPAS). Mental Health Services Research, 6(2), 61–74. https://doi.org/10.1023/b:mhsr.0000024351.12294.65

Alegría

NeMoyer

Falgàs Bagué

Wang

Alvarez

(2018). Social determinants of mental health: Where we are and where we need to go. Current Psychiatry Reports, 20(11), 95–95. https://doi.org/10.1007/s11920-018-0969-9

Alegría

Pérez

D. J.

Williams

(2003). The role of public policies in reducing mental health status disparities for people of color. Health Affairs, 22(5), 51–64. https://doi.org/10.1377/hlthaff.22.5.51

Allen

Balfour

Bell

Marmot

(2014). Social determinants of mental health. International Review of Psychiatry, 26(4), 392–407. https://doi.org/10.3109/09540261.2014.928270

Allen

Pilar

Walsh-Bailey

Hooley

Mazzucca

Lewis

C. C.

Mettert

K. D.

Dorsey

C. N.

Purtle

Kepper

M. M.

Baumann

A. A.

Brownson

R. C.

(2020). Quantitative measures of health policy implementation determinants and outcomes: A systematic review [journal article]. Implementation Science, 15(1), 1–17. https://doi.org/10.1186/s13012-020-01007-w

Allen

Walsh-Bailey

Purtle

Pilar

Hooley

Brownson

(2021). S-66 Implications for policy implementation measurement from a systematic review. 13th Annual Conference on the Science of Dissemination and Implementation.

Auerbach

Miller

B. F.

(2020). COVID-19 Exposes the cracks in our already fragile mental health system. American Journal of Public Health, 110(7), 969–970. https://doi.org/10.2105/ajph.2020.305699

Awenva

A. D.

Read

U. M.

Ofori-Attah

A. L.

Doku

V. C. K.

Akpalu

Osei

A. O.

Flisher

A. J

. (2010). From mental health policy development in Ghana to implementation: What are the barriers? African Journal of Psychiatry, 13(3), 184–191. https://pubmed.ncbi.nlm.nih.gov/20957317/

Baumann

A. A.

Long

P. D.

(2021). Equity in implementation science is long overdue. Stanford Social Innovation Review, 19(3), A15–A17. https://doi.org/10.48558/GG1H-A223

10.

Beidas

R. S.

Aarons

Barg

Evans

Hadley

Hoagwood

Marcus

Schoenwald

Walsh

Mandell

D. S.

(2013). Policy to implementation: Evidence-based practice in community mental health – study protocol. Implementation Science, 8(1), 1–9. https://doi.org/10.1186/1748-5908-8-38

11.

Berenson

Lynch

Pagán

J. A.

(2017). Identifying policy levers and opportunities for action across states to achieve health equity. Health Affairs, 36(6), 1048–1056. https://doi.org/10.1377/hlthaff.2017.0004

12.

Bielsa

V. C.

Braddick

Jané-Llopis

Jenkins

Puras

(2010). Child and adolescent mental health policies, programmes and infrastructures across Europe. International Journal of Mental Health Promotion, 12(4), 10–26. https://doi.org/10.1080/14623730.2010.9721822 (The CAMHEE (Child and Adolescent Mental Health in the Enlarged European Union: Development of effective policies and services) project)

13.

Boateng

G. O.

Neilands

T. B.

Frongillo

E. A.

Melgar-Quiñonez

H. R.

Young

S. L.

(2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer [Review]. Frontiers In Public Health, 6, (149), https://doi.org/10.3389/fpubh.2018.00149

14.

Braddick

Carral

Jenkins

Jané-Llopis

(2009). Child and Adolescent Mental Health in Europe: Infrastructures, Policy and Programmes .

15.

Brownson

R. C.

Kumanyika

S. K.

Kreuter

M. W.

Haire-Joshu

(2021). Implementation science should give higher priority to health equity. Implementation Science, 16(1), 28. https://doi.org/10.1186/s13012-021-01097-0

16.

Brownson

R. C.

Shelton

R. C.

Geng

E. H.

Glasgow

R. E.

(2022). Revisiting concepts of evidence in implementation science. Implementation Science, 17(1), 26. https://doi.org/10.1186/s13012-022-01201-y

17.

Bruns

E. J.

Parker

E. M.

Hensley

Pullmann

M. D.

Benjamin

P. H.

Lyon

A. R.

Hoagwood

K. E.

(2019). The role of the outer setting in implementation: Associations between state demographic, fiscal, and policy factors and use of evidence-based treatments in mental healthcare. Implementation Science, 14(1), 96. https://doi.org/10.1186/s13012-019-0944-9

18.

Bullock

H. L.

Lavis

J. N.

Wilson

M. G.

Mulvale

Miatello

(2021). Understanding the implementation of evidence-informed policies and practices from a policy perspective: A critical interpretive synthesis. Implementation Science, 16(1), 18. https://doi.org/10.1186/s13012-021-01082-7

19.

Callaghan

Repper

Clifton

Stacey

Carter

(2012). Evaluation of the chief nursing officer's review of mental health nursing in England: Findings from case studies in mental health trusts. Journal of Psychiatric and Mental Health Nursing, 19(5), 455–465. https://doi.org/10.1111/j.1365-2850.2011.01841.x

20.

Castillo

E. G.

Chung

Bromley

Kataoka

S. H.

Braslow

J. T.

Essock

S. M.

Young

A. S.

Greenberg

J. M.

Miranda

Dixon

L. B.

Wells

K. B.

(2018). Community, public policy, and recovery from mental illness: Emerging research and initiatives. Harvard Review of Psychiatry, 26(2), 70–81. https://doi.org/10.1097/HRP.0000000000000178

21.

Centers for Disease Control and Prevention. (2019a). School Health Policies and Practices Study (SHPPS). U.S. Department of Health & Human Services. https://www.cdc.gov/healthyyouth/data/shpps/index.htm

22.

Centers for Disease Control and Prevention. (2019b). School Health Profiles. U.S. Department of Health & Human Services. https://www.cdc.gov/healthyyouth/data/profiles/index.htm

23.

Chaudoir

S. R.

Dugan

A. G.

Barr

C. H. I.

(2013). Measuring factors affecting implementation of health innovations: A systematic review of structural, organizational, provider, patient, and innovation level measures. Implementation Science, 8(1), 1–20. https://doi.org/10.1186/1748-5908-8-22

24.

Chinman

Woodward

E. N.

Curran

G. M.

Hausmann

L. R. M.

(2017). Harnessing implementation science to increase the impact of health equity research. Medical Care, 55(Suppl 9 2), S16–S23. https://doi.org/10.1097/MLR.0000000000000769

25.

Chisholm

Flisher

A. J.

Lund

Patel

Saxena

Thornicroft

Tomlinson

. (2007). Scale up services for mental disorders: A call for action. Lancet, 370North American Edition(9594), 1241–1252. https://doi.org/10.1016/S0140-6736(07)61242-2

26.

Chor

K. H. B.

Wisdom

J. P.

Olin

S.-C. S.

Hoagwood

K. E.

Horwitz

S. M.

(2015). Measures for predictors of innovation adoption. Administration and Policy in Mental Health and Mental Health Services Research, 42(5), 545–573. https://doi.org/10.1007/s10488-014-0551-7

27.

Clinton-McHarg

Yoong

S. L.

Tzelepis

Regan

Fielding

Skelton

Kingsland

Ooi

J. Y.

Wolfenden

(2016). Psychometric properties of implementation measures for public health and community settings and mapping of constructs against the consolidated framework for implementation research: A systematic review. Implementation Science: IS, 11(1), 148–148. https://doi.org/10.1186/s13012-016-0512-5

28.

Compton

M. T.

Shim

R. S.

(eds.). (2015). The social determinants of mental health. American Psychiatric Publishing, Inc.

29.

Conley

D. L.

(2021). The impact of structural stigma and other factors on state mental health legislative outcomes during the Trump administration. Stigma and Health, 6(4), 476–486. https://doi.org/10.1037/sah0000331

30.

Cook

J. M.

Thompson

Schnurr

P. P.

(2014). Perceived characteristics of intervention scale: Development and psychometric properties. Assessment, 22(6), 704–714. https://doi.org/10.1177/1073191114561254

31.

Corrigan

P. W.

Watson

A. C.

(2003). Factors that explain how policy makers distribute resources to mental health services. Psychiatric Services, 54(4), 501–507. https://doi.org/10.1176/appi.ps.54.4.501

32.

Covidence Systematic Review Software. (2021). Veritas Health Innovation. In. Melbourne, Australia: Available at www.covidence.org

33.

Cusack

Killoury

Nugent

L. E.

(2017). The professional psychiatric/mental health nurse: Skills, competencies and supports required to adopt recovery-orientated policy in practice. Journal of Psychiatric & Mental Health Nursing, 24(2/3), 93–104. https://doi.org/10.1111/jpm.12347

34.

Czeisler

M. E.

Lane

R. I.

Petrosky

Wiley

J. F.

Christensen

Njai

Weaver

M. D.

Robbins

Facer-Childs

E. R.

Barger

L. K.

Czeisler

C. A.

Howard

M. E.

Rajaratnam

S. M. W.

(2020). Mental Health, Substance Use, and Suicidal Ideation During the COVID-19 Pandemic — United States, June 24–30, 2020 (Morbidity and Mortality Weekly Report, Issue. http://dx.https://doi.org/10.15585/mmwr.mm6932a1

35.

Damschroder

L. J.

Aron

D. C.

Keith

R. E.

Kirsh

S. R.

Alexander

J. A.

Lowery

J. C.

(2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science: IS, 4, 50–50. https://doi.org/10.1186/1748-5908-4-50

36.

Dawes

D. E.

(2016). 150 Years of Obamacare. Johns Hopkins University Press.

37.

DeGroff

Cargo

(2009). Policy implementation: Implications for evaluation. New Directions for Evaluation, 2009(124), 47–60. https://doi.org/10.1002/ev.313

38.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2009). Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method. John Wiley & Sons.

39.

Dlouhy

(2014). Mental health policy in Eastern Europe: A comparative analysis of seven mental health systems. BMC Health Services Research, 14, 42. https://doi.org/10.1186/1472-6963-14-42

40.

Dorsey

C. N.

Mettert

K. D.

Puspitasari

A. J.

Damschroder

L. J.

Lewis

C. C.

(2021). A systematic review of measures of implementation players and processes: Summarizing the dearth of psychometric evidence. Implementation Research and Practice, 2, 26334895211002474. https://doi.org/10.1177/26334895211002474

41.

Douglas

M. D.

Josiah Willock

Respress

Rollins

Tabor

Heiman

H. J.

Hopkins

Dawes

D. E.

Holden

K. B.

(2019). Applying a health equity Lens to evaluate and inform policy. Ethnicity & Disease, 29(Suppl 2), 329–342. https://doi.org/10.18865/ed.29.S2.329

42.

Eaton

McCay

Semrau

Chatterjee

Baingana

Araya

Ntulo

Thornicroft

Saxena

(2011). Scale up of services for mental health in low-income and middle-income countries. Lancet, 378(9802), 1592–1603. https://doi.org/10.1016/s0140-6736(11)60891-x

43.

Eisman

A. B.

Kilbourne

A. M.

Ngo

Fridline

Zimmerman

M. A.

Greene

Cunningham

R. M.

(2020). Implementing a state-adopted high school health curriculum: A case study. Journal of School Health, 90(6), 447–456. https://doi.org/10.1111/josh.12892

44.

Emmons

K. M.

Chambers

D. A.

(2021). Policy implementation science - an unexplored strategy to address social determinants of health. Ethnicity & Disease, 31(1), 133–138. https://doi.org/10.18865/ed.31.1.133

45.

Emmons

K. M.

Chambers

Abazeed

(2021). Embracing policy implementation science to ensure translation of evidence to cancer control policy. Translational Behavioral Medicine, 11(11), 1972–1979. https://doi.org/10.1093/tbm/ibab147

46.

Emmons

K. M.

Weiner

Fernandez

M. E.

S.-P.

(2011). Systems antecedents for dissemination and implementation: A review and analysis of measures. Health Education & Behavior, 39(1), 87–105. https://doi.org/10.1177/1090198111409748

47.

Exworthy

(2008). Policy to tackle the social determinants of health: Using conceptual models to understand the policy process. Health Policy and Planning, 23(5), 318–327. https://doi.org/10.1093/heapol/czn022

48.

Eyler

Brownson

R. C.

(2016). The power of policy to improve health. In Eyler

A. A.

Chriqui

J. F.

Moreland-Russell

Brownson

R. C.

(Eds.), Prevention, Policy, and Public Health (8–9). Oxford University Press.

49.

Gaglio

Shoup

J. A.

Glasgow

R. E.

(2013). The RE-AIM framework: A systematic review of use over time. American Journal of Public Health, 103(6), e38–e46. https://doi.org/10.2105/AJPH.2013.301299

50.

Gerber

(2018). Practitioners’ experience of the integration of mental health into primary health care in the West Rand District, South Africa. Journal of Mental Health, 27(2), 135–141. https://doi.org/10.1080/09638237.2017.1340604

51.

Glasgow

R. E.

Chambers

(2012). Developing robust, sustainable, implementation systems using rigorous, rapid and relevant science. Clinical and Translational Science, 5(1), 48–55. https://doi.org/10.1111/j.1752-8062.2011.00383.x

52.

Glasgow

R. E.

Riley

W. T.

(2013). Pragmatic measures: What they are and why we need them. American Journal of Preventive Medicine, 45(2), 237–243. https://doi.org/10.1016/j.amepre.2013.03.010

53.

Glisson

Landsverk

Schoenwald

Kelleher

Hoagwood

K. E.

Mayberg

Green

(2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health, 35(1-2), 98–113. https://doi.org/10.1007/s10488-007-0148-5

54.

Goldman

H. H.

Azrin

S. T

. (2003). Public policy and evidence-based practice. The Psychiatric Clinics Of North America, 26(4), 899–917. http://libproxy.wustl.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=cmedm&AN=14711127&site=ehost-live&scope=site https://doi.org/10.1016/S0193-953X(03)00068-6

55.

Guerra

L. A.

Rajan

Roberts

K. J.

(2019). The implementation of mental health policies and practices in schools: An examination of school and state factors. Journal of School Health, 89(4), 328–338. https://doi.org/10.1111/josh.12738

56.

Hinshaw

S. P.

(2006). The mark of shame: Stigma of mental illness and an agenda for change. Oxford University Press.

57.

Hoagwood

K. E.

Purtle

Spandorfer

Peth-Pierce

Horwitz

S. M.

(2020). Aligning dissemination and implementation science with health policies to improve children's mental health. The American Psychologist, 75(8), 1130–1145. https://doi.org/10.1037/amp0000706

58.

Holmes

E. A.

O'Connor

R. C.

Perry

V. H.

Tracey

Wessely

Arseneault

Ballard

Christensen

Cohen Silver

Everall

Ford

John

Kabir

King

Madan

Michie

Przybylski

A. K.

Shafran

Sweeney

Bullmore

(2020). Multidisciplinary research priorities for the COVID-19 pandemic: A call for action for mental health science. The Lancet Psychiatry, 7(6), 547–560. https://doi.org/10.1016/s2215-0366(20)30168-1

59.

Kaplan

H. C.

Brady

P. W.

Dritz

M. C.

Hooper

D. K.

Linam

W. M.

Froehle

C. M.

Margolis

(2010). The influence of context on quality improvement success in health care: A systematic review of the literature. The Milbank Quarterly, 88(4), 500–559. https://doi.org/10.1111/j.1468-0009.2010.00611.x

60.

Kenicer

McClay

C.-A.

Williams

(2012). A national survey of health service infrastructure and policy impacts on access to computerised CBT in Scotland. BMC Medical Informatics & Decision Making, 12(1), 102–106. https://doi.org/10.1186/1472-6947-12-102

61.

Kessler

R. C.

Demler

Frank

R. G.

Olfson

Pincus

H. A.

Walters

E. E.

Wang

Wells

K. B.

Zaslavsky

A. M

. (2005). Prevalence and treatment of mental disorders, 1990 to 2003. The New England Journal Of Medicine, 352(24), 2515–2523. http://libproxy.wustl.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=cmedm&AN=15958807&site=ehost-live&scope=site https://doi.org/10.1056/NEJMsa043266

62.

Kilbourne

A. M.

Beck

Spaeth-Rublee

Ramanuj

O'Brien

R. W.

Tomoyasu

Pincus

H. A

. (2018). Measuring and improving the quality of mental health care: A global perspective. World Psychiatry : Official Journal of the World Psychiatric Association (WPA), 17(1), 30–38. https://doi.org/10.1002/wps.20482

63.

Labott

(2021). Get ready for a spike in global unrest. Foreign Policy. https://foreignpolicy.com/2021/07/22/covid-global-unrest-political-upheaval/

64.

Lau

A. S.

Brookman-Frazee

(2016). The 4KEEPS study: Identifying predictors of sustainment of multiple practices fiscally mandated in children’s mental health services. Implementation Science, 11(1), 31. https://doi.org/10.1186/s13012-016-0388-4

65.

Lehman

W. E. K.

Greener

J. M.

Simpson

D. D.

(2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22(4), 197–209. https://doi.org/10.1016/s0740-5472(02)00233-7

66.

Lewis

C. C.

Fischer

Weiner

B. J.

Stanick

Kim

Martinez

R. G.

(2015). Outcomes for implementation science: An enhanced systematic review of instruments using evidence-based rating criteria. Implementation Science, 10(1), 155. https://doi.org/10.1186/s13012-015-0342-x

67.

Lewis

C. C.

Mettert

K. D.

Dorsey

C. N.

Martinez

R. G.

Weiner

B. J.

Nolen

Stanick

Halko

Powell

B. J.

(2018). An updated protocol for a systematic review of implementation-related measures. Systematic Reviews, 7(1), 66–66. https://doi.org/10.1186/s13643-018-0728-3

68.

Lewis

C. C.

Mettert

K. D.

Stanick

C. F.

Halko

H. M.

Nolen

E. A.

Powell

B. J.

Weiner

B. J.

(2021). The psychometric and pragmatic evidence rating scale (PAPERS) for measure development and evaluation. Implementation Research and Practice, 2, 26334895211037391. https://doi.org/10.1177/26334895211037391

69.

Lewis

C. C.

Weiner

B. J.

Stanick

Fischer

S. M.

(2015). Advancing implementation science through measure development and evaluation: A study protocol. Implementation Science, 10(1), 1–10. https://doi.org/10.1186/s13012-015-0287-0

70.

Lloyd-Evans

Lamb

Barnby

Eskinazi

Turner

Johnson

(2018a). Mental health crisis resolution teams and crisis care systems in England: A national survey. BJPsych Bulletin, 42(4), 146–151. https://doi.org/10.1192/bjb.2018.19

71.

Lloyd-Evans

Paterson

Onyett

Brown

Istead

Gray

Henderson

Johnson

(2018b). National implementation of a mental health service model: A survey of crisis resolution teams in England. International Journal of Mental Health Nursing, 27(1), 214–226. https://doi.org/10.1111/inm.12311

72.

Loper

Woo

Metz

(2021). Equity is fundamental to implementation science. Stanford Social Innovation Review, 19(3), A3–A5. https://doi.org/10.48558/QNGV-KG05

73.

Lund

Kleintjes

Kakuma

Flisher

A. J.

(2010). Public sector mental health systems in South Africa: Inter-provincial comparisons and policy implications. Social Psychiatry & Psychiatric Epidemiology, 45(3), 393–404. https://doi.org/10.1007/s00127-009-0078-5

74.

Malekinejad

Horvath

Snyder

Brindis

C. D.

(2018). The discordance between evidence and health policy in the United States: The science of translational research and the critical role of diverse stakeholders. Health Research Policy And Systems, 16(1), 81–81. https://doi.org/10.1186/s12961-018-0336-7

75.

Martin

Mullan

Horton

(2019). Overcoming the research to policy gap. The Lancet Global Health, 7, S1–S2. https://doi.org/10.1016/s2214-109x(19)30082-8

76.

Martinez

R. G.

Lewis

C. C.

Weiner

B. J.

(2014). Instrumentation issues in implementation science. Implementation Science, 9(1), 118. https://doi.org/10.1186/s13012-014-0118-8

77.

McGinty

E. E.

Kennedy-Hendricks

Linden

Choksy

Stone

Daumit

G. L.

(2018). An innovative model to coordinate healthcare and social services for people with serious mental illness: A mixed-methods case study of Maryland's Medicaid health home program. General Hospital Psychiatry, 51, 54–62. https://doi.org/10.1016/j.genhosppsych.2017.12.003

78.

McGinty

E. E.

Seewald

N. J.

Bandara

Cerdá

Daumit

G. L.

Eisenberg

M. D.

Griffin

B. A.

Igusa

Jackson

J. W.

Kennedy-Hendricks

Marsteller

Miech

E. J.

Purtle

Schmid

Schuler

M. S.

Yuan

C. T.

Stuart

E. A.

(2022). Scaling Interventions to Manage Chronic Disease: Innovative Methods at the Intersection of Health Policy Research and Implementation Science. Prevention Science, https://doi.org/10.1007/s11121-022-01427-8

79.

McGuire

A. B.

White

D. A.

White

L. M.

Salyers

M. P.

(2013). Implementation of illness management and recovery in the veterans administration: An on-line survey. Psychiatric Rehabilitation Journal, 36(4), 264–271. https://doi.org/10.1037/prj0000032

80.

McHugh

Dorsey

C. N.

Mettert

Purtle

Bruns

Lewis

C. C.

(2020). Measures of outer setting constructs for implementation research: A systematic review and analysis of psychometric quality. Implementation Research and Practice, 1, 2633489520940022. https://doi.org/10.1177/2633489520940022

81.

McLoughlin

G. M.

Allen

Walsh-Bailey

Brownson

R. C.

(2021). A systematic review of school health policy measurement tools: Implementation determinants and outcomes. Implementation Science Communications, 2(1), 67. https://doi.org/10.1186/s43058-021-00169-y

82.

Mettert

Lewis

Dorsey

Halko

Weiner

(2020). Measuring implementation outcomes: An updated systematic review of measures’ psychometric properties. Implementation Research and Practice, 1, 2633489520936644. https://doi.org/10.1177/2633489520936644

83.

Metz

Woo

Loper

(2021). Equitable implementation at work. Stanford Social Innovation Review, 19(3), A29–A31. https://doi.org/10.48558/R793-6704

84.

Moher

Liberati

Tetzlaff

Altman

D. G.

(2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. BMJ, 339, b2535. https://doi.org/10.1136/bmj.b2535

85.

Mokkenstorm

Franx

Gilissen

Kerkhof

Smit

J. H.

(2018). Suicide prevention guideline implementation in specialist mental healthcare institutions in The Netherlands. International Journal of Environmental Research and Public Health, 15(5), 910, https://doi.org/10.3390/ijerph15050910

86.

Moullin

J. C.

Sabater-Hernández

Fernandez-Llimos

Benrimoj

S. I.

(2015). A systematic review of implementation frameworks of innovations in healthcare and resulting generic implementation framework. Health Research Policy & Systems, 13(1), 1–11. https://doi.org/10.1186/s12961-015-0005-z

87.

Natesan

S. M.

Marathe

R. R

. (2015). Literature review of public policy implementation. International Journal of Public Policy, 11(4/5/6), 219–241. https://doi.org/10.1504/IJPP.2015.070550

88.

Nayak

S. S.

Fraser

Panagopoulos

Aldrich

D. P.

Kim

(2021). Is divisive politics making Americans sick? Associations of perceived partisan polarization with physical and mental health outcomes among adults in the United States. Social Science & Medicine, 284, 113976. https://doi.org/https://doi.org/10.1016/j.socscimed.2021.113976

89.

Ngui

E. M.

Khasakhala

Ndetei

Roberts

L. W.

(2010). Mental disorders, health inequalities and ethics: A global perspective. International Review of Psychiatry, 22(3), 235–244. https://doi.org/10.3109/09540261.2010.485273

90.

M. Y.

Kim

McDowell

Wong

Qiu

Wong

I. O.

Galea

Leung

G. M.

(2020). Mental health during and after protests, riots and revolutions: A systematic review. Australian & New Zealand Journal of Psychiatry, 54(3), 232–243. https://doi.org/10.1177/0004867419899165

91.

Nilsen

Bernhardsson

(2019). Context matters in implementation science: A scoping review of determinant frameworks that describe contextual determinants for implementation outcomes. BMC Health Services Research, 19(1), 189. https://doi.org/10.1186/s12913-019-4015-3

92.

Nilsen

Ståhl

Roback

Cairney

(2013). Never the twain shall meet? - a comparison of implementation science and policy implementation research. Implementation Science, 8(1), 1–12. https://doi.org/10.1186/1748-5908-8-63

93.

Abazeed

Chambers

D. A.

(2021). Policy implementation science to advance population health: The potential for learning health policy systems. Frontiers in Public Health, 9, 740. https://doi.org/10.3389/fpubh.2021.681602

94.

Park

M. M.

Zafran

Stewart

Salsberg

Ells

Rouleau

Estein

Valente

T. W.

(2014). Transforming mental health services: A participatory mixed methods study to promote and evaluate the implementation of recovery-oriented services. Implementation Science, 9(1), 1–21, 119. https://doi.org/10.1186/s13012-014-0119-7

95.

Patalay

Giese

Stanković

Curtin

Moltrecht

Gondek

(2016). Mental health provision in schools: Priority, facilitators and barriers in 10 European countries. Child & Adolescent Mental Health, 21(3), 139–147. https://doi.org/10.1111/camh.12160

96.

Patterson

M. G.

West

M. A.

Shackleton

V. J.

Dawson

J. F.

Lawthom

Maitlis

Robinson

D. L.

Wallace

A. M.

(2005). Validating the organizational climate measure: Links to managerial practices, productivity and innovation. Journal of Organizational Behavior, 26(4), 379–408. https://doi.org/https://doi.org/10.1002/job.312

97.

Peters

D. T.

Tran

N. T.

Adam

(2013). Implementation Research in Health: A Practical Guide. World Health Organization and the Alliance for Health Policy and System Research.

98.

Phulkerd

Lawrence

Vandevijvere

Sacks

Worsley

Tangcharoensathien

(2016). A review of methods and tools to assess the implementation of government policies to create healthy food environments for preventing obesity and diet-related non-communicable diseases. Implementation Science, 11, 15. https://doi.org/10.1186/s13012-016-0379-5

99.

Pollack Porter

K. M.

Rutkow

McGinty

E. E

. (2018). The importance of policy change for addressing public health problems. Public Health Reports, 133(1_suppl), 9S–14S. https://doi.org/10.1177/0033354918788880

100.

Powell

B. J.

Beidas

R. S.

Rubin

R. M.

Stewart

R. E.

Wolk

C. B.

Matlin

S. L.

Weaver

Hurford

M. O.

Evans

A. C.

Hadley

T. R.

Mandell

D. S.

(2016). Applying the policy ecology framework to Philadelphia's behavioral health transformation efforts. Administration and Policy in Mental Health, 43(6), 909–926. https://doi.org/10.1007/s10488-016-0733-6

101.

Powell

B. J.

Mettert

K. D.

Dorsey

C. N.

Weiner

B. J.

Stanick

C. F.

Lengnick-Hall

Ehrhart

M. G.

Aarons

G. A.

Barwick

M. A.

Damschroder

L. J.

Lewis

C. C.

(2021). Measures of organizational culture, organizational climate, and implementation climate in behavioral health: A systematic review. Implementation Research and Practice, 2, 26334895211018862. https://doi.org/10.1177/26334895211018862

102.

Powell

B. J.

Stanick

C. F.

Halko

H. M.

Dorsey

C. N.

Weiner

B. J.

Barwick

M. A.

Damschroder

L. J.

Wensing

Wolfenden

Lewis

C. C.

(2017). Toward criteria for pragmatic measurement in implementation research and practice: A stakeholder-driven approach using concept mapping. Implementation Science, 12(1), 118. https://doi.org/10.1186/s13012-017-0649-x

103.

Proctor

Silmere

Raghavan

Hovmand

Aarons

Bunger

Griffey

Hensley

(2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 38(2), 65–76. https://doi.org/10.1007/s10488-010-0319-7

104.

Purtle

Brownson

R. C.

Proctor

E. K.

(2017). Infusing science into politics and policy: The importance of legislators as an audience in mental health policy dissemination research. Administration and Policy in Mental Health, 44(2), 160–163. https://doi.org/10.1007/s10488-016-0752-3

105.

Purtle

Nelson

K. L.

Counts

N. Z.

Yudell

(2020). Population-Based approaches to mental health: History, strategies, and evidence. Annual Review of Public Health, 41(1), 201–221. https://doi.org/10.1146/annurev-publhealth-040119-094247

106.

Purtle

Lê-Scherban

Wang

Shattuck

P. T.

Proctor

E. K.

Brownson

R. C.

(2018). Audience segmentation to disseminate behavioral health evidence to legislators: An empirical clustering analysis. Implementation Science, 13(1), 121. https://doi.org/10.1186/s13012-018-0816-8

107.

Purtle

Peters

Brownson

R. C.

(2015). A review of policy dissemination and implementation research funded by the National Institutes of Health, 2007-2014. Implementation Science, 11, 1. https://doi.org/10.1186/s13012-015-0367-1

108.

Rabin

B. A.

Lewis

C. C.

Norton

W. E.

Neta

Chambers

Tobin

J. N.

Brownson

R. C.

Glasgow

R. E.

(2015). Measurement resources for dissemination and implementation research in health. Implementation Science, 11, 42. https://doi.org/10.1186/s13012-016-0401-y

109.

Raghavan

Bright

C. L.

Shadoin

A. L.

(2008). Toward a policy ecology of implementation of evidence-based practices in public mental health settings. Implementation Science, 3, 1–9. https://doi.org/10.1186/1748-5908-3-26

110.

Rust

Zhang

Malhotra

Reese

McRoy

Baltrus

Caplan

Levine

R. S.

(2015). Paths to health equity: Local area variation in progress toward eliminating breast cancer mortality disparities, 1990-2009. Cancer, 121(16), 2765–2774. https://doi.org/10.1002/cncr.29405

111.

Sabatier

P. A

. (1988). An advocacy coalition framework of policy change and the role of policy-oriented learning therein. Policy Sciences, 21(2-4), 129–168. http://libproxy.wustl.edu/login?url=http://search.ebscohost.com/login.aspx?direct=true&db=snh&AN=16845332&site=ehost-live&scope=site https://doi.org/10.1007/BF00136406

112.

Sabatier

Mazmanian

(1980). Implementation of Public Policy - A Framework of Analysis (From Policy Studies, P 181-203, 1980, Bertram H Raven, ed. - See NCJ-74695). In. United States.

113.

Saxena

Thornicroft

Knapp

Whiteford

(2007). Resources for mental health: Scarcity, inequity, and inefficiency. Lancet, 370(9590), 878–889. https://doi.org/10.1016/S0140-6736(07)61239-2

114.

Schnake-Mahl

A. S.

Jahn

J. L.

Purtle

Bilal

(2022). Considering multiple governance levels in epidemiologic analysis of public policies. Social Science & Medicine, 314, 115444. https://doi.org/https://doi.org/10.1016/j.socscimed.2022.115444

115.

Shera

Ramon

(2013). Challenges in the implementation of recovery-oriented mental health policies and services. International Journal of Mental Health, 42(2-3), 17–42. https://doi.org/10.2753/IMH0020-7411420202

116.

Shim

R. S.

Compton

M. T.

(2020). Addressing the social determinants of mental health: If not now, when? If not us, who? FOCUS, 18(1), 75–76. https://doi.org/10.1176/appi.focus.18104

117.

Smith

S. N.

Lai

Almirall

Goodrich

D. E.

Abraham

K. M.

Nord

K. M.

Kilbourne

A. M.

(2017). Implementing effective policy in a national mental health reengagement program for veterans. Journal of Nervous & Mental Disease, 205(2), 161–170. https://doi.org/10.1097/NMD.0000000000000592

118.

Stanick

Halko

Mettert

Dorsey

Moullin

Weiner

Powell

Lewis

C. C.

(2021). Measuring characteristics of individuals: An updated systematic review of instruments’ psychometric properties. Implementation Research and Practice, 2, 26334895211000458. https://doi.org/10.1177/26334895211000458

119.

Stanick

C. F.

Halko

H. M.

Dorsey

C. N.

Weiner

B. J.

Powell

B. J.

Palinkas

L. A.

Lewis

C. C.

(2018). Operationalizing the ‘pragmatic’ measures construct using a stakeholder feedback and a multi-method approach. BMC Health Services Research, 18(1), 882. https://doi.org/10.1186/s12913-018-3709-2

120.

Stanick

C. F.

Halko

H. M.

Nolen

E. A.

Powell

B. J.

Dorsey

C. N.

Mettert

K. D.

Weiner

B. J.

Barwick

Wolfenden

Damschroder

L. J.

Lewis

C. C.

(2019). Pragmatic measures for implementation research: development of the Psychometric and Pragmatic Evidence Rating Scale (PAPERS). Translational Behavioral Medicine, 11(1), 11–20. https://doi.org/10.1093/tbm/ibz164

121.

Stirman

S. W.

Gutner

C. A.

Langdon

Graham

J. R.

(2016). Bridging the gap between research and practice in mental health service settings: An overview of developments in implementation theory and research. Behavior Therapy, 47(6), 920–936. https://doi.org/10.1016/j.beth.2015.12.001

122.

Stirman

S. W.

Miller

C. J.

Toder

Calloway

(2013). Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implementation Science, 8(1), 65. https://doi.org/10.1186/1748-5908-8-65

123.

Strohecker

(2021). Analysis: Pandemics & protests: Unrest grips developing countries. Reuters. https://www.reuters.com/world/pandemics-protests-unrest-grips-developing-countries-2021-07-28/

124.

Substance Abuse and Mental Health Services Administration (2018). Key substance use and mental health indicators in the United States: Results from the 2017 National Survey on Drug Use and Health. In (HHS Publication No. SMA 18-5068, NSUDH Series H-53). Rockville, MD Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration.

125.

Thomann

Zhelyazkova

(2017). Moving beyond (non-)compliance: The customization of European Union policies in 27 countries. Journal of European Public Policy, 24(9), 1269–1288. https://doi.org/10.1080/13501763.2017.1314536

126.

Tummers

(2009). Policy alienation of public professionals: The development of a scale Netherlands Institute of Governance Annual Work Conference Leiden. https://www.utwente.nl/en/nig/research/archive/2009/Papers/panel10papertummers.pdf

127.

Tummers

(2011). Explaining the Willingness of public professionals to implement new policies: A policy alienation framework. International Review of Administrative Sciences, 77(3), 555–581. https://doi.org/10.1177/0020852311407364

128.

Tummers

(2012). Policy alienation of public professionals: The construct and its measurement. Public Administration Review, 72(4), 516–525. https://doi.org/10.1111/j.1540-6210.2011.02550.x

129.

Tummers

Bekkers

(2014). Policy implementation, street-level bureaucracy, and the importance of discretion. Public Management Review, 16(4), 527–547. https://doi.org/10.1080/14719037.2013.841978

130.

Tummers

L. G.

Van de Walle

(2012). Explaining health care professionals’ resistance to implement diagnosis related groups: (No) benefits for society, patients and professionals. Health Policy, 108(2/3), 158–166. https://doi.org/10.1016/j.healthpol.2012.08.024

131.

U.S. Department of Health and Human Services. (2021). Healthy People 2030: Social Determinants of Health. Retrieved July 22, 2021 from https://health.gov/healthypeople/objectives-and-data/social-determinants-health

132.

U.S. Department of Veteran Affairs. (2021). National Center for Organization Development: VA All Employee Survey. U.S. Department of Veteran Affairs. https://www.va.gov/NCOD/VAworkforcesurveys.asp

133.

United Nations (2020). Policy brief: COVID 19 and the need for action on mental heath. United Nations.

134.

Walsh-Bailey

Allen

Kepper

M. M.

Vogel

M. T.

Pilar

McLoughlin

G. M.

Brownson

R. C.

(2020). Measurement of chronic disease policy implementation determinants and outcomes: A systematic review 13th Annual Conference on the Science of Dissemination and Implementation in Health, Virtual (due to COVID-19).

135.

Watson

D. P.

Adams

E. L.

Shue

Coates

McGuire

Chesher

Jackson

Omenka

O. I.

(2018). Defining the external implementation context: An integrative systematic literature review. BMC Health Services Research, 18(1), 209–209. https://doi.org/10.1186/s12913-018-3046-5

136.

Weinberg

Whiteford

de Almeida

J. C.

Aguilar-Gaxiola

Levinson

O'Neill

Kovess-Masfety

(2012). Translation of the world mental health survey data to policies: An exploratory study of stakeholders’ perceptions of how epidemiologic data can be utilized for policy in the field of mental health. Public Health Reviews (2107-6952), 34(2), 1–21. https://doi.org/10.1007/BF03391672

137.

Weiner

B. J.

Amick

Lee

S.-Y. D.

(2008). Review: Conceptualization and measurement of organizational readiness for change: A review of the literature in health services research and other fields. Medical Care Research and Review, 65(4), 379–436. https://doi.org/10.1177/1077558708317802

138.

Weiner

B. J.

Mettert

K. D.

Dorsey

C. N.

Nolen

E. A.

Stanick

Powell

B. J.

Lewis

C. C.

(2020). Measuring readiness for implementation: A systematic review of measures’ psychometric and pragmatic properties. Implementation Research and Practice, 1, 2633489520933896. https://doi.org/10.1177/2633489520933896

139.

Wensing

Pfaff

Grol

(2020). Health system strategies for implementation. In Improving Patient Care (pp. 299–311). Wiley Blackwell. https://doi.org/10.1002/9781119488620.ch17

140.

Williams

N. J.

(2016). Multilevel mechanisms of implementation strategies in mental health: Integrating theory, research, and practice. Administration And Policy In Mental Health, 43(5), 783–798. https://doi.org/10.1007/s10488-015-0693-2

141.

Wittchen

H. U.

Jacobi

Rehm

Gustavsson

Svensson

Jönsson

Olesen

Allgulander

Alonso

Faravelli

Fratiglioni

Jennum

Lieb

Maercker

van Os

Preisig

Salvador-Carulla

Simon

Steinhausen

H. C.

(2011). The size and burden of mental disorders and other disorders of the brain in Europe 2010. European Neuropsychopharmacology, 21(9), 655–679. https://doi.org/10.1016/j.euroneuro.2011.07.018

142.

Woodward

E. N.

Singh

R. S.

Ndebele-Ngwenya

Melgar Castillo

Dickson

K. S.

Kirchner

J. E.

(2021). A more practical guide to incorporating health equity domains in implementation determinant frameworks. Implementation Science Communications, 2(1), 61. https://doi.org/10.1186/s43058-021-00146-5

143.

World Health Organization (2007). Monitoring and evaluation of mental health policies and plans. World Health Organization.

144.

World Health Organization, & Calouste Gulbenkian Foundation. (2014). Social determinants of mental health . https://www.who.int/health-topics/social-determinants-of-health#tab=tab_1

145.

Wright

(2017). What's so important about health policy implementation? https://digitalpublications.parliament.scot/ResearchBriefings/Report/2017/9/7/What-s-so-important-about-health-policy-implementation-#Executive-Summary

146.

Yan

B. W.

Hsia

R. Y.

Yeung

Sloan

F. A.

(2021). Changes in mental health following the 2016 presidential election. Journal of General Internal Medicine, 36(1), 170–177. https://doi.org/10.1007/s11606-020-06328-6

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB

0.02 MB

0.01 MB

0.03 MB