Provider perspectives and experiences following the integration of the prescription drug monitoring program into the electronic health record

Abstract

Prescription drug monitoring programs (PDMPs) are a public health tool for prescribers and pharmacists to monitor controlled substance use at the patient level. The objectives of this study were to (1) assess attitudes about the PDMP and perceived changes in experience and decision-making following integration into the electronic health record (EHR), and (2) identify barriers to optimal PDMP use and user recommendations for improvement. This descriptive study used an electronic survey to obtain feedback from prescribers and pharmacists at a single academic medical center. Descriptive statistics were calculated, and textual data were analyzed. Of 1480 PDMP users 208 responded (14.1%). Responses demonstrated the integration of the PDMP into the EHR simplified log-in and access to PDMP information resulting in more frequent access and perceived improvement of care quality. Barriers included technical issues accessing the PDMP within the EHR and the lack of integration of other state PDMPs. Overall, the benefit of integrating the PDMP into the EHR was positive but largely limited to streamlining log-in and patient selection. Recommendations for improvement include addressing technological issues and education on PDMP interpretation and integrating new features that may modify prescribing, referral, and co-prescribing behaviors.

Keywords

Prescription drug monitoring program opioid prescriber electronic health record controlled substance

Introduction

In 2016, an estimated 12 million persons aged 12 and older misused prescription opioids and other pain relievers.¹ In 2016 the CDC provided new guidance for opioid prescribing.² Recent estimates suggest a significant increase in opioid deaths after the emergence of the COVID-19 pandemic.^3,4 People obtain prescriptions for opioids and other controlled substances from prescribing physicians, nurse practitioners, and physician assistants who work in primary care, pain medicine, emergency departments, and other specialties. Controlled substances are dispensed from pharmacies, emergency departments, and other avenues, including through unregulated transactions. This complex system of controlled substance prescribing, and distribution necessitated the creation of centralized monitoring of prescribed controlled substances.

Efforts to monitor controlled substance prescribing in the U.S. originated in the early 20th century with various requirements that physicians or pharmacists submit copies of controlled substance prescriptions to central databases.^5,6 Modern state-level electronic prescription drug monitoring programs (PDMP or PMP) emerged in the 1990s to aggregate controlled substance prescription data from various sources that prescribers, pharmacists, law enforcement, licensing boards, and other registrants can use to query controlled substance prescribing and dispensing histories for individuals in a given state.^5,6 Such PDMP queries can help identify potentially concerning patterns like the use of multiple prescribers and pharmacies, and potentially harmful medication combinations. Studies suggest PDMPs are associated with reduced opioid prescribing and decreased occurrence of people using multiple prescribers for controlled substances,^7–9 although the association to decreasing opioid-related overdoses is still unclear given the complexity of the opioid prescribing and use process.¹⁰

Like other electronic health data programs, modern PDMPs include analytic tools to alert the authorized user if an individual’s controlled substance prescription data indicated high-risk use such as multiple prescribers and pharmacies, high doses, and contraindicated medication combinations. These are sometimes displaced graphically or with an indexed risk score.

While PDMPs contain valuable information, the data is often limited to a single state given the patchwork of state legislation establishing their creation, maintenance, and use.⁵ Barriers to PDMP use have been reported such as busy work environments that make logging in to access PDMP information in a time-efficient manner a challenge.^11-13 One of the most cited and critical barriers is access requiring a log-in to a separate website and re-entering patient information to gain access to the PDMP.^{8,11,12,14–17} In response, some providers and health systems have implemented workarounds.^13,18 A recent enhancement is to integrate the PDMPs into Electronic Health Records (EHR) which has the potential to streamline PDMP access by using the providers existing EHR credentials and patient records. This however can be costly and is still not the norm across health systems and states.^19–22

Since the integration of PDMPs into electronic health records is an emerging process, the present study aims to contribute to the PDMP literature by evaluating the potential benefits and impacts of integrating the PDMP into the EHR for a single academic medical center in the Midwest U.S. Obtaining feedback directly from PDMP users is expected to identify areas for improvement and areas of strength and limitation that can help stakeholders better understand the impact of PDMP integration into the EHR.

Objectives

The objectives of this study were to (1) assess attitudes about the PDMP and perceived changes in experience and decision-making following integration into the EHR and (2) identify barriers to optimal PDMP use and user recommendations for improvement.

Methods

This descriptive study used an electronic survey to gather feedback from physicians, nurse practitioners, pharmacists, and physician assistants employed at a single academic medical center that provides both inpatient and outpatient services. Integration of the PDMP into the EHR occurred in October 2020 and this electronic survey followed in December. This study was deemed non-human subject research by the University Institutional Review Board given the focus on process improvement.

Study sample and recruitment

The study sample consisted of 1480 Email addresses for PDMP users at the study academic medical center who had accessed the PDMP through the integrated functionality in the electronic health record platform. These email addresses were provided by the vendor to the State Board of pharmacy who sent the emails with the survey link. The State Board of Pharmacy emailed potential participants on behalf of the evaluation team in December 2020. One reminder was sent a week after the first contact.

Survey development

The authors were not able to identify a survey instrument that aligned with the study objectives, therefore, survey development was informed by 15 in-depth qualitative interviews of healthcare providers from family medicine, pain management, pharmacy, and emergency medicine. The Interviews asked about how informants use the PDMP in their role, examples of how the PDMP has been useful (including with patients with substance use disorders), and perceived barriers to optimal PDMP usability. A further rationale for the interviews was to better ground the survey in the lived experiences of PDMP users.

These semi-structured interviews were conducted by two qualitative researchers (MW and BSM) and lasted 12–57 min. Interview recordings were transcribed and inputted into MAXQDA^TM, a qualitative data management software. The two researchers reviewed the transcripts and used a descriptive content analysis approach to code the textual data. Open coding was performed independently by the two researchers and consensus was established through regular meetings. Subcategories and categories were developed and thematic groupings were identified, including domains for writing the survey items.

Interview transcripts were reviewed to inform the survey items in using a process of mixed method item development.^23–25 For example, the interviews suggested some users found the graphical representations of PDMP data to be helpful whereas others said they ignored the visuals. The authors captured this variation with the strongly disagree to strongly agree item “The visual aids are an important part of how I interpret the PDMP.” The pool of generated items were generated by one study author and reviewed, edited, and prioritized for inclusion by the other authors. The results of these interviews are available upon request.

Survey domains

The electronic survey used closed and open-ended (free text) items, and completion took 5–10–min. The first survey domain measured participant attitudes about the PDMP (11 items) using a Likert scale (1= strongly disagree to 4 = strongly agree or NA). The second domain measured changes in the PDMP experience (4 items) using a selection of three potential responses: harder, stayed the same, or easier. The third domain measured changes in their process and decision-making since integration (9 items) using a selection of three potential responses: decreased, stayed the same, or increased. There were three open-ended prompts at the end of the survey: “Since the PDMP was integrated into the EHR, what challenges or barriers have emerged or remain related to using the PDMP?” “What features or aspects of the PDMP do you find most useful?” and “What features, or functionalities could further enhance your ability to care for patients requiring controlled substances?” Demographics data were collected and included years in role, provider type, care setting (primary, acute, both, neither).

Survey analysis

Descriptive statistics were calculated for numerical data using SPSS version 22. Open-ended responses to the three free text prompts were analyzed by two investigators using a descriptive qualitative approach^26,27 Two researchers, independently read and re-read text to code the answers to the open-ended questions, then compared and established consensus of the coding. These codes were sorted through consensus into sub-categories, categories, and themes emerged to further enable interpretation. This survey dataset was separate from the interview dataset used to develop the survey items. Consensus by the two researchers facilitated validation. Furthermore, validation occurred by the third author who assessed the relevance of results to our research questions. Exemplary quotes were identified to ground the data interpretation in the voices of the participants.

Results

The state Board of Pharmacy sent survey emails to 1480 PDMP user emails who logged into the integrated PDMP for the study academic medical center. Of these, 208 surveys were completed for a 14.1% response rate (Table 1). The average years in practice for the respondents was 12.3 (SD = 10.9). The most common respondent were physicians (60.6%). Half of the respondents practice in both an acute care and chronic care setting (49.5%). Most respondents reported accessing the PDMP through the EHR since integration (79.8%).

Table 1.

Characteristics of respondents (N = 208).

Characteristic	N (%)	Mean (SD)
Years in role		12.3 (11.9)
Provider type		—
Physician	126 (60.6)	—
Advanced practice nurse (APRN)	45 (21.6)	—
Physician assistant (PA)	21 (10.1)	—
Pharmacist	15 (7.2)	—
Medical assistant	1 (0.5)	—
Care setting
Primary/chronic care	69 (33.2)	—
Acute care	29 (13.9)	—
Both	103 (49.5)	—
Neither	7 (3.4)	—
How access the PMP
Through integration into the electronic medical record (EPIC)	80 (38.5)	—
Through the [Iowa] PMP website (iowa.pmpaware.net)	27 (13.0)	—
Both through EPIC and through the Iowa PMP website	86 (41.3)	—
Neither I do not access the PMP myself	15 (7.2)	—

Attitudes about PDMPs

Respondents overall had positive attitudes about the PDMP with average responses indicating somewhat strong to strong agreement on a scale of 1–4 (Table 2). The most strongly positive responses included: “Overall, integrating the PDMP into the EHR has improved the quality of care I provide” (mean = 3.48; SD = 0.89); “Integrating the PDMP into the EHR has resulted in me checking the PDMP more often” (mean = 3.31; SD = 1.03). Less positive agreement responses occurred in statements: “I was adequately trained to use the PDMP integrated into the EHR” (mean=2.91; SD=1.04) and “Consistent PDMP use is the norm among my colleagues” (mean=2.98; SD = 0.93).

Table 2.

PDMP attitudes.

Item	N	NA	Mean	Std. Deviation
I was adequately trained to use the PMP integrated into the EHR	187	8	2.91	1.035
I am confident that the data in the PMP is accurate	187	6	3.49	0.765
I am confident in my ability to interpret the data presented in the PMP	187	5	3.47	0.805
The visual aids are an important part of how I interpret the PMP	187	10	2.90	0.911
My healthcare organization has clear expectations for how I should use the PMP	187	7	2.80	0.942
Consistent PMP use is the norm among my colleagues	186	9	2.98	0.929
Integrating the PMP into the EHR has improved my efficiency	185	19	3.60	0.838
Integrating the PMP into the EHR has resulted in me checking the PMP more often	187	21	3.31	1.032
Integrating the PMP into the EHR makes me more confident in how I manage patients taking controlled substances	186	17	3.30	0.973
Integrating the PMP into the EHR has improved my ability to decide on appropriate quantities of controlled substances	187	24	3.24	0.980
Overall, integrating the PMP into the EHR has improved the quality of care I provide	186	19	3.48	0.897
N providing answers to all items	182

1=Strongly disagree, 2=Somewhat disagree, 3=Somewhat agree, 4=Strongly agree

NA=Not applicable; N for these items do not equal 208 because of missing data.

Changes in experiences using PDMP following EHR integration

After the integration of PDMP into the EHR, participants reported logging in and accessing information on the correct patient was easier (78.3%, 76.5%, respectively) (Figure 1). Participants’ responses were more split about the following comments: “Determining if a patient’s PDMP profile suggests there may be a concern with their controlled substance history” and “Determining what controlled substances someone is taking” was easier or stayed the same. Few reported any of these were harder following integration.

Figure 1.

Change in PDMP experience following integration.

For the nine PDMP processes and decision-making items, three items suggested changes in behavior (Figure 2). Notably, 74.5% reported a decrease in “The amount of time it takes me to evaluate the PDMP for the average patient,” 58.1% reported an increase in “Reviewing the PDMP myself,” and 38.6% reported an increase in “Entering PDMP information into a patient’s medical record.” The self-reported behaviors of prescribing controlled substances, co-prescribing naloxone, and referring individuals to substance use treatment, were not perceived to have changed following EHR integration of the PDMP (89–94% reported these stayed the same).

Figure 2.

How has your process and decision-making changed since the PDMP was integrated into the EHR?.

Summary of open-ended survey responses

The open-ended survey questions allowed the participant to expand on their answers to each question. The questions covered the remaining challenges and barriers to using the PDMP, the most useful features of the PDMP, and features and functionality that would enhance care for the patients requiring controlled substances. Exemplary quotes were used to facilitate understanding of these answers.

Question 1: “Since the PDMP was integrated into the EHR, what challenges or barriers have emerged or remain related to using the PDMP?

The lack of EHR integration of other neighboring state PDMPs was the most common challenge. This was seen as a significant deficiency, especially for communities that closely bordered other states. Also, while most found the integrated PDMP made access to the database easier, some respondents continued to report technical challenges that interfered with how they preferred to use the PDMP. One participant wrote, “You also have to be in a patient clinic or inpatient encounter to see the PMP, which is annoying--as I may want to check the PMP outside of an encounter if I’m double-checking something before or after a clinic visit or inpatient encounter.” Another participant wrote, “I can find it for some encounters, but not as easily on others. I see patients in multiple settings that have a different Epic layout, so I think I just need to learn where to look.” Additionally, there were several comments about a patient not being automatically pulled in, requiring the provider to go to the website as before. One participant described, “Some patients’ record does not pull in, and [you] have to manually log into [the] website to check.”

There were less commonly reported barriers by the respondents which warrant attention because they may hinder PDMP interpretation. For example, if a resident physician is the prescriber, it may look like the patient is seeing multiple doctors, but in reality, it is the same clinic or the same supervising physician since specialty information is not provided about the prescriber. One participant wrote, “It is difficult to tell which resident prescribed a controlled substance and the specialty of that resident; this would improve patient care to know who writes each script and their specialty.” Furthermore, there were reported inconsistencies or incorrect information about the days supplied by the prescription. This specific information may offer clarity in “… determining gaps in time between prescriptions (i.e. # days or weeks) to translate from the SIG and amount dispensed how much more or less the patient might be using over a defined period of time.”

There were two more challenges mentioned following the integration of PDMP into EHR. One respondent from the Emergency Department reported that entering a patient’s zip code in the process of accessing the PDMP was sometimes challenging. Another mentioned that printing reports and handouts to show the patient has become more difficult after integration.

Question 2: What features or aspects of the PDMP do you find most useful?

The most common responses were general enthusiasm for the single click, one-button access. They described this feature as, “Easy and fast, don’t need another password,” “Love single click,” and “I can access the right patient in PMP quickly. I can’t tell you how much this has improved my quality of life. Thank you.”

There were positive responses to specific features. These features included the ability to easily document the PDMP was reviewed and the ease of accessing patients who were previously difficult to access. One participant wrote, “It [integrated PDMP] uses chart info to auto search the patient I’m looking for. Previously I had trouble when trying to search using a partial name to avoid errors.” Comments were made about documenting their review “with a click.” And a few respondents mentioned liking the graphical elements, e.g. fill dates, graphs of MME/LMEs/day, and the “scores” (opioid risk scores).

Question 3. What features, or functionalities could further enhance their ability to care for patients requiring controlled substances?

Many of the desired features addressed barriers from the other prompts. Additional comments included expanding access to the integrated PDMP to delegates, having the PDMP button always available in the same place in the EHR regardless of the task, making it easier to denote that the PDMP was reviewed, additional search filters (medication, prescriber, etc.), a better explanation of risk scores, and printouts for patient education.

Discussion

This study characterized the perceptions and experiences of integrating the PDMP into the EHR at a large academic medical center. While there were gains in efficiency related to integration, providers did not self-report changes to behaviors based on PDMP content, including perceptions that their rate of prescribing controlled substances, co-prescribing naloxone, and referring individuals to substance use treatment each largely stayed the same following integration (Figure 2). Achieving increases in these behaviors likely will require enhancements to the PDMP interface or changes, to the PDMP use process, or other changes to practice. The findings are organized in a structure-process-outcome²⁸ framework (Figure 3) which highlights areas where the integration appeared to have a positive impact and areas where integration did not appear to make a significant difference.

Figure 3.

Summary structure, process, and outcome findings from the PDMP integration evaluation survey.

Structure

For structural factors, respondents reported the organization did not provide much additional education or training related to integration. This may be a result of the integration occurring during the COVID-19 pandemic which was a significant strain on healthcare workers and their processes. Nevertheless, additional guidance and training are recommended in the future because a study among community pharmacists identified a lack of training as a barrier to optimal PDMP use.²⁹ A toolkit designed for community pharmacists demonstrated improvements in knowledge and self-efficacy related to PDMP use,²⁹ and similar approaches could be used for other providers and settings.

Process

Most respondents found the process of using the PDMP within the EHR to be a welcome improvement and increased their efficiency. The two process concepts addressed were logging in and interpreting the PDMP profile. Logging in and accessing a specific patient profile was generally made easier with the integration which increased efficiency. Half of the providers reported an increase in checking the PDMP themselves. Three barriers were raised, however. First, only the home state PDMP was integrated into the EHR. Single state PDMPs have been reported by others as limiting information, as well.¹² This barrier could be rectified by simplifying access to neighboring states. Second, the PDMP was only accessible within a visit encounter. This feature made it challenging to check the PDMP outside of a scheduled visit. A recommendation is to create universal access to the PDMP within a patient’s chart. Third, once providers reached the process of reviewing a PDMP profile, the impact of integration went away and the act of interpreting PDMP information remained dependent on the healthcare provider. This result could have been anticipated because the outputs or functionality do not provide interpretation skills. Possible improvements with interpretation may be case-based education that can be accessed within the PDMP website and create full information access for the healthcare team by streamlining the ability to copy PDMP information into a patient’s EHR.

Outcome

The outcomes of the PDMP integration into the EHR focused on two general areas: quality of care and self-reported changes to prescribing behaviors. These provided conflicting feedback. Providers claimed PDMP-EHR integration improved the quality of care they provide. However, when asked specifically about prescribing practice changes, there was no perceived influence on controlled substance prescribing rates, co-prescribing naloxone, or making referrals to substance use treatment. This may be a result of the third process finding mentioned previously. The lack of change to process once the provider is viewing the PDMP profile. While the present study was not able to report changes in actual prescribing behavior, a recent study by Weiner et al. demonstrated PDMP integration into the EHR had increased PDMP use and decreased opioid prescriptions, and decrease morphine equivalents for a health system.²¹ Our study may require direct measurement of prescriptions and MME in the EHR pre and post-integration.

Limitations

The generalizability of this survey is limited beyond the single, Midwestern US academic medical center, other systems and regions may have different PDMP experiences. This evaluation focused on one state’s PDMP and one electronic health record and only providers who had accessed the integrated PDMP at least once were invited to participate. HCPs who had not used the integrated PDMP may have different perspectives. Survey items were self-reported and may be subject to social desirability or accessibility bias. Also, there is the potential for non-response bias, including bias that may occur related to the electronic nature of the survey and the use of email recruitment.

Future research

These evaluation data suggest several opportunities for improvement. First, the technical and logistical barriers and concerns should be evaluated, and the system should be redesigned when feasible. Second, providers, the health system, EHR designers, and PDMP designers should collaborate to pilot new features and functionalities that capitalize on the integrated EHR and PDMP datasets. Other studies have trialed enhancements to PDMPs with some success, especially when including users in the design process.^30,31 Examples could include using best practice advisory (BPA) alerts for naloxone co-prescribing, referral to substance use treatment, or deprescribing interventions. BPAs have been shown to facilitate assessment and action³² and may be relevant based on patient data.³³ Using newly embedded assessments and tools could be more feasible since providers reported the integrated PDMP saves time and increases their efficiency. More training also would likely be required for these new features. Methodologically, subsequent work should include electronic records to corroborate perceived changes in behavior which were not feasible for the present evaluation given dynamics related to COVID-19.

Conclusions

These evaluation data provide evidence supporting the integration of the PDMP into EHRs to overcome a common barrier to PDMP use by simplifying the login and patient selection process. This improvement in efficiency, however, does not appear sufficient to fundamentally change how providers and pharmacists use the PDMP to make decisions or collaborate with patients who use controlled substances. Improvements to the user interface, incorporation of multiple state PDMPs, and incorporating other decision support aids likely will be needed for PDMPs to achieve their full potential as a clinical tool.

Footnotes

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by an award from the Iowa Board of Pharmacy. The funder had no role in the analysis or publication of the research.

ORCID iD

Matthew Witry

References

Substance Abuse Mental Health Services Administration . Key substance use and mental health indicators in the United States: Results from the 2016 National Survey on Drug Use and Health. Rockville, MD: Center for Behavioral Health Statistics and Quality, 2017. HHS Publication No. SMA 17-5044, NSDUH Series H-52)Substance Abuse and Mental Health Services Administration Retrieved from.https://www/samhsa/gov/data

Dowell

Haegerich

Chou

. CDC guideline for prescribing opioids for chronic pain—United States, 2016. JAMA 2016; 315(15): 1624–1645.

Rosenbaum

Lucas

Zandrow

, et al. Impact of a shelter-in-place order during the COVID-19 pandemic on the incidence of opioid overdoses. Am J Emerg Med 2021; 41: 51–54.

Ahmad

Rossen

Sutton

. Provisional drug overdose death counts. National Center for Health Statistics. In: Statistics, Centers for Disease Control and Prevention. CDC, 2021.

Holmgren

Botelho

Brandt

. A history of prescription drug monitoring programs in the United States: political appeal and public health efficacy. Am J Public Health 2020; 110(8): 1191–1197.

Prescription Drug Monitoring Program Training and Technical Assistance Center . History of prescription drug monitoring programs. Massachusetts: Brandeis University Heller School for Social Policy and Management, 2018.

Wilson

Hayden

Rhodes

, et al. Effectiveness of prescription monitoring programs in reducing opioid prescribing, dispensing, and Use outcomes: a systematic review. J Pain 2019; 20(12): 1383–1393.

Deyo

Hallvik

Hildebran

, et al. Association of prescription drug monitoring program use with opioid prescribing and health outcomes: a comparison of program users and nonusers. J Pain 2018; 19(2): 166–177.

Bao

Wen

Johnson

, et al. Assessing the impact of state policies for prescription drug monitoring programs on high-risk opioid prescriptions. Health Aff 2018; 37(10): 1596–1604.

10.

Fink

Schleimer

Sarvet

, et al. Association between prescription drug monitoring programs and nonfatal and fatal drug overdoses: a systematic review. Ann Intern Med 2018; 168(11): 783–790.

11.

Reist

Frazier

Rottingham

, et al. Provider beliefs on the Barriers and facilitators to prescription monitoring programs and mandated use. Subst Use Misuse, 2019, pp. 1–11.

12.

Mastarone

Wyse

Wilbur

, et al. Barriers to utilization of prescription drug monitoring programs among prescribing physicians and advanced practice registered nurses at veterans health administration facilities in oregon. Pain Med 2019; 21(4): 695–703.

13.

Freeman

Curran

Drummond

, et al. Utilization of prescription drug monitoring programs for prescribing and dispensing decisions: results from a multi-site qualitative study. Res Soc Administrative Pharm 2019; 15(6): 754–760.

14.

Blum

Nelson

Hoffman

. A survey of physicians’ perspectives on the New York State mandatory prescription monitoring program (ISTOP). J Subst Abuse Treat 2016; 70: 35–43.

15.

Carnes

Wright

Norwood

. A qualitative analysis of prescribers' and dispensers’ views on improving prescription drug monitoring programs. Res Soc Admin Phar 2017; 13(6): 1167–1174.

16.

Doyle

Leichtling

Hildebran

, et al. Research to support optimization of prescription drug monitoring programs. Pharmacoepidemiol Drug Saf 2017; 26(11): 1425–1427.

17.

Hildebran

Cohen

Irvine

, et al. How clinicians use prescription drug monitoring programs: a qualitative inquiry. Pain Med 2014; 15(7): 1179–1186.

18.

Greenwood-Ericksen

Poon

Nelson

, et al. Best practices for prescription drug monitoring programs in the emergency department setting: results of an expert panel. Ann Emerg Med 2016; 67(6): 755–764. e754.

19.

Ashburn

. The evolution of prescription drug monitoring programs. Pharmacoepidemiol Drug Safety 2016; 25(7): 852–853.

20.

Centers for Disease Control and Prevention . Integrating and expanding prescription drug monitoring program data: lessons from 9 states, 2017.

21.

Weiner

Kobayashi

Reynolds

, et al. Opioid prescribing after implementation of single click access to a State prescription drug monitoring program database in a health system’s electronic health record. Pain Med 2021; 22(10): 2218–2223.

22.

Holmgren

Apathy

. Evaluation of prescription drug monitoring program integration with hospital electronic health records by US county-level opioid prescribing rates. JAMA Netw Open 2020; 3(6): e209085e209085.

23.

Onwuegbuzie

Bustamante

Nelson

. Mixed research as a tool for developing quantitative instruments. J Mixed Methods Res 2010; 4(1): 56–78.

24.

Luyt

. A framework for mixing methods in quantitative measurement development, validation, and revision: a case study. J Mixed Methods Res 2012; 6(4): 294–316.

25.

Witry

Wesely

Goedken

, et al. Development of a medication monitoring attitude measure using a mixed methods item development process. Int J Pharm Pract 2015; 24(1): 49–59.

26.

Sandelowski

. Qualitative analysis: what it is and how to begin. Res Nurs Health 1995; 18(4): 371–375.

27.

Thomas

. A general inductive approach for analyzing qualitative evaluation data. Am J Eval 2006; 27(2): 237–246.

28.

Donabedian

. Evaluating the quality of medical care. Milbank Memorial Fund Quarterly 1966; 44(3): 166–206.

29.

Alley

Novak

Havlin

, et al. Development and pilot of a prescription drug monitoring program and communication intervention for pharmacists. Res Soc Admin Phar 2020; 16(10): 1422–1430.

30.

Leichtling

Hildebran

Novak

, et al. Physician responses to enhanced prescription drug monitoring program profiles. Pain Med 2019; 21(2): e9–e21.

31.

Weiner

Sherritt

Tseng

, et al. Advanced visualizations to interpret prescription drug monitoring program information. Drug Alcohol Depend 2019; 201: 260–265.

32.

Chanas

Volles

Sawyer

, et al. Analysis of a new best-practice advisory on time to initiation of antibiotics in surgical intensive care unit patients with septic shock. J Intensive Care Soc 2019; 20(1): 34–39.

33.

Webster

. Risk factors for opioid-use disorder and overdose. Anesth Analg 2017; 125(5): 1741–1748.