Abstract
Objectives:
Quality improvement is a critical mechanism to manage public health agency performance and to strengthen accountability for public funds. The objective of this study was to evaluate a relatively new quality improvement resource, the Public Health Quality Improvement Exchange (PHQIX), a free online communication platform dedicated to making public health quality improvement information accessible to practitioners.
Methods:
We conducted an internet-based survey of registered PHQIX users (n = 536 respondents) in 2013 and key informant interviews with PHQIX frequent users (n = 21) in 2014, in the United States. We assessed use of the PHQIX website, user engagement and satisfaction, communication and knowledge exchange, use of information, and impact on quality improvement capacity and accreditation readiness.
Results:
Of 462 respondents, 369 (79.9%) browsed quality improvement initiatives, making it the most commonly used site feature, and respondents described PHQIX as a near-unique source for real-world quality improvement examples. Respondents were satisfied with the quality and breadth of topics and relevance to their settings (average satisfaction scores, 3.9-4.1 [where 5 was the most satisfied]). Of 407 respondents, 237 (58.2%) said that they had put into practice information learned on PHQIX, and 209 of 405 (51.6%) said that PHQIX had helped to improve quality improvement capacity. Fewer than half of respondents used the commenting function, the Community Forum, and the Ask an Expert feature.
Conclusions:
Findings suggest that PHQIX, particularly descriptions of the quality improvement initiatives, is a valued resource for public health practitioners. Users reported sharing information with colleagues and applying what they learned to their own work. These findings may relate to other efforts to disseminate quality improvement knowledge.
Quality improvement is a critical mechanism to manage public health agency performance and to strengthen accountability for public funds, 1 and it is a central component of the accreditation process developed by the Public Health Accreditation Board. 2 However, the public health workforce may lack sufficient training and experience in quality improvement to implement programs as needed across governmental public health systems. 3
Opportunities to enhance quality improvement in practice include the development of in-depth resources and networks of professionals to support quality improvement learning and implementation. 4 Launched in 2012, the Public Health Quality Improvement Exchange (PHQIX), funded by the Robert Wood Johnson Foundation (RWJF), is a free and openly available online communication platform dedicated to enhancing the performance, effectiveness, and efficiency of public health systems by providing in-depth stories about quality improvement as well as an online mechanism for interacting with other quality improvement professionals, known as a community of practice.
The objective of this work was to describe the main results of an early evaluation of PHQIX conducted in fall 2013. PHQIX is one of a small number of major initiatives conducted by nonprofit organizations 5 –7 and governmental public health entities 8 to support training in quality improvement and dissemination of quality improvement resources to public health practitioners.
Program Description
PHQIX (www.phqix.org) describes quality improvement efforts in a standardized format so that practitioners and others can find real-world examples relevant to their quality improvement work. To encourage interaction and collaboration among public health professionals, the website hosts a Community Forum for dialogue among site users, the ability to comment on and pose questions about quality improvement initiatives, and an Ask an Expert feature in which quality improvement experts answer questions posed by site users. PHQIX publishes a monthly newsletter, and the website offers periodic webinars on quality improvement topics. Users do not need to register to view content, but they may wish to do so to post a comment, use the Ask an Expert feature, or receive the newsletter. PHQIX is promoted primarily through booths at national public health meetings, internet-based activities, and RWJF.
Purpose of Evaluation
The purpose of this evaluation was program improvement. We developed a program logic model (Figure) and draft evaluation questions to guide the evaluation—with input from RWJF, the National Network of Public Health Institutes (NNPHI), and the project advisory committee, which comprised national experts in quality improvement and public health practice. The evaluation focused on the following priority areas: indicators of website use, user engagement and satisfaction, content of the quality improvement initiative descriptions submitted by practitioners, interactions (communication, collaboration, and knowledge exchange) among users, and use of information from PHQIX. We asked users about PHQIX impact on quality improvement capacity and accreditation readiness, although we did not expect to see any effect only 1 year after program implementation.
Program logic model for the Public Health Quality Improvement Exchange (PHQIX). Abbreviations: ASTHO, Association of State and Territorial Health Officials; NACCHO, National Association of County and City Health Officials; NNPHI, National Network of Public Health Institutes; QI, quality improvement; RTI, Research Triangle Institute; RWJF, Robert Wood Johnson Foundation.
Methods
We collected data using an internet-based survey of registered website users, key informant interviews with a small group of PHQIX frequent users, information from program records, and analysis of website data. We drew extensively from literature on quality improvement and adopted questions from national surveys of public health agencies to compare our findings with other work. Examples of constructs on which we relied for preexisting definitions and/or measurement included current quality improvement activities, 9,10 accreditation status, 9,10 quality improvement capacity, 9,11,12 implementation of quality improvement, 13,14 and accreditation readiness. 9 We reviewed and incorporated items from the National Public Health Improvement Initiative’s Annual Assessment of Performance Management and Improvement Practices. 15 To understand the role of PHQIX in supporting the quality improvement community of practice, we gave survey respondents and interviewees the following definition of quality improvement community of practice: the group of people who share a professional role in public health quality improvement and who are connected by the goal of gaining knowledge related to this field; this includes registered users of PHQIX, visitors to the PHQIX site, RWJF grantees (Community of Practice for Public Health Improvement and the Multi-state Learning Collaborative), National Public Health Improvement Initiative grantees, and all others in the practice of public health quality improvement. The Research Triangle Institute Institutional Review Board evaluated the methods and instruments used for all data collection activities and determined that the evaluation was not human subjects research.
Internet-Based Survey Methods
We conducted an internet-based survey with a volunteer sample from registered PHQIX users, limiting eligibility to those employed at federal, state, local, tribal, or territorial public health agencies, because they are the primary target audience for PHQIX. We developed the survey in consultation with RWJF and NNPHI. We also excluded members of the PHQIX team, staff members from RWJF and NNPHI, and people affiliated with PHQIX as members of the User Group or Advisory Committee. The Research Triangle Institute team extensively tested the survey, and we conducted the survey using SurveyMonkey. 16 The estimated time to complete the survey was 15 minutes. Three members of the PHQIX user group (ie, a group of practitioners who had helped test versions of the website) piloted the survey and then made minor clarifications to several questions.
After posting a notice on the PHQIX website and in the monthly newsletter, we sent the survey to all registered PHQIX users via email. We could not use registration data to establish eligibility, because nearly half of registered users did not name their worksites when they registered. Therefore, we emailed an invitation to participate in the survey to all registered users, informed them of the eligibility criterion, and asked them to self-identify eligibility. As an incentive for participation, we gave respondents a chance to win 1 of 5 iPads at the close of the survey. To further improve the response rate, we set a clear deadline and sent out 3 reminders at weekly intervals during the survey period (4 weeks during November and December 2013). We included in the survey a question asking for permission to be contacted to participate in key informant interviews.
Key Informant Interview Methods
We conducted interviews with 21 key informants who were frequent users of the site or frequently engaged with the PHQIX community, as defined later. The interviews facilitated an in-depth exploration of the survey topics and included additional questions not covered in the survey. The interviews included “how” questions (eg, how did PHQIX facilitate communication among community-of-practice members)?
The Research Triangle Institute developed the interview questions in consultation with RWJF and NNPHI, and we piloted it with 2 members of the user group. We sent a recruitment letter to 136 people identified as frequent users through our review of website use data or their responses to the PHQIX survey, offering a $50 gift card to people who participated in a 1-hour interview. We enrolled the first 21 volunteers as participants.
We conducted interviews in January 2014, and each interview lasted about 1 hour. The interview questionnaire consisted of semistructured questions, which we tailored using individual responses to the survey questions before the interview. One of 3 Research Triangle Institute team members conducted each interview, and a second team member recorded the conversation and took typewritten notes. We used audio recordings to fill any gaps in the notes. We organized a matrix of de-identified interviewee responses by research question and reviewed it for content themes and relevant quotes.
Results
Respondents
Response Rate
The survey was emailed to 1016 potentially eligible registered users of PHQIX. Of the 1016 users contacted, 609 (59.9%) responded, 73 of whom did not meet the eligibility criteria, leaving 536 eligible respondents. We do not know how many eligible people received the survey, so we could not calculate the response rate among eligible people. However, we could estimate the response rate by examining the subset of 554 registered users for whom workplace information, provided as part of their registration data, indicated that they met the eligibility criterion. Of these 554 eligible users, 336 (60.6%) responded to the survey.
Characteristics of Survey Respondents
Respondents had an average of 13.4 years of public health practice experience and 10.1 years at their current organizations. Of 517 respondents, 363 (70.2%) had recent (within 2 years) training in quality improvement (Table 1). Most respondents worked at either local health departments (n = 397, 74.1%) or state health departments (n = 111, 20.7%); the remainder worked at federal, tribal, and territorial public health agencies. The northeast region was less well represented than other regions (n = 45 of 499, 8.4%). Respondents’ organizations had high levels of (1) quality improvement activity, with 368 of 472 (78.0%) respondents implementing at least quality improvement in programmatic areas, and (2) interest in accreditation, with 215 of 463 (46.4%) having submitted a statement of intent, filed an application, or achieved accreditation (Table 2).
Characteristics of 536 registered users who responded to a survey on use of and engagement in the Public Health Quality Improvement Exchange,a United States, 2013
Abbreviations: NNPHI, National Network of Public Health Institutes; QI, quality improvement.
aThe Public Health Quality Improvement Exchange is a free online communication platform dedicated to making public health QI information accessible to practitioners.
bValues presented as No. (%) unless noted otherwise. Percentages may not total to 100 because of rounding.
cMean (standard deviation).
Characteristics of the organizations represented by respondents (n = 536) to a survey of registered users of the Public Health Quality Improvement Exchange organizations,a United States, 2013
Abbreviations: FTE, full-time equivalent; QI, quality improvement.
aThe Public Health Quality Improvement Exchange is a free online communication platform dedicated to making public health QI information accessible to practitioners.
bValues presented as No. (%) unless noted otherwise. Percentages may not total to 100 because of rounding.
cUS Census Bureau, “Regions and Divisions,” http://www.census.gov/econ/census/help/geography/regions_and_divisions.html.
dMean (standard deviation).
Characteristics of Interviewees
Of the 20 interviewees, 18 were female. Interviewees had an average of 12.1 years of public health practice experience and 6.7 years at their current organizations. Thirteen interviewees worked at local health departments and 7 at state health departments; 7 respondents reported that the health departments served >1 million people.
User Engagement and Satisfaction
The most commonly used website features among 462 respondents were browsing through quality improvement initiatives (369, 79.9%), searching for initiatives (224, 48.5%), finding additional quality improvement resources (186, 40.3%), and reading the Community Forum (175, 37.9%) and Ask an Expert (116, 25.1%) pages. Only 57 (12.3%) respondents had posted to the Community Forum, 31 (6.7%) had posted a comment elsewhere on the site, and 12 (2.6%) had posted a question to the Ask an Expert page. Additionally, 105 of 462 (22.7%) respondents reported submitting a quality improvement initiative to PHQIX. Of 412 respondents, a small number followed PHQIX on Twitter (54, 13.1%) or LinkedIn (18, 4.4%; Table 3).
User engagement and satisfaction with PHQIX,a by site characteristics, from a survey of 536 registered users, United States, 2013
Abbreviations: PHQIX, Public Health Quality Improvement Exchange; QI, quality improvement.
aThe Public Health Quality Improvement Exchange is a free online communication platform dedicated to making public health QI information accessible to practitioners.
bValues presented as No. (%) unless noted otherwise.
cMean score (standard deviation) on a scale of 1 to 5 (1 = not at all satisfied, 5 = very satisfied).
dMean score (standard deviation) on a scale of 1 to 5 (1 = not at all useful, 5 = very useful).
Overall, respondents were satisfied with the quality improvement initiative descriptions and website functionalities. On a scale of 1 to 5 (with 5 being very satisfied), respondents gave an average rating >4 for the following: breadth of topics, types of quality improvement methods, overall quality of quality improvement initiatives, write-ups (organization, clarity, and comprehensiveness), and innovativeness and usefulness of information. Respondents were also satisfied with the usefulness of features such as the Search function (4.0), Community Forum (3.9), ability to comment on posts and initiatives (3.9), and Ask an Expert (3.9; Table 4).
Use of information from PHQIXa by respondents to a survey of 536 registered users, United States, 2013
Abbreviations: PHQIX, Public Health Quality Improvement Exchange; QI, quality improvement.
aThe PHQIX is a free online communication platform dedicated to making public health QI information accessible to practitioners.
Interviewee responses on satisfaction with the website were similar to responses of the survey respondents. Most interviewees indicated that the quality improvement initiative descriptions, Ask an Expert, and Community Forum were the most helpful features for practitioners. Several respondents also noted the importance of the ability to review and share information with other health departments; the more practical the approach, the better. Interviewees indicated that even more examples of the practical tools would be welcome—especially storyboards, charts, and diagrams. Several interviewees mentioned that PHQIX’s exclusive focus on quality improvement set it apart from other sources of similar information.
Impact on Quality Improvement Capacity, Implementation, and Accreditation Readiness
Important goals of PHQIX are to increase quality improvement capacity, quality improvement implementation, and accreditation readiness. Survey results suggest high rates of use and sharing of PHQIX information (Table 4). Of 407 respondents, 10 (2.5%) and 21 (5.2%) reported adopting and adapting quality improvement initiatives, respectively, but 237 (58.2%) reported putting information learned on PHQIX into practice. The most common types of information put into use were a general approach to a quality improvement problem (150, 36.9%), a quality improvement method (85, 20.9%), or a quality improvement tool (123, 30.2%). A total of 275 (67.6%) respondents reported sharing information learned on PHQIX. The most common reasons for sharing information included informal dissemination to staff members within the organization (190, 46.7%), using information to influence leadership (83, 20.4%), and mentoring others in quality improvement (75, 18.4%). Forty-six (11.3%) respondents had shared information from PHQIX with colleagues outside their organizations. Respondents reported that PHQIX helped them personally perform their quality improvement roles (57%) or branch out to perform new ones (29%).
Interviewees gave examples of how they shared information learned on PHQIX. Most sharing was done informally with internal teams. In some cases, initiative examples or tools were shared as a group in a meeting; in others, links to examples were shared with disease-specific team leaders. Often, content-specific information was directed to program staff members, as when HIV quality improvement examples were forwarded to an HIV team. Another interviewee indicated looking for examples of approaches for developing a quality improvement initiative.
Of 405 respondents, 209 (51.6%) and 130 (32.1%) reported that PHQIX had increased quality improvement capacity and implementation, respectively (Table 4). PHQIX reportedly had an effect on multiple aspects of organizational capacity; for example, of 207 respondents, 160 (77.3%) reported that PHQIX had improved the workforce and human resources of their organizations in terms of quality improvement. Of 405 respondents, 113 (27.9%) said that PHQIX had increased accreditation readiness.
Data from the interviews mirrored the survey findings. Twenty interviewees indicated that PHQIX helped influence organizational culture to facilitate quality improvement implementation by providing motivation and by lending structure and direction to their quality improvement efforts. According to 1 interviewee, recognition by being published on PHQIX had a positive effect on the quality improvement culture because published initiatives are shared internally and other groups are encouraged to submit their quality improvement stories to PHQIX. Five interviewees commented that the formal process of submitting quality improvement initiative descriptions on PHQIX led to capacity improvements in workforce and human resources. Nine interviewees cited PHQIX as being important in helping them develop their first quality improvement plans.
Supporting the Community of Practice
Most survey respondents (379 of 412, 92.0%) indicated that PHQIX supports the community of practice. Reviews of the open-ended comments from the survey asking how PHQIX supports the community of practice described information sharing on the website, the newsletter, email communications, the Community Forum, Twitter, and the presence of PHQIX at conferences. Many interviewees said that PHQIX is only beginning to leverage its role in facilitating a quality improvement community of practice. Several interviewees said that involving experts in the quality improvement field (for Ask an Expert) was a great way to foster communication and collaboration among the community of practice. The qualitymatters newsletter was an important resource for 19 interviewees. Interviewees appeared to play a key role in disseminating quality improvement information within their health departments, and the newsletter represented an opportunity to raise awareness among colleagues and staff members.
Suggestions for Program Improvement
The interviews provided insights into the types of information that practitioners wished to receive and how they wished to receive it. Interviewees suggested that PHQIX should keep information quality and focus foremost in its design, as well as speed and efficiency of information retrieval. Many interviewees indicated that more pushes of information from PHQIX would be helpful, especially tailored information or information specific to an individual’s quality improvement role or domain.
Lessons Learned
The primary goal of our evaluation was to improve PHQIX, but findings may inform other agencies and organizations—such as the Public Health Foundation, 17 National Association of County and City Health Officials, 18 NNPHI, 19 and the Centers for Disease Control and Prevention 20 —about actionable information to support quality improvement practice.
The main findings suggest that PHQIX was a valued resource among users. The quality improvement initiative descriptions were the most valued aspect of the site, and PHQIX was a distinctive source for these real-world examples of quality improvement. However, the settings and methods of the examples lacked variety. Findings also suggested that, even at this early stage of development, PHQIX supplies information that practitioners are using and sharing and that it can influence the downstream goals of increasing quality improvement capacity and implementation as well as accreditation readiness.
PHQIX’s role in supporting a quality improvement community of practice is evident, but it can be strengthened. For example, features that could support the community of practice (eg, Community Forum and Ask an Expert feature) were valued but accessed less frequently than other features. The data also suggest that the initiative’s reach can be expanded because users appeared to be experienced in public health and quality improvement, according to characteristics such as self-report of quality improvement training, quality improvement role in the health department, and status of quality improvement and accreditation in the organization.
These findings have implications for the PHQIX program and other initiatives engaged in similar work. Changes in the PHQIX initiative during its first year included expanding its focus on the newsletter, tailoring a shortened version of the newsletter to specific domains, and revising the submission review process and online search features. Longer-term evolution of the program will include efforts to streamline the submission process, continuation of marketing campaigns with better incentives for engagement on the website (eg, conference attendance), and reduced focus on social media until health departments are more socially connected.
Other governmental and nongovernmental organizations working to enhance quality improvement in public health practice might take away several lessons. In particular, the interviews provided insights into the types of information that practitioners wished to receive and how they wished to receive it. These insights can inform PHQIX and other dissemination efforts.
Other suggestions from interviewees provided insight into system-level mechanisms to enhance dissemination of quality improvement information. Interviewees requested resources (eg, webinars, trainings, handouts, or briefs) to help quality improvement practitioners educate their colleagues about quality improvement, particularly resources targeted toward leadership or management. This finding highlights the importance of the dissemination of quality improvement information within and across organizations and suggests a role that national organizations can fill. Interviewees recommended that PHQIX continue to work with organizations such as the National Association of County and City Health Officials, the Public Health Accreditation Board, the Centers for Disease Control and Prevention, and state and regional public health organizations to develop and implement technical assistance opportunities. Interviewees recommended that individual programs can amplify their impact by developing complementary resources and coordinating opportunities to reach practitioners.
Finally, interviewees suggested activities that could be implemented by PHQIX or other initiatives to enhance the quality improvement community of practice. Interviewees called for having content-specific updates (eg, quality improvement in maternal and child health programs), newsletters, or other materials to connect people with similar areas of expertise or interest. Suggestions also included developing and disseminating region-specific information and introducing a mechanism to foster communication and collaboration within geographic regions. These findings highlight that public health quality improvement is a relatively new area of practice and an area of rapid growth and that there is a demand for knowledge and an interest in sharing best practices.
Limitations
Our methods had several limitations. First, because the survey was based on a convenience sample, we were unable to assess the extent to which respondents represented PHQIX users or quality improvement practitioners. A high percentage of respondents had quality improvement training or participation in programs such as the Multi-state Learning Collaborative, likely representing practitioners with above-average exposure to and training in quality improvement. Therefore, opinions about the website may not represent those of all PHQIX users or all public health quality improvement practitioners. Second, reports of use of information or adaptation and adoption in the survey may be overestimates, as suggested by a comparison of responses provided in the interviews with those in the survey for the subset of people who participated in both study components. Third, less information was available for some questions than for others because response rates declined somewhat in the latter portion of the survey. Fourth, we were limited by the small number of interviews conducted. Fifth, we selected interviewees based on markers of the high use of PHQIX, which introduced selection bias.
Conclusion
This initial evaluation of PHQIX demonstrated early progress in making public health quality improvement information accessible to public health practitioners, identified areas for growth, and led to recommendations for mechanisms to enhance quality improvement in public health practice.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded under contract to the Robert Wood Johnson Foundation (ID 716 647, Sustaining National Efforts at Quality Improvement in Public Health Through the Public Health Quality Improvement Exchange 2014-2016).