Abstract
The ability to analyze data to identify best practices is key to improving quality of care for community-based health care organizations (CBOs). Leading commercial statistical software remains too costly for many CBOs operating in underserved communities. The St Louis Integrated Health Network (IHN) collaborates with CBOs to increase access to health care. IHN and a local university developed the Community Analytics Academy (CAA), a training collaborative designed to meet the need for data-informed decision making among CBOs. Establishing analytics training collaboratives for CBOs empowers organizations to respond to the ever-growing amounts of health care data and the need for data-driven decision making.
Introduction
Empowering CBO Data Managers
Health care is moving toward a population health management framework, where reimbursement is tied to outcomes and value-based payment initiatives are becoming more common. The ability to analyze available data to identify treatments, practices, and processes that optimize benefit to an organization’s population is paramount to providing cost-effective, high-quality care. 1 The need to manage larger amounts of data, to share and coordinate data between organizations, and to conduct more complex data analysis is increasing.
Many community-based health care organizations (CBOs) experience challenges to training staff in data analysis skills, date presentation, and interpretation. 2 The volume of data, challenges of sharing and coordination, and complexity of predictive analysis require increasingly advanced software with specialized training. Unfortunately, leading commercial statistical software remains too costly for many not-for-profits (NFPs) and CBOs operating in underserved communities.
Community Capacity Building
This encompasses the expansion of skills and expertise by an organization to address the issues they face, such as analytic and data processing capacity. 3 To successfully reduce health disparities and increase organizational sustainability, CBOs need to build capacity by acquiring new skill sets and resources. 4 While the concept of capacity building among CBOs is not new, the ability of a CBO to make data-driven, evidence-based decisions to address health equity issues and improve patient and client services is becoming increasingly critical in the era of increasing data availability and complexity.5,6
Addressing Access Through Education: Need and Desire
For data analytics to be effective, health care organizations need access to the right tools and the right training. Community-based, NFP health care organizations that care for underserved communities and serve as a region’s health care safety net often lack the financial resources to procure tools and training; hence, there is a demonstrated need. The St Louis Integrated Health Network (IHN) has created a collaborative model among CBO leaders, who routinely share concerns and challenges impacting the ability to model and forecast advances in care management; through these meetings, a desire to use data analytics in modeling has been emphasized. Health care safety net organizations employ masters-level individuals who are highly skilled in their field, but a lack of access to technological resources limits their ability to optimize their organization’s data analytics. 7 To address this issue, the Community Analytics Academy was developed as a collaboration between regional safety net health care organizations and university partners, with a goal of training local safety net data analysts in a free software platform with advanced analytic capacity, to facilitate the growing need for data informed decision making.
The Partnership
A local university departments of health policy and data science collaborated with the St Louis IHN. The IHN networks community health centers, hospital systems, academic medical schools, public health departments, and other safety net institutions to increase access to high-quality, affordable healthcare for all residents of Metropolitan St Louis. 7
Methods: Program Overview
All program components were developed using community-based participatory research principles. 8 Participants, course content, and schedule were all informed by the CBO pilot course participants.
Who Are the Targeted Participants and What Are Their Specific Needs?
Participants were data managers and data analysts at NFP health care and public health CBOs, including federally qualified health centers, local health departments, and other health intermediary organizations that serve underserved communities. Participants demonstrated a varied degree of proficiency with statistical analysis and represented a spectrum of experience, longevity, and education levels (see “Experience” in the Appendix A). It was important to respond to each individual’s comfort level with the software chosen, and to provide an inclusive environment that not only facilitated but motivated continued learning as well as to encourage program alumni to provide mentoring to future program participants.
Why R?
The traditional statistical software packages such as SPSS, SAS, and Stata are too costly for many NFPs to purchase 9 ; however, today’s data cleaning and analysis tasks are too difficult or impossible for Microsoft Excel to perform efficiently. R is optimal because it is free, open-source, and robust—since its inception in 1993, a strong, participatory community of scientists and scholars have continually refined it into a tool that is easily adapted to an organization, if the staff has the knowledge and skill to properly utilize it. 10
How Were Participants Recruited?
Recruitment for the CAA pilot R Workshop occurred in a mostly organic way via 3 routes: (1) IHN membership organizations were invited to select staff to attend; (2) The IHN Director of Evaluation, Quality & Learning shared the opportunity with an informal network of peers employed at local NFP public health/community based organizations, who have collaborated over the years to optimize their data collection, analysis, and evaluation processes; and (3) IHN shared a workshop flyer at various advisory meetings within the region’s safety-net community. All participants who wanted to attend were accommodated and accepted into the program.
How Was the Educational Activity Implemented?
IHN and St Louis University stakeholders met regularly over the course of 4 months to develop a training implementation plan. An online survey was first developed and distributed to gauge interest in learning R. IHN acted on behalf of the partnership, leveraging established relationships to distribute the survey as broadly as possible through their existing relationships within the safety net (see Appendix A, “Survey Results”). Responses to the survey indicated a very wide range of experience with analytical software. Furthermore, the survey queried which analytic procedures were needed in day-to-day work and should be taught in the workshops. The content of the workshops was driven by the participants.
Program Participation
Per respondent preferences, sessions were scheduled for Saturday mornings. A total of 12 participants registered to attend one or all the sessions, at a charge of $20.00 per session for instructor compensation. Participants could choose their level of attendance; slides and supporting information was made available to participants who missed sessions for free. The session was taught by a masters-prepared PhD candidate in data analytics with significant training in R, STATA, and advanced analytic procedures, as well as previous experience in conducting research and data analysis related to quality and access to care in safety net health care facilities. Note that the IHN and partnering CBOs advertised the workshop to their staff and stakeholders whom they thought would benefit from the workshop.
What Are the Specific Objectives of the Education/Training?
The training was designed to meet participant preferences, including
how to download free R software and additional plugins,
how to import and clean data, and
how to conduct both basic and multivariate statistical analyses.
Three-hour sessions were held weekly for 4 weeks. Participants were provided exercises to walk through with the instructor for each session’s designated topics of interest, with all materials available prior to the start of the sessions. All sessions included a break, question and answer sessions and time for students to receive help from the instructor. All students brought their own personal laptop.
Prior to the Workshops
Participants were sent instructions via email for downloading materials and software prior to their first session. Instructions were given for using Anaconda to download R, and the open source platform Jupyter Notebook for file and code sharing. Participants brought their own laptops in order to utilize their own organizational data during in-class tutorials, which also allowed them to save any code they would build during class for future use. All workshop materials were made available via an online portal designed by the instructor.
Workshop Structure
An adapted flipped learning model was used to limit in-class lectures, in favor of using the time for exercises with instructor presence and assistance. While the schedule for each workshop had been planned prior to the first session, participant feedback was solicited at the end of each workshop and future course content was adjusted accordingly (Table 1). With only 4 days to deliver the content, the program prioritized introducing attendees to the tools available for working with R. Becoming aware of the extensive R-related help resources available would thus foster self-efficacy in troubleshooting challenges with the application of the tools. There was a strong focus on understanding the strengths and weaknesses of the R software, the importance and depth of the R community, and the overall functionality of the software in addition to the exploration and application of R tools. Each workshop was developed so that new lessons built upon content introduced in previous sessions, allowing an opportunity to review and reinforce prior knowledge as well as exploring new concepts.
Chronological Order of Each Workshop.
Lesson Content
In alignment with the flipped learning model, the workshop was designed to limit total lecturing time to 1 hour at most, with 2 hours of questions and answers (Q&A) and hands-on activities. All workshop materials can be found at https://osf.io/bdzfm/.
A brief syllabus is included in Appendix B. As needed to improve comprehension and retention, the lecture segment would be divided with material pertinent to Activity 2 presented after Activity 1 was completed. The instructor was also able to tailor subsequent classes to fit the needs of the participants as indicated in session feedback responses.
Next Steps
As evidenced by positive evaluations and continued interest, The CAA pilot was a success in terms of participant satisfaction and desire for additional training using the format. There remain, however, opportunities and challenges to its sustainability. While the pay-per-session model covered the instructor’s actual time spent in sessions, lesson preparation in advance of the program required a significant investment. Funding sources including grants are currently being sought. Additionally, researcher partners are investigating the creation of additional R modules, as well as plans for developing pedagogy for other freeware.
While upskilling existing data managers and analytics in CBOs is a necessary first step, building analytics capacity throughout CBOs, including data awareness, use, and integrity is integral to achieving the CBOs’ strategic goals and objectives. We suggest both increasing the skill of CBO staff, whose main task is data analysis and management, and increasing the analytic capability of end users and staff who make data-informed decisions. The latter would involve training of non-analytics focused staff in best practices for data use and integration in daily decision making.
Measures for increasing the sustainability, reach, and effectiveness of the R Workshop are based in part on recommendations derived from organizer and participant experience through feedback and formal evaluations collected after each session and are currently under consideration for implementation. These measures largely refine workshop content and CBO community support for continued training. Additional feedback should be solicited from CBO leadership to ensure the skills learned are retained and to record distribution to nonworkshop attendees within the organization.
Workshop Structure
Offer the introductory workshop annually, with additional single-session workshops for special or advanced topics, and offer various levels to accommodate beginning and more advanced participants with face-to-face troubleshooting.
Explore additional freeware options that support SQL (structured query language), data visualization, and other emerging needs in health analytics.
Expand workshop participants to both data manager and end-user groups so that increased analytic capacity can be disseminated more efficiently throughout participating organizations.
CBO Community Support
As this project was a pilot study, further actions are needed to extend the work and assure integration into CBOs:
Set up coalition of CBO data managers to help guide the CAA, maintain and lead ongoing trainings, and stay abreast of CBO data analysis needs.
Continue to document and manualize training so that it can be replicated in other communities.
Develop ongoing evaluation, including continued use of R in the workplace.
As recommended by participants: “Allow time during trainings for us to bring specific questions from our CBOs to the class so we can get started on our own projects with assistance”.
Develop additional metrics for assessing uptake of newly acquired skills and appropriate timing for booster and additional sessions. Metrics will include organizational readiness to advance organizational capacity for advanced analytics and data-based decision making.
Work directly with CBO leadership to assure ongoing support for ongoing analytics training.
Work directly with CBO leadership to assess post-workshop retention and application of skills learned to organization mission.
In addition, we recognize that to fully understand the impact of the program, longitudinal assessment is necessary to determine how skills acquired in the workshop are practiced in the workplace. For the next iteration of the workshop, we plan to incorporate a one and three-month post-assessment to understand how participants used their new skills in the workplace.
Conclusion
Having audience-appropriate technical training sessions is important and not to be underestimated. It is rare to have this level of training focused on the needs and use cases relevant to CBOs, and the evidence of need and feasibility of such program implementation indicate it is warranted. This is an important gap to fill as CBOs fill an important gap in our communities, especially for serving underserved populations—they too need to be equipped to optimally use advanced tools to mitigate disparities. Initial needs assessment identified that there is a need and desire for such training. Survey results showed that most organization wanted to expand into more advanced statistics application in the future but lack training in these areas. Feedback from sessions showed that the content of this training was relevant to participants’ job and would be beneficial. This project highlights the need for such training, shows that these programs are feasible, and provides a model for reproducibility across the nation.
Footnotes
Appendix A
Appendix B
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.
