A Qualitative Process Evaluation and Quality Improvement of a State Drug Overdose Reporting System Data Collection Approach

Abstract

Tennessee registered the second-highest drug overdose mortality rate of any state in 2022; such deaths have grown by 90% from 2019 to 2022. Tennessee has participated in the State Unintentional Drug Overdose Reporting System (SUDORS) since 2018. An abstraction team synthesizes data for SUDORS from death certificates, autopsies, and other sources. The purpose of this study was to engage in a comprehensive qualitative evaluation of our SUDORS process to distill insights that could improve abstraction speed and quality while reducing abstractor burnout. We conducted 2 rounds of interviews and focus groups with all 7 members of the Tennessee abstraction team in late 2023 and early 2024. The researcher asked questions regarding the adequacy of the current project management approach and team communication level, abstractors’ stress levels, the need for further training, and whether additional data science-based tools could be deployed to increase the speed and accuracy of abstraction. The study yielded several actionable insights for improving abstraction capacity and lessening stress burdens. Accordingly, we made changes to our project management, team communication, and training approaches; worked to better integrate our extant data science tools; and took steps to improve mental well-being. As a result, the average time it takes for an abstractor to enter a case into the NVDRS platform dropped from 12.2 to 6.5 minutes, and all team members noted a decrease in stress levels. The changes made as a result of the findings of this study will help the State keep pace with a high case load and will support abstractors’ mental well-being.

• 1. What do we already know about this topic?

• Extant scholarship emphasizes the importance of efficacious project management and team communication approaches to the success of key objectives; this research applies these findings in the context of public health surveillance.

• 2. How does your research contribute to the field?

• We conducted a qualitative process evaluation of the way the State of Tennessee collects and manages State Unintentional Drug Overdose Reporting System (SUDORS) data; this research has the capacity to inform other public health surveillance teams on the optimum path to improving data quality and increasing overall capacity.

• 3. What are your research’s implications towards theory, practice, or policy?

• Our research findings consist of specific insights in the areas of project management, team communication, training, and data science enhancements that can help improve public health management across a variety of content areas.

Introduction

Fatal drug overdoses continue to pose a major public health threat to the United States (US). Tennessee (TN) registered the second-highest drug overdose mortality rate of any state in 2022. ¹ In addition, such deaths have grown by 90% in TN from 2019 to 2022. ² As a result of this significant public health challenge, TN has been participating in the State Unintentional Drug Overdose Reporting System (SUDORS) since 2018. Under the larger umbrella of the National Violent Death Reporting System (NVDRS), this project collects detailed information on every unintentional and undetermined overdose death in TN. Our abstractors synthesize data from death certificates, autopsies, toxicology reports, and a prescription monitoring database to complete the abstraction process.

In order to improve the speed, quality, and accuracy of our abstraction efforts and keep pace with a rising caseload, the TN SUDORS team has implemented a variety of data science tools. These include a REDCap data collection portal and a suite of R-based scripts that automate approximately 45% of the existing process. Nevertheless, it became clear that a comprehensive qualitative evaluation of the entirety of the team’s project management approach would also be necessary to meet case submission deadlines and to safeguard abstractors’ emotional and mental well-being. On numerous occasions prior to this evaluation project, key deadlines were missed by the team, causing stress levels to rise and continued funding to be jeopardized. In addition, there was an enduring sense that not all abstractors understood the full process, that there was an insufficient degree of day-to-day predictability, and that communication between team members was often inadequate. These challenges made evident the need to re-evaluate our process.

A process evaluation consists of a systematic, qualitative review of an existing program—including interviews with current participants, observations of key activities, and collection of relevant data—and tying the key features of the program to its intended outcomes. ³ The goal is to understand what components of a current program contribute to its success and which detract. Often, these evaluations are attached to community interventions, such as academic detailing, ⁴ but they can be used in any program with a defined process and measurable outcomes, ⁵ including those dedicated to public health surveillance. Accordingly, the purpose of this study was to evaluate the TN SUDORS process and to generate recommendations for improving the long-term sustainability of the program.

Methods

This qualitative process evaluation took place over the period between July 2023 and June 2024. The project commenced with the detailed creation of logic models mapping out the steps of the existing program. Figure 1 illustrates how abstractors synthesize information from a variety of documents to complete the abstraction process into REDCap, how they verify each others’ work, and how they ultimately enter the data in the NVDRS portal; at each step of the process, data science integrations complement the abstractors’ manual entry by automating a percentage of the process. Interview questions focused extensively on each of these program steps in order to determine where successes existed and where shortfalls occurred.

Figure 1.

SUDORS process in Tennessee: a general overview.

The data for the study came from interviews and focus groups conducted with all 7 members of the TN SUDORS abstraction team. One interview was conducted with each abstractor at the beginning of the study period (September and October 2023), as was a focus group with all members of team. A second focus group and set of interviews were conducted toward the end of the study time frame (February 2024) to evaluate improvements made in the intermediary period. This research plan was designed and implemented by the first author, a mixed methods expert and program evaluator. The study was designated “Not Research, Quality Improvement” and was exempted from IRB review by the Tennessee Department of Health Institutional Review Board. All participants provided oral consent to be interviewed, recorded, and quoted in this study. These verbal statements were documented in the researcher’s field notes, including the name, date, and statement being consented to.

The development of the interview questionnaire was informed by the principles of the RE-AIM framework. ⁶ This approach is designed to encourage researchers and evaluators to “pay more attention to essential program elements, including external validity that can improve the sustainable adoption and implementation of effective, generalizable, evidence-based interventions.” ⁷ The RE-AIM framework studies the reach, effectiveness, adoption, implementation, and maintenance of a particular policy or intervention. While primarily designed for the evaluation of public health programs and interventions, we found this approach useful in our surveillance work as well, as it ensured that interviews were conducted in a comprehensive manner. The questionnaire went through a rigorous validation process that included a close review by other Tennessee Department of Health team members.

The interview guide for this study consisted of open-ended questions seeking to identify successes, shortfalls, and challenges in the existing TN SUDORS process, as well as questions aiming to identify potential improvements to our existing approach. In each interview and focus group, abstractors were queried about what works and what does not work, and they were asked to provide their thoughts on the project management approach, extant team communication level and quality, and the efficacy of trainings for new and existing team members. Each was asked to identify factors that contributed to the program success and factors that detracted. In addition, abstractors were asked questions about their emotional and mental well-being, and they were provided space to discuss ways in which challenges related to high stress levels and burnout could be mitigated. While most of the data for this study was drawn from interviews, 2 focus groups were also conducted, as each medium has certain strengths and weaknesses. Interviews provide respondents with a safer space to express themselves in which they are only addressing 1 other individual; however, focus groups enjoy the added benefit of generating interaction between respondents, which can be fruitful in identifying new ideas that individuals might not have considered on their own. ⁸

Once interviews and focus groups were conducted, the audio files were transcribed using either Microsoft Teams or NVivo Transcription, and any errors were then manually corrected. Next, the transcripts were imported into NVivo 14 software for analysis. The principles of grounded theory and inductive reasoning were deployed to generate scientific theory from the collected data. ⁹ The coder, the first author of this study, is a researcher with a doctoral degree trained in qualitative research design and methodologies who began work on this study with limited preconceived notions of the substantive topic or of the way the TN SUDORS team works together. The coder reviewed the transcripts line-by-line and coded nodes that constitute identifiable categories; memos were regularly written to identify new themes and ideas. This iterative process revolves around the search for major patterns, which are typically identified by a particular topic or some surprising new finding. Examples of themes for this study include team communication and the project management approach. Afterwards, the functions of the NVivo software are used to analyze repeated words or phrases that may indicate a pattern. More difficult to spot are deviations from established patterns; this requires the researcher to create a note or memo wherever 1 is spotted. The NVivo software is also used to create hierarchy charts, logic models, and mind maps to better visualize the data and help uncover further insights. Once this iterative process is complete, we assembled key quotes under major thematic headings that best illustrate the causal processes at play.

Results

This study was driven by the principles of phenomenology and action-oriented research. The abstractors were asked about their perceptions of the system in which they were involved, and the findings were in turn used to engender practical and actionable change. Each component of the RE-AIM framework (reach, effectiveness, adoption, implementation, and maintenance) was utilized to engender a rich discussion of each of the process steps outlined in Figure 1. The principal finding was not that any 1 of the steps was breaking down or particularly lacking in the way it was implemented, but rather that at each step of the way, repeated shortcomings and errors were leading to major challenges that continued to snowball and exacerbate over time.

Several major themes emerged from the interviews and focus groups that pointed to potential ways of improving the process. New approaches to project management, team communication, and abstractor training were proposed. In addition, the team members also proposed potential new data science automations and identified areas where automated imports were incorrectly populating into the REDCap data collection portal. Finally, respondents highlighted ways to lessen the mental health burden of abstraction, focusing on improved communication and more manageable work burdens.

Recent SUDORS submission cycles have seen a large and growing caseload, and the Centers for Disease Control and Prevention (CDC)’s guidance is constantly evolving. As a result, it is necessary to manage the workflow effectively to ensure accuracy, meet deadlines, and reduce the risk of burnout. In the first round of interviews, abstractors emphasized the need for cases to be assigned on a weekly basis to make the workload more manageable and predictable; our team implemented this change afterwards to great success, as noted in this second-round interview:

So the assignment of cases I really like. . . you know what the previous process was like, but yeah, it’s just that big Excel sheet where we would all go and just self-assign cases and it wasn’t the best. . . So this new system that we have with the cases assigned on a weekly basis, I really do like that. I find that it’s overall just a better system, you know, be it in a way to like allows you to be held accountable for your own cases, you know. Because before we would just say, “Hey, make sure you aim to get this many abstracted, this many verified.” But you know, the last submission that we had that was in August, there were a lot of people that I don’t think were able to meet their self-assigned numbers. So having it this way I think is going to be better.

In Tennessee, autopsies that come from certain forensic centers contain much more detail than those that come from other centers. In some cases, abstracting a case from Forensic Center A could take 3 times longer than those from Forensic Center B. Consequently, a major finding has been that these more difficult Forensic Center A cases should be evenly interspersed throughout all cases in the REDCap portal to ensure that no 1 team member gets mired in a set of difficult abstractions and falls behind, something the second-round interviews confirmed constituted a helpful new policy.

Some additional suggestions to improve project management and team communication have involved streamlining the meeting schedule, maintaining a working document for troubleshooting, and scheduling regular team-building activities. Given the complexity and intensity of thought that the abstraction process demands, it is important that team members’ processes do not get interrupted by unnecessary meetings. As a result, we made non-SUDORS meetings optional for abstractors, but clarified that they are welcome to attend them. We also began maintaining a working document in a shared drive with the purpose of providing the team with an information hub where they can remind themselves of common errors and points of misunderstanding related to the CDC’s SUDORS guidance; this complements abstractors’ existing ability to ask questions using Microsoft Teams:

Yeah, I love using Teams. I think having that instant messaging between us is helpful. . . we still do have our Monday and Friday meetings where if there’s like a larger issue at hand or something that we want to go over or clarify. . . I don’t personally think we need more than that. . . I do like the idea of though, like a working document because. . . Like I’ll come across a certain situation that’s just a bit unique. . . and having that working document there can really help quickly remind myself what to do.

In addition, regular opportunities for non-work-related team-building activities can enhance team communication, particularly in a remote working environment. Examples of such activities include regularly-scheduled team coffee chats or games where discussion surrounding work is kept to a minimum. Beyond enhancing team cohesiveness, these activities can also support abstractors’ mental and social well-being.

Respondents also emphasized that they would like to see improvements made to the process for training and onboarding new abstractors. In the past, new team members were asked to begin by reading the SUDORS and NVDRS manuals, which are 158 and 271 pages in length, respectively. Abstractors reported that this left them feeling confused and overwhelmed. Team members proposed that shadowing abstractors in their daily work and abstracting test cases would constitute a better introduction to the SUDORS process:

The manuals are super, super long and you’re kind of just told, hey, read the manuals. . . and it just kind of makes it seem like the whole thing is scarier than it actually is. . . it would be better if we could like, you know, watch a training video or shadow [another abstractor] or even have some test cases that we could practice on.

In addition, abstractors provided information related to specific import errors in the REDCap data collection portal, as well as ideas for particular variables that currently require manual entry that could potentially be automated by R code. These were valuable insights that helped improve the cohesiveness between the qualitative abstraction work and the data science tools deployed to accelerate and strengthen the sustainability of the process. Over the course of the research period, the abstractors were able to benefit from learning both about the data science approaches used by our team and about the end-to-end TN SUDORS process, in which they play a pivotal role. This resolved existing misconceptions and greatly strengthened the abstractors’ overall understanding of our extant procedures, as well as of what is possible to accomplish automatically.

Finally, abstractors indicated that the work they do is mentally draining and that they often struggle with feelings of burnout. These arise both from the heavy workload and the challenging content that they consume on a daily basis. Respondents emphasized that making enhancements to the project management approach, as previously discussed, would greatly help in reducing their stress levels. Furthermore, opportunities for breaks, whether in the form of a coffee chat with other team members, a meditation session, or simply a walk outside are greatly welcomed. In the first-round interviews, they acknowledged that there is little that can be done to alleviate the burden of reading multiple autopsies and death certificates every day; nevertheless, a few best practices were distilled. First, there should be more frequent recognition by others on the team of the difficult nature of the work that abstractors perform. Second, cases involving children are particularly challenging, and no team member should abstract multiple of these in 1 day. Finally, any pictures should be kept out of the process:

I’m pretty much desensitized to it, but it is mentally draining. Some of them are just kids. . . sometimes after those cases, I just sort of have to sit down and say ok, or maybe go to the park for a walk. . . I know it’s hard, but I wish we could have some sort of more open discussion of mental health and all that with our team. . . I definitely never want to see any autopsy pictures ever.

Second-round interviews on this topic continued to emphasize the need to address mental health challenges, but all abstractors did note that stress levels were declining as a result of the increased predictability of the weekly case assignments. In addition, perhaps due to their increased familiarity and comfort level with the interviewer, they were more willing to say that abstraction itself is mentally harmful:

I would say we’re at a point now where we can feel pretty confident. . . that the end of the submission cycle isn’t gonna be a mad rush to the finish line with crossed fingers and toes. . . We as abstractors know what to expect from week to week and cases get reassigned if someone is out. . . it’s definitely a better process than what we had before. . .

But it’s still so hard to just have to sit there, day in day out, having to read about death for a living.

Discussion

The present study utilized interviews and focus groups with members of the TN SUDORS abstraction team to distill best practices and to find ways to strengthen the overall process. The themes and codes identified—including new project management approaches, team communication, training, improved integration of data science automations, and emotional health—largely aligned with existing scholarship on these topics and allowed our team to make major and lasting improvements to our approach.

The implementation of weekly case assignments constituted 1 of the most impactful changes we made to our SUDORS process as a result of the qualitative findings. Along with enhancements to our data science tools, this change is now allowing us to meet project deadlines without the undue stress or workloads that used to characterize the end of each submission cycle. This change resulted from the increased predictability of the process at the individual level, alongside improvements in focus arising from the elimination of nonproductive multitasking—including attending unnecessary meetings, which disrupt abstractors’ ability to enter and maintain flow states.^10,11 Coupled with other project management enhancements, these changes addressed extant weaknesses and increased the levels of overall clarity, planning, and organization. ¹²

A large and growing body of literature finds that team communication is a major component of overall team effectiveness. ¹³ This is even more relevant for remote and distributed teams, whose members may struggle with uncertainty and a lack of clarity. Existing guidance regarding the management of remote teams highlights the prime importance of being transparent in communication, providing timely feedback, utilizing a variety of tools, establishing clear deadlines and expectations, scheduling team-building activities that foster community, providing access to other team members, and embracing flexibility.^14,15 These elements are also the building blocks of a coherent and efficacious SUDORS process.

In line with both the extant literature on the subject of employee trainings and the interview findings for this study, we modified our training procedures to be more dynamic and embracing of experiential learning. Effective training and onboarding procedures prioritize “hands-on” learning experiences, mentorship, and opportunities to shadow experienced team members. ¹⁶ Our team anticipates that these changes will result in both faster training and reduced stress for new abstractors in the future.

A final and particularly impactful outcome of this study has been an increased understanding of the role played by mental health concerns in the TN SUDORS process. Abstractors explained on numerous occasions that while they understand that relatively little can be done to mitigate the emotional impacts of their daily reading material, they would benefit greatly from a more manageable workload and a greater team-wide acknowledgement and discussion of mental well-being. In a 2022 review article by Edú-Valsania, Laguía, and Moriano, the authors emphasize that “Work environments with excessive work schedules and high levels of demands. . . leave workers emotionally drained, cynical about work, and with a low sense of personal accomplishment.” ¹⁷ They discuss numerous findings that highlight the efficacy of diminished workloads on improved mental health. In addition, the authors note the potential for exercise and mindfulness to lessen emotional burdens during the workday. These findings from the literature coincide with the insights drawn from the in-depth interviews with the TN SUDORS team. By implementing some of the best practices distilled by this study, such as providing opportunities to de-stress and working to lessen and better manage workloads, teams can reduce burnout and team attrition and improve mental wellness and retention. ¹⁸

The implementation of these new findings into our existing approach, coupled with better integration of our data science tools, greatly improved the sustainability of the overall SUDORS process. The time it now takes a TN SUDORS abstractor to fully enter a case into the NVDRS platform has dropped from an average of 12.2 to 6.5 minutes, and 100% of team members reported decreased stress levels and improved well-being over the course of the study period.

Implications for Practice

The lessons of this research study are applicable not only to the TN SUDORS team but—through a close understanding of our process—can be transferred to and applied by others doing public health surveillance work, particularly other states’ SUDORS and NVDRS teams. Relatively simple reforms have the power to improve capacity, accuracy, and speed, as well as to reduce team member burnout and enhance mental well-being.

Specifically, teams should ensure that their members have a mechanism to apportion their daily and weekly workloads in a manner that is far, equitable, and predictable. They should work to improve team communication and training while also reducing the number of distractions, including unnecessary and unhelpful meetings that disrupt workflow. Increased attention to, and discussion of, mental health issues is also an important step toward supporting the team’s emotional well-being. Evaluating each of these improvements after they have been made will provide further feedback and clarity, helping leaders optimize their existing approaches.

Limitations

The study participants were relatively few in number (N = 7) and all work for the TN Department of Health. In addition, they all knew each other prior to the commencement of the study and engaged in discussions frequently with 1 another regarding their SUDORS work. As a result, the range of perspectives was limited at times. Nevertheless, team members regularly proposed creative and innovative solutions to existing problems, likely as a result of the high caseloads in TN and out of a desire among all abstractors to see a better process implemented. In addition, there were occasionally disagreements between team members about optimal solutions; these were discussed and addressed in a collaborative manner with the goal of arriving at an agreement that gave all abstractors flexibility to complete their work in the manner that is most optimal to them.

Conclusions

Interviews and focus groups with the TN SUDORS abstraction team yielded creative findings in the areas of project management, team communication, training, the integration of data science tools, and mental health and wellness. The implementation of these findings helped enhance the sustainability of the TN SUDORS process. Other jurisdictions could benefit from the transferability of these best practices, particularly those working in public health surveillance. Future studies should examine these findings in the context of other areas of public health practice.

Supplemental Material

sj-docx-1-inq-10.1177_00469580241298145 – Supplemental material for A Qualitative Process Evaluation and Quality Improvement of a State Drug Overdose Reporting System Data Collection Approach

Supplemental material, sj-docx-1-inq-10.1177_00469580241298145 for A Qualitative Process Evaluation and Quality Improvement of a State Drug Overdose Reporting System Data Collection Approach by Mircea Lazar, Joshua Jayasundara, Jessica Korona-Bailey and Sutapa Mukhopadhyay in INQUIRY: The Journal of Health Care Organization, Provision, and Financing

Supplemental Material

sj-docx-2-inq-10.1177_00469580241298145 – Supplemental material for A Qualitative Process Evaluation and Quality Improvement of a State Drug Overdose Reporting System Data Collection Approach

Supplemental material, sj-docx-2-inq-10.1177_00469580241298145 for A Qualitative Process Evaluation and Quality Improvement of a State Drug Overdose Reporting System Data Collection Approach by Mircea Lazar, Joshua Jayasundara, Jessica Korona-Bailey and Sutapa Mukhopadhyay in INQUIRY: The Journal of Health Care Organization, Provision, and Financing