Abstract
Background/aims:
The Food and Drug Administration’s final rule on investigational new drug application safety reporting, effective from 28 March 2011, clarified the reporting requirements for serious and unexpected suspected adverse reactions occurring in clinical trials. The Clinical Trials Transformation Initiative released recommendations in 2013 to assist implementation of the final rule; however, anecdotal reports and data from a Food and Drug Administration audit indicated that a majority of reports being submitted were still uninformative and did not result in actionable changes. Clinical Trials Transformation Initiative investigated remaining barriers and potential solutions to full implementation of the final rule by polling and interviewing investigators, clinical research staff, and sponsors.
Methods:
In an opinion-gathering effort, two discrete online surveys designed to assess challenges and motivations related to management of expedited (7- to 15-day) investigational new drug safety reporting processes in oncology trials were developed and distributed to two populations: investigators/clinical research staff and sponsors. Data were collected for approximately 1 year. Twenty-hour-long interviews were also conducted with Clinical Trials Transformation Initiative–nominated interview participants who were considered as having extensive knowledge of and experience with the topic. Interviewees included 13 principal investigators/study managers/research team members and 7 directors/vice presidents of pharmacovigilance operations from 5 large global pharmaceutical companies.
Results:
The investigative site’s responses indicate that too many individual reports are still being submitted, which are time-consuming to process and provide little value for patient safety assessments or for informing actionable changes. Fewer but higher quality reports would be more useful, and the investigator and staff would benefit from sponsors’“filtering” of reports and increased sponsor communication. Sponsors replied that their greatest challenges include (1) lack of global harmonization in reporting rules, (2) determining causality, and (3) fear of regulatory repercussions. Interaction with the Food and Drug Administration has helped improve sponsors’ adherence to the final rule, and sponsors would benefit from increased communication with the Food and Drug Administration and educational materials.
Conclusion:
The goal of the final rule is to minimize uninformative safety reports so that important safety signals can be captured and communicated early enough in a clinical program to make changes that help ensure patient safety. Investigative staff and sponsors acknowledge that the rule has not been fully implemented although they agree with the intention. Clinical Trials Transformation Initiative will use the results from the surveys and interviews to develop new recommendations and educational materials that will be available to sponsors to increase compliance with the final rule and facilitate discussion between sponsors, investigators, and Food and Drug Administration representatives.
Introduction
The US Food and Drug Administration (FDA) published a “final rule,” effective from 28 March 2011, clarifying the reporting requirements for serious and unexpected suspected adverse reactions occurring in clinical trials conducted under an investigational new drug (IND) application (21 Code of Federal Regulations part 312) 1 and a related final guidance in 2012. 2 The rule 1 was intended to improve the quality of safety reporting by reducing the number of uninformative reports generated by trial sponsors, allowing for easier and quicker detection of true safety signals.
The final rule clarified that sponsors should only generate expedited safety reports for individual cases of serious, unexpected, and suspected adverse reactions when the relationship of the drug to the adverse event (AE) was clear because the event was one that is almost always drug-related (e.g. Stevens–Johnson Syndrome). In other cases, it implicitly required the sponsor to review safety data collected across all studies in an IND, analyze these data in the aggregate, and make a judgment whether there is a reasonable possibility that the drug caused the serious AE.
The Clinical Trials Transformation Initiative (CTTI), a public–private partnership that aims to identify and promote practices that will increase the quality and efficiency of clinical trials, undertook two projects to address the challenges associated with expedited safety reporting.3,4 The first project developed recommendations
5
that offered an approach for sponsors to monitor the safety of an IND throughout the development program. FDA audit data of oncology trials has indicated no change in report volume, despite FDA guidance and CTTI recommendations, as illustrated in Jarow et al.:
6
… in a random audit of expedited safety reports, only 14% (22/160) met the criteria of serious, unexpected, suspected adverse reactions with the remainder not providing any useful information for understanding the safety profile of the investigational drug.
The second project (“IND Safety Advancement”) 4 focused specifically on safety reporting in oncology trials to determine barriers to and identify solutions for compliance with the final rule. The IND Safety Advancement Project aimed to identify (1) sponsor challenges to full implementation and motivation to change practice in order to fully comply with the final rule and (2) challenges to investigator receipt and management of safety reports. The goal of this study was to gather evidence in order to characterize these challenges. The results described herein will be used to develop resources to support sponsors and investigators in creating the most efficient, informative, and high-quality safety reporting system possible.
Methods
Oncology trials were the focus of investigation because of the magnitude of issues with safety reporting in this therapeutic area. 6 In this article, “safety reports” refer to expedited (7- to 15-day) IND safety reports that are submitted to the FDA and clinical trial investigators. Safety terminology is provided in Table 1.
Safety reporting definitions. a
Surveys
The project conducted two online surveys to assess challenges and motivations related to management of IND safety reporting processes; they were intended to gather opinions rather than provide quantitative conclusions. One survey targeted clinical trial sponsors, and one survey targeted investigative staff conducting oncology clinical trials. CTTI members and contacts identified by the project team were invited to participate. Investigators’ responses were pooled and reported separately from the pooled results of “other trial staff.” Survey recipients were able to freely forward it to others; therefore, no data are available describing the number of potential respondents who had access to complete the survey. The complete surveys for investigators and sponsors are available in the online supplementary material.
Survey questions were created and reviewed by members of the CTTI IND Safety Advancement Project Team (see section “Acknowledgments”). The survey was pilot-tested on a subset of the intended population; however, no formal validation or internal consistency checks were performed. The survey consisted of a combination of demographic (Table 2) questions and questions aimed at understanding how IND safety reporting is conducted at their site. The survey tools were not configured to require a response to all questions in order for the survey to be completed and submitted. Participation was voluntary, anonymous, and uncompensated. The survey was distributed via Constant Contact® and completed through Qualtrics Version, November 2014 (Qualtrics, Provo, UT). Survey data were collected from 18 November 2014 through 30 December 2014. Data were then aggregated by the Duke Center for Learning Health Care and distributed to project team members for descriptive analysis.
Roles and clinical trial experience of investigative staff survey respondents.
Interviews
A total of 20 in-depth interviews were also conducted to gather qualitative information about safety reporting processes. The CTTI project team nominated interview participants who were considered to have extensive knowledge of and experience with the topic. Survey respondents were also able to volunteer for interview participation. In January and February 2015, 13 principal investigators and/or other research staff working on oncology clinical trials and 7 pharmacovigilance leaders from 5 large global pharmaceutical companies were interviewed. The duration of each interview was approximately 1 h and was conducted by a professional interviewer using a pre-specified interview guide. Sponsor interview questions focused on implementation of the final rule by the sponsor, changes resulting from the final rule, and ways that the expedited safety reporting system can be improved or clarified. Investigator interview questions focused on the use of expedited safety reporting at the interviewee’s site and ways that the process can be improved. The interview questions for investigators and sponsors are available in the supplementary material.
The surveys and interviews were designated as exempt research by the Duke University Institutional Review Board.
Expert meeting
CTTI products represent a consensus opinion among selected experts in a particular field. In the case of the IND safety reporting, experts included investigative staff, sponsor representatives, and FDA members. Survey and interview responses were reviewed and discussed by the CTTI project team and CTTI-affiliated experts at an expert meeting held on 21–22 July 2015. 8 The method for building consensus follows similar guidelines as those used by the American Heart Association, 9 which encourages inclusiveness, accountability, and commitment to implementation. Once consensus is achieved, these expert opinions are integrated into CTTI products. Following release, CTTI hosts educational lectures on how to implement the recommendations. When appropriate, CTTI project team works with experts to create tools (e.g. case study examples, mock-ups of forms, templates for trial documents, and process maps) that can be adapted to a particular trial.
Results
Survey responses of investigative staff
Survey population and site characteristics
A total of 201 respondents participated in the online survey for investigators and research staff. Of which, 47 were principal investigators or sub-investigators, and 154 represented other investigative site staff. Other characteristics are provided in Table 2.
Safety report workload and processing responses
Overall, investigative staff survey respondents indicated that the workload associated with processing safety reports is substantial. Approximately 80% of sites receive more than 20 safety reports monthly (the survey selection with the highest number). More than 60% of investigators and staff reported that they spend more than 10 h per month overall processing safety reports. In total, 20% of sites have refused to process IND safety reports because (1) the institutional review board does not need to review the same report, (2) the site was experiencing workload issues, and (3) the report did not comply with the FDA reporting rule.
The “investigator survey” also revealed variability in investigator engagement with safety report processing. In all, 72% of investigators and 84% of other trial staff indicated that their site has a standard operating procedure for the management of safety reports. A consistent feature of these standard operating procedures is an initial review of the reports by the trial or regulatory coordinator before submission to the principal investigator (91%); however, 61% of investigators and 75% of other trial staff reported that every safety report is still sent to the principal investigator (Figure 1). Regarding the observation that the reports are triaged and not reviewed by principal investigator, responses indicate that there is a disconnection between the perceptions of investigators and other trial staff. Almost 70% of the investigators polled reported that they review every safety report submitted, whereas less than half (49%) of other trial staff reported this to be the case.
Percentage of IND safety reports that are reviewed by the principal investigator.
Impact of the FDA final rule on safety report workload or utility
Most surveyed investigators (54%) and staff (63%) were aware of the final rule, and changes made to safety reporting after the final rule was in effect (72% investigators and 81% other staff). Regardless of the respondents’ knowledge of the final rule, 73% of investigators and 54% of other staff reported no reduction in the volume of safety reports or change in the quality of safety reports (80%–83%) over the past year.
Safety reports that did not trigger protocol and/or consent changes were overwhelmingly viewed as not useful by investigative staff, as they rarely if ever influenced patient management. Respondents who viewed these reports as useful reported that the reports help inform conversations with research participants about new safety information. These sentiments were re-iterated by the interview respondents.
Interview responses of investigative staff
A total of 13 experts from the investigative staff population were interviewed to elaborate on safety reporting issues. All interviewees thought the intent of the safety reporting rule was laudable; however, none believed it had achieved its goal. A key theme that emerged from the “investigator interviews” was the lack of utility of safety reports. Interviewees described the current system of safety reporting as “failed,” in that it requires a huge time commitment on the part of the investigative sites without yielding useful information.
Many respondents indicated that handling safety reports has become an exercise in “just checking the boxes.” According to those interviewed, because investigators typically do not have enough time to thoroughly review all of the material, and do not find it worthwhile, most sign off on the reports without thoughtful consideration. Furthermore, responses indicated that none of the investigators have ever used any information from these reports to improve their trials or make patients safer.
Summary of investigative staff’s perceptions
Free-text responses from the investigative staff survey identified many problems with the current state of IND safety reporting, all around the recurring themes of excessive volume and insufficient context. Problems reported included the following: too many or duplicate reports, overly time-consuming, individual reports have little value, sponsors or data-monitoring committees should review and only send unexpected and possibly related reports, reports with excess detail can hide important and useful information, not actionable (i.e. not requiring changes to protocol or consent), and too many websites to access.
Solutions proposed by investigators and trial staff suggest greater adherence to the final rule with IND safety reporting: only serious, unexpected, possibly related AEs should be reported.
Sponsor survey responses
Sponsor survey population and organization characteristics
In total, 29 respondents participated in the sponsor survey: 66% (n = 18) represented large biopharmaceutical companies (≥US$10 billion in annual revenue), 10% (n = 3) were employed by midsized biopharmaceutical companies (US$1–US$10 billion in annual revenue), and 24% (n = 7) were from small biopharmaceutical companies (<US$1 billion in annual revenue). More than half (n = 10) of the large companies had greater than 50 active oncology trials; midsized and small companies ranged from 11 to 30 active oncology trials. Of those polled, a majority had a safety/pharmacovigilance role in their organization (total n = 15). Regardless of the size of the organization, all companies sponsored trials across all phases of development.
Sponsors’ perceptions of adherence to the final rule
A majority of the respondents indicated that the pharmacovigilance department was responsible for interpreting FDA safety reporting requirements (63%) and for creating organizational policy to implement the requirements (58%).
From 2012 to 2014, the majority of organizations generated greater than 1200 safety reports per year (Figure 2). For large companies, 46% of respondents reported no numerical reduction in reports; yet, when qualitatively asked whether a reduction in the volume of initial safety reports was achieved after the rule was made final, approximately 86% of large company respondents indicated they had. Most indicated that the volume of reports was reduced by 50%–75%, and a few noted a reduction beyond 75%. In contrast, both midsized and small companies reported no reduction. In fact, some had an increase in the number of safety reports in 2014 compared with 2012 (Figure 2).
Mean number of safety reports per year.
Organizational barriers to full implementation of the final rule according to sponsors
The main barriers to full implementation of the final rule were compliance concerns with differing international regulations (n = 12) and difficulties establishing thresholds for reporting (n = 7). Infrastructural or technological limitations, liability concerns, and a concern that the reduction in reporting to the FDA would be interpreted as non-compliance with regulations were also reported. Respondents from large companies indicated that an organizational change would be difficult (Figure 3).
Internal organizational barriers to implementing the FDA’s final rule.
Despite these barriers, many organizations have implemented changes within their company to improve adherence to the final rule, including pre-specification in the protocol of anticipated/disease-related events, submission of reports to FDA based on the sponsor’s determination of causality rather than relying solely on the Investigator’s assessment, technology enhancements to enable compliance, standard operating procedures to enhance compliance with final rule, filters to determine clinically significant follow-up, and review of study cases by physicians to determine causality instead of defaulting to investigator causality.
Most sponsors recognize that full implementation has not been achieved. Many large companies indicated (n = 6) that they would benefit from pre-specifying in the protocol which events are anticipated for the population and/or disease under study, and therefore not reportable, as suggested in the FDA guidance. Additional changes suggested by respondents included more interaction with the FDA to ensure clarity on the final rule and greater alignment with other global regulatory bodies. Some respondents indicated they have done everything possible to comply.
When questioned on the potential 90% reduction in safety reporting volume estimated by the FDA following adherence to the final rule, most survey respondents thought this was unrealistic for several reasons, including that a high volume of personnel involved in the reporting process in large companies would need to be trained and/or become comfortable with the new reporting standards, a lack of clarity on how to interpret the final rule and how follow-up reports should be treated, and misalignment of global reporting requirements.
When asked what would help with implementation of the final rule, respondents from large companies desired more guidance from the FDA, more education for their staff, and alignment with the European Medicines Agency and other regulatory agency requirements.
Sponsor interview responses
Sponsor interview responses generally aligned with survey results. Many said that their companies reduced the volume of individual safety reports by 40%–75%, and two interviewees indicated that their organization reduced the volume of reports by more than 90%. One interviewee noted a decrease from approximately 1400 reports in 2013 to 200 reports in 2014; another cited a reduction of 91% after speaking directly with the FDA to clarify requirements. One respondent indicated no decrease in reports due to the complicated nature of determining causality with oncology therapies.
Interviewees expressed that the greatest challenges with implementing the final rule were (1) lack of global harmonization with reporting requirements, (2) determining causality with the investigational drug (especially in blinded studies), and (3) the burden associated with performing an “analysis of similar events.”
Lack of standardization in regulatory requirements across countries with different reporting rules increases the workload of the sponsor. One interviewee described this challenge as the biggest challenge in drug safety, especially when regulations for different regions contradict one another.
Regarding determination of causality, most interviewees indicated that they have used investigator brochures to identify known adverse reactions that do not meet the “unexpected criteria” and automatically exclude those events from IND expedited reporting. Yet, sponsors may still conservatively report events for fear of regulatory consequences should they misjudge causality. According to the interviewees, this approach also prevents regulators from assuming that the company is “concealing” AEs. Furthermore, determining causality may require that aspects of the trial or certain data be unblinded, which sponsors prefer to avoid if possible.
Summary of sponsors’ perceptions
Perceived and/or real challenges can be categorized into regulatory, legal, technological, and organizational issues (Table 3). Despite these barriers, most sponsors believe the rule is reasonable and can reduce the number of uninformative safety reports. Some sponsors focused on the estimated 90% reduction goal, stating that this goal was unrealistic. Finally, respondents indicated that sponsors may benefit from additional direct interaction with the FDA to clarify nuances of the final rule.
Summary of sponsors’ perceptions.
Discussion
Using surveys and interviews of clinical trial staff and sponsors, CTTI identified some of the barriers for implementation of the FDA final rule on safety reporting. The final rule is intended to minimize uninformative safety reports so that urgent, important safety signals can be captured and communicated in a manner that will ensure patient safety. Many stakeholders trust that the safety reporting guidelines are being implemented appropriately, and that systems in place for safety reporting are effective. However, results from this project indicate that many investigators are still receiving a high volume of reports that, in general, investigators do not believe are valuable for ensuring patient safety. In addition, ineffective implementation consumes time and resources for activities viewed as not meaningful (e.g. “box-checking”).
One FDA audit reviewing 160 safety reports submitted to INDs in the Office of Oncology and Hematology Products found that only 14% (22/160) met reporting criteria while the remaining 86% of reports were uninformative. Such data suggest that full implementation by sponsors of processes to send only reports required by the regulations could reduce the volume of reports by as much as 90%. However, the FDA acknowledges that this percentage is a general estimation, that it may not apply to all sponsors, and that the main concern is that the reports be informative and of high quality. Moreover, the FDA emphasizes that this estimation should not guide reporting decisions, as the primary goal of the final rule is to improve the quality and utility of safety reports.
Implementing a change in organizational processes requires a cultural shift and considerable effort on the part of the sponsor. Sponsors may choose to preserve the status quo for several reasons: (1) an overly conservative approach may prevent FDA audits, inspections, and sanctions; (2) it will protect them from liability; and (3) change is difficult, and personnel may be comfortable with the current standard operating procedures. While some sponsors may still employ a conservative approach, many believe that the final rule is sensible, that it would help minimize uninformative reports, and potentially improve their relationship with investigators as a result.
Adoption of the final rule can be particularly challenging for large companies as it may require changes to infrastructure. An electronic format may be more time-efficient and practical 10 than a paper-based format. In addition, resistance to change may stem from lack of clarity. Although the FDA guidance on safety reporting has clarified many points, some respondents suggested more clarity on the consequences if a sponsor determines that an event is not causally related to an investigational drug but the FDA disagrees, the threshold for aggregate analysis and reporting, and best practices for submitting “other previously unreported events,” along with an expedited report.
In both the survey and interviews, investigative staff offered several suggestions to improve the utility of safety reports, including summaries of trending data with conclusions, and reports of only AEs that are serious, unexpected, and probably related to the drug and would trigger a change in protocol. Respondents who had conversations with the FDA on the final rule found those discussions very helpful and suggested that webinar series or workshops with FDA officials be made available to sponsors. Overall, there was a desire for more direct interaction with the FDA in this capacity.
In 2013, CTTI released recommendations of best practices for IND safety reporting; 5 however, it was clear that further information was needed. The authors believe that results from this study are generalizable to therapeutic areas other than oncology because although the magnitude of the problem is especially challenging in oncology, 6 the principles for improving safety reporting are the same. In hopes of increasing compliance to the final rule, CTTI plans to develop and release additional resources based, in part, on the results reported in this article. In addition, CTTI-hosted webinars are planned to educate attendees on specifics of the final rule, present findings from CTTI’s evidence-gathering activities, and facilitate discussion between sponsors, investigators, and FDA representatives.
Conclusion
Both the sponsor survey and the interviews highlight the need for further clarity on the final rule in order to meet the goal of reducing the number of uninformative safety reports. Investigators offered potential solutions; however, the onus of implementing the final rule is on the sponsors. Different sponsors have different viewpoints, which may be influenced by the size and reach of their organization; however, most sponsors agree with the intention of the final rule. Yet, difficult challenges may deter some sponsors from timely implementation of the final rule. Findings suggest that the clinical trial enterprise can benefit considerably from additional educational materials on the final rule and proactive, bidirectional communication between sponsors and the FDA in order to clarify practical application of the final rule.
Footnotes
Acknowledgements
The authors wish to acknowledge the contributions of the CTTI IND Safety and Safety Advancement Project Teams and those individuals who participated in the surveys and interviews. Also, the authors acknowledge Susan Locke at the Duke Center for Learning Healthcare for her assistance in refining, executing, and analyzing the surveys and Diane Bloom for her assistance in conducting the qualitative interviews. Medical writing assistance was provided by Kelly Kilibarda, in affiliation with Whitsell Innovations, Inc., and was sponsored by CTTI. Views expressed in this publication do not necessarily reflect the official policies of the Department of Health and Human Services, nor does any mention of trade names, commercial practices, or organizations imply endorsement by the US Government.
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for this manuscript was made possible, in part, by the Food and Drug Administration through grant R18FD005292. Partial funding was also provided by pooled membership fees from CTTI’s member organizations.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
