Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Abstract

Background:

The American Academy of Orthopaedic Surgeons (AAOS) publishes clinical practice guidelines (CPGs) for various pathologies. An extension to the Consolidated Standards for Reporting (CONSORT) checklist provides authors with a framework for reporting harms in randomized controlled trials (RCTs).

Objectives:

The purpose of this study was to measure harms reporting among RCTs cited as supporting evidence for the AAOS CPG on the treatment of clavicle fractures. This study also sought to determine whether these reporting rates changed over time.

Methods:

Adhering to Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we used the AAOS CPG for clavicle fractures and conducted a linear regression analysis to model the relationship between the year of publication and the total CONSORT checklist percentage adherence over time.

Results:

There were 28 RCTs cited as supporting evidence for the clavicle fractures CPG included in the final sample. The average number of the 18 CONSORT Extension for Harms items appropriately reported across all included RCTs was 9.32 (51.8%). Nine checklist items had more than 50% compliance (50%), 3 items had between 20% and 50% compliance (16.7%), and 6 items had less than 20% compliance (33.3%). The linear regression model demonstrated no significant improvement in CONSORT Harms reporting over time.

Conclusions:

Adverse events are inadequately reported in RCTs cited as supporting evidence for the AAOS Treatment of Clavicle Fractures CPGs. We recommend the utilization of standardized adverse event reporting tools specific to orthopedic trauma literature to facilitate ease in adverse event reporting among RCTs.

Introduction

Clinical practice guidelines (CPGs) are a set of rigorously developed recommendations on the management of a disease or condition that can promote beneficial patient outcomes if applied properly [11,28]. The American Academy of Orthopaedic Surgeons (AAOS) has published a series of guidelines for various orthopedic conditions based on a comprehensive systematic search with formal peer review and public commentary [27]. The AAOS has published 14 recommendations for the treatment of clavicle fractures (Supplemental Table 1).

The clavicle is one of the most frequently fractured bones in the human body, accounting for 3% to 5% of all fractures [21]. Historically, clavicle fractures have been treated nonoperatively with good functional outcomes, but recent research highlighting an increased risk of nonunion has prompted a rise in operative treatment [20,22,26]. Although several studies have demonstrated positive results after surgical intervention [9,23,29], there is an inherent risk of complications such as hardware irritation, infection, and neurovascular injury [16,24]. The high incidence of this injury coupled with the variability in management between providers makes clavicle fractures an ideal candidate for a CPG. For each CPG topic, experts in the field evaluate the literature on the topic to assign a strength of each recommendation labeled as inconclusive, consensus, limited, moderate, or strong.

Many of the studies underpinning the CPGs are randomized controlled trials (RCT) [10]. Because RCTs are considered the gold standard for assessing the efficacy of a treatment, it is important to analyze the extent to which they adhere to appropriate adverse event (AE) (harms) reporting [14]. The Consolidated Standards for Reporting (CONSORT) checklist was developed with the goal of improving the quality of reporting in RCTs. The original checklist was developed in 2001, a CONSORT extension has since been developed in 2004 and later updated in 2010 with the specific aim to improve harms reporting in RCTs. The CONSORT Extension for Harms is an 18-item checklist that offers a standardized method for authors to report AEs in RCTs to facilitate transparent reporting [12]. Although the checklist has existed for many years, several studies have demonstrated poor reporting of harms among RCTs in the fields of internal medicine [3,15], psychology [13], and orthopedics [1]. In light of this trend, the purpose of this study was to measure the harms-related reporting among RCTs cited as supporting evidence for the AAOS CPG regarding Treatment of Clavicle Fractures.

Methods

In this study, no institutional review board oversight was required as it did not involve human subjects. Throughout the investigation, we followed the guidelines for reporting meta-research [19] and adhered to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [7].

To identify relevant CPGs, we utilized the AAOS CPG for the treatment of clavicle fractures available on the AAOS website. Specifically, we referred to the list of studies included in the AAOS CPG titled “Quality Evaluation-Intervention-Randomized” [10]. Two authors (A.T. and A.I.) independently reviewed the studies listed in this table to identify all RCTs cited as supporting evidence for at least 1 CPG recommendation. The following inclusion criteria were applied to each RCT: (1) the trial met the definition of an RCT according to the International Committee of Medical Journal Editors; (2) the trial was available in English (for feasibility); and (3) the trial was cited as evidence supporting one or more CPG recommendations. After the screening process, the results from the 2 data extractors were compared, and any discrepancies were resolved through group discussion.

Two authors (A.T. and A.I.) independently conducted the data collection process. The following information for each study was collected: title; PMID; year of publication; journal of publication; CPG for which RCT was cited as supporting evidence; RCT funding source (if applicable); country RCT was conducted in; sample size; blinding; and number of participating centers. Additionally, each trial was evaluated based on the CONSORT Harms checklist, which consists of 10 recommendations for reporting harms that occur over the course of an RCT. Each item was scored as “yes” (1 point) or “no” (0 points) based on adherence to the predefined requirements. The overall percentage adherence was calculated out of a possible 18 points, considering that some CONSORT Harms Extension items have multiple subcategories. Scoring was discussed between the 2 authors, and any discrepancies were resolved through group discussion.

Statistical Analysis

Descriptive statistics were used to characterize the overall compliance with the CONSORT Extension for Harms. To demonstrate the relationship between the year of publication and the total CONSORT percent adherence over time, a linear regression analysis was performed.

Results

The AAOS guidelines for clavicle fracture treatment included 63 references cited as supporting evidence for at least 1 CPG. We compared this list with the “Quality Intervention—Randomized” table in the AAOS CPG, which presented a comprehensive list of all RCTs included in the guidelines. This process resulted in 28 RCT studies being included in the final sample (Fig. 1).

Fig. 1.

Flowchart for study selection.

Each of the included studies contained a sample size of over 50 patients, with most containing a range of 101 to 250 patients (14/28, 50%). Studies were predominantly conducted at multiple centers (15/28, 53.6%). A total of 19 studies provided a funding statement (19/28, 67.9%), of which private, nonindustry funding was the most common source (9/19, 47.4%). Only 5 studies mentioned adherence to CONSORT guidelines (5/28, 17.9%). Each study examined surgical interventions except for 1 (27/28, 96.4%). Finally, the included studies were conducted in 13 different countries, with the Netherlands being the most common location (6/28, 21.4%). Additional RCT characteristics are included in Supplemental Table 2.

The average number of appropriately reported CONSORT Extension for Harms items across the included studies was 9.32 (9.32/18, 51.8%) (range = 4-14). None of the included trials sufficiently reported all 18 items. The item most successfully adhered to was item 7a (“Provides denominators for [adverse events (AEs)]”) (27/28, 96.4%), followed by items 4a (“Includes harm-associated mode of data collection”), 4b (“Includes harm-associated timing of data collection”), and 5 (“Includes plans for presenting and analyzing information on harm”), each of which 26 studies successfully fulfilled (26/28, 92.9%). Zero studies adhered to checklist item 3c (“Mentions use of a validated instrument to measure AE severity”). Only 1 study fulfilled the requirements of item 3b (“Distinguishes between expected and unexpected AEs”). Nine items had compliance on >50% of CONSORT items (9/18, 50%), 3 items had between 20% and 50% compliance (3/18, 16.7%), and 6 items had <20% compliance (6/18, 33.3%) (Supplemental Table 3). The results of the linear regression model indicated that there was no significant improvement observed in CONSORT Harms reporting over time (R² = 0) (Fig. 2).

Fig. 2.

Linear regression model of mean percent adherence to the CONSORT Extension for Harms checklist. CONSORT, Consolidated Standards of Reporting Trials.

Discussion

We sought to assess the quality and comprehensiveness with which the RCTs supporting the AAOS CPG for the treatment of clavicle fractures appropriately reported AEs. We found that AEs were incompletely reported among these studies to a variable degree, as no study fulfilled all of the categories in the CONSORT Extension for Harms checklist. Item 3c (“Mentions use of a validated instrument to measure AE severity”) exhibited the lowest adherence, but given that there is currently no gold standard tool for recording AEs in orthopedic trauma, this may not be an important reflection of overall AE reporting in RCTs. Item 3b (“Distinguishes between expected and unexpected AEs”) demonstrated the second lowest percent adherence (3.6%), indicating a consistent deficit in qualifying the anticipation of certain AEs. With an overall adherence rate of 52% to CONSORT guidelines, it suggests that there may be considerable room for improvement in the way AEs are reported in RCTs concerning clavicle fractures.

Our study was not without limitations. Although the CONSORT Extension for Harms checklist clearly describes the requirements for each item, some items were more subjective than others. We attempted to mitigate this risk of bias by having multiple authors review and grade the RCTs according to a strict definition of each criterion. In addition, we excluded studies of lower-level evidence underpinning the AAOS guideline for the treatment of clavicle fractures, which may introduce a degree of selection bias. However, we chose to assess only the RCTs due to the high level of evidence and regard in which they are held in clinical practice and determined that they should be assessed independently. Finally, while assigning 1 point to each item provides a standardized approach to grading, it may not account for variations in the relative significance and impact of different checklist items on the overall assessment of AE reporting quality.

The result that AEs are incompletely reported is widely corroborated in the orthopedic literature. Upon examining the RCTs supporting the AAOS CPG for hip and knee osteoarthritis, Anderson et al found an overall compliance of 45% to CONSORT guidelines [2]. In the AAOS CPG for rotator cuff injuries, the same authors reported a 32.6% overall adherence [1]. In addition, in a systematic review of RCTs focusing on analgesics after orthopedic surgery, Montané et al found that less than one half of all CONSORT items were adequately reported [18]. Although a trend of insufficient AE reporting is apparent, it is important to recognize that it may not be reflective of the overall study quality. A lower overall CONSORT score does not necessarily imply that AEs were not reported, but it rather indicates that the transparency and comprehensiveness with which they were reported warrant further development. The findings of this study are in agreement with the previously published literature and should continue to encourage surgeons to follow CONSORT guidelines to improve AE reporting.

Low adherence to item 3b (“Distinguishes between expected and unexpected AEs”), of which only 1 RCT successfully fulfilled, may have important clinical relevance. Given the continued debate between operative and nonoperative management in the treatment of clavicle fractures, it is important for both physicians and patients to be aware of potential sequelae, so as to make an informed treatment decision. Although AEs are inevitable in the practice of orthopedic surgery, it is essential to distinguish certain AEs that might be more expected from a specific operation, as preparing patients for possible complications is often associated with increased reporting of AEs [25]. Although it is possible that all AEs were unexpected, each study should clarify that information within the manuscript.

To strengthen the quality of RCTs used to develop evidence-based guidelines, we recommend that journals require CONSORT adherence in RCTs as a prerequisite for submission. Currently, there is no consistent validated instrument or scale to measure AE severity for clavicle fracture treatment, evident in that none of the RCTs fulfilled item 3c in the CONSORT checklist and only 1 study reported unexpected versus expected AEs. One such method to report AEs is the use of electronic health record (EHR)-based triggers to detect these events after surgery [5]. Severity of outcomes can also be weighted to differentiate between outcomes of different clinical significance [4]. Similarly, the use of validated tools such as the Global Trigger Tool [6] and Orthopedic Surgical Adverse Events Severity System (OrthoSAVES) [17] should be strongly considered, as they have comprehensively reported a greater number of AEs in other orthopedic subspecialties [8]. Implementation of these strategies will serve to further standardize research criteria and strengthen the CPGs which cite RCTs as supporting evidence.

In conclusion, AEs are inadequately reported among the RCTs cited as supporting evidence for the AAOS Treatment of Clavicle Fractures CPGs. We recommend the use of standardized AE reporting tools specific to orthopedic trauma literature to facilitate ease in AE reporting among RCTs. In addition, we advise orthopedic trauma journals to encourage or require CONSORT adherence as a prerequisite for submission, as adherence to these guidelines may result in more accurate representations of the efficacy and risks of an intervention.

Supplemental Material

sj-docx-1-hss-10.1177_15563316231222484 – Supplemental material for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Supplemental material, sj-docx-1-hss-10.1177_15563316231222484 for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures by Ashley A. Thompson, Avinash Iyer, Jacob L. Kotlier, Cory K. Mayfield, Frank A. Petrigliano and Joseph N. Liu in HSS Journal®

Supplemental Material

sj-docx-2-hss-10.1177_15563316231222484 – Supplemental material for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Supplemental material, sj-docx-2-hss-10.1177_15563316231222484 for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures by Ashley A. Thompson, Avinash Iyer, Jacob L. Kotlier, Cory K. Mayfield, Frank A. Petrigliano and Joseph N. Liu in HSS Journal®

Supplemental Material

sj-docx-3-hss-10.1177_15563316231222484 – Supplemental material for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Supplemental material, sj-docx-3-hss-10.1177_15563316231222484 for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures by Ashley A. Thompson, Avinash Iyer, Jacob L. Kotlier, Cory K. Mayfield, Frank A. Petrigliano and Joseph N. Liu in HSS Journal®

Supplemental Material

sj-docx-4-hss-10.1177_15563316231222484 – Supplemental material for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Supplemental material, sj-docx-4-hss-10.1177_15563316231222484 for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures by Ashley A. Thompson, Avinash Iyer, Jacob L. Kotlier, Cory K. Mayfield, Frank A. Petrigliano and Joseph N. Liu in HSS Journal®

Supplemental Material

sj-docx-5-hss-10.1177_15563316231222484 – Supplemental material for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Supplemental material, sj-docx-5-hss-10.1177_15563316231222484 for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures by Ashley A. Thompson, Avinash Iyer, Jacob L. Kotlier, Cory K. Mayfield, Frank A. Petrigliano and Joseph N. Liu in HSS Journal®

Supplemental Material

sj-docx-6-hss-10.1177_15563316231222484 – Supplemental material for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Supplemental material, sj-docx-6-hss-10.1177_15563316231222484 for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures by Ashley A. Thompson, Avinash Iyer, Jacob L. Kotlier, Cory K. Mayfield, Frank A. Petrigliano and Joseph N. Liu in HSS Journal®

Supplemental Material

sj-docx-7-hss-10.1177_15563316231222484 – Supplemental material for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures

Supplemental material, sj-docx-7-hss-10.1177_15563316231222484 for Harms Reporting in the RCTs Underpinning the AAOS Clinical Practice Guidelines for Clavicle Fractures by Ashley A. Thompson, Avinash Iyer, Jacob L. Kotlier, Cory K. Mayfield, Frank A. Petrigliano and Joseph N. Liu in HSS Journal®