Abstract
Randomized controlled trials (RCTs) are important for evidence-based medicine; however, their quality of reporting remains to be evaluated. The aim of this study was to assess the quality of the report concerning solid tumor medication. Articles were searched in PubMed to identify all oncology phase III RCTs published from 2011 to 2015, and the results were classified manually through Endnote X7.0 software. Registration rate, primary end point (PEP) consistency, positive result rate, enrollment time point, outcome feedback in the registry, and publish time zone were extracted and assessed. The overall registration rate was higher than years before; nevertheless, a portion of trials showed PEP discrepancies and enrolled patients before registration in either journal formats. Trials published in top 5 general medical journals paid more attention to results feedback on registration websites and were more prompt with publication after study accomplishment. Our data suggested general medical journals may be more rigorous compared to oncology journals but identified a preference for positive results. On the whole, RCTs published between 2011 and 2015 seemed fairly standardized. Surveillance in registry and outcome feedback still needs to be strengthened for the stringency and reliability of clinical trials in solid tumor medication territory.
Introduction
Based on GLOBOCAN estimates, about 14.1 million new cancer cases and 8.2 million deaths occurred in 2012 worldwide, which constitutes an enormous burden on society. 1 The past 2 decades witnessed the ascension of new drugs 2 and the increase in 5-year relative survival rates. 3 New cancer drug development is characterized by a high-investment (averagely US$1042 million), long-cycle, high-risk, complicated process of research and development with low approval rates (13.4%). 4 Randomized controlled trials (RCTs) are the gold standard to assess the effectiveness and safety of new treatments. Unfortunately, scandals such as selective reporting of trials as well as neglecting unfavorable evidence did occur, 5,6 which would lead to overestimates of the benefits of treatments and underestimates of their harmful effects, distort the body of evidence available for clinical decision-making, and put patients at risk and waste health-care resources.
Honest reporting begins with revealing the existence of all clinical studies, as studies cannot influence clinical practice guidelines when concealed by research sponsors or investigators. However, trial registration was largely voluntary a decade ago; registry data sets, completion quality, and public access varied. To make all trials and clinical evidence searchable by anyone, the members of the International Committee of Medical Journal Editors (ICMJE) published a joint editorial in September 2004 aimed at fostering a comprehensive, publicly available database of clinical trials. 7 To eliminate substandard and perfunctory registry, the International Clinical Trials Registry Platform (the Registry Platform), based at World Health Organization, achieved consensus on a minimum trial registration data set of 20 items. 8 Also, the Food and Drug Administration Amendments Act (FDAAA) of 2007 mandated public registration and the deadline of results disclosure and paper publishing after trial completion. 9
The ultimate goal of registration is full transparency and performance guarantee of clinical trials. Previous studies had disclosed significant deficiencies such as incomplete registration and primary end point (PEP) discrepancies (at an average rate of 14%), but with miscellaneous research objectives. Some of these studies had problems with retrieval method and journal selection. 10 -13 An updated assessment restricted to solid tumor medication RCTs is indispensable for guiding present perceptions and promoting the advancement of future research.
The specific objectives of this study are to examine and compare the report quality of registered RCTs concerning solid tumor medications published both in leading general medical journals and leading oncology journals, to assess the consistency of registered and published PEPs in RCTs, and to analyze the reasons of PEP discrepancies.
Methods
Selection of Articles
Articles were searched in PubMed to identify all phase III RCTs of oncology published between January 1, 2011, and December 31, 2015. “Controlled Clinical Trial” and “Clinical Trial, Phase III” were selected for screening relevant articles. All the relative articles published in the top 5 general medical journals and the top 5 oncology journals (Table 1) according to the Journal Citation Report 2014 released by Thomson Reuters were included in our study. The search result was systematically reviewed by 2 authors (H.Z. and S.C.) through the use of Endnote X7.0 software. Articles with studies identified as phase III RCTs (which assigned participants randomly and implemented different interventions) were included. We reviewed the title and abstract of each article and the full-text if necessary to select out articles about oncology. Any disagreement was resolved by consulting Y.D. and H.Z. Exclusion criteria included hematologic or pediatric studies; meta-analyses, overviews, or studies with ≥2 trials; phase I, II, or IV trials; pilot, protocol, ongoing or follow-up trials; treatment solely with radiotherapy or surgery trials; screening or diagnostic trials; secondary reports of completed trials; supportive or care; or prevention trials.
Distribution of Included Trials in the Selected Journals According to the Journal Citation Report 2014.
Data Extraction
First author’s name, journal name, publication year, first author’s origin, study type (international, which means centers were from more than 1 country; multicenter, which means patients were enrolled in at least 2 sites; or study group, which means the study was designed and conducted by a study group), number of study arms, type of control arm, type of blinding method, sample size, type of result, funding source, PEP, and the result for sample size calculation were extracted separately by 2 authors (H.Z. and S.C.). Registration information for each included article was also checked by 2 authors separately. Firstly, the article was read to see if provided trial registration number; if not, we searched the article in the registries that were accepted by the ICMJE. 14 Articles were considered as did not registered, if registration number could not be found after reading and searching. Secondly, trial results were checked to justify whether they were posted in the registries and cross-checked basic information in the registries to ensure the registration number matched the published articles. Each trial was checked whether it was published within 24 months after completion of the study. We also recorded the registered PEP with a clear description and extracted PEP described in registries and classified into 2 groups according to the registered number, separately. As the ICMJE required, the ICMJE journals accept “retrospective registration” of trial that began before July 1, 2005 (retrospective means registration occurs after patient enrollment begins); and for trials began on or after July 1, 2005, registration should occur before the first patient was enrolled (prospective registration). 8 All the registered trials were divided into 3 groups “registered before the enrollment of the first patient,” “registered after study begin but before study ends,” and “registered after study ends.” We compared registered with published PEP to analyze the consistency of each article. Different types of inconsistencies were used as previously defined. 15 All discrepancies that appeared between the registered and published PEP were recorded, and inconsistent PEPs were examined to identify whether the discrepancies favor statistically significant results. Others could not judge for what reasons the author changed the PEP, which was classified as “impossible to conclude.”
Statistical Analysis
The number of articles (percentage) and median were used for categorical variables to describe the basic characteristics of included studies. Comparisons of categorical variables between general medical journals and oncology journals were performed using χ2 test and Fisher exact test where appropriate. P < .05 (2 tailed) was considered statistically significant. Analyses were performed using SPSS 21.0 software (Chicago, Illinois).
Results
A total of 381 RCTs were included in this analysis (Figure 1). Sixty-six (17.3%) were published in general medical journals, and the other 315 (82.7%) were published in oncology journals. The number of included trials of each journal and the percentage of the total are listed in Table 1. The basic characteristics of the included studies are presented in Table 2. From 2011 to 2015, the number of oncology RCTs published is roughly consistent. About half of the first authors’ affiliations were in European countries (193/381, 50.7%). Most of the trials are multicenter studies with 2 study arms. Regarding the funding source, 223 (58.5%) reported a sole industry funding source, with 49 published in general medical journals and the other 174 published in oncology journals; 168 of 381 included studies published positive results; 358 (94.0%) studies published only 1 PEP.
Flowchart for study screening and selection of solid tumor randomized controlled trials (RCTs).
Characteristics of the 381 Included Studies.
Abbreviations: IF, impact factor; PEP, primary end point.
a Two hundred twenty-six articles have 2 types, with 37 in general medical journals and 189 in oncology journals; 67 articles have 3 types, with 17 in general medical journals and 50 in oncology journals.
b In 381 articles, 4 studies have 2 PEPs with different types of result; P < .0001, for comparison between general medical journals and oncology journals.
Registration Information of the Included Studies
Table 3 shows the registration information of the included studies. A total of 339 studies were found to have registration numbers, either identified by reading (312/339, 92.0%) or by searching (27/339, 8.0%). The http://ClinicalTrials.gov was the most popular registry for authors of oncology RCTs (302/339, 89.1%). Of the included 339 RCTs published, 174 (51.3%) of the trials put results in registries; 219 (64.6%) trials were published within 24 months after study completion; and 255 (75.2%) registered trials with adequate information about PEP. Most RCTs registered 1 PEP in the registry (294/355, 87.8%), a few others registered 2 (41/355, 12.2%) or more. For the trial registration time, 55.8% (188/337) RCTs met the ICMJE’s requirement on prospective registration.
Registration Characteristics of 339 Registered Studies.
Abbreviations: ANZCTR, Australian New Zealand Clinical Trials Registry; ISRCTN, International Standard Randomized Controlled Trial Number Registry; PEP, primary end point; UMIN, University Hospital Medical Information Network Clinical Trial Registry.
a P < .0001, for comparison between general medical journals and oncology journals.
b P < .0001, for comparison between general medical journals and oncology journals.
c Four studies in oncology journals could not find information about PEP.
d P = .003, for comparison between general medical journals and oncology journals.
e Two studies in oncology journals could not find registration time.
Primary End Point Consistency Analysis
A total of 335 registered RCTs were included in the PEP consistency analysis; 4 of 339 registered studies cannot find detail information about PEP and were excluded. Table 4 shows the results of PEP consistency analysis of the 325 RCTs. A total of 35 (10.4%) RCTs had discrepancies between registered and published PEP. The discrepancies included omitting registered PEP in the text (21/335, 6.3%), introducing a new PEP into text (10/335, 3.0%), reporting registered PEP as secondary in the text (3/335, 0.9%), different timing of assessment (1/335, 0.3%), and reporting registered secondary outcome as PEP in the text (1/335, 0.3%). There were 17.1% (6/35) inconsistent PEPs published due to statistical reasons.
Difference Between PEP in Trial Registration and in Published Article.
Abbreviation: PEP, primary end point.
a Two articles have 2 reasons for difference in PEP, both in oncology journals.
b P = .03, for comparison between general medical journals and oncology journals.
General Medical Journals Versus Oncology Journals
All of the articles published in general medical journals were multicenter studies; 81.8% (54/66) and 27.3% (18/66) were international cooperative studies and published in a study group, respectively, compared with 95.9% (302/315), 55.9% (176/315), and 36.2% (114/315) of those published in oncology journals, respectively. Fifty-nine (86.8%) study results published in general medical journals were positive, compared with 109 (34.4%) in oncology journals (P < .0001). All the included RCTs published in general medical journals provided the trial registration number in the published articles, while 90.1% (246/273) of those published in oncology journals. As the included RCTs have already been published, 72.7% (48/66) of trials in general medical journals put results in registries, significantly higher than those published in oncology journals (126/273, 46.2%; P < .0001); 97.0% (64/66) of trials were published within 24 months after study completion in general medical journals, while 56.8% (155/273) were published in oncology journals (P < .0001); the amount of trials with adequate information about the assessment period in registry was similar in general medical journals (53/66, 80.3%) and oncology journals (202/273, 74.0%; P = .287). For the trial registration time, 72.7% (48/66) of RCTs published in general medical journals met the ICMJE’s requirement on prospective registration higher than oncology journals (140/271, 51.7%) significantly (P = .003). Three (4.5%) and 32 (11.9%) RCTs had discrepancies between registered and published PEP in general medical journals and oncology journals, respectively. The inconsistent PEPs due to statistical reasons were 66.7% (2/3) for the articles published in general medical journals and 12.5% (4/32) for those in oncology journals (P = .03).
Discussion
Over the past 3 decades, the 5-year survival rate for all cancers combined has increased. Progress has been most rapid for hematopoietic and lymphoid malignancies, while it has been slow for solid tumors such as lung and pancreatic cancers. 8,14 With improvements in new treatment, a growing number of oncology trials are being conducted, but vary in quality. Randomized controlled trials are considered to provide evidence of the highest grade in the hierarchy of research designs, yet rely on accurate reporting and correct interpretation. Trial results may not coincide with each other 15 ; thus, we usually rely on a body of evidence from many studies to guide medical practice. Decreasing publication of trials with negative results would distort the body of evidence available for clinical decision-making, lead to overestimate or underestimate of the real effects, and violate patients’ interests. 16
The introduction of the ICMJE registration policy in 2004 has helped provide an open access repository of registered trials and promote transparency and accountability in the planning, conducting, and reporting of clinical trials, 7 thus minimizing selective reporting bias. Assessing characteristics of published RCTs and comparing their consistencies with originally registered versions could provide deep insight into the status of global health research over time, highlighting both progress and disparity. Recent studies focusing on registration rates and outcome consistency in high-impact medical journals in different specialties have demonstrated some deficiencies. 11,17 This study conducted the most comprehensive assessment about solid tumor RCTs in the past 5 years and is the first study to make comparisons between leading general medical journals and oncology journals.
We identified a sample of 381 RCTs by searching PubMed. An increase in registration rate (89.0%, 339/381) compared with former research (58.7%, 215/366) was found, 10 and the rate in 2015 (97.7%, 86/88) was significantly higher than that in 2011 (62.3%, 38/61). Clinicaltrials.gov remained the most popular registration site. Some data in registries were neither qualified nor adequate and even lacked important items such as the primary outcome. 18 Strengthening registry systems and making every item of a data set explicitly specified could be a method to enhance standards of the implementation of research registration.
Authors are required to report results for trials of drugs in the United States subject to Food and Drug Administration regulation and there is a deadline for research publishing. 19 We recorded that 51.3% of investigators upload result information in ClinicalTrials.gov and 64.4% of articles were published within 24 months after study completion by comparing the publication date and the study completion date. This study demonstrated a fair percentage of negative results (54.8%) overall. What is noteworthy is that 86.8% of RCTs published in general medical journals showed positive results, which indicated that statistically significant outcomes of a new treatment were more likely to be published in top general medical journals than nonsignificant outcomes.
Using multiple PEPs in a clinical trial often indicates that investigators have potential result reporting biases. It should be noted that each clinical trial should only have a single PEP, which should be defined before initiating the study. 20 In this study, 41 RCTs registered and published more than 1 PEP. Discrepancy types range from omitting registered PEP in the text to report registered secondary outcome as PEP in the text. According to this study, 10.4% of included articles had PEP discrepancies in trial registration and the published version, among which 17.1% favored significant primary outcomes.
Besides, most of the trials’ sponsors and principal investigators were from Europe and America; Asia only accounts for 12.9%. Therapeutic reaction varies with race, ethnicity, and socioeconomic status. Although multicenter studies may have Asian participants, we appeal to see some more rigorous RCTs reflecting Asian populations.
In general, RCTs concerning solid tumor medication are in continuous improvement. Randomized controlled trials published in the top 3 general medical journals are considered more standardized as compared with oncology journals, reflected by the number of PEPs, clear PEP description, PEP consistency, timely publishing rate, higher result uploading rate, and a fair percentage of the positive result. However, as previously stated, editors and reviewers of leading general medical journals seemed more enthusiastic about positive results.
What we want to emphasize here is that the trial registration policy is neither a trivial process to take nor a meaningless procedure merely to upload information; it is an inseparable part of a canonical clinical trial. The policy and concrete practice make every trial searchable to anyone and make clinical research more standardized and more reliable, which provides more powerful evidence to guide clinical practice and ultimately benefit patients. We maintain that quality assurance and management cannot be left only to the registries, and the investigators and journals should continue to carry the burden for the transparency of clinical research. The sponsor and principal investigator should ensure trial registration and comprehensiveness of registries before enrolling participants. The completion quality of trial registration and consistency 21 between the registered protocol and the submitted manuscript should be routinely checked by editors and reviewers, especially regarding important items as PEP. Explanations for significant discrepancies are required before final approval for publication.
This study has some limitations. Firstly, we only selected trials published in journals with high impact factors, most of which were members of the ICMJE and required trial registration. We were not able to speculate the results from normal impact journals. But we have our own considerations; journals with high impact factors have a wider range of readers and have a greater influence on clinical practice and decision-making. Secondly, a proportion of RCTs included in our review initiated before 2005. Therefore, the improvement of RCT registration was expected to be even more remarkable since the implement of ICMJE policy, and the incidence of enrollment prior to registration was overestimated to some extent. Thirdly, we only included phase III RCTs of solid tumor medication, while the characteristics of trials in other territories remained unknown. Furthermore, our study found that a high percentage of studies reported negative results. Whereas, it is not qualified to evaluate the publication bias and to what extent the direction of the results is associated with a higher probability of publishing or not.
Conclusion
In this study, we have provided the most comprehensive assessment about solid tumor medication RCTs published in leading journals. The registration and completion status had improved concerning the implementation of ICMJE policy, the minimum reporting standards by CONSORT Group, and the FDAAA legal requirements for Clinical Trials Registration and Results Information Submission. Problems such as unsatisfactory outcome feedback in the registry and general medical journals’ preference for reporting positive results remain to improve. Joint efforts of investigators, medical journal editors, peer reviewers, as well as related institutions and organizations and consensus 22 are needed to achieve full transparency of clinical trials.
Footnotes
Author Contributions
Huiyun Zhu, Si Chen, and Pei Xie contributed equally to this article.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Dr. Huiyun Zhu is supported by Innovative Fund for PhD granted by Second Military Medical University/Naval Medical University. Dr. Yiqi Du is supported by the National Natural Science Foundation of China (Grant no 81770643).