Sage Journals: Discover world-class research

Abstract

Objectives

We sought to examine the prevalence of prospective registration and publication rates in digital health trials.

Materials and Methods

We included 417 trials that enrolled participants in 2012 and were registered in any of the 17 WHO data provider registries. The evaluation of the prospective trial registration was based on the actual difference between the registration and enrollment dates. We identified existing publications through an automated PubMed search by every trial registration number as well as a pragmatic search in PubMed and Google based on extracted metadata from the trial registries.

Results

The prospective registration and publication rates were at (38.4%) and (65.5%), respectively. We identified a statistically significant (p < 0.001) “Selective Registration Bias” with 95.7% of trials published within a year after registration, were registered retrospectively. We reported a statistically significant relationship (p = 0.003) between prospective registration and funding sources, with industry-funded trials having the lowest compliance with prospective registration at (14.3%). The lowest non-publication rates were in the Middle East (26.7%) and Europe (28%), and the highest were in Asia (56.5%) and the U.S. (42.5%). We found statistically significant differences (p < 0.001) between trial location and funding sources with the highest percentage of industry-funded trials in Asia (17.4%) and the U.S. (3.3%).

Conclusion

The adherence of investigators to the best practices of trial registration and result dissemination is still evolving in digital health trials. Further research is required to identify contributing factors and mitigation strategies to low compliance rate with trial publication and prospective registration in digital health trials.

Keywords

randomized controlled trials clinical trials publication bias publications trial registries trial registration digital health eHealth mHealth mobile health

Introduction

In the realm of scientific research, study results should be made available and accessible to the broader research community to assess the evidence around the efficacy and potential harm of healthcare interventions.¹ Emanuel et al.'s framework defines seven ethical guidelines for clinical research involving human subjects.² The first of which describes the value derived from disseminating research results as “Only if society will gain knowledge, which requires sharing results, whether positive or negative, can exposing human subjects to risk in clinical research be justified”.

A number of studies have examined the ramification of publication bias and debated a number of contributing factors to non-publication of clinical trial results such as recruitment challenges, funding sources and study design.^3–10 Other challenges pertaining to investigators were also identified as contributing factors to non-publication of clinical trial results, such as lack of time or disagreement with coauthors.^11,12

The notion of clinical trial registration was conceived in 1986, when Simes suggested that the act of registration would help control the risk of publication bias by providing a new data source for trial information and results.^13–15 The trial registries would also help mitigate selective reporting of positive outcome by comparing outcomes reported in trial publication versus outcome measures indicated in the trial registration.^16–21 In 2004, the International Committee of Medical Journal Editors (ICMJE) introduced a new mandate to promote prompt registration of all clinical trials.²² In 2005, the World Health Organization (WHO) started an initiative to standardize trial registrations and trial registry datasets across multiple national and international registries.²³ In October 2008, the 7^th revision of the Declaration of Helsinki (DoH) was adopted with new requirements focusing on the importance of prospective registration of clinical trials and reporting of their results.²⁴ The prospective registration of clinical trial means that investigators should register their trials prior to the enrollment date of the first trial participant, otherwise, the registration would be considered retrospective. In 2015, the WHO announced a new statement on public disclosure of clinical trial results with more guidelines on trial registration, publication of results, and the inclusion of the trial registration number (TRN) in respective publications to enable linking of trial reports with clinical trial registry information.²⁵

Two studies reported publication rates between 66% and 68% for clinical trials registered in the US-based clinical trials registry, ClinicalTrials.gov.^26,27 Another study reported publication rates at 73% for clinical trials registered in at least one of several clinical trial registries (ClinicalTrials.gov, Current Controlled Trials, WHO International Clinical Trials Registry Platform (ICTRP), EU Clinical Study Register, and Indian, Australian-New Zealand, and Chinese Clinical Trial Registries).²⁸ Three other studies investigated prospective trial registration and the quality of registration in the International Standard Randomized Controlled Trial Number (ISRCTN) registry and the ICTRP registries.^29–31 The results of these studies reported that prospective trial registration was between 37.8% and 53.4%. A recent global study published in the British Medical Journal in 2020 reported that only 41.7% of the included randomized controlled trials were registered prospectively.³²

However, investigators could be overwhelmed with challenges that may be particularly problematic in digital health trials, such as the usability of the intervention under test, participant recruitment, and retention challenges that may contribute to non-publication rate and prospective trial registration.^33–39 To our knowledge, this is the first global review to analyze the non-publication rate and prospective registration of digital health clinical trials across the seventeen WHO data provider registries.⁴⁰

Research objectives

The primary research objective was to examine the prospective trial registration and publication rates of digital health randomized clinical trials registered in any registry that is part of the WHO data provider registries. The secondary research objectives were (1) to investigate the compliance with recruitment and inclusion of the TRN in the published trials, (2) to explore the relationship between publication rates, trial size and the duration from the enrollment of the first participant to the publication of trail results (enrollment-to-publication duration), and (3) to analyze the relationship between retrospective trial registration and the duration from trial registration to the publication of trial results (registration-to-publication duration).

Material and methods

Data source

The WHO ICTRP is a free online portal that provides unified access to trial registration information across different clinical trial data provider registries.^[40,41] To identify the registered digital health trials, we utilized the advanced search feature in the ICTRP search portal. We applied our search terms and phrases to the [Title] or [Intervention] field and downloaded the matching trials in XML format for further analysis. To identify journal publications of included trials we searched trial papers in PubMed and Google as described in the “Identification of Publication” subsection.

Statistical analysis

We used Pearson Chi-square statistic for bivariate analyses. These statistical analyses were performed in SPSS Statistics version 24 (IBM Corporation, Armonk, NY). We used Fisher's exact test statistic for bivariate analyses where the expected cell value was less than 5. The Fisher's exact test was performed in STAT Statistics version 14.2 (StataCorp LLC, TX).

Inclusion criteria

We included all eHealth, mHealth, telehealth, and digital health-related randomized clinical trials that are registered in any of the 17 trial registries within the ICTRP database and include any information and communication technology component, such as:

Online web and mobile application

Internet, websites and personal computer application

Digital games and social media application

Telehealth and telemedicine components

We included trials irrespective of their recruitment status or trial phases. The enrollment date in the trial registries was used to indicate the start date of the included trials. The inclusion criteria were limited to trials that started the participants enrollment between January 1^st, 2012 and December 31^st, 2012 as described in Online Appendix I.

Exclusion criteria

We excluded registered clinical trials that were not randomized or did not include any digital components in their intervention. Trial publications were only identified in peer-reviewed journals, excluding pre-print servers and conference abstracts. Trials that merely utilized short messages service (SMS), phone calls, emails or video communication without any other interactive components were also excluded. We excluded trials that only reported on computerized or online surveys and questionnaires.

Search terms for digital health clinical trials

We developed a comprehensive list of 86 search terms and phrases through an iterative process as explained in Online Appendix II.

Data extraction

We downloaded the XML files for the 417 matching clinical trials in the ICTRP search portal. These files were transformed into a tabular format and imported into a local relational SQL Server Database for further data extraction and analysis. The details regarding extracting the trial condition, primary sponsor and major technology are provided in Online Appendix III.

Prospective trial registration

The data export from the ICTRP dataset did include a field to indicate whether a trial was registered retrospectively.^41,56 However, our analysis showed that this field did not include correct information for a substantial number of trials within our sample of 417 included trials. We therefore chose to evaluate prospective trial registration based on the actual difference between the registration and enrollment dates as described in Online Appendix IV.

Identification of randomized trials

We were able to identify randomized trials through a text match search in the following three fields in the data export from the ICTRP dataset: “Study_design,” “Public_title,” and “Scientific_title”. We searched for matching words that included the term ‘Random’ and did not include the term ‘Non-Random’.

Identification of publication

We identified existing publications through a two-step iterative process. The first step included an automated identification process that included a PubMed search by every TRN. The second step included a pragmatic search in PubMed and Google based on extracted metadata from the trial registries; trial title, investigators, location/city and institution. We only considered trial publications that reported at least one of the primary outcome measures of the underlying trials. Complete details of the publication identification processes are described in Online Appendix V.

Results

We utilized the advanced search functionality in the ICTRP search portal to apply our 86 search terms and phrases including all trial phases and recruitment status. There were 22,859 unique trials that matched our search terms, within which, 15,096 trials were randomized, and 1018 trials were enrolled in 2012. After screening against our inclusion and exclusion criteria, 417 trials were included, and 601 trials were excluded as per the following breakdown (Figure 1).

Figure 1.

Included trials from the search results.

Prospective trial registration and publication rates

In summary, 160/417 (38.4%) trials were registered prospectively and 273/417 (65.5%) trials were associated with identified outcome publications. Of the 273 identified trial publications, 8 were identified in references in the ClinicalTrials.gov registry, 199 were identified in PubMed and 66 were identified through Google search as described in Figure 2.

Figure 2.

Publication identification results.

We examined the relationship between trial characteristics and the non-publication rate and prospective registration of all included trials as summarized in Table 1 and detailed in Online Appendix VI.

Table 1.

Relationship between trial characteristics and prospective registration and non-publication rates of included trials (details provided in Online Appendix VI).

Trial characteristics	Number of prospectively registered trials/n total (%)	Pearson chi-square test P value [Fisher's exact test, P value]^a	Number of non-published trials/n total (%)	Pearson Chi-square test P value [Fisher's exact test P value]^a
Overall	160/417 (38.4%)		144/417 (34.5%)
Age groups		0.31 [0.26]		0.40 [0.36]
Condition		0.02		0.042
Continent		<0.001 [<0.001]		0.01 [0.01]
Trial size		0.40		0.32
Enrollment year		0.13		0.88
Funding source		0.003 [0.003]		0.17 [0.16]
Gender		0.07 [0.06]		0.56 [0.58]
Major technology		0.06 [0.06]		0.78 [0.81]
Phase		0.44 [0.45]		0.86 [0.80]
Registration year		n/a		0.88
Trials registry		<0.001 [<0.001]		0.13 [0.16]
Number of study arms		0.58 [0.59]		0.02 [0.02]

^aP value was provided from Fisher’s exact test when the expected cell size is less than 5.

There were significant relationships (p < 0.05) between trial non-publication rate and trial characteristics including trial conditions, locations, and number of study arms. The highest non-publication rates were reported in Asia and the U.S. at 13/23 (56.5%) and 51/120 (42.5%), respectively. We did not consider the 100% non-publication rate of digital health trials in Africa due to the limited number of two trials included in our study. The highest percentage of industry-funded trials was reported in Asia and the U.S. at 4/23 (17.4%) and 4/120 (3.3%), respectively. To interpret these results, we examined the relationship, and found statistically significant differences (p < 0.001), between the trial location and funding sources as indicted in Online Appendix VIII.

Only 160/417 (38.4%) of the included trials were registered prospectively. There were significant relationships (p < 0.05) between prospective trial registration and trial characteristics, including trial conditions, funding source, location, and trial registry.

Enrollment-to-publication duration and trial size

We postulated that smaller trials would be easier to conduct and publish as they have less recruitment and enrollment challenges. To support our postulate, we found statistically significant differences (p = 0.002) between the trial size and the enrollment-to-publication duration, details provided in Online Appendix IX. We also explored the trend between enrollment-to-publication duration, trial size, number of published trials and the cumulative percentage of non-publication rate of all published trials. Details can be found in Online Appendix X. The results are depicted in Figure 3 revealing an incremental trend of published trial until a critical time point at the 4^th year, where fewer trials were published into the 5^th and 6^th year after enrollment.

Figure 3.

Relationship between trial enrollment-to-publication duration and trial size, number of published trials and non-publication rate.

Registration-to-publication duration and retrospective registration

We examined the prevalence of retrospective trial registration and its relationship to trial publication and the registration-to-publication duration. Overall, 168/273 (61.5%) of published trials were registered retrospectively. We found a statistically significant relationship (p < 0.001) between retrospective trial registration and the registration-to-publication duration. The vast majority 44/46 (95.7%) of digital health clinical trials that were published within a year after trial registration, were registered retrospectively as shown in Table 2.

Table 2.

Relationship between trial registration-to-publication duration and retrospective trial registration.

Registration to publication (years)	Number of retrospectively registered trials/number of published trials (%)^a	Pearson Chi-square test P value [Fisher's exact test P value]^b
Overall	168/273 (61.5%)
1 or less year	44/46 (95.7%)	<0.001 [<0.001]
2 to <3 years	30/48 (62.5%)
3 to <4 years	34/52 (65.4%)
4 to <5 years	30/57 (52.6%)
5 to <6 years	21/45 (46.7%)
6 to <7 years	9/22 (40.9%)
7	0/3 (0%)

Non-cumulative values. The number of retrospectively registered trials, the number of published trials, and the reported percentage is not cumulative within every year level.

P value was provided from Fisher’s exact test when the expected cell size is less than 5.

Compliance with participants recruitment and inclusion of the TRN in published trials

When evaluating the compliance with participants recruitment, we found that 111/273 (40.7%) of the published trials reported fewer subjects who were actually recruited than the target size indicated in the trial registry, 96/273 (35.2%) published trials reported actual recruitment matching the trial target size, and 66/273 (24.2%) published trial recruited more participants (over-recruitment) than the target size in the trial registry. Ensuring adequate participant recruitment is critical to support the statistical power and internal validity of the trial results. Enrollment of fewer participants could introduce type II errors in reported study results.⁶¹ We suggest that over-recruitment could be appropriate and would empower further assessment of secondary hypotheses. Therefore, we considered trial recruitment as compliant if the actual recruitment was equal to or more than the target size as defined in the trial registration.

We verified whether a reference to the TRN was indicated in the papers of the 273 published trials. Nearly half of the trials 142/273 (52%) provided a reference to their respective TRN in their publications. We examined the relationship between trial characteristics and the compliance with participants recruitment and inclusion of the TRN in published trials as summarized in Table 3 and detailed in Online Appendix XI.

Table 3.

Compliance with participants recruitment and inclusion of trial identification in published trials (details provided in Online Appendix XI).

Trials characteristics	Number of trials compliant with participants recruitment /number of all published trials (%)	Pearson Chi-square test P value [Fisher's exact test P value]^a	Number of publications that included trial ID/number of all published trials (%)	Pearson Chi-square test P value [Fisher's exact test P value]^a
Overall	162/273 (59.3%)		142/273 (52%)
Age groups		0.42 [0.44]		0.02 [0.009]
Condition		0.85 [0.85]		0.12 [0.12]
Continent		0.46 [0.46]		<0.001 [<0.001]
Trial size		0.009		<0.001
Enrollment year		0.61		0.003
Funding source		0.02 [0.01]		0.03 [0.03]
Gender		0.90 [0.94]		0.002 [0.001]
Major technology		0.60 [0.59]		0.29 [0.28]
Phase		0.44 [0.51]		0.054 [0.03]
Prospective registration		0.28		0.69
Registration year		0.45		0.63
Trials registry		0.16 [0.14]		<0.001 [<0.001]
Study arms		0.32 [0.41]		0.002 [0.001]

^aP value was provided from Fisher’s exact test when the expected cell size is less than 5.

Discussion

We found that 34.5% of all included trials remain unpublished six years after the enrollment of trial participants. Nearly half of our included trials 56.1% were registered in the ClinicalTrials.gov registry, of which 36.3% remain unpublished. Our non-publication rate was higher compared to a 2018 study that reported a 27% non-publication rate for digital health trials registered in ClinicalTrials.gov.⁶² Although both studies focused on examining non-publication rates in digital health trials, their study design, data source and approach were quite different, hence a direct comparison would not be valid. The current study has a broader scope including 417 trials that started participants recruitment in 2012 and were registered in any of the seventeen WHO data provider registries. The former 2018 study included 556 trials that were registered in the ClinicalTrials.gov registry only and were completed within three years, between 2010 and 2013.⁶²

Our finding is also higher than that reported in another study with a non-publication rate at 29% for large randomized clinical trials, i.e. trials with at least 500 enrolled participants.³ Other studies reported that nearly half of the included trials were not published, which is considerably higher than our own findings.^1,4,28 To explain these differences in non-publication rates, we postulate that the rapidly evolving technology elements of digital health trials could introduce extrinsic motivation to investigators to publish their results and stay ahead of the technology innovation curve.

Our results also showed that the vast majority 96.6% of digital health trials are funded by non-industry sponsors, such as universities, hospitals, medical institutes and research centers. We also found that trials funded by university sponsors were 1.5 times more compliant with trial recruitment compared to trials funded by any other sponsors. This finding indicates that investigators of university sponsored trials are excelling at adopting best practices and strategies in trial design to improve participant recruitment, which is to be expected within the academic context of the university sponsors. We speculate that industry sponsors are more interested in the broader opportunity in the digital health marketplace beyond the realm of academia and best practices of randomized trials design.

The lowest compliance with prospective trial registration was at 14.3% for industry-funded trials. In contrast to our present study, other studies assessing the prospective registration of clinical trials found that industry-funded trials were more likely to be compliant with prospective trial registration compared to non-industry-funded trials.^63–70 However, we also found that only 14 (3.4%) trials of the included 417 trials were industry-funded which may limit the generalizability of our finding in comparison to other studies.

The compliance with participants recruitment was at 59.3%, which is higher than that of the McDonald et al. study that reported 31.1% and 46.7% compliance in participants recruitment compared to the original and revised target size respectively.^71,72 However, the design of that study was limited to 114 muti-center trials which has its unique recruitment challenges.⁷³ The compliance with trial recruitment was significantly associated with the trial size (p = 0.009) and funding source (p = 0.02). Smaller trials were 1.3 times more compliant with recruitment than larger trials. We suspect that it would be easier to adequately recruit, manage and retain participants of smaller trials with clear logistical, operational and feasibility advantages over larger trials.

Selective registration bias of digital health clinical trials

Despite the emphasis on prospective trial registration introduced by the 2004 ICMJE mandate and the 2008 DoH, we found that only 38.4% of all included trials were registered prospectively.^22,24 Similar findings were reported by two independent studies indicating that compliance with prospective trial registration was at 31%.^74,75

We hypothesize that investigators may be more inclined, or biased, to registering their trial only when submitting their results to peer-reviewed journals for publication. To support our hypothesis, we found a statistically significant relationship (p < 0.001) between retrospective registration and the registration-to-publication duration in digital health clinical trials, for which we coined a new term as “Selective Registration Bias”.

The vast majority 95.7% of trials registered within one year before publication were registered retrospectively. Our results showed that investigators, who did not register their trials promptly prior to enrollment, were required or motivated to do so only when they submit their results to scholarly journals. There may be a number of contributing factors to this selective registration bias. Firstly, the investigator may have deferred the trial registration task until the completion of the trial and only when the results are finalized and ready to be published. Secondly, the investigator may also be not aware of the ethical expectation to register their trial promptly. Lastly, journal editors and peer-reviewers are likely to suggest registration of the submitted trial publication prior to accepting the publication submission.⁷⁶ It's important to establish and broaden the adoption of prompt trial registration requirement in ethical approval and guidelines of clinical trial design at the institutional and academic level.

Challenges in oncology trials

The trial condition groups were statistically associated with the non-publication rate (p = 0.042) as well as compliance with prospective trial registration (p = 0.02). Our results indicate that investigators of oncology trials in digital health are the most compliant with prospective trial registration at 47.4% compared to investigators of trials of other conditions in digital health trials, but they seem to face more challenges to publish their oncology trial results with the highest non-publication rate reported in oncology trials in digital health at 63.2%. The significantly higher underreporting of oncology trial results suggests challenges in conducting oncology studies in the realm of digital health trials. These challenges may align with other studies indicating a number of barriers to conducting traditional oncology trials including eligibility, recruitment, follow-ups, and oncologist and patient attitudes.^64–66 However, we postulate that there may be a few other challenges that are specific to digital health oncology trials, particularly in the recruitment and enrollment stage of those trials. These explicit barriers may be explained by the treating oncologist, or patient, preferences to enroll in other non-digital health trials, such as experimental drug trials, with more measurable clinical outcomes. The results from our study validate this postulate through two distinct data points. First, we found only one oncology trial in our study that was funded by a pharmaceutical sponsor, indicating the lack of interest from pharmaceutical sponsors to invest in digital health trials. Second, we reported the lowest compliance with participants recruitment for published oncology trials in digital health 42.9%, which is a clear indicator to enrollment barriers to digital health oncology trials.

Compliance with prospective trial registration in Australia and New Zealand

The ANZCTR leads with 89.7% compliance with prospective trial registration. Also, trials from Australia and New Zealand were leading with 89.5% compliance with prospective trial registration (the 0.2 percent decline is due to one trial that was registered in the ANZCTR registry and was not located in Australia or New Zealand). The high compliance with prospective trial registration in the ANZCTR registry was acknowledged by another study indicating an incremental trend in prospective trial registration from 48% to 63% for trials registered in the ANZCTR registry between 2006 and 2012.⁶³

Compliance with participants recruitment in the Middle East

Digital health trials in the Middle East had the lowest non-publication rate and the highest compliance with participants recruitment at 26.7% and 81.1%, respectively. These results are influenced by the Iranian trials registered in the IRCT registry with the lowest non-publication rate and the highest compliance with participants recruitment at 25% and 88.9% respectively. All trials registered in the IRCT were funded by university sponsors. This finding aligns with another study that reported that 97% of IRCT trials were funded by university and other governmental institutions.⁵² The academic and public sponsorship may be a contributing factor to low non-publication rate and high compliance in recruitment of trials registered in the IRCT as they would likely encourage a culture of adherence with best practices in trial design.

Several studies discussed the legacy of the IRCT and its role in upholding best practices in the field of clinical research in Iran.^67–69 However, we found that prospective trial registration was the lowest at 8.3% for trials registered in the IRCT. Our results support findings from another study that reported on the rationale behind the low prospective trial registration rate at 8.3% for trials registered in the IRCT between 2008 and 2011.⁷⁰ The low rate in prospective trial registration in the IRCT may have contributed to highest compliance in participants recruitment. When registering the trial retrospectively, in particular towards the end of the trial, the authors would have the opportunity to adjust the target trial size in the submitted registry information to match the actual number of participants.

The inclusion of TRN in published trials was the lowest for trials registered in the IRCT at 22.2%. Combined with the lowest prospective trial registration of those trials, we postulate that investigators of IRCT trials may be unaware of the ICMJE recommendation for prospective trial registration and the WHO re-affirmation to include the TRN in the trial publication.

Adherence to best practices in clinical trials from Europe

In our study, Europe was the region with (1) the largest number (43.6%) of digital health trials, (2) the highest compliance with the inclusion of TRN (68.7%), and the second-lowest non-publication rate (28%) tied with the Middle East The overall lead in European trials is influenced by the Netherlands National Trial Register which had the second-lowest non-publication rate at 25%, and the highest compliance with inclusion of TRN at 87.5%.

Enrollment to publication duration

The majority (88.3%) of the published trials in our study were published within five years after the enrollment of the first participant. We postulated that small trials would be easier to conduct as they have fewer recruitment and enrollment challenges, hence they would likely be published in a shorter time compared to large trials. To validate our postulate, we found statistically significant differences (p = 0.002) between trial size and enrollment-to-publication duration. We observed a reversed trend in publication between large and small trials at the fourth-year mark after trial enrollment. Small trials published their results 1.3 times more than large trials within the first four years after enrollment, after which large trials published their results 2.2 times more than small trials.

Limitations

Our study is the first to our knowledge that analyzed global digital health clinical trials registered in all the WHO data provider registries.⁴⁰ Our study was limited to 417 trials registered in any of the seventeen WHO data provider registries and did not examine other registries, such as the federal office of public health's portal for human research in Switzerland, and the Philippine health research registry.^77,78 As such, when identifying published trials in peer-reviewed journals, PubMed and Google search were utilized. We acknowledge that not considering trials registries other than the WHO data provider registries, not including databases other than PubMed and Google, and the limited number of included trials may have impacted the external validity of our study results. There are known sensitivity limitations with the ICTRP search portal.⁷⁹ We followed the recommendation and utilized the advance search function as described in Appendix II.

Despite the ICMJE and WHO emphasis on trial registration, not all randomized trials are registered.^75,80 We did not include unregistered trials in our analysis, which may impact the internal validity of our results.

We considered the enrollment and/or trial start date as provided in the registration information of the included trials. The registration information is provided manually and voluntarily by the registering investigators who are often overwhelmed with competing priorities and limited resources. Therefore, the enrollment date provided in the trial registries may not always be up-to-date or maintained promptly.

Lastly, our study only included registered trials enrolled in 2012. Our findings predate the recent 2015 WHO calls for improving public disclosure of trial results and the linkages between these results and the trial registry entries.²⁵

Conclusion

In the field of digital health randomized clinical trials, the adherence of investigators to the best practices of trial registration and result dissemination is still evolving. Further research is required to identify contributing factors and mitigation strategies to low compliance rate with trial publication and prospective registration in digital health clinical trials.

Supplemental Material

sj-docx-1-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-1-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-2-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-2-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-3-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-3-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-4-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-4-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-5-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-5-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-6-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-6-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-7-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-7-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-8-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-8-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-9-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-9-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-10-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-10-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Supplemental Material

sj-docx-11-dhj-10.1177_20552076221090034 - Supplemental material for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries

Supplemental material, sj-docx-11-dhj-10.1177_20552076221090034 for Prospective trial registration and publication rates of randomized clinical trials in digital health: A cross-sectional analysis of global trial registries by Mustafa Al-Durra, Robert P Nolan, Emily Seto and Joseph A Cafazzo in Digital Health

Footnotes

Conflict of interest

The authors have no conflicts of interest to declare.

Funding

The authors received no financial support for the research, authorship, and/or publication of this article.

Contributorship

The lead author (MAD) and senior author (JC) conceived the study idea and developed the study design with contributions from all authors. MAD executed the search queries, extracted data, analyzed the results, and drafted the manuscript. All authors interpreted results were involved in the reporting of the study, reviewing and editing the manuscript, and approved the final version of the manuscript. All authors agree to take responsibility for the work.

Ethical approval

Not applicable, because this article does not contain any studies with human or animal subjects.

Guarantor

MAD.

ORCID iD

Mustafa Al-Durra

Supplemental material

Supplemental material for this article is available online.

References

Blümle

Meerpohl

Schumacher

, et al. Fate of clinical research studies after ethical approval - follow-up of study protocols until publication. PLoS One 2014. PMID: 24586265.

Emanuel

Wendler

Grady

. What makes clinical research ethical? JAMA 283: 2701–2711. http://www.ncbi.nlm.nih.gov/pubmed/10819955. Accessed December 8, 2018.

Jones

Handler

Crowell

, et al. Non-publication of large randomized clinical trials: Cross sectional analysis. BMJ Open 2013. PMID: 24169943.

Lee

Bacchetti

Sim

. Publication of clinical trials supporting successful new drug applications: A literature analysis. PLoS Med 2008. PMID: 18816163.

Carter

Griffin

Carter

. A survey identified publication bias in the secondary literature. J Clin Epidemiol 2006. PMID: 16488354.

Liebeskind

Kidwell

Sayre

, et al. Evidence of publication bias in reporting acute stroke clinical trials. Neurology 2006. PMID: 17000963.

Moscati

Jehle

Ellis

, et al. Positive-outcome bias: Comparison of emergency medicine and general medicine literatures. Acad Emerg Med 1994. doi:10.1111/j.1553-2712.1994.tb02443.x. PMID: 7621207.

Sterling

Rosenbaum

Weinkam

. Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. Am Stat 1995.

Parekh-Bhurke

Kwok

Pang

, et al. Uptake of methods to deal with publication bias in systematic reviews has increased over time, but there is still much scope for improvement. J Clin Epidemiol 2011. PMID: 20800992.

10.

Song

Parekh

Hooper

, et al. Dissemination and publication of research findings: An updated review of related biases. Health Technol Assess (Rockv) 2010. PMID: 20181324.

11.

Dickersin

Min

. NIH Clinical trials and publication bias. Online J Curr Clin Trials. 1993. PMID: 8306005

12.

Weber

Callaham

Wears

, et al. Unpublished research from a medical specialty meeting. Why investigators fail to publish. J Am Med Assoc 1998. PMID: 9676674.

13.

Harrison

Mayo-Wilson

. Trial registration: Understanding and preventing reporting bias in social work research. Res Soc Work Pract 2014.

14.

Wager

Williams

. “Hardly worth the effort”? Medical journals’ policies and their editors’ and publishers’ views on trial registration and publication bias: quantitative and qualitative study. Br Med J 2013. PMID: 24014339.

15.

Simes

. Publication bias: The case for an international registry of clinical trials. J Clin Oncol. 1986. PMID: 3760920

16.

Chan

Hróbjartsson

Haahr

, et al. Empirical evidence for selective reporting of outcomes in randomized trials: Comparison of protocols to published articles. J Am Med Assoc 2004. PMID: 15161896.

17.

Dwan

Altman

Arnaiz

, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 2008. PMID: 18769481.

18.

Moher

Hopewell

Schulz

, et al. CONSORT 2010 Explanation and elaboration: Updated guidelines for reporting parallel group randomised trials. Br Med J 2010. PMID: 22036893.

19.

Dwan

Gamble

Williamson

, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias – An updated review. PLoS One 2013. PMID: 23861749.

20.

Jones

Keil

Holland

, et al. Comparison of registered and published outcomes in randomized controlled trials: A systematic review. BMC Med 2015. PMID: 26581191.

21.

Ioannidis

Caplan

Dal-Ré

. Outcome reporting bias in clinical trials: Why monitoring matters. Br Med J 2017. PMID: 28196819.

22.

De Angelis

Drazen

Frizelle

, et al. Clinical trial registration: A statement from the international committee of medical journal editors. Ann Intern Med 2004; 141: 477. PMID: 15364170.

23.

Gülmezoglu

Pang

Horton

, et al. WHO Facilitates international collaboration in setting standards for clinical trial registration. Lancet (London, England) 2005; 365: 1829–1831.

24.

Krleza-Jerić

Lemmens

. 7th Revision of the declaration of Helsinki: Good news for the transparency of clinical trials. Croat Med J 2009; 50: 105–110. PMID: 19399942.

25.

Moorthy

Karam

Vannice

, et al. Rationale for WHO’s new position calling for prompt reporting and public disclosure of interventional clinical trial results. PLoS Med 2015; 12: e1001819.

26.

Ross

Tse

Zarin

, et al. Publication of NIH funded trials registered in ClinicalTrials.gov: Cross sectional analysis. Br Med J 2012. PMID: 22214755.

27.

Chapman

Shelton

Mahmood

, et al. Discontinuation and non-publication of surgical randomised controlled trials: Observational study. Br Med J 2014. PMID: 25491195.

28.

Manzoli

Flacco

D’Addario

, et al. Non-publication and delayed publication of randomized trials on vaccines: survey. Br Med J 2014. PMID: 24838102.

29.

Faure

Hrynaszkiewicz

. The ISRCTN register: Achievements and challenges 8 years on. J Evid Based Med 2011; 4: 188–192.

30.

Viergever

Ghersi

. The quality of registration of clinical trials. PLoS One 2011; 6: e14701.

31.

Viergever

Karam

Reis

, et al. The quality of registration of clinical trials: Still a problem. PLoS One 2014; 9: e84727.

32.

Al-Durra

Nolan

Seto

, et al. Prospective registration and reporting of trial number in randomised clinical trials: global cross sectional study of the adoption of ICMJE and Declaration of Helsinki recommendations BMJ 2020; 369: m982. doi:10.1136/bmj.m982

33.

Donkin

Christensen

Naismith

, et al. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011. PMID: 21821503.

34.

Brouwer

Kroeze

Crutzen

, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011. PMID: 21212045.

35.

Kelders

Kok

Ossebaard

, et al. Persuasive system design does matter: A systematic review of adherence to web-based interventions. J Med Internet Res 2012. PMID: 23151820.

36.

Crutzen

de Nooijer

Brouwer

, et al. Strategies to facilitate exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults: A systematic review. Health Educ Behav 2011. PMID: 21189422.

37.

Clough

Casey

. Technological adjuncts to increase adherence to therapy: A review. Clin Psychol Rev 2011. PMID: 21497153.

38.

Chib

Van Velthoven

Car

. MHealth adoption in low-resource environments: A review of the use of mobile healthcare in developing countries. J Health Commun 2015. PMID: 24673171.

39.

Eysenbach

. Tackling publication bias and selective reporting in health informatics research: Register your eHealth trials in the international eHealth studies registry. J Med Internet Res 2004. PMID: 15471761.

40.

WHO. Primary registries. WHO 2018. https://www.who.int/ictrp/network/primary/en/. Accessed December 8, 2018.

41.

International Clinical Trials Registry Platform (ICTRP) Glossary. World health Organization. http://www.who.int/ictrp/glossary/en/. Accessed November 16, 2018.

42.

Pham

Wiljer

Cafazzo

. Beyond the randomized controlled trial: A review of alternatives in mHealth clinical trial methods. JMIR mHealth UHealth 2016; 4: e107.

43.

Ross

Mulvey

Hines

, et al. Trial publication after registration in ClinicalTrials.gov: A cross-sectional analysis. PLoS Med 2009. PMID: 19901971.

44.

Huser

Cimino

. Translational science on trialome: Retrolective study of publishing results of clinical trials (podium presentation). AMIA Annu Symp Proc. 2012.

45.

Hopewell

Clarke

Stewart

, et al. Time to publication for results of clinical trials. Cochrane Database Syst Rev 2007; 2: MR000011.

46.

Cugelman

Thelwall

Dawes

. Online interventions for social marketing health behavior change campaigns: A meta-analysis of psychological architectures and adherence factors. J Med Internet Res 2011. PMID: 21320854.

47.

Kohl

LFM

Crutzen

De Vries

. Online prevention aimed at lifestyle behaviors: A systematic review of reviews. J Med Internet Res 2013. PMID: 23859884.

48.

Schubart

Stuckey

Ganeshamoorthy

, et al. Chronic health conditions and internet behavioral interventions: A review of factors to enhance user engagement. Comput Inform Nurs 2011. PMID: 21164337.

49.

Källander

Tibenderana

Akpogheneta

, et al. Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low and middle-income countries: A review. J Med Internet Res 2013. PMID: 23353680.

50.

Clough

Casey

. Technological adjuncts to enhance current psychotherapy practices: A review. Clin Psychol Rev 2011. PMID: 21382535.

51.

Davies

. The information-seeking behaviour of doctors: A review of the evidence. Health Info Libr J 2007. PMID: 17584211.

52.

van den Berg

Schoones

Vliet Vlieland

TPM

. Internet-based physical activity interventions: A systematic review of the literature. J Med Internet Res 2007. PMID: 17942388.

53.

Lehto

Oinas-Kukkonen

. Persuasive features in web-based alcohol and smoking interventions: A systematic review of the literature. J Med Internet Res 2011. PMID: 21795238.

54.

Revere

Dunbar

. Review of computer-generated outpatient health behavior interventions: Clinical encounters “in absentia”. J Am Med Inform Assoc 2001. PMID: 11141513.

55.

Younger

. Internet-based information-seeking behaviour amongst doctors and nurses: A short review of the literature: review article. Health Info Libr J 2010. PMID: 20402799.

56.

WHO Trial Registration Data Set (Version 1.3.1). World health organization. http://www.who.int/ictrp/network/trds/en/. Accessed November 16, 2018.

57.

Huser

Cimino

. Linking ClinicalTrials.gov and PubMed to track results of interventional human clinical trials. PLoS One 2013; 8: e68409.

58.

Wood

AJJ

. Progress and deficiencies in the registration of clinical trials. N Engl J Med 2009; 360: 824–830.

59.

FDAAA 801 and the Final Rule. Clinicaltrials.gov. https://clinicaltrials.gov/ct2/manage-recs/fdaaa. Accessed November 17, 2018.

60.

HHS take steps to provide more information about clinical trials to the public. National institutes of health (NIH). https://www.nih.gov/news-events/news-releases/hhs-takes-steps-provide-more-information-about-clinical-trials-public. Accessed November 17, 2018.

61.

Page

Persch

. Recruitment, retention, and blinding in clinical trials. Am J Occup Ther 2013; 67: 154–161.

62.

Al-Durra

Nolan

Seto

, et al.

Nonpublication rates and characteristics of registered randomized clinical trials in digital health: Cross-sectional analysis

J Med Internet Res 2018;20:e11924, URL: https://www.jmir.org/2018/12/e11924, PMID: 30485832 PMCID: 6315268

63.

Hunter

Seidler

Askie

. Prospective registration trends, reasons for retrospective registration and mechanisms to increase prospective registration compliance: Descriptive analysis and survey. BMJ Open 2018; 8: e019983. PMID: 29496896.

64.

Ajithkumar

Gilbert

. Modern challenges of cancer clinical trials. Clin Oncol 2017; 29: 767–769. PMID: 29066171.

65.

Institute of Medicine. Transforming clinical research in the United States: Challenges and opportunities: workshop summary (forum on drug discovery, development, and translation). Washington, DC: National Academies Press (US), 2010.

66.

Unger

Cook

Tai

, et al. The role of clinical trial participation in cancer research: Barriers, evidence, and strategies. Am Soc Clin Oncol Educ Book 2016; 35: 185–198. Medline: 27249699.

67.

Solaymani-Dodaran

Vasei

Ghanei

. Iranian Registry of clinical trials: A four-year steady progress. Arch Iran Med 2013; 16: 671–674.

68.

Solaymani-Dodaran

Ostovar

Khalili

, et al. Iranian Registry of clinical trials: Path and challenges from conception to a world health organization primary register. J Evid Based Med 2009; 2: 32–35.

69.

Feizabadi

Fahimnia

Mosavi Jarrahi

, et al. Iranian Clinical trials: An analysis of registered trials in international clinical trial registry platform (ICTRP). J Evid Based Med 2017; 10: 91–96.

70.

Solaymani-Dodaran

Khalili

Hosseini

, et al. Iranian Registry of clinical trials two years on and the timing of registrations. J Evid Based Med 2011; 4: 168–171.

71.

McDonald

Knight

Campbell

, et al. What influences recruitment to randomised controlled trials? A review of trials funded by two UK funding agencies. Trials 2006; 7: 9.

72.

Campbell

Snowdon

Francis

, et al. Recruitment to randomised trials: Strategies for trial enrollment and participation study. The STEPS study. Health Technol Assess 2007; 11: iii, ix–105. http://www.ncbi.nlm.nih.gov/pubmed/17999843. Accessed December 7, 2018.

73.

Gillan

Ross

Gilbert

, et al. Recruitment to multicentre trials: the impact of external influences. Health Bull (Edinb) 2000; 58: 229–34. PMID: 12813832.

74.

Scott

Rucklidge

Mulder

. Is mandatory prospective trial registration working to prevent publication of unregistered trials and selective outcome reporting? An observational study of five psychiatry journals that mandate prospective clinical trial registration. PLoS One 2015; 10: e0133718. PMID: 26287998.

75.

Harriman

Patel

. When are clinical trials registered? An analysis of prospective versus retrospective registration. Trials 2016; 17: 187. PMID: 27079379.

76.

Manuscript Submission Guidelines. Digital Health Journal. https://journals.sagepub.com/author-instructions/dhj#ClinicalTrials. Accessed December 9, 2021

77.

The Federal Office of Public Health’s (FOPH) portal for human research in Switzerland. https://www.kofam.ch/en/swiss-clinical-trials-portal.html. Accessed December 8, 2018.

78.

The Philippine Health Research Registry (PHRR). http://registry.healthresearch.ph/. Accessed December 8, 2018.

79.

Waffenschmidt

Sow

Hausner

. The ICTRP search portal as an information source for systematic reviews - current problems and recommendations on how to deal with them. Abstracts of the global evidence summit, Cape Town, South Africa. Cochrane Database of Systematic Reviews 2017; (9 Suppl 1). https://doi.org/10.1002/14651858.CD201702.

80.

Drain

Robine

Holmes

, et al. Trial watch: Global migration of clinical trials. Nat Rev Drug Discov 2014; 13: 166–167.