Communicating Time-to-Event Treatment Effects in Randomized Trials: A Randomized Experiment among General Practitioners

Abstract

Background

Time-to-event outcomes from randomized controlled trials (RCTs) are often communicated without clearly conveying how treatment effects evolve over time. This can limit clinicians’ ability to interpret results and support patient decision making.

Methods

We conducted an online experiment with 250 German general practitioners in April 2024. Participants evaluated treatment effects presented in 4 common formats: hazard ratios, prolongation of life, restricted mean survival time (RMST), and absolute risk reduction. We assessed 1) understanding, defined as the ability to correctly compare effect sizes (small, medium, large), and 2) acceptability of each format. We also tested whether providing baseline risk information (control group outcomes) improved performance.

Results

Participants’ effectiveness ratings did not differ between small, medium, and large treatment effects in any format. RMST presentations were judged less effective but more acceptable than the other formats. Providing baseline risk information did not influence effectiveness ratings or acceptance.

Limitations

The use of a convenience sample may limit generalizability.

Conclusions

General practitioners struggled to interpret time-to-event treatment effects across all formats. Although RMST was preferred, no format supported accurate understanding of effect size.

Implications

Current approaches may not adequately support communication of time-to-event outcomes in clinical practice. More effective strategies are needed, likely combining absolute time-based measures with clear contextual information such as baseline risk.

Registration:

OSF 10.17605/OSF.IO/U69YM

Highlights

• Time-to-event outcomes from randomized trials are difficult for clinicians to interpret.

• General practitioners were unable to distinguish between small, medium, and large treatment effects across formats.

• Restricted mean survival time was preferred but did not improve understanding of time-to-event effects.

• Current formats do not support communication of time-to-event outcomes in clinical decision making.

Clinicians need to understand results from randomized controlled trials (RCTs) in order to recommend treatments and make decisions that are consistent with current evidence. ¹ However, many clinicians have difficulty interpreting trial statistics. ² One reason is that treatment effects are often reported in relative formats, such as odds ratios, relative risks, or hazard ratios (HRs), which can be hard to interpret and difficult to relate to patient care.^3–5

To improve understanding, many authors recommend presenting treatment effects in absolute terms, ideally showing outcomes in both the treatment and control groups.^3–7 Absolute formats can make evidence more concrete and easier to use in clinical communication. However, many clinical trials assess time-to-event outcomes, such as death or disease progression, and simple absolute risk displays often do not show how treatment effects develop over time. At the start of a trial, there is no difference between groups by definition. Differences emerge during follow-up, and their size depends on when outcomes are assessed. As a result, effect estimates can vary substantially depending on the chosen time point. For example, a treatment that reduces mortality from 8.3% to 5.7% after 4 y could also be described as reducing mortality from 2.4% to 1.9% after 1 y.^7,8 This makes communication of time-to-event outcomes especially challenging.

The HR is one of the most common summary measures for time-to-event outcomes, but it is also one of the most difficult to communicate and understand. ⁸ One problem is that the HR is a relative measure. Like other relative effect measures, it does not by itself show how many patients are likely to benefit. Without additional information about outcomes in the control group, clinicians cannot judge the practical magnitude of the effect. Relative measures may also make treatments appear more effective than absolute measures because their numerical values are often more striking.^5,8–10 A second problem is conceptual. Hazards are event rates, not probabilities, and HRs should not be interpreted as direct statements about risk reduction. ¹¹ This distinction is often misunderstood. Measures based on survival time may therefore offer an advantage because they express treatment effects on a time scale that is more intuitive and closer to everyday experience.^12,13

However, previous comparisons of HRs with other effect measures have often mixed together 2 separate issues: whether the format is relative or absolute and whether it is based on hazards or survival time.^8–10 For clinicians, both issues may matter, but their separate contributions to understanding have not been well examined.

In the current study, we examined how general practitioners interpret time-to-event treatment effects when presented in 4 commonly used formats: HR, prolongation of life (POL), restricted mean survival time (RMST), and absolute risk reduction (ARR). In addition, we tested whether providing baseline risk information from the control group influences interpretation and evaluation of these formats. Our primary outcome was understanding, defined as the ability to distinguish between small, medium, and large treatment effects. We assessed this by examining whether participants’ effectiveness ratings reflected the correct ordering of these effect sizes. As a secondary outcome, we assessed acceptance of each format, including perceived understandability, transparency, trustworthiness, honesty, and informativeness. Based on prior research, we expected that relative formats would lead to higher perceived effectiveness than absolute formats, while absolute formats would better support accurate comparisons of effect sizes. We also expected that providing baseline risk information would improve both understanding and acceptance. Finally, we explored whether participants’ knowledge of risk statistics influenced these outcomes and whether the different formats affected recall of the presented information.

Methods

Participants

We conducted an online study with general practitioners in Germany between April 23 and 29, 2024. We recruited participants through Bilendi, a Berlin-based panel provider with approximately 250,000 active panelists. We aimed to recruit 250 general practitioners. Eligibility was confirmed at the beginning of the survey. A total of 250 participants completed the study and were included in the analyses. Power calculations indicated that at least 222 participants would be required to detect a medium-sized effect (f = 0.25) with 80% power at a significance level of 0.05. ¹⁴

Design and Procedure

We conducted an online experiment with a mixed design. Each participant evaluated treatment effects presented in 4 formats: 1) HR, 2) POL, 3) RMST, and 4) ARR. Participants were randomly assigned to 1 of 2 groups: a) baseline risk provided (control group outcomes included) or b) baseline risk not provided. In addition, participants were assigned to 1 of 3 effect sizes (small, medium, large).

After providing informed consent, participants were presented with treatment effects in all 4 formats in randomized order. Each participant saw only 1 level of effect size (small, medium, or large), but this same effect was shown across all formats. After each presentation, participants rated the perceived effectiveness of the treatment and the acceptability of the communication.

Participants then completed questions on demographics and knowledge of risk statistics. At the end of the survey, they were asked to recall the effect sizes presented in each format. The study was approved by the responsible ethics committee of Charité– Universitätsmedizin Berlin. All materials and data are available at osf.io/n5vy9.

Stimulus Materials

The 4 communication formats described the same underlying treatment effect using different statistical representations. We adapted the wording from previous research. ⁹

In half of the sample, descriptions additionally included baseline risk information, defined as the outcome in the control group. We derived this information from the Empagliflozin Cardiovascular Outcome Event Trial in Type 2 Diabetes Mellitus Patients (EMPA-REG OUTCOME) trial. ¹⁵ We based effect sizes on HRs of 0.9 (small), 0.7 (medium), and 0.5 (large), and we translated these values into the other formats by simulation.^12,15–17 Table 1 shows examples of the wording used for each format.

Table 1

Example Wording of Time-to-Event Treatment Effects across Communication Formats

Format	Baseline Risk Provided	Baseline Risk Not Provided
HR	A patient treated with F reduces their risk of death (hazard) by [10%/30%/50%] compared to a similar patient receiving placebo. In the placebo group, the monthly risk of death was 0.3% over 5 y.	A patient treated with F reduces their risk of death (hazard) by [10%/30%/50%] compared to a similar patient receiving placebo.
POL	A patient treated with S lives [8%/26%/44%] longer than a similar patient receiving placebo, who lives on average 22.3 y.	A patient treated with S lives [8%/26%/44%] longer than a similar patient receiving placebo.
RMST	A patient treated with O lives [0.4/1.2/2.0] months longer over a 5-y period than a similar patient receiving placebo, who lives on average 22.3 y.	A patient treated with O lives [0.4/1.2/2.0] months longer over a 5-y period than a similar patient receiving placebo.
ARR	A patient treated with V reduces their risk of death by [1.3/4.1/6.9] percentage points over 5 y compared to a similar patient receiving placebo. In the placebo group, the 5-y risk of death was 14%.	A patient treated with V reduces their risk of death by [1.3/4.1/6.9] percentage points over 5 y compared to a similar patient receiving placebo.

ARR, absolute risk reduction; HR, hazard ratio; POL, prolongation of life; RMST, restricted mean survival time. Each participant saw all 4 formats. Baseline risk information (control group outcomes) was provided to half of the sample. Values in brackets represent small, medium, and large effects.

Measures

Primary and secondary outcomes

Perceived effectiveness of time-to-event treatment effects (primary outcome)

We assessed the perceived effectiveness using two 100-point semantic differential scales (perceived effectiveness and recommendation): “How effective is this treatment in preventing premature death?” and “Would you recommend this treatment to a patient?” Given their high internal consistency (Cronbach’s α = 0.87–0.95), we averaged responses to form a single effectiveness score, which we used as the dependent variable in mixed analyses of variance (ANOVAs) with communication format, effect size, and baseline risk as factors.

Acceptance of the communication of time-to-event treatment effects

Participants rated each format on five 100-point scales assessing understandability, transparency, trustworthiness, honesty, and informativeness. We averaged responses to create an overall acceptance score (Cronbach’s α = 0.95–0.96).

Recall of time-to-event treatment effects

At the end of the survey, participants were asked to recall the effect size presented in each format. Responses were recorded in the same format as originally presented.

Measures to describe patient characteristics

We collected additional information to describe the sample to examine if these aspects were associated with the primary outcomes. First, we assessed clinicians’ understanding of risk statistics using 3 items: 1) interpretation of a HR (correct answer: the effect cannot be determined in absolute terms), 2) conversion of relative risk reduction into absolute numbers, ¹ and 3) interpretation of increases in life expectancy (mean, median, or modal survival). Second, we assessed potential exposure to pharmaceutical industry influence by asking participants how often they were visited by pharmaceutical representatives within a given time period. ¹⁸ Finally, we recorded medical experience as the number of years participants had worked in medicine.

Statistical Analysis

We analyzed the data using mixed-effects models to examine the effects of communication format, effect size, and baseline risk information on outcomes. We used mixed ANOVAs to assess perceived effectiveness and acceptance and a logistic mixed model for recall, with communication formats, effect sizes, and baseline risk as factors. We conducted all analyses in R (version 4.2.2), and we defined statistical significance as P < 0.05. All materials, data, and code are publicly available at osf.io/n5vy9.

Results

Sample Characteristics

A total of 317 individuals were invited to participate. Of these, 260 confirmed that they were general practitioners and agreed to take part. In total, 250 participants completed the study and were included in the analyses. Among participants, 61.6% (n = 154) were male and 1.2% (n = 3) identified as nonbinary. Participants had a mean of 10.1 y of medical experience. Most participants (66.0%) reported being visited by pharmaceutical representatives at least once per month.

Understanding of risk statistics in general was limited. Only 12.8% of participants correctly identified that an HR does not directly indicate an ARR or a time-based effect. In addition, 28.0% correctly converted a relative risk reduction into an absolute risk. When asked about life expectancy, participants most commonly interpreted increases as changes in the most likely (modal) age at death (48.4%), followed by the mean age (37.6%). Sample characteristics were similar across experimental groups (Table 2).

Table 2

Descriptive Information per Experimental Cell

Reference Provision	Reference						No Reference
Effect Size	Small		Medium		High		Small		Medium		High
	M	SD	M	SD	M	SD	M	SD	M	SD	M	SD
Final n	41		42		42		42		41		42
Dropout	1		2		3		0		1		3
Male	58%		62%		69%		64%		63%		57%
Years of medical experience	11.02	11.42	9.93	9.52	10.26	10.55	10.33	10.63	9.61	9.2	9.14	9.04
Pharmaceutical exposure	2.88	1.33	2.76	1.25	3.07	1.49	3.24	1.49	3.07	1.42	3.4	1.36
Correct hazard ratio understanding	12%		14%		12%		14%		10%		14%

Perceived Effectiveness of Time-to-Event Treatment Effects (Primary Outcome)

Perceived effectiveness differed depending on how treatment effects were presented, F(2.5, 610.35) = 10.37, P < 0.001, η² = 0.041). Treatments described using RMST were judged as less effective than those presented using HRs, POL, or ARR. The latter 3 formats did not differ from each other (Figure 1A).

Figure 1

Effects of communication format on perceived effectiveness and acceptance of time-to-event treatment effects. (A) Perceived effectiveness of the time-to-event formats (combined rating of effectiveness and recommendation). (B) Acceptance of the communication (combined rating of understandability, transparency, trustworthiness, honesty, and informativeness). Scores range from −50 (lowest) to 50 (highest). Across time-to-event formats, restricted mean survival time (RMST) was rated as less effective but more acceptable than the other formats. Differences between effect sizes and baseline risk conditions were not observed. ARR, absolute risk reduction; HR, hazard ratio; POL, prolongation of life.

There was no evidence that the size of the treatment effect (small, medium, large) influenced perceived effectiveness, F(2, 244) = 1.43, P = 0.24, and no interaction effects were observed. Providing baseline risk information also had no effect (all P ≥ 0.66). These findings were similar regardless of participants’ understanding of HRs or relative probabilities.

Acceptance of the Communication of Time-to-Event Treatment Effects

Acceptance of the information also depended on the time-to-event formats (Figure 1B), F(2.64, 643.63) = 10.44, P < 0.001, η² = 0.041. RMST was rated as more acceptable than the other formats, F(2.64, 643.63) = 10.44, P < 0.001 (Figure 1B).

Neither the size of the treatment effect nor the provision of baseline risk information influenced acceptance (all P ≥ 0.172). Overall, ratings were below the midpoint of the scale, indicating that the communication was generally perceived as more unacceptable than acceptable.

Recall of the Time-to-Event Treatment Effects

Overall, recall of effect sizes was low across all formats. Participants were more likely to correctly recall HRs than the other formats, likely because these were presented as round numbers (χ²[3] = 28.24, P ≤ 0.001; 14% v. 7.2%, 5.2%, and 4.4% for the other formats). A full model including all factors could not be estimated because in some conditions, no correct responses were observed.

Discussion

Main Results

In this study, the findings indicate that commonly used presentations of time-to-event outcomes (HRs, POL, RMST, and ARR) do not support clinicians in differentiating between larger and smaller treatment effects based on their effectiveness ratings. Notably, this pattern was observed across all formats examined and was not modified by the provision of additional contextual information such as baseline risk. While participants differed in their evaluation of format acceptability, with RMST being considered most acceptable, these preferences were not associated with judgments that more closely aligned with the magnitude of the underlying effect.

The lower perceived effectiveness of RMST compared with the other formats may reflect differences in familiarity. Relative measures such as HRs are commonly used in clinical research and practice, which may influence how results are interpreted. In contrast, time-based absolute measures such as RMST may appear less familiar and therefore lead to lower effectiveness ratings, even when they represent the same underlying effect. At the same time, the absence of differences between HRs, POL, and ARR suggests that commonly assumed advantages of absolute measures—as reported in many other studies^5,8–10—were not evident in this context. This may be due to differences in the comparison formats used or to an increasing awareness among clinicians of the distinction between relative and absolute effects.

Implications for Risk Communication

Taken together, the findings suggest that current approaches to presenting time-to-event outcomes may not be sufficient to ensure that clinicians’ judgments reflect differences in effect magnitude. This has potential implications for clinical decision making, as accurate assessment of treatment benefits is important for weighing options and communicating expected outcomes to patients.

Improving communication may require more than selecting a single preferred metric. Although RMST was rated as more acceptable, this did not translate into judgments that were more sensitive to differences in effect size. This suggests that acceptability and interpretability may represent distinct dimensions of communication quality. Approaches that combine multiple elements—such as expressing effects on an absolute scale, presenting outcomes over time, and providing clear contextual benchmarks—may be more effective than relying on a single summary measure.

In addition, providing baseline risk information alone did not influence judgments in this study, indicating that simply adding information may not be sufficient if the overall presentation remains difficult to interpret. Future research should examine how different forms of contextualization (e.g., comparisons with familiar treatments or reference values) affect clinicians’ interpretation of time-to-event outcomes.

Limitations

Our study has several limitations. First, participants were recruited from an online panel and self-identified as general practitioners. Although their responses were consistent with prior research, the sample may not fully represent the broader population of clinicians. Second, the study used a simplified and abstract treatment scenario. In clinical practice, additional contextual factors—such as disease severity, competing risks, and patient preferences—may influence how treatment effects are evaluated. Third, the study assessed judgments of effectiveness rather than directly measuring understanding of statistical concepts. The absence of differences in effectiveness ratings across effect sizes suggests that participants’ judgments were not sensitive to effect magnitude, but this should not be interpreted as a direct measure of underlying comprehension. Finally, participants evaluated only 1 effect size, which precludes within-subject comparisons and may have increased the difficulty of identifying differences in magnitude.

Conclusion

General practitioners’ judgments of treatment effectiveness did not reflect differences in effect size across 4 commonly used formats for time-to-event outcomes. Although RMST was rated as more acceptable, no format led to judgments that were more closely aligned with the magnitude of the underlying effect. These findings suggest that current approaches to communicating time-to-event outcomes may not adequately support clinicians in assessing treatment benefits. Future work should focus on developing and testing communication strategies that improve the alignment between presented evidence and clinicians’ judgments.