Using causal models to show the effect of untestable assumptions on effect estimates in randomized controlled trials

Abstract

Background The methods by which randomized controlled trials (RCTs) are analyzed rest on several assumptions, most of which are untestable.

Purpose To show how estimates of the net effect of treatment on survival can be obtained requiring only the assumption that randomization produced equivalent groups.

Methods The assumptions underlying ratio measures of effect, based on disease occurrence times (DOT) obtained from survival curves, are identified and cumulatively removed.

Results The four assumptions usually made are that (1) the ratio of disease incidence rates under treatment and under reference exposure is constant over time, (2) the groups being compared are exchangeable (equivalent), (3) a subject's DOT under treatment is independent of what his DOT would have been under the reference exposure, and (4) the treatment effect, if any, is in the same direction in all subjects. Removing Assumption 4 leads to an estimator of effect resembling the etiologic fraction, but able to accommodate both causative and preventive effects. Removing all assumptions but that of exchangeability still permits the estimation, directly from the survival curves, of a range of effect magnitudes, causative or preventive, compatible with the observed DOTs. The exchangeability assumption is the easiest to meet, by randomizing enough subjects.

Limitations The statistical uncertainty that affects the estimates of survival probabilities has been ignored. Taking uncertainty into account further widens the range of effects compatible with the observations.

Conclusions Retaining only the exchangeability assumption allows for a range of possible treatment effects to be estimated, although it may be wide. Readers of RCT reports should understand that the determination of a point estimate of effect within this range is entirely a function of unverifiable analytic assumptions. Clinical Trials 2007; 4: 611—620. http://ctj.sagepub.com

Get full access to this article

View all access options for this article.

References

Editors. Looking back on the millennium in medicine. New Engl J Med 2000; 342: 42—48.

Collins R. , MacMahon S. Reliable assessment of the effects of treatment on mortality and major morbidity, I: clinical trials. Lancet 2001; 357: 373—80.

Grann A. , Grann VR The case for randomized trials in cancer treatment: New is not always better . JAMA 2005; 293: 1001—003.

Kramer MS , Shapiro SH Scientific challenges in the application of randomized trials. JAMA 1984; 252: 2739—45.

Rothman KJ , Greenland S. Modern Epidemiology (2nd edn) Lippincott-Raven, Philadelphia, PA, 1998.

Gray-Donald K. , Kramer MS Causality inference in observational vs. experimental studies: An empirical comparison. Am J Epidemiol 1988; 127: 885—92.

Greenland S. , Robins JM Identifiability, exchangeability, and epidemiological confounding. Int J Epidemiol 1986; 15: 413—19.

Holland PW Statistics and causal inference. J Am Statistical Assoc 1986; 81: 945—70.

Rubin DB Bayesian inference for causal effects: the role of randomization. Annals Stat 1978; 6: 34—58.

10.

Kaplan EL , Meier P.

Nonparametric estimation from incomplete observations. Amer Stat

Assoc J 1958; 53: 457—81.

11.

Cox DR Regression models and life-tables. J R Statist Soc B 1972; 34: 187—202.

12.

Kramer MS , Lane DA Causal propositions in clinical research and practice. J Clin Epidemiol 1992; 45: 639—49.

13.

Walker AM Observation and Inference: An Introduction to the Methods of Epidemiology . Epidemiology Resources Inc., Chestnut Hill MA, 1991.

14.

Greenland S. Relation of probability of causation to relative risk and doubling dose: A methodological error that has become a social problem . Am J Public Health 1999; 89: 1166—69.

15.

Robins J. , Greenland S. The probability of causation under a stochastic model for individual risk . Biometrics 1989; 45: 1125—38.

16.

Halloran ME Concept and estimation of attributable risks in HIV epidemiologic research . In Nicolosi E. (ed). HIV Epidemiology: Models and Methods. Raven Press, Ltd , New York, 1994, pp. 211—227.

17.

MacMahon B. , Trichopoulos D. Epidemiology : Principles and Methods (2nd edn), Brown and Company, Boston: Little, 1996.

18.

Miettinen OS Proportion of disease caused or prevented by a given exposure, trait or intervention. Am J Epidemiol 1974; 99: 325—32.

19.

Allard R. , Boivin JF Measures of effect based on the sufficient causes model. 2. Risks and rates of disease associated with a single preventive agent. Epidemiology 1993; 4: 517—23.

20.

Allard R. , Boivin JF Measures of effect based on the sufficient causes model. 1. Risks and rates of disease associated with a single causative agent. Epidemiology 1993; 4: 37—42.

21.

Allard R. , Boivin JF , Lepage Y. Measures of effect based on the sufficient causes model. 3. Multivariate analysis . Epidemiology 1997; 8: 93—98.

22.

Fischl MA , Richman DD , Grieco MH , et al. The efficacy of azidothymidine (AZT) in the treatment of patients with AIDS and AIDS-related complex: A double-blind, placebo-controlled trial . New Engl J Med 1987; 317: 185—91.

23.

Greenland S. , Robins JM Conceptual problems in the definition and interpretation of attributable fractions. Am J Epidemiol 1988; 128: 1185—97.

24.

Writing Group for the Women's Health Initiative Investigators. Risks and benefits of estrogen plus progestin in healthy postmenopausal women: Principal results from the women's health initiative randomized controlled trial. JAMA 2002; 288: 321—33.

25.

Gillespie MJ , Fisher L. Confidence bands for the Kaplan-Meier survival curve estimate. Annals Stat 1979; 7: 920—24.

26.

MacLehose RF , Kaufman S. , Kaufman JS , Poole C. Bounding causal effects under uncontrolled confounding using counterfactuals. Epidemiology 2005; 16: 548—55.

27.

King G. A Solution to the Ecological Inference Problem. Princeton University Press, Princeton NJ, 1997.

28.

Rothman KJ Causes. Am J Epidemiol 1976; 104: 587—92.

29.

Weinberg CR Applicability of the simple independent action model to epidemiologic studies involving two factors and a dichotomous outcome . Am J Epidemiol 1986; 123: 162—73.

30.

Little RJ , Rubin DB Causal effects in clinical and epidemiological studies via potential outcomes: concepts and analytical approaches. Annu Rev Public Health 2000; 21: 121—45.

31.

Rubin DB Comment [on Dawid AP: Causal inference without counterfactuals]. J Amer Stat Assoc 2000; 95: 435—38.