Beyond Randomized Controlled Trials

Abstract

Complementary and alternative medicine is criticized by some as lacking evidence to support the effectiveness of its methods and medicines. Such critics typically point to mixed results from using randomized controlled trials to test complementary and alternative medicine. Randomized controlled trials have been held to be the “gold standard” in pharmaceutical research, but a growing body of evidence in orthodox journals has identified their limitations. Here, 5 fundamental flaws in the randomized controlled trial–based model are discussed as well as the impact on its relevance for testing complementary and alternative medicine therapies. A better way to evaluate complementary and alternative medicine therapies is also proposed. A 7-item checklist is suggested to quantify the strength of an area of complementary and alternative medicine research.

Over the past 50 years, randomized controlled trials have become the “gold standard” of orthodox medical research. However, such research is costly, and much of it has been driven by pharmaceutical companies racing to find profitable (ie, patentable) drugs to add to their offerings while focusing on increasingly narrow definitions of benefits. Rarely is research done simply to determine the best or even good treatments for patients because there is frequently no way to recoup the investment through the sales of proprietary products.

Complementary and alternative medicine takes a holistic approach and offers natural products that are “generally recognized as safe” and physical treatments and behavioral interventions (such as dietary changes) that are not well evaluated by narrowly defined randomized controlled trials. This article critiques the use of randomized controlled trials as a primary model in complementary and alternative medicine research and proposes that there is a better way to evaluate complementary and alternative medicine therapies.

The Value of Randomized Clinical Trials

The real-world validity of clinical findings based on randomized controlled trials is being increasingly questioned in orthodox journals. ¹ In fact, randomized controlled trials have validated a number of unsafe drugs that have later been withdrawn from the market or had warnings issued because of adverse effects that randomized controlled trials failed to uncover. Recent examples include Vioxx, Avandia, Posicor, Wellbutrin, Depakote, Accutane, Reglan, and Nuvaring. In 2009 alone, the US Food and Drug Administration (FDA) received 19 551 reports of patient deaths associated with drugs that had passed randomized controlled trials. ² In 2003, the head of GlaxoSmithKline stated that more than 90% of drugs only work in 30% to 50% of people. ³ It would be irresponsible to ignore such facts given the potential harm that the use of unsafe medication could cause to patients as well as the economic waste produced nationally from often ineffective medication causing further health problems for consumers.

Evidence shows that the randomized controlled trial–based model is flawed for both methodological and ethical reasons in 5 main areas, which also makes it unsuitable to test some complementary and alternative medicine interventions:

Suitability. Randomized controlled trials were designed initially to test the potential toxicity of chemicals (pharmaceutical drugs) that had never been used widely in practice. However, complementary and alternative medicine medicines are most often nontoxic and have a long history of safe, traditional use. Furthermore, when looking at drug benefits, randomized controlled trials generally assess a single, specifically defined potential benefit of the chemical while ignoring any other positive or negative nonspecific effects. Complementary and alternative medicines often produce many related positive nonspecific effects, thereby having broader, more general positive health effects. The “efficacy paradox” described by Walach ⁴ shows that randomized controlled trials can favor a method that provides fewer total benefits to the patient to an alternative method with large, positive nonspecific effects. Randomized controlled trials can also miss serious and even fatal side effects, such as in the cases of the drugs mentioned above.

Relevance. The model contains a trade-off between technical rigor and real-world relevance. For a randomized controlled trial to deliver a technically strong result, many restrictive assumptions are needed; but the more the assumptions made, the less relevant the results become in the real world. For example, only 1 of the potentially many effects of the substance being tested is measured, but it is assumed that this is sufficient to assess its real-world value. In particular, issues of homogeneity (or heterogeneity) and generalizability are significant in human trials, but this means that there is bias in every randomized controlled trial. For example, homogeneity relates to the need to have participants in the verum (active treatment) group matched with participants in the placebo group. This is almost impossible except in the most general of terms (ie, the same ratio of men and women in both groups). Similarly with generalizability, which ideally requires the characteristics of the group of people receiving treatment to match the characteristics of the general population. In practice, this can be done in very broad terms only, once again lessening the reliability and the relevance of the results.

Structural. Ioannidis ⁵ has published rigorous analyses showing that “in modern research false findings may be the majority or even the vast majority of published research claims.” Ioannidis identified 6 “corollaries” causing research findings to be less likely to be true in a scientific field: (a) small study size; (b) small effect sizes; (c) a large number of untested relationships (ie, confirmatory studies are more reliable); (d) great flexibility in designs, definitions, outcomes, and analytical modes; (e) significant financial and other interests and prejudices; and (f) a “hot” scientific field (with more scientific teams involved, resulting in inappropriate competition). He then suggested 3 characteristics that would improve the poststudy probability results: (a) better-powered evidence—that is, large studies or low-bias meta-analyses; (b) most research questions being addressed by many teams, with no single finding being emphasized; and (c) more focus on prestudy odds that a true relationship is being tested, instead of chasing statistical significance. This last point will be considered below, and is potentially contentious. A few contributors have contested the findings of Ioannidis. ⁵ The most well known, by Goodman and Greenland, ⁶ has been rebutted convincingly by Ioannidis. ⁷

Integrity. The randomized controlled trial model is susceptible to abuse by dishonest researchers. A good example is the now discredited meta-analysis by Shang and colleagues ⁸ published in 2005, on the basis of which the editors of Lancet heralded the death of homeopathy. ^8–10 Investigation has uncovered sponsorship bias ^11,12 ; reporting bias ¹³ ; fabrication and falsification of research data ^14,15 ; ethical issues, such as plagiarism and data fabrication in medical establishment peer-reviewed journals ¹⁶ ; and the fact that most randomized controlled trials are now conducted by offshore contract research organizations with questionable checks and balances. ¹⁷

Probity. The major multinational pharmaceutical companies who now fund more than 60% of all medical research have a history of civil and criminal convictions for dishonest activities. ¹⁸ Recent fines levied against Pfizer and GlaxoSmithKline totaled billions of dollars. The US Department of Justice reported recovering US $18 517 016 689 from October 1, 1987, to September 30, 2010, in 4668 cases involving the Department of Health and Human Services as the primary client. The amounts are staggering, but a small amount for an industry that has annual drug sales of US $300 billion in the United States alone. ¹⁹ And these are not accidental oversights. For example, GlaxoSmithKline knew in 1989 that its antidepressant drug Paxil increased the risk of suicide by 8 times compared with placebo. The FDA approved Paxil on December 29, 1992, on the basis of “bad” data from GlaxoSmithKline, which hid this fact. ²⁰ Pfizer was accused of paying kickbacks to doctors for prescribing their drugs. ²¹

Implications for Complementary and Alternative Medicine

Complementary and alternative medicine practitioners and their associations face 2 related challenges: (a) a complementary and alternative medicine therapy is evaluated based on randomized controlled trials, which makes a negative result almost inevitable, and then the complementary and alternative medicine therapy is criticized as being unsound by advocates of randomized controlled trials; and (b) complementary and alternative medicine groups feel obliged to use the randomized controlled trial–based approach to justify their therapies when, in fact, validation is best conducted using other research techniques. Complementary and alternative medicine practitioners should hold firm in their support of therapies that deserve to be tested and validated for 1 simple reason—they work.

On the other hand, the analysis by Ioannidis ⁵ means that the results of many complementary and alternative medicine trials are unreliable for the same reasons that many drug trials cannot be trusted. Novella ²² reviewed and extended the work of Ioannidis ⁸ and suggested 7 ways to improve results. However, some in conventional medicine believe that this means that complementary and alternative medicines should not be tested at all. ²² This shows the classic mindset of the “prejudiced skeptic”—prevent complementary and alternative medicine being researched simply because it is different from the randomized controlled trial–based system, then criticize it for being untested. This is not objective and intelligent science.

Complementary and alternative medicine has an obligation to provide reasonable and objective evidence that its methods are safe and effective and offer value (in every way that can be measured) to the consumer/patient and to national health schemes. It is clear that complementary and alternative medicine does very well using case-controlled studies but less well when evaluated using only randomized controlled trials. ²³ What is needed are effective evaluation methods for complementary and alternative medicine modalities that do not focus on a limited specific effect but fully evaluate relevant nonspecific effects.

Walach et al ²⁴ have suggested a “circular hierarchical model” that recognizes all levels of evidence. This model uses results from randomized controlled trials, large long-term observational studies, comparative studies in pragmatically selected cohorts, appropriate cohort studies, retrospective audits of large well-conducted data sets, prospective documentation of pragmatically treated cohorts, and large single-group observational studies, which means that all relevant data are appropriately incorporated in the evaluation. What is needed most is objective common sense. This will also be appreciated by the average consumer who wants his or her information in plain language and not inaccessible technical speak, which often hides what the consumer really wants to know (“Will it make me feel better, as well as helping my specific complaint?”).

The above discussion suggests that complementary and alternative medicine researchers should attempt to conduct research in the following way to obtain the most meaningful results:

study mainly complementary and alternative medicine interventions/medicines that are known from considerable experience to be effective;

use the largest sample sizes that can be afforded; large outcome studies combined with economic cost–benefit analyses are ideal;

research those interventions/medicines with a significant effect size before studying those with low or uncertain levels of effectiveness;

study an intervention/medicine using a variety of research methods. This should include outcome comparisons between pharmaceutical and complementary and alternative medicine interventions for particular conditions;

evaluate the intervention/medicine based on the totality of evidence available, not just individual studies;

focus on studies testing an established intervention/medicine in preference to new interventions/medicines; and

publish and consider all studies, whether they support or reject the hypothesis.

Complementary and alternative medicine researchers can and should do this and should without hesitation contest accusations that complementary and alternative medicines are “unscientific” by advocates of randomized controlled trials when the accusations are based on inappropriate testing or an ignorance of the evidence that already exists.

As an example, in Table 1 , we evaluate the evidence to date supporting the effectiveness of homeoprophylaxis, or the prevention of communicable illness in populations using homeopathically prepared oral immunizations, in terms of the 7 factors above.

Table 1.

Evidence Matrix Supporting the Effectiveness and Safety of Homoeoprophylaxis (HP)

Number	Tests	Evidence/Experience
1	Prestudy odds of positive effect	HP has been used successfully since 1798 when Dr Hahnemann first used Belladonna to prevent scarlet fever. There have been a number of studies using different statistical techniques producing consistent results. “Plausibility” is not relevant when evaluating objective data
2	Sample sizes (n)	Many studies, n < 100; Golden, n > 2400; Mroninski, n > 80 000. Finlay Institute, n > 2.2 million (2007), n > 2.2 million (2008), n > 9.8 million (2009/2010)
3	Effect size	Studies suggest 90% effectiveness. This is equivalent to that of most vaccines and is substantial
4	Variety of methods	Methods include pseudorandomized clinical trials, prospective and retrospective cohort studies, questionnaire studies, and descriptive studies as well as 2 centuries of clinical experience
5	Totality of studies	Four types of evidence: (1) 200 years of clinical evidence; (2) short-term epidemic; (3) long-term endemic; (4) provincial and national. There is consistency of results across all types of studies
6	Test established methods	Whereas HP is used in a variety of ways depending on individual circumstances, the method is well established and is based on the principle of similars
7	Publish all studies	The number of human studies is relatively small, and all known studies in English were listed in Golden's review 25

Homeoprophylaxis probably represents the most challenging of all complementary and alternative medicine areas of research for a range of philosophical, methodological, and medicopolitical reasons; however, it is possible to tick most of the boxes with this difficult example. This suggests that rigorous, objective, “common sense” testing can be done for most if not all complementary and alternative medicine methods and medicines.

Conclusions

Recurring attacks on the evidence base of complementary and alternative medicine are often a result of personal bias and at times unreasoned (and therefore unscientific) opposition to tested therapies that offer patients choice in their methods of health care. Although the majority of orthodox physicians are motivated people who genuinely care for the well-being of their patients, their training directs them into a mindset that allows only a very narrow range of views to be held.

It can be a challenge to remain objective and reasonable in the face of such unscientific attacks. Complementary and alternative medicine leaders must ensure that complementary and alternative medicine interventions/medicines are rigorously tested but in ways that are appropriate as well as being thorough. Checklists should be developed, similar to the above, that complementary and alternative medicine researchers can use to tick off and then stand by the results as being at least as reliable as those presented by pharmaceutical trials. We have to change entrenched beliefs and build a health care system based on relevant evidence that aims solely to promote optimal community well-being.