Abstract
The previous article in this series explained basic concepts related to statistical signals and statistical noise in research. This article explains statistical noise in the context of randomized controlled trials (RCTs) and observational studies and offers suggestions on how noise in such studies may be reduced so as to better detect and understand the signal. Postrandomization bias related to RCTs and confounding in observational studies are discussed. Examples are provided to facilitate understanding.
Keywords
The previous article in this series introduced and explained basic concepts related to statistical noise in research. 1 In summary, in research, we look for signals in our data. The signal may be a mean or a proportion, that is, a descriptive statistic. Or, it may be the identification of a relationship between variables, that is, the outcome of an inferential statistical procedure. Statistical signals may be distorted by noise from extraneous variables. Such noise may be random or nonrandom, and may be adequately measured, inadequately measured, unmeasured, or unknown (Appendix). The present article examines statistical noise in the context of clinical trials and observational studies and offers suggestions on how noise may be reduced.
Clinical Trials and Noise
In randomized controlled trials (RCTs), a property of randomization is that noise from inadequately measured, unmeasured, and unknown biases tends to be equally distributed between groups at baseline. So, when groups are compared, because the noise is similar between groups, the noise cancels out, making the signal easier to detect. Unfortunately, although the RCT is considered a gold standard for research, RCTs are not 24-carat gold.
Noise in RCTs can arise in three ways. First, randomization is never perfect, especially in smaller samples, and so the noise is never perfectly balanced between groups. Second, especially in multicenter RCTs, methodological variations because of inadequately standardized operating procedures, interrater variations, and between-site variations can all generate noise; this was explained in detail in an earlier article. 2 Third, postrandomization bias 3 can create noise that compromises the internal validity of RCTs.
Observational Studies and Noise
It would now be obvious to readers that, when comparing groups in observational studies such as cohort and case-control studies, because of the absence of randomization to the groups of interest, noise is never balanced between groups even at baseline. Confounding by adequately measured, inadequately measured, unmeasured, and unknown variables can therefore be substantial in such studies.
As with RCTs, the longer the duration of follow up in observational studies, the greater the accumulation of additional noise due to changes from baseline in patient and environmental variables. Here is an example. We study a cohort of healthy elderly individuals to determine the effects of diet, physical exercise, and blood pressure on the risk of mild cognitive decline and dementia across a 10-year follow up. We use appropriate instruments to measure diet, physical exercise, blood pressure, and a wide range of relevant variables that might be a source of noise when testing relationships between the risk factors of interest and the outcomes of interest. We perform all these measurements at the time of recruitment into the cohort. We can be certain that most of these measurements will change, and perhaps substantially, across the 10-year follow up. For example, subjects may change their diet and level of exercise, some for the better and some for the worse. Many subjects who were normotensive at baseline may become hypertensive during follow up; not all will be identified and receive adequate antihypertensive treatment, let alone early antihypertensive treatment. Some subjects will develop dyslipidemia or diabetes that may or may not be detected and adequately treated. Some subjects may experience stroke or head injury. It would be very difficult to capture all these changes and to adjust for all of them in statistical analyses.
In the situation described above, we look for a signal through noise generated by a large number of adequately measured, inadequately measured, unmeasured, and unknown variables. How well we succeed in detecting the signal, should one exist, depends on how strong the signal is and how well we can detect and statistically adjust for the noise. As a parallel, if a man is standing in a pool of water, we can spot his head above the surface only if he is very tall (the signal is strong) or if we are able to drain some of the water (reduce noise by adjusting for measured sources of noise).
Addressing Noise in Research
No population is homogenous; for example, for any given continuous vari- able, there is always a standard deviation. For every relationship between variables in a study, which is what hypotheses examine, there is always subject-to-subject variation even within groups. So, it is impossible to eliminate noise in research. Noise is part of the real world.
It is possible to preemptively reduce noise in RCTs through various methods such as selection of homogenous samples and standardization of procedures.
1
Noise can be statistically reduced in observational studies by propensity score matching.
5
Finally, noise can be reduced in RCTs and in observational studies through the use of linear, logistic, proportional hazards, and other forms of regression in which
Parting Notes
Unmeasured and unknown confounds and biases create noise that can never be adjusted for. As a result, when researchers fail to detect a signal, they need to consider whether failure to control for noise was responsible. And, when they do detect a signal, they must consider whether the signal may be false because its value was distorted by noise (Appendix).
Finally, if a signal cannot be reliably detected through noise, one must ask whether the signal, if any, was worth detecting. Weak signals and real-world noise may explain why a treatment that worked well in valid animal models does not work in humans, why statistically significant findings may not be clinically significant, and why a drug that had clinically significant effects in homogeneous samples in RCTs does not work well in real-world settings.
Footnotes
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author received no financial support for the research, authorship, and/or publication of this article.