To the Editor:
I read with interest the “Managing Bias in Research” editorial in the March issue of Wilderness & Environmental Medicine.
1
It correctly noted “we are awash” in potential bias. The most comprehensive catalog of bias is the 2002 paper by Hartman et al,
2
which is an elaboration on Sackett’s classic.
3
However, this list is so overwhelming that it is not helpful to reviewers or readers as they assess the completeness of the “limitations of the study” paragraph of the discussion section.
The Cochrane Collaboration has a useful tool for recognizing the main sources of potential bias: the ROBINS-I tool for assessing risk of bias in nonrandomized studies of interventions.
4
Although it addresses studies of interventions, it may also be applied to studies of exposures. The Cochrane Collaboration suggests focusing on reporting potential bias in the following 7 domains: bias in selection of participants; bias due to confounding; bias associated with interventions; bias due to deviations from intended interventions; bias due to missing data; bias in measurement of outcomes; and bias in the selection of the reported result.
The Cochrane Collaboration has a series of focusing questions in its online tool (available at: riskofbias.info). That is, a reviewer or reader could assess the internal validity of a study by considering the following questions; specifically, they can assess potential bias due to the following:
Selection of participants: Is the population eligibility and participant selection clearly defined and based solely only on characteristics known at the beginning of the study? Is eligibility for the study dependent on the intervention/exposure, the confounders, or the outcome?
Confounding: A potential confounder is any factor correlated with the intervention and the outcome. Were analysis methods that attempt to control for the important confounding domains used? Where this was not possible or questionable, to what extent could confounding be associated with the outcome? For time-varying confounding, is it possible that the intervention/exposure changed as a result or that the outcome may have been biased?
Interventions/Exposures: Did any change in the definition of the classification of exposures or interventions occur after the study began? Could a subsequent outcome affect the classification of exposure? Is the person measuring the outcome also determining the classification of the exposure?
Deviations from intended interventions: If interventions or exposures are apparent to the participant or the health practitioner, were deviations potentially due to expectations? Were deviations from usual practice potentially due to real or anticipated outcome? Was adherence to an intervention or duration of exposure comparable across study groups?
Missing data: Unless analysis data are available for nearly all of the participants, what is the level of confidence in the findings? What is the level of missingness for intervention classification, confounder determination, or outcome measurement? Is there evidence for robustness (eg, inclusion of similar findings for complete-case analysis, imputation-based analysis, and sensitivity analysis)?
Outcome measurement: Could outcome measurements be influenced by the knowledge of the intervention/exposure or the confounders? Were the methods for outcome assessment identical across intervention/exposure groups?
Reported outcomes: For prospective studies, were the emphasized conclusions those that arose from the primary outcome as analyzed in the prespecified statistical analysis plan? For retrospective studies, were the findings from multivariable models or other methods controlling for potential confounding, effect modification, and dependencies in the data? Were findings based on unadjusted analyses with a single predictor in which confounding cannot be excluded?
Restating the first line of “managing bias in research”: All studies are subject to bias. But this does not mean all studies are useless. The potential for bias does necessarily imply misfeasance or malfeasance. Research on human participants is hard, and truth is elusive. The best we can do is to execute a study the best we can, acknowledge its shortcomings, and present the study results in a fair manner. A modest conclusion should neither overstate effects nor understate uncertainty.
