Abstract
Studies into participation bias have examined participation trends, where it occurs, the factors affecting it, and methods to try to reduce it. However, some authors only discuss participation bias at the end of the study, some acknowledge it and apply a method to try to reduce it, while others ignore it or dismiss it as negligible. Issues of three high-impact epidemiology journals were examined; 81 articles were read and reviewed for potential participation bias. Categories were used to classify the approach taken to participation bias and the results recorded. Of the 81 articles considered, 42 (51%) were eligible and could have suffered from participation bias. It was found that 57% of these articles ignored the effects of participation bias, while 17% only considered it briefly in the discussion. Few articles (22%) attempted to reduce the participation bias, with over half of these using unsuitable methods (55%). This review highlights how participation bias is often not considered and hence the conclusions drawn from these studies may not be correct.
Introduction
There have been various studies that have considered recent trends in participation, such as the effort required to recruit subjects (Rogers, Murtaugh, Edwards, & Slattery, 2004) or how participation rates have declined (Hartge, 2006). Problems arise in epidemiological studies when the participation is not randomly distributed across the different groups being studied and can lead to bias (Law, Smith, & Roman, 2001). Therefore, participation bias can lead to a sample that is not representative of the population which is being studied and hence affect the results and conclusions drawn from the study. There have been investigations into where participation bias occurs (Hennekens & Buring, 1987), as well as which factors affect participation rates (Galea & Tracy, 2007). Various formulae have been proposed to calculate participation rates (Galea & Tracy, 2007). There are also several methods suggested to help reduce participation bias, which include the variable associated with participation being adjusted for like a confounder (Breslow & Day, 1980; Kleinbaum, Morgenstern, & Kupper, 1981), the “bias breaking” model (Geneletti, Richardson, & Best, 2009), using sensitivity analysis to estimate the bias (Geneletti, Mason, & Best, 2011; Kleinbaum et al., 1981), using weights (Hernan, Hernandez-Diaz, & Robins, 2004), stratification (Hernan et al., 2004), imputation (Lessler & Kalsbeck, 1992; Sarndal, 1992), and maximum likelihood to develop estimating equations (Jiany, Scott, & Wild, 2011). Some suggest simply predicting the amount of bias for discussion (Hatch et al., 2000; Madigan et al., 2000; Wrensch, 2000). Others suggest there is no solution and recommend a sensitivity analysis instead (Hennekens & Buring, 1987; Kirkwood, 1988). Further details of these methods can be found in Table 1. However, how frequently is participation bias considered? Is it only discussed at the end of a study? Is it acknowledged and a suitable method applied? Or is it acknowledged and a possibly unsuitable method applied? Is it simply ignored or dismissed?
A Brief Description of Commonly Used Methods to Try to Reduce the Effects of Participation Bias.
Here the current status of participation bias is assessed. This is not intended as an attempt to disregard the findings from studies nor an attempt to criticize thoroughly planned and well-conducted studies. It is instead a general assessment of how much attention is given to the possibility of participation bias and whether attempts are made to reduce it.
We begin by explaining the sources used and how the assessment was carried out. Next we present and discuss the results from the review. Finally, we draw conclusions of the impact this may have on research.
Method
Three journals in epidemiology were used for this assessment, with a recent issue of each selected. The journals used were
Epidemiology, November 2011, 22(6):753-881;
American Journal of Epidemiology, December 1, 2011, 174(11):1211-1325; and
International Journal of Epidemiology, October 2011, 40(5):1135-1428.
These journals were selected based on their impact factors; they were the top three for impact factors and 5-year impact factors in epidemiology (ISI Web of Knowledge, 2010), shown in Table 2. As the journals used have different publication frequencies, different months’ issues were used, but the issue was the most recent at the time of data collection.
Impact Factors (2010) of the Journals Assessed (ISI Web of Knowledge, 2010).
Each article in each issue was read thoroughly to assess the potential participation bias. The entire article was read as participation could be discussed in the abstract, methods, results, or discussion sections and not all authors use the specific term “participation bias.” Some of the articles in each of the journals did not contain any data or methods that could be linked to participation bias, for example, letters to the editor; hence, these were classed as “N/A.” Any data used within the article was considered for participation bias. This included data collection that was conducted specifically in relation to the article or a larger previous study from which the data were taken. The selection process was considered in addition to the participation rates. To gain a greater overview of the problem and increase the sample of articles, all study types were considered, such as case-control and cohort. The articles were grouped by the categories shown in Table 3. The categories were chosen to try to present all possible approaches to participation bias. The same researcher categorized all the articles in each of the three journals to minimize observer bias, although we accept this may have introduced subjectivity.
The Codes Used to Categorize the Journal Articles.
Results
The results are presented for each journal separately and then combined for an overview of participation bias across the journals.
Epidemiology
Epidemiology had articles that covered a wide variety of topics, with some more medically orientated and others more theoretical. The results for participation bias consideration are shown in Table 4. There were 18 articles classed as “N/A” as some were purely mathematical or used simulations, along with a high number of letters (7) as well as some corrections (2). Of the 9 articles considered to be relevant, only 3 (33%) used a reasonable method; a method deemed appropriate based on the information provided, holding the required assumptions as described in the literature for the method. Of those remaining, 4 articles (44%) ignored the potential problem and 2 (22%) dismissed it as irrelevant. This finding showed that although there were some attempts made to account for potential participation bias, there were still studies published that either did not consider the bias at all or which concluded it to be insignificant. However, it is possible that the results from these studies could have been altered if the participation bias was taken into account in a “reasonable” manner.
Numbers (and Percentages) of Articles in Each Category From Each Journal and Combined Results From All Three Journals.
Note. Percentages are only calculated over those articles for which participation bias is relevant.
The most commonly used method used to try to reduce participation bias in Epidemiology was to include covariates in the regression model during analysis. However, this may only be suitable if the covariates included are true confounders (Keeble, Law, Baxter, & Barber, in press; Law, Baxter, & Gilthorpe, 2011). Very few of the articles that applied this method gave a thorough justification for the covariates included. This may suggest it is a method that has simply become a common practice and that some authors may be using this idea of adjusting for potential confounders or participation-related covariates without careful thought.
American Journal of Epidemiology
The American Journal of Epidemiology contained fewer articles than Epidemiology and less articles were categorized as “N/A.” As with Epidemiology, there were theoretical articles, but some of these considered data sets to demonstrate their ideas and these data sets were investigated for participation bias. There were also original articles that could be assessed for bias. The results for the number of articles in each category are shown in Table 4. Table 4 shows there were 6 articles classed as “N/A,” which was fewer than for Epidemiology, allowing a more detailed investigation of potential participation bias in the issue. Of the 10 articles considered to be relevant, there were 7 (70%) that ignored the possibility of participation bias, 1 (10%) that used a potentially unsuitable method to try to reduce the bias based on the information provided, and 2 (20%) that dismissed that it was an issue. It can be seen that for many authors, participation bias was not a consideration when displaying their results, or possibly it was considered and deemed to not be a problem and hence not mentioned. The results from the studies that did not consider potential participation bias or that used a potentially unsuitable method to try to reduce it, should be treated with caution.
Throughout the issue, there was little mention of participation bias, suggesting it may not be considered or alternatively that authors were unsure of how to tackle or report it. There may also be concerns that their results may be disregarded if the article shows high rates of non-participation. However, this information should be displayed for the reader so they can make their own, informed decision regarding the validity of the results. Again, the method adopted to try to reduce participation bias was to use a regression model and include any relevant covariates. However, no justification of which covariates to include was given.
International Journal of Epidemiology
The International Journal of Epidemiology was the journal that contained the most articles (38) of the three considered. Fifteen articles were classed as “N/A”; some being letters (7), others being editorial (2), and others for reasons such as being entirely theoretical. There was more variability in the categories for this journal compared with the previous two as to how to deal with the issue of participation bias; see Table 4. It can be seen that of the 23 articles considered to be relevant, 4 (17%) ignored potential participation bias, 1 (4%) used a seemingly reasonable method to try to reduce the bias, 4 (17%) used possibly unsuitable methods to try to reduce the bias, 7 (30%) acknowledged there may be participation bias in the study but made no attempt to reduce it, 5 (22%) dismissed participation bias as a problem, and the remaining 2 (9%) proposed methods relating to participation bias. This may suggest a greater awareness of participation bias in the articles in the given issue of the International Journal of Epidemiology compared with the previous two journals, or may be due to the larger sample size considered. However, this may not be representative of all issues of these three journals and is not intended to compare or rank the journals for their treatment of participation bias.
It was noted that in the International Journal of Epidemiology, there appeared to be more studies that used national databases as a source of data. This can help to reduce participation bias as there is not the need to obtain permission from each of the subjects as the data are anonymized. Therefore, provided the database captures the entire nation, which many did by using identification numbers for all residents assigned at birth or during immigration, and provided that the database is accurate, this is one reasonable method to reduce participation bias. Of those studies that attempted to reduce the bias, again many chose to use a regression model and include potential confounders and other covariates, but there was little justification for why these covariates were included whilst others were excluded.
Combined Results
The results from all three journals were combined to give an overview of how participation bias was assessed by articles published in the field of epidemiology. The results are shown in Table 4. It can be seen that 39 articles had to be excluded on the basis that they were not connected to participation bias. Of the 42 that could be related to participation bias, there were 15 (36%) that ignored participation bias may be a problem; note that this is more than a third of the articles considered. Only 4 (10%) made a reasonable attempt to correct for the bias. This leaves 5 (12%) that used methods that may be unsuitable and as a consequence may have increased the bias relating to participation. There were 7 (17%) that acknowledged participation bias may have affected the results, which is useful and allows the reader to treat the results with caution. Of those left, 9 (21%) discussed participation bias and concluded that it could be dismissed as it was negligible, which if true, shows the reader that the author has considered the effects of the bias. Finally, 2 (5%) of the articles were specifically related to participation bias and proposed methods to help reduce it.
Overall, there appeared to be limited awareness of participation bias, but attempts were being made by some authors to investigate potential participation bias in their studies, with some trying to reduce it. However, there are still a large proportion of authors who did not appear to consider participation bias in their studies.
Conclusion
The results show that participation bias is ignored by many authors, although some are attempting to reduce it or mention it as a limitation in their study. By realizing participation bias may have an effect on their study, the author is allowing the readers to judge for themselves whether the results are valid.
There appears to be some uncertainty in how to deal with participation bias if it is suspected, which may be why some authors prefer to acknowledge it rather than attempt to reduce it. Many authors who attempted to reduce the bias used a regression model including several covariates but with often little reasoning for the included and excluded covariates. It may be useful for the author to provide a set of results: initial results and those reanalyzed using possibly more than one method to try to reduce the bias, allowing the reader to draw their own conclusions. This approach has been adopted by some of the authors in these three journals and could be a useful idea for future articles. Sensitivity analyses have also been used by some authors and often compared the different unadjusted and adjusted results, hence providing more information to the reader.
The articles were carefully planned and well conducted. There was also attention paid to minor details to produce accurate results and to minimize many different forms of bias. However, the results of this review showed that participation bias is a form of bias that is often ignored or dismissed as negligible, but in some cases it may be extremely important to findings.
Although there were a total of 81 articles in the three journal issues considered, 39 could not be considered for participation bias due to the lack of a data set or the nature of the article. This reduced the sample size of the study, but we believe the sample still allowed an overview of the issues surrounding participation bias and showed how often participation bias has the potential to appear in an issue of a journal. Unfortunately, this did prevent any further analysis, such as by each study design. It is appreciated that the summaries and categories used are subjective; however, one researcher was used for all data collection to minimize observer bias and maintain consistency. Each article was also read thoroughly to ensure all references to participation bias were considered, regardless of the section in the article or the terminology used to describe the bias. The journals used for analysis may not be representative of all journals that could be affected by participation bias, but it would be impractical to consider all journals, so we selected those with the highest impact factors. It is also accepted that the selected articles from each journal, which were the most recent at the time of data collection, may not reflect the overall articles that the journal publishes. As mentioned previously, this is not an attempt to compare journals, nor is it a criticism of the articles that have been published. It is instead an assessment of how carefully participation bias is considered in a range of typical articles. The results showed little evidence that participation bias is thoroughly considered. When the bias is investigated, there seems to be uncertainty in how to tackle the issue, which can result in the bias either being dismissed or a method used without careful justification. Participation bias remains a serious problem in epidemiology and it would appear that more awareness and clearer guidelines on how to reduce or acknowledge it are required.
Footnotes
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: Claire Keeble is a PhD student funded by a MRC Capacity Building Studentship. Graham Richard Law, Stuart Barber, and Paul D Baxter are funded by HEFCE.