How have selection bias and disease misclassification undermined the validity of myalgic encephalomyelitis/chronic fatigue syndrome studies?

Abstract

Myalgic encephalomyelitis/chronic fatigue syndrome has been a controversial diagnosis, resulting in tensions between patients and professionals providing them with care. A major constraint limiting progress has been the lack of a ‘gold standard’ for diagnosis; with a number of imperfect clinical and research criteria used, each defining different, though overlapping, groups of people with myalgic encephalomyelitis or chronic fatigue syndrome. We review basic epidemiological concepts to illustrate how the use of more specific and restrictive case definitions could improve research validity and drive progress in the field by reducing selection bias caused by diagnostic misclassification.

The case presented

Controversies surrounding myalgic encephalomyelitis or chronic fatigue syndrome (ME/CFS) diagnosis and management (Geraghty, 2017; Geraghty and Esmail, 2016) have affected clinical practice and public perceptions and have led to disagreements between doctors and patients (Campion, 2016). There is still a lack of understanding and recognition of ME/CFS among many general practitioners (GPs) (Horton et al., 2010).

One of the main issues has been the absence of established diagnostic biomarkers and the reliance on diagnostic criteria (with over 20 proposed to date) largely based on clinical symptoms for diagnosis and research purposes. This has been problematic, especially when such criteria have been broad (Jason et al., 2014; Morris and Maes, 2013) and have not stratified cases into sub-groups. The challenge presented by diagnosis misclassification, which results in ‘false positives’ (or ‘spurious’ cases) is evident when prevalence rates of ME/CFS using different criteria are compared. A systematic review showed a greater than 100-fold variation in disease prevalence across studies (Brurberg et al., 2014), ranging from 0.1 per cent (Nacul et al., 2011) using the Canadian Consensus Criteria (Carruthers et al., 2003), a relatively specific classification which is considered by many to most closely represent ‘genuine’ ME/CFS cases, to 3.7 and 7.6 per cent in a study using the Oxford and Australian definitions, respectively (Lindal et al., 2002).

Even if we assume differences in methodology and geographical variations in prevalence, it is clear that rates are influenced markedly by the diagnostic criteria used. Brurberg et al. (2014) also showed that the Oxford criteria (Sharpe et al., 2015) yielded the highest median prevalence across studies, 1.5 per cent, 15 times greater than that obtained using the Canadian criteria, albeit in different populations. If it is assumed that the Oxford criteria always capture Canadian-positive cases, then, based on the above figures, we expect that of a sample of 15 cases selected using the Oxford criteria, 14 will not meet the Canadian criteria. Therefore, if Oxford-positive cases are used to test a hypothesis related to a specific pathophysiological process observed in Canadian-positive cases, this could lead to the selection of 14 non-cases (false positives) for every 15 recruited; an unacceptable level of misclassification.

Implications of diagnosis misclassification in observational studies

The following hypothetical example illustrates the level of bias that could be generated using inappropriate case definitions. If the odds ratio (OR) for the association of ME/CFS with certain exposure variable (V) is 4.0, then the true association between V and disease, ascertained using a standard diagnostic criteria considered to reflect ‘actual’ cases, in a case–control study with 300 incident cases and 300 controls can be represented in a 2 × 2 table (Table 1). Table 2 illustrates the results of a hypothetical study where biased case selection resulted in the recruitment of one ‘actual’ case (associated with V; OR = 4) and 14 ‘spurious’ cases (not associated with V; OR = 1) for every 15 cases recruited. The result is a highly underestimated OR.

Table 1.

Actual association between exposure to variable V and case of disease.

Exposure status	Cases	Controls
Exposed to V	240	150
Not exposed to V	60	150

OR = (240 × 150)/(150 × 60) = 4.

Table 2.

Association between exposure to V and case of disease resulting from misclassification of cases.

Exposure status	Cases		Controls
	Actual cases	Spurious cases (false +)
Exposed to V	16	140	150
Not exposed to V	4	140	150

Cases include 14/15 ‘artificial’ (‘spurious’) cases with no actual association with V.

OR = (156 × 150)/(150 × 144) = 1.08.

OR (excluding spurious cases) = (16 × 150)/(150 × 4) = 4.

Conversely, if the association exists in ‘spurious’ cases but not in ‘actual’ cases, the result could be the finding of false association, with an overestimated OR.

Diagnosis misclassification in clinical trials

The peril of diagnosis misclassification can be illustrated with a hypothetical example on diabetes mellitus. While both the better-known types of diabetes (types 1 and 2) share hyperglycaemia as the common factor defining the diagnosis, the classification of the disease into subtypes is essential for effective management and prediction of prognosis. The distinct pathophysiologies of the two types of diabetes demand quite different approaches: targeting insulin resistance with lifestyle changes and/or oral medication for the initial management of type 2 diabetes, in contrast to mandatory insulin usage for type 1.

Consider a clinical trial of a non-insulin hypoglycaemic agent effective against type 2 diabetes (but not type 1) recruiting cases of both type 1 and type 2 diabetes. If recruitment were population-based, and because type 2 diabetes is much more common than type 1, in the absence of stratification, it could be concluded that the hypoglycaemic agent is effective in reducing blood glucose concentration in all cases, as the average decrease in glycaemia would be driven by the predominance of study participants with type 2 diabetes. This would mask the complete ineffectiveness of the treatment among the sub-group of participants with type 1 diabetes, for whom the treatment could lead to dangerous increases in glucose levels.

Similar problems may well be happening in research on other less well-understood diseases including ME/CFS, where the danger lies in generalising the results of studies using patients with unspecific ‘chronic fatigue’ (which could include people with a range of diagnoses, including mental health conditions) to people with ME/CFS. Beth Smith et al. (2014, addendum 2016) recently reappraised the evidence for ME/CFS treatments. When studies using the broad Oxford criteria (Sharpe et al., 1991) were excluded, a virtual disappearance of effect for graded exercise therapy (GET), cognitive behaviour therapy (CBT) and other psychological therapies recommended by the NICE guidelines (National Institute for Health and Care Excellence (NICE), 2007) was revealed. Studies included the pacing, graded activity, and cognitive behaviour therapy: a randomised evaluation (PACE) trial (White et al., 2011) where psychological and exercise-based treatments had some efficacy against chronic fatigue as a symptom in the sample., but were shown to have little effect for those people with ME/CFS when defined according to more restrictive criteria (Beth Smith et al., 2014, addendum 2016; Geraghty and Esmail, 2016).

The revised understanding of the evidence will likely result in changes to ME/CFS treatment guidelines, illustrating the potentially far-reaching repercussions of diagnostic misclassification and selection bias.

A strategy for research

The understanding of the significant impact diagnosis misclassification can have on policy and patient care will lead to new research opportunities as we embrace well-designed and powered studies that recruit patients compliant with more specific definitions and include detailed phenotyping of participants (Jason et al., 2015). One approach to reducing misclassification in observational and interventional studies would be to improve specificity in case selection by, for example, requiring participants to simultaneously meet a combination of selected internationally agreed diagnostic criteria to be considered as cases. This would

1. Minimise the recruitment of ‘spurious’ cases for studies;

2. Avoid results that are difficult to interpret and represent a waste of precious resources;

3. Avoid fallacies in the ‘evidence’, which have served neither patients nor health professionals.

A natural concern of using very stringent diagnostic criteria for research studies is that genuine cases could be excluded for not meeting inclusion criteria. While this can happen, the main consequence in analytical studies would be some reduction in study power, a price worth paying for a robust analysis that enables unbiased inference. If more laborious recruitment is required (as many cases may not meet more stringent criteria for inclusion), any increase in study costs to achieve a given sample size would outweigh expenditure on research that is flawed by being too inclusive in recruitment of cases, but could generate spurious results. Those results, if interpreted uncritically, may drain precious resources to no effect.

Priorities for clinical practice

Nevertheless, it is important to distinguish research from clinical practice. While the former should focus on better definition of disease status, sub-groups and the trialling of preventative and treatment interventions, the main role of the clinician is to provide the best care and support to their patients, irrespective of a diagnosis or lack of it. Therefore, clinical services should be open to people with a broader range of conditions, presenting with, for example chronic fatigue.

Historically, patients accepted by ME/CFS Specialist Services in the United Kingdom have often been required to meet the centers for disease control and prevention (CDC)-1994 criteria (Fukuda et al., 1994) or even broader case criteria (NICE, 2007). We propose that criteria such as the Institute of Medicine (Institute of Medicine, 2015) or the CDC-1994 could still be used as a guide for primary care professionals to refer patients to Specialist Services, provided an adequate workout of cases conducted in primary care to enable the practitioner to suspect a diagnosis of ME/CFS. This could be the case until we have a better understanding of ME/CFS and are in a position to diagnose reliably and offer specific treatments. It is also important to acknowledge that many with chronic fatigue currently referred to ME/CFS specialist services would benefit from alternative care pathways, avoiding overloading already stretched services. This is particularly important for those with an alternative diagnosis explaining their symptoms, including some chronic medical and psychiatric diseases. It has been suggested that between 40 and 64 per cent of cases referred to CFS Specialist services do not meet diagnostic criteria for CFS, so robust referral procedures need to be established (Devasahayam et al., 2012; Newton et al., 2010).

Conclusion

The inclusion in research studies of only those patients that simultaneously meet a small number of selected case definitions could improve research cost-effectiveness and, by reducing bias, optimise the chances of diagnostic biomarkers discovery and the development of effective treatments. A research strategy that values robustness of methods will speed the process of knowledge generation and its translation to better clinical practice. The care of those with chronic fatigue should continue, based on the best existing practice and evidence, when available, in open and transparent dialogue with patients. This will enable positive relationships built on trust between patients and professionals, with informed disease management decisions taken in partnership.