Real World Studies: What They Are and What They Are Not

Abstract

Patients are filtered by rigorously defined study selection criteria for recruitment into research; this is necessary to improve signal detection, improve internal validity, reduce study-related risks, and meet ethical standards. Research patients are assessed and managed in ways that differ from usual practice. So, neither patients nor the treatment environment resembles everyday patients treated in everyday practice. This diminishes the generalizability of study findings; that is, their external validity. There is, therefore, an increasing trend to conduct “real-world studies.” In this context, “real-world patients” are those who are not filtered by restrictive study selection criteria, and “real-world settings” are those in which patients are managed with few study-related guidelines and restrictions. The elephant in the room is that the glamour associated with such real-world studies is an illusion. This is because real-world patients in one real-world setting can differ widely from real-world patients in another real-world setting. So, even in real-world studies, we can only generalize study findings to the population from which the sample was drawn and the setting in which the sample was managed. As a final note, many assessments in research, such as computerized or pen-and-paper neuropsychological tests, are not real-world measures as are, for example, measures of activities of daily living or quality of life.

Medical and mental health research is usually conducted in tertiary care settings and patients are recruited only if they meet rigid study selection criteria (Supplementary Materials [SM], Box S1). Such criteria usually exclude patients who are likely to show ceiling or floor effects and those who may experience atypical or adverse outcomes. Such criteria also seek to make the sample homogenous in order to reduce statistical noise in study outcomes. The objectives of such sample filtration are to improve signal detection, improve the internal validity of the study, reduce risks to patients, and maintain ethical standards.

The biggest limitation of such studies is that the patients recruited are not characteristic of patients in the community. ¹ So, generalization of the study findings becomes difficult; that is, external validity is compromised. In this context, many investigating teams seek to recruit “real-world patients” in “real-world studies” conducted in “real-world settings.” What do these terms mean?

Real-world patients are those whose identification has not been constrained by restrictive study selection criteria. A real-world patient can be male or female, child or adult, treatment responsive or treatment-refractory, with or without personality disorder, with or without sub-stance use disorder, with or without medical comorbidity, and so on. There is no restriction.

As a digression, randomized controlled trials (RCTs) conducted with restrictive selection criteria are called efficacy trials; RCTs conducted on real-world patients are called effectiveness trials.

Real-world settings are those in which patients are managed with a minimum of study-related guidelines and restrictions. The patients do not need to be real-world patients as defined above; thus, we can choose to study only young, adult, treatment-resistant schizophrenia males in any real-world setting of our choice. The setting does not need to be in general practice or in the community; it can be any environment in which patients are seen, including a specialist hospital.

In explanation, the larger the number of study-related guidelines and restrictions imposed, the more the study environment becomes a research setting and not a real-world setting. Another way of explaining it is that patient care in a real-world setting is naturalistic or as close to treatment as usual as the study requirements permit.

As a digression, this is important because study-related guidelines and restrictions related to, for example, additional assessments and interventions will change the treatment environment, making the setting less natural or “real”. So, generalization of the study findings to the natural setting becomes difficult, and external validity is again compromised.

To summarize, real-world patients in real-world settings are patients who have not been filtered by selection criteria and who are managed as usual in their setting.

The elephant in the room is that the phrase “real-world” research carries the glamour of being representative of and hence generalizable to everyday clinical practice. This glamour is an illusion because real-world patients in one real-world setting may be very different from real-world patients in another real-world setting. For example, real-world patients in one setting may be enriched for treatment-refractoriness; in another setting, for medical comorbidities; in a third setting, for the mildness of illness but a larger burden of psychosocial stress; and so on. And, different real-world settings may have different standard operating procedures for assessments, hierarchy of treatments, use of psychosocial interventions, follow-up, and so on. Furthermore, obvious though unappreciated variables such as travel and waiting time, quality of engagement with the treating team, treatment costs, and others also characterize the treatment environment.

So, real-world patients and real-world practice in a tertiary care psychiatric hospital will be quite different from real-world patients and real-world practice in the psychiatry unit of a general hospital; and both of these will be different from real-world patients and real-world practice in general practice or primary care clinics. Who is to say which is the “real” real-world patient and the “real” real-world setting? Thus, generalization of real-world research is difficult unless we know all the obvious and not-so-obvious variables involved.

The take-home message is that whether we study a purposive sample in a research setting or real-world patients in a real-world setting, we can only generalize our conclusions to the population from which our sample was drawn and the setting in which the study was conducted. So, a real-world study is not necessarily a “better” study. A real-world study merely answers a different research question by moderately broadening the population and setting to which we generalize our conclusions. That is, real-world research may improve but does not assure external validity.

Readers may note that the results of “real-world” studies such as CATIE, CUtLASS, and STAR*D (SM Box S2) do not generalize well to India because real-world patients and real-world practice differ in many important ways between the USA and India. ²

Finally, readers may also note that many outcomes that are assessed in psychiatry are not real-world outcomes. As the most obvious example, oral, pen and paper, or computerized neuropsychological tests do not represent the cognitive demands of everyday life. Beyond studying neuropsychological or rating scale outcomes, it is necessary to study autobiographical memory in patients treated with electroconvulsive therapy, activities of daily living in patients with dementia, or health-related quality of life or measures of disability or burden in patients with major mental illness, as real-world outcomes.

Supplemental Material

Supplemental material for this article is available online.