Guidelines of the International Headache Society for Real-World Evidence studies in migraine and cluster headache

Abstract

Real-world evidence studies are an important tool for investigating the effectiveness, safety and tolerability of treatments for headache disorders in clinical practice. These studies use data generated from routine clinical practice from a variety of sources and can help to identify unmet needs in headache care and guide new therapies, including populations not always included in randomized clinical trials of different ages, comorbidities, and concomitant stable treatments. Real-world evidence pertains to the clinical evidence surrounding the utilization, as well as the potential advantages or drawbacks of treatment, which is extracted through an analysis of real-world data. Real-world evidence studies can present unique challenges, including timing of data collection (prospective vs retrospective), data selection, data quality issues and potential biases (selection, confounding factors, etc.). The International Headache Society has developed these guidelines for optimizing the design and conduct of real-world evidence studies, and therefore the strength and reliability of their output.

Introduction

Health data generated from routine clinical practice or related to patient health status, such as health records and claims data, are defined as real-world data (RWD). Studies based on RWD are an important tool for investigating the effectiveness and safety of treatments in the real world and can help to identify unmet needs in healthcare and guide new therapies. Clinically, real-world evidence (RWE) pertains to the evidence surrounding the utilization, as well as the potential advantages or drawbacks of treatments, which is extracted through an analysis of RWD (1,2). Studies using RWD to provide RWE are defined as RWE studies. Indeed, RWD can also be obtained from outside clinical practice, such as pharmacy registries, financial registries and genetic registries. Beyond effectiveness and safety of treatments, RWD are a valuable instrument to evaluate the natural history of the disease, enriching existing knowledge about epidemiological data (prevalence, incidence), integrate diversity and complexity (different races, ages and presence of comorbidities) and disease management (diagnostic, follow-up, treatment patterns and optimal duration, polytherapy, treatment persistence and predictors of response). Treatment cost-effectiveness analyses can also be conducted using RWD. For all of these reasons, RWE is especially important and can impact regulatory decisions of medical therapies (3) and through RWE, it is possible to shape healthcare policies and public health decisions. In other terms, RWE by reflecting actual clinical, logistical, and financial aspects of treatments/health practices, may provide information to policymakers to limit substandard, unsafe, ineffective, or cost-inefficacious therapeutics in favor of valuable ones, as well as guiding decisions concerning, for example, reimbursements. Moreover, RWE also reflects local culture, values, priorities and emphasizes patients’ experience, as they have a greater voice in making decisions and reporting their own outcomes (patient-reported outcomes - PROMs).

RWE is not a substitute for evidence from randomized controlled trials (RCT), rather it is complementary. There are several differences between RWE studies and RCTs (Online Supplementary Table 1). One of the main features of RWE studies, justifying their potential use for health decision making, is their higher generalizability to clinical practice. For instance, RWD can be instrumental to accurately address questions related to potential safety issues or adverse events as they collect data from a less selected population compared to RCTs. Furthermore, while RCTs concentrate on appraising therapy efficacy, studies using RWD have the capacity to gauge their effectiveness on the wider spectrum of end-users, encompassing factors such as effect size, comorbidities, concomitant treatments, if data collection adheres to the appropriate study design. A large sample size also facilitates sub-population analyses, such as specific under-represented populations or high-risk patient groups that are excluded from RCTs or subjected to off-label use. Studies using RWD also enables the evaluation of sustained therapeutic effectiveness, as data can be easily collected over longer observation periods compared to RCTs.

However, there are some aspects to consider for RWE studies. RWD can be collected through various methods (Online Supplementary Table 2), each corresponding to a distinct level of evidence and data quality issues that may occur. Although studies using RWD may provide insights into a broad range of research questions that cannot be addressed by RCTs, there is often redundancy, i.e., similar studies not providing any novelty or new information to the current literature, which can be balanced against the importance of reproducing findings. The latter is seldom achieved by RCTs. Moreover, there may be important biases: since randomization is not possible, confounding by indication often poses a challenge in performing causal inference, especially concerning treatment. In many cases, data is readily available during the conduct of the research questions, introducing the risk of selection bias, as the researcher may not define the population of interest before selecting the patients (records) for inclusion in their analysis, may exclude cases or controls that are lost to follow-up during participant selection or may have imbalanced groups. They often present wider inclusion criteria compared to RCTs, with a consequent difficulty in proper data correction according to patients’ comorbidities and/or other treatments. Furthermore, data collection is often reliant on self-reporting, which carries the risk of information bias.

Concerns on data quality, transparency and analytical methodologies have therefore raised skepticism about RWE studies. Nevertheless, studies using RWD have evolved as well. For example, the incorporation of digitalization in terms of electronic registries and, in headache care, electronic headache diaries represent advancements and opportunities to improve the amount and quality of RWD for research purposes.

Advances in the quality of RWD, not surprisingly, have led to more robust RWE studies and for RWE to be increasingly used for the original or supplementary approval of medical products (4). The United States Food and Drug Administration (FDA) has developed a series of draft guidance for industry on the aspects of RWD and RWE that are relevant for regulatory decision making (5,6). Additionally, the European Medicines Agency (EMA) has released a report of the experience gained with regulator-led studies conducted between September 2021 and February 2023 (7). All this highlights the importance of using RWE, which must be translated to the headache field as well.

To harmonize and guarantee meticulous and transparent execution of RWE studies in the realm of headache in order to achieve high levels of RWE with meaningful impact in health care practice, the International Headache Society (IHS) has formulated the current comprehensive guideline.

This guideline aims to provide recommendations for conducting RWE studies (study protocol, endpoints etc.) and suggests implementing a system to assess the quality of data collection that should be included in published results. This practice will assist readers in interpreting the obtained result based on the corresponding level of evidence. Moreover, this guideline is focused on two main primary headache disorders, migraine and cluster headache, thus complementing, in line with the purpose of RWE, the RCT guidelines already published by IHS (8–13). IHS is committed to helping provide the strongest evidence for patient care, encompassed by providing recommendations to the scientific community for both RCTs and RWE studies. Of note, although (health) registries, such as prescription registries, diagnosis registries or socio-economic registries, are part of RWD, research using these sources will not be covered in this guideline. The main focus of this guideline is the conduct of cohort longitudinal studies using clinical RWD. This study-type is usually more complex to conduct in the real world but ensures the highest degree of RWE for headache treatments, these being the specific focus of the guideline. The International Headache Society has another guideline focused on clinic-based registries (14).

Another specification is that in this guideline the term “treatments” is used to refer to all types of therapeutics: medications, devices, procedures or other non-pharmacological treatments, such as physical therapy modalities/techniques, and biofeedback. Moreover, this guideline is suitable for studies in pediatric, adolescence and ≥65 year-old populations, with the only remark that caregivers may be involved in data collection process (e.g., completing headache diaries)

For this guideline, we adopted a consensus-based methodology, utilizing an iterative process that involves circulating among experts’ subsequent versions for comment and refinement.

Finally, with this guideline we recognize the importance of RWE in headache, the need to ensure a comprehensive framework for guiding RWE design decisions and to enhance transparency, implementing a system to assess quality of data collection, able to assist readers in interpreting the obtained result.

1. Selection of subjects

A. Demographics

i. Age

Recommendation

The age range for participation should be specified, and appropriate age strata must be defined for studies in children, adolescents, and adults (≥18 years), respectively.

Comments

RWE studies are important for obtaining data from the elderly population (>65 years), which is usually poorly represented in headache RCTs and which, in clinical practice, may be challenging due to comorbidities and concomitant treatments. It could be considered to conduct subgroup analysis in the ≥65 year-old population when these subjects are well-represented in the study sample. Children should be defined as participants aged 6 to 11 years, and adolescents as participants aged 12 to 17 years (12,15). It is essential to consider the appropriate age strata to obtain meaningful results.

ii. Sex

Recommendation

To improve generalizability of study results, RWE studies should aim to include both female and male subjects, in a manner that reflects the epidemiology of the disease. However, in some cases, studies can be restricted to a single sex, such as in studies conducted within a specific population. In such cases, it is important to acknowledge that the generalizability of the findings might be limited due to the restricted sample. Moreover, if the sample is small, it might make sense to have specific studies focused on one sex, considering that migraine affects more females than males and vice versa for cluster headache.

Comments

Sex must always be documented (female, male or intersex, if applicable) as based on a biologically driven classification. Gender (women, men, and other gender-diverse populations) is not a synonym for sex and has social and cultural determinants and should be collected/reported if relevant for the study question.

In female adults, it may be relevant to collect information on their reproductive life stages such as fertile, premenopausal, perimenopausal and postmenopausal stages, as well as ongoing hormonal therapies and information about their menstruation cycle. For the pediatric population, also consider subdividing the female population before and after the menarche.

Unless the RWE study is specifically aimed at assessing the teratogenic or obstetric risk of a given treatment, it is important to consider the need for adequate contraception use in women during childbearing age when using potentially teratogenic treatments. For the same reason, pregnant and lactating women are often excluded from treatments with potential toxicity to the infant or unknown potential for toxicity. If sex-based analyses have been performed, results should be reported. Results of studies should provide separate data on females and males, which can be provided in the supplementary appendix. If the power is adequate, planned post-hoc analyses should be conducted to identify differences in treatment effects by biological sex or by reproductive condition.

World Health Organization definitions on hormonal status (16):

Natural menopause is deemed to have occurred after 12 consecutive months without menstruation for which there is no other obvious physiological or pathological cause and in the absence of clinical intervention. The menopausal transition can be gradual, usually beginning with changes in the menstrual cycle (16).

Perimenopause: the period from when these signs are first observed and ends one year after the final menstrual period. Perimenopause can last several years and can affect physical, emotional, mental and social well-being (16).

iii. Ethnicity

Recommendations.

Inclusion of minority participants is encouraged.

Reporting participant's race/ethnicity is encouraged, whenever possible

Comments

Inclusion and reporting of minority participants in RWE studies are critical for understanding how knowledge applies to diverse populations. Race/ethnicity information can be reported using standard NIH/OMB classifications or customized categories (17)

iv. Anthropometric measures

Recommendation.

It is advisable to include and report the body mass index (BMI) of participants, along with the distribution of BMI categories when available. This is especially important in those studies involving children and adolescents.

Comments

Body weight influences the pharmacokinetics and pharmacodynamics of drugs. Several studies have shown a significant relationship between obesity and headaches, particularly migraine and idiopathic intracranial hypertension. The specific measures collected may vary depending on the goals of the study, but core anthropometry measures are height, weight, body mass index (BMI) and body circumferences to assess for adiposity (waist, hip, and limbs). Researchers should not exclude participants above a specific BMI in order to include body size diversity.

B. Migraine and cluster headache definition

Recommendations.

The diagnosis of the headache disorder under evaluation must use the most current version of the International Classification of Headache Disorders (ICHD) (Online Supplementary Table 3) criteria (18)

The migraine diagnosis should be always reported, specifying whether it is episodic migraine with or without aura, chronic migraine or medication overuse headache, at the time study is conducted.

The cluster headache diagnosis should always be reported, specifying whether it is episodic or chronic.

Eligible subjects should fulfil the diagnostic criteria for the headache disorder under evaluation as in (a).

If the ICHD criteria are not fully fulfilled, the diagnosis should be considered as “probable”.

Comments

Episodic migraine and, specifically, high frequency or low frequency episodic migraine are not currently defined by the International Classification of Headache Disorders (18). A possible cut-off of eight migraine days/month for low-frequency (<8 days/month) and high-frequency (8–14 days/month) (19). To diagnose Probable Migraine and Probable Cluster Headache the latest version of the ICHD should be followed.

C. Comorbid headache disorders

Recommendations

Include and report relevant information about:

Pre-existing primary or secondary headache disorders

Concomitant primary or secondary headache disorders

Family history of primary and secondary headache disorders, if relevant

Comments

Patients included in studies with RWD may suffer from other headache disorders. Depending on the research question, its presence may justify exclusion due to the potential for exacerbation or confounding. Headache definition must follow the recommendations stated in 1.B (see also Online Supplementary Table 3). Family history of headache disorders should especially be assessed for first degree relatives.

D. Duration of the headache disorder under evaluation

Recommendations

The following should be considered and documented for study inclusion:

Onset of the headache under evaluation

Onset of the chronic presentation of headache under evaluation, if applicable

E. Comorbidities

Recommendations.

Relevant comorbidities, with special attention to psychiatric disorders and other chronic pain syndromes, should be thoroughly documented. Depending on the research question and the nature of the medical condition, its presence may justify exclusion due to the potential for exacerbation or confounding of the results (20)

Prior to data collection, it is recommended to create a Directed Acyclic Graph to identify concomitant medical conditions that may serve as potential confounders (Online Supplementary Figure 1). A Directed Acyclic Graph can facilitate adequate adjustment for confounding variables by avoiding adjustment for variables in the causal pathway that could otherwise dilute the treatment effect estimation (21).

Participants with relevant comorbid diseases, with a special attention to active psychiatric disorders, can be included in RWE studies if on a stable treatment regimen for at least three months prior to study enrollment, with no anticipated changes during the study.

Comments

Note that the inclusion or exclusion criteria of concomitant medical conditions should be predetermined prior to data-collection and analysis, particularly in retrospective studies, which are susceptible to selection bias. Pre-specifying these conditions can help to reduce bias.

Depression is an example of comorbidity that should be carefully assessed. In this context, major depressive disorder (MDD) should be distinguished from the presence of depressive symptoms. Whereas MDD is a formal clinical diagnosis, depressive symptoms refer to encompassing low mood, feelings of sadness, loss of interest or pleasure in activities, that may not be as severe or with the required characteristics to meet the criteria for MDD (22). Specific and validated questionnaires should be used to specify and preferably quantify comorbid depressive or anxiety symptoms using, e.g., HADS (23), CES-D (24), PHQ-9 (25), BDI (26) and GAD-7 (27).

Overall, if comorbid conditions are expected to influence treatment outcomes, exclusion of rare disorders and stratification for more common disorders should be considered. Alternatively, post-hoc analysis of effect modification by treatment is an option. The effect of treatment on common comorbid conditions can be studied as secondary outcomes, moderators, or mediators of treatment effects.

F. Current treatment history

Recommendations.

Collection of information on all treatments, whether used for the headache disorder under evaluation or for any other indication. This includes over-the-counter, prescription drugs, and devices, and the like.

The use of concomitant treatments should be reported with type and frequency (see also Paragraph 2. Study Protocol, section: Use of Concomitant Acute and Preventive Treatments for further recommendations on concomitant treatments)

Prior to data analysis it is recommended to create a Directed Acyclic Graph, to identify which treatments are potential confounders. If adjustment is necessary, this should be mentioned in the statistical analysis or initial data-analysis plan.

Comments

Concomitant treatment use can impact the effectiveness or safety of the therapy under evaluation, either positively or negatively, therefore this interaction should be considered during the analysis of the treatment effect. Note it is important to identify whether the interaction effect has a biological or statistical origin. It is important to consider carefully the potential confounding effect of concomitant treatment use, as it may lead to biased or misleading results if not adequately addressed. Concomitant treatments may also affect other specific evaluations carried out in RWE studies, such as neurophysiological assessments.

Their potential impact should be preliminarily evaluated and considered for inclusion and exclusion criteria.

G. Response to previous treatments

Recommendations.

Collect data on all previous preventive treatments that the patient has received for the condition of interest, including the type of treatment, dose, duration, degree of response and reason for discontinuation.

Preventive treatment failure is defined by any of the situations shown in Table 1. Treatment failure must be reported.

If relevant for the study question, specify if preventive treatment refractory patients are included and if a consensus definition has been used for refractoriness (e.g., European Headache Federation) (28).

If relevant for the study question, collect data on previous acute treatments as in (a) above.

Acute treatment failure is defined by any of the situation shown in Table 1.

Comments

To document previous failures of preventive treatments, it is generally recommended to use the participant's medical records. If the medical records are unavailable or unreliable, the investigator can obtain this information from the participant as an alternative method, although this is less desirable. It could be useful to report the proportion of refractory, resistant or treatment-naïve patients.

Table 1.

Acute and preventive treatment failure criteria.

	Any of the following situations	Definitions for migraine	Definitions for cluster headache
Preventive treatments	No meaningful reduction in headache frequency, duration, and/or severity after an adequate trial of treatment	No meaningful reduction is defined based on participant's experience that the treatment does not provide enough benefit in terms of reduction in headache frequency, duration, and/or severity to be continued* Adequate trial is defined as: 1–3 months for oral drugs (8,13) 2–3 cycles of onabotulinumtoxinA (CM) (30) 3–6 months for subcutaneously or intravenously-administered drugs or neuromodulation (11,31,32)	No meaningful reduction is defined based on participant's experience that the treatment does not provide enough benefit in terms of reduction in headache frequency, duration, and/or severity to be continued* Adequate trial is defined as: 1–3 months for oral drugs (10) 3–6 months for subcutaneous, intravenously-administered drugs or neuromodulation (10)
	Intolerable adverse events as defined by the participant who experienced them
	Contraindications precluding use as defined by relevant local prescribing regulations
Acute treatments	No meaningful relief after an adequate trial of medication	No meaningful relief is defined as: Patient´s well-being† not restored within 2 hours and for at least 24 hours (34) Adequate trial is defined as: At least 3 treated attacks	No meaningful relief is defined as: No pain-relief or pain-freedom within 15 min and/or recurrence of any level of pain within 3 hours Adequate trial is defined as: At least 3 treated attacks
	Intolerable adverse events as defined by the participant who experienced them
	Contraindications precluding use as defined by relevant local prescribing regulations

*Subjects’ global impression of change with the treatment (PGIC) scale can be used (33).

†Patient well-being is defined as all of the following conditions: a) improvement of headache; b) relief of non-pain symptoms; c) absence of adverse events (34).

2. Study protocol

A. Study design

Recommendations.

RWE studies should be planned considering the already available literature, introducing novelty by the research question and avoiding redundancy.

When planning RWE studies, the study design with the most adequate and highest degree of evidence that is possible to achieve for the research question should be selected.

The multicenter design should be considered whenever possible, as it could allow more extensive and impactful studies.

It may be advisable to involve a clinical epidemiologist or methodologist in the study to ensure methodological rigor where the study design or research questions require such expertise.

When reporting the methodology, the study design used must be always specified as well as study setting, such as outpatient clinic, or inpatients, and the dates when the study was conducted.

Comments

In the literature, terminology used for study designs often leads to confusion. Therefore, it is recommended to employ exclusively official methodology terms for study designs, such as cohort studies (e.g., observational retrospective longitudinal, observational prospective longitudinal), case-control studies and cross-sectional studies (29). Additionally, the case series design is recognized. It is crucial to note that whether the data collection is rooted in a registry or clinical practice does not determine the study design; rather it pertains to a description of the data collection process. Consequently, it is advised to discern thoughtfully to which of these main designs (cohort study, case-control study, cross-sectional studies, case series) the study belongs and carefully describe the data collection process, the origin and the quality of the data.

i. Cohort study

A cohort study is an observational study, that can either be prospective or retrospective. In a cohort study, participants are selected based upon exposure, and the outcome measured is the occurrence of disease outcome or the outcome of a treatment (Online Supplementary Table 4). Additionally, as the groups are considered either exposed or unexposed, information on exposed and unexposed individuals allow for the calculation of absolute or relative risk and rate measures. It is crucial to be mindful that the selection procedure employed determines the study design. Even if poorly executed or with a limited number of participants, the design remains unchanged (29,35). Prospective and retrospective are terms to describe the timing (how and when) data was collected (See Data collection section)

ii. Case control study

A case control study is an observational study, that is always retrospective as participants are selected based on the occurrence of a disease (disease outcome) or treatment outcome. The exposure of interest is then compared between the cases and controls. It is essential that controls are sampled independently of the exposure data. Both cases and controls are classified as exposed or unexposed. The purpose of including a control group is to ascertain the relative proportion of exposed and unexposed individuals in the source population. It is noteworthy that, even in a case-control study, defining the source population is imperative and aids in the selection of the controls. Note: the process of selecting controls in a case-control study can be challenging, and consulting with an epidemiologist and methodologist is recommended. The odds ratio can be calculated in this study design and not the absolute risk (29).

iii. Case-series

Participants are selected based on the presence of a disease or a disease-related outcome. Case-series sample participants with both a specific outcome and specific exposure or with the outcome of interest, regardless of their exposure status, similar to the approach in selecting cases in case-control studies (in other terms, this type of case series can be seen as the case group from a case–control study). It is important to recognize that, due to the absence of information on individuals without the outcome, the study is categorized as a case-series rather than the exposure group of a cohort study, as information of the unexposed cases is missing. In this study design, it is important to note that only the odds ratio can be calculated, and not the absolute risk (35).

B. Application of RWD

i. Comparative studies

Recommendations.

To conduct comparative studies using RWD, a longitudinal design is recommended thanks to its ability to establish the temporality between exposure and outcome better.

Identification and comparison with external controls can be considered and applied. External controls are identified as a cohort of patients comparable to the patients under investigation, not receiving any intervention. They are usually treated following usual care. International and national societies could favor data collection for a proper selection of external controls.

A prospective comparative effectiveness research (CER) study is better conducted by comparing centers or institutions that have different usual care/health care strategies, which is often referred to as a ‘natural experiment’ (36,37).

When comparing various treatments, it is important to exercise caution, because confounding by indication makes causal inference about treatment effect challenging.

Given that acute medications often have a short half-life and do not significantly impact the course of the disease, it becomes feasible in certain situation to compare these treatment options in an observational real-world setting. While causal inference might be attainable in rare cases, it is advisable to involve a clinical epidemiologist or methodologist in the study to ensure methodological rigor.

Comments

Due to the nature of RWE study designs, placebo-controlled studies may not be feasible or ethical to conduct. A comparative study is feasible with observational data, although ensuring causal inference might be challenging, as all observation data on treatment are subject to confounding by indication. Despite these challenges, comparative studies using RWD can still be useful to generate hypotheses and identifying potential treatment effects. Additionally, in particular situations, an adequate study design that simulates randomization can be used to restrict the influence of confounding by indication.

Considering the aforementioned study designs, it is important to note that the level of evidence differs between the retrospective, prospective, single center or multicenter designs.

A prospective CER study or ‘natural design’ holds, when correctly executed, a higher level of evidence (36). This is because CER or a ‘natural design’ study simulates randomization, as different centers provide different strategies of care treatment (36). So, the difference in treatment is not intentional, whereby the unknown confounding factors are likely to be equally divided among the treatment groups. Alternatively, propensity score matching or inversed probability weighting are used to emulate the balance between treatment and control group in an observational study. At its simplest, propensity score matching aligns each individual in the treatment group to an individual in the control group based on their propensity score. Briefly, the propensity score is the conditional probability of someone having a specific treatment given a set of variables known about this person. In studies using a propensity method, it is only possible to control for the known confounders, which may be a very significant limitation. However, depending on the research question and available population, the best study design and methods should be chosen (36).

Of note, in CER or a natural experiment, the outcomes of patients who receive different treatments or care at different centers are compared without intervening or manipulating the treatment being provided. However, it is important to ensure that the populations visiting the different centers are similar, as the chance of being treated with one of the interventions should be random. For example, the population of a hospital located in a low socio-economic area may differ from that of a hospital located in a high socio-economic area. Another option is that treatment allocation is random due to a specific situation, such as medication not being available during a certain time period, resulting in a group using an alternative treatment. This group can be used as a control group, as long as it can be proven that the treatment allocation was given by external factors unrelated to the condition of the patient, simulating randomization (36).

ii. Study design for adverse events and safety issues

Recommendations.

If the adverse event is anticipated to occur frequently, a cohort study and case control study are both viable options.

In case of relatively rare occurrence of adverse events, a case-control design is the most viable option with a relatively small sample size.

Association studies or explorative analysis can be performed for hypothesis generation.

Comments

In a case control study, cases with an adverse event are selected and compared to the controls. Selecting controls in this study design is difficult and requires the assistance of a clinical epidemiologist, since the controls are not simply the non-cases that were exposed.

A cohort study design could also be suitable for causal interpretations of the results when the adverse events are anticipated to occur frequently. If the adverse events are expected at a low rate, a large cohort will be necessary to have enough power for drawing a conclusion.

iii. Topics of interest for impactful RWE studies

When it comes to RWE studies, redundancy and replication are important issues. Although having multiple studies using RWD and coming to the same conclusion is advantageous to support RWE, when planning a new study, literature redundancy may risk a reduction in final impact. Impactful research using RWD can provide novel information that shape clinical practice. Table 2 outlines some of the research gaps in migraine and cluster headache, where studies with RWD may provide valuable RWE.

Table 2.

Research gaps with opportunities for RWE studies.

A non-exhaustive list of research gaps with opportunities for RWE studies in migraine and cluster headache
Effectiveness of oral preventive treatments
Effectiveness of new acute treatments
Effectiveness of new acute treatments in the prodromal phase
Compared effectiveness between migraine-specific and non-migraine-specific preventive treatments
Compared effectiveness between acute treatments
Cost-effectiveness analysis of treatments
Patient treatment preferences
Long-term tolerability and safety of migraine-specific and non-migraine specific preventive treatments
Understudied migraine and cluster headache populations (children, adolescents, ≥65-year-old group, specific racial/ethnic groups)

C. Data collection

Recommendations.

When planning RWE studies, the highest quality level of data collection that is possible to achieve should be implemented (see Figure 1 and comments).

When reporting the methodology, it is recommended to specify the quality level of data collection that was used.

Comments

Not only data can be collected prospectively (data gathered forward-looking) or retrospectively (data gathered backward-looking), but the collection methods of RWD encompass a wide range of varieties, leading to evidence of varying quality. Currently, interpreting the level of evidence can be challenging due to the diverse array of data collection methods employed. To ensure transparency, the Clinical Guidelines Committee of the IHS created a system to assess the quality of data collection, whose levels, that the study achieved through its data collection methodology, should be specified in the paper. This practice will offer readers guidance in interpreting the study's findings (Figure 1 and Data Quality Level Checklist in the Online Supplementary Material).

Figure 1.

Quality levels of data collection.

Data Quality Level 1: the RWD is collected following a protocol that outlines a structured data collection method, employing validated assessment and validated patient reported outcome measures.

Data Quality Level 2: the RWD is collected using a structured protocol with predefined structured data collection methods, incorporating standardized questions and standardized notation. It is important to note that these components have not yet undergone validation.

Data Quality Level 3: the RWD is collected following a structured protocol, where all patients undergo a predefined assessment (e.g., routine clinic visits can be standardized with standard text format for reporting in the electronic health record). The information collected is uniform and across all patients, yet standardized questions and notation methods are not utilized.

Data Quality Level 4: the RWD is collected without a predefined method, particularly in studies that extract data from patients’ clinical health records without adhering to a standardized format. However, prior to data extraction, a predefined protocol is established for extracting the data, ensuring uniform data collection for all patients, and preventing any selection bias based on data availability.

Data Quality Level 5: the RWD is collected without a predefined method and without a predefined plan for data extraction. Data and patients are selected based on their availability.

For studies classified as Data Quality Level 1 to 3, it is crucial to address the following steps when developing a structured study protocol and planning data collection. This approach will facilitate the utilization of collected data for high-quality studies in the future:

To ensure the prospective collection of data, it is strongly recommended to use an electronic headache diary that is capable of time stamps, remote monitoring, and alerts. While paper diaries are less desirable, they can be used if electronic diaries are not available.

To foster conduct of high-quality studies, a pre-determined set of data should be collected in all RWE studies. This will allow a baseline set of data comparable among all studies and essential for future reviews or meta-analysis. This essential data can be further implemented with specific information related to primary and secondary endpoints. An example of essential data to be collected for RWE studies in headache are detailed in Table 3 and Online Supplementary Table 5 shows a practical example of suggested variables and data to be collected for RWE study on anti-CGRP drugs for migraine prevention.

Subject can be instructed how to record headache characteristics, such as monthly headache days, pain quality, pain intensity, relationship with routine physical activity and associated symptoms, use of acute headache medications and compliance with the treatment, as well as to complete clinical questionnaires.

Subjects should be instructed to report any adverse events they experience using an (electronic) diary. It is preferable to confirm the characteristics of the adverse events and their relation to the treatment under investigation during follow-up visits. To ensure comprehensive recording of adverse events, it is recommended to use a list of specific side effects in addition to open-ended questions to evaluate clinical aspects, duration, severity of the adverse events, eventual need to take medication and possible consequences (in this case a follow-up after weeks/months should be recommended).

If feasible, ascertain the relevant health outcome pertaining to the disease of interest, along with the intended measurement approach, before data collection. This practice is pivotal for elevating the level of evidence of the RWE studies. It holds significance for both prospective and retrospective studies, as predefined outcome measures guarantee a high level of quality, thus aligning the level of evidence of prospective and retrospective studies (38).

Whenever feasible, establishing a distinct operational definition for the exposure (treatment) and outlining its follow-up procedure is crucial. In cases where a subject undergoes repeated exposure and measurements over time owing to a specific treatment, it is ideal for the measurement process and timing to be independent of the disease status. This approach enhances the potential for conducting high-quality research when utilizing this data in RWE studies.

For multicenter studies, establish consistent and standardized data collection procedures across all study sites to reduce measurement variability and ensure the accuracy and completeness of data.

Table 3.

Types of data to be collected for RWE studies in migraine and cluster headache.

Feature	Definition
Monthly headache days (MHDs)	Monthly days with headache
Monthly migraine days (MMDs)*	Monthly days with migraine
Number attacks / day	Number of attacks with trigeminal autonomic features / day
Attacks / week	Number of attacks with trigeminal autonomic features / week
Pain intensity	Pain intensity assessed through 4-point scale (0 = no headache, 1 = mild headache, 2 = moderate headache, 3 = severe headache, 4 = extreme severe headache) or 11-point or visual analogical scale
Pain quality	Pulsating, dull (non-throbbing), stabbing
Associated symptoms/features	Nausea Vomiting Photophobia Phonophobia Osmophobia Allodynia Vertigo/dizziness Worsening by routine physical activity Cranial autonomic symptoms Restlessness
Monthly doses of acute drugs intake	NSAIDs Triptans Ditans Gepants Combination Other drug classes (paracetamol, opioids)
Monthly days of acute drugs intake
Adverse events	Adverse event type and features Adverse event severity Time-relation to the studied treatment Duration Need to take acute medication Eventual consequences Causality in relation to the studied treatment Adverse event leading to treatment discontinuation
Concomitant Treatment	Concomitant treatment changes

* Distinguishing between headache days and migraine days in real-world settings may pose difficulties. Headache days should be used instead of migraine days if data collection cannot allow proper distinction of the two. Migraine day can be considered as any day fulfilling the latest ICHD criteria for migraine without aura, for migraine with aura, probable migraine or any day of migraine-specific acute treatment use. Diaries, especially e-diaries, may allow the collection of all necessary variables to properly distinguish headache and migraine days.

General recommendation for all Data Quality Levels:

It is recommended to include repeated measurements (daily or weekly preferably) for the duration of a treatment phase and follow-up, to enable measuring change over time and to increase statistical power.

Determine with a Directed Acyclic Graph (DAG) the potential confounding factors that need to be controlled for (see Online Supplementary Figure 1), without adjusting for a collider, which can induce a distorted association by introducing collider stratification bias.

When certain confounding factors cannot be directly measured or determined, consider instrumental variable analysis if it is feasible. Instrumental variable analysis is a statistical method that can be used to estimate treatment effects in the presence of unmeasured confounding variables. This method requires the identification of an instrumental variable that is associated with the treatment but not directly related to the outcome (39).

Monitor data quality throughout the study by performing regular quality control checks, such as data validation and cleaning, to ensure that the data is accurate and complete.

Monitor patient enrollment and retention to ensure that the study is on track and to identify any potential recruitment or retention issues.

D. Enrollment

Recommendation.

Participants must fulfill all the predefined inclusion criteria and should not meet any of the predefined exclusion criteria. Preferably this information is to be documented at the time of data collection to ensure that the baseline characteristics of interest are available for all eligible participants. In retrospective studies, it is also recommended to predefine the inclusion and exclusion criteria for a specific research question before selecting the data. This practice enhances the reliability and reproducibility of the results.

It is recommended to create a flowchart of study participants (subjects screened, included, excluded, enrolled and who terminated the study), when presenting the results.

Comment

Predefining the research question and thus the inclusion and exclusion criteria is essential to extract the right data and for a proper initial data-analysis plan and sample size calculation. An overview of the recommended patients’ characteristics regarding the inclusion criteria is provided in Table 4.

Table 4.

Overview of potential subjects’ characteristics regarding inclusion and exclusion criteria.

Demographics	Headache and medical history	Medication use
Age Sex Ethnicity Height Weight Body mass index	Headache disorder under study Disease duration Eventual duration of a chronic disease state (e.g., chronic migraine & chronic cluster headache) Headache frequency per month or another specified period Headache characteristics Concomitant primary or secondary headache disorders Comorbidities	Concomitant preventive treatments Failure(s) of preventive treatments
		Type of acute treatment Acute medication frequency per month Number of acute medications per month (compounds)

E. Duration of screening and baseline phase

Recommendations.

Baseline data must be collected to generate RWE.

Prospective cohort RWE studies investigating the effectiveness of preventive treatments for headache should include a baseline data collection phase (whose duration may depend on the headache disorder under study) before the initiation of the treatment under evaluation. This phase should be used to determine whether a subject is eligible, to assess baseline headache characteristics and headache medication intake.

Headache related data of the screening and baseline phase should ideally be derived from an electronic diary and compliance should be considered using a threshold of at least 80% on the prespecified baseline period. While paper diaries are less desirable, they can be used if electronic diaries are not available.

A baseline phase could be considered for studies investigating acute treatment as well to confirm the baseline diagnosis.

Comments

Depending on the study design a screening or baseline phase is recommended:

For migraine and chronic cluster headache prevention, we recommend a 4-week (28 days) baseline phase before the initiation of the treatment under evaluation, without any change in concomitant acute or preventive treatments or their dosage. To avoid confusion caused by variations in the length of a calendar month, we recommend using the term “4 weeks” instead of “1 month” so to establish a common duration of 28 days.

For episodic cluster headache prevention, we recommend a one-week baseline phase before the initiation of the treatment under evaluation.

A baseline phase is necessary to establish the baseline frequency of headache days/attacks and to classify each headache day/attack based on pain intensity (mild/moderate/severe/extreme), as well as to check inclusion and exclusion criteria. Using a diary during the baseline phase is also important for assessing headache characteristics such as pain quality, intensity, relationship with routine physical activity, associated symptoms as well as (acute) headache medication intake. In prospective studies, a baseline period is also useful to assess subject compliance, using a threshold of at least 80% on a 4-week period, for a proper selection of the candidate. If required, other baseline periods can be employed to align better with the specific characteristics of the headache under evaluation.

F. Duration of treatment phases

Recommendations.

The duration of the treatment phase must be adequate to respond to the objective of the study, should be predetermined in the analysis plan and should not depend on the availability of data.

When feasible, a structured protocol is advisable in the clinic account, in part, for the time needed for the full treatment effects to become manifest. When studying preventive treatments longer observation periods are recommended for collecting additional information on the persistence or delayed emergence of treatment effects, as well as assessing long-term adherence, tolerability and safety (Table 5).

It is recommended to consider that the setting when an acute treatment is used is more prone to confounding factors. Collection of RWD from more than one attack for the same subject, should be considered (Table 5).

In retrospective studies the median or mean follow-up (depending on variable distribution) time should be reported.

Comments

Prolonging the duration of a preventive treatment period can enhance the statistical power by allowing sufficient time for delayed treatment benefits to fully emerge. Certain preventive medications require gradual dose titration and adjustment before an optimal dosage is attained. Subsequently, treatment effects may accrue gradually, especially for orally administered medications, and some drugs may require several weeks or longer at an optimal dosage before reaching their full preventive potential. A longer phase may result in higher drop-out rates, which should be well documented in the study results.

Table 5.

Minimum duration of treatment phases in migraine and cluster headache studies.

	Migraine (Episodic, Chronic)	Cluster Headache
Preventive treatments	Short term effects: at least 12 weeks (if feasible 24-week recommended) Long term effects: at least 12 months	Episodic: at least 3 weeks Chronic: at least 24weeks
Acute treatments	At least 3 separate attacks in a standardized time period	At least 3 separate attacks in a standardized time period

G. Study visits

Recommendations.

In prospective studies, the frequency of study visits should be appropriately tailored to meet the study's objectives and data collection protocol. The time intervals between visits and window period for each visit need to be planned, keeping in mind that visit intervals should align with the ones of a real-world setting as much as possible. In retrospective studies, periodicity between visits must be recorded. Document the frequency and type of healthcare interactions that patients receive. This can help to control for potential confounding factors, such as differences in care between treatment groups, and ensure that study results are reliable and reproducible.

Documentation of study visits can be extracted from the patients’ electronic health records or specific study research files.

Patient-reported outcomes such as E-questionnaires and E-headache diaries can be used to capture information on symptoms and quality of life between study visits, providing a more accurate and standardized representation of patient outcomes.

It is important to consider the potential impact of missing data on study results, as patients may not attend all scheduled study visits or may drop out of the study entirely. Sensitivity analyses can be used to assess the robustness of study results to different assumptions about missing data and to evaluate the potential impact of selection on study outcomes.

Comments

In contrast to traditional clinical trials, study visits in RWE studies are often not standardized, and patients may receive different levels of care depending on their individual needs.

H. Treatment indication and dosage

Recommendations.

Drug dose and treatment regime should be determined according to local regulatory, consensus clinical practice or previous safety/tolerability data.

Prescription and reimbursement criteria, if applicable, should be reported.

If treatment is not approved for the specific indication of the headache disorder under evaluation, this should be stated (e.g., compassionate use, off-label use).

I. Use of concomitant acute and preventive treatments

Recommendations.

In case a preventive treatment is studied, acute treatment should be permitted. Preferably the agent and dosage remain consistent throughout the baseline phase, which can be achieved selecting the baseline period carefully. Additionally, the agent and dosage of the acute treatment should preferably remain consistent for the entire duration of the study. To ensure that the effect of interest, the preventive treatment, is not distorted by alterations in the acute medications or vice versa.

Preferably the use of both preventive and acute treatment is recorded carefully.

Preferably concomitant preventive headache treatments that could affect the outcomes of interest should not be changed or discontinued during the study period. If concomitant preventive treatment is changed, it is recommended to account for this in the analysis and mentioned in the discussion, as presence of an interaction cannot be excluded. Ideally, preventive treatment and its dosage should be stable for at least two months before the start of the baseline phase.

In case an acute treatment is studied, it should be avoided to use other acute treatments before the administration of the one under evaluation to treat an attack. The need for acute rescue medications after the use of the treatment under evaluation should be reported.

Comments

Standardization of instruction on acute treatment usage can vary across treatment centers. It is therefore important to take this into account for interpretation of the study results.

3. Evaluation of endpoints

Recommendations.

All primary and secondary endpoints need to be decided prior to data collection. This does not imply that RWD must solely be collected prospectively. Historically collected data can also be utilized, provided that a plan outlining primary and secondary endpoints is predefined prior to extracting data from healthcare records. In specific scenarios, dealing with missing data might pose challenges in historically collected data, which could necessitate adjustments to the primary or secondary outcomes. Nonetheless, the endpoint should invariably be pre-defined before data analysis, as stipulated in a data analysis plan. This includes identifying specific comparison groups and time points for each endpoint. These definitions ensure that the study's objectives are clear and that the collected data are relevant and reliable for analysis.

The selection of endpoints should be based on the study objectives.

Endpoints should be clinically relevant and reliably measured.

Primary and secondary endpoints in RWE studies, should align with the recommendations outlined for the RCT guidelines. Although some outcome measures, such as the Patient Global Impression of Change, are recommended as PROMS or secondary outcome (40).

Prior to the initiation of a RWE study, in both prospectively and retrospectively collected data, the power and sample size calculations must be performed for the primary and key secondary endpoints.

Comments

In RWE studies, it is important to carefully define the study endpoints. Endpoints can be clinical, patient-reported, or related to healthcare resource utilization. The selection of endpoints should be based on the study objectives, the available data sources, and the feasibility of data collection.

A. Primary and secondary endpoints

Recommendations

The primary and secondary endpoints differ among the various headache diagnosis. The recommended measures for primary headaches are outlined in the specified clinical trial guidelines provided by the International Headache Society for each distinct headache diagnosis (Migraine (8,9,11,13), Cluster Headache (10) and Children and adolescent (12), Guidelines available at: https://ihs-headache.org/en/resources/guidelines/). Primary and secondary endpoints in RWE studies for migraine and cluster headache are outlined in Table 6 and 7, respectively.

Table 6.

Recommended primary and secondary endpoints in migraine RWE studies.

	Endpoint	Definition
Migraine acute treatment	Pain freedom at 2 h*	Subjects experiencing an attack who become pain free at 2 h after treatment, before the use of any rescue medication. Pain freedom at times before 2 h should be considered as an endpoint in trials of non-oral treatments.
	Absence of the most bothersome symptom (MBS)*	Absence of the most bothersome migraine-associated symptom at 2 h after treatment. The MBS should be selected at baseline and measured on a binary scale (present or absent) over attacks.
	Pain relief at 2 h	Decrease in headache pain from moderate or severe at baseline to mild or none at 2 h or defined time point after treatment.
	Relapse	Subjects who were 2-h pain free presenting headache within 24 or 48 h of the treatment administration
	Sustained pain freedom/relief	Subjects experiencing an attack who were pain free/relief at 2 h with no relapse or use of rescue medication within 24 or 48 h of the administration of the treatment under evaluation.
	Total freedom from migraine	Subjects experiencing an attack presenting absence of pain, nausea, vomiting, photophobia and phonophobia (even any migraine-associated symptoms) at a defined time point.
	Time to pain freedom/relief	Time interval between treatment intake to treatment response (pain free or relief)
	Associated symptoms	Change in presence of migraine associated symptoms (including but not restricted to nausea, vomiting, photophobia or phonophobia) at a defined time point after the treatment administration.
	Rescue medication	Subjects taking rescue medication 2 h (or earlier) after administration of the treatment under evaluation.
	Time to treatment intake	Time interval between the onset of headache and the use of the treatment
Migraine preventive treatment	Change in headache days/month*	Change in number of days with any headache, regardless being migraine or non-migraine
	Change in migraine days/month*	Change in number of days with a headache that meets the latest ICHD criteria† for migraine without aura, migraine with aura or probable migraine; or a day with a headache that is successfully treated with migraine-specific acute medication over 28 days.
	Change in moderate/severe headache days/month*	Change in number of days with moderate or severe pain or a day with a headache that is successfully treated by an acute headache treatment over 28 days. Compared to migraine days, this endpoint is relatively simple to record by using headache diaries.
	Responder rate*	Percentage of subjects achieving 50% or other percent reductions (e.g., 30%, 75%, and 100%) in reduction of headache days or migraine days or moderate/severe days.
	Cumulative hours per 28 days of moderate/severe pain	Number of hours with moderate/severe pain per 28 days. If a subject goes to sleep and awakens with headache, sleep hours are counted as headache hours. If a subject goes to sleep with headache and awakens pain-free, half the sleeping hours are counted as headache hours.
	Change in migraine attacks	Defined by episodes of headache or migraine-specific medication intake. A migraine attack that is interrupted and recurs within 48 h is considered as one attack.
	Conversion to episodic migraine	Proportion of chronic migraine subjects at baseline that have fewer than 14 migraine or headache days per 4 weeks of a defined time point.
	Effect on the most bothersome symptom (MBS)	Change in presence of days with the most bothersome migraine-associated symptom. The MBS should be selected at baseline and daily recorded on a binary scale (present or absent).
	Headache and symptom-free days	Change in number of days with no headache or associated symptoms (including premonitory, aura, headache, and postdrome symptoms) that are directly attributable to migraine.
	Onset of effect	Time interval between treatment start and achievement of pre-defined meaningful efficacy endpoint.
	Acute treatment use	Change in number of days of specific acute drug use.
	Conversion of medication overuse to non-medication overuse	Proportion of patients with medication overuse at baseline that cease overuse of acute medications in the last 12 weeks of a defined time point.
General	Headache intensity	Subjects should note the intensity of headache using 4-point numerical rating scale (or 11-point or visual analogical scale) for the maximum intensity of the attack or at pre-specified time points during the attack.
	Global evaluation*	Subjects’ global impression of change with the treatment (PGIC, see Online Supplementary Table 6).
	Global impact	Functional disability assessment before and after the treatment at pre-defined time points using a 4-points scale (FIS, see Online Supplementary Table 6 also for other scales).
	Treatment preference	Subjects’ treatment preference among treatment options or over usual subjects’ treatment
	Adverse events*	Reporting adverse events should be evaluated by responses to open questions, including recording of event severity (mild, moderate, severe); event seriousness (serious, non-serious); likelihood of being treatment-related; time of onset and time of resolution (see section Adverse Events).

* Recommended primary endpoint/Most relevant endpoints.

The non-selected endpoint as the primary should be considered as secondary endpoints.

† For ICHD-III, criteria C and D for migraine without aura (1.1), B and C for migraine with aura (1.2), or ICHD-III criteria for probable migraine (1.5).

Table 7.

Recommended primary and secondary endpoints in cluster headache RWE studies.

	Endpoint	Definition
Cluster headache acute treatment	Sustained 15-min pain freedom*	Subjects experiencing an attack who become pain free at 15 min after treatment, before the use of any rescue medication, without recurrence of any level of pain or use of rescue medication – including oxygen - within 3 h in the treated attack.
	Pain freedom at 15 min	Subjects experiencing an attack that become pain-free at 15 min after treatment.
	Sustained 15-min pain relief	Subjects experiencing an attack who report 2-point decrease of headache intensity on the 5-point severity scale† at 15 min of treatment administration (or other defined time point after treatment), without recurrence of any level of pain or use of rescue medication within 3 h in the first treated attack.
	Pain relief at 15 min	Subjects experiencing an attack who report 2-point decrease of headache intensity on the 5-point severity scale† at 15 min of treatment administration (or other defined time point after treatment).
	Time to pain freedom/relief	Time interval between treatment intake to treatment response (pain free or meaningful relief).
	Attack responder rate	Percentage of subjects achieving treatment response (15-min pain-freedom/relief) in 50% or other percentages (e.g., 30%, 75%, and 100%) of treated attacks.
	Rescue therapy	Proportion between number of treated attacks and number of attacks with use of rescue medication including oxygen.
Cluster headache preventive treatment	Change in number of attacks*	Change from baseline in the number of weekly/monthly‡ attacks over a pre-specified period of time.
	Responder rate	Proportion of participants achieving at least a 50% or other percent reductions (e.g., 30%, 75%, and 100%) from baseline in the number of weekly/monthly‡ attacks over a pre-specified period of time.
	Change in number of severe attacks	Change from baseline in the number of weekly/monthly‡ severe attacks (pain rated as 3, 4 or 5 – if cluster headache) over a pre-specified period of time.
	Time to attack freedom	Time interval to sustained attack freedom for ≥2 months.
	Onset of effect	Time interval to onset of 50% or other percent reductions (e.g., 30%, 75%, and 100%) in weekly/monthly‡ attack frequency.
	Mean intensity of attacks	Change from baseline in the mean intensity of attacks over a pre-specified period of time.
	Acute treatment utilization	Change from baseline in number of attacks treated with a specific acute treatment.
	Efficacy of acute treatment	Change in mean number of attacks effectively treated (e.g., 15-min pain-freedom/relief) with a specific acute treatment.
General	Headache intensity	Subjects should note the intensity of headache using 5-point numerical rating scale† (or 11-point or visual analogical scale) for the maximum intensity of the attack or at pre-specified time points during the attack.
	Global evaluation*	Subjects’ global impression of change with the treatment (PGIC, see Online Supplementary Table 6).
	Global impact	Functional disability assessment before and after the treatment at pre-defined time points using a 4-points scale (FIS, see Online Supplementary Table 6 also for other scales).
	Treatment preference	Subjects’ treatment preference among treatment options or over usual subjects’ treatment
	Adverse events*	Reporting adverse events should be evaluated by responses to open questions, including recording of event severity (mild, moderate, severe); event seriousness (serious, non-serious); likelihood of being treatment-related; time of onset and time of resolution (see section Adverse Events).

* Recommended primary endpoint/Most relevant endpoints.

† For cluster headache a 5-point rather than a 4-point scale is preferred (0 = no headache, 1 = mild headache, 2 = moderate headache, 3 = severe headache, 4 = very severe headache).

‡

It is preferred to use the number of attacks per week in episodic cluster headache studies and number of attacks per month in chronic cluster headache studies.

These recommended primary and secondary endpoints cannot always be completely applied to data available using retrospective data collection, such as electronic medical records (EMR), claims and pharmacy.

B. Patient-reported outcome measures (PROMs)

Patient-reported outcome measures (PROMs) play a crucial role in evaluating the burden of illness and the benefits of treatment for headache. While several PROMs have been included in the label for migraine treatments, they have not been deemed fit nor validated yet for most other types of headache. However, it is recommended to compare PROMs at baseline to the end of the treatment period, and across the pre-defined timepoint of PROMs collection.

Depending on the investigators’ priorities, different PROMs may be considered (See Online Supplementary Table 6 for general and disease-specific PROMs)

C. Pharmacoeconomic endpoints

Studies evaluating the economic value of preventive treatment for headache should consider both the direct costs of medical treatment and health care resource utilization and the indirect costs of lost productivity.

The reduction in work productivity and activity due to headache should be considered as important components of disability and headache-associated costs.

The Work Productivity and Activity Impairment (WPAI) instrument can be used to measure the mean change from baseline in work productivity and activity (41).

D. Adverse events

Documentation of adverse events (AEs) and serious adverse events (SAEs) during treatment should follow the requirements of local institutional review boards, regulatory authorities, and Good Clinical Practice guidelines.

Acceptable methods for documenting AEs include compiling lists of events, collecting spontaneous reports from study participants, recording observations made during physical exams, using open-ended questions (if the event is not covered by the AE listing), and conducting direct questioning. The method used should be specified.

Comments

Adverse events are a common occurrence during preventive treatment, and they can occur before maximum effectiveness is reached. Therefore, the incidence of AEs, particularly those that lead to subject discontinuation from a study, should be considered a critical measure of the tolerability of preventive treatments.

Treatment-emergent adverse events may not always be related to the treatment. Investigators need to assess the likelihood of AEs being treatment related. It should be noted that regulatory authorities mandate detailed reporting of AEs using the Medical Dictionary for Regulatory Activities (MedDRA) system. To report AEs MedDRA terminology can be used.

4. Statistics

Recommendation

The analysis of data for RWE studies should be preplanned, and issues that need to be prospectively defined should be considered. Table 8 provides a list of recommended issues that should be addressed during preplanning of the data analysis for RWE studies.

Table 8.

Preplanning real-world data study analysis, an overview of the steps that need to be taken before the data-analysis.

Methods
Validation of used dataset (e.g., headache diagnosis, primary and secondary outcomes)
Selection of population; without restrictions based on post baseline (avoid selection bias)
Define baseline (period); eligibility criteria need to be met at that time point and not later (treatment studies: baseline is when eligible patient initiates treatment strategy)
If applicable: Treatment strategies definition, comparison of patients who are new users of various treatment strategies, to avoid selection bias
Create an overview of all possible confounding factors, mediator and causal pathways for the exposure and outcome of interest in a Directed Acyclic Graphs (DAG)
Definition of the primary / secondary outcome and minimal important difference
Plan for handling different types of missing data (e.g., multiple imputation, instrumental variable analysis)
Statistics
Create an Initial Data Analysis plan based on the method for the primary and secondary analysis
Sample size calculation based on minimal important difference
Inventory of which analysis will be done and if multiple-testing correction should be applied
Missing data inventory, report the missing mechanism (missing completely at random, missing at random, missing not at random)

Comments

Statistical analyses are dependent on certain assumptions, and statistical analysis plans should include methods and tests that are designed to evaluate these assumptions. Investigators should be ready to propose an alternative analysis plan if assumptions are not met during the study.

Proper sample size calculation should be conducted whenever possible to ensure that a study is adequately powered to detect meaningful differences or associations. Sample size calculations should be calculated based on the specific research question.

Summary tables for treatment and measurement timepoints should present the number of participants analyzed, as well as descriptive statistics such as mean, standard deviation, median, minimum, and maximum values, and/or response frequencies.

If significant imbalances are found in key variables of interest when comparing treatment interventions, regression methods, inversed probability weighting or matching propensity scores should be used for analysis. Researchers should be careful in interpreting the results, as those imbalances in RWD can be explained by selection of confounding by indication, which are factors that are not resolved by adjusting in your analysis.

To improve evaluations of the effectiveness of different interventions, the effect size for the primary outcome measure(s) should be calculated with available statistical methods. This approach will also facilitate comparisons of findings from different studies (42).

To include all participants in the full analysis set, imputing missing data for key variables is essential, as excluding records with missing data can severely bias results. Imputation can be done as either a primary analysis or a sensitivity analysis, and it should be reported. In RWE studies, imputing missing data is crucial to minimize bias and improve accuracy. However, caution is advised when the proportion of missing data exceeds 10%, as imputation may lead to unreliable results. Addressing missing data requires careful consideration of why data is missing and the availability of validation datasets, ensuring robust and reliable analyses and more valid conclusions about real-world treatment effects and outcomes. To properly handle missing data, it is advisable to consult with a statistician/methodologist.

Analysis of RWD that compare treatment options should be approached with caution. It is advisable to employ appropriate statistical techniques such as propensity score matching, multivariate regression models, causal inference modelling or mixed modelling methods to ensure accurate results.

Specifically, it is important not to forget that RWE studies are not randomized clinical trials. Propensity score matching is used to reduce or eliminate the effects of confounding when using observational data to estimate treatment effects. The propensity score is the probability of treatment assignment conditional on observed baseline characteristics. It allows one to design and analyze an observational (nonrandomized) study. It is a balancing score conditional on the propensity score, the distribution of observed baseline covariates which will be similar between treated and untreated subjects.

Advanced and innovative statistical techniques could be also applied to RWD. Artificial Intelligence (AI) is a comprehensive field exploring machine intelligence, while Machine Learning (ML) is a subset of AI that employs computer algorithms integrating mathematical models and data for predictions. If equipped with adequate and pertinent data and appropriate computational algorithms, ML models excel at interpreting high-dimensional datasets with numerous variables (features), even when complex biological associations are distributed across many features. ML models represent a methodological innovation that could be used in RWD studies in headache, for example, to predict response to preventive treatment. However, these techniques require experts in the field.

5. Publication of results

Recommendations.

All research results, including primary and secondary endpoints and safety data, whether positive or negative, must be published in manuscript form.

Authorship should follow the recommendations of the International Committee of Medical Journal Editors.

To ensure transparency, all authors must declare all their conflicts of interest, not only the ones that are considered as relevant for the study.

Investigators should avoid entering into agreements with sponsors, both for-profit and non-profit, that restrict access to study data, limit its analysis and interpretation, or interfere with the independent preparation and publication of manuscripts.

Study protocol should be registered and the database should be available in public repositories or should be at least available upon reasonable request.

Comments

Where applicable, RWE should be published on behalf of a study group consisting of investigators who have enrolled participants in the study. Conflicts of interest may arise when a professional's judgment concerning a primary interest, such as participants’ welfare or research validity, may be influenced by a secondary interest, such as a financial relationship with the sponsor. Potential conflicts of interest arise from financial ties such as employment, consultancies, grants, fees and honoraria, patents, royalties, stock or share ownership, and paid expert testimony. Therefore, it is essential to disclose all potential conflicts of interest to ensure transparency and integrity in reporting clinical trial results.

6. Data governance and ethics

Research ethics govern the standards of conduct for scientific researchers to protect the dignity, rights and welfare of research participants. Medical research involving human subjects should adopt the principles of the Declaration of Helsinki (43). Data governance represents the sets of strategies for the usability, availability, integrity, quality, and security of data to ensure their maximum potential (44)

A. Ethical committees

Recommendations.

Approval from the Institutional Review Board or Independent Ethics Committee is required for the study and the necessity for obtaining written informed consent depends on the study design, data source and methods of data collection (45).

Consider the support of a Contract Research Organization (CRO) in conducting and/or managing the study.

Disclose all sources of funding.

B. Informed consent

Recommendations

The appropriate patient consent for the collected RWD must be obtained. Informed consent is required for participation in a registry and a registry-based study. However, the Institutional Review Board or Independent Ethics Committee has the authority to waive informed consent if data collection is limited to clinical information and anonymized (45,46). If informed consent is necessary, it is important to follow the Good Clinical Practice Guidelines. This involves detailing the utilization of data, ensuring participants’ data privacy, and clarifying the procedure of the study (46). If a secondary data use is planned on the dataset, either secondary consent has been obtained, or the data use fits into the criteria to be exempt from obtaining explicit consent.

C. Data privacy and security

Recommendations.

Ensure that all data collected is kept confidential and that patient privacy is protected in accordance with relevant regulations and guidelines. All of the information is stored in a confidential manner.

Ensure that data are adequately protected through a validated process (e.g., anonymization, pseudonymization, codification).

Dataset should have clear and transparent data protection measures according to highest quality standards.

Comments

Anonymization is the process of irreversibly removing all personally identifiable information from a dataset, making it impossible to re-identify individuals, whereas in pseudo-anonymization personal identifiers (such as names or social security numbers) are replaced so that individuals cannot be directly identified, but the technical process allows the restoration of the original data, if needed. Finally, codification is a process where data cannot be associated with an individual because the information that identifies that person has been replaced with a code. However, the coded data is still linkable to the original information if the code key is accessible.

D. Data access and sharing

Recommendations.

Data ownership in a dataset and who is responsible for data should be stated.

To access/share data, requirements must be clear and transparent and according to best available quality standards.

To access/share data, agreements must be in place in which the integrity and data security are established, in accordance with all necessary ethical and law regulations.

The information received/accessed must be used solely for the prespecified purposes and objectives.

It is recommended to establish a data management plan to ensure appropriate data access and sharing.

It is advisable to ask for support from the legal department of the institution where the study is conducted, especially in case of planning multicenter studies that require cross-institutional collaboration and sharing of data.

7. Methodology used for guideline development

The IHS Clinical Trials Standing Committee developed the present guideline independently, without financial support and without bias. The process of developing this guideline involved two phases. In the first phase, the initial draft of the guideline was reviewed by the Clinical Guidelines Committee of the IHS, who proposed multiple changes. The Clinical Guidelines Committee of the IHS created the Data Quality Level system, based on consensus. This version of the guideline was then shared with representatives of the European Medicines Agency, the US Food and Drug Administration, pharmaceutical manufacturers, and patient associations. Their suggestions were discussed in a series of meetings with members of the Committee. After incorporating the views of these stakeholders, the Committee posted a pre-final version on the IHS website in April 2024, inviting comments from IHS members. The Committee incorporated member comments to finalize this edition, ensuring that it represents a consensus of stakeholders in the field of Headache Real-World Data research.

By following these recommendations, RWE studies can ensure that the data collected is of high quality and can be used to generate reliable and valid results. In this guideline, recommendations provide specific guidance or advice on what actions should be taken, whereas the comments provide explanatory or contextual information related to a recommendation. Furthermore, the implementation of a system to assess quality of data collection will enhance transparency regarding the data collection, as it significantly impacts the level of evidence and provides guidance for interpreting the results of RWE studies. Table 9 summarizes the most important recommendations for the highest-quality RWE.

Table 9.

Main recommendations to perform the highest-quality RWE.

Area	Recommendations
Research question	Solving research gaps (Novel research questions)
Study design	Cohort, prospective
Setting	Multicenter
Data collection	Grading level 1
Endpoints	Based on research question, a priori, clinically relevant and measurable
Statistics	Preplanned statistical analysis
Data governance and ethics	Must be guaranteed (Aspects such as Ethics approval, Informed consent, Data privacy and security, Data access and sharing are standards for research)

Supplemental Material

sj-docx-1-cep-10.1177_03331024251318016 - Supplemental material for Guidelines of the International Headache Society for Real-World Evidence studies in migraine and cluster headache

Supplemental material, sj-docx-1-cep-10.1177_03331024251318016 for Guidelines of the International Headache Society for Real-World Evidence studies in migraine and cluster headache by Patricia Pozo-Rosich, Nancy van Veelen, Edoardo Caronna, Gloria Vaghi, Marta Torres-Ferrus, Britt W.H. van der Arend, Peter J. Goadsby, Messoud Ashina, Shuu-Jiun Wang, Hans-Christoph Diener, Cristina Tassorelli and Gisela M. Terwindt in Cephalalgia