Abstract
Objective: To assess validity of a self-administered web-based migraine-questionnaire in diagnosing migraine aura for the use of epidemiological and genetic studies.
Methods: Self-reported migraineurs enrolled via the LUMINA website and completed a web-based questionnaire on headache and aura symptoms, after fulfilling screening criteria. Diagnoses were calculated using an algorithm based on the International Classification of Headache Disorders (ICHD-2), and semi-structured telephone-interviews were performed for final diagnoses. Logistic regression generated a prediction rule for aura. Algorithm-based diagnoses and predicted diagnoses were subsequently compared to the interview-derived diagnoses.
Results: In 1 year, we recruited 2397 migraineurs, of which 1067 were included in the validation. A seven-question subset provided higher sensitivity (86% vs. 45%), slightly lower specificity (75% vs. 95%), and similar positive predictive value (86% vs. 88%) in assessing aura when comparing with the ICHD-2-based algorithm.
Conclusions: This questionnaire is accurate and reliable in diagnosing migraine aura among self-reported migraineurs and enables detection of more aura cases with low false-positive rate.
Introduction
Migraine is a common brain disorder characterized by recurrent, disabling attacks of headache, autonomic features (migraine without aura; MO), and, in one-third of patients, transient neurological aura symptoms (migraine with aura; MA). In Western countries, the overall migraine prevalence in the general population is at least 12%, two-thirds of which concerns females (1–4). Since no biomarker for migraine exists, diagnosis according to the headache classification of the International Headache Society (IHS) (5) relies exclusively on the headache history. A careful history taken by a headache specialist is the gold standard for making a valid migraine and aura diagnosis.
Large scale studies with several thousands of participants are important to obtain information for epidemiological and genetic migraine research and may yield important insights in migraine pathophysiology. Migraine is a complex genetic disorders, i.e. multiple genetic and environmental factors contribute to migraine susceptibility.
Twin- and population-based family studies showed that genetic factors play an important role in migraine susceptibility, especially in the MA subtype (6–12). However, genetic linkage studies using migraine subtypes as an end diagnosis did not yield gene variants thus far. Clinical heterogeneity in migraine and aura diagnosis may have hampered the identification of such variants. Recently, in a large genome-wide association analysis with a large set of clinic-based migraineurs, a first-ever genetic risk factor was identified associated with common types of migraine, in patients that were largely recruited from specialist headache clinics with a clinic-based migraine diagnosis (13). However, population-based large-scale studies exclude the possibility of a face-to-face examination, and, therefore, a less time-consuming and less costly diagnostic strategy has to be chosen. A web-based questionnaire represents an attractive and inexpensive alternative for a clinic interview. Several groups have reported on the use of internet to recruit headache and other patients for clinical research (14–18). However, reliably diagnosing aura remains an issue.
The availability of a validated, aura-specific questionnaire is important when large numbers of cases are needed, especially in studies with self-reported migraineurs from the general population (19,20). We developed the LUMINA (Leiden University MIgraine Neuro-Analysis) website and designed and validated a self-reporting, web-based questionnaire to reliably diagnose migraine headache and aura symptoms, using only a limited number of questions. In this paper, we will present the validation of this web-based migraine and aura questionnaire.
Methods
Subjects
Participants were Dutch adults aged 18−74 years with migraine (MA and MO), who were invited via the lay press nationwide to enrol via the especially designed LUMINA website. Additionally, patients from our outpatient headache clinic were invited by a letter. In this clinic-based study, all participants were self-reporting migraineurs, of which approximately 90% had previously been diagnosed with migraine by a physician.
Study flow
Study flow is depicted in Figure 1. Patients who visited the website were informed about the study and could enrol directly. The first step was to fulfil the screening criteria, using a simple screening questionnaire that was validated previously in the population-based GEM study (3). This screening questionnaire included five questions asking: (i) whether the patient had severe headaches in the past 12 months; (ii) what the headache severity was; (iii) had suffered from headaches which were preceded by visual disturbances; (iv) had been diagnosed with migraine by a physician; and (v) had ever used anti-migraine medication. After fulfilling these criteria, cases received a unique user-ID code via email to log on to the study website, where theycould participate in an extended, web-based questionnaire study. Having completed the extended questionnaire, a number of randomly selected participants were contacted by telephone by WPJvO, CMW, and AHS, who are experienced in diagnosing migraine. This semi-structured telephone interview detailed questions on headache and aura characteristics, including ICHD-2 migraine and aura criteria (5) with special attention for visual, sensory, motor, and speech aura symptoms, was used as the gold standard. Median interview duration was 10−15 minutes, ranging up to 30 minutes if necessary. Afterwards, a final diagnosis was made: in case of ambiguity, a headache specialist (GMT) was consulted. Patients were excluded from theanalysis if they could not be reached by telephoneafter five failed telephone contact attempts. The study was approved by the local medical ethics committee. All participants provided written informed consent.
Flowchart of (semi-)automated study flow. Alg. = ICHD-2-based algorithm diagnosis; Int. = interview diagnosis; MA, migraine with aura; MO, migraine without aura; Questionnaire, extended questionnaire; Screening, screening questionnaire. *In the total MA group, 91.6% (447/488) reported visual aura symptoms.
Construction of questionnaire
The extended questionnaire (accessible via www.lumc.nl/hoofdpijn) was based on the ICHD-2 (5) and incorporated 127 items on migraine headache and aura characteristics, premonitory symptoms, trigger factors, allodynia, and medication use and was presented to participants as a digital web form. The questions were to be answered by choosing from categorical alternatives. On the web form, multicolour exemplary illustrations were shown with the most characteristic visual aura features (hemianopsia, scotoma, fortification spectra, visual blurring) and sensory aura features (anatomical distribution).
ICHD-2-based algorithm
After completion of the extended questionnaire, an algorithm based on ICHD-2 (5) migraine criteria wasrun and individual diagnosis was determined. Thealgorithm had the following possible outcomes: ‘no migraine’; ‘migraine without aura’; and ‘migraine with aura’. In the analysis, the algorithm outcomes were dichotomized into ‘aura’ and ‘no aura’ (Supplementary Figure e-1).
Statistical analysis
Descriptive statistics
Descriptive statistics were performed on demographic and clinical variables, on the algorithm-based diagnoses and on the interview-derived diagnoses. Results are reported as mean ± SD or percentage. Differences in between-groups means were analysed with independent sample t-tests and ANOVAs. Proportions were compared using Chi-squared tests. All items from the extended questionnaire thatconcerned ICHD-2 migraine criteria were evaluated separately. Likelihood ratios were calculated using standard formulas for positive likelihood ratio [LR+, sensitivity/(1−specificity)] and negative likelihood ratio [LR−, (1⊟sensitivity)/specificity].
Questionnaire validation process
The questionnaire validation process was divided into two phases and was aimed at identifying a combination of items that were better predictors for diagnosing migraine aura than the ICHD-2-based algorithm, with the interview-derived diagnosis as the gold standard. In phase I, a sample of 838 self-reported migraineurs (approximately 80% of total group) was randomly selected and used asa training sample (see Figure 1) to derive a predictivemodel. These patients fulfilled set screening criteria from the five-item LUMINA screener before they could enter the extended questionnaire. Logistic regression (see below) was used to develop the predictive model that included questionnaire items most contributing to predict subcategories ‘aura’ and‘no aura’. Subsequently, we compared both the ICHD-2-based algorithm diagnoses and the diagnoses predicted by the logistic model, to the gold standard. In phase II, we validated this derived predictive model in an independent validation sample, consisting of 200patients, approximately 20% of our sample (see Figure 1).
Phase I: development of prediction rule
In phase I, a prediction rule for the aura subcategories ‘aura’ vs. ‘no aura’ was developed using a multivariate logistic regression analysis. Relevant, individual, dichotomized items (n = 33) were selected from the extended questionnaire and were used as predictor variables for aura in the model. Selection of items was made by the authors (WPJvO; CW; GMT) and was based on clinical relevance to migraine aura and sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratios of individual items. Inter-item correlation was assessed for relevant items using Spearman’s rank coefficients and when items correlated with coefficients >0.9, one of these items was excluded from the analysis. A forward selection strategy using the likelihood ratio test was performed to identify items that were significant (p < 0.05) predictors for the outcome of aura. For each subject in this sample (n = 838), a prediction score was calculated using these items. Subsequently, a receiver operating characteristic (ROC) curve was generated to assess the optimum cut-off point for this prediction score. Using the method proposed by Halpern et al. (21), an optimum cut-off (highest sensitivity and specificity) was determined from the ROC curve. Therefore, the logistic model resulted in a selection of the 33 items with significant (p < 0.05) contribution in the aura prediction.
Phase II: validation of prediction rule
The derived predictive rule was was subsequently validated in the second sample (validation sample; n = 200; see Figure 1). Validity of this predictive model was assessed by checking whether the selected items contributed significantly (p < 0.05) for the prediction in the second sample too. Subsequently, the sensitivity and specificity from the ROC optimum in the training sample were compared with these parameters in the validation sample, using the same cut-off value.
Overall outcome measures
Sensitivity, specificity, PPV and NPV were calculated to compare the fit of the three different models with the interview-derived aura diagnosis as the gold standard. These models were: (1) ICHD-2-based algorithm; (2) predictive model from phase I; and (3) validation of predictive rule in phase II.
All data analyses were performed using SPSS 16.0.2 (SPSS, IBM, USA). p-values less than 0.05 were considered significant. When appropriate, categorical items were dichotomized into binary variables for the analysis in an attempt to simplify the instrument.
Receiver operating characteristic curve
From the data in the training sample, we generated an ROC curve by plotting the sensitivity of the questionnaire against 1−specificity. As a graphical representation of the trade-off between false-negative and false-positive rates for every possible cut-off point, the ROC curve reflects the trade-offs between sensitivity and specificity and plots the false-positive rate on the X axis and the true-positive rate on the Y axis. The area under the curve is a measure of correlation between the prediction of the questionnaire and the gold standard diagnosis. The closer the area under the curve (AUC) is to 1, the better the test is. To validate the derived logistic model, we compared the ROC from the prediction sample (n = 838) to the ROC of the validation sample (n = 200).
Results
General results
Over a 1-year period, from April 2008 until April 2009, 2397 subjects fulfilled the set screening criteria and completed the extended questionnaire (Figure 1). During this time period, a total of 1067 subjects (44.5%) were randomly selected for the semi-structured telephone interview, of which 1038 (97.3%) were reached and could be used in the analysis. A total of 29 subjects (2.7%) were not included in the analysis because they could not be reached by telephone, after having tried at least five times. From these 1038 subjects, 838 (79.4%) were randomly selected and used for the prediction model and the remaining sample of 200 subjects (18.9%) was used for validation (Figure 1).
Baseline characteristics of total study population and separate study samples
Values are mean ± standard deviation or %.
p < 0.001 (χ2-test).
M, migraine.
Screening questionnaire
In total, 94.6% of subjects (982/1038) fulfilling the screening criteria, fulfilled ICHD-2 migraine criteria in the telephone interview. We considered everyone fulfilling the screening criteria to be migraineur. We used a logistic model to predict individual aura vs. no aura status.
Algorithm diagnosis
Sensitivity, specificity, and positive and negative predictive values as well as the corresponding likelihood ratios for diagnosis of migraine aura based on the IHCD-II-based algorithm (in both the total group and training sample) and the derived seven-item prediction model (in both the training sample and in the validation sample)
MA, migraine with aura; MO, migraine without aura; NPV, negative predictive value; PPV, positive predictive value.
Phase I: derivation of predictive model
Significantly correlated questions (n = 7) with regression coefficients, odds ratios and significance levels derived from the logistic regression model (training sample; n = 838)
B, regression coefficient; CI, confidence interval; OR, odds ratio.
ROC curve
From the data in the predictive cohort, we generated an ROC curve by plotting the sensitivity of the questionnaire against 1−specificity (Figure 2a). This analysis resulted in an optimal cut-off point for the used logistic model at 0.35 with AUC of 0.85 (95% CI 0.83⊟0.88), yielding a seven-item questionnaire with a sensitivity of 0.83 and a specificity of 0.74. Compared to the ICHD-2-based algorithm outcome, this approach therefore resulted in a vast increment in sensitivity, with only small decrement of specificity (Table 2).
Receiver operating characteristic curves for (a) the derived prediction rule in the initial training sample (n = 838) and (b) in the validation sample (n = 200). The area under the ROC curve (C-statistic) for the prediction rule was 0.85 (95% CI 0.83⊟0.88) in the training sample and 0.87 (95% CI 0.82⊟0.92) in the validation sample.
Phase II: validation of derived prediction rule
Using the predictive model and cut-off point (0.35) derived from the training sample (n = 838), we validated this model in a second, independent sample (n = 200) of subjects who also fulfilled the set screening criteria. This analysis showed the model to have approximately similar sensitivity and specificity in this validation sample (Table 2). In the validation cohort, the ROC curve yielded an AUC of 0.87 (95% CI 0.82⊟0.92), which is comparable to the output from the training cohort (Figure 2b). When using this cut off from the training cohort, migraine aura diagnosis was predicted correctly in 160/200 (80.0%) of subjects.
Test−retest reliability
For a random selection of 44 patients who completed the extended questionnaire, a second time, with a mean test⊟retest interval of 155 days (median 89 days, range 1–422 days), test⊟retest reliability was found to be good with a test⊟retest kappa for algorithm diagnostic group of 0.59 (95% CI 0.38−0.80). Test⊟retest interval did not influence agreement (linear regression, p = 0.852).
Discussion
Our study has been the first one to validate a web-based questionnaire for purposes of diagnosing aura cases using a large sample of self-reported migraineurs. Few previous studies on migraine screeners and questionnaires have focused on migraine aura and the numbers of MA cases used to validate the questionnaire instruments in these studies were limited to n = 8⊟186 (17,19,22–24) respectively, in comparison to the large number of 488 aura cases in our study. Physicians frequently rely on aura as a cardinal symptom of migraine, as suggested by the 1.9-fold higher rate of medical diagnosis in interview settings when comparing MA cases to cases of MO (25). Our study shows that, in self-reported migraineurs, a distinction between MA and MO can be made via a self-administered web-based questionnaire, with a focus on visual aura symptoms. The difficulty in diagnosing other aura types might be explained by the lack of perception and recognition of verbal and other non-visual auras by patients (26). For diagnosing patients with these specific aura symptoms a clinical interview is needed. However, since the vast majority of the self-reported aura cases suffer from visual auras and only a small minority suffers from non-visual auras (27), we believe the number of non-visual aura cases is negligible when recruiting from a population of self-reported migraineurs. Perhaps the most helpful item identifying aura cases is the duration of the aura phenomena, since this question enables to distinguish visual aura symptoms from non-specific visual disturbances. Additionally, our data show aura patients are less likely to use triptans for rescue medication, which might be an indicator of lower headache severity.
We show that the question addressing the duration of the headache may hamper correct identification of migraine cases in a web-based questionnaire setting because some migraineurs overestimate the duration of an attack. Conversely, a question addressing headache severity should be included because this is helpful in distinguishing aura cases with migraineous headache from patients with non-specific headache.
The strength of our study includes the large samples of both the training (n = 838) and validation sample (n = 200), which are representative for the population studied. Both outclinic patients and other patients (most of whom are treated by their own GP or neurologist elsewhere) were included via the same web-based flow. We found no clinical or demographic differences between these populations that could have affected the predictive model. Secondly, the use of a telephone interview as a gold standard by well-trained physicians with consultation of a headache specialist assured precise categorization of migraineurs. Although we did not have a face-to-face interview as gold standard, we feel that our thorough semi-structured telephone interview safeguarded a very reliable migraine and aura diagnosis. Thirdly, the use of a validated screening instrument prior to our new questionnaire resulted in a group of self-reported migraineurs in which 95% could in fact be diagnosed with migraine. Fourth, we used a web-based questionnaire that was easy to fill out and send in for participants. With this approach, we successfully recruited large samples of migraineurs and contributed to the identification of the first genetic risk factor for the common forms of migraine (13). We included a selected population of self-reported migraineurs, that had already been diagnosed with migraine by a physician, or otherwise thought they suffered from migraine, in which our questionnaire shows a high reliability in diagnosing aura. Our study did not aim to validate the questionnaire as a screening instrument for migraine in a naive, general population.
The World Wide Web as a tool for recruiting patients and conducting research has several advantages. First, a large and diverse subject population can be reached at low cost (16). Second, internet research imposes fewer burdens on participants compared to non-internet research (15). Third, available software permits data entry and analysis in a secure web database. Fourth, investigators may be able to increase patient awareness and participation on clinical research. However, there might be certain challenges too (28). Internet users tend to be younger and better educated than the patient population as a whole; visually impaired and minority groups may be underrepresented; and the symptoms expressed by participants may be more severe then is typical. We feel, however, these potential biases have not pivotally influenced our data. Additionally, the so-called ‘virtual Munchhausen syndrome’, i.e. individuals referring themselves for studies for which they are not truly eligible, may compromise the validity of results (29). In our study, we have no evidence that data have been influenced by subjects masquerading electronically as patients. This is in accordance with previous migraine research (15). Even with such biases, altogether, the internet represents an appropriate aid to conduct research aimed at collecting clinical headache data from large numbers of patients.
We conclude that our web-based recruitment system in combination with an automated study flow is a very successful instrument to truly distinguish between MA and MO in self-reported migraine patients. We propose to use our identified seven questions that have a higher accuracy in identifying aura cases from a population of self-reported migraineurs than an ICHD-2-based algorithm.
Footnotes
Funding
This work was supported by grants of the Netherlands Organisation for Scientific Research (NWO) (MDF: 903-52-291, Vici 918.56.602; GMT: 907-00-217, Vidi 917-11-319) and by a grant from the Centre for Medical Systems Biology (CMSB) established by the Netherlands Genomic Initiative/Netherlands Organisation for Scientific Research (NGI/NWO).
Conflicts of interest
MDF, has, in the past 3 years, received grants and consultancy/industry support from Almirall, Coherex, Colucid, Eisai, GlaxoSmithKline, Linde, MAP, Medtronic, Menarini, Merck, Minster, Pfizer, and St Jude, and independent support from the Netherlands Organisation for Scientific Research (NOW). GMT has received grants and consultancy/industry support from Merck, Janssen-Cilag, and Almirall, and independent support from NWO. The other authors report no conflicts of interest.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
