Abstract
Objective
To identify migraineurs and headache-free individuals with an online questionnaire and automated analysis algorithm.
Methods
We created a branching-logic, web-based questionnaire – the Penn Online Evaluation of Migraine – to obtain standardized headache history from a previously studied cohort. Responses were analyzed with an automated algorithm to assign subjects to one of several categories based on ICHD-3 (beta) criteria. Following a pre-registered protocol, the primary outcome was sensitivity and specificity for assignment of headache-free, migraine without aura, and migraine with aura labels, as compared to a prior classification by neurologist interview.
Results
Of 118 subjects contacted, 90 (76%) completed the questionnaire; of these 31 were headache-free controls, 29 migraine without aura, and 30 migraine with aura. Mean age was 41 ± 6 years and 76% were female. There were no significant demographic differences between groups. The median time to complete the questionnaire was 2.5 minutes (IQR: 1.5–3.4 minutes). Sensitivity of the Penn Online Evaluation of Migraine tool was 42%, 59%, 70%, and 83%, and specificity was 100%, 84%, 93%, and 90% for headache-free controls, migraine without aura, migraine with aura, and migraine overall, respectively.
Conclusions
The Penn Online Evaluation of Migraine web-based questionnaire, and associated analysis routine, identifies headache-free and migraine subjects with good specificity. It may be useful for classifying subjects for large-scale research studies.
The following research study is a not a clinical trial.
Introduction
Migraine is a highly prevalent, complex neurologic disorder (1), and includes variants with and without antecedent aura. Migraine research studies require a means to classify subjects by headache type and to identify headache-free subjects as a comparison group. Traditionally, a clinical interview by a neurologist is used for this purpose, often guided by the diagnostic criteria of the International Classification of Headache Disorders (ICHD) (2). While considered the “gold standard”, a limitation of this approach is that it can be time-consuming, resource-limited, and difficult to standardize.
Numerous screening tools have been proposed as an alternative to identify patients with migraine. Many of these are designed to screen for migraine within the primary care setting (3–11), and involve three to five questions regarding duration of, or disability from, headaches and associated headache features including nausea, photophobia, and/or phonophobia. Several surveys have been designed specifically for research purposes (1,12,13). While these tools can identify migraineurs, they generally are not designed to distinguish between migraine with aura (MwA) and migraine without aura (MwoA) (except for DMQ-3, which is no longer available online) (13), do not identify headache-free individuals, and are manually administered and analyzed.
Here we present a web-based questionnaire and automated analysis code that applies the ICHD-3 (beta) criteria to subject responses; we dub the instrument the “Penn Online Evaluation of Migraine”, or POEM. Our goal is to provide an efficient means to classify subjects as migraineurs or headache-free controls. The questionnaire makes use of branching logic to minimize test duration and the presentation of irrelevant questions. The responses are processed with unit-tested, open-source routines that automate headache classification. To validate the POEM, we identified subjects who had previously been assigned a headache classification (including the classification of headache-free) by one of us as part of a prior research study (14). Following a pre-registered protocol (15), we contacted these subjects and invited them to complete the POEM. We then compared the headache classification provided by the POEM with their prior classification and computed the sensitivity and specificity of the test.
Methods
Ethics statement
The study was approved by the University of Pennsylvania Institutional Review Board; all participants provided written informed consent for participation in the original study from which subjects were drawn. For assessment of the web-based questionnaire, a waiver of written consent was granted by our Institutional Review Board.
Design of the POEM instrument
Using a commercially available survey tool (Qualtrics, Provo, UT), we created a branching-logic, web-based questionnaire that guides subjects through a series of questions regarding headache history and associated symptoms. The question text and available responses were designed to assign patients to the ICHD-3 (beta) categories of MwoA or MwA, and to further distinguish between visual and non-visual aura. In addition, questions attempted to differentiate between individuals who are headache-free (HA-free) or experience headaches that are mild and have non-migrainous features (mild non-migrainous headache, MNMH) versus those people with headaches that are more severe or have some migrainous features (headache not otherwise specified, NOS). In preparation for future research but unrelated to the development of POEM, subjects who endorsed photophobia were also asked questions regarding their ictal photophobia symptoms taken from a prior report (16); these responses are not further considered here. Finally, the survey included questions regarding headache recency, medication use, childhood motion sickness, menstrual headaches, and family history of migraine. This final set of questions did not inform the assignment of diagnostic category described here except for menstrual headaches.
As a validity check, subjects were asked “Did you take this survey seriously and are your answers sincere? Are all your answers correct?” and offered three response choices: 1) “Yes, I took the survey seriously and my answers are correct”; 2) “I made an error and want to retake the survey”; 3) “I didn’t take this seriously. Don’t use my answers”. If a subject responded with the second response, they were invited to retake the survey, and the second set of responses were retained. If they failed to successfully retake the survey, they were excluded from the study. No subjects replied with the third response. For an illustration of the branching-logic scheme, see Figure 1; for a complete list of the survey questions and available responses, see Supplemental Figure 1.
Diagram of POEM questionnaire.
Categorical assignment was performed using automated analysis routines implemented in MATLAB (17). While the code (and associated unit tests) provide the canonical description of the algorithm, we outline here the principles followed for categorical assignment. Inclusion criteria for MwoA, migraine with visual aura, and migraine with other aura are outlined in Supplemental Tables 1–3 and are based on ICHD-3 (beta) criteria. If a subject meets the criteria for both MwA and MwoA, they are classified as MwA. A subject can receive a classification of both migraine with visual aura and migraine with other aura. For headache-free, subjects can meet criteria based on three potential pathways (A, B, or C) within the branching logic structure (Supplemental Table 4) in which they deny any headaches or facial pain/discomfort/pressure or deny having headaches that were not caused by head injury, hangover, or illness. If subjects do not meet the criteria for headache-free, they can be considered for mild non-migrainous headache unless their responses meet any exclusion criteria that suggest moderate to severe headaches and/or headaches with at least one migrainous feature (Supplemental Table 5). Finally, if subjects do not meet the criteria for any of the prior classifications, subjects are given a classification of headache NOS.
Validation study
Subjects were recruited from a prior observational cohort, the Anatomy and Cerebral Hemodynamic Evaluation of Migraine (ACHE-M) study, which included both migraine and headache-free control subjects. A detailed description of ACHE-M has been published previously (14); briefly, this was a prospective case-control study using magnetic resonance imaging (MRI) to compare vascular structure and function between headache-free controls and MwA and MwoA patients. Participants were 25–50 years old at the time of ACHE-M enrollment, which occurred between March 2008 and June 2012. Participants were screened and examined by a single board-certified neurologist with expertise in headache to determine headache status; subjects were classified as MwA or MwoA using International Classification of Headache Disorders criteria (edition 2) (18). Note that there have been no changes in the criteria in migraine without aura and only minor changes for migraine with aura between editions 2, 3 (beta), and 3, which has been subsequently accepted and published after this study was conducted (19). Subjects classified as headache-free healthy controls could not have any history of potentially migrainous headaches (including ‘sinus’ headaches or any type of headache requiring medication) and could not have tension-type headache unless it occurred exceedingly rarely (defined as less than once a year). Isolated, infrequent, provoked headaches associated with excessive alcohol consumption or transient illness other than reported sinusitis were allowed. Family history of migraine was not used to exclude headache-free controls in this study.
Subjects from ACHE-M were contacted by email and invited to participate in the current research project; each email included an individualized web link for the online questionnaire. Subjects were initially contacted on September 15, 2017. No formal a priori power calculation was conducted; thus, sample size was dictated by prior ACHE-M study enrollment (14), so efforts were made to recruit as many prior ACHE-M subjects as possible to participate in this validation study. Thus, subjects who failed to respond to the initial email were re-contacted twice more, roughly 2 weeks and 4 weeks after the initial email. Following the pre-registered protocol, subjects who failed to complete the entire survey by 1 November 2017 were excluded.
Data analysis
Groups of participants were compared using the chi-squared or Fisher’s exact test for dichotomous or categorical variables and the t-test or Wilcoxon ranked-sum tests for continuous variables as appropriate. For comparison of group differences between responders and non-responders, standardized mean difference was calculated for continuous variables with normal distributions. Means and standard deviations or medians and interquartile ranges (IQR) are presented as appropriate. An association was considered significant if p < 0.05. All tests were two-sided. Statistical analyses were performed using JMP (Version 9, SAS Institute Inc., Cary, NC, USA). Specificity and sensitivity were calculated for each ACHE-M classification group (i.e. MwoA, MWA, headache-free control). For this analysis, we collapsed the POEM labels of migraine with visual aura and migraine with non-visual aura into a single category. The category of headache-free controls also included patients with mild, infrequent, non-migrainous headache, as was used in ACHE-M (20). Sensitivity was calculated by dividing the number of true positives by the number of true positives and false negatives. Therefore, for headache-free controls, true positives included the number of ACHE-M headache-free control subjects correctly classified by POEM as HA-free and MNMH, and false positives included the number of ACHE-M headache-free control subjects incorrectly classified by POEM as NOS, MwoA, and MwA. The same approach was used to calculate sensitivity for MwoA and MwA. Specificity was calculated by dividing the number of true negatives by the number of true negatives and false positives. So, for instance for MwoA, the true negatives included ACHE-M headache-free control subjects correctly classified by POEM as something other than MwoA (i.e. HA-free, MNMH, NOS, MwA) and ACHE-M MwA subjects correctly classified by POEM as something other than MwoA (i.e. HA-free, MNMW, NOS, MwA), and the false positives included subjects who were ACHE-M headache-free controls or MwA subjects and incorrectly classified by POEM as MwoA. The same approach was used to calculate specificity for headache-free control and MwoA.
Data and code availability
The analysis code is open source and available for modification, distribution, and use with attribution (17). The repository includes the questionnaire itself in “qsf” format, which may be used to implement the instrument in Qualtrics. The anonymized raw data and code that produced the results presented in the current report are also available (21).
Pre-registration
The study and methodology for analysis were pre-registered prior to data collection and analysis (15). There were no deviations from the study protocol.
Results
Study subjects
Of the original 172 ACHE-M subjects, valid email addresses were available for 118 subjects, and of these 90 (76%) completed the POEM questionnaire. Subjects who completed the POEM questionnaire were approximately evenly distributed among the three ACHE-M classification groups. Subject recruitment is illustrated in Figure 2. Clinical characteristics of subjects are described in Table 1. Aside from age, which reflects current age at time of completion of the POEM questionnaire, other subject characteristics were collected at time of enrollment in ACHE-M. There was no significant difference between the headache-free control and migraine groups in terms of age, sex, migraine duration since age of onset, blood pressure, or history of smoking. There was no significant difference in migraine frequency or use of migraine-prophylactic medication between MwA and MwoA groups. Also, there were no significant differences in demographics (age, sex) or headache characteristics (migraine duration, frequency, prophylactic medication use) between the 90 subjects who completed the POEM questionnaire and the other 82 ACHE-M subjects who did not complete the POEM questionnaire (Supplemental Table 6).
Flow diagram of subject recruitment. Clinical characteristics. Data recorded at time of enrollment in ACHE-M with the exception of age, which was calculated at time of POEM completion based on date of birth. Control: headache-free control; MwoA: migraine without aura; MwA: migraine with aura; SD: standard deviation; n: number; IQR: interquartile range (25% percentile – 75% percentile).
Completion of the POEM questionnaire required a median time of 2.5 minutes (IQR: 1.5–3.4 minutes). Only one subject reported making an error during completion of the questionnaire; this subject successfully completed the questionnaire on a second attempt. No subjects indicated they did not take the survey seriously.
Validation of POEM instrument
Confusion matrix comparing classification by direct neurologist interview versus POEM.
Control: headache-free control; HA-free: headache-free; MNMH: mild non-migrainous headache; NOS: headache not otherwise specified; MwoA: migraine without aura; MwA: migraine with aura.
POEM sensitivity and specificity for headache-free controls, migraine without aura, migraine with aura, and migraine with or without aura combined.
Control: headache-free control; MwoA: migraine without aura; MwA: migraine with aura.
We examined the 14 cases in which there was disagreement between the POEM and the original headache classification by the ACHE-M study neurologist. Comparing the misclassified subjects to those correctly classified, there was no significant difference in age, sex, migraine frequency, or years with migraine. Three headache-free control ACHE-M subjects now reported symptoms consistent with migraine and were classified by the POEM instrument as having MwA (one subject) or MwoA (two subjects). Also, three MwoA ACHE-M subjects now reported aura symptoms and were classified as MwA by the POEM: Two endorsed visual aura and one endorsed numbness or tingling. Eight subjects classified as MwA by ACHE-M were classified by the POEM instrument as MwoA. All eight subjects endorsed visual aura symptoms in the questionnaire but failed to meet strict POEM criteria for MwA for the following reasons: Two subjects indicated they had aura only once in their lifetime (and thus did not meet criteria for a minimum of two lifetime episodes); one subject reported having visual aura typically greater than 60 min in duration, and five subjects reported having visual aura typically less than 5 min in duration (and thus did not meet criteria for aura symptoms lasting between 5 and 60 minutes).
Discussion
The POEM is an open-source, automated implementation of the ICHD-3 (beta) criteria for migraine classification. Based on responses from a self-administered, online questionnaire using branching logic, the POEM instrument classifies subjects into one of six categories: Headache-free, mild non-migrainous headache, headache not otherwise specified, migraine without aura, migraine with visual aura, and migraine with other aura. Subjects are generally able to complete the POEM in less than 3 minutes and with rare self-reported errors. The POEM is moderately sensitive but is highly specific in identifying subjects as headache-free controls (headache free or mild non-migrainous headache), MwoA, or MwA. It was designed to strictly apply the ICHD-3 (beta) criteria to potential subjects.
We conducted a validation of the POEM by testing a group of subjects who had received a prior headache classification. Non-responders to the POEM questionnaire could potentially introduce bias in our validation study. However, response rates were similar between the three ACHE-M classification groups and there were no significant clinical differences between responders and non-responders in demographics (age, sex) or headache characteristics (migraine duration, frequency, prophylactic medication use; see Supplementary Table 6). One limitation of our validation study is that the original ACHE-M subjects were screened extensively and selected based on ACHE-M study requirements and may not be representative of a broader general population of headache-free and migraine individuals. Further testing of POEM in a larger general population would be desirable.
Another limitation of this validation effort is the time interval (5–9 years) between classification by the study neurologist for ACHE-M and completion of the POEM questionnaire. This may contribute to the lower sensitivity obtained for the POEM as compared to prior screening tools. While the POEM questions were designed to capture headache and associated symptoms throughout a subject’s lifetime, there is evidence that headache status can change (22–26), and it is likely that recall of prior headache symptoms degrades over time. Prior epidemiological studies have reported variable reproducibility in the diagnosis of migraine. In one study, migraine diagnosis was reproducible in 78% of the population over 12 months when using strict ICHD criteria (26), but in another study that decreased to only 48% after more than two years (22). Another population-based study found that in a 3-year period the incidence of migraine in individuals under 45 years of age increased by a third in women and doubled in men (24). We did not ask subjects to state if or how their headaches had changed since enrollment in ACHE-M, so we cannot directly address the question of how this contributed to misclassification. However, it is perhaps not surprising that a subset of patients initially classified in ACHE-M as headache-free controls went on to develop migraine over the ensuing years.
Also, this study was not powered to validate whether POEM can sufficiently classify migraine with aura subjects as migraine with visual aura or migraine with other aura. Additional studies with a larger pool of migraineurs are needed. Nevertheless, subcategorizing migraine phenotypes is critical for research studies of pathophysiology, such as studying migraine with visual aura individuals to better understand visual cortex hypersensitivity.
Of the 14 subjects who were misclassified by the POEM compared to the neurologist’s classification for ACHE-M, eight of the subjects were initially classified as MwA, while the POEM classified them as MwoA. In all eight cases, the subjects endorsed aura symptoms but failed to meet ICHD criteria for MwA, although they would have met criteria for probable MwA. In six cases, the POEM instrument excluded those subjects based on the duration of their symptoms. The specific question posed to subjects was “How long do these phenomena typically last?”. The ICHD criteria do not require that aura phenomena “typically” last between 5 and 60 minutes, only that some episodes do so. We suspect that the particular wording of this question was responsible for the overly conservative behavior of the POEM for the diagnosis of MwA. We have since modified the POEM questionnaire so that, if the subject indicates a typical aura duration of less than 5 or greater than 60 minutes, they are asked “Do these phenomena ever last 5–60 minutes?” An affirmative response is taken as consistent with a classification of MwA. Using this criteria for aura duration, we suspect that six of eight subjects who had been incorrectly classified as MwoA would instead be correctly classified as MwA by the POEM tool, increasing specificity for both MwA (94%) and MwoA (93%), and increasing sensitivity for MwA (90%).
The POEM instrument provides a “NOS” classification for individuals with headaches that are more significant than mild non-migrainous headaches but do not meet strict criteria for migraine with or without aura. In our validation study, 25 ACHE-M subjects were classified as NOS by the POEM instrument. Fifteen of those subjects had been initially classified in ACHE-M as controls but now were endorsing symptoms that did not meet criteria for headache-free or mild-non migrainous headaches. Some of this discrepancy may be related to differences in how ACHE-M defined headache-free and the POEM definitions. Changes in headache characteristics over time may also account for these differences. In all 15 cases, subjects endorsed one or more symptoms that excluded them from the mild-non-migrainous headache category. An additional nine subjects were classified by ACHE-M as migraine without aura but classified by POEM as NOS; seven of those subjects missed migraine without aura by one criterion. The one MwA subject who was classified by POEM as NOS denied aura symptoms and missed one criterion for migraine without aura. While the NOS classification may not be clinically useful, it serves the important function of providing a category for individuals who do not meet criteria for either headache-free or migraine.
Ultimately, the POEM is likely best used for the efficient screening of a large population of individuals to identify with high specificity both control and migraine subjects for research studies. Decreased sensitivity is a consequence of strictly applying ICHD criteria.
Comparison of migraine screening tools.
Photo: photophobia; Phono: phonophobia; MwoA: migraine without aura; MwA: migraine with aura; HA-free: headache free; ICHD: International Classification of Headache Disorders. “Other” refers to additional questions beyond the characteristics listed. Manual administration may be paper form and/or interview.
Trademarked by Pfizer.
These different instruments have been validated in various ways. Study populations have included clinical (6,12) and non-clinical populations, with the latter composed of Pfizer employees (9), university students (11), or the general US population (1). Sample sizes have ranged from 140 (9) to 942 (11) with a median of 200 subjects in each validation study. Five of the six studies used a clinical diagnosis by a neurologist as the gold standard, whereas the Migraine-4 (11) used the Standard-Diagnostic Interview for Headache-Revised (29).
In contrast to currently available tools, the POEM offers several advantages. First, the POEM not only identifies patients with migraine, but provides sub-classification for migraine without aura, migraine with visual aura, and migraine with other aura. Second, the POEM is the only tool that specifically identifies headache-free subjects. Third, the POEM (like the Migraine-4) implements ICHD-3 (beta) criteria. Finally, the POEM is an open-source, automated, web-based tool, allowing for easy adoption by clinicians and researchers.
Conclusions
The POEM is a new instrument to classify subjects into headache categories, including migraine with visual aura, migraine with other aura, migraine without aura, headache not otherwise specified, mild non-migrainous headache, and headache free. The migraine classifications meet ICHD-3 (beta) criteria (2). The tool is self-administered, avoiding time- and cost-intensive interviews. The categorization is automated via open-source code. In our validation study, we find the POEM to have good specificity, and thus to be suitable for the enrollment of subjects in migraine research trials.
Article highlights
We have developed a self-administered, online survey and automated, open-source tool entitled the Penn Online Evaluation of Migraine (POEM) that provides six distinct headache classifications. To validate the POEM, 90 subjects completed the instrument and their classifications were compared to those given by a study neurologist during the ACHE-M study. While moderately sensitive, the POEM instrument is highly specific in classifying subjects into the categories of headache-free, migraine without aura, and migraine with aura, and is thus useful for enrolling subjects for research studies.
Supplemental Material
Supplemental Material1 - Supplemental material for A web-based, branching logic questionnaire for the automated classification of migraine
Supplemental material, Supplemental Material1 for A web-based, branching logic questionnaire for the automated classification of migraine by Eric A Kaiser, Aleksandra Igdalova, Geoffrey K Aguirre and Brett Cucchiara in Cephalalgia
Supplemental Material
Supplemental Material2 - Supplemental material for A web-based, branching logic questionnaire for the automated classification of migraine
Supplemental material, Supplemental Material2 for A web-based, branching logic questionnaire for the automated classification of migraine by Eric A Kaiser, Aleksandra Igdalova, Geoffrey K Aguirre and Brett Cucchiara in Cephalalgia
Footnotes
Declaration of conflicting interests
The authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: EAK has received royalties from a patent shared with Alder Biopharmaceuticals. AI, GKA, and BC declare that they have no relevant financial interests that relate to the research described in this paper.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by grants from the National Eye Institute (R01EY024681 to GKA), National Institute of Neurological Disorders and Stroke (R25 NS065745 to EAK, R01 NS061572 to BC) and Department of Defense (W81XWH-151-0447 to GKA).
Institutional Review Board approval
This study (IRB #806975; Title: Circle of Willis Variability and Migraine) was approved by the University of Pennsylvania Institutional Review Board; all participants provided written informed consent.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
