Abstract
Background
Cluster headache (CH) is a primary headache disorder that is diagnosed based on the patient's history. For large-scale epidemiologic and genetic studies, a web-based, preferably short, questionnaire can be a feasible alternative to replace time-consuming clinical interviews.
Methods
Self-reported CH patients were enrolled via our research website. Participants meeting screening criteria were directed to the Leiden University Cluster headache Analysis program (LUCA) questionnaire. Individual diagnoses were calculated using an algorithm based on International Headache Society criteria. Subsequently, semi-structured telephone interviews were carried out to validate the LUCA questionnaire. The shorter Quick Ascertainment of Cluster Headache (QATCH) questionnaire for diagnosing CH was constructed by using logistic regression to select the most predictive questions.
Results
Via our website 437 self-reported CH patients were recruited. Of these, 291 patients were included in this cross-sectional study. The LUCA questionnaire was valid and accurate. Using logistic regression, three questions (QATCH) provided similar sensitivity (53.8% vs. 57.2%), specificity (88.9% vs. 87.5%), positive predictive value (95.5% vs. 95.9%) and negative predictive value (30.8% vs. 28.8%) compared with the LUCA questionnaire.
Conclusion
The web-based LUCA questionnaire was accurate and reliable in diagnosing CH among self-reported patients. Males with headache attacks of short duration and long headache-free intervals (months to years) are very likely to have CH.
Introduction
Cluster headache (CH) is one of the so-called ‘trigeminal autonomic cephalgias’ (TACs) that are characterized by severe, short-lasting headache attacks accompanied by ipsilateral facial autonomic symptoms (1). CH consists of attacks of severe, strictly unilateral, orbital, supraorbital and/or temporal pain, lasting 15 to 180 minutes and occurring from once every other day to a maximum of eight times a day (2). The attacks are associated with one or more of the following symptoms: restlessness or ipsilateral autonomic symptoms, i.e. conjunctival injection, lacrimation, nasal congestion, rhinorrhea, forehead and facial sweating, miosis, ptosis, and eyelid edema. CH can be either episodic or chronic, with the vast majority of patients having episodic CH. Episodic CH is characterized by periods of weeks to months with frequent attacks that alternate with symptom-free periods of several months to years. About 10% of CH patients have chronic CH without attack-free periods or attack-free periods of less than one month. Recent epidemiologic studies have documented that the life-time prevalence of CH ranges from 0.05 to 0.4% (3). CH is more prevalent in men (ratio male to female of 4.4:1) with a peak age of onset between the ages of 20 and 29 years (3,4).
CH is a complex genetic disorder; i.e. multiple genetic and environmental factors contribute to CH susceptibility (3). Based on a prevalence of 0.2%, the relative risk for first-degree family members of CH patients varies between five and 18 and for second-degree relatives between one and three (3). Thus far one genetic factor, i.e. a variant in the hypocretin type 2 receptor gene HCRTR2, was found to be associated with CH, albeit inconsistently (5).
More research is necessary to elucidate the genetics of CH. Genome-wide association studies (GWAS) aim to identify genetic factors by testing hundreds to thousands of single-nucleotide polymorphisms of affected individuals for associations with complex genetic diseases or particular traits (6). Recent GWAS have been successful for migraine (7–9). However, at least several hundred but preferably thousands of patients are needed for this type of study. Direct diagnostic interviews of such a large number of subjects are time consuming and expensive; therefore, a short reliable questionnaire would be preferable. An important pitfall of using such self-administered questionnaires is the inclusion of false-positive cases. This is a major concern for genetic research for which it is of the utmost importance to obtain reliable diagnoses.
Studies validating self-administered screening questionnaires for CH so far were small and included highly selected patients (10–13). We aimed to develop web-based questionnaires for recruiting large numbers of patients with self-reported primary headache syndromes such as CH and migraine. Recently, we published the first results of our Leiden University Migraine Neuro Analysis (LUMINA) study in which we described a reliable web-based screening questionnaire for migraine (14) that has been used in our GWAS for migraine (7,9).
In the present cross-sectional study, we developed and validated our Dutch, web-based CH questionnaire ‘Leiden University Cluster headache Analysis program’ (LUCA questionnaire) to diagnose CH patients. In addition, we assessed which questions from the LUCA questionnaire contribute most in assessing CH diagnosis to develop an ultra-short ‘Quick Ascertainment of Cluster Headache’ (QATCH) questionnaire to select patients in a large cohort.
Materials and methods
Subjects
Male and female patients aged 18 years and older were recruited via our headache research website (www.lumc.nl/hoofdpijn), which was developed in 2008 for research of primary headache disorders in the Netherlands. We aimed to recruit self-reported CH patients as our previous efforts to collect a population-based cohort in the Netherlands in the large population-based GEM-study (15) led to identification of only a few CH patients (personal communication, GMT and MDF, 1996). CH patients who participated in previous studies by our group, in which patients were recruited via general practitioners and tertiary headache centers throughout the Netherlands, were invited by letter to complete the web-based questionnaire (16). Announcements were made in local newspapers and on national medical TV programs. Self-reported CH patients were able to participate on their own initiative. Approval for this study was obtained by the local medical ethical committee. All participants provided written informed consent.
Study flow
The study flow is depicted in the Figure 1. Our study consisted of a three-step procedure. Phase 1 consisted of the actual enrollment of participants with a screening questionnaire, the LUCA questionnaire and a calculated diagnosis by algorithm based on the International Classification of Headache Disorders: 2nd edition (ICHD-II) (the LUCA questionnaire is available by request at www.lumc.nl/clusterheadache). In phase 2 we validated the accuracy of the extended questionnaire (LUCA) by performing a direct interview. In the last phase of the study we aimed to develop a shorter questionnaire (QATCH) for diagnosing CH with at least the same accuracy as the LUCA questionnaire.
Flowchart study flow.
Phase I: Enrollment of patients
First the subjects were informed about the study and were asked to complete a short screening questionnaire (Supplementary Figure 1) on the headache website. The purpose of this screening questionnaire, which was minimally adapted from previous questionnaires in migraine (questions on restlessness, autonomic symptoms and attack duration were added and questions on possible aura symptoms were removed) (15), was to exclude participants who were very unlikely to suffer from CH. The screening questionnaire was validated previously in 31 CH patients, 29 migraine patients and four tension-type headache patients at our outpatient clinic and was found to have a sensitivity of 100%, a specificity of 58%, a positive predictive value (PPV) of 69% and a negative predictive value (NPV) of 100% (unpublished data). Because we have an open-access website, we decided that if the screening conditions were not met, the data of the subjects were not to be stored in our database. This prevents storage of junk information of subjects from whom we could not obtain informed consent, and who could not be used for validation of the questionnaire. If the screening conditions were met, patients were registered in our system and received an email in which they were asked to complete the LUCA questionnaire.
The LUCA questionnaire was based on the ICHD-II criteria (2) and consisted of 142 items on CH and more specific and detailed questions concerning CH symptoms, demographical information, co-existing headaches and treatments of CH. The answers consisted of categorical alternatives.
Upon completion of the LUCA questionnaire, a second algorithm based on the ICHD-II criteria was run automatically to determine the individual diagnoses (not shown). The following criteria, all to be fulfilled for receiving CH diagnoses, were used for the algorithm: severe pain, unilateral pain, temporal or orbital pain, presence of one or more autonomic symptoms or restlessness, occurrence of at least five CH attacks in the past, untreated attack duration between 15 minutes and three hours and a headache attack frequency of at least one on every other day to a maximum of eight times a day. The outcome categories of the algorithm were “cluster headache” (all items fulfilled) and “no cluster headache” (not all items fulfilled).
We also used a more lenient algorithm; criterion “untreated attack duration between 15 minutes and three hours” was changed to “treated or untreated attack duration between 15 minutes and three hours.”
Phase II: Validation of the LUCA questionnaire
Semi-structured telephone interview
Within two months after completion of the LUCA questionnaire, enrolled participants were approached for a semi-structured telephone interview to diagnose CH according to the ICHD-II criteria (2). The interview diagnosis was used as the gold standard. Interviews were performed by a medical student (CC) trained in diagnosing CH under supervision of the study physicians (LAW and CMW). The interviewer and supervisors were blinded for the automatically calculated algorithm diagnosis. If participants were not reached after two attempts, they were excluded from the validation procedure. Final diagnoses were made directly after the interview by CC and LAW/CMW. In case of ambiguous symptoms or when the diagnoses determined by CC and one of the study physicians did not correspond, a headache specialist (JH), also blinded for the calculated diagnoses, was consulted, and a final diagnosis was made.
Statistical analysis of the LUCA questionnaire
All data analyses were performed using SPSS 17 (SPSS, IBM, US). Baseline characteristics of participants who were successfully interviewed and participants who were not reached were compared using two-sided χ2 tests for categorical data and Student's t tests for continuous variables. Alpha was set to 0.05. Algorithm and interview diagnoses were compared to assess the sensitivity, specificity, the (prevalence dependent) PPV and NPV, respectively, and the positive and negative likelihood ratios for the entire LUCA questionnaire.
Phase III: Toward the short QATCH questionnaire
Development of a prediction rule
In this phase of the study, we aimed to identify a subset of questions from the extended LUCA questionnaire to construct a shorter questionnaire (named Quick Ascertainment of Cluster Headache, abbreviated as QATCH) that predicts the CH diagnoses in our participants equally well or even better. To select these questions, we assessed the contribution of the individual 142 LUCA items to the CH diagnoses by calculating sensitivity, specificity, PPV and NPV, and positive and negative likelihood ratios, and selected questions based on PPVs of more than 0.90 and/or a positive likelihood ratio of more than 1.5.
To develop a prediction rule, the successfully interviewed study population was randomly divided into a training set (80% of participants) and a validation set (20% of participants) to construct and validate the generated regression model. The selected LUCA items were incorporated in a forward logistic regression model on the 80% sample to assess their contribution to discriminating CH patients from non-CH patients.
Validation of the prediction rule
The questions that contributed significantly to the CH diagnoses in the forward regression model were incorporated in the prediction rule. The regression coefficients of these items were used to assign weights to the questions and to calculate the predicted probabilities for a CH diagnosis based on this model. Subsequently, a receiver operating characteristics (ROC) curve was computed for determining the optimal cut-off value for a CH diagnosis based on our prediction rule in the 80% group according to the method of Halpern et al. (17).
The area under the curve (AUC) was assessed as a measure of correlation between the prediction of the short questionnaire and the gold standard (interview) diagnosis. Finally, interview diagnoses were compared with diagnoses obtained by our new model to determine sensitivity and specificity of our newly generated short questionnaire. In addition, we assessed the performance of this test in our population by means of the PPV and NPV in the 20% validation sample.
Results
Phase I: Enrollment of patients
Baseline characteristics of total study population and separate study samples.
CH: cluster headache; ap = 0.018 (t test); bp = 0.046 (t test
Phase II: Validation of LUCA questionnaire
Telephone interview
From these 437 participants, a total of 146 (33%) were excluded in the analysis because they could not be reached by telephone after at least two attempts. Thus 291 participants (67%) were interviewed. In total 83.5% of subjects (243/291) fulfilled ICHD-II criteria in the telephone interview.
Baseline characteristics of 291 included patients were compared to 146 excluded patients. Interviewed subjects were significantly older (p = 0.018), had longer duration of CH (p = 0.046) and fewer years of education (p = 0.0049) but absolute differences were small. There were no significant differences with respect to gender, proportions of patients using anti-CH medication (prophylactic and acute), the proportion of patients with a physician diagnosis of CH, or algorithm diagnosis (Table 1).
Statistical analysis of the LUCA questionnaire
Sensitivity, specificity, positive and negative predictive values as well as the corresponding likelihood ratios for diagnosis of cluster headache based on: 1) the ICHD-II-based algorithm; 2) the ICHD-II-based lenient algorithm, including patients with headache duration of 15–180 minutes upon treatment.
ICHD: International Classification of Headache Disorders: 2nd edition; PPV; positive predictive value; NPV: negative predictive value; Gold standard: direct interview.
With the use of the more lenient algorithm, which also includes patients with headache duration between 15 to 180 minutes upon treatment, the sensitivity increased to 70.4%, and the NPV increased to 67.6% with only a slightly decrease of the PPV to 92.5% (Table 2).
Phase III: Toward a short CH questionnaire
Development of a prediction rule
Nine variables from our LUCA questionnaire met our selection criteria for PPV of more than 0.90 and/or a positive likelihood ratio of more than 1.5. These nine variables were: untreated attack duration between 15 minutes to three hours, average attack-free period of four months to three years, pain on top of the eyeball, unilateral miosis, moderate to good response to oxygen, good response to sumatriptan, male sex, and smoking. Of the nine variables incorporated in our regression analysis, three contributed significantly to the model (QATCH questionnaire): i) untreated attack duration between 15 minutes and three hours (p < 0.001); ii) pain-free period between four months and three years (p = 0.007); and iii) male gender (p = 0.053). The regression coefficients were rounded off and used to assign points to the three items. Predicted probabilities for CH were calculated for the validation set using the regression coefficients of the model.
Validation of the prediction rule
Short diagnostic cluster headache questionnaire: QATCH.
QATCH: Quick Ascertainment of Cluster Headache; Cluster headache is diagnosed with a score ≥1.5 (after fulfilling the screening questionnaire). Scores are regression coefficients of the model. P values can be found in the text in the Results section.
Discussion
The main objective of this study was to validate a web-based questionnaire to diagnose CH patients for future large-scale epidemiologic and genetic studies. The LUCA questionnaire proved to be a valid and reliable method for diagnosing CH in a population of self-reported CH patients. Our second objective was to construct a shorter questionnaire to diagnose CH in large samples. The QATCH questionnaire represents a practical alternative for diagnosing CH patients as the combination of the screening and QATCH questionnaire works as well as combining the screening questionnaire with the much longer LUCA questionnaire. The QATCH questionnaire indicates that males with untreated headache attacks lasting 15–180 minutes and attack-free periods of four months to three years are very likely to have CH under the condition that the including criteria are fulfilled. This does not imply that females or chronic CH patients are excluded by using this questionnaire, as the questions are weighted and not all three items are obligatory to receive the diagnosis of CH. Male gender alone is not enough to receive the diagnosis of CH. In fact, untreated attack duration of 15 minutes to three hours or attack-free periods of four months to three years are enough to receive the diagnosis of CH in a pre-screened population for research purposes.
Regression analysis indicated that these three questions about gender, untreated attack duration and duration of attack-free periods are good predictors of CH diagnosis in our training set (80% of study population) and that the resulting QATCH questionnaire performed equally well in the validation set (20% of study population). A similar approach was successfully applied to select items for migraine questionnaires by our group as well as by others (14,18). However, it is important to keep in mind that the QATCH questionnaire was developed in a population of self-reported CH patients with enriched CH prevalence by application of a screening questionnaire, resulting in a study population of only 48 non-CH subjects. Therefore, the screening questionnaire should be included as a primary step, and the results cannot be generalized to application in the general population. The screening questionnaire was based on the validated migraine screening questionnaire and adapted for CH (15). Its sole purpose is exclusion of self-reported CH patients that were highly unlikely to have CH, and it was validated in our outpatient headache clinic. A drawback is that the QATCH questionnaire is not validated without the use of a screening questionnaire as a first step. It is uncertain if our ultra-short three-item QATCH questionnaire performs equally well without application of the screening questionnaire, which has 10 items and a more complicated algorithm. Another limitation could be the length of our LUCA questionnaire, which could have led to a low response. Despite the 20 minutes that it takes to fill in the questionnaire, we have a response rate of approximately 90%. In our opinion the length of our LUCA questionnaire would not be a major drawback.
The LUCA questionnaire was designed to diagnose CH fulfilling all ICHD-II criteria (2). As a consequence, the questionnaire had high specificity but low sensitivity. This is mostly explained by the observation that many patients use medication to treat their attacks, resulting in unknown untreated attack duration, which makes it hard to establish a diagnosis that is in full agreement with ICHD-II criteria as these require knowledge of untreated attack duration. Fifteen percent of participants responded to the question about attack duration without medication “do not know,” of whom 80% answered they did not know because they always use medication. This leads to an underestimation of the number of CH patients diagnosed by our LUCA questionnaire, resulting in a relatively low sensitivity. Because of this shortcoming we also calculated sensitivity, specificity, PPV and NPV for the questionnaire using a more lenient algorithm. When altering the necessary item “duration of attack without medication use between 15 minutes and three hours” to “duration of attack without or with medication use between 15 minutes and three hours,” the sensitivity increased to ∼70% with the PPV remaining as high as ∼92%. For genetic studies in CH, both algorithms have advantages and disadvantages. The strict algorithm reduces the number of false-positives in this study, which is important for identifying genetic variants with a small effect on the disease. However, using the strict algorithm also means that many CH patients will be excluded because of the low sensitivity. The lenient algorithm could be an attractive alternative because it has a much higher sensitivity and for GWAS recruitment of sufficient numbers of patients is a concern in rare disorders such as CH.
The strength of our study consists of the large sample size in comparison with other questionnaire studies performed in CH that included only up to 30 CH patients and that did not validate their findings on a randomly selected validation sample as we did (10–12). Secondly, the web-based step-wise procedure with a screening questionnaire as the first step prevents the collection of junk information and non-CH headache patients in a very easy and semi-automated way. Thirdly, the application of a semi-structured telephone interview as a gold standard ensured precise categorization of CH patients. Our LUCA questionnaire focused on the detection of self-reported CH patients from the general adult population, but did not aim to diagnose CH in the naïve general population. Further validation studies are needed to assess the generalizability of this model (screening questionnaire plus the QATCH) or using the QATCH on its own in other study designs.
The LUCA questionnaire proved to be a valid and reliable method for diagnosing CH in a population of self-reported CH patients who filled out a screening questionnaire. Being a male and suffering from headache attacks of 15 minutes to three hours with pain-free periods (of four months to three years) are good predictors for a valid diagnosis of CH according to our QATCH questionnaire. Our web-based, step-wise procedure is an easy and semi-automated way to easily collect large numbers of CH patients for genetic-epidemiologic studies.
Clinical implications
Our web-based, step-wise procedure is an easy and semi-automated way to easily collect large numbers of CH patients for genetic-epidemiologic studies. Being a male and suffering from headache attacks of 15 minutes to three hours with pain-free periods (of four months to three years) are good predictors for a valid diagnosis of CH according to our QATCH questionnaire.
Footnotes
Acknowledgments
We thank Drs AH Stam and WPJ van Oosterhout for critically reviewing the manuscript.
Funding
This work was supported by grants from the Netherlands Organization for Scientific Research (NWO) (MDF: 903-52-291, Vici 918.56.602; GMT: 907-00-217, Vidi 917-11-319) and by a grant from the Centre for Medical Systems Biology (CMSB) established by the Netherlands Genomic Initiative/Netherlands Organization for Scientific Research (NGI/NWO).
Conflicts of interest
MDF has, in the past three years, received grants and consultancy/industry support from Almirall, Coherex, Colucid, Eisai, GlaxoSmithKline, Linde, MAP, Medtronic, Menarini, Merck, Minster, Pfizer, and St Jude, and independent support from the Netherlands Organization for Scientific Research (NWO). GMT has received grants and consultancy/industry support from Merck, Janssen-Cilag, Menarini, and independent support from NWO (Clinical Fellowship 90700217 GMT and Vidi 91711319 GMT). JH has received consultancy support from Merck. LAW has received industry support from Medtronic, Menarini, Allergan and independent support from Fonds NutsOhra. The other authors have nothing to declare.