Abstract
We have assessed the validity and reliability of a self-administered headache questionnaire used in the 'Nord-Trøndelag Health Survey 1995-97 (HUNT)' in Norway, by blindly comparing questionnaire-based headache diagnoses with those made in a clinical interview of a sample of the participants. Restrictive questionnaire-based diagnostic criteria for migraine, assessed according to modified criteria of the International Headache Society, performed excellently in selecting 'definite' migraine patients (100% positive predictive value). The best agreement concerning migraine diagnoses was achieved by using a liberal set of criteria (k 0.59). Similar agreement was found evaluating patient status as headache sufferers, and as sufferers from frequent headaches (>6 days per month) (k 0.57 and 0.50, respectively). The k values of nonmigrainous headache and chronic headache (>14 days per month) were 0.43 and 0.44, respectively. The results suggest that our self-administered questionnaire may be suitable in identifying a population with 'definite' migraine, and the questionnaire is an acceptable instrument in determining the prevalence in Nord-Trøndelag of headache sufferers, migraine, non-migrainous headache, and frequent or chronic headache sufferers.
Introduction
A self-administered questionnaire is an attractive instrument for epidemiological studies, being inexpensive and relatively easy to use. However, even after the introduction of diagnostic criteria for all headache disorders in 1988 (1), questionnaire studies must be evaluated with caution. The validity and reliability of such questionnaires have been assessed by comparing questionnaire-based diagnoses with the diagnoses made in a clinical interview (e.g. (2–11)). These studies have shown variable sensitivity, specificity, predictive values, and agreement rates. If the objective of a questionnaire study is to estimate the prevalence of different headache disorders, the agreement between questionnaire diagnoses and a ‘gold standard’ should be high. If the goal is to identify subjects with a definite diagnosis of migraine, the positive predictive value of the questionnaire-based diagnosis of migraine should be high.
In order to evaluate a questionnaire used in a large population-based study of headache disorders in Nord-Trøndelag County in Norway, telephone interviews and clinical interviews were performed on population samples. The objective of the present study was to assess the validity and reliability of the questionnaire-based diagnosis of migraine. In addition, we attempted to estimate the reliability of questionnaire information concerning being a headache sufferer and concerning the reported frequency of headache.
Materials and methods
Study populations
All inhabitants 20 years and older in Nord-Trøndelag county of Norway were invited to participate in theNord-Trøndelag Health Survey between 1995 and 1997 (‘Helseundersøkelsen i Nord-Trøndelag’= HUNT). Nord-Trøndelag is one of 19 Norwegian counties, located in the central part of Norway. The habitation is scattered, and no city has more than 21 000 residents. The Nord-Trøndelag Health Survey was carried out by the National Health Screening Service in co-operation with the National Institute of Public Health, Unit for Health Services Research, Nord-Trøndelag county, and the Norwegian University of Science and Technology (NTNU).
In Fig. 1, the population samples that participated in the different parts of the study are shown. Out of 92 566 invited individuals, 64 560 (70%) answered the first questionnaire (Q1) that was enclosed with the invitation letter, and participated in a brief medical examination, at which they were given a second questionnaire (Q2) to be filled in and returned from home. The two questionnaires included more than 200 health-related questions, and the Q2 included 13 headache questions (Table 1), mainly designed to determine whether the person suffered from headache or not, and whether a possible headache fulfilled the migraine criteria of the International Headache Society (IHS) (1). A total of 51 383 subjects (56% of the total adult population) completed the headache questions in Q2, and these constituted the ‘head-HUNT’ study.

Diagram of the population according to type of participation.
Headache questions in the HUNT questionnaire
In order to reduce the time span between headache questionnaire and a clinical interview, a validation study in two phases was performed before the data from the headache questions were available. First a random sample of those who completed the Q2 were given a five-question interview by telephone concerning headache during the last year (whether the person suffered from headache or not, and intensity/frequency of headache). The questions were not designed to give information on migraine diagnosis. Among those interviewed by telephone, a sample of those with no headache (n = 25) and a sample with annoying headache of moderate or severe intensity (n = 147) were invited to the second phase consisting of a clinical interview. In order to obtain a sufficient number of subjects without headache, 642 subjects were interviewed altogether. The clinical interviews were semistructured, lasted about 30 min, and were performed by physicians with several years of experience in clinical management of headache disorders. The interview included 52 questions concerning headache complaints, focusing on headache onset, temporal pattern, localization, pain characteristics, accompanying features, triggering factors, medication and other treatment. The series of questions was followed by additional items if relevant to the individual case. The interviewers were blinded as to the results of the self-administered questionnaire and the telephone interview. Patients with a problematic headache diagnosis were discussed among the physicians before a final diagnosis was made.
The self-administered headache questionnaire (Q2) was filled in 4–8 months prior to the telephone interview. Between the telephone interview and the clinical interview, 2–4 weeks elapsed.
The regional committee for ethics in medical research, and the Norwegian Data inspectorate approved the study.
Headache classification
Based on the information from the clinical interview, headache disorders were classified in accordance with the criteria of the IHS (1). Based on information in Q2, the diagnoses of migraine with or without aura were made according to three different sets of criteria (1, most restrictive; 2, medium restrictive; and 3, liberal criteria) (Table 2). Subjects reporting ‘Yes, another type of headache’ (not migraine) in the questionnaire were classified as ‘non-migraineurs’ (other headaches) if they did not fulfil the medium restrictive set of migraine criteria.
Different sets of criteria for migraine diagnosis based on information in the questionnaire
Migraine with aura: identical to the criteria for migraine without aura, except:
1. Headache attacks lasting not more than 72 h, and
2. Often visual disturbances prior to headache.
In the questionnaire, a frequent headache sufferer was defined as an individual reporting more than six headache days per month during the last year. In the clinical interview, these individuals confirmed their status as frequent headache sufferers if they reported a mean of six or more headache days per month within the last year. Similarly, chronic headache was defined as headache more than 14 days per month.
Statistical analysis
Sensitivity, specificity, predictive values, and Cohen's kappa (= κ) statistics with 95% confidence interval (CI) were calculated for the diagnosis of migraine. κ statistics was used to compare answers given in Q2, the clinical interview and in the telephone interview with respect to headache suffering, non-migrainous headache, and frequent headache. κ is the observed agreement rate corrected for chance agreement (12).
Results
Among the 172 subjects who participated in the clinical interview, five subjects had not answered Q2, leaving 167 respondents available for comparison of information from Q2 with that of the clinical interview. In the clinical interview, 151 out of the 172 (88%) reported having had headache within the last year. Forty-four subjects out of 151 (29%) had two different headache diagnoses.
Headache suffering
The chance-corrected agreement (κ value) between information obtained in the questionnaire, the clinical interview, and the telephone interview, is given in Table 3. The agreement (κ) between Q2 and the clinical interview concerning suffering from headache or not within the last year, was 0.57 (95% CI 0.41–0.73). Headache during the last year reported in the Q2 was confirmed by 117 out of 122 subjects in the clinical interview. Among the 45 subjects who reported not to suffer from headache in the Q2, 20 (44%) reported some headache in the interview, but only four (9%) had migraine.
Change-corrected agreement (κ value) between two information methods (number of subjects is given)
The agreement between the questionnaire and the telephone interview was only 0.26 (95% CI 0.18–0.34).
In the questionnaire, 34 (21%) of 165 clinically interviewed subjects reported headache for > 6 days per month. Among the 34, 23 confirmed frequent headache (> 6 days/month) in the clinical interview (κ= 0.50, 95% CI 0.34–0.66). For chronic headache (> 14 days/month), the κ value was 0.44 (Table 3).
Migraine
In the clinical interview, 75 subjects (44%) were classified as having migraine (51 without and 24 with aura). In addition, 10 subjects (6%) fulfilled all migraine criteria but one (IHS diagnosis 1.7). In Table 4 the agreement between the migraine diagnosis made in the clinical interview and the Q2 is given.
Predictive values (PV) and κ values of questionnaire-based diagnosis of migraine. The interview-based diagnosis is used as the gold standard (n = 167, whereof 75 got a diagnosis of migraine)
The total number of subjects with questionnaire-based migraine diagnosis who fulfilled the International Headache Society (HIS) criteria for migraine in the clinical interview.
Total number of those self-reported migraine in the questionnaire, irrespective of whether they fulfilled the restrictive or medium questionnaire-based criteria.
A higher proportion got the migraine diagnosis by the clinical interview (75/167) than by the questionnaire, even using the liberal set of criteria (62/167). Among 75 persons classified as having migraine in the interview, 52 were diagnosed applying the liberal criteria to the questionnaire data.
By using the most restrictive questionnaire-based criteria for migraine, the positive predictive value was 100% (Table 4). Using the liberal set of criteria, positive predictive value was 84% and the negative predictive value was 78%. The chance-corrected agreement was 0.59 (Table 4).
In the questionnaire, subjects were also asked directly whether they thought they had migraine or not. Compared with the interview diagnosis, the positive predictive value was high (89%), although κ was moderate (0.43). The latter was due to many migraine sufferers without self-reported migraine (false negative) (Table 4).
A total of 20 subjects reported visual disturbance before headache in the clinical interview. In the Q2, 10 subjects reported that they often had visual disturbances prior to headache, and they were all diagnosed as suffering from migraine with aura in the interview (100% positive predictive value). The sensitivity to the diagnosis migraine with visual aura in the Q2 however, was only 50%.
Non-migraineurs (other headaches)
Sixty subjects out of 167 reporting headache in the Q2 did not fulfil the liberal set of criteria for migraine, and these were classified as non-migraineurs. In the interview, 41 subjects (68%) were classified as non-migraineurs (κ= 0.43, Table 3). Thirty-three of these 41 (80%) were diagnosed as having tension-type headache (24 episodic and nine chronic tension-type headache).
Discussion
Traditionally, headache questionnaires have been validated by comparing responses to a self-administered questionnaire with a clinical examination used as a gold standard. However, direct comparison of validation studies is difficult because of differences in time delay between the investigations, sampling methods, and in diagnostic criteria.
The level of agreement between two types of diagnostic instruments depends on differences between the instruments, and on different patient response to the same question at different times (13). By interviewing patients three times in 2 years, Stang et al. found that some of the key diagnostic features of headache were not consistently reported or experienced over time (14). In the present study, the self-administered headache questionnaire was filled in 5–9 months prior to the semistructured interview. In assessing the accuracy of the questionnaire as a diagnostic tool, it is therefore important to keep in mind that the time interval between questionnaire and interview may reduce the agreement between responses.
We recruited subjects for the clinical interview from the population who participated in the questionnaire survey. In order to interview a sufficient number of individuals with different types of headache (i.e. migraine and frequent headache sufferers), we enriched the group with persons with moderate to severe headache. Almost all such individuals agreed to participate in the clinical interview. Compared with all headache sufferers who participated in Head-HUNT, the group with moderate to severe headache had similar distributions of gender and age. Thus, this group is a representative sample of participants in the Head-HUNT with moderate to severe headache. Among those with no headache invited to the clinical interview, the proportion of men was higher than among participants without headache in Head-HUNT. We did not invite subjects with mild headache, and therefore the interview sample did not cover the whole spectrum of headache. Thus, our sampling method may result in a slight overestimation of agreement between methods, and the positive predictive value for questionnaire information may be overestimated.
Headache suffering
The agreement between the questionnaire and clinical interview on suffering from headache or not, was moderate (κ 0.57). According to Rasmussen et al. (2), a κ value of at least 0.5–0.6 is usually considered to represent the minimal acceptable agreement between methods. As previously mentioned, the selection of subjects to the clinical interview may have contributed to increase the κ value. This is demonstrated by the fact that the agreement between the questionnaire and the telephone interview (random sample) was only fair (0.26). On the other hand, the telephone interviews were performed only 2–4 weeks before the clinical interview, and the agreement between them was very good (0.79). However, this short interval may introduce a recall bias, which may give a slight overestimation of the agreement, because at the time of clinical interview many of the participants may have remembered their responses in the previous telephone interview.
The group that reported no headache within the last year in the questionnaire should be evaluated with caution. Among the 45 who did not suffer from headache in the questionnaire, 44% reported having an annoying headache in the clinical interview. This is in accordance with a Danish study (2), in which 51% of those who had reported no headache in a questionnaire reported headache in the clinical interview. Similarly, Wittrock et al. reported that not all members of a headache-free control group really are headache-free (15). Thus, the proportion of headache sufferers in Nord-Trøndelag may be underestimated based on the questionnaire answers.
The agreement between questionnaire and clinical interview concerning frequent headache was moderate (κ 0.50). Some disagreement may be due to subjects having developed or being relieved from frequent headache during the interval between answering the Q2 and the clinical interview. Unfortunately, only a few studies have evaluated the incidence and spontaneous relief from frequent headache. In a small prospective cohort study in Lithuania, 193 controls answered a headache questionnaire twice with a 1-year interval (16). Of the 13 subjects who reported frequent headache (> 7 days/month) after 1 year, six did not report frequent headache in the first questionnaire. In our study, the mean time span between investigations was more than half a year. If we assume that the time span could explain the discrepancy for four subjects who did not report frequent headache in the questionnaire and four subjects who did, the calculated κ would increase from 0.50 to 0.64.
Migraine
We attempted to be as close as possible to the IHS definition in constructing criteria for migraine based on questionnaire data, but we had to make some adaptations. In the two restrictive sets of criteria, the presence of at least one of the following pain characteristics was required: unilateral location, pulsating quality, or aggravation by physical activity. Unfortunately, the questionnaire had no question about pain intensity. Thus, by using the two restrictive sets of criteria (Table 2), we were likely to include subjects only fulfilling the IHS criteria 1.7 (migrainous disorder) as well as subjects fulfilling IHS criteria for migraine.
For migraine without aura, the presence of photophobia and phonophobia are required in the IHS criteria. However, we accepted phonophobia and/or photo-phobia in the questionnaire-based migraine criteria.
The pain phase in migraine with aura is often shorter than in migraine without aura (17). Therefore, we accepted a diagnosis of migraine with aura in subjects who reported ‘often visual disturbance prior to headache’ if they otherwise fulfilled the criteria for migraine without aura, even if they had attacks that lasted for less than 4 h. The group of migraineurs diagnosed according to the liberal criteria included those with self-reported migraine, and nearly all of them fulfilled the diagnosis in the interview (positive predictive value 89%). In two Danish studies, the agreement between self-reported migraine in a questionnaire and migraine diagnosis in a clinical interview was very high (κ= 0.81 and 0.77, respectively) (2, 6). The moderate κ value (0.43) in the present study was mainly due to many migraine sufferers without self-reported migraine (false negative). This may in part be explained by the fact that in Nord-Trøndelag, a relatively small proportion of these patients had previously visited a doctor for their migraine. In Denmark, 56% of the migraine sufferers had consulted their general practitioner at some time because of migraine (18).
In previous studies, comparisons of questionnaire-based migraine with migraine diagnosed in a clinical interview have shown variable predictive values and agreement rates (Table 5). Rasmussen et al. (2) expressed scepticism towards using a self-administered questionnaire as an instrument for epidemiological headache research. Later, several other studies reported much better agreement (6, 9, 11). However, the agreement may tend to be overestimated in validation studies based on interview of a clinical sample instead of using a random sample of the population (e.g. (4, 7)).
Predictive values (PV) and agreement (κ value) between questionnaire-based diagnoses and clinical interview-based diagnoses of migraine in various validation studies
Clinical interview of a random sample of the study population (general population).
Validation study of a clinical sample (not population-based).
Telephone interview compared with clinical interview of a subgroup of headache sufferers.
590 subjects with self-reported migraine and 137 responders who did not have self-reported migraine invited to a semistructured interview, whereof 40% were interviewed by telephone.
Previously, Rasmussen et al. concluded that a main problem in diagnosing headache by a questionnaire was the frequent coexistence of different types of headache, that the respondents could not distinguish (2). In this study however, we found comparable agreement in respondents who only had migraine with that of respondents who had migraine and another coexisting headache. Possibly, most respondents focused on the most annoying headache (migraine) in the questionnaire. The risk of losing some patients with true migraine may be highest among individuals with very few migraine attacks and more frequent non-migrainous headache. Because the participants had to indicate whether their headache characteristics occurred ‘often’, ‘sometimes’ or ‘seldom/never’, typical migraine features may be reported to occur only ‘seldom/never’. Thus, the design of the questionnaire may lead to a slight underestimation of the migraine prevalence.
When we included self-reported migraine in the liberal migraine criteria, the influence of missing values in the questionnaire was eliminated, and the κ value was better (0.59). Thus, we conclude that the liberal criteria are the most suitable in order to estimate the prevalence of migraine in Nord-Trøndelag.
By applying the most restrictive criteria to the questionnaire, the positive predictive value was 100%. This value may be overestimated, due to enrichment of migraine sufferers in the clinical interview. However, these criteria may be utilized to identify a subpopulation with ‘definite’ migraine.
Since the sensitivity of the questionnaire-based diagnosis of migraine with visual aura was only 50%, the true prevalence of this condition is probably underestimated. However, the positive predictive value was 100%, which may allow us to define a pure group of patients with migraine with aura.
Non-migraineurs (other headaches)
The agreement between the questionnaire and clinical interview concerning the category non-migrainous headache was moderate (κ 0.43). In the questionnaire-based group of 60 non-migraineurs, 19 (32%) were diagnosed as having migraine in the interview. One reason for the moderate κ value was the 16 subjects who reported as not suffering from headache in the questionnaire, but reported as having a non-migrainous headache in the interview. Thus, based on information from the questionnaire, the prevalence of non-migrainous headache will probably be underestimated.
Conclusion
Our self-administered questionnaire may be a useful tool in identifying a population with ‘definite’ migraine as defined by the most restrictive criteria. This will be useful in evaluating factors suspected to be associated with migraine. The liberal criteria are the most suitable in order to estimate the prevalence of migraine in Nord-Trøndelag. Moreover, the questionnaire is an acceptable instrument in determining the prevalence in Nord-Trøndelag of headache sufferers, non-migrainous headache, frequent headache sufferers (headache > 6 days per month), and chronic headache (headache > 14 days per month).
Footnotes
Acknowledgements
This study was partially financed by a scholarship to KH from the Research Council of Norway, and partly by Glaxo-Wellcome, Norway.
