Abstract
Low utilization of hearing aids has drawn increased attention to the study of consumer preferences using both simple ratings (e.g., Likert scale) and conjoint analyses, but these two approaches often produce inconsistent results. The study aims to directly compare Likert scales and conjoint analysis in identifying important attributes associated with hearing aids among those with hearing loss. Seven attributes of hearing aids were identified through qualitative research: performance in quiet settings, comfort, feedback, frequency of battery replacement, purchase price, water and sweat resistance, and performance in noisy settings. The preferences of 75 outpatients with hearing loss were measured with both a 5-point Likert scale and with 8 paired-comparison conjoint tasks (the latter being analyzed using OLS [ordinary least squares] and logistic regression). Results were compared by examining implied willingness-to-pay and Pearson’s Rho. A total of 56 respondents (75%) provided complete responses. Two thirds of respondents were male, most had sensorineural hearing loss, and most were older than 50; 44% of respondents had never used a hearing aid. Both methods identified improved performance in noisy settings as the most valued attribute. Respondents were twice as likely to buy a hearing aid with better functionality in noisy environments (p < .001), and willingness to pay for this attribute ranged from US$2674 on the Likert to US$9000 in the conjoint analysis. The authors find a high level of concordance between the methods—a result that is in stark contrast with previous research. The authors conclude that their result stems from constraining the levels on the Likert scale.
Background
About 16% of Americans suffer some kind of hearing loss (Agrawal, Platz, & Niparko, 2008). Hearing impairment affects an individual’s social, physical, and economic wellbeing (Green & Pope, 2001). Evidence shows that hearing aids, when properly fitted and consistently utilized, can increase a person’s quality of life (Chia et al., 2007; Mulrow et al., 1990) in a cost-effective manner (Gates & Mills, 2005; McPherson & Wong, 2005). However, hearing aids remain underpurchased and underutilized (Chia et al., 2007; Kochkin, 2007; Meister, Lausberg, Kiessling, von Wedel, & Walger, 2002a; Popelka et al., 1998).
A simple explanation of the low rate of purchase is the cost, as most public and private insurers do not reimburse the purchase cost of hearing aids (Abrams, Chisolm, & McArdle, 2002; Gandel, 2005; National Institute for Occupational Safety and Health, 2007). However, purchase cost is not the complete story. It does not explain the underutilization of hearing aids already purchased, nor does cost explain the equally low utilization of hearing aids in countries where public funds are used to pay for them (Smith, Mitchell, Wang, & Leeder, 2005).
Previous studies on underutilization have focused on how well the aid improves hearing (Gates & Mills, 2005; U.S. Congress, Office of Technology Assessment, 1986) as well as on process issues such as ease of use and stigma (Cohen-Mansfield & Taylor, 2004). Kochkin (2007) stresses multiple stages through which patients must pass before seeking care, including denial, stigma, understanding of the possible solutions, and cost.
The lackluster demand for hearing aids has drawn increased attention toward the consumer’s perspective (Meister et al., 2002b; Meister, Lausberg, Walger, & Von Wedel, 2001). Consumer preference studies can be beneficial in identifying important outcomes and the tradeoffs among them. Preference studies can also measure consumers’ willingness to pay for improvements. Stated-preference methods can be a simple Likert scale, which asks respondents to rate particular characteristics of an aid on a scale of 1 to 5. Another, more complicated method, called conjoint analysis, forces respondents to trade one attribute for another allowing researchers to determine an attribute’s relative importance. Both methods have been used to assess patient preferences related to hearing aids and hearing loss. Likert scales can provide a simple method for evaluating consumers’ preferences for a variety of dimensions associated with hearing loss and hearing aid utilization. For example, Kochkin (2007) utilizes a 7-point Likert scale in assessing a variety of dimensions of satisfaction. Other examples of Likert or ratings scales include Bagatto et al. (Bagatto, Moodie, Seewald, Bartlett, & Scollie, 2011), Arehart et al. (Arehart, Kates, & Anderson, 2010), and Moeller et al. (Moeller, Hoover, Peterson, & Stelmachowicz, 2009). Conjoint analysis utilizes pairwise comparisons to estimate preferences (Thurstone, 1929). The method has demonstrated validity in understanding preferences in the fitting of hearing aids, including that of Meister et al. (2001) and Fitzpatrick et al. (2007).
Most studies have found that the two methods can reveal different valuations. Likert scales are supposed to value the space between each choice equally, so that the difference between a 1 and a 2 means the same thing as the difference between a 4 and a 5. Johnson et al. (2006) examined whether chemotherapy patients rated their quality of life differently on a Likert scale versus a conjoint survey. The authors found that patients’ valuations in the conjoint analysis did not correspond with an equally spaced, linear Likert scale. This indicated that the Likert was over- or under-estimating the importance of improvements in perceived health status. Ryan et al. (2001) conducted a systematic review of patient preference elicitation methods and found conjoint analysis to be one of the more sensitive methods. According to the authors, Likert scales demonstrated limitations in accounting for strength of preferences but were useful for eliciting patient attitudes. In one further study, Phillips et al. (Phillips, Johnson, & Maddala, 2002) found similarities in the way the Likert scales and the conjoint analysis ranked attributes but that there were some inconsistencies between the two methods.
The purpose of this study is to compare the results of Likert scale and conjoint analysis methods for measuring consumers’ valuations of hearing aid attributes. The results of this study have important substantive and methodological implications for those interested in the consumer’s perspective on hearing aids and more broadly as researchers aim to conduct consumer- and patient-centered outcomes research.
Method
Whereas other, often more complex protocols for stated-preference research have been outlined (Bridges, 2003, 2011; Louviere, Hensher, & Swait, 2000), our study design follows a five-stage approach outlined by Ryan and Farrar (2000) involving (a) identifying attributes, (b) assigning levels, (c) formulating scenarios, (d) establishing preferences, and (e) data analysis. Given our aim to measure preferences with both traditional Likert scales and conjoint analysis, we modified the protocol to accommodate both approaches.
Identifying Attributes
One of the most important stages of any preference elicitation study is the identification of attributes, a process that is often guided by extensive qualitative analysis (Coast & Horrocks, 2007; Kinter, Schmeding, Rudolph, dosReis, & Bridges, 2009). We began this process with in-depth, open-ended interviews of a sample of outpatients with hearing loss recruited at the Johns Hopkins Hospital, Baltimore, Maryland. This qualitative approach has been used in other preference studies and is similar to other qualitative methods that have been employed to study hearing loss and hearing aids (Hétu, Riverin, Lalande, Getty, & St-Cyr, 1998; Lockey, Jennings, & Shaw, 2010).
Trained fieldworkers conducted the semistructured interviews and encouraged participants to discuss their feelings and experiences with hearing loss and/or hearing aids. A series of questions provided structure across interviews, while leaving room for respondents to address their individual concerns with their aids. Interviews were recorded, transcribed, and analyzed using interpretive phenomenological analysis (Smith, Flowers, & Larkin, 2009).
Attribute Selection
From the patient interviews, Table 1 presents the final seven attributes chosen for analysis, the description used in the survey instrument, and a number of quotes that illustrate how the respondents discussed the factors. From these interviews, four major themes developed. The first theme is the overall hearing aid performance. Respondents characterized this attribute as how well the hearing aid assisted them in various listening environments. Performance was also reflected in the hearing aid user’s description of the aid’s sound quality. Respondents discussed hearing aid performance in the context of three separate attributes: noisy settings, quiet settings, and sound quality. The second theme focuses on the aid’s aesthetic and how that affects purchase and usage. Features of this theme include the appearance of the hearing aid, its physical comfort, and the feedback noise that may be emitted by the hearing aid under certain conditions.
Attribute/Level Descriptions and Representative Quotes
The third theme focuses on the overall cost of the aid, which includes initial purchase costs, usage costs, and the cost of follow-up care. Purchase price was defined as the monetary cost of the hearing aid along with the costs associated with the initial consultations and fittings as well as maintenance items such as batteries. The fourth theme focuses on specific features of the aid and how these features can either enhance or negatively affect an individual’s day-to-day life. Hearing aid features include the handling of the aid, multiple settings in the aid, as well as the water and sweat resistance capabilities of the hearing aid. Respondents defined “handling of aid” as the convenience and ease of operating, wearing, and maintaining the hearing aid.
During the process of refining the attribute levels, we determined that some attributes were not relevant or were too complicated to include in our analysis. Many that we examined in early piloting included the method of cleaning, having a trial period, continuity of care/follow-up care, and style. With style, we found that there was relatively little difference between on-the-ear and in-the-ear formats. There was often ambiguity as to how patients valued these factors. As one respondent put it,
The first one I had was a big piece behind my ear. When I came back the second time, I debated between this one and a smaller one. I thought, if the smaller one will work then I’ll go for that. [My audiologist] told me about this one has two speakers, whereas the smaller one had only one speaker. With two speakers, I heard better than I did with the smaller one.
We decided to exclude style from our conjoint analysis and devoted a section of the survey to exploring attitudes and experiences with different hearing aid styles (which, for brevity are not explored in detail here).
Assigning Levels
Assigning levels to attributes is an equally important aspect of conjoint analysis (Bridges et al., 2011; Ryan & Farrar, 2000). In addition to soliciting attribute levels from the qualitative data, we also rigorously piloted a draft survey instrument to further develop the levels. Four versions of the survey instrument were tested during piloting, with refinement made to the wording of the attributes and levels. To avoid additional complexity, we focused on the development of two levels for each attribute, which are also summarized in Table 1. As a means to avoid extreme levels, which could serve as anchoring points, we developed characteristics that represented a moderately above and below average level for each factor (Bridges et al., 2011).
Formulating Scenarios
Given the aim to compare the two methods, the scenario formulation required the same attributes and levels to be used in both the development of Likert scales and conjoint tasks. For the purpose of the Likert scale, we chose to place boundaries on each attribute so as to make them comparable to the levels presented in the conjoint analysis. For example, rather than simply asking respondents to rate on a scale from 1 to 5 how important feedback was to them (irrespective of the frequency) we specified ranges in which respondents could place their preferences:
Feedback is the high-pitched squealing noise that a hearing aid can make. In some hearing aids, feedback occurs 4 times per month and in others feedback occurs 12 times a month.
A similar approach was presented for all seven attributes. These were valued on a 5-point scale (see Figure 1), ranging from not important at all to very important.
An example of attribute valuation via rating on a Likert scale
Conjoint choice tasks sort the various levels of the seven attributes into different combinations, and respondents are asked to make a series of choices. Through this process of being forced to make trade-offs, researchers are able to estimate the relative importance of the attributes. To create the conjoint analysis choice tasks, we used a full-profile, paired comparison. The experimental design utilized a minimal, main-effects orthogonal array that identified eight uncorrelated scenarios (Bridges et al., 2011; Louviere et al., 2000), each paired with its complete opposite. An example of this pair comparison is presented in Figure 2.
An example of attribute valuation via choice-based conjoint analysis
The scales and conjoint tasks were presented with careful definitions on what each attribute meant and specific instructions on how to complete the Likert scale questions and the conjoint choice tasks.
Sample
Respondents with sensorineural hearing loss beyond the 30dB speech reception threshold were recruited from the Johns Hopkins Hospital, Department of Otolaryngology. Clinicians informed potential respondents of the study during a scheduled visit, referring those who agreed to participate to study staff. Respondents could either complete the survey immediately following their appointment at the outpatient center or take it home with them to complete and return to their clinician using a prepaid envelope that was provided. As compensation for their time, respondents were offered a validated parking voucher or a gift card from a national coffee chain.
Although formal sample size calculations are rare in stated-preference consumer studies (Bridges, 2003), a number of rules of thumb have been described in the literature (Bridges et al., 2011). Applying a simple rule of thumb based on the number of attributes and levels (Orme, 2009), a minimum sample size for our relatively simple experiment is 32 respondents. Alternatively, a parametric approach has been proposed by Louviere et al. (2000), implying a minimum sample size of 48 respondents. We thus aimed to have a sample size between 50 and 60 respondents. Although this is relatively small compared to the average study in health (Marshall et al., 2010), it has been successful in other stated-preference studies of comparable complexity (Akkazieva, Gulacsi, Brandtmuller, Pentekk, & Bridges, 2006).
Analysis
The preferences of the respondents for hearing aids were elicited using both the Likert scales and the conjoint analysis choice tasks. These were administered as part of a survey of respondents with hearing loss. In addition to the Likert scales and conjoint-analysis choice tasks, we asked respondents about their race, gender, income, educational levels, and several questions related to hearing aid use.
Data were analyzed with simple OLS (ordinary least squares) and logit models, with the independent variables being the attributes, and the dependent variable being the choice of hearing aid in the conjoint task or the rating in the Likert scale. Choice models can be estimated with either OLS or as logit models, since the outcome is binary—the respondent chose either Hearing Aid A or Hearing Aid B. We present both for ease of inference.
We also estimated the willingness to pay for the various attributes. WTP allows both scales to be converted to the same monetary metric to ease comparison between methods. Knowing how much someone is willing to pay for a characteristic is a way to determine the relative value of the attributes. This is calculated by dividing the coefficient on the attribute by the coefficient on the purchase cost attribute (Louviere et al., 2000). The methods were compared using the Pearson’s correlation coefficient. A statistical test such as the Pearson’s correlation coefficient will tell us whether the two methods (three analytic models) are producing results that are similar. The Johns Hopkins Institutional Review Board reviewed and approved the study.
Results
Demographics
Respondents (Table 2) tended to be white, male, older, and upper income: 66% of the respondents were male, 84% were white, and 46% were between the ages of 51 and 70; 28% had 4 years of college and nearly 43% had some kind of postgraduate degree; 48% had incomes more than US$100,000; 44% had never worn a hearing aid. Most respondents (62.5%) said they suffered from sensorineural hearing loss. Health insurance covered all or a portion of the cost of a hearing aid in almost 36% of the cases. Respondents were shown pictures of six possible types of hearing aids: in-the-ear or full shell, half-shell, in-the-canal, completely-in-the-canal, behind-the-ear, and bone-anchored. They were allowed to select more than one choice. Most respondents said they would like to wear a completely-in-the-canal or a behind-the-ear style.
Demographic Data, Percent of Respondents
Preferences
Both methods returned similar results in the orders of which attributes were most important. Figure 3 lays out the relative rankings of each characteristic and respondents’ willingness-to-pay for each attribute. Willingness-to-pay in this context is how much more a consumer is willing to pay for a given attribute. Respondents most valued the hearing aid’s ability to perform well in noisy settings. Batteries were the least important, revealed similarly through both methods. Battery life was not a significant predictor in either the logistic or the OLS regressions. Although the rating and conjoint methods ranked feedback very differently, they both had a similar willingness-to-pay value. Water or sweat proof was also valued equally at Rank 4, and cost was valued at Rank 6. Effectiveness of the hearing aids in quiet settings was ranked 5 in the rating scale and ranked 3 in the conjoint method. Comfort was also valued differently between the two.
Estimates of willingness-to-pay for attributes
The difference in cost between the two hearing aids presented to respondents was US$2,000. Table 3 displays the various willingness-to-pay values calculated from the three methods. The conjoint method values the ability for the hearing aid to work well in noisy settings at a much greater level than the rating scale. The mean on the rating scale is 3.768, indicating a ceiling effect. If respondents were given a higher value on the scale, they would likely select the higher option. This would increase the willingness-to-pay. The three methods resulted in rankings of the attributes that are highly correlated (Figure 3). The correlations for the ratings compared with the OLS was .72 (p = .0699), .78 (p = 0.0375) for the ratings compared with the logistic, and .93 (p = .0023) for the OLS compared with the logistic regression.
Summary of Results for Each Method
Note: WTP = willingness-to-pay; AIC = Akaike Information Criterion; OLS = ordinary least squares estimation.
Each respondent answered 8 choice tasks, so there were 8*56 comparisons made for a total N of 448
Discussion
Given the literature on the relationship between conjoint analysis and rating scales for measuring preferences (Johnson et al., 2006; Phillips et al., 2002; Ryan et al., 2001), we hypothesized that there would be differences between the two methods in assessing hearing aid preferences (Arehart et al., 2010; Fitzpatrick et al., 2007; Meister et al., 2001; Moeller et al., 2009). This was not in fact the case. As seen in Figure 3, all of the willingness-to-pay values are ordered similarly in terms of preferences for each attribute regardless of the method of measurement. For example, both methods revealed that patients valued noise reduction the most and battery life the least. Conjoint analysis did produce higher willingness-to-pay values, though these could be the result of some floor and ceiling effects evidenced in the Likert scales. The attributes of noisy settings and feedback have mean values nearing 4, the maximum available choice. The means are 3.77 and 3.55, respectively. Conjoint analysis thus allowed for more accurate valuations of certain attributes, as it did not produce floor and ceiling effects.
It seems that giving respondents very clear definitions of the questions enables greater precision in the resulting scores based in Likert-type scales. Such questions elicit preferences for constructs with defined boundaries. For example, asking someone what fuel efficiency they would prefer is rather difficult unless you give them a range from which to pick. Most people would not make this choice in the real world without considering other attributes such as the price of the car. Trading then occurs among attributes to allow the consumer to achieve more of his or her preferred characteristics.
In contrast with previous literature on the subject of rating scales versus conjoint analysis, this study revealed high levels of concordance between valuations related to preferences for hearing care. We believe that our results stem from our constraining the rating of attributes to policy relevant levels. Although conjoint analysis does provide accurate assessments of preferences, our study shows that ratings scales can be as accurate, if the consumers are presented with ranges of each attribute from which to choose. This is an important finding as Likert scales are more commonly used in health and are simpler to employ. Our findings show the scales can work as well as conjoint experiments as long as the scales are bounded.
In previous literature, cost is often thought to be the biggest barrier to use of hearing aids. The sample surveyed here tended to be overwhelmingly white, educated, and wealthy, which limits the generalizability of the findings. However, this analysis answers an important question of why nearly half of a group of individuals who could afford a hearing aid have never worn one. This analysis shows that respondents were willing to pay more for hearing aids allowing them to hear conversations with their friends or family under challenging listening conditions. This finding fits in well with previous research on the effects of underutilization of hearing aids, including increased rates of depression, decreased quality of life, and increased utilization of some medical services (Abrams, Barnet, Hoth, Schultz, & Kaboli, 2006; Arlinger, 2003; Ishine, Okumiya, & Matsubayashi, 2007). Patients truly want their devices to work well in a variety of listening environments. If hearing aids do not work well from the users’ point of view, then patients will have limited benefit from their use.
Footnotes
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The research was supported by a grant from InHealth, the Institute for Health Technology Studies.