Heterogeneity in the Consistency of Best–Worst Scale Responses

Abstract

Best–worst Scaling is one of the dominant measurement approaches in choice experimentation. When employed it provides substantial information on peoples’ preferences without making choice tasks prohibitively long. However, one concern with this method is that peoples’ selection of a best may not reflect the same preferences as when a worst is selected. Research into such an inconsistency between best and worst responses has found it to be a non-trivial and persistent problem. This research further investigates these inconsistencies and finds that they can largely be attributed to a relatively small group of people in the sample who do not anchor their worst responses onto their best responses as literature suggests they would. In fact, 25% of the participants in a sample account for between 50 and 60% of the inconsistent responses recorded. The presence of this group and their disproportionate contribution to the number of inconsistencies in best and worst responses provide strong evidence that there is heterogeneity in how consistently people formulate responses in best–worst tasks. Recommendations are made regarding how to accommodate this phenomenon in utility based choice models so that better predictions of choices can be made.