Gremlins in the Data: Identifying the Information Content of Research Subjects

Abstract

Empirical demand functions, such as those from choice-based conjoint analyses, are critical to many aspects of marketing. Approaches have been developed to ensure that research subjects provide honest and thoughtful responses. However, to reduce costs, researchers increasingly collect data online, under conditions that compromise the value of the information provided. Objective measures related to how the study is completed, such as latency (how quickly answers are given), can only be tied to other objective measures (such as the consistency of the answers), but ultimately their relationship to the subject’s utility function is questionable. To address this problem, the authors introduce a mixture modeling framework that clusters subjects based on variances. The proposed model naturally groups subjects based on their internal consistency. The authors argue that a higher level of internal consistency (i.e., lower variance) reflects more engaged consumers who have sufficient experience with the product category and choice task. “Gremlins,” in contrast, behave such that the noise in their responses overwhelms any signal, leading to a lack of predictive power. This approach provides an automated way to determine which respondents are relevant. The authors discuss the conceptual and modeling framework and illustrate the method using both simulated and commercial data.