Detecting falsification in a television audience measurement panel survey

Abstract

Case studies reporting real-world experiences with survey falsification are uncommon. In this article, we document the experience of a panel survey in India that produced TV viewing estimates (“TV ratings”) where external parties were illegitimately trying to influence respondents’ behavior. The usual method to detect possible falsifications was that of analysts poring through data to find suspicious viewing patterns. Here, we develop a method using multilevel models and illustrate its use in the detection of an actual incident. We report how the model-based method was used to direct on-ground investigations that ultimately supported our analytic inferences. The model-based method offers four advantages over the usual method. First, by approximating an interpenetrated sample, the model simultaneously controls for several household characteristics. Second, Empirical Best Linear Unbiased Predictors (EBLUPs) of random effects can be examined separately at both the household and interviewer level, thus suggesting where further investigation efforts should be directed. Third, the method is faster and more objective than the usual method. Fourth, the method is easily implemented and can provide regular quality control for survey organizations.