An Empirical Evaluation of In-Depth Probes Used to Pretest Survey Questions

Abstract

A growing number of survey methodologists have advocated the use of in-depth probes for identifying comprehension problems, inadequate response categories, the perspectives adopted by respondents, and the strategies respondents employ when retrieving information from memory. This article reports the results of an empirical evaluation of the relative effectiveness of a number of the probes that have been used for the first three of these tasks. This work confirms that the traditional practice of field testing a questionnaire on a subsample drawn from the population of interest is not an effective way of detecting shortcomings in survey questions. It also indicates that the effectiveness of a probe is directly related to its specificity; the most effective comprehension probes are those directed at exploring the ways in which respondents interpret key concepts; probes designed to get at respondents' interpretations of key concepts are the most effective means of identifying perspectives respondents have adopted; and well-educated respondents are most likely to make substantive responses to in-depth probes.