Advertising for Cognitive Interviews

Abstract

Researchers commonly rely on relatively small convenience samples for cognitive pretesting questionnaires. Methods used to recruit these samples vary depending on the population of interest, study timeline, study budget, and other factors. Over the past decade, one method that has become popular because of these considerations is online classified advertisements (e.g., Craigslist ads). A concern with the use of this recruitment method is that it leads to a set of participants who repeatedly participate in cognitive interview studies, changing the cognitive processes used in interviews, potentially resulting in misleading findings. Advertisements placed on social networking sites, such as Facebook, may give researchers more control over targeting recruitment advertisements, produce more participant diversity, and reduce the prevalence of “professional participants” who respond to ads. Recent research has shown that Craigslist and Facebook advertisements do result in selection pools with different demographic characteristics and experiences as study participants. However, we are not familiar with any research that has attempted to address concerns about data quality as a result of (a) professional participant cognitive bias or (b) recruitment method. Using data from two studies for which recruitment advertisements were placed on Craigslist and Facebook, we assess whether there are differences in recruitment speed, demographic diversity, the extent to which professional participants comprise the recruitment pool, and the extent to which a geographically dispersed recruitment pool can be attained. Evidence across the measures of quality was mixed. Facebook advertisements resulted in much faster recruitment than Craigslist advertisements among an online population in which the study topic was virtual worlds and avatars (Study 1), but the inverse was true among an older population in which the study topic was long-term care (Study 2). Mixed evidence was also found for relationships between recruitment platform and demographic composition. In Study 1, we found relationships between recruitment method and education, ethnicity, and race. In Study 2, there was only a relationship between recruitment platform and marital status and employment status. Furthermore, Facebook recruits were significantly younger than Craigslist recruits in Study 1, while in Study 2, Craigslist recruits were younger. Professional participants were identified in the recruitment pools when the concept was operationalized as attempts to deceive the researchers in how they learned about the study. No evidence was found, however, when professional participant was operationalized as the number of times one had participated in research in the past 12 months. Finally, while no comparison is available between platforms, we found that Facebook advertisements resulted in a geographically dispersed recruitment pool with per capita rates ranging from 0.05 to 1.6 and only one state having no representation. The findings from this research will help survey practitioners who conduct cognitive interviews make important decisions in which platforms to expend limited resources for the best recruitment pools from which to draw interview participants.