Restricted accessResearch articleFirst published online 2022-6
Evaluating the Data Quality of a National Sample of Young Sexual and Gender Minorities Recruited Using Social Media: The Influence of Different Design Formats
Social media recruitment is no longer an uncharted avenue for survey research. The results thus far provide evidence of an engaging means of recruiting hard-to-reach populations. Questions remain, however, regarding whether the data collected using this method of recruitment produce quality data. This article assesses one aspect that may influence the quality of data gathered through nonprobability sampling using social media advertisements for a hard-to-reach sexual and gender minority youth population: recruitment design formats. The data come from the Survey of Today’s Adolescent Relationships and Transitions, which used a variety of forms of advertisements as survey recruitment tools on Facebook, Instagram, and Snapchat. Results demonstrate that design decisions such as the format of the advertisement (e.g., video or static) and the use of eligibility language on the advertisements impact the quality of the data as measured by break-off rates and the use of nonsubstantive responses. Additionally, the type of device used affected the measures of data quality.
ArigoD.PagotoS.Carter-HarrisL.SarahL.NebekerC. (2018). Using social media for health research: Methodological and ethical considerations for recruitment and intervention delivery. Digital Health, 4. https://doi.org/10.1177/2055207618771757
2.
CallegaroM. (2013). Paradata in web surveys. In FraukeK. (Ed.), Improving surveys with paradata: Analytic uses of process information (pp. 261–279). John Wiley & Sons.
3.
CataniaJ.DolciniM.OrellanaR.NarayananV. (2015). Nonprobability and probability-based sampling strategies in sexual science. Journal of Sex Research, 52(4), 396–411.
4.
ChangL.KrosnickJ. A. (2009). National surveys via RDD telephone interviewing versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly, 73(4), 641–678.
5.
ConverseP. E. (1964). The nature of belief systems in mass publics. In ApterD. E. (Ed.), Ideology and discontent (pp. 206–261). Free Press.
6.
ConverseP. E. (1970). Attitudes and non-attitudes: Continuation of a dialogue. In TufteE. R. (Ed.), The quantitative analysis of social problems (pp. 168–189). Addison-Wesley.
7.
CouperM. P. (2000). Web surveys: A review of issues and approaches. Public Opinion Quarterly, 64(4), 464–494.
8.
CouperM. P.MillerP. V. (2008). Web survey methods: Introduction. Public Opinion Quarterly, 72(5), 831–835.
9.
CowieJ. M.GurneyM. E. (2018). The use of Facebook advertising to recruit healthy elderly people for a clinical trial: Baseline metrics. JMIR Research Protocols, 7(1), e20. http://www.researchprotocols.org/2018/1/e20/doi:10.2196/resprot.7918 PMID:29367186
10.
de LeeuwE. D. (2005). To mix or not to mix data collection modes in surveys. Journal of Official Statistics, 21(2), 233–255.
11.
DillmanD. A.SmythJ. D.ChristianL. M. (2014). Internet, phone, mail, and mixed-mode surveys: The tailored design method. John Wiley & Sons.
12.
DusekG. A.YurovaY. V.RuppelC. P. (2015). Using social media and targeted snowball sampling to survey a hard-to-reach population: A case study. International Journal of Doctoral Studies, 10, 279–299.
13.
EisenhowerD.MathiowetzN.MorgansteinD. (1991). Recall error: Sources and bias reduction techniques. In BiemerP. P.GrovesR. M.LybergL. E.MathiowetzN. A.SudmanS. (Eds.), Measurement errors in surveys (pp. 127–144). Wiley.
14.
FordyceE.Heim VioxM.SternM. J.AvripasS.BilgenI.FlowersV.MichaelsS.HarperC.JohnsM.DunvilleR. (2018). Recruitment for adolescent sexual minority males and transgender youth [Paper presentation]. American Association for Public Opinion Research Meetings. Denver, CO, United States.
15.
GrovesR. M.PresserS.DipkoS. (2004). The role of topic interest in survey participation decisions. Public Opinion Quarterly, 68(1), 2–31.
16.
GrovesR.SingerE.CorningA. (2000). Leverage-salience theory of survey participation: Description and an illustration. Public Opinion Quarterly, 64, 299–308.
17.
JohnsM. M.LowryR.AndrzejewskiJ.BarriosL. C.DemissieZ.McManusT.RasberryC. N.RobinL.UnderwoodJ. M. (2019). Transgender identity and experiences of violence victimization, substance use, suicide risk, and sexual risk behaviors among high school students—19 States and large urban school districts, 2017. MMWR Morbidity and Mortality Weekly Report, 68, 67–71. http://dx.doi.org/10.15585/mmwr.mm6803a3externalicon
18.
KayrouzR.DearB. F.KarinE.TitovN. (2016). Facebook as an effective recruitment strategy for mental health research of hard to reach populations. Internet Interventions, 4, 1–10.
19.
KeeterS.ChristianL. (2012). A comparison of results from surveys by the pew research center and Google consumer surveys. Pew Research Center for the People and The Press.
20.
KrosnickJ. A.HolbrookA. L.BerentM. K.CarsonR. T.Michael HanemannW.KoppR. J.MitchellR. C.PresserS.RuudP. A.SmithV. K.MoodyW. R.GreenM. C.ConawayM. (2002). The impact of “no opinion” response options on data quality: Non-attitude reduction or an invitation to satisfice?Public Opinion Quarterly, 66(3), 371–403.
21.
LinkM. W.MurphyJ.SchoberM. F.BuskirkT. D.ChildsJ. H.TesfayeC. L.PasekJ. (2014). Mobile technologies for conducting, augmenting and potentially replacing surveys: Report of the AAPOR task force on emerging technologies in public opinion research. AAPOR.
22.
LongJ. S. (1997). Regression models for categorical and limited dependent variables. Sage.
23.
MavletovaA. (2013). Data quality in PC and mobile web surveys. Social Science Computer Review, 31(6), 725–743.
24.
MavletovaA.CouperM. P. (2015). A meta-analysis of breakoff rates in mobile web surveys. In ToninelliD.PinterR.de PedrazaP. (Eds.), Mobile research methods: Opportunities and challenges of mobile research methodologies (pp. 81–98). Ubiquity Press.
25.
PeytchevA. (2009). Survey breakoff. Public Opinion Quarterly, 73(1), 74–97.
26.
PeytchevA. (2011). Breakoff and unit nonresponse across web surveys. Journal of Official Statistics, 27(1), 33.
27.
PeytchevA.CouperM. P.McCabeS. E.CrawfordS. D. (2006). Web survey design: Paging versus scrolling. International Journal of Public Opinion Quarterly, 70(4), 596–607.
28.
RamoD. E.ProchaskaJ. J. (2012). Broad reach and targeted recruitment using Facebook for an online survey of young adult substance use. Journal of Medical Internet Research, 14(1), e28.
29.
ShropshireK.HawdonJ.WitteJ. (2009). Web survey design: Balancing measurement, response, and topical interest. Sociological Methods & Research, 37(3), 344–370.
30.
SternM. J.BilgenI.DillmanD. A. (2014). The state of survey methodology in the 2010s. Field Methods, 26(3), 284–301.
31.
SternM. J.DillmanD. A.SmythJ. D. (2007). Visual design, order effects and respondent characteristics in a self-administered survey. Survey Research Methods, 1(3), 121–138.
32.
SternM. J.SmythJ. D.MendezJ. (2012). The effects of item saliency and question design on measurement error in a self-administered survey. Field Methods, 24(1), 3–27.
33.
SudmanS.BradburnN. (1974). Response effects in surveys. Aldine.
34.
YbarraM.MitchellK. (2016). A national study of lesbian, gay, bisexual (LGB), and non-LGB youth sexual behavior online and in-person. Archives of Sexual Behavior, 45(6), 1357–1372.
35.
ZhangB.MildenbergerM.HoweP. D.MarlonJ.RosenthalS.LeiserowitzA. (2018). Quota sampling using Facebook advertisements. Political Science Research and Methods, 1, 1–7.
36.
ZlotorzynskaM.SullivanP.SanchezT. (2019). The annual American men’s internet survey of behaviors of men who have sex with men in the United States: 2016 key indicators report. JMIR Public Health Surveillance. Retrieved February 20, 2019, fromhttps://doi.org/10.2196/11313
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
