When and how to survey potential respondents is often determined by budgetary and external constraints, but choice of survey modality may have enormous implications for data quality. Different survey modalities may be differentially susceptible to measurement error attributable to interviewer assignment, known as interviewer effects. In this paper, we leverage highly similar surveys, one conducted face-to-face (FTF) and the other via phone, to examine variation in interviewer effects across survey modality and question type. We find that while there are no cross-modality differences for simple questions, interviewer effects are markedly higher for sensitive questions asked over the phone. These findings are likely explained by the enhanced ability of in-person interviewers to foster rapport and engagement with respondents. We conclude with a thought experiment that illustrates the potential implications for power calculations, namely, that using FTF data to inform phone surveys may substantially underestimate the necessary sample size for sensitive questions.
AbateG. T.de BrauwA.HirvonenK.WolleA. (2023). Measuring consumption over the phone: Evidence from a survey experiment in urban Ethiopia. Journal of Development Economics, 161(Mar), 103026.
2.
AbayK. A.BerhaneG.HoddinottJ.TafereK. (2021). Assessing response fatigue in phone surveys (p. 9636). World Bank Policy Research Working Paper.
3.
AdidaC. L.FerreeK. E.PosnerD. N.RobinsonA. L. (2016). Who’s asking? Interviewer coethnicity effects in African survey data. Comparative Political Studies, 49(12), 1630–1660. https://doi.org/10.1177/0010414016633487
4.
AquilinoW. S. (1994). Interview mode effects in surveys of drug and alcohol use: A field experiment. Public Opinion Quarterly, 58(2), 210–240.
5.
BarnowB. S.GreenbergD. (2015). Do estimated impacts on earnings depend on the source of the data used to measure them? Evidence from previous social experiments. Evaluation Review, 39(2), 179–228.
6.
BeegleK.DehejiaR. H.GattiR. (2003). Child labor, crop shocks, and credit constraints. National Bureau of Economic Research Cambridge.
7.
BiemerP. P. (2010). Total survey error: Design, implementation, and evaluation. Public Opinion Quarterly, 74(5), 817–848.
8.
BischopingK.SchumanH. (1992). Pens and polls in Nicaragua: An analysis of the 1990 preelection surveys. American Journal of Political Science, 36(2), 331. https://doi.org/10.2307/2111480
9.
BloomH. S. (1995). Minimum detectable effects. Evaluation Review, 19(5), 547–556.
10.
Brunton-SmithI.SturgisP.LeckieG. (2017). Detecting and understanding interviewer effects on survey data by using a cross-classified mixed effects location-scale model. Journal of the Royal Statistical Society - Series A: Statistics in Society, 180(2), 551–568. https://doi.org/10.1111/rssa.12205
11.
CantyA.RipleyB. (2022). boot: Bootstrap R (S-Plus) functions. R package version 1.3–28.1.
12.
CohenA.IrwinJ. V.GallE. (2017). From paper to pixels: The effects of modality on end-of-course survey responses. Asian Journal of Assessment in Teaching and Learning, 7(Dec), 32–43.
13.
CrossleyT. F.SchmidtT.TzamouraniP.WinterJ. K. (2021). Interviewer effects and the measurement of financial literacy. Journal of the Royal Statistical Society - Series A: Statistics in Society, 184(1), 150–178. https://doi.org/10.1111/rssa.12617
14.
CrossleyT. F.WinterJ. K. (2015). Asking Households about Expenditures. NBER.
15.
CullenC. (2020). Method matters: Underreporting of intimate partner violence in Nigeria and Rwanda. The World Bank. https://doi.org/10.1596/1813-9450-9274
16.
DavisonA. C.HinkleyD. V. (1997). Bootstrap methods and their application. Cambridge University Press.
17.
DillmanD. A.ChristianL. M. (2005). Survey mode as a source of instability in responses across surveys. Field Methods, 17(1), 30–52.
18.
Di MaioM.FialaN. (2020). Be wary of those who Ask: A randomized experiment on the size and determinants of the enumerator effect. The World Bank Economic Review, 34(3), 654–669. https://doi.org/10.1093/wber/lhy024
19.
FogartyB. J.KellyN. J.KilburnH. W. (2005). Issue attitudes and survey continuity across interview mode in the 2000 NES. Political Analysis, 13(1), 95–108.
20.
FootmanK. (2021). Interviewer effects on abortion reporting: A multilevel analysis of household survey responses in côte d'Ivoire, Nigeria and Rajasthan, India. BMJ Open, 11(11), e047570. https://doi.org/10.1136/bmjopen-2020-047570
21.
GibsonJ.KimB. (2007). Measurement error in recall surveys and the relationship between household size and food demand. American Journal of Agricultural Economics, 89(2), 473–489. https://doi.org/10.1111/j.1467-8276.2007.00978.x
22.
HansenM. H.HurwitzW. N.BershadM. A. (1961). Measurement errors in censuses and surveys. Bureau of the Census.
23.
HansonR. H.MarksE. S. (1958). Influence of the interviewer on the accuracy of survey results. Journal of the American Statistical Association, 53(283), 635–655.
24.
HarlingG.ChandaM. M.OrtbladK. F.MwaleM.ChongoS.KancheleC.KamungomaN.BarresiL. G.BärnighausenT.OldenburgC. E. (2019). The influence of interviewers on survey responses among female sex workers in Zambia. BMC Medical Research Methodology, 19(1), 60. https://doi.org/10.1186/s12874-019-0703-2
25.
HimeleinK. (2016). Interviewer effects in subjective survey questions: Evidence from Timor-Leste. International Journal of Public Opinion Research, 28(4), 511–533.
26.
HolbrookA. L.GreenM. C.KrosnickJ. A. (2003). Telephone versus face-to-face interviewing of national probability samples with long questionnaires. Public Opinion Quarterly, 67(1), 79–125. https://doi.org/10.1086/346010
27.
HoxJ. (1994). Hierarchical regression models for interviewer and respondent effects. Sociological Methods & Research, 22(3), 300–318. https://doi.org/10.1177/0049124194022003002.
28.
JeongD.AggarwalS.RobinsonJ.KumarN.SpearotA.ParkD. S. (2023). Exhaustive or exhausting? Evidence on respondent fatigue in long surveys. Journal of Development Economics, 161(Mar), 102992.
29.
KarlanD. S.ZinmanJ. (2012). List randomization for sensitive behavior: An application for measuring use of loan proceeds. Journal of Development Economics, 98(1), 71–75. https://doi.org/10.1016/j.jdeveco.2011.08.006
30.
KreuterF.PresserS.TourangeauR. (2008). Social desirability bias in CATI, IVR, and web surveys: The effects of mode and question sensitivity. Public Opinion Quarterly, 72(5), 847–865.
31.
KrumpalI. (2013). Determinants of social desirability bias in sensitive surveys: A literature review. Quality and Quantity, 47(4), 2025–2047.
32.
LeeR. M.RenzettiC. M. (1990). The problems of researching sensitive topics. American Behavioral Scientist, 33(5), 510–528.
33.
LiuM.StainbackK. (2013). Interviewer gender effects on survey responses to marriage-related questions. Public Opinion Quarterly, 77(2), 606–618.
34.
LoftusE. F.MarburgerW. (1983). Since the eruption of Mt. St. Helens, has anyone beaten you up? Improving the accuracy of retrospective reports with landmarkevents. Memory & Cognition, 11(2), 114–120. https://doi.org/10.3758/BF03213465
35.
LupuN.WolskyA. (2022) Survey mode effects in a developing country: comparing phone and face-to-face surveys in Costa Rica. Revista Latinoamericana de Opinión Pública, 11(2), 109–132. https://doi.org/10.14201/rlop.30159
36.
MangioneT. W.FowlerF. J.LouisT. A. (1992). Question characteristics and interviewer effects. Journal of Official Statistics-Stockholm, 8(3), 293–293.
37.
MarsdenP. V.WrightJ. D. (2010). Handbook of survey research. Emerald Group Publishing.
38.
NandiA.PlattL. (2017). Are there differences in responses to social identity questions in face-to-face versus telephone interviews? Results of an experiment on a longitudinal survey. International Journal of Social Research Methodology, 20(2), 151–166. https://doi.org/10.1080/13645579.2016.1165495
39.
OlsonK.SmythJ. D.DykemaJ.HolbrookA. L.KreuterF.WestB. T. (2020). The past, present, and future of research on interviewer effects. In OlsonK.SmythJ. D.DykemaJ.HolbrookA. L.KreuterF.WestB. T. (Eds.), Interviewer effects from a total survey error perspective (1st ed., pp. 3–16). Chapman and Hall/CRC. https://doi.org/10.1201/9781003020219-2
40.
ParkD. S.AggarwalS.JeongD.KumarN.RobinsonJ.SpearotA. (2021). Private but misunderstood? Evidence on measuring intimate partner violence via self-interviewing in rural Liberia and Malawi. National Bureau of Economic Research.
41.
PietrelliR.d’ErricoM.DassesseK. (2021). Measuring household food security through surveys: Do the characteristics of the enumerators matter?Development Policy Review, 39(6), 911–925. https://doi.org/10.1111/dpr.12534
42.
Rodriguez-SeguraD.SchuelerB. E. (2023). Assessors influence results: Evidence on enumerator effects and educational impact evaluations. Journal of Development Economics, 163(June), 103057.
ScottC.AmenuvegbeB. (1990). Effect of recall duration on reporting of household expenditures. World Bank.
45.
ShuyR. W. (2003). In-person versus telephone interviewing (pp. 175–193). Inside Interviewing: New Lenses.
46.
SimmonsA. D.BoboL. D. (2015). Can non-full-probability internet surveys yield useful data? A comparison with full-probability face-to-face surveys in the domain of race and social inequality attitudes. Sociological Methodology, 45(1), 357–387. https://doi.org/10.1177/0081175015570096
47.
SturgesJ. E.HanrahanK. J. (2004). Comparing telephone and face-to-face qualitative interviewing: A research note. Qualitative Research, 4(1), 107–118.
48.
SundströmA.StockemerD. (2022). Measuring support for women’s political leadership: Gender of interviewer effects among African survey respondents. Public Opinion Quarterly, 86(3), 668–696.
49.
TeppingB. J.BailarB. A. (1973). Enumerator variance in the 1970 census. In Proceedings of the Social Statistics Section, American Statistical Association Annual Meeting (pp. 160–169). New York City.
50.
VandenplasC. (2019). The impact of time and timing on the survey process and data quality. KU Leuven.
51.
WestB. T.BlomA. G. (2017). Explaining interviewer effects: A research synthesis. Journal of Survey Statistics and Methodology, 5(2), 175–211. https://doi.org/10.1093/jssam/smw024
52.
WestB. T.ElliottM. R. (2014). Frequentist and Bayesian approaches for comparing interviewer variance components in two groups of survey interviewers. Survey Methodology, 40(2), 163–189.
53.
WestB. T.OlsonK. (2010). How much of interviewer variance is really nonresponse error variance?Public Opinion Quarterly, 74(5), 1004–1026.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
