In mixed-mode surveys, it is difficult to separate sample selection differences from mode-effects that can occur when respondents respond in different interview settings. This paper provides a framework for separating mode effects from selection effects by matching very similar respondents from different survey modes using propensity score matching. The answer patterns of the matched respondents are subsequently compared. We show that matching can explain differences in nonresponse and coverage in two Internet samples. When we repeat this procedure for a telephone and Internet sample however, differences persist between the samples after matching. This indicates the occurrence of mode effects in telephone and Internet surveys. Mode effects can be problematic; hence we conclude with a discussion of designs that can be used to explicitly study mode effects.
Get full access to this article
View all access options for this article.
References
1.
AAPOR (2008) Standard Definitions: Final Dispositions of Case Codes and Outcome Reports for Surveys (5th edn). Lenexa, Kansas: American Association for Public Opinion Research.
2.
BiemerP.B. (2001) Non-response bias and measurement bias in a comparison of face to face and telephone interviewing.Journal of Official Statistics, 17, 2, pp. 295–320.
3.
BlythB. (2008) Mixed mode: the only ‘fitness’ regime?International Journal of Market Research, 50, 2, pp. 241–266.
4.
CookT.D., ShadishW.R., & WongV.C. (2008) Three conditions under which experiments and observational studies produce comparable causal estimates: new findings from within-study comparisons.Journal of Policy Analysis and Management, 27, 4, pp. 724–750.
5.
DehejiR.H., & WahbaS. (2002) Propensity score matching methods for non-experimental causal studies.Review of Economics and Statistics, 84, 1, pp. 151–161.
6.
de LeeuwE.D. (2005) To mix or not to mix data collection modes in surveys.Journal of Official Statistics, 21, 2, pp. 233–255.
7.
de LeeuwE.D., & de HeerW. (2002) Trends in household survey non-response: a longitudinal and international comparison. In: GrovesR.M., DillmanD.A., EltingeJ.L., & LittleR.J.A. (eds) Survey Non-response.New York: Wiley, pp. 41–54.
8.
de LeeuwE.D., & van der ZouwenJ. (1989) Data quality in telephone and face to face surveys: a comparative meta-analysis. In: GrovesR.M., BiemerP.B., LybergL.E., MasseyJ.T., NicholsW.L.I., & WaksbergJ. (eds) Telephone Survey Methodology.New York: Wiley, pp. 283–299.
9.
DillmanD.A., & ChristianL.M. (2005) Survey mode as a source of instability in responses across surveys.Field Methods, 17, 1, pp. 30–52.
10.
DillmanD.A., PhelpsG., TortoraR., SwiftK., KohrellJ., BerckJ., & MesserB.L. (2009) Response rate and measurement differences in mixed-mode surveys using mail, telephone, interactive voice response (IVR) and the internet.Social Science Research, 38, 1, pp. 1–18.
11.
GrovesR.M. (2005) Survey Errors and Survey Costs (2nd edn). Hoboken, New Jersey: John Wiley & Sons.
12.
GrovesR.M., & PeytchevaE. (2008) The impact of non-response rates on non-response bias – a meta analysis.Public Opinion Quarterly, 72, 2, pp. 167–189.
13.
HeerweghD. (2009) Mode differences between face-to-face and web surveys: an experimental investigation of data quality and social desirability effects.International Journal of Public Opinion Research, 21, 1, pp. 111–121.
KoolL., MarisA., & MunckS.D. (2009) Marktrapportage elektronische communicatie (market report on electronic communication).Netherlands Organisation for the Advancement of Science (TNO).
17.
KreuterF., PresserS., & TourangeauR. (2008) Social desirability bias in CATI, IVR and web surveys.Public Opinion Quarterly, 72, 5, pp. 847–865.
18.
LeeS. (2006) Propensity score adjustment as a weighting scheme for volunteer panel web surveys.Journal of Official Statistics, 22, 2, pp. 329–349.
19.
R Core Development Team (2009) R: a language and environment for statistical computing, version 2.9.1. Vienna, Austria (www.r-project.org).
20.
SchonlauM., van SoestA., KapteynA., & CouperM. (2009) Selection bias in web-surveys and the use of propensity scores.Sociological Methods & Research, 37, 3, pp. 291–318.
VoogtR.J.J., & SarisW.E. (2005) Mixed mode designs: finding the balance between non-response bias and mode effects.Journal of Official Statistics, 21, pp. 367–387.
