Web surveys are a popular survey mode, but the subpopulation with Internet access may not represent the population of interest. The authors investigate whether adjusting using weights or matching on a small set of variables makes the distributions of target variables representative of the population. This application has a rich sampling design; the Internet sample is part of an existing probability sample, the Health and Retirement Study, that is representative of the U.S. population aged 50 and older. For the dichotomous variables investigated, the adjustment helps. On average, the sample means in the Internet access sample differ by 6.5 percent before and 3.7 percent after adjustment. Still, a large number of adjusted estimates remain significantly different from their target estimates based on the complete sample. This casts doubt on the common procedure to use only a few variables to correct for the selectivity of convenience samples.
Battaglia, Michael P., Donald J. Malec, Bruce D. Spencer, David C. Hoaglin, and Joseph Sedransk .1995. ``Adjusting for Noncoverage of Nontelephone Households in the National Immunization Survey'' Pp. 678-83 in Proceedings of the Survey Research Methods Section, American Statistical Association.
2.
Bellemare, Charles and Sabine Kroeger.2007. ``On Representative Social Capital . '' European Economic Review51:183-202.
3.
Berrens, Robert P., Alok K. Bohara, Hank Jenkins-Smith, Carol Silva, and David L. Weimer .2003. ``The Advent of Internet Surveys for Political Research: A Comparison of Telephone and Internet Samples .'' Political Analysis11:1-22.
4.
Birnbaum, Michael H.2004. ``Human Research and Data Collection via the Internet '' Annual Review of Psychology55:803-32.
5.
Blumberg, Stephen J., Julian V. Luke, and Marcie L. Cynamon .2004. ``Has Cord-Cutting Cut Into Random-Digit-Dialed Health Surveys? The Prevalence and Impact of Wireless Substitution'' Pp. 137-42 in Eighth Conference on Health Survey Research Methods, edited by S. B. Cohen and J. M. Lepkowski.Hyattsville, MD: National Center for Health Statistics.
6.
Butz, William P. and Barbara B. Torrey .2006. ``Some Frontiers in Social Science'' Science312:1898-900.
7.
Cochran, W.G.1968. ``The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies.'' Biometrics24:295-313.
8.
Couper, Mick P.2000. ``Web Surveys. A Review of Issues and Approaches '' Public Opinion Quarterly64:464-94.
9.
---, Arie Kapteyn, Matthias Schonlau, and J. Winter.2007. ``Noncoverage and Nonresponse in an Internet Survey'' Social Science Research36:131-48.
10.
Czajka, John L., Sharon M. Hirabayash, Roderick J. A. Little, and Donald B. Rubin .1992. ``Projecting From Advanced Data Using Propensity Modeling: An Application to Income and Tax Statistics '' Journal of Business and Economic Statistics10:117-32.
11.
Danielsson, Stig.2004. ``The Propensity Score and Estimation in Nonrandom Surveys: An Overview'' Modern Statistical Survey Methods Project Report No. 18, Department of Statistics, University of Linkoeping. Retrieved August 2004 from http://www.statistics.su.se/modernsurveys/publ/11.pdf
12.
De Heer, Wim.1999. ``International Response Trends: Results of an International Survey.'' Journal of Official Statistics15:129-42.
13.
Diamond, Alexis and Jasjeet Sekhon.2005. ``Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies'' Working paper, Berkeley University. Available from http://sekhon.berkeley.edu/papers/GenMatch.pdf
14.
Duffy, Bobby, Kate Smith, George Terhanian, and John Bremer.2005. ``Comparing Data From Online and Face-to-Face Surveys'' International Journal of Market Research47:615-39.
15.
Duncan, Kristin B. and Elizabeth A. Stasny .2001. ``Using Propensity Scores to Control Coverage Bias in Telephone Surveys'' Survey Methodology27:121-30.
16.
Garren, Steven T. and Ted C. Chang .2002. ``Improved Ratio Estimation in Telephone Surveys Adjusting for Noncoverage'' Survey Methodology28:63-76.
17.
Groves, Robert M.1989. Survey Errors and Survey Costs. New York: John Wiley.
18.
--- and Mick P. Couper.1998. Nonresponse in Household Interview Surveys . New York: John Wiley.
19.
Health and Retirement Study.2002. Sampling Weights Revised for Tracker 2.0 and Beyond. Ann Arbor, MI: University of Michigan, Survey Research Center. Available from http://hrsonline.isr.umich.edu/meta/tracker/desc/wghtdoc.pdf
20.
Huggins, Vicki J. and Karol Krotki.2001. ``Implementation of Nationally Representative Web-Based Surveys'' In Proceedings of the Annual Meeting of the American Statistical Association. Available from http://www.amstat.org/Sections/Srms/Proceedings/y2001/Proceed/00299.pdf
21.
Iannacchione, Vincent G.2003. ``Sequential Weight Adjustments for Location and Cooperation Propensity for the 1995 National Survey of Family Growth'' Journal of Official Statistics19:31-43.
22.
Isaksson, Annika and Gosta Forsman.2003. ``A Comparison Between Using the Web and Using the Telephone to Survey Political Opinions'' Pp. 100-106 in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association. Available from http://www.amstat.org/sections/SRMS/Proceedings
23.
Juster, F. Thomas and Richard Suzman.1995. ``An Overview of the Health and Retirement Study'' Journal of Human Resources [Special Issue on the Health and Retirement Study: Data Quality and Early Results]30:S7-56.
24.
Kish, Leslie.1965. Survey Sampling. New York: John Wiley.
25.
Lee, Sunghee.2004. ``Statistical Estimation Methods in Volunteer Panel Web Surveys.'' Ph.D. dissertation, Joint Program of Survey Methodology, University of Maryland.
26.
---. 2006. ``Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys.'' Journal of Official Statistics22:329-49.
27.
Link, Michael W. and Robert W. Oldendick .1999. ``Call Screening: Is It Really a Problem for Survey Research?'' Public Opinion Quarterly63:577-89.
28.
Little, Roderick A. and Donald B. Rubin .2002. Statistical Analysis With Missing Data. New York: John Wiley .
29.
McCaffrey, Daniel F., Greg Ridgeway, and Andrew R. Morral .2004. ``Propensity Score Estimation With Boosted Regression for Evaluating Causal Effects in Observational Studies '' Psychological Methods9:403-25.
30.
Oldendick, Robert W. and Michael W. Link .1994. ``The Answering Machine Generation: Who Are They and What Problem Do They Pose for Survey Research? '' Public Opinion Quarterly58:264-73.
31.
Rosenbaum, Paul R.1987. ``Model-Based Direct Adjustment'' Journal of the American Statistical Association82:387-94.
32.
---. 2002. Observational Studies. 2d ed. New York: Springer-Verlag.
33.
--- and Donald B. Rubin.1983. ``The Central Role of the Propensity Score in Observational Studies for Causal Effects'' Biometrika70:41-55.
34.
Rubin, Donald B. and Neal Thomas.1996. ``Matching Using Estimated Propensity Scores: Relating Theory to Practice'' Biometrics52:249-64.
35.
Schonlau, Matthias, Beth J. Asch, and Can Du.2003. ``Web Surveys as Part of a Mixed Mode Strategy for Populations That Cannot Be Contacted by E-Mail'' Social Science Computer Review21:218-22.
36.
---, Ronald Fricker, and Marc Elliott.2002. Conducting Research Surveys via Email and the Web. Santa Monica, CA: RAND.
37.
---, Kinga Zapert, Lisa Payne Simon, Katherine Sanstad, Sue Marcus, John Adams, Mark Spranca, Hongjun Kan, Rachel Turner, and Sandra Berry.2004. ``A Comparison Between a Propensity Weighted Web Survey and an Identical RDD Survey'' Social Science Computer Review22:128-38.
38.
Schwarz, Norbert and Seymour Sudman, eds. 1992. Context Effects in Social and Psychological Research. New York: Springer-Verlag.
39.
Smith, Philip J., J.N.K. Rao, Michael P. Battaglia, Dani Daniels, and Trena Ezzati-Rice.2001. ``Compensating for Provider Nonresponse Using Response Propensities to Form Adjustment Cells: The National Immunization Survey'' Vital and Health Statistics2 (133): 1-17.
40.
Sobel, Michael E.1995. ``Causal Inference in the Social and Behavioral Sciences '' Pp. 1-38 in Handbook of Statistical Modeling for the Social and Behavioral Sciences , edited by Gerhard C. Arminger, Clifford C. Clogg, and Michael E. Sobel.New York : Plenum.
41.
Taylor, Humphrey.2000. ``Does Internet Research `Work'? Comparing On-Line Survey Results With Telephone Surveys.'' Journal of Marketing Research42:51-64.
42.
Valliant, Richard.2004. ``The Effect of Multiple Weighting Steps on Variance Estimation .'' Journal of Official Statistics20:1-18.
43.
Varedian, Maria and Gosta Forsman.2003. ``Comparing Propensity Score Weighting With Other Weighting Methods: A Case Study on Web Data'' Presented at the American Association for Public Opinion Research Conference , St. Petersburg Beach, FL.
44.
Winship, Christopher and Stephen L. Morgan .1999. ``The Estimation of Causal Effects From Observational Data'' Annual Review of Sociology25:659-707.
45.
________ and Michael Sobel.2006. ``Causal Inference in Sociological Studies'' Retrieved December 2006 from http://www.wjh.harvard.edu/∼winship/cfa_papers/causalinference.pdf
46.
Yoshimura, Osamu.2004. Pp. 4660-65 in ``Adjusting Responses in a Non-probability Web Panel Survey by the Propensity Score Weighting'' Proceedings of the Section on Survey Statistics, American Statistical Association . Available from http://www.amstat.org/sections/SRMS/Proceedings
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
