Sage Journals: Discover world-class research

Abstract

A combination of propensity score and calibration adjustment is shown to reduce bias in volunteer panel Web surveys. In this combination, the design weights are adjusted by propensity scores to correct for selection bias due to nonrandomized sampling. These adjusted weights are then calibrated to control totals for the target population and correct for coverage bias. The final set of weights is comprised of multiple components, and the estimator of a total no longer takes a linear form. Therefore, approximate methods are needed to derive variance estimates. This study compares three variance estimation methods through simulation. The first method resembles what is used in commercial statistical software based on squared residuals. The second approach uses a variance estimator originally derived for the generalized regression estimator. The third method uses jackknife replication. Results indicate bias reduction is crucial for valid variance estimation and favor the replication method over the other approaches.

Keywords

Web surveys nonprobability sampling post-survey adjustment

Get full access to this article

View all access options for this article.

References

Breslow, N.E. and N.E. Day. 1980. Statistical Methods in Cancer Research. Vol. 1, The Analysis of Case-Control Studies. Lyon, France: IARC Scientific Publications.

Centers for Disease Control and Prevention. 2003. Overview: BRFSS 2003. Atlanta, GA: Author. Retrieved from http://www.cdc.gov/brfss/technical_infodata/surveydata/2003/overview_03.rtf

Cochran, W.G. 1968. ``The Effectiveness of Adjustment by Subclassification in Removing Bias in Observational Studies.'' Biometrics 24:295-313.

Cook, M.L. and A.P. Rafferty. 2004. Health Risk Behaviors in the State of Michigan: 2003 Behavioral Risk Factor Survey. Lansing: Michigan Department of Community Health, Bureau of Epidemiology, Epidemiology Services Division. Retrieved from http://www.michigan.gov/documents/2003_MI_BRFS_Annual_Report_121101_7.pdf

Czajka, J.L. , S.M. Hirabayashi , R.J.A. Little , and D.B. Rubin. 1992. ``Projecting From Advance Data Using Propensity Modeling: An Application to Income and Tax Statistics'' Journal of Business and Economic Statistics 10:117-32.

Deville, J.C. and C.E. Sarndal. 1992. ``Calibration Estimators in Survey Sampling .'' Journal of the American Statistical Association 87:376-82.

Deville J.C. , C.E. Sarndal , and O. Sautory. 1993. ``Generalized Raking Procedures in Survey Sampling.'' Journal of the American Statistical Association 88:1013-20.

Fuller, W.A. 1998. ``Replication Variance Estimation for the Two-Phase Samples .'' Statistica Sinica 8:1153-64.

Hosmer, D.W. and S. Lemeshow. 2000. Applied Logistic Regression. New York: John Wiley.

10.

Jayasuriya, B. and R. Valliant. 1996. ``An Application of Restricted Regression Estimation to Post-Stratification in a Household Survey.'' Survey Methodology 22:127-37.

11.

Kim, J.J. , J. Li , and R. Valliant. 2007. ``Cell Collapsing in Poststratification .'' Survey Methodology 33:139-50.

12.

Kott, P.S. 2006. ``Using Calibration Weighting to Adjust for Nonresponse and Coverage Errors.'' Survey Methodology 32:133-42.

13.

Lee, S. 2004. ``Statistical Estimation Methods in Volunteer Panel Web Surveys.'' Ph.D. dissertation, Joint Program in Survey Methodology, University of Maryland.

14.

---. 2006. ``Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys.'' Journal of Official Statistics 22:329-49.

15.

Lee, S. and R. Valliant. 2007. ``Weighting Telephone Samples Using Propensity Scores'' Pp. 170-83 in Advances in Telephone Survey Methodology , edited by J. M. Lepkowski , C. Tucker , J. M. Brick , E. De Leeuw , L. Japec , P. J. Lavrakas , M. W. Link , and R. L. Sangster. New York: John Wiley.

16.

R Development Core Team. 2005. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing . Available from http://www.R-project.org

17.

Rosenbaum, P.R. and D.B. Rubin. 1983. ``The Central Role of the Propensity Score in Observational Studies for Causal Effects.'' Biometrika 70:41-55.

18.

---. 1984. ``Reducing Bias in Observational Studies Using Subclassification on the Propensity Score.'' Journal of the American Statistical Association 79:516-24.

19.

Schonlau, M. , A. van Soest , and A. Kapteyn. 2007. ``Are `Webographic' or Attitudinal Questions Useful for Adjusting Estimates From Web Surveys Using Propensity Scoring? '' Survey Research Methods 1:155-63.

20.

Schonlau, M. , K. Zapert , L.P. Simon , K. Sanstad , S. Marcus , J. Adams , M. Spranca , H. Kan , R. Turner , and S. Berry. 2004. ``A Comparison Between a Propensity Weighted Web Survey and an Identical RDD Survey'' Social Science Computer Review 22:128-38.

21.

Taylor, H. , J. Bremer , C. Overmeyer , J.W. Siegel , and G. Terhanian. 2001. ``The Record of Internet-Based Opinion Polls in Predicting the Results of 72 Races in the November 2000 US Elections '' International Journal of Market Research 43:127-35.

22.

Terhanian, G. and J. Bremer. 2000. ``Confronting the Selection-Bias and Learning Effects Problems Associated With Internet Research'' White paper, Harris Interactive, Rochester, NY .

23.

Terhanian, G. , J. Bremer , R. Smith , and R. Thomas. 2000. ``Correcting Data From Online Surveys for the Effects of Nonrandom Selection and Nonrandom Assignment'' White paper, Harris Interactive, Rochester, NY.

24.

Valliant, R. 2004. ``The Effect of Multiple Weighting Steps on Variance Estimation .'' Journal of Official Statistics 20:1-18.

25.

Wolter, K.M. 1985. Introduction to Variance Estimation. New York: Springer-Verlag.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.32 MB