Sage Journals: Discover world-class research

Abstract

Nonresponse bias is a widely prevalent problem for data on education. We develop a ten-step exemplar to guide nonresponse bias analysis (NRBA) in cross-sectional studies and apply these steps to the Early Childhood Longitudinal Study, Kindergarten Class of 2010–2011. A key step is the construction of indices of nonresponse bias based on proxy pattern-mixture models for survey variables of interest. A novel feature is to characterize the strength of evidence about nonresponse bias contained in these indices, based on the strength of the relationship between the characteristics in the nonresponse adjustment and the key survey variables. Our NRBA improves the existing methods by incorporating both missing at random and missing not at random mechanisms, and all analyses can be done straightforwardly with standard statistical software.

Keywords

nonresponse bias analysis (NRBA)missing not at random (MNAR)strong predictors proxy pattern-mixture model sensitivity analysis

Get full access to this article

View all access options for this article.

References

Andridge

R. R.

Little

R. J. A.

(2011). Proxy pattern-mixture analysis for survey nonresponse. Journal of Official Statistics, 27, 153–180.

Andridge

R. R.

Little

R. J. A.

(2020). Proxy pattern-mixture analysis for a binary variable subject to nonresponse. Journal of Official Statistics, 36(3), 703–728.

Andridge

R. R.

Thompson

K. J.

(2015). Assessing nonresponse bias in a business survey: Proxy pattern-mixture analysis for skewed data. The Annals of Applied Statistics, 9(4), 2237–2265.

Binder

D. A.

(1983). On the variances of asymptotically normal estimators from complex surveys. International Statistical Review, 51(3), 279–292.

Brick

J. M.

Tourangeau

(2017). Responsive survey design for reducing nonresponse bias. Journal of Official Statistics, 33(3), 735–752.

Deming

W. E.

Stephan

F. F.

(1940). On a least squares adjustment of a sampled frequency table when the expected marginal totals are known. The Annals of Mathematical Statistics, 11, 427–444.

Groves

R. M.

(2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70, 646–675.

Hambleton

R. K.

Swaminathan

Rogers

H. J.

(1991). Fundamentals of item response theory. Sage Press.

Hedlin

(2020). Is there a “safe area” where the nonresponse rate has only a modest effect on bias despite non-ignorable nonresponse? International Statistical Review, 88(3), 642–657.

10.

Kreuter

Olson

Wagner

Yan

Ezzati-Rice

T. M.

Casas-Cordero

Lemay

Peytchev

Groves

R. M.

Raghunathan

T. E.

(2010). Using proxy measures and other correlates of survey outcomes to adjust for non-response: Examples from multiple surveys. Journal of the Royal Statistical Society, Series A, 173, 389–407.

11.

Little

R. J. A.

(1994). A class of pattern-mixture models for normal incomplete data. Biometrika, 81, 471–483.

12.

Little

R. J. A.

Carpenter

Lee

(2022). A comparison of three popular methods for handling missing data: Complete-case analysis, weighting and multiple imputation. Sociological Methods & Research. https://doi.org/10.1177/00491241221113873

13.

Little

R. J. A.

David

(1983). Weighting adjustments for nonresponse in panel survey. U.S. Department of Commerce, Bureau of the Census. https://www2.census.gov/library/working-papers/1987/demo/SEHSD-WP1987-04.pdf

14.

Little

R. J. A.

Rubin

D. B.

(2019). Statistical analysis with missing data (3rd ed.). Wiley.

15.

Little

R. J. A.

Vartivarian

(2005). Does weighting for nonresponse increase the variance of survey means? Survey Methodology, 31, 161–168.

16.

Little

R. J. A.

West

B. T.

Boonstra

P. S.

(2020). Measures of the degree of departure from ignorable sample selection. Journal of Survey Statistics and Methodology, 8, 932–964.

17.

Montaquilia

J. M.

Brick

J. M.

(2009). The use of an array of methods to evaluate nonresponse bias in an ongoing survey program. ISI Proceedings. https://2009.isiproceedings.org/A5%20Docs/0215.pdf

18.

National Academies of Sciences, Engineering, and Medicine. (2017). Innovations in federal statistics: Combining data sources while protecting privacy. The National Academies Press. https://doi.org/10.17226/24652

19.

National Center for Education Statistics. (2013). Early Childhood Longitudinal Study, Kindergarten Class of 2010–11. https://nces.ed.gov/ecls/dataproducts.asp#K-5

20.

Rubin

D. B.

(1976). Inference and missing data. Biometrika, 63(3), 581–592.

21.

Rubin

D. B.

(1987). Multiple imputation for nonresponse in surveys. John Wiley & Sons.

22.

Särndal

C.-E.

Lundquist

(2014). Accuracy in estimation with nonresponse: A function of the degree of imbalance and degree of explanation. Journal of Survey Statistics and Methodology, 2, 361–387.

23.

Schouten

Cobben

Bethlehem

(2009). Indicators for the representativeness of survey response. Survey Methodology, 35, 101–113.

24.

Heeringa

Johnson

Little

R. J. A.

Liu

Pfeffer

Raghunathan

. (2022). Multiple imputation with massive data: An application to the panel study of income dynamics. Journal of Survey Statistics and Methodology. https://doi.org/10.1093/jssam/smab038

25.

Reiter

J. P.

Hillygus

(2015). Semi-parametric selection models for potentially non-ignorable attrition in panel studies with refreshment samples. Political Analysis, 23, 92–112.

26.

Reiter

J. P.

Hillygus

(2016). Bayesian latent pattern mixture models in panel studies with refreshment samples. The Annals of Applied Statistics, 10, 118–143.

27.

Tibshirani

(1996). Regression shrinkage and selection via the LASSO. Journal of the Royal Statistical Society. Series B (Methodological), 58(1), 267–288.

28.

Tourangeau

Nord

Lê

Sorongon

A. G.

Hagedorn

M. C.

Daly

Mulligan

G. M

. (2013). User’s manual for the ECLS-K:2011 Kindergarten data file and electronic codebook. National Center for Education Statistics. https://nces.ed.gov/pubs2015/2015074.pdf

29.

Wagner

(2010). The fraction of missing information as a tool for monitoring the quality of survey data. Public Opinion Quarterly, 74, 223–243.

30.

West

B. T.

Little

R. J. A.

Andridge

R. R.

Boonstra

P. S.

Ware

E. B.

Pandit

Alvarado-Leiton

. (2021). Assessing selection bias in regression coefficients estimated from non-probability samples, with applications to genetics and demographic surveys. Annals of Applied Statistics, 15(3), 1556–1581.

31.

Yucel

R. M

. (2011). State of the multiple imputation software. Journal of Statistical Software, 45(1), v45/i01.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.01 MB

A Case Study of Nonresponse Bias Analysis in Educational Assessment Surveys

Abstract

Keywords

Get full access to this article

References

Supplementary Material