Review of Richard Valliant and Jill A. Dever’s Survey Weights: A Step-by-Step Guide to Calculation

Abstract

In this article, I review the Stata Press publication Survey Weights: A Step-by-Step Guide to Calculation by Valliant and Dever (2018).

1 Introduction

Several years ago in South Africa, I had just completed the first two days of a three-day workshop on sampling design and the analysis of complex sample survey data. A workshop participant from one of southern Africa’s national statistical offices approached me and asked whether there was any single published source that would guide him through all the practical steps required to design, select, weight, and analyze survey data. After doing a quick mental inventory, I answered no; to the best of my knowledge, no single source could reliably provide all the statistical theory and particularly the practical knowledge and tools required for the steps beginning with sample design and progressing to weighting and analysis of the survey data. Early the next morning, that same student rushed up to me, laptop in hand, to show me the webpage for a new book, Practical Tools for Designing and Weighting Survey Samples—a text authored by Rick Valliant, Jill Dever, and Frauke Kreuter (2013). I never learned if that text measured up to his true need, but it drove home the point to me that our field has historically lacked such comprehensive publications on practical tools. That void of practical guidance on survey weighting is now further addressed in Survey Weights: A Step-by-Step Guide to Calculation, a Stata Press publication that is an abridged, easily accessible adaptation of Valliant, Dever, and Kreuter (2013) for Stata users.

2 Motivation for the book

From the perspective of the survey data user, for many decades statisticians specializing in sampling design and related statistical methodologies were exclusively members of a team of data producers. Sample designs, selection procedures, sample weighting, and imputation of item missing data were the domain of the specialists typically working with in-house and often “one-off” software tools. Beginning in the 1990s, standard software tools for sample selection appeared in major statistical software systems followed a decade later by software for imputation of item missing data and analysis of the imputed datasets. During this same period, Stata and other major software systems made important advances in supporting data users’ ability to conduct weighted estimation of population statistics and standard errors of the estimates. Although the past 30 years have seen major advances in theory and methods for weighting survey data with many statistical publications on weight calculation and weighted analysis methods, there has been no single-source treatment of the subject as comprehensive as Valliant, Dever, and Kreuter (2013) and Survey Weights: A Step-by-Step Guide to Calculation.

3 Contents

The subject matter of Survey Weights: A Step-by-Step Guide to Calculation goes far beyond covering techniques for calculating weights. This new guide provides the reader with a comprehensive overview of the role of weighting in population estimation from survey data (including Stata code to produce survey weights) and illustrates how weights are incorporated in the estimation of standard errors for population estimates. It clearly explains weighting concepts and methods. It does not ignore supporting theory for the methods but does leave details for additional reading and study of carefully cited references in the statistical literature. In addition to the extensive bibliography, the guide includes many useful reader support features, including a glossary of terms and complete online access to the Stata code and datasets used in the worked examples and illustrations included in the text.

Chapter 1 addresses the bias and variance reduction roles that weights play in population estimation and inference. It covers key features in the taxonomy of the wide range of probability sample designs that are used in practice–randomization in selection, stratification and clustering of elements, and multistage sampling. This initial overview chapter also introduces the reader to the important relationship between the weighting and imputation approaches to unit nonresponse and item missing data in surveys. It closes with a practical discussion of the importance of careful case disposition coding for sample cases and presents a step-by-step flow diagram for a typical survey weight calculation process.

Chapters 2–4 cover the basic sequence of weight calculation steps—base weight computation, adjustment for nonresponse, and calibration to external population controls. With careful record keeping during sample design and selection, calculating base weights (sometimes labeled “selection weights”) is primarily an accounting function that follows algebraic rules for computing the probability that a population element is selected to the sample. The final sections of chapter 2 provide guidance regarding two sample-selection and implementation features (replicated sample release, unknown eligibility of frame elements) that can require additional adjustments to the base weights after the survey field period is complete.

While computing base weights for sample cases is highly rule driven, the step of creating an adjustment factor to attenuate the potential bias arising from differential nonresponse or nonobservation of selected sample elements is model dependent. Chapter 3 opens with a general introduction to nonresponse and various underlying mechanisms (missing completely at random, missing at random, and missing not at random) and then transitions to in-depth coverage with Stata examples of how to derive effective nonresponse adjustments using three methods for modeling the conditional probability that a selected sample element responded to the survey. It describes traditional weighting class methods, response propensity modeling, and both old (classification and regression trees) and newer (random forest, boosting) recursive partitioning and classification approaches to estimating response probabilities and illustrates the Boosting approach using an example.

Provided that an accurate and precise external source of data is available for key variables, the final step in the standard weight calculation sequence is the calibration of the nonresponse-adjusted base weights to population proportions or totals for these key variables. In chapter 4, the authors cover the major calibration methods, including poststratification, raking (iterative proportional fitting), and generalized regression. The chapter’s text and examples clearly describe each of these methods and provide readers with the guidance needed to apply these calibration weighting methods to their survey datasets.

In chapter 4 (example 4.7) and again later in chapter 5, the authors strongly warn the reader against ignoring the impact that the separate weighting steps (base weights, nonresponse adjustment, and calibration) exert on the variances of weighted sample estimates. The message to the reader is that estimated sampling errors for survey estimates may be seriously biased if the variance estimation method does not reflect variability in the nonresponse adjustment or fails to distinguish the variance-reducing potential of the calibration step. Their argument certainly has theoretical support, and empirically it is easy to demonstrate cases where ignoring the separate stages of weighting leads to biased (often overestimated) standard errors. In my work experience, the potential for such bias can be important in surveys with highly differential nonresponse across major domains or in educational and business or nonhousehold surveys where the calibration step is highly effective at reducing variances. Following the authors’ guidance to the letter implies that the sequence of weighting steps will be performed for the full sample and repeated separately for r = 1,…, R replicate samples and that a replicated variance estimator (jackknife repeated replication [JRR], balanced repeated replication [BRR], or bootstrap) will be used to estimate standard errors of survey estimates.

A guide to the calculation of survey weights should never lose sight of the purpose behind weighted estimation or how the weights are correctly applied in survey estimation and inference. Chapter 5 covers major linearization and replication approaches to the estimation of standard errors for weighted survey estimates. Historically, Stata and other major software systems chose the first-order Taylor series linearization method as the default for estimation of standard errors for estimates from complex sample survey data. Capabilities for replicated variance estimation using the full sample weight and replicate weighting options for JRR, BRR, and ultimately bootstrap replication are available in Stata, providing survey analysts with a full range of robust options for computing standard errors for weighted estimates. Given its traditional importance and wide application as the default method in Stata commands that apply the svy: prefix, section 5.3 on linearization approaches to variance estimation is brief. This brevity is consistent with the authors’ theme that whenever possible, replicate weights—derived separately by repeating each weighting step for replicate sample cases—should be constructed and employed in JRR, BRR, or bootstrap estimation of standard errors for weighted sample estimates (section 5.6). Section 5.4 provides an excellent introduction to variance estimation using JRR, BRR, or the bootstrap and includes example Stata syntax for specifying user-provided replicate weights.

The final three chapters are an extremely valuable addition to this guide to calculation and use of weights in Stata analyses of survey data. Increasingly, researchers, media, policy analysts, and even official statisticians are turning to nonprobability sample sources of observational data to estimate population characteristics or to make inferences concerning population behaviors and associated outcomes. Chapter 6 draws on methodologies described in Elliott and Valliant (2017) to develop quasirandomization weighting and model-based approaches to weighted estimation from nonprobability sample surveys and illustrates their implementation in Stata using data from an optin volunteer Internet panel. Chapter 7 covers a number of special topics in weighted analysis that survey analysts may not encounter often but should be acquainted with: normalization of weights, multiple weights for different stages or phases of data collection, and compositing of weights when combining two or more sources of survey data. The final section of chapter 7 provides excellent coverage of the role of survey weights in regression modeling. It uses Stata code and example data to illustrate techniques for empirically comparing weighted and unweighted fits of linear and logistic regression models and performing a statistical check on whether the introduction of weights alters the model fit to the survey data.

Although not a serious omission, chapter 7 might have included coverage of the special topic of how survey weights are constructed and applied for multilevel analyses in Stata (using the mixed command).

Throughout the text, the authors repeatedly emphasize the importance of quality controls and checks on the process of weight calculation and the application of weights in analysis. Chapter 8 adds a final important emphasis of the need for careful attention throughout the weight calculation process, including accurate disposition coding, carefully constructed and annotated weight calculation code, thorough checks against external benchmarks, clear technical documentation of the entire weight calculation process, and a guide to users on the application of the weights in analysis.

4 Conclusion

Survey Weights: A Step-by-Step Guide to Calculation makes a major contribution to the field by organizing and presenting the highly diverse and widely distributed literature on the subject of survey weight calculation in a single well-written volume.

For survey statisticians and specialists, this text is an excellent reference on exactly how to develop and apply specific survey weighting methods in Stata. Instructors for courses on statistical methods for survey data could well choose this short guide as the text for a course module on weighting and weighted estimation and inference. Stata users who may not have training or experience in sample design and weighting will find this guide to be an excellent introduction to the subject—hopefully one that opens the lid of the black box and provides real insight into the mystery that often surrounds the subject of survey weights.

Supplemental Material

Supplemental Material, gr0079 - Review of Richard Valliant and Jill A. Dever’s Survey Weights: A Step-by-Step Guide to Calculation

Supplemental Material, gr0079 for Review of Richard Valliant and Jill A. Dever’s Survey Weights: A Step-by-Step Guide to Calculation by Steven G. Heeringa in The Stata Journal