Sage Journals: Discover world-class research

Abstract

The United States (U.S.) Department of Agriculture's National Agricultural Statistics Service (NASS) conducts hundreds of surveys each year to estimate nearly every facet of U.S. agriculture. Once data have been collected, responses are then reviewed and edited by NASS's regional field offices for outliers, edit failures, and item imputation. Comparable to other National Statistical Institutes, historically the review process and item-level imputations were done though a manual, interactive process, which can be time consuming and costly. NASS has developed a generalized system called IDEAL (Imputation, Deterministic Edits, Automation and Logic) to automate the imputation and error correction occurring after data collection, but before the outlier review process. Implementation of the first phase of IDEAL, called JIMMY, has occurred on multiple production surveys to inform establishment statistics. In this paper, efficiency and data quality of edits and imputation produced by JIMMY are examined and evaluated. Furthermore, results of using the new system in multiple production surveys are highlighted, and the successes and challenges of using the system in the future are discussed.

Keywords

Automation editing imputation outliers survey data

Get full access to this article

View all access options for this article.

References

de Waal

Pannekoek

Scholtus

. Handbook of statistical data editing and imputation. In: Wiley handbooks in survey methodology. Hoboken, New Jersey, United States: John Wiley & Sons, Inc, 2011, pp. ix.

Johanson

. Banff automated edit and imputation on a hog survey. In: Proceedings of the fourth international conference of establishment surveys, Montréal, Canada, 11–14 June 2012, Alexandria, Virgina, United States: American Statistical Association, 2012.

Kozak

. The banff system for automated editing and imputation. In: Statistical Society of Canada annual meeting, Saskatoon, Canada, 12–15 June 2005, 2005.

Barboza

. Utilizing automated statistical edit changes in significance editing. In: Proceedings of the 2011 joint statistical meetings, 2011.

Miller

Young

. Imputation at the national agricultural statistics service. In: United Nations statistical commission and economic commission for Europe, conference for European statisticians, work session on statistical data editing, Budapest, Hungary, 14–16 September 2015, Geneva, Switzerland: United Nations statistical commission and economic commission for Europe, 2015.

Miller

. Growing a modern editing and imputation system. In: 2021 Federal committee on statistical methodology conference, 2021.

Lipke

Miller

Wagner

, et al. Growing a modern edit and imputation system. In: United Nations statistical commission and economic commission for Europe, conference for European statisticians, work session on statistical data editing, Virtual, 3–5 October 2022, 2022.

van der Loo

MPJ

de Jonge

. Data validation infrastructure for R. J Stat Software 2021; 97: 1–31.

de Jonge

van der Loo

. errorlocate: Locate errors with validation rules. R package version 1.1.1, https://CRAN.R-project.org/package=errorlocate (2023).

10.

de Jonge

van der Loo

. validatetools: Checking and simplifying validation rule sets. R package version 0.5.2, https://CRAN.R-project.org/package=validatetools (2023).

11.

van der Loo

de Jonge

. _dcmodify: Modify data using externally defined modification rules_. R package version 0.9.0, https://CRAN.R-project.org/package=dcmodify (2024).

12.

van der Loo

de Jonge

. _deductive: Data correction and imputation using deductive methods_. R package version 1.0.1, https://CRAN.R-project.org/package=deductive (2025).

13.

van der Loo

. _simputation: Simple imputation_. R package version 0.2.9, https://CRAN.R-project.org/package=simputation (2024).

14.

R Core Team. _R: A language and environment for statistical computing_. R Foundation for Statistical Computing, Vienna, Austria, https://www.R-project.org/ (2024).

15.

Laird

Maiwurm

Lipke

, et al. Growing and testing a modern edit and imputation system. In: International conference on agricultural statistics, Washington DC, United States, 17–19 May 2023, Washington DC, United States: United States Department of Agriculture (USDA), 2023.

16.

Fellegi

Holt

. A systematic approach to automatic edit and imputation. J Am Stat Assoc 1976; 71: 17–35.

Evaluation of automatic editing and imputation to inform establishment statistics

Abstract

Keywords

Get full access to this article

References