Evaluation of automatic editing and imputation to inform establishment statistics

Abstract

The United States (U.S.) Department of Agriculture's National Agricultural Statistics Service (NASS) conducts hundreds of surveys each year to estimate nearly every facet of U.S. agriculture. Once data have been collected, responses are then reviewed and edited by NASS's regional field offices for outliers, edit failures, and item imputation. Comparable to other National Statistical Institutes, historically the review process and item-level imputations were done though a manual, interactive process, which can be time consuming and costly. NASS has developed a generalized system called IDEAL (Imputation, Deterministic Edits, Automation and Logic) to automate the imputation and error correction occurring after data collection, but before the outlier review process. Implementation of the first phase of IDEAL, called JIMMY, has occurred on multiple production surveys to inform establishment statistics. In this paper, efficiency and data quality of edits and imputation produced by JIMMY are examined and evaluated. Furthermore, results of using the new system in multiple production surveys are highlighted, and the successes and challenges of using the system in the future are discussed.