Abstract
The role of statistical tolerance intervals for developing ratio edit tolerances in a parametric setup is investigated. The performance of the methodology is assessed for the normal and Weibull distributions. The numerical results show that in terms of Type I and Type II errors, statistical tolerance intervals exhibit better performance compared to other ratio edit procedures available in the literature. The methodology is illustrated using 2010 and 2011 data from the Annual Survey of Manufacturers.
References
1.
Aggarwal, C.C. 2013. Outlier Analysis. New York: Springer. Bain, L.J. and M. Engelhardt. 1981. “Simple Approximate Distributional Results for Confidence and Tolerance Limits for the Weibull Distribution Based on Maximum Likelihood Estimators.” Technometrics 23: 15-20. DOI: http://dx.doi.org/10.1080/ 00401706.1981.10486231
2.
Barnett, V. and T. Lewis. 1994. Outliers in Statistical Data, 3rd ed. Wiley Series in Probability and Mathematical Statistics. Chichester: John Wiley & Sons.
3.
Belcher, R. 2003. “Application of the Hidiroglou-Berthelot Method of Outlier Detection for Periodic Business Surveys.” In Proceedings of the Survey Methods Section: Statistical Society of Canada Annual Meeting, June, 2003. 25-30 Halifax, Nova Scotia, Canada. Available at: http://www.ssc.ca/survey/documents/ SSC2003_R_Belcher.pdf
4.
Chawla, S. and A. Gionis. 2013. “k-means-: A Unified Approach to Clustering and Outlier Detection.” In Proceedings of the 2013 SIAM International Conference on Data Mining: Society for Industrial and Applied Mathematics, May, 2013. 187-197 Austin, Texas, USA. Available at http://epubs.siam.org/doi/pdf/10.1137/1.9781611972832.21
5.
Cornett, E., J.F. McLaughlin, and C. R. Hogue. 2006. “A Comparison of Two Ratio Edit Methods for the Annual Survey of Government Finances.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, August, 2006. 2878-2883 Seattle, WA, USA. Available at: http://www.amstat.org/sections/SRMS/ Proceedings/y2006/Files/JSM2006-000199.pdf
6.
Franklin, S., S. Thomas, and M. Brodeur. 2000. “Robust Multivariate Outlier Detection Using Mahalanobis' Distance and Modified Stahel-Donoho Estimators.” In Proceedings of the Second International Conference on Establishment Surveys (ICES-II), Survey Methods for Businesses, Farms, and Institutions, June, 2000. 697-706 Buffalo, NY, USA. Available at: http://www.amstat.org/meetings/ices/2000/proceedings/S33.pdf
7.
Ghosh-Dastidar, B. and J.L. Schafer. 2006. “Outlier Detection and Editing Procedures for Continuous Multivariate Data.” Journal of Official Statistics 22: 487-506.
8.
Hidiroglou, M.A. and J.-M. Berthelot. 1986. “Statistical Editing and Imputation for Periodic Business Surveys.” Survey Methodology 12: 73-83.
9.
Hido, S., Y. Tsuboi, H. Kashima, M. Sugiyama, and T. Kanamori. 2011. “Statistical Outlier Detection Using Direct Density Ratio Estimation.” Knowledge and Information Systems 26: 309-336. DOI: http://dx.doi.org/10.1007/s10115-010-0283-2 Hoaglin, D.C., B. Iglewicz, and J.W. Tukey. 1986. “Performance of Some Resistant Rules for Outlier Labeling.” Journal of the American Statistical Association 81: 991-999.
10.
DOI: http//dx.doi.org/10.1080/01621459.1986.10478363 Iglewicz, B. and D.C. Hoaglin. 1993. How to Detect and Handle Outliers, vol. 16. Milwaukee, WI: American Society for Quality Control.
11.
Kokic, P.N. and P.A. Bell. 1994. “Optimal Winsorizing Cutoffs for a Stratified Finite Population Estimator.” Journal of Official Statistics 10: 419-435.
12.
Krishnamoorthy, K. and T. Mathew. 2009. Statistical Tolerance Regions: Theory, Applications, and Computation. Hoboken, NJ: Wiley. Journal of Official Statistics Latouche, M. and J.-M. Berthelot. 1992. “Use of a Score Function to Prioritize and Limit Recontacts in Editing Business Surveys.” Journal of Official Statistics 8: 389-400.
13.
Montgomery, D.C. 2013. Introduction to Statistical Quality Control, 7th ed. Hoboken, NJ: Wiley. R Development Core Team. 2014. R: A Language and Environment for Statistical Computing, R Foundation for Statistical Computing, Vienna, Austria. Available at: http://www.R-project.org/ ISBN 3-900051-07-0 (accessed February 13, 2015) Rais, S. 2008. “Outlier Detection for the Consumer Price Index.” In Proceedings of the Survey Methods Section: Statistical Society of Canada Annual Meeting, May, 2008. 1-10 Ottawa, Ontario, Canada. Available at: http://www.ssc.ca/survey/documents/ SSC2068_5_Rais.pdf.
14.
Rivest, L.-P. and M. Hidiroglou. 2004. “Outlier Treatment for Disaggregated Estimates.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, August, 2004. 4248-4256 Toronto, Ontario, Canada. Available at: http://www.amstat.stat.org/sections/SRMS/Proceedings/y2004/files/Jsm2004-000149.pdf
15.
Rousseeuw, P. and A.M. Leroy. 2003. Robust Regression and Outlier Detection. Hoboken, NJ: Wiley Series in Probability and Mathematical Statistics.
16.
Shewhart, W.A. 1939. Statistical Method from the Viewpoint of Quality Control. Washington, DC: Dover.
17.
Sigman, R.S. 2002. “Statistical Methods Used to Detect Cell-Level and Respondent-Level Outliers in the 2002 Economic Census of the Services Sector.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, August 2002. 3566-3573 Minneapolis, MN, USA. Available at: https://www.amstat.org/ sections/SRMS/Proceedings/y2005/Files/JSM2005-000465.pdf
18.
Tambay, J.-L. 1988. “An Integrated Approach for the Treatment of Outliers in Sub-Annual Economic Surveys.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 229-234. Available at: http://www.amstat.org/ sections/SRMS/Proceedings/papers/1988-040.pdf
19.
Thompson, K.J. 1999. “Ratio Edit Tolerance Development Using Variations of Exploratory Data Analysis (EDA) Resistant Fences Methods.” In Proceedings of the 1999 Federal Committee on Statistical Methodology Research Conference, November 1999. 1-10. Arlington, VA, USA. Available at: https://fcsm.sites.usa.gov/ files/2014/05/VII-B_Thompson_FCSM1999.pdf.
20.
Thompson, K.J. 2007. “Investigation of Macro Editing Techniques for Outlier Detection in Survey Data.” In Proceedings of the Third International Conference on Establishment Surveys (ICES-III), Survey Methods for Businesses, Farms, and Institutions, June 2007. 1186-1193. Montreal, Quebec, Canada. Available at http://www.amstat.org/meetings/ ices/2007/proceedings/ICES2007-000071.pdf.
21.
Thompson, K.J. and S.A. Adeshiyan. 2003. “Data Quality Effects of Alternative Edit Parameters.” Journal of Data Science 1: 1-25.
22.
Thompson, K.J. and R.S. Sigman. 1999. “Statistical Methods for Developing Ratio Edit Tolerances for Economic Data.” Journal of Official Statistics 15: 517-535.
23.
Tukey, J.W. and D.H. McLaughlin. 1963. “Less Vulnerable Confidence and Significance Procedures for Location Based on a Single Sample: Trimming/Winsorization 1.” Sankhyā: The Indian Journal of Statistics (Series A) 25: 331-352.
24.
Young, D.S. 2010. “Tolerance: An R package for Estimating Tolerance Intervals.” Journal of Statistical Software 36: 1-39. Available at: http://www.jstatsoft.org/v36/i05/ (accessed February 13, 2015) Yuen, K.-V. and H.-Q. Mu. 2012. “A Novel Probabilistic Method for Robust Parametric Identification and Outlier Detection.” Probabilistic Engineering Mechanics 30: 48-59. DOI: http://dx.doi.org/10.1016/j.probengmech.2012.06.002