An iterative refinement approach for data cleaning 1

Abstract

Data cleaning is an important step in the data mining process. Successful data mining applications require good quality data. In this paper, we propose a data cleaning technique that smoothes out a substantial amount of attribute noise and handles missing attribute values as well. Our approach is inspired by the Expectation-Maximization (EM) algorithm. It iteratively refines each attribute-value using a predictor constructed from the previously refined values (known values in the first iteration). We demonstrate the effectiveness of our technique in smoothing out attribute noise and corroborate the efficacy of our technique by showing improved classification accuracy on a number of real world data sets from UCI repository [2]. Moreover, we show that our technique can easily be adapted to fill up missing attribute-values in classification problems more effectively than other standard approaches.