Data deduplication is process of discovering multiple representations of same entity in an information system. Blocking has been a benchmark technique for avoiding the pair-wise record comparisons in data deduplication. Standard blocking (SB) aims at putting the potential duplicate records in the same block on the basis of a blocking key. Afterwards, the detailed comparisons are made only among the records residing in the same block. The selection of blocking key is a tedious process that involves exponential alternatives. The outcome of SB varies considerably with a change in blocking key. To this end, we have proposed a robust blocking technique called Locality Sensitive Blocking (LSB) that does not require the selection of blocking key. The experimental results show an increase of up to 0.448 in F-score as compared with SB. Furthermore, it is found that LSB is more robust towards blocking parameters and data noise.
MuellerTLaskeyJBaillieKet al. Opportunities and challenges when using record linkage of routinely collected electronic health care data to evaluate outcomes of systemic anti-cancer treatment in clinical practice. Health Informatics J. Epub ahead of print 23February2022. DOI: 10.1177/14604582221077055.
2.
HarronKDibbenCBoydJet al. Challenges in administrative data linkage for research. Big Data Soc. Epub ahead of print 5December2017. DOI: 10.1177/2053951717745678.
3.
ZapounidouSSfakakisMPapatheodorouC. Representing and integrating bibliographic information into the Semantic Web: a comparison of four conceptual models. J Inf Sci2017; 43(4): 525–553.
4.
XinXWuJHuangLet al. Effect of multi-channel integration on customer purchase intention. J Comput Inform Syst. Epub ahead of print 19August2021. DOI: 10.1080/08874417.2021.1965052.
5.
ElmagarmidAKIpeirotisPGVerykiosVS. Duplicate record detection: a survey. IEEE T Knowl Data En2007; 19(1): 1–16.
6.
NaumannFHerschelM. An introduction to duplicate detection (Synthesis lectures on data management), vol. 2, No. 1. San Rafael, CA: Morgan & Claypool Publishers, 2010, pp. 1–87.
7.
VarolCHariS. Detecting near-duplicate text documents with a hybrid approach. J Inf Sci2015; 41(4): 405–414.
8.
TaniguchiS. Duplicate bibliographic record detection with an OCR-converted source of information. J Inf Sci2013; 39(2): 153–168.
9.
BindingCTudhopeDVlachidisA. A study of semantic integration across archaeological data and reports in different languages. J Inf Sci2019; 45(3): 364–386.
RahmEDoHH. Data cleaning: problems and current approaches. IEEE Data Eng Bull2000; 23(4): 3–13.
12.
LeeYWPipinoLLFunkJDet al. Journey to data quality. Cambridge: The MIT Press, 2006, pp. 1–26.
13.
HerzogTNScheurenFJWinklerWE. Data quality and record linkage techniques. 1st ed. New York: Springer Publishing Company, 2007, pp. 7–26.
14.
ImranMAhmadA. Enhancing data quality to mine credible patterns. J Inf Sci. Epub ahead of print 6June2021. DOI: 10.1177/01655515211013693.
15.
TalburtJR. Entity resolution and information quality. Amsterdam: Elsevier, 2011, pp. 39–61.
16.
ChristenP. Data matching: concepts and techniques for record linkage, entity resolution, and duplicate detection. Berlin: Springer Science+Business Media, 2012.
17.
WangXHuangLXuXet al. A solution for data inconsistency in data integration. J Inf Sci Eng2011; 27(2): 681–695.
18.
IbrahimHKarasnehYMirabiMet al. An automatic domain independent schema matching in integrating schemas of heterogeneous relational databases. J Inf Sci Eng2014; 30(5): 1505–1536.
19.
NewcombeHB. Handbook of record linkage: methods for health and statistical studies, administration, and business. Oxford: Oxford University Press, 1988.
20.
KılınçD. An accurate toponym-matching measure based on approximate string matching. J Inf Sci2016; 42(2): 138–149.
21.
HanJKamberMPeiJ. Data preprocessing. In: HanJKamberMPeiJ (eds) Data mining: concepts and techniques (The Morgan Kaufmann series in data management systems). 3rd ed. Waltham, MA: Morgan Kaufmann, 2011, pp. 83–124.
22.
AlaouiSSFarhaouiYAksasseB. A comparative study of the four well-known classification algorithms in data mining. In: Proceedings of the international conference on advanced information technology, services and systems, Tangier, 14–15 April 2017, pp. 362–373.
23.
KöpckeHRahmE. Frameworks for entity matching: a comparison. Data Knowl Eng2010; 69(2): 197–210.
24.
GiangPH. A machine learning approach to create blocking criteria for record linkage. Health Care Manag Sci2015; 18(1): 93–105.
25.
HernándezMAStolfoSJ. The merge/purge problem for large databases. ACM SIGMOD Rec1995; 24: 127–138.
26.
ChristenP. A survey of indexing techniques for scalable record linkage and deduplication. IEEE T Knowl Data En2012; 24(9): 1537–1555.
27.
PapadakisGSvirskyJGalAet al. Comparative analysis of approximate blocking techniques for entity resolution. Proc VLDB Endow2016; 9(9): 684–695.
28.
AnsolabehereSHershED. ADGN: an algorithm for record linkage using address, date of birth, gender, and name. Stat Public Policy2017; 4(1): 1–10.
29.
WhangSEMenestrinaDKoutrikaGet al. Entity resolution with iterative blocking. In: Proceedings of the 2009 ACM SIGMOD international conference on management of data, Providence, RI, 29 June–2 July 2009, pp. 219–232. New York: ACM.
30.
GalhotraSFirmaniDSahaBet al. Efficient and effective ER with progressive blocking. VLDB J2021; 30(4): 537–557.
31.
LehtiPFankhauserP. A precise blocking method for record linkage. In: Proceedings of the international conference on data warehousing and knowledge discovery, Copenhagen, 22–26 August 2005, pp. 210–220. Berlin; Heidelberg: Springer-Verlag.
32.
YanSLeeDKanM-Yet al. Adaptive sorted neighborhood methods for efficient record linkage. In: Proceedings of the 7th ACM/IEEE-CS joint conference on digital libraries, Vancouver, BC, Canada, 18–23 June 2007, pp. 185–194. New York: ACM.
33.
DraisbachUNaumannFSzottSet al. Adaptive windows for duplicate detection. In: Proceedings of the 2012 IEEE 28th international conference on data engineering, Arlington, VA, 1–5 April 2012, pp. 1073–1083. New York: IEEE.
34.
QuantinCBinquetCBourquardKet al. Which are the best identifiers for record linkage?Med Inform Internet Med2004; 29(3–4): 221–227.
35.
PrasadKHChaturvediSFaruquieTAet al. Automated selection of blocking columns for record linkage. In: Proceedings of the 2012 IEEE international conference on service operations and logistics, and informatics, Suzhou, China, 8–10 July 2012, pp. 78–83. New York: IEEE.
36.
SouzaLMuraiFDa SilvaAPCet al. Automatic identification of best attributes for indexing in data deduplication. In: Proceedings of the 12th Alberto Mendelzon international workshop on foundations of data management, Cali, 21–25 May 2018.
37.
KenigBGalA. MFIBlocks: an effective blocking algorithm for entity resolution. Inform Syst2013; 38(6): 908–926.
38.
BilenkoMKamathBMooneyRJ. Adaptive blocking: learning to scale up record linkage. In: Proceedings of the 6th international conference on data mining (ICDM’06), Hong Kong, China, 18–22December2006, pp. 87–96. New York: IEEE.
39.
KejriwalMMirankerDP. An unsupervised algorithm for learning blocking schemes. In: Proceedings of the 2013 IEEE 13th international conference on data mining, Dallas, TX, 7–10 December 2013, pp. 340–349. New York: IEEE.
40.
RamadanBChristenP. Unsupervised blocking key selection for real-time entity resolution. In: CaoTLimE-PZhouZ-Het al. (eds) Advances in knowledge discovery and data mining. Cham: Springer, 2015, pp. 574–585.
41.
MichelsonMKnoblockCA. Learning blocking schemes for record linkage. In: Proceedings of the 21st national conference on artificial intelligence, Boston, MA, 16–20 July 2006, vol. 1, pp. 440–445. Palo Alto, CA: AAAI Press.
42.
O’HareKJurek-LoughreyADe CamposC. An unsupervised blocking technique for more efficient record linkage. Data Knowl Eng2019; 122: 181–195.
43.
LeskovecJRajaramanAUllmanJD. Mining of massive datasets. Cambridge: Cambridge University Press, 2011.
SohailAYousafMM. A proficient cost reduction framework for de-duplication of records in data integration. BMC Med Inform Decis2016; 16: 42.
46.
DraisbachUNaumannF. A generalization of blocking and windowing algorithms for duplicate detection. In: Proceedings of the 2011 international conference on data and knowledge engineering (ICDKE), Milan, 6 September 2011, pp. 18–24. New York: IEEE.
47.
ChristenP. Febrl – an open source data cleaning, deduplication and record linkage system with a graphical user interface. In: Proceedings of the 14th ACM SIGKDD international conference on knowledge discovery and data mining, Las Vegas, NV, 24–27 August 2008, pp. 1065–1068. New York: ACM.
