Focus of this research work is optimizing the deduplication system by adjusting the pertinent factors in content defined chunking (CDC) to identify as the key ingredients by declaring chunk cut-points and efficient fingerprint lookup using bucket based index partitioning. For efficient chunking, proposed Differential Evolution (DE) algorithm based approach is optimized Two Thresholds Two Divisors (TTTD-P) CDC algorithm where significantly it reduces the number of computing operations by using single dynamic optimal parameter divisor D with optimal threshold value exploiting the multi-operations nature of TTTD. Therefore, proposed DE based TTTD-P optimize chunking to maximize chunking throughput with increased deduplication ratio (DR); and bucket indexing approach reduces hash values judgment time to identify and declare redundant chunk about 16 times faster than Rabin CDC, 5 times than Asymmetric Extremum (AE) CDC, 1.6 times than FAST CDC. Experimental results comparative analysis reveal that TTTD-P using fast BUZ rolling hash function with bucket indexing on Hadoop Distributed File System (HDFS) provide a comparatively maximum redundancy detection with higher throughput, higher deduplication ratio, lesser computation time and very low hash values comparison time as being best distributed deduplication for big data storage systems.
ChenM., MaoS. and LiuY., Big data: A survey, Mobile Networks and Applications Journal, Springer19(2) (2014), 171–209.
2.
ManyikaJ., ChuiM., BrownB., BughinJ., DobbsR., RoxburghC. and ByersA.H., Big data: The next frontier for innovation, competition, and productivity, McKinsey Company (2011), 1–156.
3.
StornR., Differential Evolution: A simple and efficient heuristic strategy for global optimization over continuous spaces, Journal of Global Optimization11 (1997), 341–359.
4.
MinJ., YoonD. and WonY., Efficient deduplication techniques for modern backup operation, IEEE Trasactions on Computers60(6) (2011), 824–840.
5.
GantzJ. and ReinselD., The digital universe decade are you ready? IDC White Paper, 2011.
6.
BiggarH., Experiencing data deduplication: Improving efficiency and reducing capacity requirements, White Paper, The Enterprise Strategy Group, 2012.
7.
WuT.-Y., PanJ.-S. and LinC.-F., Improving accessing efficiency of cloud storage using deduplication and feedback schemes, IEEE Systems Journal8(1) (2014), 208–218.
8.
MuthitacharoenA., ChenB. and MazieresD., A Low-bandwidth Network File System, Proceeding of the 18th ACM Symposium on Operating System Principle (Sosp’01), Chateau Lake Louise, Banff, Canada, 2001, pp. 174–187.
9.
DeepakumaraJ., HeysH.M. and VenkatesanR., FPGA Implementation of MD5 Hash Algorithm, IEEE Canadian Conference on Electrical and Computer Engineering, 2001, pp. 919–924.
10.
NingD.Z.Z., FPGA Implementation of SHA-1 Algorithm, IEEE 5th International Conference on ASIC, 2003, pp. 1321–1324.
11.
BhagwatD., EshghiK., LongD.D.E. and LillibridgeM., Extreme Binning: Scalable, Parallel De-duplication for Chunk- based File Backup, Proceeding of 17th IEEE/ACM International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MAsCOTS’2009), 2009, pp. 1–9.
12.
FuY., TianL., LiuF., JiangH. and XiaoN., AA-Dedupe: An Application-Aware Source Deduplication Approach for Cloud Backup Services in the Personal Computing Environment, IEEE International Conference on Cluster Computing (CLUSTER), 2013, pp. 112–120.
13.
El-ShimiA., KalachR., KumarA., OlteanA., LiJ. and SenguptaS., Primary Data Deduplication – Large Scale Study and System Design, 2012 USENIX Federated Conferences Conference Week, 2012, pp. 285–296.
14.
WilliamsR.N., Method for partitioning a block of data into sub-blocks and for storing and communicating such sub-blocks, Patent US5990810 A, 1999.
15.
KulkarniP., LaVoieF.D.J. and TraceyJ.M., Redundancy Elimination Within Large Collections of Files, 04 Proceedings of the Annual Conference on USENIX Annual Technical Conference, General Track, 2004, pp. 59–72.
16.
MeisterD., KaiserJ., BrinkmannA., KuhnM., KunkelJ.M. and CortesT., A Study on Data Deduplication in HPC Storage Systems, Conference Proceedings of the ACM/IEEE Conference on High Performance Computing (HPC), 2012.
17.
LuG., An Efficient Data Deduplication Design with Flash-Memory Based Solid State Drive, A Dissertation Submitted to The Faculty of The Graduate School Of The University Of Minnesota, 2012, pp. 1–114.
18.
BjornerN., BlassA. and GurevichY., Content-dependent chunking for differential compression, the local maximum approach, Journal of Computer and System Sciences, Elsevier79(3) (2010), 154–203.
19.
LuG., JinY. and DuD.H.C., Frequency Based Chunking for Data De-Duplication, 18th Annual IEEE/ACM International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, 2010, pp. 287–296.
20.
LkhagvasurenI., SoJ.M., LeeJ.G., YooC. and KoY.W., Byte-index chunking algorithm for data deduplication system, International Journal of Security and Its Applications, SERSC7(5) (2013), 415–424.
21.
LkhagvasurenI., SoJ.M., LeeJ.G., KimJ. and KoY.W., Multi-level byte index chunking mechanism for file synchronization, International Journal of Software Engineering and Its Applications8(3) (2014), 339–350.
22.
YuC., ZhangC., MaoY. and LiF., Leap Based Content Defined Chunking- Theory and Implementation, IEEE 31st Symposium on Mass Storage Systems and Technologies (MSST), 2015, pp. 1–12.
23.
KruusE., UngureanuC. and DubnickiC., Bimodal Content Defined Chunking for Backup Streams, Fast 10 Preceding of the 8th USENIX Conference on file and Storage Technologies, USENIX, 2010, pp. 239–252.
ZhangY., JiangH., FengD., XiaW., FuM., HuangF. and ZhouY., AE: An Asymmetric Extremum Content Defined Chunking Algorithm for Fast and Bandwidth-Efficient Data Deduplication, IEEE Conference on Computer Communications (INFOCOM), 2015, pp. 1337–1345.
MohT.-S. and ChangB.C., A Running Time Improvement forthe Two Thresholds Two Divisors Algorithm, ACMSE ’10, 2010.
28.
XiaW., JiangH., FengD., TianL., FuM. and ZhouY., Ddelta: A deduplication-inspired fast delta compression approach Performance Evaluation, 15 Proceedings of the 7th USENIX Conference on Hot Topics in Storage and File Systems, 2015, pp. 258–272.
29.
AggarwalB., AkellaA., AnandA., et al., EndRE: An end-system redundancy elimination service for enterprises, In Proceedings of the 7th USENIX conference on Networked Systems Design and Implementation (NSDI’10) (San Jose, CA, USA), USENIX Association, 2010, pp. 14–28.
30.
CuiY., LalZ., WangX., DaiN. and MiaoC., QuickSync: Improving Synchronization Efficiency for Mobile Cloud Storage Services, In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (Paris, France), ACM, 2015, pp, 592–603.
31.
XiaW., ZhouY., JiangH., FengD., HuaY., HuY., ZhangY. and LiuQ., FastCDC: A Fast and Efficient Content-Defined Chunking Approach for Data Deduplication, USENIX Open Access to the Proceedings of USENIX Annual Technical Conference (USENIX ATC ’16), 2016, pp. 101–114.
32.
BroderA.Z., Some applications of Rabin’s fingerprinting method, Sequences II: Methods in Communications, Security and Computer Science, Springer, 1993, pp. 143–152.
33.
MichaelO. and Rabin, Fingerprinting by random polynomials, Center for Research in Computing Technology.” Aiken Computation Laboratory, Univ, 1981.
34.
DubnickiC., KruusE., LichotaK. and UngureanuC., Methods and systems for data management using multiple selection criteria, US Patent App 11/566,122, 2006.
35.
MinJ., YoonD. and WonY., Efficient deduplication techniques for modern backup operation, IEEE Transactions on Computers60(6) (2011), 824–840.
36.
DerviskarabogaS., A simple and Global optimization algorithm for engineering problem: Differential evolution algorithm, Turk Journal Electrical Engineering12(1) (2004).
37.
An Introduction to Differential Evolution, http://www.maths.uq.edu.au/MASCOS/Multi Agent04/Fleetwood.pdf
Cortes-AntonioP., Rangel-GonzalezJ., Villa-VargasL.A., Ramirez-SalinasM.A., LozanoH.M. and BatyrshinI., Design and implementation of differential evolution algorithm on FPGA for doublePrecision floating-point representation, Acta Polytechnic Hungarica11(4) (2014).
40.
DeanJ. and GhemawatS., Map-reduce: Simplified Data Processing on Large Clusters, To appear in OSDI, 2004.
KumarN., RawatR. and JainS.C., Bucket Based Data Deduplication Technique for Big Data Storage System, IEEE 5th International Conference on Reliability, Infocom Tech and Optimization, 2016.
46.
KumarN. and KumarR., Improved join operations using ORC in HIVE, Springer Transactions on Information Communication Technology (ICT), Springer4(2) (2016), 209–215.
XiaW., JiangH. and FengD., et al., A comprehensive study of the past, present, and future of data deduplication, Proceedings of the IEEE104(9) (2016), 1681–1710.
49.
QuinlanS. and DorwardS., Venti: A New Approach to Archival Storage, in Proc 1st USENIX Conf File Storage Technologies, Monterey, CA, USA, 2002, pp. 89–101.
50.
KubiatowiczJ., BindelD. and ChenY., Oceanstore: An architecture for global-scale persistent storage, ACM Sigplan Notices35(11) (2000), 190–201.
51.
HongB., PlantenbergD. and LongD., et al., Duplicate Data Elimination in a SAN file system, in Proc 21st IEEE/12th NASA Goddard Conf Mass Storage Syst Technol, Greenbelt, MD, USA, 2004, pp. 301–314.
52.
ZhangY., WuY. and YangG., Droplet: A Distributed Solution of Data Deduplication, in Proc ACM/IEEE 13th Int Conf Grid Computing, 2012, pp. 114–121.
