Many extreme-scale applications require the movement of large quantities of data to, from, and among leadership computing facilities, as well as other scientific facilities and the home institutions of facility users. These applications, particularly when leadership computing facilities are involved, can touch upon edge cases (e.g., terabyte files) that had not been a focus of previous Globus optimization work, which had emphasized rather the movement of many smaller (megabyte to gigabyte) files. We report here on how automated client-driven chunking can be used to accelerate both the movement of large files and the integrity checking operations that have proven to be essential for large data transfers. We present detailed performance studies that provide insights into the benefits of these modifications in a range of file transfer scenarios.
AlanIArslanEKosarT (2015) Energy-aware data transfer algorithms. In: International conference for high performance computing, networking, storage and analysis, Austin, Texas, 15 - 20 November 2015, IEEE, 1–12.
2.
AlexanderFAlmgrenABellJ, et al. (2020) Exascale applications: skin in the game. Philosophical Transactions of the Royal Society A378(2166): 20190056.
3.
AllcockW (2003) GridFTP: protocol extensions to FTP for the grid. Global Grid Forum. Available at: https://zenodo.org/records/6828.
4.
AllcockWBresnahanJKettimuthuR, et al (2005) The Globus striped GridFTP framework and server. In: ACM/IEEE conference on supercomputing, Seattle, Washington, USA, 12 - 18 November 2005. IEEE Computer Society,54.
5.
AllcockWFosterITueckeS, et al (2001) Protocols and services for distributed data- intensive science. In AIP Conference Proceedings, volume 583, pages 161–163. Melville NY: American Institute of Physics.
6.
ArifuzzamanMDArslanE (2021) Online optimization of file transfers in high-speed networks. International conference for high performance computing, networking, storage and analysis. St Louis, Missouri, USA.
7.
ArslanEAhmedA (2018) A low-overhead integrity verification for big data transfers. In: International conference on big data, Seattle, WA, USA, 10–13 December 2018. IEEE, 4227–4236.
8.
ArslanEKosarT (2018) High-speed transfer optimization based on historical analysis and real-time tuning. IEEE Transactions on Parallel and Distributed Systems29(6): 1303–1316.
9.
ArslanEPehlivanBAKosarT (2018) Big data transfer optimization through adaptive parameter tuning. Journal of Parallel and Distributed Computing120: 89–100.
10.
BairavasundaramLNArpaci-DusseauACArpaci-DusseauRH, et al. (2008) An analysis of data corruption in the storage stack. ACM Transactions on Storage4(3): 1–28.
11.
BullotHLes CottrellRHughes-JonesR (2003) Evaluation of advanced TCP stacks on fast long-distance production networks. Journal of Grid Computing1(4): 345–359.
12.
ChardKTueckeSGlobusIF (2016) Recent enhancements and future plans. In: XSEDE16 conference on diversity, big data, and science at scale, Miami, USA, 17 July 2016, 1–8.
13.
CharyyevBArslanE (2020) RIVA: robust integrity verification algorithm for high-speed file transfers. IEEE Transactions on Parallel and Distributed Systems31(6): 1387–1399.
14.
DartERotmanLTierneyB, et al. (2013) The Science DMZ: a network design pattern for data-intensive science. In: International conference on high performance computing, networking, storage and analysis, Denver, CO, USA, 17 Nov 2013, 1–10.
15.
DartEAllcockWBhimjiW, et al. (2021a) The Petascale DTN project: high performance data transfer for HPC facilities. arXiv preprint arXiv:2105.12880.
16.
DartEAllcockWBhimjiW, et al. (2021b) Petascale DTN project host network configurations and test data set. https://zenodo.org/records/3880669.
FosterI (2011) Globus Online: accelerating and democratizing science through cloud-based services. IEEE Internet Computing15(3): 70–73.
19.
FosterIAinsworthMAllenB, et al (2017) Computing just what you need: Online data analysis and reduction at extreme scales. In: 23rd international conference on parallel and distributed computing, Santiago de CompostelaSpringer, Spain, Pages 3–19.
20.
GuYGrossmanRL (2007) UDT: UDP-based data transfer for high-speed wide area networks. Computer Networks51(7): 1777–1799.
21.
HackerTAtheyBDNobleB (2002) The end-to-end performance effects of parallel TCP sockets on a lossy wide-area network. In: 16th International Parallel and Distributed Processing Symposium, Ft. Lauderdale, FL, 15 2001 to April 19 2001: IEEE, 10.
22.
HeELeighJYuO, et al. (2002) Reliable blast UDP: predictable high performance bulk data transfer. In: International conference on cluster computing. Chicago, Illinois, 23 September 2002: IEEE, 317–324.
23.
HeitmannKUramTDFinkelH, et al. (2019) HACC cosmological simulations: first data release. The Astrophysical Journal - Supplement Series244(1): 17.
24.
ItoTOhsakiHImaseM (2005) On parameter tuning of data transfer protocol GridFTP for wide-area grid computing. In: 2nd international conference on broadband networks, Boston, MA, 3-7 October 2005. New York, NY, USA: IEEE, 1338–1344.
25.
KettimuthuRSimAGunterD, et al. (2010) Lessons learned from moving Earth System Grid data sets over a 20 Gbps wide-area network. In: 19th ACM international symposium on high performance distributed computing, Chicago, Il, 21-25 June 2010, 316–319.
26.
KettimuthuRVardoyanGAgrawalG, et al. (2014) Modeling and optimizing large-scale wide-area data transfers. In: 14th IEEE/ACM international symposium on cluster, cloud, and grid computing, CCGRID ’14, Chicago, IL, 26 May 2014. IEEE Press, 196–205. ISBN 9781479927838.
27.
KettimuthuRVardoyanGAgrawalG, et al. (2015) An elegant sufficiency: load-aware differentiated scheduling of data transfers. In: International conference for high performance computing, networking, storage and analysis, SC ’15, Austin, TX, 15 Nov 2015: Association for Computing Machinery.
28.
KettimuthuRLiuZWheelerD, et al. (2018) Transferring a petabyte in a day. Future Generation Computer Systems88: 191–198.
29.
LacinskiLLeeLTuroscyS, et al. (2024) Automated, reliable, and efficient continental-scale replication of 7.3 PB climate simulation data. A case study, Arxiv 2404.19717.
30.
LeithDShortenR (2004) H-TCP: TCP for high-speed and long-distance networks. In: 2nd international workshop on protocols for fast long-distance networks,Argonne, Illinois, USA, 26-30January 2004.
31.
LiuSJungE-SKettimuthuR, et al. (2016) Towards optimizing large-scale data transfers with end-to-end integrity verification. In: International conference on big data, Washington DC, USA, 15-18 Dec 2023. IEEE, 3002–3007.
32.
LiuZBalaprakashPKettimuthuR, et al. (2017) Explaining wide area data transfer performance. In: 26th international symposium on high-performance parallel and distributed computing, Washington D.C, 26 June 2017, 167–178.
33.
LiuZKettimuthuRFosterI, et al. (2018a) Cross-geography scientific data transferring trends and behavior. In: 27th international symposium on high-performance parallel and distributed computing, Tempe, AZ, USA, 11-15 June 2018. IEEE, 267–278.
34.
LiuZKettimuthuRFosterIT, et al. (2018b) A comprehensive study of wide area data movement at a scientific computing facility. In: 38th international conference on distributed computing systems, Vienna, Austria, 2 July 2018, 1604–1611.
35.
LiuYLiuZKettimuthuR, et al. (2019) Data transfer between scientific facilities–bottleneck analysis, insights and optimizations. In: 19th IEEE/ACM international symposium on cluster, cloud and grid computing, Larnaca, Cyprus, 14-17 May 2019. IEEE, 122–131.
36.
LiuZAliAKeneseiP, et al. (2021a) Bridging data center AI systems with edge computing for actionable information retrieval. In: 3rd annual workshop on extreme-scale experiment-in-the-loop computing, St. Louis, MO, USA, 19-19 November 2021. IEEE, 15–23.
37.
LiuZKettimuthuRChungJ, et al. (2021b) Design and evaluation of a simple data interface for efficient data transfer across diverse storage. ACM Transactions on Modeling and Performance Evaluation of Computing Systems6(1): 1–25.
38.
LuDQiaoYDindaPA, et al. (2005) Modeling and taming parallel TCP on the wide area network. In: 19th IEEE international parallel and distributed processing symposium, Denver, CO, 4 April 2005: IEEE, 10.
39.
PrasadRSJainMDovrolisC (2003) Socket buffer auto-sizing for high-performance data transfers. Journal of Grid Computing1(4): 361–376.
40.
ReichsteinMCamps-VallsGStevensB, et al. (2019) Deep learning and process understanding for data-driven Earth system science. Nature566(7743): 195–204.
SivakumarHBaileySPsocketsRLG (2000) The case for application-level network striping for data intensive applications using high speed wide area networks. In: ACM/IEEE conference on supercomputing, Kyoto, Japan, 17 Nov, 2024: IEEE, 38.
43.
StoneJPartridgeC (2000) When the CRC and TCP checksum disagree. ACM SIGCOMM - Computer Communication Review30(4): 309–319.
44.
TierneyBLeeJChenLT, et al. (1994) Distributed parallel data storage systems: a scalable approach to high speed image servers. In: 2nd ACM international conference on multimedia, San Francisco California, 15 - 20 October 1994, 399–405.
45.
TierneyBLLeeJCrowleyB, et al. (1999) A network-aware distributed storage cache for data intensive environments. In: 8th international symposium on high performance distributed computing, Salt Lake City, Utah, 10 November 2012. IEEE, 185–193.
46.
WeiDXJinCLowSH, et al. (2006) Fast TCP: motivation, architecture, algorithms, performance. IEEE/ACM Transactions on Networking14(6): 1246–1259.
47.
YildirimEKimJYKosarT (2012) How GridFTP pipelining, parallelism and concurrency work: a guide for optimizing large dataset transfers. In: 2012 SC companion: high performance computing, networking storage and analysis, Salt Lake City, Utah, 10 November 2012. IEEE, 506–515.
48.
YildirimEArslanEKimJ, et al. (2015) Application-level optimization of big data transfers through pipelining, parallelism and concurrency. IEEE Transactions on Cloud Computing4(1): 63–75.
49.
ZhangMLaiJKrishnamurthyA, et al. (2004) A transport layer approach for improving end-to-end performance and robustness using redundant paths. In: USENIX annual technical conference, Hyatt Regency Santa Clara, 10–12 July 2024.: USENIX, 99–112.
