Traditional workload managers do not have the capacity to consider how IO contention can increase job runtime and even cause entire resource allocations to be wasted. Whether from bursts of IO demand or parallel file systems (PFS) performance degradation, IO contention must be identified and addressed to ensure maximum performance. In this paper, we present AI4IO (AI for IO), a suite of tools using AI methods to prevent and mitigate performance losses due to IO contention. AI4IO enables existing workload managers to become IO-aware. Currently, AI4IO consists of two tools: PRIONN and CanarIO. PRIONN predicts IO contention and empowers schedulers to prevent it. CanarIO mitigates the impact of IO contention when it does occur. We measure the effectiveness of AI4IO when integrated into Flux, a next-generation scheduler, for both small- and large-scale IO-intensive job workloads. Our results show that integrating AI4IO into Flux improves the workload makespan up to 6.4%, which can account for more than 18,000 node-h of saved resources per week on a production cluster in our large-scale workload.
AhnDHBassNChuA, et al. (2020) Flux: overcoming scheduling challenges for exascale workflows. Future Generation Computer Systems110: 202–213. DOI: 10.1016/j.future.2020.04.006.
3.
BlagodurovSFedorovaA (2012) Towards the contention aware scheduling in HPC cluster environment. Journal of Physics: Conference Series385; 012010.
4.
BlagodurovSFedorovaA (2013) Optimizing shared resource contention in HPC clusters. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), Denver, CO, 22 November 2013, pp. 1–25.
5.
BoehmeDGamblinTBeckingsaleD, et al. (2016) Caliper: performance introspection for HPC software stacks. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC). Salt Lake City, UT, 13–18 November 2016.
6.
BrinkSLumsdenIScully-AllisonC, et al. (2020) Usability and performance improvements in Hatchet. In: Proceedings of the IEEE/ACM International Workshop on HPC User Support Tools and Workshop on Programming and Performance Visualization Tools (HUST/ProTools@SC), Atlanta, GA, pp. 49–58. Washington, DC: IEEE Computer Society. DOI: 10.1109/HUSTProtools51951.2020.00013.
7.
ChenXLuCDPattabiramanK (2013) Predicting job completion times using system logs in supercomputing clusters. In: Proceedings of the 43rd Annual IEEE/IFIP Conference on Dependable Systems and Networks Workshop (DSN-W), Valencia, Spain, 29 June–2 July 2020, pp. 1–8.
8.
CunhaRLFRodriguesERTizzeiLP, et al. (2017) Job placement advisor based on turnaround predictions for HPC hybrid clouds. Future Generation Computer Systems67: 35–46.
9.
Di MartinoCKalbarczykZIyerRK, et al. (2014) Lessons learned from the analysis of system failures at petascale: the case of blue waters. In: Proceedings of the 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, Atlanta, GA, 23–26 June 2014, pp. 610–621. Washington, DC: IEEE.
10.
DietzeRHofmannMRüngerG (2018) Analysis and modeling of resource contention effects based on benchmark applications. In: Proceedings of the 2018 International Conference on High Performance Computing & Simulation (HPCS), Orléans, France, 16–20 July 2018, pp. 330–337. Washington, DC: IEEE.
11.
DorierMAntoniuGRossR, et al. (2014) CALCioM: mitigating I/O interference in HPC systems through cross-application coordination. In: Proceedings of the 2014 IEEE 28th International Parallel and Distributed Processing Symposium, Phoenix, AZ, 19–23 May 2019, pp. 155–164. Washington, DC: IEEE.
12.
DowneyAB (1997) Using queue time predictions for processor allocation. In: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, Geneva, Switzerland, pp. 35–57.
13.
HerbeinSAhnDHLipariD, et al. (2016) Scalable I/O-aware job scheduling for burst buffer enabled HPC clusters. In: Proceedings of the 25th ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC), Kyoto, Japan, 31 May 2016, pp. 69–80. New York, NY: ACM. DOI: 10.1145/2907294.2907316.
14.
KhetawatHZimmerCMuellerF, et al. (2019) Evaluating burst buffer placement in HPC systems. In: Proceeding of the 2019 IEEE International Conference on Cluster Computing (CLUSTER), Albuquerque, NM, 23–26 September 2019, pp. 1–11. Washington, DC: IEEE.
15.
KrishnaswamySLokeSWZaslavskyA (2004) Estimating computation times of data-intensive applications. IEEE Distributed Systems Online5(4): 1.
16.
LockwoodGKSnyderSWangT, et al. (2018) A year in the life of a parallel file system. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), Denver, CO, 12–17 November 2017, pp. 74. Manhattan, NY: IEEE Press.
17.
LofsteadJZhengFLiuQ, et al. (2010) Managing variability in the IO performance of petascale storage systems. In: Proceedings of the 22nd ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC, 13–19 November 2010. ISBN 978-1-4244-7559-9, pp. 1–12.
18.
MartinassoMMéhautJF (2011) A contention-aware performance model for HPC-based networks: a case study of the InfiniBand network. In: Proceedings of the European Conference on Parallel Processing, Warsaw, Poland, 24–28 August 2011, pp. 91–102. Berlin, Germany: Springer.
19.
MatsunagaAFortesJA (2010) On the use of machine learning to predict the time and resources consumed by applications. In: Proceedings of the 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (CCGrid), Melbourne, VIC, 17–20 May 2010, pp. 495–504.
20.
McKennaRHerbeinSMoodyA, et al. (2016) Machine learning predictions of runtime and IO traffic on high-end clusters. In: Proceedings of the 2016 IEEE International Conference on Cluster Computing (CLUSTER), Taipei, Taiwan, pp. 255–258. Washington, DC: IEEE Computer Society. DOI: 10.1109/CLUSTER.2016.58.
21.
MikolovTSutskeverIChenK, et al. (2013) Distributed representations of words and phrases and their compositionality. In: Proceedings of the 26th Neural Information Processing Systems Conference, Lake Tahoe, Nevada, 5–10 December 2013, pp. 3111–3119.
22.
MillerM (2015) Design and Implementation of MACSio. Technical report. Livermore, CA: Lawrence Livermore National Lab.(LLNL).
23.
MohammedAEleliemyACiorbaFM, et al. (2019) An approach for realistically simulating the performance of scientific applications on high performance computing systems. Future Generation Computer Systems111: 617–633.
24.
MubarakMCarnsPJenkinsJ, et al. (2017) Quantifying I/O and communication traffic interference on dragonfly networks equipped with burst buffers. In: Proceedings of the 2017 IEEE International Conference on Cluster Computing (CLUSTER), Honolulu, HI, 5–8 September 2017, pp. 204–215. Washington, DC: IEEE.
25.
MubarakMCarothersCDRossRB, et al. (2016) Enabling parallel simulation of large-scale HPC network systems. IEEE Transactions on Parallel and Distributed Systems28(1): 87–100.
26.
NeuwirthSWangFOralS, et al (2017) Automatic and transparent resource contention mitigation for improving large-scale parallel file system performance. In: Proceedings of the 2017 IEEE 23rd International Conference on Parallel and Distributed Systems (ICPADS), Shenzhen, China, 15–17 December 2017, pp. 604–613. Washington, DC: IEEE.
27.
NeuwirthSWangFOralS, et al. (2016) Using balanced data placement to address I/O contention in production environments. In: Proceedings of the 2016 28th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD), Los Angeles, CA, 26–28 October 2016, pp. 9–17. Washington, DC: IEEE.
28.
NurmiDBrevikJWolskiR (2007) QBETS: queue bounds estimation from time series. In: Proceedigns of the the Workshop on Job Scheduling Strategies for Parallel Processing, Saint-Malo, France, 26 June 2006, pp. 76–101.
29.
RodriguesERCunhaRLNettoMA, et al. (2016) Helping HPC users specify job memory requirements via machine learning. In: Proceedings of the Third International Workshop on HPC User Support Tools, Salt Lake City, UT, 13 November 2016, pp. 6–13.
30.
SmithWFosterITaylorV (1998) Predicting application runtimes using historical information. In: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, Orlando, FL, 30 March 1998, pp. 122–142.
31.
SmithWTaylorVFosterI (1999) Using run-time predictions to estimate queue wait times and improve scheduler performance. In: Proceedings of the Workshop on Job Scheduling Strategies for Parallel Processing, San Juan, Puerto Rico, 16 April 1999, pp. 202–219.
32.
SungHBangJSimA, et al. (2019) Understanding parallel I/O performance trends under various HPC configurations. In: Proceedings of the ACM Workshop on Systems and Network Telemetry and Analytics, Phoenix, AZ, 25 June 2019, pp. 29–36.
33.
TsafrirDEtsionYFeitelsonDG (2007) Backfilling using system-generated predictions rather than user runtime estimates. IEEE Transactions on Parallel and Distributed Systems18(6): 789–803.
34.
WyattMRHerbeinSAhnD, et al. (2017) Unstructured Data Analytics for Next-Generation HPC Schedulers: Capturing Jobs’ Needs from Unstructured Job Scripts. Technical report. Livermore, CA: Lawrence Livermore National Lab.(LLNL).
35.
WyattMRHerbeinSGamblinT, et al. (2018) PRIONN: predicting runtime and IO using neural networks. In: Proceedings of the 47th International Conference on Parallel Processing, (ICPP), Eugene, OR, p. 46:1–46:12. New York, NY: ACM. DOI: 10.1145/3225058.3225091.
36.
WyattMRHerbeinSShogaK, et al. (2020) CanarIO: sounding the alarm on IO-related performance degradation. In: Proceedings of the 2020 IEEE International Parallel and Distributed Processing Symposium (IPDPS), New Orleans, LA, 13–16 August 2018. IEEE Computer Society, pp. 73–83. DOI: 10.1109/IPDPS47924.2020.00018.
37.
XuYArteagaDZhaoM, et al (2012) vPFS: bandwidth virtualization of parallel storage systems. In: Proceedings of the IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST), Pacific Grove, CA, 16–20 April 2012, pp. 1–12. Washington, DC: IEEE.
38.
ZhaoMXuY (2019) vPFS+: managing I/O performance for diverse HPC applications. In: Proceedings of the 2019 35th Symposium on Mass Storage Systems and Technologies (MSST), Santa Clara, CA, 24 May 2019, pp. 51–64. Washington, DC: IEEE.
