Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. This paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.
AdriaensFGionisA (2022) Diameter minimization by shortcutting with degree constraints. In: 2022 IEEE International Conference on Data Mining (ICDM), Orlando, FL, USA, 28 November–01 December 2022, IEEE, pp. 843–848.
2.
AlsaadiAWardLMerzkyA, et al. (2022) RADICAL-pilot and parsl: executing heterogeneous workflows on HPC platforms. In: 2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS), Dallas, TX, USA, 13–18 November 2022, IEEE, pp. 27–34.
3.
AmirkhaniABarshooiAH (2022) Consensus in multi-agent systems: a review. Artificial Intelligence Review55(5): 3897–3935.
4.
AttarHKhosraviMRIgorovichSS, et al. (2020) Review and performance evaluation of FIFO, PQ, CQ, FQ, and WFQ algorithms in multimedia wireless sensor networks. International Journal of Distributed Sensor Networks16(6): 1550147720913233.
5.
AuYoungAChunBNSnoerenAC, et al. (2004) Resource allocation in federated distributed computing infrastructures. In: Proceedings of the First Workshop on Operating System and Architectural Support for the on demand IT InfraStructure,San Francisco, CA, December 6-82004.
6.
BirmanK (2007) The promise, and limitations, of gossip protocols. ACM SIGOPS - Operating Systems Review41(5): 8–13.
7.
BlandWLuHSeoS, et al. (2015) Lessons learned implementing user-level failure mitigation in mpich. In: 2015 15th IEEE/ACM international symposium on cluster, cloud and grid computing, Shenzhen, China, 04–07 May 2015, IEEE, pp. 1123–1126.
8.
BłażewiczJPeschESternaM (2000) The disjunctive graph machine representation of the job shop scheduling problem. European Journal of Operational Research127(2): 317–331.
9.
BockelmanBLivnyMLinB, et al. (2021) Principles, technologies, and time: the translational journey of the HTCondor-CE. Journal of Computational Science52: 101213.
10.
BrambillaMFerranteEBirattariM, et al. (2013) Swarm robotics: a review from the swarm engineering perspective. Swarm Intelligence7: 1–41.
CappelloFAlGGroppW, et al. (2014) Toward exascale resilience: 2014 update. Supercomputing Frontiers and Innovations: An International Journal1(1): 5–28.
13.
CastroMLiskovB (2002) Practical byzantine fault tolerance and proactive recovery. ACM Transactions on Computer Systems20(4): 398–461.
14.
ChardKBubendorferK (2019) Co-operative resource allocation: building an open cloud market using shared infrastructure. IEEE Transactions on Cloud Computing7(1): 183–195.
15.
ChenYLiuZZhangY, et al. (2021) Deep reinforcement learning-based dynamic resource management for mobile edge computing in industrial Internet of Things. IEEE Transactions on Industrial Informatics17: 4925–4934.
16.
CliffordJGarfieldKTowhidnejadM, et al. (2017) Multi-layer model of swarm intelligence for resilient autonomous systems. In: 2017 IEEE/AIAA 36th Digital Avionics Systems Conference (DASC), St. Petersburg, FL, USA, 17–21 September 2017, IEEE, pp. 1–4.
DaiHKhalilEZhangY, et al. (2017) Learning combinatorial optimization algorithms over graphs. Advances in Neural Information Processing Systems30: 6351–6361.
19.
DeelmanEVahiKJuveG, et al. (2015) Pegasus, a workflow management system for science automation. Future Generation Computer Systems46: 17–35.
20.
DeelmanEVahiKRyngeM, et al. (2019) The evolution of the Pegasus workflow management software. Computing in Science & Engineering21(4): 22–36.
21.
DeelmanEda SilvaRFVahiK, et al. (2021) The pegasus workflow management system: translational computer science in practice. Journal of Computational Science52: 101200.
22.
DellTJ (1997) A white paper on the benefits of chipkill-correct ecc for pc server main memory. IBM Microelectronics division11(1-23): 5–7.
DouYXingGMaA, et al. (2024) A review of event-triggered consensus control in multi-agent systems. Journal of Control and Decision12: 1–23.
25.
FloydSHendersonTGurtovA (2004) Rfc3782: the newreno modification to tcp’s fast recovery algorithm.
26.
FreyJTannenbaumTLivnyM, et al. (2002) Condor-G: a computation management agent for multi-institutional grids. Cluster Computing5: 237–246.
27.
Galaxy Community (2022) The Galaxy platform for accessible, reproducible and collaborative biomedical analyses: 2022 update. Nucleic Acids Research50(W1): W345–W351.
28.
GareyMRJohnsonDSSethiR (1976) The complexity of flowshop and jobshop scheduling. Mathematics of Operations Research1(2): 117–129.
GuptaHRamachandranU (2018) FogStore: a geo-distributed key-value store guaranteeing low latency for strongly consistent access. In: Proceedings of the 12th ACM International Conference on Distributed and Event-Based Systems, DEBS ’18. New York, NY, USA: Association for Computing Machinery, 148–159.
31.
HaSRheeIXuL (2008) Cubic: a new TCP-friendly high-speed tcp variant. ACM SIGOPS - Operating Systems Review42(5): 64–74.
32.
HarenslakBPde RuiterJ (2021) Data Pipelines with Apache Airflow. NY: Manning, Shelter Island.
33.
JainAKGuptaNGuptaBB (2025) A survey on scalable consensus algorithms for blockchain technology. Cyber Security and Applications3: 100065.
34.
JihadAAAl-JanabiSTFYassenET (2022) A survey on provisioning and scheduling algorithms for scientific workflows in cloud computing. AIP Conference Proceedings, Volume 2400. Melville, NY: AIP Publishing.
35.
JinHRaghavanKPapadimitriouG, et al. (2022) Workflow anomaly detection with graph neural networks. In: 2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS). Dallas, TX, USA, 13–18 November 2022, pp. 35–42.
36.
JinHPapadimitriouGRaghavanK, et al. (2024) Large language models for anomaly detection in computational workflows: from supervised fine-tuning to in-context learning. In: SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, Atlanta, GA, USA, 17–22 November 2024, IEEE, pp. 1–17.
37.
JohnstonJLiuXYWuS, et al. (2023a) A curriculum learning approach to optimization with application to downlink beamforming. IEEE Transactions on Signal Processing72: 84–98.
38.
JohnstonJLiuXYWuS, et al. (2023b) Downlink beamforming optimization via deep learning. In: 2023 59th Annual Allerton Conference on Communication, Control, and Computing (Allerton), Monticello, IL, USA, 26–29 September 2023, IEEE, pp. 1–5.
39.
KarydisKKannappanPTannerHG, et al. (2016) Resilience through learning in multi-agent cyber-physical systems. Frontiers in Robotics and AI3: 36.
40.
KennedyJ (2006) Swarm intelligence. In: Handbook of Nature-Inspired and Innovative Computing: Integrating Classical Models with Emerging Technologies. New York, NY: Springer, 187–219.
41.
KühnDOsthusD (2012) A survey on Hamilton cycles in directed graphs. European Journal of Combinatorics33(5): 750–766.
42.
LakshmanAMalikP (2010) Cassandra: a decentralized structured storage system. ACM SIGOPS - Operating Systems Review44(2): 35–40.
43.
LeithDShortenR (2004) H-TCP: TCP for high-speed and long-distance networks. Proceedings of PFLDnet, Volume 2004. Argonne, IL: Citeseer.
44.
LiuNLiZXuZ, et al. (2017) A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning. In: 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), Atlanta, GA, USA, 5–8 June 2017, pp. 372–382.
45.
LiuXYLiZWuS, et al. (2023) Stationary deep reinforcement learning with quantum k-spin Hamiltonian regularization. In: ICLR 2023 Workshop on Physics for Machine Learning, Kihali, Rwanda, May 1-5, 2023.
46.
LosadaNGonzálezPMartínMJ, et al. (2020) Fault tolerance of mpi applications in exascale systems: the ulfm solution. Future Generation Computer Systems106: 467–481.
47.
MahmudIZukPWangC, et al. (2024) DISTRI: development and integration of simulation tools for resilient infrastructure. In: The 5th International Workshop on Big Data & AI Tools, Models, and Use Cases for Innovative Scientific Discovery (BTSD), Washington DC, USA, 15–18 December 2024.
48.
MaoYDebSVenkatakrishnanSB, et al. (2020) Perigee: efficient peer-to-peer network design for blockchains. In: Proceedings of the 39th Symposium on Principles of Distributed Computing, Virtual, August 3-7,2020, pp. 428–437.
49.
MitchellRPottierLJacobsS, et al. (2019) Exploration of workflow management systems emerging features from users perspectives. In: 2019 IEEE International Conference on Big Data (Big Data), Los Angeles, CA, USA, 09–12 December 2019, IEEE, pp. 4537–4544.
50.
NovellaJAEmami KhoonsariPHermanS, et al. (2019) Container-based bioinformatics with pachyderm. Bioinformatics35(5): 839–846.
ParkKLeeDHWooY, et al. (2009) Reliability and performance enhancement technique for ssd array storage system using raid mechanism. In: 2009 9th International Symposium on Communications and Information Technology, Icheon, Korea (South), 28–30 September 2009, IEEE, pp. 140–145.
53.
PattersonDAGibsonGKatzRH (1988) A case for redundant arrays of inexpensive disks (raid). In: Proceedings of the 1988 ACM SIGMOD International Conference on Management of Data, Chicago, IL, June1109-3116, 1988, pp. –.
54.
PhadkeAMedranoFA (2022) Towards resilient UAV swarms—a breakdown of resiliency requirements in UAV swarms. Drones6(11): 340.
55.
PoolaDSalehiMARamamohanaraoK, et al. (2017) Chapter 15 – a taxonomy and survey of fault-tolerant workflow management systems in cloud and distributed computing environments. In: MistrikIBahsoonRAliN, et al. (eds) Software Architecture for Big Data and the Cloud. Boston: Morgan Kaufmann, 285–320. https://www.sciencedirect.com/science/article/pii/B9780128054673000156
56.
SaeediSKhorsandRBidgoliSG, et al. (2020) Improved many-objective particle swarm optimization algorithm for scientific workflow scheduling in cloud computing. Computers & Industrial Engineering147: 106649.
57.
SardarazMTahirM (2019) A hybrid algorithm for scheduling scientific workflows in cloud computing. IEEE Access7: 186137–186146.
SridharanVLibertyD (2012) A study of dram failures in the field. In: SC’12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT, USA, 10–16 November 2012, IEEE, pp. 1–11.
61.
StoicaIMorrisRKargerD, et al. (2001) Chord: a scalable peer-to-peer lookup service for internet applications. ACM SIGCOMM - Computer Communication Review31(4): 149–160.
62.
SureshLMalkhiDGopalanP, et al. (2018) Stable and consistent membership at scale with rapid. In: 2018USENIX Annual Technical Conference (USENIX ATC 18),Boston, MA, July11-13, 2018.
63.
SvirinPDeKFortiA, et al. (2019) BigPanDA: panda workload management system and its applications beyond ATLAS. EPJ Web of Conferences, Volume 214. EDP Sciences, Les Ulis, France. pp. 03050.
64.
TaillardE (1993) Benchmarks for basic scheduling problems. European Journal of Operational Research64(2): 278–285.
65.
TaylorIJDeelmanEGannonDB, et al. (2007) Workflows for E-Science: Scientific Workflows for Grids, Volume 1. Springer, London, UK.
66.
TharejaKRaghavanKMandalA, et al. (2025) A greedy consensus-based approach to distributed job selection: toward fully-decentralized workload management system. In: Proceedings of the 25th IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid’25), Tromso, Norway, May 19-22, 2025,IEEE.
67.
TopcuogluHHaririSWuMY (2002) Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Transactions on Parallel and Distributed Systems13(3): 260–274.
68.
TylerNKnopRBardD, et al. (2022) Cross-facility workflows: case studies with active experiments. In: 2022 IEEE/ACM Workshop on Workflows in Support of Large-Scale Science (WORKS), Dallas, TX, USA, 13–18 November 2022, IEEE, pp. 68–75.
69.
Van BeekVOikonomouGIosupA (2019) Portfolio scheduling for managing operational and disaster-recovery risks in virtualized datacenters hosting business-critical workloads. In: 2019 18th International Symposium on Parallel and Distributed Computing (ISPDC), Amsterdam, Netherlands, 03–07 June 2019, IEEE, pp. 94–102.
WrattenLWilmAGökeJ (2021) Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers. Nature Methods18(10): 1161–1168.
72.
WuSZhaiYHuangJ, et al. (2023a) FT-GEMM: a fault tolerant high performance GEMM implementation on x86 CPUs. In: Proceedings of the 32nd International Symposium on High-Performance Parallel and Distributed Computing,Orlando, FL, June20-23, 2023, pp. 323–324.
73.
WuSDingYZhaiY, et al. (2024a) Ft k-means: a high-performance k-means on gpu with fault tolerance. In: 2024 IEEE International Conference on Cluster Computing (CLUSTER), Kobe, Japan, 24-27 September 2024, IEEE, pp. 322–334.
74.
WuSRaghavanKDiS, et al. (2024b) Dgro: diameter-guided ring optimization for integrated research infrastructure membership. https://arxiv.org/abs/2410.11142
75.
WuSZhaiYLiuJ, et al. (2023b) Anatomy of high-performance gemm with online fault tolerance on gpus. In: Proceedings of the 37th International Conference on Supercomputing,Orlando, FL, June21-23, 2023, pp. 360–372.
76.
WuSZhaiYLiuJ, et al. (2024c) Turbofft: Co-designed high-performance and fault-tolerant fast fourier transform on gpus. arXiv preprint arXiv:2412.05824.
77.
WuSZhaiYLiuJ, et al. (2024d) Turbofft: a high-performance fast fourier transform with fault tolerance on gpu. arXiv preprint arXiv:2405.02520.
78.
XieYZhuYWangY, et al. (2019) A novel directional and non-local-convergent particle swarm optimization based workflow scheduling in cloud–edge environment. Future Generation Computer Systems97: 361–378.
79.
YangPCPurawatSIeongPU, et al. (2019) A demonstration of modularity, reuse, reproducibility, portability and scalability for modeling and simulation of cardiac electrophysiology using kepler workflows. PLoS Computational Biology15(3): e1006856.
80.
ZouYYangLJingG, et al. (2024) A survey of fault tolerant consensus in wireless networks. High-Confidence Computing4: 100202.
