Computational workflows are a common class of application on supercomputers, yet the loosely coupled and heterogeneous nature of workflows often fails to take full advantage of their capabilities. We created Colmena to leverage the massive parallelism of a supercomputer by using Artificial Intelligence (AI) to learn from and adapt a workflow as it executes. Colmena allows scientists to define how their application should respond to events (e.g., task completion) as a series of cooperative agents. In this paper, we describe the design of Colmena, the challenges we overcame while deploying applications on exascale systems, and the science workflows we have enhanced through interweaving AI. The scaling challenges we discuss include developing steering strategies that maximize node utilization, introducing data fabrics that reduce communication overhead of data-intensive tasks, and implementing workflow tasks that cache costly operations between invocations. These innovations coupled with a variety of application patterns accessible through our agent-based steering model have enabled science advances in chemistry, biophysics, and materials science using different types of AI. Our vision is that Colmena will spur creative solutions that harness AI across many domains of scientific computing.
AlexanderFAlmgrenABellJ, et al. (2020) Exascale applications: skin in the game. Philosophical Transactions of the Royal Society A378(2166): 20190056.
2.
AlexanderFJAngJBilbreyJA, et al. (2021) Co-design center for exascale machine learning technologies (ExaLearn). The International Journal of High Performance Computing Applications35(6): 598–616.
3.
AlsaadiAWardLMerzkyA, et al. (2022) RADICAL-Pilot and Parsl: executing heterogeneous workflows on HPC platforms. In: IEEE/ACM Workshop on Workflows in Support of Large-Scale Science. Piscataway: IEEE. DOI: 10.1109/works56498.2022.00009.
4.
BabujiYWoodardALiZ, et al. (2019) Parsl: pervasive parallel programming in Python. In: ACM International Symposium on High-Performance Parallel and Distributed Computing, Pisa, Italy, June 3-7.
5.
BadraJOwoyeleOPalP, et al. (2022) A machine learning-genetic algorithm approach for rapid optimization of internal combustion engines. In: Artificial Intelligence and Data Driven Optimization of Internal Combustion Engines. Amsterdam: Elsevier, 125–158. DOI: 10.1016/b978-0-323-88457-0.00003-5.
6.
BalaprakashPSalimMUramTD, et al. (2018) DeepHyper: Asynchronous hyperparameter search for deep neural networks. In: 2018 IEEE 25th International Conference on High Performance Computing (HiPC). Piscataway: IEEE, 42–51.
7.
BhowmikDGaoSYoungMT, et al. (2018) Deep clustering of protein folding simulations. BMC Bioinformatics19: 484–558.
BraceASalimMSubbiahV, et al. (2021) Stream-AI-MD: streaming AI-driven adaptive molecular simulations for heterogeneous computing platforms. In: Proceedings of the Platform for Advanced Scientific Computing Conference. New York, USA: Association for Computing Machinery, 1–13.
10.
BraceAYakushinIMaH, et al. (2022) Coupling streaming AI and HPC ensembles to achieve 100–1000× faster biomolecular simulations. In: IEEE International Parallel and Distributed Processing Symposium. Piscataway: IEEE, 806–816.
11.
CaccinMLiZKermodeJR, et al. (2015) A framework for machine-learning-augmented multiscale atomistic simulations on parallel supercomputers. International Journal of Quantum Chemistry115(16): 1129–1139. DOI: 10.1002/qua.24952.
12.
CasalinoLDommerACGaiebZ, et al. (2021) AI-driven multiscale simulations illuminate mechanisms of SARS-CoV-2 spike dynamics. The International Journal of High Performance Computing Applications35(5): 432–451.
13.
ChardRBabujiYLiZ, et al. (2020) funcX: a federated function serving fabric for science. In: 29th Intl Symp on High-Performance Parallel Dist Computing, Stockholm Sweden, June 23 - 26, 2020.
14.
CurtaroloSMorganDPerssonK, et al. (2003) Predicting crystal structures with data mining of quantum calculations. Physical Review Letters91(13): 135503. DOI: 10.1103/physrevlett.91.135503.
15.
DhakalARaithPWardL, et al. (2023) Fine-grained accelerator partitioning for machine learning and scientific computing in function as a service platform. In: Proceedings of the SC’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023. New York: ACM. DOI: 10.1145/3624062.3624238.
16.
DharumanGWardLMaH, et al. (2023) Protein generation via genome-scale language models with bio-physical scoring. In: Proceedings of the SC’23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC-W 2023. New York: ACM. DOI: 10.1145/3624062.3626087.
17.
DoanHAAgarwalGQianH, et al. (2020) Quantum chemistry-informed active learning to accelerate the design and discovery of sustainable energy storage materials. Chemistry of Materials32(15): 6338–6346. DOI: 10.1021/acs.chemmater.0c00768.
18.
DommerACasalinoLKearnsF, et al. (2023) # COVIDisAirborne: AI-enabled multiscale computational microscopy of delta SARS-CoV-2 in a respiratory aerosol. The International Journal of High Performance Computing Applications37(1): 28–44.
19.
DrorRODirksRMGrossmanJ, et al. (2012) Biomolecular simulation: a computational microscope for molecular biology. Annual Review of Biophysics41: 429–452.
20.
DunnABrenneckJJainA (2019) Rocketsled: a software library for optimizing high-throughput computational searches. Journal of Physics: Materials2(3): 034002.
21.
EgeleRMaulikRRaghavanK, et al. (2022) AutoDEUQ: automated deep ensemble with uncertainty quantification. In: 26th International Conference on Pattern Recognition (ICPR). Piscataway: IEEE, 1908–1914.
22.
FoxGAdigaAChenJ, et al. (2019) Learning everywhere: pervasive machine learning for effective high-performance computation. In: IEEE International Parallel and Distributed Processing Symposium Workshops. Piscataway: IEEE. DOI: 10.1109/ipdpsw.2019.00081.
Gómez-BombarelliRWeiJNDuvenaudD, et al. (2018) Automatic chemical design using a data-driven continuous representation of molecules. ACS Central Science4(2): 268–276. DOI: 10.1021/acscentsci.7b00572.
25.
HarbHElliottSNWardL, et al. (2023) Uncovering novel liquid organic hydrogen carriers: a systematic exploration of chemical compound space using cheminformatics and quantum chemical methods. Digital Discovery2(6): 1813–1830. DOI: 10.1039/d3dd00123g.
26.
HerathCPlaleB (2010) Streamflow programming model for data streaming in scientific workflows. In: 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing. Piscataway: IEEE. DOI: 10.1109/ccgrid.2010.116.
27.
HospitalAGoñiJROrozcoM, et al. (2015) Molecular dynamics simulations: advances and applications. Advances and Applications in Bioinformatics and Chemistry8: 37–47.
28.
HudsonSLarsonJNavarroJL, et al. (2022) libEnsemble: a library to coordinate the concurrent evaluation of dynamic ensembles of calculations. IEEE Transactions on Parallel and Distributed Systems33(4): 977–988. DOI: 10.1109/tpds.2021.3082815.
JacobsenTJørgensenMHammerB (2018) On-the-fly machine learning of atomic potential in density functional theory structure optimization. Physical Review Letters120(2): 026102. DOI: 10.1103/physrevlett.120.026102.
31.
KamatarASakarvadiaMHayot-SassonV, et al. (2023) Lazy python dependency management in large-scale systems. In: 2023 IEEE 19th International Conference on E-Science (E-Science). Piscataway: IEEE. DOI: 10.1109/e-science58273.2023.10254910.
32.
LeeHTurilliMJhaS, et al. (2019) DeepDriveMD: deep-learning driven adaptive molecular simulations for protein folding. In: 3rd Workshop on Deep Learning on Supercomputers. Piscataway: IEEE.
33.
LinZAkinHRaoR, et al. (2023) Evolutionary-scale prediction of atomic-level protein structure with a language model. Science379(6637): 1123–1130. DOI: 10.1126/science.ade2574.
34.
LiuJPacittiEValduriezP, et al. (2015) A survey of data-intensive scientific workflow management. Journal of Grid Computing13(4): 457–493. DOI: 10.1007/s10723-015-9329-8.
35.
MontoyaJHWintherKTFloresRA, et al. (2020) Autonomous intelligent agents for accelerated materials discovery. Chemical Science11(32): 8517–8532. DOI: 10.1039/d0sc01101k.
36.
MoritzPNishiharaRWangS, et al. (2018) Ray: a distributed framework for emerging AI applications. In: 13th USENIX OSDI, Boston, MA, United States, July 7, 2025–July 9, 2025, 561–577.
37.
ParkHXiaoliYRuijieZhu, et al. (2024) A generative artificial intelligence framework based on a molecular diffusion model for the design of metal-organic frameworks for carbon capture. Communications Chemistry7: 21. DOI: 10.1038/s42004-023-01090-2.
38.
ParteeSEllisMRigazziA, et al. (2022) Using machine learning at scale in numerical simulations with SmartSim: an application to ocean climate modeling. Journal of Computational Science62: 101707. DOI: 10.1016/j.jocs.2022.101707. https://www.sciencedirect.com/science/article/pii/S1877750322001065.
39.
PauloskiJGHayot-SassonVWardL, et al. (2023) Accelerating communications in federated applications with transparent object proxiesProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23. New York, NY, USA: Association for Computing Machinery. DOI: 10.1145/3581784.3607047.
40.
PhillipsJCZhengGKumarS, et al. (2002) NAMD: biomolecular simulation on thousands of processors. In: SC’02: Proceedings of the 2002 ACM/IEEE Conference on Supercomputing. Piscataway: IEEE, 36.
41.
PhungTSWardLChardK, et al. (2021) Not all tasks are created equal: adaptive resource allocation for heterogeneous tasks in dynamic workflows. In: IEEE Workshop on Workflows in Support of Large-Scale Science. Piscataway: IEEE. DOI: 10.1109/works54523.2021.00008.
42.
ReyesKGLiuJVargasCJD (2022) Decision-making under uncertainty for multi-stage pipelines: simulation studies to benchmark screening strategies. Journal of Occupational Medicine74(8): 2897–2907. DOI: 10.1007/s11837-022-05368-z.
43.
RussoJDZhangSLeungJM, et al. (2022) Westpa 2.0: high-performance upgrades for weighted ensemble simulations and analysis of longer-timescale applications. Journal of Chemical Theory and Computation18(2): 638–649.
44.
ShamsiZChengKJShuklaD (2018) Reinforcement learning based adaptive sampling: REAPing rewards by exploring protein conformational landscapes. The Journal of Physical Chemistry B122(35): 8386–8395.
45.
St JohnPPhillipsCKemperTW, et al. (2019) Message-passing neural networks for high-throughput polymer screening. Journal of Chemical Physics150(23): 234111.
46.
TrifanAGorgunDSalimM, et al. (2022) Intelligent resolution: integrating Cryo-EM with AI-driven multi-resolution simulations to observe the severe acute respiratory syndrome coronavirus-2 replication-transcription machinery in action. The International Journal of High Performance Computing Applications36(5-6): 603–623.
47.
WardLSivaramanGPauloskiJG, et al. (2021) Colmena: scalable machine-learning-based steering of ensemble simulations for high performance computing. In: IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments. Piscataway: IEEE. DOI: 10.1109/mlhpc54614.2021.00007.
48.
WardLPauloskiJGHayot-SassonV, et al. (2023) Cloud services enable efficient AI-guided simulation workflows across heterogeneous resources. In: IEEE International Parallel and Distributed Processing Symposium Workshops, Milan, Italy, June 3-7, 2025. DOI: 10.1109/ipdpsw59300.2023.00018.
49.
WooHMQianXTanL, et al. (2023) Optimal decision-making in high-throughput virtual screening pipelines. Patterns4(11): 100875. DOI: 10.1016/j.patter.2023.100875.
50.
WozniakJMJainRBalaprakashP, et al. (2018) CANDLE/Supervisor: a workflow framework for machine learning applied to cancer research. BMC Bioinformatics19(S18): 491.
51.
YildizOMorozovDNicolaeB, et al. (2021) Dynamic heterogeneous task specification and execution for in situ workflows. In: Workshop on Workflows in Support of Large-Scale Science, Denver, Colorado, USA, Nov 12, 2023 - Nov 17, 2023.
52.
ZamoraYWardLSivaramanG, et al. (2021) Proxima: accelerating the integration of machine learning in atomistic simulations. In: Proceedings of the ACM International Conference on Supercomputing, ICS ’21. New York: ACM. DOI: 10.1145/3447818.3460370.
53.
ZhangYWangHChenW, et al. (2020) DP-GEN: a concurrent learning platform for the generation of reliable deep learning based potential energy models. Computer Physics Communications253: 107206. DOI: 10.1016/j.cpc.2020.107206.
54.
ZvyaginMBraceAHippeK, et al. (2023) GenSLMs: genome-scale language models reveal SARS-CoV-2 evolutionary dynamics. The International Journal of High Performance Computing Applications37(6): 683–705. DOI: 10.1177/10943420231201154.
