Scientific workflows are frequently being used to model complex phenomena, to analyze instrumental data, to tie together information from distributed sources, and to pursue other scientific endeavors. Out of necessity, these complex applications need to execute in distributed environments and make use of a number of heterogeneous resources. In this paper we describe some of these applications and illustrate techniques that improve their performance and reliably in distributed environments, such as grids and clouds. Although clouds were first introduced in the business arena, they have a potential to be provide on-demand resources for scientific computations.
Andrews, T., Curbera, F., Dholakia, H., Goland, Y., Klein, J., Leymann, F., Liu, K., Roller, D., Smith, D., Thatte, S., Trickovic, I. and Weerawarana, S. ( 2003). Specification: business process execution language for Web Services Version 1.1, http://www.-106.ibm.com/developerworks/webservices/library/ws-bpel/
4.
Barish, B.C. and Weiss, R. ( 1999). LIGO and the detection of gravitational waves . Phys Today52: 44.
5.
Berriman, B., Bergou, A., Deelman, E., Good, J., Jacob, J., Katz, D., Kesselman, C., Laity, A., Singh, G., Su, M.-H. and Williams, R. ( 2003). Montage: a grid-enabled image mosaic service for the NVO. In the Proceedings of Astronomical Data Analysis Software & Systems (ADASS) XIII.
6.
Berriman, G.B., Deelman, E., Good, J., Jacob, J., Katz, D.S., Kesselman, C., Laity, A., Prince, T.A., Singh, G. and Su, M.-H. ( 2004). Montage: a grid enabled engine for delivering custom science-grade mosaics on demand. In Proceedings of SPIE Conference 5487: Astronomical Telescopes.
7.
Berriman, G.B., Good, J.C., Deelman, E., Singh, G. and Livny, M. ( 2008). A cost benefit study of doing astrophysics on the cloud: production of image mosaics. In the Proceedings of Astronomical Data Analysis Software and Systems (ADASS).
8.
Bharathi, S., Chervenak, A., Deelman, E., Mehta, G., Su, M.-H. and Vahi, K. ( 2008). Characterization of scientific workflows. In Proceedings of the 3rd Workshop on Workflows in Support of Large-Scale Science (WORKS08) .
9.
Brown, D.A., Brady, P.R., Dietz, A., Cao, J., Johnson, B. and McNabb, J. ( 2006). A case study on the use of workflow technologies for scientific analysis: gravitational wave data analysis. In Workflows for e-Science, edited by I. Taylor , E. Deelman, D. Gannon and M. Shields , Springer.
10.
Callaghan, S., Maechling, P., Deelman, E., Vahi, K., Mehta, G., Juve, G., Milner, K., Graves, R., Field, E., Okaya, D., Gunter, D., Beattie, K. and Jordan, T. ( 2008). Reducing time-to-solution using distributed high-throughput mega-workflows - experiences from SCEC CyberShake. In Proceedings of the 4th IEEE International Conference on e-Science (e-Science 2008), Indianapolis, IN.
11.
Callahan, S.P., Freire, J., Santos, E., Scheidegger, C.E., Silva, C.T. and Vo, H.T. ( 2006). VisTrails: visualization meets data management . In the Proceedings of ACM SIGMOD.
12.
Chervenak, A., Schuler, R., Kesselman, C., Koranda, S. and Moe, B. ( 2005). Wide area data replication for scientific collaborations . In Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing, pp. 1-8.
13.
Churches, D., Gombas, G., Harrison, A., Maassen, J., Robinson, C., Shields, M., Taylor, I. and Wang, I. ( 2006). Programming scientific and distributed workflow with Triana services. Concurrency Comput. Pract. Ex. 18: 1021-1037.
Couvares, P., Kosar, T., Roy, A., Weber, J. and Wenger, K. ( 2006). Workflow management in Condor. In Workflows in e-Science, edited by I. Taylor , E. Deelman, D. Gannon and M. Shields, Springer, pp. 357-375.
Deelman, E., Callaghan, S., Field, E., Francoeur, H., Graves, R., Gupta, N., Gupta, V., Jordan, T.H., Kesselman, C., Maechling, P., Mehringer, J., Mehta, G., Okaya, D., Vahi, K. and Zhao, L. ( 2006a). Managing large-scale workflow execution from resource provisioning to provenance tracking: the CyberShake example. In E-SCIENCE '06: Proceedings of the 2nd IEEE International Conference on e-Science and Grid Computing, p. 14.
19.
Deelman, E., Gannon, D., Shields, M. and Taylor, I. ( 2008a). Workflows and e-Science: an overview of workflow system features and capabilities. Future Generat. Comput Syst , doi:10.1016/j.future.2008.06.012.
20.
Deelman, E., Mehta, G., Singh, G., Su, M.-H. and Vahi, K. ( 2006b) Pegasus: mapping large-scale workflows to distributed resources. In Workflows in e-Science, edited by I. Taylor, E. Deelman, D. Gannon and M. Shields , Springer.
21.
Deelman, E., Singh, G., Livny, M., Berriman, B. and Good, J. ( 2008b). The cost of doing science on the cloud: the Montage example. In the Proceedings of SC'08, Austin, TX.
22.
Deelman, E., Singh, G., Su, M.-H., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Vahi, K., Berriman, G.B., Good, J., Laity, A., Jacob, J.C. and Katz, D.S. ( 2005). Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci. Program. 13: 219-237.
23.
Enabling Grids for E-sciencE (EGEE). http://www.eu-egee.org/
Flexible Image Transport System. http://fits.gsfc.nasa.gov/
26.
Globus. http://www.globus.org
27.
28.
Graves, R. ( 2008). Physics based probabilistic seismic hazard calculations for Southern California. In Proceedings of the 14th World Conference on Earthquake EngineeringBeijing , China.
29.
Hoffa, C., Mehta, G., Freeman, T., Deelman, E., Keahey, K., Berriman, B. and Good, J. ( 2008). On the Use of cloud computing for scientific workflows . In Proceedings of the 3rd International Workshop on Scientific Workflows and Business Workflow Standards in e-Science (SWBES), in conjunction with the 4th IEEE International Conference on e-Science (e-Science 2008), Indianapolis, IN.
30.
Ipaw (2006). Provenance and Annotation of Data, International Provenance and Annotation Workshop (IPAW 2006), Chicago, IL, May 3-5, Revised Selected Papers.
31.
Juve, G. and Deelman, E. ( 2008). Resource provisioning options for large-scale scientific workflows. In Proceedings of the 3rd International Workshop on Scientific Workflows and Business Workflow Standards in e-Science (SWBES), in conjunction with the 4th IEEE International Conference on e-Science (e-Science 2008), Indianapolis, IN.
32.
Juve, G., Deelman, E., Vahi, K., Mehta, G., Berriman, B., Berman, B.P. and Maechling, P. ( 2009). Scientific workflow applications on Amazon EC2 . In Cloud Computing Workshop in Conjunction with e-Science , Oxford, UK: IEEE.
33.
Kertész, A., Sipos, G. and Kacsuk, P. ( 2006). Brokering multi-grid workflows in the P-GRADE portal . In the Proceedings of Euro-Par2006 : Parallel Processing , vol. 4375, Berlin: Springer.
34.
Laszewski, G. v. and Hategan, M. ( 2006). Workflow concepts of the Java CoG kit. J. Grid Comput3: 239-258.
35.
Litzkow, M., Livny, M. and Mutka, M. ( 1988). Condor - a hunter of idle workstations. In Proceedings of the 8th International Conference on Distributed Computing Systems, pp. 104-111.
36.
Ludäscher, B., Altintas, I., Berkley, C., Higgins, D., Jaeger-Frank, E., Jones, M., Lee, E., Tao, J. and Zhao, Y. ( 2005). Scientific workflow management and the Kepler system . Concurrency Comput. Pract. 18: 1039-1065.
37.
Maechling, P., Deelman, E., Zhao, L., Graves, R., Mehta, G., Gupta, N., Mehringer, J., Kesselman, C., Callaghan, S., Okaya, D., Francoeur, H., Gupta, V., Cui, Y., Vahi, K., Jordan, T. and Field, E. ( 2006). SCEC CyberShake workflows-automating probabilistic seismic hazard analysis calculations. In Workflows for e-Science , edited by I. Taylor, E. Deelman, D. Gannon and M. Shields, Springer.
38.
Mandal, A., Kennedy, K., Koelbel, C., Marin, G., Mellor-Crummey, J., Liu, B. and Johnsson, L. ( 2005). Scheduling strategies for mapping application workflows onto the grid. In Proceedings of the IEEE International Symposium on High Performance Distributed Computing (HPDC) .
39.
Microsoft Workflow Foundation. (2007). http://msdn2.microsoft.com/en-us/netframework/aa663328.aspx
40.
Michener, W., Beach, J., Bowers, S., Downey, L., Jones, M., Ludäscher, B., Pennington, D., Rajasekar, A., Romanello, S., Schildhauer, M., Vieglais, D. and Zhang, J. ( 2005). Data Integration and Workflow Solutions for Ecology. Data Integration in the Life Sciences (DILS), San Diego, California, USA, July 20-22, 2005, pp. 321-324: Springer.
41.
Miles, S., Papay, J., Wroe, C., Lord, P., Goble, C. and Moreau, L. ( 2004). Semantic description, publication and discovery of workflows in myGrid. Technical Report ECSTR-IAM04-001, Electronics and Computer Science , University of Southampton.
42.
Moreno-Vozmediano, R., Montero, R. and Llorente, I. ( 2009). Elastic management of cluster-based services in the cloud . In Proceedings of the 1st Workshop on Automated Control for Datacenters and Clouds (ACDC09), pp. 19-24.
Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L. and Zagorodnov, D. ( 2008). The Eucalyptus Open-source Cloud-computing System . In the Proceedings of Cloud Computing and its Applications , Chicago.
46.
Oinn, T., Li, P., Kell, D.B., Goble, C., Goderis, A., Green-wood, M., Hull, D., Stevens, R., Turi, D. and Zhao, J. ( 2006). Taverna/myGrid: aligning a workflow system with the life sciences community. In Workflows in e-Science, edited by I. Taylor, E. Deelman, D. Gannon and M. Shields.Springer.
47.
Open Science Grid. www.opensciencegrid.org
48.
Pegasus . http://pegasus.isi.edu
49.
Piccoli, L. ( 2008). Lattice QCD workflows: a case study. In SWBES08: Challenging Issues in Workflow Applications, Indianapolis, IN.
50.
Plale, B., Gannon, D., Brotzge, J., Droegemeier, K., Kurose, J., McLaughlin, D., Wilhelmson, R., Graves, S., Ramamurthy, M. and Clark, R. ( 2006). CASA and LEAD: adaptive cyberinfrastructure for real-time multiscale weather forecasting. Computer39: 56-64.
51.
Ramakrishnan, A., Singh, G., Zhao, H., Deelman, E., Sakellariou, R., Vahi, K., Blackburn, K., Meyers, D. and Samidi, M. ( 2007). Scheduling data-intensive workflows onto storage-constrained distributed resources. In Proceedings of the 7th IEEE International Symposium on Cluster Computing and the Grid - CCGrid2007.
Shoshani, A., Bernardo, L.M., Nordberg, H., Rotem, D. and Sim, A. ( 1998). Storage management for high energy physics applications . In the Proceedings of Computing in High Energy Physics 1998 (CHEP 98).
54.
Singh, G., Kesselman, C. and Deelman, E. ( 2006). Application-level resource provisioning on the grid. In the Proceedings of e-Science, Amsterdam, The Netherlands.
55.
Singh, G., Su, M.H., Vahi, K., Deelman, E., Berriman, B., Good, J., Katz, D.S. and Mehta, G. ( 2008). Workflow task clustering for best effort systems with Pegasus. Proceedings of the 15th ACM Mardi Gras conference: From lightweight mash-ups to lambda grids: Understanding the spectrum of distributed computing requirements, applications, tools, infrastructures, interoperability, and the incremental adoption of key capabilities.
56.
Singh, G., Vahi, K., Ramakrishnan, A., Mehta, G., Deelman, E., Zhao, H., Sakellariou, R., Blackburn, K., Brown, D., Fairhurst, S., Meyers, D., Berriman, G.B., Good, J. and Katz, D.S. ( 2007). Optimizing workflow data footprint. Sci. Program. 15: 249-268.
57.
Skrutskie, M.F., Schneider, S.E., Stiening, R., Strom, S.E., Weinberg, M.D., Beichman, C., Chester, T., Cutri, R., Lonsdale, C. and Elias, J. ( 1997). The Two Micron All Sky Survey (2MASS): overview and status . In The Impact of Large Scale Near-IR Sky Surveys , (Garzon F, Epchtein, N, Omont, A, Burton, B, Persi, P eds), Dordrecht: Kluwer Academic Publishers, pp. 25.
58.
Slominski, A. ( 2006). Adapting BPEL to scientific workflows. In Workflows for e-Science, edited by I. Taylor , E. Deelman, D. Gannon and M. Shields , Springer.
59.
Southern California Earthquake Center (SCEC) ( 2006). http://www.scec.org/
60.
Stevens, R.D., Robinson, A.J. and Goble, C.A. ( 2003). myGrid: personalised bioinformatics on the information grid. In Bioinformatics (11th International Conference on Intelligent Systems for Molecular Biology), vol. 19.
61.
Taylor, I., Deelman, E., Gannon, D. and Shields, M. eds. (2006). Workflows in e-Science, Springer.
62.
Taylor, I., Shields, M., Wang, I. and Philp, R. ( 2003). Distributed P2P computing within Triana: a galaxy visualization test case. In the Proceedings of IPDPS2003 .
63.
TeraGrid . (2004). http://www.teragrid.org/
64.
Walker, E. ( 2008). Continuous adaptation for high performance throughput computing across distributed clusters. In Proceedings of the 2008 IEEE International Conference on Cluster Computing, pp. 369-375.
65.
Walker, E. and Guiang, C. ( 2007). Challenges in executing large parameter sweep studies across widely distributed computing environments. In Proceedings of the 5th IEEE workshop on challenges of large applications in distributed environment, pp. 11-18.
66.
Wang, L., Tao, J., Kunze, M., Rattu, D. and Castellanos, A.C. ( 2008). The Cumulus Project: build a scientific cloud for a data center. In the Proceedings of Cloud Computing and its Applications, Chicago, IL.
67.
Wieczorek, M., Prodan, R. and Fahringer, T. ( 2005). Scheduling of scientific workflows in the ASKALON grid environment. ACM SIGMOD SIGMOD Rec.34.
