Approaches to Distributed Execution of Scientific Workflows in Kepler

• Autorzy: Płóciennik M. , Żok T. , Altintas I. , Wang J. , Crawl D. , Abramson D. , Imbeaux F. , Guillerminet B. , Lopez-Caniego M. , Campos Plasencia I. , Pych W. , Ciecieląg P. , Palak B. , Owsiak M. , Frauel Y.
• Języki publikacji: EN

Abstrakty

EN

The Kepler scientific workflow system enables creation, execution and sharing of workflows across a broad range of scientific and engineering disciplines while also facilitating remote and distributed execution of workflows. In this paper, we present and compare different approaches to distributed execution of workflows using the Kepler environment, including a distributed data-parallel framework using Hadoop and Stratosphere, and Cloud and Grid execution using Serpens, Nimrod/K and Globus actors. We also present real-life applications in computational chemistry, bioinformatics and computational physics to demonstrate the usage of different distributed computing capabilities of Kepler in executable workflows. We further analyze the differences of each approach and provide a guidance for their applications.

Słowa kluczowe

EN

distributed execution; Kepler; scientific workflow

• Wydawca: IOS Press
• Czasopismo: Fundamenta Informaticae
• Rocznik: 2013
• Tom: Vol. 128, nr 3
• Strony: 281--302
• Opis fizyczny: Bibliogr. 28 poz., rys., tab.

Twórcy

autor

Płóciennik M.

• Poznań Supercomputing and Networking Center, IChB PAS, Poland

autor

Żok T.

• Poznań Supercomputing and Networking Center, IChB PAS, Poland

autor

Altintas I.

• San Diego Supercomputer Center, University of California San Diego, USA

autor

Wang J.

• San Diego Supercomputer Center, University of California San Diego, USA

autor

Crawl D.

• San Diego Supercomputer Center, University of California San Diego, USA

autor

Abramson D.

• Faculty of Information Technology, Monash University, Clayton, Australia

autor

Imbeaux F.

• CEA, IRFM, France

autor

Guillerminet B.

• CEA, IRFM, France

autor

Lopez-Caniego M.

• Instituto de Fisica de Cantabria, CSIC, Spain

autor

Campos Plasencia I.

• Instituto de Fisica de Cantabria, CSIC, Spain

autor

Pych W.

• Nicolaus Copernicus Astronomical Center PAS, Poland

autor

Ciecieląg P.

• Nicolaus Copernicus Astronomical Center PAS, Poland

autor

Palak B.

• Poznań Supercomputing and Networking Center, IChB PAS, Poland

autor

Owsiak M.

• Poznań Supercomputing and Networking Center, IChB PAS, Poland

autor

Frauel Y.

• CEA, IRFM, France

Bibliografia

• [1] Abramson, D., Bethwaite, B., Enticott, C., Garic, S., Peachey, T.: Parameter Exploration in Science and Engineering Using Many-Task Computing, IEEE Transactions on Parallel and Distributed Systems, 22, 2011, 960-973, ISSN 1045-9219.
• [2] Abramson, D., Enticott, C., Altinas, I.: Nimrod/K: towards massively parallel dynamic grid workflows, SC ’08: Proceedings of the 2008 ACM/IEEE conference on Supercomputing, IEEE Press, Piscataway, NJ, USA, 2008.
• [3] Alard, C., Lupton, R. H.: A Method for Optimal Image Subtraction, The Astrophysical Journal, 503(1), 1998, 325-331.
• [4] Altintas, I., Berkley, C., Jaeger, E., Jones, M., Ludaescher, B., Mock, S.: Kepler: An Extensible System for Design and Execution of Scientific Workflows, IN SSDBM, 2004.
• [5] Altintas, I., Wang, J., Crawl, D., Li, W.: Challenges and approaches for distributed workflow-driven analysis of large-scale biological data, Proceedings of the 2012 Joint EDBT/ICDT Workshops, ACM, 2012.
• [6] Altschul, S. F., Gish, W., Miller, W., Myers, E. W., Lipman, D. J.: Basic Local Alignment Search Tool, Journal of Molecular Biology, 215(3), 1990, 403 -410, ISSN 0022-2836.
• [7] Battre, D., Ewen, S., Hueske, F., Kao, O., Markl, V., Warneke, D.: Nephele/PACTs: A Programming Model and Execution Framework for Web-Scale Analytical Processing, Proceedings of the 1st ACM symposium on Cloud computing, SoCC ’10, ACM, New York, NY, USA, 2010, ISBN 978-1-4503-0036-0.
• [8] Cabellos, L., Campos, I., del Castillo, E. F., Owsiak, M., Palak, B., Płóciennik, M.: Scientific workflow orchestration interoperating HTC and HPC resources, Computer Physics Communications, 182(4), 2011, 890 - 897, ISSN 0010-4655.
• [9] Chapman, B., Jost, G., van der Pas, R., Kuck, D.: Using OpenMP: Portable Shared Memory Parallel Programming, The MIT Press, Cambridge, MA, USA, 2007.
• [10] Churches, D., Gombas, G., Harrison, A., Maassen, J., Robinson, C., Shields, M., Taylor, I., Wang, I.: Programming scientific and distributed workflow with Triana services: Research Articles, Concurr. Comput. : Pract. Exper., 18, August 2006, 1021-1037, ISSN 1532-0626.
• [11] Crawl, D., Altintas, I.: A Provenance-Based Fault Tolerance Mechanism for Scientific Workflows, Provenance and Annotation of Data and Processes (IPAW 2008, Revised Selected Papers) (J. Freire, D. Koop, L. Moreau, Eds.), 5272, Springer, 2008.
• [12] Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters, Communications of the ACM, 51(1), 2008, 107-113.
• [13] Deelman, E., Blythe, J., Gil, Y., Kesselman, C., Mehta, G., Patil, S., Su, M.-H., Vahi, K., Livny, M.: Pegasus: Mapping Scientific Workflows onto the Grid, in: Grid Computing (M. Dikaiakos, Ed.), vol. 3165 of Lecture Notes in Computer Science, chapter 2, Springer Berlin / Heidelberg, Berlin, Heidelberg, 2004, ISBN 978-3540-22888-2, 131-140.
• [14] Gropp, W., Lusk, E., Skjellum, A.: Using MPI: Portable Parallel Programming with the Message Passing Interface, Scientific and Engineering Computation Series, 2nd edition edition, MIT Press, Cambridge, MA, USA, 1999.
• [15] Gu, Y., Grossman, R.: Sector and Sphere: The Design and Implementation of a High Performance Data Cloud, Philosophical Transactions of the Royal Society A, 367(1897), June 2009, 2429-2445.
• [16] Guillerminet, B., Plasencia, I. C., Haefele, M., Iannone, F., Jackson, A., Manduchi, G., Plociennik, M., Sonnendrucker, E., Strand, P., Owsiak, M.: High Performance Computing tools for the Integrated Tokamak Modelling project, Fusion Engineering and Design, 85(34), 2010, 388 - 393, ISSN 0920-3796, Proceedings of the 7th IAEA Technical Meeting on Control, Data Acquisition, and Remote Participation for Fusion Research.
• [17] Hull, D., Wolstencroft, K., Stevens, R., Goble, C., Pocock, M. R., Li, P., Oinn, T.: Taverna: a tool for building and running workflows of services, Nucleic Acids Research, 34(suppl 2), 1 July 2006, W729-W732.
• [18] Kacsuk, P., Sipos, G.: Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal, Journal of Grid Computing, 3(3), September 2005, 221-238, ISSN 1570-7873.
• [19] Köhler, S., Riddle, S., Zinn, D., McPhillips, T., Ludascher, B.: Improving workflow fault tolerance through provenance-based recovery, Proceedings of the 23rd international conference on Scientific and statistical database management, SSDBM’11, Springer-Verlag, Berlin, Heidelberg, 2011, ISBN 978-3-642-22350-1.
• [20] Kozlovszky, M., Karoczkai, K., Marton, I., Balasko, A., Marosi, A., Kacsuk, P.: Enabling Generic Distributed Computing Infrastructure Compatibility for Workflow Management Systems, Computer Science, 13(3), 2012, 61-78.
• [21] Moretti, C., Bui, H., Hollingsworth, K., Rich, B., Flynn, P., Thain, D.: All-Pairs: An Abstraction for Data- Intensive Computing on Campus Grids, IEEE Transactions on Parallel and Distributed Systems, 21, 2010, 33-46, ISSN 1045-9219.
• [22] Scott, B. D.: Free-energy conservation in local gyrofluid models, Physics of Plasmas, 12(10), 2005, 102307.
• [23] Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: The Condor experience, Concurrency Computation Practice and Experience, 17(2-4), 2005, 323-356, Cited By (since 1996) 440.
• [24] Wang, J., Altintas, I.: Early Cloud Experiences with the Kepler Scientific Workflow System, Procedia Computer Science, 9, 2012, 1630-1634.
• [25] Wang, J., Altintas, I., Hosseini, P. R., Barseghian, D., Crawl, D., Berkley, C., Jones, M. B.: Accelerating Parameter Sweep Workflows by Utilizing Ad-Hoc Network Computing Resources: An Ecological Example, IEEE Congress on Services, IEEE Computer Society, 2009.
• [26] Wang, J., Crawl, D., Altintas, I.: Kepler + Hadoop: A General Architecture Facilitating Data-Intensive Applications in Scientific Workflow Systems, Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS ’09, ACM New York, NY, USA, Portland, Oregon, 2009.
• [27] Wang, J., Crawl, D., Altintas, I.: A framework for distributed data-parallel execution in the Kepler scientific workflow system, Procedia Computer Science, 9, 2012, 1620-1629.
• [28] Wang, J., Korambath, P., Kim, S., Johnson, S., Jin, K., Crawl, D., Altintas, I., Smallen, S., Labate, B., Houk, K.: Facilitating e-Science Discovery Using Scientific Workflows on the Grid, Guide to e-Science: Next Generation Scientific Research and Discovery, 2011, 353-382.