Multi-queue service for task scheduling based on data availability

• Autorzy: Rybiński Kamil , Śmiałek Michał

Identyfikatory

DOI

10.15439/2023F8957

• Języki publikacji: EN

Abstrakty

EN

Large-scale computation (LSC) systems are often performed in distributed environments where message passing is the key to orchestrating computations. In this paper we present a new message queue concept developed within the context of an LSC system (BalticLSC). The concept consists in proposing a multi-queue, where queues are grouped into families. A queue family can be used to distribute messages of the same kind to multiple computation modules distributed between various nodes. Such message families can be synchronised to implement a mechanism for initiating computation jobs based on multiple data inputs. Moreover, the proposed multi-queue has built-in mechanisms for controlling message sequences in applications where complex data set splitting is necessary. The presented multi-queue concept was implemented and applied with success in a working LSC system.

Słowa kluczowe

EN

computer science; processor; message passing; distributed databases; data processing; task analysis

PL

informatyka; procesor; komunikowanie; rozproszone bazy danych; przetwarzanie danych; analiza zadań

• Wydawca: Polskie Towarzystwo Informatyczne
• Czasopismo: Annals of Computer Science and Information Systems
• Rocznik: 2023
• Tom: Vol. 35
• Strony: 699--709
• Opis fizyczny: Bibliogr. 25 poz., il.

Twórcy

autor

Rybiński Kamil

• Warsaw University of Technology pl. Politechniki 1, 00-661 Warszawa, Poland

• https://orcid.org/0000-0002-5543-790X

autor

Śmiałek Michał

• Warsaw University of Technology pl. Politechniki 1, 00-661 Warszawa, Poland

• https://orcid.org/0000-0001-6170-443X

Bibliografia

• 1. Adam Barker, Paolo Besana, David Robertson, and Jon B. Weissman. The benefits of service choreography for data-intensive computing. In Proceedings of the 7th International Workshop on Challenges of Large Applications in Distributed Environments, CLADE ’09, page 1–10, New York, NY, USA, 2009. Association for Computing Machinery.
• 2. Li Chunlin, Tang Jianhang, and Luo Youlong. Multi-queue scheduling of heterogeneous jobs in hybrid geo-distributed cloud environment. The Journal of Supercomputing, 74:5263–5292, 2018.
• 3. Philippe Dobbelaere and Kyumars Sheykh Esmaili. Kafka versus RabbitMQ: A comparative study of two industry reference publish/subscribe implementations: Industry paper. In Proceedings of the 11th ACM International Conference on Distributed and Event-Based Systems, DEBS ’17, page 227–238, New York, NY, USA, 2017. Association for Computing Machinery.
• 4. Nishant Garg. Apache Kafka. Packt Publishing Birmingham, UK, 2013.
• 5. Mohammad Hedayati, Kai Shen, Michael L Scott, and Mike Marty. Multi-queue fair queuing. In USENIX Annual Technical Conference, pages 301–314, 2019.
• 6. Péter Kacsuk, editor. Science Gateways for Distributed Computing Infrastructures. Springer International Publishing, 2014.
• 7. Peter Kacsuk, Zoltan Farkas, Miklos Kozlovszky, Gabor Hermann, Akos Balasko, Krisztian Karoczkai, and Istvan Marton. WS-PGRADE/gUSE generic DCI gateway framework for a large variety of user communities. Journal of Grid Computing, 10(4):601–630, nov 2012.
• 8. Peter Kacsuk, József Kovács, and Zoltán Farkas. The Flowbster cloud-oriented workflow system to process large scientific data sets. Journal of Grid Computing, 16(1):55–83, jan 2018.
• 9. A V Karthick, E Ramaraj, and R Ganapathy Subramanian. An efficient multi queue job scheduling for cloud computing. In 2014 World Congress on Computing and Communication Technologies, pages 164–166, 2014.
• 10. Haopeng Li and Hui Li. A scheduling strategy based on multi-queues of cassandra. In 2017 IEEE International Conference on Big Data (Big Data), pages 2664–2669, 2017.
• 11. Ji Liu, Esther Pacitti, Patrick Valduriez, and Marta Mattoso. A survey of data-intensive scientific workflow management. Journal of Grid Computing, 13:457–493, 2015.
• 12. Pedro García López, Aitor Arjona, Josep Sampé, Aleksander Slominski, and Lionel Villard. Triggerflow: Trigger-based orchestration of serverless workflows. In Proceedings of the 14th ACM International Conference on Distributed and Event-Based Systems, DEBS ’20, page 3–14, New York, NY, USA, 2020. Association for Computing Machinery.
• 13. Krzysztof Marek, Michał Śmiałek, Kamil Rybiński, Radosław Roszczyk, and Marek Wdowiak. BalticLSC: Low-code software development platform for large scale computations. Computing and Informatics, 40(4):734–753, 2021.
• 14. Chris Peltz. Web services orchestration and choreography. Computer, 36(10):46–52, 2003.
• 15. Anastasiia Postnikova, Nikita Koval, Giorgi Nadiradze, and Dan Alistarh. Multi-queues can be state-of-the-art priority schedulers. In Proceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 353–367, 2022.
• 16. Stephen Ross-Talbot. Orchestration and choreography: Standards, tools and technologies for distributed workflows. In NETTAB Workshop-Workflows management: new abilities for the biological information overflow, volume 1, page 8, Naples, Italy, 2005.
• 17. Maciej Rostanski, Krzysztof Grochla, and Aleksander Seman. Evaluation of highly available and fault-tolerant middleware clustered architectures using RabbitMQ. In 2014 Federated Conference on Computer Science and Information Systems, pages 879–884, 2014.
• 18. Radoslaw Roszczyk, Marek Wdowiak, Michal Smialek, Kamil Rybinski, and Krzysztof Marek. BalticLSC: A low-code HPC platform for small and medium research teams. In 2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, oct 2021.
• 19. Kamil Rybiński, Michał Śmiałek, Agris Sostaks, Krzysztof Marek, Radosław Roszczyk, and Marek Wdowiak. Visual low-code language for orchestrating large-scale distributed computing. Journal of Grid Computing, 21(3), jul 2023.
• 20. Gaurav Sharma, Neha Miglani, and Ajay Kumar. PLB: a resilient and adaptive task scheduling scheme based on multi-queues for cloud environment. Cluster Computing, 24(3):2615–2637, 2021.
• 21. T Sharvari and Nag K Sowmya. A study on modern messaging systems-Kafka, RabbitMQ and NATS streaming. 2019.
• 22. Jaspreet Singh and Deepali Gupta. An smarter multi queue job scheduling policy for cloud computing. International Journal of Applied Engineering Research, 12(9):1929–1934, 2017.
• 23. John Vineet and Liu Xia. A survey of distributed message broker queues. 2017.
• 24. Steve Vinoski. Advanced Message Queuing Protocol. IEEE Internet Computing, 10(6):87–89, 2006.
• 25. Guozhang Wang, Joel Koshy, Sriram Subramanian, Kartik Paramasivam, Mammad Zadeh, Neha Narkhede, Jun Rao, Jay Kreps, and Joe Stein. Building a replicated logging system with Apache Kafka. Proceedings of the VLDB Endowment, 8(12):1654–1655, aug 2015.

Uwagi

1. Thematic Tracks Regular Papers

2. Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2024).