Influence of YARN schedulers on power consumption and processing time for various big data benchmarks

• Autorzy: Drypczewski Krzysztof , Proficz Jerzy , Stepnowski Andrzej

Identyfikatory

DOI

10.17466/tq2018/22.4/c

• Języki publikacji: EN

Abstrakty

EN

Climate change caused by human activities can influence the lives of everybody on the planet. The environmental concerns must be taken into consideration by all fields of study includingICT. Green Computing aims to reduce negative effects of IT on the environment while, at the same time, maintaining all of the possible benefits it provides. Several Big Data platforms like Apache Spark or YARN have become widely used in analytics and High-Performance Computing systems due to the reliability and usability of Map Reduce implementations. The authors research the power consumption and energy efficiency of Hadoop YARN schedulers using Apache Spark under three different workloads. The test cases include: sorting large binary files,counting unique words in large text files and processing satellite imagery from the Sentinel-2mission. The presented results show small (2%–11%) but distinct differences in the power consumption of FIFO and FAIR schedulers.

Słowa kluczowe

EN

Apache Spark; YARN; big data; green computing; Sentinel; Tera Sort; word count; benchmarks; scheduler

• Wydawca: Politechnika Gdańska
• Czasopismo: TASK Quarterly : scientific bulletin of Academic Computer Centre in Gdansk
• Rocznik: 2018
• Tom: Vol. 22, No 4
• Strony: 303--312
• Opis fizyczny: Bibliogr. 24 poz., rys., tab.

Twórcy

autor

Drypczewski Krzysztof

• Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Narutowicza 11/12, 80–233, Gdańsk, Poland

autor

Proficz Jerzy

• Centre of Informatics – Tricity Academic Supercomputer & Network, Gdansk University of Technology, Narutowicza 11/12, 80–233, Gdańsk, Poland

autor

Stepnowski Andrzej

• Faculty of Electronics, Telecommunications and Informatics, Gdansk University of Technology, Narutowicza 11/12, 80–233, Gdańsk, Poland

Bibliografia

• [1]USGCRP 2017 Climate Science Special Report: Fourth National Climate Assessment, U. S. Global Change Research Program, Washington,I 470doi: https://doi.org/10.7930/J0J964J6
• [2] Biswajit S 2014 Green Computing. International Journal of Computer Trends and Technology (IJCTT) 14(2)
• [3] Molla A, Cooper V and Pittayachawan S 2009 IT and Eco-sustainability: Developing and Validating a Green IT Readiness Model, ICIS 2009 Procs.
• [4] Pence H 2014 Journal of Educational Technology Systems 43(2)159 doi: https://doi.org/10.2190/ET.43.2.d
• [5] Apache Hadoop Homepage [online] http://hadoop.apache.org/ [Accessed: 28-March-2018]
• [6] Apache Spark Homepage [online] https://spark.apache.org/ [Accessed: 28-March-2018]
• [7] Almeida F, Arteaga J, Blanco V and Cabrera A 2005 Supercomputing Frontiers AndInnovations 2(2)64 doi: http://dx.doi.org/10.14529/jsfi150204
• [8] Proficz J and Czarnul P 2016 Performance and Power-Aware Modeling o fMPI Applications for Cluster Computing, in Wyrzykowski R, Deelman E, Dongarra J, Karczewski K, Kitowski J and Wiatr K (eds.), Parallel Processing and Applied Mathematics. Lecture Notes in Computer Science 95 74, Springer, Cham, https://link.springer.com/chapter/10.1007%2F978-3-319-32152-319
• [9] Czarnul P, Kuchta J, Rościszewski P and Proficz J 2016 Procs. 2016 Federated Conferenceon Computer Science and Information Systems Ganzha M, Maciaszek L and Paprzycki M (eds.)88 55
• [10] Czarnul P, Kuchta J, Matuszek M, Proficz J, Rościszewski P, Szymański J and Wójcik M 2017 Simulation Modelling Practice and Theory 7 7124 doi: 10.1016/j.simpat.2017.05.009
• [11] Appuswamy R, Gkantsidis C, Narayanan D, Hodson O and Rowstron A 2013 Scale-upvs scale-out for Hadoop: time to rethink?,SoCC’13, Santa Clara, California, USA. ACM 978-1-4503-2428-1doi: http://dx.doi.org/10.1145/2523616.2523629
• [12] Schall D, Hudlet V and Härder T 2010 Procs. Third C* Conference on Computer Science and Software Engineering,ACM, New York 1 doi: 10.1145/1822327.1822328
• [13] Schall D and Hudlet V 2011 Watt DB: an energy-proportional cluster of wimpy nodes,in Proceedings of the 2011 ACM SIGMOD International Conference on Management of data, SIGMOD’11 1229, publisher ACM
• [14] Haidar A, Jagode H, YarKhan A, Vaccaro P, Tomov S and Dongarra J 2017 Power-aware Computing: Measurement, Control, and Performance Analysis for Intel Xeon Phi,IEEE High Performance Extreme Computing Conference (HPEC’17)
• [15]PAPI homepage [online] http://icl.cs.utk.edu/papi
• /[16] Krzywaniak A, Proficz J and Czarnul P 2018 Analyzing energy/performance trade-off swith power capping for parallel applications on modern multi and many core processors15 339 doi: https://doi.org/10.15439/2018f177
• [17] Karau H 2013 Fast data processing with spark, Packt Publishing Ltd.
• [18] Yarn and Map Reduce Schedulers [online] https://www.cloudera.com/documentation/en-terprise/5-8-x/topics/adminschedulers.html [Accessed: 28-March-2018]
• [19] Apache Hadoop YARN– Resource Manager, Vinod Kumar Vavilapalli, Hortonworks Homepage [online] https://hortonworks.com/blog/apache-hadoop-yarn-resourcemanager/[Accessed: 24-Februrary-2018]
• [20] HDFS Introduction [online] https://hortonworks.com/apache/hdfs/[Accessed: 20-March-2018
• ][21] HDFS design [online] https://hadoop.apache.org/docs/r1.2.1/hdfsdesign.html[Accessed: 10-March-2018]
• [22] Drypczewski K and Proficz J 2017 TASK Quarterly 21(4)365
• [23] Krawczyk H, Nykiel M and Proficz J 2015 Polish Maritime Research 22(3)99doi: https://doi.org/10.1515/pomr-2015-0062
• [24] Hart M 2004 Project Gutenberg Mission Statement [online] https://www.gutenberg.org/wiki/Gutenberg:ProjectGutenbergMissionStatementbyMichaelHart [ Accessed: 20-March-2018]

Uwagi

PL

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2019).