Infrastructure and Energy Conservation in Big Data Computing: A Survey

• Autorzy: Niewiadomska-Szynkiewicz Ewa , Karpowicz Michał P.

Identyfikatory

DOI

10.26636/jtit.2019.132419

• Języki publikacji: EN

Abstrakty

EN

Progress in life, physical sciences and technology depends on efficient data-mining and modern computing technologies. The rapid growth of data-intensive domains requires a continuous development of new solutions for network infrastructure, servers and storage in order to address Big Datarelated problems. Development of software frameworks, include smart calculation, communication management, data decomposition and allocation algorithms is clearly one of the major technological challenges we are faced with. Reduction in energy consumption is another challenge arising in connection with the development of efficient HPC infrastructures. This paper addresses the vital problem of energy-efficient high performance distributed and parallel computing. An overview of recent technologies for Big Data processing is presented. The attention is focused on the most popular middleware and software platforms. Various energy-saving approaches are presented and discussed as well.

Słowa kluczowe

EN

big data; cloud; cluster; energy-efficient computation; grid; HPC; software platform

• Wydawca: Instytut Łączności - Państwowy Instytut Badawczy
• Czasopismo: Journal of Telecommunications and Information Technology
• Rocznik: 2019
• Tom: nr 2
• Strony: 73--82
• Opis fizyczny: Bibliogr. 90 poz., rys.

Twórcy

autor

Niewiadomska-Szynkiewicz Ewa

• Institute of Control and Computation Engineering, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, Poland

autor

Karpowicz Michał P.

• Institute of Control and Computation Engineering, Warsaw University of Technology, Nowowiejska 15/19, 00-665 Warsaw, Poland

Bibliografia

• [1] P. D. Healy, T. Lynn, E. Barrett, and J. P. Morrison, “Single system image: A survey”, J. Parallel Distrib. Comput., vol. 90–91, pp. 35–51, 2016 (10.1016/j.jpdc.2016.01.004).
• [2] A. Oussous, F. Z. Benjelloun, A. A. Lahcen, and S. Bel ih, “Big data technologies: A survey”, J. of King Saud Univer. – Comp. and Inform. Sci., vol. 30, no. 4, pp. 431–448, 2018 (doi: 10.1016/j.jksuci.2017.06.001).
• [3] ETP4HPC Strategic Research Agenda achieving HPC leadership in Europe [Online]. Available: www.etp4hpc.eu
• [4] IEEE 802.3az-2010 – IEEE standard for information technology [Online]. Available: https://standards.ieee.org/standard/ 802 3az-2010.html
• [5] Mosix home page [Online]. Available: www.mosix.org
• [6] OpenSSI home page [Online]. Available: www.openssi.org/cgi-bin/ view?page=openssi.html
• [7] Kerrighed home page [Online]. Available: www.kerrighed.org
• [8] F. Berman, G. Fox, and A. J. G. Hey, Eds., Grid Computing: Making the Global Infrastructure a Reality. Wiley, 2003 (ISBN: 978-0-470-85319-1).
• [9] Unicore home page [Online]. Available: www.unicore.eu
• [10] Globus toolkit home page [Online]. Available: toolkit.globus.org
• [11] M. Cannataro, Handbook of Research on Computational Grid Technologies for Life Sciences, Biomedicine, and Healthcare. Hershey, PA, USA: IGI Global, 2009 (ISBN-13: 978-1605663746).
• [12] R. J. Walters, S. Crouch, and P. Bennett, “Building computational grids using ubiquitous Web technologies”, in Collaborative Networks in the Internet of Services. 13th IFIP WG 5.5 Working Conference on Virtual Enterprises, PRO-VE 2012, Bournemouth, UK, October 1-3, 2012. Proceedings, L. M. Camarinha-Matos, L. Xu, and H. Afsarmanesh, Eds. IFIPAICT, vol. 380, pp. 254–261. Berlin, Heidelberg: Springer, 2012 (doi: 10.1007/978-3-642-32775-9 26).
• [13] S. Chaudhary, G. Somani, and R. Buyya, Eds., Research Advances in Cloud Computing. Springer, 2017 (doi: 10.1007/978-981-10-5026-8).
• [14] N. Sehgal and P. Ch. P. Bhatt, Cloud Computing. Concepts and Practices. Springer, 2018 (ISBN: 978-3-319-77839-6).
• [15] W.-M. Hwu, Ed., GPU Computing Gems Emerald Edition. Morgan Kaufman, 2011 (ISBN: 9780123849885).
• [16] A. B. Singh, J. S. Bhat, R. Raju, and R. D’Souza, “Survey on various load balancing techniques in cloud computing”, Adv. in Comput., vol. 7, no. 2, pp. 28–34, 2017 (doi: 10.5923/j.ac.20170702.04).
• [17] A. Thakur and M. S. Goraya, “A taxonomic survey on load balancing in cloud”, J. of Netw. and Comp. Appl., vol. 98, pp. 43–57, 2017 (doi: 10.1016/j.jnca.2017.08.020).
• [18] J. Zhang et al., “Load balancing in data center networks: A survey”, IEEE Commun. Surv. Tutor., vol. 20, no. 3, pp. 2324–2352, 2018 (doi: 10.1109/COMST.2018.2816042).
• [19] G. Staples, “TORQUE resource manager”, in Proc. of the 2006 ACM/IEEE Conf. on Supercomput. SC’06, Tampa, FL, USA, 2006, Article no. 8 (doi: 10.1145/1188455.1188464).
• [20] Portable batch system home page [Online]. Available: www.pbspro.org
• [21] Slurm workload manager home page [Online]. Available: slurm.schedmd.com
• [22] E. Mohamed and Z. Hong, “Hadoop-mapreduce job scheduling algorithms survey”, in Proc. 2016 7th Int. Conf. on Cloud Comput. and Big Data CCBD 2016, Macau, China, 2016, pp. 237–242, 2016 (doi: 10.1109/CCBD.2016.054).
• [23] T. White, Hadoop: The Definitive Guide. O’Reilly Media, 2015 (ISBN: 9781491901687).
• [24] Apache Hadoop home page [Online]. Available: https://hadoop.apache.org
• [25] Apache Spark home page [Online]. Available: spark.apache.org
• [26] Apache Storm home page [Online]. Available: storm.apache.org
• [27] Apache Flink home page [Online]. Available: https://flink.apache.org
• [28] Rapidminer studio home page [Online]. Available: https://rapidminer.com/
• [29] Orange home page [Online]. Available: orange.biolab.si
• [30] E. Frank, M. A. Hall, and I. H. Witten, Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, 2016 (ISBN: 9780123748560).
• [31] I. Milne, M. Bayer, L. Cardle, P. Shaw, G. Stephen, F. Wright, and D. Marshall, “Tablet – next generation sequence assembly visualization”, Bioinformatics, vol. 26, no. 3, pp. 401–403, 2010 (doi: 10.1093/bioinformatics/btp666).
• [32] T. Carver, T. D. Bohme, U. Otto, J. Parkhill, and M. Berriman, “Bamview: viewing mapped read alignment data in the context of the reference sequence”, Bioinformatics, vol. 26, no. 5, pp. 676–673, 2010 (doi: 10.1093/bioinformatics/btq010).
• [33] K. Rutherford et al., “Artemis: sequence visualization and annotation”, Bioinformatics, vol. 16, no. 10, pp. 944–949, 2000 (doi: 10.1093/bioinformatics/16.10.944).
• [34] T. Carver, S. R. Harris, M. Berriman, J. Parkhill, and J. A. McQuillan, “Artemis: an integrated platform for visualization and analysis of high-throughput sequence-based experimental data”, Bioinformatics, vol. 28, no. 4, pp. 464–469, 2012 (doi: 10.1093/bioinformatics/btr703).
• [35] D. Desale, Top tools for social network analysis and visualisation, 2018 [Online]. Available: https://www.kdnuggets.com/software/ social-network-analysis.html
• [36] A. Sikora and E. Niewiadomska-Szynkiewicz, “A federated approach to parallel and distributed simulation of complex systems. Int. J. of Appl. Mathem. and Comp. Sci., vol. 17, no. 1, pp. 99–106, 2007 (doi: 10.2478/v10006-007-0009-0).
• [37] A. Inostrosa-Psijas, V. Gil-Costa, M. Marin, and G. Wainer, “Semiasynchronous approximate parallel DEVS simulation of Web search engines”, Concurr. and Comput.: Pract. and Exper., vol. 30, no. 7, 2018 (doi: 10.1002/cpe.4149).
• [38] J. A. Miller, M. E. Cotterell, and S. J. Buckley, “Supporting a modeling continuum in scalation: from predictive analytics to simulation modeling”, in Proc. of 2013 Winter Simulations Conference WSC 2013, Washington, DC, USA, 2013, pp. 1191–1202 (doi: 10.1109/WSC.2013.6721507).
• [39] E. Niewiadomska-Szynkiewicz and A. Sikora, “A software tool for federated simulation of Wireless Sensor Networks and mobile ad hoc networks”, in Applied Parallel and Scientific Computing 10th International Conference, PARA 2010, Reykjavík, Iceland, June 6-9, 2010, Revised Selected Papers, Part I, K. Jónasson, Ed. LNCS, vol. 7133, pp. 303–313. Berlin, Heidelberg: Springer, 2012 (doi: 10.1007/978-3-642-28151-8 30).
• [40] X. Song, Y. Wu, Y. Ma, Y. Ciu, and G. Gong, “Military simulation big data: Background, state of the art, and challenges”, Mathem. Problems in Engin., vol. 2015, Article ID 298356, pp. 1–20, 2015 (doi: 10.1155/2015/298356).
• [41] A. K. Fidjeland, E. B. Roesch, M. P. Shanahan, and W. Luk, “Nemo: A platform for neural modelling of spiking neurons using GPUS”, in Proc. 2009 20th IEEE Int. Conf. on Appl.-specif. Syst., Architec. and Process., Boston, MA, USA, 2009, pp. 137–144 (doi: 10.1109/ASAP.2009.24).
• [42] P. Szynkiewicz, “A novel GPU-enabled simulator for large scale spiking neural networks”, J. of Telecommun. and Inform. Technol., no. 2, pp. 34–42, 2016 [Online]. Available: https://www.itl.waw.pl/ czasopisma/JTIT/2016/2/34.pdf
• [43] M. A. Martinez-del Amor et al., “Accelerated simulation of P systems on the GPU: A survey”, in Bio-Inspired Computing – Theories and Applications. 9th International Conference, BIC-TA 2014, Wuhan, China, October 16-19, 2014. Proceedings, L. Pan, G. P˘ãun, M. J. Perez-Jim ´ enez, ´ and T. Song, Eds. Communications in Computer and Information Science, vol. 472, pp. 308–312. Springer, 2014 (doi: 10.1007/978-3-662-45049-9 50).
• [44] J. Beyer, M. Hadwiger, and H. Pfister, “A survey of GPU-based large-scale volume visualization” in Proc. of the Eurograph. Conf. on Visual. Eurovis 2014, Swansea, UK, 2014, pp. 1–19 (doi: 10.2312/eurovisstar.20141175).
• [45] R. K. V. Maeda et al., “JADE: a heterogeneous multiprocessor system simulation platform using recorded and statistical application models”, in Proc. of the 1st Int. Worksh. on Adv. Interconn. Solutions and Technol. for Emerg. Comput. Syst. AISTECS’16, Prague, Czech Republic, 2016 (doi: 10.1145/2857058.2857066).
• [46] H. Casanova, A. Giersch, A. Legrand, M. Quinson, and F. Suter, “Versatile, scalable, and accurate simulation of distributed applications and platforms”, J. of Parallel and Distrib. Comput., vol. 74, no. 10, pp. 2899–2917, 2014 (doi: 10.1016/j.jpdc.2014.06.008).
• [47] R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. F. De Rose, and R. Buyya, “CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms”, Software: Pract. and Exper. (SPE), vol. 41, no. 1, pp. 23–50, 2011 (doi: 10.1002/spe.995).
• [48] Multi2sim workload manager home page [Online]. Available: www.multi2sim.org
• [49] Energy Efficiency [Online]. Available: https://ec.europa.eu/energy/ en/topics/energy-efficiency
• [50] Code of Conduct for Energy Efficiency in Data Centres [Online]. Available: https://ec.europa.eu/jrc/en/energy-efficiency/ code-conduct/datacentres
• [51] Carbon Abatement Handbook [Online]. Available: https://gesi.org/report/detail/carbon-abatement-handbook
• [52] M. Avgerinou, P. Bertoldi, and L. Castellazzi, “Trends in data centre energy consumption under the European code of conduct for data centre energy efficiency”, Energies, vol. 10, no. 10, pp. 1–18, 2017 (doi: 10.3390/en10101470).
• [53] B. Subramaniam, W. Saunders, T. Scogland, and W. Feng, “Trends in energy-efficient computing: A perspective from the Green500”, in 2013 Int. Green Comput. Conf. Proc., Arlington, VA, USA 2013, pp. 1–8 (doi: 10.1109/IGCC.2013.6604520).
• [54] D. L. Beaty, “Internal IT load profile variability”, ASHRAE J., vol. 55, no. 2, pp. 72–74, 2013.
• [55] J. S. Vetter, Contemporary High Performance Computing: From Petascale toward Exascale. Chapman and Hall/CRC Computational Science series, CRC Press, 2013 (ISBN: 9781466568341).
• [56] M. P. Karpowicz, P. Arabas, and E. Niewiadomska-Szynkiewicz, “Energy-aware multilevel control system for a network of Linux software routers: Design and implementation”, IEEE Systems J., vol. 12, no. 1, pp. 571–582, 2018 (doi: 10.1109/JSYST.2015.2489244).
• [57] ETSI ES 203 237 v1.1.1 (2014-03) standard [Online]. Available: www.etsi.org
• [58] L. A. Barroso and U. Holzle, “The case for energy-proportional computing”, Computer, vol. 40, no. 12, pp. 33–37, 2007 (doi: 10.1109/MC.2007.443).
• [59] H. Lim, A. Kansal, and J. Liu, “Power budgeting for virtualized data centers”, in Proc. of the 2011 USENIX Ann. Tech. Conf. USENIX ATC’11, Portland, OR, USA, 2011 [Online]. Available: https://www.microsoft.com/en-us/research/wp-content/uploads/ 2011/06/VPSUsenix11.pdf
• [60] L. Chiaraviglio, M. Mellia, and F. Neri, “Reducing power consumption in backbone networks”, in Proc. of the 2009 IEEE Int. Conf. on Commun. ICC’09, Piscataway, NJ, USA, 2009, pp. 2298–2303 (doi: 10.1109/ICC.2009.5199404).
• [61] M. Karpowicz, “Energy-efficient CPU frequency control for the Linux system”, Concurr. and Comput.: Pract. and Exper., vol. 28, no. 2, pp. 420–437, 2016 (doi: 10.1002/cpe.3476).
• [62] R. Bolla et al., “Large-scale validation and benchmarking of a network of power-conservative systems using ETSI’s green abstraction layer”, Trans. on Emerg. Telecommun. Technol., vol. 27, no. 3, pp. 451–468, 2016 (doi: 10.1002/ett.3006).
• [63] P. Arabas, “Energy aware data centers and networks: a survey”, J. of Telecommun. and Inform. Technol.”, no. 4, pp. 26–36, 2019 (doi: 10.26636/jtit.2018.129818).
• [64] A. Y. Zomaya and J. Ch. Lee, Eds., Energy-Efficient Distributed Computing Systems. Wiley, 2012 (ISBN: 978-0-470-90875-4).
• [65] J. C. McCullough, Y. Agarwal, J. Chandrashekar, S. Kuppuswamy, A. C. Snoeren, and R. K. Gupta, “Evaluating the effectiveness of model-based power characterization”, in in Proc. of the 2011 USENIX Ann. Tech. Conf. USENIX ATC’11, Portland, OR, USA, 2011 [Online]. Available: https://www.synergylabs.org/yuvraj/ docs/Agarwal USENIX11 Evaluating-Power-Models.pdf
• [66] J.-M. Pierson, Large-scale Distributed Systems and Energy Efficiency: A Holistic View. Wiley, 2015 (ISBN: 9781118864630).
• [67] S. Mittal, “A survey of architectural techniques for DRAM power management”, Int. J. of High Perform. Syst. Archit., vol. 4, no. 2, pp. 110–119, 2012 (doi: 10.1504/IJHPSA.2012.050990).
• [68] K. Chalmers et al., Eds., Communicating Process Architectures 2015 & 2016: WoTUG-37 & WoTUG-38. Concurrent Systems Engineering Series, vol. 69. IOS Press, 2018 (ISBN 978-1-61499-885-3).
• [69] R. Bolla, R. Bruschi, A. Carrega, and F. Davoli, “Green network technologies and the art of trading-off”, in Proc. 2011 IEEE Conf. on Comp. Commun. Worksh. INFOCOM WKSHPS 2011, Shanghai, China, 2011, pp. 301–306 (doi: 10.1109/INFCOMW.2011.5928827).
• [70] V. Pallipadi, S. Li, and A. Belay, “cpuidle: Do nothing, efficiently...”, in Proc. Linux Symposium, Ottawa, Ontario, Canada, 2007, vol. 2, pp. 119–125.
• [71] V. Pallipadi and A. Starikovskiy, “The ondemand governor”, in Proc. Linux Symposium, Ottawa, Ontario, Canada, 2006, vol. 2, pp. 215–230.
• [72] M. Karpowicz, E. Niewiadomska-Szynkiewicz, P. Arabas, and A. Sikora, “Energy and power efficiency in cloud”, in Resource Management for Big Data Platforms: Algorithms, Modelling, and High-Performance Computing Techniques, F. Pop, J. Kołodziej, and B. Di Martino, Eds. Springer, 2016, pp. 97–127 (doi: 10.1007/978-3-319-44881-7 6).
• [73] I. Manousakis, M. Marazakis, and A. Bilas, “FDIO: A feedback driven controller for minimizing energy in I/O-intensive applications”, in Proc. of the 5th USENIX Conf. on Hot Topics in Storage and File Syst. HotStorage’13, San Jose, CA, USA 2013 [Online]. Available: https://www.usenix.org/system/files/conference/ hotstorage13/hotstorage13-manousakis.pdf
• [74] M. Kondo, H. Sasaki, and H. Nakamura, “Improving fairness, throughput and energy-efficiency on a chip multiprocessor through DVFs”, SIGARCH Comp. Archit. News, vol. 35, no. 1, pp. 31–38, 2007 (doi: 10.1145/1241601.1241609).
• [75] Q. Zhang, L. Cheng, and R. Boutaba, “Cloud computing: stateof-the-art and research challenges”, J. of Internet Serv. and Appl., vol. 1, no. 1, pp. 7–18,
• [76] H. Jung and M. Pedram, “Supervised learning based power management for multicore processors”, IEEE Trans. on Comp.-Aided Design of Integr. Circ. and Syst., vol. 29, no. 9, pp. 1395–1408, 2010 (doi: 10.1109/TCAD.2010.2059270).
• [77] J. Howard et al., “A 48-core IA-32 processor in 45 nm CMOS using on-die message-passing and DVFs for performance and power scaling”, IEEE J. of Solid-State Circ., vol. 46, no. 1, pp. 173–183, 2011 (doi: 10.1109/JSSC.2010.2079450).
• [78] M. E. Salehi et al., “Dynamic voltage and frequency scheduling for embedded processors considering power/performance tradeoffs”, IEEE Trans. on Very Large Scale Integr. (VLSI) Syst., vol. 19, no. 10, pp. 1931–1935, 2011 (doi: 10.1109/TVLSI.2010.2057520).
• [79] M. Karpowicz, P. Arabas, and E. Niewiadomska-Szynkiewicz, “Design and implementation of energy-aware application-specific CPU frequency governors for the heterogeneous distributed computing systems”, Future Gener. Comp. Syst., vol. 78, pp. 302–315, 2018 (doi: 10.1016/j.future.2016.05.011).
• [80] L. Wang and S. U. Khan, “Review of performance metrics for green data centers: a taxonomy study”, J. of Supercomput., vol. 63, no. 3, pp. 639–656, 2003 (doi: 10.1007/s11227-011-0704-3). 2010 (doi: 10.1007/s13174-010-0007-6).
• [81] I. T. Cotes-Ruiz et al., “Dynamic voltage frequency scaling simulator for real workflows energy-aware management in green cloud computing”, PLoS ONE, vol. 12, no. 1, 2017 (doi: 10.1371/journal.pone.0169803).
• [82] Y. Chiang, Y. Ouyang, and C. Hsu, “An efficient green control algorithm in cloud computing for cost optimization”, IEEE Trans. on Cloud Comput., vol. 3, no. 2, pp. 145–155, 2015 (doi: 10.1109/TCC.2014.2350492).
• [83] E. Niewiadomska-Szynkiewicz, A. Sikora, P. Arabas, M. Kamola, M. Mincer, and J. Kołodziej, “Dynamic power management in energy-aware computer networks and data intensive systems”, Future Gener. Comp. Syst., vol. 37, pp. 284–296, 2014 (doi: 10.1016/j.future.2013.10.002).
• [84] J. Kołodziej, S. Khan, L. Wang, and A. Zomaya, “Energy efficient genetic-based schedulers in computational grids”, Concurr. and Comput.: Pract. and Exper., vol. 27, no. 4, pp. 809–829, 2015 (doi: 10.1002/cpe.2839).
• [85] M. Kamola and P. Arabas, “Shortest path green routing and the importance of traffic matrix knowledge”, in Proc. 2013 24th Tyrrhenian Int. Worksh. on Digit. Commun. – Green ICT TIWDC 2013, Genoa, Italy, 2013 (doi: 10.1109/TIWDC.2013.6664215).
• [86] K. Govindarajan, V. S. Kumar, and T. S. Somasundaram, “A distributed cloud resource management framework for highperformance computing (HPC) applications”, in Proc. 2016 8th Int. Conf. on Adv. Comput. ICoAC 2017, Chennai, India, 2017, pp. 1–6 (doi: 10.1109/ICoAC.2017.7951735).
• [87] E. Niewiadomska-Szynkiewicz and P. Arabas, “Resource management system for HPC computing”, in Automation 2018. Advances in Automation, Robotics and Measurement Techniques, R. Szewczyk, C. Zieliński, and M. Kaliczyńska, Eds. Advances in Intelligent Systems and Computing, vol. 743, pp. 52–61. Springer, 2018 (doi: 10.1007/978-3-319-77179-3 5).
• [88] L.-D. Radu, “Green cloud computing: A literature survey”, Symmetry, vol. 9, no. 12, pp. 1–20, 2017 (doi: 10.3390/sym9120295).
• [89] N. Akhter and M. Othman, “Energy aware resource allocation of cloud data center: review and open issues”, Cluster Comput., vol. 19, no. 3, pp. 1163–1182, 2016 (doi: 10.1007/s10586-016-0579-4).
• [90] T. Mastelic, A. Oleksiak, H. Claussen, I. Brandic, J.-M. Pierson, and A. Vasilakos, “Cloud computing: survey on energy efficiency”, ACM Comput. Surv., vol. 47, no. 2, Article no. 33, 2015 (doi: 10.1145/2656204).

Uwagi

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2019).