Big data platform for smart grids power consumption anomaly detection

• Autorzy: Lipcàk Peter , Macak Martin , Rossi Bruno

Identyfikatory

DOI

10.15439/2019F210

• Konferencja: Federated Conference on Computer Science and Information Systems (14 ; 01-04.09.2019 ; Leipzig, Germany)
• Języki publikacji: EN

Abstrakty

EN

Big data processing in the Smart Grid context has many large-scale applications that require real-time data analysis (e.g., intrusion and data injection attacks detection, electric device health monitoring). In this paper, we present a big data platform for anomaly detection of power consumption data. The platform is based on an ingestion layer with data densification options, Apache Flink as part of the speed layer and HDFS/KairosDB as data storage layers. We showcase the application of the platform to a scenario of power consumption anomaly detection, benchmarking different alternative frameworks used at the speed layer level (Flink, Storm, Spark).

Słowa kluczowe

EN

big data; cloud computing; data analysis; power engineering computing; power system security; software; security of data; smart power grid

PL

big data; chmura obliczeniowa; analiza danych; informatyka energetyczna; bezpieczeństwo systemu elektroenergetycznego; bezpieczeństwo danych; inteligentna sieć energetyczna

• Wydawca: Polskie Towarzystwo Informatyczne
• Czasopismo: Annals of Computer Science and Information Systems
• Rocznik: 2019
• Tom: Vol. 18
• Strony: 771--780
• Opis fizyczny: Bibliogr. 41 poz., rys., tab.

Twórcy

autor

Lipcàk Peter

• Faculty of Informatics, Masaryk University Brno, Czech Republic

autor

Macak Martin

• Institute of Computer Science, Masaryk University
• Faculty of Informatics, Masaryk University Brno, Czech Republic

autor

Rossi Bruno

• Institute of Computer Science, Masaryk University
• Faculty of Informatics, Masaryk University Brno, Czech Republic

Bibliografia

• 1. Y. Demchenko, P. Grosso, C. De Laat, and P. Membrey, “Addressing big data issues in scientific data infrastructure,” in 2013 International Conference on Collaboration Technologies and Systems (CTS). IEEE, 2013. http://dx.doi.org/10.1109/CTS.2013.6567203 pp. 48–55.
• 2. A. Radenski, T. Gurov, K. Kaloyanova, N. Kirov, M. Nisheva, P. Stanchev, and E. Stoimenova, “Big data techniques, systems, applications, and platforms: Case studies from academia,” in 2016 Federated Conference on Computer Science and Information Systems (FedCSIS). IEEE, 2016. http://dx.doi.org/10.15439/2016F91 pp. 883–888.
• 3. Y. Zhang, T. Huang, and E. F. Bompard, “Big data analytics in smart grids: A review,” Energy Informatics, vol. 1, no. 1, p. 8, 2018. http://dx.doi.org/10.1186/s42162-018-0007-5
• 4. K. Zhou, C. Fu, and S. Yang, “Big data driven smart energy management: From big data to big insights,” Renewable and Sustainable Energy Reviews, vol. 56, pp. 215–225, 2016. http://dx.doi.org/10.1016/j.rser.2015.11.050
• 5. B. Rossi and S. Chren, “Smart grids data analysis: A systematic mapping study,” arXiv preprint https://arxiv.org/abs/1808.00156, 2018. [Online]. Available: https://arxiv.org/abs/1808.00156
• 6. M. Ge, H. Bangui, and B. Buhnova, “Big data for internet of things: a survey,” Future Generation Computer Systems, vol. 87, pp. 601–614, 2018. http://dx.doi.org/10.1016/j.future.2018.04.053
• 7. X. Liu and P. S. Nielsen, “Regression-based online anomaly detection for smart grid data,” arXiv preprint https://arxiv.org/abs/1606.05781, 2016.
• 8. B. Rossi, S. Chren, B. Buhnova, and T. Pitner, “Anomaly detection in smart grid data: An experience report,” in 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2016. http://dx.doi.org/10.1109/SMC.2016.7844583 pp. 2313–2318.
• 9. Z.-H. Yu and W.-L. Chin, “Blind False Data Injection Attack Using PCA Approximation Method in Smart Grid,” IEEE Transactions on Smart Grid, vol. 6, no. 3, pp. 1219–1226, 2015. http://dx.doi.org/10.1109/TSG.2014.2382714
• 10. J. Lee, Y.-c. Kim, and G.-L. Park, “An Analysis of Smart Meter Readings Using Artificial Neural Networks,” in Convergence and Hybrid Information Technology. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012, vol. 7425, pp. 182–188.
• 11. F. McLoughlin, A. Duffy, and M. Conlon, “A clustering approach to domestic electricity load profile characterisation using smart metering data,” Applied energy, vol. 141, pp. 190–199, 2015. http://dx.doi.org/10.1016/j.apenergy.2014.12.039
• 12. P. Pääkkönen and D. Pakkala, “Reference architecture and classification of technologies, products and services for big data systems,” Big data research, vol. 2, no. 4, pp. 166–186, 2015. http://dx.doi.org/10.1016/j.bdr.2015.01.001
• 13. N. Marz and J. Warren, Big Data: Principles and best practices of scalable real-time data systems. New York; Manning Publications Co., 2015.
• 14. J. Lin, “The lambda and the kappa,” IEEE Internet Computing, vol. 21, no. 5, pp. 60–66, 2017. http://dx.doi.org/10.1109/MIC.2017.3481351
• 15. S. Shahrivari, “Beyond batch processing: towards real-time and stream- ing big data,” Computers, vol. 3, no. 4, pp. 117–129, 2014. http://dx.doi.org/10.3390/computers3040117
• 16. J. Kreps, “Questioning the lambda architecture,” Online article, July, 2014. [Online]. Available: https://www.oreilly.com/ideas/questioning-the-lambda-architecture
• 17. M. Kiran, P. Murphy, I. Monga, J. Dugan, and S. S. Baveja, “Lambda architecture for cost-effective batch and speed big data processing,” in 2015 IEEE International Conference on Big Data (Big Data). IEEE, 2015. http://dx.doi.org/10.1109/BigData.2015.7364082 pp. 2785–2792.
• 18. J. van Rooij, V. Gulisano, and M. Papatriantafilou, “Locovolt: Distributed detection of broken meters in smart grids through stream processing,” in Proceedings of the 12th ACM International Conference on Distributed and Event-based Systems. ACM, 2018. http://dx.doi.org/10.1145/3210284.3210298 pp. 171–182.
• 19. H. Sequeira, P. Carreira, T. Goldschmidt, and P. Vorst, “Energy cloud: Real-time cloud-native energy management system to monitor and analyze energy consumption in multiple industrial sites,” in 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing. IEEE, 2014. http://dx.doi.org/10.1109/UCC.2014.79 pp. 529–534.
• 20. M. Mayilvaganan and M. Sabitha, “A cloud-based architecture for big-data analytics in smart grid: A proposal,” in 2013 IEEE International Conference on Computational Intelligence and Computing Research, Dec 2013. http://dx.doi.org/10.1109/ICCIC.2013.6724168 pp. 1–4.
• 21. A. A. Munshi and Y. A. I. Mohamed, “Data lake lambda architecture for smart grids big data analytics,” IEEE Access, vol. 6, pp. 40 463–40 471, 2018. http://dx.doi.org/10.1109/ACCESS.2018.2858256
• 22. X. Liu and P. Nielsen, “Streamlining smart meter data analytics,” in Proceedings of the 10th Conference on Sustainable Development of Energy, Water and Environment Systems. International Centre for Sustainable Development of Energy, Water and Environment Systems, 2015.
• 23. N. Balac, T. Sipes, N. Wolter, K. Nunes, B. Sinkovits, and H. Karimabadi, “Large scale predictive analytics for real-time energy management,” in 2013 IEEE International Conference on Big Data, Oct 2013. http://dx.doi.org/10.1109/BigData.2013.6691635 pp. 657–664.
• 24. A. R. Al-Ali, I. A. Zualkernan, M. Rashid, R. Gupta, and M. Alikarar, “A smart home energy management system using iot and big data analytics approach,” IEEE Transactions on Consumer Electronics, vol. 63, no. 4, pp. 426–434, November 2017. http://dx.doi.org/10.1109/TCE.2017.015014
• 25. H. Daki, A. El Hannani, A. Aqqal, A. Haidine, and A. Dahbi, “Big data management in smart grid: concepts, requirements and implementation,” Journal of Big Data, vol. 4, 12 2017. http://dx.doi.org/10.1186/s40537-017-0070-y
• 26. C. Yang, W. Chen, K. Huang, J. Liu, W. Hsu, and C. Hsu, “Implementation of smart power management and service system on cloud computing,” in 2012 9th International Conference on Ubiquitous Intelligence and Computing and 9th International Conference on Autonomic and Trusted Computing, Sep. 2012. http://dx.doi.org/10.1109/UIC-ATC.2012.160 pp. 924–929.
• 27. M. Ali, Z. Y. Dong, X. Li, and P. Zhang, “Rsa-grid: a grid computing based framework for power system reliability and security analysis,” in 2006 IEEE Power Engineering Society General Meeting, June 2006. http://dx.doi.org/10.1109/PES.2006.1709374. ISSN 1932-5517
• 28. T. Rajeev and S. Ashok, “A cloud computing approach for power management of microgrids,” in ISGT2011-India, Dec 2011. http://dx.doi.org/10.1109/ISET-India.2011.6145354 pp. 49–52.
• 29. S. Chren, B. Rossi, and T. Pitner, “Smart grids deployments within eu projects: The role of smart meters,” in 2016 Smart cities symposium Prague (SCSP). IEEE, 2016. http://dx.doi.org/10.1109/SCSP.2016.7501033 pp. 1–5.
• 30. M. Schvarcbacher, K. Hrabovská, B. Rossi, and T. Pitner, “Smart grid testing management platform (sgtmp),” Applied Sciences, vol. 8, no. 11, p. 2278, 2018. http://dx.doi.org/10.3390/app8112278
• 31. K. Hrabovská, N. Šimková, B. Rossi, and T. Pitner, “Smart grids and software testing process models,” in 2019 Smart cities symposium Prague (SCSP). IEEE, 2019, pp. 1–5.
• 32. J. Peppanen, X. Zhang, S. Grijalva, and M. J. Reno, “Handling bad or missing smart meter data through advanced data imputation,” in IEEE Innovative Smart Grid Technologies Conference. IEEE, 2016. http://dx.doi.org/10.1109/ISGT.2016.7781213 pp. 1–5.
• 33. X. Liu, N. Iftikhar, H. Huo, R. Li, and P. S. Nielsen, “Two approaches for synthesizing scalable residential energy consumption data,” Future Generation Computer Systems, vol. 95, pp. 586–600, 2019. http://dx.doi.org/10.1016/j.future.2019.01.045
• 34. M. Zaharia, R. S. Xin, P. Wendell, T. Das, M. Armbrust, A. Dave, X. Meng, J. Rosen, S. Venkataraman, M. J. Franklin et al., “Apache spark: a unified engine for big data processing,” Communications of the ACM, vol. 59, no. 11, pp. 56–65, 2016. http://dx.doi.org/10.1145/2934664
• 35. P. Carbone, A. Katsifodimos, S. Ewen, V. Markl, S. Haridi, and K. Tzoumas, “Apache flink: Stream and batch processing in a single engine,” Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, vol. 36, no. 4, 2015.
• 36. S. T. Allen, M. Jankowski, and P. Pathirana, Storm Applied: Strategies for real-time event processing. Manning Publications Co., 2015.
• 37. J. Karimov, T. Rabl, A. Katsifodimos, R. Samarev, H. Heiskanen, and V. Markl, “Benchmarking distributed stream processing engines,” arXiv preprint https://arxiv.org/abs/1802.08496, 2018.
• 38. Y. Wang, “Stream processing systems benchmark: Streambench,” G2 Pro gradu, diplomityö, Aalto University, Finland, 2016-06-13. [Online]. Available: http://urn.fi/URN:NBN:fi:aalto-201606172599
• 39. M. A. Lopez, A. G. P. Lobato, and O. C. M. Duarte, “A performance comparison of open-source stream processing platforms,” in 2016 IEEE Global Communications Conference (GLOBECOM). IEEE, 2016. http://dx.doi.org/10.1109/GLOCOM.2016.7841533 pp. 1–6.
• 40. S. Chintapalli, D. Dagit, B. Evans, R. Farivar, T. Graves, M. Holderbaugh, Z. Liu, K. Nusbaum, K. Patil, B. J. Peng et al., “Benchmarking streaming computation engines: Storm, flink and spark streaming,” in 2016 IEEE international parallel and distributed processing symposium workshops (IPDPSW). IEEE, 2016. http://dx.doi.org/10.1109/IPDPSW.2016.138 pp. 1789–1792.
• 41. Smart* data set for sustainability. [Online]. Available: http://traces.cs.umass.edu/index.php/Smart/Smart

Uwagi

1. Track 5: Software and System Engineering

2. Technical Session: Joint 39th IEEE Software Engineering Workshop (SEW-39) and 6th International Workshop on Cyber-Physical Systems (IWCPS-6)

3. Opracowanie rekordu ze środków MNiSW, umowa Nr 461252 w ramach programu "Społeczna odpowiedzialność nauki" - moduł: Popularyzacja nauki i promocja sportu (2020).