A Python library for the Jupyteo IDE Earth observation processing tool enabling interoperability with the QGIS System for use in data science

• Autorzy: Bednarczyk Michał

Identyfikatory

DOI

10.7494/geom.2022.16.1.117

• Języki publikacji: EN

Abstrakty

EN

This paper describes JupyQgis – a new Python library for Jupyteo IDE enabling interoperability with the QGIS system. Jupyteo is an online integrated development environment for earth observation data processing and is available on a cloud platform. It is targeted at remote sensing experts, scientists and users who can develop the Jupyter notebook by reusing embedded open-source tools, WPS interfaces and existing notebooks. In recent years, there has been an increasing popularity of data science methods that have become the focus of many organizations. Many scientific disciplines are facing a significant transformation due to data-driven solutions. This is especially true of geodesy, environmental sciences, and Earth sciences, where large data sets, such as Earth observation satellite data (EO data) and GIS data are used. The previous experience in using Jupyteo, both among the users of this platform and its creators, indicates the need to supplement its functionality with GIS analytical tools. This study analyzed the most efficient way to combine the functionality of the QGIS system with the functionality of the Jupyteo platform in one tool. It was found that the most suitable solution is to create a custom library providing an API for collaboration between both environments. The resulting library makes the work much easier and simplifies the source code of the created Python scripts. The functionality of the developed solution was illustrated with a test use case.

Słowa kluczowe

EN

Earth observation data processing; IDE; IPython; Jupyter notebook; web processing service; GIS; data science; machine learning; API

• Wydawca: AGH University of Science and Technology Press
• Czasopismo: Geomatics and Environmental Engineering
• Rocznik: 2022
• Tom: Vol. 16, no. 1
• Strony: 117--144
• Opis fizyczny: Bibliogr. 40 poz., fot., rys., tab.

Twórcy

autor

Bednarczyk Michał

• University of Warmia and Mazury in Olsztyn, Faculty of Geoengineering, Institute of Geodesy andCivil Engineering, Department of Geodesy, Olsztyn, Poland

• https://orcid.org/0000-0002-0450-5327

Bibliografia

• 1. Aalst W.M.P., Bichler M., Heinzl A.: Responsible Data Science. Business & Information Systems Engineering, vol. 59, 2017, pp. 311–313. https://doi.org/10.1007/s12599-017-0487-z.
• 2 Janowski A., Szulwic J., Tysiąc P.: Spatial Modelling in Environmental Analysis and Civil Engineering. Applied Sciences, vol. 11(9), 2021, 3945. https://doi.org/10.3390/app11093945.
• 3. Bacao F., Santos M.Y., Behnisch M.: Spatial Data Science. ISPRS International Journal of Geo-Information, vol. 9, 2020, 428. https://doi.org/10.3390/ijgi9070428.
• 4. Gibert K., Horsburgh J.S., Athanasiadis I.N., Holmes G.: Environmental Data Science. Environmental Modelling & Software, vol. 106, 2018, pp. 4–12. https://doi.org/10.1016/j.envsoft.2018.04.005.
• 5. Cheng Y., Zhou K., Wang J., Yan J.: Big Earth Observation Data Integration in Remote Sensing Based on a Distributed Spatial Framework. Remote Sensing, vol. 12, 2020, 972. https://doi.org/10.3390/rs12060972.
• 6. Mattmann C.: A vision for data science. Nature, vol. 493, 2013, pp. 473–475. https://doi.org/10.1038/493473a.
• 7. Janowski A., Bobkowska K., Szulwic J.: 3d Modelling of Cylindrical-Shaped Objects from Lidar Data – an Assessment Based on Theoretical Modelling and Experimental Data. Metrology and Measurement Systems, vol. 25(1), 2018, pp. 47–56. https://doi.org/10.24425/118156.
• 8. Janowski A., Szulwic J., Ziółkowski P.: Combined Method of Surface Flow Measurement Using Terrestrial Laser Scanning and Synchronous Photogrammetry. [in:] 2017 Baltic Geodetic Congress (Geomatics): BGC Geomatics 2017: proceedings: 22–25 June 2017, Gdansk University of Technology, Poland, IEEE, Piscataway 2017, pp. 110–115. https://doi.org/10.1109/BGC.Geomatics.2017.54.
• 9. Ossowski R., Przyborski M., Tysiac P.: Stability Assessment of Coastal Cliffs Incorporating Laser Scanning Technology and a Numerical Analysis. Remote Sensing, vol. 11(16), 2019, 1951. https://doi.org/10.3390/rs11161951.
• 10. Tysiac P.: Bringing Bathymetry LiDAR to Coastal Zone Assessment: A Case Study in the Southern Baltic. Remote Sensing, vol. 12(22), 2020, 3740. https://doi.org/10.3390/rs12223740.
• 11. Guo H., Nativi S., Liang D., Craglia M., Wang L., Schade S., Corban C., He G., Pesaresi M., Li J., Shirazi Z., Liu J., Annoni A.: Big Earth Data science: an information framework for a sustainable planet. International Journal of Digital Earth, vol. 13(7), 2020, pp. 743–767. https://doi.org/10.1080/17538947.2020.1743785.
• 12. Dumitru C.O., Schwarz G., Castel F., Lorenzo J., Datcu M.: Artificial Intelligence Data Science Methodology for Earth Observation. [in:] Soofastaei A. (ed.), Advanced Analytics and Artificial Intelligence Applications, IntechOpen, London 2019. https://doi.org/10.5772/intechopen.86886.
• 13. Artiemjew P., Chojka A., Rapiński J.: Deep Learning for RFI Artifact Recognition in Sentinel-1 Data. Remote Sensing, vol. 13(1), 2021, 7. https://doi.org/10.3390/rs13010007.
• 14. Janowski A., Renigier-Biłozor M., Walacik M., Chmielewska A.: Remote Measurement of Building Usable Floor Area – Algorithms Fusion. Land Use Policy, vol. 100, 2021, 104938. https://doi.org/10.1016/j.landusepol.2020.104938.
• 15. Soille P., Burger A., De Marchi D., Kempeneers P., Rodriguez D., Syrris V., Vasilev V.: A versatile data-intensive computing platform for information retrieval from big geospatial data. Future Generation Computer Systems, vol. 81, 2018, pp. 30–40. https://doi.org/10.1016/j.future.2017.11.007.
• 16. Stromann O., Nascetti A., Yousif O., Ban Y.: Dimensionality Reduction and Feature Selection for Object-Based Land Cover Classification based on Sentinel-1 and Sentinel-2 Time Series Using Google Earth Engine. Remote Sensing, vol. 12, 2020, 76. https://doi.org/10.3390/rs12010076.
• 17. Müller M., Bernard L., Brauner J.: Moving code in spatial data infrastructures–web service based deployment of geoprocessing algorithms. Transactions in GIS, vol. 14, 2010, pp. 101–118. https://doi.org/10.1111/j.1467-9671.2010.01205.x.
• 18. Gomes V.C.F., Queiroz G.R., Ferreira K.R.: An Overview of Platforms for Big Earth Observation Data Management and Analysis. Remote Sensing, vol. 12(8), 2020, 1253. https://doi.org/10.3390/rs12081253.
• 19. Kadiyala A., Kumar A.: Applications of Python to evaluate environmental data science problems. Environmental Progress & Sustainable Energy, vol. 36(6), 2017, pp. 1580–1586. https://doi.org/10.1002/ep.12786.
• 20. Raschka S., Patterson J., Nolet C.: Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence. Information, vol. 11(4), 2020, 193. https://doi.org/10.3390/info11040193.
• 21. Hao J., Ho T.K.: Machine Learning Made Easy: A Review of Scikit-learn Package in Python Programming Language. Journal of Educational and Behavioral Statistics, vol. 44(3), 2020, pp. 348–361. https://doi.org/10.3102/1076998619832248.
• 22. Bisong E.: Google Colaboratory. [in:] Building Machine Learning and Deep Learning Models on Google Cloud Platform, Apress, Berkeley 2019, pp. 59–64. https://doi.org/10.1007/978-1-4842-4470-8_7.
• 23. Pimentel J.F., Murta L., Braganholo V., Freire J.: A Large-Scale Study about Quality and Reproducibility of Jupyter Notebooks. [in:] MSR 2019: 2019 IEEE/ACM 16th International Conference on Mining Software Repositories: proceedings: 26–27 May 2019, Montreal, Canada, IEEE Computer Society, Conference Publishing Services (CPS), Los Alamitos 2019, pp. 507–517. https://doi.org/10.1109/MSR.2019.00077.
• 24. Cook J.: Docker for Data Science: Building Scalable and Extensible Data Infrastructure around the Jupyter Notebook Server, Apress, Santa Monica 2017. https://doi.org/10.1007/978-1-4842-3012-1.
• 25. Rapiński J., Bednarczyk M., Zinkiewicz D.: JupyTEP IDE as an Online Tool for Earth Observation Data Processing. Remote Sensing, vol. 11, 2019, 1973. https://doi.org/10.3390/rs11171973.
• 26. Fernández L., Hagenrud H., Zupanc B., Laface E., Korhonen T., Andersson R.: Jupyterhub at the ESS. An Interactive Python Computing Environment for Scientists and Engineers. [in:] Proceedings of the 7th International Particle Accelerator Conference (IPAC2016), Busan, Korea, 8–13 May 2016, pp. 2778–2780. https://doi.org/10.18429/JACOW-IPAC2016-WEPOR049.
• 27. Zuhlke M., Fomferra N., Brockmann C., Peters M., Veci L., Malik J., Regner P.: SNAP (Sentinel Application Platform) and the ESA Sentinel 3 Toolbox. [in:] Ouwehand L. (ed.), Sentinel-3 for Science Workshop, ESA Special Publication, vol. 734, Venice 2015.
• 28. Grizonnet M., Michel J., Poughon V., Inglada J., Savinaud M., Cresson R.: Orfeo ToolBox: open source processing of remote sensing images. Open Geospatial Data, Software and Standards, vol. 2, 2017, 15. https://doi.org/10.1186/s40965-017-0031-6.
• 29. Warmerdam F.: The Geospatial Data Abstraction Library. [in:] Hall G.B., Leahy M.G. (eds.), Open Source Approaches in Spatial Data Handling, Advances in Geographic Information Science, Springer-Verlag Berlin Heidelberg 2008, pp. 87–104. https://doi.org/10.1007/978-3-540-74831-1_5.
• 30. Unofficial Jupyter Notebook Extensions: https://jupyter-contrib-nbextensions. readthedocs.io [access: 30.05.2021].
• 31. 3liz/qgis-nbextension: https://github.com/3liz/qgis-nbextension [access: 30.05.2021].
• 32. Geospatial Python Tutorial – Install Jupyter Notebook in QGIS3: https://lerryws.xyz/posts/Install-Jupyter-Notebook-in-QGIS3 [access: 30.05.2021].
• 33. PyQGIS Developer Cookbook: https://docs.qgis.org/3.22/en/docs/pyqgis_developer_cookbook/index.html [access: 30.05.2021].
• 34. Pandas website: https://pandas.pydata.org [access: 30.05.2021].
• 35. Statistics Poland: https://stat.gov.pl [access: 30.05.2021].
• 36. Renigier-Biłozor M., Janowski A., Walacik M.: Geoscience Methods in Real Estate Market Analyses Subjectivity Decrease. Geosciences, vol. 9(3), 2019, 130. https://doi.org/10.3390/geosciences9030130.
• 37. Zydroń A., Walkowiak R.: Analiza atrybutów wpływających na wartość nieruchomości niezabudowanych przeznaczonych na cele budowlane w gminie Mosina [Analysis of Factors Affecting Value of Undeveloped Plots Allocated for Buildings Development in Mosina Municipality]. Rocznik Ochrona Środowiska, t. 15, cz. 3, 2013, pp. 2911–2924.
• 38. Kucharska-Stasiak E.: Odwzorowanie cech nieruchomości w cenach i skutki dla procesu wyceny [Reflection of Real Estate Attributes in Prices and Consequences for Valuation Process]. Studia i Materiały Towarzystwa Naukowego Nieruchomości, t. 18, nr 3, 2010, pp. 7–16.
• 39. Muratorplus: Ceny mieszkań w Polsce – prognozy na 2021. https://www.muratorplus.pl/inwestycje/inwestycje-mieszkaniowe/ceny-mieszkan-w-2018-r-nowe-mieszkania-mocno-podrozaly-gdzie-ceny-mieszkan-wzrosly-najbardziej-aa-BJAZ-hHVK-d4wc.html [access: 30.05.2021].
• 40. Pracuj.pl: Ceny mieszkań w 2020 – ile średnich pensji potrzeba, aby kupić włas- ne lokum? 6.11.2020, https://zarobki.pracuj.pl/raporty-i-trendy-placowe/ceny-mieszkan-2020-ile-srednich-pensji-potrzeba-aby-kupic-wlasne-lokum/ [access: 30.05.2021].

Uwagi

PL

Opracowanie rekordu ze środków MEiN, umowa nr SONP/SP/546092/2022 w ramach programu „Społeczna odpowiedzialność nauki” - moduł: Popularyzacja nauki i promocja sportu (2022-2023)