Processing of satellite data in the cloud

• Autorzy: Proficz J. , Drypczewski K.

Identyfikatory

DOI

10.17466/tq2017/21.4/y

• Języki publikacji: EN

Abstrakty

EN

The dynamic development of digital technologies, especially those dedicated to devices generating large data streams, such as all kinds of measurement equipment (temperature and humidity sensors, cameras, radio-telescopes and satellites – Internet of Things) enables more in-depth analysis of the surrounding reality, including better understanding of various natural phenomenon, starting from atomic level reactions, through macroscopic processes (e.g. meteorology) to observation of the Earth and the outer space. On the other hand such a large quantitative improvement requires a great number of processing and storage resources, resulting in the recent rapid development of Big Data technologies. Since 2015, the European Space Agency (ESA) has been providing a great amount of data gathered by exploratory equipment: a collection of Sentinel satellites – which perform Earth observation using various measurement techniques. For example Sentinel-2 provides a stream of digital photos, including images of the Baltic Sea and the whole territory of Poland. This data is used in an experimental installation of a Big Data processing system based on the open source software at the Academic Computer Center in Gdansk. The center has one of the most powerful supercomputers in Poland – the Tryton computing cluster, consisting of 1600 nodes interconnected by a fast Infiniband network (56 Gbps) and over 6 PB of storage. Some of these nodes are used as a computational cloud supervised by an OpenStack platform, where the Sentinel-2 data is processed. A subsystem of the automatic, perpetual data download to object storage (based on Swift) is deployed, the required software libraries for the image processing are configured and the Apache Spark cluster has been set up. The above system enables gathering and analysis of the recorded satellite images and the associated metadata, benefiting from the parallel computation mechanisms. This paper describes the above solution including its technical aspects.

Słowa kluczowe

EN

Apache Spark; satellite data; Sentinel-2; ESA; big data; cloud; OpenStack

PL

Apache Spark; dane satelitarne; Sentinel-2; ESA; zbiór danych; chmura; OpenStack

• Wydawca: Politechnika Gdańska
• Czasopismo: TASK Quarterly : scientific bulletin of Academic Computer Centre in Gdansk
• Rocznik: 2017
• Tom: Vol. 21, No 4
• Strony: 365--377
• Opis fizyczny: Bibliogr. 31 poz., rys.

Twórcy

autor

Proficz J.

• Gdansk University of Technology, Academic Computer Centre TASK, Narutowicza 11/12, 80-233 Gdansk, Poland

autor

Drypczewski K.

• Gdansk University of Technology, Academic Computer Centre TASK, Narutowicza 11/12, 80-233 Gdansk, Poland

Bibliografia

• [1] Demchenko Y, De Laat C and Membrey P 2014 CTS, IEEE 104
• [2] Apache Spark – Lightning-fast cluster computing [Online] available at: https://spark.apache.org/ [Accessed: 25-July-2017]
• [3] Apache Hadoop – website [Online] available at: https://hadoop.apache.org/[Accessed: 23-June-2017]
• [4] Thensorflow Homepage [Online] available at: https://www.tensorflow.org/[Accessed: 23-June-2017]
• [5] Torch – a scientific computing platform for Lua JIT [Online] available at: http://torch.ch/ [Accessed: 23-June-2017]
• [6] Theano – a library to define, optimize, and evaluate mathematical expressions [Online] available at: http://deeplearning.net/software/theano/ [Accessed: 26-June-2017]
• [7] Caffe – Deep learning framework [Online] available at: http://caffe.berkeleyvision.org/ [Accessed: 26-June-2017]
• [8] Copernicus – observing the Earth, ESA programe [Online] available at: http://www.esa. int/Our Activities/Observing the Earth/Copernicus [Accessed: 26-June-2017]
• [9] Sentinel-2 Homepage [Online] available at: http://www.esa.int/Our Activities/ Observing the Earth/Copernicus/Sentinel-2 [Accessed: 23-June-2017]
• [10] Vega – Arianspace [Online] available at: http://www.arianespace.com/vehicle/vega/ [Accessed: 06-July-2017]
• [11] Rockot launch vehicle [Online] available at: http://www.russianspaceweb.com/rockot.html [Accessed: 06-July-2017]
• [12] Copernicus Open Access Hub [Online] available at: https://scihub.copernicus.eu/ [Accessed: 27-July-2017]
• [13] Sentinel-2 MSI, Data formats [Online] available at: https://sentinel.esa.int/web/sentinel/user-guides/sentinel-2-msi/data-formats [Accessed: 23-July-2017]
• [14] Tryton Supercomputer, Academic Computer Centre TASK [Online] available at: https://task.gda.pl/kdm/sprzet/tryton/
• [15] OpenStack, Open Source Cloud Computing Software [Online] available at: https://www.openstack.org/ [Accessed: 27-June-2017]
• [16] Ceph Homepage [Online] available at: http://ceph.com/ [Accessed: 27-June-2017]
• [17] Swift – OpenStack [Online] available at: https://wiki.openstack.org/wiki/Swift [Accessed: 27-June-2017]
• [18] Apache Hadoop YARN [Online] available at: https://hadoop.apache.org/docs/r2.7.3/hadoop-yarn/hadoop-yarn-site/YARN.html [Accessed: 27-June-2017]
• [19] Apache Mesos [Online] available at: http://mesos.apache.org/ [Accessed: 28-June-2017]
• [20] Kubernetes – Production-Grade Container Orchestration [Online] available at: https://kubernetes.io/ [Accessed: 28-June-2017]
• [21] HDFS Architecture Guide [Online] available at: https://hadoop.apache.org/docs/r1. 2.1/hdfs design.html [Accessed: 28-June-2017]
• [22] Apache HBase Home [Online] available at: https://hbase.apache.org/ [Accessed: 28-June-2017]
• [23] Apache Cassandra Homepage [Online] available at: http://cassandra.apache.org/ [Accessed: 28-June-2017]
• [24] SNAP – Sentinel Application Platform [Online] available at: http://step.esa.int/main/toolboxes/snap/ [Accessed: 28-June-2017]
• [25] JPEG 2000 Homepage [Online] available at: https://jpeg.org/jpeg2000/ [Accessed: 28-June-2017]
• [26] Java Advanced Imaging API Home Page [Online] available at: http://www.oracle.com/technetwork/java/iio-141084.html [Accessed: 28-June-2017]
• [27] The ImageIO-Ext Project Home [Online] available at: https://github.com/geosolutions-it/imageio-ext/ [Accessed: 29-June-2017]
• [28] Open JPEG, an open-source JPEG 2000 codec [Online] available at: http://www.openjpeg.org/ [Accessed: 29-June-2017]
• [29] Odersky M 2014 Scala by Example, Programing Methods Laboratory [Online] available at: http://www.scala-lang.org/docu/files/ScalaByExample.pdf, EPFL [Accessed: 26-July-2017]
• [30] Sentinel-2 User Handbook [Online] available at: https://sentinel.esa.int/documents/247904/685211/Sentinel-2 User Handbook, ESA [Accessed: 03-July-2017]
• [31] H2O Homepage [Online] available at: https://www.h2o.ai/ [Accessed: 03-July-2017]

Uwagi

PL

Opracowanie rekordu w ramach umowy 509/P-DUN/2018 ze środków MNiSW przeznaczonych na działalność upowszechniającą naukę (2018).