Wyniki wyszukiwania - BazTech

Ograniczanie wyników

Znaleziono wyników: 2

Liczba wyników na stronie

Wyniki wyszukiwania

Wyszukiwano:
w słowach kluczowych: user defined function

Sortuj według:

Ogranicz wyniki do:

Parallelizing user-defined functions in the ETL workflow using orchestration style sheets

Ali Syed Muhammad Fawad, Mey Johannes, Thiele Maik

International Journal of Applied Mathematics and Computer Science

2019

Vol. 29, no. 1

69--79

Today’s ETL tools provide capabilities to develop custom code as user-defined functions (UDFs) to extend the expressiveness of the standard ETL operators. However, while this allows us to easily add new functionalities, it also comes with the risk that the custom code is not intended to be optimized, e.g., by parallelism, and for this reason, it performs poorly for data-intensive ETL workflows. In this paper we present a novel framework, which allows the ETL developer to choose a design pattern in order to write parallelizable code and generates a configuration for the UDFs to be executed in a distributed environment. This enables ETL developers with minimum expertise in distributed and parallel computing to develop UDFs without taking care of parallelization configurations and complexities. We perform experiments on large-scale datasets based on TPC-DS and BigBench. The results show that our approach significantly reduces the effort of ETL developers and at the same time generates efficient parallel configurations to support complex and data-intensive ETL tasks.

Application of database technology to analysis of rock structure images

Mlynarczuk M., Ladniak M., Piorkowski A.

Physicochemical Problems of Mineral Processing

2014

Vol. 50, iss. 2

563--573

The aim of this paper is to present a possibility to use information technology in the form of databases for processing and analyzing large image sets based on methods of image analysis and mathematical morphology. Up to now the use of databases in the image analysis process has been reduced to storing large amounts of data in the form of images. However, all transformations and analyses of such sets are made on user’s computers. This requires a large data set (images) to be sent by network each time, and also it may possess the problems resulting from managing such large amounts of analyzed photographs on a computer. The proposed approach completely eliminates these problems by moving all transformations of image analysis to a database platform. For this purpose a set of routines realizing transformations of the image analysis and mathematical morphology was developed. The proposed approach allows the unification of the image processing and analysis area and advanced statistical analyses of obtained parameters describing geometrical sizes of objects on photographs. The proposed methodology was illustrated by practical realization of two measurement types for a simple structure of copper concentrate and more complicated, from the point of view of image analysis, structures such as dolomites from Redziny and Laskowa Gora and sandstones from Tumlin and Wisniowka.