Choose timezone
Your profile timezone:
European HTCondor Workshop 2019
→
Europe/Rome
EC-JRC Ispra
EC-JRC Ispra
European Commission – Joint Research Centre
Via Enrico Fermi, 2749
I - 21027 Ispra (VA)
Italy
N45° 48' 36.09'' E008° 37' 16.72''
N45° 48.601 E008° 37.278
45.80998, 8.62135
https://www.openstreetmap.org/#map=17/45.80998/8.62135
, ,
Description
|
The European HTCondor Workshop 2019 was held in Italy, hosted by the European Commission's Joint Research Centre in Ispra, Lombardy, Italy, very close to the shores of Lago Maggiore. The Joint Research Centre (JRC) is the European Commission's science and knowledge service which employs scientists to carry out research in order to provide independent scientific advice and support to EU policy. The JRC is located in 6 different sites (Brussels, Geel, Ispra, Karlsruhe, Petten, Seville), and employs around 3000 staff coming from throughout the EU. |
 |
|
The workshop was the fifth edition in Europe after the successful events at CERN in December 2014, ALBA in February 2016, DESY in June 2017 and RAL in September 2018. The workshops are opportunities for novice and experienced users of HTCondor to learn, get help and have exchanges between them and with the HTCondor developers and experts. They are primarily addressed at users from EMEA, but open to everyone. The workshops consist of presentations, tutorials and "office hours" for consultancy, covering the HTCondor CE (Compute Element) covered prominently as well. The workshops address participants from academia and research as well as from commercial entities alike. |
|
-
-
Antonio
Convener: Catalin Condurache (EGI)-
The spatial distribution of built-up areas and their expansion represent one of the most important forms of land use/ land cover changes confronting climate, environmental and socio-economic systems at a global scale. Characterizing the status and dynamics of built-up areas over large areas is technically feasible thanks to the availability of a panoply of earth observation data with different spatial, spectral and temporal characteristics. In the last 10 years, at the Joint Research Centre of the European Commission, the Global Human Settlement Layer project has been exploiting different sources of satellite imagery to monitor changes in the European and Global built-up landscapes to better inform policies and decision making (Corbane et al. 2017) (Florczyk et al. 2016).
To meet the demands of large-scale mapping of human settlements from space, not only mass storage infrastructures are needed but also novel data analytics combined with high-performance computing platforms have to be designed. The JRC EO Data and Processing Platform (JEODPP) developed in the framework of JRC Big Data Analytics (BDA) project provides petabyte scale storage coupled to high throughput computing capacities enabling and facilitating the extraction of built-up areas from large volumes of satellite data both at the European and Global scales (Soille et al. 2018). In this work, we present the JEO-batch feature of the JEODPP; a low-level batch processing orchestrated by a dedicated workload manager and its utility for the execution of two main automated workflows for extracting built-up areas over large zones. The first workflow is implemented on a pan-European coverage of Very High Resolution satellite data from the Copernicus contributing missions acquired in 2015; the second workflow exploits a global Sentinel-2 pixel-based composite from the Copernicus constellation of satellites acquired mainly in 2018.
Although the two workflows build on the same classifier, the number of images/tiles to be processed and their projections, the characteristics of remote sensing sensors, in particular their spatial resolutions and the derived outputs required different configurations of the workload automations. Taking advantage of the Docker universe of the JEO-batch that relies on the HTCondor architecture, the workflows, originally coded in Matlab, were compiled and successfully run on the JEODPP. The following table summarizes the main characteristics of the massive batch processing that allowed extracting built-up areas at the European and global scales:
References:
- Corbane, Christina, Martino Pesaresi, Panagiotis Politis, Vasileios Syrris, Aneta J. Florczyk, Pierre Soille, Luca Maffenini, et al. 2017. “Big Earth Data Analytics on Sentinel-1 and Landsat Imagery in Support to Global Human Settlements Mapping.” Big Earth Data 1 (1–2): 118–44. https://doi.org/10.1080/20964471.2017.1397899.
- Florczyk, Aneta Jadwiga, Stefano Ferri, Vasileios Syrris, Thomas Kemper, Matina Halkia, Pierre Soille, and Martino Pesaresi. 2016. “A New European Settlement Map From Optical Remotely Sensed Data.” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 9 (5): 1978–92. https://doi.org/10.1109/JSTARS.2015.2485662.
- Soille, P., A. Burger, D. De Marchi, P. Kempeneers, D. Rodriguez, V. Syrris, and V. Vasilev. 2018. “A Versatile Data-Intensive Computing Platform for Information Retrieval from Big Geospatial Data.” Future Generation Computer Systems 81: 30–40. https://doi.org/10.1016/j.future.2017.11.007.
Speakers: Dr Christina Corbane (Joint Research Centre), Dr Dario Rodriguez (European Commission - DG JRC)
-
-
