Speakers
Description
The Copernicus programme of the European Union with its fleet of
Sentinel satellites for monitoring land, ocean, and atmosphere with
applications from environment monitoring to emergency response is
generating Terabytes of free and open data on a daily basis. The
Joint Research Centre (JRC) of the European Commission has developed a
prototype Joint Earth Observation Data and Processing platform
(JEODPP) to enable its knowledge production units to process and
analyse global geospatial data at Petabyte scale in support to EU
policy needs. In the framework of a collaboration between CERN and
JRC, the EOS distributed file system enables high data throughput
between the processing nodes and the storage servers. The performance
of the JEODPP is analysed on use-cases in the context of interactive
and batch processing based on docker containerisation and managed by
the HTCondor workload manager. Web-based interactive analysis and
visualisation of the Copernicus data on the EOS repository is obtained
via Jupyter Notebooks connected to distributed backend processing
servers. The process distribution and real-time visualisation of the
results on interactive maps is achieved via custom interactive widget
like ipyleaflet used in the Jupyter notebooks. This way the data
analysis capability of the JEODPP can be shared with internal or
remote user groups.
