Big Data processing and analysis challenges in mega-science experiments

29 Jan 2015, 09:00 → 31 Jan 2015, 15:00 Europe/Moscow

NRC KI and JINR

Alexei Klimentov (Brookhaven National Laboratory (US)), Vladimir Korenkov (Joint Inst. for Nuclear Research (RU))

How to come to NRC KI NRC_KI_scheme_eng_.doc NRC_KI_scheme_eng_.pdf

how to find LIT JINR map_JINR.bmp

link to hotel aerostar http://www.aerostar.ru

link to hotel Pekin http://hotelpeking.ru/en/

link to Mariott (Tverskaya Yamskaya) http://marriottmoscowtverskaya.ru

Схема проезда к НИЦ КИ Схема_расположения_Курчатовского_центра_культуры_и_проходной_НИЦ_«Курчатовский_институт».docx Схема_расположения_Курчатовского_центра_культуры_и_проходной_НИЦ_«Курчатовский_институт».pdf

Thursday 29 January ¶

09:45 → 10:00 Welcome. Dr.V.Velikhov, NRC-KI vice-director; Dr.V.Demin, NBICS vice-director¶

Introduction and logistics. A.Klimentov NRC-KI-workshop_Jan29_2015.pdf NRC-KI-workshop_Jan29_2015.pptx

10:00 → 11:50 Database Technologies¶

Convener: Dr Maria Grigorieva (NRC KI)

slides

• database_discussion.pdf
• Gancho_Dimitrov_PanDA_DB_implementation_Moscow_workshop_29thJan2015.pdf
• HStorage_development_state.pdf
• lassnig_db_teg.pdf
• Our_experience_in_NoSQL_integration.pdf
• Our_experience_in_NoSQL_integration.pptx
• swc20120614_lassnig.pdf

Topics for discussion:

1. Our experience of the developing of Hybrid SQL/NoSQL Storage, and NoSQL database integration in BigPanDA monitor

2. Database performance tests : SQL - NoSQL, NoSQL - NoSQL

3. Technology evaluation tests results for NoSQL databases: MongoDB, HBase, Cassandra, Dremel, CouchDB, MariaDB

4. Experience of using Hadoop/Spark/MapReduce in PanDA Infrasctucture. Use cases.

5. Foreseen performance and possible changes in PanDA Oracle archived database schema during/after the LHC Run2

6. Query routing strategy in BigPanDA applications (BigPanDA Monitor in particular)

7. How to implement cross database requests in heterogeneous architecture

8. Strategies of the data modelling for NoSQL databases 

10:00 → 11:00 EOS and Data Store¶

Convener: Eygene Ryabinkin (National Research Centre Kurchatov Institute (RU))

10:00 → 11:50 PanDA@NRC-KI¶

Convener: Ruslan Mashinistov (Russian Academy of Sciences (RU))

10:00 PanDA @ NRC-KI¶

PanDA instance installation and commissioning status PanDA for ATLAS PanDA beyond LHC and HEP

Speaker: Ruslan Mashinistov (Russian Academy of Sciences (RU))

Slides PanDA_instance_@_RRC-KI_-_Jan29.pdf

10:50 Discussion¶

PanDA beyond ATLAS : ALICE Do we need version w/o VOMS, Grid Services… Do we need Light weight DDM Staging and I/O matters

10:00 → 11:00 webFTS¶

Convener: Mr Andrey Kiryanov (B.P. Konstantinov Petersburg Nuclear Physics Institute - PNPI ()

10:00 Introduction: current status of FTS3/WebFTS¶

Speaker: Oliver Keeble (CERN)

Slides Presentation1.pdf

10:10 FTS readiness for transfers to/from non-Grid resources (VOMS/non-VOMS credentials, etc.)¶

10:20 Impact of Federated Identity support in WebFTS for non-Grid users¶

Slides http://cern.ch/kiryanov/WebFTS-IdF.pdf

10:30 FTS & HPC: old/new APIs, lightweight clients, Python client library for PanDA integration¶

10:40 Usage scenario for Titan: requirements from both sides¶

Speakers: Mr Andrey Kiryanov (B.P. Konstantinov Petersburg Nuclear Physics Institute - PNPI (), Danila Oleynik (Joint Inst. for Nuclear Research (RU))

10:50 Feature requests for FTS/WebFTS¶

11:00 → 11:50 Meeting with Big Data Lab PhD students (round table)¶

Conveners: Alberto Pace (CERN), Dr Markus Schulz (CERN), Massimo Lamanna (CERN), Prof. Shantenu Jha (Rutgers U), Simon C. LIN (Academia Sinica)

round table panel and participants list Students_and_PhD_students_list_for_round_table-2.docx Students_and_PhD_students_list_for_round_table-2.pdf

11:50 → 13:30 BigPanDA¶

Convener: Dr Alexei Klimentov (Brookhaven National Laboratory (US))

11:50 Introduction¶

11:55 webFTS highlights¶

Speakers: Mr Andrey Kiryanov (B.P. Konstantinov Petersburg Nuclear Physics Institute - PNPI (), Dr Markus Schulz (CERN), Oliver Keeble (CERN)

12:05 Database session highlights¶

Speakers: Gancho Dimitrov (CERN), Dr Maria Grigorieva (NRC KI), Ms Marina Golosova (NRC KI), Dr Mario Lassnig (CERN)

Slides Status_report.pdf Status_report.pptx

12:15 PanDA session highlights¶

Speakers: Danila Oleynik (Joint Inst. for Nuclear Research (RU)), Kaushik De (University of Texas at Arlington (US)), Ruslan Mashinistov (Russian Academy of Sciences (RU))

Slides panda-session-summary-jan15.pdf panda-session-summary-jan15.pptx

12:25 PanDA HPC pilot¶

Speakers: Danila Oleynik (Joint Inst. for Nuclear Research (RU)), Prof. Shantenu Jha (Rutgers U)

Slides PanDA_Pilot_HPC_(KI).pdf PanDA_Pilot_HPC_(KI).pptx

12:45 webFTS integration with PanDA¶

Speakers: Mr Andrey Kiryanov (B.P. Konstantinov Petersburg Nuclear Physics Institute - PNPI (), Danila Oleynik (Joint Inst. for Nuclear Research (RU)), Kaushik De (University of Texas at Arlington (US)), Oliver Keeble (CERN)

13:30 → 14:30 Lunch

14:30 → 16:50 Modeling Distributed Computing Systems¶

Conveners: Dr Eugene Burnaev (IPPI RAS), Kaushik De (University of Texas at Arlington (US)), Prof. Shantenu Jha (Rutgers U)

14:30 → 16:30 visit to I.V.Kurchatov museum and NRC KI Supercomputing center¶

doodle poll to visit I.V.Kurchatov museum http://doodle.com/hmwa3377raash3ne

doodle poll to visit NRC KI Tier1 and supercomputi http://doodle.com/bmksyczw8hye5waw

14:30 → 16:00 Nano-,bio-,information and cognitive technologies Institute seminar¶

14:30 "PanDA, A New Paradigm for Computing in HEP"¶

Speaker: Kaushik De (University of Texas at Arlington (US))

Slides panda-ki-jan15.pdf panda-ki-jan15.pptx

15:10 CERN and Computing¶

Speaker: Alberto Pace (CERN)

Slides 2015-01-29-Alberto-Pace-Moscow-Big_Data_challenges_in_mega-science-V03.pdf 2015-01-29-Alberto-Pace-Moscow-Big_Data_challenges_in_mega-science-V03.pptx

17:00 → 17:20 Leave to JINR by bus from NRC KI

Friday 30 January ¶

10:00 → 10:15 Welcome from Prof.N.Rusakovich, JINR Chief Scientific Secretary¶

10:15 → 11:10 Modeling in mega-projects¶

10:15 NICA mega-project¶

Speaker: Prof. O.Rohachevsky (JINR)

10:35 Modeling of grid-cloud systems¶

Speaker: Prof. Gennady Ososkov (JINR)

11:10 → 11:25 coffee break

11:25 → 12:25 BigPanDA project¶

11:25 Project overview : Growing PanDA ecosystem, PanDA beyond ATLAS, prospects for ALICE, COMPASS, NICA.¶

Speaker: Kaushik De (University of Texas at Arlington (US))

Slides panda-dubna-jan15.pdf panda-dubna-jan15.pptx

11:40 Towards an Abstractions-driven and Standards-based Next-generation of Distributed Cyberinfrastructure¶

Speaker: Prof. Shantenu Jha (Rutgers University)

12:05 Network awareness¶

Speaker: Artem Petrosyan (Joint Inst. for Nuclear Research (RU))

Slides network_awareness_2015.01.30.pdf network_awareness_2015.01.30.pptx

12:25 → 14:00 Technology trends¶

12:25 webFTS and FTS3¶

Speaker: Oliver Keeble (CERN)

Slides FTS3_Trends_v3.pdf

12:45 EOS¶

Speakers: Alberto Pace (CERN), Massimo Lamanna (CERN)

Slides DUBNA20150129.pdf

13:05 ORACLE and Hadoop/HBASE in ATLAS¶

Speakers: Gancho Dimitrov (CERN), Dr Mario Lassnig (CERN)

Slides Gancho_Dimitrov_Oracle_in_ATLAS_Dubna_Workshop30thJan2015_2.pdf hadoop_atlas.pdf

13:40 CLAVIRE Highlights¶

Speakers: Prof. Alexander Boukhanovsky (ITMO), Dr Denis Nasonov (ITMO)

Slides CLAVIRE_2015.pdf CLAVIRE_2015.pptx

14:00 → 15:00 lunch

15:00 → 15:30 Monitoring¶

15:00 NoSQL for the storage of monitoring data:¶

Speaker: Ivan Kadochnikov (Joint Inst. for Nuclear Research (RU))

15:15 Monitoring in DIRAC environment for the BES3 experiment¶

Speaker: Igor Pelevanyuk (Joint Inst. for Nuclear Research (RU))

Slides

• JINR_Meeting_BESIII_Dark.pdf
• JINR_Meeting_BESIII_Light.pdf
• JINR_Meeting_BES_Light.pptx

15:30 → 16:20 Reports on Tier1¶

15:30 ASGC TW Tier-1¶

Speaker: Simon Lin (Academia Sinica (TW))

Slides 20150130-ASGC-SimonLin.pdf 20150130-ASGC-SimonLin.pptx

15:50 NRC KI RF Tier-1¶

Speaker: Dr Vassily Velikhov (NRC KI)

16:05 JINR Tier-1¶

Speaker: Dr Tatiana Strizh (JINR)

16:20 → 16:35 tea break

16:35 → 16:55 Presentation of the heterogeneous cluster of LIT¶

Speaker: Dr D.Podgainy (JINR)

16:55 → 17:30 Excursion in LIT¶

18:00 → 21:00 Beer party¶

doodle poll http://doodle.com/b7qnir6hw4h2epwk#table

Saturday 31 January ¶

09:30 → 13:00 Technical splinter meetings¶

09:30 Distributed Modeling splinter meeting¶