Workshop on the future of Big Data management

27 Jun 2013, 08:00 → 28 Jun 2013, 20:30 Europe/London

Lecture Theatre 3 (LT3) in the Blackett Laboratory (Imperial College London)

David Colling (Imperial College Sci., Tech. & Med. (GB)), Jens Jensen (CLRC-RAL), Wahid Bhimji (University of Edinburgh (GB))

"Big Data" is now being managed by various academic and industry groups. This workshop is organised by the LHC community and will bring together a range of different participants from different disciplines working with Big Data. The aim is to explore the current and future challenges in data processing, storage, transfer and preservation. The workshop focuses on the infrastructure, technologies, and tools, with a view of bringing together communities. This meeting will seek to achieve the following outputs: - To build a cross-disciplinary community in Big Data who can exchange knowledge and best practice and work together as this field evolves in the future. - The meeting discussion will be processed afterwards to form a working document that represent the current state of knowledge and future plans for big data communities. The video of the event is now available: Thu: http://tinyurl.com/BigDataImpThu Fri: http://tinyurl.com/BigDataImpFri

Minutes Big-Data-workshop-2013-06-27.txt Big-Data-workshop-2013-06-28.txt

Thursday 27 June

10:00 Coffee

1 Welcome and Scene Setting

Speaker: Dr David Colling (Imperial College Sci., Tech. & Med. (GB))

Slides Welcome!.pdf Welcome!.pptx

Big data needs of different communities

To establish the requirements driving the later discussion.

Convener: Dr David Colling (Imperial College Sci., Tech. & Med. (GB))

Questionnaire Responses QuestionairreResponses.xlsx

2 High Energy Physics inc LHC

Speaker: Dr Richard Philip Mount (SLAC National Accelerator Laboratory (US))

Slides Big-Data-HEP-v01.pdf Big-Data-HEP-v01.pptx

3 Astronomy inc SKA

Speaker: Dr Paul Calleja (University of Cambridge)

Slides Calleja_Imperial.pdf Calleja_Imperial.ppt

4 Cloud computing and data intensive research

Speaker: Dr Kenji Takeda (Microsoft Research)

5 Earth Observation and Climate Modelling

Speaker: Phil Kershaw (CEMS)

Slides 20130627_Imperial_College_Big_Data,_Philip_Kershaw.pdf 20130627_Imperial_College_Big_Data,_Philip_Kershaw.pptx

6 Weather forcasting

Speaker: Baudouin Raoult (European Centre for Medium-Range Weather Forecasts)

7 PanData and the Research Data Alliance

Speaker: Juan Bicarregui (STFC)

Slides

• BigData2013BicarreguiJuan.pdf
• BigData2013BicarreguiJuan.pptx
• Video

8 Economic and Social Science

Speaker: Fiona Armstrong (ESRC)

Slides Data_Landscape_ESRC_27_June_2013-1.pdf

13:00 Lunch

Big data needs of different communities

To establish the requirements driving the later discussion.

Convener: Jens Jensen (CLRC-RAL)

Questionnaire Responses QuestionairreResponses.xlsx

9 Bioinformatics

Speaker: Dr Guy Coates (Wellcome Trust Sanger Institute)

Slides life-sciences-overview.pdf life-sciences-overview.ppt

10 ELIXIR: An infrastructure for biological information in Europe

Speaker: Andrew Lyall (EMBL-EBI)

Slides 2013_06_27_Imperial_Big_Data_SIG.pdf 2013_06_27_Imperial_Big_Data_SIG.pptx

11 Arts and Humanities

Speaker: Prof. Andrew Prescott (King's College London/AHRC)

Slides prescottimperialbigdata.pdf prescottimperialbigdata.pptx

15:00 Tea

Data Storage: Advanced filesystems and interfaces

• Advances in cluster Filesystems : Lustre; Ceph; HDFS ; GPFS
• Data access interfaces and protocols.
• Storage management interfaces
• Advances in storage hardware.
• High-throughput storage strategies, caching,

Convener: Shaun De Witt (Unknown)

12 GPFS

Speaker: Vic Cornell (DataDirect Networks)

Slides GridScaler_Presentation_Imperial_Jun_2013.pdf GridScaler_Presentation_Imperial_Jun_2013.pptx

13 HDFS

Speaker: Steve Loughran (Hortonworks)

Slides 2013-06-26-ATLAS-HDFS-1.pdf 2013-06-26-ATLAS-HDFS-1.pptx

14 Large scale solutions with Lustre

Speaker: John Swinburne (Intel)

Slides Intel_HPDD_Lustre_Presentation_GridPP.pdf

15 An Intro to Ceph and Big Data

Speaker: Patrick McGarry (inktank)

Slides 20130627_CERN_big_data.odp 20130627_CERN_big_data.pdf

16 CERN experiences with EOS, S3 and Ceph

Speaker: Dirk Duellmann (CERN)

Slides dirkd-eos-cloud-ceph.pdf

17 Discussion: Filesystem needs for different communities

18 Dinner

The Workshop Dinner will be held at Med Kitchen on Gloucester Road and is kindly sponsored by DDN (http://www.ddn.com/). A map to Med Kitchen (with walking directions) can be found in the material attached to this event.

Slides Map from physics to Med Kitchen

Friday 28 June

Data Processing: Toolkits,Data structures, I/O optimisation

• analysis packages and tools for data processing.
• data visualisation
• Serialisation formats
• Layout and access optimisations
• Benchmarking

Convener: Wahid Bhimji (University of Edinburgh (GB))

19 ROOT current architecture and plans

Speaker: Fons Rademakers (CERN)

Slides ROOT-BigData-Analysis-London-2013.pdf

20 Hadoop data processing

Speaker: Steve Loughran (Hortonworks)

Slides 2013-06-26-ATLAS-YARN_(1).pdf 2013-06-26-ATLAS-YARN_(1).pptx

21 Contrast between big data processing in academia and industry

Speaker: Simon Metson (Cloudant)

Slides imperial-bigdata.pdf

22 Optimising bioinformatics pipelines for clinical genomics

Speaker: Dr Michael Mueller (Imperial College)

23 Marmal-aid: a tool for genomics processing

Speaker: Dr Rob Lowe (QMUL)

Slides big_data_rlowe.pdf big_data_rlowe.pptx

24 Astronomy toolkits and data structures

Speaker: Dr Adrian Jenkins (Durham University)

Slides BigData.pdf BigData.ppt

25 Discussion: Building on strengths of tools for all communitiies

11:00 Coffee

Data Storage: Hardware

• Advances in cluster Filesystems : Lustre; Ceph; HDFS ; GPFS
• Data access interfaces and protocols.
• Storage management interfaces
• Advances in storage hardware.
• High-throughput storage strategies, caching,

Convener: Wahid Bhimji (University of Edinburgh (GB))

26 Hardware for big data: lessons learned

Speaker: Marcel van Drunen (Dell)

Slides Marcel_van_Drunen_Dell_Hadoopsizing.pdf Marcel_van_Drunen_Dell_Hadoopsizing.pptx

27 High performance storage solutions

Speaker: James Coomer

Slides

• bd1.pdf
• bd1.pptx
• WOSTest.pdf
• WOSTest.ppt

12:10 Lunch

Data Transfer: Protocols and tools

• File transfer services: (FTS iRods...)
• Remote access (Federated data stores...)

Convener: Roger Jones (Lancaster University (GB))

28 Network developments

Speaker: Paul Lewis (JANET)

Slides BigDataImperial.pdf BigDataImperial.pptx

29 The Evolution of the FTS File Transfer Service Tool

Speaker: Michail Salichos (CERN)

Slides fts3.pdf

30 Federated Data Stores - Volume, Velocity & Variety

Speaker: Andrew Hanushevsky (STANFORD LINEAR ACCELERATOR CENTER)

Slides BDMWS-London.pdf BDMWS-London.pptx

31 EUDAT technology choices

Speaker: Mark van de Sanden (SURFsara)

Slides EUDAT_Workshop_on_Big_Data_management_-_Imperial_College_-_27-28_June_2013.pdf EUDAT_Workshop_on_Big_Data_management_-_Imperial_College_-_27-28_June_2013.pptx

32 Discussion: Future of data transfer

Data Management: meta-data, data discovery and preservation

Convener: Richard Bantges

33 Environmental Data Archival

Speaker: Stephen Pascoe (CEDA)

Slides Environmental_Data_Archival_2013-06-1.pdf

34 The Application of Raimes' Rules to Long-Term Data Preservation

Speaker: Dr Jamie Shiers (CERN)

DPHEP@CHEP2013 https://indico.cern.ch/sessionDisplay.py?sessionId=14&confId=214784#20131016

DPHEP Indico https://indico.cern.ch/categoryDisplay.py?categId=4458

DPHEP.org http://www.dphep.org/

G8 Statement https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/206801/G8_Science_Meeting_Statement_12_June_2013.pdf

H2020@CHEP https://indico.cern.ch/contributionDisplay.py?contribId=494&confId=214784

Slides DPHEP-GridPP.pdf DPHEP-GridPP.pptx

15:35 Tea

Data management 2: Open access and preservation

Convener: Jens Jensen (CLRC-RAL)

35 DOIs for tracking data

Speaker: Mr Matthew James Viljoen (STFC - Science & Technology Facilities Council (GB))

Slides 20130629_big_data_DOIs_for_tracking_data.pdf 20130629_big_data_DOIs_for_tracking_data.pptx

36 Digital preservation

Speaker: Jonathan Tilbury (Tessela)

Slides BigDataPreservation-1.pdf BigDataPreservation-1.pptx

37 Discussion

Roundup and ways forward

Convener: Dr David Colling (Imperial College Sci., Tech. & Med. (GB))

slides Wrap_up_–_what_now.pdf Wrap_up_–_what_now.pptx