WLCG Service Reliability Workshop

Nov 26, 2007, 8:00 AM → Nov 30, 2007, 6:30 PM Europe/Zurich

IT Auditorium (CERN)

NEW! - VRVS has been booked as follows:

• Monday - Island
• Tuesday - Island
• Wednesday - Saturn
• Thursday - Plane
• Friday - Car

Tentative list of topics:

• Critical Services - Experiments' Viewpoint
• Reliability by design - follow-up on issues from WLCG Collaboration workshop in Victoria / CHEP
• Monitoring & end-to-end Service Reliability
• Middleware development - tips & techniques related to reliable by design.
  (Hopefully including a session on developing DB apps).
• WLCG Medium Term Requirements for Operations & Support

Target attendance: 30-50(?) people

Make your suggestions here

mailing list

Monday, November 26

9:00 AM → 6:00 PM Critical services - Requirements

VRVS details: Island

9:00 AM Introduction and Idea of the Workshop

N.B. workshop summary will be given at Overview Board and GDB 1st week of December (next week)

Slides

• wlcg-ob-october.pdf
• wlcg-ob-october.ppt
• workshop-intro.pdf
• workshop-intro.ppt

9:20 AM Critical services - Requirements of the Experiments

10:20 AM coffee break

10:40 AM Techniques for implementing & running robust and reliable services

12:00 PM lunch break

1:30 PM Case Studies - WLCG Services (part 1)

3:00 PM coffee break

3:30 PM Case Studies - WLCG Services (part 2)

4:00 PM Case Studies - Experiment Services

FIO best practices wiki page

Speaker: Veronique Lefebure

Slides Critical_Services_new.pdf Critical_Services_new.ppt

Tuesday, November 27

9:00 AM → 7:00 PM WLCG Operations - What is Required to support LHC experiments?

VRVS details: Island

9:00 AM Inter-site Problem Determination

Slides http://indico.cern.ch/materialDisplay.py?contribId=8&materialId=slides&confId=3578

9:40 AM FTS transfers - debugging tools

• Prototype tools and procedures on T0-export - Alexander Uzhinskiy
• Prototype tools and procedures at SARA - Ron Trompert
• Prototype tools and procedures at IN2P3 - David Bouvet
• Plans and direction (discussion) - Gavin McCance

Slides

• FTSMonitor_at_IN2P3-CC.pdf
• FTSMonitor_at_IN2P3-CC.ppt
• ftsmonitoring_nov2007.pdf
• ftsmonitoring_nov2007.ppt
• FTS_monitoring.pdf
• FTS_monitoring.ppt
• Monitoring_directions.pdf
• Monitoring_directions.ppt

10:40 AM coffee break

11:10 AM Mind the Gap

What can we do to prevent cracks opening (or widening) in the services?

Specific examples from recent times (i.e. during EGEE '07) include:

• SAM unavailability
• GridView - change of availability algorithm
• LFC - affected by Oracle client bug in 'old' versions

More communication and better planning would likely help. How (concretely) do we fix these problems before the deluge of data arrives?

11:40 AM Requirements for WLCG management dashboards

Speaker: John Shade (CERN)

Slides EGEE_Management_Dashboard.pdf EGEE_Management_Dashboard.ppt

12:20 PM lunch break

1:30 PM WLCG / EGEE / OSG operations and evolution in the coming years

WLCG / EGEE / OSG operations are now well established, through:

• Weekly joint operations meetings
• Bi-annual (roughly) workshops
• Sessions at WLCG collaboration workshops
• A set of tools, procedures and documentation.

This session summarises the current state of global operations as well as the possible evolution beyond (or even during) the next phase of EGEE (III) - in the light of the EGI Design Study.

In particular, we need to establish a clear view of our current needs in terms of efficient operations and how this would map to a model where National Grid Initiatives (NGIs) play a significant role.

The issues of 24x7 operations also needs to be discussed with priority.

3:30 PM coffee break

4:00 PM Experiment Operations

What is it that the experiments hate most about the current operations setup?

What explicitly is missing from the point of view of the experiments?

What can be done better? What (perhaps?) should not be done at all?

Should we somehow integrate global / experiment operations? e.g. via repeat consoles in the various operations rooms?

Wednesday, November 28

9:00 AM → 6:00 PM Monitoring - What is Required to run Reliable Services?

VRVS details: Saturn

Morning: Outstanding requirements for current projects and discussion of where this might go - e.g. SAM/ gridview, nagios-based prototype, GOCDB, CIC Portal, Experiment Dashboards

Afternoon: discussion of the requirements identified during Tuesday's sessions, building a medium-long term plan.

9:00 AM Introduction

Speaker: Ian Neilson (CERN)

Slides WLCG_Reliability_Monitoring.pdf WLCG_Reliability_Monitoring.ppt

9:15 AM Management reporting requirements

Speaker: Alberto Aimar (CERN)

Slides LCG_management_20071128.pdf LCG_management_20071128.ppt

9:35 AM OSG Requirements

Speaker: Rob Quick (OSG - Indiana University)

Slides OSG_Monitoring_Requirements.pdf OSG_Monitoring_Requirements.ppt

9:55 AM Requirements from CIC Portal users

Speaker: Gilles Mathieu (IN2P3/CNRS Computing Centre, Lyon, France)

Slides CICportalRequirements_WLCGworkshop112007.pdf CICportalRequirements_WLCGworkshop112007.ppt

10:15 AM Coffee

10:30 AM GOCDB Requirements

Slides Gordon-GOCDB-req.pdf Gordon-GOCDB-req.ppt

10:50 AM EGEE Metrics Implementation Group

Speaker: Etienne Urbah (LAL)

Slides WLCG-Service-Reliability-Workshop-Metrics-Implementation-Group-V8.pdf WLCG-Service-Reliability-Workshop-Metrics-Implementation-Group-V8.ppt

11:10 AM Site Monitoring Prototype Feedback

Speaker: Emir Imamagic (Unknown)

Slides Site_Monitoring_Prototype_Feedback.pdf Site_Monitoring_Prototype_Feedback.ppt

11:30 AM Discussion

12:00 PM lunch break

1:00 PM Service availability calculation methods - now and the future

Speaker: Piotr Nyczyk (CERN)

Slides SAM-availability-calculation.pdf SAM-availability-calculation.ppt

1:20 PM Security for Grid Sites

Speaker: Louis PONCET (CERN)

keynote presentaion SecurityWLCG_workshop.key.zip

Slides

• Security_2007_WLCG_WS.pdf
• SecurityWLCG_workshop.pdf
• SecurityWLCG_workshop.ppt

1:40 PM Use cases for GridMap visualization

Speaker: Max Boehm (Unknown)

Slides GridMaps_UseCases_2007-11-28.pdf GridMaps_UseCases_2007-11-28.ppt

2:00 PM Models for monitoring

Speaker: James Casey (CERN)

Slides 0711-WLCG-Models_for_Monitoring.pdf 0711-WLCG-Models_for_Monitoring.ppt

2:20 PM coffee break

2:40 PM Experiment Critical Services and Monitoring - What's Missing for CCRC'08 (and beyond)?

Speaker: Julia Andreeva (CERN)

4:10 PM Prioritization of requirements raised during the day

4:40 PM LHC VO User Support evolution update

Speaker: Maria Dimou (CERN)

Slides http://cern.ch/dimou/lcg/UserSupport/ServiceReliability-UserSupport-20071128.ppt

5:00 PM Mid-workshop checkpoint

Speaker: Jamie Shiers (CERN)

Slides SoNowWhat.pdf SoNowWhat.pptx

Thursday, November 29

9:00 AM → 2:00 PM Robust Services - Middleware Developers' Techniques & Tips

VRVS details: Plane

Key techniques from middleware / storage-ware developers for making services robust by design

9:00 AM CASTOR

Speaker: Dennis Waldron (CERN)

Slides WLCG_CASTOR2.pdf WLCG_CASTOR2.ppt

9:30 AM LFC / DPM

Speaker: Jean-Philippe Baud (CERN)

Slides LFC-DPM_ReliabilityWorkshop.pdf LFC-DPM_ReliabilityWorkshop.ppt

10:00 AM FTS

Speaker: Paolo Tedesco (Unknown)

Slides FTS_reliability.pdf FTS_reliability.ppt

10:30 AM coffee break

11:00 AM Other m/w sessions: BDII, WMS/LB, VOMS, R-GMA, Logging format

12:30 PM lunch break

2:00 PM → 6:00 PM DB application design issues

2:00 PM DB design (Tips & tricks)

• schema design,
• columns and constraints,
• indexes and partitioning,
• proper use of PL/SQL (bind variables, static and dynamic SQL),
• writing robust applications

Speaker: Dawid Wojcik (CERN)

Slides Workshop_DB_Design.pdf Workshop_DB_Design.ppt

3:30 PM coffee break

3:50 PM DB development tools (with demo)

• bentic
• application express
• sql developer </UL

Speaker: Balys Sulmanas (CERN)

Slides DevTools.pdf DevTools.ppt

4:50 PM DB Monitoring tools (demo)

• session manager
• OEM
• 3D

Speaker: Dawid Wojcik (CERN)

Slides Workshop_Mon_Tools.pdf Workshop_Mon_Tools.ppt

5:30 PM CORAL

Speaker: Andrea Valassi (CERN)

Slides WLCG_CORAL_2007.pdf WLCG_CORAL_2007.ppt

Friday, November 30

9:00 AM → 1:30 PM DB - performance and tuning issues

VRVS details: Car

9:00 AM DB performance/tuning in a RAC environment for developers

- Advanced SQL - Optimization - Examples from production

Speaker: Miguel Anjo (CERN)

Slides WLCG_Reliability_db-performance-tuning.pdf WLCG_Reliability_db-performance-tuning.pptx

10:15 AM coffee break

10:40 AM DB performance tuning in a RAC environment for administrators

• Service architecture for performance
• Monitoring and reporting performance issues
• Reactive performance tuning, case studies
• New HW sizing and capacity planning

Speaker: Luca Canali (CERN)

Slides WLCG_Oracle_perf_for_admin_Luca_Nov07.pdf WLCG_Oracle_perf_for_admin_Luca_Nov07.ppt

12:00 PM Dataguard for RAC Migration

Speaker: Jacek Wojcieszuk (CERN)

Paper CERN_Dataguard_Migration2.pdf

Slides DataGuard_for_RAC_migrations_v2.pdf DataGuard_for_RAC_migrations_v2.ppt

12:30 PM lunch break

1:30 PM → 5:30 PM DB - service issues

1:30 PM Service plans for 2008

Speaker: Maria Girone (CERN)

Slides DBservices_2008_SRW.pdf DBservices_2008_SRW.ppt

1:45 PM Service Recommendations

• Security of machines and authentication techniques
• How to manage your logs (listener.log, crs logs, alert logs etc)
• How to manage your Oracle environment (host environment)
• A quick recap on the backup emails (from the talk at CNAF)
• Managing your targets in Grid Control Security of machines and authentication techniques

Speaker: Gordon Brown (CCLRC)

Slides CERN_WLCG_2007_1.pdf CERN_WLCG_2007_1.ppt

2:45 PM streams service review

Speaker: Dafonte Perez Eva (CERN)

Slides StreamsServiceReview.ppt

3:45 PM coffee break

4:05 PM adding/removing nodes plus demo

Speaker: Jacek Wojcieszuk (CERN)

Slides RAC-adding_and_removing_nodes.pdf RAC-adding_and_removing_nodes.ppt