Operations team & Sites

Tuesday 27 Sept 2011, 11:00 → 12:16 Europe/London

EVO - GridPP Operations team meeting

- This is the weekly ops & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 108203 with code: 4880. Apologies: Raja, Jeremy, Andy W, Wahid

Minutes gridpp-ops-minutes-2011-09-27.txt

11:00 → 11:20 Meetings & updates

- ROD team update -- Issues from move over to NGI_UK - Nagios status - Tier-1 update - Security update -- T2 issues Input from Mark M who is away at an LHCONE meeting: 1) Grid Mon. Andy Pickford, at glasgow, has managed to get the site up and running again. It is not in its final format but some of the tests do work between sites. The URL is http://gm001.gla.scotgrid.ac.uk/gridmon/graph.html 2) IPv6 Work. Ongoing, we should have several boxes up and running at Glasgow by the end of October 3) CVMFS role out. All three Scotgrid sites will have CVMFS either running or in process of installation by the end of October 4) Durham Update. Durham will be in downtime until the 30th of September. The site DPM has lost its meta data. Atlas to be informed. SE at Durham will be updated Sl5 1.8.1 EMI 5) LHCONE Strategy Document for GridPP. I will be out at SARA, as I previously stated, I will report back to the collaboration on the 4th of October. JANET appear to have a cunning plan and I have a proposal for a new test network infrastructure which should make the file transfers quicker. -- General notes. Checking Red/Amber tickets for NGI_UK:http://tinyurl.com/3endfg7 Or go to https://ggus.eu/ws/ticket_search.php and select Support Unit:NGI_UK and Creation date: Any and Status: open states - then click Go. Checking Red/Amber tickets for ROC UKI: http://tinyurl.com/3ctjrna

11:20 → 11:40 Experiment problems/issues

Review of weekly issues by experiment/VO - LHCb - CMS - ATLAS - Other -- Pheno have been having some proxy/site issues recently and may join the meeting to discuss some tickets. - Experiment blacklisted sites - Experiment known events affecting job slot requirements - Site performance/accounting issues - Metrics review

Atlas Report Atlas Report

11:40 → 11:50 Ops team core tasks - overview

Quick look back at what the posts are about. Operations falls under "WP-C Grid Deployment, Operations and Support". It was defined in the GridPP4 proposal as "Coordination across Grid operational tasks including infrastructure wide monitoring, data handling, security, release management, training, documenting and support of institute based system administrators. The GridPP “Ops-Team” and NGI-based operations will be essential for the coherent and secure operation of a UK Grid. The support for data handling is mission-critical for the efficient extraction of physics results." "In GridPP3, there are 21 FTE supporting the Tier-2s; 13 System Administrators, 4 Tier-2 coordinators; and 4 deputy coordinators funded by EGEE. ... Instead, this proposal requests just 21 FTE, consisting of 13 system-administrators and 8 more senior posts that will, in addition, take specific national responsibilities in the core Grid Operations team described in Section 8. These 8 operations experts or coordinators effectively replace the roles of the existing 8 Tier-2 coordinators and their deputies, building on that successful structure but engaging more sites directly in the national operations." Then for the section 8 summary see the attached PDF. ATLAS defined 5 'group analysis sites' and CMS 3. These provide the 8 mentioned above. In addition there are others who contribute to the core activities by virtue of having a cross site role or being part NGI/EGI funded. There are no clear distinctions in several cases. What we arrived at so far is this breakdown for the initial allocations. Region On Duty - shifts Daniela – Stuart – John – Andrew – Kashif Ops tasks Core grid services/Monitoring/Regional tools = Mark/David/Kashif Security = Ewan(/Rob) Documentation/website = Andrew/Stephen Wider VO services = Chris Staged rollout = Daniela/Duncan Ticket follow-up = Stephen/Matt On-duty coordination = Jeremy Accounting = Rob/Alessandra Other grids = Stuart It needs to be clear that the names are the people COORDINATING the task area. So they are the ones expected to report on those areas at ops team meetings. What will they be reporting, what tasks define the coordination role and activity in that area? That is what we need to define and conclude on at the F2F core team meeting at QMUL on 11th November. The tasks are currently 'defined' in the wiki with linked pages that are dedicated to defining that particular task. https://www.gridpp.ac.uk/wiki/Category:GridPP_Operations is the main page. Then an example linked page is: https://www.gridpp.ac.uk/wiki/Staged_rollout. The text in these pages should evolve to reflect what the *team* believe is required from these tasks (and want to happen) and in this and some of the upcoming meetings there is a focus section on the agenda that should be led by the task leads to get input on the task from the whole team (Jeremy's initial thoughts are the text to kick off the discussions). Everyone in the team can be expected to contribute in some way to several tasks but the named people are the ones responsible for developing it (with Jeremy... but since he is away you have your chance to set out some of your own ideas!). At the F2F meeting we will go through each area (including the special one of strategy) and agree what we expect from each area, who will provide extra effort and how we will monitor progress. We need a coherent structure of work in place and showing results by April when we will report at GridPP28. The focus for all of this is improving and streamlining operations and the better we couple with the experiment needs the better (so for example the monitoring activity might include working with people on the WLCG dashboard and the staged rollout might include providing some effort to test requested experiment deployments). If all this sounds rather wooly then that's because it is! It is up to the ops-team to build better processes for monitoring tickets, validating accounting and spotting problems, getting missing documentation written etc. We are already doing some of the activities but in an adhoc and uncoordinated way.

Core-ops-team-text operations-team.pdf

11:50 → 12:00 Ops team core tasks

This is a first attempt to expose and develop some feedback in some of the core tasks. In this case: Staged rollout/Ticket coordination/Documentation Refer to: http://www.gridpp.ac.uk/wiki/Category:GridPP_Operations Details of the tasks as defined at the moment: http://www.gridpp.ac.uk/wiki/Staged_rollout http://www.gridpp.ac.uk/wiki/Ticket_follow-up http://www.gridpp.ac.uk/wiki/Documentation Please can the task leads try to lead a discussion on some of the things they see as needed in their areas? So for example with staged rollout: - do all sites know what UK sites are already involved? - do people know the sorts of findings testing EMI-1 releases? - do site admins know what changes are being introduced in upcoming releases? This task could for example (perhaps by way of a summary webpage): - bring together the sites carrying out the testing - provide summaries to the ops team meeting on a biweekly basis about changes and findings with recent releases - gather comments about recent production releases to feedback to EGI - request additional support to test an important release - follow-up on questions about changes to the experiment frameworks that affect site deployments ..... The other areas have similar and more questions so please explore them and feedback to the task lead who will record them in the wiki and present the best ideas/use of time at the F2F.

11:55 → 12:00 Actions

- http://www.gridpp.ac.uk/wiki/Operations_Team_Action_items

12:00 → 12:01 AOB

- Conclusions on dates for HEPSYSMAN and core ops team F2F.