Choose timezone
Your profile timezone:
Operations meeting
→
Europe/Zurich
CERN
CERN
Description
Phone: +41 22 767 7000 \"Grid Operations/Ian Bird\"
ACTION LIST
Weekly Operations' Meeting 2005-03-21
Agenda: http://agenda.cern.ch/fullAgenda.php?ida=a045859
Operations' Manual: http://lcgdeploy.cvs.cern.ch/cgi-bin/lcgdeploy.cgi/lcg-docs/EGEE-CIC-Operational-Manual/opMan.pdf
CA rpm instructions: http://cern.ch/grid-deployment/lcg2CAlist.html and https://cic.in2p3.fr/index.php?id=rc&subid=rc_config
Template for weekly report submission by the ROC: http://cern.ch/egee-docs/list.php?dir=.\operational_tools\&
Participants: Stephen Burke (RAL, UK), Alessandro Cavalli (CNAF, IT), Maria Dimou (CERN, Secretary), Sven Gabriel (FZK), Jorge Gomez (SWE), Alexander Kryukov (RU), Holger Marten (FZK), Gilles Mathieu (IN2P3,FR), Ian Neilson (CERN), Piotr Nyczyk (CERN), Per Öster (Stockholm, NE), Ognjen Prnjat (SEE), Nick Thackray (CERN, chair), Steve Traylen (RAL, UK), Min Tsai (CERN & Taiwan), Jules Wolfrat (SARA, NE).
Apologies: Marcin Radecki (CE)
Absences:
Comments on the notes of the last meeting:
This was not discussed.
CIC & ROC reports:
CIC-on-duty report (Gilles from IN2P3):
A full report is in the CIC-on-duty log, starting as [2005-03-21 10:31] - Jonathan Schaeffer
ROC reports must be uploaded to this meeting's agenda as "More Information" by the CIC-on-duty and the ROC managers themselves by 11am CET. The report template, is linked from the ROC views of the CIC web site and from the 2005-02-28 meeting's agenda.
Issues discussed in addition:
Gilles reported that GGUS is used in operation this week. Gilles and Piotr are working together at CERN this week on hand-over from savannah. Piotr reminded the ROCs that old tickets have to be handled in savannah and new ones in GGUS so ROC managers should look in both places for some time in the near future.
It can now be documented in the Operations Manual that, after initial contact with the site and the ROC, escalation will go to the ROC only (and no more to the site) by the CIC-on-duty. DE-CH have a rotational system of its "sub-ROCs" so they have to work on a procedure to hand pending problems to their successor.
Russia will be CIC-on-duty in the week of May 2nd. Alex will discuss in the Athens EGEE conference with experienced CICs their feedback from operations. Meanwhile, they translated the Operations Manual in russian.
NB! All CICs, please, re-visit the on-duty rotation agenda. The author of these notes has no "Authorisation" to view this agenda. Hoping that the relevant people can view it.
Issue that came up from the SEE report:
We upgraded our regional BDIIs to lcg-bdii-3.2.1-1 because it resolves a number of race conditions, therefore it is much more stable and helps the respective RB become rock solid.
Laurence reported though that this version has still one or two minor glitches thus its not ready for mass deployment.
On Laurence Field's advice, don't install bdii-3.2.1-1 if not yet done. Wait for the one that comes with LCG2 2_4_0.
Sites are reminded to move Service Nodes to SLC3 a.s.a.p.
If the Site Functional Tests (SFT) show a site as non-functional, the present procedure suggests to contact the CIC-on-duty. They may re-run the daily tests if they have a hint that the problem may have been temporary. Experiments (Atlas, maybe others) were asking if the CIC-on-duty could re-run the tests before contacting the site. As these tests need 30 minutes to complete as we often have 40% non-functional, it is impossible to re-run the tests by hand for every failing site. They will contact Piotr to tailor their scripts that black/white list sites and configure their BDII appropriately.
Ognjen expressed concern about VO managers black-listing sites with arbitrary criteria, even if the sites are fully functional. Steve said there is no obligation from the VOs to use all sites. Ognjen and Sven said a fair and coherent use of the infrastructure should be done. Raise at the ROC managers meeting and in Experiment coordination meetings (which?)
Gilles explained access permissions to the new web portal. Restriction failed so read access is proposed for people with certificates. Restricted access should only apply to the usage of tools.
Synchronisation of the cic portal with people's information in GOC db will be discussed at RAL next week.
CIC portal contains the RBs found in the Information System (IS). This doesn't match with manually prepared lists from information in GOCdb. The only valid list is the one from the IS. Piotr pointed out that if a site 'promised' in GOCdb to makes available a RB, then it should indeed be found in the IS. So any such mismatch should not happen and, in the future, it should be detected by the monitoring tools.
Security report (Ian Neilson):
- The UK report included an incident reported on March 12th concerning the Lancaster site. Ian and Steve will follow-up what exactly happened and why it was discovered on a later date.
Next CIC-on-duty:
- The CIC-on-duty this week is CERN.
- The actual work is be done by the ROCs and the follow-up by the CICs. The CIC-on-duty just does the monitoring.
- The ROC managers should remember that when a problem at a site is solved the site needs to go through a period of quarantine before joining production again. This is mentioned in the Operations' Manual.
- The CIC site (requires personal certificate loaded) is http://cic.in2p3.fr .
- Link to the daily Site Status reports.
- Savannah should be used for the Tasks' hand-over and follow-up. The Operations' Manual should be the basis.
- When the CICs report also as ROCs, they should do this in separate reports.
- Sites' status of updates can be checked in the daily reports e.g. http://cern.ch/lcg-testzone-reports/cgi-bin/lastreport.cgi
Next meeting:
- The next regular telephone meeting will take place on Monday April 4th 2005 from 14:00 to 16:00 in conference room 28-R-015. PLEASE NOTE: NO MEETING ON EASTER MONDAY!
Maria Dimou, IT/GD, Grid Infrastructure Services
There are minutes attached to this event.
Show them.