Operations Meeting
CERN
Weekly Operations' Meeting 2005-05-02
Agenda: http://agenda.cern.ch/fullAgenda.php?ida=a045865
Operations' Manual: http://lcgdeploy.cvs.cern.ch/cgi-bin/lcgdeploy.cgi/lcg-docs/EGEE-CIC-Operational-Manual/opMan.pdf
CA rpm instructions: http://cern.ch/grid-deployment/lcg2CAlist.html and https://cic.in2p3.fr/index.php?id=rc&subid=rc_config
Template for weekly report submission by the ROC: http://cern.ch/egee-docs/list.php?dir=.\operational_tools\&
Participants:
Asia Pac.:
CE: Marcin Radecki
CERN: Simone Campana, Maria Dimou (Secretary), Ian Neilson, Piotr Nyczyk.
CH/DE: Holger Marten, Sven Hermann
FR: Pierre Girard, Gilles Mathieu, Rolf Rumler, Osman Aidel, Frederic Schaer
IT: Paolo Veronesi
NE: Per Öster
SEE:
SWE: Gonzalo Merino
Russia: Alex Kryukov, Victor Edneral
UK/IR:
Apologies: Patricia Mendez Lorenzo (CERN, Alice VO support), Antonio Delgado Peris (CERN. Biomed VO support).
Absences: Nobody joined from the UK (CIC-on-Duty). Bank holiday (May 1st). Nobody joined from SEE. Bank holiday (Orthodox Easter Monday). Min Tsai (Asia Pacific) is in Taipei.
Comments on the notes of the last meeting:
No comment.
CIC & ROC reports:
Exiting CIC-on-duty (Russia):
A full report is in the CIC-on-duty log, starting as [2005-05-02 10:41] - Victor Edneral
ROC reports must be uploaded to this meeting's agenda as "More Information" by the CIC-on-duty and the ROC managers themselves by 11am CET.
The New report template, written by Osman Aidel, is linked from the ROC views of the CIC web site. Feedback received from Sven and Marcin since the last meeting is integrated in the present version.
Issues discussed in addition:
IT: LSF Info Provider bugs in savannah recently submitted but still with no comment received. Maria, while writing these notes, sent a reminder to Laurence. His answer (savannah tickets updated):
Bug #8332, is a request for new functionality. Currently the dynamic plug-in forLSF does not produce a value for GlueCEStateStatus. This will be done for the
next release.
Bug #8296 shows up due to the way the site is configuring of LSF. In similar
cases where a site has found a problem due to the local configuration of LSF,
the local LSF expert fixed the problem and submitted a patch to be included. Thenew rpm can be then made very quickly. As I am not an LSF batch system expert
and I can't reproduce the problem locally, it is diffuclt for me to solve. Thereare a couple of INFN people how could problem help find a fix,
Stefano Borgia
Andrea Chierici
Alessandro Italiano
These people are running LSF at their sites and have supply bug fixes and patches.
CMS requires a normalisation of OS used by the sites. Example:
ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b mds-vo-name=local,o=grid| grep -e
OperatingSystemName -e OperatingSystemRelease -e OperatingSystemReleaseVersion
GlueHostOperatingSystemName: SLC
GlueHostOperatingSystemRelease: 3.0.4
GlueHostOperatingSystemVersion: 3
GlueHostOperatingSystemName: SL
GlueHostOperatingSystemRelease: 3.0.2
GlueHostOperatingSystemVersion: 3
GlueHostOperatingSystemName: CentOS
GlueHostOperatingSystemRelease: 3.0.4
GlueHostOperatingSystemVersion: 3
GlueHostOperatingSystemName: RHEL
GlueHostOperatingSystemRelease: 3.0.3
GlueHostOperatingSystemVersion: 3
As this couldn't be sorted out as fast, it is now ACTION 2005-05-02--2.
CE: Marcin asked which is the policy concerning VO data loss on a SE. Simone suggested that the site should submit the problem in ggus, asking the VO to clean-up their catalog. The policy, in general, depends on the VO-site relationship. Once this point is clarifed, Maria offered to include it in the VO Registration Procedure, if people think this is appropriate. ACTION 2005-05-02--3.
LCG2_4_0 Status:
DE: Holger said that all sites in their region upgraded to LCG2 2_4_0 but 3 sites remain, which are actually major.
These are Aachen, DESY & FZK. The reasons vary. Some were at the EGEE conference in Athens while the upgrade deadline was approaching. Others wish to modify the installation procedure by using Quattor. Most importantly DESY has now a large production (Zeus and H1 Experiments' Monte Carlo) running with LCG2 2_3_1. Upgrade is not possible because if the WNs pass on 2_4_0 and the Service Nodes stay on 2_3_1, inter-operability issues will arise.
NB! All CICs, please, re-visit the on-duty rotation agenda.
Security report (Ian Neilson):
- Nothing to report.
Next CIC-on-duty:
- The CIC-on-duty this week is the UK.
- The actual work is done by the ROCs and the follow-up by the CICs. The CIC-on-duty just does the monitoring.
- The ROC managers should remember that when a problem at a site is solved the site needs to go through a period of quarantine before joining production again. This is mentioned in the Operations' Manual.
- The CIC site (requires personal certificate loaded) is http://cic.in2p3.fr .
- Link to the daily Site Status reports.
- Savannah should be used for the Tasks' hand-over and follow-up. The Operations' Manual should be the basis.
- When the CICs report also as ROCs, they should do this in separate reports.
- Sites' status of updates can be checked in the daily reports e.g. http://cern.ch/lcg-testzone-reports/cgi-bin/lastreport.cgi
Action List now in a separate file.
A.O.B.
Frederic reminded that CA 0.28 should now be installed everywhere. Piotr said that beginning of May CA and R-GMA client availability on WN becomes critical and, if found not configured, the site will be marked as non-functional.The VOs should make the union of tests and decide which ones are critical. They will then ask the Operations' Team to run them. Gonzalo asked to have the result of Site Functional Tests (SFT) per VO, up-to-date and accessible on the web at any time. via cic portal or on a SFT/VO page. Piotr will make SFT easily configurable for VOs.ACTION 2005-05-02--4.
- While discussing ACTION 2005-02-28--1, all participants expressed doubt on how to identify the VO responsibles for relations with the sites. Maria will ask the people who made the GGUS supporters' categories and Simone will relay the request to the rest of EIS.
Next meeting:
- The next regular telephone meeting will take place on Monday May 9th 2005 from 14:00 to 16:00 in conference room 28-R-015.
Maria Dimou, IT/GD, Grid Infrastructure Services