Go to the Indico Home Page Oldest event Older event Up to category Newer event Newest event Printable version
  • Compact style
  • Indico style
  • Indico style - inline minutes
  • Indico style - numbered
  • Indico style - numbered + minutes
  • Indico Weeks View
    • Switch to the management area of this event
    Login

    Operations meeting

    → Europe/Zurich
    CERN

    CERN
    Ian Bird
    Description
    Phone: +41 22 767 7000 \"Grid Operations/Ian Bird\"
    CERN meeting room 28-R-006
    NOTE THE CHANGE OF ROOM
    Hide

    Weekly Operations' Meeting 2005-02-28

    Agenda: http://agenda.cern.ch/fullAgenda.php?ida=a045856
    Operations' Manual: http://lcgdeploy.cvs.cern.ch/cgi-bin/lcgdeploy.cgi/lcg-docs/EGEE-CIC-Operational-Manual/opMan.pdf
    Template for weekly report submission by the ROC: http://cern.ch/egee-docs/list.php?dir=.\operational_tools\&

    Participants: Simone Campana (CERN, ATLAS support, part-time), Alessandro Cavalli (INFN), Jeremy Coles (RAL, UK), Maria Dimou (CERN, Secretary), David Kant (RAL GOC, for the Action list discussion), Kostas Koumantaros (SEE), Holger Marten (FZK), Gonzalo Merino (PIC, SWE), Piotr Nyczyk (CERN), Marcin Radecki (CE), Rolf Rumler (IN2P3, FR), Frederic Schaer (IN2P3,FR), Philippa Strange (RAL, UK), Nick Thackray(CERN, Chair), Ron Trompert (SARA, NE),Min Tsai (CERN & Taiwan). There was a report but no participant from Russia.

    Corrections to the notes of the last meeting:
    1. The CIC-on-duty report was given by Frederic and not by Piotr, this was an editing mistake by the editor of the notes.
    2. The person on the phone who had suggested to re-try VRVS was Gonzalo (PIC) and not Alessandro (CNAF), this was an auditing mistake by the editor of the notes:-)

    CIC & ROC reports:

    CIC-on-duty report (Piotr Nyczyk from CERN):
    A full report is in the CIC-on-duty log, starting as [2005-02-28 11:53] - Piotr Nyczyk 9654. To navigate from the entry page http://cic.in2p3.fr, click on CIC views, On Duty Dashboard, FollowUp, FollowUp using LogFiles. This is the last entry on page 01/2005 : html

    ROC reports are now submitted by the CIC-on-duty and the ROCs according to this template, linked from the ROC views of the CIC web site and from this meeting's agenda.

    Reports should be sent to the project-egee-roc-managers@cern.ch list by 11am CET. They are linked from the meeting agenda. CERN submitted no report again because it is not a ROC but a single site. Kostas and other ROC managers asked for CERN reports in the future.

    Issues discussed in addition:
     

    1. Piotr changed the RB to CNAF during the CERN network problems. More than one BDII is needed in the Replica Manager configuration.

    2. The CIC managers met last week and decided to, each, run a RB and a BDII.

    3. The problem with the Toronto site that didn't respond to escalation as reported by the last CIC-on-duty (IN2P3 for the week of Feb. 14th week) is now suspended because they surfaced on Feb. 25th.

    4. Simone Campana, member of the Deployment Team dedicated to ATLAS Experiment Integration Support (EIS) came to the meeting to convey a request from the ATLAS collaboration (appended). The original text is left unchanged. The editor of these notes supposes that 'raking' stands for 'ranking' etc. Action 2005-02-28--1resulted from this discussion.

    From: Laura Perini [mailto:laura.perini@mi.infn.it]
    Sent: Monday, February 28, 2005 1:34 PM
    To: Ian Bird; Simone Campana; alessandro.desalvo@roma1.infn.it; Miguel
    De Oliveira Branco; cristina.vistoli@cnaf.infn.it; Alessandro Cavalli
    Cc: Gilbert Poulard; Dario Barberis
    Subject: ATLAS production and requests for operation

    Dear Ian,
    here are some points for the attentions of the operation group:

    For the ATLAS production it is important we are able to distribute efficiently the jobs via RB.

    For using well the raking mechanims we need the sites to pubblish a
    parameter VO specific, specifying the max number of CPU that ATLAS could
    occupy.

    When an ATLAS dedicated queue exists, as it should always be the case, we think that
    the PBS parameter max_running that reflects itself in the GlueCEPolicyMaxRunningJobs
    information, would be a good choice. I suppose a similar parameter exists for LSF...
    would you agree asking the sites, starting from CERN, to pubblish a correct value in this field?
    We are now installing ATLAS sw on SLC3 WN too; our installation however
    assumes all the WN belonging to a single CE have the same OS, and would
    not work if it is not the case.

    Could you please all the sites are configured in this way?

    Hoping for positive input, and thanks for your attention
    Laura
     

    5. The UK participants reported that the Birmingham SE is full. It isn't yet clear to which VO belong the data. As a general remark on the reporting template, it is unfair to report a site as "non functional" if the reason for that is external to the site.

    6. The CESGA site in Spain (SWE) had a non-responding BDII during its upgrade. This was a case of a BDII being used as GIIS. A bug was submitted in Savannah on this. 
    Gonzalo relayed the problem submitted to the rollout list by Mario David on RLS service set-up. This was followed up by email with Maria, James Casey and Laurence Field.

    7. The Wiki pages are a good information source for installation problems at a site. The ROC managers can register and edit existing pages with up-to-date information. There is a page version control but no quality control or moderation at the moment.

     

    Security report (Ian Neilson absent):
    - No report.

    Next CIC-on-duty:

    • The CIC-on-duty this week is RAL.
    • The actual work is be done by the ROCs and the follow-up by the CICs. The CIC-on-duty just does the monitoring.
    • The ROC managers should remember that when a problem at a site is solved the site needs to go through a period of quarantine before joining production again. This is mentioned in the Operations' Manual.
    • The CIC site (requires personal certificate loaded) is http://cic.in2p3.fr .
    • Link to the daily Site Status reports.
    • Savannah should be used for the Tasks' hand-over and follow-up. The Operations' Manual should be the basis.
    • When the CICs report also as ROCs, they should do this in separate reports.
    • Sites' status of updates can be checked in the daily reports e.g. http://cern.ch/lcg-testzone-reports/cgi-bin/lastreport.cgi

    Action List:

    (*** ACTION 2004-11-29--1 ***)Vincent Breton to check with the other application managers the "Migration to SLC3 plan" of non-HEP VOs. LHC experiments are, up to now, mid February 2005, still not clear about the status of their software. IN2P3 reported on the 2005-02-21 meeting that the BIOMED VO hasn't yet started. OPEN

    (*** ACTION 2004-12-06--1 ***) Sites should accelerate migration to SLC3, at least on the service nodes due to security considerations. OPEN

    (*** ACTION 2004-12-13--1 ***) Escalation procedures in the Operations' Manual should clarify that sites running outdated LCG2 versions or don't respond to CIC-on-duty action prompt will be disclosed in this meeting. OPEN

    (*** ACTION 2005-01-03--1 ***) Grid Infrastructure Support Section to give generic (functional) DNS aliases to important service nodes, ensuring transparent service changes.
    Conclusion:
    This was discussed after the meeting with Markus Schulz. The network group can't propagate the new host alias as fast as needed when the physical machines change. The solution we adopted (without being a complete fail-over) is to put the RB files on a shared filesystem and bring another machine up, when necessary with the same identity.
    DONE Close at next meeting?

    (*** ACTION 2005-01-10--3 ***) The asian sites' support needs to be formalised. Requested by CERN at the relevant CIC-on-duty report. Assigned to Taiwan, the asian sites' ROC. OPEN

    (*** ACTION 2005-02-07--2 ***) Gilles Mathieu and Markus to put the web page with CA rpms, currently under http://cern.ch/markusw, on the CIC web site. 
    Conclusion:
    Put by Oliver Keeble in http://lcgdeploy.cvs.cern.ch/cgi-bin/lcgdeploy.cgi/lcg2/rpmlist/security-rpm.h
    DONE Close at next meeting

    (*** ACTION 2005-02-07--3 ***) GOC database manager David Kant to add an operational days field, in addition to operational hours, e.g. Operational hours: 0900 - 1700 (GMT), for every site. Sites to correct their timezone in GOC db. ROC managers should do the same, each on their page under http://cern.ch/egee-sa1/ROC-support.htm. 
    Progress:
    This is done so far by DE&CH ROC, NE ROC, SEE ROC and SWE ROC.
    OPEN

    (*** ACTION 2005-02-07--4 ***) John Gordon using some information circulated by Ian Bird to define metrics on site performance. OPEN

    (*** ACTION 2005-02-14--1 ***) Min Tsai (CERN) noticed that the CNAF LCG2 Release shown in the monitoring tools is 2_2_0. This is an editing mistake and it should be corrected by CNAF. 
    CONCLUSION:
    This was not an error. As we have to get away from LCG2 2.2 release, and as the site remains in this release due to LSF problems, CNAF is advised to consult lcg-deployment-support@cern.ch.
    OPEN

    (*** ACTION 2005-02-14--2 ***) David Kant's comment is needed here: ROCs to make sure that sites in their region which are in the GOC database, also appear in the BDII configuration file which is a prerequisite status for a 'certified' site according to the Site Registration Requirements document. 
    Conclusion:
    David Kant aid that a GOC tool is available for the ROC manages with a list of sites in their region and an easy way to change the site status flag to 'certified'.
    OPEN

    (*** ACTION 2005-02-14--3 ***) Nick to clarify the future release strategy. Ian Bird presented different plans at the EGEE review to the ones that Maarten Litmaath published in email. OPEN

    (*** ACTION 2005-02-14--4 ***) Min to add the colour code to the Regions' page as well. 
    DONE but should remain OPEN and be combined with the action on metrics.

    (*** ACTION 2005-02-14--5 ***)Participants requested a historical site view, i.e. one that shows how long a given site has been out last week to appear on the weekly report template. The request came out of discussions about the weekly operations report required by Nick. Philippa and Steve clarified its content:
    Figures for sites with any problems during the week are required and the only way to presently find that information is to click on all the test history links for all the sites. It was considered that maybe having a single page with a flag to say whether a site had been down over a given period would be a useful tool for reporting. This kind of page would also highlight patterns of failure across more than one site maybe hinting at bugs or more widespread issues. 
    On the Site Functional Test results page you can either:
    1. look at a page of all the sites and their latest SFT results.
    2. look at a page of one site and all of the complete history of that site.
    It would be useful if possible/easy to be able to see a summary of history for all sites. A simple suggestion might be that there is an extra column in page number (1) above that shows a counter for the number of consecutive days that a
    site has passed critical test. However, in this way "flapping" sites are hidden. The work on metrics will help to define the granularity of data required for classifying the site performance.
    OPEN

    (*** ACTION 2005-02-21--1 ***) Nick and Ian Bird to define and document in the Operations' Manual the penalty for sites not responding to escalation, e.g. Toronto case of week 2005-02-14. Philippa suggested the introduction of a "last chance information" step in the Escalation steps as described in the Operations Manual, to inform the site it now goes in front of the GDB or EGEE PMB. Nick reminded the ROCs that after the 1st contact from the CIC-on-duty to the site and its ROC, it is up to the ROC to pursue the case.
    OPEN

    (*** ACTION 2005-02-21--2 ***) Steve to ask David Kant to adapt GOC db not to allow sites to turn their monitoring flag off while in production. 
    DONE. Close at the next meeting.

    (*** ACTION 2005-02-21--3 ***) Piotr to ask Judit to exclude 'non functional' sites when building the BDII from GOC db. 
    DONE. Close at the next meeting.

    (*** ACTION 2005-02-21--4 ***) Piotr to make a list of sites that don't allow (long) monitoring jobs to complete and will send it to the ROC managers. 
    Conclusion:
    The list was made but was empty.
    DONE.Close at the next meeting.

    (*** ACTION 2005-02-21--5 ***) A re-incarnation of (*** ACTION 2005-02-07--1 ***): Who will wite a policy and procedure of data migration possibility for a site? OPEN

    (*** ACTION 2005-02-28--1 ***) ROC managers to contact their sites asking to set the value of max_running_jobs to the number of available CPUs instead of 9999 for those cases when there is no limit. This is an ATLAS requst. OPEN

    AOB

    • Trinity College Dublin tries a port to the Fedora OS that fails due to problems with t.he Replica Manager. To be further discussed between Nick and Jeremy.
    • The next regular telephone meeting will take place on Monday March 7th 2005 from 14:00 to 16:00 in conference room 28-R-015.

    Maria Dimou, IT/GD, Grid Infrastructure Services

    There are minutes attached to this event. Show them.