WLCG-OSG-EGEE Operations meeting

→ Europe/Zurich
28-R-15 (CERN conferencing service (joining details below))

28-R-15

CERN conferencing service (joining details below)

Description
grid-operations-meeting@cern.ch
Weekly OSG, EGEE, WLCG infrastructure coordination meeting.
We discuss the weekly running of the production grid infrastructure based on weekly reports from the attendees. The reported issues are discussed, assigned to the relevant teams, followed up and escalated when needed. The meeting is also the forum for the sites to get a summary of the weekly WLCG activities and plans
Attendees:
  • OSG operations team
  • EGEE operations team
  • EGEE ROC managers
  • WLCG coordination representatives
  • WLCG Tier-1 representatives
  • other site representatives (optional)
  • GGUS representatives
  • VO representatives
  • To dial in to the conference:
    a. Dial +41227676000
    b. Enter access code 0157610

    OR click HERE

    list of actions
    minutes
      • 16:00 → 16:05
        Feedback on last meeting's minutes 5m
        minutes
      • 16:05 → 16:30
        EGEE Items 25m
        • <big> Grid-Operator-on-Duty handover </big> 5m
          From ROC SEE (backup: ROC CE) to ROC France (backup: ROC Russia)

          Tickets:
          New tickets :
          2nd email:
          Qurantine:

          Notes:

        • There were troubles in the SAM jobs submission on Monday. These troubles were quickly resolved for the production sites, but they were not resolved for the PPS sites, and it took quite some time to have the corresponding GGUS ticked resolved.
        • Some SAM tests fail with JS or RM due to "timeout executing..." It is not clear whether this is a site problem or not, and such failures make difficult to create or update tickets.
        • Lots of time between consecutive SAM tests for the same site - from 4 hours to days for some sites. This problem was partly resolved later in the week. It could be that the sites that have large intervals between tests have not enabled priority for ops VO?
        • On Thursday afternoon there was a problem with GGUS, that prevented creation of tickets.
  • <big>yaim-3.0.1 preview for extended testing available </big> 10m
    Dear all,
    we would like to announce you that the preview version of YAIM 3.0.1 is available for extended testing. We make it available to give sites an early opportunity to give their feedback.
    We will particularly appreciate the testing of new functionality (DNS-VO configuration, hierarchical config, VOMS configs), as well as any general feedback.
    !!! Since it is a preview release please don't install it on your production system !!!
    The package can be downloaded from /afs/cern.ch/project/gd/www/robert/glite-yaim-3.0.1- 6.noarch.rpm or http://project.web.cern.ch/project/gd/www/robert/glite-yaim- 3.0.1-6.noarch.rpm
    We keep the documentation on a WIKI page: https://uimon.cern.ch/twiki/bin/view/LCG/YaimGuide301 This page documents the new features implemented in YAIM and also the known issues. We will also add and update the section with troubleshooting information in order to facilitate this testing.
    All your questions and feedback send please to: robert.harakaly@cern.ch
    Speaker: Robert Harakaly (CERN)
  • <big>lcg-maganeVOTag </big> 5m
    lcg-maganeVOTag is in LCG/EGEE middleware (so far the only tool to publish tags). The current version is bugged. The patch for this bug is under certification at the moment but it looks like there are issues which the certifier is sorting out together with the developer.
    There is also another tool, which is about to become LCG/EGEE middleware as well, which is lcg-tags. This tool has just been flagged for certification. The certification process could take more or less time depending on the issues encountered.
    In general, once a tool is certified, it is entered in the update repository. This happens not instantaneously, but once per week.
    After that the question is deployment and how long does it take to have every site upgrading.
    IIt will take some time to have the tool (one of the two) deployed on a reasonable number of sites. For the moment, it would be convenient for Atlas to ship lcg-tags with the installation job and use it from the WN. The only dependencies on externals are grid-proxy-info and globus- url-copy (both from the VDT stuff). The perl modules used are standard perl and should exist in all linux distributions.
  • <big> Maintenance of external packages </big> 15m
  • <big> Update on SLC4 migration </big> 5m
    Speaker: Alberto Di Meglio
  • <big> EGEE issues coming from ROC reports </big> 20m
    Reports were not received from these ROCs: UKI (apologies sent)
    Reports were not received from these non-HEP VOs: VO 1

    1. The SAM failures included in the site reports could be filtered so they are more useful, e.g. to show only test failures that happen more than 3 consecutive times (Nb Days=0 & Nb Hours=3); it does not make any sense to analyze failures that happen randomly or less often than that, and it leads to site reports full of failure that are difficult to trace(CERN-PROD, CERN)


    2. what is a proper portal for monitoring?
      https://lcg-sam.cern.ch:8443/sam/sam.py
      https://lcg-sft.cern.ch/sft/lastreport.cgi
      There two have different result for the same VO (ops), the same CE at the same time.(Russia)


    3. There are problems with the calculation of unavailability hours for SFT failures in the CIC reporting tool.
      Example: SEE ROC site AEGIS01-PHY-SCL and its glite-CE g02.phy.bg.ac.yu had SAM test failure on 2006-11-29 19:08:57, and the next successful SAM test was on 2006-11-29 21:07:15, as can be seen from SAM.
      However, CIC daily report for 2006-11-29 lists this failure to last for 4 hours!
      GGUS ticket submitted on this issue: https://gus.fzk.de/pages/ticket_details.php?ticket=16241 (SEE)


  • 16:30 → 16:50
    OSG Items 20m
    No items for discussion.
  • 16:50 → 17:35
    WLCG Items 45m
    • <big><a href="https://twiki.cern.ch/twiki/bin/view/LCG/WeeklyServiceReports">WLCG Service Report</a></big> 15m
    • <big> WLCG Service Commissioning report and upcoming activities </big> 15m
      Speaker: Harry Renshall
      document
    • From TRIUMF: FTS configuration still problematic. Dynamically created services.xml seems crazy. Not all sites in bdii at the same time. Why not maintain one centrally and distribute that?

    • From Gridka: Since two days dteam/ops transfers from CERN to FZK are timing out for unknown reason. Transfers from CERN for other VO''''s as well as transfers from other centers are OK. Of course this is reflected in the SFT pages.

    • From Gridka: Announcement Scheduled Downtime: The pre-planned upgrade to dcache version 1.7.0 has been postponed to 12/12/06 (!).
    Tier-1 reports
  • 17:35 → 17:55
    Review of action items 20m
    list of actions
  • 17:55 → 18:00
    AOB 5m
    Reminder of VOMRS intervention going on now:
    The lcg-voms.cern.ch vomrs service will be unavailable for, we hope, a maximum period of 2 hours, on December 4th 2006 @16hrs CET
    Reason: Upgrade to vomrs-1.3.0.
    Contents of this Release in
    https://twiki.cern.ch/twiki//bin/view/LCG/VomrsUpd ateLog
    Users obtaining voms-proxies will not be affected by the interruption. Site admins, users, Representatives and VO Admins trying to use the web interface or build the gridmap file are kindly asked to be patient.
    This applies to VOname = ALICE, ATLAS, CMS, LHCb, DTEAM, OPS, Sixt, Unosat, Geant4