Go to the Indico Home Page Oldest event Older event Up to category Newer event Newest event Printable version
  • Compact style
  • Indico style
  • Indico style - inline minutes
  • Indico style - numbered
  • Indico style - numbered + minutes
  • Indico Weeks View
    • Switch to the management area of this event
    Login

    WLCG-OSG-EGEE Operations meeting

    → Europe/Zurich
    28-R-15 (CERN conferencing service (joining details below))

    28-R-15

    CERN conferencing service (joining details below)
    Description
    grid-operations-meeting@cern.ch
    Weekly OSG, EGEE, WLCG infrastructure coordination meeting.
    We discuss the weekly running of the production grid infrastructure based on weekly reports from the attendees. The reported issues are discussed, assigned to the relevant teams, followed up and escalated when needed. The meeting is also the forum for the sites to get a summary of the weekly WLCG activities and plans
    Attendees:
  • OSG operations team
  • EGEE operations team
  • EGEE ROC managers
  • WLCG coordination representatives
  • WLCG Tier-1 representatives
  • other site representatives (optional)
  • GGUS representatives
  • VO representatives

    • To dial in to the conference:
    a. Dial +41227676000
    b. Enter access code 0157610

    OR click HERE

    list of actions
    Minutes
      • 16:00 → 16:05
        Feedback on last meeting's minutes 5m
        Minutes
      • 16:05 → 16:30
        EGEE Items 25m
        • <big> Grid-Operator-on-Duty handover </big> 5m
          From ROC CE (backup: ROC SEE) to ROC DECH (backup: ROC SWE)

          Lead team handover
          Tickets:
          Open tickets : 23
          2nd email: 17
          Quarantine: 17
          Site OK: 26
          Close: 20

          Notes:

        • RB time_to_match failures - some tickets affected (delayed) due to insufficient output of the test. We created a new ticket (#17725) for that because we cannot chase sites if tests don't show enough information about the failure. RB-ime_to_match test for some sites just show no output. Few days ago Dave Kant fixed that, so should be ok from now.
        • Backup-team handover:
          The electricity problems at CERN affected everybody, obviously, but the service was restored shortly. Tickets 17612 and 16652 do not look as suspension case to me, that is why I did not issue "suspension to be discussed" e-mails.
  • <big> PPS reports </big> 5m
    PPS reports were not received from these ROCs: AsiaPacific France Italy NorthernEurope Russia SouthEasternEurope SouthWesternEurope
    New gLite CE (ce111) giving access to production SLC4 WNs available at CERN_PPS.
    Volunteer sites wanted in order to install 2/3 LCG RBs in PPS.
    Speaker: Nicholas Thackray (CERN)
  • <big> EGEE issues coming from ROC reports </big> 10m
    Reports were not received from these ROCs: Italy


    1. It is difficult to fix OPS-VO-related problems at a site due to inability to run OPS VO tests. We had a problem with RB time-to-match test which somehow fails for OPS VO. Other VO jobs are passing through without problems. Is that acceptable we ask a sensor developer (who typically belongs to OPS VO) to assist the site in finding the problem (i.e. to execute some simple tests like try to do gridftp-ls against RB)? Even CIC-on-duty person could not help us as they have no access to OPS VO also. (CE ROC)
      ANSWER from SAM team:
      Normally SAM Admin page (accessible from CIC portal) should be used to run OPS VO tests on demand. However there are several sensors that are not covered by SAM Admin and unfortunately RB sensor is one of these. In that case if site admins are unable to find real problems, I think there are 3 solutions:
      1) either RB sensor developers/maintainers (I think it is still Dave Kant) will provide a mechanism for on demand execution with some more debugging information (integration with CIC Portal/SAM Admin)
      2) or we decide to include COD people in OPS VO (which would extend OPS VO quite significantly)
      3) or site admins have no other choice but to contact sensor developer/maintainer directly


    2. Please could people tell what are the latest developements to not only improve, but also fix and transcend the information system design flaws ? All but one errors encountered this week were timeouts on BDIIs : how can the information system be considered as stable ?(France ROC)


    3. A patch for lcg-info-generic has been developed and tested succesfully, in order to keep up with the high load on the CE.
      The patch improves the speed by a factor of about 20 in our case (7seconds -> 0.3 s). It is a drop in replacement, does not change case of DNs, does not need extra rpms.
      In see we''ve been using the new script in a couple of site, we''ve attached the results at
      http://wiki.egee-see.org/index.php?title=SpeedUP_LCG_INFO_GENERIC
      Please advise on how to pass this certification. (SEE ROC)


    4. It would be nice if the CERN power cut SAM failures were not submitted for sites to comment on: e.g.,
      https://lcg-sam.cern.ch:8443/sam/sam.py? funct=TestResult&nodename=svr016.gla.scotgrid.ac.uk&vo=ops&testname=CE- sft-lcg-rm-cp&testtimestamp=1169707232. Can errors that are not site related be filtered out? (UKI ROC) ANSWER: SAM has no input on power cuts at CERN or any other incidents that can occasionally happen. Avoiding dependencies and filtering out errors caused by 3rd party or central services has been planned since quite a long time. This is also a goal of a new TIC group (Tools Improvements for COD) who wants to improve existing tests and provide new ones.
      But in general I think that there is also a problem with ROC Report that is generated automatically and contains any single failures. I think there should be a possibility to record there such "central" failures like CERN power cuts before the report is given to site admins to comment. But it is rather a question to people from CIC Portal and not SAM


  • 16:30 → 16:50
    OSG Items 20m
    Item 1
  • 16:50 → 17:35
    WLCG Items 45m
    • <big><a href="https://twiki.cern.ch/twiki/bin/view/LCG/WeeklyServiceReports">WLCG Service Report</a></big> 15m
    • <big> WLCG Service Commissioning report and upcoming activities </big> 15m
      See new and updated information at https://twiki.cern.ch/twiki/bin/view/LCG/SC4ExperimentPlans
      Speaker: Harry Renshall
      document
  • 17:35 → 17:55
    Review of action items 20m
    list of actions
  • 17:55 → 18:00
    AOB 5m