UKI Monthly Operations Meeting (TB-SUPPORT)

→ Europe/Zurich
VRVS - Moon virtual room

VRVS - Moon virtual room

Description
Monthly review and discussion meeting for those involved with UKI EGEE/WCG deployment and operations.
UKI Monthly meeting (28/06/2007) Duncan Rand Philippa Strange stephen mcallister Derek Ross Peter Gronbech Jeremy Coles Olivier van der Aa Gianfranco Sciacca Christopher Brew Brian Davies Yves Coppens Greig Cowan Santanu Das Matt Doidge Winnie Lacesso Mona Aggarwal Alessandra Forti Accounting problem in London. Mainly UCL-CENTRAL Birmingham: Problem with the config file on the mon box. Trying to solve them. JC: The view I am getting now is completly different than yesterday. The agenda has been corrected. PG: Birmingham not stable. YC, problem with the monbox. Steve Loyd tests: OV: QMUL is installing sge. Will need help to install their ce. Knowledge at imperial. JC: UCL-HEP something broke two days ago. JC: Sheffield ? GC: in downtime PG: Downtime in oxford and now stuck jobs. Problem copying sandboxes seen at Birmingham. More and more often. Have seen similar problems elsewhere. At oxford it was cern rb and at also at ral. JC: Action open CB: Problem at the user point of view ? did you submit a ticket. I tend to raise tickets internally. Strange problem for the moment where rb02 is not accepting my proxy. DR: In downtime at the moment (rb02). CB: Who is the expert ? OV: Marteen Lindmaath JC: The database can get to full. SAM tests: Lancaster: Having problem with dCache. Problem with gPlazma. JC: Was this an existing installation. Lancaster: one change broke everything. JC: Are you still looking at DPM. Lancaster: Not had time. Tickets: Open tickets (recent). At T1 long tickets (1week). Derek: Mainly done but not being closed. JC: We should close more quickly. Mainly waiting for responses. There is one for PG PG: Will have a look. Other concerns: PG: What is the current position with DPM ? GC: Cambridge 1-5-10 which was well behing the production release. 1-6-4 came (but broken) Cambridge having problem passing sam tests. Wanted to upgrade to the last release, during the upgrade to the 1-6-4 broke things. Required fair amount of work to fix the db. Now we know what to do to go from 1-5-10 to 1-6-4. GC: We can start to address to upgrade problem at each sites. Santanu: They where some table that where missing. Everyone: Yaim changes important. CB: updates are very frequent difficult to keep up Current experiment/VO activities and issues LHCB: Please publish your install system correctly. JC: MINOS has been changed to minos.vo.gridpp.ac.uk to avoid future conflicts with Fermilab VO JC: Quite a lot of people willing to get GridPP running. Please more sites install the GridPP one. JC: One particular concern is that lhcb could not run everywhere. OV: Sam tests for LHCB not frequent enough. Should be automatic. JC: I'll find out. CB: LHCB seems to have problem with their install but it seems a problem with them. Middleware upgrades and move to SL4 JC: Wn is availaible. UI is in the pps. JC: Which sites are upgrading and when ? PG: Are we going to have sl3 type jobs which fail ? PG: Is DPM on SL4 supported ? GC: Works. Does require a couple of additional sl3 rpms ? There is one being tested now which does not require those additional packages from sl3. CB: Don't know if we will do a different ce our a subcluster. DR: LHCB has asked to have a different CE for SL4. OV: Tags will appear in both subcluster CB: Will go for a different CE. AF: Atlas is only trying now... AF: For release 13 is should work on SL4. CB: Will do a big bang upgrade. JC: What do you plan to do to SL4. AF: I am going to wait. Don't want to cope because of bugs. JC: Will wait for tests at RAL and PPD. DR: Running sam tests for the moment. JC: New Yaim on pps now YC: The YAIM 3-1-1 seems ok. Some minor problem with voview. YC: I can explain on the GridPP wiki about the different options for accounts. CB: I basically catching up with the last YAIM. GC: Difficult to understand the versions of YAIM. AF: 3-1-1 is only on the PPS. CB: Did anyone document how to enable a dns type vo. JC: Send a link for the minutes. Helpdesk workflows PS: Presentation of the slides. Presented Stephen McAllister, Support Centre Manager NGS JC: Agent ? PS: The person to which a ticket is assigned to. PS: To close tickets From email: status=closed From footprint web interface PS: Please take more responsibility of tickets. PS: Email us if you have any question about the slides. JC: Do we agree that if after 3 days the user did not respond then we close the ticket. Yes What I referred to was slide 8 of the presentation where it says: PS: " If a solution to a ticket appears to have been delivered for more than three working days with no activity on the ticket and the ticket is not at status=Solved, then the Helpdesk should contact the Agent to ask them to progress the ticket, or grant the Helpdesk permission to close the ticket." PS: So, not a blanket closure on tickets apparently solved for more than 3 days, but a definitive process can be initiated after 3 days which may lead to the closure of the ticket. JC: When is it going to implement. We started before Anne left. JC: Santanu is worried that tickets repeat each other. PS: The problem with the cic on duty tickets is that it is automatic. If it is automatic we cannot check that there is no duplicate. JC: The purpose of the presentation is to let you know about the ticket process. PS: Where we are now is clearly an improvement. AF: I think it has improved quite a lot. Storage accounting JC: Questions about checking that the values in the portal is correct. (ACtion) please each site check that the figures are correct. JC: Sam portal moved to gridview. Will be availaible in three weeks. JC: Moved to Gocdb3 JC: dCache at Imperial ? OV: Now fine GC: recommend to upgrade to the last dCache version. JC: from the review, would be usefull to train sysadm to nagios. AF: Would be useful. JC: Will ask on TB support. PG: Putting it with hepsysman would be a solution. JC: It seems there is general support. AOB ? - GC: Please check the accounting for the storage. ACTIONS: JC: Send a mail to TB-SUPPORT to ask about the interested in organizing a monitoring day. TC: Check the storage accounting ALL: Install the GridPP VO. ?: Check the RB status at RAL. PG: One ticket open for 9 days.
There are minutes attached to this event. Show them.
    • 10:30 → 10:45
      Site status review and issues 15m
      - Accounting (http://www3.egee.cesga.es/gridsite/accounting/CESGA/egee_view.php) -- UKI-LT2-UCL-CENTRAL* -- UKI-NORTHGRID-MAN-HEP All seem to have stopped since yesterday? - Monitoring status SL tests: http://hepwww.ph.qmul.ac.uk/~lloyd/atlas/atest.php -- UKI-LT2-QMUL -- UKI-LT2-UCL-HEP -- UKI-NORTHGRID-SHEF-HEP -- UKI-SOUTHGRID-OX-HEP SAM: -- UKI-NORTHGRID-LANCS-HEP Tickets: -- Several tickets on Tier-1 >1week and marked urgent. Several of these are "waiting for rep". -- Several tickets on Olivier - also "waiting for rep" -- Ticket 1345 against EFDA still in progress after 9 days - Site concerns
    • 10:45 → 10:55
      Current experiment/VO activities and issues 10m
      - ATLAS - CMS # Job processing: Spring07 MC production activities last week focussed on on-going production of extra events in addition to the 35M produced since Spring. In addition, also 21M HLT-re-reco events were produced. MC production for CSA07 wasof ~23M (~20 merged) Minbias GEN-SIM events produced in 4.5 days (that is a rateof ~150M (125M merged) evts/month, also thanks to more reliable sites. # Datatransfers: the \'extended\' LoadTest is converging into CSA07 preparationactivities, and an extended plan to continue debugging transfer links is beingdesigned. - LHCb 1. LHCb would like to remind all sites that *before* moving all their WN resourcesto SLC4 they should provide for a limited time another CE pointing to arestricted part of their farm running the implementation of SL(C)4 they willinstall in production. This will allow VOs to test the actual OS installation atsites before fully migrating. 2. SL(C)4 WNs should be installed with 32-bitcompatibility even if on 64-bit nodes as currently there is no 64-bit nativemiddleware, although VO\'s applications are certified on 64-bit and can actuallyrun in native mode when they do not make use of middleware. 3. Of course theplatform used by the WNs of the CEs (OS, architecture and possibly compiler usedfor the client libraries) should be advertised in a consistent way according toa well defined convention and this should be enforced before the CE is put intoproduction. The current definition of the Glue schema information are still veryvague and difficult to use for matching resources. Many equivalent platformshave different advertisements and the version of the compiler is not published(which is not yet relevant but will in the near future when applications mightmove to gcc 4.1 for example). We remind that the LCG has defined a convention avery long ago for the applications (__) that is usedby the LHC VOs and matching capabilities should be provided based on thisnomenclature. Links TBC - Regional VOs -- MINOS has been changed to minos.vo.gridpp.ac.uk to avoid future conflicts with Fermilab VO - Others -- Many requests to join gridpp VO. Please enable it! -- How far have the regional VOs got in being setup?
    • 10:55 → 11:05
      Middleware upgrades and move to SL4
      • Status and plans to move to SL4 on WNs.

      YAIM update


      YAIM 3.1.1 changes regarding 3.0.x

      • One yaim that will configure both glite 3.0 and glite 3.1 services (get rid of yaim 3.0.1 and yaim 3.1.0)
      • Main feature: Modularity (yaim core + yaim clients + one yaim module per service)
      • Won't support glite CE configuration and 3.0 WMS configuration (this will be done by yaim 3.0.1)
      • Doesn't contain substantial new code (except for 3.1 WMS configuration). It's a merge of yaim 3.0.1-22 + yaim 3.1.0 (for UI and WN) configuration
      • Timeline:
        • 25th to 29th June -> Certification
        • 2nd July -> Ready for PPS

      YAIM 3.0.1-22 content

      • Most remarkable:
        • VOViews for FQANs are not configured
        • Security bug fix with YAIMLOG file
        • Configuration of static mapping of special accounts like sgm/prd
        • Bug fixes from previous obsoleted yaim versions: 17/18/19/21
      • Timeline:
        • 25th June -> Ready for PPS

      More details about the content and timelines of the releases in the YAIM Planning page: https://twiki.cern.ch/twiki/bin/view/LCG/YaimPlanning

    • 11:05 → 11:20
      Helpdesk workflows 15m
      - Proposed workflows for helpdesk processes
      Slides
    • 11:20 → 11:25
      Storage accounting
      • Cross-checking figures
    • 11:25 → 11:30
      AOB 5m
      - SAM portal will be replaced by a new GridView portal in the coming months. GridView will have a new version available in 3 weeks time. - Move to GOCDB3 - SRM problems - recent dCache upgrade(s) and DPM upgrade(s) have problems. - Is there support for a UK tutorial in using monitoring tools (esp. Nagios?) Next meeting: Provisionally 26th July @ 10:30.