Deployment team

Tuesday 7 Oct 2008, 10:00 → 11:30 Europe/London

EVO - GridPP Deployment team meeting

Jeremy Coles

- This is the weekly DTEAM meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +41 22 76 71400. The phone bridge ID is 553553 with code: 4880.

Experiment Roundup LHCb - Raja * Renaming 1.5M files from SRMv1 to v2. Otherwise nothing much * Few small production runs * Don't understand why pilots not being picked up by sites (inc cern) - big issue. Lots of jobs waiting. RB? Dirac? still investigating. (both dirac 3 and 2 affected - they have shared algorithms). CMS - Duncan * Can't send a replenishment supply of jobs. - Poss issue with the submitter? Jobs getting stuck * Brunel were low on disk space (now upgraded) - Also network capping been ongoing for a while. Aircon - ongoing issue. (were running at low capacity) ATLAS - Brian * have been doing reprocessing tests on T1 - staging files back from disk * T1 issues with castor. Under investigation at the moment * T2 activity last week - worked well * Closing SRM endpoints / diskspace thats non-space tokened.-- getting SURLS from sites and will follow up. (he can see most of the DPM sites by setting DPNS_HOST and DPM_HOST variables -- are sites aware?) * Frederic - Space Tokens. -- Don't phase out SRMv1 until all v2 issues are sorted :-) -- WIP. Will mail list when done. * Having to be 'gentle' to CASTOR just now -- it'd be nice to stress test it just now but it's too fragile and breaks. --T1 Memory Requirements * 3GB at Tier1s? - 2G should do 'most' jobs - some need 3G (5% of jobs?) - tagged as a SW bug now but memory requirements always increase. There has been no request (yet) that sites have to buy 3G/core. requirement is 2G now. --Manchester * discussion about job profile / startup at manchester. UKQCD * To be enabled on RAL WMS? * will be on the imperial one. ROC UPDATE (see agenda for URLs) * missasigned ticked (RGMA Failure at QM) - Roc people don't get reminders? -- discussion continued on footprints / GGUS / GOCDB * Check consistency with publishing * New LFC Sam tests. -- Going critical but ignored for availability. VO Cards * Changes based on this should be discussed at WLCG ops meetings Depreciated Services * more gLite3.0 getting old. GDB/Pre-GDB * Will prorities alter with beam delay till april -- LHCb - No change to Tier1 - procure as normal -- Ditto atlas Ticket Roundup * Brians space tokens for ATLAS GGUS / RSS feed * see URL and monitor feed to keep track. GDB / Tier2 issues * see agenda -- Experiment T2 issues -- GRIF example -- Accounting / Monitoring / Storage -- Batch / Middleware ** If any issues Bring em up NOW! * Oxford having MAUI problems - not filling Q and segfaults * Glasgow - soft / hard job limirts not working properly * need a decent 'batch knowledge transfer' Middleware - Is there a need / requirement for newer Supported OS's by middleware - SL5? 64 Bit WNs -- ATLAS will have issues with python (LFC) - but otherwise no major expected hassles Output sandbox for WMS unlimited - potental problem. * use quotaing just now. * real fix planned (in developemnt) Actions -- Postponed till JC has more time (they're on him anyway) GANGA workshop * Can we have one in the UK. -- Get experiment and sysadmins involved. -- Funding? Externals? Network Fitness * Get input from sites -- Status of their neyworks (1 para) and a brief forward look * edi-gla lightpath -- vanished into dark data :-) GridPP VO -- Security challenge 3 Real soon now * please can all sites enable it (Cambridge, UCL-*, IC-HEP, QMUL and GridIreland - this means you!) * which VO / DN shall he use -- perhaps hide it within a 'large' VO GlueSite / BDII * see http://goc.grid.sinica.edu.tw/gocwiki/How_to_publish_my_site_information Meeting closed 11:25

10:00 → 10:20 Experiment problems/issues

Review of weekly issues by experiment/VO - LHCb - CMS -- Brunel issues - ATLAS -- memory limits -- Manchester "sluggish" - Other

10:20 → 10:35 ROC update

ROC update *************** UK issue: " UKI-LT2-QMUL: RGMA failure on mon01.esc.qmul.ac.uk GGUS Ticket #40945 Ticket submitted on 16/09/08 Answered on 04/10/08: site did not receive the ticket => ROC_UKI seems not answering. It seems ROC_UKI does not receive GGUS notifications. This should be fixed." * Comparison of BDII and GOCDB entries for LFC in GSTAT Some sites may have noticed that GSTAT is now comparing LFC entries in the GlueService of the BDII and the nodenames in GOCDB. Example: http://gstat.gridops.org/gstat/CERN-PROD shows prod-lfc-atlas-local.cern.ch as being present in the BDII as a GlueServiceType: lcg-local-file-catalog but in the BDII this host is entered as node type LFC. Assuming it is a local LFC it should be a node type Local-LFC. History: GGUS:38053 While this test does produce in Error in gstat it is not critical in the sense of availability or for the CODs. i.e fix it in your own time. To pass this test the following comparison is made. Service BDII Service Type GOCDB Node Type Central LFC lcg-file-catalog LFC Local LFC lcg-local-file-catalog Local-LFC If you have an LFC which is both a central and local LFC for different VOs then you should enter the node in the GOCDB as both an LFC and a Local-LFC. * Comparison of BDII and GOCDB Entries for bdii_site and bdii_top Services. Similar to the LFC test above another test is also done by gstat to compare the BDII entries for SiteBDII and TopBDII endpoints. The conditions that will pass are. Service GlueServiceType GOCDB Node Type Top BDII bdii_top Top-BDII Site BDII bdii_site Site-BDII History: GGUS:40475 In the case of the top_bdii there is an existing bug that can make this harder to resolve than it should be when you wish to publish a host alias as the service endpoint. BUG:41361. A fix for this trivial bug will pushed forward. * New LFC SAM tests Later this week, two new services will be added to SAM production: LFC_L and LFC_C. The associated tests will be made critical so that history can be viewed in the SAM portal, but they will be ignored for availability calculations, and COD alarms will be supressed. At some stage in the future, and after suitable notifications, they will replace the existing LFC service. The new tests avoid trying to write to read-only LFCs, and include an lfc-ping test on which the others are dependent. VO card --------- VOs wishing to change their needs to be supported by a site should of course use the VO cards as the definitive reference. Any change to the VO card by any VO which would trigger site action should be discussed first at the weekly EGEE/WLCG operations meeting. gLite 3.0 ---------- These services will be obosoleted: * glite-SE_classic * glite-VOBOX * glite-WMS * glite-PX * glite-MON WLCG update ***************** GDB tomorrow. Ticket status *************** https://gus.fzk.de/download/escalationreports/roc/html/20081006_EscalationReport_ROCs.html Open tickets: https://gus.fzk.de/pages/rss/rss_open_uki_sites.php

10:35 → 10:40 T2 issues for the October GDB

10:40 → 10:45 Priorities given LHC delays

- SL5 and 64-bit WNs - Newly procured hardware installation - gLite components?

10:45 → 10:55 New problems

- unlimited Output SandBox for WMS. How do we deal with this until a fix is implemented in the WMS release?

10:55 → 11:05 Actions review

11:05 → 11:10 AOB

- Ganga workshop. Monthly for ATLAS at CERN. If hosting in the UK include the experts (like Ulrich for LHCb). - Network fitness for purpose - A review. Plan is to get sites to write a short summary on their network status highlighting any problems seen in the last year (such as impacts on campus) and a forward look to changes expected in the coming year.