Deployment team & sites

→ Europe/London
EVO - GridPP Deployment team & sites meeting

EVO - GridPP Deployment team & sites meeting

Jeremy Coles
Description
- This is the biweekly DTEAM & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 77907 with code: 4880. Apologies: Kashif (Pete & Ewan also if temp alarm issue not resolved)

Deployment team & sites meeting – 1st March 2011
Attendees:
Brian Davies
Derek Ross
Jeremy Coles (chair + minutes)
Govind Songara
Stephen Burke
Daniela Bauer
John Bland
Richard Hellier
Graeme Stewart
Peter Grandi
Andrew McNab
Santanu Das
Winnie Lacesso
Raja Nandakumar
Raul Lopes
Stuart Wakefield
Ben Waugh
Andrew Washbrook
Sam Skipsey
Alessandra Forti
Stephen Jones
Kashif Mohammad
Chris Brew
Chris Curtis
Mingchao Ma
Ewan Mac Mahon
Wahid Bhimji
Pete Gronbech
Chris Walker
Rob Harper
Mark Mitchell
Elena Korolkova
Stuart Purdie
 
 
 
Experiment problems/issues(20')
Review of weekly issues by experiment/VO 
 
-LHCb  Cream CEs at Birmingham, Oxford, QMUL and Imperial are "buggy".  From Stuart's EGI ops meeting summary we learn that the "CREAM 1.6.4 is buggy, unsuitable for use with LHCb (and possibly other users), due to a problem with argument parsing.  People using it should upgrade to 1.6.5 urgently."
 
QMUL is now fine.
The problem with 1.6 for LHCb relates to argument passing breaking when passing the certificate DN for LHCb. LHCb has this DN in their JDL.
 
Raja mentioned the status of his action from last week concerned with whether the LHCb pilots could take a signal from the batch system to stop pilots (sites found LHCb jobs made slot management difficult as they frequently filled 48 hr queues right up to the limit by back filling). LHCb does not have any plans to implement a change here any time soon. Jobs are typically less than 24hrs.
 
Sam mentioned the origin of the issue last week was that sites had set up 48 hr queues as this was requested in the LHCb VO card. If this was changed and only 24hr queues were required this would help. Action: Raja to contact Roberto to update the VO ID card.
 
Kashif confirmed that Oxford is on 1.6.4 and will update.
 
 
 - CMS
 
Stuart mentioned a few issues with IC. First dCache problems – not all services returned cleanly from the power down last week. A headnode out of memory problem also had to be fixed. Also an SGE-CREAM plugin is causing about 50% of jobs to be aborted.
 
 
 
 - ATLAS
 
Graeme mentioned a problem with Lancaster’s CREAM CE. Jobs run but do not stay running – it is being followed up. Elena was overseeing things last week and continued to report. Many job failures seen yesterday. QMUL had the wrong certificate installed but now fine (Chris explained this was due to mixing up the signature and key when updating the SE). Lancaster had a corrupted release and had to reinstall (version 15.6.3-10). During the night IC lost 3000 jobs and the site is now marked offline. The related GGUS ticket is 68119. The headnode had run out of memory…
 
There was a discussion about running YAIM on CREAM after observations at Oxford indicated jobs were lost on the WN when this process was used. It removes the sandbox so when the job completes and it tries to send the sandbox there is no directory and the job fails. There was no obvious conclusion as to why this should happen. Other sites were asked if they ran YAIM on a running CE and what was the best practice.
 
Alessandra said they did this yesterday with no jobs lost. Sam indicated Glasgow also did this but advised against any automatic updates. Glasgow do not drain the CE just to run YAIM and run things manually.
 
 
 - Other
 
Peter mentioned that at Durham he spotted a problem on the GridPP approved VOs page related to the entire dteam VO. This is now fixed. He also mentioned a problem with nmr enablement and bizarre logs being generated on the Vo server.
 
 
- Experiment blacklisted sites  - Experiment known events affecting job slot requirements  - Site performance issues 
 
No additional discussion required on these items
 
Meetings & updates(15')
- ROD team issues to review
 
Andrew mentioned things were good and he is on shift this week. From last week, there were a few problems at different sites but all were fixed in time. Durham had some problem in publishing correct value for GlueSAPath and some other issues  but finally  fixed. For reference https://gus.fzk.de/ws/ticket_info.php?ticket=67843.
 
- Tier-1 update 
 
Derek said that ATLAS had been experiencing slow transfers. The T1 is now fully Quattorised. FTS has been updated. CASTOR updates include moving the name server to 2.1.10 are scheduled. Electrical work will be undertaken to address noise issues on the UPS – one server will be moved over to evaluate the changes. CASTOR 2.1.10 instances will be upgraded. The site is at risk during a top-BDII update. A new FTS “group” feature has been enabled following the update.
 
Jeremy asked about the progress with full node reservation at the T1. Derek replied that this will be covered in an upcoming blog article.
 
Brian noted that recent slow FTS transfers required some iperf tests to be carried out. Pete asked if results could be seen already from GridMon. There was then a brief discussion about GridMon which is being moved from Daresbury to Glasgow (the hardware has arrived).
 
 
- Operational security
 
Mingchao mentioned an issue currently under discussion on the list concerned with CREAM. The matter has been escalated to the vulnerability group. There was a problem notified to sites on 11th February and a list of IPs to check was circulated but no responses were received despite many US sites being compromised. Jeremy suggested the request be resent as sites may have missed the Friday afternoon message. Action: Mingchao to resend IP check request.
 
 
-- Checking results at https://pakiti.egi.eu  
 
One site appeared recently but the problem has been fixed.
 
 
-   Upcoming meetings -- The next GDB is in Lyon on 9th March: http://indico.cern.ch/conferenceDisplay.py?confId=126130. Topics to be covered:  CREAM - all sites should now have at least one instance of glexec (& MUPJ) - all sites should now be looking at this (with ARGUS). Deadline for T2s is the end of June.  LHCONE - T2/country networking  File system stress tesing  Experiment operations - any matters to raise?  
 
On the topic of glexec Daniela mentioned an issue for IC in that ARGUS will not work in its present form and Dave C already indicated the site may not install it.
Pete commented that the deadline was just for glexec not a requirement to have ARGUS (gums, SCAS or something else can be used). There was then a discussion about ARGUS’s role – that it only gets the user ID for each user and it is only used by glexec. Only pilot roles are allowed to use it. Pilot jobs use it to switch to the original ID of the submitter. Kashif added that reconfiguration was not necessary to install ARGUS – it gets the mapping to local pool accounts. Ewan added that the credential is mapped to a pool such that another job from the same submitter gets the same account.
It was pointed out that if different pool accounts were used by different CEs then a problem may arise. IC and Birmingham seem to be in this position. Perhaps two ARGUS servers could be used!? Disjoint accounts are used by the different CEs to avoid conflicts. Daniela was concerned about glexec working with dCache.
Kashif added that one ARGUS server per CE is not possible with the current setups. Glexec will not know which CE is submitting. One option would be to wrap the glexec call. Ewan suggested that you could pick just one set of pool accounts for use with ARGUS.
Graeme mentioned how this works for ATLAS. Glexec is used for executing user code. Error handling and staging is done by the pilot. No file handling is done by the mapped accounts.
Birmingham and IC and ECDF use the tarball install. Ewan wondered whether an rpm install could be tarballed up and said this is a wide problem for sites where root access is not permitted. Binary suid with root is needed. Pete had asked about tarball installs at the last GDB and the developer replied that it was not yet considered but now noted it as a requirement.
Action: Jeremy to check on tarball install status
 
ATLAS sonar matrix(10') Results Matrixdf filenknown type file 
Graeme gave a summary of the results and tests. He pointed out that sonar is useful but difficult to get the matrix plots from directly so he had got programmatic access to the data in order to produce the tables shown. In the pdf the top table shows the UK sites and the rows give the source sites. Note that the units should be MB/s not GB/s! The colours go from green to red. <3 is red. >10 is green. Reds through yellows and orange give an indication of the relative position of the results (intense yellow being better). The average of about 6MB/s for UK T2s to T1s is lower than seen for remote site links (see second table). Triumph and Taiwan T1s show general problems and should perhaps not be considered for these comparisons. Also note that the rates shown are for transfers via the FTS. To improve rates some tuning (of for example the disk servers) is required. The results do not suggest anything is network limited.
 
The T2Ds for comparison show better rates. All except for Napoli have 10GB external connectivity.
 
Brian: one issue is that we don’t control all the factors where red is seen. For example we don’t control the FTS of other T1s. And before we start tuning we need to be aware that ATLAS is also interested in overall throughput not just bare transfer rates.
 
Jeremy asked about CMS experiences with transfers from sites that are common to ATLAS and CMS – which is only RALPP! Chris B answered that he/CMS had not looked at remote T1 rates for a while now and the CMS requirements were anyway quite low and of order 10MB/s. Brian thought the findings were similar to ATLAS with perhaps good peak speeds. He pointed out that the sonar tests only use small files and he intended to expand the exercise and so we should return to a general discussion of the more general results in 1-2 weeks time. Graeme indicated that he was already in discussion with some good external sites such as NIKHEF with whom we could work more closely on expanded tests – these being typical sites where good rates should be expected. The general requirement is going to be data exchange between the bigger T2s.
 
Sam noted that the Glasgow asymmetry in results for in and outbound transfer rates may now have been solved. Since an opposite asymmetry was noted for Manchester he intended to circulate some details of what they did in Glasgow (config changes) to improve things.
 
 
WLCG Tier-2 availability, accounting, publishing and misc results(05')
- February results not yet released. 
 
-   Accounting: R-GMA is to end service today (https://operations-portal.egi.eu/broadcast/archive/id/328). Still 2 UK sites, UCL and JET using it to publish accounts! There is no further extension.
 
Ben said that UCL were now working on gLite-APEL and this should be ready in the coming days.
 
 
 
  Publishing gaps/stopped for: RHUL; Glasgow; Cambridge  https://gus.fzk.de/ws/ticket_info.php?ticket=67954 The following site do not publish the mandatory GlueSATotalOnlineSize attributed:  UKI-LT2-QMUL UKI-SOUTHGRID-BRIS-HEP  https://gus.fzk.de/ws/ticket_info.php?ticket=67945 The following sites do not publish the WLCG mandatory attributed GlueSAReservedOnlineSize  UKI-LT2-QMUL UKI-SOUTHGRID-BRIS-HEP  https://gus.fzk.de/ws/ticket_info.php?ticket=67935 The following WLCG sites do not publish the mandatory attribute: GlueSACapability:InstalledOnlineCapacity= value  UKI-LT2-Brunel UKI-LT2-QMUL UKI-LT2-UCL-HEP UKI-NORTHGRID-SHEF-HEP UKI-SCOTGRID-DURHAM UKI-SCOTGRID-ECDF UKI-SCOTGRID-GLASGOW UKI-SOUTHGRID-BRIS-HEP
 
 
 
 
 
 
 
EVO chat:
[10:56:05] Brian Davies joined
[10:56:54] Govind Songara joined
[10:57:27] Stephen Burke joined
[10:57:28] Daniela Bauer left
[10:57:44] John Bland joined
[10:57:45] Richard Hellier joined
[10:58:30] Graeme Stewart joined
[10:59:20] Peter Grandi joined
[10:59:48] Andrew McNab joined
[10:59:53] Santanu Das joined
[11:00:16] Daniela Bauer joined
[11:00:18] Winnie Lacesso joined
[11:00:39] Raja Nandakumar joined
[11:01:18] raul lopes joined
[11:01:32] Stuart Wakefield joined
[11:02:04] Ben Waugh joined
[11:02:34] Sam Skipsey joined
[11:02:52] raul lopes left
[11:03:51] Alessandra Forti joined
[11:04:35] Stephen Jones Is only CREAM 1.6.4 problematic, or all CREAMs = 1.6.4?
[11:04:51] Wahid Bhimji joined
[11:05:17] Stephen Jones Sorry, is is all CREAM less than or equal to 1.6.4
[11:05:43] Jeremy Coles I am taking minutes.
[11:06:45] Chris Curtis joined
[11:07:21] Mohammad kashif joined
[11:10:04] Chris Brew joined
[11:10:23] Skype Bridge joined
[11:10:51] Ewan Mac Mahon joined
[11:11:37] Chris Curtis Not sure whats wrong with audio, sorry
[11:12:02] Chris Curtis I don't know what the mapping is between Cream release numbers and yum versions
[11:12:14] Raja Nandakumar And off I go - bye bye ...
[11:12:16] Raja Nandakumar left
[11:12:24] Chris Curtis If we're out of date I'll upgrade, no problems
[11:12:56] Sam Skipsey Chris : quote from Stuart's email to dteam:
[11:12:59] Sam Skipsey This means that glite-CREAM-3.2.9-1 is buggy, and anyone using it should upgrade to glite-CREAM-3.2.10-0. 3.2.9-1 was released on the 9th of Feb.)
[11:13:05] Mingchao Ma joined
[11:13:51] Chris Curtis Ah, we have 3.2.9-1 - I'll upgrade.
[11:15:44] Pete Gronbech joined
[11:16:21] Christopher Walker joined
[11:17:07] Alessandra Forti problem with dccp
[11:21:08] Chris Curtis I'm running yaim right now - guess we'll find out in about 3 minutes!
[11:21:59] Mark Mitchell joined
[11:22:47] Mark Mitchell sorry I am late Jeremy got caught in another meeting
[11:23:11] Stephen Jones Ideally, we would drain before YAIM, but that is not practical due to the downtime needed. There is a risk.
[11:24:31] Elena Korolkova I have asked to submit test jobs to IC (for ATLAS)
[11:24:56] Elena Korolkova Daniela, please update the ticket. Thanks
[11:25:11] Elena Korolkova https://gus.fzk.de/ws/ticket_info.php?ticket=68119
[11:26:37] Rob Harper joined
[11:28:05] Peter Grandi log entry that seems to cause problems with "enmr.eu" and a couple others: https://voms.cnaf.infn.it:8443/voms/planck/Role=SofwareManager/services/VOMSCompatibility?method=getGridmapUsers
[11:28:50] Peter Grandi the error I get is from 'edg-mkgridmap' and is "Not found".
[11:29:03] Christopher Walker QMUL runs yaim on running machines. I've been assuming it didn't cause problems. It isn't practicable with the current amount of time I have to do differently and apply timely updates.
[11:29:33] Andrew Washbrook joined
[11:29:55] Sam Skipsey Peter: is it doing it for the "special" user mappings (SGM, PRD) or for the VO in general?
[11:30:14] Sam Skipsey If it's just for special roles, it can just be that the VO doesn't define them.
[11:30:47] Sam Skipsey (So, the line you posted is just for the SGM role mappings for planck. Presumably they don't have SGM role mappings...)
[11:31:40] Chris Curtis Ok, yaim finished running on CreamCE epgr05.ph.bham.ac.uk (which has just been upgraded to 3.2.10) and my running test jobs are all still intact. I'll check again once they "complete" - I guess they could run to completion and then fail to return output
[11:31:44] Peter Grandi it is doing it for the special roles, and they are complaining. It is not just for planck, it is a few others on the same server. Similar also for 'dzero' at FNAL.
[11:31:45] Christopher Walker Can you give a link to gridmon please
[11:32:54] Pete Gronbech http://gridmon3.dl.ac.uk/gridmon/graph.html
[11:33:08] Sam Skipsey Peter, I'd not worry about special roles being not present. It just means that they don't require them (you could remove the relevant entries from the mkgridmap config if they annoy you.)
[11:34:04] Peter Grandi Sam I am not worrying too much bu they keep complaining: https://gus.fzk.de/ws/ticket_info.php?ticket=67164
[11:36:35] Sam Skipsey Hm. Have you tried the configuration that Marco suggested? You run the edg-mkgridmap script manually (it's just a cron job).
[11:38:26] Sam Skipsey (Let's take this off the text panel here, and to email, Peter)
[11:42:11] Alessandra Forti different uid
[11:42:25] Peter Grandi yes thanks sam. I have just done a couple of updates to the GGUS ticket.
[11:44:34] Alessandra Forti I can hear echos
[11:47:36] Stephen Jones There seem to be quite a few unresolved issues with this proposal. Does anyone have an overview of how this works in all cases?
[11:49:07] Wahid Bhimji We dont have that problem - but I am sure we will have other issues with this
[11:49:14] Wahid Bhimji well we have / need tarball
[11:49:18] Wahid Bhimji ECDF that is
[11:49:37] Christopher Walker QMUL uses the tarball too
[11:49:44] Christopher Walker We could install an rpm
[11:50:03] Peter Grandi I have to check our setup. I think we have a single set of grid-to-posix accounts, but will have to double check.
[11:50:48] Andrew Washbrook indeed!
[11:51:50] John Bland liverpool have the same problem on shared clusters, we may have a big job getting central admins to agree to glexec
[11:55:26] Ewan Mac Mahon These glExec objections have been raised (loudly) over a period of what must now be years. IIRC the end result was that some people would support it, some wouldn't and anyone with a workload that required it would be restricting themselves to running only on supporting sites, and they were OK with that.
[11:55:45] Alessandra Forti production doesn't require glexec.
[11:55:55] Duncan Rand joined
[11:57:52] Alessandra Forti meaning: in case they refuse completely production can still run
[12:07:53] Govind Songara i have fixed it now..
[12:08:36] Duncan Rand sounds like a storm issue
[12:09:45] Skype Bridge left
[12:09:54] Wahid Bhimji 68095
[12:10:04] Peter Grandi my capacity ticket is 68094
[12:10:15] Wahid Bhimji possibly
[12:11:07] Jeremy Coles Sorry meeting is over running! Will be another 5 mins.
[12:12:07] Peter Grandi I have a nice ldapsearch output in 68094...
[12:12:31] Peter Grandi I do suspect iot is a middleware issue, probably some GIP plug8ins/script needs to be extended
[12:16:38] Alessandra Forti manchester is going ahead next week
[12:17:12] Alessandra Forti it will make easier to concentrate on one ce type
[12:17:53] Elena Korolkova left
[12:19:10] Wahid Bhimji left
[12:20:35] Chris Brew left
[12:21:58] Brian Davies left
[12:21:58] Mark Mitchell left
[12:22:00] Richard Hellier left
[12:22:01] Peter Grandi left
[12:22:02] Mohammad kashif left
[12:22:02] Andrew McNab left
[12:22:03] Derek Ross left
[12:22:03] John Bland left
[12:22:03] Andrew Washbrook left
[12:22:05] Winnie Lacesso left
[12:22:05] Daniela Bauer left
[12:22:06] Ben Waugh left
[12:22:06] Mingchao Ma left
[12:22:06] Sam Skipsey left
[12:22:06] Matthew Doidge left
[12:22:07] Graeme Stewart left
[12:22:08] Ewan Mac Mahon left
[12:22:08] Govind Songara left
[12:22:08] Rob Harper left
[12:22:10] Santanu Das left
[12:22:13] Stephen Jones left
[12:22:15] Chris Curtis left
[12:22:16] Stuart Purdie left
[12:22:18] Christopher Walker left
[12:22:23] Stuart Wakefield left--
 
 
There are minutes attached to this event. Show them.
    • 11:00 → 11:20
      Experiment problems/issues 20m
      Review of weekly issues by experiment/VO - LHCb Cream CEs at Birmingham, Oxford, QMUL and Imperial are "buggy". From Stuart's EGI ops meeting summary "CREAM 1.6.4 is buggy, unsuitable for use with LHCb (and possibly other users), due to a problem with arguement parsing. People using it should upgrade to 1.6.5 urgently." - CMS - ATLAS - Other - Experiment blacklisted sites - Experiment known events affecting job slot requirements - Site performance issues
    • 11:20 → 11:35
      Meetings & updates 15m
      - ROD team issues to review - Tier-1 update Operational security -- Checking results at https://pakiti.egi.eu - Upcoming meetings -- The next GDB is in Lyon on 9th March: http://indico.cern.ch/conferenceDisplay.py?confId=126130. Topics to be covered: CREAM - all sites should now have at least one instance glexec (& MUPJ) - all sites should now be looking at this (with ARGUS). Deadline for T2s is the end of June. LHCONE - T2/country networking File system stress tesing Experiment operations - any matters to raise?
    • 11:35 → 11:45
      ATLAS sonar matrix 10m
      - Comments and discussion based on the results circulated recently by Graeme. - Experiment & GridPP plans for further testing and tuning.
      Results Matrix
    • 11:45 → 11:50
      WLCG Tier-2 availability, accounting, publishing and misc results 5m
      - February results not yet released. - Accounting: R-GMA is to end service today (https://operations-portal.egi.eu/broadcast/archive/id/328). Still 2 UK sites, UCL and JET using it to publish accounts! There is no further extension. Publishing gaps/stopped for: RHUL; Glasgow; Cambridge https://gus.fzk.de/ws/ticket_info.php?ticket=67954 The following site do not publish the mandatory GlueSATotalOnlineSize attributed: UKI-LT2-QMUL UKI-SOUTHGRID-BRIS-HEP https://gus.fzk.de/ws/ticket_info.php?ticket=67945 The following sites do not publish the WLCG mandatory attributed GlueSAReservedOnlineSize UKI-LT2-QMUL UKI-SOUTHGRID-BRIS-HEP https://gus.fzk.de/ws/ticket_info.php?ticket=67935 The following WLCG sites do not publish the mandatory attribute: GlueSACapability:InstalledOnlineCapacity= value UKI-LT2-Brunel UKI-LT2-QMUL UKI-LT2-UCL-HEP UKI-NORTHGRID-SHEF-HEP UKI-SCOTGRID-DURHAM UKI-SCOTGRID-ECDF UKI-SCOTGRID-GLASGOW UKI-SOUTHGRID-BRIS-HEP
    • 11:50 → 12:00
      Site installs/experience with 10m
      CREAM - note Glasgow problem after moving to all CREAM http://scotgrid.blogspot.com/2011/02/ce-is-dead-long-live-ce-nos-paenitet.html. egi.eu have prepared this: "information page about the CREAM-CE software and about the transfer of applications from LCG-CE to CREAM-CE: https://wiki.egi.eu/wiki/FAQ:_lcg-ce_to_cream-ce" ARGUS - SouthGrid update here http://southgrid.blogspot.com/2011/02/going-through-argus-valley.html. glexec - ?
    • 12:00 → 12:05
      AOB 5m
      - Richard "GGUS tkt on the information system"