Choose timezone
Your profile timezone:
Deployment team
→
Europe/Zurich
EVO - GridPP Deployment team meeting
EVO - GridPP Deployment team meeting
Description
- This is the weekly DTEAM meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area
- The phone bridge number is +41 22 76 71400. Th phone bridge ID is 344539 with code: 4880
Present
Andrew Elwell
Graeme Stewart
Pete Gronbech
Jeremy Coles
Brian Davies
Raja Nandakumar
Frederic Brochu
Duncan Rand
Steven Burke
Mingchao Ma
Derek Ross
Greig Cowan
Jens Jensen
Dave Colling [part]
Experiment status
----------------------
LHCb - Raja
CCRC08:
going well files into ral, reconstruction OK
using castor
old reconstruction in parallell - staging in off dcache - seems slow - big backlog - talking to Derek and Tim to resolve.
CMS - Dave C
CCRC - going reasonably well - most sites doing well - some sites have minor issues which are being dealt with.
seeing 0.1 to 0.2% failure on jobs so far. (from mail - 28010 jobs over 21 siteas with a total of 71 failures)
larger scale Multiuser analyis due to start early june - only a few people doing analysis just now.
Greig - do you have an update on imperial dCache? - No. Duncan doesn't know either. Acknowledge its flakey - seeing timeouts on gridFTP managed by FTS. bugs open with developers. Mona and Kostas following up.
ATLAS - Graeme
production was going well last week until Sat when UK DDM box locked up. (known bugs) - restored on Sat PM then RAL Oracle failed and FTS locked up. discovered ~10pm at night. then discovered credential delegattion had got corrupted.
Switched from Marios to Kors' certificates for transport within UK - went sorta OK. Major outage fixed out of hours - process now works - good improvement.
one falloout thats affecting *some* T2s with FTS - delegated credentials missing on the channels.
Brunel, ECDF, lancaster, ucl-hep. manchester, oxford. -- sems to be SRMv1 endpoints affected.
Spoken to Matt to switch any transfer involving a space token to use SRM2 - may be that if it uses SRM1 then they havent been passed source space token? ongoing....
(no space token used on GET) may be issue wiuth URL copy channel from 1->2?
Production therefore only at ~50% of UK BUT they got a whole pile of bad tasks (4000 jobs) into Q - expect 3* that failures - should be killed officially by atlas MC coordinator rather than Graeme (just to let sites know)
CCRC - T1-T1 transfers started today - RAL looks good. 98% as dest ~20 errors of src timeouts (SARA / FZK / Taiwan)
RAL as source - Mostly ok apart from CNAF and FZK.
[aside from Derek - They had already started working on the nagios alert before GS's RT ticket hit]
Greig - Gridview not showing any atlas data but it's there in the dashboard. - perhaps some lag?]
... discussion ensued about weaknesses of gridview - esp wrt to dcache parsing
April reliability figures:
Scotgrid poor as we were closing old edinburgh. (also had poor reliability)
ECDF getting better. (last week bad again)
LT2 - QMUL and UCL-Cent pulled figures down.
Brunel poor too - Still missing a member of staff. Henry doing in spare time (as is Duncan as coord)
Northgrid - Liverpool in feb bad, otherwise back up to 93%
Southgrid - Oxford upgraded in April + migrating nodes between sites.
RAL dcache issues and JET - unknown.
-- but Accounting in April = matched scotgrid :-)
Storage Avail plots - http://www.gridpp.ac.uk/wiki/GridPP_storage_availability_monitoring
ANALYSIS WORKSHOP
---------------------------
-- not going ahead but a larger user session at gridpp21 mtg
-- expansion of UIs (say onto laptops etc)
* Rather than full UI on local machine (need an SL4 VM ideally)
* offer UI at sites that users can GSISSH into <<-- recommendatiion
* tarball install available on all group machines in depts. <<-- recommendation
* need CRL / Configs - not trivial task
Most people submit from CERN - more now starting in UK from
Site breakdown - 3 at lancs, how many are needed?
what should we offer - one per site, one per T2 - realistically one (or more) per site depending on install method
for the UI you need a *real* account not a pool a/c - may be issue with site admins?
Team update: -
------------------
Pete G - HEPIX last week. benchmark talks good - quad core AMD chips _can_ perform better than intel but v close. (don't panic if you got intel) - SL update - xfs will go into vanilla SL5 and in contribs on SL4. Security patches for SL309 for consiiderable time (oct 2009/2010?). Oxford work - Older nodes on SL4. New torque server for all 3 CEs. Greig helping with DPM. Need to move 1/2 cluster up to new site.
Andrew - Nagios / WLCG stuff. DPM usage and deliverables for next funding
Duncan @brunel CCRC08, getting CMS sam tests passed. bandwith issues etc.
@imperial - Mona installed tarball - getting atlas jobs running
@rhul - maui issues - working with vendor
@QM - atlas install issues last week - coulnt install as atlassgm.
@ucl-cent - VOMS issues. CE waiting for network ports to be opened
Steven B - Docs / User support.
userguide supposed gto be taken over. Gridpp website maintenance
Been working on glue schema (glue2.0) - public comment from monday
Mingchao - security workshop in oxford last week
Derek R - new CE for atlas (2 more this week one CMS one LHCb) - should GS use ce03? - waiting to watch load.
watching CCRC08
dCache issues - stageoff issues - ongoing
Greig - HEPIX working group - benchmarking f/s and storage tech. Working on LHCb mc data storage at edinburgh + grif. (slight change to computing model). SLC vs SL4? had probs getting jobs running on ECDF. official line was we don't support non-slc distros. [RAJA - who said this? email exchange -- they consider SL / SLC equal] - seems like they were wrong. Manpower forcussed on dirac3 development not dirac2 support. no prodn jobs in Q anyway. hopefully will start prodn soon on dirac3
Graeme - atlas + procurement. Formal kickoff starts tomorrow. Debugging ECDF problems with gpfs. (load on gpfs daemons sucking CPU) - 2 of the 4 sites that had atlas s/w on gpfs space have abandoned it. seems to happen at all points in the job - doesn't make much sense.- will be running some standalone jobs to test on gpfs / local disk. Durham - all OK and quiet.
Jens - updating venezuala CA. new castor info provider now live (docs to follow). busy with docs for allhands, OGF, pma etc etc.
TICKET UPDATES
---------------------
28457 - Jeremy to follow up
33373 - Raja thinks exchange of emails - roberto thinks it can be closed.
32029 - old site. Greig to investigate.
... more next week
AOB
-----
GDB tomorrow - Alessandra volunteered to go 1st representing T2s
Agenda at http://indico.cern.ch/conferenceDisplay.py?confId=20229
Peter Love to attend Dteam as lack of info about say CCRC progress?
-- can attend the daily CCRC specific calls.
-- JC will discuss with alessandra for T2 dissemination.
-- reluctance to turn dteam into tb-support meeting
All Hands - anyone submitting?
(~4 from gridpp)
focus has changed - more about what users can do? some of the accepted ones are more user-specifiic.
hepsysman - meeting mid June at RAL?
(post CCRC and last minute before data starts)
GS and AE were volunteered. Thanks...
There are minutes attached to this event.
Show them.