Deployment team

→ Europe/Zurich
EVO - GridPP Deployment team meeting

EVO - GridPP Deployment team meeting

Jeremy Coles
Description
- This is the weekly DTEAM meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area
minutes for 25 Sept 2007 present ====== Jens Jensen, [left early for anoyther meeting] Mingchao Ma, Jeremy Coles (chair) Graeme Stewart, Andrew Elwell Derek Ross Stephen Burke, Frederic Brochu Barney Garret, Dave Newbold, Alessandara Forti Pete Gronbech Apologies - Raja, Greig UK root CA key - update from Jens ===================== 2 copies disappeared (still not sure if this was writing failure / sticks swapped / malicious deliberate) - does not affect escience CA Don't know exactly what happened hence treat as security incident Will rekey root Can't explot this - May affect Java as it doesn't check signing files root CA is in distributiuon to build chain, but it isn't accredited (can't issue certs) Netscape bug *may* affect us, but VO side should be simple as the name isn't changing Timescale - Weeks (delayed slightly following redesign of security policy - new safe perhaps) Overall 'Don't Panic!' - key protected by strong passphrase if it went maliciously investigations ongoing - no certain trail of events yet. Will check if he can use the mailing address of individual grid cert owners in UK Experiment updates ============ LHCb ------- Crashing jobs - Poss envornment issue at T1 causing jobs to crash CMS [Dave Newbold] ---- csa07 - should havce started yesterday [dave newbold] DPM issues in UK - T2 4 sites - imperial, brunel, PPD, bristol no issue with dcache, DPM needs to be fixewd in the cms application software next week - needs to be told wich rfio type is used T1 meeting - scheduled for December 3rd (ish) at RAL T2 meetings = Requested that they attend, not sure if they need to - UK t2s are 'known' and well connected Non-UK T2's estonia, Finland - just coming on stream Other T2 sites - rest of UK will probably run MC - they still have ussues with software installs - They would rather concentrate on the Main 4 sites for now. 'rogue' CMS jobs - they prefer they use the official cms tools - users who can run on the non-std software / data / etc. They use the CMS VO but not the 'preferred' way - mop up other CMS fairshares at sites. CMS resources - Hundreds of TB of datasets - 40TB to PPD. Significant resouucrces at sites - for analysis. MC lower sitye resource requirements barrier for entry to Analysis... Transfer test Phedex etc T1-T2 needs links are classed as decomissioned - comissioned depending on reliability, Real data will only go over comissioned tagged links 50-60 links are presently there - seems to drop out when network bod stops intervening T1 - CASTOR appears to be adequate. High load but robust and stays up! (yay) Now coming to issiues with mysterious FTS errors certificate problem - many services running under personal certificates (renewal etc) They need some system in place VO box services that need myproxy - should it be personal or server certs? (discuss! - SB thnks it should be server - Check with Jens - DN will) debugging FTS / Phedex / CASTOR / SRM - PITA as there are no people who have high level overview of the whole logs - setting FTS retry and backoff off. Seems to be better (if an abuse of FTS design) problen aren't 'network' problems per se, but more likely to be interactions between the middleware layers All RAL traffic going through site Janet firewall - why not through the OPN (Andrew Samsum knows and is aware) the T2 data challenges aren't anywhere near the gigabit levels - 3MB/s is the threshold to stay up = and many aren't. would we be better extending the transfer tests to be longer duration rather than bandwith max challenge, ATLAS ------ UK - overall - No data sets gone to T2's in last 7 days. eg, glasgow has ~4 files from M4 RAL issue? - could be VO box cron job? - VO box will be updated as part of thje new DDM software. ATLAS -- CASTOR - not sure of latest status. Assume OK. The SL4_64 recipies *do* work - on the gridpp wiki - Atlass software jobs running OK biomed -------- horribly inefficent - 50-75% jobs don't start correctly - we don't automatically kill at Glasgow - needs (manpower intensive) sysadmin intervention Glasgow - review of the switch of the individual USER jobs to a max of 1000 rather than per VO limit of 1000 (which hit the atlas SAM / steve lloyd jobs) ROC updates - ======== No meeting in sept - mostly admin in egee meeting tickets - 7 open few still open against olivier - follow up offline with Dace Colling pps - update 6 lcg-rm - zero maint mode atlas - several probs with the info published per VO cf the overall. ... Graeme / Pete -= will investigate vo views information provider bug - will eventually need to go to Jeff Templon - will need info from sites as to why some sites don't work Security ====== * Root CA - see above * Changing ownership of gridmapdir - prcocessed as vuln and followed up OK. * OSCT incident handling procedure - Mingchao - sent it out last week - Policy dated 10 Sept - no sig difference but we need to define a security incident contact for the UKI ROC * XSS problems - potential ones are GOCDB (NGS), GridPP (mcr) and VOMS (mcr) - not a 'vuln' but needs code audit to be sure we're safe. (romain) - Andrew McNab has already fixed. HEPSYSMAN ======== * date confirmed as 31 oct - oone day event - 10am start or so - no site report s in Autumn meeting - Chris Brew, AE prepared to talk. - Ian neilson avail to talk too. trasfer tests - / gridmon =============== AE to arramge milestones for testing need to concentrate on longer term tests support of gridmon boxes actions ===== fewer than before JC took a hol EGEE ==== GS, AF, PG, MM, SB, JC going SL4 on WN's ========= * Oxford - new machines running SL4 * Glasgow - alewdy done * Manchester - some WN's SL4 - one of the CE's to be upgraded. Colin Morey now away * Lancs already upgraded. * Shef / Liv have other issues just now * Birm = Some SL4 in testing - prod syaying for now * Bris - devel with SL4 * Camb - Se's SL4, WN still 3 * RalPPd - >50% sl4 already * T1 - 60-80% SL4 just now Condor Vulns ======== Pete will make sure santanu aware AOB === UKI roc meeting thurs anything to raise let JC know
There are minutes attached to this event. Show them.
    • 11:00 → 11:20
      Experiment problems/issues 20m
      Review of weekly issues by experiment/VO - LHCb -- Analysis jobs (primarily root jobs) crashed over the weekend - possibly because of a job environment problem. A GGUS ticket (27224) has been submitted - CMS -- The CMS CSA07 service challenge Tier 0 reconstruction and Tier 1 data export phase should now start on Tuesday 25 September and run for 30 days. - ATLAS - Other
    • 11:20 → 11:35
      ROC update 15m
      ROC manager update ************************* Next ROC manager's meeting will be a brief review meeting at EGEE'07: http://indico.cern.ch/conferenceDisplay.py?confId=14081 Ticket status *************** GOCDB - 7 Pete - 1 Alessandra - 2 Olivier - 2 Tier-1 - 2 Imperial - 1 Birmingham - 1 EGEE ops meeting items of interest ****************************************** # Release of gLite 3.1.0 Update06 to PPS done * new voms certificate for US-ATLAS server (sync to production) * uberftp for the glite-UI node * lcg-tags added to glite 3.1 UI, WN * lcg-infosites added to the glite 3.1 WN # JRA1 and SA3 have now moved the old lcg-RB to zero maintenance mode. glite 3.1 WMS whether on SL3 or SL4 is the supported solution. # ATLAS - A user with EMail in the DN had problems at 2 sites, SARA and BNL, to access dcache. # n many sites there is a mismatch between all inclusive information published for a CE and information published in the VOViews. As an example, the queue mentioned below supports only ATLAS, therefore, the number of waiting jobs in the inclusive view should be the same as the one for the ATLAS VoView. But it is not. The VOView publishes all zeroes. Moreover, there are some queues where the number of waiting jobs for all views do not add up to the total published in the inclusive view. In total more than 130 ATLAS queues are affected, among which almost all T1s. Since the WMS uses information in the VOView and the latest one is generally the wrongly published one, ATLAS is submitting jobs almost randomly with accumulation of jobs at small sites. The issue is extremely severe. We would like the operation team to investigate the reason for so many mismatches and chase site by site to have the problem cured. Attached below is a list of currently problematic CEs as of today. ===> CE:ce.epcc.ed.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:3 TOTVOrun:0 TOTwait:29 TOTVOwait:0 ===> CE:ce00.hep.ph.ic.ac.uk:2119/jobmanager-sge-72hr TOTrun:177 TOTVOrun:3717 TOTwait:0 TOTVOwait:0 ===> CE:epgce1.ph.bham.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:6 TOTVOrun:0 TOTwait:457 TOTVOwait:0 ===> CE:epgce1.ph.bham.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:3 TOTVOrun:0 TOTwait:1 TOTVOwait:0 ===> CE:fal-pygrid-18.lancs.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:13 TOTVOrun:7 TOTwait:0 TOTVOwait:0 ===> CE:hep-ce.cx1.hpc.ic.ac.uk:2119/jobmanager-pbs-heplt2 TOTrun:274 TOTVOrun:5206 TOTwait:353 TOTVOwait:6707 ===> CE:heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:19 TOTVOrun:0 TOTwait:0 TOTVOwait:0 ===> CE:heplnx206.pp.rl.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:8 TOTVOrun:0 TOTwait:2 TOTVOwait:0 ===> CE:heplnx207.pp.rl.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:19 TOTVOrun:0 TOTwait:0 TOTVOwait:0 ===> CE:heplnx207.pp.rl.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:8 TOTVOrun:0 TOTwait:2 TOTVOwait:0 ===> CE:lcgce01.phy.bris.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:3 TOTVOrun:0 TOTwait:0 TOTVOwait:0 ===> CE:lcgce01.phy.bris.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:13 TOTVOrun:0 TOTwait:24 TOTVOwait:0 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-10min TOTrun:1 TOTVOrun:18 TOTwait:0 TOTVOwait:0 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-12hr TOTrun:40 TOTVOrun:720 TOTwait:1 TOTVOwait:18 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-1hr TOTrun:6 TOTVOrun:108 TOTwait:0 TOTVOwait:0 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-24hr TOTrun:70 TOTVOrun:1260 TOTwait:33 TOTVOwait:594 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-30min TOTrun:3 TOTVOrun:54 TOTwait:0 TOTVOwait:0 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-3hr TOTrun:14 TOTVOrun:252 TOTwait:1 TOTVOwait:18 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-6hr TOTrun:25 TOTVOrun:450 TOTwait:1 TOTVOwait:18 ===> CE:mars-ce2.mars.lesc.doc.ic.ac.uk:2119/jobmanager-sge-72hr TOTrun:99 TOTVOrun:1782 TOTwait:33 TOTVOwait:594 ===> CE:pc90.hep.ucl.ac.uk:2119/jobmanager-lcgpbs-lcgatlas TOTrun:13 TOTVOrun:0 TOTwait:32 TOTVOwait:0 ===> CE:serv03.hep.phy.cam.ac.uk:2119/jobmanager-lcgcondor-atlas TOTrun:3 TOTVOrun:0 TOTwait:4 TOTVOwait:4444 ===> CE:svr016.gla.scotgrid.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:13 TOTVOrun:0 TOTwait:0 TOTVOwait:0 ===> CE:t2ce02.physics.ox.ac.uk:2119/jobmanager-lcgpbs-atlas TOTrun:7 TOTVOrun:0 TOTwait:1 TOTVOwait:0 ===> CE:t2ce02.physics.ox.ac.uk:2119/jobmanager-lcgpbs-short TOTrun:0 TOTVOrun:0 TOTwait:2 TOTVOwait:0
    • 11:45 → 11:50
      HEPSYSMAN & monitoring 5m
      - Confirmation of the date - What should be covered
    • 11:50 → 11:55
      The status and future of network tests 5m
      ... what next for data management
    • 11:55 → 12:00
      AOB 5m
      - Actions http://www.gridpp.ac.uk/wiki/Deployment_Team_Action_items - EGEE'07 http://indico.cern.ch/conferenceTimeTable.py?confId=18714&showDate=all&showSession=all&detailLevel=contribution&viewMode=parallel - SL4 on WNs - progress and issues (to be reviewed on Thursday - UKI ROC meeting)