Deployment team

→ Europe/Zurich
EVO - GridPP Deployment team meeting

EVO - GridPP Deployment team meeting

Jeremy Coles
Description
- This is the weekly DTEAM meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +41 22 76 71400. The phone bridge ID is 581947 with code: 4880.
Minutes
    • 11:00 → 11:20
      Experiment problems/issues 20m
      Review of weekly issues by experiment/VO - LHCb - CMS - ATLAS - Other VOs Site Issues: - JET - Brunel (jobs waiting on grid-40) and FCR for CMS (http://pprc.qmul.ac.uk/~lloyd/gridpp/fcrtest.html). Also 100% failure on UK tests. - Lancaster UK test result (failures look a bit high) - London Tier-2 RB. Should this be removed from the monitoring? http://pprc.qmul.ac.uk/~lloyd/gridpp/rbtest.html. WLCG accounting - The September Tier-2 accounting report shows the following usage as a percentage of pledged resources: London 108%; Northgrid 137%; ScotGrid 69% and SouthGrid 69%.
    • 11:20 → 11:35
      ROC update 15m
      ROC update *************** Picked up by COD at yesterday's WLCG/EGEE ops meeting: Additional information about ticket set to 'Case transfered to political instances': Site name: UKI-LT2-QMUL ROC : UKI Ticket id: 8997 (GGUS id: 40945) Problem : RGMA-host-cert-valid 'Case transfered to political instances' status from 2008-09-30 and no progress. Now node is in Scheduled Downtime. Release: 2008-10-18: gLite 3.1 Update 34 was released to production The update contains: * LB: A number of non-critical bug fixes (PATCH:1803 , PATCH:2380) * glite-UI: glite-brokerinfo added * glite-BDII: obsoleted lcg-info packages removed fixe for a critical bug which could potentially cause problems on any node type if the variables BDII_VAR_DIR or BDII_DIR are not set in the configuration file (not an issue for sites using YAIM). This update also fixes the problem spotted in the previous release). The release notes given there are now valid for the glite-BDII service node. In addition the the attributed set_lk_max_locks 2000 has been set in the DB_CONFIG file which is a partial solution to Bug #42727. * glite-VOMS (oracle and mysql): obsoleted lcg-info packages removed * FTS for SL4: - Some UK sites see ongoing problems with the site report information. Also mentioned by CE ROC. - IN2P3-CC T1 has now succeeded in configuring its CEs GIP to restrict CMS access to both VOMS:/cms/Role=production and VOMS:/cms/Role=lcgadmin. - The way the configuration has been made (with help of Steve Traylen) can be found in GGUS ticket #37102. gLite 3.0 services NOW OBSOLETE * glite-SE_classic * glite-VOBOX * glite-WMS * glite-PX * glite-MON From UK site reports (http://tinyurl.com/6m6bx8): UCL-HEP: Many entries in this report do not match the SAM database UCL-HEP: Some ops SAM jobs reported failed with user proxy expired. Watching the batch system, I see jobs being ran timely and never waiting in the queues as batch system has had many empty slots throughout. I have also noticed from Steve Lloyd s Atlas tests <http://pprc.qmul.ac.uk/~lloyd/gridpp/atest.html> that jobs are flagged as failed because "Failed to run after 8 Hr 3 Min 12 Sec" which is clearly not the case. I also see that a few other sites have had exactly the same issue as us with Steve s Atlas tests, not only in London. BHAM-HEP: - Birmingham had suffered a lot of networking and dns instabilities. Core switch needs rebooting by IS. DNS problem have practically disappeared since IS upgraded BIND from version BIND 9.4.2-P1 to BIND 9.4.2-P2. - GPFS with software area was made available only midweek by BlueBear/ClusterVision team after a shutdown of over two weeks. - epgce1 suffered for Marodona errors. I tested scp, globus-url-copy on all the workers, ntp, ... all the tests were successful. I migrated the CE to new server and SL4. Problems seems to have disappeared suddenly. Issue with SL3 CE? Bristol and JET reported lots of Maradona failures too; some were also observed on SL4 CEs -> middleware issue???? Difficult to say at this point, epgce1 hardware was misbehaving at times, will do memtests. - The scattered software area due to absence of large disk remains an issue. I will relocate it to newly purchase raid. OX-HEP: We have had problems with instability of maui. We have to restart it regularly. WLCG update ***************** - GDB rota Ticket status *************** https://gus.fzk.de/download/escalationreports/roc/html/20081020_EscalationReport_ROCs.html
      Ticket metrics
    • 11:35 → 11:45
      ATLAS multi-user jobs 10m
      - the proposal (what and why) - the implementation (how) - the response
    • 11:45 → 11:50
      Inefficient jobs 5m
      - The document was reviewed by the DB in March. - Are there any changes that anyone wants to recommend. See the high-level summary here: http://www.gridpp.ac.uk/db/Inefficient_Jobs.html - In particular does this still hold? " Any job consuming <10 minutes CPU over a given 6 hour period (efficiency < 0.027) is considered stalled." - What problems are being seen at the moment?
    • 11:50 → 11:55
      Disaster planning 5m
      - Brief overview of where this is going!
      DP template
    • 11:55 → 12:05
      Actions review 10m
    • 12:05 → 12:10
      AOB 5m