Operations team & Sites

Tuesday 17 Jul 2012, 11:00 → 12:16 Europe/London

EVO - GridPP Operations team meeting

- This is the biweekly ops & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 126540 with code: 4880. Apologies:

11:00 → 11:20 Experiment problems/issues

Review of weekly issues by experiment/VO - LHCb - CMS - ATLAS 9/7/2012-16/7/2012 UKI-SOUTHGRID-OX-HEP and UKI-SOUTHGRID-BHAM-HEP are requested to provide a syncat dump of the storage element so that atlas can eliminate dark data. Instruction about how to do it can be found https://twiki.cern.ch/twiki/bin/view/Atlas/DDMOperationsScripts#Tools_to_dump_SE. If you can't access it let us know. UKI-SCOTGRID-DURHAM: 30% of job failures is currently in test mode. Elena reported it's few nodes misbehaving. Perhaps they can be set offline. UKI-LT2-QMUL: cmtsite timeout error might have appeared again but it was due to a user code with memory leaks and using all the memory on those nodes. (more on memory later) UKI-LT2-IC-HEP: there has been a answer from AdS and they should check the NFS mount they are using for cvmfs. Memory, CPU resources and other Atlas problems. There has been a big wave of errors caused by a number of atlas problems that should be ignored when looking at stats - Group production incorrectly configured this has caused thousands of failures all over the place last week. All the problems have been so far reported by the shifters and fixed. - lost heartbeat and CPU consumption a new algorithm is being tested to broker jobs to sites with enough CPU. Atlas sends out scout jobs for each task and then it uses what they report to broker the bulk of the jobs. Among the things reported is the time they take to run and the average will be used to make a best guess on how much cpu the jobs use. The tasks with scouts that never end will be aborted. The algorithm is clearly an approximation so they will use something like 2*cpucount less than maxtime where the maxtime is a site parameter extracted from the BDII. So far few tasks have been tested though. We are also working to understand why the pilots don't get the SIGTERM signal that is sent to the jobs by the batch system before it definitely kills them and why the batch system errors are not correctly passed back most of the time and to understand which other signals need to be caught to replace the "lost heartbeat" error with well defined errors and behaviour. I setup a test queue in Manchester for this purpose I need to enable it in the cream CE. - Memory consumption: QMUL reported excessive memory consumption from the jobs of a user - this might have hit also other sites. The problem is most likely memory leak but there seem to be no protection against this in UK sites other than the 3GB queues at RAL. The pilots should set a limit but they don't seem to have done so in this case. Alastair will investigate what needs to be done to limit jobs with memory leaks in the future. - cmtsite timeout: QMUL installed new cvmfs on 1/3 of the nodes and in Manchester I installed now 5 nodes which had a problem with autofs with SL5.8 to hopefully cure the autofs hanging. - Other CW - 'Snoplus have asked about blinding data. AIUI, they want to hide about half of the data and not look at it until they have developed their analysis algorithms. They then want to unlock the data to allow them to rerun their analysis on the whole data set. So, do we have a recommendation on the best way for them to do this?' So far: spacetokens or LFN.

11:20 → 11:40 Meetings & updates

With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest - Tier-1 status - Accounting - Documentation - Interoperation UKI-LT2-Brunel dgc-grid-35.brunel.ac.uk bdii_site dgc-grid-50.brunel.ac.uk SRM UKI-SCOTGRID-ECDF srm.glite.ecdf.ed.ac.uk SRM UKI-SCOTGRID-GLASGOW svr022.gla.scotgrid.ac.uk org.glite.lb.server svr022.gla.scotgrid.ac.uk org.glite.wms.WMProxy UKI-SOUTHGRID-BRIS-HEP lcgse02.phy.bris.ac.uk SRM UKI-SOUTHGRID-CAM-HEP serv07.hep.phy.cam.ac.uk bdii_site For gLite 3.2 stuff, with some repitition skipped, only listing one example of each service, and limiting to the ones that a currently unsupported: RAL-LCG2 lcgce03.gridpp.rl.ac.uk org.glite.ce.CREAM lcgvo-alice.gridpp.rl.ac.uk VOBOX srm-atlas.gridpp.rl.ac.uk SRM (Is the SRM one a false hit? I though it was a recent version of CASTOR?) UKI-LT2-IC-HEP topbdii01.grid.hep.ph.ic.ac.uk bdii_top UKI-LT2-RHUL sbdii2.ppgrid1.rhul.ac.uk bdii_site UKI-NORTHGRID-MAN-HEP site-bdii.tier2.hep.manchester.ac.uk bdii_site top-bdii.tier2.hep.manchester.ac.uk bdii_top UKI-NORTHGRID-SHEF-HEP lcg.shef.ac.uk bdii_site UKI-SCOTGRID-ECDF info.glite.ecdf.ed.ac.uk bdii_site UKI-SCOTGRID-GLASGOW svr030.gla.scotgrid.ac.uk bdii_site UKI-SOUTHGRID-CAM-HEP vserv02.hep.phy.cam.ac.uk bdii_site UKI-SOUTHGRID-OX-HEP t2bdii04.physics.ox.ac.uk bdii_site UKI-SOUTHGRID-RALPP heplnv145.pp.rl.ac.uk bdii_site There's also quite a few SRM (read: DPM) instances, that aught to be updated to the EMI version before end of Oct. I know Brian had some tickets tracking that. Generally, however, it looks like it's primarily BDII's that need a round of updating. I am surprised by the gLite 3.1 SRM at ECDF, Brunel and Bristol - in particular, is this a false hit at Edinburgh? There is a starting point table for reccomended versions for upgrade in the agenda https://wiki.egi.eu/wiki/Agenda-16-07-2012 LCG-CE is not mentioned, but the reccomendation for sites that want to offer a Globus interface is to deploy the IGE Globus GRAM-5 from UMD-1. The policy from EGI is that if gLite 3.1 services are running after end of September, then the site will be de-certified. (which is rather drastic, but that is their policy, on the grounds that there is no security support in them). We need to examine the update of VO testing on EMI-1 WN's - as the gLite 3.2 WN will be EOL'd soon. Who has EMI-1 WN's for VOs to test on, and what VO's have tested their jobs on EMI-1, and can we build a list of 'current' users that haven't? - Monitoring - On-duty - Rollout - Security - Services - Tickets - Tools - VOs - SIte updates

11:40 → 12:00 Review of July GDB

Daniela's report: https://www.gridpp.ac.uk/wiki/GDB_11th_July_2012

12:00 → 12:05 Actions

To be completed: https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items Completed: https://www.gridpp.ac.uk/wiki/Operations_Team_Completed_Actions

12:05 → 12:06 AOB