Ops team minutes  22 Jul 14 

Experiment problems/issues 20' 

CMS (Daniela) 
Nothing exciting.
Just Rauls DPM ticket that is more or less resolved.

ATLAS (Alessandra) 
Not much 
Ticket for Glasgow - one server overloaded with too many requests.
	Also happened to Manchester last week - do we have to dig further ?
	(Sam) in this case maybe a backlog of deletion - definately deletion load.
	In Manchester case it was xrootd...
	Alessandra - deletion service did have a problem last week so is catching up.
RAL ; Shef and Royal Holloway need to restart xrootd redirectors.

Other VO (Chris) 
No response from VO admins.
Nothing to Report.
hyper-K - would be appreciated if one or two sites enabled hyper-K
	Can copy files back to iRods at QMUL .
Sites might (e.g. Lancs/Ed ) but not until Sept.
(Chris) - If RAL do that would be significant resources..

Meetings & updates 20' 
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest

RAL FTS2 Server being switched off (believe its only T2K and they are aware). 
Please let people know if others are using FTS2 (and copy in Jeremy) 

Chris - RAL have moved to 3.2 - which allows multi stage transfers so can put in LFC. Might be useful to non-LHC VOs - have told T2K .

Bi-weekly WLCG ops meeting.

Personal detials going to public lists (not aimed at anyway in particular). 
Traffic light protocol 

Message from Pete Clarke - want IPv6 session at GridPP33 and Janet are participating 
	Looking for ideas.
(Closing date for GridPP33 is this Fri..) 

There was a Technical meeting on Fri 
WLCG figures for June are finalised.

HEP Software Foundation - initial meeting. 
Trying to put together a website 

WLCG Ops coord meeting :
	Input from Sites ?	

Tier 1 
Castor instance updating. 
Software service for small VOs removed at start of Sept. Move them to CVMFS.
At-risk today on network

Apel
Alessandra - should we change the algorithm for not publishing in Nagios to look at all CEs that should be publishing.
Didn't notice as some CEs were publising 
Jeremy - could look at this .

Gareth - had some similar but with ARC CEs - as there was no check that it was publishing.

Documentation 
Andrew - issue with alerts - will look at.

Interoperation / Monitoring (David) 
Can't make next interops meeting if next week.
For Monitoring - if any feedback on cvmfs proposal do send it.

Gstat pages obsolete ? 
Ewan - they are not supported
Chris spotted things .Agree if these are out of date then it is better to get rid of them
Jeremy will take that back to meeting.

Security 
EGI policy requirement for argus - or use of suspension list in some way.
In future might just be assumed that this works and no emails going around 

(Chris) Storm following up in ticket. Noone has tested his xrootd setup
(Ewan) Believe the DPM module is in a useable repo - but not sure anyone has setup.
(Sam) yes it is in the EMI repo - noone has done it.

Multicore
Now regular running jobs from ATLAS . 
In UK 4 sites running (5 enabled)
October may seem far away.. 
Sites should have a go at least by then 
It will help with reduced memory etc.

Solution for HT-Condor ..
Oxford is the enabled but not running - not clear whether its site or ATLAS side - 
perfect candidate to be the next running(!)
Ewan - OK on boad in principle - if we know what the problem is.
	Resources we have cream ce and torque maui - that doesn't support multicore
	Arc-ce and condor does - now 1/3 total capacity - migrating over.

Andrew L - Condor defrag deamon so that jobs can be drained.
Ewan - ok what would be the symptom if don't have that...
Steve jones - what was the RAL setup info:
Andrew Lahiff: (11:51 AM)
Information about our multicore setup is here:
https://www.gridpp.ac.uk/wiki/RAL_HTCondor_Multicore_Jobs_Configuration
Sam - other systems are designed to run mcore payloads . So don't need this deamon.
Sam - Torque reserves blocks - it maybe could be tuned to be more optimal ...

Alessandra - Jeff showed some script to move resources around - runs in a cron.
Sam - Would prefer to backfill - but if you can't then this is the best. 
Alessandra - First step this is what we should do. Also use another Nikhef script so parameters passed to batch system. CREAM ce throws away walltime , doesn't pass to torque.

Jeremy - Can use SGE "out of box"? 
Chris Walker - had to use cron job to reserve slots - but otherwise worked.
	Can "best practise" be collected on a wiki.

Alessandra - yes there is a wiki in development.

Durham is SLURM (probably only site in UK) . .. Brunel was testing.

Jeremy -  We don't need all sites...

Alessandra - Tier 1 running both CMS and ATLAS jobs 
LHCb not actively looking at.
Mostly an ATLAS site issue.
At some point CMS will look at T2s..

Glasgow - Gareth - just looking at Arc and Condor - happy to do it - just dragging on a bit.
IC - haven't looked at but can.
Liv - Steve - Condor server following on from others - just 5-6 nodes now - but will enable everything and make it known. In near future - ~month away.
Sussex - Matt - In principe yeah - but have other issue with ATLAS jobs - so need to sort that first.

Alessandra/ Jeremy - regular update? Yes.

Alessandra  - not just site as ATLAS have to setup queues... Start with OX and then follow up in future weeks.
Kashif - don't have defrag deamon so will look at that.

Jeremy - also gather together links etc - advertise those pages. 
And someone with Torque setup look at scripts etc... (just manchester?) 

Alessandra - ultimately will also move to condor....

Extra Q in ATLAS needed?
Alessadra - depends on site - e.g. Nikhef does, but QMUL doesn't 
Chris - Don't need it - but they may have one for other reasons.

Matt - have equal resources on both Torque , SGE.. would be good to use both ... maybe the box hasn't been ticked on ATLAS for SGE to use - or maybe a throughput problem.

Tickets (Matt et al.) 

See Matts post.
RAL - 3 tickets, 2 need updates . Andrew will do so.
Shef - Elena away...
Raul / DPM one - worth recording outcomes - tweaking for good performance gains.

Ewan (107007) - cvmfs client versions - asking for a version they already had. Why did they get the ticket. 
Jeremy - prob a mistake but will check.
Chris - Stay on an old version until it unmounts - is that the issue?
	Ewan - prob rebooted since then 
Jeremy will follow up since with Andrea.

Actions: 
Reminder on sites to say what they  are doing for IPV6.
dirac file catalog open action to evaluate consistancy checking - Sam .
kashif on puppet module for cvmfs - could maybe be closed ? follow up offline .

Register for GridPP Meeting and Send pete Items for IPV6 discussion . 

wahid: (22/07/2014 11:04)
iRods eh ?
we might but not now
Matt Doidge: (11:06 AM)
We'd like to enable them - but we're looking at late August/September
Alessandra Forti: (11:09 AM)
sorry
I'm late
wahid: (11:18 AM)
there is a plan for deletion to use dav
whether that will help or not who knows
Chris Walker2: (11:19 AM)
Is the problem due to putting one new disk server on - and that getting the majority of the traffic?
Alessandra Forti: (11:19 AM)
RAL-LCG2 UKI-NORTHGRID-SHEF-HEP UKI-LT2-RHUL 
wahid: (11:20 AM)
RAL-LCG2 might have a bigger problem with FAX - Alastair and Shaun were looking at it
Ewan Mac Mahon: (11:28 AM)
AIUI the cluster publisher doesn't support ARC CEs.
Chris Walker2: (11:30 AM)
Alessandra - are you going to file the ticket against nagios. 
I think you should as I'm sure it will catch me out at some point. 
Alessandra Forti: (11:30 AM)
I may add to the ticket I've opened to correct the manchester accounting.
Chris Walker2: (11:36 AM)
Personally, I'd open a separate ticket - it is likely to remain open for some time as they will take months to fix it. Then the ticket will be only on one thing and they won't close it because your accounting is fixed. 
wahid: (11:38 AM)
they will still have to email 
how will it be banned on storage for example
Ewan Mac Mahon: (11:38 AM)
Well, it's been used a couple of times and they didn't email.
wahid: (11:39 AM)
well it won't be banned then 
on DPM you have to runa command to get the list
could be done in a cron but I doubt anyone is 
Duncan Rand: (11:40 AM)
what's the dpm command?
Chris Walker2: (11:40 AM)
https://ggus.eu/index.php?mode=ticket_info&ticket_id=106379 - StoRM frontend memory leak with argus. 
Duncan Rand: (11:41 AM)
and which rpm?
wahid: (11:41 AM)
You can then 
;-) 
Ewan Mac Mahon: (11:41 AM)
I do plan to.
There's probably a blog post in it, too :-P
Matt Doidge: (11:43 AM)
That's fine.
Samuel Cadellin Skipsey: (11:43 AM)
duncan: the package is dpm-argus-1.8.8-1.el6.x86_64.rpm
It's in the EMI3 repo
(the poller it installs is called dpm-arguspoll, apparently)
Duncan Rand: (11:43 AM)
Thanks, it doesn't appear to be brought in automatically with dpm...
Samuel Cadellin Skipsey: (11:44 AM)
No, it isn't
(Thanks to packaging etc)
But we are told it can be installed and configured and work.
Duncan Rand: (11:45 AM)
Error: Package: dpm-argus-1.8.8-1.el6.x86_64 (EMI-3-updates) Requires: dpm-mysql-nameserver = 1.8.8-1.el6 Installed: dpm-name-server-mysql-1.8.8-4.el6.x86_64 (@epel) dpm-mysql-nameserver = 1.8.8-4.el6
Samuel Cadellin Skipsey: (11:46 AM)
...
Jeremy Coles: (11:46 AM)
Alessandra F, Andy W, Andrew L, Andrew M, Chris W, Daniela B, David C, Duncan R, Ewan M, Gang Q, Gareth R, Govind, Jeremy C, John B, Kashif M, Matt D, Matt RB, Rob H, Rob F, Sam S, Steve J, Wahid B
Samuel Cadellin Skipsey: (11:46 AM)
Oh deary me: I see the packaging is not the best, then, Duncan
I've prodded some people, Duncan, we'll see if we can fix packages.
Andrew Lahiff: (11:51 AM)
Information about our multicore setup is here:
https://www.gridpp.ac.uk/wiki/RAL_HTCondor_Multicore_Jobs_Configuration
Matt Doidge: (11:52 AM)
SGE works quite well out of the box for multicore.
wahid: (11:56 AM)
Chris can you put in this window what you jsut said re script
Samuel Cadellin Skipsey: (11:57 AM)
there's Nikhef stuff on their wiki, https://wiki.nikhef.nl/grid/Enabling_multicore_jobs_and_jobs_requesting_large_amounts_of_memory but as Alessandra notes, he's mostly distributed scripts on the list.
Chris Walker2: (11:57 AM)
[root@fe03 cron.hourly]# cat /etc/cron.hourly/gridengine_reserve  #!/bin/bash # reserve slots for jobs requesting 8 cores /usr/local/sbin/do_reservation > /var/log/gridengine_reserve.log

AIUI - and Dan's the one that did this, this is a bodge to ensure that slots are actually reserved for jobs. 
It should really be in the submit script. 
Duncan Rand: (11:58 AM)
I think Brunel mighyt have a test slurm
Ask Raul.
Chris Walker2: (12:00 PM)
This came from one of the german sites IIRC
Daniela Bauer: (12:03 PM)
We haven't looked into this yet, but we can
Duncan Rand: (12:03 PM)
Sam: why is it listed as dpm-mysql-nameserver and sometimes dpm-name-server-mysql 
Matt Doidge: (12:03 PM)
Chris does "do_reservation" use qrsub to setup things?
Chris Walker2: (12:04 PM)
[root@fe03 cron.hourly]# cat /usr/local/sbin/do_reservation #!/bin/bash # reserve slots for jobs requesting 8 cores . /etc/profile.d/sge.sh /usr/bin/qstat -u \* | grep " 8 " | awk '{ system("/usr/bin/qalter " $1 " -R y")}' /usr/bin/qstat -u \* | grep " 2 " | awk '{ system("/usr/bin/qalter " $1 " -R y")}' /usr/bin/qstat -u \* | grep " 4 " | awk '{ system("/usr/bin/qalter " $1 " -R y")}'
Matt Raso-Barnett: (12:04 PM)
i think it is doing a qalter -R 
ah thanks
Chris Walker2: (12:04 PM)
It's a horrendous bodge - and one that should be fixed in the submit scripts
THe rest of it is in the EGI multicore/mpi 
Matt Doidge: (12:05 PM)
Thanks Chris.
Samuel Cadellin Skipsey: (12:05 PM)
Duncan: well spotted, I think it's probably a typo in the dependancy
I see only dpm-name-server-mysql actually existing.
wahid: (12:06 PM)
I'll type
Andy has left - had multi core queues for a while - so not sure why he hasn't enabled this for atlas
maybe the shared nature means reservations don't work out well
Matt Doidge: (12:08 PM)
Is it easier if sites setup seperate multicore queues, or if sites enable multicore on currently existing queues?
*easier for atlas
Samuel Cadellin Skipsey: (12:09 PM)
Duncan: no, actually, it appears that dpm-name-server-mysql actually provides both dpm-name-server-mysql and dpm-mysql-nameserver
For "reasons", we assume.
Daniela Bauer: (12:15 PM)
Apologies, I need to go - I'm going on holiday today :-) (I won't be here next week either )
wahid: (12:16 PM)
Closed
Matt Doidge: (12:19 PM)
https://ggus.eu/index.php?mode=ticket_info&ticket_id=107007
Ewan Mac Mahon: (12:24 PM)
http://southgrid.blogspot.co.uk/2014/05/configuring-cvmfs-for-smaller-vos.html
^ I think that closes that action.