grid ops meeting 23/06/2015

Action Items :

Glue 2 testing - remove test as critical this will get rid of alerts in dashboard 

attended
Alessandra Forti, Andrew John Washbrook, Andrew McNab, Brian Davies, Catalin Condurache, Chris Brew, Daniel Traynor, Daniela Bruer, David Crooks, Elena Korolkova, Ewan Mac Mahon, Gang Qin, Gareth Douglas Roy, Garath Smith, Gorden Stewart, Govind, Ian Robert Neilson, Jeremy Coles, John Bland, John Hill, Kashif, Liam Skinner, Matt Doidge, Matt Raso-Barnett, Paige Winslowe Lacesso, Raul, Robert Fray, Robert Wolfgang Frank, Samiel Cadellin Skipsey, Steve Jones, Tom Whyntie

minutes by Daniel Traynor

Experiment problems/issues

LHCB NTR

CMS NTR

ATLAS 

Atlas T1 meeting noted FTS issues in general, large number of files waiting to be transferred, submission is slow as a result, 

Atlas Tickets discussion

RAL request update for ticket 112721. - ticket seems to conflate different issues.

Disk pool issues (full) RAL are in the process of deploying new disk servers

A specific transfer was slow from Glasgow to continental Europe, but other transfers fine, not sure what the issue is, not easy to debug. Perfsonar might help to debug

DIRAC: 
	see chat

-- LIGO: Catalin
	progress ongoing, python packages added to CVMFS that are  not available on RAL WN, see chat

-- LOFAR: George 
	NTR

-- LSST: Alessandra
	NTR, Liverpool enabled other VOMS
 
-- LZ: David 
	NTR

-- UKQCD: 
	NTR

-- UCLan/GalDyn: Tom
	Deploying software via CVMFS, fist test jobs ran

	CTSCAN work

-  DIRAC status
	Glasgow something changed on the 8th June - last cream ce turned off - ARC CEs not in yet, therefore no jobs running. VAC at Lancaster in progress, gridpp  VO needs pilot role at site, tickets sent

Meetings & updates

http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest

General updates
General request for minor VOs CVMFS to pint to egi,eu not gridpp.ac.uk, To be checked?
GridPP VO software directory using cvmfs not yet needed.
Discussion on the other VO storage space use and the clean up after the VO ends.

WLCG Operations Coordination

BDII discussion - tbsupport contains extensive discussion

Tickets

MANCHESTER 114444

Talking to DPM SRMs by LHCb checksums were not being stored (now fixed). Sites being asked to crete them. 

Site roundtable & general discussion

Manchester: miss what was said

ECDF: woking with central batch system to make sure new computing cluster is comparable with grid setup.

RALPPD: Deploy ELK, getting rid of SL5, network tweaks to increase bandwidth to T1, noticed link was reaching 10Gs.

QMUL: IPV6 dual stack, Lustre work

IC: new Dirac version investigation.

Glasgow - HTCondor arc transfers finalising, refactor cluster management, CEPH studies, cloud studies.

Sheffield: move storage servers and ARGUS off of SL5. HTCondor deployment starting. IPV6 pefsonar enabled, get experience then deploy more.

Oxford: clean up grid config e.g. pilot jobs configuration, getting rid of CREAMCE- ALICE issue, SL5 storage nodes staying for a while, 

RHUL: Setup test HTCondor system, moving storage nodes outside of firewall. 

Liverpool: GridPP 35, general site updates,  Upgrade of Network, upgrade of cooling, 

Cambridge: ARC CE investigation, SL5 storage nodes staying, general site updates, Plan move to new datacenter.

Durham: Lot of new nodes arriving.

Lancaster: Shift to new infrastructure setup using VMWare. and reinstall of cluster( a few days downtime). Overhall of tarball. SL5 storage hanging around till retirement. IPV6 discussion.

Sussex: Lustre migration in progress including monitoring update. 

Brunel: IPV6 migration advanced, no SL5, move from cream to ARC CE almost finished, general site updates, CMS testbed for DPM middleware readiness group request. 

Chat

Jeremy Coles: (23/06/2015 11:00)
Dan is taking minutes today. Thanks Dan.
Matt Doidge: (11:01 AM)
That's better - I forgot you could turn off the "when people join/leave" tone in Vidyo.
Jeremy Coles: (11:01 AM)
Useful for me to know when people leave!
Brian Davies @RAL-LCG2: (11:02 AM)
vo.dirac.ac.uk report: Now sucesfully transfering files from durahm to tape at RAl using FTS
now investigating monitoring/lproxy renewal/retries /checksums/lfc registration webdav view /webFTS ( last two or three are possible not necessary)
Jeremy Coles: (11:06 AM)
https://ggus.eu/?mode=ticket_info&ticket_id=112721
Alessandra Forti: (11:07 AM)
I was only help the debugging
from Manc it was ok ~100MB/s
Brian Davies @RAL-LCG2: (11:10 AM)
perfsonar should also be able to see if there is a change in rates
you should be able to look on th eglasgow perfsonar bw box to see what is is beoing tested
Samuel Cadellin Skipsey: (11:14 AM)
(I also note: re Glasgow transfers, that we also have good rates to Nikhef)
Brian Davies @RAL-LCG2: (11:14 AM)
yes
Ewan Mac Mahon: (11:15 AM)
On Glasgow transfers; aside from the various dashboards you should be able to see what tests your perfsonar in running directly on its own web interface, and if need be add some more.
If you can replicate the problem with perfSonar, that would give a bit of a base to work from and maybe start thinking about traceroute traces and so forth.
Tom Whyntie: (11:16 AM)
Catalin: another reason ot use the CernVM
No, they're not
but
it let's you know which ones are missing
without having to test it by running it on the worker nodes
Sounds like excellent progress though, thanks! :-)
Samuel Cadellin Skipsey: (11:18 AM)
Ewan: sure, but I'm not actually sure there's a big issue with us specifically - I also see transfers to DESY-HH today which were pretty fast.)
Brian Davies @RAL-LCG2: (11:19 AM)
For ticket section. Re CMS slow transfers from fnal tro RALggus ticket(GGUS #114275), We at RAL are investigating fixing packet loss , but also trying to see if it is possible to fix tcp streams per file to be higher to try anf get these slow transfers to complete. Hava checked routing is ok. Looking at seeing if there are any good transfer rates at the moment for any file from FNAL to RAL.
Tom Whyntie: (11:20 AM)
Sounds exciting - looking forward to it
Ewan Mac Mahon: (11:22 AM)
@Sam - I agree it's probably not a Glasgow specific problem, but if you can find something like a pair of destinations, one that's good and one that's bad, then we can all try them and see if we can spoty the differences/commonalities.
Samuel Cadellin Skipsey: (11:23 AM)
Ewan: sure, I'm having a prod now. (Surprised to see that actually we're also pretty fast to the USA at the moment, incidentally)
Liam Skinner: (11:25 AM)
@Glasgow people, can I establish comms please ?, I'm liam.skinner@durham.ac.uk, just I need some advise please ?
Gareth Douglas Roy: (11:26 AM)
Hi Liam, easiest thing is to email uki-scotgrid-glasgow@physics.gla.ac.uk that'll get to all of us
Liam Skinner: (11:28 AM)
@Gareth, many thanks - expect a mail soon!
Ewan Mac Mahon: (11:30 AM)
I think there are two parts to this cvmfs switch:
- Make sure your config supports both the gridpp.ac.uk domain and the egi.eu domain,
- The for the VOs that are present in both, change their software area environment variable as set in grid-env.sh to point to the egi.eu version, and not the gridpp.ac.uk one.
For the really UK specific VO repos (e.g. southgrid.gridpp.ac.uk), they won't appear under the egi.eu path so their environment variables still need to point to the gridpp.ac.uk paths.
Catalin Condurache (RAL): (11:36 AM)
all UK sites should have and keep the configs for the gridpp.ac.uk domain
Brian Davies @RAL-LCG2: (11:43 AM)
Re slow transfer agian , not related to waitIO as problem seen also when servers are relatively quiet
Steve Jones: (11:47 AM)
mcore accounting turned on at Liverpool now. 
Ewan Mac Mahon: (11:52 AM)
:-)
we did the 'multicore' thing too because it was so easy, but we're still going to decomission it imminently.
Gareth Douglas Roy: (11:53 AM)
Apologies, I have another meeting I need to attend. Need to go thanks!
Ewan Mac Mahon: (11:53 AM)
And it was mostly that easy because Alessandra posted a very simple 'just do this' instruction to tb-support, so I just did that.
Tom Whyntie: (11:56 AM)
Have to leave now - thanks for the new VO updates, cheers, Tom
Brian Davies @RAL-LCG2: (12:02 PM)
What is the modfication key for the minutes upload?
Jeremy Coles: (12:03 PM)
dteam?
Ewan Mac Mahon: (12:03 PM)
Yay for IPv6 stuff.
It's probably a good idea for (at least) one person from eachh new IPv6 site to join the Hepix IPv6 working group list, if you haven't already.
Even if you don't want to do too much, it's a good way to keep in touch with issues and best practice from everyone else
Hmm.
It also might not though :-P