Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the Janet(UK) Community area.
Meeting URL
SeeVogh R.N. http://research.seevogh.com/join?meeting=M2MvMB2a2iDDDl929BDM92
EVO International http://evo.caltech.edu/evoNext/koala.jnlp?meeting=M2MvMB2a2iDDDl929BDM92
- Phone Bridge (+44 (0)161 306 6802.)
ID: 100 3161
Password: 4880
Apologies: David C
OPs team and sites meeting minutes Tuesday, 18 March 2014
Attendance
B Davies (Minutes)
A Forti
A McNab
A Washbrook
C Walker
D Traynor
D Bauer
E Korolkova
E MacMahon
E steele
G Qin
G Roy
G smith
G Songara
I Loader
J Coles
J Bland
J Hill
J Martyniak
M Mitchell
M Slater
M Doidge
M Kasif
QMUL
R Nandakumar
R Foy
R Frank
S Skipsey
S Jones
------------------------------------------------------------
Experiment problems/issue
- LHCb
Raja Nandakumar LHCb running MC simulation and restripping.
Mostly smooth going across all sites.
Bristol alone having "failed" pilots - though the jobs are still running.
Investigation of this is ongoing.
- CMS
Nothing to Report
- ATLAS
https://indico.cern.ch/event/291931/contribution/0/material/slides/0.pdf
KM Oxford have new ARC CE; can this be tested please?
- Other
------------------------------------------------------------
Meetings & updates (20')
General updates
The GridPP website was upgraded over the weekend (it is now SHA-2 ready). Please inform Andrew if you encounter any problems with it.
Last week there was a pre-GDB on batch systems and a GDB. The March GDB meeting summary is now available. The GDB actions list has been updated.
There was a GridPP Strategic Review yesterday. Recommendations will be shared with the PMB and CB in due course.
The CERN VOMS service will move to new hosts whose host certificates are signed by the new (SHA-2) CERN CA during 2014. First VOMS-aware services in WLCG need to be aware of these hosts. See the EGI broadcast message which indicates a timeline of 6th May for services to be updated.
The GridPP IPv6 site status table has been updated to provide an 'allocation' column. This follows discussion of allocation strategies across sites. This IETF document may be of interest.
EGI invites entries to win a funded trip to the community forum in May.
The next WLCG middleware readiness WG meeting takes place this afternoon at 13:30 UK time. There are pre-meeting updates in the wiki.
Alarms (and then tickets) are now being raised against EMI-2 services at sites. Please respond to the tickets quickly and indicate your plans for removing the EMI-2 based node - replies are expected within 10 working days. If you believe alarms/tickets result from false-positives then please indicate this in the ticket... we are aware of some examples already.
France is deploying an EGI wide Dirac instance for 'other communities' and EGI is considering to include this as part of a H2020 proposal.
CVMFS v2.1.17 was recently released. Ian reported that RAL T1 had been using it stably for several weeks.
The EGI availability/reliability figures for February have been added to the reports wiki page. The UK services show 100%. Ops for NGI_UK is 98%:98%. No sites were below the EGI 70% target
---------------------------
Tier-1 - Status Page
The Tier1 moved to use the new site firewall yesterday (Monday 17th March). As announced FTSs (2 & 3) were drained/stopped around the change and Batch jobs were not started during it. There was one problem after the change that affected the LFC in particular and external access from many worker nodes but that was fixed in around an hour.
The software server used by the small VOs will be withdrawn from service (aiming for June).
A replacement MyProxy server has been put into production (to resolve the MyProxy issues raised in GGUS ticket 97025). This new service is called myproxy.gridpp.rl.ac.uk. Sites and VOs need to make appropriate reconfigurations to use this. We plan to turn the old one (lcgrbp01.gridpp.rl.ac.uk) off at the end of March.
The Atlas file deletion rate has been improved. However, there is still a problem that needs to be understood.
There have been problems with the CMS Castor instance at the end of last week and the start of this. The Castor /Database teams have some ideas for the cause of this which looks to be load related. Work is ongoing.
EMI-3 WN has been rolled out to one batch of WNs which have behaved OK so far.
There is a UPS/Generator load test tomorrow morning (Wednesday 19th March). A 'warning' has been announced in the GOC DB.
---------------------------
Storage & Data Management - Agendas/Minutes
Chris noticed some of Steve's tests failing. At IC this related to a full spacetoken. Bristol is not working as there is no SCRATCHDISK spacetoken. Durham fails with a no space left on device error message.
---------------------------
Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06
A review of the HEPSPEC page reveals that most sites now have SL6 entries with the exceptions being: UCL-HEP; Manchester and RALPP. We will be checking HS06 figures in the 2014 Q1 quarterly reports so the deadline for action is 31st March 2014.
APEL publishing appears up-to-date for all sites.
---------------------------
Documentation - KeyDocs
Keydocs action needed by: Mark M; Jens J; Rob H/Security T; Alessandra F; Wahid B; David C and Matt D.
---------------------------
On-duty - Dashboard ROD rota
Many EMI-2 tickets created during the week. Some false positives add to confusion!
Tickets outstanding (as on 15th March) for Brunel (2); Oxford - Although system in downtime until yesterday and Sheffield.
---------------------------
Rollout Status WLCG Baseline
The EMI-2 decommissioning task has started.
The next WLCG middleware readiness WG meeting takes place this afternoon at 13:30 UK time.
---------------------------
Services - PerfSonar dashboard | GridPP VOMS
A reminder of this perfSONAR overview for UK sites. Username is WLCGps. Currently shows a number of problems that need to be addressed at various sites.
---------------------------
Tickets
47 Open UK tickets this week, a dozen of them are EMI2 retirement tickets so they'll get the lion's share of our attention.
NGI
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101502 (24/2)
The ILC software area move ticket. IC, RAL, QMUL, Cambridge and Liverpool have moved. Lancaster moved but is (hopefully was) broken for ILC (pardon my noise). Assuming that anyone not mentioned on the ticket hasn't migrated the ILC SW_DIR yet that leaves the following list of uk sites to migrate their ILC software area:
OXFORD
GLASGOW
BRISTOL
RALPP
BIRMINGHAM
BRISTOL
BRUNEL - Moved but hadn't updated the ticket
RHUL
DURHAM
MANCHESTER
(I might have missed some of you out, this list is from lcg-infosites and grepped using my admittedly poor eyeballs. A braver man would have crafted his own ldapsearch to glean this info).
If you don't want to make the change then removing support for ILC is a viable course of action, if you have made the change please update the ticket to let ILC know. In progress (17/3)
EMI ALARMS
Remember that you need to have *at least* your upgrade plans in these tickets within a fortnight of the ticket's submission - so by the 28th of March.
SUSSEX
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102212 (14/3)
BDII is the culprit here, ticket acknowledged but no other news (or plan). In Progress (17/3)
RALPP
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102207 (14/3)
ARC CE, a few CREAMs, site BDII and WNs - Chris reports that this is probably a false alarm, but is looking into it (in case the publishing is off). Chris has included a plan for the other components (if I'm reading right, is RALPP ditching all CREAMs?). In progress (14/3)
BRISTOL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102205 (14/3)
site BDII (seems to be a common one), some CEs and the WNs. Winnie has posted an assurance that the upgrade will be done in time, but I'm not sure if that'll count as a plan to the powers that be. In progress (17/3)
GLASGOW
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102202 (14/3)
Lots of services, but Dave has given a detailed upgrade battleplan. In progress (17/3)
EDINBURGH
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102201 (14/3)
I think site-BDII and WNs. Wahid has given assurances, but (sorry to be a pedantic patsy) not sure if that'll count as a "upgrade plan". In other news the testing for the SL6 WN tarball is going well. In progress (14/3)
DURHAM
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102199 (14/3)
The Durham DPM, CE and site BDII are on the list. Assigned (14/3)
SHEFFIELD
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102197 (14/3)
Some CEs and the APEL box (that's a guess). Elena has given a good plan, there's some hassle as their APEL and BDII box are shared. In progress (14/3)
UCL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102193 (14/3)
DPM, BDII, CE and WNs. Ben has said he will upgrade in the next few weeks. On hold (14/3)
RHUL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102189 (14/3)
A CREAM and the site BDII. Govind is planning his upgrade plans. In progress (14/3)
IMPERIAL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102185 (14/3)
Just some WNs. Engaged in testing and plan to upgrade the last cluster this week. In progress (14/3)
BRUNEL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102184 (14/3)
Just the DPM I think. Henry remarks that they're just about to embark on a physical server move and doesn't want to change anything significant before the move. I gave my recipe for the EMI3 dpm move in case it helps. In progress (14/3)
EFDA-JET
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102166 (14/3)
Again just the DPM. Acknowledged, but no plan. In progress (17/3)
I'm actually pretty sure Lancaster should have got a ticket as we still have one cluster on the EMI2 tarball. I'm not going to complain though.
NORMAL TICKETS
QMUL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101639 (26/2)
This ticket about jobs using RFC3820 style proxies not working at QM is in an odd state. The user seemed to be confused as to what feedback he should give. In progress (17/3)
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101916 (8/3)
Sorry to be picking on QM, but this 444444 publishing jobs ticket is looking neglected, I suspect you've been frying bigger fish but can you please show it (or even better, the underlaying issue!) some love. Is this linked to your other information publishing problems? In progress (10/7)
UCL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101285 (16/2)
UCL Perfsonar ticket. After having his perfsonar box whacked by a power outage Ben is reinstalling, but is seeing some odd hardware issues. Has anyone else seen their R610s PERCs play up like this (only showing 3G partitions)? In progress (12/3)
MANCHESTER
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102394 (18/3)
Just in, but similar to the ILC ticket - Catalin has asked Manchester to deploy cvmfs for t2k. In progress (18/3)
---------------------------
Tools - MyEGI Nagios
Monday 17th March
The GridPP DIRAC service is now able to submit jobs to VMs on Vac sites. Vac configuration for GridPP DIRAC explains how to configure a Vac site to run GridPP DIRAC jobs in VMs. More volunteer sites would be useful.
EGI has published a new roadmap for operations tools.
------------------------------------------------------------
March GDB overview
Summary notes are here:
http://indico.cern.ch/event/291931/contribution/4/material/0/0.pdf
Agenda: http://indico.cern.ch/event/272619/other-view?view=standard
WLCG summary: https://twiki.cern.ch/twiki/bin/view/LCG/GDBMeetingNotes20140312
------------------------------------------------------------
Summary of pre-GDB on batch systems (10')
Agenda: http://indico.cern.ch/event/272785/
------------------------------------------------------------
AOB
* Jeremy is still waiting on several sites to respond to the email of 10th March (11:45).
* Please complete the poll regarding GridPP34 being in Ambleside on 19th August: http://doodle.com/bgcdgghasqatbbq5
* There is no ops meeting next week due to GridPP33.
Mark Mitchell Last meeting .
------------------------------------------------------------
Chat Window:
[11:01:48] Mark Mitchell yep can hear you
[11:02:46] Brian Davies i am speaking
[11:02:46] Sam Skipsey Brian is speaking and audible
[11:05:44] Matt Doidge No news is good news?
[11:05:49] Jeremy Coles Brian is taking minutes this week. Thanks Brian.
[11:06:11] Daniela Bauer No news = "we haven't got a clue what is going on" ?!
[11:06:44] Christopher Walker Echo was my fault. 2 speakers speaking
[11:10:40] Brian Davies For minutes: Who is on SIP , QMUL and hepvc please?
[11:11:20] Brian Davies Raja I was still tseting up for minutes Can you text what you want ion minutes of rLHCb report
[11:11:48] Raja Nandakumar LHCb running MC simulation and restripping.
[11:11:55] Raja Nandakumar Mostly smooth going across all sites.
[11:12:31] Raja Nandakumar Bristol alone having "failed" pilots - though the jobs are still running. Investigation of this is ongoing.
[11:15:44] Ewan Mac Mahon I think we were considering the argument that a T2 (as in a regional T2) has a 40GB link in aggregate already.
[11:15:57] Ewan Mac Mahon I.e. 4x sites ~10Gb each. Ish.
[11:16:07] Sam Skipsey Well, depending on the regional Tier 2, Ewan
[11:16:12] Christopher Walker I'm on hepvc, QMUL and there in my own right.
[11:16:22] Mark Mitchell Which is what I was going to say
[11:16:33] Ewan Mac Mahon Indeed, but realisticly, it's clearly scales with the resources.
[11:17:01] Mark Mitchell Big ask. What is the JANET backbone speed into Europe?
[11:17:05] Ewan Mac Mahon I think we only really have the one T2 with a lot of resources and a bit of a potential network bottleneck.
[11:18:04] Ewan Mac Mahon It also depends where you want the data to go; if it's for internal suffling within the cloud (e.g. always-on-FAX) then the international links are less important.
[11:18:36] Alessandra Forti NNW was on 20GB only few months ago iI'd like to see their laugh if I ask for 40GB
[11:18:42] Mark Mitchell The figures from the network team at Glasgow are interesting
[11:18:58] Christopher Walker QMUL routinely peaks at 8 GBit.
[11:19:08] Mark Mitchell As do we Chris
[11:19:15] Ewan Mac Mahon I suspect that we're in pretty good shape for bandwidth per HS06/disk compared to the big US T2s already though, and this is a worry over nothing.
[11:19:27] Mark Mitchell What is of interest is the Glasgow network Team figures on Network use
[11:19:56] Mark Mitchell We are the largest research user but we pale into insignificance in comparasion to I-Tunes and social Media Traffic
[11:20:08] Matt Doidge I'll conver ILC in the tickets section.
[11:20:13] Matt Doidge *cover
[11:20:35] Mark Mitchell From JANET
[11:20:37] Mark Mitchell Our backbone runs at 100Gbit/s (with an interconnect capacity of around 40Gbit/s)
[11:21:02] Ewan Mac Mahon 'Our' = your cluster, or your university backbone?
[11:21:23] Mark Mitchell So the Tier-2s would be University Scale (ish) I would say Ewan's suggestion is sensible as it means we could aggregate the Tier-2s
[11:21:33] Mark Mitchell Sorry Ewan that is from the JANET website
[11:21:43] Mark Mitchell https://www.ja.net/products-services/janet-connect/janet-connections
[11:21:48] Ewan Mac Mahon Oh, right - Janet's backbone.
[11:22:21] Mark Mitchell Glasgow's core backbone is multiple 40 Gigs in a series of ring topologies which are using Spanning tree to affect failover.
[11:22:29] Christopher Walker QMUL that isn't the case - our traffic is more than twice the rest of the university
[11:23:15] Ewan Mac Mahon I'm a bit surprised that you're shifting so much more itunes etc. than grid traffic; I'm fairly sure Oxford doesn't do that much of that. I wonder if we're just giving all of our students really crap connections.
[11:23:44] Mark Mitchell What Colin Cooper did was take the annual aggregate of traffic across the campus. We are moving to a dedicated dual layer 3 connection which will change this as we are still technically in a shared environment at present.
[11:23:48] Ewan Mac Mahon If we were saturating the grid link a lot, we'd dominate,
[11:24:34] Ewan Steele so we have been told we can have multiple /48 blocks
[11:24:45] Ewan Mac Mahon One would do
[11:25:05] Mark Mitchell For a long, long time
[11:25:08] Mark Mitchell
[11:25:27] Ewan Steele sorry not /48 multi /64
[11:25:36] Ewan Steele just a small difference
[11:26:10] Ewan Mac Mahon TBH, for a cluster, I'd say a /64 isn't enough because you can't subnet it, a /56 is more than plenty. Bigger than that is total waste.
[11:26:46] Ewan Steele Ive been discouraged from asking for a /56 but I can put my foot down if people think its worth it
[11:26:52] Ewan Mac Mahon Multiple /64s sounds a bit silly, but probably not a total catastrophe.
[11:27:10] Ewan Mac Mahon Somewhere between a /56 and a /64 would do.
[11:27:13] Mark Mitchell Block allocations and prefix tables here: http://www.ripe.net/internet-coordination/press-centre/understanding-ip-addressing
[11:27:28] Ewan Steele our networks team have said a /56 for us would be a bit of a waste
[11:27:54] Ewan Mac Mahon A /56 gives you 256 subnets, whereas something between the two could give you several, but not as many.
[11:28:58] Ewan Mac Mahon e.g. a /61 would give you eight subnets, which seems fairly manageable.
[11:29:27] Ewan Steele yeah I think 256 subnets is a little excesive a /61 sounds like a good plan. I might request it even if I cant route the traffic yet
[11:30:16] John Hill We've been allocated a /64, with the option of increasing it to a /60 if necessary.
[11:30:37] Ewan Mac Mahon The argument in favour of having a /56 isn't so much that you need it, but that it might be simpler for a University to give everyone uniform allocations - it's simpler to manage if it's all equal sized blocks, but that's for the university networking people to worry about.
[11:31:12] Ewan Mac Mahon I would argue with anyone that suggested allocating a /64 - it seems to me to betray a fundamental misunderstanding of how to wield IPv6 in practice.
[11:31:32] John Hill That's basically the view here: the University is allocating /56 to departments whatever their size
[11:31:37] Ewan Mac Mahon You don't want to be putting stateless autoconfiguraton addresses in your DNS, for example. That way lies much hassle.
[11:32:56] Alessandra Forti with a /48 there isn't much the universities can do but give /64
[11:32:57] Daniel Traynor janet has a /32 in ipv6 space, that means it can make 65,536 /48 or 16,777,216 /56
[11:33:08] Daniel Traynor alocations
[11:33:25] Ewan Mac Mahon Indeed, but only 4096 /44s.
[11:33:26] Alessandra Forti /48 = only 256 /56
[11:34:03] Christopher Walker http://pprc.qmul.ac.uk/~lloyd/gridpp/nettest.html
[11:34:13] Ewan Mac Mahon This is why Oxford asked for more than a /48 - we have more than 200 'units' already.
[11:34:31] Ewan Mac Mahon But Oxford is a larger and more complex institution than a lot of places.
[11:34:44] Alessandra Forti I'm sure Manchester has the same problem but haven't made the argument as convincingly as oxford
[11:35:07] Ewan Mac Mahon Indeed.
[11:35:37] Ewan Steele wht are we testing networking if we have perfsonar??
[11:35:42] Ewan Steele why*
[11:36:08] Duncan Rand it predates perfsonar
[11:36:49] Duncan Rand plus it tests something subtly different WN-SE rather than perfsonar-perfsonar
[11:36:53] Raja Nandakumar It was to all SEs from the WN. Not just to RAL
[11:36:57] Ewan Mac Mahon It's also this difference between types of endpoints. perfSonar is usually located equivelently to the SE (more or less) whereas WNs are quite often behind NATs etc.
[11:37:15] Ewan Mac Mahon And on slower switches.
[11:37:16] Sam Skipsey Although, to be fair, there are lots of things that are different to perfsonar that we retired.
[11:37:34] Sam Skipsey (mostly with arguments about how they were custom and no-one else used them)
[11:38:21] Ewan Mac Mahon This is true. I think there's an argument for picking the good bits out of Steve's tests and having them added to the central VO tests (they're not uniquely beneficial to the UK) and retiring the rest.
[11:41:11] Steve Jones https://www.gridpp.ac.uk/wiki/ARGUS_deployment
[11:42:04] Steve Jones https://www.gridpp.ac.uk/wiki/Argus_Server#Configuring_Argus_for_Central_Banning
[11:43:18] John Hill " perfSONAR overview" in the bulletin points to www.example.com
[11:46:03] Chris Brew RALPP has moved
[11:46:05] Jeremy Coles I've added the correct link now. Sorry! It is http://tinyurl.com/o3f3ntr The username is WLCGps and pwd WLCG
[11:49:08] Pete Gronbech ILC are the biggest VO after the LHC VOs over the GridPP4 project so far
[11:58:13] Ewan Mac Mahon He vanished.
[12:02:31] Ewan Mac Mahon IIRC we specified 5yr warranties on the perfSonar boxes, so if it looks like the hardware's playing up, he should probably just poke Dell support; it's what they're for.
[12:02:42] Jeremy Coles On perfSONAR. The sites with tickets still appear in the dashboard with problems: UCL-HEP; RALPP; ECDF; Brunel; Lancaster and Sheffield.
[12:03:00] Elena Korolkova What's the ticket number for t2k/Man?
[12:03:06] Ewan Steele we have just had dell support out to replace the motherboard on ours :/
[12:03:18] Matt Doidge Elena - https://ggus.eu/index.php?mode=ticket_info&ticket_id=102394
[12:03:27] Jeremy Coles Sites without tickets but showing CRIT in the dashboard: Birmingham; Liverpool; Imperial; RHUL and Oxford.
[12:03:41] Elena Korolkova Thanks, Matt
[12:03:50] John Bland ours is because they're monitoring on port 80 for web access, but we only had port 443 open
[12:04:18] Duncan Rand I will ask Shawn about that
[12:04:49] John Bland i've asked to open up port 80, but I would have thought the secure link would have been the default
[12:04:57] Ewan Steele I think I can probably close my ticket Duncan?
[12:05:12] Jeremy Coles Yes. Durham is green from the dashboard.
[12:06:08] Duncan Rand close it please Ewan
[12:06:32] Ewan Mac Mahon Looks like the ps problem at Oxford is that we're on 3.3.1 rather than 3.3.2 - if that's right I think that should be an easy fix with a yum update.
[12:08:38] Ewan Mac Mahon Great - this is the critical bit I've missed.
[12:08:47] Andrew McNab https://www.gridpp.ac.uk/wiki/Quick_Guide_to_Dirac
[12:08:58] Ewan Mac Mahon I thought there was such a thing, but I've defitely not read it.
[12:10:58] Ewan Mac Mahon I am in the GridPP VO, so sounds like that should work for me straight off.
[12:11:05] Ewan Mac Mahon I'll give it a go when I can.
[12:11:14] Ewan Mac Mahon Will do
[12:11:49] Ewan Mac Mahon Vac is a Good Thing(TM) IMO; it's well worth a look at.
[12:13:17] Ewan Mac Mahon It's got most of the good bits or 'cloud' type computing done in a way that's a better fit for High Throughput use than the normal virtual infrastructure focussed APIs.
[12:13:17] Andrew McNab Ewan: you are enabled as a user in the GridPP DIRAC in the default group so yes it should just work for you
[12:14:22] Ewan Mac Mahon Frankly, I think vac should see the sort of adoption that cvmfs did - it's just clearly the better way to do things now.
[12:14:37] Ewan Mac Mahon *jumps up and down waving pom-poms*
[12:16:00] Sam Skipsey Is it still a relatively static allocation of VMs, Ewan?
[12:16:12] Sam Skipsey (Also, does it support people who want to do things like MPI?)
[12:16:47] Ewan Mac Mahon Depends what you mean by static. You give it a target ratio of VM types in its config file
[12:16:59] Ewan Mac Mahon (but you can tweak that, it re-reads changed configs at runtime)
[12:17:18] Sam Skipsey Right, but how does it adapt to having no work of (say) the largest type?
[12:17:28] Ewan Mac Mahon and if you want (say) 50% LHCb VMs, but there's no LHCb work, it'll ease off running those and do something else.
[12:17:40] Sam Skipsey ...by "ease off"...
[12:17:48] Ewan Mac Mahon The shutdown scripts are supposed to quit with an exit code, one of which is 'there's no work'
[12:17:58] Sam Skipsey Okay, but how fast is that?
[12:18:12] Sam Skipsey That seems wasteful compared to conventional batch scheduling.
[12:18:33] Ewan Mac Mahon As for MPI; it supports multicore VMs now, doing communication between VMs would be difficult/impossible/out of scopoe.
[12:18:37] Sam Skipsey (but then, VM startup and shutdown is also wasteful)
[12:19:05] Ewan Mac Mahon And it's not that slow; you're looking at a few minutes, so it's on the order of time an idle pilot takes to quit on an ordinary batch system.
[12:19:07] Sam Skipsey I note that supporting multicore VMs is very different to supporting MPI.
[12:19:08] Ewan Mac Mahon More or less.
[12:19:27] Andrew McNab it uses less CPU than having a dedicated headnode, because the info about VMs that "fizzle" without getting any work is shared across the whole Vac space.
[12:20:02] Sam Skipsey (and, in particular, MPI support needs orchestration of VMs with an explicit nodelist, which is I guess not something that the Vac needs to worry about at the moment)
[12:20:44] Sam Skipsey ...I'm not sure I understand your point, Andrew.
[12:21:08] Sam Skipsey That's only relevant if all of your users are using pilot frameworks, surely?
[12:22:00] Andrew McNab the "waste" / cost of running a VM to see if there is any work is less than the cost of maintaining a substantial multicore headnode with the batch / gateway stuff on it in a conventional CREAM/batch site
[12:22:38] Sam Skipsey Is this research published anywhere?
[12:22:54] Sam Skipsey And does it still apply to the many small VOs who don't use pilot models?
[12:23:04] Sam Skipsey (or Dirac)
[12:23:06] Andrew McNab the assumption is that the VOs are using a pilot framework of some kind, which is all of LHC, and the DIRAC ones like biomed, ILC, etc
[12:23:18] Sam Skipsey But not lots of other VOs.
[12:23:25] Ewan Mac Mahon And in the future, everyone else too. The WMS is doomed.
[12:23:42] Ewan Steele I'm sorry I need to go.
[12:23:42] Sam Skipsey ...the WMS being doomed does not mean pilot models are better solutions.
[12:23:55] Sam Skipsey That's clearly a false equivalence if I've ever seen one.
[12:24:21] Sam Skipsey (Although I agree that if *everyone* agreed on a common pilot framework, we might be getting somewhere.)
[12:25:10] Ewan Mac Mahon Well it sounds like both GridPP and apparantly EGI are pushing towards dirac for 'other' VOs, so I think we're more or less getting there on that front.
[12:25:24] Ewan Mac Mahon It's not clear that there's much of an alternative that actually exists atm.
[12:25:26] Christopher Walker https://twiki.cern.ch/twiki/bin/view/LCG/GDBMeetingNotes20140311
[12:25:31] Sam Skipsey Sure, but I'd prefer it if ATLAS and CMS held their noses and used it too
[12:25:58] Sam Skipsey Plus, it would be horribly inefficient for our local users, who we do have.
[12:26:24] Ewan Mac Mahon And as a transition thing, I don't think there's anything that would stop you using vac nodes as a client to a local batch system if you had other VO/local users, though.
[12:26:52] Andrew McNab there's also the possibility to put other things in the VMs that know to shutdown when they are idle. So batch worker nodes, condor nodes, even PROOF workers could be made into VMs and run this way as part of a Vac space.
[12:27:21] Sam Skipsey Well, in fact, for ATLAS and CMS, they *are* just using Condor glideins, effectively.
[12:27:38] Ewan Mac Mahon Depends what you want for the local users; if they're happy on a conventional cluster that's OK, but vac gives a lot of the benefits of cloudy stuff (isolation, choice of OS/environment etc.).
[12:27:39] Sam Skipsey (Which leads to my other question of how this is better than, say, Condor with the VM universe )
[12:28:46] Ewan Mac Mahon The problem with a lot of VI could systems is that they're really, really complicated. Plus the 'please start me a VM' model doesn't fit resources that are usually full terribly well.
[12:29:13] Sam Skipsey sure.
[12:29:24] Ewan Mac Mahon Anyway, I believe there's a talk on this at the GridPP meeting, so I expect some beer assisted thradhing out of this stuff.
[12:30:12] Ewan Mac Mahon But on a practical level, you can grab a spare WN and getting it running LHCb jobs on vac this week, and get a bit of a hands-on feel for it. It's very lightweight to get it going.
[12:34:13] Ewan Mac Mahon At this point, I'm pretty sure the pain of getting off torque/maui is less than the pain of staying.
[12:34:27] Ewan Mac Mahon It's a matter of short term pain vs long term, but still.
[12:34:29] Alessandra Forti I agree
[12:35:47] Ewan Mac Mahon I like the way Dave just organised a meeting in Pitlochry, but felt the need to Doodle poll Ambleside on the grounds that it's hard to get to.
[12:35:55] Ewan Mac Mahon Also, Oh Noes!
[12:35:59] Ewan Mac Mahon
[12:36:35] Alessandra Forti Good luck for the future!
[12:37:13] Mark Mitchell Thanks Alessandra
Attendance
B Davies (Minutes)
A Forti
A McNab
A Washbrook
C Walker
D Traynor
D Bauer
E Korolkova
E MacMahon
E steele
G Qin
G Roy
G smith
G Songara
I Loader
J Coles
J Bland
J Hill
J Martyniak
M Mitchell
M Slater
M Doidge
M Kasif
QMUL
R Nandakumar
R Foy
R Frank
S Skipsey
S Jones
------------------------------------------------------------
Experiment problems/issue
- LHCb
Raja Nandakumar LHCb running MC simulation and restripping.
Mostly smooth going across all sites.
Bristol alone having "failed" pilots - though the jobs are still running.
Investigation of this is ongoing.
- CMS
Nothing to Report
- ATLAS
https://indico.cern.ch/event/291931/contribution/0/material/slides/0.pdf
KM Oxford have new ARC CE; can this be tested please?
- Other
------------------------------------------------------------
Meetings & updates (20')
General updates
The GridPP website was upgraded over the weekend (it is now SHA-2 ready). Please inform Andrew if you encounter any problems with it.
Last week there was a pre-GDB on batch systems and a GDB. The March GDB meeting summary is now available. The GDB actions list has been updated.
There was a GridPP Strategic Review yesterday. Recommendations will be shared with the PMB and CB in due course.
The CERN VOMS service will move to new hosts whose host certificates are signed by the new (SHA-2) CERN CA during 2014. First VOMS-aware services in WLCG need to be aware of these hosts. See the EGI broadcast message which indicates a timeline of 6th May for services to be updated.
The GridPP IPv6 site status table has been updated to provide an 'allocation' column. This follows discussion of allocation strategies across sites. This IETF document may be of interest.
EGI invites entries to win a funded trip to the community forum in May.
The next WLCG middleware readiness WG meeting takes place this afternoon at 13:30 UK time. There are pre-meeting updates in the wiki.
Alarms (and then tickets) are now being raised against EMI-2 services at sites. Please respond to the tickets quickly and indicate your plans for removing the EMI-2 based node - replies are expected within 10 working days. If you believe alarms/tickets result from false-positives then please indicate this in the ticket... we are aware of some examples already.
France is deploying an EGI wide Dirac instance for 'other communities' and EGI is considering to include this as part of a H2020 proposal.
CVMFS v2.1.17 was recently released. Ian reported that RAL T1 had been using it stably for several weeks.
The EGI availability/reliability figures for February have been added to the reports wiki page. The UK services show 100%. Ops for NGI_UK is 98%:98%. No sites were below the EGI 70% target
---------------------------
Tier-1 - Status Page
The Tier1 moved to use the new site firewall yesterday (Monday 17th March). As announced FTSs (2 & 3) were drained/stopped around the change and Batch jobs were not started during it. There was one problem after the change that affected the LFC in particular and external access from many worker nodes but that was fixed in around an hour.
The software server used by the small VOs will be withdrawn from service (aiming for June).
A replacement MyProxy server has been put into production (to resolve the MyProxy issues raised in GGUS ticket 97025). This new service is called myproxy.gridpp.rl.ac.uk. Sites and VOs need to make appropriate reconfigurations to use this. We plan to turn the old one (lcgrbp01.gridpp.rl.ac.uk) off at the end of March.
The Atlas file deletion rate has been improved. However, there is still a problem that needs to be understood.
There have been problems with the CMS Castor instance at the end of last week and the start of this. The Castor /Database teams have some ideas for the cause of this which looks to be load related. Work is ongoing.
EMI-3 WN has been rolled out to one batch of WNs which have behaved OK so far.
There is a UPS/Generator load test tomorrow morning (Wednesday 19th March). A 'warning' has been announced in the GOC DB.
---------------------------
Storage & Data Management - Agendas/Minutes
Chris noticed some of Steve's tests failing. At IC this related to a full spacetoken. Bristol is not working as there is no SCRATCHDISK spacetoken. Durham fails with a no space left on device error message.
---------------------------
Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06
A review of the HEPSPEC page reveals that most sites now have SL6 entries with the exceptions being: UCL-HEP; Manchester and RALPP. We will be checking HS06 figures in the 2014 Q1 quarterly reports so the deadline for action is 31st March 2014.
APEL publishing appears up-to-date for all sites.
---------------------------
Documentation - KeyDocs
Keydocs action needed by: Mark M; Jens J; Rob H/Security T; Alessandra F; Wahid B; David C and Matt D.
---------------------------
On-duty - Dashboard ROD rota
Many EMI-2 tickets created during the week. Some false positives add to confusion!
Tickets outstanding (as on 15th March) for Brunel (2); Oxford - Although system in downtime until yesterday and Sheffield.
---------------------------
Rollout Status WLCG Baseline
The EMI-2 decommissioning task has started.
The next WLCG middleware readiness WG meeting takes place this afternoon at 13:30 UK time.
---------------------------
Services - PerfSonar dashboard | GridPP VOMS
A reminder of this perfSONAR overview for UK sites. Username is WLCGps. Currently shows a number of problems that need to be addressed at various sites.
---------------------------
Tickets
47 Open UK tickets this week, a dozen of them are EMI2 retirement tickets so they'll get the lion's share of our attention.
NGI
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101502 (24/2)
The ILC software area move ticket. IC, RAL, QMUL, Cambridge and Liverpool have moved. Lancaster moved but is (hopefully was) broken for ILC (pardon my noise). Assuming that anyone not mentioned on the ticket hasn't migrated the ILC SW_DIR yet that leaves the following list of uk sites to migrate their ILC software area:
OXFORD
GLASGOW
BRISTOL
RALPP
BIRMINGHAM
BRISTOL
BRUNEL - Moved but hadn't updated the ticket
RHUL
DURHAM
MANCHESTER
(I might have missed some of you out, this list is from lcg-infosites and grepped using my admittedly poor eyeballs. A braver man would have crafted his own ldapsearch to glean this info).
If you don't want to make the change then removing support for ILC is a viable course of action, if you have made the change please update the ticket to let ILC know. In progress (17/3)
EMI ALARMS
Remember that you need to have *at least* your upgrade plans in these tickets within a fortnight of the ticket's submission - so by the 28th of March.
SUSSEX
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102212 (14/3)
BDII is the culprit here, ticket acknowledged but no other news (or plan). In Progress (17/3)
RALPP
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102207 (14/3)
ARC CE, a few CREAMs, site BDII and WNs - Chris reports that this is probably a false alarm, but is looking into it (in case the publishing is off). Chris has included a plan for the other components (if I'm reading right, is RALPP ditching all CREAMs?). In progress (14/3)
BRISTOL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102205 (14/3)
site BDII (seems to be a common one), some CEs and the WNs. Winnie has posted an assurance that the upgrade will be done in time, but I'm not sure if that'll count as a plan to the powers that be. In progress (17/3)
GLASGOW
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102202 (14/3)
Lots of services, but Dave has given a detailed upgrade battleplan. In progress (17/3)
EDINBURGH
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102201 (14/3)
I think site-BDII and WNs. Wahid has given assurances, but (sorry to be a pedantic patsy) not sure if that'll count as a "upgrade plan". In other news the testing for the SL6 WN tarball is going well. In progress (14/3)
DURHAM
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102199 (14/3)
The Durham DPM, CE and site BDII are on the list. Assigned (14/3)
SHEFFIELD
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102197 (14/3)
Some CEs and the APEL box (that's a guess). Elena has given a good plan, there's some hassle as their APEL and BDII box are shared. In progress (14/3)
UCL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102193 (14/3)
DPM, BDII, CE and WNs. Ben has said he will upgrade in the next few weeks. On hold (14/3)
RHUL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102189 (14/3)
A CREAM and the site BDII. Govind is planning his upgrade plans. In progress (14/3)
IMPERIAL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102185 (14/3)
Just some WNs. Engaged in testing and plan to upgrade the last cluster this week. In progress (14/3)
BRUNEL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102184 (14/3)
Just the DPM I think. Henry remarks that they're just about to embark on a physical server move and doesn't want to change anything significant before the move. I gave my recipe for the EMI3 dpm move in case it helps. In progress (14/3)
EFDA-JET
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102166 (14/3)
Again just the DPM. Acknowledged, but no plan. In progress (17/3)
I'm actually pretty sure Lancaster should have got a ticket as we still have one cluster on the EMI2 tarball. I'm not going to complain though.
NORMAL TICKETS
QMUL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101639 (26/2)
This ticket about jobs using RFC3820 style proxies not working at QM is in an odd state. The user seemed to be confused as to what feedback he should give. In progress (17/3)
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101916 (8/3)
Sorry to be picking on QM, but this 444444 publishing jobs ticket is looking neglected, I suspect you've been frying bigger fish but can you please show it (or even better, the underlaying issue!) some love. Is this linked to your other information publishing problems? In progress (10/7)
UCL
https://ggus.eu/index.php?mode=ticket_info&ticket_id=101285 (16/2)
UCL Perfsonar ticket. After having his perfsonar box whacked by a power outage Ben is reinstalling, but is seeing some odd hardware issues. Has anyone else seen their R610s PERCs play up like this (only showing 3G partitions)? In progress (12/3)
MANCHESTER
https://ggus.eu/index.php?mode=ticket_info&ticket_id=102394 (18/3)
Just in, but similar to the ILC ticket - Catalin has asked Manchester to deploy cvmfs for t2k. In progress (18/3)
---------------------------
Tools - MyEGI Nagios
Monday 17th March
The GridPP DIRAC service is now able to submit jobs to VMs on Vac sites. Vac configuration for GridPP DIRAC explains how to configure a Vac site to run GridPP DIRAC jobs in VMs. More volunteer sites would be useful.
EGI has published a new roadmap for operations tools.
------------------------------------------------------------
March GDB overview
Summary notes are here:
http://indico.cern.ch/event/291931/contribution/4/material/0/0.pdf
Agenda: http://indico.cern.ch/event/272619/other-view?view=standard
WLCG summary: https://twiki.cern.ch/twiki/bin/view/LCG/GDBMeetingNotes20140312
------------------------------------------------------------
Summary of pre-GDB on batch systems (10')
Agenda: http://indico.cern.ch/event/272785/
------------------------------------------------------------
AOB
* Jeremy is still waiting on several sites to respond to the email of 10th March (11:45).
* Please complete the poll regarding GridPP34 being in Ambleside on 19th August: http://doodle.com/bgcdgghasqatbbq5
* There is no ops meeting next week due to GridPP33.
Mark Mitchell Last meeting .
------------------------------------------------------------
Chat Window:
[11:01:48] Mark Mitchell yep can hear you
[11:02:46] Brian Davies i am speaking
[11:02:46] Sam Skipsey Brian is speaking and audible
[11:05:44] Matt Doidge No news is good news?
[11:05:49] Jeremy Coles Brian is taking minutes this week. Thanks Brian.
[11:06:11] Daniela Bauer No news = "we haven't got a clue what is going on" ?!
[11:06:44] Christopher Walker Echo was my fault. 2 speakers speaking
[11:10:40] Brian Davies For minutes: Who is on SIP , QMUL and hepvc please?
[11:11:20] Brian Davies Raja I was still tseting up for minutes Can you text what you want ion minutes of rLHCb report
[11:11:48] Raja Nandakumar LHCb running MC simulation and restripping.
[11:11:55] Raja Nandakumar Mostly smooth going across all sites.
[11:12:31] Raja Nandakumar Bristol alone having "failed" pilots - though the jobs are still running. Investigation of this is ongoing.
[11:15:44] Ewan Mac Mahon I think we were considering the argument that a T2 (as in a regional T2) has a 40GB link in aggregate already.
[11:15:57] Ewan Mac Mahon I.e. 4x sites ~10Gb each. Ish.
[11:16:07] Sam Skipsey Well, depending on the regional Tier 2, Ewan
[11:16:12] Christopher Walker I'm on hepvc, QMUL and there in my own right.
[11:16:22] Mark Mitchell Which is what I was going to say
[11:16:33] Ewan Mac Mahon Indeed, but realisticly, it's clearly scales with the resources.
[11:17:01] Mark Mitchell Big ask. What is the JANET backbone speed into Europe?
[11:17:05] Ewan Mac Mahon I think we only really have the one T2 with a lot of resources and a bit of a potential network bottleneck.
[11:18:04] Ewan Mac Mahon It also depends where you want the data to go; if it's for internal suffling within the cloud (e.g. always-on-FAX) then the international links are less important.
[11:18:36] Alessandra Forti NNW was on 20GB only few months ago iI'd like to see their laugh if I ask for 40GB
[11:18:42] Mark Mitchell The figures from the network team at Glasgow are interesting
[11:18:58] Christopher Walker QMUL routinely peaks at 8 GBit.
[11:19:08] Mark Mitchell As do we Chris
[11:19:15] Ewan Mac Mahon I suspect that we're in pretty good shape for bandwidth per HS06/disk compared to the big US T2s already though, and this is a worry over nothing.
[11:19:27] Mark Mitchell What is of interest is the Glasgow network Team figures on Network use
[11:19:56] Mark Mitchell We are the largest research user but we pale into insignificance in comparasion to I-Tunes and social Media Traffic
[11:20:08] Matt Doidge I'll conver ILC in the tickets section.
[11:20:13] Matt Doidge *cover
[11:20:35] Mark Mitchell From JANET
[11:20:37] Mark Mitchell Our backbone runs at 100Gbit/s (with an interconnect capacity of around 40Gbit/s)
[11:21:02] Ewan Mac Mahon 'Our' = your cluster, or your university backbone?
[11:21:23] Mark Mitchell So the Tier-2s would be University Scale (ish) I would say Ewan's suggestion is sensible as it means we could aggregate the Tier-2s
[11:21:33] Mark Mitchell Sorry Ewan that is from the JANET website
[11:21:43] Mark Mitchell https://www.ja.net/products-services/janet-connect/janet-connections
[11:21:48] Ewan Mac Mahon Oh, right - Janet's backbone.
[11:22:21] Mark Mitchell Glasgow's core backbone is multiple 40 Gigs in a series of ring topologies which are using Spanning tree to affect failover.
[11:22:29] Christopher Walker QMUL that isn't the case - our traffic is more than twice the rest of the university
[11:23:15] Ewan Mac Mahon I'm a bit surprised that you're shifting so much more itunes etc. than grid traffic; I'm fairly sure Oxford doesn't do that much of that. I wonder if we're just giving all of our students really crap connections.
[11:23:44] Mark Mitchell What Colin Cooper did was take the annual aggregate of traffic across the campus. We are moving to a dedicated dual layer 3 connection which will change this as we are still technically in a shared environment at present.
[11:23:48] Ewan Mac Mahon If we were saturating the grid link a lot, we'd dominate,
[11:24:34] Ewan Steele so we have been told we can have multiple /48 blocks
[11:24:45] Ewan Mac Mahon One would do
[11:25:05] Mark Mitchell For a long, long time
[11:25:08] Mark Mitchell
[11:25:27] Ewan Steele sorry not /48 multi /64
[11:25:36] Ewan Steele just a small difference
[11:26:10] Ewan Mac Mahon TBH, for a cluster, I'd say a /64 isn't enough because you can't subnet it, a /56 is more than plenty. Bigger than that is total waste.
[11:26:46] Ewan Steele Ive been discouraged from asking for a /56 but I can put my foot down if people think its worth it
[11:26:52] Ewan Mac Mahon Multiple /64s sounds a bit silly, but probably not a total catastrophe.
[11:27:10] Ewan Mac Mahon Somewhere between a /56 and a /64 would do.
[11:27:13] Mark Mitchell Block allocations and prefix tables here: http://www.ripe.net/internet-coordination/press-centre/understanding-ip-addressing
[11:27:28] Ewan Steele our networks team have said a /56 for us would be a bit of a waste
[11:27:54] Ewan Mac Mahon A /56 gives you 256 subnets, whereas something between the two could give you several, but not as many.
[11:28:58] Ewan Mac Mahon e.g. a /61 would give you eight subnets, which seems fairly manageable.
[11:29:27] Ewan Steele yeah I think 256 subnets is a little excesive a /61 sounds like a good plan. I might request it even if I cant route the traffic yet
[11:30:16] John Hill We've been allocated a /64, with the option of increasing it to a /60 if necessary.
[11:30:37] Ewan Mac Mahon The argument in favour of having a /56 isn't so much that you need it, but that it might be simpler for a University to give everyone uniform allocations - it's simpler to manage if it's all equal sized blocks, but that's for the university networking people to worry about.
[11:31:12] Ewan Mac Mahon I would argue with anyone that suggested allocating a /64 - it seems to me to betray a fundamental misunderstanding of how to wield IPv6 in practice.
[11:31:32] John Hill That's basically the view here: the University is allocating /56 to departments whatever their size
[11:31:37] Ewan Mac Mahon You don't want to be putting stateless autoconfiguraton addresses in your DNS, for example. That way lies much hassle.
[11:32:56] Alessandra Forti with a /48 there isn't much the universities can do but give /64
[11:32:57] Daniel Traynor janet has a /32 in ipv6 space, that means it can make 65,536 /48 or 16,777,216 /56
[11:33:08] Daniel Traynor alocations
[11:33:25] Ewan Mac Mahon Indeed, but only 4096 /44s.
[11:33:26] Alessandra Forti /48 = only 256 /56
[11:34:03] Christopher Walker http://pprc.qmul.ac.uk/~lloyd/gridpp/nettest.html
[11:34:13] Ewan Mac Mahon This is why Oxford asked for more than a /48 - we have more than 200 'units' already.
[11:34:31] Ewan Mac Mahon But Oxford is a larger and more complex institution than a lot of places.
[11:34:44] Alessandra Forti I'm sure Manchester has the same problem but haven't made the argument as convincingly as oxford
[11:35:07] Ewan Mac Mahon Indeed.
[11:35:37] Ewan Steele wht are we testing networking if we have perfsonar??
[11:35:42] Ewan Steele why*
[11:36:08] Duncan Rand it predates perfsonar
[11:36:49] Duncan Rand plus it tests something subtly different WN-SE rather than perfsonar-perfsonar
[11:36:53] Raja Nandakumar It was to all SEs from the WN. Not just to RAL
[11:36:57] Ewan Mac Mahon It's also this difference between types of endpoints. perfSonar is usually located equivelently to the SE (more or less) whereas WNs are quite often behind NATs etc.
[11:37:15] Ewan Mac Mahon And on slower switches.
[11:37:16] Sam Skipsey Although, to be fair, there are lots of things that are different to perfsonar that we retired.
[11:37:34] Sam Skipsey (mostly with arguments about how they were custom and no-one else used them)
[11:38:21] Ewan Mac Mahon This is true. I think there's an argument for picking the good bits out of Steve's tests and having them added to the central VO tests (they're not uniquely beneficial to the UK) and retiring the rest.
[11:41:11] Steve Jones https://www.gridpp.ac.uk/wiki/ARGUS_deployment
[11:42:04] Steve Jones https://www.gridpp.ac.uk/wiki/Argus_Server#Configuring_Argus_for_Central_Banning
[11:43:18] John Hill " perfSONAR overview" in the bulletin points to www.example.com
[11:46:03] Chris Brew RALPP has moved
[11:46:05] Jeremy Coles I've added the correct link now. Sorry! It is http://tinyurl.com/o3f3ntr The username is WLCGps and pwd WLCG
[11:49:08] Pete Gronbech ILC are the biggest VO after the LHC VOs over the GridPP4 project so far
[11:58:13] Ewan Mac Mahon He vanished.
[12:02:31] Ewan Mac Mahon IIRC we specified 5yr warranties on the perfSonar boxes, so if it looks like the hardware's playing up, he should probably just poke Dell support; it's what they're for.
[12:02:42] Jeremy Coles On perfSONAR. The sites with tickets still appear in the dashboard with problems: UCL-HEP; RALPP; ECDF; Brunel; Lancaster and Sheffield.
[12:03:00] Elena Korolkova What's the ticket number for t2k/Man?
[12:03:06] Ewan Steele we have just had dell support out to replace the motherboard on ours :/
[12:03:18] Matt Doidge Elena - https://ggus.eu/index.php?mode=ticket_info&ticket_id=102394
[12:03:27] Jeremy Coles Sites without tickets but showing CRIT in the dashboard: Birmingham; Liverpool; Imperial; RHUL and Oxford.
[12:03:41] Elena Korolkova Thanks, Matt
[12:03:50] John Bland ours is because they're monitoring on port 80 for web access, but we only had port 443 open
[12:04:18] Duncan Rand I will ask Shawn about that
[12:04:49] John Bland i've asked to open up port 80, but I would have thought the secure link would have been the default
[12:04:57] Ewan Steele I think I can probably close my ticket Duncan?
[12:05:12] Jeremy Coles Yes. Durham is green from the dashboard.
[12:06:08] Duncan Rand close it please Ewan
[12:06:32] Ewan Mac Mahon Looks like the ps problem at Oxford is that we're on 3.3.1 rather than 3.3.2 - if that's right I think that should be an easy fix with a yum update.
[12:08:38] Ewan Mac Mahon Great - this is the critical bit I've missed.
[12:08:47] Andrew McNab https://www.gridpp.ac.uk/wiki/Quick_Guide_to_Dirac
[12:08:58] Ewan Mac Mahon I thought there was such a thing, but I've defitely not read it.
[12:10:58] Ewan Mac Mahon I am in the GridPP VO, so sounds like that should work for me straight off.
[12:11:05] Ewan Mac Mahon I'll give it a go when I can.
[12:11:14] Ewan Mac Mahon Will do
[12:11:49] Ewan Mac Mahon Vac is a Good Thing(TM) IMO; it's well worth a look at.
[12:13:17] Ewan Mac Mahon It's got most of the good bits or 'cloud' type computing done in a way that's a better fit for High Throughput use than the normal virtual infrastructure focussed APIs.
[12:13:17] Andrew McNab Ewan: you are enabled as a user in the GridPP DIRAC in the default group so yes it should just work for you
[12:14:22] Ewan Mac Mahon Frankly, I think vac should see the sort of adoption that cvmfs did - it's just clearly the better way to do things now.
[12:14:37] Ewan Mac Mahon *jumps up and down waving pom-poms*
[12:16:00] Sam Skipsey Is it still a relatively static allocation of VMs, Ewan?
[12:16:12] Sam Skipsey (Also, does it support people who want to do things like MPI?)
[12:16:47] Ewan Mac Mahon Depends what you mean by static. You give it a target ratio of VM types in its config file
[12:16:59] Ewan Mac Mahon (but you can tweak that, it re-reads changed configs at runtime)
[12:17:18] Sam Skipsey Right, but how does it adapt to having no work of (say) the largest type?
[12:17:28] Ewan Mac Mahon and if you want (say) 50% LHCb VMs, but there's no LHCb work, it'll ease off running those and do something else.
[12:17:40] Sam Skipsey ...by "ease off"...
[12:17:48] Ewan Mac Mahon The shutdown scripts are supposed to quit with an exit code, one of which is 'there's no work'
[12:17:58] Sam Skipsey Okay, but how fast is that?
[12:18:12] Sam Skipsey That seems wasteful compared to conventional batch scheduling.
[12:18:33] Ewan Mac Mahon As for MPI; it supports multicore VMs now, doing communication between VMs would be difficult/impossible/out of scopoe.
[12:18:37] Sam Skipsey (but then, VM startup and shutdown is also wasteful)
[12:19:05] Ewan Mac Mahon And it's not that slow; you're looking at a few minutes, so it's on the order of time an idle pilot takes to quit on an ordinary batch system.
[12:19:07] Sam Skipsey I note that supporting multicore VMs is very different to supporting MPI.
[12:19:08] Ewan Mac Mahon More or less.
[12:19:27] Andrew McNab it uses less CPU than having a dedicated headnode, because the info about VMs that "fizzle" without getting any work is shared across the whole Vac space.
[12:20:02] Sam Skipsey (and, in particular, MPI support needs orchestration of VMs with an explicit nodelist, which is I guess not something that the Vac needs to worry about at the moment)
[12:20:44] Sam Skipsey ...I'm not sure I understand your point, Andrew.
[12:21:08] Sam Skipsey That's only relevant if all of your users are using pilot frameworks, surely?
[12:22:00] Andrew McNab the "waste" / cost of running a VM to see if there is any work is less than the cost of maintaining a substantial multicore headnode with the batch / gateway stuff on it in a conventional CREAM/batch site
[12:22:38] Sam Skipsey Is this research published anywhere?
[12:22:54] Sam Skipsey And does it still apply to the many small VOs who don't use pilot models?
[12:23:04] Sam Skipsey (or Dirac)
[12:23:06] Andrew McNab the assumption is that the VOs are using a pilot framework of some kind, which is all of LHC, and the DIRAC ones like biomed, ILC, etc
[12:23:18] Sam Skipsey But not lots of other VOs.
[12:23:25] Ewan Mac Mahon And in the future, everyone else too. The WMS is doomed.
[12:23:42] Ewan Steele I'm sorry I need to go.
[12:23:42] Sam Skipsey ...the WMS being doomed does not mean pilot models are better solutions.
[12:23:55] Sam Skipsey That's clearly a false equivalence if I've ever seen one.
[12:24:21] Sam Skipsey (Although I agree that if *everyone* agreed on a common pilot framework, we might be getting somewhere.)
[12:25:10] Ewan Mac Mahon Well it sounds like both GridPP and apparantly EGI are pushing towards dirac for 'other' VOs, so I think we're more or less getting there on that front.
[12:25:24] Ewan Mac Mahon It's not clear that there's much of an alternative that actually exists atm.
[12:25:26] Christopher Walker https://twiki.cern.ch/twiki/bin/view/LCG/GDBMeetingNotes20140311
[12:25:31] Sam Skipsey Sure, but I'd prefer it if ATLAS and CMS held their noses and used it too
[12:25:58] Sam Skipsey Plus, it would be horribly inefficient for our local users, who we do have.
[12:26:24] Ewan Mac Mahon And as a transition thing, I don't think there's anything that would stop you using vac nodes as a client to a local batch system if you had other VO/local users, though.
[12:26:52] Andrew McNab there's also the possibility to put other things in the VMs that know to shutdown when they are idle. So batch worker nodes, condor nodes, even PROOF workers could be made into VMs and run this way as part of a Vac space.
[12:27:21] Sam Skipsey Well, in fact, for ATLAS and CMS, they *are* just using Condor glideins, effectively.
[12:27:38] Ewan Mac Mahon Depends what you want for the local users; if they're happy on a conventional cluster that's OK, but vac gives a lot of the benefits of cloudy stuff (isolation, choice of OS/environment etc.).
[12:27:39] Sam Skipsey (Which leads to my other question of how this is better than, say, Condor with the VM universe )
[12:28:46] Ewan Mac Mahon The problem with a lot of VI could systems is that they're really, really complicated. Plus the 'please start me a VM' model doesn't fit resources that are usually full terribly well.
[12:29:13] Sam Skipsey sure.
[12:29:24] Ewan Mac Mahon Anyway, I believe there's a talk on this at the GridPP meeting, so I expect some beer assisted thradhing out of this stuff.
[12:30:12] Ewan Mac Mahon But on a practical level, you can grab a spare WN and getting it running LHCb jobs on vac this week, and get a bit of a hands-on feel for it. It's very lightweight to get it going.
[12:34:13] Ewan Mac Mahon At this point, I'm pretty sure the pain of getting off torque/maui is less than the pain of staying.
[12:34:27] Ewan Mac Mahon It's a matter of short term pain vs long term, but still.
[12:34:29] Alessandra Forti I agree
[12:35:47] Ewan Mac Mahon I like the way Dave just organised a meeting in Pitlochry, but felt the need to Doodle poll Ambleside on the grounds that it's hard to get to.
[12:35:55] Ewan Mac Mahon Also, Oh Noes!
[12:35:59] Ewan Mac Mahon
[12:36:35] Alessandra Forti Good luck for the future!
[12:37:13] Mark Mitchell Thanks Alessandra
There are minutes attached to this event.
Show them.