Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the weekly GridPP ops & sites meeting
- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6
-- The PIN is 1234.
To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.
-- The London (UK) service is on +442030510622
-- The meeting extension is 9308582.
Apologies:
Minutes:
Present:
A Forti
A Washbrook
A Lahiff
B Davies (Minutes)
C Brew
D Bauer
E Korolokova
E Mac Mahon
F Melaacio
G Qian
D Roy
G Smith
G stewart
Govind
I loader
I nielson
J Coles (Chair)
J Bland
J hill
Marcus
O Smith
P Lcacesso
P Gronbech
R Nandakumar
R Frank
S Skipsey
S Jones
T Whyntie
--------------------------------------------------------------------
--------------------------------------------------------------------
11:01 - 11:20 Experiment problems/issues 19'
Review of weekly issues by experiment/VO
- LHCb
RN- Thanks to RAL Tier2
- CMS
https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T2_UK_London_Brunel
DB Nothing to Report
Bristol was low %
QMUL asking to runb more CMS jobs
- ATLAS
EK- PRODDISK fully decommed , recretaed at QMUL to allow jobs to run ( temporarily.)
Reminder to sites to delete PRODDISKS
Request for T2 dumps for consistent checking.
Analysis of why Analysis jobs fail:
https://indico.cern.ch/event/462436/contribution/1/attachments/1188471/1724289/ADCWeekly-10-11-15.pdf
- Other
-- DEAP3600: Pending
-- HPC DiRAC: Jens
"now archiving codes and .svn directories by tar'ing and gzip'ing them first.
It needs more preparation, but increased the i/o performance for archives by a factor of 4-10 at least."
GS-worried about number of files in vo.dirac and wether it scales.
BD- Median filers size .5MB, largest 480GB
taring should help
-- LIGO: Catalin
-- LOFAR: George
-- LSST: Alessandra
AF- looking at using Ganga, trouble with proxies
ARC sites not publishing VOView
Oxford and liverpoool pilot roles (dirac) doesn't work
Issues with FNAL side of LSST
-- LZ: David
Looking to be enabled CPU not storage at sites. ( copy data back to IC.)
-- UKQCD: Jeremy - no update
-- UCLan/GalDyn: Tom
With regards to further sites supporting: "For GalDyn there are no requirements as of yet - in fact there is not yet galdyn.egi.eu VO. They are still in the "incubator" stage of testing with the NorthGrid VO (which have been very successful) as they based at the University of Central Lancashire (UCLan)."
-- PRaVDA: Mark/Matt
- GridPP DIRAC status
-- http://www.gridpp.ac.uk/php/gridpp-dirac-sam.php?action=view
DR- AMcN builf ing a new image
3 sites that don't have pilots so don't run jobs.
T1(GGUS ticket), Sheffield and sussex.
http://www.gridpp.ac.uk/php/gridpp-dirac-sam.php?action=view
https://www.gridpp.ac.uk/wiki/Cloud_%26_VM_status
- Status of pilot enabling across sites.
tracking page in gridpp wiki
https://www.gridpp.ac.uk/wiki/Batch_system_status
-------------------------------------------------------------------
--------------------------------------------------------------------
11:20 - 11:40 Meetings & updates 20'
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
- General updates
- WLCG ops coordination
- Tier-1 status
- Storage and data management
- Tier-2 Evolution
- Accounting
- Documentation
- Interoperation
- Monitoring
- On-duty
- Rollout
- Security
- Services
- Tickets
- Tools
- VOs
- Site updates
-------------------------------
General updates
Tuesday 17th November
The GalDyn case study has been written up
There is now a job adverts section on the new GridPP website. Now we need a process to populate the area....
There is a summary available of the LHCOPN-LHCONE meeting in Amsterdam on the 28-29 of October 2015.
Ewan: Pilot accounts for everyone
Welcome to Markus Ebert who has started at Edinburgh (designated GridPP/LSST DAC person).
-------------------------------
WLCG Operations Coordination - AgendasWiki Page
Tuesday 17th November
There is a WLCG ops coordination meeting this week on Thursday.
-------------------------------
Tier-1 - Status Page
Tuesday 17th November A reminder that there is a weekly Tier-1 experiment liaison meeting. Notes from the last meeting here
We are investigating why LHCB batch jobs sometimes fail to write results back to Castor (and the sometimes fail to write remotely as well).
We are continuing with some detailed network changes needed to remove our old core switch from the network. There is a site 'warning' tomorrow morning for one of the steps in this.
A week or so ago we saw some Atlas Hammercloud tests failing (loss of heartbeat) - although the problem seems to have gone away now. This is not understood yet.
We have found a problem on some disk servers of one particular batch that have been updated to SL6. The servers can run slowly and individual commands hang (until a timeout) while making name lookups. So far a total of three servers have been affected spread over a couple of weeks. We can easily fix the problem but do not yet know why it occurs.
Two of the tape servers are now running Castor version 2.1.15 - and we are rolling this out to more. These can be updated independently of the rest of Castor as they have no interaction with the Castor database. Work is getting underway to test Castor 2.1.15.
-------------------------------
Storage & Data Management - Agendas/Minutes
Tuesday 17th November
Getting Input for next DPM F2F meeting.
Configuring DPM with puppet
Limiting connections per disk server
Still looking for feedback.
Lock-in to storage limiting effect to move away from DPM.
-------------------------------
Tier-2 Evolution - GridPP JIRA
Tuesday 17 Nov
New depo.gridpp.ac.uk service for uploading files to via HTTPS
ATLAS VMs now upload log files to depo.gridpp.ac.uk for debugging (GRIDPP-24)
-------------------------------
Accounting - UK Grid Metrics HEPSPEC06 Atlas Dashboard HS06
Tuesday 3rd November
APEL delay (normal state) Lancaster and Sheffield.
-------------------------------
Documentation - KeyDocs
Friday 6th Nov, 2015
SteveJ: Advice to admins published about a common GSS error, globus_gsi_callback_module: Could not verify credential etc.
https://www.gridpp.ac.uk/wiki/Security_system_errors_and_workarounds
General note
See the worst KeyDocs list for documents needing review now and the names of the responsible people.
-------------------------------
Interoperation - EGI ops agendas
Tuesday 10th November
Meeting yesterday was cancelled in favour of OMB; next meeting scheduled for 14/12.
-------------------------------
Monitoring - Links MyWLCG
I got cutoff exactly at monitoring. No special news. Just a notification that condor_gangliad
tool for jobs monitoring is officially available from Condor 8.2+ stable release
Tuesday 16th June
F Melaccio & D Crooks decided to add a FAQs section devoted to common monitoring issues under the monitoring page.
Feedback welcome.
-------------------------------
On-duty - Dashboard ROD rota
Tuesday 17th November
Tuesday 3rd November
The long-standing UCL availability alarm went green yesterday on 29th October. We are not sure why!
Quite a lot of activity on the dashboard this week, but only one or two new tickets.
Tickets: Five for availability / reliability: Sussex, Sheffield, Liverpool, Lancaster and UCL. Two for GLUE2 validation: Liverpool and QMUL. One for the CEs at QMU
-------------------------------
Rollout Status WLCG Baseline
Tuesday 15th September
UMD 3.13.3 is available.
References
Staged Rollout pages (now separated into EMI1 & 2), and the page listing the deployed versions is extractable from the bdii, so they should all be reasonably up-to-date:
http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout.html
http://www.hep.ph.ic.ac.uk/~dbauer/grid/staged_rollout_emi2.html
http://www.hep.ph.ic.ac.uk/~dbauer/grid/state_of_the_nation.html
-------------------------------
Security - Incident Procedure Policies Rota
Tuesday 17th November
Advisory-SVG-2015-CVE-2015-7183 issued 06/11/2015: a few UK sites show as unpatched in EGI monitoring. WNs, as tested by the monitoring, may be less vulnerable than affected middleware services but they could be taken as an indication of general site readiness and sites are encouraged to check their status.
MD- not only security group to be notified.( Gridpp EGI-SVG were first to see this.)
IN- Check Jenkins announce
-------------------------------
Services - PerfSonar dashboard | GridPP VOMS
- This includes notifying of (inter)national services that will have an outage in the coming weeks or will be impacted by work elsewhere. (Cross-check the Tier-1 update).
JC- Is NDN work for LHCONE/OPN in conjunction or parallel to Imperial College (DR) work.
Work to connect RAL so as to test aability of LHCONE.
Tuesday 6th October
A reminder, the next LHCOPN and LHCONE joint meeting: Science Park Amsterdam (NL) 28-29 of October 2015
-------------------------------
Tickets
Monday 16th November 2015, 15.45 GMT
Only 17 Open UK tickets this week.
LSST FRIDAY 13th TEETHING TROUBLES
117585 (Liverpool)
Solution was Accounts in gridmapdir
117586 (Oxford)
Daniela has been testing out the LSST Dirac pilots - Ewan's fighting the good fight at Oxford getting his ARC to work, maybe Liverpool's problem has a similar root cause (we should be so lucky)?
TIER 1
116866 (12/10)
In a similar vein (and thanks again to Daniela for the effort with preparing our pilots) is this ticket about getting "other" pilots enabled at the Tier 1 - particularly with a view of these tests: http://www.gridpp.ac.uk/php/gridpp-dirac-sam.php?action=viewlcg Any news? On hold (6/11)
ECDF(-RDF)
I think this new atlas ticket:
117606 (15/11)
is a duplicate of this current atlas ticket:
117447 (8/11)
So I suspect it can be closed!
I see from 117642 that the other VO nagios has been recently rebooted which might explain the lot of failures I see, I'll check again before the meeting.
-------------------------------
Tools - MyEGI Nagios
Tuesday 6 Oct 2015
Moved Gridpp nagios instance back to Oxford from Lancaster. It was kind of double whammy as both sites went down together. Fortunately Oxford site was partially working so we managed to start SAM Nagios at Oxford. Sam tests were unavailable for few hours but no affect on egi availability/reliability. Sites can have a look at https://mon.egi.eu/myegi/ss/ for a/r status.
T
-------------------------------
VOs - GridPP VOMS VO IDs Approved VO table
Tuesday 19th May
There is a current priority for enabling/supporting our joining communities.
-------------------------------
Site Updates
Meeting Summaries
Project Management Board -GRIDPP5 notices gone to Tier2s
Dave Britton to send info regarding hardware purchases.
HEPSYSMAN still looking for agenda input
--------------------------------------------------------------------
--------------------------------------------------------------------
11:40 - 12:00 Possible discussion items 20'
GR-dirac for small VOs, schedule/ETA for decommisioning of WMS?
pheno ans ops still use it.( but most users should have switched to dirac.)
IC and RAL T1 still running
1. Site roundtable
Manchster
AF -NTR- no recent work on IPV6 at the moment. All machines are dual stacked, but not in the DNS
EMcM- perfSonar and storage the priority for IPV6ing ( for new sites who only have ipV6 WNs.)
GR-will perfsonar boxes work without reverse DNS. apparently not. Glasgow need this.
JC- Please update https://www.gridpp.ac.uk/wiki/IPv6_site_status
and also https://www.gridpp.ac.uk/wiki/Batch_system_status
Edinburgh-
AW- New cluster install (local issues.)
running SL7, working with ATLAS
EMcM do we get EGI security updates for SL7 ( for security team meeting, report back here.)
AL testing SL7 with jobs in Docker containers. CMS jobs working successfully. Limitation in condor that CPU time not working so accounting no good at the moment.
RALPP
CB- redeployed xrootd reverse proxies for FAX/AAA ( plan to put lhcb stuff on same boxes.) seem happy co-existing.
Imperial College
DB- dirac, glexec issue, making small VO work.
DR- FTS transfers within dirac, with developers. new nagios servers to test ATLAS/LHCb services using http.
SS- http deployment Task Force in testing mode, some sites did not make there endpoints writable.
Sheffield
EK-IPV6 address for perfSonar looking for instructions.CCan get for Storage if needed. ( Site Availability effect?)
DR- Marian at CERN has setup a nagios ipv6 system . sites can have services added to this service.
QMUL
TW- Re New website; SJ has provided feedback, Thanks. ( mentioned at PMB).
--------------------------------------------------------------------
--------------------------------------------------------------------
12:00 - 12:05 Actions & AOB 5'
* https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
CHAT Window:
Chris Brew: (17/11/2015 11:20:49)
Can't FT3 talk to https endpoints?
Samuel Cadellin Skipsey: (11:21 AM)
Chris: well, the documentation claims it can (as it's also based on GFAL2). Not sure I've seen it done ever.
Ewan Mac Mahon: (11:23 AM)
AIUI the whole VO authentication didn't work.
I think we've fixed it now, but I need to go update the ticket.
Jeremy Coles: (11:30 AM)
http://www.gridpp.ac.uk/php/gridpp-dirac-sam.php?action=view
https://www.gridpp.ac.uk/wiki/Cloud_%26_VM_status
Ewan Mac Mahon: (11:33 AM)
https://www.gridpp.ac.uk/wiki/Batch_system_status
Federico Melaccio: (11:38 AM)
there was one at sussex I believe
but I do not know if it was gridpp
Jeremy Coles: (11:39 AM)
https://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
Matt Doidge: (11:40 AM)
Sorry I'm late
John Hill: (11:41 AM)
Got to leave early - another commitment
Ewan Mac Mahon: (11:42 AM)
I think the theme of the feedback on DPM puppet ws essentially "WAHHH!".
Certainly from me, at least.
Tom Whyntie: (11:44 AM)
Hello, sorry I'm later
*late
Ewan Mac Mahon: (11:45 AM)
I have no clue.
Jeremy Coles: (11:46 AM)
https://www.gridpp.ac.uk/php/KeyDocs.php?sort=area
Gareth Douglas Roy: (11:49 AM)
Sorry this request seems not valid or not matching to your scope. Please check the url set in your browser and the roles related to your DN.
Is what I get
Alessandra Forti: (11:49 AM)
sites usually can see only their site
Elena Korolkova: (11:49 AM)
For Sheffield: orry this request seems not valid or not matching to your scope. Please check the url set in your browser and the roles related to your DN.
Federico Melaccio: (11:49 AM)
I got cutoff exactly at monitoring. No special news. Just a notification that condor_gangliad tool for jobs monitoring is officially available from Condor 8.2+ stable releases
Ewan Mac Mahon: (11:51 AM)
Ooooh. Does that count as impact?
I think it does. Totally impact.
Steve Jones: (11:55 AM)
Accounts in gridmapdir
Daniela Bauer: (11:57 AM)
@Ewan, sorry I still get "nothing to serve"
Ewan Mac Mahon: (12:03 PM)
IMO just do it.
It's the right direction to be moving, and all the while it's not the last one you can just make it as a site decision.
If you're still running when all the other WMSes have gone you're in a much more awkward position.
Daniela Bauer: (12:09 PM)
our WMS apparently gets used by enterprising CMS users circuventing the central queue :-D
Alessandra Forti: (12:10 PM)
I'd rather not to put production stuff in ipv6 without more having testing
/having/heavy/
Jeremy Coles: (12:11 PM)
Please update https://www.gridpp.ac.uk/wiki/IPv6_site_status
and also https://www.gridpp.ac.uk/wiki/Batch_system_status
Ewan Mac Mahon: (12:14 PM)
Feel free to minute 'utterly batshit insane' if you want.
Gareth Douglas Roy: (12:14 PM)
Please do, it's driving me up the wall
Tom Whyntie: (12:21 PM)
Just to say re. GridPP DIRAC - there are some really nice features being implemented, like automatic registration and better security settings on the DIRAC File Catalog - thanks!
It's also good to get the updates from the GridPP DIRAC mailing list - if you're using it it's worth subscribing to: https://mailman.ic.ac.uk/mailman/listinfo/gridpp-dirac-users
Steve Jones: (12:28 PM)
no prob
Federico Melaccio: (12:28 PM)
thanks
Tom Whyntie: (12:29 PM)
Thanks, bye!
There are minutes attached to this event.
Show them.