Ops Meeting 080113
Experiments
CMS quiet RALPP problems.
ATLAS quiet over xmas, problems with alas group production which filled prod disk at several sites.
System kicked in excluding space tokens.
Stephan looking at ways to clear up, prod disk will move to data disk.
Glasgow problem, prod disk blacklisted, set site offline . Lost a few days.
Compiling report for December availability , looks better than previous month.
Three sites , with tickets about move of space from group disk to data disk. Sheffield, RALPP, and Glasgow ?
Glasgow ticket is in progress.
Other VOs issues:
T2k copied lots of files, including some which already existed. Storage group should help them do better.
Neiss VO having problems getting going again. QM and LANCS not in NGS WMS so some work required. Need to talk to Glasgow.
NGS VOMS server to retire this week?
Chris will file tickets.
VOs should be asked to site GridPP in publications.
Meetings and updates:
Tier 1 status:
Atlas storage blew up, error message caused by expired password, which should not have expired, but update to oracle 11 caused passwords to be reset to expire on Christmas day.
Some problems with top bdii’s, aware of this. Sometimes need restarting.
On duty.
On duty ROD news (Thanks )
Problem last week Kashif was on duty , alarms were not reaching the dashboard. He filed a ticket which has now been solved. Not many UK .
Rollout (Danielia state of Nation page)
Update pushed out on 20
th Dec, for ce and LB.
State of Nation not up to date.
Did manage to install Matts tarball. Tested ok others could try.
https://www.sysadmin.hep.ac.uk/wiki/EMI2Tarball
Need proper Atlas testing and a queue enabling.
AF upgraded the Wiki page yesterday.
Bham ok now, Sheffield have done the WN’s.
QM WN’s running EMI2 but fragile. Some config issues as installed with rpms. (Beta/deployed)
Issues around the deployment system but as Chris states it is a fragile deployment.
UCL no comments (Failing DPM tests also)
LANCS one cluster using EMI tar ball, other one needs to be upgraded as its on glite this week or next.
Glasgow, partially built, want to be ready by next week, will also install ARGUS at the same time.
Sussex, space tokens still need testing, so not in production.
Durham no updates at the moment, still low on manpower. Glite CE and bdii. EM should we mark them offline. To take the pressure off until they are ready. Mark will talk to Mike to then discuss whether they should be marked off line. Durham in test for ATLAS. Ongoing issues have not been resolved completely.
ECDF waiting for SL6 tar ball which is being worked on. Next 2 weeks?
Security
Quiet, one EGI site compromised in Estonia over the Christmas period.
No reported issues at any other Grid site.
Tickets
Covered by Matt in the Agenda
GDB Report
Covered by Matt in the Agenda
Post EMI working life cycle. Community Support model for ending of various projects seen as the way to go.
Chris Walker LFC support will be ongoing.
Virtualised Worker Nodes. Working group update on how to build trusted images and associated traceability. VM WNs will be a possibility in the future. However ongoing development is required.
Software defined Networks. Matt recommended reviewing the slides.
EGI and EMI rolls still to be defined post EMI decline. EGI and LCG product stacks are becoming closer.
YAIM vs Puppet support is being considered. Product teams are still to support YAIM.
UK to feedback to the GDB on these changes.
WN EGI deadlines.
End of January deadline, however sites can give technical response if required to avoid being blacklisted after this date.
Underway at all sites.
No sites have been suspended.
Durham transition to be co-ordinated from Glasgow.
January EMI 1 Probes will stop working in January. (No date given)
AOB
Peter G. Eval section in quarterly reports is designed to deliver additional information to STFC at the end of the project. Covering collaboration, impact, joint ventures and Citations, etc.
Chat log:
[11:01:56] Brian Davies if people have joined via evo, I have joined via seevogh , so it appears they work together
[11:02:16] Alessandra Forti I think it is the same
[11:02:16] RECORDING Mark joined
[11:02:30] Alessandra Forti I've used a jlnp file that I had on disk
[11:03:01] Mark Slater joined
[11:04:08] Rob Harper joined
[11:04:14] Matt Doidge joined
[11:04:33] Linda Cornwall I think RAL PP had an air conditioning problem over christmas
[11:04:45] Ewan Mac Mahon joined
[11:05:12] Gareth Smith joined
[11:05:17] Queen Mary, U London London, U.K. joined
[11:07:17] Rob Harper Yes, AC failure at RALPP on Xmas day. We basically just turned the site off until the new year.
[11:09:01] Elena Korolkova for Sheffield I'll remove GROUPDISK this week.
[11:09:11] Alessandra Forti ok
[11:12:20] Andrew McNab For LHCb, I can't see anything particularly UK related in the logbook or the ops meeting yesterday.
[11:12:37] Daniel Traynor joined
[11:15:23] Andrew McNab mike problem
[11:15:35] Andrew McNab was basically ok over christmas week, with problems here and there
[11:18:50] Matt Doidge https://www.sysadmin.hep.ac.uk/wiki/EMI2Tarball
[11:19:10] Matt Doidge for anyone interested
[11:19:24] Alessandra Forti https://www.gridpp.ac.uk/wiki/UK_EMI2_Deployment
[11:21:00] Mark Slater Bham is now correct. Everything UMD1 except DPM + WNs which are EMI2
[11:22:56] Ewan Mac Mahon On upgrades, can I just add that DPM pool nodes count as service nodes too - they're just harder to monitor automatically.
[11:23:11] Ewan Mac Mahon They're still grid accessible and need to be upgraded.
[11:23:19] Ewan Mac Mahon (or turned off)
[11:30:11] Stephen Jones left
[11:30:18] Stephen Jones joined
[11:31:40] Ewan Mac Mahon Well, they're not making progress.
[11:31:50] Ewan Mac Mahon The BDII should have been done months ago.
[11:32:08] Ewan Mac Mahon Constantly going "We're doing it" and not doing it isn't progress.
[11:32:47] Ewan Mac Mahon So, talk to Mike and we'll re-visit this next week at this meeting?
[11:32:50] Mark Mitchell Chris I agree
[11:33:15] Ewan Mac Mahon Yup; Chris is right - if we don't deal with this it makes the NGI look bad, not just the site.
[11:34:16] Alessandra Forti ECDF was skipped
[11:34:45] Ewan Mac Mahon I think they're just another taeball site.
[11:34:52] Ewan Mac Mahon *tarball
[11:38:15] Elena Korolkova I think Liverpool should increased PRODDISK by 5TB for now as it's in test mode because the dusk is full
[11:38:18] Elena Korolkova http://panda.cern.ch/server/pandamon/query?job=*&site=UKI-NORTHGRID-LIV-HEP&type=production&jobStatus=failed&hours=12&user=gangarbt
[11:39:08] John Bland Elena: which other token loses 5TB?
[11:40:15] Ewan Mac Mahon Common practice is to take it off DATADISK
[11:40:19] Alessandra Forti I've just closed it
[11:40:38] Elena Korolkova DATADISK.
[11:41:42] Elena Korolkova I think we will be able to reduce PRODDISK next week when the problem with GROUP production will be sorted out
[11:44:12] Matt Doidge https://www.gridpp.ac.uk/wiki/GDB_12th_December_2012
[11:48:47] Ewan Mac Mahon It used to be less trivial pre-shared caching, but having that wil make it easier.
[11:50:10] Govind Songara joined
[12:02:36] Ewan Mac Mahon The last time there was a 'rollback' it did work as you'd hope, it has not done in the dim and distant past.
[12:02:47] Ewan Mac Mahon I object a bit.
[12:03:10] Ewan Mac Mahon We shouldn't be painting ourselves into that sort of corner in the first place.
[12:03:28] Ewan Mac Mahon If only one person can release a package, we're already stuffed.
[12:03:34] Alessandra Forti it's matter of resources I guess
[12:03:47] Alessandra Forti the only protection is to have local repositories
[12:03:58] Alessandra Forti and don't update from the master without testing
[12:03:59] Ewan Mac Mahon The processes should be designed to allow multiple people to be able to bump the packages.
[12:04:17] Stuart Purdie I should add: The principle of 'yum update' always being correct is one I subscribe to - so take the old one and boost the version number is the right approach.
[12:04:32] Ewan Mac Mahon This isn't complicated - once again we just need to nick the procedures from other projects that get this right.
[12:04:58] Ewan Mac Mahon There should be no 'rollback' on point of principle, and everything else should be set up to make that work.
[12:05:16] Ewan Mac Mahon Not least because 'rollback' causes immense problems and more-or-less doesn't work.
[12:05:36] Alessandra Forti indeed
[12:06:02] Alessandra Forti but they don't want to do tht
[12:06:05] Alessandra Forti that
[12:07:02] Ewan Mac Mahon Well, I think we should clearly state our UK view that we strongly believe they should.
[12:07:35] Ewan Mac Mahon Most of the problems should be historical anyway - AIUI most of it was related to the grief of usign ETICS for everything.
[12:08:57] Alessandra Forti I meant they don't want to do the roll-back
[12:10:20] Ewan Mac Mahon Ah right. I think we're probably mostly in agreement, but I would like to strengthen "Rollbacks only in exeptional circumstances" to "No rollbacks, ever"
[12:10:37] Ewan Mac Mahon Where 'rollback' = the bad version number decreasing version.
[12:12:31] Ewan Mac Mahon I think we're virtually all in the same place here, tbh.