Operations team & Sites

→ Europe/London
EVO - GridPP Operations team meeting

EVO - GridPP Operations team meeting

Description
- This is the weekly ops & sites meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the Janet(UK) Community area. - Direct EVO link http://evo.caltech.edu/evoNext/koala.jnlp?meeting=eseieIvnv9aiaeIla8Is - The phone bridge number is +44 131 474 4520 (CERN number +41 22 76 71400). The phone bridge ID is 154732 with code: 4880. Apologies: Matt D. John G.
Ops Minutes 02/04/2013

Audio Issues for Jeremy and Chris
ATLAS
Issues with Glasgow over the weekend due to storage being overloaded by the VO. Not a site issue.
Notification procedure for black listing covered.
FTS Testing successful.
Storage issues at Lancaster, Liverpool and QMUL. Now fixed.
File copying issue at Glasgow under investigation.
LHCB
No issues
GGUS Tickets against 2 sites
CMS
No issues
Small VOs
No Issues

EMI Dcache
No Update

Key tokens page generated by Jens.
To be discussed in the storage Group.

Chris Walker advised that Small Vos are looking for guidance in this area. To be discussed in the Storage Group.

WLCG Coordination Section added to the bulletin
Task force for HTTP proxy Discovery. If interested please contact Alessandra.
SL6 Task Force. Please contact Alessandra.

Tier-1
Minor Network issues. No other updates.

DDN Discussion
Storage Group to discuss the implications of the technology on offer.
Detailed discussion to be held in the future.
Brian Davies multiple technology groups should be evaluated.
Pete Gronbech to distribute presentation from DDN

Documentation
Limited Updates due to holiday period.
Reminders to be sent out on areas such as monitoring and security

Interoperation
EGI Summary to be issued by Stuart Purdie. If any questions arise contact Stuart directly.

Monitoring
No Issues

Operations
ROD Email issue fixed

Roll Out
EMI-1 Nagios Probes go critical this month.
EMI-1 services after the end of April must go into down time
Stuart issued a short note on services still on EMI-1 to TB Support.
Brian Davies Dcache at RALPP to be off lined.

Security
Thank you for the security challenge work

Services
Manchester Electrical Work will impact core services such as the VOMs server. Oxford and Imperial backups need to be tested prior to the Manchester work going ahead.

Tickets
ECDF Availability ticket still upon surrounding the Unknown State.
Team and Site tickets to be reviewed.
QMUL EMI2 Update to be investigated for remaining components. Concerns around any security risks.


Site Updates
No Issues

Current Status
EMI-3 and Yaim support.
Large scale and generalised discussion around the deployment of EMI-3 and YAIM.
YAIM supported in EMI-3 environment for CEs.
Discussion around Puppet and YAIM interoperability.
Further Investigation required.

Actions and Site Updates
DN moves without email addresses discussed.






CHAT LOG
[11:01:16] Jeremy Coles We'll start in 2 minutes.
[11:02:27] John Hill joined
[11:02:28] Rob Fay joined
[11:02:28] Elena Korolkova joined
[11:02:30] Andrew Washbrook joined
[11:02:32] John Bland joined
[11:02:32] Ewan Mac Mahon joined
[11:02:35] Robert Frank joined
[11:02:40] Jeremy Coles For minutes: Mark=5 Duncan=5 Catalin=5 Andrew=5 Stuart=5 Ewan=5 Matt=6 David=6 Stephen=6 Brian=6
[11:03:20] Wahid Bhimji joined
[11:03:21] Jeremy Coles Mark is taking minutes.
[11:03:30] RECORDING Mark joined
[11:03:35] Linda Cornwall joined
[11:03:38] Sam Skipsey joined
[11:03:46] Raja Nandakumar joined
[11:05:24] Pete Gronbech joined
[11:05:32] Christopher Walker joined
[11:05:50] Brian Davies joined
[11:05:52] Elena Korolkova https://twiki.cern.ch/twiki/bin/viewauth/Atlas/StorageAreaAutomaticBlacklisting.
[11:06:19] Elena Korolkova http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteviewhistorywithstatistics?columnid=10074
[11:06:47] Rob Harper joined
[11:07:52] Jeremy Coles Report uploaded: https://indico.cern.ch/getFile.py/access?contribId=1&resId=0&materialId=0&confId=240010
[11:10:07] Stuart Purdie joined
[11:10:44] Gareth Smith joined
[11:11:53] Jeremy Coles Is my audio getting through?
[11:12:02] John Hill No
[11:12:12] Jeremy Coles Will restart. Yes Elena.
[11:12:15] Ewan Mac Mahon Yes.
[11:12:25] Ewan Mac Mahon I can hear Elena, but not Jeremy
[11:13:05] Jeremy Coles Still not working.... I was going to ask about SAAB. Will rejoin again.
[11:13:14] Jeremy Coles left
[11:14:24] Jeremy Coles joined
[11:18:49] Pete Gronbech darth vader
[11:18:50] Mark Mitchell Brian could you mute please
[11:19:18] Christopher Walker My audio should be better now.
[11:19:49] Christopher Walker Only news from other VOs is that Jens has written a document on using keytokens.
[11:20:01] Chris Brew joined
[11:21:17] Ewan Mac Mahon Plus we all sortof agreed to set up a dirac last week.
[11:26:02] Ewan Mac Mahon Have they cut their prices by about two thirds?
[11:26:15] Ewan Mac Mahon That's basically the key determinant of interestingness.
[11:26:23] Ewan Mac Mahon We knew they had cool stuff.
[11:33:04] Jeremy Coles https://www.gridpp.ac.uk/php/KeyDocs.php?sort=reviewed
[11:37:11] Christopher Walker EMI-3 StoRM won't work in production.
[11:37:31] Elena Korolkova I'm wondering why EMI2 ce is shown as a n EMI1 CE?
[11:38:03] Rob Harper  
[11:38:15] Christopher Walker QMUL is now extremely likely to miss the deadline for StoRM.
[11:42:44] Christopher Walker Will reasses the situation when I come back from annual leave and hepix.
[11:45:18] Wahid Bhimji waiting for reply for a long time - so can I reassign or close ?
[11:46:54] Wahid Bhimji "red" doesn't seem right
[11:47:01] Ewan Mac Mahon I think we've only got one at oxf and it's Brian - I'll deal with that today.
[11:47:04] Wahid Bhimji for irrelevant tickets
[11:48:56] Ewan Mac Mahon If I were you Wahid, I think I'd close that at 'Not solved', with a note that it doesn't seem to be your problem, and see if anyone re-opens it.
[11:49:40] Ewan Mac Mahon We don't have a good GGUS resolution that indicates something like 'invalid' or 'not a bug'.
[11:53:31] Christopher Walker https://ggus.eu/ws/ticket_info.php?ticket=92498 is tracking QMUL's APEL update
[11:53:55] Mohammad kashif Wahid, I will have a look at this ticket
[11:55:31] Ewan Mac Mahon I suspect we may have to settle for a documented mess though.
[11:56:45] Wahid Bhimji thanks kashif
[12:01:17] Wahid Bhimji all of that is possible in puppet...
[12:01:31] Ewan Mac Mahon I think we could maybe go for "YAIM or an equivalent" ?
[12:02:43] Wahid Bhimji I don't think there is anything that can't (much more nicely) done with puppet - but of course it needs to be provided - and work - and not break everything
[12:04:17] Ewan Mac Mahon AIUI puppet can run helper scripts - I suspect that we may wind up with puppet modules that run some bash helper scripts that happen to strongly resemble the existing YAIM modules.
[12:04:30] Wahid Bhimji (what for ..)
[12:05:29] Christopher Walker http://www.youtube.com/watch?v=ehVMxbJdld8 - cern puppet talk
[12:06:05] Christopher Walker AIUI the particular issue with APEL is that it's a complete rewrite from the ground up.
[12:06:28] Gareth Smith left
[12:08:04] Ewan Mac Mahon I think we should have some generic wording rather than necessarily demand puppet, but puppet clearly will be the favourite option.
[12:08:08] Christopher Walker Writing Yaim (and indeed puppet) from scratch is going to be non negligible manpower.
[12:08:11] Wahid Bhimji With DPM we are in a driving seat of course
[12:08:13] Steve Jones Yes, APEL is all new.
[12:09:24] Wahid Bhimji with DPM the plan was new compontents would only be puppet configurable from now on - but to be clear you can run puppet apply without a master - which is yaim like
[12:09:52] Wahid Bhimji potentially there was talk of wrapping somehting where yaim ran puppet to make transistion easier ...
[12:09:59] Jeremy Coles https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
[12:10:28] Ewan Mac Mahon Indeed; self contained locally usable puppet modules that could optionally be pulled out into a site puppet sounds ideal.
[12:11:21] Ewan Mac Mahon Provided that any config options come from something similar to site-info.def
[12:11:39] Robert Frank left
[12:11:49] Steve Jones Yes - the site-info.def is a useful database of config data that abstracts the whole site-management problem.
[12:13:48] Steve Jones Without it, admins need to know a lot more about the middleware.
[12:14:08] Christopher Walker Would it be worth separating out the closed items?
[12:14:48] Ewan Mac Mahon He may well have done - this is probably just me....
[12:14:59] Steve Jones https://www.gridpp.ac.uk/wiki/Scheduled_Downtimes
[12:17:48] Daniel Traynor left
[12:17:48] Steve Jones cheers
[12:17:49] Brian Davies left
[12:17:52] Mohammad kashif left
[12:17:53] Rob Harper left
[12:17:53] John Hill left
[12:17:55] Chris Brew left
[12:17:55] Rob Fay left
[12:17:55] John Bland left
[12:17:56] Raja Nandakumar left
[12:17:58] Andrew Washbrook left
[12:17:59] Linda Cornwall left
[12:17:59] Elena Korolkova left
[12:18:15] Stuart Purdie left
[12:18:19] Wahid Bhimji left
[12:18:28] Christopher Walker left
There are minutes attached to this event. Show them.
    • 11:00 → 11:20
      Experiment problems/issues 20m
      Review of weekly issues by experiment/VO - LHCb - CMS - ATLAS - Other
      ATLAS report
    • 11:20 → 11:40
      Meetings & updates 20m
      With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest - General updates - WLCG ops coordination - Tier-1 status - Accounting - Documentation - Interoperation - Monitoring - On-duty - Rollout - Security - Services - Tickets - Tools - VOs - SIte updates
    • 11:40 → 12:00
      Current status/issues 20m
      - Lack of YAIM support in EMI-3. - Please report any projects at your site that may be a good fit with the GridPP impact and KE agenda.
    • 12:00 → 12:05
      Actions & site updates 5m
      https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
    • 12:05 → 12:06
      AOB 1m