Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the biweekly ops & sites meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area.
- The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 115728 with code: 4880.
Apologies: Mark M; Mingchao M; Mark S
GridPP Ops team minutes 2011 12 06 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ROD updates ------------ Some ROD team tickets were filed against Glasgow CEs that were marked as being in scheduled downtime. After some discussion it would appear that the ROD dashboard will raise alarms for any service that appears in the GOC db as 'in production' regardless of downtime status, and that while downtimes are visible via the dashboard, they're shown separately. ROD people have to manually check the automatically raised alarms against downtimes and only file tickets when appropriate. EGI Ops meeting ---------------- There was no EGI operations meeting yesterday, the next one will be held next week. On an EGI related note, Jeremy highlighted a reference to the Cream CE manual circulated by Chris Walker, who'd been pointed to it in response to a ticket he'd raised - it would appear that some of the hitorical gaps in documentation are being filled in and may merit a further look. UK Nagios Status ---------------- gridppnagios has been running uneventfully, but Kashif is planning an update for today - this should be a simple process, but he warned that it might cause a brief flurry of false alarms. It was agreed that Kashif would publicise this (and similar future upgrades) to the ROD team list. There has been some progress concerning the backup UK nagios to be hosted at Lancaster. There has been a query from Lancaster's systems people as to whether it could be hosted as a VM instead of as hardware (since they're generally pushing towards virtualised services). We're keen for it to be hardware so as to keep it isolated from any interdependency on Lancaster systems as far as possible. Matt Doidge continues to take the discussion forward at Lancaster. Tier 1 updates ------------------ - An 'operatinal error' occasioned an outage affecting the 'other VOs' software area that took it out of commission for the weekend. It was not entirely clear that this actually had much impact, however. - Some old MINOS data is in the process of being migrated for an NFS store into Castor. - Site BDIIs have now all been updated to the EGI UMD release, and the top level BDII servers are under way. Chris Walker noted that he'd been seeing issues with the RAL top BDIIs (GGUS #77026) - Routine rolling updates of glite-WN and CVMfs have been taking place across the batch farm. Kashif quiried what monitoring the Tier 1 has in place for WMSes since the ROD dashboard does not raise alarms for WMS failures since they're not considered critical services. Gareth replied that the Tier 1 use their own internal alarm systems, as well as manually keeping an eye on the gridppnagios test results via its web interface. Jeremy asked what monitoring is in place for other UK WMS servers, and whether they should be considered critical tests? They are not at the moment, since a single WMS failing doesn't make any services unavailable since users can simply fail over to another one. There are some caveats with that, including the lack of automatic WMS fail over (user jobs would generally need manual modification), and the potential issue of VOs supported on only one or a few WMS instances. Jeremy Coles will follow up. Elena asked about an issue from this morning with ATLAS transfers into RAL failing. Gareth said he'd check, and later reported via that chat window: "I have checked with Castor team. There was a locking issue for the database behind the Atlas SRM between about 07:45 and 08:30. resolved when staff arrived at work." Security --------------- Mingchao was not present to deliver a report directly, but had sent a written one to Jeremy. Edited highlights are that: - There are no incidents of note, with one UK university having a host with a connection to a compromised machine elsewhere, but the UK host was not part of a grid site, and has since been secured anyway. - There are some ongoing issues with the results that the security dashboard probes are detecting the glibc version(s) installed at Durham. - Some hosts are showing alarms due to CRLs not having been updated. Chris Walker noted that QMUL were showing some alarms specifically due to out of date CRL for the Thai CA, ultimately due to the CRL update source being offline due to flooding. Absent Mingchao there was no definite conclusion on how best to handle this sort of case. It was noted that there is a new release of the EGI trust anchors package(s) out. Operational/Tier 2 issues. ============================ Jeremy recapped and summarised the recent discussion on tb-support around ticket assignment issues. Key points are: - Everyone should be able to re-assign tickets if they've made it to the wrong place. The necessary 'ticket relations' section seems not to appear in all web browsers though. - It seems that the 'on solution' notification option doesn't actually mean that, though it's not entirely clear what it does mean. - Alessandra related the ongoing saga of test ticket (number 77082) which had appeared to be automatically assigned to her. There would appear to be complexities introduced by the differences between team tickets and 'normal' ones, and (possibly) between things done via the web interface and things done via email. More work is needed to undertand the current situation - Jeremy to follow up. The Doidge Report -------------------- This week Matt's tour of interesting and outstanding tickets took us via: - 77059 : Submitted against QMUL against Cream, this resulted in the aforementioned pointer to the Cream docs. The specific issue was related to the amount of memory Java is configured to use, and it was suggested that sites check that their configs are in line with the documented recommendations. - 74353 : The Pheno/MyProxy ticket. Had been closed as 'unsolved' by Maarten Litmaath (WTF?!), but then re-opened. People continue to follow up on the issue(s), but it was noted that the Pheno folks seem to have gone a bit quiet on us. - 75320 : The ticket required involvement from an INFN VOMS server administrator, but it was not clear how best to go about that. There was some discussion on potential use of the 'involve others' field, noting that it requires email addresses rather than support units. It was also suggested to just email the INFN folks and ask them to involve themselves. - 77008 : Cambridge's SE is still advertising itself as available to biomed despite the fact that it is not supposed to be, and in fact does not work for them. It was agreed that Santanu would take the issue to the gridpp storage group. Experimental issues: LHCb ------ No issues, but it was noted that there's not a lot going on at the moment either. CMS ----- Had a minor issue at Brunel, but it was fixed. Nothing else. ATLAS ------ - CVMfs is now in production at a further two sites, and others are ongoing. Imperial had implemented it, but chose to mount it at a location other than the default /cvmfs which seems to not work with ATLAS' code. It was reported that ECDF had apparantly also tried this and failed similarly. Alessandra asked about getting it moved to /cvmfs, which Duncan said he would pursue, while also noting that ATLAS really should be more flexible about this. This is of particular interest to sites with shared clusters not run purely for GridPP. Deployment at Durham has been postponed for now, given their other issues and lack of available time. Cambridge have been having some issues (possibly) with the automounter which are not entirely clear. It was decided to take this to email. Both Birmingham and Brunel are planning to deploy CVMfs today, both were asked to update their tracking tickets to say so. - Alessandra reported on a talk from the recent ADC meeting about users' behaviour with targetting sites. ATLAS statistics show that in the vast majority of cases, users are not specifically targetting a single site to run their jobs, but that they are often blacklisting ones that don't work well. - Alessandra gave a link to the new ATLAS monitoring page that shows the test results that will now be used to classify sites into 'a' 'b' and 'c' categories on a monthly basis: http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteviewhistorywithstatistics?columnid=562&view=Shifter%20view She will also add it to the monitoring links page. Guidelines on retiring services -------------------------------- Jeremy has been looking into the issue of how to retire grid service nodes; there is no documentation on this from EGI. We have previously had guidelines on removing SEs, but the general issue is worth considering, particularly in light of the fact that people will be retiring lcg-CEs (if they haven't already). Sam summarised the guidelines for storage elements (essentially that warnings need to be given to the VOs several months in advance) and Chris Walker posted the URL (https://www.gridpp.ac.uk/wiki/SE_Shutdown) noting that it's somewhat hidden under the Storage Group pages. The discussion then moved on to the case of lcg-CEs. The discussion mostly centred on a consensus that CEs are disposable items and there is no requirement in principle to give notice. Gareth Smith described the Tier 1's procedure as being to mark the CE in scheduled downtime, to drain it, then remove it from the gocdb. He pointed out though that this had the effect of removing all (visible) trace of the node from the gocdb, losing access to any history of (e.g.) downtimes; there appears to be no support for marking a node as 'retired', but retaining a record of its previous existence. The discussion then moved on to the mechanism of documenting the shutdown procedures, rather than the content of the procedures themselves. It was suggested that there should be a page (on the wiki?) for each node type containing pointers to documentation of each stage of the lifecycle (e.g. installation, monitoring, decomissioning) with the hope that much of this would be references to external docs, but with locally written ones where necessary. It was agreed to continue consideration of this area by email and other channels. Hardware updates ------------------- The state of the STFC/BIS e-infrastructure funding bid was queried, and it was noted that there had been no announceable developments, but that it was hoped that news would be forthcoming shortly. There was some discussion of how to plan GridPP4 hardware spending without knowing the outcome of the networking bids. Alessandra described her approach, which was to prepare two versions of the GridPP4 tender request, one for the case of getting the networking money, the other not. Alternatively, Ewan suggested simply sitting tight and waiting for news as it isn't expected to be much longer. Jeremy noted that there had been some discussions about future management of site hardware pledges, particularly with regard to taking account of decomissioning end-of-life hardware. Something has been sent to CB members, and site admins were advised to talk to their local CB person for further details. Actions review ---------------- - glExec tarball release : this has stalled given the uncertainty over which, if any, VOs are going to use it. The discussion over possible future glExec use cases is currently with the security TEG. - DPM/LFC checker : Sam reported that this is still pending, but remains someway down his todo list, now behind the more pressing DPM space rebalancer. - Moving existing data into spacetokens : Sam and Wahid noted that this requirement came from t2k, and they seem to have rather cooled on using space tokens, so the motivation has gone away. - Security measures for wiki pages : This is done, and now just requires people to use them. Andrew noted particularly that some pages exist in both secured and non-secured versions, and that the former will need to be removed. Andrew reminded everyone of how to use the new infrastructure - the UI is simply to set the first word of the page title to be 'Protected', which causes the beginning of the page URL to be 'Protected_', which being the security restrictions into play. - There was some discussion of the Steve Lloyd tests which are still being used for figures in the the quarterly reports. They are currently failing on some sites in a manner which seems to be dependent on which release(s) are installed. Alessandra suggested that Steve's tests should be replaced by something else (maybe the Hammercloud tests). However, the short-term problem is a change to the layout of ATLAS software installs that Steve's tests do not take into account. Chat window logs: ----------------- [11:01:12] Stephen Jones joined [11:01:12] Sam Skipsey joined [11:01:13] David Crooks joined [11:01:19] Santanu Das joined [11:01:22] Wahid Bhimji ooops [11:01:27] Elena Korolkova joined [11:01:34] Andrew Washbrook i think he meant to say he would be happy to take the minutes [11:01:40] Jeremy Coles https://indico.cern.ch/conferenceDisplay.py?confId=164310 [11:01:59] Duncan Rand joined [11:02:08] Govind Songara joined [11:02:12] Ewan Mac Mahon Sorry - who's doing the minutes? [11:02:16] Wahid Bhimji me [11:02:23] Daniela Bauer joined [11:02:26] Ewan Mac Mahon Right-o. *puts down pen* [11:02:29] Jeremy Coles Wahid but he would prefer you to do them! [11:02:53] Ewan Mac Mahon *picks pen back up* [11:03:22] Ewan Mac Mahon So; I'm fine with minuting, if you'd rather not Wahid. [11:03:38] Wahid Bhimji hah OK - go ahead then [11:03:54] Wahid Bhimji I wont look a gift horse in the mouth and actually I have other minutes to type up... [11:04:17] Stuart Wakefield joined [11:04:26] Mohammad kashif joined [11:04:54] Wahid Bhimji so are you doing it - just so I pick my pen back up if need be [11:05:51] Wahid Bhimji Ewan? [11:06:07] raul lopes joined [11:06:23] Ewan Mac Mahon Yes; I'm minuting. [11:07:02] Wahid Bhimji ok great. thanks [11:08:43] Queen Mary, U London London, U.K. joined [11:10:10] Ewan Mac Mahon There have been a couple of email about this. [11:10:34] Alessandra Forti joined [11:11:09] Gareth Smith joined [11:13:37] Jeremy Coles Hi Mark. Welcome. For others - Mark is a new sysadmin at Liverpool. [11:14:41] Ewan Mac Mahon Ticket number for this anyone? [11:15:44] Jeremy Coles 77026 [11:15:47] Ewan Mac Mahon Thanks. [11:16:06] Queen Mary, U London London, U.K. BDII problem GGUS:77026 [11:21:52] Mark Norman Thanks Jeremy, look forward to working with everyone. [11:27:13] Ewan Mac Mahon The metrics are clearly not useful if we're gaming them. [11:27:18] Ian Collier left [11:27:33] Ewan Mac Mahon However, gaming the metrics might be useful in other ways, and to us, but it clearly breaks the metric. [11:27:57] Elena Korolkova if there are metrics for the tickets why then we assign tickets to install cvmfs by ourselves? [11:28:44] Alessandra Forti https://ggus.eu/ws/ticket_info.php?ticket=77082 [11:29:08] Alessandra Forti left [11:29:12] Stephen Jones we can hear you [11:29:18] Alessandra Forti joined [11:40:31] Wahid Bhimji matt was a bit quiet for me [11:42:35] Gareth Smith For Elanea (Atlas) - Regarding your comment on failing transfers to RAL Tier1 this morning. I have checked with Castor team. There was a locking issue for the database behind the Atlas SRM between about 07:45 and 08:30. resolved when staff arrived at work. [11:43:00] Elena Korolkova thanks, Gareth [11:49:43] Jeremy Coles Matt - I think you may have a problem with your mic today as you were faint. For next week perhaps we can check tickets 5-10 mins before the meeting and discuss via email on Monday afternoon. [11:49:56] Alessandra Forti https://www.gridpp.ac.uk/wiki/UK_CVMFS_Deployment#Current_Deployment_status [11:50:52] Ian Collier joined [11:54:39] RECORDING Ewan joined [11:55:00] Santanu Das left [11:56:33] Santanu Das joined [11:57:00] Elena Korolkova there is no ADC today [11:57:38] Alessandra Forti http://dashb-atlas-ssb.cern.ch/dashboard/request.py/siteviewhistorywithstatistics?columnid=562&view=Shifter%20view [12:01:42] Ewan Mac Mahon I'm sure I've seen a 'how to remove an SE' page somewhere before. [12:02:22] Stephen Jones Sam, could you give us the link? [12:03:33] Queen Mary, U London London, U.K. https://www.gridpp.ac.uk/wiki/SE_Shutdown [12:04:27] Sam Skipsey As Chris notes. The minimum notice required is a month; I'd actually probably give 6 weeks. [12:04:30] Stephen Jones Cheers Chris. [12:04:38] Alessandra Forti https://indico.cern.ch/materialDisplay.py?contribId=6&materialId=slides&confId=163359 [12:04:55] Ewan Mac Mahon I agree, It should just cease to exist. [12:04:57] Alessandra Forti This is the talk about atlas brokerage [12:05:04] Ewan Mac Mahon Without killing jobs, for preference. [12:05:53] Alessandra Forti this is the live monitoring link [12:05:55] Alessandra Forti https://atlasbrokeragemonitoring.web.cern.ch/atlasbrokeragemonitoring/weekly.html [12:06:06] Ian Collier left [12:06:13] Ewan Mac Mahon You don't mark things as decomissioned, you just disappear them. [12:06:32] Ewan Mac Mahon Once something's not advertised, it doesn't exist. People aren't supposed to remmeber things. [12:06:51] Ewan Mac Mahon They're supposed to look them up 'live'. [12:07:48] Gareth Smith left [12:08:43] Stephen Jones yes [12:09:21] raul lopes left [12:14:07] Elena Korolkova how we can plan networking upgrade from gridpp4 grant if we doesn't know about networking grant? [12:18:07] Jeremy Coles https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items [12:19:02] Matthew Doidge talking of tarballs, any news on an EMI WN tarball? [12:20:56] Wahid Bhimji done [12:21:08] Wahid Bhimji (oh maybe not done) [12:23:35] Elena Korolkova Why should we have special tests for quarter reports? [12:23:51] Elena Korolkova we have nagios tests [12:25:17] Duncan Rand also, why atlas? [12:27:05] Jeremy Coles historical [12:31:20] Matthew Doidge sorry, got to run [12:31:28] Matthew Doidge left [12:33:04] Stuart Wakefield left [12:33:09] Mohammad kashif left [12:33:13] Elena Korolkova left [12:33:13] Rob Harper left [12:33:14] Andrew McNab left [12:33:14] John Bland left [12:33:16] David Crooks left [12:33:16] Duncan Rand left [12:33:16] Sam Skipsey left [12:33:17] Emyr James left [12:33:18] Chris Brew left [12:33:19] Govind Songara left
There are minutes attached to this event.
Show them.