UKI Monthly Operations Meeting (TB-SUPPORT)

Thursday 16 Oct 2008, 10:30 → 12:00 Europe/London

EVO - GridPP Deployment team meeting

Jeremy Coles

- This is the monthly UKI meeting - The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area. - The phone bridge number is +41 22 76 71400. The phone bridge ID is 482161 with code: 4880.

TB-Support 16th October Minutes Derek Ross Jeremy Coles Elena Korolkova Winnie Lacesso Mike Kenyon Rob Harper Matthew Doidge Stephen Childs Brian Davies Graeme Stewart Chris Brew Yves Coppens Duncan Rand Gianfranco Sciacca Peter Love Stephen Burke Pete Gronbech Sam Skipsey Ewan MacMahon Stephen Jones Rob Fay Simon George Alessandra Forti IPP1 Durham Stephen Dallison Santanu Das Site Availability ================= Problems at RAL-Tier 1 - Downtime EFDA-JET - unknown UK Tests -------- - Brunel - problem with file Steve uses on down filesystem - QMUL - same problem Atlas Tests ----------- Brunel - as above Imperial (both) - RHUL - BHAM - DNS problems, also problematic switch, GPFS software area on one cluster disappearing, replacing a CE. EFDA-JET - down at the moment Oxford - intermittent issues with maui crashing RAL T1 - Banned normal atlas users, Steve tried with a /atlas/uk/Role=production but had problems with T2 sites so switched back Acounting --------- UCD - should be unregistered IC-LESC - tomcat failure, being reinstalled QMUL - Monbox taken offline WLCG Draft Availabilty reprt for September - contact T2 coordinator about discrepancies Experiment Problems/Issues ========================== CMS --- CRAFT, lot of MC production, activity mainly at T1. No problem maintaing links in UK, low rate monitoring of links starting. Brunel issue - Atlas space tokens took all space - now fixed LHCB ---- Gridmap file changes Atlas ----- Detector taking comsic rays, high quality cosmic rays taken till end of month. RAW sizes are bigger, data being take 24hrs rather than 14hr, so data in T0 much higher - buffers stretched. Issues exporting to T1, data sometimes has to be recalled from tape. MC production bursty - 2-3 days of activity. Problems installing latest Atlas releases at ECDF. 2008 MC AOD being distributed over T2 in next few weeks, need 5TB in mCDISK Will then have a analysis challenge to get users to use different tools. Want GLEXEc and SCAS. Oxford may replicate data, if not centrally distributed. ROC/WLCG stuff ============== WLCG Update ----------- - GDB on T2 issues, experiments (apart from Alice) happy with T2s, some middleware will be brought forward - FTS on SL4, GLEXEC ready but SCAS needs more development. Missing procedures for killing jobs, poor traceability of jobs to catch errors. Expectation for data taking is now April/May Alice has asked sites to volunteer to deploy CREAM CE, request was made at Weekly Monday Operations meeting Will Alice accept pool accounts for vobox ssh access? Gathering Site Information ========================== Want to gather info about memory limits on batch systems - Atlas are interested in this information, jobs have occasional memory spikes, so need to know batch system policies about killing jobs. Want to know about problems with site WAN in the last year - capping, outages, anything leading to WAN becoming a bottleneck, plus forward look for 1-2 yearsi.e. increasing WAN connectivity. Middleware questions from EGEE'08 ================================= iii Acceptable versions Would like checkpointed releases which would solve these problems AOB === Please share experiences of recent purchases on Wiki HEPIX in Taipei next week HEPSYSMAN - suggestions if/when/where/topics for next HEPSYSMAN meeting conseunsus that it Would be useful to have one Chat Window =========== [10:30:02] Winnie Lacesso joined [10:30:04] Mike Kenyon joined [10:30:09] Matthew Doidge joined [10:30:09] Rob Harper joined [10:30:21] Stephen Childs joined [10:30:28] Brian Davies joined [10:31:39] Graeme Stewart joined [10:31:49] Chris Brew joined [10:31:50] Yves Coppens joined [10:33:53] Duncan Rand joined [10:34:12] Stephen Childs left [10:34:53] Stephen Childs joined [10:35:08] Gianfranco Sciacca joined [10:35:10] Peter Love joined [10:37:03] Stephen Burke joined [10:37:32] Pete Gronbech joined [10:38:56] Sam Skipsey joined [10:39:45] Ewan Mac Mahon joined [10:43:52] Winnie Lacesso Same intermittent SAM maradonna errors at LeSC .. probably also still SL3? [10:46:51] Stephen Jones joined [10:47:18] Rob Fay joined [10:48:23] Ewan Mac Mahon For the record the Oxford maui had died again; it's been kicked. [10:53:48] Simon George joined [10:56:10] Alessandra Forti joined [10:58:34] Alessandra Forti I can hear someone talking on top of graeme [10:58:44] Sam Skipsey We actually think we know why there's an issue at ECDF now. [10:58:48] Sam Skipsey But it's on going. [10:58:55] Elena Korolkova the same for me [10:59:53] Chris Brew left [11:01:44] IPPP1 Durham joined [11:02:11] Chris Brew joined [11:10:48] Stephen Burke I'm not sure if sites can see that page, the access is protected ... [11:11:04] Alessandra Forti no they can't [11:11:20] Derek Ross I certainly can't [11:12:00] Stephen Burke Actually I don't think there's a tier-1 address on it anyway - maybe that should be added ... [11:17:31] Winnie Lacesso Did the sound shut off? Got not sound. [11:18:22] Ewan Mac Mahon There's still sound - try turning off with F8 and back on again. [11:19:04] Winnie Lacesso Brilliant Ewan! Tx. [11:22:23] Stephen Dallison joined [11:22:34] Santanu Das joined [11:23:53] Duncan Rand back in a minute [11:37:12] Winnie Lacesso I'm listening &Manchester or Durham would be good. [11:37:26] Rob Harper Still here! I'll come if I can. [11:37:29] IPPP1 Durham Durham is listening... and would be interested to attend [11:37:31] Elena Korolkova yes, I'm listening [11:38:36] Rob Fay Yes [11:38:38] Stephen Dallison yes to sys man [11:38:40] IPPP1 Durham left [11:38:40] Santanu Das Jeremy, will be here for a min? [11:38:42] Ewan Mac Mahon HepSysMan: yes, [11:38:43] Alessandra Forti left [11:38:45] Graeme Stewart left [11:38:45] Sam Skipsey In principle, yes, in practise, might not make it. [11:38:48] Yves Coppens yes, could hold it in Bham subject to room availability [11:38:48] Matthew Doidge yes [11:38:50] Stephen Jones left [11:38:56] Derek Ross yes (someone from RAL-LCG2 would be there) [11:38:56] Stephen Burke left [11:38:57] Elena Korolkova yes [11:38:57] Sam Skipsey left [11:39:01] Duncan Rand left [11:39:02] Simon George maybe my mic doesn't work ... in principle interetsed in HEPSYSMAN, depends on the date and location as I have a busy schedule [11:39:08] Yves Coppens left [11:39:16] Chris Brew yes [11:39:17] Stephen Dallison left [11:39:22] Chris Brew left [11:39:28] Rob Fay left [11:39:33] Elena Korolkova left [11:39:38] Mike Kenyon HEPSYSMAN. Attend: yes; hold: probably not [11:39:38] Winnie Lacesso left [11:39:38] Rob Harper left [11:39:52] Peter Love left [11:40:01] Brian Davies might be able to attend hepsysman [11:40:06] Simon George left [11:40:26] Gianfranco Sciacca left [11:40:55] Mike Kenyon left [11:41:07] Ewan Mac Mahon left [11:41:09] Santanu Das hi Pete

10:30 → 10:45 Site availability

SAM tests: http://pprc.qmul.ac.uk/~lloyd/gridpp/samtest.html - EFDA - RAL Tier-1 UK tests: http://pprc.qmul.ac.uk/~lloyd/gridpp/uktest.html - Brunel - very high failure rate - QMUL - high failure rate ATLAS tests: http://pprc.qmul.ac.uk/~lloyd/gridpp/atest.html Problems at: - Brunel - Imperial (both) - RHUL-01 - UCL-HEP - BHAM (both) - JET - RAL T1 LHCb tests; http://pprc.qmul.ac.uk/~lloyd/gridpp/lhcb_samtest.html Most have warnings but fails at: - IC-LeSC - BHAM Accounting: http://www3.egee.cesga.es/gridsite/accounting/CESGA/egee_view.php - mpUCDie - IC-LeSC - QMUL - We looked at what APEL tells us about site efficiencies at the last DTEAM meeting: http://tinyurl.com/3s5u5u. The main observation was that sites showing low efficiencies for a given VO tend to only receive test or pilot jobs. - GridMap: http://gridmap.cern.ch/gm/ Nothing additional on above. - WLCG draft availability report for September: https://twiki.cern.ch/twiki/bin/viewfile/LCG/SamMbReports?filename=tier2_rel_sep2008.pdf

10:45 → 10:55 Experiment problems/issues

CMS: Now starting cosmic data taking with higher magnetic field. Some issues with maintaining site links. LHCb: Ran in to two problems with SAM tests last week. One due to submission problem and one due to scripts not running properly. They have a proposal for populating the gridmap file which will be circulated in a broadcast shortly: http://indico.cern.ch/getFile.py/access?subContId=6&contribId=2&resId=0&materialId=0&confId=42750. ATLAS: Ran well on UK T2s until hit problems with CASTOR at T1. Currently they have issues with T0 exports and use of containers (collection of data sets). A reminder of ATLAS storage requirements: https://twiki.cern.ch/twiki/bin/view/Atlas/StorageSetUp.

10:55 → 11:05 ROC/WLCG stuff

ROC update *************** 1) There were some Oracle DB errors last week which led to the site and ROC reports missing the SAM error information. This has now been fixed. 2) There has been a changeover of staff who deal with tickets in the ROC heldesk. As a consequence and to aid ticket assignments Alessandra set up this page: https://www.gridpp.ac.uk/deployment/support/index.html. So, if you are wondering where the ticket assignment addresses are coming from, now you know! 3) We (DTEAM+) are holding a joint workshop with NGS ops people next week. If your site has become an affiliate or you have other comments on working with NGS, please let Jeremy or your Tier-2 coordinator know. 4) A reminder of how to schedule a downtime to satisfy EGEE/WLCG: https://edms.cern.ch/file/829986/0.1/EGEE-intervention-procedures.pdf. It covers different services and intervention types with different lead times for the announcement. T1 news ********** Last Monday Catalin announced the following: "Following a successful trial period of the gLite WMS, we are announcing the decommissioning of the EDG Resource Brokers located at RAL Tier1 (lcgrb01, lcgrb02 and lcgrb03.gridpp.rl.ac.uk)". The plan is to drain the RBs on Friday 28th November. The machines will be switched to WMS services. Users were advised to switch job submission to the gLite WMS server at RAL." WLCG update *************** 1) There was a GDB on 8th October that looked at T2s. The agenda is here: http://indico.cern.ch/conferenceDisplay.py?confId=20234. You may find the experiment talks interesting (basically review how T2s fit into their models) but the most useful talks for UK sysadmins were Oliver Keeble's update on the middleware: http://indico.cern.ch/materialDisplay.py?sessionId=7&materialId=1&confId=20234 Main points: and Steve T's review of batch systems: http://indico.cern.ch/getFile.py/access?sessionId=6&resId=0&materialId=3&confId=20234. Main points: procedures for killing jobs at memory limits are not well establised and more work needs to be done on tracing and profiling jobs (to catch frequent errors in jobs etc.). There was talk about planning changes given that the LHC schedule has changed, but the overall conclusion seems to be carry on as we are. From John Shade's talks note these useful links on Nagios: - Multi-level monitoring framework (James Casey) http://indico.cern.ch/getFile.py/access?contribId=344&sessionId=53&resId=4&materialId=slides&confId=32220 – Introduction to Nagios (Ronald Starink) http://indico.cern.ch/materialDisplay.py?contribId=239&sessionId=54&materialId=slides&confId=32220 – Grid Site Monitoring with Nagios (Emir Imamagic) http://indico.cern.ch/getFile.py/access?contribId=240&sessionId=54&resId=0&materialId=slides&confId=32220 ESPECIALLY: – Grid Site Monitoring Demo (Steve Traylen) https://twiki.cern.ch/twiki/bin/view/LCG/GridMonitoringNcgYaimTutorial?slideshow=on;skin=print#GoSlide1

WLCG downtime procedure SC4-scheduled-maintenance-June21.pdf

11:05 → 11:15 Gathering site information

There are two areas where GridPP wishes to have sysadmin input on site information. One concerns batch system memory limits and the other network connectivity. - This wiki area has been setup for you to complete: https://www.gridpp.ac.uk/wiki/Site_information

11:15 → 11:25 Middleware questions from EGEE'08

The main session on deployment and operations issues http://indico.cern.ch/materialDisplay.py?contribId=342&sessionId=60&mate rialId=slides&confId=32220 raised the following questions: i) gLite version - how do you want versions published and to what level is required/acceptable? ii) How should WN version(s) be published. [Note that several versions may be installed on a WN. VOs would like their jobs to have more information about the environment before submission]. iii) Acceptable versions - previously this was defined by a "minimum" version number and was clearest for the SEs. How should this be published and used? How do you want to know? iv) End-of-life. When a middleware version is no longer supported, what process should be employed to remove it? Nb: If these points are familiar it is because they were posted on TB-SUPPORT. One response came from Henry.

11:25 → 11:30 AOB

- Please try to add information to this page if and when you make new purchases http://www.gridpp.ac.uk/wiki/Guidance_and_recent_purchases. Please! - The Autumn HEPiX is in Taipei next week. Some of the talks are uploaded already and worth reviewing: http://tinyurl.com/4wm7lv