Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
Description
- This is the biweekly Ops-team & sites meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area.
- The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 77907 with code: 4880.
Apologies:
GridPP Operations team meeting 24/05/11 11:00 In attendance: Jeremy Coles (Chair), Stephen Jones (Minutes), Alessandra Forti, Andrew Washbrook, Ben Waugh, Catalin Condurache, Chris Brew, Christopher Walker, Daniela Bauer, David Crooks, Duncan Rand, Emyr James, Ewan Mac Mahon, Gareth Smith, Govind Songara, Jeremy Maris, Mark Mitchell, Mark Slater, Matthew Doidge, Mingchao Ma,Mohammad kashif, Peter Grandi, Raja Nandakumar, Raul Lopes, Rob Fay, Rob Harper, Sam Skipsey, Santanu Das, Stuart Purdie, Stuart Wakefield, Wahid Bhimji Meetings & updates - ROD team update Kashif: Nothing much to report - going fine. - EGI operations update Jeremy: No meeting this week. Next meeting 30th. Stuart: Nothing new to report. -- EMI 1/Kebnekaise released (notes: http://www.eu-emi.eu/emi-1-kebnekaise) Jeremy: Would early adopters please provide their input? Stuart: EMI not yet in staged role out. Jeremy: The release system is confusing at present. Duncan: Can CREAM be used with ARGUS? Kashif: Not Yet. Wait for EMI. Raul Lopes: The CREAM+ARGUS configuration is now in test. - Nagios status Kashif: Changes made to incorporate glexec tests into Nagios. Linked to GOCDB. Only Liverpool and Oxford registered at present. - Tier-1 update Gareth: Generally quiet. A new Castor instance is being prepared for internal use only. Some problems have been seen with timeouts on CVMFS, causing lhcb SAM tests to fail. This sporadically affects availability, but doesn't have a major effect. WMS03 has been reinstalled after problems; CEs have been converted to CREAM and configured in Quattor. There have been complaints from Alice about access to RAL clusters. Problem traced to slow rate of starts versus short job duration. To be fixed in MAUI. - Security update <nothing to add> -- T2 issues Jeremy: Manchester, Sheffield and RALPP publishing is behind. Alessandra: Publishing is behind because of hardware problems at RAL. Christina has fixed it now. - HEPSYSMAN -- 30th June - 1st July (RAL): http://hepwww.rl.ac.uk/sysman/June2011/main.html Gareth: Please apply now for accommodation block bookings. Gareth/Jeremy: Agenda still being formed. It will include site reports. Please give agenda ideas to Pete and/or Gareth/Jeremy. - Next GDB 8th June : http://indico.cern.ch/conferenceDisplay.py?confId=106645 Jeremy: Will a member of the Core Ops Team please volunteer to be a T2 rep for the GBD? Experiment problems/issues Review of weekly issues by experiment/VO (Agreement to be moved to 11:20) - LHCb Raja: Nothing major. CVMFS issue is not causing major problems. Cambridge is moving to PBS, and has had some problems running jobs. Santanu Das: Some problems with lhcb pilots due to group issues. Under investigation. Raja: Imperial is fine for lhcb, suffering less job aborts. - CMS Stuart Wakefield: Things are running well. Running lots of jobs. Imperial has implemented a new dcache fairly smoothly. - ATLAS Alessandra: ANALY_RAL: transfer to and from RAL SCRATCHDISK started to fail due to atlas analysis jobs writing aggressively to the space token. Jobs slots have been throttled to reduce the load and 7 disk servers have been put in read only mode to drain them and spread the load to more servers. Atlas should start a cleanup process on their side but better wait for a further update from RAL. GGUS: https://gus.fzk.de/ws/ticket_info.php?ticket=69863 Alessandra: UKI-SCOTGRID-GLASGOW: Bad transfer efficiency due to RFIO/File Delete interaction. Some badly deleted files were cleaned up by Sam. Alessandra: IMPERIAL: 100% production failure due to software area being full. This caused an error during software install, and a clean-up is required. Duncan: 250 GB not enough. Increase or clean systemically. Jeremy: Update the VOID card to show more disk needed. Duncan: It should check the space before trying, then it won't break. Jeremy: A ticket is required. Duncan: Provide more than we'll ever need. Royal Holloway has 1TB. Chris: QUEEN MARY: Occasional transfer time-outs due to time taken for storage checksum generation. Further ATLAS Notes: Alessandra: Increase UK cloud space on production Space to a safer value, as it is better to have some margin than for a site to get blacklisted. Atlas requires a better clean-up policy, which will soon be implemented. Clean-up will occur as soon as possible after job completes. Factory configuration should be kept in sync. For example, RH is in one, but not the other). Lancs transition to T2D status delayed due to network issues,. - Other Alessandra: Massive number of pheno jobs currently queued. - Experiment blacklisted sites - Experiment known events affecting job slot requirements - Site performance/accounting issues - Metrics review CREAM status Snapshot at http://www.gridpp.ac.uk/wiki/Site_status_and_plans Jeremy: GDB wishes CREAM to be deployed by July. Jeremy: Switch to CREAM from lcg-CE likely around August/September. Jeremy: Sites in progress are Durham, Cambridge ... CREAM Problems: Stuart Purdie: There may be glibc issues with newest release. Investigating. Chris Walker: Problems observed with SGE CREAM under heavy load. Stuart: Somehow related to BLAH interaction. Kashif: Also seen problem, no pattern except heavy load. Andrew Washbrooke: VOMSValidator falls back to PEMs for one user. Working with devel. glexec & ARGUS - Site checks on status - Any developments Daniela: Hacks are required for operation with SGE. Taking material direct from developers. - News on the relocatable install Jeremy: Still no news on relocatable (tarball). Waiting on Marin Litmaath. If not delivered, the June deadline may slip due to factors outside our control. Ewan: I shall not create our own tar ball. We now depend on delivery of official version. Noted changes to status: Several sites waiting for tarball and/or SGE hacks. Liverpool: ARGUS and glexec installed on whole of main cluster. Still waiting for plan on whether/how to install on SGE cluster (tarball reqd). Manchester: ARGUS installed. glexec in plan (1 node done). Durham: No plans (not needed as no analysis). May need eventually (PG). Can start with single user pilot for Atlas. ECDF: No plans (but no objections). Would need SGE version. Glasgow: SCAS & glexec deployed. Further rollout with ARGUS May/June. Birmingham: Deployed for local cluster. Tarball needed for shared cluster. Bristol: No current plans. Cambridge: Condor has been swapped to PBS. Planning Argus/GlExec ~ 1 month. EFDA: Low priority. Not planned. Oxford: Deployed. RALPP: Installed and working. Tier-1: Deployed with SCAS. ARGUS in planning. AOB - WLCG workshop http://indico.desy.de/conferenceDisplay.py?confId=4019. Will shortly confirm places (of order 15). Jeremy: Please let me know if anyone else is interested - places are short. David Crooks: Accounting at Glasgow for new nodes was not being scaled properly for one week. A correction has been made that makes future reading correct, but history is off by a factor. :--- CHAT WINDOW --- 10:56:27] Stuart Purdie joined [10:56:34] Jeremy Maris joined [10:56:39] Queen Mary, U London London, U.K. joined [10:56:50] Peter Grandi your audio fine here [10:57:37] Peter Grandi having difficulty with the microphone device here... [10:58:12] Jeremy Coles Hi Peter. I'll watch the chat window in case you type a comment/response/question..... [10:58:31] Daniela Bauer joined [10:59:23] Mark Mitchell joined [10:59:35] Sam Skipsey joined [11:00:15] Mark Slater joined [11:00:16] Peter Grandi Ok that was better, thanks. I'll get the volume right now. [11:00:31] Matthew Doidge joined [11:01:10] Duncan Rand joined [11:01:12] Gareth Smith joined [11:01:35] David Crooks joined [11:01:38] Mohammad kashif joined [11:01:40] raul lopes joined [11:02:26] Jeremy Coles Stephen is taking minutes. [11:02:52] Ewan Mac Mahon joined [11:03:04] Stuart Wakefield joined [11:03:22] Rob Harper joined [11:04:18] Chris Brew joined [11:04:36] Ben Waugh joined [11:05:16] Alessandra Forti joined [11:05:52] Christopher Walker joined [11:06:08] Christopher Walker left [11:06:14] Andrew Washbrook joined [11:06:27] RECORDING Christopher joined [11:06:36] Christopher Walker joined [11:07:05] Christopher Walker left [11:07:06] Christopher Walker joined [11:08:12] Raja Nandakumar joined [11:09:31] Wahid Bhimji joined [11:15:19] Peter Grandi BTW I recently discovered in the APEL pages somewhere per-site list of APEL syncs, e.g. for Durham: http://goc-accounting.grid-support.ac.uk/rss/UKI-SCOTGRID-DURHAM_Sync.html [11:16:09] Ewan Mac Mahon Pete's on a train. [11:22:58] Santanu Das where is the HEPSYSMAN this time? [11:23:04] Jeremy Coles RAL [11:25:24] Mingchao Ma joined [11:29:40] Peter Grandi we only run production (mostly ATLAS) and 490GB used of exp-sw area. [11:30:28] Mark Mitchell Jeremy we have about 3.5 TB for the software releases [11:30:42] Sam Skipsey That's "available space" not "used". [11:30:52] Jeremy Coles Thanks for checking. [11:31:55] Peter Grandi BTW 490GB of used here and 90GB of free. I need anyhow to move that area to a new server, and it will become bigger. [11:32:08] Sam Skipsey (They're actually using 1.4T of space for all experiment software across all VOs, but ATLAS are the majority of that.) [11:38:20] Queen Mary, U London London, U.K. Alessandra, can you mute please [11:40:14] Mohammad kashif t2ce04.physics.ox.ac.uk is our new creamce, we are getting atlasprd jobs but no atlaspilot [11:43:25] Mohammad kashif glexec services tested by gridppnagios https://gridppnagios.physics.ox.ac.uk/nagios/cgi-bin/status.cgi?servicegroup=org.sam.glexec.CE&style=detail [11:46:34] Mark Mitchell Jeremy we are getting audio break up at Glasgow [11:46:56] Mark Mitchell It appears to be a Glasgow issue [11:47:30] Jeremy Coles Hi Mark - yes it has been fine for me. [11:48:12] Andrew Washbrook Cream@ECDF issues: persistent load issues under VM instances, VOMSValidator fails for specific users (needs fallback to pem files) [11:52:00] Stephen Jones Glexec + Argus now deployed in L'pool on all WNs in main cluster (not small SGE cluster) [11:53:53] Ewan Mac Mahon Andrew: What sort of disk system does that heavily loaded CREAM have under its VM? [11:54:06] Catalin Condurache joined [11:54:08] Catalin Condurache left [11:54:16] Ewan Mac Mahon We've got two CREAM CEs at Oxford and the one with slower disks always shows high load. [11:54:51] Ewan Mac Mahon We think it's probably something to do with the MySQL keeping trying to write things to disk, but we haven't absolutely confirmed. [11:55:47] Andrew Washbrook hmm good question - not sure offhand but I'll ask our systems team for confirmation [11:56:05] Andrew Washbrook i thought it might be mysql, but seeing interesting issues with tomcat server as well [11:56:38] Andrew Washbrook (service gLite stop does not stop the tomcat owned processes after timeout) [11:56:47] Andrew Washbrook dont know if this is all related.. [11:57:20] Mohammad kashif https://samnag023.cern.ch/nagios/cgi-bin/status.cgi?hostgroup=United+Kingdom&style=detail [11:57:38] Wahid Bhimji Stable SGE version [12:03:23] Duncan Rand multi-user pilot [12:08:30] Mohammad kashif https://gridppnagios.physics.ox.ac.uk/nagios/cgi-bin/status.cgi?servicegroup=org.sam.glexec.CE&style=detail [12:12:00] Alessandra Forti bye [12:12:01] David Crooks left [12:12:01] Wahid Bhimji bye [12:12:02] Stephen Jones thanks [12:12:02] Catalin Condurache left [12:12:02] Raja Nandakumar left [12:12:03] Alessandra Forti left [12:12:03] Wahid Bhimji left [12:12:03] Ben Waugh left [12:12:04] Govind Songara left [12:12:05] Daniela Bauer left [12:12:05] Mark Slater left [12:12:06] Sam Skipsey left [12:12:07] Matthew Doidge left [12:12:07] Mohammad kashif left [12:12:08] Peter Grandi left [12:12:08] Mark Mitchell left [12:12:08] Stuart Wakefield left [12:12:10] Andrew Washbrook left [12:12:12] Ewan Mac Mahon left [12:12:13] Rob Harper left [12:12:13] Gareth Smith left [12:12:16] Emyr James left [12:12:21] raul lopes left [12:12:27] Chris Brew left [12:12:28] Mingchao Ma left [12:12:32] Duncan Rand left [12:12:53] Santanu Das left [12:14:33] Jeremy Maris left [12:15:21] Rob Fay left [12:15:26] Stuart Purdie left
There are minutes attached to this event.
Show them.