Choose timezone
Your profile timezone:
Operations team (& sites)
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the biweekly OPS team meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area.
- The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 78425 with code: 4880.
Apologies: Raja, Alessandra, Mingchao
GridPP Operations team meeting 17/05/11
In attendance: Jeremy Coles (Chair), Mark Mitchell, Brian Davies, Daniela Bauer, Mark Slater, Govind Songara, Catalin Condurache, Stuart Wakefield, Pete Gronbech, David Crooks (Minutes), Gareth Smith, Matthew Doidge, Stephen Jones, Andrew McNab, Wahid Bhimji, Sam Skipsey, Duncan Rand, Chris Walker, Stuart Purdie
Apologies: Raja, Alessandra, Mingchao
11:00
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
-- There is nothing special to report for LHCb - some issues with Tier-2 sites which are being dealt with either by GGUS or directly by email. Nothing of note in the pipeworks.
- CMS
Stuart: Discussion within UK about which physics groups to support. Sites have more disk space, looking at more groups, ongoing. Other issue, discovered problem with CMS/gsi dcap setup at Imperial, turned on dcap access after discussion. CMS now running happily.
Jeremy: Not just disk space, also quality of service.
Stuart: Allocation made a while ago for physics groups. Some sites not working as well as expected, so some rebalancing to take place.
Jeremy: CMS worldwide?
Stuart: Right.
- ATLAS
Brian: ATLAS were having an issue at T1 which caused many jobs to go out to T2 which filled a lot of PRODDISKs. ATLAS Central had database intervention at CERN at midnight which meant all sites offline. T2 work dropped dramatically over last 12 hours. Panda/job management down at the moment. Cause similar to issue of tags for releases at T1 meant that jobs were then reallocated from T1 to T2s but jobs reallocated meant that they needed a lot more data so spacetokens filled up.
Jeremy: Is this reallocation part of the model?
Brian: yes, if jobs can't run at T1 then it can run at T2. Higher requirements for these jobs
Jeremy: This has happened before, T2s get blacklisted because they fill up.
Brian: Yes, there’s an issue of the spacetoken space availability at which a site is blacklisted, and also rate at which spacetoken is cleaned up. Within UK, monitoring situation, asking sites to increase spacetoken size as needed. Greater issue, being discussed tomorrow. Manchester has been blacklisted as a result.
- Other
- Experiment blacklisted sites
- Experiment known events affecting job slot requirements
- Site performance/accounting issues
- Metrics review
Jeremy: Metrics in terms of accounting period will be reviewed at end of the month. Durham fell out because VO share wasn't published.
Mark Mitchell: Fixed on Friday. YAIM was issue, waiting for full report. Problem at site, resolved quickly.
11:20
Meetings & updates (20')
- ROD team update
- Nagios status
Jeremy: Not sure we need to keep this as a standing item.
- Tier-1 update
Gareth: Fairly quiet week, 2 things:
1) Moved ATLAS as well as LHCb to use CVMFS for software area. Going ok but some SAM test failures, in some cases it needs a kick on particular worker node. May be fixed with latest version of client. Pleased because problems with ATLAS software server (NFS) for some time.
2) Operational problem yesterday morning caused by castor database, took some castor instances off air for ~ 1 hour. Problem somewhere in oracle system.
Some other internal changes. Some work on WMS ongoing, rebuilding one of the CEs.
Chris: CVMFS problems: I’ve noticed some errors in the logs, could you expand? Gareth: Problems we see were in LHCb SAM tests. The issues were that the ls recursion timed out. CVMFS may not be responding. Haven't seen any in last 24 hours.
- Security update
-- Still need a London SSC5 site.
London SSC5 site to be arranged – discussed, but to be confirmed.
-- T2 issues
-- General notes.
- Escalated tickets https://gus.fzk.de/download/escalationreports/roc/html/20110516_EscalationReport_ROCs.html
57746 - Camont issue with Cam-HEP. Condor related? More job outputs given so back to the site - Santanu!
Jeremy: Problem with way condor reads logs. Original ticket submitted by Carl Harrison, back with Santanu because original problem not resolved.
64995 - RAL T1 No GlueSACapability defined for WLCG Storage Areas. On hold but now progressing. Change ticket status?
Jeremy: Jens says this is in progress
Gareth: On hold because it's in progress but in development, I still get note of it in weekly report
65991 - pheno unable to see DNs for some site accounting:
I can successfully identify the numbers for RAL and GLASGOW from your list.
All sites that have reported something during the time period 1-30 April:
Netlab1.UKI-LT2-Brunel
RAL-LCG2
Triton.UKI-LT2-Brunel
UKI-LT2-IC-HEP
UKI-LT2-QMUL
UKI-LT2-RHUL
UKI-NORTHGRID-LIV-HEP
UKI-SCOTGRID-GLASGOW
UKI-SOUTHGRID-BHAM-HEP
UKI-SOUTHGRID-OX-HEP
UKI-SOUTHGRID-RALPP
not visible for me are:
UKI-NORTHGRID-LANCS-HEP: Matt – I haven’t made the changes, I need to do that, will do so today.
UKI-NORTHGRID-MAN-HEP: Duncan: It has been discussed, need to check status
UKI-NORTHGRID-SHEF-HEP
UKI-SCOTGRID-DURHAM: Mark Mitchell: I’ll check with Peter Grandi
66306 - Creation of Irish NGI. Child tickets almost complete.
Jeremy: A series of child tickets about what needs to be set up, almost complete
66564 - WLCG sites not publishing the GlueCECapability shared
ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=grid '(&(objectclass=gluece)(glueceuniqueid=*.ac.uk*)(!(gluececapability=share=*)))' | grep GlueCEHostingCluster: | sort | uniq
GlueCEHostingCluster: ce01.dur.scotgrid.ac.uk
GlueCEHostingCluster: ce02.dur.scotgrid.ac.uk
GlueCEHostingCluster: ce2.glite.ecdf.ed.ac.uk
GlueCEHostingCluster: ce3.glite.ecdf.ed.ac.uk
GlueCEHostingCluster: dgc-grid-44.brunel.ac.uk
GlueCEHostingCluster: hepgrid3.ph.liv.ac.uk
GlueCEHostingCluster: lcg-ce02.hep.ucl.ac.uk
GlueCEHostingCluster: lcg-ce04.hep.ucl.ac.uk
GlueCEHostingCluster: lcgce0.shef.ac.uk
Jeremy: Probably needs to be enabled on all CEs
Chris: I think so
Sam: Talked to Peter Grandi, fixed this as part of YAIM fix, will check.
Chris: Looks like Durham is fixed
Wahid: Can I check what this is needed for?
Jeremy: Management board for WLCG looking for consistency and accuracy in the information system.
Chris: Just a small change required.
Wahid: I’ll look into it
67322 - Wrong value for GlueHostProcessorOtherDescription
Derek can this be marked solved?
Catalin: Will check with Derek on the status of this.
67365 - WLCG sites not publishing the GlueCECapability Share
Looks like this is almost there.
68077 - Mandatory WLCG InstalledOnlineCapacity not published RAL-LCG2
Jeremy: In hand
69028 - GGUS-FAQ about Responsible Units
This looks pretty much resolved.
Several other tickets on hold.
68853 - Retirenment of SL4 and 32bit DPM Head nodes and Servers
Awaits end of SL4 gLite 3.1 T2 machines.
Brian : Should be on hold
68858 - Retirenment of SL4 and 32bit DPM Head nodes and Servers - Glasgow sub-ticket
Sam: Should be on hold, Brian knows the situation at Glasgow
68859 - Durham sub-ticket
68865 - UCL-HEP sub-ticket.
Then the currently open GGUS tickets summary:
http://tinyurl.com/6z6uq5v
Catalin: I updated all tickets regarding WMS, should be able to close them later in the week
Mark Slater: 70521: (Release 16.6.4 install failed on UKI-SOUTHGRID-BHAM-HEP ) Difficulty if you have 2 clusters with 2 software areas with different installs. Got much more of an idea now how to keep this from happening. In the future.
Duncan: 69336: (rfcp failures at MAN-HEP) Partly due to network issues: need to reboot servers for other changes. Should be dealt with soon.
11:40
GDB overview (10')
GDB agenda page:http://indico.cern.ch/conferenceDisplay.py?confId=106644.
Jeremy's summary: https://www.gridpp.ac.uk/wiki/GDB_11th_May_2011
Discussion:
Duncan: Where do we look to find availability? Steve Lloyd’s pages, Gridmap don’t include CREAM CE data. Could Gridmap include CREAM?
Jeremy: Check on GridPP Nagios. I’ll look into this
DPM/Nagios: Wahid: Similar things discussed to those mentioned in Storage Workshop about nagios probes that can be used for DPM. Will talk to Ricardo about including them. PUPPET not really used in the UK. Feedback from storage workshop was that we want more internal DPM Nagios probes.
Chris: QMUL having problem with CREAM CE, SAM tests seem to be fine but Steve Lloyd’s tests seem to be failing. Is this a known problem?
Stuart Purdie: There was something about this in the EGI SA-1 meeting yesterday. In staged rollout, if you are using most up-to-date glibc, they saw some crashes with blah against SGE, may affect all batch systems. Keep an eye on BUpdater, which might have some issues.
Chris: Seeing 100% CPU every time I’ve looked recently.
Matt: Also could be what we’re seeing at Lancaster. Also seeing >100% CPU on CREAM CEs . Posted ticket, gist is that bupdater gets a backlog of work which has to be worked through (intensive), best left alone to work through the backlog. Fix is "optimise code", not sure about the timeline for this.
SA-1 Meeting: Stuart: EMI1 is released, but if you are in staged rollout wait for another stage.
11:50
Accounting/publishing/etc. (05')
Just a quick check of:
http://www3.egee.cesga.es/gridsite/accounting/CESGA/egee_view.php
UCL-HEP not publishing. CE problems?
Daniela: CE problems not fixed, but publishing is fixed
Lancaster - Database problems affecting CREAM CE.
Matt: Looked like the database, but noted bug with Bupdater in CREAM
Glasgow - Sick server (network or disk?)
Mark: Network: Rebooted last Thursday, took cluster down and restarted. Memory error reoccurred. Some other corrective actions can be taken, we’ll be in at risk tomorrow. Buffers are running out of memory on some ports. Not getting many transfer errors. May need a code upgrade.
Sam: Disk server appears to be fine - all ATLAS based, so know exactly which files were affected. Not the disk server, problem was probably network issues. All production jobs affected were rerun.
Brunel - globus gatekeeper fails to start after upgrade
EFDA-JET - new BDII node
RHUL - nfs server problems
Govind: Condor problem with CREAM CE, fixed.
Birmingham - Disk/controller failure - impacts mySQL; BDII & APEL
Mark Slater: Down for about 4 days from Sunday-Thursday. Got new disk, looks like it was disk, not the controller. Affected SQL server which we weren't taking backups of (which we are now doing). All seems fine now, up since middle of last week. Having replaced disk has stayed up longer with replaced disk, still not sure.
Brian: Issue tracked by CMS concerning data transfers to Bristol, wondering if SouthGrid members not at Bristol were aware of the issue.
Ewan: Please escalate to GGUS so that we can track it and look into it.
11:55
Actions (05')
- http://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
12:00
AOB (01')
- WLCG workshop in July https://indico.desy.de/conferenceTimeTable.py?confId=4019#all. Currently expect GridPP travel funding to support the core ops-team as a first priority. For those on the ops-team with an interest but no response yet please let me know!
http://indico.cern.ch/event/126705
Jeremy: Asked about attendance to the workshop.
Gareth: HEPSYSMAN, June 30th/July 1, registration page will hopefully go up today or tomorrow.
Chat transcript:
[11:01:58] Mark Slater joined
[11:02:09] Queen Mary, U London London, U.K. joined
[11:02:29] Gareth Smith joined
[11:02:36] Catalin Condurache nope
[11:02:44] Brian Davies joined
[11:03:30] Daniela Bauer joined
[11:03:32] Catalin Condurache 've got it now
[11:03:33] Andrew McNab got that email now, thanks
[11:04:04] Duncan Rand joined
[11:04:04] Duncan Rand left
[11:04:04] Wahid Bhimji joined
[11:04:35] David Crooks joined
[11:04:35] Govind Songara joined
[11:05:01] Stuart Wakefield joined
[11:05:30] RECORDING David joined
[11:05:32] Jeremy Coles David is taking minutes.
[11:07:47] Jeremy Coles Will return to first agenda item "Experiment problems/issues"
[11:11:42] Duncan Rand david says: "one"
[11:13:12] Govind Songara Can someone paste agenda here please
[11:14:26] Matthew Doidge http://indico.cern.ch/conferenceDisplay.py?confId=126705
[11:16:21] Govind Songara Its old one
[11:19:29] Wahid Bhimji (my connection dropped- back now)
[11:19:57] Matthew Doidge Mine did too
[11:22:45] Pete Gronbech joined
[11:23:46] Pete Gronbech left
[11:24:34] Stephen Jones joined
[11:25:16] Pete Gronbech joined
[11:26:40] Stephen Jones Sorry for lateness - just read email
[11:41:42] Stuart Purdie joined
[11:43:07] Stuart Purdie Sorry I'm late - previous meeting overran
[11:43:49] Gareth Smith Some availability/reliability reports (incl. Tier2s) available at https://twiki.cern.ch/twiki/bin/view/LCG/SamMbReports
[11:43:59] Gareth Smith But the most recent seems to be February 2011
[11:58:51] Matthew Doidge https://savannah.cern.ch/bugs/?func=detailitem&item_id=79324
[12:03:24] Brian Davies left
[12:03:45] Brian Davies joined
[12:05:28] Mark Mitchell Jeremy I need to head off. Speak to you next week.
[12:05:34] Mark Mitchell left
[12:12:26] Brian Davies left
[12:12:29] Daniela Bauer left
[12:12:29] Mark Slater left
[12:12:29] Stephen Jones Thanks
[12:12:30] Govind Songara left
[12:12:30] Catalin Condurache left
[12:12:33] Stuart Wakefield left
[12:12:33] Pete Gronbech left
[12:12:34] David Crooks left
[12:12:34] Gareth Smith left
[12:12:36] Matthew Doidge left
[12:12:39] Stephen Jones left
[12:12:39] Andrew McNab left
[12:12:44] Wahid Bhimji left
[12:12:45] Sam Skipsey left
[12:12:52] Duncan Rand left
In attendance: Jeremy Coles (Chair), Mark Mitchell, Brian Davies, Daniela Bauer, Mark Slater, Govind Songara, Catalin Condurache, Stuart Wakefield, Pete Gronbech, David Crooks (Minutes), Gareth Smith, Matthew Doidge, Stephen Jones, Andrew McNab, Wahid Bhimji, Sam Skipsey, Duncan Rand, Chris Walker, Stuart Purdie
Apologies: Raja, Alessandra, Mingchao
11:00
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
-- There is nothing special to report for LHCb - some issues with Tier-2 sites which are being dealt with either by GGUS or directly by email. Nothing of note in the pipeworks.
- CMS
Stuart: Discussion within UK about which physics groups to support. Sites have more disk space, looking at more groups, ongoing. Other issue, discovered problem with CMS/gsi dcap setup at Imperial, turned on dcap access after discussion. CMS now running happily.
Jeremy: Not just disk space, also quality of service.
Stuart: Allocation made a while ago for physics groups. Some sites not working as well as expected, so some rebalancing to take place.
Jeremy: CMS worldwide?
Stuart: Right.
- ATLAS
Brian: ATLAS were having an issue at T1 which caused many jobs to go out to T2 which filled a lot of PRODDISKs. ATLAS Central had database intervention at CERN at midnight which meant all sites offline. T2 work dropped dramatically over last 12 hours. Panda/job management down at the moment. Cause similar to issue of tags for releases at T1 meant that jobs were then reallocated from T1 to T2s but jobs reallocated meant that they needed a lot more data so spacetokens filled up.
Jeremy: Is this reallocation part of the model?
Brian: yes, if jobs can't run at T1 then it can run at T2. Higher requirements for these jobs
Jeremy: This has happened before, T2s get blacklisted because they fill up.
Brian: Yes, there’s an issue of the spacetoken space availability at which a site is blacklisted, and also rate at which spacetoken is cleaned up. Within UK, monitoring situation, asking sites to increase spacetoken size as needed. Greater issue, being discussed tomorrow. Manchester has been blacklisted as a result.
- Other
- Experiment blacklisted sites
- Experiment known events affecting job slot requirements
- Site performance/accounting issues
- Metrics review
Jeremy: Metrics in terms of accounting period will be reviewed at end of the month. Durham fell out because VO share wasn't published.
Mark Mitchell: Fixed on Friday. YAIM was issue, waiting for full report. Problem at site, resolved quickly.
11:20
Meetings & updates (20')
- ROD team update
- Nagios status
Jeremy: Not sure we need to keep this as a standing item.
- Tier-1 update
Gareth: Fairly quiet week, 2 things:
1) Moved ATLAS as well as LHCb to use CVMFS for software area. Going ok but some SAM test failures, in some cases it needs a kick on particular worker node. May be fixed with latest version of client. Pleased because problems with ATLAS software server (NFS) for some time.
2) Operational problem yesterday morning caused by castor database, took some castor instances off air for ~ 1 hour. Problem somewhere in oracle system.
Some other internal changes. Some work on WMS ongoing, rebuilding one of the CEs.
Chris: CVMFS problems: I’ve noticed some errors in the logs, could you expand? Gareth: Problems we see were in LHCb SAM tests. The issues were that the ls recursion timed out. CVMFS may not be responding. Haven't seen any in last 24 hours.
- Security update
-- Still need a London SSC5 site.
London SSC5 site to be arranged – discussed, but to be confirmed.
-- T2 issues
-- General notes.
- Escalated tickets https://gus.fzk.de/download/escalationreports/roc/html/20110516_EscalationReport_ROCs.html
57746 - Camont issue with Cam-HEP. Condor related? More job outputs given so back to the site - Santanu!
Jeremy: Problem with way condor reads logs. Original ticket submitted by Carl Harrison, back with Santanu because original problem not resolved.
64995 - RAL T1 No GlueSACapability defined for WLCG Storage Areas. On hold but now progressing. Change ticket status?
Jeremy: Jens says this is in progress
Gareth: On hold because it's in progress but in development, I still get note of it in weekly report
65991 - pheno unable to see DNs for some site accounting:
I can successfully identify the numbers for RAL and GLASGOW from your list.
All sites that have reported something during the time period 1-30 April:
Netlab1.UKI-LT2-Brunel
RAL-LCG2
Triton.UKI-LT2-Brunel
UKI-LT2-IC-HEP
UKI-LT2-QMUL
UKI-LT2-RHUL
UKI-NORTHGRID-LIV-HEP
UKI-SCOTGRID-GLASGOW
UKI-SOUTHGRID-BHAM-HEP
UKI-SOUTHGRID-OX-HEP
UKI-SOUTHGRID-RALPP
not visible for me are:
UKI-NORTHGRID-LANCS-HEP: Matt – I haven’t made the changes, I need to do that, will do so today.
UKI-NORTHGRID-MAN-HEP: Duncan: It has been discussed, need to check status
UKI-NORTHGRID-SHEF-HEP
UKI-SCOTGRID-DURHAM: Mark Mitchell: I’ll check with Peter Grandi
66306 - Creation of Irish NGI. Child tickets almost complete.
Jeremy: A series of child tickets about what needs to be set up, almost complete
66564 - WLCG sites not publishing the GlueCECapability shared
ldapsearch -x -h lcg-bdii.cern.ch -p 2170 -b o=grid '(&(objectclass=gluece)(glueceuniqueid=*.ac.uk*)(!(gluececapability=share=*)))' | grep GlueCEHostingCluster: | sort | uniq
GlueCEHostingCluster: ce01.dur.scotgrid.ac.uk
GlueCEHostingCluster: ce02.dur.scotgrid.ac.uk
GlueCEHostingCluster: ce2.glite.ecdf.ed.ac.uk
GlueCEHostingCluster: ce3.glite.ecdf.ed.ac.uk
GlueCEHostingCluster: dgc-grid-44.brunel.ac.uk
GlueCEHostingCluster: hepgrid3.ph.liv.ac.uk
GlueCEHostingCluster: lcg-ce02.hep.ucl.ac.uk
GlueCEHostingCluster: lcg-ce04.hep.ucl.ac.uk
GlueCEHostingCluster: lcgce0.shef.ac.uk
Jeremy: Probably needs to be enabled on all CEs
Chris: I think so
Sam: Talked to Peter Grandi, fixed this as part of YAIM fix, will check.
Chris: Looks like Durham is fixed
Wahid: Can I check what this is needed for?
Jeremy: Management board for WLCG looking for consistency and accuracy in the information system.
Chris: Just a small change required.
Wahid: I’ll look into it
67322 - Wrong value for GlueHostProcessorOtherDescription
Derek can this be marked solved?
Catalin: Will check with Derek on the status of this.
67365 - WLCG sites not publishing the GlueCECapability Share
Looks like this is almost there.
68077 - Mandatory WLCG InstalledOnlineCapacity not published RAL-LCG2
Jeremy: In hand
69028 - GGUS-FAQ about Responsible Units
This looks pretty much resolved.
Several other tickets on hold.
68853 - Retirenment of SL4 and 32bit DPM Head nodes and Servers
Awaits end of SL4 gLite 3.1 T2 machines.
Brian : Should be on hold
68858 - Retirenment of SL4 and 32bit DPM Head nodes and Servers - Glasgow sub-ticket
Sam: Should be on hold, Brian knows the situation at Glasgow
68859 - Durham sub-ticket
68865 - UCL-HEP sub-ticket.
Then the currently open GGUS tickets summary:
http://tinyurl.com/6z6uq5v
Catalin: I updated all tickets regarding WMS, should be able to close them later in the week
Mark Slater: 70521: (Release 16.6.4 install failed on UKI-SOUTHGRID-BHAM-HEP ) Difficulty if you have 2 clusters with 2 software areas with different installs. Got much more of an idea now how to keep this from happening. In the future.
Duncan: 69336: (rfcp failures at MAN-HEP) Partly due to network issues: need to reboot servers for other changes. Should be dealt with soon.
11:40
GDB overview (10')
GDB agenda page:http://indico.cern.ch/conferenceDisplay.py?confId=106644.
Jeremy's summary: https://www.gridpp.ac.uk/wiki/GDB_11th_May_2011
Discussion:
Duncan: Where do we look to find availability? Steve Lloyd’s pages, Gridmap don’t include CREAM CE data. Could Gridmap include CREAM?
Jeremy: Check on GridPP Nagios. I’ll look into this
DPM/Nagios: Wahid: Similar things discussed to those mentioned in Storage Workshop about nagios probes that can be used for DPM. Will talk to Ricardo about including them. PUPPET not really used in the UK. Feedback from storage workshop was that we want more internal DPM Nagios probes.
Chris: QMUL having problem with CREAM CE, SAM tests seem to be fine but Steve Lloyd’s tests seem to be failing. Is this a known problem?
Stuart Purdie: There was something about this in the EGI SA-1 meeting yesterday. In staged rollout, if you are using most up-to-date glibc, they saw some crashes with blah against SGE, may affect all batch systems. Keep an eye on BUpdater, which might have some issues.
Chris: Seeing 100% CPU every time I’ve looked recently.
Matt: Also could be what we’re seeing at Lancaster. Also seeing >100% CPU on CREAM CEs . Posted ticket, gist is that bupdater gets a backlog of work which has to be worked through (intensive), best left alone to work through the backlog. Fix is "optimise code", not sure about the timeline for this.
SA-1 Meeting: Stuart: EMI1 is released, but if you are in staged rollout wait for another stage.
11:50
Accounting/publishing/etc. (05')
Just a quick check of:
http://www3.egee.cesga.es/gridsite/accounting/CESGA/egee_view.php
UCL-HEP not publishing. CE problems?
Daniela: CE problems not fixed, but publishing is fixed
Lancaster - Database problems affecting CREAM CE.
Matt: Looked like the database, but noted bug with Bupdater in CREAM
Glasgow - Sick server (network or disk?)
Mark: Network: Rebooted last Thursday, took cluster down and restarted. Memory error reoccurred. Some other corrective actions can be taken, we’ll be in at risk tomorrow. Buffers are running out of memory on some ports. Not getting many transfer errors. May need a code upgrade.
Sam: Disk server appears to be fine - all ATLAS based, so know exactly which files were affected. Not the disk server, problem was probably network issues. All production jobs affected were rerun.
Brunel - globus gatekeeper fails to start after upgrade
EFDA-JET - new BDII node
RHUL - nfs server problems
Govind: Condor problem with CREAM CE, fixed.
Birmingham - Disk/controller failure - impacts mySQL; BDII & APEL
Mark Slater: Down for about 4 days from Sunday-Thursday. Got new disk, looks like it was disk, not the controller. Affected SQL server which we weren't taking backups of (which we are now doing). All seems fine now, up since middle of last week. Having replaced disk has stayed up longer with replaced disk, still not sure.
Brian: Issue tracked by CMS concerning data transfers to Bristol, wondering if SouthGrid members not at Bristol were aware of the issue.
Ewan: Please escalate to GGUS so that we can track it and look into it.
11:55
Actions (05')
- http://www.gridpp.ac.uk/wiki/Operations_Team_Action_items
12:00
AOB (01')
- WLCG workshop in July https://indico.desy.de/conferenceTimeTable.py?confId=4019#all. Currently expect GridPP travel funding to support the core ops-team as a first priority. For those on the ops-team with an interest but no response yet please let me know!
http://indico.cern.ch/event/126705
Jeremy: Asked about attendance to the workshop.
Gareth: HEPSYSMAN, June 30th/July 1, registration page will hopefully go up today or tomorrow.
Chat transcript:
[11:01:58] Mark Slater joined
[11:02:09] Queen Mary, U London London, U.K. joined
[11:02:29] Gareth Smith joined
[11:02:36] Catalin Condurache nope
[11:02:44] Brian Davies joined
[11:03:30] Daniela Bauer joined
[11:03:32] Catalin Condurache 've got it now
[11:03:33] Andrew McNab got that email now, thanks
[11:04:04] Duncan Rand joined
[11:04:04] Duncan Rand left
[11:04:04] Wahid Bhimji joined
[11:04:35] David Crooks joined
[11:04:35] Govind Songara joined
[11:05:01] Stuart Wakefield joined
[11:05:30] RECORDING David joined
[11:05:32] Jeremy Coles David is taking minutes.
[11:07:47] Jeremy Coles Will return to first agenda item "Experiment problems/issues"
[11:11:42] Duncan Rand david says: "one"
[11:13:12] Govind Songara Can someone paste agenda here please
[11:14:26] Matthew Doidge http://indico.cern.ch/conferenceDisplay.py?confId=126705
[11:16:21] Govind Songara Its old one
[11:19:29] Wahid Bhimji (my connection dropped- back now)
[11:19:57] Matthew Doidge Mine did too
[11:22:45] Pete Gronbech joined
[11:23:46] Pete Gronbech left
[11:24:34] Stephen Jones joined
[11:25:16] Pete Gronbech joined
[11:26:40] Stephen Jones Sorry for lateness - just read email
[11:41:42] Stuart Purdie joined
[11:43:07] Stuart Purdie Sorry I'm late - previous meeting overran
[11:43:49] Gareth Smith Some availability/reliability reports (incl. Tier2s) available at https://twiki.cern.ch/twiki/bin/view/LCG/SamMbReports
[11:43:59] Gareth Smith But the most recent seems to be February 2011
[11:58:51] Matthew Doidge https://savannah.cern.ch/bugs/?func=detailitem&item_id=79324
[12:03:24] Brian Davies left
[12:03:45] Brian Davies joined
[12:05:28] Mark Mitchell Jeremy I need to head off. Speak to you next week.
[12:05:34] Mark Mitchell left
[12:12:26] Brian Davies left
[12:12:29] Daniela Bauer left
[12:12:29] Mark Slater left
[12:12:29] Stephen Jones Thanks
[12:12:30] Govind Songara left
[12:12:30] Catalin Condurache left
[12:12:33] Stuart Wakefield left
[12:12:33] Pete Gronbech left
[12:12:34] David Crooks left
[12:12:34] Gareth Smith left
[12:12:36] Matthew Doidge left
[12:12:39] Stephen Jones left
[12:12:39] Andrew McNab left
[12:12:44] Wahid Bhimji left
[12:12:45] Sam Skipsey left
[12:12:52] Duncan Rand left
There are minutes attached to this event.
Show them.