Choose timezone
Your profile timezone:
Operations team & Sites
→
Europe/London
EVO - GridPP Operations team meeting
EVO - GridPP Operations team meeting
Description
- This is the biweekly ops & sites meeting
- The intention is to run the meeting in EVO: http://evo.caltech.edu/evoGate/. Join the meeting in the GridPP Community area.
- The phone bridge number is +44 (0)161 306 6802 (CERN number +41 22 76 71400). The phone bridge ID is 5559787 with code: 4880.
Apologies:
Tuesday, July 10, 2012
11:00
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
UK running smoothly
Oxford was not in job submissions. Now included.
Brunel had problems with aborted pilots.
B’ham had CVMFS issues.
VO card update (Storage needed by lhcb job requirement.)
- CMS
NTR
Imperial still would like to iperf 10GB test their network.
BD top contact SW with regard to RAL
Mark M willing for Glasgow to test during downtime
Oxford stated they had already tried them.
- ATLAS
UKI-SCOTGRID-GLASGOW: in downtime for 10 days for an air conditioning upgrade. Site is running limited production and no analysis for this period. There have been few problems with exhausted space and queues getting closed during the weekend. Now solved.
UKI-LT2-UCL-HEP: analysis queue have been put back online last Thursday and so far the site has had no problems.
UKI-LT2-QMUL: testing beta version of cvmfs on 3 nodes so far they had no problems while other nodes without beta version had. Need more stat.
WB I would like ECDF to try it (beta cvmfs) - but Andy would be the one to do it and he may not be so keen
UKI-LT2-IC-HEP: no progress on the release issue in the past week.
- Other
11:20
Meetings & updates (20')
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
Certificate update
JJ
Robot certs are issued by hand; corrupted DB. Backup restored.
KM reported that email contacts corrupted, fixed.
No loss of requests, can delete re-submissions.
AF has not received notification for cert renewal.
Discussion regarding removal of old CA cert.
JJ This is coming… Will keep cert whilst certs valid against it.
Can track CRL downloads.
¾ sites updated within one week.
EGI? WLCG availability figures. No known issues.
Meeting today regarding extending CA capabilities.
- Tier-1 status
As in section.
-Storage and data management
Review of DPM community support
- Accounting
- Documentation
- Interoperation
- Monitoring
- On-duty
Dashboard still using Lancaster Instance
- Rollout
- Security
- Services
perSonar has been added at some sites.
New link:
https://perfsonar.racf.bnl.gov:8443/exda/?page=25&cloudName=UK
July deadline for perfSonar on track
AF addition to Twiki
Gridmon boxes to be returned to RAL
PG Can all sites let me know if they have their gridmon box ASAP
Sites reported to PG in meeting of result.
- Tickets
-84066
-Ticket regarding location of site in GOCDB
-Qmul cvmfs issue
-snoplus putting git in SW area
-Sussex has expired certs for CE.
KM has access to machine.
DPM crash at RHUL
- Tools
- VOs
- Site updates
Sussex covered
GDB tomorrow
https://indico.cern.ch/conferenceDisplay.py?confId=155070
11:40
Monitoring - links, ranking and discussion (20') Slides pdf file}
DC
Trying to find pages in common for people
Taken monitoring page links from wiki:
https://www.gridpp.ac.uk/wiki/Links_Monitoring_pages
sections:
Gstat; site operations; diagnosis pages; security pages.
Relevance wanted.
12:00
AOB (1')
Quarterly reports due. End of this week for initial drafts.
Some sites have stopped receiving nagios alerts. Only effect sites which have more than one site email address. If site only has one address, then it works. Effects Oxford, RHUL and Sheffield.
VOMS - part 2 (12:01 ->12:30 )
12:01
Review of findings (29')
Brian Davies BD, Kashif Mohammad KM, Jeremy Coles JC, Rob Fay RF, Alessandra Forti AF, Raja Nandakumar RN, Andrew McNab AM, Matthew Doidge MD, Pete Gronbech PG, Jens Jensen JJ, Robert Frank RF, David Wallom DW, Claire Devereux CD, John Gordon JG, Mike Jones MJ.
Discussion summary from before:
https://indico.cern.ch/materialDisplay.py?contribId=5&materialId=1&confId=192186
-Check VOMS rationalization options
- Plan Manchester transition from Computer Science to Physics
RF/ AW Replication is working. No work in production for next 5 weeks.
KM using SSL over WAN for replication.
-Consolidation of VOMS servers.
NGS and GridPP servers.
AF, RF has seen that only 4-5 VOs are active on NGS voms server.
DW- Long term strategy is 30-150. Follow up; replication is purely for redundancy of a single service rather than scalability issues?
KM it is also scalable.
RF voms-admin is scalability issue.
JG interesting to see what IT has done.
AM Memory footprint per VO in tomcat supported leads to scaling issue.
CD Vo.ngs.ac.uk splitting leads to increase in new VOs
MJ UAS uses only the VOMS admin tools to talk to VOMS-Admin
JG if we hit a ceiling can we start up new voms server?
Not enough resources to run further.
JC decommission non-active VOs and move NGS VOS to gridpp service.
- Understand work to be done to introduce new backup VOMS servers
- Confirm secondary sites (Oxford & Imperial)
Need access to voms-admin rights?
-Need for certificates
KM Issue with host domain Issuing cert.
- Agree timetable for implementations
- Discuss best option for data replication
- Setup VOMS at additional sites
JC reconvene in 3 weeks time.
- Start replication
- Announcements to community
- Configure VOMS clients
- Switchover
- Follow up to include review of VO membership dates and configurations
11:00
Experiment problems/issues (20')
Review of weekly issues by experiment/VO
- LHCb
UK running smoothly
Oxford was not in job submissions. Now included.
Brunel had problems with aborted pilots.
B’ham had CVMFS issues.
VO card update (Storage needed by lhcb job requirement.)
- CMS
NTR
Imperial still would like to iperf 10GB test their network.
BD top contact SW with regard to RAL
Mark M willing for Glasgow to test during downtime
Oxford stated they had already tried them.
- ATLAS
UKI-SCOTGRID-GLASGOW: in downtime for 10 days for an air conditioning upgrade. Site is running limited production and no analysis for this period. There have been few problems with exhausted space and queues getting closed during the weekend. Now solved.
UKI-LT2-UCL-HEP: analysis queue have been put back online last Thursday and so far the site has had no problems.
UKI-LT2-QMUL: testing beta version of cvmfs on 3 nodes so far they had no problems while other nodes without beta version had. Need more stat.
WB I would like ECDF to try it (beta cvmfs) - but Andy would be the one to do it and he may not be so keen
UKI-LT2-IC-HEP: no progress on the release issue in the past week.
- Other
11:20
Meetings & updates (20')
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
Certificate update
JJ
Robot certs are issued by hand; corrupted DB. Backup restored.
KM reported that email contacts corrupted, fixed.
No loss of requests, can delete re-submissions.
AF has not received notification for cert renewal.
Discussion regarding removal of old CA cert.
JJ This is coming… Will keep cert whilst certs valid against it.
Can track CRL downloads.
¾ sites updated within one week.
EGI? WLCG availability figures. No known issues.
Meeting today regarding extending CA capabilities.
- Tier-1 status
As in section.
-Storage and data management
Review of DPM community support
- Accounting
- Documentation
- Interoperation
- Monitoring
- On-duty
Dashboard still using Lancaster Instance
- Rollout
- Security
- Services
perSonar has been added at some sites.
New link:
https://perfsonar.racf.bnl.gov:8443/exda/?page=25&cloudName=UK
July deadline for perfSonar on track
AF addition to Twiki
Gridmon boxes to be returned to RAL
PG Can all sites let me know if they have their gridmon box ASAP
Sites reported to PG in meeting of result.
- Tickets
-84066
-Ticket regarding location of site in GOCDB
-Qmul cvmfs issue
-snoplus putting git in SW area
-Sussex has expired certs for CE.
KM has access to machine.
DPM crash at RHUL
- Tools
- VOs
- Site updates
Sussex covered
GDB tomorrow
https://indico.cern.ch/conferenceDisplay.py?confId=155070
11:40
Monitoring - links, ranking and discussion (20') Slides pdf file}
DC
Trying to find pages in common for people
Taken monitoring page links from wiki:
https://www.gridpp.ac.uk/wiki/Links_Monitoring_pages
sections:
Gstat; site operations; diagnosis pages; security pages.
Relevance wanted.
12:00
AOB (1')
Quarterly reports due. End of this week for initial drafts.
Some sites have stopped receiving nagios alerts. Only effect sites which have more than one site email address. If site only has one address, then it works. Effects Oxford, RHUL and Sheffield.
VOMS - part 2 (12:01 ->12:30 )
12:01
Review of findings (29')
Next steps
Present,Brian Davies BD, Kashif Mohammad KM, Jeremy Coles JC, Rob Fay RF, Alessandra Forti AF, Raja Nandakumar RN, Andrew McNab AM, Matthew Doidge MD, Pete Gronbech PG, Jens Jensen JJ, Robert Frank RF, David Wallom DW, Claire Devereux CD, John Gordon JG, Mike Jones MJ.
Discussion summary from before:
https://indico.cern.ch/materialDisplay.py?contribId=5&materialId=1&confId=192186
-Check VOMS rationalization options
- Plan Manchester transition from Computer Science to Physics
RF/ AW Replication is working. No work in production for next 5 weeks.
KM using SSL over WAN for replication.
-Consolidation of VOMS servers.
NGS and GridPP servers.
AF, RF has seen that only 4-5 VOs are active on NGS voms server.
DW- Long term strategy is 30-150. Follow up; replication is purely for redundancy of a single service rather than scalability issues?
KM it is also scalable.
RF voms-admin is scalability issue.
JG interesting to see what IT has done.
AM Memory footprint per VO in tomcat supported leads to scaling issue.
CD Vo.ngs.ac.uk splitting leads to increase in new VOs
MJ UAS uses only the VOMS admin tools to talk to VOMS-Admin
JG if we hit a ceiling can we start up new voms server?
Not enough resources to run further.
JC decommission non-active VOs and move NGS VOS to gridpp service.
- Understand work to be done to introduce new backup VOMS servers
- Confirm secondary sites (Oxford & Imperial)
Need access to voms-admin rights?
-Need for certificates
KM Issue with host domain Issuing cert.
- Agree timetable for implementations
- Discuss best option for data replication
- Setup VOMS at additional sites
JC reconvene in 3 weeks time.
- Start replication
- Announcements to community
- Configure VOMS clients
- Switchover
- Follow up to include review of VO membership dates and configurations
There are minutes attached to this event.
Show them.