Operations team & Sites

Tuesday 20 Nov 2018, 11:00 → 12:30 Europe/London

EVO - GridPP Operations team meeting

- This is the weekly GridPP ops & sites meeting

- The intention is to run the meeting in Vidyo: https://vidyoportal.cern.ch/flex.html?roomdirect.html&key=zXhsqAxVnaT6

-- The PIN is 1234. To join via phone see http://information-technology.web.cern.ch/services/fe/howto/users-join-vidyo-meeting-phone for dial in numbers.

-- The London (UK) service is on +44 (0)161 306 6802. Phone bridge ID 1001002

-- The meeting extension is 109308582. PIN 1234

Chair:  Jeremy

Minutes: Brian D

Apologies:

Ops team and sites minutes 20-11-2018
Present
Alessandra Forti
Brian Davies
Chris Brew
Daniela Bauer
Darren Moore
David Crooks
Elena Korolkova
Emanuele Simili (New @ Glasgow)
Gareth Roy
Jeremey Coles
John Hill
Kashif Mohammad
Matt Doidge
Raja Naandakumar
Raul Lopez
Robert Frank
Sam Skipsey
Winnie Lacesso

------------------------------------------------------------------------------------------------------
Experiment problems/issues
    LHCb
RN: MC running,  pilots aborted at sites , Steve Jones working at it. LHCb replied to ticket raised by Bristol. ARC directory should be being cleared but isn't.
KM: Oxford queue rate low. RN to investigate.
    CMS
DB: Reported added to minutes
    T1: https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL
    T2: https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T2_UK_London_Brunel
5 tickets for RAL/RALPP
CB:  Ipv6 issue
DM: T1 ticks are work in progress
    ATLAS
EK:
QMUL squid ticket issue, can be closed.
Ticket for  RALPP ( IPV6 related.) Transfers have improved but still problems.
Harvester migration in progress , most sites have moved to UCORE queue . Those done are:
Brunel , QMUL RHUL , Liverpol, ECDF, Glasgow,  Manchester Oxford Ralpp Sussex
Analysis ques for SL7 for Oxford  created. needs testing
In general:
Heavy Ion data taking.
ATLAS hevay Ion , Issue at SARA.
CMS 2.4-3GB/s of heavy Ion Rate.
ALICE have issue at PIC
    Other:
 Updates should be recorded in https://www.gridpp.ac.uk/wiki/GridPP_VO_Incubator.
Are we not picking up issues earlier enough ("Ganga issue for LSST" as an example.) Need to find a better way to escalate issues with these communities.
EUCLD found quick solution elsewhere so not used gridpp
SKA rucio issue at RAL.
vo.dirac.ac.uk BD to test endpoint after endpoint change.

GridPP DIRAC status [Andrew McNab]
    -- https://www.gridpp.ac.uk/gridpp-dirac-sam
No known issues
------------------------------------------------------------------------------------------------------
Meettings & updates
With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest
    General updates

Glasgow, Liverpool, Manchester, RAL have created CE JSON files, there will be a discussion at the next WLCG Information System this week. https://indico.cern.ch/event/774735/
Flat json structure used  (different implemantions.)
Hireachy structure added. thursday meeting to discuss WLCG-ISTF.
EGI OMB meeting:  KM gave UK position.
Deciding whether to keep BDII format. Ohter NGIs aren't looking into it. Monitoring issue raised ( they did noyt know about it.)
IN2P3 is looking itnto it. Nagios plugins using bdii info to get endpoints.
Argos uses message BUS.
Plugins not maintained by argo team but by resource provider. ( Cream, DPM dCache example etc .)
Mandate from WLCG needed to get people to move. Evolution is needed.
Aim is middle of next year?? ( perhaps too optimisitic?)
Teams need effort to make changes.
    WLCG ops coordination
There was an EGI OMB last week
https://indico.egi.eu/indico/event/4309/
 Schedule for T2K LFC migration agreed at technical meeting last week.
WLCG ops meeting
Final figures for Availablity, U met targets.
    Tier-1 status
On Thursday 15th November, all non-LHC VOs were migrated to the new consolidated Castor tape instance.
CMS AAA issues are ongoing, although we may have finally turned a corner! It was found that certain CMS requests do not specify the Ceph pool to use (in this case it should be “cms”).
 There was no default pool set, thus the request fails. It was found that a bug meant that any subsequent requests would be sent to the (non-set) default pool causing all transfers to fail.
This required a restart of the XRootD process. For the CMS AAA service we have added a default pool and monitoring of the logs, which will restart the process should it detect that transfers are using a non-set pool.
SAM test results were at 96% over the weekend.
We have been experiencing issues with Ceph/Echo ClusterVision 17 disk servers that were occasionally/randomly rebooting.
This has now been resolved with a firmware patch that was rolled out last week.
The hardware continues to be weighted up and we exceed 1Tb/s aggregated throughput on Friday 16th - graph below.
Schedule downtime for ORACLE patching.
    Storage and data management
Tracking XDC within DOMA.
background document ready keeping track
    Tier-2 Evolution
NTR
    Accounting
NTR
    Documentation

    Interoperation
UMD product ID cards, software devlopers may be interested.
https://indico.egi.eu/indico/event/4309/

    Monitoring

    On-duty
Add to rota
    Security
NTR
    Services

    Tickets
44 Open UK Tickets this week
QMUL
138292 (14/11)
It looks like Dan's tweak to his storm setup worked and the LHCB user's jobs are working - another ticket that can be solved. In progress (16/11)
TIER 1
138033 (1/11)
This Singularity ticket from atlas has become quite in depth, and perhaps worth a look to anyone trying to get docker to run atlas singularity jobs. In progress (15/11)
THE IPV6 TICKETS, Dunan R reported over half way in terms of volume.
https://www.gridpp.ac.uk/wiki/IPv6_site_status
RALPP: 131616
Some great progress with dual stacking perfsonar and storage, but Chris notes some problems with packet loss and high RTT. Did the router restart noted in other tickets soothe these? In progress (16/11)
CB: RALPP might back out due to other issues. Routing layer issue under heavy network traffic. Maybe need new router...
OXFORD: 131615
The last update was the end of October with no news (and none expected?). On hold (25/10) De-proritised by University
CAMBRIDGE: 131614
John has dual-stacked his storage, and is just waiting to hear back from the user groups that all is working well. I assume from the lack of tickets it is! Waiting for reply (8/11)
BRISTOL: 131613
The picture at Bristol is still unclear from the ticket, with the storage dual-stacked but it seems like gridftp isn't working via v6? Some possible causes of this were listed in the ticket. In progress (9/10)
WL: CMS working on gridftp. plan reinstall of perfsonar on a ipv6 networked box.
BIRMINGHAM: 131612
No news on this sine August. How goes it Mark? Still the same issue as before (central IT can't supply v6 DNS entries for the site). On Hold (27/8)
GLASGOW: 131611
There's been a big push at Glasgow on this front, although tests weren't looking great despite the efforts. Any luck towards fixing the perfsonar errors? In progress (12/10)
GR: maddash seems broken? so can't see errors.
ECDF: 131610
The last update was back in the start of September, with plans to make plans were being made to dual-stack the SE. Any plans that can be shared? In progress (10/9)
DURHAM: 131609
A recent update was that things were being held back by no reverse-DNS, and none likely to be rolled out until mid-2019 at least. Could the JANET reverse DNS service fill in for this? In progress (17/10)
SHEFFIELD: 131608
There was a plan to dual-stack the perfsonar boxes mid-November (so now-ish). Did you have any luck with that? In progress (30/10)
MANCHESTER: 131607
IIUI another site being held back by no v6 reverse DNS. Any news on this front? On hold (25/4)
LIVERPOOL: 131606
Any sign of those new routers yet, or any other news? In progress (4/6)
RHUL: 131603
Waiting on reverse DNS, but some positive news is that their central IT have decided to outsource this to JANET so it should be smooth sailing... right? In progress (29/10)

ipv6 test results:
http://pprc.qmul.ac.uk/~lloyd/gridpp/ukgrid.html

    Tools

    VOs

    Site updates

------------------------------------------------------------------------------------------------------
Discussion topics
 GDB overview
https://indico.cern.ch/event/651359/
IC listed upcoming meetings
https://indico.cern.ch/event/651359/contributions/3208544/attachments/1752599/2840300/GDB-Introduction-2018-11-14.pdf
XDC talk: Oliver keeble, Aim Federating Scalable Storage
 Network Virtualisation WG reportSpeaker: Marian Babik (CERN)
DOMA TPC update Speaker: Alessandra Forti (University of Manchester (GB))
AF: Increasing number of sites, 23 now.
Alternative ways how to do AA for http . Works with macaroons.
xrootd will work with delegation.
CERN SSO Project Speakers: Hannah Short (CERN), Paolo Tedesco (CERN)
ATLAS data carousel report¶

EGI updates

------------------------------------------------------------------------------------------------------
Actions & AOB

------------------------------------------------------------------------------------------------------
Chat dump:
Gareth
Emanuele Simili
For Brian
GR
Jeremy
Brian is taking minutes today.
https://twiki.cern.ch/twiki/bin/view/LCG/WLCGOpsMeetingWeek181119
https://www.gridpp.ac.uk/wiki/GridPP_VO_Incubator
JC
Raja
Kashif, could you point me to the Oxford number?
R
Alessandra
https://www.gridpp.ac.uk/wiki/GridPP_VO_Incubator#LSST
AF
Raja
I see online for Oxford / lhcb :
http://wlcg-sam-lhcb.cern.ch/templates/ember/#/historicalsmry/heatMap?end_time=2018%2F09%2F30%2023%3A00&profile=LHCb_CRITICAL&site=LCG.Oxford.uk&start_time=2018%2F07%2F01%2000%3A00&time=manual
R
Jeremy
https://indico.egi.eu/indico/event/4309/
For BDII: https://indico.egi.eu/indico/event/4309/contribution/4/material/slides/0.pdf
JC
Alessandra
https://gitlab.com/uki-scotgrid-glasgow/information-system/raw/master/uki-scotgrid-glasgow-ce.json
AF
Jeremy
https://indico.egi.eu/indico/event/4309/contribution/1/material/slides/2.pdf
JC
Ste
Sorry I'm very late.... got wrapped up in ER modelling for JSON bdii replacement... tempus fugit!
SJ
Jeremy
http://pprc.qmul.ac.uk/~lloyd/gridpp/ukgrid.html
JC
Kashif
Raja, I will send you a separate email about Oxford/lhcb
K
John
I believe we're waiting for ATLAS Central to confirm that they are happy before the ticket can be closed
I
JH
Jeremy
http://pprc.qmul.ac.uk/~lloyd/gridpp/ukgrid.html
JC
Ste
No change on IPv6. Still waiting for better infrastructure.
SJ
Jeremy
https://indico.cern.ch/event/651359/
JC
Duncan
Gareth, I emailed the appropriate people about the maddash server...
DR

 

11:00 → 11:01 Ops meeting minutes

• This is a reminder that this is an important task. The minute taker gives access to the discussions for those not present and provides a reference for others to refer back to afterwards.

• The team composition has been changing. If everybody contributes then the task comes around less often.

• Please extract actions from the meeting and add them to our table here: https://www.gridpp.ac.uk/wiki/Operations_Team_Action_items#Action_list.

• Recent allocations: See above link. The page should be updated each week by the minute taker (if they don't the task will keep coming to them!).

• Upcoming allocations:

11:01 → 11:20 Experiment problems/issues

Review of weekly issues by experiment/VO

• LHCb

• CMS
  T1: https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T1_UK_RAL
  T2: https://cms-site-readiness.web.cern.ch/cms-site-readiness/SiteReadiness/HTML/SiteReadinessReport.html#T2_UK_London_Brunel

Please see attached notes.

• ATLAS

• Other: Updates should be recorded in https://www.gridpp.ac.uk/wiki/GridPP_VO_Incubator.

• GridPP DIRAC status [Andrew McNab]
  -- https://www.gridpp.ac.uk/gridpp-dirac-sam

Daniela: I'm still having trouble with my which was killed in a recent update.

CMS report:

From the computing operations meeting:

The CONDOR-JobSubmit SAM test was stuck over the weekend, and this has not yet been resolved, but is being actively investigated.  Stay tuned.

For those interested in the heavy ion run, we’re keeping up with archiving the very high RAW data rate to tape at CERN, although as expected falling behind on making a tape copy at FNAL and on prompt reconstruction.

CPU utilization has been good over the past week, with 180k cores used in production and 50k in analysis.  Many workflows that had previously been problematic are now running.
 

Current CMS UK tickets (problems at RAL?):

(new): https://ggus.eu/?mode=ticket_info&ticket_id=138393 RALPPD

https://ggus.eu/?mode=ticket_info&ticket_id=138302 RALPPD

https://ggus.eu/?mode=ticket_info&ticket_id=138276 RALPPD

https://ggus.eu/?mode=ticket_info&ticket_id=138327 RAL-LCG2

https://ggus.eu/?mode=ticket_info&ticket_id=137650 RAL-LCG2

https://ggus.eu/?mode=ticket_info&ticket_id=138041  Bristol

 

 

 

11:20 → 11:40 Meetings & updates

With reference to: http://www.gridpp.ac.uk/wiki/Operations_Bulletin_Latest

• General updates
• WLCG ops coordination
• Tier-1 status
• Storage and data management
• Tier-2 Evolution
• Accounting
• Documentation
• Interoperation
• Monitoring
• On-duty
• Security
• Services
• Tickets
• Tools
• VOs
• Site updates

11:40 → 12:20 Discussion topics

• GDB overview
• EGI updates

12:20 → 12:25 Actions & AOB