Choose timezone
Your profile timezone:
HEPiX IPv6 Working Group - in person at CERN
→
Europe/Zurich
31/3-004 - IT Amphitheatre (CERN)
Description
We will meet in person at CERN. Zoom connectivity will also be available for those of you who cannot attend in person.
Please register to say you will attend the meeting (select whether in person or not).
The start times and length of slots are estimates and may be adjusted on the day depending on discussion
Participants
hepix-ipv6-wg
Zoom Meeting ID
64006864374
Host
Edoardo Martelli
Alternative host
Bruno Heinrich Hoeft
Useful links
Join via phone
Zoom URL
Minutes still being modified -
contributions of notes gratefully received from Raja, Duncan and Martin.
HEPiX IPv6 Working Group meeting
CERN - Tuesday/Wednesday 28/29 June 2022
Tuesday 28 June - 1st Day
Present: Martin Bly, Costin Grigoras, Bruno Hoeft, David Kelsey, Edoardo Martelli, Carmen Misa Moreira, Kars Ohrenberg, Duncan Rand, Andrea Sciaba
Remote: Nick Buraglio, Tim Chown, Rob Currie, Phil Demar, Raja Nandakumar, Francesco Prelz
Apologies: Marian Babik
RajaN's notes merged with notes from MartinB and DaveK
1. Welcome, agenda, etc.
DaveK welcomed all to the meeting noting how great it is to be able to meet together in person after all this time in the CERN IT Auditorium. He also welcomed remote participants noting that some were prohibited from attending in person because of COVID issues and hoped they would be able to attend in person next time.
The agenda was presented and agreed. See https://indico.cern.ch/event/1169293/
RajaN and DuncanR agreed to take notes on first day.
2. Round table updates
CERN (EdoardoM)
Nothing much to report. Occasional bugs in data centre.
Some big bugs with DHCPv6 - problem with vxlan, routers sharing mac addresses
-- option 79 needed, but does not work with vxlan due to possible packet duplication
-- Fixed in a previous version. Upgrade for other issues broke it.
-- Still an open issue
-- code for ipv6 far from perfect!
-- Newer versions of linux do not use uuid with mac address and causes the problem.
ESnet/DOE (NickB)
ESnet update: Moving ahead with ipv6-only testing. Management network is 99.9% ipv6-only. Last element due to be removed, functionally not active.
Next steps - running a ipv6-only trial as part of deliverables. Finished, doing performance testing. After that, write a report (internal?)
DOE update: Slowly moving to what deliverables should be. Postponed data call. Next goal : 20% of clients being ipv6-only. Setting up a few community practice meetings to provide advice with implementation.
FermiLab (PhilD)
FNAL ipv6 update :
Milestones for ipv6-only clients - FY23:20%, FY24:50%, FY25:80%. Only for US Tier-1s (national labs). University run Tier-2s not included in this.
-- Transitions mostly during natural hardware lifecycle refresh
-- Enforced by OBR (office of Budget Responsibility)
-- US CMS Tier-1 fully dual-stack
-- non-CMS computing systems mostly dual-stack
---- Data servers are all dual-stack
-- FNAL Traffic : 33% ipv6 overall. Inbound ~80% ipv6-only - investigating why (dCache java issue?).
Some details on ipv6-only strategy - dual stack and ipv6-only share physical links, but use separate virtual links.
-- Monitoring web sites :
https://my.es.net/
https://public.stardust.es.net/d/XkxDL5H7z/esnet-public-dashboards?orgId=1
KIT (BrunoH)
KIT is already dual-stack for some time
Moving all management systems to ipv6 as far as possible. Not there yet, but looks promising.
Deployed campus vslan overlay.
Tried to implement ACLs, but found bugs. Mostly solved and are going towards ipv6-only
Will also be talking about Tier-1 tomorrow
RAL - MartinB
Nothing special regarding ipv6 at RAL.
New network at Tier-1. Going to use that in moving forward with ipv6.
Plan to bring in next gen of networking over the next 3 years or so.
-- Lobbied for making it ipv6-only. Unlikely to happen.
-- Will have to have a few different networks (security systems, normal network, ...)
-- Eventually will be able to do dual-stack WNs in ~ end 2022 (maybe)
Discussion : Security best with pure systems. Dual-stack worst of all worlds from security standpoint.
Work ongoing on zero trust systems - just because you are behind a firewall does not mean that your
system is trustworthy. A problem once you have BYOD and suchk.
LHCb (RajaN)
https://ggus.eu/index.php?mode=ticket_info&ticket_id=153653
-- vague rumbling that it is due to ipv6
FNAL dCache dual stack now.
-- as reported by Nick and Philip
ALICE (CostinG)
Alice VO-box at KIT receiving data from jobs at KIT
-- running only over ipv4. Not sure why, given that the machines are all dual stacked.
-- Standard choice is to require ipv4 for all Alice VO-boxes.
-- Want to understand workings, before switching default to ipv6
Ongoing issue with Alice servers at CERN
-- Not able to log on to the services even from within CERN
-- Started over the last weekend.
-- Eduardo Note - CERN is currently upgrading the campus routers. Known issues regarding this upgrade are probably causing this.
INFN (FrancescoP) (from sick bed)
Mysterious issues with CMS workloads spotted with Bruno's packed snapshots
-- Known by CMS experts!
-- Not much support from INFN Torino.
Internal staff meeting within INFN
-- ipv6 not very popular!
INFN Milano ipv6 advertisements were silently failing due to filtering
-- Incorrect configuration by Francesco (?)
-- But ... no ticket / alarm / ...
-- no functioning ipv6 for 10 days
-- Found because of latency in connecting
Duncan - one of the two perfsonars is down
-- Francesco is the contact! Will investigate.
UK Jisc (TimC)
100Gbps coming in from CERN to Imperial. Often ~100% ipv6 and the link can be saturated.
See blog at:
Regarding other research communities in UK (big data), sadly both SKA and the UKAEA (Fusion) experiments are not thinking of ipv6 right now.
Imperial/UK/dCache (DuncanR)
Monitoring dCache update status for versions which are ipv6-compatible (7.2.17).
-- 27 out of 31 were compatible last week.
-- Now all are. The new reports are from KIT
-- Bruno - there was a problem with bdii which is fixed now.
CMS/LHCb (AndreaS)
Now responsible for ETF tests for LHCb.
-- writing storage probes using CMS as a model
-- Will be requesting Marian to resurrect the ipv6-only ETF instance.
---- coffee break ----
Notes from DuncanR - second half of first day
3. Fall back to IPv4 when IPv6 is broken
Rob Currie presented his slides on “Fallback to IPv4 when IPv6 is broken?”. Standards: happy eyeballs is a standard. But questionable whether happy eyeballs is actually useful for us in our move to IPv6 only - we need to know IPv6 is failing. What should we recommend - is it up to the implementation? In the end it is TCP timeout, happy eyeballs is a wrapper around TCP timeout. What is a standard TCP time out? There is a maximum timeout of 5 mins. Xrottd seems to use the same failover between v4 and v6 as for multiple round-robin hosts. Want to prefer IPv6. Do we as a group want to define normal behaviour. Don’t believe we should lobby for happy eyeballs in our code. Classic dual-stack situation - IPv6 fails then it works because IPv4 exists and works. Agree happy eyeballs is for interactive situations. With 90s issue, it would help to be able to diagnose exact issue.
4. Status of Tier-2s
Andrea Sciaba presented the current status of WLCG Tier-2s.
Good news, two US ATLAS sites have completed IPv6 deployment of storage. Also, three German sites: Wuppertal, GoeGrid, but as yet still untested and unconfirmed. So percentage moves from 82% to 85% Tier-2s completed and 83% of ATLAS storage is now available over v6. Four sites in UK still in progress: Glasgow, Bham, RHUL and Liverpool. Some other countries such as Canada still in progress.
5. Dates of next meetings
DaveK asked: could we combine with LHCOPN/LHCONE meeting on 24-25 October 2022? - makes it more worthwhile for attendees from far away to attend both meetings. All felt this to be a good idea.
Dates of future meetings were agreed as follows:
Next “in person” meeting at CERN will be a full day (09:00 to 17:00 CEST) on Wednesday 26 October 2022. This is timed to follow the LHCOPN/LHCONE meeting (also in person at CERN) on Monday/Tuesday 24/25 October. The agenda for LHCONE is at https://indico.cern.ch/event/1146558/
Regular monthly one-hour Zoom meetings (on last Thursday of month) were defined for 28 July 2022, 25 August, and 29 September. All those one hour meetings run from 16:00 to 17:00 CEST.
It was realised of course that many will take vacations during the summer months and will therefore miss one or more of these one-hour meetings, but we will meet with those who are available.
6. Analysis and results of CMS condor jobs
Speaker: Carmen Misa Moreira (CERN)
Notes from Martin.
Slides shown are sensitive so not uploaded to agenda page - they were distributed to all attendees. Others wishing to gain access, please contact Edoardo Martelli.
Big Db query: query monit_pod_raw_metric. Recovers 500 lines per job. Dataset contains 7813 jobs, from 6th to 27th June ordered by end points, CMS jobs. For each check for dual stack, get the ip4 and ipv6 addresses. IN this case, dual-stacked means that the DNS responds with both types but the machine may have an ipv6 address but is not dual stacked. Results ordered by CMS_Type. Reviewed results: dual stacked 3850. Schedd_name dual_stacked (9197), Startd_principle dual_stacked (9792). Conclusions were made but not documented in the minutes. Also looked at xrootd transfers: data now available but results not ready for presentation. Conclusion: two different infrastructures, one production (usually dual stacked) and one analysis (usually not dual stacked).
End of Day 1 of meeting
Wednesday 29 June - 2nd Day
Present: Martin Bly, Bruno Hoeft, David Kelsey, Edoardo Martelli, Carmen Misa Moreira, Kars Ohrenberg, Andrea Sciaba
Remote: Marian Babik, Tim Chown, Raja Nandakumar, Francesco Prelz
Apologies: Duncan Rand
1. MartinB agreed to take notes for the second day.
2. Plans to investigate more reasons for ongoing use of IPv4 in data transfers: Discussion
Some discussion on use of jumbo frames and the topic of why stuff is still going over ipv4. Carmen’s slides will be distributed to meeting attendees with suitable request not to distribute to wider audiences.
3. Detailed monitoring to prepare for IPv6: Bruno Heinrich Hoeft (KIT)
KIT looking at ipv6 only systems at KIT for testing. Decied to dual stack – need a monitoring system to monitor all coms between WNs and administration, job sbmussion, storages, …with packetbeat collecting network data, logstash and elastic search, kibana for visualising. DNS: all via ipv4 even though host and DNS server are dual stack – needs ipv6 resolve address in resolve.conf (first line). Squids: some older squids are ipv4 only, significant connections via public ipv4 – checked cvmfs prefer can prefer ipv6 – cms_ipfamily_prefer6. Alice VOBoxes: need to accommodate ipv4 only sites so need to cechk ipv6 migration/preference with Alice. Xrootd: Looks like an upgrade to XrootD 5 clients will fix access to storage – all Alice SEs are dual stacked. NTP interesting: on port 123: all ipv4. Some hosts connected to as ntp servers. Looking at processes that are doing NTP requests. Done by alice because they don’t trust local time. Additional issues with LRMS, general WN infrastructure, local talking to itself on high ports, HTCondor now scheduling jobs via ipv6. Further work going – WNs still resolving DNS via ipv4, NTP destinations change quickly, LRMS. Closer scrutiny is necessary. Deeps dives will require time and manpower.
Discussion: Set up a page on the web/wiki that lists the various problems that could show up and possible solutions. Francesco reports there is a single ntp pool address 2.europe.pool.ntp that resolves to ipv6 and well as ipv4.
4. Update on monitoring (perfSONAR, ETF, etc.): Marian Babik (CERN)
Reports Checkmk2.1 has been released.
ETF central instance show s test form both ipv6 and ipv4 from atlas and CMS. Many CEs dual stack but blocking ipv6 (any number of reasons which but nmostly likely to be configuration issues) Middleware: Atlas/CMS plan to migrate to ARC REST with the new HTCondor ARC backend including support for SciTokens. Storage probes – CMS running new probes for WebDAV/xroot/gsiftp (ipv4/6). Atlas/Lhcb developing new storage probes for py3, gfal2, WebDAV/xroot. SciTokens – Atlas and CMS both submitting jobs with both x509 and tokens. ETF -> MONIT/Sitemon interface – ipv6 metrics in MONIT, Atlas testing ipv6 profile/aggregations.
PerfSONAR: 238 active instances, 207 production end points for T1/T2. Various rounds of HW server upgrades in progress, including 100G NICs. Some 100G tuning to give 10G stream via 100G NIC. PerfSONAR 5 beta now out. Toolkit supports CC7, latest Debian, Ubuntu 18/20, RHEL 8 (Alma/Rocky). New dashboard – Grafana – host based, particularly for 100G mesh, very flexible. Does ipv4 and ipv6. Evolution: collects, stores, configures and transports all network metrics – distributed deployment operated in collaboration. Alarms and alerts service (https://aaas/atlas-ml.org/ ). User-subscribeable alerting service for specific alerts types. Looking at alerts for path changes when traffic starts to take a new routes, traceroute, based on AS numbers – look at changes in sequence of AS numbers. Also looking at triggering traceroute tests from alarms.
Duncan: subscriptions? Subscribe from interface. Sign up via SSO for your site. Filtering possible. Note that existing subscribers need to re-subscribe because alarm hierarchy has changed.
Tim: historic data preserved? Yes – historical data is in ElasticSearch centrally but updating to perfsonar5 will change how it is kept locally so will lose historical local data.
Break
5. Working towards wider adoption of IPv6 within Jisc: Tim Chown
Planning for wider ipv6 adoption in Jisc – why is it important, current position, problems and pain points, implications and risks, areas for action. Though VERY DRAFT and not official position.
Why does it matter: latest version of IP, IP is fundamental to Jisc’s business, ipv4 exhaustions,. Matters for members because communities are adopting it for their needs, cyber security, students etc., should have access to state of the art technology.
Currently, ipv6 is a free component of Janet IP connection service, ipv6 /48 assignments…
Problems and pains: various pains and issues. Particularly students entering an IPv6-dominated word with no IPv6 experience. Network needs to be state of the art.
Implications and risks of doing nothing for members follow on from the pains to Jisc. Particularly need to participate in ipv6only collaborations.
Actions? Include ipv6 in all services, services must run dual stack etc. Including: auditing Jisc services for support, raising levels of knowledge and understanding in ipv6 internally, ensuring ipv6 specified in procurements from outset…
Areas to work on: ipv6 in Jisc offices and systems internally, track ipv6 gaps (if not why not), track standards and polices, provide advice and support for members, baselining status of deployment and use by members.
Discussion of science data transfer and NRENs encouraging adoption of IPv6 in their members.
6. Update from RNTWG and SCITAGS activities: Marian Babik (CERN)
Scitags.org. initiative promoting identification of the science domains and their high level activities at the network level. Enable tracking and correlation of transfers with network flow monitoring. Expts can better understand how their net flows perform along the path. Provide visibility of flows. Reviewed how scitags work: both packet marking in ipv6 and flow marking with udp fireflies for ipv4. Starting with dCache and Xrootd. Reported progress on Flow marking and packet marking. Xrootd 5.4.0 supports UDP fireflies. dCache – discussions on steps to implementation. Packet marking: flowd service generic service for packet and flow marking that storage can use via local api, testing a plugin that provides for marking of packets for 3rd party processes. Collectors: Various collector initiatives, ESNET and Jisc/Janet deployments. Registry– provide a list for experiments and activities supported, exposed by JSON at api.scitags.org. Various next steps: test/validate propagation flow identifiers; xRootD implementation; eBPF-TC to mark packets (demo at SC22); next release of flowd service.
7. AOB, future plans and next meetings
Next meetings – see above.
Future plans.
- Continue Carmen’s work on datamining data transfer data to look at XroodD transfers from the other experiments (Atlas, LHCb) then do the same for FTS transfers, the top talkers for LHCOPN /LCHONE.
- Francesco: Continue work on additional islands of ipv6-only to have more than just Cern, Brunel, Nebraska.
Discussion on which is the most useful of these approaches. Of similar importance. Examining existing data traffic will find bad config choices in existing software stacks whereas an ipv6-only cluster will show up issues in software not so far examined. Raja noted that LHCb happily running jobs on ipv6-only cluster at Brunel.
Possible wider use of DM to track down problems at sites due to reasons other than networking. MJB will talk to Katie and James at RAL.
Edoardo and Carmen will continue their activities.
Bruno will update the wiki to record his findings from his work at KIT.
End of Day2 meeting
-----------------------------------
Additional notes from Martin for Day 1 - DaveK will merge later.
Round table
CERN:
Issues with VXLAN and option 79 with IPv6 with adding the MAC address of the client. There was a fix but a subsequent update to fix other issues broke it again.
ESNET update: (Nick)
Moving ahead with ipv6 only network. Sone to have ipv6 only mgmt net for international backbone.
DOE update: (nick)
Slowing moving on deliverables. Next level is 20% implementation.
ITEF: (Nick)
Some work on dual stack networks, to be presented at next IETF meeting in 3 weeks.
Noted that the memo doesn't require 100% IPV6 only - some stuff that just has to be ipv4 can remain ipv4. Likely to use a NAT64 load balance to provide IPV4 access to ipv6 only connections.
Phil DeMar: summarised federal mandate:
FY23: 20%, FY24 50%, FY25 80.
US National labs and SLAC T2 covered, university T2s are not covered. DOE following OMB (Office of management and Budget) mandated. Office of Science advice is to concentrate of business and administration systems as ipv6-only and scientific computing on dual stack for now – ipv6only as collaborations allow.
US-CMS T1 fully dual stack. Stats on ipv4/ipv6 traffic. Notes on FNAL ipv6-only strategy. Separate VRFs for ipv6-only. NAT64 and DNS64 to facilitate interwork between ipv4 and ipv6.
Brno:
Update on Karlsruhe.
Francesco: Milan, INFN etc.
Accidentally advertised an ipv6 subnet to GARR which was silently dropped because it was smaller than the allocated subnet – so no functioning IPv6 for 10 days or so. No one noticed until started to investigate some network delays. (@Milan)
Tim Chown reports SKA not thinking of ipv6 to start with.
Duncan (Imperial):
Has been monitoring dCache for ipv6 compatible sites (running compatible version of dCache). Week of 23rd: 27/53 sites, this week – 31/57, all KIT. Bruno asserts that this is likely due to a broken BDII having been fixed.
Break
Fallback to IPv4 when IPv6 is broken: Robert Currie
Report from Rob on the fallout form a networking inclident which impacted only ipv6. Running DPM which didn’t react well to broken ipv6. Quick testing showd ed that ipv4 hosts could access nay hosts that were dual stacked from testing, but external access from dual stacked clients proved problematic. XrootD and some https transfers seemed to fail being marked as timed out. Expected; Site should have been marked as unavailable, wasn’t initially clear what services were impacted or how. Should HEP tools be expected follow the example of openssh or curl?
Francesco: in the end it is tcp timeout that would govern connectivity. Implementation specific.
Status of Tier1=2/LHCOPN/LHCONE: Andrea Sciabà, Bruno Hoeft, Edoardo Martelli
There are minutes attached to this event.
Show them.