- Compact style
- Indico style
- Indico style - inline minutes
- Indico style - numbered
- Indico style - numbered + minutes
- Indico Weeks View
In place of the normal face to face meeting at CERN - a fully virtual Zoom meeting.
Please register to say you will attend the Zoom meeting - connection details will be available to those who register.
The agenda is DRAFT - still being defined
October 14, 2021, 15-18
first session of another two half-day IPv6 meeting on Zoom.
(notes by Francesco P)
In attendance: Marian Babik, Andrey Bobyshev, Nick Buraglio, Tim Chown, Dave Kelsey, Edoardo Martelli, Raja Nandakumar, Kars Ohrenberg, Francesco Prelz, Duncan Rand, Andrea Sciaba'.
Agenda at: https://indico.cern.ch/event/1083277/
Dave reviews the agenda.
20' for a first round of roundtable updates:
Francesco P. (INFN): No news from INFN-Turin on the storage IPv6 migration. They stopped even sending apologies via e-mail... No worker node PCAP file was dropped in for IPv4 application analysis . Just a reminder, the dropbox is available here (link to CERNbox): https://l.infn.it/ipv6dumpdrop
Apologies from Martin B. RAL report was sent via e-mail:
+--------[TODO: Please paste Martin's e-mail here] [DONE: see below]---------+
The Tier1 is progressing with the implementation of the new Leaf/Spine network – we now have connectivity between the new L/S network and the Legacy Network via the SCD Super Spine (no hosts on the L/S network as yet). The CTA (new tape service) network is also connected to the Super Spine and it has some hosts with Tier1 addresses on it as well as its own separate address space. We now have connectivity between the L/S network and the site core, and are currently working on connectivity to the border routers. This takes time as there are change control processes to go through and scheduling to avoid collisions with other central networking changes windows. When the connectivity is fully implemented we anticipate changing the connectivity for the Tier1 over to use the new exit routers in all cases and the legacy side will access ‘outside’ via the Super Spine to the new L/S network. This will provide L/S/SS link and external links over 100Gbps pipes and remove all of the current exit choke points (40Gbps).
I think I reported last time that our current T1 <-> Site Core pipe (40Gbps) now carries both ipv4 and ipv6 traffic removing the 10Gbps choke for ipv6. We had hoped that this would solve the problem of ipv6 not switching to the standby path in the event of an issue – this has not proved to be the case. Given these links will be retired in the next few weeks, we chose not to spend effort investigating this issue.
We intend to have a ritual immolation of the Extreme x670v routers on their retirement, no flowers by request. The OPNR s4810p units will be allowed a graceful retirement
Edoardo M.: No IPv6 news from CERN.
Kars O.: Things are happening at DESY, but nothing related with IPv6.
Bruno H.: There are slight improvements here and there, but we are still on the move at getting workable administrative access via IPv6 to our network equipment. There there is still legacy hardware that will never support IPv4.
Duncan R.: no news from Imperial College.
Tim C.: No news besides the IETF activity that will be reported tomorrow.
Dave K.: May want to ping GridPP and see what they are up to, or what they are actually monitoring.
Duncan R.: Can try to revive the links between Imperial and Queen Mary, and perhaps provide some WN PCAP dump for analysis.
Francesco P.: To add a figure besides our generalised lack of news,
the usual Google statistics
(https://www.google.com/intl/en/ipv6/statistics.html)
show that the linear increase in IPv6 adoption continues, and the global IPv6 traffic fraction, as seen by Google, has now crossed the 35% mark.
Dave K.: Any news from the experiments? Dimitrios and Atlas aren't here.
Raja N. shows a few slides (wearing first his LHC-B hat, then his DUNE hat...):
Duncan will encourage the addition of more IPv6-only worker nodes at Brunel.
Dave K.: Is the dCache IPv6-only testbed something that the developer runs ?
Raja N. I suspect that Fermilab has a dual-stack dCache instance available, and this is used for testing from IPv6-only.
Andrea S.: No news from CMS - CMS is good!
Marian B. talks about the packet-marking activities at the Research Network
Technical WG and shows these slides:
Tim C.: You mentioned you could include other information in the UDP fireflies, of interest of the far end - or people in the middle - security allowing.
Marian B.: There is a linux tool called 'ss' that gets a full dump of all open sockets. I hope to be able to add the flows and explore various options. We definitely want to discuss This.
Tim C.: The technique has a lot of potential, bu the interesting question is what should be include in the metadata.
Dave K.: whichever of the two techniques is used, monitoring has to be done at different places. What can cause UDP and TCP traffic to take different paths ?
Marian B.: It's unlikely that a TCP transfer takes a completely different path than UDP.
Dave K.: Are the two techniques somehow in competition ?
MArian B.: We will be pursuing both techniques. We need components with the ability to access the network packets at various places. The extension header with the destination label is new.
Tim C.: This issue has come out within IETF too, to help with the
monitoring of different applications. The nice thing about the fireflies
is that they can use more than the 20 bits allowed by the packet header.
Marian B.: The 2-million UDP fireflies that ESnet has captured provide info
on both IPv4 and IPv6.
Francesco P.: Out of curiosity: did the usage of the term 'firefly' for this
kind of marker UDP packets start within this worker group? I never
heard about this term before.
Marian B.: Fundamentally yes, it was invented here.
Tim C.: A better name could be 'tracer packets', like 'tracer bullets'.
Phil de Mar shares a few slides on the IPv6 mandate on US labs:
(to be uploaded: *** add link here)
(Andrey comments on the details on slide 7)
Questions on Phil's talk:
Duncan R.: Why were the US national labs left out last time around?
Phil D.M.: They probably knew it would be hard, and that DOE could have
used some more time to get ready.
Dave K.: Similar to WLCG, and this group's approach, where we threaten to
go IPv6-only so as to get at least dual-stack.
Cybersecurity concerns are prevalent everywhere.
1+1 > 2 in terms of vulberability.
Nick Buraglio introduces himself:
I'm leading the implementation of the ESNET IPv6-only program. I wrote the
implementation program, which is now pending DOE approval - will be
submitted to the US OMB (Office of Management and Budget).
My background is in supercomputing in science. Worked @ NCSA and University
of Illinois - so I'm keeping a keen eye on the scientific community needs.
ESNET had been working for 1 1/2 years to make the network management layer
IPv6-only. My office has been IPv6-only for about 18 months (since the
beginning of the pandemics). We've made enough experience up from Layer-1
and all the way through the network stack.
I've been involved with IPv6 since 2002: I understand this is an iterative
process. You cannot write a document with nice plans without making provisions
for changes. Every document has to be a living document, and be adjusted
promptly in case of need. We want this process to succeed as best as we can:
the labs will likely not be excluded from this process.
One caveat: there's one exclusion on the memo: National Security Systems -
as they include specialty, uncommon items. Hovever science is similar,
in a different way.
Note: My office is behind a DNS64/NAT64, as certain things "out there"
just don't work with IPv6 only.
Francesco P.: We've been discussing time and again about the class
of applications that cannot work behind NAT64 (IPv4 literals in the
payload or in databases, etc...). What did you find in your experience ?
Nick B.: House-developed applications are always the "long pole in the tent",
and there's no incentive to change them.
We experienced significantly less problems this time around w.r.t
the first time DNS64/NAT64 was tested 10 years ago.
The biggest problem we encountered was with Spotify (which is important in
campus environments): the desktop application doesn't work behind NAT64.
The mobile app is better. Github, too is irksome here and there.
'Tectonic shifts' are occurring in that organisation...
( - 10 minutes coffee break - )
As we are a bit behind with the agenda, and Andrey may want to
hear this today, Ben Jones presents the slides on IPv6-only right away:
Andrea S.: How likely will the new computer centre be IPv6-only ?
Ben J. There are things we need that will be IPv4-only, as far as I can see.
There's no way we can be out of AFS by the time new computing centre
is turned on.
Andrea S.: Doesn't AFS support IPv6 now ?
Ben J.: The new kernel module may support that, but we haven't checked it out.
Edoardo M.: It's in the roadmap, but not implemented yet.
Ben J.: Fascinating to see how this "impossible" task is now progressing.
Francesco P., wearing his small CERN/AD experiment data 'manager' hat: I've
been threatened with unpredictable consequences if we didn't move out
of AFS 6 years ago, and we moved out... How come AFS is still an issue ?
Ben J. I've been seeing plans to move out of AFS going nowhere for most of
my career in IT. Very realistically, it's not going - for certain
things it's just too good... It's still the filesystem used for batch
submission.
Dave K.: Where ?
Ben J.: In the Atlas Tier-0.
And... CERN is not running out of IPv4 addresses - is actually handing
them back. But probably we need an optimist here, not a cynicist...
Tim C.: If you tweak the address selection recipe the fraction of
data transfers may increase. There may be transfers that 'could'
be using IPv6.
Nick B.: Getting IPv6 deployed is largely an issue of management buy-in, and
management will buy if they are forced to (e.g. by the US Government) or
if there is a business driver (better performance, etc.), that may cause
it to turn into a fire that need to be taken care of. Otherwise the
process will keep lagging.
Tim C.: The carrot of getting access to IPv6-only resources or running
out of IPv4 addresses may have to turn into a stick.
Nick B.: At NCSA the biggest issue is that it was a big AFS shop. We put IPv6
everywhere it could go, but not "elsewhere".
Edoardo shows the CERN graph of LHCONE+LHCOPN, showing a 50% IPv4 share
https://twiki.cern.ch/twiki/bin/view/LHCOPN/LHCOPNEv4v6Traffic
Dave K.: Why is that, as 100% of LHCOPN should be IPv6 capable ?
Is that because the LHCONE data are included ?
Tim C.: More data could be fed into netsage, that I'm a big fan of.
The netsage ingest pipeline can also be run a local container, if privacy
concerns have to be addressed.
Francesco P.: I can also look at a few PCAP snippets - or other flow data,
if they are shareable and/or available.
Dave K.: It would be nice to see the LHCOPN data only, disentangled from
LHCONE.
Edoardo M.: Ok.
Andrea S. shows the Tier-2 migration status, as of this morning, on the
usual page:
https://twiki.cern.ch/twiki/bin/view/LCG/WlcgIpv6#WLCG_Tier_2_IPv6_deployment_stat
Dave K.: Didn't USATLAS, one of the main 'culprits', have a plan to be finished
by the end of the year?
Andrea S.: There is a Google doc maintained by US-ATLAS, referenced in the
page above (https://docs.google.com/spreadsheets/d/1d2FbmFoXZkBP_cAmJ5q5kWgdsGnWuyFT0ot1n9Gf4ns/edit?usp=sharing).
There was some progress w.r.t. last year, but don't know how often this
document is updated.
Some sites just don't see the need to use IPv6, so there's no workable
stick that can be used. Some Tier-2's are living inside a campus, and have
no leverage to impose IPv6 on the campus.
Bruno H.: Freiburg plans to get IPv6 ready by either end of 2021 or 2022.
Progressing slowly but steadily...
Dave K.: No news on Tier-1 status ?
Bruno H.: They are all done. Kurchatow Institut in Russia, the last missing
T1 site, communicated they are now IPv6 ready and tested.
------------------------------
HepIX ipv6 meeting minutes : 15 Oct 2021
notes by Raja N
Monitoring including perfSONAR & ETF
Speaker: Marian Babik (CERN)
IPv4 transfers, FTS, XrootD monitoring etc
Discussion
[Break for 15 minutes]
IPv6 at IETF - DHCPv6 and other issues
Speaker: Tim Chown
Conference submissions
HEPiX, ISGC2022, TNC22?
Roundtable updates (continued)
Next meetings:
AOB, future plans