Migrating to 100GE WAN Infrastructure at Fermilab

Apr 14, 2015, 3:00 PM
C209 (C209)



oral presentation Track6: Facilities, Infrastructure, Network Track 6 Session


Mr Phil Demar (Fermilab)


Fermilab is in the process of upgrading its wide-area network facilities to 100GE technology. One might assume that migration to be relatively straightforward, with forklift upgrades of our existing network perimeter devices to 100GE-capable platforms, and accompanying deployment of 100GE WAN services. However, our migration to 100GE WAN technology has proven to be significantly more complicated than that. Several key factors are driving this complexity: [1] The Laboratory has historically separated its high impact science (LHC) data movement from its general internet traffic. For a decade, this has been implemented with additional physical infrastructure, included a separate border router and multiple 10GE WAN links dedicated to the science data traffic. However, the extreme cost of 100GE routers, coupled with corresponding high cost of 100GE WAN links, has necessitated a consolidation of the current WAN infrastructure into a simpler configuration, one in which science data and general internet traffic must share the same 100GE infrastructure. [2] The Laboratory has also been able to rely on the diversity inherent in its 8x10GE WAN link configuration to provide acceptable resiliency in its WAN services. Replacement of the multiple 10GE links with a single 100GE link has the potential to seriously compromise that diversity. To alleviate that vulnerability, plans for deployment of a second 100GE WAN link in early 2015 are already under way. However, preserving full geographic diversity with limited 100GE equipment has presented additional challenges. [3] With the availability of 100GE WAN infrastructure, as well as 40GE-attached network R&D systems, traffic generated as part of network research activities also potentially becomes much more significant. While this is a positive thing for the Laboratory’s network R&D efforts, it adds yet another class of high-impact traffic competing for the new 100GE WAN resources. This talk will describe how Fermilab is consolidating its WAN infrastructure in order to implement 100GE technology, while still preserving its long term philosophy about keeping high impact science data traffic, its general internet traffic, and it network R&D traffic logically isolated from each other. Current virtualization techniques being deployed will be discussed, as will investigations into use of emerging Software Defined Networking (SDN) and OpenFlow technologies as a longer term solution.

Primary author

Mr Phil Demar (Fermilab)


Mr Andrey Bobyshev (FERMILAB) Vyto Grigaliunas (Fermilab)

Presentation materials