Kubernetes Tutorial/Hackathon- Please sign up by Friday, April 5 especially if you plan to attend in person.  Will send an email out to this effect. 

Multi-site stretched cluster assembled with Kubespray, and using Wireguard as the fundamental network layer.

Wireguard is a VPN technology. We can assemble a VPN mesh that encrypts all interal cluster traffic and requires a site to only expose 1 UDP port to the public internet for the most essential connectivity. Wireguard is built into the Kernel (above v5.6?) creates a private interface on each node. To Kubernetes it appears to be all on 1 private network. However we need to understand what it looks like to expose public services. Public-facing services where we can, and tunneled private traffic where we have to? 

Wireguard config example:

[Peer]
PublicKey = xxjmp6WyT7IU/9hffUjyV0uj8sfYzR6G3C/I3yt+Qxk= # Elliptic curve public key
AllowedIPs = 192.168.0.6/32 # INTERNAL IP assigned to the 'wg0' interface 
Endpoint = 192.41.231.216:51820 # EXTERNAL IP and UDP port assigned for negotiating Wireguard traffic
PersistentKeepalive = 30 # Periodic ping between nodes to keep the conenction alive

[Peer]
PublicKey = oVVQuMR2hHCW+a5y0w4BS9ySOQK2pp8Tkba4RP5TByM=
AllowedIPs = 192.168.0.7/32
Endpoint = 192.41.237.213:51820
PersistentKeepalive = 30

[Peer]
PublicKey = BFh6AaxOf8rmDE68BtRcdcEIrQRrx6TklfZozLm3d28=
AllowedIPs = 192.168.0.8/32
Endpoint = 206.12.98.227:51820



Kubespray config sample - each site has a label corresponding to its site in CRIC as well as the institution where it sits:

# ...
   uchicago005.hl-lhc.io:
     ansible_host: 192.168.0.5
     ip: 192.168.0.5
     access_ip: 192.168.0.5
     node_labels:
       site: mwt2
       institution: uchicago
   umich001.hl-lhc.io:
     ansible_host: 192.168.0.6
     ip: 192.168.0.6
     access_ip: 192.168.0.6
     node_labels:
       site: aglt2
       institution: umich
   msu001.hl-lhc.io:
     ansible_host: 192.168.0.7
     ip: 192.168.0.7
     access_ip: 192.168.0.7
     node_labels:
       site: aglt2
       institution: msu
   uvic001.hl-lhc.io:
     ansible_host: 192.168.0.8
     ip: 192.168.0.8
     access_ip: 192.168.0.8
     node_labels:
       site: uvic
       institution: uvic
# ...



Kubectl:

[root@uchicago002 ~]# kubectl get nodes
NAME                    STATUS   ROLES           AGE     VERSION
msu001.hl-lhc.io        Ready    <none>          6d21h   v1.28.6
uchicago002.hl-lhc.io   Ready    control-plane   6d21h   v1.28.6
uchicago003.hl-lhc.io   Ready    control-plane   6d21h   v1.28.6
uchicago004.hl-lhc.io   Ready    control-plane   6d21h   v1.28.6
uchicago005.hl-lhc.io   Ready    <none>          6d20h   v1.28.6
umich001.hl-lhc.io      Ready    <none>          6d21h   v1.28.6
uvic001.hl-lhc.io       Ready    <none>          6d21h   v1.28.6