In the previous blog in this series, I examined enabling communication between SRIOV-based pods. But the reality is that data centers (especially ones needed for 5G deployment) have a mixture of workloads/pods with different network connectivity:
· Pods that have interfaces attached to a Virtual-Function(VF) of a SRIOV-enabled NIC which provides connectivity directly via the fabric underlay. We call these SRIOV pods. An example of such a pod is the User-Plane Function (UPF) in 5G where SRIOV can essentially take performance to line rate. While there are advantages, there are certain disadvantages as well like the need for fabric provisioning and a limited number of VFs in an SRIOV-enabled NIC
· Pods that have interfaces attached to virtualized router in the compute node (eg: CN2 vRouter) which provide connectivity via overlays (over a fabric underlay). We call these non-SRIOV pods
· Bare Metal Servers (BMS) that are not part of the Kubernetes clusters. The customer could be running applications directly on the native OS or the customer might have created VMs (eg: on ESXi server) or containers for running applications
Irrespective of the network connectivity to the workloads, we need to enable communication across such a heterogeneous mix. Also, there are use cases where the heterogeneous workloads can belong to the same Virtual-Network/Subnet or to a different Virtual-Network/Subnet. For example, the UPF can be an SRIOV pod, the Intermediate-UPF (I-UPF) and Gi-LAN services can be non-SRIOV pods all belonging to the same Virtual-Network/Subnet.
Given the above complex networking requirements especially with 5G deployments gaining traction, there is a need to address these requirements in order to simplify things for the customer. That is why we have provided user-friendly Kubernetes constructs and a CNI that hides all the above complexities and manual fabric provisioning. As part of CN2 release 23.1, Juniper now offers such Kubernetes-based constructs to allow Intra-VN and Inter-VN communication across a heterogeneous mix of SRIOV pods, non-SRIOV pods and BMS workloads.
Intra-VN Communication Between SRIOV Pods, non-SRIOV Pods and BMS
All steps below are done in Kubernetes unless specified.
One-time setup steps:
· Change CN2's encapsulation priority to vxlan in global-vrouter-config
· In Apstra, create a Remote EVPN Gateway referencing CN2 networking controllers
· Create a CN2 BGPRouter, referencing the Remote EVPN Gateway from the previous step. It will exchange EVPN Type-2 (ie MAC-IP) routes needed for intra-VN communication
Intra-VN provisioning steps:
· Create a common CN2 VN and add Apstra plugin label to extend the VN to the fabric
· Create SRIOV NAD and reference the common VN
· Create SRIOV pods and attach to the SRIOV NAD
· Create non-SRIOV NAD and reference the common VN
· Create non-SRIOV pods and attach to the non-SRIOV NAD
· In Apstra, attach fabric ports (connecting to BMS) to the VN created in Apstra via CN2
Inter-VN Communication Between SRIOV Pods, non-SRIOV Pods and BMS
All steps below are done in Kubernetes unless specified.
One-time setup steps:
· Change CN2's encapsulation priority to vxlan in global-vrouter-config
· In Apstra, create a Remote EVPN Gateway referencing CN2 networking controllers
· Create a CN2 BGPRouter, referencing the Remote EVPN Gateway from the previous step. It will exchange EVPN Type-5 (ie IP-Prefix) routes needed for inter-VN communication
Inter-VN provisioning steps:
· Create a CN2 VNR (with a vn selector label of your choosing) with routingType set to evpn to enable routing between all the VNs attached to this VNR (using a label selector)
· Create SRIOV NAD (and add Apstra plugin label and VNR selector label to match)
· Create SRIOV pods and attach to the SRIOV NAD
· Create non-SRIOV NAD (and add VNR selector label to match)
· Create non-SRIOV pods and attach to the non-SRIOV NAD
· Create reference NAD (and add VNR selector label to match) in CN2 with ssor label (Sytem Source Of Record) equal to Asptra to alias the BMS VN in Apstra into CN2
Sample Topology to Validate Intra/Inter-VN Communication Between SRIOV Pods, non-SRIOV Pods and BMS
------------------------------
Pradeep H Krishnamurthy
------------------------------