Figure 2: Reference Topology for EZ-LAG Configuration
EZ-LAG requires direct links between the two leafs where dual homing is needed. Only one is required, but more can be specified. One can notice the similarity here with MC-LAG that requires an Inter-Chassis Link (ICL) to run the Inter-Chassis Control Protocol (ICCP), but the major difference is that, unlike MC-LAG, that uses a proprietary protocol (ICCP), EZ-LAG only uses standard protocols, like OSPF and BGP.
The user must specify at least one such link in the EZ-LAG configuration of both peers, but any number of links can be specified as a list. Each peer will bundle these links into a single Aggregated Ethernet interface ae0, even when it is only one link.
IP addresses of both ends of ae0 are derived from the specified subnet in the configuration. The addresses will be used for underlay protocol peering, most commonly EBGP.
EZ-LAG also derives loopback addresses from the specified configuration and configures an overlay BGP sessions between these loopbacks. Overlay can be either EBGP or IBGP, as specified in the configuration.
The excerpt below shows all knobs under the new configuration stanza “services evpn”. Some of these parameters are mandatory and some are optional. The excerpt shows all possible configuration knobs, even though some of these are mutually exclusive. We will specifically call out the parameters from each category.
Table 1: Parameters in Services EVPN Configuration Hierarchy
Table 2 : Default and Auto-Derived Parameters
In the setup we used for this post, these are the configured attributes:
These are the auto derived attributes:
Basic EZ-LAG Configuration
The excerpt below provides the bare minimum EZ-LAG configurations of both peers that will lead to a working dual-homed scenario, starting from the base configurations shown in the excerpts. For the sake of conciseness, these configurations only show a working example of EZ-LAG between two peers, and do not include the portions that are needed to work with the rest of the fabric.
Peer 1 Configuration
Peer 1 Base Configuration
set version 23.2R1.13
set system host-name qfx1
set system root-authentication encrypted-password "<secret data>"
set system services ssh root-login allow
set system syslog user * any emergency
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set interfaces em0 unit 0 family inet address 192.168.1.101/24
set forwarding-options evpn-vxlan shared-tunnels
set routing-options static route 0.0.0.0/0 next-hop 192.168.1.1
set protocols lldp port-id-subtype interface-name
set protocols lldp interface all
Peer 1 EZ-LAG Configuration
set system scripts commit allow-transients
set system scripts commit file services_evpn_commit_script.py
set services evpn device-attribute peer-id 1
set services evpn device-attribute loopback-subnet 111.1.1.10/32
set services evpn device-attribute system-id 00:14:15:16:17:11
set services evpn device-attribute peer-to-peer peer-subnet inet 172.16.12.0/31
set services evpn device-attribute peer-to-peer peer-subnet interface-name xe-0/0/46
set services evpn evpn-vxlan irb IRB_100 vlan-id 100
set services evpn evpn-vxlan irb IRB_100 subnet-address inet 10.0.1.11/24
set services evpn evpn-vxlan irb IRB_200 vlan-id 200
set services evpn evpn-vxlan irb IRB_200 subnet-address inet 10.0.2.11/24
set services evpn evpn-vxlan server SERVER_1 esi-lag-id 1
set services evpn evpn-vxlan server SERVER_1 vlan-id-list 100
set services evpn evpn-vxlan server SERVER_1 interface xe-0/0/45
set services evpn evpn-vxlan server SERVER_2 esi-lag-id 2
set services evpn evpn-vxlan server SERVER_2 vlan-id-list 200
set services evpn evpn-vxlan server SERVER_2 interface xe-0/0/44
Peer 2 Configuration
Peer 2 Base Configuration
set version 23.2R1.13
set system host-name qfx2
set system root-authentication encrypted-password "<secret data>"
set system services ssh root-login allow
set system syslog user * any emergency
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set interfaces em0 unit 0 family inet address 192.168.1.102/24
set forwarding-options evpn-vxlan shared-tunnels
set routing-options static route 0.0.0.0/0 next-hop 192.168.1.1
set protocols lldp port-id-subtype interface-name
set protocols lldp interface all
Peer 2 EZ-LAG Configuration
set system scripts commit allow-transients
set system scripts commit file services_evpn_commit_script.py
set services evpn device-attribute peer-id 7
set services evpn device-attribute loopback-subnet 111.1.1.10/32
set services evpn device-attribute system-id 00:14:15:16:17:10
set services evpn device-attribute peer-to-peer peer-subnet inet 172.16.12.0/31
set services evpn device-attribute peer-to-peer peer-subnet interface-name xe-0/0/46
set services evpn evpn-vxlan irb IRB_100 vlan-id 100
set services evpn evpn-vxlan irb IRB_100 subnet-address inet 10.0.1.10/24
set services evpn evpn-vxlan irb IRB_200 vlan-id 200
set services evpn evpn-vxlan irb IRB_200 subnet-address inet 10.0.2.10/24
set services evpn evpn-vxlan server SERVER_1 esi-lag-id 1
set services evpn evpn-vxlan server SERVER_1 vlan-id-list 100
set services evpn evpn-vxlan server SERVER_1 vlan-id-list 200
set services evpn evpn-vxlan server SERVER_1 interface xe-0/0/45
set services evpn evpn-vxlan server SERVER_2 esi-lag-id 2
set services evpn evpn-vxlan server SERVER_2 vlan-id-list 100
set services evpn evpn-vxlan server SERVER_2 vlan-id-list 200
set services evpn evpn-vxlan server SERVER_2 interface xe-0/0/44
Host-1 Configuration Files (Centos 7)
/etc/sysconfig/network-scripts/ifcfg-ens2f0
NAME=ens2f0
UUID=4f8772aa-1c36-44d8-a1d5-8da3d7921eec
DEVICE=ens2f0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
NM_CONTROLLED=no
SLAVE=yes
MASTER=bond0
/etc/sysconfig/network-scripts/ifcfg-ens2f1
NAME=ens2f1
DEVICE=ens2f1
TYPE=Ethernet
UUID=9b89bf2d-2d26-49e8-b1d1-c532816e1dcb
BOOTPROTO=none
ONBOOT=no
NM_CONTROLLED=no
SLAVE=yes
MASTER=bond0
/etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
TYPE=Bond
ONBOOT=yes
BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=fast"
NM_CONTROLLED=no
MTU="9000"
/etc/sysconfig/network-scripts/ifcfg-bond0.100
DEVICE=bond0.100
ONPARENT=yes
BOOTPROTO=none
DEFROUTE=yes
IPADDR=10.0.1.111
PREFIX=24
VLAN=yes
NM_CONTROLLED=no
ZONE=trusted
/etc/sysconfig/network-scripts/route-bond0.100
10.0.2.0/24 via 10.0.1.254
10.0.3.0/24 via 10.0.1.254
10.0.4.0/24 via 10.0.1.254
Host-2 Configuration (Ubuntu 20.04)
network:
version: 2
renderer: networkd
ethernets:
ens2f0:
match:
macaddress: a0:42:3f:3a:f7:76
set-name: ens2f0
ens2f1:
match:
macaddress: a0:42:3f:3a:f7:77
set-name: ens2f1
bonds:
bond0:
dhcp4: false
dhcp6: false
interfaces: [ ens2f0, ens2f1 ]
parameters:
mode: 802.3ad
mii-monitor-interval: 100
vlans:
bond0.200:
id: 200
link: bond0
addresses : [10.0.2.2/24]
routes:
- to: 10.0.1.0/24
via: 10.0.2.1
- to: 10.0.3.0/24
via: 10.0.2.1
- to: 10.0.4.0/24
via: 10.0.2.1
Verifications
Let’s now look at the details of the configuration that has been generated by the commit script and verify that the setup is working as expected.
As explained at the beginning of this post, since the whole objective of this feature is to not expose the complexity of EVPN-VXLAN configuration to the user, we chose to implement it by using a commit script. The curious user can refer to the document listed in the “Useful Links” section to learn more about how commit scripts work on a Junos device. For the purpose of this post, we can simply say that a commit script is a tool that allows the user to verify at commit time that the configuration complies with certain user defined rules, and generate warnings, logs, or even change the configuration itself. It is the latter option that is used to implement EZ-LAG feature. In this case, when the user commits the configuration above, the commit script parses “services evpn” section of the configuration and generates other configuration sections that are not exposed by default in the configuration displayed with “show configuration” command. However, the user can always use “display commit-scripts” to see the entire configuration, the original one and the commit-script generated one, all merged into one configuration, with certain markers for easier reading.
The excerpt below shows the output of this special “show” command on Leaf-1 after committing the configuration listed above.
We can see that the commit script auto generated CLI configuration commands under interfaces, forwarding-options, policy-options, routing-instances, routing-options, and protocols stanzas.
{master:0}
root@qfx1> show configuration | display commit-scripts | display set
set version 23.2R1.13
set system host-name qfx1
set system root-authentication encrypted-password "$6$8xY0iVur$w8.f59mL/q7w4VtO4KZCCSape0vDokgktMxfJ2Z.MY.1pBjvVR5wPlhYiG.3sg31UsxjHnnWjHezBJ6kAQYnd0"
set system scripts commit allow-transients
set system scripts commit file services_evpn_commit_script.py
set system services ssh root-login allow
set system syslog user * any emergency
set system syslog file interactive-commands interactive-commands any
set system syslog file messages any notice
set system syslog file messages authorization info
set chassis aggregated-devices ethernet device-count 255
set chassis network-services enhanced-ip
set services evpn device-attribute peer-id 1
set services evpn device-attribute loopback-subnet 111.1.1.10/32
set services evpn device-attribute system-id 00:01:02:03:04:05
set services evpn device-attribute peer-to-peer peer-subnet inet 172.16.12.0/31
set services evpn device-attribute peer-to-peer peer-subnet interface-name xe-0/0/46
set services evpn evpn-vxlan irb IRB_100 vlan-id 100
set services evpn evpn-vxlan irb IRB_100 subnet-address inet 10.0.1.11/24
set services evpn evpn-vxlan irb IRB_200 vlan-id 200
set services evpn evpn-vxlan irb IRB_200 subnet-address inet 10.0.2.11/24
set services evpn evpn-vxlan server SERVER_1 esi-lag-id 1
set services evpn evpn-vxlan server SERVER_1 vlan-id-list 100
set services evpn evpn-vxlan server SERVER_1 interface xe-0/0/45
set services evpn evpn-vxlan server SERVER_2 esi-lag-id 2
set services evpn evpn-vxlan server SERVER_2 vlan-id-list 200
set services evpn evpn-vxlan server SERVER_2 interface xe-0/0/44
set interfaces xe-0/0/44 ether-options ethernet-switch-profile storm-control __SERVICES_EVPN_EVPN_VXLAN_STORM_CONTROL
set interfaces xe-0/0/44 ether-options 802.3ad ae2
set interfaces xe-0/0/45 ether-options ethernet-switch-profile storm-control __SERVICES_EVPN_EVPN_VXLAN_STORM_CONTROL
set interfaces xe-0/0/45 ether-options 802.3ad ae1
set interfaces xe-0/0/46 ether-options 802.3ad ae0
set interfaces ae0 aggregated-ether-options lacp active
set interfaces ae0 unit 0 family inet address 172.16.12.0/31
set interfaces ae1 vlan-tagging
set interfaces ae1 encapsulation flexible-ethernet-services
set interfaces ae1 esi auto-derive type-1-lacp
set interfaces ae1 esi all-active
set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 aggregated-ether-options lacp system-id 00:01:02:03:04:06
set interfaces ae1 unit 1 family ethernet-switching interface-mode trunk
set interfaces ae1 unit 1 family ethernet-switching vlan members 100
set interfaces ae2 vlan-tagging
set interfaces ae2 encapsulation flexible-ethernet-services
set interfaces ae2 esi auto-derive type-1-lacp
set interfaces ae2 esi all-active
set interfaces ae2 aggregated-ether-options lacp active
set interfaces ae2 aggregated-ether-options lacp system-id 00:01:02:03:04:07
set interfaces ae2 unit 1 family ethernet-switching interface-mode trunk
set interfaces ae2 unit 1 family ethernet-switching vlan members 200
set interfaces em0 unit 0 family inet address 10.87.11.81/25
set interfaces irb unit 100 virtual-gateway-accept-data
set interfaces irb unit 100 family inet address 10.0.1.11/24 virtual-gateway-address 10.0.1.254
set interfaces irb unit 100 virtual-gateway-v4-mac 00:00:5e:00:01:01
set interfaces irb unit 200 virtual-gateway-accept-data
set interfaces irb unit 200 family inet address 10.0.2.11/24 virtual-gateway-address 10.0.2.254
set interfaces irb unit 200 virtual-gateway-v4-mac 00:00:5e:00:01:01
set interfaces lo0 unit 0 family inet address 111.1.1.10/32 primary
set interfaces lo0 unit 0 family inet address 111.1.1.10/32 preferred
set forwarding-options storm-control-profiles __SERVICES_EVPN_EVPN_VXLAN_STORM_CONTROL all bandwidth-percentage 1
set forwarding-options evpn-vxlan shared-tunnels
set forwarding-options vxlan-routing next-hop 45056
set forwarding-options vxlan-routing interface-num 8192
set forwarding-options vxlan-routing overlay-ecmp
set policy-options policy-statement EXPORT-LO0 term LOOPBACK from interface lo0.0
set policy-options policy-statement EXPORT-LO0 term LOOPBACK then accept
set policy-options policy-statement EXPORT-LO0 term REJECT then reject
set policy-options policy-statement pplb then load-balance per-packet
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 instance-type mac-vrf
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 protocols evpn encapsulation vxlan
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 protocols evpn default-gateway no-gateway-community
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vtep-source-interface lo0.0
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 service-type vlan-aware
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 interface ae1.1
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 interface ae2.1
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 route-distinguisher 111.1.1.10:1
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vrf-target target:1:1
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vlans SERVICES_EVPN_EVPN_VXLAN_VLAN_100 vlan-id 100
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vlans SERVICES_EVPN_EVPN_VXLAN_VLAN_100 l3-interface irb.100
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vlans SERVICES_EVPN_EVPN_VXLAN_VLAN_100 vxlan vni 10100
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vlans SERVICES_EVPN_EVPN_VXLAN_VLAN_200 vlan-id 200
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vlans SERVICES_EVPN_EVPN_VXLAN_VLAN_200 l3-interface irb.200
set routing-instances __SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1 vlans SERVICES_EVPN_EVPN_VXLAN_VLAN_200 vxlan vni 10200
set routing-options router-id 111.1.1.10
set routing-options autonomous-system 65001
set routing-options static route 0.0.0.0/0 next-hop 10.87.11.126
set routing-options forwarding-table export pplb
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_UNDERLAY type external
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_UNDERLAY local-address 172.16.12.0
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_UNDERLAY export EXPORT-LO0
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_UNDERLAY local-as 65004
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_UNDERLAY multipath multiple-as
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_UNDERLAY neighbor 172.16.12.1 peer-as 65003
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_OVERERLAY type external
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_OVERERLAY multihop ttl 2
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_OVERERLAY multihop no-nexthop-change
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_OVERERLAY local-address 111.1.1.10
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_OVERERLAY family evpn signaling
set protocols bgp group __SERVICES_EVPN_EVPN_VXLAN_EBGP_OVERERLAY neighbor 112.1.1.10 peer-as 65002
set protocols evpn no-core-isolation
set protocols lldp port-id-subtype interface-name
set protocols lldp interface all
set protocols loop-detect enhanced interface ae1 vlan-id 100
set protocols loop-detect enhanced interface ae1 loop-detect-action interface-down
set protocols loop-detect enhanced interface ae1 transmit-interval 1s
set protocols loop-detect enhanced interface ae1 revert-interval 60
set protocols loop-detect enhanced interface ae2 vlan-id 200
set protocols loop-detect enhanced interface ae2 loop-detect-action interface-down
set protocols loop-detect enhanced interface ae2 transmit-interval 1s
set protocols loop-detect enhanced interface ae2 revert-interval 60
You see the benefits of EZ-LAG feature by comparing the length of the original config to that of the auto-generated one and understand the time saving to configure it and having less chances to make configuration mistakes at the same time.
Notice the highlighted system-ids for both ae interfaces that are autogenerated and must match on the two peers.
Let’s now verify that the setup is working as expected by running some operational CLI commands.
Leaf-1
{master:0}
root@qfx1> show interfaces terse
Interface Admin Link Proto Local Remote
<snip>
xe-0/0/44 up up
xe-0/0/44.1 up up aenet --> ae2.1
xe-0/0/44.32767 up up aenet --> ae2.32767
xe-0/0/45 up up
xe-0/0/45.1 up up aenet --> ae1.1
xe-0/0/45.32767 up up aenet --> ae1.32767
xe-0/0/46 up up
xe-0/0/46.0 up up aenet --> ae0.0
<snip>
ae0 up up
ae0.0 up up inet 172.16.12.0/31
ae1 up up
ae1.1 up up eth-switch
ae1.32767 up up
ae2 up up
ae2.1 up up eth-switch
ae2.32767 up up
<snip>
irb up up
irb.100 up up inet 10.0.1.11/24
10.0.1.254/24
irb.200 up up inet 10.0.2.11/24
10.0.2.254/24
<snip>
lo0.0 up up inet 111.1.1.10 --> 0/0
lo0.16384 up up inet 127.0.0.1 --> 0/0
lo0.16385 up up inet
<snip>
Leaf-2
{master:0}
root@qfx2> show interfaces terse
Interface Admin Link Proto Local Remote
gr-0/0/0 up up
pfe-0/0/0 up up
pfe-0/0/0.16383 up up inet
inet6
pfh-0/0/0 up up
pfh-0/0/0.16383 up up inet
pfh-0/0/0.16384 up up inet
xe-0/0/44 up up
xe-0/0/44.1 up up aenet --> ae2.1
xe-0/0/44.32767 up up aenet --> ae2.32767
xe-0/0/45 up up
xe-0/0/45.1 up up aenet --> ae1.1
xe-0/0/45.32767 up up aenet --> ae1.32767
xe-0/0/46 up up
xe-0/0/46.0 up up aenet --> ae0.0
et-0/0/48 up up
et-0/0/48.16386 up up
et-0/0/50 up up
et-0/0/50.16386 up up
ae0 up up
ae0.0 up up inet 172.16.12.1/31
ae1 up up
ae1.1 up up eth-switch
ae1.32767 up up
ae2 up up
ae2.1 up up eth-switch
ae2.32767 up up
bme0 up up
bme0.0 up up inet 128.0.0.1/2
128.0.0.4/2
128.0.0.63/2
cbp0 up up
dsc up up
em0 up up
em0.0 up up inet 10.87.11.82/25
em1 up down
em2 up up
em2.32768 up up inet 192.168.1.2/24
em3 up up
esi up up
fti0 up up
gre up up
ipip up up
irb up up
irb.100 up up inet 10.0.1.11/24
10.0.1.254/24
irb.200 up up inet 10.0.2.11/24
10.0.2.254/24
jsrv up up
jsrv.1 up up inet 128.0.0.127/2
lo0 up up
lo0.0 up up inet 112.1.1.10 --> 0/0
lo0.16384 up up inet 127.0.0.1 --> 0/0
lo0.16385 up up inet
lsi up up
mtun up up
pimd up up
pime up up
pip0 up up
tap up up
vme up down
vtep up up
vtep.32768 up up
vtep.32769 up up eth-switch
vtep.32771 up up
{master:0}
root@qfx2>
The above excerpt shows that the script created interfaces ae0.0, ae1.1, ae2.1, irb.100, irb.200, and lo0.0 and assigned the proper addresses.
It’s also important to verify that ae1 and ae2 interfaces are “up up” on both leafs, because ESI is configured in all-active mode.
Let’s first look at LACP status and statistics. It is important that Host-1 sees Leaf-1 and Leaf-2 as the same LACP system, therefore it does not know that it is dual-homed to two separate leafs. This is verified below because the System ID is identical for the “Actor” role on both leafs.
Leaf-1
{master:0}
root@qfx1> show lacp interfaces ae1 extensive
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/45 Actor No No Yes Yes Yes Yes Fast Active
xe-0/0/45 Partner No No Yes Yes Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/45 Current Fast periodic Collecting distributing
LACP info: Role System System Port Port Port
priority identifier priority number key
xe-0/0/45 Actor 127 00:01:02:03:04:06 127 2 2
xe-0/0/45 Partner 65535 90:e2:ba:d9:30:64 255 1 15
{master:0}
root@qfx1>
Leaf-2
{master:0}
root@qfx2> show lacp interfaces ae1 extensive
Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/45 Actor No No Yes Yes Yes Yes Fast Active
xe-0/0/45 Partner No No Yes Yes Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/45 Current Fast periodic Collecting distributing
LACP info: Role System System Port Port Port
priority identifier priority number key
xe-0/0/45 Actor 127 00:01:02:03:04:06 127 2 2
xe-0/0/45 Partner 65535 90:e2:ba:d9:30:64 255 2 15
{master:0}
root@qfx2>
Let’s now have a closer look at one of the AE interfaces.
Leaf-1
{master:0}
root@qfx1> show interfaces ae1 extensive
Physical interface: ae1, Enabled, Physical link is Up
Interface index: 692, SNMP ifIndex: 522, Generation: 183
Link-level type: Flexible-Ethernet, MTU: 1518, Speed: 10Gbps, BPDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 1bps
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x4000
Current address: 80:7f:f8:d2:d0:29, Hardware address: 80:7f:f8:d2:d0:29
Ethernet segment value: 01:90:e2:ba:d9:30:64:00:0f:00, Mode: all-active
Last flapped : 2023-08-28 09:28:48 UTC (02:31:09 ago)
Statistics last cleared: Never
Traffic statistics:
Input bytes : 1297388 1256 bps
Output bytes : 1332988 1256 bps
Input packets: 9983 1 pps
Output packets: 9957 1 pps
IPv6 transit statistics:
Input bytes : 0
Output bytes : 0
Input packets: 0
Output packets: 0
Input errors:
Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Giants: 0, Policed discards: 0, Resource errors: 0
Output errors:
Carrier transitions: 1, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0
Egress queues: 10 supported, 5 in use
Queue counters: Queued packets Transmitted packets Dropped packets
0 566 566 0
3 0 0 0
4 0 0 0
7 9386 9386 0
8 9 9 0
Queue number: Mapped forwarding classes
0 best-effort
3 fcoe
4 no-loss
7 network-control
8 mcast
Logical interface ae1.1 (Index 878) (SNMP ifIndex 537) (HW Token 1) (Generation 339)
Flags: Up SNMP-Traps 0x20024000 Encapsulation: Ethernet-Bridge
Statistics Packets pps Bytes bps
Bundle:
Input : 0 0 0 0
Output: 3 0 138 0
Adaptive Statistics:
Adaptive Adjusts: 0
Adaptive Scans : 0
Adaptive Updates: 0
Link:
xe-0/0/45.1
Input : 0 0 0 0
Output: 3 0 138 0
Aggregate member links: 1
Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx
xe-0/0/45.1 0 0 0 0
Protocol eth-switch, MTU: 1518, Generation: 305, Route table: 8, Mesh Group: __all_ces__, EVPN multi-homed status: Forwarding, EVPN multi-homed ESI Split Horizon Label: 0, Next-hop: 1804,
vpls-status: up
Local Bias Logical Interface Name: vtep.32770, Index: 881, VXLAN Endpoint Address: 112.1.1.10
Flags: Trunk-Mode, 0x4000000
Logical interface ae1.32767 (Index 879) (SNMP ifIndex 529) (Generation 340)
Flags: Up SNMP-Traps 0x4004000 VLAN-Tag [ 0x0000.0 ] Encapsulation: ENET2
Statistics Packets pps Bytes bps
Bundle:
Input : 0 0 0 0
Output: 0 0 0 0
Adaptive Statistics:
Adaptive Adjusts: 0
Adaptive Scans : 0
Adaptive Updates: 0
Link:
xe-0/0/45.32767
Input : 0 0 0 0
Output: 329 0 134873 0
LACP info: Role System System Port Port Port
priority identifier priority number key
xe-0/0/45.32767 Actor 127 00:01:02:03:04:06 127 2 2
xe-0/0/45.32767 Partner 65535 90:e2:ba:d9:30:64 255 1 15
LACP Statistics: LACP Rx LACP Tx Unknown Rx Illegal Rx
xe-0/0/45.32767 9048 9007 0 0
Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx
xe-0/0/45.32767 0 0 0 0
{master:0}
root@qfx1>
EZ-LAG sets LACP auto-derive mode to “type-1-lacp” and, in this mode, the ESI is the system ID and the administrative key on the remote CE device in the LACP link (partner). In our case, the remote CE device is Host-1
- 90:e2:ba:d9:30:64 - MAC address of the bond on Host-1
- 00:0f - admin key on Host-1
- 00 - the last octet is always 0
These information can be easily verified on Host-1
[root@5a7s38 ~]# ifconfig bond0 | grep ether
ether 90:e2:ba:d9:30:64 txqueuelen 1000 (Ethernet)
[root@5a7s38 ~]#
[root@5a7s38 ~]# tcpdump -i ens2f0 -vvv
tcpdump: listening on ens2f0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:53:42.884097 LACPv1, length 110
Actor Information TLV (0x01), length 20
System 90:e2:ba:d9:30:64 (oui Unknown), System Priority 65535, Key 15, Port 1, Port Priority 255
State Flags [Activity, Timeout, Aggregation, Synchronization, Collecting, Distributing]
0x0000: ffff 90e2 bad9 3064 000f 00ff 0001 3f00
0x0010: 0000
Partner Information TLV (0x02), length 20
System 00:01:02:03:04:06 (oui Unknown), System Priority 127, Key 2, Port 2, Port Priority 127
State Flags [Activity, Timeout, Aggregation, Synchronization, Collecting, Distributing]
0x0000: 007f 0001 0203 0406 0002 007f 0002 3f00
0x0010: 0000
Collector Information TLV (0x03), length 16
Max Delay 0
0x0000: 0000 0000 0000 0000 0000 0000 0000
Terminator TLV (0x00), length 0
Verify that ESI is identical on Leaf-2
{master:0}
root@qfx2> show interfaces ae1 extensive | match segment
Ethernet segment value: 01:90:e2:ba:d9:30:64:00:0f:00, Mode: all-active
{master:0}
root@qfx2>
It is important to verify that for each ae interface, only one leaf is forwarding BUM traffic to the server and the other leaf is blocking it.
{master:0}
root@qfx2> show interfaces ae1 extensive | match multi-homed
Protocol eth-switch, MTU: 1518, Generation: 301, Route table: 8, Mesh Group: __all_ces__, EVPN multi-homed status: Blocking BUM Traffic to ESI, EVPN multi-homed ESI Split Horizon Label: 0,
{master:0}
root@qfx2>
In our case, we can see that EVPN multi-homed status on Leaf-1 is “Forwarding” and that on Leaf-2, the status is “Blocking BUM traffic to ESI”.
From BGP perspective, we need to have two BGP sessions, one for the underlay and another for the overlay. Here both are “Established” and Leaf-1 received 12 EVPN routes.
{master:0}
root@qfx1> show bgp summary
Threading mode: BGP I/O
Default eBGP mode: advertise - accept, receive - accept
Groups: 2 Peers: 2 Down peers: 0
Table Tot Paths Act Paths Suppressed History Damp State Pending
bgp.evpn.0
12 12 0 0 0 0
inet.0
1 1 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
112.1.1.10 65002 130 131 0 0 55:16 Establ
bgp.evpn.0: 12/12/12/0
__SERVICES_EVPN_EVPN_VXLAN_MAC_VRF_1.evpn.0: 12/12/12/0
__default_evpn__.evpn.0: 0/0/0/0
172.16.12.1 65003 127 125 0 0 55:19 Establ
inet.0: 1/1/1/0
{master:0}
root@qfx1>
Now, it’s important to check EVPN routes that have been exchanged between the two peers, after verifying reachability between Host-1 and Host-2.
In the context of this post, we’re more particularly interested in Type 1 EVPN routes that are used for per Ethernet segment auto-discovery to advertise EVPN multi-homing mode.
{master:0}
root@qfx1> show route table bgp.evpn.0 protocol bgp
bgp.evpn.0: 44 destinations, 44 routes (44 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
1:112.1.1.10:0::0190e2bad1e230000f00::FFFF:FFFF/192 AD/ESI
*[BGP/170] 02:25:52, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
1:112.1.1.10:0::0190e2bad93064000f00::FFFF:FFFF/192 AD/ESI
*[BGP/170] 02:25:53, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
1:112.1.1.10:0::050000fdea0000277400::FFFF:FFFF/192 AD/ESI
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
1:112.1.1.10:0::050000fdea000027d800::FFFF:FFFF/192 AD/ESI
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
1:112.1.1.10:1::0190e2bad1e230000f00::0/192 AD/EVI
*[BGP/170] 02:25:53, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
1:112.1.1.10:1::0190e2bad93064000f00::0/192 AD/EVI
*[BGP/170] 02:25:54, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
2:112.1.1.10:1::10100::00:00:5e:00:01:01/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 631
2:112.1.1.10:1::10100::80:7f:f8:d2:02:60/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 631
2:112.1.1.10:1::10100::90:e2:ba:d9:30:64/304 MAC/IP
*[BGP/170] 01:34:16, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 631
2:112.1.1.10:1::10200::00:00:5e:00:01:01/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 637
2:112.1.1.10:1::10200::80:7f:f8:d2:02:60/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 637
2:112.1.1.10:1::10200::90:e2:ba:d1:e2:30/304 MAC/IP
*[BGP/170] 01:34:16, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 637
2:112.1.1.10:1::10100::00:00:5e:00:01:01::10.0.1.254/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 631
2:112.1.1.10:1::10100::80:7f:f8:d2:02:60::10.0.1.11/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 631
2:112.1.1.10:1::10100::90:e2:ba:d9:30:64::10.0.1.111/304 MAC/IP
*[BGP/170] 00:19:07, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 631
2:112.1.1.10:1::10200::00:00:5e:00:01:01::10.0.2.254/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 637
2:112.1.1.10:1::10200::80:7f:f8:d2:02:60::10.0.2.11/304 MAC/IP
*[BGP/170] 04:09:41, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 637
2:112.1.1.10:1::10200::90:e2:ba:d1:e2:30::10.0.2.112/304 MAC/IP
*[BGP/170] 00:19:07, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0, Push 637
3:112.1.1.10:1::10100::112.1.1.10/248 IM
*[BGP/170] 04:09:42, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
3:112.1.1.10:1::10200::112.1.1.10/248 IM
*[BGP/170] 04:09:42, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
4:112.1.1.10:0::0190e2bad1e230000f00:112.1.1.10/296 ES
*[BGP/170] 02:25:53, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
4:112.1.1.10:0::0190e2bad93064000f00:112.1.1.10/296 ES
*[BGP/170] 02:25:54, localpref 100, from 112.1.1.10
AS path: 65002 I, validation-state: unverified
> to 172.16.12.1 via ae0.0
{master:0}
root@qfx1>
In the CLI output above, we notice that there are two Type-1 EVPN routes per host, these are AD/ESI and AD/EVI routes. the AD/ESI route advertises the Ethernet segment to which the host is connected and the AD/EVI route advertises the remote EVPN to which the route belongs.
By looking at the ESIs in these routes and recalling what we explained above about how are these Type-1 ESI are constructed, we can identify that the 1st and 5th Type-1 routes are relative to Host-2, and that the 2nd and 6th Type-1 routes are relative to Host-1.
Now, how about the 3rd and 4th routes? Let’s dissect the ESI of these routes and figure it out.
ESI “050000fdea0000277400” can be deconstructed as follows:
- 0x05 : Autonomous System Number (ASN) based ESI
- 0x0000fdea : ASN = 65002 (decimal)
- 0x00002774 : Local Discriminator = 10100 (decimal). In this case, the VNI is used as a Local Discriminator
- 0x00 : the last octet is always zero
This Type-1 route is in fact relative to the Virtual Gateway that was auto-derived from the EZ-LAG configuration. by default, EZ-LAG configures IRB interfaces with a virtual gateway, but there are CLI knobs available to change that into anycast mode. Those commands are:
set services evpn global-parameters anycast-mac <mac_address>
set services evpn evpn-vxlan irb <irb_instance> use-anycast-address
We verified by testing, that when this mode is used, the 3rd and 4th Type-1 routes above do not show up anymore.
Conclusion
In this post, we explained EZ-LAG feature in details. This new feature was introduced by Juniper as an alternative to MC-LAG and based on EVPN-VXLAN technology. We showed that the configuration of this feature has been made much simpler than ESI-LAG, which requires a full-fledged EVPN-VXLAN configuration. EZ-LAG hides the complexity of an ESI-LAG configuration while still providing the same feature and the same service. The knobs supported by EZ-LAG CLI allow the user to tailor the configuration and avoid conflicts with existing one for other features on the same switch.
Useful links
Glossary
- AD : Auto Discovery
- ASN : Autonomous System Number
- BGP : Border Gateway Protocol
- CLI : Command Line Interface
- ESI : Ethernet Segment Identifier
- EVI : Ethernet VPN Instance
- EVPN : Ethernet Virtual Private Network
- LAG : Link Aggregation Group, aka bond or port channel
- MC-LAG : Multi-Chassis LAG
- OSPF : Open Shortest Path First
- VNI : Virtual Network Identifier
- VXLAN : Extended Virtual Local Area Network
Acknowledgments
Many thanks to Aninda Chatterjee, Santhosh Somasekharan Nair and Selvakumar Sivaraj
Comments
If you want to reach out for comments, feedback or questions, drop us a mail at:
Revision History
| Version |
Author(s) |
Date |
Comments |
| 1 |
Ridha Hamidi |
October 2023 |
Initial Publication |
#SolutionsandTechnology