Switching

Hello, we are trying to get an MC-LAG configuration to work and seeing some issues. We have a server with 2 25G links to a pair of QFX switches labeled as Leaf1A and Leaf1B. These are stand alone switches directly connect to each other via a LACP link of 2 100G ports (et-0/0/30 and et-0/0/31) using ae100 as the aggregator interface. This ae100 is a trunk that is carrying vlan 50 and vlan 100. leaf1a is configured with iccp address 10.0.0.1 on vlan 50 and switch leaf1b is iccp addressed 10.0.0.2 on vlan 50. We have the server connected to ports et-0/0/0:0 on both switches and the LACP is using ae1. 

ae1 is an access port with vlan 100. The gateway for the server is 10.0.1.254 and mcae is configured on the ae1. 

What we are seeing is pinging from the server to a different irb or l3 interface we get 90% packet loss to switch B. Pinging from the switches to the server sourced from the irb.100 vlan 100 gateway address of 10.0.1.254 about 99% of the time packets make it from switch A but only about 1% of the time does it work from switch B. 

JTAC is telling us that you can not have the same gateway for the server on both switches which would completely negate the LACP function. Also, we can not run this with vrrp as we need active/active, not active/standby.

Running a tap on the links, we can see that the packets are both being sourced from the same mac address and IP, so there should not be any arp or IP conflicts. We have tested this to make sure it is not a server config issue by also connecting a QFX5200 switch as if it is a server doing LACP to the two leaf switches and it exhibits the same behavior ruling out the possibility of it being anything on the server side of things. 

Anyone have any ideas or have a working config that you might be able to share?

Hi,

The two QFX5200 will have different MAC addresses for IRB.100 but the ARP table on the server will use only one MAC, which is why this will not work.

The solution for active-active with L3 on the Leaf is something that would use the same MAC, which is EVPN, ESI-LAG, and with two Leaf just "EVPN collapsed core", but the QFX5200 does not support EVPN, as far as I know.

I wonder what would happen, of it is possible, to manually configure the MAC manually. Never seen it tested...

Normally you would just run the Leaf in L2 active-active and the L3 at the next hop, so the spine in your diagram. If your Spine supports EVPN, then you can run active-active L3 there, so the Leaf would have MC-LAG also towards the Spines.

Another option is to use VRRP on the Leafs, but with different IP active on different Leaf, and then ideally distribute your workloads on the server to different IP, default gateway.

Hi,

Since your switches are directly connected, you can utilize the Virtual Chassis function. Two QFX5200 will work as a single switch. 

BR

hello Ihor,

the vc method has a disadvantage in the leaf/spine setup when needing to do a os/firmware upgrade on the switches that we are looking to avoid. It does appear that the mcae mac syncronize function does work (sort of) when the ICCP link is not running on the ICL link. The mac address gets sync'ed (so long as you don't have the ICCP running on the ICL), but the issue seems to be a bug that takes down LACP links stating that the IP address is not the same on both links even though it is. The consistency check fails on this. We have requested clarification with JTAC as when you turn consistency checks off, it all works fine. We ran for 28 hours with no issues in this setup. 

Hello,

When using same ip address on the irb interfaces you should use mac-sync feature to announce the same ip and mac-address for the the irb interface

https://www.juniper.net/documentation/us/en/software/junos/mc-lag/topics/concept/mc-lag-feature-concepts.html#mc-lag-concepts__d3103e76

I hope this helps with your problem.

Regards,