SD-WAN

Hi All,

I would like to know how the fabric fragmentation works and how a network-interface MTU configuration affects this feature. 

Currently I have a LAN and WAN interface for 2 routers which peer with each other. LAN MTU on both interfaces are configured as 1600, and WAN is configured as 1500 (left as default). When I enable path-mtu-discovery for the neighborhood used to facilitate the peering, I get a maximum path MTU of 1998. 

In this case, does the network-interface MTU get overwritten for traffic between these two peers? Technically if I generate traffic on the LAN side of router-1, destined for a service on the LAN side of router-2 (with DF bit set), I should theoretically be able to send 1600 byte packets end to end, correct? 

Also - is there something I need to enable in configuration for path-mtu to be taken into account, or does that happen automatically? 

Regards

------------------------------
Morne Vermeulen
Core Engineer
+27 (0) 10 141 8512
------------------------------

Hello Morne,

The short answer is yes you are right: you should be able to send your 1600-byte DF-set packets end to end. Since the transit MTU over the fabric has been discovered to be 1998, your 1600-byte packet will be sent without fabric fragmentation. There is no additional configuration required: discovered MTU supersedes configured MTU automatically.

Fabric fragmentation and reassembly comes into play when the fabric packet exceeds the egress MTU, whether that MTU is configured or discovered. The fragmentation algorithm is very similar to standard IPv4 fragmentation, but expanded to accommodate for limitations in RFC 791. Be aware that LAN-side packets, once encapsulated in the fabric, can expand due to encryption, HMAC digest addition, and metadata addition. Once reassembled, the original packet is restored.

Path-mtu-discovery is designed to override configured (or default) MTU. The reason for this is that if the discovered MTU is smaller than the configured one, traffic can be silently dropped along the fabric path (ICMP destination-unreachable packets may be generated) and that is very hard to diagnose. Conversely, if the discovered MTU is larger than the configured one, why not use it?

Regards,

------------------------------
Dennis G Montgomery
Principal Software Engineer
MA
(781) 203-8378
------------------------------

Hello Morne,

Additionally, as far as fabric fragmentation is concerned, we sort of take the notion of a "fragment" out of the visible network between 128T instances. Instead of the standard grouping of packets associated with a fragment (first packet with an L4 header, subsequent packets without) we segment a jumbo frame in such a manner that all "fragments" have a standard L4 header as they traverse the fabric as well as the additional metadata necessary to properly reassemble these fabric fragments at the next hop 128T. If one were to look at these packets in a capture, they would look like non-fragmented packets. We do this in hopes to avoid having any third party entity between the two instance of 128T from dropping packets due to them being fragments. Similar to typical IP fragmentation, the network interfaces MTU configuration dictates the largest sized fabric fragment that will be sent out of 128T.

Thanks,

Scott McCulley

------------------------------
Scott A McCulley
Senior Software Engineer
MA
(781) 203-8377
------------------------------

Original Message:
Sent: 06-25-2019 13:04
From: Dennis G Montgomery
Subject: Fabric Fragmentation and MTU

Hello Morne,

The short answer is yes you are right: you should be able to send your 1600-byte DF-set packets end to end. Since the transit MTU over the fabric has been discovered to be 1998, your 1600-byte packet will be sent without fabric fragmentation. There is no additional configuration required: discovered MTU supersedes configured MTU automatically.

Fabric fragmentation and reassembly comes into play when the fabric packet exceeds the egress MTU, whether that MTU is configured or discovered. The fragmentation algorithm is very similar to standard IPv4 fragmentation, but expanded to accommodate for limitations in RFC 791. Be aware that LAN-side packets, once encapsulated in the fabric, can expand due to encryption, HMAC digest addition, and metadata addition. Once reassembled, the original packet is restored.

Path-mtu-discovery is designed to override configured (or default) MTU. The reason for this is that if the discovered MTU is smaller than the configured one, traffic can be silently dropped along the fabric path (ICMP destination-unreachable packets may be generated) and that is very hard to diagnose. Conversely, if the discovered MTU is larger than the configured one, why not use it?

Regards,

------------------------------
Dennis G Montgomery
Principal Software Engineer
MA
(781) 203-8378
------------------------------

Thanks Dennis, 

Based on your and Scott's answer, I would imagine then that even traffic with the df-bit set would be forwarded and fragmented successfully over the WAN? Seeing as the end user will never see the fragmented traffic, there is no reason to honour this, and I should be able to send 1546 MTU df traffic over 2 128T nodes that have a WAN MTU of 1500 and a LAN MTU of 1600 respectively?

I'll get a lab environment set up with this as well to test.

Regards,

------------------------------
Morne Vermeulen
Core Engineer
+27 (0) 10 141 8512
------------------------------

Original Message:
Sent: 06-25-2019 13:04
From: Dennis G Montgomery
Subject: Fabric Fragmentation and MTU

Hello Morne,

The short answer is yes you are right: you should be able to send your 1600-byte DF-set packets end to end. Since the transit MTU over the fabric has been discovered to be 1998, your 1600-byte packet will be sent without fabric fragmentation. There is no additional configuration required: discovered MTU supersedes configured MTU automatically.

Fabric fragmentation and reassembly comes into play when the fabric packet exceeds the egress MTU, whether that MTU is configured or discovered. The fragmentation algorithm is very similar to standard IPv4 fragmentation, but expanded to accommodate for limitations in RFC 791. Be aware that LAN-side packets, once encapsulated in the fabric, can expand due to encryption, HMAC digest addition, and metadata addition. Once reassembled, the original packet is restored.

Path-mtu-discovery is designed to override configured (or default) MTU. The reason for this is that if the discovered MTU is smaller than the configured one, traffic can be silently dropped along the fabric path (ICMP destination-unreachable packets may be generated) and that is very hard to diagnose. Conversely, if the discovered MTU is larger than the configured one, why not use it?

Regards,

------------------------------
Dennis G Montgomery
Principal Software Engineer
MA
(781) 203-8378
------------------------------

Hello Morne,

Yes that is correct.  The fabric/SVR will preserve the DF bit when the packet is delivered to the destination LAN, even if the packet had to be fabric-fragmented in transit.  If, of course, you have path-mtu-discovery enabled and your path MTU is 1998, as you described in your initial post, fabric fragmentation wouldn't be needed.

Regards,

Dennis Montgomery

------------------------------
Dennis G Montgomery
Principal Software Engineer
MA
(781) 203-8378
------------------------------

Original Message:
Sent: 07-10-2019 11:22
From: Morne Vermeulen
Subject: Fabric Fragmentation and MTU

Thanks Dennis,

Based on your and Scott's answer, I would imagine then that even traffic with the df-bit set would be forwarded and fragmented successfully over the WAN? Seeing as the end user will never see the fragmented traffic, there is no reason to honour this, and I should be able to send 1546 MTU df traffic over 2 128T nodes that have a WAN MTU of 1500 and a LAN MTU of 1600 respectively?

I'll get a lab environment set up with this as well to test.

Regards,

------------------------------
Morne Vermeulen
Core Engineer
+27 (0) 10 141 8512
------------------------------


Original Message:
Sent: 06-25-2019 13:04
From: Dennis G Montgomery
Subject: Fabric Fragmentation and MTU

Hello Morne,

The short answer is yes you are right: you should be able to send your 1600-byte DF-set packets end to end. Since the transit MTU over the fabric has been discovered to be 1998, your 1600-byte packet will be sent without fabric fragmentation. There is no additional configuration required: discovered MTU supersedes configured MTU automatically.

Fabric fragmentation and reassembly comes into play when the fabric packet exceeds the egress MTU, whether that MTU is configured or discovered. The fragmentation algorithm is very similar to standard IPv4 fragmentation, but expanded to accommodate for limitations in RFC 791. Be aware that LAN-side packets, once encapsulated in the fabric, can expand due to encryption, HMAC digest addition, and metadata addition. Once reassembled, the original packet is restored.

Path-mtu-discovery is designed to override configured (or default) MTU. The reason for this is that if the discovered MTU is smaller than the configured one, traffic can be silently dropped along the fabric path (ICMP destination-unreachable packets may be generated) and that is very hard to diagnose. Conversely, if the discovered MTU is larger than the configured one, why not use it?

Regards,

------------------------------
Dennis G Montgomery
Principal Software Engineer
MA
(781) 203-8378