Routing

Expand all | Collapse all

QFX10002 VXLAN ":vxlan.inet.0" and a labeled next-hop.

  • 1.  QFX10002 VXLAN ":vxlan.inet.0" and a labeled next-hop.

    Posted 28 days ago
    Hey all,

    I'm working on creating some configurations for using a QFX10002 as leaf within an existing QFX5100 EVPN/VXLAN topology. In my current topology we do not use a classical leaf-spine layout but a more randomly connected mesh using MPLS for path optimization.

    Everything seems to be working fine on EVPN/VXLAN level, putting all the existing QFX5k configuration within a "virtual-switch" routing instance on the QFX10k. I'm experienced 1 big problem:
    -OSPF/LDP/RSVP as "underlay"
    - iBGP as "overlay" (not only evpn but also inet-vpn, inet6-vpn, ..), RR's in the core, route-reflectors clients at the edge (the qfx10k in this case is rr-client)
    - traffic engineering (mpls)  is configured as bgp

    After a clean reboot of the qfx10k layer2 communication between ports on qfx5k (192.0.2.2) and qfx10k (192.0.2.11) are working fine. I'm seeing the next-hop in the ":vxlan.inet.0" as unicast:

    root@ed1.lab# run show route forwarding-table matching 192.0.2.2 
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    192.0.2.2/32   user     1 192.0.2.206    ucst     1741    58 ae2.0
    
    Routing table: __master.anon__.inet
    Internet:
    
    Routing table: __pfe_private__.inet
    Internet:
    
    Routing table: :vxlan.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    192.0.2.2/32   user     0                    indr  2097181     3
                                  192.0.2.206    ucst     1741    58 ae2.0​



    When I make any change on one of the ports on the qfx10k (or creating flap of ldp) we do see the next-hop moving to a labeled switched path. This makes more sense as the vtep's are learned via bgp and should be resolved out of "inet.3":

    root@ed1.lab# run show route forwarding-table matching 192.0.2.2 
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    192.0.2.2/32   user     1 192.0.2.206    ucst     1741    54 ae2.0
    
    Routing table: __master.anon__.inet
    Internet:
    
    Routing table: :vxlan.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    192.0.2.2/32   user     0                    indr  2097198     3
                                  192.0.2.206   Push 30     1777     4 ae2.0
    ​


    At this point traffic flows are broken:
    - Ports on the qfx10k are still able to talk to eachother
    - Traffic from the qfx5k ports still arrives on qfx10k but the response from those systems behind the qfx10k does never reaches the qfx5k ports and is dropped somewhere.


    Questions:
    - After the reboot shouldn't we automatically move to the broken situation? It looks like for some reason the ldp route is not superseding the old unicast route that came up first. After the first change it behaves like expected and we can flip/flop the situation by disabling/enabling LDP.
    - Is this broken situation a limitation of the q5 chips to not be able to put a vxlan and mpls label at the same time? Did I overlook it in the list of vxlan en mpls limitations?


    Full configuration:
    root@ed1.lab> show configuration 
    ## Last commit: 2021-02-06 21:55:20 CET by root
    version 20.4R1.12;
    system {
        host-name ed1.lab;
        root-authentication {
            encrypted-password "****"; ## SECRET-DATA
        }
        services {
            ssh {
                root-login allow;
            }
        }
        time-zone Europe/Brussels;
        ntp {
            server x.x.x.x;
            server x.x.x.x;
        }
    }
    chassis {
        aggregated-devices {
            ethernet {
                device-count 30;
            }
        }
        fpc 0 {
            pic 0 {
                port 0 {
                    channel-speed 10g;
                }
                port 33 {
                    channel-speed 10g;
                }
                port 37 {
                    speed 100g;
                }
                port 41 {
                    speed 100g;
                }
            }
        }
        alarm {
            management-ethernet {
                link-down ignore;
            }
            ethernet {
                link-down ignore;
            }
        }
    }
    security {
        authentication-key-chains {
            key-chain upstream-bfd {
                key 0 {
                    secret "****"; ## SECRET-DATA
                    start-time "1970-1-1.01:00:01 +0100";
                }
            }
        }
        ipsec {
            security-association core {
                mode transport;
                manual {
                    direction bidirectional {
                        protocol ah;
                        spi 256;
                        authentication {
                            algorithm hmac-sha1-96;
                            key ascii-text "****"; ## SECRET-DATA
                        }
                    }
                }
            }
        }
    }
    interfaces {
        xe-0/0/2:0 {
            flexible-vlan-tagging;
            mtu 9216;
            encapsulation flexible-ethernet-services;
            esi {
                00:01:99:00:00:00:01:99:11:02;
                all-active;
            }
            unit 1 {
                encapsulation vlan-bridge;
                vlan-id 127;
            }
        }
        xe-0/0/33:0 {
            ether-options {
                802.3ad ae2;
            }
        }
        xe-0/0/33:1 {
            ether-options {
                802.3ad ae2;
            }
        }
        xe-0/0/33:2 {
            ether-options {
                802.3ad ae2;
            }
        }
        xe-0/0/33:3 {
            ether-options {
                802.3ad ae2;
            }
        }
        ae2 {
            mtu 9216;
            aggregated-ether-options {
                lacp {
                    active;
                    periodic fast;
                }
            }
            unit 0 {
                family inet {
                    address 192.0.2.207/31;
                }
                family inet6 {
                    address AAAA:BBBB:0000:0067::3/127;
                }
                family mpls;
            }
        }
        em0 {
            disable;
        }
        lo0 {
            unit 0 {
                family inet {
                    address 192.0.2.11/32 {
                        primary;
                    }
                }
                family inet6 {
                    address AAAA:BBBB:0:0::11/128 {
                        primary;
                    }
                }
            }
        }
    }
    policy-options {
        prefix-list interco_list;
        prefix-list interco6_list;
        policy-statement export6_core {
            term connected {
                from {
                    protocol direct;
                    prefix-list interco6_list;
                }
                then {
                    community set interco;
                    next-hop self;
                    accept;
                }
            }
            term static {
                from {
                    protocol static;
                    tag 10;
                }
                then {
                    community set statics;
                    accept;
                }
            }
            term bgp {                      
                from protocol bgp;
                then accept;
            }
            then reject;
        }
        policy-statement export_core {
            term connected {
                from {
                    protocol direct;
                    prefix-list interco_list;
                }
                then {
                    community set interco;
                    next-hop self;
                    accept;
                }
            }
            term static {
                from {
                    protocol static;
                    tag 10;
                }
                then {
                    community set statics;
                    accept;
                }
            }
            term bgp {
                from protocol bgp;
                then accept;
            }
            then reject;
        }
        policy-statement import_vxlan_lab {
            term vxlan_esi {
                from community vxlan_esi;
                then accept;
            }
            term vxlan_127 {
                from community vxlan_127;
                then accept;
            }
            then reject;
        }
        policy-statement load-balance {
            then {
                load-balance per-packet;
            }
        }
        policy-statement multipath-resolv {
            from {
                route-filter 192.0.2.0/28 exact;
            }
            then multipath-resolve;
        }
        community aggregates members 64512:5;
        community interco members 64512:2;
        community statics members 64512:10;
        community vxlan_127 members target:64512:127;
        community vxlan_esi members target:64512:1;
    }
    routing-instances {
        lab {
            protocols {
                evpn {
                    extended-vni-list 127;
                    encapsulation vxlan;
                    default-gateway no-gateway-community;
                    vni-options {
                        vni 127 {
                            vrf-target target:64512:127;
                        }
                    }
                }
            }
            vtep-source-interface lo0.0;
            instance-type virtual-switch;
            route-distinguisher 192.0.2.11:1;
            vrf-import import_vxlan_lab;
            vrf-target target:64512:1;
            vlans {
                lan {
                    interface xe-0/0/2:0.1;
                    vxlan {
                        vni 127;
                        ingress-node-replication;
                    }
                }
            }
        }
    }
    routing-options {
        resolution {
            rib :vxlan.inet.0 {
                import multipath-resolv;
            }
        }
        router-id 192.0.2.11;
        autonomous-system 64512;
        forwarding-table {
            export load-balance;
            ecmp-fast-reroute;
            indirect-next-hop;
        }
    }
    protocols {
        bgp {
            family inet {
                labeled-unicast {
                    aggregate-label {
                        community aggregates;
                    }
                    per-prefix-label;
                }
            }
            family inet6 {
                labeled-unicast {
                    aggregate-label {
                        community aggregates;
                    }
                }
            }
            group CORE {
                type internal;
                local-address 192.0.2.11;
                hold-time 30;
                family inet {
                    any;
                }
                family inet-vpn {
                    any;                    
                }
                family inet6-vpn {
                    any;
                }
                family evpn {
                    signaling;
                }
                authentication-key "****"; ## SECRET-DATA
                export export_core;
                peer-as 64512;
                multipath;
                neighbor 192.0.2.1;
                neighbor 192.0.2.2;
                neighbor 192.0.2.3;
                neighbor 192.0.2.4;
                neighbor 192.0.2.5;
            }
            group CORE6 {
                type internal;
                local-address AAAA:BBBB:0:0::11;
                hold-time 30;
                family inet6 {
                    any;
                }
                authentication-key "****"; ## SECRET-DATA
                export export6_core;
                peer-as 64512;
                neighbor AAAA:BBBB:0:0::1;
                neighbor AAAA:BBBB:0:0::2;
                neighbor AAAA:BBBB:0:0::3;
                neighbor AAAA:BBBB:0:0::4;
                neighbor AAAA:BBBB:0:0::5;
            }
            traceoptions {
                file bgp.log size 1m files 2;
            }
            log-updown;
            graceful-restart;
        }
        ldp {
            deaggregate;
            explicit-null;
            transport-address router-id;
            interface ae2.0;
            interface lo0.0;
            family {
                inet;
                inet6;
            }
            transport-preference ipv4;
        }
        mpls {
            traffic-engineering {
                bgp;
            }
            no-propagate-ttl;
            explicit-null;
            icmp-tunneling;
            no-decrement-ttl;
            no-cspf;
            ipv6-tunneling;
            interface lo0.0;
            interface ae2.0;
        }
        ospf {
            area 0.0.0.0 {
                interface lo0.0 {
                    passive;
                }
                interface ae2.0 {
                    interface-type p2p;
                    authentication {
                        simple-password "****"; ## SECRET-DATA
                    }
                }
            }                               
            reference-bandwidth 1000g;
        }
        ospf3 {
            area 0.0.0.0 {
                interface lo0.0 {
                    passive;
                }
                interface ae2.0 {
                    interface-type p2p;
                    ipsec-sa core;
                }
            }
            reference-bandwidth 1000g;
        }
        rsvp {
            interface lo0.0;
            interface ae2.0 {
                authentication-key "****"; ## SECRET-DATA
            }
        }
    }​



    Thanks,

    Thomas


  • 2.  RE: QFX10002 VXLAN ":vxlan.inet.0" and a labeled next-hop.

    Posted 28 days ago
    Just als fyi, you can limit the entries in ":vxlan.inet.0" to resolve only to inet.0 entries as workaround: 
    set routing-options resolution rib :vxlan.inet.0 resolution-ribs inet.0​