Routing

Expand all | Collapse all

IPv4 over MPLS: P/PE Egress routing problem

  • 1.  IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-06-2020 09:04

    We run a multivendor mpls network spread across around 8 DC's. Currently I'm redesigning our "internet breakout" and I'm puzzled about some odd behavior I'm seeing (or misunderstanding the output).

    I can simulate our problem in a very simple topology with only Juniper equipment:


    Customer CPE/Router <-> QFX5100 <-> MX204

     

    - Our IGP is OSPF, we run MPLS with LDP and BGP on top
    - MPLS traffic engineering is on BGP, we configured explicit-null's
    - Routing options are configured with indirect-next-hop, we use per-prefix-labels

     

    We put the "connected" interface interco's on the MX and QFX in BGP to satisfy the route lookups with the right next-hop (OSPF is only responsible for backbone interfaces + loopbacks)
    On the MX204 we also announce an aggregate by redistributing a static 0/0 towards the network. This problem happens on the default and as well on upstream interface ranges on the MX.


    Problem:
    When sending traffic out the customer CPE towards something behind the MX we get a loop:

    X.X.X.2 = intero between CPE and QFX
    Y.Y.Y.1 = IP behind an interface on the MX (something connected, but not the MX itself)
    Z.Z.Z.1 = IP of the MX on the link between the MX/QFX

     

    thomas@503-r51b09-2> traceroute source X.X.X.2 Y.Y.Y.1 ttl 5 
    traceroute to Y.Y.Y.1 (Y.Y.Y.1) from X.X.X.2, 5 hops max, 40 byte packets
     1  X.X.X.1 (X.X.X.1)  23.820 ms  13.688 ms  23.873 ms
     2  Z.Z.Z.1 (Z.Z.Z.1)  10.475 ms  4.051 ms  1.992 ms
     3  Z.Z.Z.1 (Z.Z.Z.1)  2.674 ms  4.898 ms  3.866 ms
     4  Z.Z.Z.1 (Z.Z.Z.1)  3.412 ms  4.424 ms  1.500 ms
     5  Z.Z.Z.1 (Z.Z.Z.1)  2.679 ms  2.483 ms  2.566 ms
    


    If we do a traceroute towards Y.Y.Y.2, the IP of the MX of that interco, it works fine.

    Output's:

    1) first the qfx does do a lookup on 0/0 in inet.0 and after that inet.3 should be checked to reach our MX-LOOPBACK:

    root@qfx> show route table inet.0 0.0.0.0 
    
    inet.0: 95 destinations, 167 routes (83 active, 0 holddown, 78 hidden)e
    + = Active Route, - = Last Active, * = Both
    
    0.0.0.0/0          *[BGP/170] 1d 11:10:16, localpref 100, from MX-LOOPBACK
                          AS path: I, validation-state: unverified
                        >  to Z.Z.Z.1 via et-0/0/10.0, Push 0
    
    
    root@qfx> show route table inet.3 MX-LOOPBACK
    
    inet.3: 9 destinations, 12 routes (9 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    MX-LOOPBACK/32  *[LDP/9] 1w0d 11:32:50, metric 1
                        >  to Z.Z.Z.1 via et-0/0/10.0, Push 0
    

     

    I guess pushing 0 to the packet isn't optimal as this router is also the last hop before our MX (they are back to back).


    2)

    The packet arrives at the MX and a lookup should be done on the mpls table:

    root@mx> show route table mpls.0 label 0   
    
    mpls.0: 28 destinations, 28 routes (28 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    0                  *[MPLS/0] 6w2d 19:46:38, metric 1
                           to table inet.0
    0(S=0)             *[MPLS/0] 6w2d 19:46:38, metric 1
                           to table mpls.0


    So label 0 makes the MX look into inet.0 that one has all the right entries. 


    So in theory, if you look at the outputs this should work, but it is not? Looking at the traceroute it looks like we are stuck on the MX itself, traffic is not bouncing back towards other interfaces.

    It also seems to work for the MX IP's on the connected interfaces but not for the peer's.

     


    a) I do get that the ingress PE is also the PHP, is this the cause of the problem? Shouldn't the fowarding logic be smart enough to handle this and just send an IP packet towards the MX?

     

    b) I guess moving towards RSVP and doing tunnel-services & UHP on the MX would solve this but I do not fully understand why the current config isnt working. How are all you solving these kind of problems?

     

    Thanks,

     

    Thomas



  • 2.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-06-2020 21:01

    Hello,

    There is not enough information to analyze Your problem.

    Please post complete sanitized configurations, topology diagram and traffic flows.

    Also please post following printouts from QFX and MX:

     

    show route extensive Y.Y.Y.1 | no-more
    show route forwarding-table destination Y.Y.Y.1 | no-more
    

     

     

     


    @thomaswoidt wrote:


    - MPLS traffic engineering is on BGP, we configured explicit-null's

    <skip>

    I guess pushing 0 to the packet isn't optimal as this router is also the last hop before our MX (they are back to back).

    <skip>

    Shouldn't the fowarding logic be smart enough to handle this and just send an IP packet towards the MX?

     

     

    This is expected if You configure explicit-null. And no, forwarding logic is doing what You instructed it to do.

     

    HTH

    Thx

    Alex



  • 3.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-07-2020 04:14
    .... wrong, see below.

    I'll work on full configurations.



  • 4.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-07-2020 04:39

    Hello,

     

     


    @thomaswoidt wrote:


    Y.Y.Y.1 = IP behind an interface on the MX (something connected, but not the MX itself)

     

    and

     

     


    @thomaswoidt wrote:
    root@mx> show route Y.Y.Y.1 extensive table inet.0 | no-more
    
    inet.0: 802801 destinations, 3192207 routes (802801 active, 0 holddown, 0 hidden)
    Y.Y.Y.1/32 (1 entry, 1 announced)
            State: <FlashAll>
            *Local  Preference: 0
                    Next hop type: Local, Next hop index: 0
    

     

    so, Y.Y.Y.1 - which one is this?  Local IP on MX, or link IP on another box directly connected to MX?

     

    HTH

    Thx

    Alex

     



  • 5.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-07-2020 06:19

    Small mistake on the Y.Y.Y.1 (bad thing of not having just a simple lab atm, I also modified the QFX config here).

     

    These are the right outputs (so the IP is behind the MX on a connected interface, I'll try to remove the others):

    root@mx> show route Y.Y.Y.1 extensive table inet.0 | no-more                                         
    
    inet.0: 802732 destinations, 3189419 routes (802732 active, 0 holddown, 0 hidden)
    Y.Y.Y.0/30 (1 entry, 1 announced)
            State: <FlashAll>
    TSI:
    Page 0 idx 1, (group CORE type Internal) Type 1 val 0xe753e28 (adv_entry)
       Advertised metrics:
         Flags: Nexthop Change
         Nexthop: Self
         Localpref: 100
         AS path: [65000] I
         Communities: 65000:2
        Advertise: 0000001f
    Path Y.Y.Y.0
    Vector len 4.  Val: 0 1
            *Direct Preference: 0
                    Next hop type: Interface, Next hop index: 0
                    Address: 0x5221b7c
                    Next-hop reference count: 1
                    Next hop: via xe-0/1/0.109, selected
                    State: <Active Int>
                    Local AS: 65000 
                    Age: 5d 2:01:10 
                    Validation State: unverified 
                    Task: IF
                    Announcement bits (4): 2-LDP 7-BGP_RT_Background 8-Resolve tree 6 9-Resolve tree 8 
                    AS path: I 
    
    
    
    root@qfx> show route Y.Y.Y.1 extensive table inet.0 
    
    inet.0: 95 destinations, 167 routes (83 active, 0 holddown, 78 hidden)
    Y.Y.Y.0/30 (1 entry, 1 announced)
    TSI:
    KRT in-kernel Y.Y.Y.0/30 -> {indirect(131093)}
    Page 0 idx 1, (group BORDER type Internal) Type 1 val 0xeb4ecfc (adv_entry)
       Advertised metrics:
         Nexthop: MX-LOOPBACK
         Localpref: 100
         AS path: [65000] I
         Communities: 65000:2
         Cluster ID: CLUSTER-ID
         Originator ID: MX-LOOPBACK
        Advertise: 00000003
    Path Y.Y.Y.0
    from MX-LOOPBACK
    Vector len 4.  Val: 1 2 3
            *BGP    Preference: 170/-101
                    Next hop type: Indirect, Next hop index: 0
                    Address: 0xc664220
                    Next-hop reference count: 14
                    Source: MX-LOOPBACK
                    Next hop type: Router, Next hop index: 2087
                    Next hop: Z.Z.Z.1 via et-0/0/10.0, selected
                    Label operation: Push 0
                    Label TTL action: prop-ttl
                    Load balance label: Label 0: None; 
                    Label element ptr: 0xbcc4458
                    Label parent element ptr: 0x0
                    Label element references: 4
                    Label element child references: 3
                    Label element lsp id: 0
                    Session Id: 0x0
                    Protocol next hop: MX-LOOPBACK
                    Indirect next hop: 0xce1dc04 131093 INH Session ID: 0x0
                    State: <Active Int Ext>
                    Local AS: 65000 Peer AS: 65000
                    Age: 2d 8:36:27         Metric2: 1 
                    Validation State: unverified 
                    Task: BGP_65000.MX-LOOPBACK
                    Announcement bits (4): 0-KRT 6-BGP_RT_Background 7-Resolve tree 4 8-Resolve tree 5 
                    AS path: I 
                    Communities: 65000:2
                    Accepted
                    Localpref: 100
                    Router ID: MX-LOOPBACK
                    Indirect next hops: 1
                            Protocol next hop: MX-LOOPBACK Metric: 1
                            Indirect next hop: 0xce1dc04 131093 INH Session ID: 0x0
                            Indirect path forwarding next hops: 1
                                    Next hop type: Router
                                    Next hop: Z.Z.Z.1 via et-0/0/10.0
                                    Session Id: 0x0
                                    MX-LOOPBACK/32 Originating RIB: inet.3
                                      Metric: 1     Node path count: 1
                                      Forwarding nexthops: 1
                                            Nexthop: Z.Z.Z.1 via et-0/0/10.0
                                            Session Id: 0
    

    And the forwarding tables:

    root@mx> show route forwarding-table destination Y.Y.Y.1 table default | no-more 
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    Y.Y.Y.1/32 dest     1 0:24:dc:ce:3e:74   ucst      828 241368 xe-0/1/0.109
    
    
    root@qfx> show route forwarding-table destination Y.Y.Y.1 table default | no-more 
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    Y.Y.Y.0/30 user     0                    indr   131093     7
                                  Z.Z.Z.1   Push 0     2087     3 et-0/0/10.0
    


  • 6.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-07-2020 05:08

    This the config for the QFX. The relevant pieces towards the MX's is called BORDER. I tried to only keep relevant pieces of config (we also do EVPN/VXLAN over loopback/MPLS and L3VPN's here).

     

    The BORDER peering group has 3 MX's, spread around some DC's. The problem only exists when the local connected MX is used. Connecting the CPE to a QFX without a local MX it all works fine (you'll see a swap 0 for the ingress label instead of push 0).

     

    Filtering is strict as we have full tables on MX and only limited routes on QFX (aggregation network).

     

     

    root@qfx> show configuration               
    ## Last commit: 2020-06-05 06:31:12 CEST by root
    ## Image name: jinstall-host-qfx-5-19.4R1.10-signed.tgz
    
    version 20191212.201431_builder.r1074901;
    system {
        host-name qfx;
        root-authentication {
            encrypted-password ****; ## SECRET-DATA
        }
        services {
            ssh {
                root-login allow;
            }
        }
        time-zone Europe/Brussels;
        ntp {
            server ****;
            server ****;
        }
    }
    chassis {
        aggregated-devices {
            ethernet {
                device-count 30;
            }
        }
        fpc 0 {
            pic 0 {
                ....
                port 16 {
                    channel-speed 10g;
                }
                ....
            }
        }
    }
    security {
        authentication-key-chains {
            key-chain upstream-bfd {                   
                key 0 {
                    secret ****; ## SECRET-DATA
                    start-time "1970-1-1.01:00:01 +0100";
                }
            }
        }
        ipsec {
            security-association core {
                mode transport;
                manual {
                    direction bidirectional {
                        protocol ah;
                        spi 256;
                        authentication {
                            algorithm hmac-sha1-96;
                            key ascii-text ****; ## SECRET-DATA
                        }
                    }
                }
            }
        }
    }
    interfaces {
        ....
        et-0/0/10 {
            description "Link to MX et-0/0/2";
            mtu 9216;
            unit 0 {
                family inet {
                    address Z.Z.Z.2/30;
                }
                family inet6 {
                    address ..../127;
                }
                family mpls;
            }
        }
        ....
        xe-0/0/16:0 {
            description "Link to CPE";
            unit 0 {
                family inet {
                    address X.X.X.1/30;
                }
                family inet6 {
                    address ..../127;
                }
            }
        }
        ....
        lo0 {
            unit 0 {
                family inet {
                    filter {
                        input protect-me;
                    }
                    address QFX-LOOPBACK/32 {
                        primary;
                    }
                }
                family inet6 {
                    address ..../128 {
                        primary;
                    }
                }
            }
        }
    }
    
    policy-options {
        prefix-list interco_list {
            X.X.X.0/30;
        }
        prefix-list interco6_list {
            ..../127;
        }
    
        ....
    
        policy-statement export_border {
            term customers {
                from {
                    protocol bgp;
                    community customers;
                }
                then accept;
            }
            term loopbacks {
                from {
                    protocol bgp;
                    community customers_loopbacks;
                }
                then accept;
            }
            term underlay {
                from {
                    protocol bgp;
                    community [ interco statics ];
                }
                then accept;
            }
            term connected {
                from {
                    protocol direct;
                    prefix-list interco_list;
                }
                then {
                    community set interco;
                    next-hop self;
                    accept;
                }                           
            }
            term static {
                from {
                    protocol static;
                    tag 10;
                }
                then {
                    community set statics;
                    accept;
                }
            }
            then reject;
        }
    
        policy-statement import_border {
            term underlay {                 
                from {
                    protocol bgp;
                    community [ interco statics ];
                }
                then accept;
            }
            term aggregate {
                from {
                    protocol bgp;
                    community aggregates;
                }
                then accept;
            }
            then reject;
        }
    
        ....
    
        policy-statement load-balance {
            then {
                load-balance per-packet;
            }
        }
    
        ....
    
        community aggregates members 65000:5;
        community all members *:*;
    
        community customers members 65000:50000;
        community customers_blackhole members 65000:5666;
        community customers_loopbacks members 65000:59999;
    
        community interco members 65000:2;
        community rtbh members 65000:666;
        community statics members 65000:10;
    }
    firewall {
        family inet {
            filter protect-me {
    ...
            }
        }
    ....
    }
    
    
    routing-instances {
    ....
    }
    
    routing-options {
        forwarding-table {
            export load-balance;
            ecmp-fast-reroute;
            indirect-next-hop;
        }
        router-id QFX-LOOPBACK;
        autonomous-system 65000;
    }
    
    
    protocols {
    
        ospf {
            area 0.0.0.0 {
                interface lo0.0 {
                    passive;
                }
                interface et-0/0/10.0 {
                    interface-type p2p;
                    authentication {
                        simple-password ****; ## SECRET-DATA
                    }
                    bfd-liveness-detection {
                        minimum-interval 1500;
                        multiplier 3;
                    }
                }
            }
            reference-bandwidth 1000g;
        }
    
        rsvp {
            interface lo0.0;
            interface et-0/0/10.0 {
                authentication-key ****; ## SECRET-DATA
            }
        }
    
        bgp {
            family inet {
                labeled-unicast {
                    aggregate-label {
                        community aggregates;
                    }
                    per-prefix-label;
                }
            }
            family inet6 {
                labeled-unicast {
                    aggregate-label {
                        community aggregates;
                    }
                }
            }
    
            group BORDER {
                local-address QFX-LOOPBACK;
                hold-time 30;
                import import_border;
                family inet {
                    any;
                }
                family inet-vpn {
                    any;
                }
                authentication-key ****; ## SECRET-DATA
                export export_border;
                cluster CLUSTER-ID;
                peer-as 65000;
                neighbor MX-1;
                neighbor MX-2;
                neighbor MX-3;
            }
            group BORDER6 {
                local-address QFX-LOOPBACK;
                hold-time 30;
                import import6_border;
                family inet6 {
                    any;
                }
                authentication-key ****; ## SECRET-DATA
                export export6_border;
                cluster CLUSTER-ID;
                peer-as 65000;
                neighbor MX-1;
                neighbor MX-2;
                neighbor MX-3;
            }
    
    ....
    
            log-updown;
            graceful-restart;
        }
    
        ldp {
            deaggregate;
            explicit-null;
            transport-address router-id;
            interface et-0/0/10.0;
            interface lo0.0;
            family {
                inet;
                inet6;
            }
            transport-preference ipv4;
        }
    
        mpls {
            traffic-engineering {
                bgp;
            }
            explicit-null;
            icmp-tunneling;
            no-decrement-ttl;
            no-cspf;
            ipv6-tunneling;
    
            interface lo0.0;
            interface et-0/0/10.0;
    
        }
    
    ....
    
        ospf3 {
            area 0.0.0.0 {
                interface lo0.0 {
                    passive;
                }
    
                interface et-0/0/10.0 {
                    interface-type p2p;
                    ipsec-sa core;
                    bfd-liveness-detection {
                        minimum-interval 1500;
                        multiplier 3;
                    }
                }
    
            }
            reference-bandwidth 1000g;
        }
    
    
    }
    
    {master:0}
    root@qfx> 


  • 7.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-07-2020 05:56
    root@mx> show configuration 
    ## Last commit: 2020-06-05 06:19:04 CEST by root
    version 20191212.201431_builder.r1074901;
    system {
        host-name mx;
        root-authentication {
            encrypted-password ****; ## SECRET-DATA
        }
        services {
            ssh {
                root-login allow;
                sftp-server;
            }
        }
        time-zone Europe/Brussels;
        ntp {
            server ****;
            server ****;
        }
    }
    chassis {
        fpc 0 {
            pic 0 {
                port 2 {
                    speed 40g;
                }
            }
            pic 1 {
                port 0 {
                    speed 10g;
                }
            }
        }
        alarm {
            management-ethernet {
                link-down ignore;
            }
        }
        network-services enhanced-ip;
    }
    security {                              
        authentication-key-chains {         
            key-chain upstream-bfd {        
                key 0 {                     
                    secret ****; ## SECRET-DATA
                    start-time "1970-1-1.01:00:01 +0100";
                }                           
            }                               
        }                                   
        ipsec {                             
            security-association core {     
                mode transport;             
                manual {                    
                    direction bidirectional {
                        protocol ah;        
                        spi 256;            
                        authentication {    
                            algorithm hmac-sha1-96;
                            key ascii-text ****; ## SECRET-DATA
                        }                   
                    }                       
                }                           
            }                               
        }                                   
    }    
    interfaces {                            
        et-0/0/2 {                          
            description "Link to QFX et-0/0/10";
            mtu 9216;                       
            unit 0 {                        
                family inet {               
                    address Z.Z.Z.1/30;
                }                           
                family inet6 {              
                    address ****/127;
                }                           
                family mpls;                
            }                               
        }                        
    ....
        xe-0/1/0 {                          
            flexible-vlan-tagging;          
            mtu 9216;         
            unit 109 {                      
                vlan-id 109;                
                family inet {               
                    mtu 1500;               
                    address Y.Y.Y.2/30;
                }                           
                family inet6 {              
                    mtu 1500;               
                    address ****/126;
                }                           
            }   
    ....
        lo0 {
            unit 0 {
                family inet {
                    address MX-LOOPBACK/32 {
                        primary;
                        preferred;
                    }
                }
                family inet6 {
                    address ****/128 {
                        primary;
                    }
                }
            }
        }
    }
    policy-options {   
        prefix-list interco_list {                       
             Y.Y.Y.0/30;             
        }                                   
        prefix-list interco6_list {         
    ....          
        }         
    
        policy-statement export6_core {     
            term aggregate {                
                from {                      
                    protocol static;        
                    tag 5;                  
                }                           
                then {                      
                    community set aggregates;
                    next-hop self;          
                    accept;                 
                }                           
            }                               
            then reject;                    
        }
    
        policy-statement export_core {
            term connected {
                from {
                    protocol direct;
                    prefix-list interco_list;
                }
                then {
                    community set interco;
                    next-hop self;
                    accept;
                }
            }
            term aggregate {
                from {
                    protocol static;
                    tag 5;
                }
                then {
                    community set aggregates;
                    next-hop self;
                    accept;
                }
            }
            term static {
                from {
                    protocol static;
                    tag 10;
                }
                then {
                    community set statics;
                    accept;
                }                           
            }                               
            term customers {                
                from {                      
                    protocol bgp;           
                    community customers;    
                }                           
                then accept;                
            }                               
            then reject;                    
        }  
    
        policy-statement import6_core {     
            term customers {                
                from {                      
                    protocol bgp;           
                    community customers;    
                }                           
                then accept;                
            }                               
            term loopbacks {                
                from {                      
                    protocol bgp;           
                    community customers_loopbacks;
                }                           
                then accept;                
            }                               
            term connected {                
                from {                      
                    protocol bgp;           
                    community interco;      
                }                           
                then accept;                
            }                               
            term static {                   
                from {                      
                    protocol bgp;           
                    community statics;      
                }                           
                then accept;                
            }                               
            then reject;                    
        }                              
    
        policy-statement import_core {      
            term customers {                
                from {                      
                    protocol bgp;           
                    community customers;    
                }                           
                then accept;                
            }                               
            term loopbacks {                
                from {                      
                    protocol bgp;           
                    community customers_loopbacks;
                }                           
                then accept;                
            }                               
            term connected {                
                from {                      
                    protocol bgp;           
                    community interco;      
                }                           
                then accept;                
            }                               
            term static {                   
                from {                      
                    protocol bgp;           
                    community statics;      
                }                           
                then accept;                
            }                               
            then reject;                    
        }                            
    
        community aggregates members 65000:5;
        community all members *:*;          
        community customers members 65000:50000;
        community customers_blackhole members 65000:5666;
        community customers_loopbacks members 65000:59999;
        community interco members 65000:2;  
        community rtbh members 65000:666;   
        community statics members 65000:10; 
        as-path private 64512-65535;   
    
         
    }                                       
    
    firewall {                              
        family inet {                       
            filter protect-me { 
    ....
            }                               
        }                                   
    }  
    
    
    
    protocols {
        ospf {
            area 0.0.0.0 {
                interface lo0.0 {
                    passive;
                }
                interface et-0/0/2.0 {
                    interface-type p2p;
                    authentication {
                        simple-password ****; ## SECRET-DATA
                    }
                    bfd-liveness-detection {
                        minimum-interval 1500;
                        multiplier 3;
                    }
                }
            }                               
            reference-bandwidth 1000g;      
        }   
         
        rsvp {                              
            interface lo0.0;                
            interface et-0/0/2.0 {          
                authentication-key ****; ## SECRET-DATA
            }    
    
        bgp {                               
            family inet {                   
                labeled-unicast {           
                    aggregate-label {       
                        community aggregates;
                    }                       
                    per-prefix-label;       
                }                           
            }                               
            family inet6 {                  
                labeled-unicast {           
                    aggregate-label {       
                        community aggregates;
                    }                       
                }                           
            }   
                             
            group CORE {                    
                local-address MX-LOOPBACK;
                hold-time 30;               
                import import_core;         
                family inet {               
                    any;                    
                }                           
                family inet-vpn {           
                    any;                    
                }                           
                family inet6-vpn {          
                    any;                    
                }                           
                authentication-key ****; ## SECRET-DATA
                export export_core;         
                cluster MX-CLUSTERID-1; 
                peer-as 65000;              
                neighbor QFX-LOOPBACK;
                ....
            }   
                                
            group CORE6 {                   
                local-address MX-LOOPBACK;
                hold-time 30;               
                import import6_core;        
                family inet6 {              
                    any;                    
                }                           
                authentication-key ****; ## SECRET-DATA
                export export6_core;        
                cluster MX-CLUSTERID-1;     
                peer-as 65000;              
                neighbor QFX-LOOPBACK;
                .... 
            }
                                   
            log-updown;                     
            graceful-restart;               
        } 
    
     
        ldp {                               
            deaggregate;                    
            explicit-null;                  
            transport-address router-id;    
            interface et-0/0/2.0;           
    
            interface lo0.0;                
            family {                        
                inet;                       
                inet6;                      
            }                               
            transport-preference ipv4;      
        }                                   
        mpls {                              
            traffic-engineering {           
                bgp;                        
            }                               
            explicit-null;                  
            icmp-tunneling;                 
    
            no-decrement-ttl;               
            no-cspf;                        
            ipv6-tunneling;                 
            interface lo0.0;                
            interface et-0/0/2.0;           
    
        }      
    
        ospf3 {                             
            area 0.0.0.0 {                  
                interface lo0.0 {           
                    passive;                
                }                           
                interface et-0/0/2.0 {      
                    interface-type p2p;     
                    ipsec-sa core;          
                    bfd-liveness-detection {
                        minimum-interval 1500;
                        multiplier 3;       
                    }                       
                }                           
          
            }                               
            reference-bandwidth 1000g;      
        }                                   
    }                      
                 
    


  • 8.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-07-2020 07:18

    Hello,

    Thanks for providing the configurations.

    First findings:

    1/ on the forward path, the traffic does not take Your 0/0 route on QFX, it takes a specific Y.Y.Y.0/30 route

    2/ I see no issues with encapsulation or forward routing on either QFX or MX

     

    Let's examine the return path.

    Please provide the following printouts from QFX and MX:

     

    show route extensive X.X.X.2 | no-more
    show route forwarding-table destination X.X.X.2 | no-more

    Also, do You have NAT configured on MX (inline or regular MS-DPC|MS-MPC NAT)?

    HTH

    Thx
    Alex

     

     



  • 9.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-07-2020 09:09

    1/ True, in this case. As I'm trying to simulate the most simple scenario where this issue occurs .

     

    Example with the default route looks like this:

    root@qfx> show route forwarding-table destination 8.8.4.4 table default | no-more 
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    indr   131093     7
                                  40:de:ad:cb:20:27 Push 0     2087     3 et-0/0/10.0
    default            perm     0                    rjct       51     1
    
    
    
    root@mx> show route forwarding-table destination 8.8.4.4 table default | no-more    
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    8.8.4.0/24         user     0 SOME-TRANSIT-PROVIDER     ucst      869 425690 xe-0/1/0.101
    
    

    2/ I've been looking at this for some days and cant figure out what I am missing, we do plenty of other stuff on mpls/l3vpn/evpn/vxlan/... that is just working fine.

     

    This label is pushed at the DC next to it and just works fine:

    root@qfx-other> show route table inet.0 0.0.0.0 
    
    inet.0: 91 destinations, 119 routes (66 active, 0 holddown, 49 hidden)
    + = Active Route, - = Last Active, * = Both
    
    0.0.0.0/0          *[BGP/170] 4d 14:02:00, localpref 100, from MX-LOOPBACK
                          AS path: I, validation-state: unverified
                        >  to P2P-QFX via xe-0/0/12:3.0, Push 53
    
    
    root@qfx> show route table mpls.0 label 53  
    
    mpls.0: 41 destinations, 41 routes (41 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    53                 *[LDP/9] 1w1d 11:36:31, metric 1
                        >  to Z.Z.Z.1 via et-0/0/10.0, Swap 0
    53(S=0)            *[LDP/9] 1w1d 11:36:31, metric 1
                        >  to Z.Z.Z.1 via et-0/0/10.0, Pop      
    

    Return traffic is currently going over this qfx-other (and is working fine). The CPE is connected to both, On the CPE I'm currently pushing all traffic out the other DC and have more specifics for the test-ip's towards the qfx on the "broken route".



  • 10.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-08-2020 00:47

    The route back (I checked and the problem also occurs if I do it with the source of the interco interface):

     

    root@mx> show route forwarding-table destination X.X.X.2 table default 
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
     X.X.X.2/30  user     0                    indr  1048578     3
                                  Z.Z.Z.2   Push 0      607     3 et-0/0/2.0
    
    
    root@qfx> show route forwarding-table destination X.X.X.2 table default 
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    X.X.X.2/32  dest     1 3c:8a:b0:d3:cc:e7  ucst     2141     2 xe-0/0/16:0.0
    
    

    It all looks perfectly fine for me :(. If you do traceroutes on the qfx or the mx with as source the same interfaces (from the RE), they also work.

     



  • 11.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-08-2020 02:17

    I went through a couple of scenarios again:

    1) QFX and QFX-OTHER are back to back like the QFX-MX. if we do the same thing here (communicate over connected interco's) the issue is not present, so pushing and pop'ing 0 on QFX is working fine.

     

    2) If the traceroute is started on something behind an interface on the MX it does work fine too, so packets arriving on the MX get label 0 pushed and this is understood by the QFX (and pop'ed before going to the cpe)

     

    3) If we start traceroute on the CPE it's broken:

    - Pushing 0 here needs to be validated (but we might conclude it works based on the QFX <-> QFX-OTHER test)

    - Lookup on MX for ingress traffic with label 0 needs more investigation 



  • 12.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-08-2020 03:22

    Hello,

     


    @thomaswoidt wrote:

     

    3) If we start traceroute on the CPE it's broken:

    - Pushing 0 here needs to be validated (but we might conclude it works based on the QFX <-> QFX-OTHER test)

    - Lookup on MX for ingress traffic with label 0 needs more investigation 



    Please check if You have any missing prefixes/labels NOT installed in MX HW

     

     

    show krt queue | no-more

     

     

     

    HTH

    Thx

    Alex

     

     

     



  • 13.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-08-2020 05:23

    I was doing some hw filtering today to confirm more things. If I understand the flexible-match filter correctly this should take all label's 0 from mpls and will look further for the source-addresss of our CPE's (ipv4 header).

     

     

    root@mx> show configuration firewall family mpls | display set 
    set firewall family mpls filter in-count interface-specific
    set firewall family mpls filter in-count term null from ip-version ipv4 protocol icmp
    set firewall family mpls filter in-count term null from ip-version ipv4 source-address X.X.X.2/32
    set firewall family mpls filter in-count term null from flexible-match-range match-start layer-3
    set firewall family mpls filter in-count term null from flexible-match-range byte-offset 0
    set firewall family mpls filter in-count term null from flexible-match-range bit-length 20
    set firewall family mpls filter in-count term null from flexible-match-range range 0
    set firewall family mpls filter in-count term null then count nullpkt
    set firewall family mpls filter in-count term null then accept
    set firewall family mpls filter in-count term accept then accept
    
    root@mx> 
    

     

    Output of our filter matches the amount of pings send from the CPE:

    root@mx> show firewall |match et-0/0/2                                               
    
    Filter: in-count-et-0/0/2.0-i                                  
    nullpkt-et-0/0/2.0-i                                    0                    0
    
    
    thomas@503-r51b09-2> ping source X.X.X.2 Y.Y.Y.2 count 1 wait 1    
    PING Y.Y.Y.2 (Y.Y.Y.2): 56 data bytes
    36 bytes from Z.Z.Z.1: Time to live exceeded
    Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
     4  5  00 0054 a47b   0 0000  01  01 ed90 X.X.X.2  Y.Y.Y.2 
    
    
    root@mx> show firewall |match et-0/0/2    
    
    Filter: in-count-et-0/0/2.0-i                                  
    nullpkt-et-0/0/2.0-i                                   88                    1
    
    
    thomas@503-r51b09-2> ping source X.X.X.2 Y.Y.Y.1 count 1 wait 1    
    PING Y.Y.Y.1 (Y.Y.Y.1): 56 data bytes
    64 bytes from Y.Y.Y.1: icmp_seq=0 ttl=63 time=1.448 ms
    
    
    root@mx> show firewall |match et-0/0/2                                               
    
    Filter: in-count-et-0/0/2.0-i                                  
    nullpkt-et-0/0/2.0-i                                  176                    2
    

     

    Output:

    root@mx> show krt queue | no-more 
    Routing table add queue: 0 queued
    Interface add/delete/change queue: 0 queued
    Top-priority deletion queue: 0 queued
    Top-priority change queue: 0 queued
    Top-priority add queue: 0 queued
    high priority V4oV6 tcnh delete queue: 0 queued
    high prioriy anchor gencfg delete queue: 0 queued
    High-priority multicast add/change: 0 queued
    Indirect next hop top priority add/change: 0 queued
    Indirect next hop add/change: 0 queued
    high prioriy anchor gencfg add-change queue: 0 queued
    MPLS add queue: 0 queued
    Indirect next hop delete: 0 queued
    High-priority deletion queue: 0 queued
    MPLS change queue: 0 queued
    High-priority change queue: 0 queued
    High-priority add queue: 0 queued
    Normal-priority indirect next hop queue: 0 queued
    Normal-priority deletion queue: 0 queued
    Normal-priority composite next hop deletion queue: 0 queued
    Low prioriy Statistics-id-group deletion queue: 0 queued
    Normal-priority change queue: 0 queued
    Normal-priority add queue: 0 queued
    Least-priority delete queue: 0 queued
    Least-priority change queue: 0 queued
    Least-priority add queue: 0 queued
    Normal-priority pfe table nexthop queue: 0 queued
    EVPN gencfg queue: 0 queued
    Normal-priority gmp queue: 0 queued
    Routing table delete queue: 0 queued
    Low priority route retry queue: 0 queued
    
    root@mx> 
    

     



  • 14.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-08-2020 06:24

    Hello,

    Ok cool so You know how to use the JUNOS flex-offset filter Smiley Happy

    Could You please check if this packet has BOS bit set?

    Also, if would be good to mirror the whole packet to see if there isn't any artefact introduced by CPE.

    HTH

    Thx

    Alex



  • 15.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-08-2020 07:53

    So what I did was create a poor-man's tap by adding a sample action to the firewall filter and use the inline ipfix to send it to some tcpdump host:

     

    root@mx# show | compare 
    [edit chassis fpc 0]
    -    sampling-instance flowlogs;
    [edit services]
    -   flow-monitoring {
    -       version-ipfix {
    -           template mplsflows {
    -               flow-active-timeout 15;
    -               flow-inactive-timeout 15;
    -               mpls-template;
    -           }
    -       }
    -   }
    [edit]
    -  forwarding-options {
    -      sampling {
    -          instance {
    -              flowlogs {
    -                  input {
    -                      rate 1;
    -                  }
    -                  family mpls {
    -                      output {
    -                          flow-server **** {
    -                              port 9011;
    -                              version-ipfix {
    -                                  template {
    -                                      mplsflows;
    -                                  }
    -                              }
    -                          }
    -                          inline-jflow {
    -                              source-address MX-LOOPBACK;
    -                              flow-export-rate 200;
    -                          }
    -                      }
    -                  }
    -              }
    -          }
    -      }
    -  }
    [edit firewall family mpls filter in-count term null then]
    -       sample;
    
    [edit]
    root@mx# 

    Wireshark can decode the frames (you also need to wait till it receives the template). Screenshot is attached. I'll do the same exercise on the working path to validate the output:

    Screenshot from 2020-06-08 16-45-13.png



  • 16.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-09-2020 13:39

    Just as small update:

    - it was impossible to find the packet from the other cpe (connected to qfx-other) when using mpls firewall filters. After adding it as plain family inet filter do get hits.

     

     

    root@mx# show firewall | display set | match count 
    set firewall family inet filter ip-count interface-specific
    set firewall family inet filter ip-count term null from destination-address */32
    set firewall family inet filter ip-count term null then count inetpkt
    set firewall family inet filter ip-count term null then accept
    set firewall family inet filter ip-count term accept then accept
    ...
    set firewall family mpls filter in-count interface-specific
    set firewall family mpls filter in-count term null from ip-version ipv4 protocol icmp
    set firewall family mpls filter in-count term null from ip-version ipv4 destination-address */32
    set firewall family mpls filter in-count term null from flexible-match-range match-start layer-3
    set firewall family mpls filter in-count term null from flexible-match-range byte-offset 0
    set firewall family mpls filter in-count term null from flexible-match-range bit-length 20
    set firewall family mpls filter in-count term null from flexible-match-range range 0
    set firewall family mpls filter in-count term null then count nullpkt
    set firewall family mpls filter in-count term null then accept
    set firewall family mpls filter in-count term accept then accept
    

     

     

    Output after 1 ping from each CPE:

     

    root@mx# run show firewall 
    
    Filter: __default_bpdu_filter__                                
    
    Filter: protect-me                                             
    
    Filter: in-count-et-0/0/2.0-i                                  
    Counters:
    Name                                                Bytes              Packets
    nullpkt-et-0/0/2.0-i                                   88                    1
    
    Filter: ip-count-et-0/0/2.0-i                                  
    Counters:
    Name                                                Bytes              Packets
    inetpkt-et-0/0/2.0-i                                   84                    1
    

     

     

    - it seems that the QFX connected to the MX is actually doing 2 different things:

    1. for the direct connected cpe it is pushing 0 to the packet (expl null option), this is to be expected

    2. for the other cpe, the packet is arriving at qfx-other, gets label 53, is being transmitted to qfx, here the output show a 53-swap-0 but in reality seems the mpls label is pop'ed and we just get an ip packet at the MX

     

     

    So it looks like we might have 2 independent issues:

    - QFX is doing some labeling wrong towards MX (not pushing the 0 for label swap's)

    - The MX works fine if it gets a normal IP packet, but has issues with label 0 (even if here we should get a normal lookup in inet.0)



  • 17.  RE: IPv4 over MPLS: P/PE Egress routing problem

    Posted 06-10-2020 03:42

    Besides the debugging, following fix actually works like expected (UHP):

     

    QFX:

    root@qfx> show configuration protocols mpls label-switched-path qfx_to_mx 
    to MX-LOOPBACK;
    ultimate-hop-popping;
    
    root@qfx> show rsvp session name qfx_to_mx extensive | match "Label out"
      Resv style: 1 FF, Label in: -, Label out: 80
    

    On the MX:

    root@mx> show route label 80 table mpls.0       
    
    mpls.0: 32 destinations, 32 routes (32 active, 0 holddown, 0 hidden)
    + = Active Route, - = Last Active, * = Both
    
    80                 *[RSVP/0] 00:01:54
                           to table inet.0, label-switched-path qfx_to_mx
    80(S=0)            *[RSVP/0] 00:01:54
                           to table mpls.0
    
    
    

    Doesn't explain the weirdness but is an acceptable workaround. I think the MX is even doing multiple label lookups at ingress so it's not using tunnel-services for recirculation (even those can do high capacity on the mx204 platform).