Routing

Expand all | Collapse all

MX/EX MPLS VPN route load balancing issue

Jump to Best Answer
  • 1.  MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 06:12

    Hello all!

     

    Thank you for reading my topic.

    Our network consists of 2x MX and 2x EX9200 directly connected to eachother in full mesh. No P (MPLS only) routers are bieng used. Popping happends directly on the same device.

     

    Our top PE routers have internet connectivity, the bottom EX9200 devices recieve 2 default routes from the PE routers. I want both the PE's to handle the internet traffic.

    Our configuration did not have multipath loadbalancing enabled yet. This caused all traffic to be routed to one PE (TC5-PE01) because it has the lower router ID.

    corenetwork.png

     

    Last night i have enabled multipathing;

    set routing-instances OUTSIDE-VRF routing-options multipath 

    I also tried to add the vpn-unequal-cost command but this did not change anything.

     

    After this command, some traffic that follow the default route is bieng loadbalanced  (unkown which and why) but most of the traffic still only has one active default route path. The next-hop router ID seems to ruin the party.

    fyi: The other xxxMbits of traffic on the "empty connections" is shortest path EVPN traffic for a known /32 destination

    I now have default routes with an @ (Routing Use Only), an > (Active Route) and an # (Forwarding use only)

     

    username@NOS722-CORE01> show route table OUTSIDE-VRF.inet.0 0.0.0.0/0
    @ = Routing Use Only, # = Forwarding Use Only
    + = Active Route, - = Last Active, * = Both
    
    0.0.0.0/0     @[BGP/170] 21w0d 23:57:56, localpref 100, from x.x.254.2
                          AS path: I, validation-state: unverified
                        > to x.x.254.137 via ae1.0, Push 16
                        [BGP/170] 21w0d 23:56:01, localpref 100, from x.x.254.3
                          AS path: I, validation-state: unverified
                        > to x.x.254.129 via ae5.0, Push 16
                       #[Multipath/255] 13:01:47, metric2 10
                        > to x.x.254.137 via ae1.0, Push 16
                          to x.x.254.129 via ae5.0, Push 16
    

    I understand the function of the routing tables (routing&  forwarding table) but i don't understand it's use/reason in the current scenario.

    Why is the routing path only active with 1 path@BGP/170 and why is there now a "ForwardingOnly" multipath path with a cost of 255?

    What is the difference between these two and why? 

    How can we make the multipath active for all traffic?

     

    Thank you in advance for thinking with me!

     

    Extra output:

    username@MGW111-CORE01> show route table OUTSIDE-VRF.inet.0 0.0.0.0/0 extensive
    
    0.0.0.0/0 (3 entries, 2 announced)
            State: <CalcForwarding>
    TSI:
    KRT in-kernel 0.0.0.0/0 -> {indirect(1048703), indirect(1048695)}
    Page 0 idx 1, (group internal-vssmgw-v4 type Internal) Type 1 val 0xb77f18c (adv_entry)
       Advertised metrics:
         Flags: Nexthop Change
         Nexthop: Self
         Localpref: 100
         AS path: [1234] I
         Communities: target:100:1009
    Page 0 idx 2, (group HST-FW-v4 type Internal) Type 1 val 0xc15b2a4 (adv_entry)
       Advertised metrics:
         Flags: Nexthop Change
         Nexthop: Self
         Localpref: 100
         AS path: [1234] I
         Communities: target:100:1009
    Path 0.0.0.0 from x.x.254.2 Vector len 4.  Val: 1 2
            @BGP    Preference: 170/-101
                    Route Distinguisher: x.x.254.2:9
                    Next hop type: Indirect
                    Address: 0xa762f40
                    Next-hop reference count: 85
                    Source: x.x8.254.2
                    Next hop type: Router, Next hop index: 1672
                    Next hop: x.x.254.133 via ae5.0, selected
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Session Id: 0x144
                    Protocol next hop: x.x8.254.2
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Indirect next hop: 0xa7a4dd0 1048703 INH Session ID: 0x167
                    State: <Secondary Active Int Ext ProtectionCand>
                    Local AS: 1234 Peer AS: 1234
                    Age: 38w2d 14:02:15     Metric2: 10
                    Validation State: unverified
                    Task: BGP_1234.x.x.254.2+179
                    Announcement bits (3): 1-RT 2-BGP_RT_Background 5-Resolve tree 8
                    AS path: I
                    AS path: Recorded
                    Communities: target:100:1009
                    Import Accepted
                    VPN Label: 16
                    Localpref: 100
                    Router ID: x.x.254.2
                    Primary Routing Table bgp.l3vpn.0
                    Indirect next hops: 1
                            Protocol next hop: x.x.254.2 Metric: 10
                            Label operation: Push 16
                            Label TTL action: prop-ttl
                            Load balance label: Label 16: None;
                            Indirect next hop: 0xa7a4dd0 1048703 INH Session ID: 0x167
                            Indirect path forwarding next hops: 1
                                    Next hop type: Router
                                    Next hop: x.x.254.133 via ae5.0
                                    Session Id: 0x144
                            x.x.254.2/32 Originating RIB: inet.0
                              Metric: 10                      Node path count: 1
                              Forwarding nexthops: 1
                                    Nexthop: x.x8.254.133 via ae5.0
             BGP    Preference: 170/-101
                    Route Distinguisher: x.x.254.3:10
                    Next hop type: Indirect
                    Address: 0x9daf530
                    Next-hop reference count: 31
                    Source: x.x.254.3
                    Next hop type: Router, Next hop index: 1963
                    Next hop: x.x.254.125 via ae1.0, selected
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Session Id: 0x1c1
                    Protocol next hop: x.x.254.3
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Indirect next hop: 0xa7a4660 1048695 INH Session ID: 0x166
                    State: <Secondary NotBest Int Ext ProtectionCand>
                    Inactive reason: Not Best in its group - Router ID
                    Local AS: 1234 Peer AS: 1234
                    Age: 25w6d 0:52:17      Metric2: 10
                    Validation State: unverified
                    Task: BGP_1234.x.x.254.3+179
                    AS path: I
                    AS path: Recorded
                    Communities: target:100:1009
                    Import Accepted
                    VPN Label: 16
                    Localpref: 100
                    Router ID: x.x.254.3
                    Primary Routing Table bgp.l3vpn.0
                    Indirect next hops: 1
                            Protocol next hop: x.x.254.3 Metric: 10
                            Label operation: Push 16
                            Label TTL action: prop-ttl
                            Load balance label: Label 16: None;
                            Indirect next hop: 0xa7a4660 1048695 INH Session ID: 0x166
                            Indirect path forwarding next hops: 1
                                    Next hop type: Router
                                    Next hop: x.x.254.125 via ae1.0
                                    Session Id: 0x1c1
                            x.x.254.3/32 Originating RIB: inet.0
                              Metric: 10                      Node path count: 1
                              Forwarding nexthops: 1
                                    Nexthop: x.x.254.125 via ae1.0
            #Multipath Preference: 255
                    Next hop type: Indirect
                    Address: 0x97b4254
                    Next-hop reference count: 2
                    Next hop type: Router, Next hop index: 1672
                    Next hop: x.x.254.133 via ae5.0, selected
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Session Id: 0x144
                    Next hop type: Router, Next hop index: 1963
                    Next hop: x.x.254.125 via ae1.0
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Session Id: 0x1c1
                    Protocol next hop: x.x.254.2
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Indirect next hop: 0xa7a4dd0 1048703 INH Session ID: 0x167
                    Protocol next hop: x.x.254.3
                    Label operation: Push 16
                    Label TTL action: prop-ttl
                    Load balance label: Label 16: None;
                    Indirect next hop: 0xa7a4660 1048695 INH Session ID: 0x166
                    State: <ForwardingOnly Int Ext>
                    Inactive reason: Forwarding use only
                    Age: 13:06:16   Metric2: 10
                    Validation State: unverified
                    Task: RT
                    Announcement bits (1): 0-KRT
                    AS path: I
                    AS path: Recorded
                    Communities: target:100:1009
    
    

    Forwarding table:

    username@MGW111-CORE01> show route forwarding-table table OUTSIDE-VRF extensive
    Routing table: OUTSIDE-VRF.inet [Index 16]
    Internet:
    
    Destination:  default
      Route type: user
      Route reference: 0                   Route interface-index: 0
      Multicast RPF nh index: 0
      Flags: sent to PFE
      Next-hop type: unilist               Index: 1048902  Reference: 1
      Next-hop type: indirect              Index: 1048703  Reference: 21
                                        Weight: 0x0
      Nexthop: 5c:45:27:cb:b0:ca
      Next-hop type: Push 16               Index: 1672     Reference: 2
      Load Balance Label: None
      Next-hop interface: ae5.0         Weight: 0x0
      Next-hop type: indirect              Index: 1048695  Reference: 10
                                        Weight: 0x0
      Nexthop: cc:e1:7f:ad:47:c1
      Next-hop type: Push 16               Index: 1963     Reference: 2
      Load Balance Label: None
      Next-hop interface: ae1.0         Weight: 0x0
    
    Destination:  default
      Route type: permanent
      Route reference: 0                   Route interface-index: 0
      Multicast RPF nh index: 0
      Flags: none
      Next-hop type: reject                Index: 669      Reference: 3

     



  • 2.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 07:35

    Hi,

     

    Just a quick query.

     

    do you have per-packet load balancing configured?

     

    in short. forwarding-only routes are used only for forwarding, they are not used by any routing-policies (redistribution)



  • 3.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 07:51

    So if i understand correctly; Forwarding-only routes only apply to packets that are bieng routed in and out? It does not apply to packets that come from a local gateway interface (irb) ? That would explain that a little traffic is bieng loadbalanced but the majority isn't because the majority comes from locally connected interfaces/gateway)

     

    I have a loadbalancing policy enabled;

     

    fsadmin@MGW111-CORE01> show configuration | display set | match LB
    set routing-options forwarding-table export LB
    set policy-options policy-statement LB then load-balance per-packet

    But as you see it is configured in global config. Should i also configure it for each VRF?

     



  • 4.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 08:05

    I don't think it's required for every vrf but can you share " show route forwarding-table destination 0.0.0.0/0 | no-more " ?

     

     



  • 5.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 08:13
    username@MGW111-CORE01> show route forwarding-table destination 0.0.0.0/0 | no-more
    Routing table: default.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            perm     0                    rjct       36     2
    0.0.0.0/32         perm     0                    dscd       34     1
    
    Routing table: __juniper_services__.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            perm     0                    dscd      571     2
    0.0.0.0/32         perm     0                    dscd      571     2
    
    Routing table: __master.anon__.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            perm     0                    rjct      606     1
    0.0.0.0/32         perm     0                    dscd      604     1
    
    Routing table: HOSTING-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     1                    indr  1048703    21
                                  5c:45:27:cb:b0:ca Push 16     1672     2 ae5.0
    default            perm     0                    rjct      615     2
    0.0.0.0/32         perm     0                    dscd      613     1
    
    Routing table: INTERNETONLY-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            perm     0                    rjct      633     1
    0.0.0.0/32         perm     0                    dscd      631     1
    
    Routing table: MEDIANET-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    indr  1048607     3
                                  5c:45:27:c8:3f:c0 Push 19      732     2 ae0.0
    default            perm     0                    rjct      642     1
    0.0.0.0/32         perm     0                    dscd      640     1
    
    Routing table: KA-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    indr  1048609     3
                                  5c:45:27:c8:3f:c0 Push 20      801     2 ae0.0
    default            perm     0                    rjct      651     1
    0.0.0.0/32         perm     0                    dscd      649     1
    
    Routing table: OMR-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    indr  1048612     5
                                  5c:45:27:c8:3f:c0 Push 21      902     2 ae0.0
    default            perm     0                    rjct      660     1
    0.0.0.0/32         perm     0                    dscd      658     1
    
    Routing table: OUTSIDE-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    ulst  1048902     1
                                                     indr  1048703    21
                                  5c:45:27:cb:b0:ca Push 16     1672     2 ae5.0
                                                     indr  1048695    10
                                  cc:e1:7f:ad:47:c1 Push 16     1963     2 ae1.0
    default            perm     0                    rjct      669     3
    0.0.0.0/32         perm     0                    dscd      667     1
    
    Routing table: INFRA-DMZ-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    indr  1048597     3
                                  0:9:f:9:a:1e       ucst      904     4 ae25.876
    default            perm     0                    rjct     2774     1
    0.0.0.0/32         perm     0                    dscd     2760     1
    
    Routing table: INFRA-SEC-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    indr  1048596     3
                                  0:9:f:9:a:1e       ucst      908     5 ae25.877
    default            perm     0                    rjct     3542     1
    0.0.0.0/32         perm     0                    dscd     3533     1
    
    Routing table: BACKEND-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            perm     0                    rjct     3753     1
    0.0.0.0/32         perm     0                    dscd     3747     1
    
    Routing table: FRONTEND-VRF.inet
    Internet:
    Destination        Type RtRef Next hop           Type Index    NhRef Netif
    default            user     0                    indr  1048863     3
                                  0:9:f:9:a:1e       ucst     1911     4 ae25.878
    default            perm     0                    rjct     3803     1
    0.0.0.0/32         perm     0                    dscd     3801     1
    

    Pretty much all VRF's get a default route leak from OUTSIDE-VRF one way or another which is in full working state. The other VRF's are not in use (fully) yet.



  • 6.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 08:34

    you have a unilist next-hop for outside vrf only. All the traffic going out (using default-route)  via outside vrf only should be load balanced ( load share to be precise ).

     

    Can you explain when you say most of the traffic is from local irb which is default-gateway?

     

    Is it in outside routing-instance?

     

    There could be a chance that some flow eating high bandwidth than other as well might be causing this issue.  what is the load sharing ratio that you see between the both ae interfaces?

     



  • 7.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 08:49

    You can also try indexed-next-hop load balancing if per-packet loadbalancing not working properly

     

    https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/indexed-next-hop-edit-forwarding-options.html



  • 8.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-15-2018 09:20

    Hi, please provide me below configuration from your box.

     

    #show configuration forwarding-options

     

    regards

    Vadivelan V



  • 9.  RE: MX/EX MPLS VPN route load balancing issue

    Posted 03-16-2018 02:06

    Thanks for the help so far!

     

    "Can you explain when you say most of the traffic is from local irb which is default-gateway?"

    The device from my output is an edge router. It serves as a gateway router for many subnets. Traffic from this subnet does not seem to be loadbalanced (uses the route-only routing entry) These subnets live in HOSTING-VRF on the same device which gets a default route from OUTSIDE-VRF.

    The device also has a few BGP sessions to external networks/firewalls. Traffic from these segments appear to be loadbalanced (uses the forwarding-only routing entry)

     

    "There could be a chance that some flow eating high bandwidth than other as well might be causing this issue.  what is the load sharing ratio that you see between the both ae interfaces?"

    We don't have a lot of big flows as our traffic is 80% web/contact hosting. The Gbit's you see are hundred thousands of flows of no more then 1.5mbit a piece.
    As you can see in the picture in the topic start; The majority of the traffic (4Gbit) follows the default route. The other 600Mbit is more-specific-route traffic (EVPN /32's) and/or loadbalanced traffic from the forwarding-only route entry.

     

    "You can also try indexed-next-hop load balancing if per-packet loadbalancing not working properly"

    I will look into this today and get back to you.

     

    "Hi, please provide me below configuration from your box."

    Yes i can but it does not show anything usefull. Global only has sampling & analizer options. The VRF forwarding-options are non-existent

     

    username@MGW111-CORE01> show configuration forwarding-options
    sampling {
        instance {
            sample-inline1 {
                input {
                    rate 1;
                    run-length 0;
                }
                family inet {
                    output {
                        flow-server x.x.71.8 {
                            port 9941;
                            version-ipfix {
                                template {
                                    ipv4;
                                }
                            }
                        }
                        inline-jflow {
                            source-address x.x.254.4;
                        }
                    }
                }
                family inet6 {
                    output {
                        flow-server x.x.71.8 {
                            port 9941;
                            version-ipfix {
                                template {
                                    ipv6;
                                }
                            }
                        }
                        inline-jflow {
                            source-address x.x.254.4;
                        }
                    }
                }
            }
        }
    }
    analyzer {
        qsightmirror1 {
            input {
                ingress {
                    interface ae355.0;
                }
                egress {
                    interface ae355.0;
                }
            }
            output {
                interface xe-1/2/1.0;
            }
        }
        qsightmirror2 {
            input {
                ingress {
                    interface ae6.530;
                    interface ae6.526;
                    interface ae6.528;
                    interface ae6.531;
                    interface ae6.532;
                    interface ae6.534;
                    interface ae6.570;
                    interface ae6.572;
                    interface ae6.573;
                    interface ae9.533;
                }
                egress {
                    interface ae6.530;
                    interface ae6.526;
                    interface ae6.528;
                    interface ae6.531;
                    interface ae6.532;
                    interface ae6.534;
                    interface ae6.570;
                    interface ae6.572;
                    interface ae6.573;
                    interface ae9.533;
                }
            }
            output {
                interface xe-1/2/2.0;
            }
        }
    }
    helpers {
        traceoptions {
            file bootp-trace;
            level all;
            flag bootp;
        }
    }
    
    username@MGW111-CORE01> show configuration routing-instances OUTSIDE-VRF forwarding-options
    
    

     

     



  • 10.  RE: MX/EX MPLS VPN route load balancing issue
    Best Answer

    Posted 06-05-2018 05:45

    The issues has been fixed by adding more multipathing commands in different VRF's. The traffic flow was a little bit different then i initially anticipated.

     

    As for the Route-Only and Forwarding-Only routes;

     

    The route that should have been elected as active by BGP if the multipath algorithm had not been configured is still displayed so, even though locally the device will install both routes in the forwarding table when the option is enabled. After multipath is enabled, the BGP router continues to advertises only the active path to its neighbors, unless add-path is in use. Here comes the difference between routing-only and forwarding-only. The routes that are downloaded in the forwarding table and are used for load balancing locally can be seen with the "forwarding-only" sign while the active routes that are used by BGP for advertisement and for route redistribution scopes are seen with the route-only sign.
     If you would want to have both routes in the routing table and to be able to advertise them further you would need to use add-path but as far as I understand the goal was to load balance the traffic locally.
    Information about add-path:
    https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/multipath-edit-routing-protocols-bgp-group.html