Routing

Hello all!

Thank you for reading my topic.

Our network consists of 2x MX and 2x EX9200 directly connected to eachother in full mesh. No P (MPLS only) routers are bieng used. Popping happends directly on the same device.

Our top PE routers have internet connectivity, the bottom EX9200 devices recieve 2 default routes from the PE routers. I want both the PE's to handle the internet traffic.

Our configuration did not have multipath loadbalancing enabled yet. This caused all traffic to be routed to one PE (TC5-PE01) because it has the lower router ID.

Last night i have enabled multipathing;

set routing-instances OUTSIDE-VRF routing-options multipath 

I also tried to add the vpn-unequal-cost command but this did not change anything.

After this command, some traffic that follow the default route is bieng loadbalanced  (unkown which and why) but most of the traffic still only has one active default route path. The next-hop router ID seems to ruin the party.

fyi: The other xxxMbits of traffic on the "empty connections" is shortest path EVPN traffic for a known /32 destination

I now have default routes with an @ (Routing Use Only), an > (Active Route) and an # (Forwarding use only)

username@NOS722-CORE01> show route table OUTSIDE-VRF.inet.0 0.0.0.0/0
@ = Routing Use Only, # = Forwarding Use Only
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0     @[BGP/170] 21w0d 23:57:56, localpref 100, from x.x.254.2
                      AS path: I, validation-state: unverified
                    > to x.x.254.137 via ae1.0, Push 16
                    [BGP/170] 21w0d 23:56:01, localpref 100, from x.x.254.3
                      AS path: I, validation-state: unverified
                    > to x.x.254.129 via ae5.0, Push 16
                   #[Multipath/255] 13:01:47, metric2 10
                    > to x.x.254.137 via ae1.0, Push 16
                      to x.x.254.129 via ae5.0, Push 16

I understand the function of the routing tables (routing&  forwarding table) but i don't understand it's use/reason in the current scenario.

Why is the routing path only active with 1 path@BGP/170 and why is there now a "ForwardingOnly" multipath path with a cost of 255?

What is the difference between these two and why? 

How can we make the multipath active for all traffic?

Thank you in advance for thinking with me!

Extra output:

username@MGW111-CORE01> show route table OUTSIDE-VRF.inet.0 0.0.0.0/0 extensive

0.0.0.0/0 (3 entries, 2 announced)
        State: <CalcForwarding>
TSI:
KRT in-kernel 0.0.0.0/0 -> {indirect(1048703), indirect(1048695)}
Page 0 idx 1, (group internal-vssmgw-v4 type Internal) Type 1 val 0xb77f18c (adv_entry)
   Advertised metrics:
     Flags: Nexthop Change
     Nexthop: Self
     Localpref: 100
     AS path: [1234] I
     Communities: target:100:1009
Page 0 idx 2, (group HST-FW-v4 type Internal) Type 1 val 0xc15b2a4 (adv_entry)
   Advertised metrics:
     Flags: Nexthop Change
     Nexthop: Self
     Localpref: 100
     AS path: [1234] I
     Communities: target:100:1009
Path 0.0.0.0 from x.x.254.2 Vector len 4.  Val: 1 2
        @BGP    Preference: 170/-101
                Route Distinguisher: x.x.254.2:9
                Next hop type: Indirect
                Address: 0xa762f40
                Next-hop reference count: 85
                Source: x.x8.254.2
                Next hop type: Router, Next hop index: 1672
                Next hop: x.x.254.133 via ae5.0, selected
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Session Id: 0x144
                Protocol next hop: x.x8.254.2
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Indirect next hop: 0xa7a4dd0 1048703 INH Session ID: 0x167
                State: <Secondary Active Int Ext ProtectionCand>
                Local AS: 1234 Peer AS: 1234
                Age: 38w2d 14:02:15     Metric2: 10
                Validation State: unverified
                Task: BGP_1234.x.x.254.2+179
                Announcement bits (3): 1-RT 2-BGP_RT_Background 5-Resolve tree 8
                AS path: I
                AS path: Recorded
                Communities: target:100:1009
                Import Accepted
                VPN Label: 16
                Localpref: 100
                Router ID: x.x.254.2
                Primary Routing Table bgp.l3vpn.0
                Indirect next hops: 1
                        Protocol next hop: x.x.254.2 Metric: 10
                        Label operation: Push 16
                        Label TTL action: prop-ttl
                        Load balance label: Label 16: None;
                        Indirect next hop: 0xa7a4dd0 1048703 INH Session ID: 0x167
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: x.x.254.133 via ae5.0
                                Session Id: 0x144
                        x.x.254.2/32 Originating RIB: inet.0
                          Metric: 10                      Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: x.x8.254.133 via ae5.0
         BGP    Preference: 170/-101
                Route Distinguisher: x.x.254.3:10
                Next hop type: Indirect
                Address: 0x9daf530
                Next-hop reference count: 31
                Source: x.x.254.3
                Next hop type: Router, Next hop index: 1963
                Next hop: x.x.254.125 via ae1.0, selected
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Session Id: 0x1c1
                Protocol next hop: x.x.254.3
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Indirect next hop: 0xa7a4660 1048695 INH Session ID: 0x166
                State: <Secondary NotBest Int Ext ProtectionCand>
                Inactive reason: Not Best in its group - Router ID
                Local AS: 1234 Peer AS: 1234
                Age: 25w6d 0:52:17      Metric2: 10
                Validation State: unverified
                Task: BGP_1234.x.x.254.3+179
                AS path: I
                AS path: Recorded
                Communities: target:100:1009
                Import Accepted
                VPN Label: 16
                Localpref: 100
                Router ID: x.x.254.3
                Primary Routing Table bgp.l3vpn.0
                Indirect next hops: 1
                        Protocol next hop: x.x.254.3 Metric: 10
                        Label operation: Push 16
                        Label TTL action: prop-ttl
                        Load balance label: Label 16: None;
                        Indirect next hop: 0xa7a4660 1048695 INH Session ID: 0x166
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: x.x.254.125 via ae1.0
                                Session Id: 0x1c1
                        x.x.254.3/32 Originating RIB: inet.0
                          Metric: 10                      Node path count: 1
                          Forwarding nexthops: 1
                                Nexthop: x.x.254.125 via ae1.0
        #Multipath Preference: 255
                Next hop type: Indirect
                Address: 0x97b4254
                Next-hop reference count: 2
                Next hop type: Router, Next hop index: 1672
                Next hop: x.x.254.133 via ae5.0, selected
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Session Id: 0x144
                Next hop type: Router, Next hop index: 1963
                Next hop: x.x.254.125 via ae1.0
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Session Id: 0x1c1
                Protocol next hop: x.x.254.2
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Indirect next hop: 0xa7a4dd0 1048703 INH Session ID: 0x167
                Protocol next hop: x.x.254.3
                Label operation: Push 16
                Label TTL action: prop-ttl
                Load balance label: Label 16: None;
                Indirect next hop: 0xa7a4660 1048695 INH Session ID: 0x166
                State: <ForwardingOnly Int Ext>
                Inactive reason: Forwarding use only
                Age: 13:06:16   Metric2: 10
                Validation State: unverified
                Task: RT
                Announcement bits (1): 0-KRT
                AS path: I
                AS path: Recorded
                Communities: target:100:1009

Forwarding table:

username@MGW111-CORE01> show route forwarding-table table OUTSIDE-VRF extensive
Routing table: OUTSIDE-VRF.inet [Index 16]
Internet:

Destination:  default
  Route type: user
  Route reference: 0                   Route interface-index: 0
  Multicast RPF nh index: 0
  Flags: sent to PFE
  Next-hop type: unilist               Index: 1048902  Reference: 1
  Next-hop type: indirect              Index: 1048703  Reference: 21
                                    Weight: 0x0
  Nexthop: 5c:45:27:cb:b0:ca
  Next-hop type: Push 16               Index: 1672     Reference: 2
  Load Balance Label: None
  Next-hop interface: ae5.0         Weight: 0x0
  Next-hop type: indirect              Index: 1048695  Reference: 10
                                    Weight: 0x0
  Nexthop: cc:e1:7f:ad:47:c1
  Next-hop type: Push 16               Index: 1963     Reference: 2
  Load Balance Label: None
  Next-hop interface: ae1.0         Weight: 0x0

Destination:  default
  Route type: permanent
  Route reference: 0                   Route interface-index: 0
  Multicast RPF nh index: 0
  Flags: none
  Next-hop type: reject                Index: 669      Reference: 3

Hi,

Just a quick query.

do you have per-packet load balancing configured?

in short. forwarding-only routes are used only for forwarding, they are not used by any routing-policies (redistribution)

So if i understand correctly; Forwarding-only routes only apply to packets that are bieng routed in and out? It does not apply to packets that come from a local gateway interface (irb) ? That would explain that a little traffic is bieng loadbalanced but the majority isn't because the majority comes from locally connected interfaces/gateway)

I have a loadbalancing policy enabled;

fsadmin@MGW111-CORE01> show configuration | display set | match LB
set routing-options forwarding-table export LB
set policy-options policy-statement LB then load-balance per-packet

But as you see it is configured in global config. Should i also configure it for each VRF?

I don't think it's required for every vrf but can you share " show route forwarding-table destination 0.0.0.0/0 | no-more " ?

username@MGW111-CORE01> show route forwarding-table destination 0.0.0.0/0 | no-more
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct       36     2
0.0.0.0/32         perm     0                    dscd       34     1

Routing table: __juniper_services__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    dscd      571     2
0.0.0.0/32         perm     0                    dscd      571     2

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      606     1
0.0.0.0/32         perm     0                    dscd      604     1

Routing table: HOSTING-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     1                    indr  1048703    21
                              5c:45:27:cb:b0:ca Push 16     1672     2 ae5.0
default            perm     0                    rjct      615     2
0.0.0.0/32         perm     0                    dscd      613     1

Routing table: INTERNETONLY-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      633     1
0.0.0.0/32         perm     0                    dscd      631     1

Routing table: MEDIANET-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    indr  1048607     3
                              5c:45:27:c8:3f:c0 Push 19      732     2 ae0.0
default            perm     0                    rjct      642     1
0.0.0.0/32         perm     0                    dscd      640     1

Routing table: KA-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    indr  1048609     3
                              5c:45:27:c8:3f:c0 Push 20      801     2 ae0.0
default            perm     0                    rjct      651     1
0.0.0.0/32         perm     0                    dscd      649     1

Routing table: OMR-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    indr  1048612     5
                              5c:45:27:c8:3f:c0 Push 21      902     2 ae0.0
default            perm     0                    rjct      660     1
0.0.0.0/32         perm     0                    dscd      658     1

Routing table: OUTSIDE-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    ulst  1048902     1
                                                 indr  1048703    21
                              5c:45:27:cb:b0:ca Push 16     1672     2 ae5.0
                                                 indr  1048695    10
                              cc:e1:7f:ad:47:c1 Push 16     1963     2 ae1.0
default            perm     0                    rjct      669     3
0.0.0.0/32         perm     0                    dscd      667     1

Routing table: INFRA-DMZ-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    indr  1048597     3
                              0:9:f:9:a:1e       ucst      904     4 ae25.876
default            perm     0                    rjct     2774     1
0.0.0.0/32         perm     0                    dscd     2760     1

Routing table: INFRA-SEC-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    indr  1048596     3
                              0:9:f:9:a:1e       ucst      908     5 ae25.877
default            perm     0                    rjct     3542     1
0.0.0.0/32         perm     0                    dscd     3533     1

Routing table: BACKEND-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct     3753     1
0.0.0.0/32         perm     0                    dscd     3747     1

Routing table: FRONTEND-VRF.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            user     0                    indr  1048863     3
                              0:9:f:9:a:1e       ucst     1911     4 ae25.878
default            perm     0                    rjct     3803     1
0.0.0.0/32         perm     0                    dscd     3801     1

Pretty much all VRF's get a default route leak from OUTSIDE-VRF one way or another which is in full working state. The other VRF's are not in use (fully) yet.

you have a unilist next-hop for outside vrf only. All the traffic going out (using default-route)  via outside vrf only should be load balanced ( load share to be precise ).

Can you explain when you say most of the traffic is from local irb which is default-gateway?

Is it in outside routing-instance?

There could be a chance that some flow eating high bandwidth than other as well might be causing this issue.  what is the load sharing ratio that you see between the both ae interfaces?

You can also try indexed-next-hop load balancing if per-packet loadbalancing not working properly

https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/indexed-next-hop-edit-forwarding-options.html

Hi, please provide me below configuration from your box.

#show configuration forwarding-options

regards

Vadivelan V

Thanks for the help so far!

"Can you explain when you say most of the traffic is from local irb which is default-gateway?"

The device from my output is an edge router. It serves as a gateway router for many subnets. Traffic from this subnet does not seem to be loadbalanced (uses the route-only routing entry) These subnets live in HOSTING-VRF on the same device which gets a default route from OUTSIDE-VRF.

The device also has a few BGP sessions to external networks/firewalls. Traffic from these segments appear to be loadbalanced (uses the forwarding-only routing entry)

"There could be a chance that some flow eating high bandwidth than other as well might be causing this issue.  what is the load sharing ratio that you see between the both ae interfaces?"

We don't have a lot of big flows as our traffic is 80% web/contact hosting. The Gbit's you see are hundred thousands of flows of no more then 1.5mbit a piece.
As you can see in the picture in the topic start; The majority of the traffic (4Gbit) follows the default route. The other 600Mbit is more-specific-route traffic (EVPN /32's) and/or loadbalanced traffic from the forwarding-only route entry.

"You can also try indexed-next-hop load balancing if per-packet loadbalancing not working properly"

I will look into this today and get back to you.

"Hi, please provide me below configuration from your box."

Yes i can but it does not show anything usefull. Global only has sampling & analizer options. The VRF forwarding-options are non-existent

username@MGW111-CORE01> show configuration forwarding-options
sampling {
    instance {
        sample-inline1 {
            input {
                rate 1;
                run-length 0;
            }
            family inet {
                output {
                    flow-server x.x.71.8 {
                        port 9941;
                        version-ipfix {
                            template {
                                ipv4;
                            }
                        }
                    }
                    inline-jflow {
                        source-address x.x.254.4;
                    }
                }
            }
            family inet6 {
                output {
                    flow-server x.x.71.8 {
                        port 9941;
                        version-ipfix {
                            template {
                                ipv6;
                            }
                        }
                    }
                    inline-jflow {
                        source-address x.x.254.4;
                    }
                }
            }
        }
    }
}
analyzer {
    qsightmirror1 {
        input {
            ingress {
                interface ae355.0;
            }
            egress {
                interface ae355.0;
            }
        }
        output {
            interface xe-1/2/1.0;
        }
    }
    qsightmirror2 {
        input {
            ingress {
                interface ae6.530;
                interface ae6.526;
                interface ae6.528;
                interface ae6.531;
                interface ae6.532;
                interface ae6.534;
                interface ae6.570;
                interface ae6.572;
                interface ae6.573;
                interface ae9.533;
            }
            egress {
                interface ae6.530;
                interface ae6.526;
                interface ae6.528;
                interface ae6.531;
                interface ae6.532;
                interface ae6.534;
                interface ae6.570;
                interface ae6.572;
                interface ae6.573;
                interface ae9.533;
            }
        }
        output {
            interface xe-1/2/2.0;
        }
    }
}
helpers {
    traceoptions {
        file bootp-trace;
        level all;
        flag bootp;
    }
}

username@MGW111-CORE01> show configuration routing-instances OUTSIDE-VRF forwarding-options

The issues has been fixed by adding more multipathing commands in different VRF's. The traffic flow was a little bit different then i initially anticipated.

As for the Route-Only and Forwarding-Only routes;

The route that should have been elected as active by BGP if the multipath algorithm had not been configured is still displayed so, even though locally the device will install both routes in the forwarding table when the option is enabled. After multipath is enabled, the BGP router continues to advertises only the active path to its neighbors, unless add-path is in use. Here comes the difference between routing-only and forwarding-only. The routes that are downloaded in the forwarding table and are used for load balancing locally can be seen with the "forwarding-only" sign while the active routes that are used by BGP for advertisement and for route redistribution scopes are seen with the route-only sign.
 If you would want to have both routes in the routing table and to be able to advertise them further you would need to use add-path but as far as I understand the goal was to load balance the traffic locally.
Information about add-path:
https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/multipath-edit-routing-protocols-bgp-group.html