Switching

We are experiencing a very odd issue with our QFX5100 switching and the routing tables.

Issue:
Ubuntu Test Server (96.126.81.60) making any TCP or UDP connection to off-net services is unable to connect.

After spending a few weeks trying to identify the issue, we found the following to be happening:

1) When no default route is populated on the QFX5100 switch, and only the full BGP table, the Loopback0.0 filter is being applied to all transit traffic.

2) When a default route is populated on the QFX5100 switch, and a full BGP table, only the default route is utilized, the BGP forwarding table appears to be ignored, and the Loopback0.0 filter is not applied.

Below is a simplified network topology showing how all of the Juniper devices are interconnected, as well as how the routing tables are being populated.

I say simplified, because there are actually two MX80 routers, with full BGP tables from two different carriers.

The QFX5100 switches are two physical switches in a VC, using LACP bonding connectivity ( ae[0-5] ) setup for "flexible-vlan-tagging".

Can anyone throw out some ideas as to why the the lo0.0 input filter is being processed on transit traffic?
To my understanding, and all of our research and training, transit traffic should never touch the RE unless exceptions in the packets are experienced.

However, when digging into this issue, I ran across this KB article, but I don't think it applies to my setup (or is even relevant) as I do not run any firewalls on this equipment other than the lo0.0 firewall policy to protect the RE (management, etc).

https://kb.juniper.net/InfoCenter/index?page=content&id=KB32041&cat=QFX_SERIES&actp=LIST

https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1080758

Let me know what you would like to see for configurations or routing table output and I will gladly show it.

Ping Google DNS from Ubuntu Test Server

# ping 8.8.8.8 -c 5
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=120 time=13.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=120 time=16.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=120 time=38.2 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=120 time=19.1 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=120 time=11.9 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 11.988/19.828/38.263/9.566 ms

Traceroute to Google DNS (default UDP mode)

# traceroute 8.8.8.8 -n
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  96.126.81.57  0.786 ms  0.806 ms  0.847 ms
 2  96.126.81.37  19.928 ms  19.916 ms  19.897 ms
 3  96.126.81.25  20.177 ms  20.207 ms  20.187 ms
 4  62.115.170.158  20.059 ms  20.054 ms  20.106 ms
 5  62.115.137.106  20.341 ms  20.463 ms  20.488 ms
 6  62.115.113.84  20.330 ms  19.770 ms *
 7  * * *
 8  * * *
 9  * * *
10  * * *

Traceroute to Google DNS (ICMP Mode)

# traceroute 8.8.8.8 -n --icmp
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  96.126.81.57  0.872 ms  0.906 ms  0.941 ms
 2  96.126.81.37  18.166 ms  18.171 ms  18.169 ms
 3  96.126.81.25  18.384 ms  18.404 ms  18.420 ms
 4  62.115.170.158  18.383 ms  18.383 ms  18.406 ms
 5  62.115.137.106  19.947 ms  20.009 ms  20.105 ms
 6  62.115.113.84  19.020 ms  18.184 ms  18.194 ms
 7  72.14.243.44  17.788 ms  22.440 ms  22.462 ms
 8  * * *
 9  72.14.236.240  22.136 ms  22.148 ms  22.115 ms
10  209.85.248.109  23.025 ms  22.848 ms  22.542 ms
11  8.8.8.8  21.378 ms  21.378 ms  20.519 ms

DIG to Google DNS

# dig google.com @8.8.8.8  

; <<>> DiG 9.10.3-P4-Ubuntu <<>> google.com @8.8.8.8
;; global options: +cmd
;; connection timed out; no servers could be reached

QFX5100 lo0.0 Firewall Filter

> show log RE_FIREWALL | match 96.126.81.60
Mar 27 20:30:24 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 49279 53 (2 packets)
Mar 27 20:30:29 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 49279 53 (2 packets)
Mar 27 20:30:34 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 56371 53 (2 packets)
Mar 27 20:30:39 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 56371 53 (2 packets)
Mar 27 21:04:52 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 52793 53 (2 packets)
Mar 27 21:04:57 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 52793 53 (2 packets)
Mar 27 21:05:02 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 45378 53 (2 packets)
Mar 27 21:05:08 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 45378 53 (2 packets)
Mar 27 21:39:20 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 39696 53 (2 packets)
Mar 27 21:39:25 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 39696 53 (2 packets)
Mar 27 21:39:30 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 58978 53 (2 packets)
Mar 27 21:39:35 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 58978 53 (2 packets)
Mar 27 22:01:48 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 37783 53 (1 packets)
Mar 27 22:01:53 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 37783 53 (1 packets)
Mar 27 22:01:58 qfx5100-01 fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0 D udp 96.126.81.60 8.8.8.8 37783 53 (1 packets)

QFX Configurations - Loopback 0

# show interfaces lo0.0 
family inet {
    filter {
        input PROTECT_RE_v4;
    }
    address 96.126.81.3/32;
}
family inet6 {
    filter {
        input PROTECT_RE_v6;
    }
    address 2607:f170:110:0::3/128;
}

QFX Configurations - Facing MX80 Router

# show interfaces ae1
flexible-vlan-tagging;
aggregated-ether-options {
    minimum-links 1;
    link-speed 10g;
    lacp {
        active;
        periodic fast;
    }
}
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members 1013;
        }
    }
}

# show interfaces irb.1013   
family inet {
    address 96.126.81.25/30;
}
family inet6 {
    address 2607:f170:110:1014::25/116;
}

QFX Configurations - Facing EX4200 Switch

# show interfaces ae4 
flexible-vlan-tagging;
aggregated-ether-options {
    minimum-links 1;
    link-speed 10g;
    lacp {
        active;
        periodic fast;
    }
}
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members [ 19-20 220-225 1016 2010 ];
        }
    }
}

# show interfaces irb.1016  
family inet {
    address 96.126.81.37/30;
}
family inet6 {
    address 2607:f170:110:1016::37/116;
}

Hello,

It looks like You got used to MX-style lo0 filters which typically do not have match on dst.IP, it is not needed there.

Well, QFX is different in this regard and KB article clearly spells the workaround

to work around the issue, set the firewall filter separately for transit traffic and for traffic to the switch. Make sure that they do not clash.

Which means - introduce the dst.IP match in Your lo0 filter and You are going to be fine. 

You can "collect" the local subnets automatically using JUNOS apply-path:

set policy-options prefix-list CONNECTED_IPv4_SUBNETS apply-path "interfaces <*> unit <*> family inet address <*>"

HTH

Thx

Alex

@Alex

I'm not understanding how this answers the question - Why is transit traffic getting to the RE?

In this above specific example, I am attempting a DNS query to Google DNS from my testing server.   

The server is on-net, connected to my EX4200 switch.  The destination server is off-net.

All traffic is "transit" of the QFX5100.

Here is a relevant snip of my term that has anything to do with DNS, with a implicit deny at the end.  This rule would allow the traffic as there is no defined source or destination and has an accept.

set firewall family inet filter PROTECT_RE_v4 term accept-dns from protocol udp
set firewall family inet filter PROTECT_RE_v4 term accept-dns from protocol tcp
set firewall family inet filter PROTECT_RE_v4 term accept-dns from source-port 53
set firewall family inet filter PROTECT_RE_v4 term accept-dns then count accept-dns
set firewall family inet filter PROTECT_RE_v4 term accept-dns then accept

Also, in some of the terms, where required, I am calling dynamic prefix-lists.

set policy-options prefix-list ROUTER_IP4 apply-path "interfaces <*> unit <*> family inet address <*>"
set policy-options prefix-list ROUTER_IP6 apply-path "interfaces <*> unit <*> family inet6 address <*>"

Attached is a full export (display set) of the PROTECT_RE_v4 policy.

Attachment(s)

PROTECT_RE_v4__PROTECT_RE_v6.txt   19 KB 1 version

Hello,

---

@sflynn1509 wrote:

@Alex

 

I'm not understanding how this answers the question - Why is transit traffic getting to the RE?

 

Because QFX5100 imlementation of lo0.0 filter is accepting+redirecting matching packets to RE.

Perhaps it would be more understandable if the QFX5100 filter CLI could say "accept+punt" rather than "accept"

So, if I rewrite it this way, would it be more understandable? (it is not an actual JUNOS CLI):

set firewall family inet filter PROTECT_RE_v4 term accept-dns from protocol udp
set firewall family inet filter PROTECT_RE_v4 term accept-dns from protocol tcp
set firewall family inet filter PROTECT_RE_v4 term accept-dns from source-port 53
set firewall family inet filter PROTECT_RE_v4 term accept-dns then count accept-dns
set firewall family inet filter PROTECT_RE_v4 term accept-dns then accept+punt

This is specific to QFX5K (AFAIK, and ACX5K too) filter implementation.  On MX or EX or SRX You won't see this behaviour because MX/EX/SRX use chipsets that are different from Broadcom one used by QFX5K/ACX5K.

And if You narrow the filter term match to only returning self-terminating DNS packets by adding/introducing dst.IP match, then You won't see this behaviour.

HTH

Thx

Alex

Alex - this does not explain why transit traffic is getting to the RE.

All documentation states that transit traffic should never touch the RE unless it is exception traffic, which this is not.

So, to prove the theory, I have modifed the PROTECT_RE_v4 filter, specifically the DNS statement to have source and destination matching .. traffic still fails to pass.  

set firewall family inet filter PROTECT_RE_v4 term accept-dns from source-prefix-list ANET_MGMT_V4
set firewall family inet filter PROTECT_RE_v4 term accept-dns from destination-prefix-list ROUTER_IP4
set firewall family inet filter PROTECT_RE_v4 term accept-dns from protocol udp
set firewall family inet filter PROTECT_RE_v4 term accept-dns from protocol tcp
set firewall family inet filter PROTECT_RE_v4 term accept-dns from source-port 53
set firewall family inet filter PROTECT_RE_v4 term accept-dns then count accept-dns
set firewall family inet filter PROTECT_RE_v4 term accept-dns then accept

My argument is this --- transit traffic (which this is because the source nor the destination IPs exist on the QFX5100 switch) should never be accepted, punted, or touched by the lo0.0 input filters.  The traffic is not destined to the RE, nor is it needed by the RE.  These packets should be hitting the FIB only and passed at wire speed.  The RE has no need to touch any of this traffic.

As an additional test, I applied a protection filter, exampled by Juniper documentation, and it blockes my SSH access to off-net servers.

https://www.juniper.net/documentation/en_US/junos/topics/example/firewall-filter-stateless-example-trusted-source-block-telnet-and-ssh-access.html

set firewall family inet filter PROTECT_RE_v4__TEST term accept-ssh from source-prefix-list ANET_MGMT_V4
set firewall family inet filter PROTECT_RE_v4__TEST term accept-ssh from destination-prefix-list ROUTER_IP4
set firewall family inet filter PROTECT_RE_v4__TEST term accept-ssh from protocol tcp
set firewall family inet filter PROTECT_RE_v4__TEST term accept-ssh from destination-port 22
set firewall family inet filter PROTECT_RE_v4__TEST term accept-ssh then count accept-ssh
set firewall family inet filter PROTECT_RE_v4__TEST term accept-ssh then accept
set firewall family inet filter PROTECT_RE_v4__TEST term deny-ssh from protocol tcp
set firewall family inet filter PROTECT_RE_v4__TEST term deny-ssh from destination-port 22
set firewall family inet filter PROTECT_RE_v4__TEST term deny-ssh then count deny-ssh
set firewall family inet filter PROTECT_RE_v4__TEST term deny-ssh then reject
set firewall family inet filter PROTECT_RE_v4__TEST term accept-all then accept

# show interfaces lo0.0 
family inet {
    filter {
        input PROTECT_RE_v4__TEST;
    }
    address 96.126.81.3/32;
}
family inet6 {
    address 2607:f170:110:0::3/128;
}

# ping 209.208.0.134 -c 5   
PING 209.208.0.134 (209.208.0.134) 56(84) bytes of data.
64 bytes from 209.208.0.134: icmp_seq=1 ttl=49 time=48.0 ms
64 bytes from 209.208.0.134: icmp_seq=2 ttl=49 time=52.8 ms
64 bytes from 209.208.0.134: icmp_seq=3 ttl=49 time=47.1 ms
64 bytes from 209.208.0.134: icmp_seq=4 ttl=49 time=56.2 ms
64 bytes from 209.208.0.134: icmp_seq=5 ttl=49 time=54.8 ms

--- 209.208.0.134 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 47.143/51.827/56.256/3.647 ms

# traceroute 209.208.0.134 -n
traceroute to 209.208.0.134 (209.208.0.134), 30 hops max, 60 byte packets
 1  96.126.81.57  0.882 ms  0.902 ms  0.913 ms
 2  96.126.81.37  27.485 ms  27.474 ms  27.455 ms
 3  96.126.81.25  27.719 ms  27.734 ms  27.759 ms
 4  62.115.170.158  27.591 ms  27.590 ms  27.591 ms
 5  62.115.137.106  28.974 ms  29.065 ms  27.825 ms
 6  62.115.113.84  28.034 ms  27.419 ms  27.370 ms
 7  4.68.74.165  27.235 ms  22.663 ms  22.842 ms
 8  4.69.137.146  55.147 ms  54.912 ms  54.866 ms
 9  67.30.142.82  54.844 ms  54.838 ms  54.748 ms
10  209.208.7.50  58.554 ms  58.547 ms  58.441 ms
11  209.208.7.130  58.901 ms  58.940 ms  58.998 ms
12  209.208.0.134  55.839 ms  55.752 ms  55.910 ms


# ssh {{redacted}}@209.208.0.134
ssh: connect to host 209.208.0.134 port 22: No route to host

# run show log RE_FIREWALL    
Mar 28 07:56:49 ds-01.dal-tx clear-log[4164]: logfile cleared
Mar 28 07:57:07  ds-01.dal-tx fpc0 PFE_FW_SYSLOG_IP: FW: ae4.0        R  tcp 96.126.81.60 209.208.0.134 54882    22 (1 packets)

---

@aarseniev wrote:

 

Because QFX5100 imlementation of lo0.0 filter is accepting+redirecting matching packets to RE.

 

Can you provide a documentation link explaining why all transit traffic would be accepted + redirected to the RE on the QFX platform?

I've not heard that before either. FWIW I have the same filter on my QFX without such issues. Perhaps post your whole config for a second set of eyes to look at. What version are you running?

test@test> show configuration interfaces lo0 |display set 
set interfaces lo0 unit 0 family inet filter input protect-re

{master:0}
test@test> show configuration firewall |display set |match dns    
set firewall family inet filter protect-re term allow-dns from source-prefix-list dns-servers
set firewall family inet filter protect-re term allow-dns from protocol tcp
set firewall family inet filter protect-re term allow-dns from protocol udp
set firewall family inet filter protect-re term allow-dns from source-port domain
set firewall family inet filter protect-re term allow-dns then count allow-dns
set firewall family inet filter protect-re term allow-dns then log
set firewall family inet filter protect-re term allow-dns then accept

---

@smicker wrote:

I've not heard that before either. FWIW I have the same filter on my QFX without such issues. Perhaps post your whole config for a second set of eyes to look at. What version are you running?

@smicker - I will get it sanatized and upload it shortly.

Given the routing weirdness you also noted it almost sounds as if your lo0 interface has become your default route (if such a thing is possible), which would explain why it is seeing all traffic (and why transit traffic isn't working).

Version

# run show version 
fpc0:
--------------------------------------------------------------------------
Hostname: ds-01
Model: qfx5100-48t-6q
Junos: 14.1X53-D47.6
JUNOS Base OS Software Suite [14.1X53-D47.6]
JUNOS Base OS boot [14.1X53-D47.6]
JUNOS Crypto Software Suite [14.1X53-D47.6]
JUNOS Online Documentation [14.1X53-D47.6]
JUNOS Kernel Software Suite [14.1X53-D47.6]
JUNOS Packet Forwarding Engine Support (qfx-ex-x86-32) [14.1X53-D47.6]
JUNOS Routing Software Suite [14.1X53-D47.6]
JUNOS SDN Software Suite [14.1X53-D47.6]
JUNOS Enterprise Software Suite [14.1X53-D47.6]
JUNOS Web Management Platform Package [14.1X53-D47.6]
JUNOS py-base-i386 [14.1X53-D47.6]
JUNOS Host Software [14.1X53-D47.6]

We have also attempted the following version with identical outcome:

14.1X53-D48.1

17.3R3.10 (was crashing the switch after BGP load up)

We have not tried this release as of yet -- 14.1X53-D49 -- this was going to be done today.

Attached is the redacted configuration of the QFX5100.   Please be aware that quite a bit of the configuration is deactivated or questionable why it may be there.  We have been fighting this issue for over a month now and have tried everything we can think of to come up with answer to this problem.

xe-0/0/48:1.1013 is the current configuration on the device as I removed the IRB.1013 to see if it would make a difference last night.  I wanted to see if the LACP bundle was possibly causing this issue.

I can place the configuration back to the attached configuration if it will help in decifering the route view.

# show interfaces xe-0/0/48:1  
description er-02__xe-0/0/1;
flexible-vlan-tagging;
unit 0 {
    family ethernet-switching {
        interface-mode trunk;
        vlan {
            members 1030;
        }
    }
}
unit 1013 {
    vlan-id 1013;
    family inet {
        address 96.126.81.26/30;
    }
    family inet6 {
        address 2607:f170:110:1013::26/116;
    }
}

If you question a config setting, ask I will explain the purpose for it or we can remove it and test further.

Also, here is a view of the route database and FIB as it sits right now.

> show route 209.208.0.134 extensive 

inet.0: 731349 destinations, 731352 routes (731349 active, 0 holddown, 0 hidden)
209.208.0.0/17 (1 entry, 1 announced)
TSI:
KRT in-kernel 209.208.0.0/17 -> {indirect(131071)}
        *BGP    Preference: 170/-101
                Next hop type: Indirect
                Address: 0x2034e740
                Next-hop reference count: 2193969
                Source: 96.126.81.25
                Next hop type: Router, Next hop index: 1723
                Next hop: 96.126.81.25 via xe-0/0/48:1.1013, selected
                Session Id: 0x0
                Protocol next hop: 96.126.81.25
                Indirect next hop: 0x1f00a750 131071 INH Session ID: 0x0
                State: <Active Int Ext>
                Local AS:  6364 Peer AS:  6364
                Age: 13:07:27   Metric2: 0 
                Validation State: unverified 
                Task: BGP_6364.96.126.81.25+60228
                Announcement bits (2): 0-KRT 6-Resolve tree 1 
                AS path: 1299 3356 6364 I
                Communities: 6364:11 6364:12
                Accepted                
                Localpref: 100
                Router ID: 96.126.81.2
                Indirect next hops: 1
                        Protocol next hop: 96.126.81.25
                        Indirect next hop: 0x1f00a750 131071 INH Session ID: 0x0
                        Indirect path forwarding next hops: 1
                                Next hop type: Router
                                Next hop: 96.126.81.25 via xe-0/0/48:1.1013
                                Session Id: 0x0
                        96.126.81.24/30 Originating RIB: inet.0
                          Node path count: 1
                          Forwarding nexthops: 1
                                Next hop type: Interface
                                Nexthop: via xe-0/0/48:1.1013

> show route 

inet.0: 731401 destinations, 731404 routes (731401 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

1.0.0.0/24         *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 13335 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.4.0/22         *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 4826 38803 56203 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.4.0/24         *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 4826 38803 56203 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.5.0/24         *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 4826 38803 56203 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.6.0/24         *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 4826 38803 56203 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.7.0/24         *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 4826 38803 56203 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.16.0/24        *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 2497 2519 2519 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.64.0/18        *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 2497 7670 18144 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.128.0/17       *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 38040 23969 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013
1.0.128.0/18       *[BGP/170] 13:06:16, localpref 100
                      AS path: 1299 38040 23969 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013

> show route forwarding-table 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct       51     2
0.0.0.0/32         perm     0                    dscd       49     1
1.0.0.0/24         user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.4.0/22         user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.4.0/24         user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.5.0/24         user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.6.0/24         user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.7.0/24         user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.16.0/24        user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.64.0/18        user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.128.0/17       user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.128.0/18       user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.128.0/19       user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013
1.0.128.0/24       user     0                    indr   131071 731327
                              96.126.81.25       ucst     1723     8 xe-0/0/48:1.1013

Attachment(s)

qfx5100_redacted_config__03282019.txt   72 KB 1 version

Hi sflynn1509,

Did you mean there is no route to the destination (no default router either)? This traffic must be  hitting the default route in the PFE/forwarding-table that has a default next action to punt to the RE.  So that control plane look-up is done to check for any special treatment (by firewall filter etc.)  for the destination prefix in question.  The RE is protected against such traffic through filters (if any) or via DDOS protection.  This default route is created when PFE comes up first, however it's not hit as long as a usable default route is programmed to the PFE either by manual configuration on the CLI or learnt via a routing-protocol.

Please check these outputs on your device to confirm this is indeed getting hit:

start shell
cprod -A fpc0 -c 'set dc bc "l3 defip show"' | grep 0.0.0.0

cprod -A fpc0 -c 'set dc bc "l3 egress show 100005"'

Example:

root@Juniper> start shell

root@Juniper:RE:0% cprod -A fpc0 -c 'set dc bc "l3 defip show"' | grep 0.0.0.0
1536 1 0.0.0.0/0 00:00:00:00:00:00 100005 0 0 0 0 n <<<<<<<<<<<<<<<will be "y" when it gets hit
1536 2 0.0.0.0/0 00:00:00:00:00:00 100004 0 0 0 0 n

root@Juniper:RE:0% cprod -A fpc0 -c 'set dc bc "l3 egress show 100005"'

HW (unit 0)
Entry Mac Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount
100005 00:00:00:01:02:03 2 1 127 0 -1 yes no 17 <<<<<<<<<<<<<<<<<<punts to cpu by default

If confirmed, please feel free to quote this for getting any further rationale from JTAC on this.

Hope this helps.

Regards,
-r.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution."
Kudos are always appreciated :).

mriyaz:

The DS-01 (QFX5100) switch does not have a default route in the routing table.  This is intentional because it has a full bgp route table.

The routes to both destination servers (209.208.0.134 and 8.8.8.8) exist in the routing table.

> show route 209.208.0.134    

inet.0: 731422 destinations, 731425 routes (731422 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

209.208.0.0/17     *[BGP/170] 15:51:09, localpref 100
                      AS path: 1299 3356 6364 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013


> show route 8.8.8.8          

inet.0: 731423 destinations, 731426 routes (731423 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

8.8.8.0/24         *[BGP/170] 15:52:48, localpref 100
                      AS path: 1299 15169 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013

Requested Output

root@ds-01:RE:0% cprod -A fpc0 -c 'set dc bc "l3 defip show"' | grep 0.0.0.0
6140  1        0.0.0.0/0            00:00:00:00:00:00 100005    0     0     0    0 y
6140  3        0.0.0.0/0            00:00:00:00:00:00 100004    0     0     0    0 n
6141  4        0.0.0.0/0            00:00:00:00:00:00 100005    0     0     0    0 n


root@ds-01:RE:0% cprod -A fpc0 -c 'set dc bc "l3 egress show 100005"'


HW (unit 0)
Entry  Mac                 Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount
100005  00:00:00:01:02:03    2    0   127    0        -1  yes   no    5

So, are you saying that because I do NOT have a default route, that it is forcing all of the traffic into the RE for processing?

---

@sflynn1509 wrote:

mriyaz:

 

The DS-01 (QFX5100) switch does not have a default route in the routing table.  This is intentional because it has a full bgp route table.

 

The routes to both destination servers (209.208.0.134 and 8.8.8.8) exist in the routing table.

 

> show route 209.208.0.134    

inet.0: 731422 destinations, 731425 routes (731422 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

209.208.0.0/17     *[BGP/170] 15:51:09, localpref 100
                      AS path: 1299 3356 6364 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013


> show route 8.8.8.8          

inet.0: 731423 destinations, 731426 routes (731423 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

8.8.8.0/24         *[BGP/170] 15:52:48, localpref 100
                      AS path: 1299 15169 I, validation-state: unverified
                    > to 96.126.81.25 via xe-0/0/48:1.1013

 

Requested Output

root@ds-01:RE:0% cprod -A fpc0 -c 'set dc bc "l3 defip show"' | grep 0.0.0.0
6140  1        0.0.0.0/0            00:00:00:00:00:00 100005    0     0     0    0 y
6140  3        0.0.0.0/0            00:00:00:00:00:00 100004    0     0     0    0 n
6141  4        0.0.0.0/0            00:00:00:00:00:00 100005    0     0     0    0 n


root@ds-01:RE:0% cprod -A fpc0 -c 'set dc bc "l3 egress show 100005"'


HW (unit 0)
Entry  Mac                 Vlan INTF PORT MOD MPLS_LABEL ToCpu Drop RefCount
100005  00:00:00:01:02:03    2    0   127    0        -1  yes   no    5

 

So, are you saying that because I do NOT have a default route, that it is forcing all of the traffic into the RE for processing?

[Ans] Yup, the highlighted output above just confirms that too.  You can configure a default route and see the behavior change for double confirmation.

---

Hope this helps.

Regards,
-r.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution."
Kudos are always appreciated :).

Ok - that's interesting.  This is the first i've ever heard that you still have to have a default route even with full bgp tables.

Is this a QFX thing?   I've never seen this before on any other Juniper or Cisco device.

I can't believe that would be expected behavior. Does

cprod -A fpc0 -c 'set dc bc "l3 defip show"' | grep 209.208.0.0

show an installed route? And when you say "When no default route is populated on the QFX5100 switch, and only the full BGP table" do you mean full internet table? That's 750K routes vs max 208K on QFX5100.

---

@smicker wrote:

I can't believe that would be expected behavior. Does

 

cprod -A fpc0 -c 'set dc bc "l3 defip show"' | grep 209.208.0.0

 

show an installed route? And when you say "When no default route is populated on the QFX5100 switch, and only the full BGP table" do you mean full internet table? That's 750K routes vs max 208K on QFX5100.

---

Yes, I am sending the full internet routing table to the QFX5100.

The routing table says there are 731K routes in the active table.

inet.0: 731533 destinations, 731536 routes (731533 active, 0 holddown, 0 hidden)

> show bgp summary
Groups: 2 Peers: 4 Down peers: 1
Table Tot Paths Act Paths Suppressed History Damp State Pending
inet.0
          731537 731537 0 0 0 0
inet6.0
          0 0 0 0 0 0
Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
96.126.81.25 6364 514314 2542 0 1 19:13:05 731537/731537/731537/0 0/0/0/0

Are you getting the max # of routes from this document? -- Page #41
https://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/qfx-series/routing-options.pdf

um --- nope

root@ds-01:RE:0% cprod -A fpc0 -c 'set dc bc "l3 defip show"' | grep 209.208
root@ds-01:RE:0%

but it exists in the routing table

> show route 209.208.0.0 detail 

inet.0: 731534 destinations, 731537 routes (731534 active, 0 holddown, 0 hidden)
209.208.0.0/17 (1 entry, 1 announced)
        *BGP    Preference: 170/-101
                Next hop type: Indirect
                Address: 0x2034e740
                Next-hop reference count: 2194521
                Source: 96.126.81.25
                Next hop type: Router, Next hop index: 1723
                Next hop: 96.126.81.25 via xe-0/0/48:1.1013, selected
                Session Id: 0x0
                Protocol next hop: 96.126.81.25
                Indirect next hop: 0x1f00a750 131071 INH Session ID: 0x0
                State: <Active Int Ext>
                Local AS:  6364 Peer AS:  6364
                Age: 18:34:22   Metric2: 0 
                Validation State: unverified 
                Task: BGP_6364.96.126.81.25+60228
                Announcement bits (2): 0-KRT 6-Resolve tree 1 
                AS path: 1299 3356 6364 I
                Communities: 6364:11 6364:12
                Accepted
                Localpref: 100
                Router ID: 96.126.81.2  

Yes agree with smicker.

@sflynn1509, I'm sorry in the excitement of explaining what's punting traffic to the CPU I've missed that a route exists for the prefix in the RIB.  Please also check if it exists in the FIB.  As I've said earlier, that default route on the PFE isn't hit unless its last to last resort i.e. when no route for the prefix exists and no default route either.

Please also check this from shell for any table full messages:
show log messages | grep lpm

start shell

cprod -A fpc0 -c "show syslog messages"

Hope this helps.

Regards,
-r.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution."
Kudos are always appreciated :).

We defintely seem to be over the scale on the no. of routes learnt, please reduce this to scale that should take care of it.  CLI to see what's configured and current usage:

show chassis forwarding-options
show pfe route summary hw

Options are explained in the doc above or here:
https://www.juniper.net/documentation/en_US/junos/topics/topic-map/layer-2-forwarding-tables.html#id-configuring-the-unified-forwarding-table-on-switches

Hope this helps.

Regards,
-r.

--------------------------------------------------

If this solves your problem, please mark this post as "Accepted Solution."
Kudos are always appreciated :).

Confirmation of the proposed issue.

> show log messages | match lpm | last 100 
Mar 28 17:57:52  ds-01 fpc0 brcm_rt_ip_uc_lpm_install:1335 (LPM route add failed) Reason : Table full unit 0
Mar 28 17:57:52  ds-01 fpc0 brcm_rt_ip_uc_entry_install:1203 brcm_rt_ip_uc_entry_install Error: lpm ip route install failed vrf 1 ip 93.170.214/24 nh-swidx 131071 nh-hwidx 100029
Mar 28 17:57:52  ds-01 fpc0 brcm_rt_ip_uc_lpm_install:1335 (LPM route add failed) Reason : Table full unit 0
Mar 28 17:57:52  ds-01 fpc0 brcm_rt_ip_uc_entry_install:1203 brcm_rt_ip_uc_entry_install Error: lpm ip route install failed vrf 1 ip 93.170.212/24 nh-swidx 131071 nh-hwidx 100029
Mar 28 17:57:53  ds-01 fpc0 brcm_rt_ip_uc_lpm_install:1335 (LPM route add failed) Reason : Table full unit 0
Mar 28 17:57:53  ds-01 fpc0 brcm_rt_ip_uc_entry_install:1203 brcm_rt_ip_uc_entry_install Error: lpm ip route install failed vrf 1 ip 201.183.255/24 nh-swidx 131071 nh-hwidx 100029
Mar 28 17:58:02  ds-01 fpc0 brcm_rt_ip_uc_lpm_install:1335 (LPM route add failed) Reason : Table full unit 0
Mar 28 17:58:02  ds-01 fpc0 brcm_rt_ip_uc_entry_install:1203 brcm_rt_ip_uc_entry_install Error: lpm ip route install failed vrf 1 ip 93.170.212/22 nh-swidx 131071 nh-hwidx 100029
Mar 28 17:58:06  ds-01 fpc0 brcm_rt_ip_uc_lpm_install:1335 (LPM route add failed) Reason : Table full unit 0
Mar 28 17:58:06  ds-01 fpc0 brcm_rt_ip_uc_entry_install:1203 brcm_rt_ip_uc_entry_install Error: lpm ip route install failed vrf 1 ip 103.132.192/24 nh-swidx 131071 nh-hwidx 100029
Mar 28 17:58:12  ds-01 fpc0 brcm_rt_ip_uc_lpm_install:1335 (LPM route add failed) Reason : Table full unit 0
Mar 28 17:58:12  ds-01 fpc0 brcm_rt_ip_uc_entry_install:1203 brcm_rt_ip_uc_entry_install Error: lpm ip route install failed vrf 1 ip 192.141.138/24 nh-swidx 131071 nh-hwidx 100029

> show chassis forwarding-options 
fpc0:
--------------------------------------------------------------------------
UFT Configurtion:
l2-profile-three. (MAC: 160K L3-host: 144K LPM: 16K) (default)
num-65-127-prefix = 1K

> show pfe route summary hw 

Slot 0

Unit: 0
Profile active: l2-profile-three
Type            Max       Used      Free      % free
----------------------------------------------------
IPv4 Host       147456    42        147370    99.94
IPv4 LPM        12288     12279     5         0.04
IPv4 Mcast      73728     0         73685     99.94

IPv6 Host       73728     22        73685     99.94
IPv6 LPM(< 64)  6144      2         3         0.05
IPv6 LPM(> 64)  1024      9         1015      99.12
IPv6 Mcast      36864     0         36843     99.94

And now I know my answer --  Why is transit traffic getting to the RE?

Transit traffic will be punted to the RE if the route does not exist in the FIB and there is no default route present.

Don't always trust what is displayed on the screen and assume that the routes are truly active in the FIB as I did.  Under the hood, they may not actually be active.

inet.0: 731349 destinations, 731352 routes (731349 active, 0 holddown, 0 hidden)

This has been a very educational post for me and I thank you all for your feedback and assistance in tracking this down.

We now have to make some design change decisions due to this limitation that I failed to be aware of (max table limit on QFX5100).

Awesome. Educational for me too.