Junos OS

Expand all | Collapse all

MX L2-over-GRE MTU issues

  • 1.  MX L2-over-GRE MTU issues

    Posted 01-04-2020 08:36

    I have two MX routers with intentical version 17.3R3.10

    First MX is border router and have the following config:

     

    show interfaces xe-1/0/0 
    flexible-vlan-tagging;
    mtu 9216;
    unit 0 {
        vlan-id 506;
        family inet {
            mtu 9000;
            address 10.0.0.18/30;
        }
    }
    
    show interfaces gr-1/3/0
    unit 0 {
        tunnel {
            source 10.0.0.18;
            destination 10.0.0.17;
        }
        family bridge {
            interface-mode trunk;
            vlan-id-list 3009;
        }
    }

    VLAN 3009 is bridged with irb with public IP address

     

     

    Router 2 config is:

     

    show interfaces xe-0/0/3                                            
    flexible-vlan-tagging;
    mtu 9216;
    unit 0 {
        vlan-id 506;
        family inet {
            mtu 9000;
            address 10.0.0.17/30;
        }
    }
    
    show interfaces gr-0/0/0 
    unit 0 {
        tunnel {
            source 10.0.0.17;
            destination 10.0.0.18;
        }
        family bridge {
            interface-mode trunk;
            vlan-id-list 3009;
        }
    }

    VLAN3009 is bridged with second interface:

     

     

    show bridge-domains
    test {
        vlan-id 3009;
    }
    show interfaces xe-0/0/1    
    unit 0 {
        family bridge {
            interface-mode trunk;
            vlan-id-list 3009;
        }
    }

    On which have connected Linux server with configured public IP and gateway irb interface on first router.

     

     

    Between routers jumbo frames is passing, but from the Linux server (default mtu 1500) to the gateway via gre tunnel largest passing packet is 1468 bytes, which create issues with tcp-mss. Already tried allow-fragmentation and do-not-fragment tunnel options, but nothing changes.

     

    In both routers gre tunnel mtu is marked as unlimited:

     

    run show interfaces gr-1/3/0      
    Physical interface: gr-1/3/0, Enabled, Physical link is Up
      Interface index: 275, SNMP ifIndex: 851
      Type: GRE, Link-level type: GRE, MTU: Unlimited, Speed: 10000mbps
      Device flags   : Present Running
      Interface flags: Point-To-Point SNMP-Traps
      Input rate     : 0 bps (0 pps)
      Output rate    : 21616 bps (58 pps)
    
      Logical interface gr-1/3/0.0 (Index 568) (SNMP ifIndex 901)
        Flags: Up Point-To-Point SNMP-Traps 0x20024000 IP-Header 10.0.0.17:10.0.0.18:47:df:64:0000000000000000 Encapsulation: L2-over-GRE
        Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off
        Gre keepalives configured: Off, Gre keepalives adjacency state: down
        Input packets : 28868
        Output packets: 3953521
        Protocol bridge, MTU: Unlimited
          Flags: Trunk-Mode
    

     

     

     

    run show interfaces gr-0/0/0   
    Physical interface: gr-0/0/0, Enabled, Physical link is Up
      Interface index: 140, SNMP ifIndex: 566
      Type: GRE, Link-level type: GRE, MTU: Unlimited, Speed: 10000mbps
      Device flags   : Present Running
      Interface flags: Point-To-Point SNMP-Traps
      Input rate     : 17752 bps (45 pps)
      Output rate    : 0 bps (0 pps)
    
      Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 621)
        Flags: Up Point-To-Point SNMP-Traps 0x0 IP-Header 10.0.0.18:10.0.0.17:47:df:64:0000000000000000 Encapsulation: L2-over-GRE
        Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off
        Gre keepalives configured: Off, Gre keepalives adjacency state: down
        Input packets : 7362259
        Output packets: 27122
        Protocol bridge, MTU: Unlimited
          Flags: Trunk-Mode

     

     

    I would appreciate any ideas



  • 2.  RE: MX L2-over-GRE MTU issues

     
    Posted 01-04-2020 13:28

    What is the MTU of  interfaces xe-0/0/1 ?

     

    Please verify the MTU of physical interface in the path.

     

    You can also ping with do-no-fragment knob and packet size to figure out the maximum possible MTU for the path. 

     

     ping 8.8.8.8 do-not-fragment size 1400 < keep increasing the size until packet drops

     

    If you have some tcp-mss value in mind for GRE interface you can try to configure that

     

    TEST# show interfaces gr-4/0/0
    unit 0 {
    family inet {
    mtu 1400;
    tcp-mss 600;
    }
    }

    Usually if ethernet header is 1500 the TCP-MSS value is 1460 (1500 - IP Header and TCP Header)

     

    Mostly, it should be some configuration issue. Please check that and let me know if you have further query.

    PS: Please accpet my response as solution if it answers your query, kuods are appreciated too!

    Thanks
    Vishal



  • 3.  RE: MX L2-over-GRE MTU issues

    Posted 01-06-2020 02:09

     


    @vlsingh wrote:

    What is the MTU of  interfaces xe-0/0/1 ?

    MTU is 1500, but this interface is connected to the Linux server and not related with the gre tunnel.

     


    @vlsingh wrote:

    You can also ping with do-no-fragment knob and packet size to figure out the maximum possible MTU for the path. 

     


    Between both routers jumbo frames is passing without issues, via tunnel max MTU is 1468 bytes.

     

     


    @vlsingh wrote:

    If you have some tcp-mss value in mind for GRE interface you can try to configure that

     

    TEST# show interfaces gr-4/0/0
    unit 0 {
    family inet {
    mtu 1400;
    tcp-mss 600;
    }
    }


    We are talking about family bridge (Layer 2 over GRE). With family inet and tcp-mss tune there is no issues.



  • 4.  RE: MX L2-over-GRE MTU issues

     
    Posted 01-06-2020 12:35

    Do you see "MTU errors" increasing on any interface involve in the topology when you send your traffic?

     

    show interfaces ge-0/0/0 extensive | match "MTU errors"

     

    Or do you see "Output MTU" counter increasing on the box when sending that traffic?

     

    show pfe statistics traffic

     

    Thanks Vishal



  • 5.  RE: MX L2-over-GRE MTU issues

    Posted 01-07-2020 05:05

    No MTU errors on both sides:

     

    # run show interfaces xe-1/0/0 extensive | match "MTU errors"
    Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0

    # run show interfaces xe-0/0/3 extensive |match "MTU errors"
    Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0

     

    On second router where is connected the Linux server output mtu counter is increasing:

     

    Packet Forwarding Engine hardware discard statistics:
    
        Timeout                    :                    0
    
        Truncated key              :                    0   
    
        Bits to test               :                    0        
    
        Data error                 :                    0
    
        TCP header length error    :                    0
    
        Stack underflow            :                    0
    
        Stack overflow             :                    0
    
        Normal discard             :               735218
    
        Extended discard           :                    0
    
        Invalid interface          :                    0
    
        Info cell drops            :                    0
    
        Fabric drops               :                    0
    
    Packet Forwarding Engine Input IPv4 Header Checksum Error and Output MTU Error statistics:
    
        Input Checksum             :                    0
    
        Output MTU                 :                 1177

     



  • 6.  RE: MX L2-over-GRE MTU issues

     
    Posted 01-07-2020 06:43

    Hello,

     

    Is that "Output MTU" drop increasing when you are sending the interested traffic?

     

    As mentioned in the following blog:

     

             http://blog.ip.fi/2014/02/junos-l3-incompletes-what-and-why.html

     

    We can check for pfe drops corresponding to "Output MTU" drop.

     

    Can you please share following output?

     

    start shell pfe network fpc0

    show jnh 0 exception terse

    exit

     

    Thanks

    Vishal



  • 7.  RE: MX L2-over-GRE MTU issues

    Posted 01-18-2020 07:24

    @vlsingh wrote:

    Is that "Output MTU" drop increasing when you are sending the interested traffic?

     Yes, on each sended packet.

     


    @vlsingh wrote:

    Can you please share following output?

    start shell pfe network fpc0

    show jnh 0 exception terse

    exit

     

     

    From MX960:

     

     

    # show jnh 0 exception terse
    
    Reason                             Type           Packets      Bytes
    ==================================================================
    
    
    PFE State Invalid
    ----------------------
    unknown iif                        DISC(  1)        47216    5683339
    
    
    Packet Exceptions
    ----------------------
    ttl expired                        PUNT(  1)       131205    7460684
    
    
    Bridging
    ----------------------
    mlp pkt                            PUNT( 11)            1         29
    
    
    Firewall
    ----------------------
    firewall discard                   DISC( 67)        29239    1659966
    
    
    Routing
    ----------------------
    hold route                         DISC( 70)     53206547 2347192094
    resolve route                      PUNT( 33)     11873244  535881749
    control pkt punt via nh            PUNT( 34)         1182      49644
    host route                         PUNT( 32)     21807739  930704110
    reject route                       PUNT( 40)      3523413  197335715

     

     

     

    From MX80:

     

     

    # show jnh 0 exception terse
    
    Reason                             Type           Packets      Bytes
    ==================================================================
    
    
    Error Parcels
    ----------------------
    from WAN                           M2L ERROR         6812
    
    
    PFE State Invalid
    ----------------------
    sw error                           DISC( 64)           42       2248
    unknown family                     DISC( 73)         1076      83912
    iif down                           DISC( 87)           11        638
    unknown iif                        DISC(  1)      2463941  134519193
    invalid L2 token                   DISC( 86)            1         46
    
    
    Packet Exceptions
    ----------------------
    mtu exceeded                       DISC( 21)        12086   18663387
    IP options                         PUNT(  2)            2         64
    my-mac check failed                DISC( 28)           12       1020
    
    
    Bridging
    ----------------------
    bridge ucast split horizon         DISC( 26)            1         90
    mlp pkt                            PUNT( 11)         5361     155469
    input trunk vlan lookup failed     DISC( 91)        75787    3509085
    
    
    Routing
    ----------------------
    hold route                         DISC( 70)            2        324
    resolve route                      PUNT( 33)            7        748
    control pkt punt via nh            PUNT( 34)     54472896 1525280380
    host route                         PUNT( 32)       942262   58209572
    reject route                       PUNT( 40)            2        188
    
    

     

    "unknown iif" is increasing on each packet.

     



  • 8.  RE: MX L2-over-GRE MTU issues

     
    Posted 01-18-2020 21:02

    Hi Lyubomir,

     

    I think from MX80 this output is interesting, does "MTU exceeded" increase with each packet that you are sending?

     

    # show jnh 0 exception terse

    mtu exceeded DISC( 21) 12086 18663387

     

    You can debug this drop counter (21) to confirm if its your GRE traffic

     

    start shell pfe network tfeb0

    debug jnh exceptions 21 discard

    debug jnh exceptions-trace

    show jnh exceptions-trace

    undebug jnh exceptions 21 discard

     

    refer: http://blog.ip.fi/2014/02/junos-l3-incompletes-what-and-why.html

     

    If you can see your packet in "show jnh exceptions-trace" output on MX80. That means packet is getting dropped on MX80 and you need to verify all interface MTU on MX80 again.

     

    Thanks

    Vishal



  • 9.  RE: MX L2-over-GRE MTU issues

    Posted 01-23-2020 11:36

    Hello,

     

    MTU on all interfaces is > 9000 bytes. Here is the debug result when I am pinging from the server via gre tunnel MX960 router with 1470 bytes:

     

    mtu exceeded - its increasing on each packet

     

    # show jnh exceptions-trace
    [Jan 23 20:30:01.891] [241572] jnh_exception_packet_trace:1800 ###############
    [Jan 23 20:30:01.891] [241573] jnh_exception_packet_trace:1805 [iif:334,code/info:149 D(mtu exceeded)/0x5ea,score:(0x1),ptype:2/0,orig_ptype:2,offset:60,orig_offset:60,len:1558,l2iif:328,oif:337,BD 3,l2-off=180388627013,token=2791800 ]
    [Jan 23 20:30:01.891] [241574] jnh_exception_packet_trace:1829 0x00: 20 01 95 00 5e a0 01 4e 2a 3c 06 16 80 01 48 20 
    [Jan 23 20:30:01.891] [241575] jnh_exception_packet_trace:1829 0x10: 3c 00 03 40 02 00 00 02 45 00 00 00 00 15 10 08 
    [Jan 23 20:30:01.891] [241576] jnh_exception_packet_trace:1829 0x20: 81 f4 89 15 f3 28 c0 da fa 28 a5 81 00 01 fa 08 
    [Jan 23 20:30:01.891] [241577] jnh_exception_packet_trace:1829 0x30: 00 45 00 06 04 6d 00 00 00 3f 2f f5 0b b9 cd d0 
    [Jan 23 20:30:01.891] [241578] jnh_exception_packet_trace:1829 0x40: 12 b9 cd d0 11 00 00 65 58 00 25 90 a2 80 ab 28 
    [Jan 23 20:30:01.892] [241579] jnh_exception_packet_trace:1829 0x50: c0 da fa 2f f0 81 00 0b c1 08 00 45 00 05 da 6c 
    [Jan 23 20:30:01.892] [241580] jnh_exception_packet_trace:1829 0x60: ff 40 00 40 01 98 20 5b 84 3c 01 5b 84 3c fa 00 
    [Jan 23 20:30:01.892] [241581] jnh_exception_packet_trace:1829 0x70: 00 6b 66 0c 21 00 55 10 f5 29 5e 00 00 00 00 bc

     

    iif:334 is the gre tunnel - Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 621)

     

     

    show interfaces gr-0/0/0   

    Physical interface: gr-0/0/0, Enabled, Physical link is Up

      Interface index: 140, SNMP ifIndex: 566

      Description: S3C-TEST

      Type: GRE, Link-level type: GRE, MTU: Unlimited, Speed: 10000mbps

      Device flags   : Present Running

      Interface flags: Point-To-Point SNMP-Traps

      Input rate     : 43096 bps (69 pps)

      Output rate    : 23664 bps (6 pps)

     

      Logical interface gr-0/0/0.0 (Index 334) (SNMP ifIndex 621)

        Flags: Up Point-To-Point SNMP-Traps Reassemble-Pkts 0x0 IP-Header xx.xx.xx.xx:xx.xx.xx.xx:47::64:0000000000000000 Encapsulation: L2-over-GRE

        Copy-tos-to-outer-ip-header: Off, Copy-tos-to-outer-ip-header-transit: Off

        Gre keepalives configured: Off, Gre keepalives adjacency state: down

        Input packets : 131830202

        Output packets: 2781297

        Protocol bridge, MTU: Unlimited

          Flags: Trunk-Mode



  • 10.  RE: MX L2-over-GRE MTU issues

     
    Posted 01-23-2020 12:24

    Hi Lyubomir,

     

    ## Here is the ICMP reply packets which is geting dropped:

     

    Frame 1 (97 bytes on wire, 97 bytes captured)
    Ethernet II, Src: 28:c0:da:fa:28:a5 (28:c0:da:fa:28:a5), Dst: 08:81:f4:89:15:f3 (08:81:f4:89:15:f3)
    802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 506
    Internet Protocol, Src: 185.205.208.18 (185.205.208.18), Dst: 185.205.208.17 (185.205.208.17)
    Generic Routing Encapsulation (Transparent Ethernet bridging)
    Ethernet II, Src: 28:c0:da:fa:2f:f0 (28:c0:da:fa:2f:f0), Dst: SuperMic_a2:80:ab (00:25:90:a2:80:ab)
    802.1Q Virtual LAN, PRI: 0, CFI: 0, ID: 3009
    Internet Protocol, Src: 91.132.60.1 (91.132.60.1), Dst: 91.132.60.250 (91.132.60.250)
    Internet Control Message Protocol
    Checksum: 0x6b66 [incorrect, should be 0xfd35]
    0000 10 f5 29 5e 00 00 00 00 bc ..)

     

     

    ## Incoming interface is 334 ( gr-0/0/0.0 ), outgoing interface is (337), packet length is 1558 and its getting dropped due to MTU issue.  Please check the MTU for interface index 337 and 328. Please check MTU for interface in that bridge domain.

     

    jnh_exception_packet_trace:1805 [iif:334,code/info:149 D(mtu exceeded)/0x5ea,score:(0x1),ptype:2/0,orig_ptype:2,offset:60,orig_offset:60,len:1558,l2iif:328,oif:337,BD 3,l2-off=180388627013,token=2791800 ]


    ## If there is no MTU misconfiguration you can open a JTAC case.



    PS: Please accpet my response as solution if it answers your query, kudos are appreciated too!

    Thanks
    Vishal



  • 11.  RE: MX L2-over-GRE MTU issues

    Posted 01-05-2020 00:00

    Hello,

     


    @Lyubomir wrote:

    Already tried allow-fragmentation and do-not-fragment tunnel options, but nothing changes.

     



    You need to use 2 knobs together: "allow-fragmentation" and "reassemble-packets" 

    https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/reassemble-packets-edit-interfaces.html

     

    One more stupid question - do You use DPC/I-CHIP cards for GRE tunneling, by any chance?

    ICHIP cards do not support inline GRE reassembly, only Trio cards do.

     

    HTH

    Thx

    Alex

     

     

     

     

     

     



  • 12.  RE: MX L2-over-GRE MTU issues

    Posted 01-06-2020 02:33

    @aarseniev wrote:

    Hello,

     


    @Lyubomir wrote:

    Already tried allow-fragmentation and do-not-fragment tunnel options, but nothing changes.

     



    You need to use 2 knobs together: "allow-fragmentation" and "reassemble-packets" 

    https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/reassemble-packets-edit-interfaces.html


    Just added in both ends "allow-fragmentation" and "reassemble-packets", but still max MTU which pass tunnel is 1468.

     

     


    @aarseniev wrote:

    One more stupid question - do You use DPC/I-CHIP cards for GRE tunneling, by any chance?

    ICHIP cards do not support inline GRE reassembly, only Trio cards do.


    Router 1 is with MPC 3D 16x 10GE

    Router 2 is MX80 with integrated 4 x 10GE