Junos OS

Expand all | Collapse all

Strange packet drops on EX4550 switch interface

  • 1.  Strange packet drops on EX4550 switch interface

    Posted 14 days ago
    Hello good folks

    I have been observing packet drops on one of the switches, the setup is as

    [EX4550] ge-0/0/25--------------ge-0/1/2[EX2200]ge-0/0/47-------[Hypervisor]

    The drops are observed on port ge-0/0/25 of the EX4550, no drops on both involved ports of the EX2200.
    Utilization on the link between the switches is hardly over 200Mbps but according to Juniper's documentations

    Drops :
    Number of packets dropped by the input queue of the I/O Manager ASIC. If the interface is saturated, this number increments once for every packet that is dropped by the ASIC's RED mechanism

    Any ideas on where to look further?

    FYI The last I cleared below statistics was 24 hours ago and there are no any other errors except for drops in the output of show interface extensive so to avoid making the output long I filtered to match the keyword drops

    EX4550# run show interfaces ge-0/0/25 extensive | match "drops: [1-9]"
    Jun 10 09:27:47
    Carrier transitions: 0, Errors: 0, Drops: 2189555101, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
    
    
    EX2200# run show interfaces ge-0/1/2 extensive | match drops    
    Jun 10 12:28:54
        Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
        Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
    
    
    [edit]
    EX2200# run show interfaces ge-0/0/47 extensive | match drops             
    Jun 10 12:29:06
        Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
        Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
    
    


    ------------------------------
    Paul Basondole
    ------------------------------


  • 2.  RE: Strange packet drops on EX4550 switch interface

     
    Posted 14 days ago
    Do you have any Class of service configured on the switch?  Some errors in that might cause this issue.

    Also confirm the interface is successfully full duplex
     show interfaces ge-0/0/25 media

    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
    http://puluka.com/home
    ------------------------------



  • 3.  RE: Strange packet drops on EX4550 switch interface

    Posted 14 days ago
    Hello,

    Same here, no Class Of Service configured.

    Thank you.

    ------------------------------
    Vladimeri Benashvili
    ------------------------------



  • 4.  RE: Strange packet drops on EX4550 switch interface

    Posted 13 days ago
    Hello Steve

    Both interfaces are full duplex and there is no QOS configuration in place

    EX4550# run show interfaces ge-0/0/25 media 
    Jun 10 14:34:19
    Physical interface: ge-0/0/25, Enabled, Physical link is Up
      Interface index: 179, SNMP ifIndex: 613
      Description: Connecting to ge-0/1/2 SNH-DC EX2200
      Link-level type: Ethernet, MTU: 9216, LAN-PHY mode, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
      Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online, Media type: Fiber
      Device flags   : Present Running
      Interface flags: SNMP-Traps Internal: 0x0
      Link flags     : None
      CoS queues     : 8 supported, 8 maximum usable queues
      Current address: f4:a7:39:19:72:5c, Hardware address: f4:a7:39:19:72:5c
      Last flapped   : 2021-05-25 10:24:18 UTC (2w2d 04:10 ago)
      Input rate     : 107230384 bps (19010 pps)
      Output rate    : 117543616 bps (19467 pps)
      Active alarms  : None
      Active defects : None                 
      MAC statistics:
        Input bytes: 1288462578494, Input packets: 2185867995, Output bytes: 2032491575732, Output packets: 2484030690
      Autonegotiation information:
        Negotiation status: Complete
        Link partner:
            Link mode: Full-duplex, Flow control: Symmetric, Remote fault: OK, Link partner Speed: 1000 Mbps
        Local resolution:
            Flow control: Symmetric, Remote fault: Link OK
      Interface transmit statistics: Disabled
    
    
    {master:0}[edit]
    paul@SNH-DC-SW-EX4550# show interfaces ge-0/0/25
    Jun 10 14:34:40 
    description "Connecting to ge-0/1/2 SNH-DC EX2200";
    mtu 9216;
    unit 0 {
        family ethernet-switching {
            port-mode trunk;
            vlan {
                members all;
            }
        }
    }
    
    
    {master:0}[edit]
    paul@SNH-DC-SW-EX4550# show class-of-service 
    Jun 10 14:34:47
    
    
    {master:0}[edit]
    paul@SNH-DC-SW-EX4550# 
    
    ----------------------------------------------------------------------------------------
    
    EX2200# run show interfaces ge-0/1/2 media
    Jun 10 17:36:21
    Physical interface: ge-0/1/2, Enabled, Physical link is Up
    Interface index: 181, SNMP ifIndex: 607
    Description: Connecting to ge-0/0/25 SNH-DC EX4550
    Link-level type: Ethernet, MTU: 9216, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
    Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online
    Device flags : Present Running
    Interface flags: SNMP-Traps Internal: 0x4000
    Link flags : None
    CoS queues : 8 supported, 8 maximum usable queues
    Current address: 54:e0:32:8f:24:35, Hardware address: 54:e0:32:8f:24:35
    Last flapped : 2021-05-25 13:24:20 EAT (2w2d 04:12 ago)
    Input rate : 148668688 bps (23979 pps)
    Output rate : 153749200 bps (24391 pps)
    Active alarms : None
    Active defects : None
    MAC statistics:
    Input bytes: 8193384421336, Input packets: 10150750028, Output bytes: 4458262288526, Output packets: 8490658783
    Autonegotiation information:
    Negotiation status: Complete
    Link partner:
    Link mode: Full-duplex, Flow control: Symmetric, Remote fault: OK, Link partner Speed: 1000 Mbps
    Local resolution:
    Flow control: Symmetric, Remote fault: Link OK
    Interface transmit statistics: Disabled
    


    ------------------------------
    Paul Basondole
    JNCIP ENT/SP
    ------------------------------



  • 5.  RE: Strange packet drops on EX4550 switch interface

    Posted 13 days ago
    Although class-of-service isn't explicitly configured, there are default settings that you need to check. I'm not sure what the default profile is on a 4550 but other EX platforms have as little as 5% allocated for best-effort. If you check the extensive output of the interface, you can see the allocations. 

      Egress queues: 12 supported, 5 in use
      Queue counters:       Queued packets  Transmitted packets      Dropped packets
        0                                0                    0                    0
        3                                0                    0                    0
        4                                0                    0                    0
        7                                0                    0                    0
        8                                0                    0                    0
      Queue number:         Mapped forwarding classes
        0                   best-effort
        3                   fcoe
        4                   no-loss
        7                   network-control
        8                   mcast​
    
    <omitted>
    
      CoS information:
        Direction : Output                  
        CoS transmit queue               Bandwidth               Buffer Priority   Limit
                                  %            bps     %           usec
        0 best-effort             5       50000000     5              0      low    none
        3 fcoe                   35      350000000    35              0      low    none
        4 no-loss                35      350000000    35              0      low    none
        7 network-control         5       50000000     5              0      low    none
        8 mcast                  20      200000000    20              0      low    none


    ------------------------------
    DANIEL HEARTY
    Principal Engineer
    ------------------------------



  • 6.  RE: Strange packet drops on EX4550 switch interface

    Posted 13 days ago
    Hello Daniel

    The output shows queue 0 has 95% of the bandwidth that's 950Mbps

    EX4550> show interfaces ge-0/0/25 extensive 
    Physical interface: ge-0/0/25, Enabled, Physical link is Up
      Interface index: 179, SNMP ifIndex: 613, Generation: 240
      Description: Connecting to ge-0/1/2 SNH-DC EX2200
      Link-level type: Ethernet, MTU: 9216, LAN-PHY mode, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
      Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online, Media type: Fiber
      Device flags   : Present Running
      Interface flags: SNMP-Traps Internal: 0x0
      Link flags     : None
      CoS queues     : 8 supported, 8 maximum usable queues
      Hold-times     : Up 0 ms, Down 0 ms
      Current address: f4:a7:39:19:72:5c, Hardware address: f4:a7:39:19:72:5c
      Last flapped   : 2021-05-25 10:24:18 UTC (2w2d 22:01 ago)
      Statistics last cleared: 2021-06-09 07:26:09 UTC (2d 00:59 ago)
      Traffic statistics:
       Input  bytes  :        2025176413133            112646992 bps
       Output bytes  :        3030985767713            138822288 bps
       Input  packets:           3282596060                20442 pps
       Output packets:           3700170748                22932 pps
       IPv6 transit statistics:
        Input  bytes  :                   0
        Output bytes  :                   0
        Input  packets:                   0
        Output packets:                   0
      Input errors:
        Errors: 1, Drops: 0, Framing errors: 1, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
      Output errors:
        Carrier transitions: 0, Errors: 0, Drops: 3715117227, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
      Egress queues: 8 supported, 4 in use
      Queue counters:       Queued packets  Transmitted packets      Dropped packets
        0                                0           3697137895           3715117226
        1                                0                    0                    0
        5                                0                    0                    0
        7                                0              3021218                    1
      Queue number:         Mapped forwarding classes
        0                   best-effort
        1                   assured-forwarding
        5                   expedited-forwarding
        7                   network-control
      Active alarms  : None
      Active defects : None
      MAC statistics:                      Receive         Transmit
        Total octets                 2025161142308    3030966948474
        Total packets                   3282573890       3700145877
        Unicast packets                 3240317648       3587342545
        Broadcast packets                  2144316         15561259
        Multicast packets                 40111926         97242073
        CRC/Align errors                         1                0
        FIFO errors                              0                0
        MAC control frames                       0                0
        MAC pause frames                         0                0
        Oversized frames                         0
        Jabber frames                            0
        Fragment frames                          0
        Code violations                          0
      Autonegotiation information:
        Negotiation status: Complete
        Link partner:                       
            Link mode: Full-duplex, Flow control: Symmetric, Remote fault: OK, Link partner Speed: 1000 Mbps
        Local resolution:
            Flow control: Symmetric, Remote fault: Link OK
      Packet Forwarding Engine configuration:
        Destination slot: 0
      CoS information:
        Direction : Output
        CoS transmit queue               Bandwidth               Buffer Priority   Limit
                                  %            bps     %           usec
        0 best-effort            95      950000000    95             NA      low    none
        7 network-control         5       50000000     5             NA      low    none
      Interface transmit statistics: Disabled
      MACSec statistics:
        Output                              
            Secure Channel Transmitted
            Protected Packets               : 0
            Encrypted Packets               : 0
            Protected Bytes                 : 0
            Encrypted Bytes                 : 0
         Input
            Secure Channel Received
            Accepted Packets                : 0
            Validated Bytes                 : 0
            Decrypted Bytes                 : 0
    
      Logical interface ge-0/0/25.0 (Index 106) (SNMP ifIndex 614) (Generation 315)
        Flags: Up SNMP-Traps 0x0 Encapsulation: ENET2
        Traffic statistics:                 
         Input  bytes  :            557796747
         Output bytes  :             13318470
         Input  packets:              1365819
         Output packets:               107405
        Local statistics:
         Input  bytes  :            557796747
         Output bytes  :             13318470
         Input  packets:              1365819
         Output packets:               107405
        Transit statistics:
         Input  bytes  :                    0                    0 bps
         Output bytes  :                    0                    0 bps
         Input  packets:                    0                    0 pps
         Output packets:                    0                    0 pps
        Protocol eth-switch, Generation: 334, Route table: 0
          Flags: Trunk-Mode
    
    {master:0}
    paul@SNH-DC-SW-EX4550> ​


    ------------------------------
    Paul Basondole
    JNCIP ENT/SP
    ------------------------------



  • 7.  RE: Strange packet drops on EX4550 switch interface

    Posted 13 days ago
    Alright, so we can see that traffic is indeed being dropped in the BE queue. Have you had any micro-burst events in your network?  Is the drop counter clocking?

    ------------------------------
    DANIEL HEARTY
    Principal Engineer
    ------------------------------



  • 8.  RE: Strange packet drops on EX4550 switch interface

    Posted 12 days ago
    Yes the drop count is clocking. I have tried to add another connection between the switches and created a 2G LAG​ but the EX4550 is still dropping traffic. Utilization at this time is less than 100Mbps. Not sure how I would check for micro-burst

    ------------------------------
    Paul Basondole
    JNCIP ENT/SP
    ------------------------------