Yes the drop count is clocking. I have tried to add another connection between the switches and created a 2G LAG but the EX4550 is still dropping traffic. Utilization at this time is less than 100Mbps. Not sure how I would check for micro-burst
Original Message:
Sent: 06-11-2021 08:52
From: DANIEL HEARTY
Subject: Strange packet drops on EX4550 switch interface
Alright, so we can see that traffic is indeed being dropped in the BE queue. Have you had any micro-burst events in your network? Is the drop counter clocking?
------------------------------
DANIEL HEARTY
Principal Engineer
Original Message:
Sent: 06-11-2021 04:31
From: Paul Basondole
Subject: Strange packet drops on EX4550 switch interface
Hello Daniel
The output shows queue 0 has 95% of the bandwidth that's 950Mbps
EX4550> show interfaces ge-0/0/25 extensive Physical interface: ge-0/0/25, Enabled, Physical link is Up Interface index: 179, SNMP ifIndex: 613, Generation: 240 Description: Connecting to ge-0/1/2 SNH-DC EX2200 Link-level type: Ethernet, MTU: 9216, LAN-PHY mode, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online, Media type: Fiber Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x0 Link flags : None CoS queues : 8 supported, 8 maximum usable queues Hold-times : Up 0 ms, Down 0 ms Current address: f4:a7:39:19:72:5c, Hardware address: f4:a7:39:19:72:5c Last flapped : 2021-05-25 10:24:18 UTC (2w2d 22:01 ago) Statistics last cleared: 2021-06-09 07:26:09 UTC (2d 00:59 ago) Traffic statistics: Input bytes : 2025176413133 112646992 bps Output bytes : 3030985767713 138822288 bps Input packets: 3282596060 20442 pps Output packets: 3700170748 22932 pps IPv6 transit statistics: Input bytes : 0 Output bytes : 0 Input packets: 0 Output packets: 0 Input errors: Errors: 1, Drops: 0, Framing errors: 1, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0 Output errors: Carrier transitions: 0, Errors: 0, Drops: 3715117227, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0 Egress queues: 8 supported, 4 in use Queue counters: Queued packets Transmitted packets Dropped packets 0 0 3697137895 3715117226 1 0 0 0 5 0 0 0 7 0 3021218 1 Queue number: Mapped forwarding classes 0 best-effort 1 assured-forwarding 5 expedited-forwarding 7 network-control Active alarms : None Active defects : None MAC statistics: Receive Transmit Total octets 2025161142308 3030966948474 Total packets 3282573890 3700145877 Unicast packets 3240317648 3587342545 Broadcast packets 2144316 15561259 Multicast packets 40111926 97242073 CRC/Align errors 1 0 FIFO errors 0 0 MAC control frames 0 0 MAC pause frames 0 0 Oversized frames 0 Jabber frames 0 Fragment frames 0 Code violations 0 Autonegotiation information: Negotiation status: Complete Link partner: Link mode: Full-duplex, Flow control: Symmetric, Remote fault: OK, Link partner Speed: 1000 Mbps Local resolution: Flow control: Symmetric, Remote fault: Link OK Packet Forwarding Engine configuration: Destination slot: 0 CoS information: Direction : Output CoS transmit queue Bandwidth Buffer Priority Limit % bps % usec 0 best-effort 95 950000000 95 NA low none 7 network-control 5 50000000 5 NA low none Interface transmit statistics: Disabled MACSec statistics: Output Secure Channel Transmitted Protected Packets : 0 Encrypted Packets : 0 Protected Bytes : 0 Encrypted Bytes : 0 Input Secure Channel Received Accepted Packets : 0 Validated Bytes : 0 Decrypted Bytes : 0 Logical interface ge-0/0/25.0 (Index 106) (SNMP ifIndex 614) (Generation 315) Flags: Up SNMP-Traps 0x0 Encapsulation: ENET2 Traffic statistics: Input bytes : 557796747 Output bytes : 13318470 Input packets: 1365819 Output packets: 107405 Local statistics: Input bytes : 557796747 Output bytes : 13318470 Input packets: 1365819 Output packets: 107405 Transit statistics: Input bytes : 0 0 bps Output bytes : 0 0 bps Input packets: 0 0 pps Output packets: 0 0 pps Protocol eth-switch, Generation: 334, Route table: 0 Flags: Trunk-Mode{master:0}paul@SNH-DC-SW-EX4550>
------------------------------
Paul Basondole
JNCIP ENT/SP
Original Message:
Sent: 06-10-2021 10:49
From: DANIEL HEARTY
Subject: Strange packet drops on EX4550 switch interface
Although class-of-service isn't explicitly configured, there are default settings that you need to check. I'm not sure what the default profile is on a 4550 but other EX platforms have as little as 5% allocated for best-effort. If you check the extensive output of the interface, you can see the allocations.
Egress queues: 12 supported, 5 in use Queue counters: Queued packets Transmitted packets Dropped packets 0 0 0 0 3 0 0 0 4 0 0 0 7 0 0 0 8 0 0 0 Queue number: Mapped forwarding classes 0 best-effort 3 fcoe 4 no-loss 7 network-control 8 mcast<omitted> CoS information: Direction : Output CoS transmit queue Bandwidth Buffer Priority Limit % bps % usec 0 best-effort 5 50000000 5 0 low none 3 fcoe 35 350000000 35 0 low none 4 no-loss 35 350000000 35 0 low none 7 network-control 5 50000000 5 0 low none 8 mcast 20 200000000 20 0 low none
------------------------------
DANIEL HEARTY
Principal Engineer
Original Message:
Sent: 06-10-2021 10:40
From: Paul Basondole
Subject: Strange packet drops on EX4550 switch interface
Hello Steve
Both interfaces are full duplex and there is no QOS configuration in place
EX4550# run show interfaces ge-0/0/25 media Jun 10 14:34:19Physical interface: ge-0/0/25, Enabled, Physical link is Up Interface index: 179, SNMP ifIndex: 613 Description: Connecting to ge-0/1/2 SNH-DC EX2200 Link-level type: Ethernet, MTU: 9216, LAN-PHY mode, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: Online, Media type: Fiber Device flags : Present Running Interface flags: SNMP-Traps Internal: 0x0 Link flags : None CoS queues : 8 supported, 8 maximum usable queues Current address: f4:a7:39:19:72:5c, Hardware address: f4:a7:39:19:72:5c Last flapped : 2021-05-25 10:24:18 UTC (2w2d 04:10 ago) Input rate : 107230384 bps (19010 pps) Output rate : 117543616 bps (19467 pps) Active alarms : None Active defects : None MAC statistics: Input bytes: 1288462578494, Input packets: 2185867995, Output bytes: 2032491575732, Output packets: 2484030690 Autonegotiation information: Negotiation status: Complete Link partner: Link mode: Full-duplex, Flow control: Symmetric, Remote fault: OK, Link partner Speed: 1000 Mbps Local resolution: Flow control: Symmetric, Remote fault: Link OK Interface transmit statistics: Disabled{master:0}[edit]paul@SNH-DC-SW-EX4550# show interfaces ge-0/0/25Jun 10 14:34:40 description "Connecting to ge-0/1/2 SNH-DC EX2200";mtu 9216;unit 0 { family ethernet-switching { port-mode trunk; vlan { members all; } }}{master:0}[edit]paul@SNH-DC-SW-EX4550# show class-of-service Jun 10 14:34:47{master:0}[edit]paul@SNH-DC-SW-EX4550# ----------------------------------------------------------------------------------------EX2200# run show interfaces ge-0/1/2 mediaJun 10 17:36:21Physical interface: ge-0/1/2, Enabled, Physical link is UpInterface index: 181, SNMP ifIndex: 607Description: Connecting to ge-0/0/25 SNH-DC EX4550Link-level type: Ethernet, MTU: 9216, Speed: 1000mbps, Duplex: Full-Duplex, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,Flow control: Enabled, Auto-negotiation: Enabled, Remote fault: OnlineDevice flags : Present RunningInterface flags: SNMP-Traps Internal: 0x4000Link flags : NoneCoS queues : 8 supported, 8 maximum usable queuesCurrent address: 54:e0:32:8f:24:35, Hardware address: 54:e0:32:8f:24:35Last flapped : 2021-05-25 13:24:20 EAT (2w2d 04:12 ago)Input rate : 148668688 bps (23979 pps)Output rate : 153749200 bps (24391 pps)Active alarms : NoneActive defects : NoneMAC statistics:Input bytes: 8193384421336, Input packets: 10150750028, Output bytes: 4458262288526, Output packets: 8490658783Autonegotiation information:Negotiation status: CompleteLink partner:Link mode: Full-duplex, Flow control: Symmetric, Remote fault: OK, Link partner Speed: 1000 MbpsLocal resolution:Flow control: Symmetric, Remote fault: Link OKInterface transmit statistics: Disabled
------------------------------
Paul Basondole
JNCIP ENT/SP
Original Message:
Sent: 06-10-2021 05:57
From: STEVE PULUKA
Subject: Strange packet drops on EX4550 switch interface
Do you have any Class of service configured on the switch? Some errors in that might cause this issue.
Also confirm the interface is successfully full duplex
show interfaces ge-0/0/25 media
------------------------------
Steve Puluka BSEET - Juniper Ambassador
IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
http://puluka.com/home
Original Message:
Sent: 06-10-2021 05:32
From: Paul Basondole
Subject: Strange packet drops on EX4550 switch interface
Hello good folks
I have been observing packet drops on one of the switches, the setup is as
[EX4550] ge-0/0/25--------------ge-0/1/2[EX2200]ge-0/0/47-------[Hypervisor]
The drops are observed on port ge-0/0/25 of the EX4550, no drops on both involved ports of the EX2200.
Utilization on the link between the switches is hardly over 200Mbps but according to Juniper's documentations
Drops :
Number of packets dropped by the input queue of the I/O Manager ASIC. If the interface is saturated, this number increments once for every packet that is dropped by the ASIC's RED mechanism
Any ideas on where to look further?
FYI The last I cleared below statistics was 24 hours ago and there are no any other errors except for drops in the output of show interface extensive so to avoid making the output long I filtered to match the keyword drops
EX4550# run show interfaces ge-0/0/25 extensive | match "drops: [1-9]"Jun 10 09:27:47Carrier transitions: 0, Errors: 0, Drops: 2189555101, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0EX2200# run show interfaces ge-0/1/2 extensive | match drops Jun 10 12:28:54 Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0 Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0[edit]EX2200# run show interfaces ge-0/0/47 extensive | match drops Jun 10 12:29:06 Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0 Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
------------------------------
Paul Basondole
------------------------------