Routing

 View Only
last person joined: 2 days ago 

Ask questions and share experiences about ACX Series, CTP Series, MX Series, PTX Series, SSR Series, JRR Series, and all things routing, including portfolios and protocols.
Expand all | Collapse all

How to stop Tail drop and Red Drop in MX204?

  • 1.  How to stop Tail drop and Red Drop in MX204?

    Posted 02-06-2023 04:50
    Hello my friends,

    I noticed recently that my Juniper MX204 et interface has some tail drop and red drop as image below:



    I have also checked my class-of-service drop-profile and found:
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
    Fill level                         Drop probability
    100                                  100

    I have not set up any Fill Level on my MX204, I know that if the queue buffer size fill level is 100, the probability of packet drop is also 100%. We use MX204 as a BNG and noticed that some customers are complaining for packet loss and low bandwidth. 

    What will be the possible solution? If anybody can guide me please.

    Thanks

    ------------------------------
    Nazmus Sakib
    ------------------------------


  • 2.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-06-2023 10:09
    Hello Sakib, 

    By default, even if you do not configure COS on your device, RED is still in effect and functions as the primary mechanism for managing congestion. In the default RED drop profile, when the fill level is 0 percent, the drop probability is 0 percent. When the fill level is 100 percent, the drop probability is 100 percent. The TAIL drop profile is also active by default acting as backup mechanisms to tackle congestion with the same principles of 0 % fill level means 0% drop probability and 100% fill level suggesting 100 % drop probability . When you have packets enqueued which is more than you could send, you might see RED drop. If the enqueueing packets come too fast and it even can't be enqueued (ie. the current queue size is reaching max), you will see tail drop.

     
    In the situation when you RED and/or TAIL drops are observed on an interface and no COS configuration is made on the box you have a number of solutions:
     
    • Increase bandwidth on egress interfaces - this can be done by replacing the interface where you observe the drops with an AE bundle
    • Use traffic shaping to throttle traffic from the source
    • Configure a classifier to put mission critical applications data to a queue other than the best effort queue. Then, allocate enough bandwidth and buffer to that queue. This effectively allows less drops on mission critical applications data at the expense of other application data that are using the best effort queue
    Thanks 
    Ruban Johnson

    ------------------------------
    RUBAN JOHNSON
    ------------------------------



  • 3.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-07-2023 12:18
    Thanks Ruban,
    Our configuration is already on et interface and the pic has been configured at 40G. We are using CoS shaping on MX204.
    One thing I doubt that we recently changed our whole FPC, PIC settings on MX204. Among the four on PIC0, port0: 40G, Port1: 40G, Port2: 100G and port3: 100G
    Although we are not using 100G ports at this moment.
    If we use classifier or policer COS shaping on QFX instead of MX204, is there any improvement can we expect?

    Thanks for your cooperation

    ------------------------------
    Nazmus Sakib
    ------------------------------



  • 4.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-08-2023 07:10
    Hello Sakib, 

    Can you please provide your cos and the interface configurations ? 

    i dont see the drops are regular as we dont see any at the time you have taken the output. Do you see your interface graphs for any micro bursts happening ?

    Do you see drops only at best effort or its happening in all your queues ?

    If we use classifier or policer COS shaping on QFX instead of MX204, is there any improvement can we expect?

    yes, you can try that as well 

    Thanks 
    Ruban Johnson


    ------------------------------
    RUBAN JOHNSON
    ------------------------------



  • 5.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-10-2023 06:25

    This is the interface which is used for customer downlink to Juniper QFX5110:

    show configuration interfaces et-0/0/0
    description "DOWNLINK SW2 et-0/0/48 40Gb";
    enable;
    traceoptions {
        flag all;
    }
    hierarchical-scheduler maximum-hierarchy-levels 3 implicit-hierarchy;
    flexible-vlan-tagging;
    auto-configure {
        stacked-vlan-ranges {
            dynamic-profile vlan-ps {
                accept pppoe;
                ranges {
                    any,any;
                }
            }
        }
        vlan-ranges {
            dynamic-profile vlan-ps {
                accept pppoe;
                ranges {
                    any;
                }
            }
        }
        remove-when-no-subscribers;
    }
    mtu 2000;
    encapsulation flexible-ethernet-services;
    unit 1 {
        vlan-id 1;
        family inet {
            address 192.168.100.2/24;
        }
        family iso;
    }
    unit 202 {
        encapsulation vlan-bridge;
        vlan-tags outer 202 inner 4093;
    }
    unit 203 {
        encapsulation vlan-bridge;
        vlan-tags outer 203 inner 4093;
    }
    unit 4093 {
        encapsulation vlan-bridge;
        vlan-id 4093;
    }

    We have configured Class of Service under the dynamic profile like that:

    class-of-service {
                traffic-control-profiles {
                    50m {
                        shaping-rate 50m;
                    }
                    200m {
                        shaping-rate 200m;
                    }
                    1g {
                        shaping-rate 2g;
                    }
                    100m {
                        shaping-rate 110m;
                    }
                    250m {
                        shaping-rate 310m;
                    }
                    300m {
                        shaping-rate 310m;
                    }
                    10g {
                        shaping-rate 12g;
                    }



    ------------------------------
    Nazmus Sakib
    ------------------------------



  • 6.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-10-2023 11:26

    Hello Sakib, 

    So i understand this is a new 40g link, dont we need a new traffic-control-profile for them with a shaping rate ? i do see for 10g but nothing added for 40g ? whats been added for this interface ? you can check with the below command 

    show class-of-service interface et-0/0/0

    show class-of-service interface et-0/0/0 detail 

    Thanks

    Ruban 



    ------------------------------
    RUBAN JOHNSON
    ------------------------------



  • 7.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-13-2023 08:16

     show class-of-service interface et-0/0/0
    Physical interface: et-0/0/0, Index: 150
    Maximum usable queues: 8, Queues in use: 4
    Exclude aggregate overhead bytes: disabled
    Logical interface aggregate statistics: disabled
      Scheduler map: <default>, Index: 2
      Congestion-notification: Disabled

      Logical interface: et-0/0/0.1, Index: 334
    Object                  Name                   Type                    Index
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: et-0/0/0.202, Index: 335

      Logical interface: et-0/0/0.203, Index: 336

      Logical interface: et-0/0/0.32767, Index: 338
    Object                  Name                   Type                    Index
    Traffic-control-profile __control_tc_prof      Output                  45866

      Logical interface: et-0/0/0.4093, Index: 337

      Logical interface: pp0.3221308416, Index: 3221308416
    Object                  Name                   Type                    Index
    Traffic-control-profile 300m_UID1201           Output              4294970596
    Scheduler-map           <default>              Output              4294967298
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: pp0.3221308419, Index: 3221308419
    Object                  Name                   Type                    Index
    Traffic-control-profile 100m_UID1199           Output              4294970601
    Scheduler-map           <default>              Output              4294967298
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: pp0.3221308422, Index: 3221308422
    Object                  Name                   Type                    Index
    Traffic-control-profile 1g_UID1198             Output              4294970599
    Scheduler-map           <default>              Output              4294967298
    Classifier              dscp-ipv6-compatibility dscp-ipv6                  9
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: pp0.3221308423, Index: 3221308423
    Object                  Name                   Type                    Index
    Traffic-control-profile 300m_UID1201           Output              4294970596
    Scheduler-map           <default>              Output              4294967298
    Classifier              dscp-ipv6-compatibility dscp-ipv6                  9
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: pp0.3221308425, Index: 3221308425
    Object                  Name                   Type                    Index
    Traffic-control-profile 1g_UID1198             Output              4294970599
    Scheduler-map           <default>              Output              4294967298
    Classifier              dscp-ipv6-compatibility dscp-ipv6                  9
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: pp0.3221308435, Index: 3221308435
    Object                  Name                   Type                    Index
    Traffic-control-profile 1g_UID1198             Output              4294970599
    Scheduler-map           <default>              Output              4294967298
    Classifier              dscp-ipv6-compatibility dscp-ipv6                  9
    Classifier              ipprec-compatibility   ip                         13

    show class-of-service interface et-0/0/0 detail
    Physical interface: et-0/0/0, Enabled, Physical link is Up
      Link-level type: Flexible-Ethernet, MTU: 2000, MRU: 2008, Speed: 40Gbps, Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled
      Device flags   : Present Running
      Interface flags: SNMP-Traps Internal: 0x4000

    Physical interface: et-0/0/0, Index: 150
    Maximum usable queues: 8, Queues in use: 4
    Exclude aggregate overhead bytes: disabled
    Logical interface aggregate statistics: disabled
      Scheduler map: <default>, Index: 2
      Congestion-notification: Disabled

      Logical interface: pp0.3221308416, Index: 3221308416
    Object                  Name                   Type                    Index
    Traffic-control-profile 300m_UID1201           Output              4294970596
    Scheduler-map           <default>              Output              4294967298
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: pp0.3221308419, Index: 3221308419
    Object                  Name                   Type                    Index
    Traffic-control-profile 100m_UID1199           Output              4294970601
    Scheduler-map           <default>              Output              4294967298
    Classifier              ipprec-compatibility   ip                         13

      Logical interface: pp0.3221308422, Index: 3221308422
    Object                  Name                   Type                    Index
    Traffic-control-profile 1g_UID1198             Output              4294970599
    Scheduler-map           <default>              Output              4294967298
    Classifier              dscp-ipv6-compatibility dscp-ipv6                  9
    Classifier              ipprec-compatibility   ip                         13



    ------------------------------
    Nazmus Sakib
    ------------------------------



  • 8.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-13-2023 09:30

    I wasn't expecting it to be a BNG from your first post!

    The "50m" shaping rate on a subscriber session will contribute drops to the parent interface when that single customer is receiving more than 50Mbps. Same goes for every other subscriber session on the interface. Well, it would for "normal" IFLs, I assume it does for ppp ones too?




  • 9.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 03-09-2023 08:16

    Hi Eusa,

    Thank you for your reply. However, in addition to predefined default shaping we specify our customers through RADIUS and Juniper Dynamic Profile COS shaping as below:


    PPP {
        predefined-variable-defaults {
            cos-traffic-control-profile 50m;
            input-filter block-WAN4;
            input-ipv6-filter block-WAN6;


    This is the shaping on dynamic profile:
    class-of-service {
            traffic-control-profiles {
                50m {
                    shaping-rate 50m;
                }
                200m {
                    shaping-rate 200m;
                }
                1g {
                    shaping-rate 2g;
                }
                100m {
                    shaping-rate 110m;
                }
                250m {
                    shaping-rate 310m;
                }
                300m {
                    shaping-rate 310m;
                }
                10g {
                    shaping-rate 12g;

    My question is that why individual customer would hit the 50m predefined shaping instead of their own in both RADIUS and BNG?

    Thanks



    ------------------------------
    Nazmus Sakib
    ------------------------------



  • 10.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-07-2023 17:08

    Hello Ruban,

    I work with Sakib.
    We dont see the interface reaching full utilisation, about 12G peak times, normally floats around 2G-4G depending on time of day. Even during normal times we see tail drops. Sure I know we can add CoS policies to limit the type of traffic affected however why would there would be such a low buffer limit by default when configuring a 40G interface? I would've thought the buffer would scale with theese interfaces. Do you know what else I could check or configure?


    Here is some stuff I have seen, also attached.

    show class-of-service interface et-0/0/0 comprehensive
    
    Physical interface: et-0/0/0, Enabled, Physical link is Up
      Interface index: 150, SNMP ifIndex: 594, Generation: 263
      Description: DOWNLINK SW2 et-0/0/48 40Gb
      Link-level type: Flexible-Ethernet, MTU: 2000, MRU: 2008, Speed: 40Gbps, BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled,
      Source filtering: Disabled, Flow control: Enabled
      Pad to minimum frame size: Disabled
      Device flags   : Present Running
      Interface flags: SNMP-Traps Internal: 0x4000
      CoS queues     : 8 supported, 8 maximum usable queues
      Schedulers     : 0
      Hold-times     : Up 0 ms, Down 0 ms
      Damping        : half-life: 0 sec, max-suppress: 0 sec, reuse: 0, suppress: 0, state: unsuppressed
      Current address: #### , Hardware address: ####
      Last flapped   : 2023-01-21 03:26:32 GMT (2w2d 07:59 ago)
      Statistics last cleared: 2023-01-31 14:48:11 GMT (5d 20:37 ago)
      Traffic statistics:
       Input  bytes  :       32143250474963            561814824 bps
       Output bytes  :      225109175654022           2054957568 bps
       Input  packets:          77302980118               128482 pps
       Output packets:         181617769082               232346 pps
       IPv6 transit statistics:
       Input  bytes  :                    0
       Output bytes  :       73211242867684
       Input  packets:                    0
       Output packets:          56356562293
      Dropped traffic statistics due to STP State:
       Input  bytes  :                    0
       Output bytes  :                    0
       Input  packets:                    0
       Output packets:                    0
      Input errors:
        Errors: 1352, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 1352, L2 channel errors: 0, L2 mismatch timeouts: 0,
        FIFO errors: 0, Resource errors: 0
      Output errors:
        Carrier transitions: 0, Errors: 0, Drops: 1849276, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 225161,
        Resource errors: 0
      Egress queues: 8 supported, 4 in use
      Queue counters:       Queued packets  Transmitted packets      Dropped packets
        0                     181545293744         181543444468              1849276
        1                                0                    0                    0
        2                                0                    0                    0
        3                         74633259             74633259                    0
      Queue number:         Mapped forwarding classes
        0                   best-effort
        1                   expedited-forwarding
        2                   assured-forwarding
        3                   network-control
      Active alarms  : None
      Active defects : None
      PCS statistics                      Seconds
        Bit errors                             0
        Errored blocks                         0
      Ethernet FEC Mode  :                   NONE
      Ethernet FEC statistics              Errors
        FEC Corrected Errors                    0
        FEC Uncorrected Errors                  0
        FEC Corrected Errors Rate               0
        FEC Uncorrected Errors Rate             0
      MAC statistics:                      Receive         Transmit
        Total octets                34313549838910  230559055235335
        Total packets                  77303043279     181617879070
        Unicast packets                77301152262     181617294980
        Broadcast packets                  1341850           423770
        Multicast packets                   549166           160327
        CRC/Align errors                         0                0
        FIFO errors                              0                0
        MAC control frames                       0                0
        MAC pause frames                         0                0
        Oversized frames                         0
        Jabber frames                            0
        Fragment frames                          0
        VLAN tagged frames             77303025037
        Code violations                          0
        Total errors                             0                0
      Filter statistics:
        Input packet count             77303218480
        Input packet rejects               1253138
        Input DA rejects                      2439
        Input SA rejects                         0
        Output packet count                            181618283774
        Output packet pad count                                   0
        Output packet error count                                 0
        CAM destination filters: 0, CAM source filters: 0
      Packet Forwarding Engine configuration:
        Destination slot: 0 (0x00)
      CoS information:
        Direction : Output
        CoS transmit queue               Bandwidth               Buffer Priority   Limit
                                  %            bps     %           usec
        0 best-effort            95    38000000000    95              0      low    none
        3 network-control         5     2000000000     5              0      low    none
      Preclassifier statistics:
        Traffic Class        Received Packets   Transmitted  Packets      Dropped Packets
        real-time                           0                      0                    0
        network-control           77296688593            77296688593                    0
        best-effort                   6295842                6295842                    0
      Interface transmit statistics: Disabled
    
    Physical interface: et-0/0/0, Enabled, Physical link is Up
      Interface index: 150, SNMP ifIndex: 594
      Description: DOWNLINK SW2 et-0/0/48 40Gb
    Forwarding classes: 16 supported, 4 in use
    Egress queues: 8 supported, 4 in use
    Queue: 0, Forwarding classes: best-effort
      Queued:
        Packets              :          181545293744                245600 pps
        Bytes                :       234171592339270            2267774080 bps
      Transmitted:
        Packets              :          181543444468                245600 pps
        Bytes                :       234168845377843            2267774080 bps
        Tail-dropped packets :               1833989                     0 pps
        RL-dropped packets   :                     0                     0 pps
        RL-dropped bytes     :                     0                     0 bps
        RED-dropped packets  :                 15287                     0 pps
         Low                 :                 10056                     0 pps
         Medium-low          :                     0                     0 pps
         Medium-high         :                     0                     0 pps
         High                :                  5231                     0 pps
        RED-dropped bytes    :              22658714                     0 bps
         Low                 :              14830537                     0 bps
         Medium-low          :                     0                     0 bps
         Medium-high         :                     0                     0 bps
         High                :               7828177                     0 bps
      Queue-depth bytes      :
        Average              :                     0
        Current              :                     0
        Peak                 :                   399
        Maximum              :             478150656
    Queue: 1, Forwarding classes: expedited-forwarding
      Queued:
        Packets              :                     0                     0 pps
        Bytes                :                     0                     0 bps
      Transmitted:
        Packets              :                     0                     0 pps
        Bytes                :                     0                     0 bps
        Tail-dropped packets :                     0                     0 pps
        RL-dropped packets   :                     0                     0 pps
        RL-dropped bytes     :                     0                     0 bps
        RED-dropped packets  :                     0                     0 pps
         Low                 :                     0                     0 pps
         Medium-low          :                     0                     0 pps
         Medium-high         :                     0                     0 pps
         High                :                     0                     0 pps
        RED-dropped bytes    :                     0                     0 bps
         Low                 :                     0                     0 bps
         Medium-low          :                     0                     0 bps
         Medium-high         :                     0                     0 bps
         High                :                     0                     0 bps
      Queue-depth bytes      :
        Average              :                     0
        Current              :                     0
        Peak                 :                     0
        Maximum              :                 32768
    Queue: 2, Forwarding classes: assured-forwarding
      Queued:
        Packets              :                     0                     0 pps
        Bytes                :                     0                     0 bps
      Transmitted:
        Packets              :                     0                     0 pps
        Bytes                :                     0                     0 bps
        Tail-dropped packets :                     0                     0 pps
        RL-dropped packets   :                     0                     0 pps
        RL-dropped bytes     :                     0                     0 bps
        RED-dropped packets  :                     0                     0 pps
         Low                 :                     0                     0 pps
         Medium-low          :                     0                     0 pps
         Medium-high         :                     0                     0 pps
         High                :                     0                     0 pps
        RED-dropped bytes    :                     0                     0 bps
         Low                 :                     0                     0 bps
         Medium-low          :                     0                     0 bps
         Medium-high         :                     0                     0 bps
         High                :                     0                     0 bps
      Queue-depth bytes      :
        Average              :                     0
        Current              :                     0
        Peak                 :                     0
        Maximum              :                 32768
    Queue: 3, Forwarding classes: network-control
      Queued:
        Packets              :              74633259                     8 pps
        Bytes                :           22812826145                 38272 bps
      Transmitted:
        Packets              :              74633259                     8 pps
        Bytes                :           22812826145                 38272 bps
        Tail-dropped packets :                     0                     0 pps
        RL-dropped packets   :                     0                     0 pps
        RL-dropped bytes     :                     0                     0 bps
        RED-dropped packets  :                     0                     0 pps
         Low                 :                     0                     0 pps
         Medium-low          :                     0                     0 pps
         Medium-high         :                     0                     0 pps
         High                :                     0                     0 pps
        RED-dropped bytes    :                     0                     0 bps
         Low                 :                     0                     0 bps
         Medium-low          :                     0                     0 bps
         Medium-high         :                     0                     0 bps
         High                :                     0                     0 bps
      Queue-depth bytes      :
        Average              :                     0
        Current              :                     0
        Peak                 :                    79
        Maximum              :              25034752
    
    Physical interface: et-0/0/0, Index: 150
    Maximum usable queues: 8, Queues in use: 4
    Exclude aggregate overhead bytes: disabled
    Logical interface aggregate statistics: disabled
    
    Scheduler map: <default>, Index: 2
    
      Scheduler: <default-be>, Forwarding class: best-effort, Index: 21
        Transmit rate: 95 percent, Rate Limit: none, Buffer size: 95 percent, Buffer Limit: none, Priority: low
        Excess Priority: low
        Drop profiles:
          Loss priority   Protocol    Index    Name
          Low             any             1    <default-drop-profile>
          Medium low      any             1    <default-drop-profile>
          Medium high     any             1    <default-drop-profile>
          High            any             1    <default-drop-profile>
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
    
      Scheduler: <default-nc>, Forwarding class: network-control, Index: 23
        Transmit rate: 5 percent, Rate Limit: none, Buffer size: 5 percent, Buffer Limit: none, Priority: low
        Excess Priority: low
        Drop profiles:
          Loss priority   Protocol    Index    Name
          Low             any             1    <default-drop-profile>
          Medium low      any             1    <default-drop-profile>
          Medium high     any             1    <default-drop-profile>
          High            any             1    <default-drop-profile>
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
    Drop profile: <default-drop-profile>, Type: discrete, Index: 1
      Fill level    Drop probability
             100                 100
      Congestion-notification: Disabled
    Forwarding class                       ID      Queue  Restricted queue  Fabric priority  Policing priority   SPU priority
      best-effort                           0       0          0             low                normal            low
      expedited-forwarding                  1       1          1             low                normal            low
      assured-forwarding                    2       2          2             low                normal            low
      network-control                       3       3          3             low                normal            low
    
    



    ------------------------------
    WILLIAM ROULLIER
    ------------------------------



  • 11.  RE: How to stop Tail drop and Red Drop in MX204?

    Posted 02-08-2023 08:06
    Hello William, 

    Found the below PR's which could contribute to the Red/tail drops 

    https://prsearch.juniper.net/problemreport/PR1358405
    https://prsearch.juniper.net/problemreport/PR1386350

    Also, its best to contact Juniper TAC on such issues 

    Thanks 
    Ruban Johnson

    ------------------------------
    RUBAN JOHNSON
    ------------------------------