Routing

 View Only
last person joined: 2 days ago 

Ask questions and share experiences about ACX Series, CTP Series, MX Series, PTX Series, SSR Series, JRR Series, and all things routing, including portfolios and protocols.

  • 1.  MX480 droping all protocal connections for few mins

    Posted 06-09-2023 08:00

    Hello, 

    We have MX480 with one FPC and one RE (17.3R3.10).

    We did replace the RE from 4GB ram to 16GB ram RE and also upgraded to junos 17.3R3.10. 

    For some reason, we randomly loose connection to the MX including all BGP/ISIS/LDP. 

    ISIS adjacency is restored immediately but it takes bit of time before the MX responds to pings on even P2P IPs that don't need protocol connectivity.

    Logs show DDOS aggregate protocol violation before the flap hits.

    Jun  8 11:06:00  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  Sample:syslog exceeded its allowed bandwidth at fpc 1 for 119 times, started at 2023-06-08 11:05:59 EAT
Jun  8 11:06:03  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  Sample:aggregate exceeded its allowed bandwidth at fpc 1 for 102 times, started at 2023-06-08 11:06:02 EAT
Jun  8 11:11:08  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception Sample:syslog has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 119 times, from 2023-06-08 11:05:59 EAT to 2023-06-08 11:06:07 EAT
Jun  8 11:11:08  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception Sample:aggregate has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 102 times, from 2023-06-08 11:06:02 EAT to 2023-06-08 11:06:07 EAT
Jun  8 18:27:21  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  resolve:ucast-v4 exceeded its allowed bandwidth at fpc 1 for 123 times, started at 2023-06-08 18:27:20 EAT
Jun  8 18:44:09  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  resolve:aggregate exceeded its allowed bandwidth at fpc 1 for 27 times, started at 2023-06-08 18:44:09 EAT
Jun  8 18:54:09  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception resolve:aggregate has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 27 times, from 2023-06-08 18:44:09 EAT to 2023-06-08 18:49:09 EAT
Jun  8 18:56:34  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception resolve:ucast-v4 has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 123 times, from 2023-06-08 18:27:20 EAT to 2023-06-08 18:51:34 EAT
Jun  8 20:24:28  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  resolve:ucast-v4 exceeded its allowed bandwidth at fpc 1 for 124 times, started at 2023-06-08 20:24:27 EAT
Jun  8 20:29:56  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception resolve:ucast-v4 has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 124 times, from 2023-06-08 20:24:27 EAT to 2023-06-08 20:24:55 EAT
Jun  9 10:40:51  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  resolve:ucast-v4 exceeded its allowed bandwidth at fpc 1 for 125 times, started at 2023-06-09 10:40:51 EAT
Jun  9 10:45:53  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception resolve:ucast-v4 has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 125 times, from 2023-06-09 10:40:51 EAT to 2023-06-09 10:40:52 EAT
Jun  9 10:48:35  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  resolve:ucast-v4 exceeded its allowed bandwidth at fpc 1 for 126 times, started at 2023-06-09 10:48:34 EAT
Jun  9 10:57:17  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception resolve:ucast-v4 has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 126 times, from 2023-06-09 10:48:34 EAT to 2023-06-09 10:52:17 EAT
Jun  9 10:58:03  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  resolve:ucast-v4 exceeded its allowed bandwidth at fpc 1 for 127 times, started at 2023-06-09 10:58:03 EAT
Jun  9 11:03:08  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception resolve:ucast-v4 has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 127 times, from 2023-06-09 10:58:03 EAT to 2023-06-09 10:58:07 EAT
Jun  9 11:09:00  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  resolve:ucast-v4 exceeded its allowed bandwidth at fpc 1 for 128 times, started at 2023-06-09 11:09:00 EAT
Jun  9 11:17:03  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception resolve:ucast-v4 has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 128 times, from 2023-06-09 11:09:00 EAT to 2023-06-09 11:12:03 EAT
Jun  9 11:35:19  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  ARP:aggregate exceeded its allowed bandwidth at fpc 1 for 159 times, started at 2023-06-09 11:35:18 EAT
Jun  9 11:45:06  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception ARP:aggregate has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 159 times, from 2023-06-09 11:35:18 EAT to 2023-06-09 11:40:05 EAT
Jun  9 12:05:46  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  TTL:aggregate exceeded its allowed bandwidth at fpc 1 for 72 times, started at 2023-06-09 12:05:45 EAT
Jun  9 12:12:51  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception TTL:aggregate has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 72 times, from 2023-06-09 12:05:45 EAT to 2023-06-09 12:07:50 EAT
Jun  9 12:17:57  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  ARP:aggregate exceeded its allowed bandwidth at fpc 1 for 160 times, started at 2023-06-09 12:17:56 EAT
Jun  9 12:27:33  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_CLEAR: INFO: Host-bound traffic for protocol/exception ARP:aggregate has returned to normal. Its allowed bandwith was exceeded at fpc 1 for 160 times, from 2023-06-09 12:17:56 EAT to 2023-06-09 12:22:32 EAT
Jun  9 12:45:32  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  ARP:aggregate exceeded its allowed bandwidth at fpc 1 for 161 times, started at 2023-06-09 12:45:32 EAT
Jun  9 12:49:23  PE-1-SP-NBO-KE-re0 jddosd[17447]: DDOS_PROTOCOL_VIOLATION_SET: Warning: Host-bound traffic for protocol/exception  TTL:aggregate exceeded its allowed bandwidth at fpc 1 for 73 times, started at 2023-06-09 12:49:23 EAT

    Is the DDOS violation causing RE crush ?  

    We also see in the logs below:

    Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 632
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 786
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 1005
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 777
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 714
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 864
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 756
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 1184
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 624
Jun  9 12:49:59  PE-1-SP-NBO-KE-re0 mib2d[17280]: SNMP_EVLIB_FAILURE: PFED ran out of transfer credits with PFE.Failed to get stats. ifl index: 861

    Regards,

    lish. 



  • 2.  RE: MX480 droping all protocal connections for few mins

    Posted 06-12-2023 05:43

    Hi, you need to review your RE protection, filters and policers.

    It's not necessarily is a DDOS, can be DOS or a Loop in your network that is causing your RE being flooded.

    Adapt your protection to your network design.



    ------------------------------
    ARTHUR GUILHERME LIMA RIBEIRO
    ------------------------------

    Original Message


  • 3.  RE: MX480 droping all protocal connections for few mins

    Posted 06-22-2023 04:18

    Hello Arthur, 

    We tried looking for any trace of broadcast storm via the graphs and couldn't find any. we do have some ERP rings plugged in to VPLS at the MX and this vpls instances have loop-detection mechanisms that drop and block incase Storm happens.

    Not sure what we could be missing. 

    Regards, 

    Lish. 


    Original Message


  • 4.  RE: MX480 droping all protocal connections for few mins

    Posted 06-28-2023 07:15

    Hi, There must some flood hitting on FPC 1 causing this DDOS violation for ucast-v4 and arp. ISIS /BGP/LDP which are flapping are on FPC 1? 

    Check the output of "show policer" if default arp-counter is incrementing. You can also check if any interface is receiving high broadcast packet under show interface extensive output.



    ------------------------------
    Avinash Birla
    ------------------------------

    Original Message