SRX

Expand all | Collapse all

SRX240 having high CPU utilization issue: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99

  • 1.  SRX240 having high CPU utilization issue: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99

    Posted 10-17-2018 19:05

    Hi, 

     

    I am facing the the CPU high utilzation issue of SRX240.

    Log messages are :

    Oct 18 11:14:55 SRX-CORP-WILL-A last message repeated 2 times
    Oct 18 11:15:15 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=87
    Oct 18 11:16:23 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:16:31 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:20:40 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=92
    Oct 18 11:21:49 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:22:20 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:23:15 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:24:49 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:25:02 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=85
    Oct 18 11:25:08 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:25:33 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=88
    Oct 18 11:25:47 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:27:18 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=94
    Oct 18 11:27:28 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:27:32 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:27:35 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=98
    Oct 18 11:27:38 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=87
    Oct 18 11:27:55 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=93
    Oct 18 11:27:59 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:28:31 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:29:23 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=93
    Oct 18 11:29:26 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:30:05 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 11:31:04 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=90
    Oct 18 11:31:07 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=97
    Oct 18 11:31:15 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=95
    Oct 18 11:33:41 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=92
    Oct 18 12:01:19 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=87
    Oct 18 12:02:19 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=94
    Oct 18 12:02:22 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 12:02:26 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=86
    Oct 18 12:02:34 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=93
    Oct 18 12:03:24 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=86
    Oct 18 12:03:46 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=93
    Oct 18 12:04:05 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 12:04:11 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=85
    Oct 18 12:04:20 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=94
    Oct 18 12:04:22 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=91
    Oct 18 12:04:25 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=89
    Oct 18 12:26:36 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 12:26:40 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=94
    Oct 18 12:31:51 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=89
    Oct 18 12:33:02 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=91
    Oct 18 12:33:04 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 12:47:30 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99
    Oct 18 12:48:59 SRX-CORP-WILL-A PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99

     

    Routing engine :

    show chassis routing-engine
    node1:
    --------------------------------------------------------------------------
    Routing Engine status:
    Temperature 35 degrees C / 95 degrees F
    CPU temperature 35 degrees C / 95 degrees F
    Total memory 2048 MB Max 1126 MB used ( 55 percent)
    Control plane memory 1088 MB Max 457 MB used ( 42 percent)
    Data plane memory 960 MB Max 672 MB used ( 70 percent)
    CPU utilization:
    User 18 percent
    Background 0 percent
    Kernel 11 percent
    Interrupt 0 percent
    Idle 71 percent
    Model RE-SRX240H2
    Serial ID ACMT9642
    Start time 2017-08-11 15:30:02 EST
    Uptime 432 days, 20 hours, 20 minutes, 39 seconds
    Last reboot reason Router rebooted after a normal shutdown.
    Load averages: 1 minute 5 minute 15 minute
    0.28 0.30 0.29

     

    Show System process Extensive:

    node1:
    --------------------------------------------------------------------------
    last pid: 86460; load averages: 0.22, 0.29, 0.28 up 432+20:22:23 13:01:42
    138 processes: 19 running, 107 sleeping, 1 zombie, 11 waiting

    Mem: 208M Active, 109M Inact, 1024M Wired, 495M Cache, 112M Buf, 136M Free
    Swap:


    PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    1430 root 7 76 0 997M 59188K RUN 0 ??? 7800.59% flowd_octeon_hm
    40 root 1 -16 0 0K 16K psleep 0 12:15 2844.34% bufdaemon
    22 root 1 171 52 0K 16K RUN 0 7090.5 68.41% idle: cpu0
    1443 root 1 139 0 10068K 2884K select 0 0:01 1.46% sdxd
    1083 root 1 76 0 12896K 4664K select 0 269.1H 0.00% eventd
    23 root 1 -20 -139 0K 16K RUN 0 82.5H 0.00% swi7: clock
    1489 root 1 76 0 28276K 12856K select 0 59.7H 0.00% mib2d

     

    packet forwarding engine (data plane)

    show security monitoring fpc 5
    node1:
    --------------------------------------------------------------------------
    FPC 0
    PIC 0
    CPU utilization : 40 %
    Memory utilization : 70 %
    Current flow session : 2666
    Current flow session IPv4: 2666
    Current flow session IPv6: 0
    Max flow session : 524288
    Total Session Creation Per Second (for last 96 seconds on average): 219
    IPv4 Session Creation Per Second (for last 96 seconds on average): 219
    IPv6 Session Creation Per Second (for last 96 seconds on average): 0

     

    show chassis forwarding
    'node1:
    --------------------------------------------------------------------------
    FWDD status:
    State Online
    Microkernel CPU utilization 15 percent
    Real-time threads CPU utilization 16 percent
    Heap utilization 70 percent
    Buffer utilization 1 percent
    Uptime: 432 days, 20 hours, 18 minutes, 55 seconds

     

    show security flow statistics
    node1:
    --------------------------------------------------------------------------
    Current sessions: 2139
    Packets forwarded: 519251685982
    Packets dropped: 1186868040
    Fragment packets: 9990718

     

    Any help would highly appreciated.

     

    Thank you



  • 2.  RE: SRX240 having high CPU utilization issue: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99

     
    Posted 10-17-2018 20:51

    Hello,

     

    When you see High PFE CPU, can you execute the command below 3 times?

     

    show pfe statistics traffic

     

    Regards,

     

    Rushi



  • 3.  RE: SRX240 having high CPU utilization issue: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 5 PIC 0 CPU utilization exceeds threshold, current value=99

    Posted 10-19-2018 18:45

     

    Hi anwarabbas,

     

    Can you share:

     

    • show chassis cluster status
    • show version

    Even though the logs are showing high CPU on dataplane, this is not seen in the outputs you provided:

     

     

    show security monitoring fpc 5 
    node1:
    --------------------------------------------------------------------------
    FPC 0
    PIC 0
    CPU utilization : 40 %
    show chassis forwarding 
    'node1:
    --------------------------------------------------------------------------
    FWDD status:
    State Online 
    Microkernel CPU utilization 15 percent
    Real-time threads CPU utilization 16 percent

     

     

    • Were the commands taken during the time of the high CPU utilization?
    • Are the logs showing always or just between 11pm-12am?

    During the time of the problem can you confirm how much traffic (bps) is the SRX receiving. See the following post:

     

    https://forums.juniper.net/t5/SRX-Services-Gateway/SRX1500-Capactiy/m-p/338608#M50534