SRX

Expand all | Collapse all

High CPU, not traceable

Jump to Best Answer
  • 1.  High CPU, not traceable

    Posted 04-02-2019 23:51

    Hello, I have a SRX 4100 and high CPU "spikes"

     

    While troubleshooting, I realised that mib2d & snmp take much utilization, & research showed me that our Check_MK plugin does snmpwalks and that may cause our high CPU, so I temporarly deactivated our plugin for Check_MK and monitored it manually with snmpgets and the CPU spikes were instantly less.. But still there are spikes above 90 percent

     

    Running following commands:

    show chassis routing-engine

    node0:
    --------------------------------------------------------------------------
    Routing Engine status:
        Temperature                 27 degrees C / 80 degrees F
        CPU temperature             27 degrees C / 80 degrees F
        Total memory              1954 MB Max   645 MB used ( 33 percent)
        Memory utilization          27 percent
        5 sec CPU utilization:
          User                      24 percent
          Background                 9 percent
          Kernel                    56 percent
          Interrupt                 11 percent
          Idle                       0 percent
        1 min CPU utilization:
          User                       4 percent
          Background                14 percent
          Kernel                    17 percent
          Interrupt                  7 percent
          Idle                      59 percent
        5 min CPU utilization:
          User                       3 percent
          Background                14 percent
          Kernel                    14 percent
          Interrupt                  6 percent
          Idle                      63 percent
        15 min CPU utilization:
          User                       2 percent
          Background                13 percent
          Kernel                    13 percent
          Interrupt                  6 percent
          Idle                      66 percent
        Model                          SRX Routing Engine
        Serial ID                      BUILTIN
        Start time                     2019-01-10 08:42:04 CET
        Uptime                         74 days, 22 hours, 57 minutes, 58 seconds
        Last reboot reason             0x8:power-button hard power off 
        Load averages:                 1 minute   5 minute  15 minute
                                           1.11       0.83       0.73

    show system processes extensive

    last pid: 25114;  load averages:  0.80,  0.65,  0.59  up 82+23:06:30    08:47:53
    172 processes: 3 running, 149 sleeping, 20 waiting
    
    Mem: 437M Active, 96M Inact, 66M Wired, 724M Cache, 69M Buf, 579M Free
    Swap: 615M Total, 615M Free
    
    
      PID USERNAME         THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       10 root               1 171   52     0K    12K RUN    1268.2 50.00% idle: cpu0
     1952 root               1 139   15 11652K  6908K RUN     85.6H  5.96% sampled
       24 root               1 -68 -187     0K    12K WAIT    86.3H  3.96% irq11: uhci0 em3++*
        4 root               1  -8    0     0K    12K -       44.6H  3.47% g_down
       11 root               1 -40 -159     0K    12K WAIT    38.6H  0.49% swi2: netisr 0
       27 root               1 -16    0     0K    12K -       21.7H  0.49% em0 taskq
     1948 root               1  76    0 33016K 21372K select  90.9H  0.00% mib2d
     1995 root               1  76    0 23240K 18492K select  24.6H  0.00% snmpd
     1581 root               1  76    0  5464K  2384K select  16.9H  0.00% sysctlrelayd
     2074 root               1  76    0     0K    12K select 793:07  0.00% peerproxy02a00001
       14 root               1 -20 -139     0K    12K WAIT   638:05  0.00% swi7: +
       45 root               1 171   52     0K    12K pgzero 513:29  0.00% pagezero
     1956 root               1  76    0 11520K  8840K select 400:24  0.00% ppmd
     2603 root               1  76    0     0K    12K select 301:59  0.00% peerproxy1000a081
       15 root               1 -16    0     0K    12K -      229:15  0.00% yarrow
     1962 root               1   4    0 41096K 11752K kqread 179:47  0.00% l2cpd
     1585 root               1  76    0  9608K  4900K select 178:11  0.00% license-check
     1578 root               1  76    0  9568K  4808K select 174:36  0.00% rtlogd
       12 root               1 -20 -139     0K    12K WAIT   165:31  0.00% swi7: clock sio
     2602 root               1  76    0     0K    12K select 159:47  0.00% peerproxy1000a082
     1982 root               7  76    0 20784K  5584K select 115:13  0.00% aamwd
     1994 root               1  76    0 47356K 18908K select  88:44  0.00% pfed
     2540 root               1  76    0 40316K 14984K select  87:15  0.00% chassisd
        3 root               1  -8    0     0K    12K -       84:27  0.00% g_up
     2160 root               1  76    0 25480K 14012K select  64:14  0.00% nsd
     1570 root               1  76    0  5264K  2832K select  52:10  0.00% pmond
       32 root               1 -16    0     0K    92K -       50:00  0.00% vtblk1 taskq
       51 root               1  76    0     0K    12K sleep   39:12  0.00% netdaemon
     1564 root               1  76    0 12376K  7628K select  38:35  0.00% jsrpd
     1951 root               1  76    0 22988K  8900K select  32:32  0.00% l2ald
     1950 root               1   4    0 90840K 34172K kqread  30:35  0.00% rpd
     1953 root               1  76    0 10836K  5564K select  29:59  0.00% rmopd
     1967 root               1  76    0 17392K  5384K select  25:42  0.00% bdbrepd
     1955 root               1  76    0 10836K  4240K select  19:34  0.00% fud
     1562 root               1  76    0 12828K  4520K select  19:06  0.00% shm-rtsdbd
       28 root               1 -16    0     0K    12K -       15:53  0.00% em1 taskq
    
    

    can someone help me with the troubleshoot? 

     

    Thanks in advance

     

     

     



  • 2.  RE: High CPU, not traceable

    Posted 04-03-2019 01:40

    sampled seems to take CPU now.

    Restart nd confirm the state:- 

    restart sampling immediately

     



  • 3.  RE: High CPU, not traceable

    Posted 04-03-2019 03:44

    i restarted it:

      PID USERNAME         THR PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       10 root               1 171   52     0K    12K RUN    1270.1 32.47% idle: cpu0
    84871 root               1  -8   15 11648K  7028K RUN      0:23  8.98% sampled
    86508 root               1 139   15  2124K  1244K RUN      0:00  8.98% gzip
       24 root               1 -68 -187     0K    12K WAIT    86.6H  5.96% irq11: uhci0 em3++*
        4 root               1  -8    0     0K    12K -       44.8H  4.49% g_down
       11 root               1 -40 -159     0K    12K WAIT    38.7H  1.95% swi2: netisr 0
       27 root               1 -16    0     0K    12K -       21.8H  0.98% em0 taskq
     1948 root               1  76    0 33016K 21372K select  90.9H  0.00% mib2d
     1995 root               1  76    0 23240K 18492K select  24.6H  0.00% snmpd
     1581 root               1  76    0  5464K  2384K RUN     16.9H  0.00% sysctlrelayd
     2074 root               1  76    0     0K    12K select 794:37  0.00% peerproxy02a00001

    seems like my pcap is the problem?



  • 4.  RE: High CPU, not traceable
    Best Answer

    Posted 04-03-2019 04:01

    you may disable sampling to confirm the cause and then think about adding/notadding finetuning the same.