SRX

Expand all | Collapse all

Brand new SRX300 high CPU usage

Jump to Best Answer
  • 1.  Brand new SRX300 high CPU usage

    Posted 09-28-2016 05:07

    Hi all,

     

    I have unboxed a new SRX300 and the CPU usage of the routing engine is about 50%

     

     

    root> show chassis routing-engine
    Routing Engine status:
        Temperature                 42 degrees C / 107 degrees F
        CPU temperature             56 degrees C / 132 degrees F
        Total memory              4096 MB Max  2417 MB used ( 59 percent)
          Control plane memory    2624 MB Max  2073 MB used ( 79 percent)
          Data plane memory       1472 MB Max   339 MB used ( 23 percent)
        5 sec CPU utilization:
          User                      56 percent
          Background                 0 percent
          Kernel                     2 percent
          Interrupt                  0 percent
          Idle                      42 percent
        Model                          RE-SRX300
        Serial ID                      CV2116AF0125
        Start time                     2016-05-29 04:43:00 UTC
        Uptime                         13 minutes, 3 seconds
        Last reboot reason             0x200:normal shutdown
        Load averages:                 1 minute   5 minute  15 minute
                                           0.68       0.98       0.95
    

    This is with the default configuration loaded.  I have also configured a very basic routed interface and the latency TO the srx is about 3ms via a directly connected EX2200 and out of the SRX (i.e back to the switch via a simple ping) is anywhere from 1-2ms through to 50-60ms.  I have logged this with JTAC and they are investigating but has anyone else seen this sort of behaviour with a brand new unit?  I have a 210HE2 in front of me that is actually in production (the SRX300 is just on the bench with nothing connected to it at the moment for example) and that is only running at about 10% utilisation and has no throughput latency...

     

    I have tried  a few different junos versions but are currently running on 15.1X49-D60.7

     

    processes are high JTAC are reporting for a unit that is not processing anything..

     

    last pid:  1774;  load averages:  0.55,  0.71,  0.82  up 0+00:18:21    05:00:51
    149 processes: 17 running, 120 sleeping, 12 waiting
    
    Mem: 282M Active, 120M Inact, 1560M Wired, 371M Cache, 112M Buf, 1644M Free
    Swap:
    
    
      PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
     1627 root        5  76    0  1552M   471M select 0  24:06 145.07% flowd_octeon_hm
       21 root        1 171   52     0K    16K RUN    0   6:35 38.33% idle: cpu0
       20 root        1 171   52     0K    16K RUN    1   2:16  0.00% idle: cpu1
       81 root        1  -8    0     0K    16K mdwait 0   0:15  0.00% md1
     1655 root        7  76    0 34492K  9124K select 0   0:08  0.00% aamwd
       23 root        1 -20 -139     0K    16K RUN    0   0:08  0.00% swi7: clock
     1613 root        1  76    0 34032K 15708K select 0   0:05  0.00% snmpd
     1636 root        1  76    0   108M 29284K select 0   0:05  0.00% authd
     1618 root        1  76    0 60544K 22744K RUN    0   0:03  0.00% pfed
     1634 root        1  76    0   104M 16208K select 0   0:02  0.00% jdhcpd
     1614 root        1  76    0 49240K 14440K select 0   0:02  0.00% mib2d
     1609 root        1  76    0 72236K 24848K select 0   0:02  0.00% chassisd
        3 root        1  -8    0     0K    16K -      0   0:02  0.00% g_up
     1664 root        1  76    0 18996K  7836K select 0   0:02  0.00% license-check
    
    root> show security flow statistics
        Current sessions: 0
        Packets forwarded: 0
        Packets dropped: 0
        Fragment packets: 0
    

    Cheers,

    Aaron



  • 2.  RE: Brand new SRX300 high CPU usage

    Posted 09-28-2016 09:14

    This is the same as mine, but I have traffic passing through the device.



  • 3.  RE: Brand new SRX300 high CPU usage

    Posted 09-28-2016 15:32

    You have it in production?  I am wondering if I am actually going to perservere with it and possibly just get them RMA'd and get another model if juniper cant sort it out.  the unit is going to replace an SRX210 which lives in Thailand and latency is already around 150-200ms.. I cant add another 70 or so to that for no good reason.



  • 4.  RE: Brand new SRX300 high CPU usage

    Posted 09-28-2016 19:24

    Hi,

     

    This is a known issue and is actively being investigated by Engineering team for a fix.

     

    Regards,

    Sahil Sharma

    ---------------------------------------------------

    Please mark my solution as accepted if it helped, Kudos are appreciated as well.



  • 5.  RE: Brand new SRX300 high CPU usage

    Posted 09-28-2016 19:57
    What do I do in the meantime? I have 3 devices to rollout. Do I send them back as DOA and purchase new ones once a fixed ? Is it just the 300? Or can i use any new 3xx devices??

    Cheers,
    Aaron

    Sent from my iPhone


  • 6.  RE: Brand new SRX300 high CPU usage

    Posted 09-28-2016 20:03

    Hi Aaron,

     

    This is a software issue and has nothing to do with the hardware of the SRX3xx devices. A software upgrade to the resolved version would fix this. I would let you know the fixed software version so that you can upgrade and get this issue fixed.

     

    This issue would be present in all the SRX3xx devices as this is a software bug.

     

    I would find out more details on this and then update you.

     

    Regards,

    Sahil Sharma

    ---------------------------------------------------

    Please mark my solution as accepted if it helped, Kudos are appreciated as well.



  • 7.  RE: Brand new SRX300 high CPU usage

    Posted 09-28-2016 20:56

    Hi Aaron,

     

    As per the latest update, the engineering team is working with the hardware team to fix this. The fix should be in software code only.

     

    I would keep you posted.

     

    Regards,

    Sahil Sharma

    ---------------------------------------------------

    Please mark my solution as accepted if it helped, Kudos are appreciated as well.



  • 8.  RE: Brand new SRX300 high CPU usage

    Posted 09-28-2016 21:52

    Thanks for the update... I have logged a P2 case for this (2016-0928-1022), could you perhaps update the case owner?  Is there going to be an alert or similar sent out about this as lets face it, from what you are saying ever 3xx series device out there is not functioning correctly?  Do I understand it correct?



  • 9.  RE: Brand new SRX300 high CPU usage

    Posted 09-29-2016 01:45

    Aaron,

     

    I would update the case owner about the PRs.

    Not sure if every 3xx device is affected by this, howeverm i have seen quite a few devices with such symptoms.

    The case owner would notify you when the PR is fixed.

     

    Regards,

    Sahil Sharma

    ---------------------------------------------------

    Please mark my solution as accepted if it helped, Kudos are appreciated as well.

     



  • 10.  RE: Brand new SRX300 high CPU usage

    Posted 09-29-2016 04:46

    I have similiar issue on my SRX300 which is running D60 software. I don't have high CPU utilization but my control plane is using 95 percent of memory:

    Routing Engine status:
        Temperature                 42 degrees C / 107 degrees F
        CPU temperature             56 degrees C / 132 degrees F
        Total memory              4096 MB Max  2908 MB used ( 71 percent)
          Control plane memory    2624 MB Max  2493 MB used ( 95 percent)
          Data plane memory       1472 MB Max   412 MB used ( 28 percent)
        5 sec CPU utilization:
          User                       9 percent
          Background                 0 percent
          Kernel                     4 percent
          Interrupt                  0 percent
          Idle                      87 percent
        Model                          RE-SRX300

    I also have srx300 which are running almost the same traffic and configuration and are on D50 software and they don't have that problem.



  • 11.  RE: Brand new SRX300 high CPU usage

    Posted 09-30-2016 02:32

    Thanks for that... is there any mitigation or workarounds that you know of as I can't imagine EVERY unit doing it?  What is the factory default configuration doing that some other customers configurations isnt?



  • 12.  RE: Brand new SRX300 high CPU usage

    Posted 12-19-2016 11:07

    @aaron@ARB JTAC resolved your problem? or the case is still open?

    We are facing the same exact problem with our SRX 300s, thanks for your input.



  • 13.  RE: Brand new SRX300 high CPU usage

     
    Posted 12-19-2016 12:28

    Hi,

     

    if you read the release notes for 15.1X49-D70 (http://www.juniper.net/techpubs/en_US/junos15.1x49-d70/information-products/topic-collections/release-notes/15.1x49-d70/junos-release-notes-15.1X49-D70.pdf) under resolved issues, the following PR is seen:

     

    • On SRX300 Series devices, high RE CPU is noticed when ethernet-switching is configured. This leads to a higher than expected latency for traffic originating from the device itself. This issue has no impact on transit traffic through the device. PR1206823

     

    It could sound like the issue people are facing. 

     

    15.1X49-D70 release notes are online but the actual software image is not. I would expect to show up shortly as D70 is expected to be release this month.



  • 14.  RE: Brand new SRX300 high CPU usage

    Posted 12-19-2016 14:24

    Jonas,

     

    Thanks for taking the time to post this. I look forward to reading over the release notes. 

     

    I see this on my srx300 @ idle as well, high cpu usage that is. We were considering deploying these for a new businsess class product and this has made up hesitant. I have a 220 running version 12 with a similar config and it only using about 3% cpu at idle. 

     

    Jarrod@FW1> show version
    Hostname: FW1
    Model: srx300
    Junos: 15.1X49-D60.7
    JUNOS Software Release [15.1X49-D60.7]

     

    Jarrod@FW1> show chassis routing-engine
    Routing Engine status:
    Temperature 50 degrees C / 122 degrees F
    CPU temperature 65 degrees C / 149 degrees F
    Total memory 4096 MB Max 2540 MB used ( 62 percent)
    Control plane memory 2624 MB Max 2152 MB used ( 82 percent)
    Data plane memory 1472 MB Max 368 MB used ( 25 percent)
    5 sec CPU utilization:
    User 57 percent
    Background 0 percent
    Kernel 3 percent
    Interrupt 0 percent
    Idle 40 percent
    Model RE-SRX300
    Serial ID CV2116AF0542
    Start time 2016-12-19 19:42:53 UTC
    Uptime 2 hours, 34 minutes, 31 seconds
    Last reboot reason 0x200:normal shutdown
    Load averages: 1 minute 5 minute 15 minute
    0.68 0.58 0.57

     

    Jarrod@FW1> show chassis forwarding
    FWDD status:
    State Online
    Microkernel CPU utilization 59 percent
    Real-time threads CPU utilization 0 percent
    Heap utilization 25 percent
    Buffer utilization 1 percent
    Uptime: 2 hours, 37 minutes, 2 seconds



  • 15.  RE: Brand new SRX300 high CPU usage

     
    Posted 12-19-2016 23:17

    Hi,

     

    it looks better in D70 - as expected:

    jh@fw> show chassis routing-engine
    Routing Engine status:
        Temperature                 60 degrees C / 140 degrees F
        CPU temperature             76 degrees C / 168 degrees F
        Total memory              4096 MB Max  1065 MB used ( 26 percent)
          Control plane memory    2624 MB Max   682 MB used ( 26 percent)
          Data plane memory       1472 MB Max   383 MB used ( 26 percent)
        5 sec CPU utilization:
          User                      15 percent
          Background                 0 percent
          Kernel                    10 percent
          Interrupt                  0 percent
          Idle                      75 percent
        Model                          RE-SRX300
        Serial ID                      XXXXX
        Start time                     2016-12-20 07:29:39 CET
        Uptime                         47 minutes, 5 seconds
        Last reboot reason             0x200:normal shutdown
        Load averages:                 1 minute   5 minute  15 minute
                                           0.41       0.31       0.27
    
    jh@fw>
    

     



  • 16.  RE: Brand new SRX300 high CPU usage

    Posted 12-19-2016 14:33

    Correct, I had a call logged with JTAC and they sent me D62.1 before it was released.  I have been running it for about 2 weeks now without issue.

     

    Current time: 2016-12-20 05:30:36 ICT
    Time Source:  NTP CLOCK
    System booted: 2016-12-18 13:20:04 ICT (1d 16:10 ago)
    Protocols started: 2016-12-18 13:20:04 ICT (1d 16:10 ago)
    Last configured: 2016-12-10 15:37:12 ICT (1w2d 13:53 ago) by arbadmin
     5:30AM  up 1 day, 16:11, 1 user, load averages: 0.13, 0.10, 0.08
    
    
    Information for junos:
    
    Comment:
    JUNOS Software Release [15.1X49-D62.1]
    

    So I am guessing that D70 is either a rollup from D62.1 or has additional fixes in it.  Either way, it works, we have our devices set for etherswitching (for vlan functionality via irb) and its behaving.



  • 17.  RE: Brand new SRX300 high CPU usage
    Best Answer

     
    Posted 12-19-2016 18:46

    D62 is a service release with just this fix ( and few more), and D70 is the regular Maintenance release with more features/ fixes.  D70 is now available for customer download http://www.juniper.net/support/downloads/?p=srx300#sw



  • 18.  RE: Brand new SRX300 high CPU usage

    Posted 12-19-2016 18:49

    Thanks for the update.  I will update my D62.1's to D70 during the next maintenance window



  • 19.  RE: Brand new SRX300 high CPU usage

    Posted 12-19-2016 19:29
    I wonder if the D70 is supported on the 300, it appears to only support 320 and higher....


  • 20.  RE: Brand new SRX300 high CPU usage

     
    Posted 12-19-2016 19:42
    Junos OS Release 15.1X49-D70 supports the following devices:
    •Branch SRX Series devices: SRX300, SRX320, SRX340, SRX345, and SRX550
    High Memory(SRX 550M)

    Ref: Release notes for D70 - http://www.juniper.net/techpubs/en_US/junos15.1x49-d70/information-products/topic-collections/release-notes/15.1x49-d70/junos-release-notes-15.1X49-D70.pdf
    Page : 4


  • 21.  RE: Brand new SRX300 high CPU usage

     
    Posted 12-19-2016 23:16

    It looks like an error on the download site. The image works for SRX300:

     

    jh@fw> show version
    Hostname: fw
    Model: srx300
    Junos: 15.1X49-D70.3
    JUNOS Software Release [15.1X49-D70.3]

     



  • 22.  RE: Brand new SRX300 high CPU usage

    Posted 12-20-2016 02:56

    I installed the D70 on my SRX 300 this morning. It did solve the CPU issue, however I had to roll back as it sadly didnt agree with my current running configuration. I have copied the config errors that occured below. Perhaps I need to read the release notes, maybe I missed something. 

     

    vlans {

    vlan1 {

    ##

    ## Warning: Default bridge-domain mode without any vlan-id configuration is not supported.

    ## Warning: statement ignored: unsupported platform (srx300)

    ##

    vlan-id none;

    ##

    ## Warning: l3-interface can be configured only under vlans with 'vlan-id'/'vlan-tags'

    ## Warning: l3-interface can be configured only under vlans with 'vlan-id'/'vlan-tags'

    ##

    l3-interface irb.0;

    }

    }

     

     



  • 23.  RE: Brand new SRX300 high CPU usage

     
    Posted 12-20-2016 03:37

    Looks like you just need to assign a vlan-id on the vlan. Then the config should be OK.



  • 24.  RE: Brand new SRX300 high CPU usage

    Posted 12-20-2016 04:48

    Yeah I am going to try that. It works fine on the firmware im running now.  Its built on vlan-id 1. How ever when I show the set it shows vlan none....which  is odd. I am going to try vlan-id 2.



  • 25.  RE: Brand new SRX300 high CPU usage

    Posted 12-20-2016 17:34

    I was able to manipulate my configuration and get version 

     

    Jarrod@FW1> show version 

    Hostname: FW1

    Model: srx300

    Junos: 15.1X49-D70.3

    JUNOS Software Release [15.1X49-D70.3]

     

    to run.  The CPU usage is now perfect.

     

    Jarrod@FW1> show chassis routing-engine
    Routing Engine status:
    Temperature 46 degrees C / 114 degrees F
    CPU temperature 59 degrees C / 138 degrees F
    Total memory 4096 MB Max 983 MB used ( 24 percent)
    Control plane memory 2624 MB Max 656 MB used ( 25 percent)
    Data plane memory 1472 MB Max 339 MB used ( 23 percent)
    5 sec CPU utilization:
    User 11 percent
    Background 0 percent
    Kernel 7 percent
    Interrupt 0 percent
    Idle 82 percent
    Model RE-SRX300
    Serial ID CV2116AF0542
    Start time 2016-12-21 01:10:57 UTC
    Uptime 18 minutes, 4 seconds
    Last reboot reason 0x200:normal shutdown
    Load averages: 1 minute 5 minute 15 minute
    0.14 0.38 0.70

     

    Awesome!



  • 26.  RE: Brand new SRX300 high CPU usage

    Posted 12-21-2016 04:08

    Thanks a lot for your inputs. We have tested the new version of the firmware, it's true that the high cpu and high memory usage is resolved.

    But, in our tests, We have seen high latency when you do ping from the router itself.

    This is a problem because We can't do benchmark or see real results from ping, mtr, etc.



  • 27.  RE: Brand new SRX300 high CPU usage

    Posted 05-28-2019 21:36

    Hello all,

     

    I am facing same issue about high CPU utilization on Juniper SRX340. Below is some output from my SRX.

    seyma@SRX340> show version 
    Hostname: SRX340
    Model: srx340
    Junos: 15.1X49-D70.3
    JUNOS Software Release [15.1X49-D70.3]
    seyma@SRX340> show chassis routing-engine                      
    Routing Engine status:
        Temperature                 39 degrees C / 102 degrees F
        CPU temperature             60 degrees C / 140 degrees F
        Total memory              4096 MB Max  1106 MB used ( 27 percent)
          Control plane memory    2624 MB Max   682 MB used ( 26 percent)
          Data plane memory       1472 MB Max   412 MB used ( 28 percent)
        5 sec CPU utilization:
          User                     100 percent
          Background                 0 percent
          Kernel                     0 percent
          Interrupt                  0 percent
          Idle                       0 percent
        Model                          RE-SRX340
        Serial ID                      CY3318AF0723
        Start time                     2019-05-18 14:24:09 HKT
        Uptime                         10 days, 22 hours, 6 minutes, 40 seconds
        Last reboot reason             0x1:power cycle/failure
        Load averages:                 1 minute   5 minute  15 minute
                                           0.99       1.00       1.00
    
    seyma@SRX340> show system processes extensive 
    last pid: 24051;  load averages:  0.99,  0.98,  0.99  up 10+22:11:07    12:25:01
    149 processes: 19 running, 118 sleeping, 12 waiting
    
    Mem: 289M Active, 169M Inact, 1579M Wired, 416M Cache, 112M Buf, 1525M Free
    Swap:
    
      PID USERNAME     THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
     1829 root           6  76    0  1554M   471M RUN    0 857.9H 369.87% flowd_octeon_hm
       21 root           1 171   52     0K    16K RUN    0 177.8H  0.00% idle: cpu0
     1859 root           7  76    0 34520K  9288K select 0 151:29  0.00% aamwd
       23 root           1 -20 -139     0K    16K RUN    0 127:39  0.00% swi7: clock
     1818 root           1  76    0 39308K 14692K select 0  52:38  0.00% l2ald
     1820 root           1  76    0 60500K 22896K select 0  47:32  0.00% pfed
       25 root           1 -20 -139     0K    16K WAIT   0  43:01  0.00% swi7: +
     1889 root           1  76    0     0K    16K select 0  40:44  0.00% peerproxy00000001
       22 root           1 -40 -159     0K    16K WAIT   0  33:29  0.00% swi2: netisr 0
     1868 root           1  76    0 19004K  7876K select 0  29:06  0.00% license-check
     1821 root           1  76    0 21060K  9216K select 0  25:42  0.00% rmopd
        5 root           1 -16    0     0K    16K rtfifo 0  16:39  0.00% rtfifo_kern_recv
     1479 root           1  76    0 18260K  6572K select 0  16:03  0.00% eventd
     1817 root           1   4    0 91876K 57380K kqread 0  10:50  0.00% rpd
       18 root           1 171   52     0K    16K RUN    3   9:46  0.00% idle: cpu3
       19 root           1 171   52     0K    16K RUN    2   9:43  0.00% idle: cpu2
     1847 root           1  76    0 19792K  4604K select 0   8:39  0.00% shm-rtsdbd
     1824 root           1  76    0 22336K 10612K select 0   7:17  0.00% ppmd
     1858 root           7   8    0 97412K  6748K nanslp 0   5:41  0.00% ipfd

    Based on above output, we could not find any process is eating much CPU except "flowd_octeon_hm". The transit traffic is not interrupt  but exception traffic is. This cause slowness of accessing to the device via SSH and our monitoring system is not be able to capure the traffic.

     

    Kindly advise the next action I should do/check in order to resolve the issue.

     



  • 28.  RE: Brand new SRX300 high CPU usage

    Posted 06-15-2019 23:11

    seyma,

     

    The output indicates 100% user CPU this generally indicate multiple user session idle on SRX box. 

    You need to look at "show systme user" to see currently logged in and idle user.

    -Logout the idle users via command "requset system logout user terminal termin-name.

     

    Regarding the flowd process going high that should not be conern becuase the SRX have multiple flowd thread and the output that you see is cumulative output of multiple threads so it can reach above 100%.



  • 29.  RE: Brand new SRX300 high CPU usage

    Posted 06-15-2019 23:30

    Hi Seyma,

     

    The RE output shows "User" to use 100% of the resource.

     

    Apart from checking if there are any logged in user. Check if you have a lot of mdg/php process active.

     

    show system processes extensive | match mgd

    show system processes extensive | match php

     

    These processes will stay in 0.0% and could be deep down in the "show .. extensive " command lisat.

    If you find many, quick way to relieve the box would be to kill the stale mgd/php process with kill -9 <pid>

     

    Regards,

     

    Rahul



  • 30.  RE: Brand new SRX300 high CPU usage

     
    Posted 06-16-2019 19:10

    Hello,

     

    This could be due to some user process being stuck. Flowd being high is normal.

     

    I would suggest to kill the existing user (mgd) sessions. Here is a sample:

     

    fw-user@srx> show system users no-resolve
    2:04AM up 5 days, 18:13, 2 users, load averages: 0.23, 0.12, 0.09
    USER     TTY     FROM      LOGIN@ IDLE WHAT
    root        u0     - Sat07AM 3days cli
    fw-user  p0 10.222.9.47    2:03AM - -cli (cli)

     

    fw-user@srx> show system processes | grep mgd | except sbin
    1958 ?? Ss 0:01.50 mgd: (mgd) (root)/dev/ttyu0 (mgd)
    14316 ?? Ss 0:00.04 mgd: (mgd) (fw-user)/dev/ttyp0 (mgd)

     

    start shell user root


    kill 1958
    kill 14316

     

    Hope this helps. Regards,

     

    Vikas