SRX

Hi all,

 

I have unboxed a new SRX300 and the CPU usage of the routing engine is about 50%

 

 

root> show chassis routing-engine
Routing Engine status:
    Temperature                 42 degrees C / 107 degrees F
    CPU temperature             56 degrees C / 132 degrees F
    Total memory              4096 MB Max  2417 MB used ( 59 percent)
      Control plane memory    2624 MB Max  2073 MB used ( 79 percent)
      Data plane memory       1472 MB Max   339 MB used ( 23 percent)
    5 sec CPU utilization:
      User                      56 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      42 percent
    Model                          RE-SRX300
    Serial ID                      CV2116AF0125
    Start time                     2016-05-29 04:43:00 UTC
    Uptime                         13 minutes, 3 seconds
    Last reboot reason             0x200:normal shutdown
    Load averages:                 1 minute   5 minute  15 minute
                                       0.68       0.98       0.95

This is with the default configuration loaded.  I have also configured a very basic routed interface and the latency TO the srx is about 3ms via a directly connected EX2200 and out of the SRX (i.e back to the switch via a simple ping) is anywhere from 1-2ms through to 50-60ms.  I have logged this with JTAC and they are investigating but has anyone else seen this sort of behaviour with a brand new unit?  I have a 210HE2 in front of me that is actually in production (the SRX300 is just on the bench with nothing connected to it at the moment for example) and that is only running at about 10% utilisation and has no throughput latency...

 

I have tried  a few different junos versions but are currently running on 15.1X49-D60.7

 

processes are high JTAC are reporting for a unit that is not processing anything..

 

last pid:  1774;  load averages:  0.55,  0.71,  0.82  up 0+00:18:21    05:00:51
149 processes: 17 running, 120 sleeping, 12 waiting

Mem: 282M Active, 120M Inact, 1560M Wired, 371M Cache, 112M Buf, 1644M Free
Swap:


  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
 1627 root        5  76    0  1552M   471M select 0  24:06 145.07% flowd_octeon_hm
   21 root        1 171   52     0K    16K RUN    0   6:35 38.33% idle: cpu0
   20 root        1 171   52     0K    16K RUN    1   2:16  0.00% idle: cpu1
   81 root        1  -8    0     0K    16K mdwait 0   0:15  0.00% md1
 1655 root        7  76    0 34492K  9124K select 0   0:08  0.00% aamwd
   23 root        1 -20 -139     0K    16K RUN    0   0:08  0.00% swi7: clock
 1613 root        1  76    0 34032K 15708K select 0   0:05  0.00% snmpd
 1636 root        1  76    0   108M 29284K select 0   0:05  0.00% authd
 1618 root        1  76    0 60544K 22744K RUN    0   0:03  0.00% pfed
 1634 root        1  76    0   104M 16208K select 0   0:02  0.00% jdhcpd
 1614 root        1  76    0 49240K 14440K select 0   0:02  0.00% mib2d
 1609 root        1  76    0 72236K 24848K select 0   0:02  0.00% chassisd
    3 root        1  -8    0     0K    16K -      0   0:02  0.00% g_up
 1664 root        1  76    0 18996K  7836K select 0   0:02  0.00% license-check

root> show security flow statistics
    Current sessions: 0
    Packets forwarded: 0
    Packets dropped: 0
    Fragment packets: 0

Cheers,

Aaron

This is the same as mine, but I have traffic passing through the device.

You have it in production?  I am wondering if I am actually going to perservere with it and possibly just get them RMA'd and get another model if juniper cant sort it out.  the unit is going to replace an SRX210 which lives in Thailand and latency is already around 150-200ms.. I cant add another 70 or so to that for no good reason.

Hi,

 

This is a known issue and is actively being investigated by Engineering team for a fix.

 

Regards,

Sahil Sharma

---------------------------------------------------

Please mark my solution as accepted if it helped, Kudos are appreciated as well.

What do I do in the meantime? I have 3 devices to rollout. Do I send them back as DOA and purchase new ones once a fixed ? Is it just the 300? Or can i use any new 3xx devices??

Cheers,
Aaron

Sent from my iPhone

Hi Aaron,

 

This is a software issue and has nothing to do with the hardware of the SRX3xx devices. A software upgrade to the resolved version would fix this. I would let you know the fixed software version so that you can upgrade and get this issue fixed.

 

This issue would be present in all the SRX3xx devices as this is a software bug.

 

I would find out more details on this and then update you.

 

Regards,

Sahil Sharma

---------------------------------------------------

Please mark my solution as accepted if it helped, Kudos are appreciated as well.

Hi Aaron,

 

As per the latest update, the engineering team is working with the hardware team to fix this. The fix should be in software code only.

 

I would keep you posted.

 

Regards,

Sahil Sharma

---------------------------------------------------

Please mark my solution as accepted if it helped, Kudos are appreciated as well.

Thanks for the update... I have logged a P2 case for this (2016-0928-1022), could you perhaps update the case owner?  Is there going to be an alert or similar sent out about this as lets face it, from what you are saying ever 3xx series device out there is not functioning correctly?  Do I understand it correct?

Aaron,

 

I would update the case owner about the PRs.

Not sure if every 3xx device is affected by this, howeverm i have seen quite a few devices with such symptoms.

The case owner would notify you when the PR is fixed.

 

Regards,

Sahil Sharma

---------------------------------------------------

Please mark my solution as accepted if it helped, Kudos are appreciated as well.

 

I have similiar issue on my SRX300 which is running D60 software. I don't have high CPU utilization but my control plane is using 95 percent of memory:

Routing Engine status:
    Temperature                 42 degrees C / 107 degrees F
    CPU temperature             56 degrees C / 132 degrees F
    Total memory              4096 MB Max  2908 MB used ( 71 percent)
      Control plane memory    2624 MB Max  2493 MB used ( 95 percent)
      Data plane memory       1472 MB Max   412 MB used ( 28 percent)
    5 sec CPU utilization:
      User                       9 percent
      Background                 0 percent
      Kernel                     4 percent
      Interrupt                  0 percent
      Idle                      87 percent
    Model                          RE-SRX300

I also have srx300 which are running almost the same traffic and configuration and are on D50 software and they don't have that problem.

Thanks for that... is there any mitigation or workarounds that you know of as I can't imagine EVERY unit doing it?  What is the factory default configuration doing that some other customers configurations isnt?

@aaron@ARB JTAC resolved your problem? or the case is still open?

We are facing the same exact problem with our SRX 300s, thanks for your input.

Hi,

 

if you read the release notes for 15.1X49-D70 (http://www.juniper.net/techpubs/en_US/junos15.1x49-d70/information-products/topic-collections/release-notes/15.1x49-d70/junos-release-notes-15.1X49-D70.pdf) under resolved issues, the following PR is seen:

 

• On SRX300 Series devices, high RE CPU is noticed when ethernet-switching is configured. This leads to a higher than expected latency for traffic originating from the device itself. This issue has no impact on transit traffic through the device. PR1206823

 

It could sound like the issue people are facing. 

 

15.1X49-D70 release notes are online but the actual software image is not. I would expect to show up shortly as D70 is expected to be release this month.

Jonas,

 

Thanks for taking the time to post this. I look forward to reading over the release notes. 

 

I see this on my srx300 @ idle as well, high cpu usage that is. We were considering deploying these for a new businsess class product and this has made up hesitant. I have a 220 running version 12 with a similar config and it only using about 3% cpu at idle. 

 

Jarrod@FW1> show version
Hostname: FW1
Model: srx300
Junos: 15.1X49-D60.7
JUNOS Software Release [15.1X49-D60.7]

 

Jarrod@FW1> show chassis routing-engine
Routing Engine status:
Temperature 50 degrees C / 122 degrees F
CPU temperature 65 degrees C / 149 degrees F
Total memory 4096 MB Max 2540 MB used ( 62 percent)
Control plane memory 2624 MB Max 2152 MB used ( 82 percent)
Data plane memory 1472 MB Max 368 MB used ( 25 percent)
5 sec CPU utilization:
User 57 percent
Background 0 percent
Kernel 3 percent
Interrupt 0 percent
Idle 40 percent
Model RE-SRX300
Serial ID CV2116AF0542
Start time 2016-12-19 19:42:53 UTC
Uptime 2 hours, 34 minutes, 31 seconds
Last reboot reason 0x200:normal shutdown
Load averages: 1 minute 5 minute 15 minute
0.68 0.58 0.57

 

Jarrod@FW1> show chassis forwarding
FWDD status:
State Online
Microkernel CPU utilization 59 percent
Real-time threads CPU utilization 0 percent
Heap utilization 25 percent
Buffer utilization 1 percent
Uptime: 2 hours, 37 minutes, 2 seconds

Hi,

 

it looks better in D70 - as expected:

jh@fw> show chassis routing-engine
Routing Engine status:
    Temperature                 60 degrees C / 140 degrees F
    CPU temperature             76 degrees C / 168 degrees F
    Total memory              4096 MB Max  1065 MB used ( 26 percent)
      Control plane memory    2624 MB Max   682 MB used ( 26 percent)
      Data plane memory       1472 MB Max   383 MB used ( 26 percent)
    5 sec CPU utilization:
      User                      15 percent
      Background                 0 percent
      Kernel                    10 percent
      Interrupt                  0 percent
      Idle                      75 percent
    Model                          RE-SRX300
    Serial ID                      XXXXX
    Start time                     2016-12-20 07:29:39 CET
    Uptime                         47 minutes, 5 seconds
    Last reboot reason             0x200:normal shutdown
    Load averages:                 1 minute   5 minute  15 minute
                                       0.41       0.31       0.27

jh@fw>

 

Correct, I had a call logged with JTAC and they sent me D62.1 before it was released.  I have been running it for about 2 weeks now without issue.

 

Current time: 2016-12-20 05:30:36 ICT
Time Source:  NTP CLOCK
System booted: 2016-12-18 13:20:04 ICT (1d 16:10 ago)
Protocols started: 2016-12-18 13:20:04 ICT (1d 16:10 ago)
Last configured: 2016-12-10 15:37:12 ICT (1w2d 13:53 ago) by arbadmin
 5:30AM  up 1 day, 16:11, 1 user, load averages: 0.13, 0.10, 0.08

Information for junos:

Comment:
JUNOS Software Release [15.1X49-D62.1]

So I am guessing that D70 is either a rollup from D62.1 or has additional fixes in it.  Either way, it works, we have our devices set for etherswitching (for vlan functionality via irb) and its behaving.

D62 is a service release with just this fix ( and few more), and D70 is the regular Maintenance release with more features/ fixes.  D70 is now available for customer download http://www.juniper.net/support/downloads/?p=srx300#sw

Thanks for the update.  I will update my D62.1's to D70 during the next maintenance window

I wonder if the D70 is supported on the 300, it appears to only support 320 and higher....

Junos OS Release 15.1X49-D70 supports the following devices:
•Branch SRX Series devices: SRX300, SRX320, SRX340, SRX345, and SRX550
High Memory(SRX 550M)

Ref: Release notes for D70 - http://www.juniper.net/techpubs/en_US/junos15.1x49-d70/information-products/topic-collections/release-notes/15.1x49-d70/junos-release-notes-15.1X49-D70.pdf
Page : 4

It looks like an error on the download site. The image works for SRX300:

 

jh@fw> show version
Hostname: fw
Model: srx300
Junos: 15.1X49-D70.3
JUNOS Software Release [15.1X49-D70.3]

 

I installed the D70 on my SRX 300 this morning. It did solve the CPU issue, however I had to roll back as it sadly didnt agree with my current running configuration. I have copied the config errors that occured below. Perhaps I need to read the release notes, maybe I missed something. 

 

vlans {

vlan1 {

##

## Warning: Default bridge-domain mode without any vlan-id configuration is not supported.

## Warning: statement ignored: unsupported platform (srx300)

##

vlan-id none;

##

## Warning: l3-interface can be configured only under vlans with 'vlan-id'/'vlan-tags'

## Warning: l3-interface can be configured only under vlans with 'vlan-id'/'vlan-tags'

##

l3-interface irb.0;

}

}

 

 

Looks like you just need to assign a vlan-id on the vlan. Then the config should be OK.

Yeah I am going to try that. It works fine on the firmware im running now.  Its built on vlan-id 1. How ever when I show the set it shows vlan none....which  is odd. I am going to try vlan-id 2.

I was able to manipulate my configuration and get version 

 

Jarrod@FW1> show version 

Hostname: FW1

Model: srx300

Junos: 15.1X49-D70.3

JUNOS Software Release [15.1X49-D70.3]

 

to run.  The CPU usage is now perfect.

 

Jarrod@FW1> show chassis routing-engine
Routing Engine status:
Temperature 46 degrees C / 114 degrees F
CPU temperature 59 degrees C / 138 degrees F
Total memory 4096 MB Max 983 MB used ( 24 percent)
Control plane memory 2624 MB Max 656 MB used ( 25 percent)
Data plane memory 1472 MB Max 339 MB used ( 23 percent)
5 sec CPU utilization:
User 11 percent
Background 0 percent
Kernel 7 percent
Interrupt 0 percent
Idle 82 percent
Model RE-SRX300
Serial ID CV2116AF0542
Start time 2016-12-21 01:10:57 UTC
Uptime 18 minutes, 4 seconds
Last reboot reason 0x200:normal shutdown
Load averages: 1 minute 5 minute 15 minute
0.14 0.38 0.70

 

Awesome!

Thanks a lot for your inputs. We have tested the new version of the firmware, it's true that the high cpu and high memory usage is resolved.

But, in our tests, We have seen high latency when you do ping from the router itself.

This is a problem because We can't do benchmark or see real results from ping, mtr, etc.

Hello all,

 

I am facing same issue about high CPU utilization on Juniper SRX340. Below is some output from my SRX.

seyma@SRX340> show version 
Hostname: SRX340
Model: srx340
Junos: 15.1X49-D70.3
JUNOS Software Release [15.1X49-D70.3]

seyma@SRX340> show chassis routing-engine                      
Routing Engine status:
    Temperature                 39 degrees C / 102 degrees F
    CPU temperature             60 degrees C / 140 degrees F
    Total memory              4096 MB Max  1106 MB used ( 27 percent)
      Control plane memory    2624 MB Max   682 MB used ( 26 percent)
      Data plane memory       1472 MB Max   412 MB used ( 28 percent)
    5 sec CPU utilization:
      User                     100 percent
      Background                 0 percent
      Kernel                     0 percent
      Interrupt                  0 percent
      Idle                       0 percent
    Model                          RE-SRX340
    Serial ID                      CY3318AF0723
    Start time                     2019-05-18 14:24:09 HKT
    Uptime                         10 days, 22 hours, 6 minutes, 40 seconds
    Last reboot reason             0x1:power cycle/failure
    Load averages:                 1 minute   5 minute  15 minute
                                       0.99       1.00       1.00

seyma@SRX340> show system processes extensive 
last pid: 24051;  load averages:  0.99,  0.98,  0.99  up 10+22:11:07    12:25:01
149 processes: 19 running, 118 sleeping, 12 waiting

Mem: 289M Active, 169M Inact, 1579M Wired, 416M Cache, 112M Buf, 1525M Free
Swap:

  PID USERNAME     THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
 1829 root           6  76    0  1554M   471M RUN    0 857.9H 369.87% flowd_octeon_hm
   21 root           1 171   52     0K    16K RUN    0 177.8H  0.00% idle: cpu0
 1859 root           7  76    0 34520K  9288K select 0 151:29  0.00% aamwd
   23 root           1 -20 -139     0K    16K RUN    0 127:39  0.00% swi7: clock
 1818 root           1  76    0 39308K 14692K select 0  52:38  0.00% l2ald
 1820 root           1  76    0 60500K 22896K select 0  47:32  0.00% pfed
   25 root           1 -20 -139     0K    16K WAIT   0  43:01  0.00% swi7: +
 1889 root           1  76    0     0K    16K select 0  40:44  0.00% peerproxy00000001
   22 root           1 -40 -159     0K    16K WAIT   0  33:29  0.00% swi2: netisr 0
 1868 root           1  76    0 19004K  7876K select 0  29:06  0.00% license-check
 1821 root           1  76    0 21060K  9216K select 0  25:42  0.00% rmopd
    5 root           1 -16    0     0K    16K rtfifo 0  16:39  0.00% rtfifo_kern_recv
 1479 root           1  76    0 18260K  6572K select 0  16:03  0.00% eventd
 1817 root           1   4    0 91876K 57380K kqread 0  10:50  0.00% rpd
   18 root           1 171   52     0K    16K RUN    3   9:46  0.00% idle: cpu3
   19 root           1 171   52     0K    16K RUN    2   9:43  0.00% idle: cpu2
 1847 root           1  76    0 19792K  4604K select 0   8:39  0.00% shm-rtsdbd
 1824 root           1  76    0 22336K 10612K select 0   7:17  0.00% ppmd
 1858 root           7   8    0 97412K  6748K nanslp 0   5:41  0.00% ipfd

Based on above output, we could not find any process is eating much CPU except "flowd_octeon_hm". The transit traffic is not interrupt  but exception traffic is. This cause slowness of accessing to the device via SSH and our monitoring system is not be able to capure the traffic.

 

Kindly advise the next action I should do/check in order to resolve the issue.

 

seyma,

 

The output indicates 100% user CPU this generally indicate multiple user session idle on SRX box. 

You need to look at "show systme user" to see currently logged in and idle user.

-Logout the idle users via command "requset system logout user terminal termin-name.

 

Regarding the flowd process going high that should not be conern becuase the SRX have multiple flowd thread and the output that you see is cumulative output of multiple threads so it can reach above 100%.

Hi Seyma,

 

The RE output shows "User" to use 100% of the resource.

 

Apart from checking if there are any logged in user. Check if you have a lot of mdg/php process active.

 

show system processes extensive | match mgd

show system processes extensive | match php

 

These processes will stay in 0.0% and could be deep down in the "show .. extensive " command lisat.

If you find many, quick way to relieve the box would be to kill the stale mgd/php process with kill -9 <pid>

 

Regards,

 

Rahul

Hello,

 

This could be due to some user process being stuck. Flowd being high is normal.

 

I would suggest to kill the existing user (mgd) sessions. Here is a sample:

 

fw-user@srx> show system users no-resolve
2:04AM up 5 days, 18:13, 2 users, load averages: 0.23, 0.12, 0.09
USER     TTY     FROM      LOGIN@ IDLE WHAT
root        u0     - Sat07AM 3days cli
fw-user  p0 10.222.9.47    2:03AM - -cli (cli)

 

fw-user@srx> show system processes | grep mgd | except sbin
1958 ?? Ss 0:01.50 mgd: (mgd) (root)/dev/ttyu0 (mgd)
14316 ?? Ss 0:00.04 mgd: (mgd) (fw-user)/dev/ttyp0 (mgd)

 

start shell user root

kill 1958
kill 14316

 

Hope this helps. Regards,

 

Vikas