SRX

Expand all | Collapse all

High CPU root cause SRX 220

  • 1.  High CPU root cause SRX 220

    Posted 11-25-2017 03:44

    Hello all,

     

    I have an SRX220 running high CPU. Find out because off complaints that the traffic between two vlan's was considerd very slow.

    Eventd takes a lot of CPU without being configured. But my best guess it is because the packet fragmentation. I'm not sure how to determine where this happens. So I changed the following settings;

    reduced the st0 interface mtu tot 1420 from default

    reduced the tcp-mms all-tcp to 1300 from default

    reduced the ipsec-vpn mss to 1350 from default

     

    But there is no major decrease in utilization. Any tips how to find out where the high cpu usage comes from?

     

    Model: srx220h
    JUNOS Software Release [12.1X46-D50.4]

     

    Process

     

      PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
     1420 root        5  76    0   516M 58864K select 0    ??? 109.72% flowd_octeon_hm
     1440 root        1 127    0 13836K  2988K RUN    0 2577.8 30.03% rtlogd
    57306 root        1 127    0 13036K  5284K RUN    0 539:25 27.78% eventd
       22 root        1 171   52     0K    16K RUN    0 3360.0  6.01% idle: cpu0

     

    forwarding 

     

    root@srx220> show chassis forwarding
    FWDD status:
      State                                 Online
      Microkernel CPU utilization        18 percent
      Real-time threads CPU utilization 100 percent
      Heap utilization                   53 percent
      Buffer utilization                  1 percent
      Uptime:                               515 days, 2 hours, 33 minutes, 55 seconds

     

    Routing-Engine (this is during the weekend, within business hours idle wil flat out 0% the entire day).

     

    root@srx220> show chassis routing-engine
    Routing Engine status:
        Temperature                 51 degrees C / 123 degrees F
        Total memory              1024 MB Max   768 MB used ( 75 percent)
          Control plane memory     544 MB Max   511 MB used ( 94 percent)
          Data plane memory        480 MB Max   254 MB used ( 53 percent)
        CPU utilization:
          User                      31 percent
          Background                 0 percent
          Kernel                    60 percent
          Interrupt                  1 percent
          Idle                       9 percent
        Model                          RE-SRX220H
        Serial ID                      AAFK6429
        Start time                     2016-06-28 07:23:36 GMT+1
        Uptime                         515 days, 2 hours, 46 minutes, 39 seconds
        Last reboot reason             0x20:power-button soft power off
        Load averages:                 1 minute   5 minute  15 minute
                                           2.10       2.12       2.13
    root@srx220> show security flow statistics
        Current sessions: 299
        Packets forwarded: 245878319729
        Packets dropped: 606183520
        Fragment packets: 18446744072410236784

     

     

    VPN assosiations
     <131076 ESP:aes-cbc-128/sha1 - 48530/unlim - root 500 x.x.x.x    (to SSG5)
      >131076 ESP:aes-cbc-128/sha1 - 48530/unlim - root 500 x.x.x.x
    <131074 ESP:3des/sha1 - 3283/ unlim - root 500 x.x.x.x (to SRX220/proposal standard) >131074 ESP:3des/sha1 - 3283/ unlim - root 500 x.x.x.x
    <131073 ESP:3des/sha1 - 1683/ unlim - root 500 x.x.x.x (to SRX220/proposal standard) >131073 ESP:3des/sha1 - 1683/ unlim - root 500 x.x.x.x

     

     

    Config items

     

    root@srx220> show configuration security flow
    tcp-mss {
        all-tcp {
            mss 1300;
        }
        ipsec-vpn {
            mss 1350;
      
    VPN Interfaces
    unit 2 {
        description gw_;
        family inet {
            mtu 1420;
        }
    root@srx220> show configuration system syslog
    archive size 100k files 3;
    user * {
    any emergency;
    }
    file messages {
    any critical;
    authorization info;
    }
    file interactive-commands {
    interactive-commands error;
    }
    file kmd-logs {
    daemon info;
    match KMD;

     

     

    Update:

    Ok this is wierd. I though of rebooting/updating so I checked the partitions. Backup partition was empty so I sliced it with the primary snapshot to be sure. After that the syslog process is using 0% CPU.

    Other values dropped also:

    Real-time threads CPU utilization 1 percent 

     

    processes:

    1420 root 5 76 0 516M 58864K select 0 ??? 99.17% flowd_octeon_hm
    22 root 1 171 52 0K 16K RUN 0 3360.5 81.20% idle: cpu0

     

    Ping trough the vpn decreased from 100ms+ to a acceptable 8ms.

    So I have to wait to business hours to check if the CPU will increase again.

     

    Any thought about the fragmentation will be appreciated. 

     



  • 2.  RE: High CPU root cause SRX 220

     
    Posted 11-25-2017 04:37

    The eventd process takes care of logging and traps.  Typically when this is running high on a branch device I have seen logging enabled on security policies that have just too many session events for the device to handle.

     

    Take a look at your security policy "then log" statements and turn off those that you can afford to not have logged any more to reduce the load.

     



  • 3.  RE: High CPU root cause SRX 220

    Posted 11-25-2017 04:49

    Hi Spuluka,

     

    Thank you, I forgot to mention I already disabled all the policy logging yesterday and restarted the syslog daemon.

     

    root@srx220> show security policies | match log

    root@srx220>

     

    Maybe syslogd restarted again because of the partition touch.



  • 4.  RE: High CPU root cause SRX 220

     
    Posted 11-25-2017 06:30

    Not much else feeds eventd.

     

    https://www.juniper.net/documentation/en_US/junos/topics/concept/junos-script-automation-event-notifications-and-policy-overview.html

     

    So with eventd causing the cpu, there needs to be something generating lots of events. Check screens and local logs to see what is showing up.

     

    Your uptime is also very high so maybe a system reboot would help.

     



  • 5.  RE: High CPU root cause SRX 220

    Posted 11-25-2017 09:23

    If you are screening good amount traffic via flow module of SRX then also you may observe the high CPU.

     

    if possible for you, you may selectively bypass the traffic across fw to reduce cpu util.

     

    Follow the kb fyr..

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB26757



  • 6.  RE: High CPU root cause SRX 220

     
    Posted 11-27-2017 21:17

    Fragmentation is on dataplane and I dont expect that to cause high control plane CPU.

     

    The process eventd and rtlogd are high, these process mainly deal with logging and if reboot fix the issue most possible reason is the process were stuck.