SRX

 View Only
last person joined: yesterday 

Ask questions and share experiences about the SRX Series, vSRX, and cSRX.
Expand all | Collapse all

SRX Branch + NSM + Logging + High CPU

  • 1.  SRX Branch + NSM + Logging + High CPU

    Posted 03-01-2011 17:12

    We are using 210B, 240h and 650 SRX's with NSM 10.2. We are running 10.3.R1.9 on the devices. Our issue is high CPU with and dropping traffic. We have been troubleshooting this issue for some time with JTAC. I'm trying to get a feel if anyone is running a similar environment with success. We are not doing IDP, ALG's, etc. These boxes are a basic firewall with logging and average 10% usage. 


    #NSM
    #logging


  • 2.  RE: SRX Branch + NSM + Logging + High CPU
    Best Answer

    Posted 03-04-2011 13:57

    SRX was designed to log using SYSLOG, so when you log useing NSM mode it logs to the local flash then during the rollup it converts the log to NSM format and sends it..

     

    Two solutions:

     

    1. Log directly to a syslog server instead

    2. NSM 2011.1 appears to support SRX sd-syslog format.



  • 3.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-10-2011 08:51

    Depends which CPU is high, the routing-engine or the flow CPU. I assume its the routing-engine CPU as that is what most of us suffer from.

     

    The best solution is to upgrade NSM to 2011.1 and use syslog to send traffic to the NSM server. This might require a few changes to the NSM setup, but will greatly reduce the load on the SRX. We are switching all our high-traffic devices to this new method, and can confirm that it works with SRX running 10.3R2 & 10.2R3. Sadly, NSM doesn't seem to parse the logs from earlier versions.



  • 4.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-20-2011 07:52

    Hi motd ,  do you mean that syslog server will receive traffic & send it to NSM , how to do that on NSM side ?



  • 5.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-20-2011 08:24

    NSM can be configured to receive syslog messages directly. See page 765 of

    http://www.juniper.net/techpubs/software/management/security-manager/nsm2011_1/nsm-admin-guide.pdf

     

    Its one setting on NSM to listen to the messages, the second part is configuring the SRX to stream messages directly to an external syslog server (nsm in this case), without passing the messages to the routing engine.



  • 6.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-23-2011 13:12

    "..with SRX running 10.3R2 & 10.2R3. Sadly, NSM doesn't seem to parse the logs from earlier versions"

     

    Do you have this working with the the SRX branch? We are running 10.2R3 and when trying to switch to "security log mode stream" the entire configuration is accepted but in the logs following the commit we see the message " Stream has no meaning when system-event-mode is on" and no logs forwarded to Syslog. We have not tested with JunOS 10.4R3 yet, we're waiting on the NSM schema to be released first.

     

    NSM will handle traffic logs from JunoS 10.x versions if you have configured the log mode to event and set the event rate to 1000 eps as recommended. NSM log categorization under event mode works well, but the standard STRM log categorization is basic. Hopefully once we can get stream working, both STRM & NSM logging will be categorized best.

     

    Here's our configuration (a reboot does not resolve this).

     

        set security log mode stream
        set security log format sd-syslog
        set security log source-address <source address of dmi device>
        set security log stream nsm-dataplane-log category all

        set security log stream nsm-dataplane-log severity info

        set security log stream nsm-dataplane-log format sd-syslog
        set security log stream nsm-dataplane-log host <ip of nsm devSvr>
        set security log stream nsm-dataplane-log port 5140


    The error in default-log-messages is

     

    "UI_CONFIGURATION_ERROR: Process: rtlogd, path: [edit security log], statement: stream nsm-dataplane-log, Stream has no meaning when system-event-mode is on"

     

     

     

     



  • 7.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-23-2011 13:46

    markpr...

       

    set security log stream nsm-dataplane-log severity info <==Change to none in NSM or omit from cli 

       



  • 8.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-23-2011 17:12

    I tried removing severity and setting the log mode to "stream" again and this this did not help on the 210H I tested with. I also tried a reboot and then adding in the minimal set of commands for a new syslog destination. This commits with success, no errors, but checking default-log-messages I get the same error.

     

    " UI_CONFIGURATION_ERROR: Process: rtlogd, path: [edit security log], statement: stream strm-stream-log, Stream has no meaning when system-event-mode is on"

     

    [edit security log]
    # show

    mode stream;
    format sd-syslog;
    source-address <source ip address of the srx210h>;

     

    stream strm-stream-log {
        format sd-syslog;
        host {
            <ip address of strm>;
        }
    }

     

    I thought this may be a problem with 10.2r3 so upgraded the same SRX210H to 10.4r3.4 but I still get the same error. Here's the syslog configuration, in event mode traffic logs are written into default-log-messages. When the configuration is in stream mode, the errors are written here but no traffic logs appear (or are forwarded to the syslog destination).

     

    [edit system syslog]
    # show

    file default-log-messages {
        any any;
        match "!(.*dfwd_ev_client.*)";
        structured-data;
    }

     

    I believe the releases notes and PR that putting the SRX into stream mode will reduce the CPU and still support logging to an NSM2011.1 - if we can get the SRX branch into stream mode. I don't have an SRX HE available in this environment to test with.

     

     

     

     



  • 9.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-29-2011 13:29

    Hi motd ,

     

    And when turning NSM to a sysog server from page   765  as  you mentioned , i will configure its ip at SRx as external syslog srever , right ?

     

    Also will i be able to view these logs from Log viewer Tab (  the same as if i was adding the SRX to NSM ) ?

     



  • 10.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-30-2011 08:01

    Yes, only one change is required on the NSM server itself.

     

    The other changes you are making is simply instructing the SRX to send its traffic logs to a syslog server instead of logging them locally to disk or flash.

     

    The logs show up in the log viewer just like before



  • 11.  RE: SRX Branch + NSM + Logging + High CPU

     
    Posted 07-04-2011 16:59

    Hello,

    I'm having issues with this.  We had our SRX set to log to NSM using event mode.  This was causing high CPU so i have made the changes suggested in this thread. 

     

    We have our SRXs running in a HA cluster.  I have put the configuration under each node in the groups configuration section.  NSM has also been changed to allow syslog over UDP. 

     

    Also - We are managing the SRX cluster via the FXP, and NSM is using the FXP of each node for management as well.

     

    Here is the config:


     show configuration groups node0 security log
    mode stream;
    format sd-syslog;
    source-address 10.150.7.18;
    stream nsm-stream {
        category all;
        host {
            10.203.0.34;
            port 5140;


    show configuration groups node0 system syslog

    file default-log-messages {
        any any;
        structured-data;

     

    This configuration is the same on node1.

    In NSM all i see is logs from source 0.0.0.0 destintion 0.0.0.0.  Nothing else.

     

    Any thoughts on this?

    Setting it to stream mode has fixed the CPU issue, but now logs do not appear properly.



  • 12.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 05-11-2011 13:39

    If anyone else is struggling with this, the configurations detailed here are good and we are successfully logging in stream mode to external syslog servers and to NSM2011.1 now - but there are a few caveats as of JunOS 10.4

     

    1) " UI_CONFIGURATION_ERROR: Process: rtlogd, path: [edit security log], statement: stream strm-stream-log, Stream has no meaning when system-event-mode is on" - this is a bogus error and can be ignored. It appears whether the system is in log mode stream or log mode event.

     

    2) default-log-messages - in log mode event, the traffic logs appear here by default (or more correctly the control plane logs appear here). When in log mode stream, the traffic logs are sent straight from the dataplane and traffic logs do not appear here. Assume this means the webui will not show the traffic logs either.

     

    3) stream mode only supported outbound on a physical interface not over a VPN (st interfaces). If you have branch offices connected via VPN you will not be able forward traffic logs in stream mode. You have to use event mode if you want to collect those traffic logs. This was our big problem in testing out stream mode.

     

    We noticed some small differences in the data contents in structured (stream) vs non-structured (event) but the STRM DSM and NSM does support both.

     



  • 13.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-23-2011 13:30

    "Our issue is high CPU with and dropping traffic"

     

    We haven't seen this in our enviroinments, but we are sticking on 10.2R3 and selectively choose NSM releases that work best for us. 2010.4q14 and above was a good release and 2011.1 is looking very good for JunOS device management at last. There is a PR that references high CPU with event mode logging to NSM and the recommendation is NSM2011.1 with the SRX in stream mode as others have stated here - if you can get this to work (let us know if you do).

     



  • 14.  RE: SRX Branch + NSM + Logging + High CPU

    Posted 03-23-2011 13:44

    UPDATE

     

    Per SomeITGuy's post I pursued to look over the release notes for 2011.1. We've upgraded to this release. I can report this is working well for us. We are still running 10.3.R1.9 on the branch devices. We found what may be a bug, that you must set the severity to none in order for this to work.