SRX

 View Only
last person joined: yesterday 

Ask questions and share experiences about the SRX Series, vSRX, and cSRX.
Back to discussions
Expand all | Collapse all

SRX 240 CPU_THRESHOLD_EXCEEDED

  • 1.  SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 00:54
      |   view attached

    Hello, somebody help me please with this problem.  Avarage CPU load was doubled. First peack on graphic (in attached file) cause is enabled security traceoptions with packet mode logging, after this traceoptions and logging was disabled and cpu loading back to normal. Second peak caused by nothing,  were not configuration changes. Reboot device didnt help.

     

    Spoiler
    show chassis routing-engine
    Routing Engine status:
    Temperature 47 degrees C / 116 degrees F
    CPU temperature 45 degrees C / 113 degrees F
    Total memory 2048 MB Max 1044 MB used ( 51 percent)
    Control plane memory 976 MB Max 488 MB used ( 50 percent)
    Data plane memory 1072 MB Max 557 MB used ( 52 percent)
    CPU utilization:
    User 19 percent
    Background 0 percent
    Kernel 42 percent
    Interrupt 0 percent
    Idle 39 percent
    Model RE-SRX240H2
    Serial ID ACAL8054
    Start time 2019-02-05 20:59:39 MSK
    Uptime 15 hours, 50 minutes, 48 seconds
    Last reboot reason Router rebooted after a normal shutdown.
    Load averages: 1 minute 5 minute 15 minute
    0.79 0.64 0.64

     

     

    Spoiler

    top -H
    last pid: 3123; load averages: 0.65, 0.64, 0.64 up 0+15:52:20 12:51:30
    79 processes: 5 running, 72 sleeping, 2 zombie
    CPU states: 78.3% user, 0.0% nice, 13.5% system, 0.3% interrupt, 7.9% idle
    Mem: 192M Active, 122M Inact, 1162M Wired, 216M Cache, 112M Buf, 281M Free
    Swap:

    PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    1656 root 139 0 1124M 88076K CPU1 1 55.1H 93.95% flowd_octeon_hm
    1656 root 139 0 1124M 88076K CPU2 2 55.1H 90.19% flowd_octeon_hm
    1656 root 139 0 1124M 88076K CPU3 3 55.1H 90.19% flowd_octeon_hm
    1656 root 123 0 1124M 88076K RUN 0 55.1H 51.86% flowd_octeon_hm
    1656 root 76 0 1124M 88076K select 0 55.1H 0.00% flowd_octeon_hm
    1656 root 76 0 1124M 88076K ucondt 0 55.1H 0.00% flowd_octeon_hm
    1656 root 76 0 1124M 88076K select 0 55.1H 0.00% flowd_octeon_hm
    1652 root 76 0 18500K 9028K select 0 23:24 0.00% ppmd
    1644 root 4 0 60200K 30316K kqread 0 7:58 0.00% rpd
    1643 root 76 0 33428K 12196K select 0 7:43 0.00% mib2d
    1642 root 76 0 23888K 14672K select 0 4:05 0.00% snmpd
    1686 root 76 0 14464K 6140K select 0 3:42 0.00% license-check
    1645 root 76 0 22644K 9584K select 0 1:45 0.00% l2ald
    1637 root 76 0 128M 19524K select 0 1:30 0.00% chassisd
    1669 root 76 0 17792K 4072K select 0 1:22 0.00% shm-rtsdbd
    1680 root 76 0 19684K 8292K select 0 1:15 0.00% utmd
    1648 root 76 0 30112K 10460K select 0 1:07 0.00% pfed
    1649 root 76 0 15888K 7308K select 0 1:06 0.00% rmopd
    1688 root 4 0 25920K 13648K kqread 0 0:53 0.00% eswd
    1685 root 4 0 11756K 5516K kqread 0 0:49 0.00% mcsnoopd
    1651 root 76 0 32812K 15552K select 0 0:46 0.00% kmd
    1638 root 76 0 14400K 5464K select 0 0:35 0.00% alarmd
    1683 root 8 0 29848K 4640K nanslp 0 0:34 0.00% wmic
    1634 root 76 0 3348K 1392K select 0 0:33 0.00% bslockd
    1677 root 76 0 78828K 5580K select 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1679 root 76 0 15524K 6580K select 0 0:27 0.00% rtlogd
    1667 root 76 0 27064K 10000K select 0 0:19 0.00% smid
    1654 root 76 0 9196K 3692K select 0 0:14 0.00% irsd
    1698 nobody 4 0 10612K 1488K kqread 0 0:13 0.00% webapid
    1636 root 76 0 39768K 9000K select 0 0:12 0.00% dcd
    1274 root 76 0 15084K 4736K select 0 0:12 0.00% eventd
    1672 root 76 0 16928K 6468K select 0 0:10 0.00% pkid
    1671 root 76 0 28788K 13248K select 0 0:07 0.00% nsd
    1640 root 76 0 7704K 5596K select 0 0:07 0.00% ntpd

     

    Spoiler
     show security monitoring fpc 0
    FPC 0
    PIC 0
    CPU utilization : 67 %
    Memory utilization : 52 %
    Current flow session : 41156
    Current flow session IPv4: 41156
    Current flow session IPv6: 0
    Max flow session : 131072
    Total Session Creation Per Second (for last 96 seconds on average): 691
    IPv4 Session Creation Per Second (for last 96 seconds on average): 691
    IPv6 Session Creation Per Second (for last 96 seconds on average): 0
    Spoiler
    show version
    Hostname: 
    Model: srx240h2
    JUNOS Software Release [12.3X48-D70.3]

     

     

    From syslog:

     

    PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87

    PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99



  • 2.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 03:00

    The process flowd is the traffic processing engine and the syslog verifies that the high cpu is from processing traffic on fpc 0 pic 0 which means interfaces starting with the 0/0/x designation.

     

    So the issue will be traffic processing related.  Do you have monitoring on the ports to see where the flows that generate the high cpu may be coming from?

     

    You will want to exam any deep inspection policies that may be hitting these flows to see if they need to be adjusted if this is a regular occurance and affecting performance.

     



  • 3.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 04:57

    Thanks, trafic on interfaces as usual, DPI disabled.

     

    Spoiler

    show security idp status
    State of IDP: Default, Up since: 2019-02-05 21:03:34 MSK (19:34:10 ago)

    Packets/second: 0 Peak: 0 @ 2019-02-06 15:23:18 MSK
    KBits/second : 0 Peak: 0 @ 2019-02-06 15:23:18 MSK
    Latency (microseconds): [min: 0] [max: 0] [avg: 0]

    Packet Statistics:
    [ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]

    Flow Statistics:
    ICMP: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]
    TCP: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]
    UDP: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]
    Other: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]

    Session Statistics:
    [ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]
    Policy Name : none

    Spoiler
    show security monitoring performance spu
    fpc 0 pic 0
    Last 60 seconds:
    0: 38 1: 37 2: 51 3: 46 4: 57 5: 48
    6: 54 7: 51 8: 48 9: 35 10: 36 11: 42
    12: 67 13: 61 14: 55 15: 54 16: 53 17: 54
    18: 42 19: 32 20: 31 21: 53 22: 51 23: 57
    24: 53 25: 49 26: 49 27: 53 28: 39 29: 39
    30: 49 31: 51 32: 58 33: 67 34: 53 35: 55
    36: 78 37: 68 38: 57 39: 57 40: 71 41: 69
    42: 68 43: 65 44: 62 45: 67 46: 59 47: 48
    48: 50 49: 55 50: 69 51: 63 52: 64 53: 64
    54: 60 55: 64 56: 63 57: 58 58: 60 59: 75


  • 4.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 10:51

    Hi, beh0ld

     

    Can you confirm if the issue is constantly happening? If so lets gather the following commands:

     

    > show system uptime

    > show log messages | match  RTPERF_CPU_THRESHOLD_EXCEEDED | last 10

    > show security monitoring performance spu

    > monitor interface traffic (Press letter B to reflect bits per second )

     

    I am looking to isolate a capacity problem as explained in the following post:

     

      https://forums.juniper.net/t5/SRX-Services-Gateway/SRX1500-Capactiy/m-p/338608#M50534

     

     

     

     



  • 5.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 22:46

    Hi, spuluka

    Yes, i confirm, constantly happen about a half month. Nothing extraordinary in NMS graphics. 

    Spoiler
    show system uptime
    Current time: 2019-02-07 10:40:00 MSK
    System booted: 2019-02-05 20:59:40 MSK (1d 13:40 ago)
    Protocols started: 2019-02-05 21:03:30 MSK (1d 13:36 ago)
    Last configured: 2019-02-06 19:22:33 MSK (15:17:27 ago) by root
    10:40AM up 1 day, 13:40, 2 users, load averages: 0.93, 0.73, 0.63
    show log messages | match RTPERF_CPU_THRESHOLD_EXCEEDED | last 10
    Feb 7 10:22:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:22:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:22:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:23:01 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 10:25:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 10:30:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:30:37 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:37:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:37:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 10:37:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85

    how security monitoring performance spu
    fpc 0 pic 0
    Last 60 seconds:
    0: 57 1: 64 2: 91 3: 97 4: 94 5: 92
    6: 69 7: 57 8: 56 9: 47 10: 41 11: 46
    12: 56 13: 61 14: 62 15: 60 16: 67 17: 59
    18: 66 19: 45 20: 45 21: 55 22: 68 23: 59
    24: 55 25: 66 26: 58 27: 55 28: 42 29: 39
    30: 51 31: 56 32: 56 33: 58 34: 64 35: 64
    36: 62 37: 65 38: 53 39: 65 40: 59 41: 52
    42: 55 43: 56 44: 54 45: 64 46: 64 47: 56
    48: 60 49: 56 50: 57 51: 68 52: 67 53: 63
    54: 66 55: 60 56: 63 57: 51 58: 47 59: 44

    Bytes=b, Clear=c, Delta=d, Packets=p, Quit=q or ESC, Rate=r, Up=^U, Down=^D
    Seconds: 9 Time: 10:44:59

    Interface Link Input bytes (bps) Output bytes (bps)
    ge-0/0/0 Up 2558263426 (497168) 19827703128 (9528848)
    gr-0/0/0 Up 73326302 (29536) 143008471 (13952)
    ip-0/0/0 Up 0 (0) 0 (0)
    lsq-0/0/0 Up 0 (0) 0 (0)
    lt-0/0/0 Up 0 (0) 0 (0)
    mt-0/0/0 Up 0 (0) 0 (0)
    sp-0/0/0 Up 0 (0) 0 (0)
    ge-0/0/1 Up 186770229752 (13372952) 281493838087 (81210024)
    ge-0/0/2 Up 207984967156 (72853152) 130825737418 (5327792)
    ge-0/0/3 Up 81465039721 (9478760) 180633447718 (15423792)
    ge-0/0/4 Up 190613271956 (28431000) 88609308918 (7764256)
    ge-0/0/5 Up 42193797476 (1521680) 8605508250 (482760)
    ge-0/0/6 Up 102543839421 (9013896) 113897375600 (11080784)
    ge-0/0/7 Up 0 (0) 0 (0)
    ge-0/0/8 Up 131347449279 (12540368) 118313240434 (10523304)
    ge-0/0/9 Down 0 (0) 0 (0)
    ge-0/0/10 Down 0 (0) 0 (0)
    ge-0/0/11 Down 0 (0) 0 (0)
    ge-0/0/12 Down 0 (0) 0 (0)
    ge-0/0/13 Down 0 (0) 0 (0)
    ge-0/0/14 Down 0 (0) 0 (0)
    ge-0/0/15 Down 0 (0) 0 (0)
    ae0 Up 268235269473 (22851712) 462127285805 (96633816)
    ae1 Up 398598239112 (101284152) 219435046336 (13092048)
    fxp2 Up 0 6784600
    gre Up 0 0
    ipip Up 0 0
    irb Up 0 0
    lo0 Up 40013092 40013092
    lsi Up 0 0
    mtun Up 0 0
    pimd Up 0 0
    pime Up 0 0
    pp0 Up 0 (0) 0 (0)
    ppd0 Up 0 (0) 0 (0)
    ppe0 Up 0 (0) 0 (0)
    st0 Up 116899963287 (10872080) 105477004230 (9058640)
    tap Up 0 0
    vlan Up 2869979589587 (107720848) 649737156915 (99937968)

     








  • 6.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 23:37

    Hi, beh0ld

     

    The sum of the bps ingressing the SRX doesnt seem to be over 600Mbps wich is the troughput (IMIX) supported for the SRX240:

     

        https://www.juniper.net/assets/uk/en/local/pdf/datasheets/1000281-en.pdf

     

    Can you let us know what interfaces are part of ae0 and ae7?

    Can you let us know that is the physical interface for the VPN related to st0 interface?

    Please take 2 more times the "monitor interface traffic" command. I want to have a better picture of the bps received by the SRX and to make sure we are not relying on an output taken when the problem was not happening.

     

    Please also share the following commands:

     

    > request pfe execute target fwdd command “show pfe statistics traffic” 

    > request pfe execute target fwdd command “show octeon cpu detail all”

    > request pfe execute target fwdd command “show octeon session detail”

    > request pfe execute target fwdd command “show threads”

    > request pfe execute target fwdd command “show services mum”

    > show system processes extensive | except 0.0

    > show system routing-engine

    > show configuration | match idp

    > show configuration | match utm

    > show configuration | match session-init

    > show configuration | match session-close

    > show configuration | match traceoptions

     

     



  • 7.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-07-2019 00:21

     

    Spoiler

    set interfaces ge-0/0/1 description "aenet --> ae0.0"
    set interfaces ge-0/0/1 gigether-options 802.3ad ae0
    set interfaces ge-0/0/2 description "aenet --> ae1.0"
    set interfaces ge-0/0/2 gigether-options 802.3ad ae1
    set interfaces ge-0/0/3 description "aenet --> ae0.0"
    set interfaces ge-0/0/3 gigether-options 802.3ad ae0
    set interfaces ge-0/0/4 description "aenet --> ae1.0"
    set interfaces ge-0/0/4 gigether-options 802.3ad ae1

    ---------------------------------------------------------------------------------------------------------------

    ISP for VPN

    set interfaces ge-0/0/8 unit 69 vlan-id 69

    -----------------------------------------------------------------------------------------------------

    Bytes=b, Clear=c, Delta=d, Packets=p, Quit=q or ESC, Rate=r, Up=^U, Down=^D
     Seconds: 22 Time: 12:04:56

    Interface Link Input bytes (bps) Output bytes (bps)
    ge-0/0/0 Up 2642119102 (318368) 20513348552 (220112)
    gr-0/0/0 Up 87306701 (1112) 218046161 (1368)
    ip-0/0/0 Up 0 (0) 0 (0)
    lsq-0/0/0 Up 0 (0) 0 (0)
    lt-0/0/0 Up 0 (0) 0 (0)
    mt-0/0/0 Up 0 (0) 0 (0)
    sp-0/0/0 Up 0 (0) 0 (0)
    ge-0/0/1 Up 196018792158 (15950280) 298257251963 (100461336)
    ge-0/0/2 Up 221258408810 (91608928) 139129419811 (7461456)
    ge-0/0/3 Up 88425583475 (10311272) 193457469800 (17072096)
    ge-0/0/4 Up 202926658401 (19643632) 96711243580 (10077384)
    ge-0/0/5 Up 47711929983 (2619400) 9261801229 (4399704)
    ge-0/0/6 Up 110443279211 (32141704) 122756010273 (22467904)
    ge-0/0/7 Up 0 (0) 0 (0)
    ge-0/0/8 Up 141960082483 (25190760) 127852128798 (34784952)
    ge-0/0/9 Down 0 (0) 0 (0)
    ge-0/0/10 Down 0 (0) 0 (0)
    ge-0/0/11 Down 0 (0) 0 (0)
    ge-0/0/12 Down 0 (0) 0 (0)
    ge-0/0/13 Down 0 (0) 0 (0)
    ge-0/0/14 Down 0 (0) 0 (0)
    ge-0/0/15 Down 0 (0) 0 (0)
    ae0 Up 284444375633 (26261552) 491714721763 (117533432)
    ae1 Up 424185067211 (111252560) 235840663391 (17538840)

    fxp2 Up 0 7024500
    gre Up 0 0
    ipip Up 0 0
    irb Up 0 0
    lo0 Up 41391244 41391244
    lsi Up 0 0
    mtun Up 0 0
    pimd Up 0 0
    pime Up 0 0
    pp0 Up 0 (0) 0 (0)
    ppd0 Up 0 (0) 0 (0)
    ppe0 Up 0 (0) 0 (0)
    st0 Up 126340558679 (22473856) 113957598801 (32240456)
    tap Up 0 0
    vlan Up 2909662905321 (114397464) 693617729443 (112920744)

    ----------------------------------------------------------------------------------------------------------------------------------------------

    Bytes=b, Clear=c, Delta=d, Packets=p, Quit=q or ESC, Rate=r, Up=^U, Down=^D
    JR2-NNDC Seconds: 43 Time: 12:06:59

    Interface Link Input bytes (bps) Output bytes (bps)
    ge-0/0/0 Up 2644516962 (493720) 20528558931 (9143064)
    gr-0/0/0 Up 87376560 (4208) 218097056 (3848)
    ip-0/0/0 Up 0 (0) 0 (0)
    lsq-0/0/0 Up 0 (0) 0 (0)
    lt-0/0/0 Up 0 (0) 0 (0)
    mt-0/0/0 Up 0 (0) 0 (0)
    sp-0/0/0 Up 0 (0) 0 (0)
    ge-0/0/1 Up 196297959772 (21395416) 299004483124 (29981136)
    ge-0/0/2 Up 221902051915 (21627344) 139338808221 (12896592)
    ge-0/0/3 Up 88644397734 (8354416) 193854648767 (10787360)
    ge-0/0/4 Up 203307326718 (24050704) 96908550983 (8726112)
    ge-0/0/5 Up 47776113639 (1414248) 9282724869 (383416)
    ge-0/0/6 Up 111029437402 (33648080) 123071911723 (25538240)
    ge-0/0/7 Up 0 (0) 0 (0)
    ge-0/0/8 Up 142336589266 (30149688) 128498323734 (37917688)
    ge-0/0/9 Down 0 (0) 0 (0)
    ge-0/0/10 Down 0 (0) 0 (0)
    ge-0/0/11 Down 0 (0) 0 (0)
    ge-0/0/12 Down 0 (0) 0 (0)
    ge-0/0/13 Down 0 (0) 0 (0)
    ge-0/0/14 Down 0 (0) 0 (0)
    ge-0/0/15 Down 0 (0) 0 (0)
    ae0 Up 284942357506 (29749832) 492859131891 (40768496)
    ae1 Up 425209378633 (45678048) 236247359204 (21622704)
    fxp2 Up 0 7030600
    gre Up 0 0
    ipip Up 0 0
    irb Up 0 0
    lo0 Up 41426516 41426516
    lsi Up 0 0
    mtun Up 0 0
    pimd Up 0 0
    pime Up 0 0
    pp0 Up 0 (0) 0 (0)
    ppd0 Up 0 (0) 0 (0)
    ppe0 Up 0 (0) 0 (0)
    st0 Up 126673111541 (27159272) 114560320377 (35151816)
    tap Up 0 0
    vlan Up 2911138287132 (69604376) 695122310536 (61814496)

    -----------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show pfe statistics traffic"
    ================ master ================
    SENT: Ukern command: show pfe statistics traffic

    PFE Traffic statistics:
    2125899558 packets input (25364 packets/sec)
    2377220990 packets output (29378 packets/sec)

    PFE Local Traffic statistics:
    9478871 local packets input
    3671662 local packets output
    0 software input control high drops
    0 software input high drops
    11591 software input medium drops
    0 software input low drops
    192 software output drops
    656394 hardware input drops
    299583093 Notification/control packet drops in ISR

    PFE Local Protocol statistics:
    0 hdlc keepalives
    0 atm oam
    0 fr lmi
    0 ppp lcp/ncp
    2725957 ospf hello
    0 ospf3 hello
    0 rsvp hello
    0 ldp hello
    0 bfd
    0 isis iih
    564197 lacp
    2786172 arp
    0 ether oam
    0 synce
    0 gre keepalives
    0 unknown

    PFE Hardware Discard statistics:
    0 timeout
    0 truncated key
    0 bits to test
    0 data error
    0 stack underflow
    0 stack overflow
    5545893 regular discard
    43416 extended/illegal nexthop discard
    0 invalid iif
    0 info cell drops
    0 input drops
    0 fabric drops
    0 aged packets


    PFE Input IPv4 Header Checksum Error and Output MTU Error statistics:
    0 IPv4 header checksum error
    0 MTU error

    ---------------------------------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show octeon cpu detail all"
    ================ master ================
    SENT: Ukern command: show octeon cpu detail all

    Average Thread(1) Utilization: 3%
    Last 60 seconds:
    0: 4 1: 4 2: 4 3: 4 4: 5 5: 4
    6: 5 7: 5 8: 4 9: 4 10: 4 11: 4
    12: 3 13: 3 14: 3 15: 4 16: 4 17: 4
    18: 3 19: 4 20: 4 21: 3 22: 3 23: 3
    24: 4 25: 4 26: 3 27: 4 28: 3 29: 3
    30: 4 31: 4 32: 3 33: 3 34: 4 35: 4
    36: 4 37: 4 38: 4 39: 4 40: 4 41: 4
    42: 4 43: 4 44: 4 45: 4 46: 4 47: 4
    48: 4 49: 4 50: 5 51: 4 52: 4 53: 3
    54: 4 55: 3 56: 4 57: 4 58: 4 59: 3

    Last 60 minutes:
    0: 3 1: 3 2: 4 3: 4 4: 5 5: 4
    6: 4 7: 6 8: 4 9: 4 10: 4 11: 4
    12: 3 13: 3 14: 3 15: 3 16: 5 17: 3
    18: 3 19: 3 20: 3 21: 3 22: 3 23: 3
    24: 3 25: 3 26: 3 27: 3 28: 3 29: 3
    30: 3 31: 3 32: 3 33: 3 34: 2 35: 4
    36: 4 37: 4 38: 3 39: 4 40: 5 41: 4
    42: 4 43: 3 44: 4 45: 4 46: 4 47: 3
    48: 4 49: 4 50: 3 51: 3 52: 3 53: 4
    54: 3 55: 3 56: 3 57: 4 58: 4 59: 5

    Last 24 hours:
    0: 3 1: 3 2: 1 3: 0 4: 0 5: 0
    6: 1 7: 0 8: 0 9: 0 10: 0 11: 0
    12: 0 13: 0 14: 0 15: 0 16: 2 17: 2
    18: 3 19: 3 20: 3 21: 3 22: 3 23: 3

    Average Thread(2) Utilization: 68%
    Last 60 seconds:
    0: 68 1: 71 2: 75 3: 61 4: 76 5: 69
    6: 84 7: 87 8: 79 9: 78 10: 76 11: 68
    12: 62 13: 57 14: 58 15: 66 16: 67 17: 68
    18: 66 19: 69 20: 74 21: 66 22: 53 23: 52
    24: 66 25: 69 26: 62 27: 66 28: 64 29: 66
    30: 65 31: 62 32: 57 33: 47 34: 78 35: 71
    36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    48: 68 49: 67 50: 76 51: 73 52: 66 53: 70
    54: 71 55: 63 56: 72 57: 69 58: 68 59: 59

    Last 60 minutes:
    0: 66 1: 67 2: 69 3: 69 4: 84 5: 72
    6: 76 7: 90 8: 78 9: 78 10: 77 11: 71
    12: 60 13: 58 14: 59 15: 60 16: 75 17: 56
    18: 62 19: 60 20: 59 21: 59 22: 56 23: 55
    24: 61 25: 59 26: 58 27: 56 28: 56 29: 58
    30: 59 31: 56 32: 60 33: 57 34: 53 35: 63
    36: 65 37: 63 38: 60 39: 61 40: 71 41: 68
    42: 68 43: 65 44: 64 45: 67 46: 63 47: 64
    48: 68 49: 67 50: 64 51: 63 52: 63 53: 69
    54: 64 55: 60 56: 63 57: 67 58: 72 59: 73

    Last 24 hours:
    0: 63 1: 59 2: 45 3: 31 4: 25 5: 30
    6: 36 7: 29 8: 27 9: 24 10: 25 11: 24
    12: 26 13: 24 14: 26 15: 32 16: 45 17: 54
    18: 58 19: 67 20: 60 21: 59 22: 64 23: 63

    Average Thread(3) Utilization: 68%
    Last 60 seconds:
    0: 69 1: 72 2: 75 3: 61 4: 76 5: 69
    6: 85 7: 87 8: 79 9: 78 10: 76 11: 68
    12: 62 13: 57 14: 58 15: 67 16: 67 17: 68
    18: 66 19: 69 20: 74 21: 65 22: 53 23: 53
    24: 66 25: 70 26: 62 27: 67 28: 63 29: 66
    30: 65 31: 63 32: 57 33: 47 34: 78 35: 71
    36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    48: 68 49: 67 50: 76 51: 73 52: 66 53: 69
    54: 71 55: 63 56: 72 57: 69 58: 68 59: 59

    Last 60 minutes:
    0: 66 1: 67 2: 68 3: 70 4: 84 5: 72
    6: 76 7: 90 8: 78 9: 78 10: 77 11: 71
    12: 60 13: 58 14: 58 15: 60 16: 75 17: 56
    18: 62 19: 60 20: 59 21: 59 22: 56 23: 55
    24: 61 25: 59 26: 58 27: 56 28: 56 29: 58
    30: 60 31: 56 32: 60 33: 57 34: 53 35: 64
    36: 65 37: 63 38: 60 39: 61 40: 71 41: 68
    42: 68 43: 65 44: 64 45: 67 46: 63 47: 64
    48: 68 49: 67 50: 64 51: 63 52: 63 53: 69
    54: 64 55: 60 56: 63 57: 67 58: 72 59: 73

    Last 24 hours:
    0: 63 1: 59 2: 45 3: 31 4: 25 5: 30
    6: 36 7: 29 8: 27 9: 24 10: 25 11: 24
    12: 26 13: 24 14: 26 15: 32 16: 45 17: 54
    18: 58 19: 68 20: 60 21: 59 22: 64 23: 63

    Average Utilization: 68%
    Last 60 seconds:
    0: 68 1: 71 2: 75 3: 61 4: 76 5: 69
    6: 84 7: 87 8: 79 9: 78 10: 76 11: 68
    12: 62 13: 57 14: 58 15: 67 16: 67 17: 68
    18: 66 19: 69 20: 74 21: 65 22: 53 23: 53
    24: 66 25: 69 26: 62 27: 66 28: 63 29: 66
    30: 65 31: 62 32: 57 33: 47 34: 78 35: 71
    36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    48: 68 49: 67 50: 76 51: 73 52: 66 53: 69
    54: 71 55: 63 56: 72 57: 69 58: 68 59: 59

    Last 60 minutes:
    0: 66 1: 67 2: 70 3: 70 4: 81 5: 75
    6: 77 7: 90 8: 77 9: 78 10: 78 11: 69
    12: 60 13: 59 14: 58 15: 60 16: 75 17: 55
    18: 62 19: 59 20: 60 21: 58 22: 56 23: 56
    24: 61 25: 59 26: 59 27: 55 28: 56 29: 59
    30: 58 31: 57 32: 59 33: 56 34: 54 35: 63
    36: 66 37: 62 38: 59 39: 62 40: 72 41: 66
    42: 69 43: 65 44: 64 45: 68 46: 63 47: 64
    48: 68 49: 68 50: 63 51: 64 52: 64 53: 68
    54: 64 55: 61 56: 63 57: 67 58: 74 59: 72

    Last 24 hours:
    0: 63 1: 59 2: 45 3: 31 4: 25 5: 30
    6: 36 7: 28 8: 27 9: 24 10: 25 11: 24
    12: 26 13: 24 14: 26 15: 32 16: 45 17: 54
    18: 58 19: 68 20: 60 21: 59 22: 64 23: 63

    ------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show octeon session detail"
    ================ master ================
    SENT: Ukern command: show octeon session detail

    Average session Utilization: 44794
    Last 60 seconds:
    0: 42711 1: 43504 2: 42762 3: 43340 4: 44166 5: 43494
    6: 44200 7: 43416 8: 44117 9: 43423 10: 44201 11: 43521
    12: 44356 13: 43575 14: 44329 15: 43469 16: 44200 17: 43415
    18: 44435 19: 45207 20: 44478 21: 45117 22: 44303 23: 45058
    24: 44123 25: 44707 26: 43926 27: 44778 28: 43996 29: 44990
    30: 44236 31: 45026 32: 44289 33: 44990 34: 44538 35: 44836
    36: 45861 37: 45144 38: 46077 39: 45392 40: 46105 41: 45391
    42: 46103 43: 45303 44: 45885 45: 45093 46: 45812 47: 45056
    48: 46296 49: 45497 50: 46231 51: 45535 52: 46224 53: 46768
    54: 45957 55: 46401 56: 45402 57: 45913 58: 45137 59: 45841

    Last 60 minutes:
    0: 45269 1: 42308 2: 40867 3: 41160 4: 41368 5: 42430
    6: 40647 7: 40208 8: 41343 9: 42520 10: 43734 11: 44906
    12: 46931 13: 42855 14: 41512 15: 40549 16: 40872 17: 40649
    18: 41408 19: 41054 20: 39974 21: 41412 22: 41099 23: 40991
    24: 40381 25: 39808 26: 40069 27: 40790 28: 40514 29: 39676
    30: 40445 31: 42210 32: 40937 33: 40594 34: 40007 35: 40201
    36: 39686 37: 39711 38: 38881 39: 39464 40: 41435 41: 41946
    42: 41600 43: 40636 44: 41792 45: 40671 46: 41216 47: 41232
    48: 40794 49: 42033 50: 45228 51: 45975 52: 44010 53: 44572
    54: 40350 55: 40103 56: 40724 57: 42055 58: 40999 59: 40786

    Last 24 hours:
    0: 41361 1: 40098 2: 32246 3: 24208 4: 20914 5: 20143
    6: 19845 7: 19981 8: 19971 9: 20225 10: 20641 11: 20427
    12: 21191 13: 21580 14: 22183 15: 24692 16: 31761 17: 38090
    18: 38925 19: 40307 20: 38770 21: 39640 22: 40529 23: 41090

    --------------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show threads"
    ================ master ================
    SENT: Ukern command: show threads

    PID PR State Name Stack Use Time (Last/Max/Total) cpu
    --- -- ------- --------------------- --------- ---------------------
    1 H ready Maintenance 1312/73824 0/8/393 ms 0%
    2 L running Idle 4720/73824 0/15/1233350 ms 0%
    3 H asleep Timer Services 1240/73824 0/8/6910 ms 0%
    5 L asleep Ukern Syslog 760/73824 0/0/0 ms 0%
    6 L asleep Sheaf Background 1112/73824 0/8/744 ms 0%
    7 M asleep mac_db 760/73824 0/0/0 ms 0%
    8 M asleep Docsis 1056/73824 0/8/2410 ms 0%
    9 M asleep ATMX 1840/73824 0/8/7650 ms 0%
    10 M asleep XDSL 4728/73824 0/8/444736 ms 0%
    11 M asleep DSX50ms 1752/73824 0/8/50120 ms 0%
    12 M asleep DSXonesec 1328/73824 0/8/2435 ms 0%
    13 M asleep SFP 1232/73824 0/8/3525 ms 0%
    14 M asleep Ethernet 4648/73824 0/15/1093209 ms 0%
    15 M asleep RSMON syslog thread 2448/73824 0/8/15 ms 0%
    16 L asleep Syslog 4640/73824 0/8/1554 ms 0%
    17 M ready Fwdd Notif Recv 5144/73824 0/16/4521382 ms 3%
    18 M asleep Forwarding Thread 4792/73824 0/8/725924 ms 0%
    19 M asleep Periodic 12992/73824 0/23/239628 ms 0%
    20 M asleep bcmCNTR.0 4912/16384 8/22/4808211 ms 3%
    21 M asleep bcmTX 912/16384 0/0/0 ms 0%
    22 M asleep bcmXGS3AsyncTX 944/16384 0/0/0 ms 0%
    23 M asleep bcmLINK.0 4352/16384 0/68/279867 ms 0%
    24 L asleep BCM SDK periodic 1784/73824 0/8/667 ms 0%
    25 M asleep FPC_IPC-Thread 2328/73824 0/0/0 ms 0%
    26 H asleep TTP Receive 4704/73824 0/30/840265 ms 0%
    27 H asleep TTP Transmit 4704/73824 0/38/742839 ms 0%
    28 M asleep UDP Input 856/73824 0/0/0 ms 0%
    29 H asleep TCP Timers 4640/73824 0/8/6450 ms 0%
    30 H asleep TCP Receive 760/73824 0/0/0 ms 0%
    33 M asleep Service Set Cleanup 880/73824 0/0/0 ms 0%
    35 L asleep Monitor sort 2480/73824 0/8/2287 ms 0%
    36 H asleep Monitor Exporter 1800/73824 0/8/4403 ms 0%
    37 M asleep Monitor Expire 1080/73824 0/8/464 ms 0%
    38 M asleep Monitor Expire TCP 2368/73824 0/8/2333 ms 0%
    39 L asleep Monitor Aged 1056/73824 0/8/38 ms 0%
    40 H asleep Monitor sampledc 864/73824 0/0/0 ms 0%
    41 M asleep Monitor v8 1072/73824 0/8/22 ms 0%
    42 M asleep Monitor v5 to v8 1784/73824 0/8/265 ms 0%
    47 L asleep Console 3504/73824 0/0/0 ms 0%
    48 M asleep L2ALM Manager 3088/73824 0/8/54081 ms 0%
    49 L asleep DFW Alert 13264/73824 0/8/4052 ms 0%
    50 L asleep CLNS Err Input 848/73824 0/0/0 ms 0%
    51 L asleep CLNS Option Input 848/73824 0/0/0 ms 0%
    52 M asleep PIC Periodic 1376/73824 0/8/3627 ms 0%
    53 M asleep PIC 760/73824 0/0/0 ms 0%
    54 M asleep FWDD CM 4632/73824 0/1542/50306 ms 0%
    55 L asleep ICMP6 Input 1016/73824 0/0/0 ms 0%
    56 L asleep IP6 Option Input 2936/73824 0/0/0 ms 0%
    57 L asleep ICMP Input 2944/73824 0/8/220358 ms 0%
    58 L asleep IP Option Input 2936/73824 0/0/0 ms 0%
    59 M asleep IGMP Input 2936/73824 0/0/0 ms 0%
    60 M asleep VRRP Manager 792/73824 0/0/0 ms 0%
    61 M asleep PPM Manager 3032/73824 0/0/0 ms 0%
    62 M asleep Wheel timer 2920/73824 0/8/2926 ms 0%
    63 H asleep JFlow v9 export 1408/73824 0/8/2293 ms 0%
    65 H asleep Jflow v9 PFE Poll 960/73824 0/0/0 ms 0%
    66 M asleep PPM Data thread 4640/73824 0/8/12032 ms 0%
    67 L asleep NH Probe Service 776/73824 0/0/0 ms 0%
    69 L asleep Virtual Console 2168/73824 0/0/0 ms 0%
    71 M asleep Msg relay 2272/73824 0/0/0 ms 0%
    72 M asleep IPSec 4504/73824 0/8/32917 ms 0%
    76 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    80 M asleep DA-PFE 2416/73824 0/8/2868 ms 0%
    81 M asleep PFE Manager 15448/73824 0/3211/480653 ms 0%
    82 M asleep L2HA TOGGLE 1800/73824 0/8/5745 ms 0%
    83 M asleep USP Trace 2384/73824 0/8/11576 ms 0%
    84 M asleep PFE Statistics 6072/73824 8/15/307012 ms 0%
    85 M asleep Userfw Msgrelay 2352/73824 0/0/0 ms 0%
    86 H asleep Services UI 2632/73824 0/0/0 ms 0%
    87 H asleep Monitor UI 2712/73824 0/0/0 ms 0%
    88 H asleep Jflow v9 UI Daemon 3256/73824 0/0/0 ms 0%
    89 H asleep USP IPC Server 3104/73824 7/7/7 ms 0%
    90 M asleep NSD-PFE 8360/73824 0/31/6094 ms 0%
    91 M asleep IDP PFEMAN Thread 13424/73824 0/8/798 ms 0%
    92 M asleep utm-pfe 4640/73824 0/8/3162 ms 0%
    93 H asleep USP IPC Server 3104/73824 0/0/0 ms 0%
    94 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    95 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    96 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    99 L asleep BCM AGING periodic 2632/73824 0/211/87310 ms 0%
    1555 L running Cattle-Prod Daemon 4240/73824 0/8/15 ms 0%
    1587 H asleep USP IPC Server 2264/73824 8/8/8 ms 0%
    1588 L asleep Cattle-Prod Daemon 3304/73824 0/0/0 ms 0%

    ----------------------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show services mum"
    ================ master ================
    SENT: Ukern command: show services mum

    Memory usage manager: gsm
    Total free space to start with: 177182704
    Active customers: 1
    Max customers: 12
    Yellow zone limit: 44295676
    Orange zone limit: 31892886
    Red zone limit: 15946443
    Operational zone: Green

    cust_id in use limit
    -------------------------------
    0 914167 177182704
    1 632 177182704
    -------------------------------

    actual free space = 135948952
    est. free space = 135948952

    Proactive reclaim : ENABLED
    Seconds between reclaims : 5
    Number of proactive reclaims : 0

    -------------------------------------------------------------------------------------------------------

    show system processes extensive | except 0.0
    last pid: 6421; load averages: 0.92, 1.05, 0.94 up 1+15:18:51 12:18:01
    153 processes: 19 running, 122 sleeping, 1 zombie, 11 waiting

    Mem: 227M Active, 122M Inact, 1170M Wired, 221M Cache, 112M Buf, 232M Free
    Swap:


    PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    1656 root 7 76 0 1124M 88140K select 0 136.7H 325.49% flowd_octeon_hm
    22 root 1 171 52 0K 16K RUN 0 843:02 33.79% idle: cpu0

    -------------------------------------------------------------------------------------------------------

    show chassis routing-engine
    Routing Engine status:
    Temperature 49 degrees C / 120 degrees F
    CPU temperature 47 degrees C / 116 degrees F
    Total memory 2048 MB Max 1065 MB used ( 52 percent)
    Control plane memory 976 MB Max 517 MB used ( 53 percent)
    Data plane memory 1072 MB Max 557 MB used ( 52 percent)
    CPU utilization:
    User 21 percent
    Background 0 percent
    Kernel 43 percent
    Interrupt 0 percent
    Idle 36 percent
    Model RE-SRX240H2
    Serial ID ACAL8054
    Start time 2019-02-05 20:59:39 MSK
    Uptime 1 day, 15 hours, 18 minutes, 39 seconds
    Last reboot reason Router rebooted after a normal shutdown.
    Load averages: 1 minute 5 minute 15 minute
    0.80 1.01 0.93

    --------------------------------------------------------------------------------------------------------------------

    root@> show configuration | match idp

    root@> show configuration | match utm

    root@> show configuration | match session-init

    root@> show configuration | match session-close

    root@> show configuration | match traceoptions

     

     



  • 8.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-07-2019 08:22