SRX

Expand all | Collapse all

SRX 240 CPU_THRESHOLD_EXCEEDED

Jump to Best Answer
  • 1.  SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 00:54
      |   view attached

    Hello, somebody help me please with this problem.  Avarage CPU load was doubled. First peack on graphic (in attached file) cause is enabled security traceoptions with packet mode logging, after this traceoptions and logging was disabled and cpu loading back to normal. Second peak caused by nothing,  were not configuration changes. Reboot device didnt help.

     

    Spoiler
    show chassis routing-engine
    Routing Engine status:
    Temperature 47 degrees C / 116 degrees F
    CPU temperature 45 degrees C / 113 degrees F
    Total memory 2048 MB Max 1044 MB used ( 51 percent)
    Control plane memory 976 MB Max 488 MB used ( 50 percent)
    Data plane memory 1072 MB Max 557 MB used ( 52 percent)
    CPU utilization:
    User 19 percent
    Background 0 percent
    Kernel 42 percent
    Interrupt 0 percent
    Idle 39 percent
    Model RE-SRX240H2
    Serial ID ACAL8054
    Start time 2019-02-05 20:59:39 MSK
    Uptime 15 hours, 50 minutes, 48 seconds
    Last reboot reason Router rebooted after a normal shutdown.
    Load averages: 1 minute 5 minute 15 minute
    0.79 0.64 0.64

     

     

    Spoiler

    top -H
    last pid: 3123; load averages: 0.65, 0.64, 0.64 up 0+15:52:20 12:51:30
    79 processes: 5 running, 72 sleeping, 2 zombie
    CPU states: 78.3% user, 0.0% nice, 13.5% system, 0.3% interrupt, 7.9% idle
    Mem: 192M Active, 122M Inact, 1162M Wired, 216M Cache, 112M Buf, 281M Free
    Swap:

    PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    1656 root 139 0 1124M 88076K CPU1 1 55.1H 93.95% flowd_octeon_hm
    1656 root 139 0 1124M 88076K CPU2 2 55.1H 90.19% flowd_octeon_hm
    1656 root 139 0 1124M 88076K CPU3 3 55.1H 90.19% flowd_octeon_hm
    1656 root 123 0 1124M 88076K RUN 0 55.1H 51.86% flowd_octeon_hm
    1656 root 76 0 1124M 88076K select 0 55.1H 0.00% flowd_octeon_hm
    1656 root 76 0 1124M 88076K ucondt 0 55.1H 0.00% flowd_octeon_hm
    1656 root 76 0 1124M 88076K select 0 55.1H 0.00% flowd_octeon_hm
    1652 root 76 0 18500K 9028K select 0 23:24 0.00% ppmd
    1644 root 4 0 60200K 30316K kqread 0 7:58 0.00% rpd
    1643 root 76 0 33428K 12196K select 0 7:43 0.00% mib2d
    1642 root 76 0 23888K 14672K select 0 4:05 0.00% snmpd
    1686 root 76 0 14464K 6140K select 0 3:42 0.00% license-check
    1645 root 76 0 22644K 9584K select 0 1:45 0.00% l2ald
    1637 root 76 0 128M 19524K select 0 1:30 0.00% chassisd
    1669 root 76 0 17792K 4072K select 0 1:22 0.00% shm-rtsdbd
    1680 root 76 0 19684K 8292K select 0 1:15 0.00% utmd
    1648 root 76 0 30112K 10460K select 0 1:07 0.00% pfed
    1649 root 76 0 15888K 7308K select 0 1:06 0.00% rmopd
    1688 root 4 0 25920K 13648K kqread 0 0:53 0.00% eswd
    1685 root 4 0 11756K 5516K kqread 0 0:49 0.00% mcsnoopd
    1651 root 76 0 32812K 15552K select 0 0:46 0.00% kmd
    1638 root 76 0 14400K 5464K select 0 0:35 0.00% alarmd
    1683 root 8 0 29848K 4640K nanslp 0 0:34 0.00% wmic
    1634 root 76 0 3348K 1392K select 0 0:33 0.00% bslockd
    1677 root 76 0 78828K 5580K select 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1677 root 8 0 78828K 5580K nanslp 0 0:30 0.00% ipfd
    1679 root 76 0 15524K 6580K select 0 0:27 0.00% rtlogd
    1667 root 76 0 27064K 10000K select 0 0:19 0.00% smid
    1654 root 76 0 9196K 3692K select 0 0:14 0.00% irsd
    1698 nobody 4 0 10612K 1488K kqread 0 0:13 0.00% webapid
    1636 root 76 0 39768K 9000K select 0 0:12 0.00% dcd
    1274 root 76 0 15084K 4736K select 0 0:12 0.00% eventd
    1672 root 76 0 16928K 6468K select 0 0:10 0.00% pkid
    1671 root 76 0 28788K 13248K select 0 0:07 0.00% nsd
    1640 root 76 0 7704K 5596K select 0 0:07 0.00% ntpd

     

    Spoiler
     show security monitoring fpc 0
    FPC 0
    PIC 0
    CPU utilization : 67 %
    Memory utilization : 52 %
    Current flow session : 41156
    Current flow session IPv4: 41156
    Current flow session IPv6: 0
    Max flow session : 131072
    Total Session Creation Per Second (for last 96 seconds on average): 691
    IPv4 Session Creation Per Second (for last 96 seconds on average): 691
    IPv6 Session Creation Per Second (for last 96 seconds on average): 0
    Spoiler
    show version
    Hostname: 
    Model: srx240h2
    JUNOS Software Release [12.3X48-D70.3]

     

     

    From syslog:

     

    PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87

    PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99



  • 2.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

     
    Posted 02-06-2019 03:00

    The process flowd is the traffic processing engine and the syslog verifies that the high cpu is from processing traffic on fpc 0 pic 0 which means interfaces starting with the 0/0/x designation.

     

    So the issue will be traffic processing related.  Do you have monitoring on the ports to see where the flows that generate the high cpu may be coming from?

     

    You will want to exam any deep inspection policies that may be hitting these flows to see if they need to be adjusted if this is a regular occurance and affecting performance.

     



  • 3.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 04:57

    Thanks, trafic on interfaces as usual, DPI disabled.

     

    Spoiler

    show security idp status
    State of IDP: Default, Up since: 2019-02-05 21:03:34 MSK (19:34:10 ago)

    Packets/second: 0 Peak: 0 @ 2019-02-06 15:23:18 MSK
    KBits/second : 0 Peak: 0 @ 2019-02-06 15:23:18 MSK
    Latency (microseconds): [min: 0] [max: 0] [avg: 0]

    Packet Statistics:
    [ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]

    Flow Statistics:
    ICMP: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]
    TCP: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]
    UDP: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]
    Other: [Current: 0] [Max: 0 @ 2019-02-06 15:23:18 MSK]

    Session Statistics:
    [ICMP: 0] [TCP: 0] [UDP: 0] [Other: 0]
    Policy Name : none

    Spoiler
    show security monitoring performance spu
    fpc 0 pic 0
    Last 60 seconds:
    0: 38 1: 37 2: 51 3: 46 4: 57 5: 48
    6: 54 7: 51 8: 48 9: 35 10: 36 11: 42
    12: 67 13: 61 14: 55 15: 54 16: 53 17: 54
    18: 42 19: 32 20: 31 21: 53 22: 51 23: 57
    24: 53 25: 49 26: 49 27: 53 28: 39 29: 39
    30: 49 31: 51 32: 58 33: 67 34: 53 35: 55
    36: 78 37: 68 38: 57 39: 57 40: 71 41: 69
    42: 68 43: 65 44: 62 45: 67 46: 59 47: 48
    48: 50 49: 55 50: 69 51: 63 52: 64 53: 64
    54: 60 55: 64 56: 63 57: 58 58: 60 59: 75


  • 4.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 10:51

    Hi, beh0ld

     

    Can you confirm if the issue is constantly happening? If so lets gather the following commands:

     

    > show system uptime

    > show log messages | match  RTPERF_CPU_THRESHOLD_EXCEEDED | last 10

    > show security monitoring performance spu

    > monitor interface traffic (Press letter B to reflect bits per second )

     

    I am looking to isolate a capacity problem as explained in the following post:

     

      https://forums.juniper.net/t5/SRX-Services-Gateway/SRX1500-Capactiy/m-p/338608#M50534

     

     

     

     



  • 5.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 22:46

    Hi, spuluka

    Yes, i confirm, constantly happen about a half month. Nothing extraordinary in NMS graphics. 

    Spoiler
    show system uptime
    Current time: 2019-02-07 10:40:00 MSK
    System booted: 2019-02-05 20:59:40 MSK (1d 13:40 ago)
    Protocols started: 2019-02-05 21:03:30 MSK (1d 13:36 ago)
    Last configured: 2019-02-06 19:22:33 MSK (15:17:27 ago) by root
    10:40AM up 1 day, 13:40, 2 users, load averages: 0.93, 0.73, 0.63
    show log messages | match RTPERF_CPU_THRESHOLD_EXCEEDED | last 10
    Feb 7 10:22:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:22:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:22:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:23:01 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 10:25:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 10:30:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:30:37 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:37:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:37:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 10:37:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85

    how security monitoring performance spu
    fpc 0 pic 0
    Last 60 seconds:
    0: 57 1: 64 2: 91 3: 97 4: 94 5: 92
    6: 69 7: 57 8: 56 9: 47 10: 41 11: 46
    12: 56 13: 61 14: 62 15: 60 16: 67 17: 59
    18: 66 19: 45 20: 45 21: 55 22: 68 23: 59
    24: 55 25: 66 26: 58 27: 55 28: 42 29: 39
    30: 51 31: 56 32: 56 33: 58 34: 64 35: 64
    36: 62 37: 65 38: 53 39: 65 40: 59 41: 52
    42: 55 43: 56 44: 54 45: 64 46: 64 47: 56
    48: 60 49: 56 50: 57 51: 68 52: 67 53: 63
    54: 66 55: 60 56: 63 57: 51 58: 47 59: 44

    Bytes=b, Clear=c, Delta=d, Packets=p, Quit=q or ESC, Rate=r, Up=^U, Down=^D
    Seconds: 9 Time: 10:44:59

    Interface Link Input bytes (bps) Output bytes (bps)
    ge-0/0/0 Up 2558263426 (497168) 19827703128 (9528848)
    gr-0/0/0 Up 73326302 (29536) 143008471 (13952)
    ip-0/0/0 Up 0 (0) 0 (0)
    lsq-0/0/0 Up 0 (0) 0 (0)
    lt-0/0/0 Up 0 (0) 0 (0)
    mt-0/0/0 Up 0 (0) 0 (0)
    sp-0/0/0 Up 0 (0) 0 (0)
    ge-0/0/1 Up 186770229752 (13372952) 281493838087 (81210024)
    ge-0/0/2 Up 207984967156 (72853152) 130825737418 (5327792)
    ge-0/0/3 Up 81465039721 (9478760) 180633447718 (15423792)
    ge-0/0/4 Up 190613271956 (28431000) 88609308918 (7764256)
    ge-0/0/5 Up 42193797476 (1521680) 8605508250 (482760)
    ge-0/0/6 Up 102543839421 (9013896) 113897375600 (11080784)
    ge-0/0/7 Up 0 (0) 0 (0)
    ge-0/0/8 Up 131347449279 (12540368) 118313240434 (10523304)
    ge-0/0/9 Down 0 (0) 0 (0)
    ge-0/0/10 Down 0 (0) 0 (0)
    ge-0/0/11 Down 0 (0) 0 (0)
    ge-0/0/12 Down 0 (0) 0 (0)
    ge-0/0/13 Down 0 (0) 0 (0)
    ge-0/0/14 Down 0 (0) 0 (0)
    ge-0/0/15 Down 0 (0) 0 (0)
    ae0 Up 268235269473 (22851712) 462127285805 (96633816)
    ae1 Up 398598239112 (101284152) 219435046336 (13092048)
    fxp2 Up 0 6784600
    gre Up 0 0
    ipip Up 0 0
    irb Up 0 0
    lo0 Up 40013092 40013092
    lsi Up 0 0
    mtun Up 0 0
    pimd Up 0 0
    pime Up 0 0
    pp0 Up 0 (0) 0 (0)
    ppd0 Up 0 (0) 0 (0)
    ppe0 Up 0 (0) 0 (0)
    st0 Up 116899963287 (10872080) 105477004230 (9058640)
    tap Up 0 0
    vlan Up 2869979589587 (107720848) 649737156915 (99937968)

     








  • 6.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-06-2019 23:37

    Hi, beh0ld

     

    The sum of the bps ingressing the SRX doesnt seem to be over 600Mbps wich is the troughput (IMIX) supported for the SRX240:

     

        https://www.juniper.net/assets/uk/en/local/pdf/datasheets/1000281-en.pdf

     

    Can you let us know what interfaces are part of ae0 and ae7?

    Can you let us know that is the physical interface for the VPN related to st0 interface?

    Please take 2 more times the "monitor interface traffic" command. I want to have a better picture of the bps received by the SRX and to make sure we are not relying on an output taken when the problem was not happening.

     

    Please also share the following commands:

     

    request pfe execute target fwdd command “show pfe statistics traffic” 

    request pfe execute target fwdd command “show octeon cpu detail all”

    request pfe execute target fwdd command “show octeon session detail”

    request pfe execute target fwdd command “show threads”

    request pfe execute target fwdd command “show services mum”

    > show system processes extensive | except 0.0

    > show system routing-engine

    > show configuration | match idp

    > show configuration | match utm

    > show configuration | match session-init

    > show configuration | match session-close

    > show configuration | match traceoptions

     

     



  • 7.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-07-2019 00:21

     

    Spoiler

    set interfaces ge-0/0/1 description "aenet --> ae0.0"
    set interfaces ge-0/0/1 gigether-options 802.3ad ae0
    set interfaces ge-0/0/2 description "aenet --> ae1.0"
    set interfaces ge-0/0/2 gigether-options 802.3ad ae1
    set interfaces ge-0/0/3 description "aenet --> ae0.0"
    set interfaces ge-0/0/3 gigether-options 802.3ad ae0
    set interfaces ge-0/0/4 description "aenet --> ae1.0"
    set interfaces ge-0/0/4 gigether-options 802.3ad ae1

    ---------------------------------------------------------------------------------------------------------------

    ISP for VPN

    set interfaces ge-0/0/8 unit 69 vlan-id 69

    -----------------------------------------------------------------------------------------------------

    Bytes=b, Clear=c, Delta=d, Packets=p, Quit=q or ESC, Rate=r, Up=^U, Down=^D
     Seconds: 22 Time: 12:04:56

    Interface Link Input bytes (bps) Output bytes (bps)
    ge-0/0/0 Up 2642119102 (318368) 20513348552 (220112)
    gr-0/0/0 Up 87306701 (1112) 218046161 (1368)
    ip-0/0/0 Up 0 (0) 0 (0)
    lsq-0/0/0 Up 0 (0) 0 (0)
    lt-0/0/0 Up 0 (0) 0 (0)
    mt-0/0/0 Up 0 (0) 0 (0)
    sp-0/0/0 Up 0 (0) 0 (0)
    ge-0/0/1 Up 196018792158 (15950280) 298257251963 (100461336)
    ge-0/0/2 Up 221258408810 (91608928) 139129419811 (7461456)
    ge-0/0/3 Up 88425583475 (10311272) 193457469800 (17072096)
    ge-0/0/4 Up 202926658401 (19643632) 96711243580 (10077384)
    ge-0/0/5 Up 47711929983 (2619400) 9261801229 (4399704)
    ge-0/0/6 Up 110443279211 (32141704) 122756010273 (22467904)
    ge-0/0/7 Up 0 (0) 0 (0)
    ge-0/0/8 Up 141960082483 (25190760) 127852128798 (34784952)
    ge-0/0/9 Down 0 (0) 0 (0)
    ge-0/0/10 Down 0 (0) 0 (0)
    ge-0/0/11 Down 0 (0) 0 (0)
    ge-0/0/12 Down 0 (0) 0 (0)
    ge-0/0/13 Down 0 (0) 0 (0)
    ge-0/0/14 Down 0 (0) 0 (0)
    ge-0/0/15 Down 0 (0) 0 (0)
    ae0 Up 284444375633 (26261552) 491714721763 (117533432)
    ae1 Up 424185067211 (111252560) 235840663391 (17538840)

    fxp2 Up 0 7024500
    gre Up 0 0
    ipip Up 0 0
    irb Up 0 0
    lo0 Up 41391244 41391244
    lsi Up 0 0
    mtun Up 0 0
    pimd Up 0 0
    pime Up 0 0
    pp0 Up 0 (0) 0 (0)
    ppd0 Up 0 (0) 0 (0)
    ppe0 Up 0 (0) 0 (0)
    st0 Up 126340558679 (22473856) 113957598801 (32240456)
    tap Up 0 0
    vlan Up 2909662905321 (114397464) 693617729443 (112920744)

    ----------------------------------------------------------------------------------------------------------------------------------------------

    Bytes=b, Clear=c, Delta=d, Packets=p, Quit=q or ESC, Rate=r, Up=^U, Down=^D
    JR2-NNDC Seconds: 43 Time: 12:06:59

    Interface Link Input bytes (bps) Output bytes (bps)
    ge-0/0/0 Up 2644516962 (493720) 20528558931 (9143064)
    gr-0/0/0 Up 87376560 (4208) 218097056 (3848)
    ip-0/0/0 Up 0 (0) 0 (0)
    lsq-0/0/0 Up 0 (0) 0 (0)
    lt-0/0/0 Up 0 (0) 0 (0)
    mt-0/0/0 Up 0 (0) 0 (0)
    sp-0/0/0 Up 0 (0) 0 (0)
    ge-0/0/1 Up 196297959772 (21395416) 299004483124 (29981136)
    ge-0/0/2 Up 221902051915 (21627344) 139338808221 (12896592)
    ge-0/0/3 Up 88644397734 (8354416) 193854648767 (10787360)
    ge-0/0/4 Up 203307326718 (24050704) 96908550983 (8726112)
    ge-0/0/5 Up 47776113639 (1414248) 9282724869 (383416)
    ge-0/0/6 Up 111029437402 (33648080) 123071911723 (25538240)
    ge-0/0/7 Up 0 (0) 0 (0)
    ge-0/0/8 Up 142336589266 (30149688) 128498323734 (37917688)
    ge-0/0/9 Down 0 (0) 0 (0)
    ge-0/0/10 Down 0 (0) 0 (0)
    ge-0/0/11 Down 0 (0) 0 (0)
    ge-0/0/12 Down 0 (0) 0 (0)
    ge-0/0/13 Down 0 (0) 0 (0)
    ge-0/0/14 Down 0 (0) 0 (0)
    ge-0/0/15 Down 0 (0) 0 (0)
    ae0 Up 284942357506 (29749832) 492859131891 (40768496)
    ae1 Up 425209378633 (45678048) 236247359204 (21622704)
    fxp2 Up 0 7030600
    gre Up 0 0
    ipip Up 0 0
    irb Up 0 0
    lo0 Up 41426516 41426516
    lsi Up 0 0
    mtun Up 0 0
    pimd Up 0 0
    pime Up 0 0
    pp0 Up 0 (0) 0 (0)
    ppd0 Up 0 (0) 0 (0)
    ppe0 Up 0 (0) 0 (0)
    st0 Up 126673111541 (27159272) 114560320377 (35151816)
    tap Up 0 0
    vlan Up 2911138287132 (69604376) 695122310536 (61814496)

    -----------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show pfe statistics traffic"
    ================ master ================
    SENT: Ukern command: show pfe statistics traffic

    PFE Traffic statistics:
    2125899558 packets input (25364 packets/sec)
    2377220990 packets output (29378 packets/sec)

    PFE Local Traffic statistics:
    9478871 local packets input
    3671662 local packets output
    0 software input control high drops
    0 software input high drops
    11591 software input medium drops
    0 software input low drops
    192 software output drops
    656394 hardware input drops
    299583093 Notification/control packet drops in ISR

    PFE Local Protocol statistics:
    0 hdlc keepalives
    0 atm oam
    0 fr lmi
    0 ppp lcp/ncp
    2725957 ospf hello
    0 ospf3 hello
    0 rsvp hello
    0 ldp hello
    0 bfd
    0 isis iih
    564197 lacp
    2786172 arp
    0 ether oam
    0 synce
    0 gre keepalives
    0 unknown

    PFE Hardware Discard statistics:
    0 timeout
    0 truncated key
    0 bits to test
    0 data error
    0 stack underflow
    0 stack overflow
    5545893 regular discard
    43416 extended/illegal nexthop discard
    0 invalid iif
    0 info cell drops
    0 input drops
    0 fabric drops
    0 aged packets


    PFE Input IPv4 Header Checksum Error and Output MTU Error statistics:
    0 IPv4 header checksum error
    0 MTU error

    ---------------------------------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show octeon cpu detail all"
    ================ master ================
    SENT: Ukern command: show octeon cpu detail all

    Average Thread(1) Utilization: 3%
    Last 60 seconds:
    0: 4 1: 4 2: 4 3: 4 4: 5 5: 4
    6: 5 7: 5 8: 4 9: 4 10: 4 11: 4
    12: 3 13: 3 14: 3 15: 4 16: 4 17: 4
    18: 3 19: 4 20: 4 21: 3 22: 3 23: 3
    24: 4 25: 4 26: 3 27: 4 28: 3 29: 3
    30: 4 31: 4 32: 3 33: 3 34: 4 35: 4
    36: 4 37: 4 38: 4 39: 4 40: 4 41: 4
    42: 4 43: 4 44: 4 45: 4 46: 4 47: 4
    48: 4 49: 4 50: 5 51: 4 52: 4 53: 3
    54: 4 55: 3 56: 4 57: 4 58: 4 59: 3

    Last 60 minutes:
    0: 3 1: 3 2: 4 3: 4 4: 5 5: 4
    6: 4 7: 6 8: 4 9: 4 10: 4 11: 4
    12: 3 13: 3 14: 3 15: 3 16: 5 17: 3
    18: 3 19: 3 20: 3 21: 3 22: 3 23: 3
    24: 3 25: 3 26: 3 27: 3 28: 3 29: 3
    30: 3 31: 3 32: 3 33: 3 34: 2 35: 4
    36: 4 37: 4 38: 3 39: 4 40: 5 41: 4
    42: 4 43: 3 44: 4 45: 4 46: 4 47: 3
    48: 4 49: 4 50: 3 51: 3 52: 3 53: 4
    54: 3 55: 3 56: 3 57: 4 58: 4 59: 5

    Last 24 hours:
    0: 3 1: 3 2: 1 3: 0 4: 0 5: 0
    6: 1 7: 0 8: 0 9: 0 10: 0 11: 0
    12: 0 13: 0 14: 0 15: 0 16: 2 17: 2
    18: 3 19: 3 20: 3 21: 3 22: 3 23: 3

    Average Thread(2) Utilization: 68%
    Last 60 seconds:
    0: 68 1: 71 2: 75 3: 61 4: 76 5: 69
    6: 84 7: 87 8: 79 9: 78 10: 76 11: 68
    12: 62 13: 57 14: 58 15: 66 16: 67 17: 68
    18: 66 19: 69 20: 74 21: 66 22: 53 23: 52
    24: 66 25: 69 26: 62 27: 66 28: 64 29: 66
    30: 65 31: 62 32: 57 33: 47 34: 78 35: 71
    36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    48: 68 49: 67 50: 76 51: 73 52: 66 53: 70
    54: 71 55: 63 56: 72 57: 69 58: 68 59: 59

    Last 60 minutes:
    0: 66 1: 67 2: 69 3: 69 4: 84 5: 72
    6: 76 7: 90 8: 78 9: 78 10: 77 11: 71
    12: 60 13: 58 14: 59 15: 60 16: 75 17: 56
    18: 62 19: 60 20: 59 21: 59 22: 56 23: 55
    24: 61 25: 59 26: 58 27: 56 28: 56 29: 58
    30: 59 31: 56 32: 60 33: 57 34: 53 35: 63
    36: 65 37: 63 38: 60 39: 61 40: 71 41: 68
    42: 68 43: 65 44: 64 45: 67 46: 63 47: 64
    48: 68 49: 67 50: 64 51: 63 52: 63 53: 69
    54: 64 55: 60 56: 63 57: 67 58: 72 59: 73

    Last 24 hours:
    0: 63 1: 59 2: 45 3: 31 4: 25 5: 30
    6: 36 7: 29 8: 27 9: 24 10: 25 11: 24
    12: 26 13: 24 14: 26 15: 32 16: 45 17: 54
    18: 58 19: 67 20: 60 21: 59 22: 64 23: 63

    Average Thread(3) Utilization: 68%
    Last 60 seconds:
    0: 69 1: 72 2: 75 3: 61 4: 76 5: 69
    6: 85 7: 87 8: 79 9: 78 10: 76 11: 68
    12: 62 13: 57 14: 58 15: 67 16: 67 17: 68
    18: 66 19: 69 20: 74 21: 65 22: 53 23: 53
    24: 66 25: 70 26: 62 27: 67 28: 63 29: 66
    30: 65 31: 63 32: 57 33: 47 34: 78 35: 71
    36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    48: 68 49: 67 50: 76 51: 73 52: 66 53: 69
    54: 71 55: 63 56: 72 57: 69 58: 68 59: 59

    Last 60 minutes:
    0: 66 1: 67 2: 68 3: 70 4: 84 5: 72
    6: 76 7: 90 8: 78 9: 78 10: 77 11: 71
    12: 60 13: 58 14: 58 15: 60 16: 75 17: 56
    18: 62 19: 60 20: 59 21: 59 22: 56 23: 55
    24: 61 25: 59 26: 58 27: 56 28: 56 29: 58
    30: 60 31: 56 32: 60 33: 57 34: 53 35: 64
    36: 65 37: 63 38: 60 39: 61 40: 71 41: 68
    42: 68 43: 65 44: 64 45: 67 46: 63 47: 64
    48: 68 49: 67 50: 64 51: 63 52: 63 53: 69
    54: 64 55: 60 56: 63 57: 67 58: 72 59: 73

    Last 24 hours:
    0: 63 1: 59 2: 45 3: 31 4: 25 5: 30
    6: 36 7: 29 8: 27 9: 24 10: 25 11: 24
    12: 26 13: 24 14: 26 15: 32 16: 45 17: 54
    18: 58 19: 68 20: 60 21: 59 22: 64 23: 63

    Average Utilization: 68%
    Last 60 seconds:
    0: 68 1: 71 2: 75 3: 61 4: 76 5: 69
    6: 84 7: 87 8: 79 9: 78 10: 76 11: 68
    12: 62 13: 57 14: 58 15: 67 16: 67 17: 68
    18: 66 19: 69 20: 74 21: 65 22: 53 23: 53
    24: 66 25: 69 26: 62 27: 66 28: 63 29: 66
    30: 65 31: 62 32: 57 33: 47 34: 78 35: 71
    36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    48: 68 49: 67 50: 76 51: 73 52: 66 53: 69
    54: 71 55: 63 56: 72 57: 69 58: 68 59: 59

    Last 60 minutes:
    0: 66 1: 67 2: 70 3: 70 4: 81 5: 75
    6: 77 7: 90 8: 77 9: 78 10: 78 11: 69
    12: 60 13: 59 14: 58 15: 60 16: 75 17: 55
    18: 62 19: 59 20: 60 21: 58 22: 56 23: 56
    24: 61 25: 59 26: 59 27: 55 28: 56 29: 59
    30: 58 31: 57 32: 59 33: 56 34: 54 35: 63
    36: 66 37: 62 38: 59 39: 62 40: 72 41: 66
    42: 69 43: 65 44: 64 45: 68 46: 63 47: 64
    48: 68 49: 68 50: 63 51: 64 52: 64 53: 68
    54: 64 55: 61 56: 63 57: 67 58: 74 59: 72

    Last 24 hours:
    0: 63 1: 59 2: 45 3: 31 4: 25 5: 30
    6: 36 7: 28 8: 27 9: 24 10: 25 11: 24
    12: 26 13: 24 14: 26 15: 32 16: 45 17: 54
    18: 58 19: 68 20: 60 21: 59 22: 64 23: 63

    ------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show octeon session detail"
    ================ master ================
    SENT: Ukern command: show octeon session detail

    Average session Utilization: 44794
    Last 60 seconds:
    0: 42711 1: 43504 2: 42762 3: 43340 4: 44166 5: 43494
    6: 44200 7: 43416 8: 44117 9: 43423 10: 44201 11: 43521
    12: 44356 13: 43575 14: 44329 15: 43469 16: 44200 17: 43415
    18: 44435 19: 45207 20: 44478 21: 45117 22: 44303 23: 45058
    24: 44123 25: 44707 26: 43926 27: 44778 28: 43996 29: 44990
    30: 44236 31: 45026 32: 44289 33: 44990 34: 44538 35: 44836
    36: 45861 37: 45144 38: 46077 39: 45392 40: 46105 41: 45391
    42: 46103 43: 45303 44: 45885 45: 45093 46: 45812 47: 45056
    48: 46296 49: 45497 50: 46231 51: 45535 52: 46224 53: 46768
    54: 45957 55: 46401 56: 45402 57: 45913 58: 45137 59: 45841

    Last 60 minutes:
    0: 45269 1: 42308 2: 40867 3: 41160 4: 41368 5: 42430
    6: 40647 7: 40208 8: 41343 9: 42520 10: 43734 11: 44906
    12: 46931 13: 42855 14: 41512 15: 40549 16: 40872 17: 40649
    18: 41408 19: 41054 20: 39974 21: 41412 22: 41099 23: 40991
    24: 40381 25: 39808 26: 40069 27: 40790 28: 40514 29: 39676
    30: 40445 31: 42210 32: 40937 33: 40594 34: 40007 35: 40201
    36: 39686 37: 39711 38: 38881 39: 39464 40: 41435 41: 41946
    42: 41600 43: 40636 44: 41792 45: 40671 46: 41216 47: 41232
    48: 40794 49: 42033 50: 45228 51: 45975 52: 44010 53: 44572
    54: 40350 55: 40103 56: 40724 57: 42055 58: 40999 59: 40786

    Last 24 hours:
    0: 41361 1: 40098 2: 32246 3: 24208 4: 20914 5: 20143
    6: 19845 7: 19981 8: 19971 9: 20225 10: 20641 11: 20427
    12: 21191 13: 21580 14: 22183 15: 24692 16: 31761 17: 38090
    18: 38925 19: 40307 20: 38770 21: 39640 22: 40529 23: 41090

    --------------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show threads"
    ================ master ================
    SENT: Ukern command: show threads

    PID PR State Name Stack Use Time (Last/Max/Total) cpu
    --- -- ------- --------------------- --------- ---------------------
    1 H ready Maintenance 1312/73824 0/8/393 ms 0%
    2 L running Idle 4720/73824 0/15/1233350 ms 0%
    3 H asleep Timer Services 1240/73824 0/8/6910 ms 0%
    5 L asleep Ukern Syslog 760/73824 0/0/0 ms 0%
    6 L asleep Sheaf Background 1112/73824 0/8/744 ms 0%
    7 M asleep mac_db 760/73824 0/0/0 ms 0%
    8 M asleep Docsis 1056/73824 0/8/2410 ms 0%
    9 M asleep ATMX 1840/73824 0/8/7650 ms 0%
    10 M asleep XDSL 4728/73824 0/8/444736 ms 0%
    11 M asleep DSX50ms 1752/73824 0/8/50120 ms 0%
    12 M asleep DSXonesec 1328/73824 0/8/2435 ms 0%
    13 M asleep SFP 1232/73824 0/8/3525 ms 0%
    14 M asleep Ethernet 4648/73824 0/15/1093209 ms 0%
    15 M asleep RSMON syslog thread 2448/73824 0/8/15 ms 0%
    16 L asleep Syslog 4640/73824 0/8/1554 ms 0%
    17 M ready Fwdd Notif Recv 5144/73824 0/16/4521382 ms 3%
    18 M asleep Forwarding Thread 4792/73824 0/8/725924 ms 0%
    19 M asleep Periodic 12992/73824 0/23/239628 ms 0%
    20 M asleep bcmCNTR.0 4912/16384 8/22/4808211 ms 3%
    21 M asleep bcmTX 912/16384 0/0/0 ms 0%
    22 M asleep bcmXGS3AsyncTX 944/16384 0/0/0 ms 0%
    23 M asleep bcmLINK.0 4352/16384 0/68/279867 ms 0%
    24 L asleep BCM SDK periodic 1784/73824 0/8/667 ms 0%
    25 M asleep FPC_IPC-Thread 2328/73824 0/0/0 ms 0%
    26 H asleep TTP Receive 4704/73824 0/30/840265 ms 0%
    27 H asleep TTP Transmit 4704/73824 0/38/742839 ms 0%
    28 M asleep UDP Input 856/73824 0/0/0 ms 0%
    29 H asleep TCP Timers 4640/73824 0/8/6450 ms 0%
    30 H asleep TCP Receive 760/73824 0/0/0 ms 0%
    33 M asleep Service Set Cleanup 880/73824 0/0/0 ms 0%
    35 L asleep Monitor sort 2480/73824 0/8/2287 ms 0%
    36 H asleep Monitor Exporter 1800/73824 0/8/4403 ms 0%
    37 M asleep Monitor Expire 1080/73824 0/8/464 ms 0%
    38 M asleep Monitor Expire TCP 2368/73824 0/8/2333 ms 0%
    39 L asleep Monitor Aged 1056/73824 0/8/38 ms 0%
    40 H asleep Monitor sampledc 864/73824 0/0/0 ms 0%
    41 M asleep Monitor v8 1072/73824 0/8/22 ms 0%
    42 M asleep Monitor v5 to v8 1784/73824 0/8/265 ms 0%
    47 L asleep Console 3504/73824 0/0/0 ms 0%
    48 M asleep L2ALM Manager 3088/73824 0/8/54081 ms 0%
    49 L asleep DFW Alert 13264/73824 0/8/4052 ms 0%
    50 L asleep CLNS Err Input 848/73824 0/0/0 ms 0%
    51 L asleep CLNS Option Input 848/73824 0/0/0 ms 0%
    52 M asleep PIC Periodic 1376/73824 0/8/3627 ms 0%
    53 M asleep PIC 760/73824 0/0/0 ms 0%
    54 M asleep FWDD CM 4632/73824 0/1542/50306 ms 0%
    55 L asleep ICMP6 Input 1016/73824 0/0/0 ms 0%
    56 L asleep IP6 Option Input 2936/73824 0/0/0 ms 0%
    57 L asleep ICMP Input 2944/73824 0/8/220358 ms 0%
    58 L asleep IP Option Input 2936/73824 0/0/0 ms 0%
    59 M asleep IGMP Input 2936/73824 0/0/0 ms 0%
    60 M asleep VRRP Manager 792/73824 0/0/0 ms 0%
    61 M asleep PPM Manager 3032/73824 0/0/0 ms 0%
    62 M asleep Wheel timer 2920/73824 0/8/2926 ms 0%
    63 H asleep JFlow v9 export 1408/73824 0/8/2293 ms 0%
    65 H asleep Jflow v9 PFE Poll 960/73824 0/0/0 ms 0%
    66 M asleep PPM Data thread 4640/73824 0/8/12032 ms 0%
    67 L asleep NH Probe Service 776/73824 0/0/0 ms 0%
    69 L asleep Virtual Console 2168/73824 0/0/0 ms 0%
    71 M asleep Msg relay 2272/73824 0/0/0 ms 0%
    72 M asleep IPSec 4504/73824 0/8/32917 ms 0%
    76 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    80 M asleep DA-PFE 2416/73824 0/8/2868 ms 0%
    81 M asleep PFE Manager 15448/73824 0/3211/480653 ms 0%
    82 M asleep L2HA TOGGLE 1800/73824 0/8/5745 ms 0%
    83 M asleep USP Trace 2384/73824 0/8/11576 ms 0%
    84 M asleep PFE Statistics 6072/73824 8/15/307012 ms 0%
    85 M asleep Userfw Msgrelay 2352/73824 0/0/0 ms 0%
    86 H asleep Services UI 2632/73824 0/0/0 ms 0%
    87 H asleep Monitor UI 2712/73824 0/0/0 ms 0%
    88 H asleep Jflow v9 UI Daemon 3256/73824 0/0/0 ms 0%
    89 H asleep USP IPC Server 3104/73824 7/7/7 ms 0%
    90 M asleep NSD-PFE 8360/73824 0/31/6094 ms 0%
    91 M asleep IDP PFEMAN Thread 13424/73824 0/8/798 ms 0%
    92 M asleep utm-pfe 4640/73824 0/8/3162 ms 0%
    93 H asleep USP IPC Server 3104/73824 0/0/0 ms 0%
    94 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    95 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    96 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    99 L asleep BCM AGING periodic 2632/73824 0/211/87310 ms 0%
    1555 L running Cattle-Prod Daemon 4240/73824 0/8/15 ms 0%
    1587 H asleep USP IPC Server 2264/73824 8/8/8 ms 0%
    1588 L asleep Cattle-Prod Daemon 3304/73824 0/0/0 ms 0%

    ----------------------------------------------------------------------------------------------------------------------

    request pfe execute target fwdd command "show services mum"
    ================ master ================
    SENT: Ukern command: show services mum

    Memory usage manager: gsm
    Total free space to start with: 177182704
    Active customers: 1
    Max customers: 12
    Yellow zone limit: 44295676
    Orange zone limit: 31892886
    Red zone limit: 15946443
    Operational zone: Green

    cust_id in use limit
    -------------------------------
    0 914167 177182704
    1 632 177182704
    -------------------------------

    actual free space = 135948952
    est. free space = 135948952

    Proactive reclaim : ENABLED
    Seconds between reclaims : 5
    Number of proactive reclaims : 0

    -------------------------------------------------------------------------------------------------------

    show system processes extensive | except 0.0
    last pid: 6421; load averages: 0.92, 1.05, 0.94 up 1+15:18:51 12:18:01
    153 processes: 19 running, 122 sleeping, 1 zombie, 11 waiting

    Mem: 227M Active, 122M Inact, 1170M Wired, 221M Cache, 112M Buf, 232M Free
    Swap:


    PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    1656 root 7 76 0 1124M 88140K select 0 136.7H 325.49% flowd_octeon_hm
    22 root 1 171 52 0K 16K RUN 0 843:02 33.79% idle: cpu0

    -------------------------------------------------------------------------------------------------------

    show chassis routing-engine
    Routing Engine status:
    Temperature 49 degrees C / 120 degrees F
    CPU temperature 47 degrees C / 116 degrees F
    Total memory 2048 MB Max 1065 MB used ( 52 percent)
    Control plane memory 976 MB Max 517 MB used ( 53 percent)
    Data plane memory 1072 MB Max 557 MB used ( 52 percent)
    CPU utilization:
    User 21 percent
    Background 0 percent
    Kernel 43 percent
    Interrupt 0 percent
    Idle 36 percent
    Model RE-SRX240H2
    Serial ID ACAL8054
    Start time 2019-02-05 20:59:39 MSK
    Uptime 1 day, 15 hours, 18 minutes, 39 seconds
    Last reboot reason Router rebooted after a normal shutdown.
    Load averages: 1 minute 5 minute 15 minute
    0.80 1.01 0.93

    --------------------------------------------------------------------------------------------------------------------

    root@> show configuration | match idp

    root@> show configuration | match utm

    root@> show configuration | match session-init

    root@> show configuration | match session-close

    root@> show configuration | match traceoptions

     

     



  • 8.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-07-2019 08:22

    Hi,

     

    The average of traffic ingresing the SRX is around 260Mbps so I dont think we are pushing too much traffic. However I am suspecting about bursts of traffic because the CPU goes high just for 1 or 2 seconds:

     

     

    	Average Thread(2) Utilization: 68%
    	Last 60 seconds:
    	0: 68 1: 71 2: 75 3: 61 4: 76 5: 69
    	6: 84 7: 87 8: 79 9: 78 10: 76 11: 68
    	12: 62 13: 57 14: 58 15: 66 16: 67 17: 68
    	18: 66 19: 69 20: 74 21: 66 22: 53 23: 52
    	24: 66 25: 69 26: 62 27: 66 28: 64 29: 66
    	30: 65 31: 62 32: 57 33: 47 34: 78 35: 71
    	36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    	42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    	48: 68 49: 67 50: 76 51: 73 52: 66 53: 70
    	54: 71 55: 63 56: 72 57: 69 58: 68 59: 59
    	
    	Average Thread(3) Utilization: 68%
    	Last 60 seconds:
    	0: 69 1: 72 2: 75 3: 61 4: 76 5: 69
    	6: 85 7: 87 8: 79 9: 78 10: 76 11: 68
    	12: 62 13: 57 14: 58 15: 67 16: 67 17: 68
    	18: 66 19: 69 20: 74 21: 65 22: 53 23: 53
    	24: 66 25: 70 26: 62 27: 67 28: 63 29: 66
    	30: 65 31: 63 32: 57 33: 47 34: 78 35: 71
    	36: 76 37: 75 38: 75 39: 71 40: 74 41: 69
    	42: 64 43: 68 44: 73 45: 66 46: 73 47: 64
    	48: 68 49: 67 50: 76 51: 73 52: 66 53: 69
    	54: 71 55: 63 56: 72 57: 69 58: 68 59: 59

     

     

    I dont see any memory problems and no processes are showing high, however I can see that User and Kernel are high:

     

     

    	show chassis routing-engine 
    	Routing Engine status:
    	Temperature 49 degrees C / 120 degrees F
    	CPU temperature 47 degrees C / 116 degrees F
    	Total memory 2048 MB Max 1065 MB used ( 52 percent)
    	Control plane memory 976 MB Max 517 MB used ( 53 percent)
    	Data plane memory 1072 MB Max 557 MB used ( 52 percent)
    	CPU utilization:
    	User 21 percent
    	Background 0 percent
    	Kernel 43 percent
    	Interrupt 0 percent
    	Idle 36 percent
    	Model RE-SRX240H2
    	Serial ID ACAL8054
    	Start time 2019-02-05 20:59:39 MSK
    	Uptime 1 day, 15 hours, 18 minutes, 39 seconds
    	Last reboot reason Router rebooted after a normal shutdown.
    	Load averages: 1 minute 5 minute 15 minute
    	0.80 1.01 0.93

     

     

    PLease gather the following commands from shell gather:

     

          > start shell  (entering shell mode)
          % ps
          % top -H

     

    Also please share the following commands:

     

    > request pfe execute command "show fwdd statistics notification" target fwdd
    > request pfe execute command "show threads" target fwdd
    > request pfe execute command "show pfe statistics notification" target fwdd
    > request pfe execute command "show fwdd statistics result" target fwdd

     

    Can you confirm if the high CPU is seeing during non-production hours? This could help us to understand if the issue can be related to specific traffic:

     

           show log messages | match RTPERF_CPU_THRESHOLD_EXCEEDED

    Was the SRX recently rebooted for testing purposes?

     

    Determining whats causing the high CPU will require a deeper inspection on the SRX. Can you share a RSI and the var/log/ directory of the SRX?

     

    • In order to get the RSI (request support information) run the following command to save the resulting output in to a text file that you could later download from the SRX:

     

                                    request support information | save /var/tmp/[Document_Name].txt

     

           Another option for getting the request support information is to set the terminal application, like putty/secure CRT, to log  the session to a file and run:

     

                                     request support information | no-more

     

    • Archive the /var/log/ contents:

     

                                    file archive compress source /var/log/* destination /var/tmp/[File_Name]

     

             To ensure the files directory was properly archived, check the file size using the command:  

     

                                    file list /var/tmp/[File_Name] detail

     

     



  • 9.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-07-2019 21:36

    Hello all! Thanks for helping.

     

    Spoiler
    ps
    PID TT STAT TIME COMMAND
    1 u0- SLs 0:01.23 /junos/sbin/init -D/junos --
    1248 u0- S 0:00.68 /usr/sbin/usbd -N
    1274 u0- S 0:57.12 /usr/sbin/eventd -N -r -s -A
    1689 u0 Ss+ 0:00.05 /usr/libexec/getty std.9600 ttyu0
    6294 p3 Ss 0:00.15 -csh (csh)
    8371 p3 R+ 0:00.03 ps
    -----------------------------------------------------------------------

    top -H
    last pid: 8375; load averages: 0.52, 0.57, 0.62 up 2+12:06:11 09:05:21
    79 processes: 5 running, 73 sleeping, 1 zombie
    CPU states: 76.2% user, 0.0% nice, 17.6% system, 0.6% interrupt, 5.5% idle
    Mem: 188M Active, 122M Inact, 1214M Wired, 180M Cache, 112M Buf, 268M Free
    Swap:

    PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    1656 root 139 0 1124M 88140K CPU1 1 209.0H 92.48% flowd_octeon_hm
    1656 root 139 0 1124M 88140K CPU2 2 209.0H 90.19% flowd_octeon_hm
    1656 root 139 0 1124M 88140K CPU3 3 209.0H 90.19% flowd_octeon_hm
    1656 root 113 0 1124M 88140K RUN 0 209.0H 47.36% flowd_octeon_hm
    1656 root 76 0 1124M 88140K ucondt 0 209.0H 0.00% flowd_octeon_hm
    1656 root 76 0 1124M 88140K select 0 209.0H 0.00% flowd_octeon_hm
    1656 root 76 0 1124M 88140K select 0 209.0H 0.00% flowd_octeon_hm
    1652 root 76 0 18500K 8956K select 0 90:44 0.00% ppmd
    1644 root 4 0 59000K 29008K kqread 0 33:00 0.00% rpd
    1643 root 76 0 33428K 12044K select 0 29:30 0.00% mib2d
    1642 root 76 0 23888K 14556K select 0 15:31 0.00% snmpd
    1686 root 76 0 14464K 6072K select 0 14:02 0.00% license-check
    1645 root 76 0 22644K 9488K select 0 6:40 0.00% l2ald
    1637 root 76 0 128M 19320K select 0 5:42 0.00% chassisd
    1680 root 76 0 19684K 8144K select 0 4:38 0.00% utmd
    1669 root 76 0 17792K 3964K select 0 4:34 0.00% shm-rtsdbd
    1649 root 76 0 15888K 7292K select 0 4:24 0.00% rmopd
    1648 root 76 0 30112K 10224K select 0 4:13 0.00% pfed
    1688 root 4 0 25920K 13552K kqread 0 3:35 0.00% eswd
    1685 root 4 0 11756K 5460K kqread 0 3:26 0.00% mcsnoopd
    1651 root 76 0 32812K 15476K select 0 2:56 0.00% kmd
    1638 root 76 0 14400K 5408K select 0 2:11 0.00% alarmd
    1634 root 76 0 3348K 1376K select 0 2:07 0.00% bslockd
    1683 root 8 0 29848K 2404K nanslp 0 2:05 0.00% wmic
    1677 root 76 0 78828K 4748K select 0 1:53 0.00% ipfd
    1677 root 8 0 78828K 4748K nanslp 0 1:53 0.00% ipfd
    1677 root 8 0 78828K 4748K nanslp 0 1:53 0.00% ipfd
    1677 root 8 0 78828K 4748K nanslp 0 1:53 0.00% ipfd
    1677 root 8 0 78828K 4748K nanslp 0 1:53 0.00% ipfd
    1679 root 76 0 15524K 6508K select 0 1:42 0.00% rtlogd
    1667 root 76 0 27064K 9912K select 0 1:12 0.00% smid
    1274 root 76 0 15084K 4836K select 0 0:57 0.00% eventd
    1654 root 76 0 9196K 3640K select 0 0:54 0.00% irsd
    1698 nobody 4 0 10612K 1488K kqread 0 0:51 0.00% webapid
    3660 root 76 0 16816K 5576K select 0 0:48 0.00% wland
    3660 root 76 0 16816K 5576K select 0 0:48 0.00% wland
    3660 root 76 0 16816K 5576K select 0 0:48 0.00% wland
    1636 root 76 0 39768K 8892K select 0 0:45 0.00% dcd
    1672 root 76 0 16928K 6300K select 0 0:38 0.00% pkid
    1671 root 76 0 29112K 13248K select 0 0:27 0.00% nsd
    -----------------------------------------------------------------------------------------------------

    request pfe execute command "show fwdd statistics notification" target fwdd
    ================ master ================
    SENT: Ukern command: show fwdd statistics notification


    Notification statistics:

    Real-time -> microkernel:
    Queue Name Sent Dropped
    ----- ------------ ---------- -----------
    0 High 7616044 0
    1 Medium 7041214 0
    2 Low 36655 0
    3 Error/ICMP 466845609 953241
    4 Discard 9945 0
    --------------------------------------------
    0 Invalid
    0 Bad stream ID
    0 Bad notif type

    Microkernel -> Realtime:
    0 No packet
    0 No IFD/IFL
    0 Dropped
    0 Dropped Keepalives
    0 L2 rewrite error
    0 No packet in notif
    0 Droppped out ifl index
    0 Droppped wrong tunnel

    L2 Microkernel -> Realtime:
    5588599 Sent
    0 No Mbuf

    L3 Microkernel -> Realtime:
    7640802 Sent
    0 No Mbuf
    0 Send error

    Notification Commands:
    0 Commands
    0 Invalid Command
    0 Unknown Comman
    -------------------------------------------------------------------------------------------------------------------------

    request pfe execute command "show threads" target fwdd
    ================ master ================
    SENT: Ukern command: show threads

    PID PR State Name Stack Use Time (Last/Max/Total) cpu
    --- -- ------- --------------------- --------- ---------------------
    1 H asleep Maintenance 1312/73824 0/8/401 ms 0%
    2 L running Idle 4720/73824 0/15/1875837 ms 0%
    3 H asleep Timer Services 1240/73824 0/8/10367 ms 0%
    5 L asleep Ukern Syslog 760/73824 0/0/0 ms 0%
    6 L asleep Sheaf Background 1112/73824 0/8/1024 ms 0%
    7 M asleep mac_db 760/73824 0/0/0 ms 0%
    8 M asleep Docsis 1776/73824 0/8/3596 ms 0%
    9 M asleep ATMX 1840/73824 0/8/11679 ms 0%
    10 M asleep XDSL 4728/73824 0/8/677261 ms 0%
    11 M asleep DSX50ms 1752/73824 0/8/76330 ms 0%
    12 M asleep DSXonesec 1328/73824 0/8/4002 ms 0%
    13 M asleep SFP 1920/73824 0/8/5249 ms 0%
    14 M asleep Ethernet 4648/73824 0/15/1700776 ms 0%
    15 M asleep RSMON syslog thread 2448/73824 0/8/37 ms 0%
    16 L asleep Syslog 4640/73824 0/8/2335 ms 0%
    17 M asleep Fwdd Notif Recv 5144/73824 0/16/6850822 ms 3%
    18 M asleep Forwarding Thread 4792/73824 0/8/1109733 ms 0%
    19 M asleep Periodic 12992/73824 0/23/365388 ms 0%
    20 M asleep bcmCNTR.0 4912/16384 15/22/7357204 ms 3%
    21 M asleep bcmTX 912/16384 0/0/0 ms 0%
    22 M asleep bcmXGS3AsyncTX 944/16384 0/0/0 ms 0%
    23 M asleep bcmLINK.0 4352/16384 8/68/434378 ms 0%
    24 L asleep BCM SDK periodic 1784/73824 0/8/911 ms 0%
    25 M asleep FPC_IPC-Thread 2328/73824 0/0/0 ms 0%
    26 H asleep TTP Receive 4704/73824 0/30/1284942 ms 0%
    27 H asleep TTP Transmit 4704/73824 0/38/1137929 ms 0%
    28 M asleep UDP Input 856/73824 0/0/0 ms 0%
    29 H asleep TCP Timers 4640/73824 0/8/9725 ms 0%
    30 H asleep TCP Receive 760/73824 0/0/0 ms 0%
    33 M asleep Service Set Cleanup 880/73824 0/0/0 ms 0%
    35 L asleep Monitor sort 2480/73824 0/8/3651 ms 0%
    36 H asleep Monitor Exporter 1800/73824 0/8/6379 ms 0%
    37 M asleep Monitor Expire 1080/73824 0/8/678 ms 0%
    38 M asleep Monitor Expire TCP 2368/73824 0/8/3591 ms 0%
    39 L asleep Monitor Aged 1056/73824 0/8/66 ms 0%
    40 H asleep Monitor sampledc 864/73824 0/0/0 ms 0%
    41 M asleep Monitor v8 1072/73824 0/8/22 ms 0%
    42 M asleep Monitor v5 to v8 1784/73824 0/8/372 ms 0%
    47 L asleep Console 3504/73824 0/0/0 ms 0%
    48 M asleep L2ALM Manager 3088/73824 0/8/82838 ms 0%
    49 L asleep DFW Alert 13264/73824 0/8/6124 ms 0%
    50 L asleep CLNS Err Input 848/73824 0/0/0 ms 0%
    51 L asleep CLNS Option Input 848/73824 0/0/0 ms 0%
    52 M asleep PIC Periodic 1800/73824 0/8/5462 ms 0%
    53 M asleep PIC 760/73824 0/0/0 ms 0%
    54 M asleep FWDD CM 4632/73824 0/1542/76704 ms 0%
    55 L asleep ICMP6 Input 1016/73824 0/0/0 ms 0%
    56 L asleep IP6 Option Input 2936/73824 0/0/0 ms 0%
    57 L asleep ICMP Input 2944/73824 0/15/335547 ms 0%
    58 L asleep IP Option Input 2936/73824 0/0/0 ms 0%
    59 M asleep IGMP Input 2936/73824 0/0/0 ms 0%
    60 M asleep VRRP Manager 792/73824 0/0/0 ms 0%
    61 M asleep PPM Manager 3032/73824 0/0/0 ms 0%
    62 M asleep Wheel timer 2920/73824 0/8/4668 ms 0%
    63 H asleep JFlow v9 export 1408/73824 0/8/3861 ms 0%
    65 H asleep Jflow v9 PFE Poll 960/73824 0/0/0 ms 0%
    66 M asleep PPM Data thread 4640/73824 0/8/18130 ms 0%
    67 L asleep NH Probe Service 776/73824 0/0/0 ms 0%
    69 L asleep Virtual Console 2168/73824 0/0/0 ms 0%
    71 M asleep Msg relay 2272/73824 0/0/0 ms 0%
    72 M asleep IPSec 4504/73824 0/8/50511 ms 0%
    76 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    80 M asleep DA-PFE 2416/73824 0/8/4454 ms 0%
    81 M asleep PFE Manager 15448/73824 0/3211/723063 ms 0%
    82 M asleep L2HA TOGGLE 1800/73824 0/8/8570 ms 0%
    83 M asleep USP Trace 2384/73824 0/8/17515 ms 0%
    84 M asleep PFE Statistics 6072/73824 0/15/475875 ms 0%
    85 M asleep Userfw Msgrelay 2352/73824 0/0/0 ms 0%
    86 H asleep Services UI 2632/73824 0/0/0 ms 0%
    87 H asleep Monitor UI 2712/73824 0/0/0 ms 0%
    88 H asleep Jflow v9 UI Daemon 3256/73824 0/0/0 ms 0%
    89 H asleep USP IPC Server 3104/73824 7/7/7 ms 0%
    90 M asleep NSD-PFE 8360/73824 0/31/9293 ms 0%
    91 M asleep IDP PFEMAN Thread 13424/73824 0/8/1270 ms 0%
    92 M asleep utm-pfe 4640/73824 0/8/4678 ms 0%
    93 H asleep USP IPC Server 3104/73824 0/0/0 ms 0%
    94 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    95 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    96 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    99 L asleep BCM AGING periodic 2632/73824 0/211/133403 ms 0%
    1615 H asleep USP IPC Server 2264/73824 0/0/0 ms 0%
    1618 L running Cattle-Prod Daemon 4240/73824 0/7/14 ms 0%
    1619 L asleep Cattle-Prod Daemon 3304/73824 0/0/0 ms 0%
    ________________________________________________________________________________________________

    request pfe execute command "show pfe statistics notification" target fwdd
    ================ master ================
    SENT: Ukern command: show pfe statistics notification

    PFE Notification statistics:
    963546010 parsed
    0 aged
    0 corrupt
    0 illegal
    459684859 sample
    0 giants
    0 transit options/ttl-exceeded (re-injected)
    0 transit options/ttl-exceeded errors
    0 svc options sent to ASP
    0 svc options sent to RE
    0 post svc options sent out
    7384704 options or ttl expired (not RE-destined)
    0 discard sample
    459423424 rate limited
    0 packet get failure
    0 DMA failure
    0 Total DMa'd packets
    0 Unknown/unclassified packets

    PFE Notification Type statistics:
    Parsed Input Failed Ignored
    Illegal 0 0 0 0
    Unclass 7453396 7453396 0 0
    Option 0 0 0 0
    Next-Hop 14630975 14601401 3521 0
    Discard 9945 9945 0 0
    Sample 0 0 0 0
    Redirect 459674914 284585 0 0
    DontFrag 254 254 0 0
    CfDF 0 0 0 0
    Poison 0 0 0 0
    MemPkt 0 0 0 0
    Autoconf 0 0 0 0
    Reject 0 0 0 0
    __________________________________________________________________________________________

    In non-proction hours no messages from perf_mon, but on NMS graphic of CPU utilization i didnt see any chages.
    show log messages | match RTPERF_CPU_THRESHOLD_EXCEEDED
    Feb 6 15:45:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 15:45:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 15:45:51 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:46:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 15:46:49 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 15:49:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 15:50:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 15:50:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 15:50:35 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 6 15:51:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:52:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:52:55 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 15:53:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 15:54:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 15:54:19 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 15:54:37 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:54:49 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:54:55 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:55:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 15:55:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 6 15:55:19 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 15:55:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 15:55:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 15:55:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 15:55:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 15:55:51 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 15:56:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 15:56:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 15:56:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:56:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 6 15:57:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 6 15:57:32 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 15:57:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 6 15:57:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 15:58:48 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 6 15:59:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 16:00:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 16:01:23 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 16:01:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 16:01:32 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 16:01:51 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 16:10:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 16:14:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 16:14:54 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 16:20:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 16:20:31 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 16:25:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 16:34:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 16:35:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 6 16:35:23 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 16:38:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 16:50:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 16:55:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 16:59:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 6 17:04:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 17:04:44 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 6 17:05:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 17:05:23 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 17:08:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 6 17:08:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 6 17:08:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 17:08:39 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 17:11:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 17:15:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 6 17:19:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 6 17:19:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 17:19:20 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 6 17:19:26 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 6 17:19:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 17:19:40 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 6 17:20:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 6 17:20:20 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 6 17:20:29 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 17:21:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 18:00:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 6 18:00:19 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 18:00:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 18:00:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 18:08:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=97
    Feb 6 19:00:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 19:00:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 19:00:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 6 19:00:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 6 19:00:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 6 19:00:40 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 19:00:49 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 19:01:00 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 19:01:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 19:01:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 19:01:37 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 19:01:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 19:02:41 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 19:03:21 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 19:03:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 19:04:13 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 6 19:04:20 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 19:04:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 6 19:04:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 6 19:04:40 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 19:04:55 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 6 19:05:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 6 19:05:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 6 19:05:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 6 19:05:20 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 6 19:05:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 6 19:08:00 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 6 19:08:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 19:08:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 6 19:08:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=97
    Feb 6 19:08:31 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 6 19:08:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 6 19:09:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 19:09:41 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 6 19:09:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 6 19:13:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 6 19:22:39 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 7 09:00:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 09:25:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 09:32:35 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 09:32:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=97
    Feb 7 09:32:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 09:34:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:05:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:08:13 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 7 10:08:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:10:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:14:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 10:15:00 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:15:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 10:15:21 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:16:19 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:16:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 10:16:41 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 7 10:17:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 10:18:01 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:18:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 10:19:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 10:21:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 10:21:21 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 10:21:28 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=98
    Feb 7 10:22:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:22:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:22:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:23:01 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 10:25:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 10:30:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:30:37 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:37:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:37:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 10:37:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:40:40 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 10:43:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:45:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:45:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:45:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:50:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 10:52:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 10:52:21 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:53:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:53:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:55:39 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:55:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 10:56:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 10:56:19 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 10:56:21 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:56:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 10:56:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:56:59 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 10:58:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:02:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 11:04:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 11:04:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:04:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 11:08:05 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 7 11:08:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 11:08:31 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 11:08:55 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:09:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 11:09:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 11:09:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 11:09:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:09:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:10:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=96
    Feb 7 11:10:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:10:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 11:10:35 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 11:11:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:11:34 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 11:15:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 11:16:23 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:16:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:17:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:20:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 11:21:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:24:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:24:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:25:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 11:25:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:27:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:27:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 11:27:55 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 11:28:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:29:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:29:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:29:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:30:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 11:30:13 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 11:35:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 11:35:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:51:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 11:54:59 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 11:55:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 7 11:55:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 11:55:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 11:55:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:01:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:01:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:01:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:01:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:01:28 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:01:35 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 12:01:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:01:53 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:02:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:02:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:02:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:02:34 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:02:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:03:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:03:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:03:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:03:26 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:03:28 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:04:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:04:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 12:04:26 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:04:34 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 12:04:44 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:04:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:05:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:05:34 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 12:05:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:05:48 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:06:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:06:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:06:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:06:44 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:06:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:06:57 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 12:07:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:07:23 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:07:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:08:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 7 12:08:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:08:57 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:09:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:10:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 12:11:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:14:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:15:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:15:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:15:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:15:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 12:15:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:17:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:17:51 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:20:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:20:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:20:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 12:20:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 12:20:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:21:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 12:21:39 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=95
    Feb 7 12:21:51 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:21:57 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:22:01 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 12:22:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:22:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:25:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:26:01 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 12:26:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 12:34:57 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:35:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:38:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:39:40 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:40:41 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:41:40 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 12:42:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 12:55:09 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 12:55:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 13:00:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 13:00:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:03:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 13:08:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 13:08:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 13:08:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 13:08:32 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 13:08:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 13:10:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 13:10:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:10:57 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 13:12:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:12:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:13:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:15:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 13:20:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 13:20:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:20:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 13:20:51 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:23:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:23:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 13:25:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 13:25:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 13:25:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 13:58:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:08:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:08:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 14:08:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 14:08:32 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 7 14:08:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 14:08:53 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 14:09:13 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:09:43 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:09:58 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:10:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 14:10:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 14:10:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:10:37 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 14:10:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:10:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 14:10:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 14:11:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 14:11:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:11:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 14:11:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 14:12:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 14:12:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 14:12:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:12:23 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:12:31 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 14:12:58 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:13:13 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 14:13:35 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:13:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 14:13:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:13:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 14:14:53 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=97
    Feb 7 14:15:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 14:15:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 14:16:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 14:16:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:16:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 14:17:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 14:18:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:19:24 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:20:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 14:21:13 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 14:21:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:21:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 14:22:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 14:22:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 14:59:20 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 15:08:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=99
    Feb 7 15:08:22 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 15:30:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 15:47:55 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 16:01:43 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 16:01:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=97
    Feb 7 16:01:53 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 16:02:01 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 16:03:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 16:07:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 16:07:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 16:08:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=97
    Feb 7 16:08:32 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 16:08:34 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 16:08:41 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 16:17:44 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 16:27:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 16:30:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 16:35:23 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 16:45:03 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 16:45:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 16:46:44 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 16:47:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 16:48:35 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 16:56:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 16:57:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 16:58:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 16:58:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 17:00:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 17:00:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 17:03:42 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 17:04:43 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 17:05:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 17:08:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 17:08:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 17:08:33 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 17:19:05 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 17:19:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 17:50:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 18:00:12 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 18:00:15 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 19:00:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 19:00:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=98
    Feb 7 19:00:43 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:00:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 19:01:06 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:01:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 7 19:01:52 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 19:02:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 19:02:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:03:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 19:03:26 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:03:28 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 19:03:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=94
    Feb 7 19:04:11 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 19:04:17 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:04:19 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=97
    Feb 7 19:04:21 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:04:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 19:04:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:04:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:05:16 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:05:21 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=96
    Feb 7 19:05:31 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 19:05:57 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:06:20 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 19:06:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=92
    Feb 7 19:06:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=91
    Feb 7 19:06:48 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:06:56 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:07:02 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 19:08:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 19:08:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 19:09:49 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 19:19:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:19:47 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 19:19:49 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:20:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:21:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 19:21:29 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 19:21:39 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:22:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:22:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=93
    Feb 7 19:22:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:22:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:23:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 19:24:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:24:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    Feb 7 19:24:28 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:24:49 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:25:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:25:14 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:28:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:28:30 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:28:38 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 19:32:46 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:47:20 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:49:54 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:51:55 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:52:19 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:54:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:56:25 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 19:56:27 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 19:56:36 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 19:56:49 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 19:59:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 20:02:00 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 20:03:07 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 20:07:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=88
    Feb 7 20:08:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 20:10:08 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=90
    Feb 7 20:12:10 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=86
    Feb 7 20:12:37 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 20:12:45 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 20:14:18 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 20:18:50 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=85
    Feb 7 20:20:04 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 7 20:20:35 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=87
    Feb 8 09:07:58 JR2-NNDC PERF_MON: RTPERF_CPU_THRESHOLD_EXCEEDED: FPC 0 PIC 0 CPU utilization exceeds threshold, current value=89
    ________________________________________________________________________________________________________________________

    SRX rebooted twice.
    ________________________________________________________________________________________________________________________

    Give me your email 

     



  • 10.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED
    Best Answer

    Posted 02-07-2019 22:54

    Hi,

     

    Based on the "messages" log file, it looks like the problem happens between 9am-8pm, can you check older logs and confirm this? Are these the production hours of your network?

     

    It looks like there are notifications/packets being shared between the RE and the PFE and this will explain the User and Kernel usage that we noticed in the previous "show chassis routing-engine" command output.

     

     

    	request pfe execute command "show threads" target fwdd 
    	================ master ================
    	SENT: Ukern command: show threads
    
    	PID PR State Name Stack Use Time (Last/Max/Total) cpu
    	--- -- ------- --------------------- --------- ---------------------
    	17 M asleep Fwdd Notif Recv 5144/73824 0/16/6850822 ms 3%
    	20 M asleep bcmCNTR.0 4912/16384 15/22/7357204 ms 3%

     

    I noticed a lot of notifications related to ICMP Errors:

     

    	request pfe execute command "show fwdd statistics notification" target fwdd 
    	================ master ================
    	SENT: Ukern command: show fwdd statistics notification
    
    
    	Notification statistics:
    
    	Real-time -> microkernel:
    	Queue Name 	       Sent       Dropped
    	----- ------------ ---------- -----------
    	0 	  High         7616044    0
    	1 	Medium         7041214    0
    	2 	Low            36655      0
    	3 	Error/ICMP     466845609  953241   <<<<<<<<<<<<<<<<<<<<<<
    	4 	Discard        9945       0

     

    And I can see notifications being rate-limited:

     

    	request pfe execute command "show pfe statistics notification" target fwdd 
    	================ master ================
    	SENT: Ukern command: show pfe statistics notification
    
    	PFE Notification statistics:
    	963546010 parsed
    	0 aged
    	0 corrupt
    	0 illegal
    	459684859 sample    <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
    	0 giants
    	0 transit options/ttl-exceeded (re-injected)
    	0 transit options/ttl-exceeded errors
    	0 svc options sent to ASP
    	0 svc options sent to RE
    	0 post svc options sent out
    	7384704 options or ttl expired (not RE-destined)
    	0 discard sample
    	459423424 rate limited   <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<,
    	0 packet get failure
    	0 DMA failure
    	0 Total DMa'd packets
    	0 Unknown/unclassified packets

     

    And a lot of redirects:

     

    	PFE Notification Type statistics:
    	Parsed 		Input 		Failed 		Ignored
    	Illegal 	0 			0 			0 0
    	Unclass 	7453396 	7453396 	0 0
    	Option 		0 			0 			0 0
    	Next-Hop 	14630975 	14601401 	3521 0
    	Discard 	9945 		9945 		0 0
    	Sample  	0 			0 			0 0
    	Redirect 	459674914 	284585 		0 0    <<<<<<<<<<<<<<<<<<<<<,
    	DontFrag 	254 		254 		0 0
    	CfDF 		0 			0 			0 0
    	Poison 		0 			0 			0 0
    	MemPkt 		0 			0 			0 0
    	Autoconf 	0 			0 			0 0
    	Reject 		0 			0 			0 0

     

    Please confirm the working hours of your network because if the problem happens only during those hours we might need to take some packet captures on the interfaces looking for ICMP redirects.

     



  • 11.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-07-2019 23:24
      |   view attached

    Yes, 9am-8pm is production hours and messeges from PFE sends only at this period, but kernel and user usage constantly on this values, two week ago CPU utlization was two times less.

     

    CPU utilization:
    User 19 percent
    Background 0 percent
    Kernel 39 percent
    Interrupt 0 percent
    Idle 42 percent

     



  • 12.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

     
    Posted 02-06-2019 16:26

    Since you are not using IDP then the cause will be from one of your traffic flows being processed.  The challenge will be to identify which it is.  If you have an NMS with bandwidth graphs where you can see what changed this can help narrow it down.

     

    Remember that total throughput for the srx240 is an aggregate 1.8 g on all interfaces.

     



  • 13.  RE: SRX 240 CPU_THRESHOLD_EXCEEDED

    Posted 02-07-2019 11:13

    Hello Beh0ld,

     

    From the graph it appears that you are not only observing a data plane high CPU but your RE also shoots high.

     

    The factors for high CPUs on RE and PFE differ a bit.

     

    1. On PFE:-

     

    - There are two basic reasons for high CPU in the data plane.

    a) High rate of cycles per second (CPS) - i.e. rate of creation or deletion of session. 

    b) High rate of packets per second (PPS) - i.e. rate of packets processed per second.

     

    - Whenever this CPU exceeds 85% , you will see a message printed in the syslog.

     

    2. On RE:-

    - Basically RE CPU depends upon the time taken by different processes.

    - A high RE CPU does NOT generate any message.

     

    Since both the CPU goes up around the same time , I would think of any traffic which invokes some additional processing on the RE too.

     

    top -H shows rpd, ppmd, snmpd and mibd as the highest utilized ones.

     

    "show chassis routing-engine" shows "Kernel" using 42%". I would expect it to be less than 10%  (when NOT using any Advanced feature).

     

    I would advise you to check the following in the configuration: -

     

    1. Any traceoptions configured on the device should be removed.

    2. Check for the  SNMP traffic. Make sure any unconfigured SNMP server is NOT polling it. (Hint: You can use RE protect filter to stop CPU impacts.)

    3. Check the rate of SNMP polling.

    4. PPMD generally controls OSPF and LACP. Look for any aggressive timers.

    5. Do your security policies have log or/and count ? If yes, please ensure that you are NOT logging it locally.

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB16502

     

    I hope this information in addition to replies above will be helpful.

     

    Cheers!

     


    #HIghCPUtroubleshooting