Junos OS

last person joined: 7 days ago 

Ask questions and share experiences about Junos OS.

  • 1.  QFX5110 high FPC CPU utilization

    Posted 07-23-2019 05:03

    Hi,

    We have freshly installed QFX5110-48S with no load at this moment. But FPC CPU load is rolling over 60%.

    What we can see to diagnose and correct the cause ?

    Model: qfx5110-48s-4c
Junos: 18.1R3-S4.2
    > show chassis fpc       
                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
  0  Online            35     60          0       59     59     59    16384      28         49
  1  Empty

     

    Spoiler
    > show chassis fpc errors    
FPC Scope    Category   Level Occurred Cleared Threshold Action-Taken Action
  0 board    functional Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             memory     Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             io         Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             storage    Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             switch     Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             processing Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
    pfe      functional Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             memory     Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             io         Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             storage    Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             switch     Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0
             processing Minor      0        0       0         0
                        Major      0        0       0         0
                        Fatal      0        0       0         0


  • 2.  RE: QFX5110 high FPC CPU utilization

     
    Posted 07-23-2019 05:26

    Good to have below shell command output.

     

    labroot@S2> start shell pfe network fpc0

    Switching platform (1799 Mhz Pentium processor, 639MB memory, 0KB flash)

    FPC0(S2 vty)# show threads
    FPC0(S2 vty)# show sched
    FPC0(S2 vty)# set parser security 10
    FPC0(S2 vty)# clear sched
    FPC0(S2 vty)# show threads
    FPC0(S2 vty)# show sched



  • 3.  RE: QFX5110 high FPC CPU utilization

    Posted 07-23-2019 05:35
    Spoiler
    #show sched
Total uptime 29+05:30:31, (2525431836 ms), 1525893213 thread dispatches
CPU load is 9% (5 second), 0% (1 minute)
Total network interrupt time 0 (usec)

 CPU       Name      Time(ms)
 89%       Idle      2248398015
 10%    Threads      277033821
  0%        ISR      0


    #show threads
    PID PR State Name Stack Use Time (Last/Max/Total) cpu
    --- -- ------- --------------------- --------- ---------------------
    1 H asleep Maintenance 828/73824 0/0/0 ms 0%
    2 L running Idle 2464/73824 8/22/2248379408 ms 89%
    3 H asleep Timer Services 2456/73824 0/7/2398400 ms 0%
    5 L asleep Ukern Syslog 708/73824 0/0/0 ms 0%
    6 L asleep Sheaf Background 796/73824 0/1/2477 ms 0%
    7 M asleep SFP 2924/73824 0/51/56595 ms 0%
    8 M asleep QSFP 3532/73824 0/64/6443642 ms 0%
    9 M asleep CXP 2576/73824 0/1/10488 ms 0%
    10 M asleep Ethernet 6456/73824 0/71/16350096 ms 0%
    11 L asleep Syslog 884/73824 0/3/8053 ms 0%
    12 M asleep mac_db 708/73824 0/0/0 ms 0%
    13 M asleep Periodic 2248/73824 0/3/67724 ms 0%
    14 M asleep PTP stack 4092/73824 0/5/1309122 ms 0%
    15 M asleep SyncApp 2908/73824 0/0/0 ms 0%
    16 H asleep TTP Receive 1832/73824 0/2/18886 ms 0%
    17 H asleep TTP Transmit 708/73824 0/0/0 ms 0%
    18 L asleep Console 3196/73824 1/1/1 ms 0%
    19 M asleep CLKSYNC Manager 2844/73824 0/0/0 ms 0%
    20 L asleep tvp_rpc_thread 2616/73824 0/4/2347862 ms 0%
    21 M asleep RPM Msg thread 708/73824 0/0/0 ms 0%
    22 H asleep Pfesvcsor 724/73824 0/0/0 ms 0%
    23 M asleep Wheel timer 2756/73824 0/3/20526 ms 0%
    24 M asleep caem ipc 4152/73824 0/4/490333 ms 0%
    25 M asleep probe processing 2148/73824 0/0/0 ms 0%
    26 M asleep probe processing 2156/73824 0/0/0 ms 0%
    27 M asleep PIC Periodic 3076/73824 0/6/515887 ms 0%
    28 M asleep PIC 700/73824 0/0/0 ms 0%
    29 M asleep CMVC 3428/73824 0/3/60137 ms 0%
    30 M asleep CMQFX 19388/73824 0/30/367723 ms 0%
    31 L asleep hfreq q stats 2164/73824 0/0/0 ms 0%
    32 L asleep DFW Alert 2704/73824 0/3/12834 ms 0%
    33 L asleep ICMP6 Input 996/73824 0/0/0 ms 0%
    34 L asleep IP6 Option Input 2780/73824 0/0/0 ms 0%
    35 L asleep ICMP Input 2748/73824 0/0/0 ms 0%
    36 L asleep IP Option Input 2780/73824 0/0/0 ms 0%
    37 M asleep IGMP Input 2780/73824 0/0/0 ms 0%
    38 L asleep NH Probe Service 724/73824 0/0/0 ms 0%
    39 L asleep Cattle-Prod Daemon 3372/73824 0/0/0 ms 0%
    40 M asleep PRIVATE PFE Manager 4448/73824 0/1/2602 ms 0%
    41 M asleep PPM Manager 2972/73824 1/2/4 ms 0%
    43 M asleep bcmPOLL 964/57440 0/1/1 ms 0%
    44 M asleep socdmadesc.0 2664/57440 0/6/5373756 ms 0%
    45 M asleep bcmDPC 948/57440 0/0/0 ms 0%
    46 M asleep PPM Data thread 740/73824 0/1/2078 ms 0%
    47 M asleep VRRP Manager 2756/73824 0/0/0 ms 0%
    48 M asleep PFE Manager 42100/73824 0/297/95848 ms 0%
    49 L asleep SFlow client 2380/73824 0/0/0 ms 0%
    50 L asleep SFlow sample deamon 2240/73824 0/2/42122 ms 0%
    52 L ready Virtual Console (private) 3396/73824 0/1/3 ms 0%
    53 M asleep L2ALM Manager 4768/73824 0/3/217333 ms 0%
    54 M asleep ARP Proxy 2600/73824 0/5/123146 ms 0%
    55 M asleep PFE Statistics 4728/73824 0/71/5001852 ms 0%
    56 H asleep brcm opus cos stats daemon 4312/73824 0/15/4785017 ms 0%
    57 M asleep bcmMEM_SCAN.0 3912/57440 47/59/4445261 ms 0%
    58 M asleep bcmSRAM_SCAN.0 2712/57440 1/7/46234 ms 0%
    59 M asleep bcmL2age.0 1060/57440 0/1/48 ms 0%
    61 M asleep bcmLINK.0 5832/73824 2/104/9335400 ms 0%
    62 L asleep BCM SDK periodic 2136/73824 0/2/14642 ms 0%
    63 L asleep DDOS Policers 4584/73824 0/4/280954 ms 0%
    64 L asleep BRCM SALM Aging 3836/73824 0/24/19913577 ms 0%
    65 M asleep BRCM SALM Pending Cleanup 3912/73824 0/6/227203 ms 0%
    66 M asleep bcmL2MOD.0 2392/57440 0/7/4239 ms 0%
    67 L asleep Rate Limit 2160/73824 0/1/2905 ms 0%
    68 L asleep Rate Limit 2176/73824 0/1/2918 ms 0%
    69 L asleep AS Packet Process 2380/73824 0/0/0 ms 0%
    70 M asleep bulkget Manager 3948/73824 0/1/2427 ms 0%
    71 M asleep bcmCNTR.0 3744/57440 95/135/176552361 ms 6%
    72 M asleep brcm ifl stats daemon 4656/73824 0/5/228924 ms 0%
    73 L asleep Virtual Console 2244/73824 0/0/0 ms 0%
    74 L asleep Virtual Console (private) 2252/73824 0/0/0 ms 0%
    Spoiler
    # show sched
Total uptime 0+00:00:27, (27435 ms), 16580 thread dispatches
CPU load is 13% (5 second), 124011725% (1 minute)
Total network interrupt time 0 (usec)

 CPU       Name      Time(ms)
 81%       Idle      22320
 18%    Threads      5115
  0%        ISR      0

#show threads 
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
  1 H  asleep    Maintenance            828/73824  0/0/0 ms  0%
  2 L  running   Idle                  2464/73824  5/9/3010 ms 51%
  3 H  asleep    Timer Services        2456/73824  0/1/6 ms  0%
  5 L  asleep    Ukern Syslog           708/73824  0/0/0 ms  0%
  6 L  asleep    Sheaf Background       796/73824  1/1/1 ms  0%
  7 M  asleep    SFP                   2924/73824  0/0/0 ms  0%
  8 M  asleep    QSFP                  3532/73824  0/28/56 ms  0%
  9 M  asleep    CXP                   2576/73824  0/0/0 ms  0%
 10 M  asleep    Ethernet              6456/73824  0/2/29 ms  0%
 11 L  asleep    Syslog                 884/73824  0/0/0 ms  0%
 12 M  asleep    mac_db                 708/73824  0/0/0 ms  0%
 13 M  asleep    Periodic              2248/73824  0/0/0 ms  0%
 14 M  asleep    PTP stack             4092/73824  0/1/3 ms  0%
 15 M  asleep    SyncApp               2908/73824  0/0/0 ms  0%
 16 H  asleep    TTP Receive           1832/73824  0/0/0 ms  0%
 17 H  asleep    TTP Transmit           708/73824  0/0/0 ms  0%
 18 L  asleep    Console               3196/73824  0/0/0 ms  0%
 19 M  asleep    CLKSYNC Manager       2844/73824  0/0/0 ms  0%
 20 L  asleep    tvp_rpc_thread        2616/73824  0/1/8 ms  0%
 21 M  asleep    RPM Msg thread         708/73824  0/0/0 ms  0%
 22 H  asleep    Pfesvcsor              724/73824  0/0/0 ms  0%
 23 M  asleep    Wheel timer           2756/73824  0/0/0 ms  0%
 24 M  asleep    caem ipc              4152/73824  0/1/1 ms  0%
 25 M  asleep    probe processing      2148/73824  0/0/0 ms  0%
 26 M  asleep    probe processing      2156/73824  0/0/0 ms  0%
 27 M  asleep    PIC Periodic          3076/73824  0/1/1 ms  0%
 28 M  asleep    PIC                    700/73824  0/0/0 ms  0%
 29 M  asleep    CMVC                  3428/73824  0/0/0 ms  0%
 30 M  asleep    CMQFX                19388/73824  0/0/0 ms  0%
 31 L  asleep    hfreq q stats         2164/73824  0/0/0 ms  0%
 32 L  asleep    DFW Alert             2704/73824  0/0/0 ms  0%
 33 L  asleep    ICMP6 Input            996/73824  0/0/0 ms  0%
 34 L  asleep    IP6 Option Input      2780/73824  0/0/0 ms  0%
 35 L  asleep    ICMP Input            2748/73824  0/0/0 ms  0%
 36 L  asleep    IP Option Input       2780/73824  0/0/0 ms  0%
 37 M  asleep    IGMP Input            2780/73824  0/0/0 ms  0%
 38 L  asleep    NH Probe Service       724/73824  0/0/0 ms  0%
 39 L  asleep    Cattle-Prod Daemon    3372/73824  0/0/0 ms  0%
 40 M  asleep    PRIVATE PFE Manager   4448/73824  0/0/0 ms  0%
 41 M  asleep    PPM Manager           2972/73824  0/0/0 ms  0%
 43 M  asleep    bcmPOLL                964/57440  0/0/0 ms  0%
 44 M  asleep    socdmadesc.0          2664/57440  0/1/6 ms  0%
 45 M  asleep    bcmDPC                 948/57440  0/0/0 ms  0%
 46 M  asleep    PPM Data thread        740/73824  0/0/0 ms  0%
 47 M  asleep    VRRP Manager          2756/73824  0/0/0 ms  0%
 48 M  asleep    PFE Manager          42100/73824  0/0/0 ms  0%
 49 L  asleep    SFlow client          2380/73824  0/0/0 ms  0%
 50 L  asleep    SFlow sample deamon   2240/73824  0/0/0 ms  0%
 52 L  ready     Virtual Console (private)  4412/73824  0/1/2 ms  0%
 53 M  asleep    L2ALM Manager         4768/73824  0/1/1 ms  0%
 54 M  asleep    ARP Proxy             2600/73824  0/0/0 ms  0%
 55 M  asleep    PFE Statistics        4728/73824  0/0/0 ms  0%
 56 H  asleep    brcm opus cos stats daemon  4312/73824  0/4/14 ms  0%
 57 M  asleep    bcmMEM_SCAN.0         3912/57440  3/3/3 ms  0%
 58 M  asleep    bcmSRAM_SCAN.0        2712/57440  0/0/0 ms  0%
 59 M  asleep    bcmL2age.0            1060/57440  0/0/0 ms  0%
 61 M  asleep    bcmLINK.0             5832/73824  2/3/13 ms  0%
 62 L  asleep    BCM SDK periodic      2136/73824  0/0/0 ms  0%
 63 L  asleep    DDOS Policers         4584/73824  0/1/1 ms  0%
 64 L  asleep    BRCM SALM Aging       3836/73824  0/12/2261 ms 38%
 65 M  asleep    BRCM SALM Pending Cleanup  3912/73824  0/1/1 ms  0%
 66 M  asleep    bcmL2MOD.0            2392/57440  0/0/0 ms  0%
 67 L  asleep    Rate Limit            2160/73824  0/0/0 ms  0%
 68 L  asleep    Rate Limit            2176/73824  0/0/0 ms  0%
 69 L  asleep    AS Packet Process     2380/73824  0/0/0 ms  0%
 70 M  asleep    bulkget Manager       3948/73824  0/0/0 ms  0%
 71 M  asleep    bcmCNTR.0             3744/57440  0/95/392 ms  6%
 72 M  asleep    brcm ifl stats daemon  4656/73824  0/1/1 ms  0%
 73 L  asleep    Virtual Console       2244/73824  0/0/0 ms  0%
 74 L  asleep    Virtual Console (private)  2252/73824  0/0/0 ms  0%


  • 4.  RE: QFX5110 high FPC CPU utilization

     
    Posted 07-23-2019 05:43

    Thank you. Do you have "accept-source-mac" knob configured?

    Regards,
    Rahul



  • 5.  RE: QFX5110 high FPC CPU utilization

    Posted 07-23-2019 05:54

    Nope

     

    > show configuration | match accept-source-mac 

{master:0}


  • 6.  RE: QFX5110 high FPC CPU utilization

     
    Posted 07-24-2019 20:50

    In case you're still seeing, i would request to open JTAC case. This issue need further troubleshooting.



  • 7.  RE: QFX5110 high FPC CPU utilization
    Best Answer

    Posted 07-24-2019 21:45
    Rnayar, thank you for your assistance. Upgrading to JUNOS 18.1R3-S6.1 solved the problem