Junos OS

Hey guys,

i just buy 2x qfx 3500 and run them as Virtual Chassis. There is no Traffic load yet but i see high cpu usage from the fxpc process

"1583 root        2 -52  -52   579M   109M umtx   0  36.7H 37.55% fxpc"
the question is, why it used upto 50% cpu and for what works this FXPC process.

Also i have huge Ping times upto 300ms, never seen it before at Juniper devices where is no load at all

Thanks

Can you provide the complete output "show system processes extensive | except 0.00" and show virtual-chassis? 

VCP stable?

Regards,

Rahul

Hi, thanks for the reply

root@qfx> show system processes extensive | except 0.00
last pid: 51323;  load averages:  1.12,  0.99,  0.87  up 3+18:05:17    15:29:57
162 processes: 34 running, 111 sleeping, 17 waiting

Mem: 540M Active, 61M Inact, 1578M Wired, 167M Cache, 112M Buf, 1722M Free
Swap: 2032M Total, 2032M Free


  PID USERNAME  THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
   35 root        1 155   52     0K    16K CPU6   6  89.9H 93.95% idle: cpu6
   36 root        1 155   52     0K    16K CPU5   5  84.9H 93.95% idle: cpu5
   34 root        1 155   52     0K    16K CPU7   7  89.9H 93.21% idle: cpu7
   37 root        1 155   52     0K    16K CPU4   4  89.6H 92.48% idle: cpu4
   38 root        1 155   52     0K    16K CPU3   3  89.1H 90.97% idle: cpu3
   39 root        1 155   52     0K    16K CPU2   2  88.7H 90.19% idle: cpu2
   40 root        1 155   52     0K    16K CPU1   1  87.1H 84.96% idle: cpu1
   41 root        1 155   52     0K    16K RUN    0  80.1H 82.67% idle: cpu0
   33 root        1 155   52     0K    16K CPU8   8  55.6H 49.61% idle: cpu8
 1583 root        2 -52  -52   579M   109M select 8  38.5H 41.31% fxpc
 1577 root        1  22    0 78876K 24108K select 2  92:14  0.73% chassisd


{master:0}
root@qfx> show virtual-chassis

Preprovisioned Virtual Chassis
Virtual Chassis ID: c206.a3b6.2a68
Virtual Chassis Mode: Enabled
                                                Mstr           Mixed Route Neighbor List
Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
0 (FPC 0)  Prsnt    R1308-C      qfx3500s       129   Master*      N  VC   1  vcp-255/1/0
                                                                           1  vcp-255/1/1
1 (FPC 1)  Prsnt    P2894-C      qfx3500s       129   Backup       N  VC   0  vcp-255/1/0
                                                                           0  vcp-255/1/1

{master:0}

 show virtual-chassis
preprovisioned;
no-split-detection;
member 0 {
    role routing-engine;
    serial-number R1308-C;
}
member 1 {
    role routing-engine;
    serial-number P2894-C;
}

{master:0}[edit]

Interesting. May i have below command output as well?

CLI

show version
show chassis hardware

start shell
vty fpc0
show syslog messages
show threads
show threads cpu
show threads verbose

Please try the following once.

Start shell

vty fpc0 

FXPC0(vty)# set parser security 10
Security level set to 10

FXPC0(vty)# test sfp diagnostics disable
SFP toolkit diagnostics: disabled

FXPC0(vty)# test sfp periodic diagnostic-interval 10
new SFP diagnostic polling interval 10

root@1fx> show version
fpc0:
--------------------------------------------------------------------------
Hostname: qfx
Model: qfx3500s
Junos: 15.1R7.9
JUNOS Base OS boot [15.1R7.9]
JUNOS Base OS Software Suite [15.1R7.9]
JUNOS Kernel Software Suite [15.1R7.9]
JUNOS Crypto Software Suite [15.1R7.9]
JUNOS Online Documentation [15.1R7.9]
JUNOS Enterprise Software Suite [15.1R7.9]
JUNOS Packet Forwarding Engine Support (qfx-3) [15.1R7.9]
JUNOS Routing Software Suite [15.1R7.9]
JUNOS py-base-xlr [15.1R7.9]

fpc1:
--------------------------------------------------------------------------
Hostname: qfx2
Model: qfx3500s
Junos: 15.1R7.9
JUNOS Base OS boot [15.1R7.9]
JUNOS Base OS Software Suite [15.1R7.9]
JUNOS Kernel Software Suite [15.1R7.9]
JUNOS Crypto Software Suite [15.1R7.9]
JUNOS Online Documentation [15.1R7.9]
JUNOS Enterprise Software Suite [15.1R7.9]
JUNOS Packet Forwarding Engine Support (qfx-3) [15.1R7.9]
JUNOS Routing Software Suite [15.1R7.9]
JUNOS py-base-xlr [15.1R7.9]

{master:0}

root@qfx> show chassis hardware
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                R1308             Virtual Chassis
Routing Engine 0          BUILTIN      BUILTIN           QFX Routing Engine
Routing Engine 1          BUILTIN      BUILTIN           QFX Routing Engine
FPC 0            REV 18   750-036931   R1308-C           QFX3500-48S4Q
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           48x 10G-SFP+
    Xcvr 0       REV 01   740-031981   F17CJU02532       SFP+-10G-LR
    Xcvr 8       REV 01   740-021308   G1806156792       SFP+-10G-SR
    Xcvr 9       REV 01   740-031980   F17CJU02970       SFP+-10G-SR
    Xcvr 13      REV 01   740-031980   F176JU00476       SFP+-10G-SR
    Xcvr 19      50A 19   NON-JNPR     X0832800612       SFP+-10G-SR
  PIC 1                   BUILTIN      BUILTIN           4x 40G-QSFP+
    Xcvr 0       REV 01   740-038624   G1804005190-2     QSFP+-40G-CU1M
    Xcvr 1       REV 01   740-038624   D87B2942199-2     QSFP+-40G-CU1M
  MGMT BRD       REV 11   750-036946   BBBE0545          QFX3500-MGMT-RJ45-AFI
  Power Supply 0 Rev 06   740-032091   WD16659           JPSU-650W-AC-AFI
  Power Supply 1 Rev 04   740-032091   VC03998           JPSU-650W-AC-AFI
  Fan Tray 0
  Fan Tray 1
  Fan Tray 2
FPC 1            REV 16   750-036931   P2894-C           QFX3500-48S4Q
  CPU                     BUILTIN      BUILTIN           FPC CPU
  PIC 0                   BUILTIN      BUILTIN           48x 10G-SFP+
    Xcvr 0       REV 01   740-031981   F17CJU02531       SFP+-10G-LR
    Xcvr 8       REV 01   740-021308   G1806156790       SFP+-10G-SR
    Xcvr 9       REV 01   740-021308   G1806157568       SFP+-10G-SR
    Xcvr 13      REV 01   740-031980   F176JU00471       SFP+-10G-SR
  PIC 1                   BUILTIN      BUILTIN           4x 40G-QSFP+
    Xcvr 0       REV 01   740-038624   D87B2942199-1     QSFP+-40G-CU1M
    Xcvr 1       REV 01   740-038624   G1804005190-1     QSFP+-40G-CU1M
  MGMT BRD       REV 11   750-036946   BBBE0602          QFX3500-MGMT-RJ45-AFI
  Power Supply 0 Rev 06   740-032091   WD16660           JPSU-650W-AC-AFI
  Power Supply 1 Rev 05   740-032091   VH08307           JPSU-650W-AC-AFI
  Fan Tray 0
  Fan Tray 1
  Fan Tray 2

{master:0}

TFXPC0(vty)# show syslog messages
[Mon Nov  5 16:17:57 2018 LOG: Debug] UPDN msg to kernel for ifd:xe-0/0/19, flag:2, speed: 10000000000, duplex:0
[Mon Nov  5 16:17:57 2018 LOG: Info] NLPHY: xcvr detected.
[Mon Nov  5 16:17:57 2018 LOG: Info] NLPHY: DEVSTART for [FPC: 0, PIC: 0, Port: 19
[Mon Nov  5 16:17:57 2018 LOG: Info] DCBCM [FPC:0 PIC:0 Port:19]: Setting PHY speed as 10Gps
[Mon Nov  5 16:17:58 2018 LOG: Err] dcbcm_drv_port_get_link_status: ifd=xe-0/0/19 rmt=2
[Mon Nov  5 16:17:58 2018 LOG: Debug] sfp-0/0/19 Link Enabled
[Mon Nov  5 16:17:58 2018 LOG: Info] DCBCM [FPC:0 PIC:0 Port:19]: Un-Squelched the netlogic PHY
[Mon Nov  5 16:17:58 2018 LOG: Err] dcbcm_drv_port_get_link_status: ifd=xe-0/0/19 rmt=2
[Mon Nov  5 16:17:59 2018 LOG: Err] dcbcm_drv_port_get_link_status: ifd=xe-0/0/19 rmt=2
[Mon Nov  5 16:17:59 2018 LOG: Notice] sfp-0/0/19 link 19 SFP voltage high warning set
[Mon Nov  5 16:17:59 2018 LOG: Notice] sfp-0/0/19 link 19 SFP voltage low  warning set
[Mon Nov  5 16:17:59 2018 LOG: Notice] sfp-0/0/19 link 19 SFP laser bias current low  warning set
[Mon Nov  5 16:17:59 2018 LOG: Notice] sfp-0/0/19 link 19 SFP output power high warning set
[Mon Nov  5 16:17:59 2018 LOG: Notice] sfp-0/0/19 link 19 SFP output power low  warning set
[Mon Nov  5 16:17:59 2018 LOG: Err] dcbcm_drv_port_get_link_status: ifd=xe-0/0/19 rmt=2
[Mon Nov  5 16:18:01 2018 LOG: Debug] UPDN msg to kernel for ifd:xe-0/0/19, flag:1, speed: 10000000000, duplex:2
[Mon Nov  5 16:23:16 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:23:16 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:23:16 2018 LOG: Debug] CMQFX: no change in uft config inside PFE. Dont reinit the asic
[Mon Nov  5 16:23:16 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:23:16 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:23:16 2018 LOG: Debug] CMQFX: no change in uft-num-prefix inside PFE. Dont reinit the asic
[Mon Nov  5 16:23:16 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:23:16 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:23:16 2018 LOG: Info] CMQFX: Auto-channel-global flag val:0
[Mon Nov  5 16:23:17 2018 LOG: Err] LBCM-L2,pfe_bcm_l2_evt_entry_unset(),1805:Egress vlan-xlat counter detach failed for port-class(0), vlan-token(5) error(4294967295)

[Mon Nov  5 16:23:17 2018 LOG: Err] LBCM-L2,brcm_port_init_def(),1056:(brcm_port_init_def:1056) Setting L2 Learn (unit 0, port_num 20), learn_flg 0
[Mon Nov  5 16:25:32 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:25:32 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:25:32 2018 LOG: Debug] CMQFX: no change in uft config inside PFE. Dont reinit the asic
[Mon Nov  5 16:25:32 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:25:32 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:25:32 2018 LOG: Debug] CMQFX: no change in uft-num-prefix inside PFE. Dont reinit the asic
[Mon Nov  5 16:25:32 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:25:32 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:25:32 2018 LOG: Info] CMQFX: Auto-channel-global flag val:0
[Mon Nov  5 16:26:55 2018 LOG: Err] LBCM-L2,brcm_irb_intf_uninit(),5866:Failed (-1) to detach counters from dev 0 vlan 5 intf 7 irb irb.5

[Mon Nov  5 16:26:56 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:26:56 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:26:56 2018 LOG: Debug] CMQFX: no change in uft config inside PFE. Dont reinit the asic
[Mon Nov  5 16:26:56 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:26:56 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:26:56 2018 LOG: Debug] CMQFX: no change in uft-num-prefix inside PFE. Dont reinit the asic
[Mon Nov  5 16:26:56 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:26:56 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:26:56 2018 LOG: Info] CMQFX: Auto-channel-global flag val:0
[Mon Nov  5 16:27:38 2018 LOG: Info] Next-hop resolution requests from interface 571 throttled
[Mon Nov  5 16:28:55 2018 LOG: Info] Next-hop resolution requests from interface 571 throttled
[Mon Nov  5 16:29:28 2018 LOG: Info] Next-hop resolution requests from interface 571 throttled
[Mon Nov  5 16:31:04 2018 LOG: Err] LBCM-L2,brcm_irb_intf_init(),5594:Failed(-1) to attach counters to dev 0 vlan 5 intf 7 irb irb.5

[Mon Nov  5 16:31:04 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:31:04 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:31:04 2018 LOG: Debug] CMQFX: no change in uft config inside PFE. Dont reinit the asic
[Mon Nov  5 16:31:04 2018 LOG: Debug] CMLC_LOCAL: Socket event, 0x8
[Mon Nov  5 16:31:04 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:31:04 2018 LOG: Debug] CMQFX: no change in uft-num-prefix inside PFE. Dont reinit the asic
[Mon Nov  5 16:31:04 2018 LOG: Debug] CMLC_LOCAL: Parsing IPC message
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: ipc message pointer = 0x79cb5aa8
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: action -> cmd
[Mon Nov  5 16:31:04 2018 LOG: Debug] IPCMSG: type: 4, subtype: 258, length: 24
[Mon Nov  5 16:31:04 2018 LOG: Info] CMQFX: Auto-channel-global flag val:0
[Mon Nov  5 16:31:04 2018 LOG: Err] LBCM-L2,brcm_port_init_def(),1056:(brcm_port_init_def:1056) Setting L2 Learn (unit 0, port_num 20), learn_flg 0
[Mon Nov  5 16:31:04 2018 LOG: Err] LBCM-L2,brcm_port_l3_intf_uninit(),5581:l3 interface deletion success. xe-0/0/19.0, (null):Ok
[Mon Nov  5 16:31:05 2018 LOG: Err] LBCM-L2,brcm_ifl_l2_init(),2204:(brcm_ifl_l2_init:2204) Setting port learning config :xe-0/0/19, learn_flg 5, move_flg 13
[Mon Nov  5 16:31:05 2018 LOG: Err] LBCM-L2,brcm_port_learning_config(),1188:(brcm_port_learning_config:1188) Setting L2 learning unit:0,port_num:20, learn_flg 5
[Mon Nov  5 16:31:05 2018 LOG: Err] LBCM-L2,pfe_bcm_l2_evt_port_class_get_first_used(),184:Assign port_class_id: 0
[Mon Nov  5 16:31:05 2018 LOG: Err] LBCM-L2,brcm_ifl_l2_init(),2204:(brcm_ifl_l2_init:2204) Setting port learning config :xe-0/0/19, learn_flg 5, move_flg 13
[Mon Nov  5 16:31:05 2018 LOG: Err] LBCM-L2,brcm_port_learning_config(),1188:(brcm_port_learning_config:1188) Setting L2 learning unit:0,port_num:20, learn_flg 5
[Mon Nov  5 17:01:44 2018 LOG: Info] Next-hop resolution requests from interface 571 throttled
[Mon Nov  5 17:02:57 2018 LOG: Info] Next-hop resolution requests from interface 571 throttled

TFXPC0(vty)# show threads
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
  1 H  asleep    Maintenance            568/73824  0/1/3 ms  0%
  2 L  running   Idle                   936/73824  2/64/15928455 ms  4%
  3 H  asleep    Timer Services         896/73824  0/1/63152 ms  0%
  5 L  asleep    Ukern Syslog           456/73824  0/0/0 ms  0%
  6 L  asleep    Sheaf Background       928/73824  0/1/2456 ms  0%
  7 M  asleep    SFP                  41488/73824  0/158/238386 ms  0%
  8 M  asleep    QSFP                  1336/73824  0/54/625023 ms  0%
  9 M  asleep    CXP                   1000/73824  0/1/11189 ms  0%
 10 M  asleep    Ethernet              6936/73824  0/7/3267791 ms  0%
 11 L  asleep    Syslog                6448/73824  0/1/560 ms  0%
 12 M  asleep    mac_db                 488/73824  0/0/0 ms  0%
 13 M  asleep    Periodic               856/73824  0/1/117673 ms  0%
 14 M  asleep    Mellanox BridgeX driver   968/73824  0/1/63163 ms  0%
 15 L  asleep    Console               6272/172128  3/3/3 ms  0%
 16 M  asleep    RPM Msg thread         504/73824  0/0/0 ms  0%
 17 H  asleep    Pfesvcsor              864/73824  0/3/67035 ms  0%
 18 M  asleep    PIC Periodic          6592/73824  1/2/159679 ms  0%
 19 M  asleep    PIC                    456/73824  0/0/0 ms  0%
 20 M  asleep    CMVC                  6776/172128  0/11/44193 ms  0%
 21 M  asleep    CMQFX                10248/172128  0/10/239586 ms  0%
 22 L  asleep    DFW Alert             1128/73824  0/1/11087 ms  0%
 23 L  asleep    ICMP6 Input           2824/73824  0/1/321 ms  0%
 24 L  asleep    IP6 Option Input      6320/73824  1/1/1 ms  0%
 25 L  asleep    ICMP Input            6336/73824  0/10/335438 ms  0%
 26 L  asleep    IP Option Input       6320/73824  1/1/1 ms  0%
 27 M  asleep    IGMP Input            6320/73824  1/1/1 ms  0%
 28 L  asleep    NH Probe Service       504/73824  0/0/0 ms  0%
 29 L  asleep    Cattle-Prod Daemon    3704/106592  2/2/2 ms  0%
 30 M  asleep    PRIVATE PFE Manager   6416/172128  0/3/229 ms  0%
 31 M  asleep    PPM Manager           6560/73824  0/11/6858 ms  0%
 33 M  asleep    bcmPOLL               1040/57440  0/1/22785 ms  0%
 34 M  asleep    bcmDPC                 664/57440  1/1/1 ms  0%
 35 M  asleep    PPM Data thread       1168/73824  0/1/55548 ms  0%
 36 M  asleep    VRRP Manager          6320/73824  4/4/4 ms  0%
 37 M  asleep    PFE Manager          28312/172128  1/843/6259638 ms  1%
 38 L  asleep    SFlow client          6408/73824  1/4/85 ms  0%
 39 L  asleep    SFlow sample deamon    960/73824  0/1/42140 ms  0%
 42 M  asleep    L2ALM Manager         6832/73824  0/6/25632 ms  0%
 43 M  asleep    PFE Statistics        8432/172128  0/103/1154221 ms  0%
 44 H  asleep    brcm opus cos stats daemon  4200/73824  16/32/3002966 ms  0%
 45 M  asleep    bcmCNTR.0             2856/57440  0/170/90747869 ms 27%
 46 M  asleep    bcmLINK.0            22840/73824  3/190/1187851 ms  0%
 47 L  asleep    BCM SDK periodic       888/73824  0/1/13387 ms  0%
 48 L  asleep    DDOS Policers         6504/73824  1/22/305863 ms  0%
 49 L  asleep    BRCM SALM Aging       1824/73824  1/50/876817 ms  0%
 50 M  asleep    BRCM SALM Pending Cleanup  1976/73824  0/4/51579 ms  0%
 51 M  asleep    bcmL2MOD.0            1336/57440  0/3/89 ms  0%
 52 L  asleep    Rate Limit             888/73824  0/1/13282 ms  0%
 53 L  asleep    AS Packet Process      528/73824  0/0/0 ms  0%
 54 M  asleep    brcm ifl stats daemon  2920/73824  1/3/17110 ms  0%
 55 M  asleep    bulkget Manager       9720/73824  0/6/1893 ms  0%
 58 L  ready     Virtual Console       4020/106592  1/21/86 ms  0%
 59 L  asleep    Virtual Console       2544/106592  1/1/1 ms  0%

TFXPC0(vty)#

TFXPC0(vty)# show threads cpu
PID PR State     Name                   Stack Use  Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------  ---------------------
 45 M  asleep    bcmCNTR.0             2856/57440  0/170/90753561 ms 27%
  2 L  running   Idle                   936/73824  4/64/15929694 ms  4%
 37 M  asleep    PFE Manager          28312/172128  0/843/6261173 ms  1%
 10 M  asleep    Ethernet              6936/73824  0/7/3268069 ms  0%
 44 H  asleep    brcm opus cos stats daemon  4200/73824  0/32/3003147 ms  0%
 46 M  asleep    bcmLINK.0            22840/73824  3/190/1187913 ms  0%
 43 M  asleep    PFE Statistics        8432/172128  0/103/1154322 ms  0%
 49 L  asleep    BRCM SALM Aging       1824/73824  0/50/876817 ms  0%
  8 M  asleep    QSFP                  1336/73824  0/54/625080 ms  0%
 25 L  asleep    ICMP Input            6336/73824  0/10/335438 ms  0%
 48 L  asleep    DDOS Policers         6504/73824  0/22/305881 ms  0%
 21 M  asleep    CMQFX                10248/172128  0/10/239605 ms  0%
  7 M  asleep    SFP                  41488/73824  0/158/238404 ms  0%
 18 M  asleep    PIC Periodic          6592/73824  0/2/159687 ms  0%
 13 M  asleep    Periodic               856/73824  0/1/117679 ms  0%
 17 H  asleep    Pfesvcsor              864/73824  0/3/67046 ms  0%
 14 M  asleep    Mellanox BridgeX driver   968/73824  0/1/63165 ms  0%
  3 H  asleep    Timer Services         896/73824  0/1/63162 ms  0%
 35 M  asleep    PPM Data thread       1168/73824  0/1/55552 ms  0%
 50 M  asleep    BRCM SALM Pending Cleanup  1976/73824  0/4/51582 ms  0%
 20 M  asleep    CMVC                  6776/172128  0/11/44194 ms  0%
 39 L  asleep    SFlow sample deamon    960/73824  0/1/42142 ms  0%
 42 M  asleep    L2ALM Manager         6832/73824  1/6/25634 ms  0%
 33 M  asleep    bcmPOLL               1040/57440  0/1/22785 ms  0%
 54 M  asleep    brcm ifl stats daemon  2920/73824  0/3/17112 ms  0%
 47 L  asleep    BCM SDK periodic       888/73824  0/1/13388 ms  0%
 52 L  asleep    Rate Limit             888/73824  0/1/13286 ms  0%
  9 M  asleep    CXP                   1000/73824  0/1/11190 ms  0%
 22 L  asleep    DFW Alert             1128/73824  0/1/11087 ms  0%
 31 M  asleep    PPM Manager           6560/73824  1/11/6859 ms  0%
  6 L  asleep    Sheaf Background       928/73824  0/1/2456 ms  0%
 55 M  asleep    bulkget Manager       9720/73824  0/6/1893 ms  0%
 11 L  asleep    Syslog                6448/73824  0/1/560 ms  0%
 23 L  asleep    ICMP6 Input           2824/73824  0/1/321 ms  0%
 30 M  asleep    PRIVATE PFE Manager   6416/172128  0/3/229 ms  0%
 58 L  ready     Virtual Console       4020/106592  1/21/118 ms  0%
 51 M  asleep    bcmL2MOD.0            1336/57440  0/3/89 ms  0%
 38 L  asleep    SFlow client          6408/73824  1/4/85 ms  0%
 36 M  asleep    VRRP Manager          6320/73824  4/4/4 ms  0%
  1 H  asleep    Maintenance            568/73824  0/1/3 ms  0%
 15 L  asleep    Console               6272/172128  3/3/3 ms  0%
 29 L  asleep    Cattle-Prod Daemon    3704/106592  2/2/2 ms  0%
 59 L  asleep    Virtual Console       2544/106592  1/1/1 ms  0%
 34 M  asleep    bcmDPC                 664/57440  1/1/1 ms  0%
 27 M  asleep    IGMP Input            6320/73824  1/1/1 ms  0%
 26 L  asleep    IP Option Input       6320/73824  1/1/1 ms  0%
 24 L  asleep    IP6 Option Input      6320/73824  1/1/1 ms  0%
 12 M  asleep    mac_db                 488/73824  0/0/0 ms  0%
 53 L  asleep    AS Packet Process      528/73824  0/0/0 ms  0%
  5 L  asleep    Ukern Syslog           456/73824  0/0/0 ms  0%
 16 M  asleep    RPM Msg thread         504/73824  0/0/0 ms  0%
 19 M  asleep    PIC                    456/73824  0/0/0 ms  0%
 28 L  asleep    NH Probe Service       504/73824  0/0/0 ms  0%

TFXPC0(vty)#

TFXPC0(vty)# show threads verbose
PID PR State     Name                   Stack Use        Started@ Time (Last/Max/Total) cpu
--- -- -------   ---------------------  ---------     -------- ------------------------
  1 H  asleep    Maintenance            568/73824         4640 ms 0/1/3 ms  0%
  2 L  running   Idle                   936/73824         4644 ms 0/64/15930771 ms  4%
  3 H  asleep    Timer Services         896/73824         4653 ms 0/1/63170 ms  0%
  5 L  asleep    Ukern Syslog           456/73824         4664 ms 0/0/0 ms  0%
  6 L  asleep    Sheaf Background       928/73824         4685 ms 0/1/2456 ms  0%
  7 M  asleep    SFP                  41488/73824         4867 ms 1/158/238417 ms  0%
  8 M  asleep    QSFP                  1336/73824         4876 ms 0/54/625121 ms  0%
  9 M  asleep    CXP                   1000/73824         4881 ms 0/1/11192 ms  0%
 10 M  asleep    Ethernet              6936/73824         4951 ms 0/7/3268327 ms  0%
 11 L  asleep    Syslog                6448/73824         4994 ms 0/1/560 ms  0%
 12 M  asleep    mac_db                 488/73824         5095 ms 0/0/0 ms  0%
 13 M  asleep    Periodic               856/73824         5153 ms 1/1/117686 ms  0%
 14 M  asleep    Mellanox BridgeX driver   968/73824         5164 ms 0/1/63169 ms  0%
 15 L  asleep    Console               6272/172128         5385 ms 3/3/3 ms  0%
 16 M  asleep    RPM Msg thread         504/73824         5389 ms 0/0/0 ms  0%
 17 H  asleep    Pfesvcsor              864/73824         5390 ms 0/3/67056 ms  0%
 18 M  asleep    PIC Periodic          6592/73824         5511 ms 1/2/159694 ms  0%
 19 M  asleep    PIC                    456/73824         5512 ms 0/0/0 ms  0%
 20 M  asleep    CMVC                  6776/172128         5794 ms 0/11/44194 ms  0%
 21 M  asleep    CMQFX                10248/172128         5820 ms 0/10/239621 ms  0%
 22 L  asleep    DFW Alert             1128/73824         5961 ms 0/1/11087 ms  0%
 23 L  asleep    ICMP6 Input           2824/73824         6117 ms 0/1/321 ms  0%
 24 L  asleep    IP6 Option Input      6320/73824         6118 ms 1/1/1 ms  0%
 25 L  asleep    ICMP Input            6336/73824         6139 ms 0/10/335438 ms  0%
 26 L  asleep    IP Option Input       6320/73824         6148 ms 1/1/1 ms  0%
 27 M  asleep    IGMP Input            6320/73824         6149 ms 1/1/1 ms  0%
 28 L  asleep    NH Probe Service       504/73824         6310 ms 0/0/0 ms  0%
 29 L  asleep    Cattle-Prod Daemon    3704/106592         6382 ms 2/2/2 ms  0%
 30 M  asleep    PRIVATE PFE Manager   6416/172128         6480 ms 0/3/229 ms  0%
 31 M  asleep    PPM Manager           6560/73824         6482 ms 0/11/6860 ms  0%
 33 M  asleep    bcmPOLL               1040/57440         6946 ms 0/1/22786 ms  0%
 34 M  asleep    bcmDPC                 664/57440         7975 ms 1/1/1 ms  0%
 35 M  asleep    PPM Data thread       1168/73824         7997 ms 0/1/55556 ms  0%
 36 M  asleep    VRRP Manager          6320/73824         8062 ms 4/4/4 ms  0%
 37 M  asleep    PFE Manager          28312/172128         8074 ms 0/843/6262546 ms  1%
 38 L  asleep    SFlow client          6408/73824         8090 ms 1/4/85 ms  0%
 39 L  asleep    SFlow sample deamon    960/73824         8091 ms 0/1/42142 ms  0%
 42 M  asleep    L2ALM Manager         6832/73824         8165 ms 0/6/25635 ms  0%
 43 M  asleep    PFE Statistics        8432/172128         8176 ms 0/103/1154322 ms  0%
 44 H  asleep    brcm opus cos stats daemon  4200/73824         8216 ms 0/32/3003308 ms  0%
 45 M  asleep    bcmCNTR.0             2856/57440        23942 ms 0/170/90758642 ms 27%
 46 M  asleep    bcmLINK.0            22840/73824        25609 ms 2/190/1187963 ms  0%
 47 L  asleep    BCM SDK periodic       888/73824        25865 ms 0/1/13388 ms  0%
 48 L  asleep    DDOS Policers         6504/73824        30225 ms 0/22/305898 ms  0%
 49 L  asleep    BRCM SALM Aging       1824/73824        30261 ms 0/50/876817 ms  0%
 50 M  asleep    BRCM SALM Pending Cleanup  1976/73824        30262 ms 0/4/51583 ms  0%
 51 M  asleep    bcmL2MOD.0            1336/57440        30279 ms 0/3/89 ms  0%
 52 L  asleep    Rate Limit             888/73824        43023 ms 0/1/13288 ms  0%
 53 L  asleep    AS Packet Process      528/73824       219332 ms 0/0/0 ms  0%
 54 M  asleep    brcm ifl stats daemon  2920/73824       219866 ms 0/3/17113 ms  0%
 55 M  asleep    bulkget Manager       9720/73824       219926 ms 0/6/1893 ms  0%
 58 L  ready     Virtual Console       4020/106592    333119483 ms 0/21/167 ms  0%
 59 L  asleep    Virtual Console       2544/106592    333521651 ms 1/1/1 ms  0%

TFXPC0(vty)#

There is a thread in the code related to the 40G interfaces where it run at 500ms which cause the CPU to run high on the QFX3500 due to its process. As a workaround, you can execute below command under the shell.

start shell user root

root@:RE:0% cprod -A fpc0 -c 'set dcbcm bcmshell "counter off"'

Regards,

Rahul

The usage goes down to 15% and the ping times are mutch better. Thanks for the help.

Hi Rahul,

Does this trick solve the issue in EX3400s as well?

Best Regards,

Supun

Hi Supun,

You need to provide below output to understand if it's same or different. Please feel free to open new thread with below output

Thanks Rahul,

PFA.,

BR

Supun

Attachment(s)

Logs-FXPC.txt   22 KB 1 version

Is there a way to essentially "undo" this command, this disabled all monitoring/snmp graphing of inbound traffic on all interfaces. I tried setting "counter on" but this did not work.