Switching

 View Only
last person joined: 2 days ago 

Ask questions and share experiences about EX and QFX portfolios and all switching solutions across your data center, campus, and branch locations.

  • 1.  High CPU and memory usage by fxpc with MACSEC_MII_1 Error on VC of EX4600 with 21.4R3-S2

    Posted 03-17-2023 10:54

    I have a problem with high CPU and memory usage seen in routing engines of both EX4600 that are stacked in VC and both are on newest/recommended Junos version 21.4R3-S2.4

    Result of this problem is slow response of ssh and very high latency when pinging this stack (900ms+) where traffic going through it is fine and without problems.

    It doesn't do much traffic for now as it is a new installation and I have this problem from start. There are no loops and RSTP is enabled.

    Below are the outputs of show commands where you can see that the fxpc process is hogging up all of this CPU and memory.

    > show chassis routing-engine 
Routing Engine status:
  Slot 0:
    Current state                  Master
    Temperature                 38 degrees C / 100 degrees F
    CPU temperature             38 degrees C / 100 degrees F
    DRAM                      1920 MB
    Memory utilization          77 percent
    5 sec CPU utilization:
      User                      63 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      34 percent
    1 min CPU utilization:
      User                      63 percent
      Background                 0 percent
      Kernel                     3 percent
      Interrupt                  0 percent
      Idle                      34 percent
    5 min CPU utilization:
      User                      64 percent
      Background                 0 percent
      Kernel                     3 percent
      Interrupt                  0 percent
      Idle                      34 percent
    15 min CPU utilization:
      User                      63 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      34 percent
    Model                          EX4600-40F
    Serial ID                      BUILTIN
    Start time                     2023-03-03 01:07:01 UTC
    Uptime                         6 days, 15 hours, 54 minutes, 41 seconds
    Last reboot reason             0x1:power cycle/failure 
    Load averages:                 1 minute   5 minute  15 minute
                                       1.28       1.16       1.02
Routing Engine status:
  Slot 1:
    Current state                  Backup
    Temperature                 39 degrees C / 102 degrees F
    CPU temperature             39 degrees C / 102 degrees F
    DRAM                      1920 MB
    Memory utilization          56 percent
    5 sec CPU utilization:
      User                      62 percent
      Background                 0 percent
      Kernel                     2 percent
      Interrupt                  0 percent
      Idle                      36 percent
    Model                          EX4600-40F
    Serial ID                      BUILTIN
    Uptime                         6 days, 15 hours, 55 minutes, 42 seconds
    Last reboot reason             0x1:power cycle/failure 
    Load averages:                 1 minute   5 minute  15 minute
                                       0.88       0.87       0.90
{master:0}

> show system processes extensive 
last pid: 17028;  load averages:  1.07,  1.12,  1.01  up 6+15:57:43    17:04:14
171 processes: 5 running, 145 sleeping, 1 zombie, 20 waiting

Mem: 964M Active, 129M Inact, 458M Wired, 290M Cache, 69M Buf, 7808K Free
Swap: 1106M Total, 1106M Free


  PID USERNAME    PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
 1951 root         68   -1  1268M   589M RUN    102.8H 61.13% fxpc
   10 root        155   52     0K    12K RUN     52.9H 34.52% idle
 1941 root         40    0   127M 37932K select  61:26  0.10% chassisd
 2185 root          4    0 55152K 41336K pfesta  22:55  0.05% mib2d
 1951 root        -52  -52  1268M   589M select 102.8H  0.00% fxpc

> show chassis fpc 
                     Temp  CPU Utilization (%)   CPU Utilization (%)  Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      1min   5min   15min  DRAM (MB) Heap     Buffer
  0  Online            39     64          0        0      0      0    1920        0         77
  1  Online            40     65          0        0      0      0    1920        0         55
  2  Empty        

> show virtual-chassis status 

Virtual Chassis ID: 9075.1b20.1236
Virtual Chassis Mode: Enabled
                                                Mstr           Mixed Route Neighbor List
Member ID  Status   Serial No    Model          prio  Role      Mode  Mode ID  Interface
0 (FPC 0)  Prsnt    TC37XXXXXXXX ex4600-40f     128   Master*      N  VC   1  vcp-255/0/26
                                                                           1  vcp-255/0/27
1 (FPC 1)  Prsnt    TC37XXXXXXXX ex4600-40f     128   Backup       N  VC   0  vcp-255/0/26
                                                                           0  vcp-255/0/27

Member ID for next new member: 2 (FPC 2)

    I tried troubleshooting the problem . There are no system or chassis alarms, VC is Established with just "no-split-detection" option in configuration.

    I think I found root of those problems as I see weird bugs after trying to vty to fpc0 and it just outputs errors regarding MACSEC which I doesn't plan of using and don't have anything in configuration for this.

    > start shell 
% vty fpc0


TOR platform (1500 Mhz Pentium processor, 511MB memory, 0KB flash)

TFXPC0( vty)# [Thu Mar  9 17:06:34.540 LOG: Debug] MMI op failed (op=0) address = 0x001000d9
[Thu Mar  9 17:06:50.410 LOG: Debug] MACSEC_MII_1 Error: Not Ready timeout
[Thu Mar  9 17:06:50.410 LOG: Debug] MMI op failed (op=0) address = 0x00100108
[Thu Mar  9 17:06:50.751 LOG: Debug] MACSEC_MII_1 Error: Not Ready timeout
[Thu Mar  9 17:06:50.751 LOG: Debug] MMI op failed (op=0) address = 0x001000d9
[Thu Mar  9 17:06:52.486 LOG: Debug] MACSEC_MII_1 Error: Not Ready timeout
[Thu Mar  9 17:06:52.486 LOG: Debug] MMI op failed (op=0) address = 0x00100108
[Thu Mar  9 17:06:52.826 LOG: Debug] MACSEC_MII_1 Error: Not Ready timeout
[Thu Mar  9 17:06:52.826 LOG: Debug] MMI op failed (op=0) address = 0x001000d9
[Thu Mar  9 17:06:53.063 LOG: Info] [DCBCM] dcbcm_eth_an_config: ifd - ge-0/0/5 AN spd = 1000000000, dpx = 2, an = 2, fc = 0
[Thu Mar  9 17:06:53.063 LOG: Info] [DCBCM] dcbcm_check_an_config_change: ifd - ge-0/0/5, OLD CFG spd = 1000000000, dpx = 2, an = 1, fc = 0 auto_speed:0 auto_mdix = 1 NEW CFG spd = 1000000000, dpx = 2, an = 1, fc = 0 auto_speed:0 auto_mdix = 1 ifd->ifd_media.ifm_speed = 100000000[Thu Mar  9 17:06:53.470 LOG: Debug] MACSEC_MII_1 Error: Not Ready timeout
[Thu Mar  9 17:06:53.470 LOG: Debug] MMI op failed (op=0) address = 0x00100108
[Thu Mar  9 17:06:53.811 LOG: Debug] MACSEC_MII_1 Error: Not Ready timeout
[Thu Mar  9 17:06:53.811 LOG: Debug] MMI op failed (op=0) address = 0x001000d9
[Thu Mar  9 17:06:54.475 LOG: Debug] MACSEC_MII_1 Error: Not Ready timeout
[Thu Mar  9 17:06:54.475 LOG: Debug] MMI op failed (op=0) address = 0x00100108

    I can't do anything on the console of fpc as it constantly just outputs errors I provided above.

    I thought, that problem is connected to my lastest/recommended software so I tried downgrading it but it fails with another, frist time for me, error - rcp which prevents me from upgrading this VC :(

    > show version 
fpc0:
--------------------------------------------------------------------------
Model: ex4600-40f
Junos: 21.4R3-S2.4
JUNOS Base OS boot [21.4R3-S2.4]
JUNOS Base OS Software Suite [21.4R3-S2.4]
JUNOS Online Documentation [21.4R3-S2.4]
JUNOS Crypto Software Suite [21.4R3-S2.4]
JUNOS Crypto Software Suite [21.4R3-S2.4]
JUNOS Kernel Software Suite [21.4R3-S2.4]
JUNOS Phone-Home Software Suite [21.4R3-S2.4]
JUNOS Packet Forwarding Engine Support (qfx-ex-x86-32) [21.4R3-S2.4]
JUNOS jsd [i386-21.4R3-S2.4-jet-1]
JUNOS Routing Software Suite [21.4R3-S2.4]
JUNOS SDN Software Suite [21.4R3-S2.4]
JUNOS Enterprise Software Suite [21.4R3-S2.4]
JUNOS Web Management Platform Package [21.4R3-S2.4]
JUNOS Openconfig [21.4R3-S2.4]
JUNOS py-base-i386 [21.4R3-S2.4]
JUNOS py-extensions-i386 [21.4R3-S2.4]
JUNOS Host Software [18.1R3-S6.1]

fpc1:
--------------------------------------------------------------------------
Model: ex4600-40f
Junos: 21.4R3-S2.4
JUNOS Base OS boot [21.4R3-S2.4]
JUNOS Base OS Software Suite [21.4R3-S2.4]
JUNOS Online Documentation [21.4R3-S2.4]
JUNOS Crypto Software Suite [21.4R3-S2.4]
JUNOS Crypto Software Suite [21.4R3-S2.4]
JUNOS Kernel Software Suite [21.4R3-S2.4]
JUNOS Phone-Home Software Suite [21.4R3-S2.4]
JUNOS Packet Forwarding Engine Support (qfx-ex-x86-32) [21.4R3-S2.4]
JUNOS jsd [i386-21.4R3-S2.4-jet-1]
JUNOS Routing Software Suite [21.4R3-S2.4]
JUNOS SDN Software Suite [21.4R3-S2.4]
JUNOS Enterprise Software Suite [21.4R3-S2.4]
JUNOS Web Management Platform Package [21.4R3-S2.4]
JUNOS Openconfig [21.4R3-S2.4]
JUNOS py-base-i386 [21.4R3-S2.4]
JUNOS py-extensions-i386 [21.4R3-S2.4]
JUNOS Host Software [18.1R3-S6.1]


> show system storage all-members 
fpc0:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             1.6G       1.4G       140M       91%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                121M       121M         0B      100%  /packages/mnt/jbase
/dev/md1                712M       712M         0B      100%  /packages/mnt/jkernel-qfx-ex-x86-32-21.4R3-S2.4
/dev/md2                138M       138M         0B      100%  /packages/mnt/jpfe-qfx-ex-x86-32-21.4R3-S2.4
/dev/md3                 13M        13M         0B      100%  /packages/mnt/jdocs-dc-re-21.4R3-S2.4
/dev/md4                290M       290M         0B      100%  /packages/mnt/jroute-qfx-ex-x86-32-21.4R3-S2.4
/dev/md5                 38M        38M         0B      100%  /packages/mnt/jcrypto-dc-re-21.4R3-S2.4
/dev/md6                 28M        28M         0B      100%  /packages/mnt/jcrypto-qfx-5-21.4R3-S2.4
/dev/md7                 12M        12M         0B      100%  /packages/mnt/jphone-home-21.4R3-S2.4
/dev/md8                 11M        11M         0B      100%  /packages/mnt/jsd-i386-21.4R3-S2.4-jet-1
/dev/md9                 17M        17M         0B      100%  /packages/mnt/jsdn-i386-21.4R3-S2.4
/dev/md10                71M        71M         0B      100%  /packages/mnt/jswitch-qfx-ex-x86-32-21.4R3-S2.4
/dev/md11                32M        32M         0B      100%  /packages/mnt/jweb-ex-21.4R3-S2.4
/dev/md12                19M        19M         0B      100%  /packages/mnt/py-base-i386-21.4R3-S2.4
/dev/md13                14M        14M         0B      100%  /packages/mnt/py-extensions-i386-21.4R3-S2.4
/dev/md14               1.1G        74K       999M        0%  /tmp
/dev/ad2s1e             504M       244K       463M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f             484M       2.1M       443M        0%  /var
/dev/ad1s1d             1.9G       598M       1.2G       33%  /var/tmp
/dev/ad1s1a             242M        20M       202M        9%  /var/log
/dev/ad1s1e             242M      10.0K       222M        0%  /altconfig
/dev/ad3s1             1006M       4.0K       926M        0%  /user
/dev/md15                98M       164K        90M        0%  /var/mfs/var/run
/var/jails/rest-api       484M       2.1M       443M      0%  /packages/mnt/jroute-qfx-ex-x86-32-21.4R3-S2.4/web-api/var
devfs                   1.0K       1.0K         0B      100%  /packages/mnt/jroute-qfx-ex-x86-32-21.4R3-S2.4/web-api/dev
/var/jail/etc           484M       2.1M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/etc
/var/jail/run           484M       2.1M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/run
/var/jail/tmp           484M       2.1M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/tmp
/var/tmp                1.9G       598M       1.2G       33%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/tmp/uploads
/var/jail/sess          484M       2.1M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/sess
/var/jail/log           484M       2.1M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/log
devfs                   1.0K       1.0K         0B      100%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/dev
/var/jail/jweb-app       484M       2.1M       443M       0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/jweb-app
/dev/md16                18M        18M         0B      100%  /packages/mnt/junos-openconfig-21.4R3-S2.4
/dev/md17               504M        95M       369M       20%  /var/rundb

fpc1:
--------------------------------------------------------------------------
Filesystem              Size       Used      Avail  Capacity   Mounted on
/dev/ad0s1a             1.6G       1.4G       140M       91%  /
devfs                   1.0K       1.0K         0B      100%  /dev
/dev/md0                121M       121M         0B      100%  /packages/mnt/jbase
/dev/md1                712M       712M         0B      100%  /packages/mnt/jkernel-qfx-ex-x86-32-21.4R3-S2.4
/dev/md2                138M       138M         0B      100%  /packages/mnt/jpfe-qfx-ex-x86-32-21.4R3-S2.4
/dev/md3                 13M        13M         0B      100%  /packages/mnt/jdocs-dc-re-21.4R3-S2.4
/dev/md4                290M       290M         0B      100%  /packages/mnt/jroute-qfx-ex-x86-32-21.4R3-S2.4
/dev/md5                 38M        38M         0B      100%  /packages/mnt/jcrypto-dc-re-21.4R3-S2.4
/dev/md6                 28M        28M         0B      100%  /packages/mnt/jcrypto-qfx-5-21.4R3-S2.4
/dev/md7                 12M        12M         0B      100%  /packages/mnt/jphone-home-21.4R3-S2.4
/dev/md8                 11M        11M         0B      100%  /packages/mnt/jsd-i386-21.4R3-S2.4-jet-1
/dev/md9                 17M        17M         0B      100%  /packages/mnt/jsdn-i386-21.4R3-S2.4
/dev/md10                71M        71M         0B      100%  /packages/mnt/jswitch-qfx-ex-x86-32-21.4R3-S2.4
/dev/md11                32M        32M         0B      100%  /packages/mnt/jweb-ex-21.4R3-S2.4
/dev/md12                19M        19M         0B      100%  /packages/mnt/py-base-i386-21.4R3-S2.4
/dev/md13                14M        14M         0B      100%  /packages/mnt/py-extensions-i386-21.4R3-S2.4
/dev/md14               1.1G        74K       999M        0%  /tmp
/dev/ad2s1e             504M       236K       463M        0%  /config
procfs                  4.0K       4.0K         0B      100%  /proc
/dev/ad1s1f             484M       2.0M       443M        0%  /var
/dev/ad1s1d             1.9G        26K       1.8G        0%  /var/tmp
/dev/ad1s1a             242M        11M       211M        5%  /var/log
/dev/ad1s1e             242M      10.0K       222M        0%  /altconfig
/dev/ad3s1             1006M       4.0K       926M        0%  /user
/dev/md15                98M       154K        90M        0%  /var/mfs/var/run
/var/jails/rest-api       484M       2.0M       443M      0%  /packages/mnt/jroute-qfx-ex-x86-32-21.4R3-S2.4/web-api/var
devfs                   1.0K       1.0K         0B      100%  /packages/mnt/jroute-qfx-ex-x86-32-21.4R3-S2.4/web-api/dev
/var/jail/etc           484M       2.0M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/etc
/var/jail/run           484M       2.0M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/run
/var/jail/tmp           484M       2.0M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/tmp
/var/tmp                1.9G        26K       1.8G        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/tmp/uploads
/var/jail/sess          484M       2.0M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/sess
/var/jail/log           484M       2.0M       443M        0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/log
devfs                   1.0K       1.0K         0B      100%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/dev
/var/jail/jweb-app       484M       2.0M       443M       0%  /packages/mnt/jweb-ex-21.4R3-S2.4/jail/var/jweb-app
/dev/md16                18M        18M         0B      100%  /packages/mnt/junos-openconfig-21.4R3-S2.4
/dev/md17               504M        95M       369M       20%  /var/rundb


> request system software add /var/tmp/jinstall-host-ex-4600-20.4R3-S6.3-signed.tgz force-host reboot no-validate force 


Checking pending install on fpc1

Checking pending install on fpc0
Pushing bundle /var/tmp/jinstall-host-ex-4600-20.4R3-S6.3-signed.tgz to fpc1
error: rcp for member 1, failed

    Can anybody help me with resolving this big issue?

    On a side note, I wonder why now, Juniper recommends newest software for devices which are mostly put in core of networks (having multiple 10Gs interfaces).



  • 2.  RE: High CPU and memory usage by fxpc with MACSEC_MII_1 Error on VC of EX4600 with 21.4R3-S2

    Posted 03-27-2023 04:28

    I resolved this issue myself. The problem was in the software and I managed to downgrade that to 20.4R3 and I don't see this high cpu usage.

    I managed to overcome the issue with upgrading by firstly doing upgrade to backup member, switching RE and then upgrading former master.

    Note to others (and myself) don't go for the newest software, even if Juniper wrongly recommends it.


    Original Message