Switching

 View Only
last person joined: 2 days ago 

Ask questions and share experiences about EX and QFX portfolios and all switching solutions across your data center, campus, and branch locations.
Expand all | Collapse all

EX3400 crashes after 4 days up

  • 1.  EX3400 crashes after 4 days up

    Posted 01-22-2020 00:24

    Hi,

    I have 4x Juniper EX3400 in a virtual chassis:

    • 1x EX3400 with POE
    • 3x EX3400 without POE

     

    Since mid december 2019 i have to reboot the chassis after 4 days uptime -> SSH & WebInterface is not reachable anymore. SNMP requests the chassis is still answering.

     

    The following things will be logged after the 4 days are reached:

     

    Jan 21 22:37:17  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:37:32  switch last message repeated 3 times
    Jan 21 22:37:37  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:37:37  switch jlaunchd: snmp (PID 17185) terminated by signal number 15!
    Jan 21 22:37:37  switch jlaunchd: Registered PID 50568(snmp): exec_command
    Jan 21 22:37:37  switch jlaunchd: snmp (PID 50568) started
    Jan 21 22:37:37  switch jlaunchd: Registered PID 50568(snmp): new process
    Jan 21 22:37:38  switch snmpd[50568]: SNMPD_TRAP_WARM_START: trap_generate_warm: SNMP trap: warm start
    Jan 21 22:37:42  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:37:47  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:37:52  switch inetd[14764]: accept (for ssh): Software caused connection abort
    Jan 21 22:37:52  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:37:57  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:37:57  switch kernel: rt_pfe_veto: Possible slowest client is mcsnoopd. States processed - 545192. States to be processed - 1
    Jan 21 22:37:57  switch kernel: rt_pfe_veto: Possible second slowest client is l2ald. States processed - 545192. States to be processed - 1
    Jan 21 22:38:02  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:38:32  switch last message repeated 6 times
    Jan 21 22:38:37  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 2, veto simulation: 0
    Jan 21 22:38:42  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 72, veto simulation: 0
    Jan 21 22:38:47  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:38:52  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:38:52  switch kernel: rt_pfe_veto: Possible slowest client is mcsnoopd. States processed - 545257. States to be processed - 1
    Jan 21 22:38:52  switch kernel: rt_pfe_veto: Possible second slowest client is l2ald. States processed - 545257. States to be processed - 1
    Jan 21 22:38:57  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:38:57  switch kernel: rt_pfe_veto: Possible slowest client is mcsnoopd. States processed - 545266. States to be processed - 1
    Jan 21 22:38:57  switch kernel: rt_pfe_veto: Possible second slowest client is l2ald. States processed - 545266. States to be processed - 1
    Jan 21 22:39:02  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:39:12  switch last message repeated 2 times
    Jan 21 22:39:13  switch sshd[50575]: sshd re-exec requires execution with an absolute path
    Jan 21 22:39:17  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:39:22  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    
    ### tried to start sshd manually ###
    Jan 21 22:39:26  switch sshd[50579]: error: Bind to port 22 on :: failed: Address already in use.
    Jan 21 22:39:26  switch sshd[50579]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use.
    Jan 21 22:39:26  switch sshd[50579]: fatal: Cannot bind any address.
    #############################
    
    Jan 21 22:39:27  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:39:32  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:39:47  switch last message repeated 3 times
    Jan 21 22:39:49  switch inetd[14764]: accept (for ssh): Software caused connection abort
    Jan 21 22:39:52  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:40:02  switch last message repeated 2 times
    Jan 21 22:40:12  switch last message repeated 2 times
    Jan 21 22:40:17  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 72, veto simulation: 0
    Jan 21 22:40:22  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:40:32  switch last message repeated 2 times
    Jan 21 22:41:52  switch last message repeated 16 times
    Jan 21 22:41:57  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 72, veto simulation: 0
    Jan 21 22:42:02  switch kernel: rt_pfe_veto: Memory over consumed. Op 1 err 12, rtsm_id 0:-1, msg type 90, veto simulation: 0
    Jan 21 22:42:32  switch last message repeated 6 times
    
    #####rebooted chassis (all-members)#######
    
    Jan 21 22:42:32  switch mgd[50581]: UI_REBOOT_EVENT: System rebooted by 'root'
    Jan 21 22:42:36  switch shutdown: reboot requested by root at Tue Jan 21 22:43:36 2020

     

    The chassis was fresh installed from USB stick on January 17 2020 but the problem still exists!

     

    I hope somebody can help me.

     

    If you need some other diagnostic logs or statistics please contact me.

     


    Thank you in advanced!

     


    #ex3400
    #crashes
    #4daysuptime


  • 2.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 00:46

    Hello,

    The syslog messages match a really old known bug

    https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR864551&smlogin=true , fixed in

     

    Release junos
    12.3R3 x
    12.3X50-D30 x
    13.1R2 x
    13.2R1 x
    13.2X50-D16 x
    12.3R2 x
    13.1R1 x
    13.2X50-D17 x
    13.1X50-D10 x
    15.1R1 x
    13.2X51-D10 x

     

    What JUNOS version are You running?

    Thanks

    Alex



  • 3.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 01:00

    Hi aarseniev,

     

    thank you for your reply!

     

    I forgot to write that. The following version is running on the switches -> 19.4R1.10

     

    The version was fresh installed from a USB stick.

     

     

    Thank you!



  • 4.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 01:21

    Hello,

    OK we can exclude that particular memory leak then.

    Now, a follow-up question - how many routes and how many MAC addresses are pushing to this box?

    EX3400 cannot possibly hold a full table, see the datasheet  https://www.juniper.net/assets/us/en/local/pdf/datasheets/1000581-en.pdf

    Pages 6-7:

    Layer 2 Features
    • Maximum MAC addresses per system: 32,000

    Layer 3 Features: IPv4
    • Maximum number of ARP entries: 16,000
    • Maximum number of IPv4 unicast routes in hardware: 14,000 prefixes; 36,000 host routes

    Layer 3 Features: IPv6
    • Maximum number of Neighbor Discovery entries: 8,000
    • Maximum number of IPv6 unicast routes in hardware: 3,500 prefixes; 18,000 host routes

    HTH

    Thx

    Alex



  • 5.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 01:43

    Hi,

     

    thank you for your answer.

     

    show ethernet-switching table brief

    gives following output:

     

    Ethernet switching table : 221 entries, 221 learned
    Routing instance : default-switch
    

    and

    show arp

    gives:

    Total entries: 7

    I will check the total entries in two days again.

    But i don't think that we have more than the maximum addresses.


    I don't have routes in the juniper switch. When I enter the following command:

    show route

    I get following output:

    inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
    Limit/Threshold: 32768/32768 destinations

     

    IPv6 is not used

     

     

    Thank you!



  • 6.  RE: EX3400 crashes after 4 days up

     
    Posted 01-22-2020 01:32

    Hi,

     

    Can you share the output of "show system virtual-memory no-forwarding"

     

     



  • 7.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 01:47

    Hi,

     

    this is the output of:

    show system virtual-memory no-forwarding
             Type InUse MemUse HighUse Requests  Size(s)
          entropy     3     1K       -     2976  16,4096
             UART     8    21K       -        8  16,1024,8192
              USB    22    36K       -       28  16,32,64,128,256,512,2048,4096,8192,16384
           USBdev    27     5K       -     8770  32,128,256
           DEVFS2   153     5K       -      515  16,32
           DEVFS3   729   183K       -      739  256
           DEVFS1   153    77K       -      162  512
            DEVFS    40     1K       -       41  16,64
           DEVFSP     0     0K       -   622403  32
      nullfs_hash     1    64K       -        1  65536
      nullfs_node    28     1K       -       66  32
     nullfs_mount     7     1K       -       10  16
        pfs_nodes    36     5K       -       36  128
      pfs_vncache    39     2K       -      203  32
      tmpfs mount     2     1K       -        2  128
       tmpfs name   340     7K       -    12137  16,32,64
    geom_flashmap     0     0K       -        7  32,256
             GEOM   694    54K       -     4607  16,32,64,128,256,512,1024,16384
        geom_uzip   189  2543K       -  1562697  32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
          CAM SIM     2     1K       -        2  128
      isofs_mount    34     5K       -       34  128
       isofs_node  8789  1099K       -     8789  128
          CAM XPT    14     2K       -      241  16,32,128,256,512,1024,2048,65536
          CAM DEV     2     4K       -        4  2048
          CAM CCB     0     0K       -    82478  2048
         CAM path     3     1K       -       15  16
             cdev     4     1K       -        4  128
         filedesc    47   113K       -      228  16,2048,4096,8192
            sigio     0     0K       -       56  32
         filecaps     0     0K       -        3  128
          kdtrace   394    67K       -    13066  64,256
             kenv    38     6K       -       38  16,32,64,256,4096
           kqueue   225    29K       -     6596  32,64,256,1024
             proc        procK       -     args  60
            hhook    16     2K       -       24  16,128
          ithread   104    14K       -      104  16,64,256
           KTRACE   100    13K       -      100  128
       CAM periph     4     1K       -       11  16,32,128
           linker   460 15647K       -     1242  16,32,64,128,256,1024,2048,4096,8192,16384
    CAM I/O Scheduler     1     1K       -        1 64
            lockf   155     9K       -    74939  32,64
       loginclass     3     1K       -       53  128
           devbuf  2063  6996K       -     4275  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
             temp  2341   145K       -  1026057  16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
        CAM queue     4     1K       -       10  16,256
           module   305    20K       -      305  64,128
         mtx_pool     2     8K       -        2  4096
         netstack     1     1K       -        1  16
              osd   651    11K       -   113799  16,32,64,128
         pmchooks     1     1K       -        1  64
             pgrp    10     1K       -      535  64
          session     9     2K       -      527  128
             proc     2     1K       -        2  512
          subproc   382   745K       -     6714  256,4096
             cred    74    19K       -     5522  256
           plimit    32     8K       -     7621  256
          uidinfo     3     1K       -        9  128
    CAM dev queue     2     1K       -        2  64
           sysctl     0     0K       -  4109531  16,32,64,128
        sysctloid  5972   192K       -     6023  16,32,64,128
        sysctltmp     0     0K       -    25703  32,64,256,512,1024
          scsi_da     0     0K       -       57  32
          tidhash     1     1K       -        1  1024
          callout     3  1124K       -        3
             umtx   552    52K       -      552  64,128
         p1003.1b     1     1K       -        1  16
              bus  1132   125K       -     2941  16,32,64,128,256,1024
              bus         busK       -       sc  46
          devstat    36    73K       -       36  16,4096
     eventhandler    87     5K       -       87  32,64,128
             kobj   115   230K       -      467  2048
              Per         PerK       -      cpu  1
             rman    66     8K       -       72  16,32,128
             sbuf     1     1K       -    49712  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
        taskqueue    30     2K       -       30  16,32,128
           Unitno    17     2K       -    29793  16,64
             vmem     3   112K       -       12  256,2048,4096,8192,16384,32768,65536
         ioctlops     0     0K       -    16687  512,2048
           select    95     6K       -       95  64
              iov     0     0K       - 36991075  16,32,64,128,256,512
              msg     4    25K       -        4  1024,4096,16384
              sem     4   101K       -        4  1024,4096
              shm    11   104K       -       84  4096,65536
              tty     3     2K       -        5  512
              pts     1     1K       -        3  128
         mbuf_tag     0     0K       -  4750498  16,32,64
            shmfd    91    43K       -      301  32,256,1024,4096
           soname    90    16K       -  1169711  16,32,64,128
              pcb   531   172K       -     5876  16,32,64,128,256,512,1024,2048,4096
         vfscache     4   153K       -        4  256,8192,16384
       cl_savebuf     0     0K       -       26  32
         vfs_hash     1    64K       -        1  65536
           vnodes     5     1K       -        9  32,128
            mount   668    20K       -      712  16,32,64,128,256
           statfs     0     0K       -     4127  512
      ddb_capture     1    64K       -        1  65536
      vnodemarker     0     0K       -    22422  512
       arc4random     1     1K       -        1  1024
    audit_evclass   187     3K       -      232  16
        ufs_mount     3    13K       -        3  256,4096,8192
        vm_pgdata     1   128K       -        1
          UMAHash     2     5K       -        6  256,512,1024,2048,4096
              iic     0     0K       -  3097168  16,32,64,128
         veriexec  7952   988K       -     7952  16,128
           busdma  2932  2237K       -     3032  32,64,128,1024,8192
           bounce  1614  6456K       -     1614
         nexusdev     2     1K       -        2  16
          md_disk    34    68K       -       34  2048
           ofwcpu     1     1K       -        1  32
         openfirm   156     3K       -      238  16,32,64
           OAMCTL     1     1K       -        5  32
           socket     2     1K       -        6  32
             mkey   561     9K       -   113209  16
          pfe_ipc     4     1K       -    57610  16,32,8192
          ifstate 21916   801K       -    64470  16,32,64,128,512,1024,2048
         ifstrace 21000  1899K       -    95196  64,128,256,512,1024
         itable16  1049   206K       -     1147  256
         itable32   548    35K       -      553  64
         itable64     1     1K       -        1  128
               lr     1     2K       -        1  2048
              pic    25    10K       -       43  32,512,1024
          pfestat   149   142K       -   101647  16,32,128,256,4096
           gencfg  7497   701K       -    36509  16,32,64,128,512,1024,2048,8192
              jsr     4     1K       -        8  16
              idl     1     8K       -     4388  16,32,128,512,1024,2048,8192
           rtsmsg     0     0K       -      921  4096,8192
           rtstat   114  3648K       -     1854  32768
    InterFaceBridgeDomain   223    35K       -      429
             mesh        meshK       -    group  201
       ifl_st_ctl   200     4K       -      351  16
     ifl_tlv_info   205     4K       -      360  16
            devrt    18   126K       -     1089  32,64,256,512,2048,8192,16384
             ddos     7   113K       -        7  32,16384,32768,65536
    vpls_lc_instance     1     2K       -        1 2048
         ifa_list     6     1K       -       10  16
    lag_list_node    10     1K       -       17  32
    gresstatevarlog     1    96K       -        1
           crypto     1     1K       -        1  512
          JFE job     1     1K       -        1  512
    sysctl policing   730    23K       -      730 32
      JFE PROFILE    45    51K       -       45  16,64,512,2048
        schedhist     1   448K       -        1
              KPF     1     1K       -        1  16
    JFE statistics     7  1353K       -        7 8192,16384
            jlock    76   152K       -      489  2048
              BPF     0     0K       -       10  64,512,1024,4096
           gtable    24     1K       -       45  32
              DDI     5     1K       -        5
            jlist     1     1K       -        1  32
      ifl_idx_mgr     1     1K       -        1  64
        iflogical   649   109K       -     1145  16,32,512,1024,4096
         iffamily   504    80K       -     1002  16,32,128,512,1024
        rtnexthop   472    63K       -    11216  16,32,64,128,2048
          metrics     8     1K       -       14
              STP   610   129K       -     1069  16,128,512
            rnode   507    10K       -     1585  16,32
           rcache     4     8K       -        4  2048
         syncache     1     8K       -        1  8192
         tlv_stat     0     0K       -    14767  16384,32768,65536
    nhack_info_entries     3     1K       -        3 16,32
         ifdevice    35     7K       -       48  16,64,2048
           ifstat  1839  1981K       -    12733  32,64,512,1024,4096,8192
     rts_osd_slot    33     1K       -       78  16
    Bridge Domain    20     6K       -       38  16,512
        hmac_drbg     0     0K       -        1  64
           rtdata   242    80K       -     1779  32,512,8192
             ipfw   212    55K       -      846  16,32,64,128,256,512,1024,2048,4096
          ifmaddr   679    18K       -     1166  16,32
           rtable   589    57K       -     2349  16,32,512
           sysctl     0     0K       -       68  256,512,1024
           ifaddr    34     2K       -       42  32,64
             bmeb    13   341K       -       14  16,32,64,256,512,1024,2048,16384
      Export Host     2     2K       -        2  1024
      inpcbpolicy   158     3K       -     2245  16
      ipsecpolicy   316    79K       -     4490  256
          solaris 177144  7907K       -   178667 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
              fbt 83139  5325K       -    83583  64
             KATS     0     0K       -       13  16,32,64,128
              SDT    34     1K       -       34  16,32
    ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
    UMA Kegs:        192,        0,     130,     17,      130
    UMA Zones:       304,        0,     131,      1,      131
    UMA Slabs:        56,        0,    3952,      8,     4346
    UMA Hash:        128,        0,      14,     17,       16
    4 Bucket:         16,        0,      53,   1459,    52403
    6 Bucket:         24,        0,       4,    668,      114
    8 Bucket:         32,        0,     137,    745,    15148
    12 Bucket:        48,        0,      14,    574,      199
    16 Bucket:        64,        0,      38,    781,     9794
    32 Bucket:       128,        0,      57,    470,    29396
    64 Bucket:       256,        0,      80,    115,      933
    128 Bucket:      512,        0,     122,     94,     2699
    256 Bucket:     1024,        0,     101,    147,     9149
    vmem btag:        28,        0,   21812,    220,    49639
    VM OBJECT:       168,        0,   15647,    529,   133270
    RADIX NODE:       48,   482748,   63087,    333,   434325
    MAP:             152,        0,       3,     75,        3
    KMAP ENTRY:       80,        0,       5,    145,        5
    MAP ENTRY:        80,        0,   11597,   2803,   454044
    VMSPACE:         248,        0,      60,    132,     6245
    fakepg:           80,        0,   16396,    254,    49303
    mt_zone:         272,        0,     350,     28,      350
    16:               16,        0,  144049,    851,  6159027
    32:               32,        0,   30368,    754,  9343859
    64:               64,        0,  100019,    718, 38155058
    128:             128,        0,   60004,   4011,   470862
    256:             256,        0,    2238,     87,    49594
    512:             512,        0,    1213,     99,    48488
    1024:           1024,        0,     584,     16,    50626
    2048:           2048,        0,     538,    192,   166322
    4096:           4096,        0,    1563,      2,    18162
    8192:           8192,        0,     299,     20,    58590
    16384:         16384,        0,     252,      4,    18348
    32768:         32768,        0,     152,      8,     3031
    65536:         65536,        0,      11,      5,     5291
    64 pcpu:           8,        0,      44,    212,       44
    ptr pcpu:          4,        0,       0,      0,        0
    kenv:            386,        0,       5,     55,      141
    SLEEPQUEUE:       44,        0,     277,    290,      277
    dma buffer 32:    32,        0,       0,      0,        0
    dma buffer 64:    64,        0,       0,      0,        0
    dma buffer 128:   128,        0,       0,      0,        0
    dma buffer 256:   256,        0,       0,      0,        0
    dma buffer 512:   512,        0,       0,      0,        0
    dma buffer 1024:  1024,        0,       0,      0,        0
    dma buffer 2048:  2048,        0,       0,      0,        0
    dma buffer 4096:  4096,        0,       0,      0,        0
    dma coherent 32:    32,        0,      11,    367,       19
    dma coherent 64:    64,        0,       0,      0,        0
    dma coherent 128:   128,        0,     129,    150,      129
    dma coherent 256:   256,        0,     287,     28,      315
    dma coherent 512:   512,        0,       6,     26,       15
    dma coherent 1024:  1024,        0,      16,     12,       28
    dma coherent 2048:  2048,        0,       6,      4,       18
    dma coherent 4096:  4096,        0,       5,      0,        5
    Files:            64,        0,    1040,    787,  2061520
    filedesc0:      1056,        0,     173,     25,     6505
    rl_entry:         32,        0,     140,    490,      140
    TURNSTILE:        72,        0,     277,    101,      277
    umtx pi:          52,        0,       0,      0,        0
    umtx_shm:         52,        0,       0,      0,        0
    Junos 48:         48,        0,    1704,    396,    10658
    Junos 80:         80,        0,   10581,    319,    47568
    Junos 96:         96,        0,    3241,    665,    17932
    Junos 120:       120,        0,    2117,   1909,    23631
    Junos 160:       160,        0,    1477,   1573,     7469
    Junos 176:       176,        0,     463,     87,     1592
    Junos 208:       208,        0,     341,    210,     2208
    Junos 232:       232,        0,      26,    110,     1438
    Junos 240:       240,        0,       4,    124,       49
    Junos 296:       296,        0,     237,     88,    10840
    jsocket:          24,        0,     559,    785,   113206
    MAC labels:       20,        0,   11500,    418,    35423
    PROC:            920,        0,     172,     36,     6504
    THREAD:          888,        0,     256,     20,      264
    cpuset:           40,        0,     153,    453,      154
    audit_record:   1160,        0,       0,      0,        0
    mbuf_packet:     256,    81930,     256,    756,  1279471
    mbuf:            256,    81930,      25,   1498,  9061886
    mbuf_cluster:   2048,    12800,    1019,    497,   441923
    mbuf_jumbo_page:  4096,     6400,       6,     14,    86553
    mbuf_jumbo_9k:  9216,     1896,       0,      0,        0
    mbuf_jumbo_16k: 16384,     1066,       0,      0,        0
    ttyoutq:         256,        0,      39,     36,       95
    g_bio:           288,        0,       0,    210,   379419
    ttyinq:          152,        0,      75,     55,      180
    cryptop:          64,        0,       0,      0,        0
    cryptodesc:       56,        0,       0,      0,        0
    GENCFG:           68,        0,    1801,    205,     5331
    VNODE:           288,        0,   11203,     53,    23241
    VNODEPOLL:        64,        0,       0,      0,        0
    BUF TRIE:         48,        0,    1022,   6034,    49097
    NAMEI:          1024,        0,       0,     40,  5242829
    rentr:            12,        0,       0,      0,        0
    S VFS Cache:      72,        0,   10951,    193,    26272
    STS VFS Cache:   116,        0,       0,      0,        0
    L VFS Cache:     292,        0,     491,     42,      501
    LTS VFS Cache:   336,        0,       0,      0,        0
    AIO:             128,        0,       0,      0,        0
    AIOP:             16,        0,       0,      0,        0
    AIOCB:           440,        0,       0,      0,        0
    AIOL:             64,        0,       0,      0,        0
    AIOLIO:          172,        0,       0,      0,        0
    pipe:            472,        0,      84,     68,     3840
    Mountpoints:     688,        0,      51,     14,       54
    procdesc:         80,        0,       0,      0,        0
    ksiginfo:         80,        0,     186,    864,    46387
    itimer:          256,        0,       0,      0,        0
    KNOTE:            72,        0,     255,    417,  3103308
    socket:          552,    62265,     559,     43,   113206
    unpcb:           176,    62282,     248,    192,   109030
    ipq:              64,      126,       0,      0,        0
    udp_inpcb:       280,    62272,      40,     58,     1309
    tcp_inpcb:       280,    62272,     113,     97,      926
    tcpcb:          1080,    62265,     113,     31,      926
    sackhole:         24,        0,       0,      0,        0
    tcptw:            60,     8241,       0,    536,       77
    syncache:        128,    15376,       0,    279,      172
    tcpreass:         20,      808,       0,      0,        0
    ripcb:           280,    62272,       6,     64,       11
    selfd:            32,        0,     531,    855, 770913944
    SWAPMETA:        280,   244202,       0,      0,        0
    FFS inode:       112,        0,    1896,    264,     2080
    FFS1 dinode:     128,        0,       0,      0,        0
    FFS2 dinode:     256,        0,    1896,     69,     2080
    TMPFS dirent:     36,        0,      11,    549,       50
    TMPFS node:      192,        0,      12,    114,       51
    TMPFS dirent:     36,        0,     325,    235,    11935
    TMPFS node:      192,        0,     331,    110,    11936
    dtrace_state_cache:   224,        0,       0,      0,        0
    taskq_zone:       24,        0,       0,      0,        0
    fuse_ticket:     136,        0,       0,      0,        0
    288656356 cpu context switches
    148727583 device interrupts
      8027836 software interrupts
      6924884 traps
    1392515875 system calls
          114 kernel threads created
         4859  fork() calls
         1531 vfork() calls
            0 rfork() calls
            0 swap pager pageins
            0 swap pager pages paged in
            0 swap pager pageouts
            0 swap pager pages paged out
        50216 vnode pager pageins
        50231 vnode pager pages paged in
          129 vnode pager pageouts
         1200 vnode pager pages paged out
            0 page daemon wakeups
      3086766 pages examined by the page daemon
            0 pages reactivated
       341051 copy-on-write faults
          219 copy-on-write optimized faults
       653450 zero fill pages zeroed
        13714 zero fill pages prezeroed
         1402 intransit blocking page faults
      1190821 total VM faults taken
        49723 page faults requiring I/O
            0 pages affected by kernel thread creation
      5184592 pages affected by  fork()
      2173073 pages affected by vfork()
            0 pages affected by rfork()
      1347314 pages freed
            0 pages freed by daemon
       599046 pages freed by exiting process
        30732 pages active
       286230 pages inactive
            0 pages in VM cache
        52034 pages wired down
       119386 pages free
         4096 bytes per page
     13219872 total name lookups
               cache hits (83% pos + 16% neg) system 0% per-directory
               deletions 0%, falsehits 0%, toolong 0%
    interrupt                          total       rate
    irq0: ipi                             45          0
    irq2: ipi                       25191018         86
    irq3: ipi                             38          0
    irq6: ipi                            583          0
    irq29: mp_tmr0                  28719224         68
    irq117: ehci0                     453094         10
    irq123: uart0                       1027          0
    irq127: iichb0                    599100         14
    irq128: iichb1                    252154          6
    irq133: bcmrng0                 94002010         88
    Total                          149218293       3453
    vm.kmem_map_free: 125878272
    


  • 8.  RE: EX3400 crashes after 4 days up

     
    Posted 01-22-2020 02:11

    Thanks for the sharing the output. At present, the memory looks normal (125 MB free)

    Since you must have very recently rebooted the device, the memory usage currently looks fine.

    I assume you currently do not see those "memory over consumed" log messages?

     

    If yes, when the problem re-surfaces again, we need to collect the same CMD output for "show system virtaul-memory no-forwarding" to compare (with current one) to see which S/w component is leaking the memory block.

     

    and JFYI.. there are known bug behaviour to this however we might addtional memory statistics to narrow down the culprit process/task which is using excess amount of memory.

     

    Suggest to you monitor and collect output for "show system virtaul-memory no-forwarding" when the problem arise again.

    Also, I would highy recommend to log a case with JTAC for faster root cause analysis.

     

     

     

     

     

     

     

     

     



  • 9.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 02:24

    Hi,

     

    thank you very much for your answers. I will try to downgrade as soon as possible.

     

    In 2-3 days the switch will have problems again, then I will post the output again.

     

    I assume you currently do not see those "memory over consumed" log messages?

    Yes, i don't see these messages at the moment. But in 2-3 days Smiley Wink

     

     

    Thank you!



  • 10.  RE: EX3400 crashes after 4 days up

     
    Posted 01-22-2020 04:44

    I am in agreement with @

     



  • 11.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 06:43

    Hi,

     

    i wrote that in my first post:

    I have 4x Juniper EX3400 in a virtual chassis:
    
    1x EX3400 with POE
    3x EX3400 without POE

     

    I have no open cases at the moment. The service contracts are already expired.

     

    Thank you!



  • 12.  RE: EX3400 crashes after 4 days up

     
    Posted 01-22-2020 07:14

    Sorry I missed that.  I would still say best approach would be to go to 18.2R3-S2, and in a week or less report back your findings.

     

    Good luck.



  • 13.  RE: EX3400 crashes after 4 days up

    Posted 01-23-2020 00:16

    Hi,

    i downloaded this image:

    junos-install-media-usb-ex-arm-32-18.2R3.4.img.gz

    Is this the right version?

     

     

    Thank you



  • 14.  RE: EX3400 crashes after 4 days up

    Posted 01-23-2020 00:26

    That is not the right release. You have found the initial Junos 18.2R3 release. You have to select "Junos SR" under OS on the download page to see the -S releases.

     

    The image you have downloaded is for usb reinstallation. You can also just do a ordinary downgrade with an install package.

    That image name should be named junos-arm-32-18.2R3-S2.9.tgz.



  • 15.  RE: EX3400 crashes after 4 days up

    Posted 01-23-2020 00:30

    Hi,

     

    thanks for the quick answer.

     

    I found the image. I will inform you when it is installed

     

     

    Thank you!



  • 16.  RE: EX3400 crashes after 4 days up

     
    Posted 01-23-2020 00:32

    Nope, that's not correct one as per JTAC Recommended.. You need Service Release also known as JUNOS-SR.

    For instance: if you're down grading via a USB, the file should be: junos-install-media-usb-ex-arm-32-18.2R3-S2.9.img.gz

     

    Select JUNOS SR at the landing page of https://support.juniper.net/support/downloads/

    enter EX3400 and drop down to JUNOS SR..

     

     



  • 17.  RE: EX3400 crashes after 4 days up

    Posted 01-22-2020 02:01

    Do you have any specific requirements to use Junos 19.4R1? The recommended production version is currently 18.2R3-S2 as per https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476 .

     

    I will suggest at downgrade to this version if you are just using switchports, vlans and PoE as major stability improvements has been implemented in this release. Some of them possibly didn't make it into 19.4R1.

     

     



  • 18.  RE: EX3400 crashes after 4 days up
    Best Answer

     
    Posted 01-23-2020 07:57

    This problem will initially be fixed in 18.2R3-S3 and 19.1R3 (and then in other branches). In the interim as a workaround you should stop polling the OID "jnxCosIfqTailDropPkts", which is what's causing the memory leak.

     

    Downgrading to 18.2R3-S2 will not help.

     

    -Alper



  • 19.  RE: EX3400 crashes after 4 days up

     
    Posted 01-29-2020 07:34

    Further investigation shows polling jnxCosQstatTailDropPkts causes the same memory leak. It would be cautious to stop polling anything under jnxCos for the time being until the full scope of the issue is understood.