Switching

 View Only
last person joined: 15 hours ago 

Ask questions and share experiences about EX and QFX portfolios and all switching solutions across your data center, campus, and branch locations.

  • 1.  Heap corruption at 0x0x0

    Posted 09-14-2021 18:48
    Hello, I have a VC with 2 EX-4300s
    Tonight at 03.02 all ports went to FLAP following a PANIC PANIC PANIC PANIC PANIC PANIC. I attach the log below, can you tell me what the problem is (JUNOS 20.4) ? Thanks

    Sep 13 23:58:04 sw1.it-vr.connectlife.it xntpd: kernel time sync enabled 6001
    Sep 14 00:15:07 sw1.it-vr.connectlife.it xntpd: kernel time sync enabled 2001
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex:
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: PANIC PANIC PANIC PANIC PANIC PANIC
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Heap corruption at 0x0x0
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Traceback:
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 00: sp = 0x06610f00, pc = 0x0184ea90
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 01: sp = 0x06610f18, pc = 0x01853d60
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 02: sp = 0x06610f60, pc = 0x01805834
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 03: sp = 0x06610f88, pc = 0x018060fc
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 04: sp = 0x06610fa8, pc = 0x01827974
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 05: sp = 0x06610fb0, pc = 0x0316715c
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 06: sp = 0x06610fb8, pc = 0x0300544c
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 07: sp = 0x06611088, pc = 0x031b5e84
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 08: sp = 0x066110a0, pc = 0x031b7530
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 09: sp = 0x066111d0, pc = 0x031b7a98
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 10: sp = 0x06611220, pc = 0x031b8e04
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 11: sp = 0x066112d8, pc = 0x031b93bc
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 12: sp = 0x06611348, pc = 0x02f9b600
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 13: sp = 0x06611468, pc = 0x02f9c248
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 14: sp = 0x066114b0, pc = 0x02f9c348
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 15: sp = 0x066114e0, pc = 0x0320d620
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 16: sp = 0x06611518, pc = 0x018125ec
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex:
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:7000) Foreign(0x80000011:59396)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:7000) Foreign(0x80000011:55391)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 1
    Sep 14 03:02:44 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(1)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:6157) Foreign(0x80000011:57272)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it mib2d[2053]: SNMP_TRAP_LINK_DOWN: ifIndex 670, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 152
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 3000/1000/3
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:6985) Foreign(0x80000011:49574)
    Sep 14 03:02:45 sw1.it-vr.connectlife.it mib2d[2053]: SNMP_TRAP_LINK_DOWN: ifIndex 671, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/1
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 153
    Sep 14 03:02:45 sw1.it-vr.connectlife.it mib2d[2053]: SNMP_TRAP_LINK_DOWN: ifIndex 672, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/2
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:6234) Foreign(0x80000011:64697)
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 1, vksid: 0, state: 1 reported a so_error 60
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 1, vksid: 0, state: 1 reported a so_error 60
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: peer_inputs: 6369: VKS0 closing connection peer class: 0, type: 17, index: 1, vksid: 0, state: 1, err 5
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 154
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 155
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 159
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae45: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:45 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperDisabled (entPhysicalIndex 41, entStateAdmin 3, entStateAlarm 0)
    Sep 14 03:02:45 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChanged
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/12 not found to be part of any LAG bundle
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/13 not found to be part of any LAG bundle
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/15 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/16 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4300-48T @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Sep 14 03:02:46 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4300-48T @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/44 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/45 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/47 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: Percentage memory available(15)less than threshold(20 %)- 2
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: pfe-manager (PID 21319) terminated by signal number 6. Core dumped!
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6111: peer class: 0, type: 17, index: 0, vksid: 1, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6111: peer class: 0, type: 17, index: 0, vksid: 1, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_inputs: 6410: Closing connection peer class: 0, type: 17, index: 0, vksid: 1, state: 1, err 5
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 0, vksid: 0, state: 1 reported a so_error 54
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 0, vksid: 0, state: 1 reported a so_error 54
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_inputs: 6369: VKS0 closing connection peer class: 0, type: 17, index: 0, vksid: 0, state: 1, err 5
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: Dump Command: /bin/sh (PID 4874) started
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: pfe-manager (PID 4875) started
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae13: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:52 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/13: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:52 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae13: Interface marked down due to lacp timeout on member ge-0/0/13
    Sep 14 03:02:53 sw1.it-vr.connectlife.it pfex: uncached heap start 0x2c33a000 0x78000000 size 0x8000000 status 0
    Sep 14 03:02:53 sw1.it-vr.connectlife.it pfex: dma desc heap start 0x3333a000 end 0x3433a000 status 0
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae12: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/12: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae12: Interface marked down due to lacp timeout on member ge-0/0/12
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae15: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/15: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae15: Interface marked down due to lacp timeout on member ge-0/0/15
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae46: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/46: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae46: Interface marked down due to lacp timeout on member ge-0/0/46
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae47: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/47: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae47: Interface marked down due to lacp timeout on member ge-0/0/47
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae16: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/16: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae16: Interface marked down due to lacp timeout on member ge-0/0/16
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae44: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/44: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae44: Interface marked down due to lacp timeout on member ge-0/0/44

    ------------------------------
    DANNY CAPPELLARI
    ------------------------------


  • 2.  RE: Heap corruption at 0x0x0

     
    Posted 09-14-2021 21:24
    Hi,

    What is the exact version for this VC: 20.4Rx-Sy ?

    Seems like heap memory got exhausted:

    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: Percentage memory available(15)less than threshold(20 %)- 2
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: pfe-manager (PID 21319) terminated by signal number 6. Core dumped!

    Do you see a pfex core dump ? If yes, can you upload here ?

    Check using command: show system core-dumps
    Original Message


  • 3.  RE: Heap corruption at 0x0x0

    Posted 09-15-2021 14:35
      |   view attached
    Hi, thanks for your reply.. the Junos version is 20.4R1.12 built 2020-12-20

    With show system core-dumps i'm obtain this:

    fpc0:
    --------------------------------------------------------------------------
    /var/crash/*core*: No such file or directory
    /var/tmp/*core*: No such file or directory
    /var/tmp/pics/*core*: No such file or directory
    /var/crash/kernel.*: No such file or directory
    /var/jails/rest-api/tmp/*core*: No such file or directory
    /tftpboot/corefiles/*core*: No such file or directory
    /jail/var/tmp/*core*: No such file or directory

    fpc1:
    --------------------------------------------------------------------------
    /var/crash/*core*: No such file or directory
    /var/tmp/*core*: No such file or directory
    /var/tmp/pics/*core*: No such file or directory
    /var/crash/kernel.*: No such file or directory
    /var/jails/rest-api/tmp/*core*: No such file or directory
    /tftpboot/corefiles/*core*: No such file or directory
    /jail/var/tmp/*core*: No such file or directory



    Cappellari Danny
    Responsabile tecnico | ConnectLife S.r.l


    Hi

    Questo documento è formato esclusivamente per il destinatario. Tutte le informazioni ivi contenute, compresi eventuali allegati, 
    sono da ritenere esclusivamente confidenziali e riservate secondo i termini del vigente D.Lgs. 196/2003 in materia di privacy e 
    del Regolamento europeo 679/2016 – GDPR-  e quindi ne è proibita l'utilizzazione ulteriore non autorizzata. Se avete ricevuto 
    per errore questo messaggio, Vi preghiamo cortesemente di contattare immediatamente il mittente e cancellare la e-mail.



    Original Message