Switching

IMPORTANT MODERATION NOTICE

This community is currently under full moderation, meaning  all posts will be reviewed before appearing in the community. Please expect a brief delay—there is no need to post multiple times. If your post is rejected, you'll receive an email outlining the reason(s). We've implemented full moderation to control spam. Thank you for your patience and participation.



Expand all | Collapse all

Heap corruption at 0x0x0

  • 1.  Heap corruption at 0x0x0

    Posted 11 days ago
    Hello, I have a VC with 2 EX-4300s
    Tonight at 03.02 all ports went to FLAP following a PANIC PANIC PANIC PANIC PANIC PANIC. I attach the log below, can you tell me what the problem is (JUNOS 20.4) ? Thanks

    Sep 13 23:58:04 sw1.it-vr.connectlife.it xntpd: kernel time sync enabled 6001
    Sep 14 00:15:07 sw1.it-vr.connectlife.it xntpd: kernel time sync enabled 2001
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex:
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: PANIC PANIC PANIC PANIC PANIC PANIC
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Heap corruption at 0x0x0
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Traceback:
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 00: sp = 0x06610f00, pc = 0x0184ea90
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 01: sp = 0x06610f18, pc = 0x01853d60
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 02: sp = 0x06610f60, pc = 0x01805834
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 03: sp = 0x06610f88, pc = 0x018060fc
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 04: sp = 0x06610fa8, pc = 0x01827974
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 05: sp = 0x06610fb0, pc = 0x0316715c
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 06: sp = 0x06610fb8, pc = 0x0300544c
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 07: sp = 0x06611088, pc = 0x031b5e84
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 08: sp = 0x066110a0, pc = 0x031b7530
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 09: sp = 0x066111d0, pc = 0x031b7a98
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 10: sp = 0x06611220, pc = 0x031b8e04
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 11: sp = 0x066112d8, pc = 0x031b93bc
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 12: sp = 0x06611348, pc = 0x02f9b600
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 13: sp = 0x06611468, pc = 0x02f9c248
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 14: sp = 0x066114b0, pc = 0x02f9c348
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 15: sp = 0x066114e0, pc = 0x0320d620
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex: Frame 16: sp = 0x06611518, pc = 0x018125ec
    Sep 14 03:02:39 sw1.it-vr.connectlife.it pfex:
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:7000) Foreign(0x80000011:59396)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:7000) Foreign(0x80000011:55391)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_IPC_CONNECTION_DROPPED: Dropped IPC connection for FPC 1
    Sep 14 03:02:44 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_IFDEV_DETACH_FPC: ifdev_detach_fpc(1)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:6157) Foreign(0x80000011:57272)
    Sep 14 03:02:44 sw1.it-vr.connectlife.it mib2d[2053]: SNMP_TRAP_LINK_DOWN: ifIndex 670, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/0
    Sep 14 03:02:44 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 152
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 3000/1000/3
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:6985) Foreign(0x80000011:49574)
    Sep 14 03:02:45 sw1.it-vr.connectlife.it mib2d[2053]: SNMP_TRAP_LINK_DOWN: ifIndex 671, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/1
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 153
    Sep 14 03:02:45 sw1.it-vr.connectlife.it mib2d[2053]: SNMP_TRAP_LINK_DOWN: ifIndex 672, ifAdminStatus up(1), ifOperStatus down(2), ifName ge-1/0/2
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep: Dropping socket connection due to keepalive timer expiration, idle/intvl/cnt: 1000/1000/5
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: tcp_timer_keep:Local(0x80000001:6234) Foreign(0x80000011:64697)
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 1, vksid: 0, state: 1 reported a so_error 60
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 1, vksid: 0, state: 1 reported a so_error 60
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: peer_inputs: 6369: VKS0 closing connection peer class: 0, type: 17, index: 1, vksid: 0, state: 1, err 5
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 154
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 155
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: stp_change_idx: stp_forw_state not found for stp_index 159
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae45: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:45 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP3: ENTITY trap generated: entStateOperDisabled (entPhysicalIndex 41, entStateAdmin 3, entStateAlarm 0)
    Sep 14 03:02:45 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP0: ENTITY trap generated: entConfigChanged
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/12 not found to be part of any LAG bundle
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/13 not found to be part of any LAG bundle
    Sep 14 03:02:45 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/15 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/16 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4300-48T @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Sep 14 03:02:46 sw1.it-vr.connectlife.it chassisd[1849]: CHASSISD_SNMP_TRAP7: SNMP trap generated: FRU removal (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: EX4300-48T @ 1/*/*, jnxFruType 3, jnxFruSlot 1)
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/44 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/45 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: if_pfe_ge_ifdpointchange_tlv: Child IFD ge-1/0/47 not found to be part of any LAG bundle
    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: Percentage memory available(15)less than threshold(20 %)- 2
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: pfe-manager (PID 21319) terminated by signal number 6. Core dumped!
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6111: peer class: 0, type: 17, index: 0, vksid: 1, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6111: peer class: 0, type: 17, index: 0, vksid: 1, state: 1 reported a sb_state 32 = SBS_CANTRCVMORE
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_inputs: 6410: Closing connection peer class: 0, type: 17, index: 0, vksid: 1, state: 1, err 5
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 0, vksid: 0, state: 1 reported a so_error 54
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_input_pending_internal: 6103: peer class: 0, type: 17, index: 0, vksid: 0, state: 1 reported a so_error 54
    Sep 14 03:02:52 sw1.it-vr.connectlife.it /kernel: peer_inputs: 6369: VKS0 closing connection peer class: 0, type: 17, index: 0, vksid: 0, state: 1, err 5
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: Dump Command: /bin/sh (PID 4874) started
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: pfe-manager (PID 4875) started
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae13: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:52 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/13: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:52 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae13: Interface marked down due to lacp timeout on member ge-0/0/13
    Sep 14 03:02:53 sw1.it-vr.connectlife.it pfex: uncached heap start 0x2c33a000 0x78000000 size 0x8000000 status 0
    Sep 14 03:02:53 sw1.it-vr.connectlife.it pfex: dma desc heap start 0x3333a000 end 0x3433a000 status 0
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae12: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/12: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae12: Interface marked down due to lacp timeout on member ge-0/0/12
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae15: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/15: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae15: Interface marked down due to lacp timeout on member ge-0/0/15
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae46: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/46: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae46: Interface marked down due to lacp timeout on member ge-0/0/46
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae47: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/47: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae47: Interface marked down due to lacp timeout on member ge-0/0/47
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae16: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/16: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae16: Interface marked down due to lacp timeout on member ge-0/0/16
    Sep 14 03:02:53 sw1.it-vr.connectlife.it /kernel: lag_bundlestate_ifd_change: bundle ae44: bundle IFD minimum bandwidth or minimum links not met, Bandwidth (Current : Required) 0 : 1000000000 Number of links (Current : Required) 0 : 1
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACPD_TIMEOUT: ge-0/0/44: lacp current while timer expired current Receive State: CURRENT
    Sep 14 03:02:53 sw1.it-vr.connectlife.it lacpd[5374]: LACP_INTF_DOWN: ae44: Interface marked down due to lacp timeout on member ge-0/0/44

    ------------------------------
    DANNY CAPPELLARI
    ------------------------------


  • 2.  RE: Heap corruption at 0x0x0

     
    Posted 11 days ago
    Hi,

    What is the exact version for this VC: 20.4Rx-Sy ?

    Seems like heap memory got exhausted:

    Sep 14 03:02:46 sw1.it-vr.connectlife.it /kernel: Percentage memory available(15)less than threshold(20 %)- 2
    Sep 14 03:02:52 sw1.it-vr.connectlife.it init: pfe-manager (PID 21319) terminated by signal number 6. Core dumped!

    Do you see a pfex core dump ? If yes, can you upload here ?

    Check using command: show system core-dumps


  • 3.  RE: Heap corruption at 0x0x0

    Posted 10 days ago
      |   view attached
    Hi, thanks for your reply.. the Junos version is 20.4R1.12 built 2020-12-20

    With show system core-dumps i'm obtain this:

    fpc0:
    --------------------------------------------------------------------------
    /var/crash/*core*: No such file or directory
    /var/tmp/*core*: No such file or directory
    /var/tmp/pics/*core*: No such file or directory
    /var/crash/kernel.*: No such file or directory
    /var/jails/rest-api/tmp/*core*: No such file or directory
    /tftpboot/corefiles/*core*: No such file or directory
    /jail/var/tmp/*core*: No such file or directory

    fpc1:
    --------------------------------------------------------------------------
    /var/crash/*core*: No such file or directory
    /var/tmp/*core*: No such file or directory
    /var/tmp/pics/*core*: No such file or directory
    /var/crash/kernel.*: No such file or directory
    /var/jails/rest-api/tmp/*core*: No such file or directory
    /tftpboot/corefiles/*core*: No such file or directory
    /jail/var/tmp/*core*: No such file or directory



    Cappellari Danny
    Responsabile tecnico | ConnectLife S.r.l


    Hi

    Questo documento è formato esclusivamente per il destinatario. Tutte le informazioni ivi contenute, compresi eventuali allegati, 
    sono da ritenere esclusivamente confidenziali e riservate secondo i termini del vigente D.Lgs. 196/2003 in materia di privacy e 
    del Regolamento europeo 679/2016 – GDPR-  e quindi ne è proibita l'utilizzazione ulteriore non autorizzata. Se avete ricevuto 
    per errore questo messaggio, Vi preghiamo cortesemente di contattare immediatamente il mittente e cancellare la e-mail.