Junos OS

Expand all | Collapse all

EX4300 Switches reset itself to version 13.2X51-D35.3. Layer 2 networking fails

Jump to Best Answer
  • 1.  EX4300 Switches reset itself to version 13.2X51-D35.3. Layer 2 networking fails

    Posted 05-06-2019 08:45

    Hi all,

     

    I had very strange things happen this weekend to our EdgeRouter and Juniper EX4300 switches. Our EdgeRouter seems to have completely lost networking capabilities and will refuse to factory reset even though the unit powers on and the lights show activity.

     

    Our Juniper switches (configured in virtual chassis) which is connected to the EdgeRouter and connected to the same UPS are also having issues. Both switches power on and show activity on the ports, but networking does not work. I am not even able to hit the juniper switch address when plugged into a port that is configured to allow access to it. I am able to access the console port though. My configuration is still there and everything looks correct. I am able to "show interface" and see the activity on a port with my computer plugged in, but when I do "show arp" these are the only entries I get even though I have multiple things plugged into my switch and showing activity:

     

     

    show arp
    MAC Address       Address         Name                      Interface               Flags
    02:00:00:00:00:0a 128.0.0.5       128.0.0.5                 bme0.0                  permanent
    02:00:00:00:00:0a 128.0.0.6       128.0.0.6                 bme0.0                  permanent
    02:00:00:00:00:0a 128.0.0.16      128.0.0.16                bme0.0                  permanent
    Total entries: 3

    When this happened, I hard restarted both switches, upon restart, it looks like the switches reset itself to version 13.2X51-D35.3. The switch actually shows a commit made by root that was made when the siwtch was restarted: 

    show system commit
    0   2019-05-04 09:19:22 EDT by root via other

    And here is the one difference from my last commit:

    how system rollback 0 compare 1
    - version 18.1R1.9;
    + version 13.2X51-D35.3;

     

    I'm not sure where to go from here. Has anyone heard of this happening before? Again, my configuration seems to be OK and the ports on the switches show activity, but my version has been set back and L2 networking seems completely broken. Any help is greatly appreciated.

     

     

     



  • 2.  RE: EX4300 Switches reset itself to version 13.2X51-D35.3. Layer 2 networking fails
    Best Answer

     
    Posted 05-06-2019 09:15

    Your switch has booted from the backup partition, which is running JUNOS 13. Your v18 configuration is incompatible with v13 so it is not getting loaded at boot. You can either try booting from the v18 partition with

    request system reboot slice alternate media internal

    or reinstall v18 over the current v13 and reboot. You should grab a copy of your configuration in case something goes wrong and you need to reinstall from scratch.

     

    In either case remember to snapshot to your backup partition so they are running the same code.

    request system snapshot slice alternate


  • 3.  RE: EX4300 Switches reset itself to version 13.2X51-D35.3. Layer 2 networking fails

    Posted 05-08-2019 07:19

    Thanks for your response smicker. Unfortuntely, the switches were not able to boot to the internal partition and continue to boot from the backup. Here is what it looks like when loading:

     

    U-Boot 2011.06 (Jan 10 2014 - 02:08:51)
    
    Unicore software on multiprocessor system!!
    To enable mutlticore build define CONFIG_MP
    CPU0:  P20BJE, Version: 2.0, (0x82190120)
    Core:  E500MC, Version: 3.2, (0x80230032)
    Clock Configuration:
           CPU0:1500 MHz, CPU1:1500 MHz,
           CCB:600  MHz,
           DDR:600  MHz (1200 MT/s data rate) (Asynchronous), LBC:75   MHz
           FMAN1: 500 MHz
    L1:    D-cache 32 kB enabled
           I-cache 32 kB enabled
    Reset Configuration Word (RCW):
           00000000: 4c580000 00000000 1e140000 00440000
           00000010: 648e20c1 ffc02000 ce000000 41000000
           00000020: 00000000 00000000 00000000 f05b4101
           00000030: 00000000 00000000 00000000 00000000
    Board: EX4300-24T 3.18
    EPLD:  Version 10.0 (0x0a)
    I2C:   ready
    DRAM:  Initializing
    Detected UDIMM TS7K8EDSQ-0000
        DDR: 2 GiB (DDR3, 64-bit, CL=8, ECC on)
    FLASH bank: 0
    Flash: 8 MiB
    L2:    128 KB enabled
    Corenet Platform Cache: 1024 KB enabled
    SERDES: bank 2 disabled
    SERDES: timeout resetting bank 2
    
    Firmware Version: 01.00.00
    PCIe2: Root Complex, x2, regs @ 0xfe201000
    PCIe2: Bus 00 - 01
    In:    serial
    Out:   serial
    Err:   serial
    USB:
    Register 10011 NbrPorts 1
    USB EHCI 1.00
    Register 10011 NbrPorts 1
    USB EHCI 1.00
    scanning bus for devices...
    Root Hub 0: 2 USB Device(s) found
    Root Hub 1: 1 USB Device(s) found
           scanning bus for storage devices... 1 Storage Device(s) found
    Net:   Initializing Fman
    Uploaded FMAN microcode version 101.8
    FM1@DTSEC1 connected to Broadcom BCM54616S
    FM1@DTSEC1 [PRIME]
    Consoles: U-Boot console
    
    FreeBSD/PowerPC U-Boot bootstrap loader, Revision 2.4
    (hbuilder@svl-junos-pool103.juniper.net, Fri Jan 10 02:08:46 PST 2014)
    Memory: 2048MB
    bootsequencing is enabled
    bootsuccess is set
    new boot device = disk0s1:
    Loading /boot/defaults/loader.conf
    /kernel data=0xb26ef0+0xd5090 syms=[0x4+0x96090+0x4+0xda8ac]
    
    
    Hit [Enter] to boot immediately, or space bar for command prompt.
    Booting [/kernel]...
    Kernel entry at 0x800000c0 ...
    GDB: no debug ports present
    KDB: debugger backends: ddb
    KDB: current backend: ddb
    Copyright (c) 1996-2015, Juniper Networks, Inc.
    All rights reserved.
    Copyright (c) 1992-2006 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    JUNOS 13.2X51-D35.3 #0: 2015-04-09 20:48:22 UTC
        builder@svl-junos-pool88.juniper.net:/volume/build/junos/13.2/service/13.2X51-D35.3/obj-powerpc/junos/bsd/kernels/JUNIPER-EX-4300/kernel
    can't re-use a leaf (all_slot_serialid)!
    Timecounter "decrementer" frequency 18750000 Hz quality 0
    cpu0: Freescale e500mc core revision 3.2
    cpu0: HID0 c0000080<EMCP,EN_L2MMU_MHD,EN_MAS7_UPDATE>
    real memory  = 1992294400 (1900 MB)
    avail memory = 1949667328 (1859 MB)
    Security policy loaded: Junos MAC/veriexec (mac_veriexec)
    Security policy loaded: JUNOS MAC/pcap (mac_pcap)
    Security policy loaded: JUNOS MAC/runasnonroot (mac_runasnonroot)
    MAC/veriexec fingerprint module loaded: SHA256
    MAC/veriexec fingerprint module loaded: SHA1
    ETHERNET SOCKET BRIDGE initialising
    Initializing EXSERIES properties ...
    nexus0: <Powerpc Nexus device>
    ocpbus0: <on-chip peripheral bus> on nexus0
    openpic0: <OpenPIC in on-chip peripheral bus> iomem 0xfe040000-0xfe06ffff on ocpbus0
    lbc0: <Freescale 8533 Local Bus Controller> iomem 0xfe124000-0xfe12410f,0xff000000-0xffffffff irq 41 on ocpbus0
    cb fpga: un-supported RE/board:148
    cfi0: <AMD/Fujitsu - 8MB> iomem 0xff800000-0xffffffff on lbc0
    cb fpga: un-supported RE/board:148
    syspld0 iomem 0xff000000-0xff00ffff on lbc0
    i2c0: <MPC85XX OnChip i2c Controller> iomem 0xfe118000-0xfe118014 irq 54 on ocpbus0
    ds1672 rtc0: <DS1672 RTC> on i2c0
    i2c1: <MPC85XX OnChip i2c Controller> iomem 0xfe118100-0xfe118114 irq 54 on ocpbus0
    poe0: <POE> on i2c1
    poe_attach: re-types method failed
    device_attach: poe0 attach returned 19
    uart0: <16550 or compatible> iomem 0xfe11c500-0xfe11c50f irq 52 on ocpbus0
    uart0: console (9600,n,8,1)
    uart1: <16550 or compatible> iomem 0xfe11c600-0xfe11c60f irq 52 on ocpbus0
    pcib0: <Freescale MPCP20BJE PCI Express host controller> iomem 0xfe201000-0xfe201fff,0xc0000000-0xc7ffffff on ocpbus0
    pci0: <PCI bus> on pcib0
    pcib1: <PCI-PCI bridge> at device 0.0 on pci0
    pci1: <PCI bus> on pcib1
    besw0: <Broadcom BCM56649J GigE Switch> mem 0xc0000000-0xc003ffff irq 57 at device 0.0 on pci1
    BCM device 0xb649 has been attachedbman0 iomem 0xfe31a000-0xfe31afff,0xfd800000-0xfd803fff,0xfd804000-0xfd804fff irq 32,121,123,125,127,129,131,133,135,137,139
    on ocpbus0
    bman0: Hardware version: 1.0.
    qman0 iomem 0xfe318000-0xfe318fff,0xfd808000-0xfd80bfff,0xfd80c000-0xfd80cfff irq 32,120,122,124,126,128,130,132,134,136,138 on ocpbus0
    qman0: Hardware version: 1.2.
    fman0 iomem 0xfe400000-0xfe4fffff irq 112,32 on ocpbus0
    fman0: Hardware version: 3.0.
    dtsec0 on fman0
    dtsec0: Configured for independent mode.
    dtsec0: hardware MAC address 80:ac:ac:5d:7f:c0
    miibus0: <MII bus> on dtsec0
    brgphy0: <BCM54616S 10/100/1000baseTX PHY> on miibus0
    brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
    ehci0: <Frescale Integrated USB 2.0 controller> iomem 0xfe210000-0xfe210fff irq 60 on ocpbus0
    usb0: EHCI version 1.0
    usb0 on ehci0
    usb0: USB revision 2.0
    uhub0: Freescale EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
    uhub0: 1 port with 1 removable, self powered
    umass0: Transcend TS2GUFM-H, rev 2.00/11.00, addr 2
    ehci1: <Frescale Integrated USB 2.0 controller> iomem 0xfe211000-0xfe211fff irq 61 on ocpbus0
    usb1: EHCI version 1.0
    usb1 on ehci1
    usb1: USB revision 2.0
    uhub1: Freescale EHCI root hub, class 9/0, rev 2.00/1.00, addr 1
    uhub1: 1 port with 1 removable, self powered
    Initializing product: 143 ..
    Loading the NETPFE fc module
    ###PCB Group initialized for udppcbgroup
    ###PCB Group initialized for tcppcbgroup
    da0 at umass-sim0 bus 0 target 0 lun 0
    da0: <TS TS2GUFM-H 1100> Fixed Direct Access SCSI-4 device
    da0: 40.000MB/s transfers
    da0: 1968MB (4030464 512 byte sectors: 255H 63S/T 250C)
    Kernel thread "wkupdaemon" (pid 46) exited prematurely.
    Trying to mount root from ufs:/dev/da0s1a
    Attaching /packages/junos via /dev/mdctl...
    Mounted junos-ex package on /dev/md0...
    Verified manifest signed by PackageProduction_13_2_0
    Verified jboot signed by PackageProduction_13_2_0
    Verified junos-ex-13.2X51-D35.3 signed by PackageProduction_13_2_0
    Mounted fips-mode-powerpc package on /dev/md2...
    Verified manifest signed by PackageProduction_13_2_0
    Verified fips-mode-powerpc-13.2X51-D35.3 signed by PackageProduction_13_2_0
    Mounted jdocs-ex package on /dev/md4...
    Verified manifest signed by PackageProduction_13_2_0
    Verified jdocs-ex-13.2X51-D35.3 signed by PackageProduction_13_2_0
    Mounted junos-ex-4300 package on /dev/md6...
    Verified manifest signed by PackageProduction_13_2_0
    Verified junos-ex-4300-13.2X51-D35.3 signed by PackageProduction_13_2_0
    Mounted jweb-ex package on /dev/md8...
    Verified manifest signed by PackageProduction_13_2_0
    Verified jweb-ex-13.2X51-D35.3 signed by PackageProduction_13_2_0
    Executing /packages/mnt/jweb-ex-13.2X51-D35.3/mount.post..
    Mounted py-base-powerpc package on /dev/md10...
    Verified manifest signed by PackageProduction_13_2_0
    Verified py-base-powerpc-13.2X51-D35.3 signed by PackageProduction_13_2_0
    Automatic reboot in progress...
    ** /dev/da0s1a
    FILE SYSTEM CLEAN; SKIPPING CHECKS
    clean, 71503 free (23 frags, 8935 blocks, 0.0% fragmentation)
    ** /dev/da0s3e
    FILE SYSTEM CLEAN; SKIPPING CHECKS
    clean, 78584 free (320 frags, 9783 blocks, 0.4% fragmentation)
    Computing slice and partition sizes for /dev/da0 ...
    savecore: Router rebooting after a normal shutdown...
    No dump exists
    ** /dev/da0s3d
    FILE SYSTEM CLEAN; SKIPPING CHECKS
    clean, 411532 free (28 frags, 51438 blocks, 0.0% fragmentation)
    ** /dev/da0s4d
    FILE SYSTEM CLEAN; SKIPPING CHECKS
    clean, 59052 free (36 frags, 7377 blocks, 0.1% fragmentation)
    rm: /var/etc/pam.conf: Operation not permitted
    mkdir: /mfs/var/sdb: No such file or directory
    mkdir: /mfs/var/sdb: No such file or directory
    Creating initial configuration...mgd: Running FIPS Self-tests
    veriexec: no fingerprint for file='/sbin/kats/cannot-exec' fsid=69 fileid=51404 uid=0 pid=401
    mgd: FIPS Self-tests Passed
    mgd: commit complete
    Setting initial options:  debugger_on_panic=NO debugger_on_break=NO.
    Starting optional daemons: .
    Doing initial network setup:.
    Initial interface configuration:
    additional daemons:.
    Additional routing options:kern.module_pathLoading the CHMIC module
    Loading the EX-series platform NETPFE modulees;/modules/peertype;/modules/ifpfe_drv;/modules/platform;/modules;
    
    kld netpfe drv: ifpfed_chmic ifpfed_eth ifpfed_irb ifpfed_ml_cmn ifpfed_svcskld platform: ex_ifpfe if_vcpkld peertype: peertype_hcm peertype_pfem peertype_sfi p
    eertype_slavere peertype_xdpc grat_arp_on_ifup=YES: net.link.ether.inet.grat_arp_on_ifup: 1 -> 1
     ipsec kld.
    Doing additional network setup:.
    Starting final network daemons:.
    starting local daemons:set cores for group access
    .
    PFEX runs as JUNOS process...
    Boot media /dev/da0 has dual root support
    ** /dev/da0s2a
    FILE SYSTEM CLEAN; SKIPPING CHECKS
    clean, 75455 free (31 frags, 9428 blocks, 0.0% fragmentation)
    setting ldconfig path: /usr/lib /opt/lib
    starting standard daemons: cron.
    Local package initialization:.
    Initialize /var subdirs
    Tue May  7 12:49:53 EDT 2019
    
    sw-latham-core (ttyu0)
    
    login: 
    Logging to master
    Password:
    
    
    ***********************************************************************
    **                                                                   **
    **  WARNING: THIS DEVICE HAS BOOTED FROM THE BACKUP JUNOS IMAGE      **
    **                                                                   **
    **  It is possible that the primary copy of JUNOS failed to boot up  **
    **  properly, and so this device has booted from the backup copy.    **
    **                                                                   **
    **  The primary copy will be recovered by auto-snapshot feature now  **
    **                                                                   **

     

    Here is what my partitions look like:

     

    show system storage partitions
    fpc0:
    --------------------------------------------------------------------------
    Boot Media: internal (da0)
    Active Partition: da0s2a
    Backup Partition: da0s1a
    Currently booted from: backup (da0s1a)
    
    Partitions information:
      Partition  Size   Mountpoint
      s1a        316M   /
      s2a        324M   altroot
      s3d        887M   /var/tmp
      s3e        170M   /var
      s4d        116M   /config
    
    fpc1:
    --------------------------------------------------------------------------
    Boot Media: internal (da0)
    Active Partition: da0s1a
    Backup Partition: da0s2a
    Currently booted from: backup (da0s2a)
    
    Partitions information:
      Partition  Size   Mountpoint
      s1a        316M   altroot
      s2a        324M   /
      s3d        887M   /var/tmp
      s3e        170M   /var
      s4d        116M   /config

     

     

    Any other recommendations? What would cause something like this? 



  • 4.  RE: EX4300 Switches reset itself to version 13.2X51-D35.3. Layer 2 networking fails

     
    Posted 05-08-2019 13:56

    Smicker is right, the switches are booting from the backup partition which wasn't upgraded

    This is shown in show system storage partitions

    show system storage partitions
    fpc0:
    --------------------------------------------------------------------------
    Boot Media: internal (da0)
    Active Partition: da0s2a
    Backup Partition: da0s1a
    Currently booted from: backup (da0s1a)
    
    Partitions information:
      Partition  Size   Mountpoint
      s1a        316M   /
      s2a        324M   altroot
      s3d        887M   /var/tmp
      s3e        170M   /var
      s4d        116M   /config
    
    fpc1:
    --------------------------------------------------------------------------
    Boot Media: internal (da0)
    Active Partition: da0s1a
    Backup Partition: da0s2a
    Currently booted from: backup (da0s2a)
    
    Partitions information:
      Partition  Size   Mountpoint
      s1a        316M   altroot
      s2a        324M   /
      s3d        887M   /var/tmp
      s3e        170M   /var
      s4d        116M   /config

     You need to follow this procedure https://kb.juniper.net/InfoCenter/index?page=content&id=KB23180



  • 5.  RE: EX4300 Switches reset itself to version 13.2X51-D35.3. Layer 2 networking fails

    Posted 05-09-2019 09:37

    I was able to fix this by copying over the backup aprtition to the primary partition and then performing an upgrade to the latest JUNOS. Then made sure my backup parititon was up to date as well.