Switching

Expand all | Collapse all

EX3400 - Recovery from failed SW upgrade fails

Jump to Best Answer
  • 1.  EX3400 - Recovery from failed SW upgrade fails

    Posted 10-10-2017 08:49

    When trying to recover from a failed  customer SW upgrade, an EX3400 switch gets a kernel panic.

    - The methods using USB and TFTP show the same result.

    - When trying to boot an snapshot created from another EX3400 switch , no access to USB image happens when booting. (can't load 'kernel').https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/ex-series-system-snapshot-boot-directory.html

     

    Any suggestions? Unfortunately this device was purchased a year ago and customer didnt buy service. 😞

     

    U-Boot 2016.01-rc1 (May 23 2016 - 08:45:58 -0700) VERSION=1.2.0
    
    DRAM:  AVS: 0x0
    DEV ID = 0xdc14
    SKU ID = 0xb547
    DDR type: DDR3
    MEMC 0 DDR speed = 750MHz
    PHY revision version: 0x00044006
    SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
    In:    serial
    Out:   serial
    Err:   serial
    FLASH bank: 1
    Board: EX3400-48P
    Net:   bcmiproc_eth-0
    Last Reset Reason: 8
    Hit ^C to stop autoboot:  0 
    SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
    device 0 offset 0x3c0000, size 0x10000
    SF: 65536 bytes @ 0x3c0000 Read: OK
    
    ## Checking Image at 61000000 ...
       Legacy image found
       Image Name:   Juniper U-Boot Script File
       Image Type:   ARM Unknown OS Script (uncompressed)
       Data Size:    1803 Bytes = 1.8 KiB
       Load Address: 00000000
       Entry Point:  00000000
       Contents:
          Image 0: 1795 Bytes = 1.8 KiB
       Verifying Checksum ... OK
    ## Executing script at 61000000
    Setting bus to 0
    SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
    device 0 offset 0x140000, size 0x140000
    SF: 1310720 bytes @ 0x140000 Read: OK
    6113fffc: f1c1fb43                               C...
    crc32 for 61000000 ... 6113ffff ==> 43fbc1f1
    Total of 1 word(s) were the same
    starting USB...
    USB0:   phy ctrl 01d36440
    Initial usb2h clock is: 000003fc
    Initial usb2h clock is: 000003fd
    iClk = 00000078, USBClk = 01536440
    
    Bring usb2h_out of reset.......
    usb2_reset_state is set and now it is: 00000000
    usb2_reset_state is: 00000000
    USB EHCI 1.00
    scanning bus 0 for devices... 4 USB Device(s) found
           scanning usb for storage devices... 2 Storage Device(s) found
    SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB, mapped at 00000088
    ## Starting application at 0x00000074 ...
    Consoles: U-Boot console  
    
    FreeBSD/armv6 U-Boot loader, Revision 1.2
    (fbsd-builder@svl-bsdx-05.juniper.net, Wed May 18 12:32:06 PDT 2016)
    
    DRAM: 2048MB
    Number of U-Boot devices: 3
    U-Boot env: loaderdev not set, will probe all devices.
    Found U-Boot device: disk
      Probing all disk devices...
      Checking unit=0 slice=<auto> partition=<auto>... good.
    /
    can't load 'kernel'
    
    Type '?' for a list of commands, 'help' for more detailed help.
    lloader> install --format file:///junos-arm-32-15.1X53-D56.tgz panic: free: guard1 fail @ 0x45c40 from /.amd/svl-engdata1vs1/occamdev/build/freebsd/head/20160518.183540_fbsd-builder_head.327854/src/lib/libstand/dosfs.c:248 --> Press a key on the console to reboot <--


  • 2.  RE: EX3400 - Recovery from failed SW upgrade fails

     
    Posted 10-10-2017 08:54

    Why not just RMA the device?  HW RMAs are covered (free of charge) under Limited Lifetime Warrenty.  



  • 3.  RE: EX3400 - Recovery from failed SW upgrade fails
    Best Answer

    Posted 10-10-2017 09:22

    Hello Frank,

     

    As EX3400 is not able to load kernel it does not go to OAM volume which can be used to install junos from bootable USB.

    Create bootable USB with install media image 15.1X53-D57 version(junos-install-media-usb-arm-32-15.1X53-D57.3-limited.img.gz) and insert usb to switch. 

    Please follow recovery instructions shown below. Install --format command should not be used for EX2300/3400. We are creating a KB for this procedure.

      

    Type '?' for a list of commands, 'help' for more detailed help.

    loader>

     

    1. Select USB manually  

    loader> set currdev="disk1s1a"

     2.  Manually boot from USB:

    loader> include /boot/loader.rc

     

     3.  you will observe following messages and switch will boot from USB:

     

    /boot/kernel data=0xe48000 syms=[0x4+0x5cdb0+0x4+0x81971]

    /boot/dtb/ex3400.dtb size=0x1f25

    /boot/contents.iso size=0x171e800

     

    Hope this will be helpful for recovery.

     

    Regards,

    Raj Maradia

     

     



  • 4.  RE: EX3400 - Recovery from failed SW upgrade fails

    Posted 10-11-2017 01:46

    Thanks for this excellent answer.

     

    Best Regards

     



  • 5.  RE: EX3400 - Recovery from failed SW upgrade fails

    Posted 01-25-2018 05:38

    Thank you for this information. It cannot get documented soon enough as JTAC was not helpful and still sending me towards the older EX recovery methods...
    Just to note, I initially downloaded the "limited" image and it seemed to not work. This could be my image or I messed up the process. I then downloaded the junos-install-media-usb-arm-32-[version].img.gz file and was successful. Just to add some details that some may not know.

     

    Uncompress the file so you are left with a .img file
    On OSX to create a bootable drive do the following:
    $diskutil list - to find where your thumb drive is located, I'll be using "disk2" in my example
    $diskutil unmountDisk /dev/disk2
    $sudo dd if=/path/to/junos-install-media-usb-arm-32-version.img of=/dev/disk2 bs=1m
        OSX will probably have a pop-up asking to eject, but just in case you can eject manually
    $diskutil eject /dev/disk2

    Plug the now bootable USB drive to the EX2300 switch, boot up with the console cable plugged in. You can either interrupt the boot process to get to the 'loader>' prompt. In my example the switch would only boot to the loader> prompt.
    With the usb drive installed on the switch type in the following
    loader> set currdev="disk1s1a"
    loader> include /boot/loader.rc

     

    Thank you again 

     

     

     

     



  • 6.  RE: EX3400 - Recovery from failed SW upgrade fails

    Posted 06-01-2018 05:05

    Thank you very much

    it worked perfect for me



  • 7.  RE: EX3400 - Recovery from failed SW upgrade fails

    Posted 07-19-2018 10:04

    Hi, I've been trying this solution on an EX2300 and I haven't been able to get it too work. Here is the following output:

     

    can't load 'kernel'

     

    Type '?' for a list of commands, 'help' for more detailed help.

    loader> set currdev="disck1s1a"

    no such file or directory

    loader> set currdev="disk1s1a"

    loader> include /boot/loader.rc

    .autoboot not found

    Error while including /boot/loader.rc, in the line:

    .autoboot

    loader>

    I am using junos-install-media-usb-arm-32-15.1X53-D55.5.img as I would like to use the D55.5. 

     

    Thanks,

    Eli Wulff

     



  • 8.  RE: EX3400 - Recovery from failed SW upgrade fails

    Posted 07-19-2018 10:30

    I would suggest re-doing the USB drive, if that doesn't work try another USB drive (I've found in the past some work, some don't). Then power off the switch, plug in the drive, then boot. 



  • 9.  RE: EX3400 - Recovery from failed SW upgrade fails

    Posted 07-19-2018 10:49

    Hi Eli,

     

    There are problems loading D55 by the method that I stated above. There are few tweaks required to install D55 which is not straight forward. 

    It is better to install 15.1X53-D58 (JTAC recommended) and then downgrade if required.

     

    Thanks,

    Raj


    ewulff@juniper.net wrote:

    Hi, I've been trying this solution on an EX2300 and I haven't been able to get it too work. Here is the following output:

     

    can't load 'kernel'

     

    Type '?' for a list of commands, 'help' for more detailed help.

    loader> set currdev="disck1s1a"

    no such file or directory

    loader> set currdev="disk1s1a"

    loader> include /boot/loader.rc

    .autoboot not found

    Error while including /boot/loader.rc, in the line:

    .autoboot

    loader>

    I am using junos-install-media-usb-arm-32-15.1X53-D55.5.img as I would like to use the D55.5. 

     

    Thanks,

    Eli Wulff

     


     



  • 10.  RE: EX3400 - Recovery from failed SW upgrade fails

    Posted 02-15-2019 04:06

    Hi @rmaradia,

    How long does this procedure takes? I followed your steps, but the switch seems to hang after the reboot (see attachement). Is this a normal behaviour?

     

    thanks for your help.ex2300.png