Switching

When trying to recover from a failed  customer SW upgrade, an EX3400 switch gets a kernel panic.

- The methods using USB and TFTP show the same result.

- When trying to boot an snapshot created from another EX3400 switch , no access to USB image happens when booting. (can't load 'kernel').https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/ex-series-system-snapshot-boot-directory.html

Any suggestions? Unfortunately this device was purchased a year ago and customer didnt buy service. 😞

U-Boot 2016.01-rc1 (May 23 2016 - 08:45:58 -0700) VERSION=1.2.0

DRAM:  AVS: 0x0
DEV ID = 0xdc14
SKU ID = 0xb547
DDR type: DDR3
MEMC 0 DDR speed = 750MHz
PHY revision version: 0x00044006
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
In:    serial
Out:   serial
Err:   serial
FLASH bank: 1
Board: EX3400-48P
Net:   bcmiproc_eth-0
Last Reset Reason: 8
Hit ^C to stop autoboot:  0 
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
device 0 offset 0x3c0000, size 0x10000
SF: 65536 bytes @ 0x3c0000 Read: OK

## Checking Image at 61000000 ...
   Legacy image found
   Image Name:   Juniper U-Boot Script File
   Image Type:   ARM Unknown OS Script (uncompressed)
   Data Size:    1803 Bytes = 1.8 KiB
   Load Address: 00000000
   Entry Point:  00000000
   Contents:
      Image 0: 1795 Bytes = 1.8 KiB
   Verifying Checksum ... OK
## Executing script at 61000000
Setting bus to 0
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB
device 0 offset 0x140000, size 0x140000
SF: 1310720 bytes @ 0x140000 Read: OK
6113fffc: f1c1fb43                               C...
crc32 for 61000000 ... 6113ffff ==> 43fbc1f1
Total of 1 word(s) were the same
starting USB...
USB0:   phy ctrl 01d36440
Initial usb2h clock is: 000003fc
Initial usb2h clock is: 000003fd
iClk = 00000078, USBClk = 01536440

Bring usb2h_out of reset.......
usb2_reset_state is set and now it is: 00000000
usb2_reset_state is: 00000000
USB EHCI 1.00
scanning bus 0 for devices... 4 USB Device(s) found
       scanning usb for storage devices... 2 Storage Device(s) found
SF: Detected MX25L6405D with page size 256 Bytes, erase size 64 KiB, total 8 MiB, mapped at 00000088
## Starting application at 0x00000074 ...
Consoles: U-Boot console  

FreeBSD/armv6 U-Boot loader, Revision 1.2
(fbsd-builder@svl-bsdx-05.juniper.net, Wed May 18 12:32:06 PDT 2016)

DRAM: 2048MB
Number of U-Boot devices: 3
U-Boot env: loaderdev not set, will probe all devices.
Found U-Boot device: disk
  Probing all disk devices...
  Checking unit=0 slice=<auto> partition=<auto>... good.
/
can't load 'kernel'

Type '?' for a list of commands, 'help' for more detailed help.

lloader> install --format file:///junos-arm-32-15.1X53-D56.tgz
panic: free: guard1 fail @ 0x45c40 from /.amd/svl-engdata1vs1/occamdev/build/freebsd/head/20160518.183540_fbsd-builder_head.327854/src/lib/libstand/dosfs.c:248
--> Press a key on the console to reboot <--

Why not just RMA the device?  HW RMAs are covered (free of charge) under Limited Lifetime Warrenty.  

Hello Frank,

As EX3400 is not able to load kernel it does not go to OAM volume which can be used to install junos from bootable USB.

Create bootable USB with install media image 15.1X53-D57 version(junos-install-media-usb-arm-32-15.1X53-D57.3-limited.img.gz) and insert usb to switch. 

Please follow recovery instructions shown below. Install --format command should not be used for EX2300/3400. We are creating a KB for this procedure.

Type '?' for a list of commands, 'help' for more detailed help.

loader>

1. Select USB manually  

loader> set currdev="disk1s1a"

 2.  Manually boot from USB:

loader> include /boot/loader.rc

 3.  you will observe following messages and switch will boot from USB:

/boot/kernel data=0xe48000 syms=[0x4+0x5cdb0+0x4+0x81971]

/boot/dtb/ex3400.dtb size=0x1f25

/boot/contents.iso size=0x171e800

Hope this will be helpful for recovery.

Regards,

Raj Maradia

Thanks for this excellent answer.

Best Regards

Thank you for this information. It cannot get documented soon enough as JTAC was not helpful and still sending me towards the older EX recovery methods...
Just to note, I initially downloaded the "limited" image and it seemed to not work. This could be my image or I messed up the process. I then downloaded the junos-install-media-usb-arm-32-[version].img.gz file and was successful. Just to add some details that some may not know.

Uncompress the file so you are left with a .img file
On OSX to create a bootable drive do the following:
$diskutil list - to find where your thumb drive is located, I'll be using "disk2" in my example
$diskutil unmountDisk /dev/disk2
$sudo dd if=/path/to/junos-install-media-usb-arm-32-version.img of=/dev/disk2 bs=1m
    OSX will probably have a pop-up asking to eject, but just in case you can eject manually
$diskutil eject /dev/disk2

Plug the now bootable USB drive to the EX2300 switch, boot up with the console cable plugged in. You can either interrupt the boot process to get to the 'loader>' prompt. In my example the switch would only boot to the loader> prompt.
With the usb drive installed on the switch type in the following
loader> set currdev="disk1s1a"
loader> include /boot/loader.rc

Thank you again rmaradia

Thank you very much

it worked perfect for me

Hi, I've been trying this solution on an EX2300 and I haven't been able to get it too work. Here is the following output:

can't load 'kernel'

Type '?' for a list of commands, 'help' for more detailed help.

loader> set currdev="disck1s1a"

no such file or directory

loader> set currdev="disk1s1a"

loader> include /boot/loader.rc

.autoboot not found

Error while including /boot/loader.rc, in the line:

.autoboot

loader>

I am using junos-install-media-usb-arm-32-15.1X53-D55.5.img as I would like to use the D55.5. 

Thanks,

Eli Wulff

I would suggest re-doing the USB drive, if that doesn't work try another USB drive (I've found in the past some work, some don't). Then power off the switch, plug in the drive, then boot. 

Hi Eli,

There are problems loading D55 by the method that I stated above. There are few tweaks required to install D55 which is not straight forward. 

It is better to install 15.1X53-D58 (JTAC recommended) and then downgrade if required.

Thanks,

Raj

---

ewulff@juniper.net wrote:

Hi, I've been trying this solution on an EX2300 and I haven't been able to get it too work. Here is the following output:

 

can't load 'kernel'

 

Type '?' for a list of commands, 'help' for more detailed help.

loader> set currdev="disck1s1a"

no such file or directory

loader> set currdev="disk1s1a"

loader> include /boot/loader.rc

.autoboot not found

Error while including /boot/loader.rc, in the line:

.autoboot

loader>

I am using junos-install-media-usb-arm-32-15.1X53-D55.5.img as I would like to use the D55.5. 

 

Thanks,

Eli Wulff

 

---

Hi @rmaradia,

How long does this procedure takes? I followed your steps, but the switch seems to hang after the reboot (see attachement). Is this a normal behaviour?

thanks for your help.