Switching

Expand all | Collapse all

EX3400 no serial login prompt after recovery from bad upgrade

Jump to Best Answer
  • 1.  EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 10:46

    After recovering from a bad upgrade by reinstalling clean from usb, EX3400 standalone (new install) can't reach login: prompt from serial console.

     

    Seems like I am missing something simple as all the ports are active. Couldn't find any solutions in my searching. Hoping someone has seen this before.

     

    Full details:

    I ran an upgrade on my EX3400 switch (new switch, was setting up first time) using

     

    request system software add /tmp/usb/junos-arm-32-18.2R3.4.tgz

    which failed because the configuration didnt match this version. I looked that up and found a thread stating this was a problem introduced in version 17 for certain configs and I could use the `no-validate` option to ignore. I tried that, install worked, but caused a kernel panic on restart (hindsight tells me I should have been more careful with this).

     

     

    So I went ahead and followed the steps to recover here (usb boot): https://kb.juniper.net/InfoCenter/index?page=content&id=KB31265&actp=METADATA

    Worked great, rebooted and no kernel panic. I tried this with an 18.x version, got the no serial console login prompt issue.

     

    So, I decided to reinstall to the original version as well just for sanity, 15.1X53-D55.5.

    Still got the no login prompt when connected to serial console. I am certain my serial console is fine because it logs output right up untill here every time I reboot.

    I tried running a factory-reset on config by holding the reset button on the front, no luck.

     

    Last few lines of output always look ike this:

     

    Starting casperd.
    Starting jlaunchhelperd.
    /etc/rc: WARNING: run_rc_command: cannot run 
    kern.rtc_retries: 0 -> 5
    Starting cron.
    
    Thu Jan  1 00:02:18 UTC 1970
                     

    Can't interact with console at all from here. Enter does nothing nor does pressing any combination of keys I could think of. My other switch shows `FreeBSD/arm (TEST) (ttyu1)` right after this and then the login prompt which this switch never reaches.

     

     



  • 2.  RE: EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 11:09

    Hi nickaielloftx,

     

    Please verify if the below is the correct image type that you tried to run on the device:

    junos-arm-32-18.2R3.4.tgz

    Did you see normal boot logs while the device was booting up?

     

    I am suspecting an issue with the file system or hardware issue. Does not seem to me like a miss with respect to something simple. 

     

    In case this is a critical box, you can try opening a JTAC case for quicker remote troubleshooting.

     

    Hope this helps 🙂

     

    Please mark "Accepted Solution" if this helps you solve your query.

    Kudos are always appreciated!

     



  • 3.  RE: EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 11:36

    Hi,

    junos-arm-32-18.2R3.4.tgz

    Was the version I installed that caused the issue. It was actually a mistake on my part as I had grabbed the wrong version. I had intended to use version 18.1R3.3, but got mixed up when downloading the version.

    For the bad versio, boot logs were fine right up until a kernel panic at

    kern.module_path: /packages/sets/active/boot/junFatal kernel mode data abort: 'Translation Fault (L2)' on read

    The odd part is that after installing other version from USB, everything seems to install and then boot fine, just can't get into it because of no login prompt over serial.

     

    I did open a JTAC case, but unfortunately the only option from that side was an RMA. Wanted to see if anyone in the community recognized the issue.

    Appreciate the feedback!



  • 4.  RE: EX3400 no serial login prompt after recovery from bad upgrade

     
    Posted 06-12-2020 12:09

    My experience shows that install formats are not successful from time to time, it helps to do it a second time after it fails completely or partially.

    Can you please confirm that you used the image file "junos-install-media-usb-ex-arm-32-1X.XRX.X.img.gz" to deploy it to an USB device and to boot the EX3400 from it? It may be worth a second try.

     

    You can try to use a very recent version for a recovery, in case you are running into any software issue, e.g. "junos-install-media-usb-ex-arm-32-19.1R3-S1.img.gz".



  • 5.  RE: EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 12:44

    Hi F1ght3r,

     

    I tried installing 2 different versions via the format USB method (3 total installs).

    The first was "junos-install-media-usb-arm-32-18.1R3.3-limited.img"

    Second, I tried the non-limited of that same version "junos-install-media-usb-arm-32-18.1R3.3.img"

    Both of these were downloaded from the "Install media" section of the downloads page for EX3400.

     

    Finally, I decided to just try going back to the original version (I actually had it saved locally, no longer available to Download from what I saw):

    extracting "junos-install-media-usb-arm-32-15.1X53-D55.5.img.gz" gave me

    "junos-install-media-usb-arm-32.img"

     

    For all 3 attempts, the output indicates success after booting to USB (shows success then reboots), but then fails to give the login prompt once booted to the new version. During boot, I was always able to see it loading the modules ending in the version.

    Example: "junos-net-prd-arm-32-20161230.114620_builder_junos_151_x53_d55"

    Where the version matched whatever I installed last.

     


    You can try to use a very recent version for a recovery, in case you are running into any software issue, e.g. "junos-install-media-usb-ex-arm-32-19.1R3-S1.img.gz".


    I will give this a try.



  • 6.  RE: EX3400 no serial login prompt after recovery from bad upgrade

     
    Posted 06-12-2020 12:58

    Hi Nick,

     

    > Second, I tried the non-limited of that same version "junos-install-media-usb-arm-32-18.1R3.3.img"

     

    Just to be safe, are we talking about the same package? The correct one for 18.1R3 would be "junos-install-media-usb-arm-32-18.1R3.3.img.gz"



  • 7.  RE: EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 13:09

    Yeah, "junos-install-media-usb-arm-32-18.1R3.3.img.gz" was the downloaded package, I extracted and used

    dd if=./junos-install-media-usb-arm-32-18.1R3.3.img of=/dev/<myusbdevice>

     

    Then plugged into the switch and at next boot, I interrupted with ctrl+c and selected boot menu -> Boot from USB

    Process came from: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31265&actp=METADATA



  • 8.  RE: EX3400 no serial login prompt after recovery from bad upgrade

     
    Posted 06-12-2020 13:15

    Ok, sounds good, please tell me when you have tested it with 19.1R3-S1 install media.



  • 9.  RE: EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 13:48

    Tried with version 19.1R3 but same problem. Though one thing I noticed now as I am replying, the media I downloaded doesn't have the "-S1" as you mentioned. Not sure if that is an issue/where to get those "-S1" versions from.

    "junos-install-media-usb-ex-arm-32-19.1R3.9.img.gz"

     

    The end of the output is

    Creating a recovery snapshot
    Creating image ...
    Compressing image ...
    Image size is 335MB
    Recovery snapshot created successfully
    @ 946685718 [2000-01-01 00:15:18 UTC] snapshot done
    
     Lock Manager
    RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
    Copyright (c) 1992-2006 Birdstep Technology, Inc.  All Rights Reserved.
    
    Unix Domain sockets Lock manager
    Lock manager 'lockmgr' started successfully.
    
    Database Initialization Utility
    RDM Embedded 7 [04-Aug-2006] http://www.birdstep.com
    Copyright (c) 1992-2006 Birdstep Technology, Inc.  All Rights Reserved.
    
    Profile database initialized
    Enhanced arp scale is disabled
    /etc/rc: WARNING: No dump device.  Savecore not run.
    Prefetching /usr/sbin/rpd ...
    Prefetching /usr/sbin/lacpd ...
    Prefetching /usr/sbin/chassisd ...
    Starting jlaunchhelperd.
    sysctl: unknown oid 'kern.rtc_retries'
    Starting cron.
    cron: cron already running, pid: 12700
    /etc/rc: WARNING: failed to start cron
    
    Thu Mar 26 00:00:00 UTC 2020
    
    


  • 10.  RE: EX3400 no serial login prompt after recovery from bad upgrade

     
    Posted 06-12-2020 13:56

    > Though one thing I noticed now as I am replying, the media I downloaded doesn't have the "-S1" as you mentioned.

    > Not sure if that is an issue/where to get those "-S1" versions from.

     

    No, this is not an issue, "-S1" is just an additional bugfixing service release. To download service releases, you have to choose "OS: Junos SR" instead of "Junos".

     

    Just to make sure, when you connect your console cable to any other device, your input is recognized normally when you press Enter or any other key, right?



  • 11.  RE: EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 14:02

    Console worked fine, just switched it to another EX3400 switch and was able to interact normally with that switch, no changes to console settings required.



  • 12.  RE: EX3400 no serial login prompt after recovery from bad upgrade

     
    Posted 06-12-2020 14:05

    Then I'm afraid that you have to RMA this device as suggested by JTAC, unless JTAC don't find another solution for that. Maybe some internal code was overwritten by using the wrong install package at the beginning, which cannot be solved by a recovery installation.



  • 13.  RE: EX3400 no serial login prompt after recovery from bad upgrade

    Posted 06-12-2020 14:07

    Thanks for the support here! Appreciate the help.



  • 14.  RE: EX3400 no serial login prompt after recovery from bad upgrade
    Best Answer

    Posted 06-13-2020 16:21

    In case anyone else comes across this in the future, I was actually able to get it working.

     

    First thing was installing clean by booting from USB installer described here: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31265&actp=METADATA

    Don't know if the version matters, but I chose to return to the original version the switch started with as I fortunately had the install media for it.

     

    After that I still got the no login prompt, so rebooted and followed these steps: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31403&actp=METADATA

    Except, instead of just setting password, I replaced the whole config with the config pulled from my other switch.

    Additionaly, I did not run reboot from the recovery cli. Instead I exited from both configuration and operational mode which quickly loaded to a "login: " prompt. I logged in with the password I had just set and did some other config/testing before rebooting. The next boot was normal and reached the "login: " prompt fine.

     

    Hope this saves someone in the future!