Switching

Expand all | Collapse all

Unable to upgrade EX3300 stack members to JunOS 12.3R12.

Jump to Best Answer
  • 1.  Unable to upgrade EX3300 stack members to JunOS 12.3R12.

    Posted 01-16-2020 04:18

    Hello everyone,

    Yesterday I tried to upgrade our EX3300 stack with Junos 12.3R12. The current fw is 12.3R6.6. I have downloaded fw to: /var/tmp/jinstall-ex-3300-12.3R12.4-domestic-signed.tgz on master switch, after that I have issed command to validate new fw:

    request system software validate /var/tmp/jinstall-ex-3300-12.3R12.4-domestic-signed.tgz

    And finally I have started upgrade of last virtual-chassis member (9th), to avoid a downtime of whole VC:

    request system software add /var/tmp/jinstall-ex-3300-12.3R12.4-domestic-signed.tgz member 9

    Unfortunately after the member 9 has been rebooted it fails to load Junos. In the console some disquieting things has appeared during start of this switch (as below). After a while I have managed to take a snapshot and start switch with previous version of Junos (12.3R12). The affected switch status of virtual-chassis was 'not present'.

    Can anybody explain me what happened and what I have done wrong?

     

    MISSING '.'  I=4097  OWNER=0 MODE=40755
    SIZE=512 MTIME=Mar 17 12:26 2014
    DIR=?
    
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    
    FIX? yes
    
    DIRECTORY CORRUPTED  I=4098  OWNER=0 MODE=40755
    SIZE=512 MTIME=Jun 14 08:27 2013
    DIR=?
    
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    
    SALVAGE? yes
    
    MISSING '.'  I=4098  OWNER=0 MODE=40755
    SIZE=512 MTIME=Jun 14 08:27 2013
    DIR=?
    
    ...
    
    UNREF DIR  I=4131  OWNER=0 MODE=40700
    SIZE=512 MTIME=Feb 12 13:49 2015
    RECONNECT? yes
    
    NO lost+found DIRECTORY
    CREATE? yes
    
    CG 0: BAD MAGIC NUMBER
    
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    SORRY. CANNOT CREATE lost+found DIRECTORY
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    
    
    UNREF DIR  I=4130  OWNER=0 MODE=40700
    SIZE=512 MTIME=Feb 12 13:49 2015
    RECONNECT? yes
    
    NO lost+found DIRECTORY
    CREATE? yes
    
    CG 0: BAD MAGIC NUMBER
    
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    SORRY. CANNOT CREATE lost+found DIRECTORY
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    
    ...
    
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    SORRY. CANNOT CREATE lost+found DIRECTORY
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    
    
    
    CLEAR? yes
    
    UNREF FILE  I=4106  OWNER=0 MODE=120755
    SIZE=25 MTIME=Feb 12 13:50 2015
    RECONNECT? yes
    
    NO lost+found DIRECTORY
    CREATE? yes
    
    CG 0: BAD MAGIC NUMBER
    
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    SORRY. CANNOT CREATE lost+found DIRECTORY
    UNEXPECTED SOFT UPDATE INCONSISTENCY
    

     

     



  • 2.  RE: Unable to upgrade EX3300 stack members to JunOS 12.3R12.

     
    Posted 01-16-2020 05:26

    Hi,

     

    That output shows file system corruption.  You need to format install that device.

     

    Follow those steps to format install.

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB20643

     

    Regards,

     

    Randall

     



  • 3.  RE: Unable to upgrade EX3300 stack members to JunOS 12.3R12.

    Posted 01-16-2020 07:09

    Thank you very much for response. Is that means that it was just a coincidence that the file system of the switch was corrupted?
    As I mentioned the switch is now working fine. But this knowledge will be helpful.



  • 4.  RE: Unable to upgrade EX3300 stack members to JunOS 12.3R12.

     
    Posted 01-16-2020 07:37

    Due to the flash technology available back in 2008/9 the file systems on the legacy EX product lines (EX2200/3300/4200) are susceptible to flash corruption over time.  Longer switch is in-service with no reboot, the more likely this may happen.  It is hit or miss, from my experience.  I believe the most often occurance is with the Master RE switch in an EX4200 VC.  Since EX4200 is the oldest product, it is most likely one to see this.  Flash corruption does not negatively affect the operation of the switch.  Despite Unix/Linux tools being present, pre-identifying is difficult at best, and (again from experience) not of the tools to fix this work on an operating switch.

     

    I have seen this most often with EX4200, few and far better on EX3300/EX2200 - you look like one of the lucky ones!  For EX4200VC I have found that if you do a partition switch (affects Master RE) that things seem to go much better.

     

    As @randero stated, once a switch is re-boot/reset and has this state (flash corruption present) generally only recovery is via a USB install which would also require local console access.

     

    HTH and Good Luck



  • 5.  RE: Unable to upgrade EX3300 stack members to JunOS 12.3R12.
    Best Answer

     
    Posted 01-16-2020 08:49

    There are even KB articles for the affected platforms:

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB20570

     

    I have lots of experience with EX4200 (VC) flash pain. The main reason for flash corruption are locally written syslog messages, which corrupt the internal flash. And the only way to recover completely is to perform an "install --format". The only way to prevent this corruption is to disable locally written syslog messages completely and never use traceoptions or anything else that writes to the local flash.

     

    By the way, flash corruption actually does affect the operation of the Switch. We saw very strange behavior and if you perform a commit on such a device, the device may crash and creates a vmcore dump.