Switching

Expand all | Collapse all

Juniper EX3400 software upgrade not possible

  • 1.  Juniper EX3400 software upgrade not possible

    Posted 06-25-2017 14:05

    Hello,

     

    we have started to take a look at the EX3400 series and encountered that is impossible to upgrade due to low storage space.

    In first place transfering the package to the device is insanly slow at 140 kb/s. We have been testing via http, ftp and scp to see if it is not a protocol regression but transfer rates stayed at 140 kb/s. The upgrade itself fails due to low storage space but it's not possible to free up more storage since all the tmp directories are already empty.

     

    Software upgrades have always been troublesome with Juniper due to low space, memory leakage and slow processing power but given the fact that the EX3400 is pretty new and already suffering from outdated hardware is not understandable and makes automatization of upgrade process impossible.

     

    We never encountered upgrade problems with other vendors even with entry level gear.

     



  • 2.  RE: Juniper EX3400 software upgrade not possible

     
    Posted 06-25-2017 18:26

    Hello,

     

    It is strange that you are experiencing very slow transfer rates.

    Did you try other methods outlined here:

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB31201&actp=RSS

     

    Regards,

     

    Rushi



  • 3.  RE: Juniper EX3400 software upgrade not possible

     
    Posted 06-26-2017 22:07

    Hi Folks,
    Please do a full SCAN of the box and delete any unwanted files,

     

    start shell user root
    cd /
    find .

     

    -rengar



  • 4.  RE: Juniper EX3400 software upgrade not possible

    Posted 07-03-2017 00:44

    HI

     

    Please check this KB KB31633

     

    Cause
    There is storage issues in EX2300 and EX3400 in /var directory. This has limited space which could be occupied by other files.
     
    Solution
    We would be getting the error when we try normal upgrade
     
    root@EX-2300> ...ar/tmp/junos-arm-32-15.1X53-D52.tgz force                 
    ERROR: not enough space to unpack /var/tmp/junos-arm-32-15.1X53-D52.tgz
     
    Workaround to fix this issue is to move the file to /mfs directory and upgrade from there.
     
    root@EX-2300:RE:0% mv junos-arm-32-15.1X53-D52.tgz /mfs
    root@EX-2300:RE:0% ls
    bcast.bdisp.log         gres-tp-tomcat          rtsdb
    bcast.disp.log          krt_rpf_filter.txt      sampled.pkts
    bcast.rstdisp.log       package.log             sd-upgrade
    bcast.undisp.log        pics                    stable
    ex_autod_config         pkg_cleanup.log         version_ex.xml
    ex_autod_rollback_cfg   pkg_cleanup.log.err
    root@EX-2300:RE:0% cli
    {master:0}
    root@EX-2300> ... add /mfs/junos-arm-32-15.1X53-D52.tgz                      
    NOTICE: Validating configuration against junos-arm-32-15.1X53-D52.tgz.
    NOTICE: Use the 'no-validate' option to skip this if desired.
    Verified junos-arm-32-15.1X53-D52 signed by PackageProductionEc_2016
    Adding junos-arm-32-15.1X53-D52 ...
    Verified fips-mode signed by PackageProductionEc_2016
    Adding fips-mode-arm-32-20161006.233631_builder_junos_151_x53_d52 ...
     

     

    Thanks

    Partha



  • 5.  RE: Juniper EX3400 software upgrade not possible

    Posted 10-09-2018 05:29

    Followed Partha advice this morning on 2300c and install works if you place the junos packages

    in /mfs/

     

     

    Before when i tried request system software add /var/tmp/junos-arm-32-18.3R1.9.tgz

    it would fail

    ERROR: insufficient space for /packages/db/junos-arm-32-18.3R1.9/contents/junos-runtime.tgz
    ERROR: insufficient space
    ERROR: Failed to add junos-arm-32-18.3R1.9.tgz

     

    start shell user root

    mv /var/tmp/junos-arm-32-18.3R1.9.tgz

    exit

    request system software add /mfs/junos-arm-32-18.3R1.9.tgz

    Now  all is well.



  • 6.  RE: Juniper EX3400 software upgrade not possible

    Posted 10-15-2018 06:33

    I still get insufficient space erros when installing from /mfs/ even when using force option. 

     

    Basically stuck on one code version. Juniper needs to address this issue asap, in my opinion.



  • 7.  RE: Juniper EX3400 software upgrade not possible

     
    Posted 10-15-2018 07:04

    The same issue exists on 2300/2300-C. One workaround is to install the upgrade from a mounted USB flash drive.



  • 8.  RE: Juniper EX3400 software upgrade not possible

    Posted 10-15-2018 09:10

    So in other words there is not fix. JTAC seems to be at a loss related to it, choosing instead to focus on system storage clean up not cleaning up 2 small 25KB log files. 



  • 9.  RE: Juniper EX3400 software upgrade not possible

    Posted 10-16-2018 05:30

    Even from usb still getting lack of space error. 



  • 10.  RE: Juniper EX3400 software upgrade not possible

     
    Posted 10-16-2018 06:54

    Did you make sure to delete recovery snapshots that might be taking up space?



  • 11.  RE: Juniper EX3400 software upgrade not possible

     
    Posted 10-29-2018 12:07
    Juniper just updated the “not enough storage” kb article with additional suggestions for freeing up space pre-install.

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198


  • 12.  RE: Juniper EX3400 software upgrade not possible

    Posted 10-31-2018 13:15

    The only thing that worked for us was following this suggestion:

    https://www.reddit.com/r/Juniper/comments/76w463/cleaning_up_old_packages_on_ex3400/

     

    Which is basically removing all older versions from /packages/db, as all older versions seem to be kept there.

     

    This is indeed extremely bad from all points, as it's a comepletly manual process that shouldn't have to exist, as it should be properly handled, and is prone to mistakes, to which I don't know the impact, since I don't know if these are temporary files or not. And of course, nothing in the KB about that either.



  • 13.  RE: Juniper EX3400 software upgrade not possible

     
    Posted 02-04-2019 16:44

    Still an issue for me under 18.4R1-S1. Was able to install by moving the installation package to /mfs and using

     

    request system software add /mfs/junos-arm-32-18.4R1-S1.1.tgz no-copy no-validate force unlink reboot


  • 14.  RE: Juniper EX3400 software upgrade not possible

    Posted 03-06-2019 15:45

    We have a bunch of ex3400's and are experiencing the not enough storage space issue.  Manually deleting files sometimes works, but that does not scale when you have a lot of switches. 

     

    I also see painfully slow transfer speeds with scp and ftp when copying the software package to the switch.  I routinely get 700 kB/s and it takes about 8 minutes to transfer a 300 mb file.  Anyone else see this and or find a solution?



  • 15.  RE: Juniper EX3400 software upgrade not possible

     
    Posted 06-23-2019 16:41

    Is it VC or stand alone device? is the traffic comming over the backup member? if is this the case, I would suggest to change the mastership of RE and transfer the traffic directly to the master.



  • 16.  RE: Juniper EX3400 software upgrade not possible

    Posted 06-23-2019 22:17

    The painfully slow scp/ftp traffic is due to the fixed PFE->CPU rate-limiter. Both ssh and ftp are limited to 100 pps towards the CPU/RE.

     

    These rate-limiters are currently not configurable so no workaround.



  • 17.  RE: Juniper EX3400 software upgrade not possible

    Posted 06-24-2019 04:23

    To share our experience, we have a USB stick plugged into the master EX3400 and use it to store the update archive.

     

    Usually this is still not enough to workaround space limitations and we also delete old packages manually. After our last update from 18.1R3-S5 to 18.1R3-S6, I put together the following command:

    ls /packages/db/ | grep -vE `ls -l /packages/sets/active/ | grep '^l' | cut -d'/' -f 4 | tr '\n' '|' | sed 's/|$//'`

    This should print all packages that are not referenced from /packages/sets/active/ which should be safe to remove. No guarantee that this works in all cases (it's your fault if you use it and it breaks), but might be easier than manually going through the entire directory manually 😉



  • 18.  RE: Juniper EX3400 software upgrade not possible

    Posted 06-26-2019 08:41

    Hi,

     

    Please ensure that the destination file location of the "zip file for upgrade" has atleast two times more space than the size of the zip file. You can always keep a check on file storage capacity to prevent such errors. 

     

    - M 🙂



  • 19.  RE: Juniper EX3400 software upgrade not possible

    Posted 06-26-2019 08:41

    Hi,

     

    Please ensure that the destination file location of the "zip file for upgrade" has atleast two times more space than the size of the zip file. You can always keep a check on file storage capacity to prevent such errors. 

     



  • 20.  RE: Juniper EX3400 software upgrade not possible

    Posted 06-24-2019 19:31

    Hello,

     

    As per the below KB,

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198

     

    If you are currently running image is 18.1R2 or later:

     

    1. root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force unlink no-copy

      The force option frees up storage space by removing the non-recovery snapshots if any. The unlink option is available starting 18.1R2. The unlink option will remove the .tgz file immediately after unpacking so that the available image size calculation takes into account the free 300M. Without using unlink, this space is not considered.

    2. If the above step fails, run the software update with the force and unlink option again (it will work the second time in some situations).

     

    The force unlink option has always worked for me if not once then the second time it has always worked. 

     

    Thanks,

    Puneet