we have started to take a look at the EX3400 series and encountered that is impossible to upgrade due to low storage space.
In first place transfering the package to the device is insanly slow at 140 kb/s. We have been testing via http, ftp and scp to see if it is not a protocol regression but transfer rates stayed at 140 kb/s. The upgrade itself fails due to low storage space but it's not possible to free up more storage since all the tmp directories are already empty.
Software upgrades have always been troublesome with Juniper due to low space, memory leakage and slow processing power but given the fact that the EX3400 is pretty new and already suffering from outdated hardware is not understandable and makes automatization of upgrade process impossible.
We never encountered upgrade problems with other vendors even with entry level gear.
It is strange that you are experiencing very slow transfer rates.
Did you try other methods outlined here:
Hi Folks,Please do a full SCAN of the box and delete any unwanted files,
start shell user rootcd /find .
Please check this KB KB31633
Followed Partha advice this morning on 2300c and install works if you place the junos packages
Before when i tried request system software add /var/tmp/junos-arm-32-18.3R1.9.tgz
it would fail
ERROR: insufficient space for /packages/db/junos-arm-32-18.3R1.9/contents/junos-runtime.tgzERROR: insufficient spaceERROR: Failed to add junos-arm-32-18.3R1.9.tgz
start shell user root
request system software add /mfs/junos-arm-32-18.3R1.9.tgz
Now all is well.
I still get insufficient space erros when installing from /mfs/ even when using force option.
Basically stuck on one code version. Juniper needs to address this issue asap, in my opinion.
The same issue exists on 2300/2300-C. One workaround is to install the upgrade from a mounted USB flash drive.
So in other words there is not fix. JTAC seems to be at a loss related to it, choosing instead to focus on system storage clean up not cleaning up 2 small 25KB log files.
Even from usb still getting lack of space error.
Did you make sure to delete recovery snapshots that might be taking up space?
The only thing that worked for us was following this suggestion:
Which is basically removing all older versions from /packages/db, as all older versions seem to be kept there.
This is indeed extremely bad from all points, as it's a comepletly manual process that shouldn't have to exist, as it should be properly handled, and is prone to mistakes, to which I don't know the impact, since I don't know if these are temporary files or not. And of course, nothing in the KB about that either.
Still an issue for me under 18.4R1-S1. Was able to install by moving the installation package to /mfs and using
request system software add /mfs/junos-arm-32-18.4R1-S1.1.tgz no-copy no-validate force unlink reboot
We have a bunch of ex3400's and are experiencing the not enough storage space issue. Manually deleting files sometimes works, but that does not scale when you have a lot of switches.
I also see painfully slow transfer speeds with scp and ftp when copying the software package to the switch. I routinely get 700 kB/s and it takes about 8 minutes to transfer a 300 mb file. Anyone else see this and or find a solution?
Is it VC or stand alone device? is the traffic comming over the backup member? if is this the case, I would suggest to change the mastership of RE and transfer the traffic directly to the master.
The painfully slow scp/ftp traffic is due to the fixed PFE->CPU rate-limiter. Both ssh and ftp are limited to 100 pps towards the CPU/RE.
These rate-limiters are currently not configurable so no workaround.
To share our experience, we have a USB stick plugged into the master EX3400 and use it to store the update archive.
Usually this is still not enough to workaround space limitations and we also delete old packages manually. After our last update from 18.1R3-S5 to 18.1R3-S6, I put together the following command:
ls /packages/db/ | grep -vE `ls -l /packages/sets/active/ | grep '^l' | cut -d'/' -f 4 | tr '\n' '|' | sed 's/|$//'`
This should print all packages that are not referenced from /packages/sets/active/ which should be safe to remove. No guarantee that this works in all cases (it's your fault if you use it and it breaks), but might be easier than manually going through the entire directory manually 😉
Please ensure that the destination file location of the "zip file for upgrade" has atleast two times more space than the size of the zip file. You can always keep a check on file storage capacity to prevent such errors.
- M 🙂
As per the below KB,
If you are currently running image is 18.1R2 or later:
root@juniper> request system software add /var/tmp/junos-arm-32-18.4R1.8.tgz force unlink no-copy
The force option frees up storage space by removing the non-recovery snapshots if any. The unlink option is available starting 18.1R2. The unlink option will remove the .tgz file immediately after unpacking so that the available image size calculation takes into account the free 300M. Without using unlink, this space is not considered.
If the above step fails, run the software update with the force and unlink option again (it will work the second time in some situations).
The force unlink option has always worked for me if not once then the second time it has always worked.