Junos OS

 View Only
last person joined: 2 days ago 

Ask questions and share experiences about Junos OS.
Back to discussions
Expand all | Collapse all

The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

  • 1.  The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-15-2021 12:45
    Hi All,

    I am using EX4300: Model: ex4300-48t Junos: 18.4R3-S1.3
    Today i cannot upgrade to version 20.4R3.8 ( Or 20.2). This is error:
    "root> ... add /var/tmp/jinstall-ex-4300-20.2R3.9-signed.tgz reboot
    NOTICE: Validating configuration against jinstall-ex-4300-20.2R3.9-signed.tgz.
    NOTICE: Use the 'no-validate' option to skip this if desired.
    The /var/tmp filesystem on JUNOS is low on free disk space.
    This package requires 12MB more space free.
    ERROR: Please use the request system storage cleanup command to delete old files from /var/tmp"

    I tried to "request system zeroize" successfully before upgrade but i met the same error.
    "request system storage cleanup" is not useful.
    I am looking forward to hearing from you soon.
    Here is show system storage:

    Best Regards,
    Ket, Vo Van.

    ------------------------------
    KET VO
    ------------------------------


  • 2.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-15-2021 20:02
    You can try the no-copy option during install which will reduce the space required for the install process.

    If that is not enough then it will require crawling the directories and deleting files.  There are a number of general kb on the topic in the Juniper documentation but they all seem to avoid specific file recommendations for the most part.

    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
    http://puluka.com/home
    ------------------------------

    Original Message


  • 3.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-16-2021 05:44
    Hi Steve Puluka,

    Option no-copy and unlink is the same.
    Which file should i delete? this is test switch before using for production.
    Thank so much.

    Best Regards,
    Ket, Vo van.

    ------------------------------
    KET VO
    ------------------------------

    Original Message


  • 4.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-16-2021 11:50
    The listed files all seem required to me but I'm not a BSD or Linux expert.  I'd say your best bet is to look at the detail list under /var/tmp as a start.

    Another way to try is to
    delete the upgrade file locally in var/tmp
    have this file on a usb drive
    mount the usb and install from there with the no-copy option

    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
    http://puluka.com/home
    ------------------------------

    Original Message


  • 5.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-17-2021 05:23

    Hi Steve Puluka,

    I used option: no-copy but it didn't working.
    --> The same result:  "This package requires 12MB more space free."
    Thanks,
    Ket, vo van.



    ------------------------------
    KET VO
    ------------------------------

    Original Message


  • 6.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-18-2021 09:03
    Hi Ket, 

    Did you try to re-install JunOS from the loader> prompt ? 
    Put the package on a usb key and connect it on the EX
    reboot and stop the boot with [space] 
    on loader> prompt use the command "install --format file:///<package>"

    It will re-install the JunOS and reboot 
    But warning all the configuration will be lost, but i see that you already try a "zeroise". 

    regards, 
    Théo

    ------------------------------
    THEO QUENNEHEN
    ------------------------------

    Original Message


  • 7.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-19-2021 05:52
    Hi Theo,

    On loader mode is a great option. It worked rapidly with successfully upgrading.
    Thank so much.
    Thank all.

    Reguards,
    Ket, Vo Van.

    ------------------------------
    KET VO
    ------------------------------

    Original Message


  • 8.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-18-2021 12:01
    Have you tried: request system storage cleanup?

    If you feel nervous about it you can append "Dry-run" and it'll tell you what it would do. I think it'll ask for file-by-file confirmation but don't have access to anything right now.

    If you feel inclined, you can also read:
    https://www.juniper.net/documentation/us/en/software/junos/junos-install-upgrade/topics/ref/command/request-system-storage-cleanup.html

    ------------------------------
    BEN BIGGIE
    ------------------------------

    Original Message


  • 9.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 10-21-2021 12:13
    I have the same issue on 2 different 4300-MP stacks.  Support recommended deleting a /var/tmp/.scheme-cache file, but neither stack has that file on it.
    Going back for round two of Support.
    Original Message


  • 10.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 05-04-2022 06:06
    Hi,

    We're having the same issues... What was the outcome for you guys?

    Cheers,
    Dante
    ​
    Original Message


  • 11.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 05-16-2022 19:50

    Hi Dante,

    I'm currently struggling with a stack of EX3400s and it looks the outcome will be upgrading via USB, this has always worked for me.

    FAT32 USB with the OS-package directly on it.

    Here is a guide if you need to prepare it on an production environment for a later reboot (not using boot loader), if you have a VC stack first run 'request session member <member id>' and add 'member <member id>' to the 'request system software add...' line to upgrade each invidually, dont forget to unmount between each switch.

    #Enter the shell
start shell
#run ls before inserting USB, this way you will know which partition is the USB.
ls /dev/da*
#Navigate to var/tmp and create a directory for the USB.
cd /var/tmp
mkdir usb
#Check the USB partition, note which one is new
ls /dev/da*
#Mount the usb folder to the USB (da1s1 in this instance)
mount -t msdosfs /dev/da1s1 /var/tmp/usb
#Double check path
mount | grep usb
#Run the following
request system software add /var/tmp/<os-package.tgz> no-copy no-validate force
#Unmount
start shell
su
umount /dev/da1s1


    ------------------------------
    Andreas
    ------------------------------

    Original Message


  • 12.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 05-18-2022 15:02
    Hi Andreas,

    Thank you for the feedback. I'm aware of the EX3400 storage issues, those have been around for a long time, and multiple options to get it working are available. The USB option works, however very inconvenient for remote upgrades ;-).

    However, this thread is about the EX4300, which haven't shown any issues until recently. A colleague tried to upgrade an EX4300 VC and when the master started to unpack the tgz on the members, the upgrade failed stating the member that failed was a few MB storage short in order to be able to complete the upgrade.

    We copied the software package manually from the master to the other VC members, logged in the members individually and started an upgrade with the no-copy option for the local member only. After all the members were ready, the VC was rebooted and upgraded fine.

    I found it dissappointing the storage seems to have become an issue another hardware type next to EX3400 and EX2300.

    FYI: I noticed that older QFX51xx and EX46xx platforms are not able to download a file an upgrade package to /var/tmp, even though they have plenty of space. The problem here was that the file copy command puts a temporary download file in the home directory of the logged in user and the partition where the home directory resides is too small. Hence the file copy fails with the message "not enough storage".
    I find it hard to understand Juniper has not fixed this yet (this issue has been there for years), seems like an easy fix to me: just do not make a temporary copy in the home dir...
    Or maybe I'm just the only one downloading images that way? :-P

    Storage and Juniper switches, not the best story so far.
    But I'm still a Juniper fan nevertheless :-)

    Cheers,
    Dante
    Original Message


  • 13.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 06-01-2022 12:13
    Minor update: we found out a little optimization is possible. It is not necessary to copy the file from the master to the other VC members. Instead you can initiate the upgrade from the master for each member with the no-copy option:


    request system software add /var/tmp/package.tgz no-copy member <VC member number>

    and repeat this for every member in the VC. Then reboot the VC as a whole and done!


    Original Message


  • 14.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 06-24-2022 14:24
    I have been following this thread and have something to add.  I have a number of EX2300 and EX3400 with the same issue.  I decided to open a JTAC case and this is what they did to get around the issue.

    1. Run the request system storage cleanup command. This will remove any unnecessary files but keep everything under /root intact:
    root@juniper> request system storage cleanup

    2. Run the request system snapshot delete snap* command. This will delete any existing recovery snapshots that are stored on the system.
    root@juniper> request system snapshot delete snap*

    Note: If you are unable to delete the recovery.ufs snapshot, please follow the steps under KB32170 - Recovery and Non-recovery snapshots on Junos devices .

    3. Copy os-package and package-hooks-ex to /var/tmp .

    4. Install the two packages by using the request system software add command:
    root@juniper> request system software add /var/tmp/os-package.tgz
    root@juniper> request system software add /var/tmp/package-hooks-ex.tgz

    5. Install the Junos OS release software by adding options no-copy , unlink , and force under the request system software add command:
    root@juniper> request system software add /var/tmp/<junos-package> force unlink

    6. After reboot, remove the interim packages by issuing the commands "request system software delete os-package" and "request system software delete package-hooks-platform"

    7. Reboot

    I do not know if the files os-package.tgz and package-hooks-ex.tgz are available for download somewhere outside of getting the direct link from support.

    I was also told that Junos version 19 and newer, using "request system software add /var/tmp/<junos-package> unlink no-copy" will update the switches without the not enough space error.

    ------------------------------
    PAUL PETERSON
    ------------------------------

    Original Message


  • 15.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 06-27-2022 06:02
    Hi Paul,

    Thanks for the feedback.

    I'd like to just point out this thread is about EX4300, which is running on a different hardware platform than EX2300/EX3400. The latter are running on ARM and the former is -uhm- PowerPC (I think... don't shoot me if I'm wrong).

    The storage issues on EX2300/EX3400 go way back, and are a known issue from at least 2016: check the creation date of this KB article. Also, I don't think you can use those "helper" packages on EX4300. I would not advise to do so.

    I haven't seen storage issues on EX4300 until recently. In my experience they show it when they have been operational for a while and are still running  an 18-ish Junos version from which I want to upgrade.

    As for the EX2300/EX3400: I see the storage issue happening in 95% of the cases, no matter what circumstances.

    Cheers,
    Dante
    Original Message


  • 16.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 06-27-2022 06:02
    Hi Paul,

    I forgot to add something: in case the steps in the  KB are not helping and you're not up for a format install, check this post. Saved me on many occasions :-)

    Cheers,
    Dante
    Original Message


  • 17.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 06-27-2022 11:18
    Paul Peterson's EX3400 memory fixes should handle the extreme cases. But with our EX3400s, we found that after a firmware update is installed, storage used is increased from about ~36% up to ~74%, which prevents a further update.

    Writing a new recovery snapshot resolves this situation, and gets the storage used back down to ~36%. 
      request system snapshot recovery

    Worth trying the other simple cleanups, but if storage took a big hit after a version update, this may be the fix.

    Steve Bohrer  (he, him, his)
    IT Infrastructure, Emerson College
    617-824-8523



    Original Message


  • 18.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 07-14-2022 06:20
    Had some issues with a 4300 today going from 18.x to 20. Fix for us was at root shell (linux cli not JUNOS cli) , cd /var/tmp, and du -hs * and du -hs .* 

    We had some old hidden directories that were taking just enough space to be problematic, apparently left over from trying JWeb on this switch long ago. rm -rf the offending directory made enough space for us to scp the file over into /var/tmp on the switch, and then do  request system software add unlink no-copy /var/tmp/jfilename...

    ------------------------------
    Steve Bohrer
    ------------------------------

    Original Message


  • 19.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 06-28-2022 16:01
    Alright.  I am running 18.4R3 on my 4300's in a 2 switch virtual chassis and am having what seems to be the same issue.  I was able to get to 19.4 but that was doing four separate, small updates.  And now I cant seem to go anywhere else.  Is this a known issue?  I haven't seen anything else on this.  If anyone has found a KB or knows of a work around I would appreciate it.

    Mitch

    ------------------------------
    Robert Mitchell
    ------------------------------

    Original Message


  • 20.  RE: The /var/tmp filesystem on JUNOS is low on free disk space when upgrading on EX4300

    Posted 06-29-2022 10:36
    Before update you probably want to remove the various hidden packages, and check what undeleted/not properly managed/not cleaned packages were left behind.

    Login and try some
    request system snapshot delete snap.*
    and
    request system snapshot delete previous


    And for the abandoned  packages from previous/old JunOS:

    file list /packages/db/ detail

    You might see some stuff with older version numbers..

    ------------------------------
    Olivier Benghozi
    ------------------------------

    Original Message