Junos OS

Expand all | Collapse all

EX3400 /var partition usage is high

Jump to Best Answer
  • 1.  EX3400 /var partition usage is high

    Posted 02-26-2020 06:01

    Hi all,

     

    I have two EX3400-48T switches configured in a VC. I struggle with disk-usage alarms for /var that don't make any sense to me. The system is very new, just the OS was updated to 18.2R3-S2.9 and some configuration was done.

     

    me@XX> show system alarms
    2 alarms currently active
    Alarm time Class Description
    2020-02-26 09:29:42 UTC Minor RE 1 /var partition usage is high
    2020-02-20 22:20:00 UTC Minor FPC 0 /var partition usage is high

    However there should be enough free space from my understanding:

     

    me@XX> show system storage
    fpc0:
    --------------------------------------------------------------------------
    Filesystem Size Used Avail Capacity Mounted on
    /dev/gpt/junos 1.3G 919M 337M 73% /.mount
    tmpfs 805M 48K 805M 0% /.mount/tmp
    tmpfs 324M 420K 324M 0% /.mount/mfs
    fpc1:
    --------------------------------------------------------------------------
    Filesystem Size Used Avail Capacity Mounted on
    /dev/gpt/junos 1.3G 916M 340M 73% /.mount
    tmpfs 564M 48K 564M 0% /.mount/tmp
    tmpfs 324M 524K 324M 0% /.mount/mfs

    Since there is no separate /var partition visible my guess is this is refering to the main partition mounted at /.mount?

     

    Manually set very low thresholds for the alarms but they did not disappear:

     

    me@XX> show configuration chassis disk-partition /var
    level full {
    free-space 50 mb;
    }
    level high {
    free-space 100 mb;
    }

    Any help is appreciated.



  • 2.  RE: EX3400 /var partition usage is high

    Posted 02-26-2020 08:02

    good luck: I'll be following this thread. I have the same issue on and off with my EX2300s . 

    Michel



  • 3.  RE: EX3400 /var partition usage is high

    Posted 02-26-2020 14:30

    @LapointeMichel wrote:

    good luck: I'll be following this thread. I have the same issue on and off with my EX2300s . 

    Michel


    The EX2300's disk is way too small, unfortunately. I often can't even install an update without using a USB stick, which is not fun for a remote switch.

     

    There's a few tips here that may help:

    https://networkdirection.net/articles/routingandswitching/juniper-routers-and-switches/junos-cleanup-storage-space/



  • 4.  RE: EX3400 /var partition usage is high
    Best Answer

     
    Posted 02-26-2020 08:50

    Hi Bananarama,

     

    Have you tried rebooting the VC for the system to recalculate?

     

    In case you want to free some space try the following:

     

    Execute storage clean up.

    root@juniper> request system storage cleanup all-members

     

    Delete the snapshot on all members:

    root@juniper> request system snapshot delete * all-members


    Check for residual packages from old codes in the /packages/db/ directory and manually delete them.
    Note: This is needed for each member.

    Example. The current version this device is running is 181_r3_s4 and the previous version was: 151_x53_d59
    root@juniper> start shell user root
    root@juniper% cd /packages/db/
    root@juniper>% ls -l
    total 224
    drwxr-xr-x 3 root wheel 512 Feb 12 20:34 jdocs-ex-arm-32-20190315.110124_builder_junos_181_r3_s4
    drwxr-xr-x 3 root wheel 512 Feb 12 20:34 jpfe-EX34XX-arm-32-20190315.110124_builder_junos_181_r3_s4
    drwxr-xr-x 3 root wheel 512 Feb 12 20:34 jsd-arm-32-18.1R3-S4.2-jet-1
    drwxr-xr-x 4 root wheel 512 Feb 12 20:34 jsdn-arm-32-18.1R3-S4.2
    drwxrwxrwx 5 root wheel 512 Feb 12 19:27 junos-arm-32-15.1X53-D59.4---> old file
    drwxr-xr-x 4 root wheel 512 Feb 12 20:34 junos-arm-32-18.1R3-S4.2
    drwxr-xr-x 3 root wheel 512 Feb 12 20:34 junos-daemons-arm-32-20190315.110124_builder_junos_181_r3_s4
    drwxr-xr-x 4 root wheel 512 Feb 12 20:35 junos-dp-crypto-support-ex-arm-32-20190315.110124_builder_junos_181_r3_s4
    drwxr-xr-x 3 root wheel 512 Feb 12 19:33 junos-dp-crypto-support-mtx-arm-32-20180914.204810_builder_junos_151_x53_d59--> old file

    Delete old files using the following command:

    root@juniper% rm -R [filename]

    Example:


    root@juniper% rm -R junos-arm-32-15.1X53-D59.4


    root@juniper% rm -R junos-dp-crypto-support-mtx-arm-32-20180914.204810_builder_junos_151_x53_d59

     


    Regards,
    Lil Dexx
    JNCIE-ENT#863, 3X JNCIP-[SP-ENT-DC], 4X JNCIA [cloud-DevOps-Junos-Design], Champions Ingenius, SSYB

     

    If this solves your problem, please mark this post as "Accepted Solution" so we can help others too \:)/

     



  • 5.  RE: EX3400 /var partition usage is high

    Posted 02-27-2020 01:44

    Thank you all for your replies.

    @LilDexx wrote:

    Have you tried rebooting the VC for the system to recalculate?




     

    I rebooted the device ealier and it did not help. Cannot reboot it again because it is a production system.


    @LilDexx wrote:

    In case you want to free some space try the following:

     

    Execute storage clean up.

    root@juniper> request system storage cleanup all-members

     

    Delete the snapshot on all members:

    root@juniper> request system snapshot delete * all-members


     

    That was done before already and did not free any significant space.

     


    @LilDexx wrote:

    Check for residual packages from old codes in the /packages/db/ directory and manually delete them.
    Note: This is needed for each member.

    Example. The current version this device is running is 181_r3_s4 and the previous version was: 151_x53_d59
    root@juniper> start shell user root
    root@juniper% cd /packages/db/
    root@juniper>% ls -l


     

    Thanks for this hint. That actually freed 226MB of space on each device. Shortly afterwards the alarms were cleared. As this solved the problem for me, I will select your answer as the selected answer. Thanks!

     

    However one question remains to me: Why did the threshold of 50mb for the full disk alarm not work? There were still 337mb free, no reason at all for the alarms to fire. Does anyone have an idea?



  • 6.  RE: EX3400 /var partition usage is high

     
    Posted 02-26-2020 19:48

    Hi,

     

    You may check the below KB for directions to troubleshoot

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB30860

     

    If you dont have any large size file in /var and the you still have high utilization then it may be because of the memory bloating. In that case a power cycle of device can help to bring the /var utlization down.

     

    Hope this helps