Switching

 View Only
last person joined: 3 days ago 

Ask questions and share experiences about EX and QFX portfolios and all switching solutions across your data center, campus, and branch locations.
  • 1.  EX4550-32F Host 0 /var partition is full

    Posted 11-23-2016 21:10

     I've got a strange problem with my EX4550

     

    I get an error saying that the storage is full.

     

    Show system storage displays this

     

    root@ex4550-sheehan> show system storage
    fpc0:
    --------------------------------------------------------------------------
    Filesystem Size Used Avail Capacity Mounted on
    /dev/da0s1a 340M 129M 183M 41% /
    devfs 1.0K 1.0K 0B 100% /dev
    /dev/md0 68M 68M 0B 100% /packages/mnt/jbase
    /dev/md1 5.8M 1.1M 4.2M 21% /packages/mfs-fips-mode-powerpc
    /dev/md2 2.9M 2.9M 0B 100% /packages/mnt/fips-mode-powerpc-12.3R5.7
    /dev/md3 9.0M 4.4M 3.9M 53% /packages/mfs-jcrypto-ex
    /dev/md4 12M 12M 0B 100% /packages/mnt/jcrypto-ex-12.3R5.7
    /dev/md5 8.1M 3.5M 4.0M 47% /packages/mfs-jdocs-ex
    /dev/md6 6.2M 6.2M 0B 100% /packages/mnt/jdocs-ex-12.3R5.7
    /dev/md7 43M 39M 700K 98% /packages/mfs-jkernel-ex
    /dev/md8 108M 108M 0B 100% /packages/mnt/jkernel-ex-12.3R5.7
    /dev/md9 12M 7.9M 3.6M 69% /packages/mfs-jpfe-ex45x
    /dev/md10 22M 22M 0B 100% /packages/mnt/jpfe-ex45x-12.3R5.7
    /dev/md11 17M 12M 3.2M 79% /packages/mfs-jroute-ex
    /dev/md12 38M 38M 0B 100% /packages/mnt/jroute-ex-12.3R5.7
    /dev/md13 12M 7.2M 3.6M 66% /packages/mfs-jswitch-ex
    /dev/md14 21M 21M 0B 100% /packages/mnt/jswitch-ex-12.3R5.7
    /dev/md15 14M 9.5M 3.4M 73% /packages/mfs-jweb-ex
    /dev/md16 25M 25M 0B 100% /packages/mnt/jweb-ex-12.3R5.7
    /dev/da0s3e 239M 239M -18.8M 109% /var
    /dev/md17 126M 12K 116M 0% /tmp
    /dev/da0s3d 702M 138K 646M 0% /var/tmp
    /dev/da0s4d 116M 154K 106M 0% /config
    /dev/md18 118M 22M 87M 20% /var/rundb
    procfs 4.0K 4.0K 0B 100% /proc
    /var/jail/etc 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/etc
    /var/jail/run 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/run
    /var/jail/tmp 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/tmp
    /var/tmp 702M 138K 646M 0% /packages/mnt/jweb-ex-12.3R5.7/jail/var/tmp/uploads
    devfs 1.0K 1.0K 0B 100% /packages/mnt/jweb-ex-12.3R5.7/jail/dev

     

    Which seems to indicate the problem lies somewhere in the /var directory

     

    du -h in /var shows this

     

    root@ex4550-sheehan:RE:0% du -h
    2.0K ./.snap
    2.0K ./bin
    2.0K ./crash
    2.0K ./cron/tabs
    4.0K ./cron
    2.0K ./db/entropy
    2.0K ./db/ext
    2.0K ./db/fsad
    2.0K ./db/gtpcd
    2.0K ./db/certs/common/key-pair
    2.0K ./db/certs/common/local
    2.0K ./db/certs/common/certificate-request
    2.0K ./db/certs/common/certification-authority
    2.0K ./db/certs/common/crl
    12K ./db/certs/common
    4.0K ./db/certs/system-key-pair
    4.0K ./db/certs/system-cert
    22K ./db/certs
    2.0K ./db/commit-queue
    306K ./db/help
    2.0K ./db/leases
    2.0K ./db/scripts/commit
    2.0K ./db/scripts/event
    2.0K ./db/scripts/op
    2.0K ./db/scripts/lib
    10K ./db/scripts
    2.0K ./db/zoneinfo
    2.0K ./db/persistent_mac
    2.0K ./db/dhcp_snoop
    386K ./db
    4.0K ./etc/filters
    2.0K ./etc/ssl-/local
    2.0K ./etc/ssl-/peer
    2.0K ./etc/ssl-/certificate-authority
    8.0K ./etc/ssl-
    2.0K ./etc/ssl/local
    2.0K ./etc/ssl/peer
    2.0K ./etc/ssl/certificate-authority
    8.0K ./etc/ssl
    2.0K ./etc/ssl+/local
    2.0K ./etc/ssl+/peer
    2.0K ./etc/ssl+/certificate-authority
    8.0K ./etc/ssl+
    228K ./etc
    2.0K ./etcroot/ssh
    98K ./etcroot
    2.0K ./home/b
    4.0K ./home
    2.0K ./log/ext
    2.0K ./log/ggsn/gtppcdr
    4.0K ./log/ggsn
    2.0K ./log/flowc/failed
    4.0K ./log/flowc
    1.2M ./log
    2.0K ./mfs/var/tmp
    82K ./mfs/var/shmipc/if-rtsdb
    84K ./mfs/var/shmipc
    88K ./mfs/var
    90K ./mfs
    2.0K ./run/ext
    2.0K ./run/named
    2.0K ./run/ppp
    154K ./run
    2.0K ./sw/pkg
    4.0K ./sw
    2.0K ./tmp/.snap
    2.0K ./tmp/vi.recover
    2.0K ./tmp/install
    2.0K ./tmp/pics
    2.0K ./tmp/gres-tp
    2.0K ./tmp/rtsdb
    14K ./tmp
    4.0K ./root
    2.0K ./empty
    2.0K ./account
    2.0K ./at/jobs
    2.0K ./at/spool
    6.0K ./at
    2.0K ./backups
    2.0K ./transfer/config
    4.0K ./transfer
    2.0K ./heimdal
    106K ./jail/etc
    4.0K ./jail/run
    2.0K ./jail/tmp/uploads
    4.0K ./jail/tmp
    116K ./jail
    2.0K ./mail
    2.0K ./msgs
    2.0K ./named
    2.0K ./preserve
    2.0K ./validate
    2.0K ./logical-systems
    2.0K ./rwho
    2.0K ./spool/lock
    2.0K ./spool/lpd
    2.0K ./spool/mqueue
    2.0K ./spool/opielocks
    2.0K ./spool/output/lpd
    4.0K ./spool/output
    14K ./spool
    2.0K ./yp
    2.0K ./rundb/.snap
    2.0K ./rundb/private
    10K ./rundb/sdb/shmem
    520K ./rundb/sdb
    22M ./rundb
    24M .

     

    Which doesn't appear to match in terms of size used

     

    Earlier I did have a 230-240Mb httpd.log in /var/log which I deleted - but it doesn't appear to have made a difference in the reported size. Does this require a restart of the switch to update?

     

    Switch is running 12.3R5.7 and has been up for 818 days, so it's probably due a reboot.

     

    Any help would be greatly appreciated



  • 2.  RE: EX4550-32F Host 0 /var partition is full

     
    Posted 11-24-2016 00:38

    Have you tried request system storage cleanup ?



  • 3.  RE: EX4550-32F Host 0 /var partition is full

    Posted 11-24-2016 13:26

    Yes, tried that. 

     

    root@ex4550-sheehan> request system storage cleanup
    Please check the list of files to be deleted using the dry-run option. i.e.
    request system storage cleanup dry-run
    Do you want to proceed ? [yes,no] (no) yes

    fpc0:
    --------------------------------------------------------------------------
    error: Rotating files failed: newsyslog: log_trim: fclose: No space left on devi ce

    {master:0}

     

    Is the response from this command.



  • 4.  RE: EX4550-32F Host 0 /var partition is full

    Posted 11-25-2016 04:03

    I assume you get the same error without the dry run option attempting the actual clean up.

     

    I would agree that a restart is a next good option to see if the issue clears or at least releases whatever locks are preventing the storage cleanup from working.



  • 5.  RE: EX4550-32F Host 0 /var partition is full

     
    Posted 11-28-2016 05:28

    I would rather manually delete old logs and then try request system storage cleanup again.

    % rm /var/log/*.gz

    With full filesystem there is risk that switch will not come up.



  • 6.  RE: EX4550-32F Host 0 /var partition is full

    Posted 11-13-2018 14:11

    Hey guys,

    if someone still has the same issue try this:

     

    du -sk /var/* | sort -nr | head

     

    It finds files which was not clearly removed.