I've got a strange problem with my EX4550
I get an error saying that the storage is full.
Show system storage displays this
root@ex4550-sheehan> show system storagefpc0:--------------------------------------------------------------------------Filesystem Size Used Avail Capacity Mounted on/dev/da0s1a 340M 129M 183M 41% /devfs 1.0K 1.0K 0B 100% /dev/dev/md0 68M 68M 0B 100% /packages/mnt/jbase/dev/md1 5.8M 1.1M 4.2M 21% /packages/mfs-fips-mode-powerpc/dev/md2 2.9M 2.9M 0B 100% /packages/mnt/fips-mode-powerpc-12.3R5.7/dev/md3 9.0M 4.4M 3.9M 53% /packages/mfs-jcrypto-ex/dev/md4 12M 12M 0B 100% /packages/mnt/jcrypto-ex-12.3R5.7/dev/md5 8.1M 3.5M 4.0M 47% /packages/mfs-jdocs-ex/dev/md6 6.2M 6.2M 0B 100% /packages/mnt/jdocs-ex-12.3R5.7/dev/md7 43M 39M 700K 98% /packages/mfs-jkernel-ex/dev/md8 108M 108M 0B 100% /packages/mnt/jkernel-ex-12.3R5.7/dev/md9 12M 7.9M 3.6M 69% /packages/mfs-jpfe-ex45x/dev/md10 22M 22M 0B 100% /packages/mnt/jpfe-ex45x-12.3R5.7/dev/md11 17M 12M 3.2M 79% /packages/mfs-jroute-ex/dev/md12 38M 38M 0B 100% /packages/mnt/jroute-ex-12.3R5.7/dev/md13 12M 7.2M 3.6M 66% /packages/mfs-jswitch-ex/dev/md14 21M 21M 0B 100% /packages/mnt/jswitch-ex-12.3R5.7/dev/md15 14M 9.5M 3.4M 73% /packages/mfs-jweb-ex/dev/md16 25M 25M 0B 100% /packages/mnt/jweb-ex-12.3R5.7/dev/da0s3e 239M 239M -18.8M 109% /var/dev/md17 126M 12K 116M 0% /tmp/dev/da0s3d 702M 138K 646M 0% /var/tmp/dev/da0s4d 116M 154K 106M 0% /config/dev/md18 118M 22M 87M 20% /var/rundbprocfs 4.0K 4.0K 0B 100% /proc/var/jail/etc 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/etc/var/jail/run 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/run/var/jail/tmp 239M 239M -18.8M 109% /packages/mnt/jweb-ex-12.3R5.7/jail/var/tmp/var/tmp 702M 138K 646M 0% /packages/mnt/jweb-ex-12.3R5.7/jail/var/tmp/uploadsdevfs 1.0K 1.0K 0B 100% /packages/mnt/jweb-ex-12.3R5.7/jail/dev
Which seems to indicate the problem lies somewhere in the /var directory
du -h in /var shows this
root@ex4550-sheehan:RE:0% du -h2.0K ./.snap2.0K ./bin2.0K ./crash2.0K ./cron/tabs4.0K ./cron2.0K ./db/entropy2.0K ./db/ext2.0K ./db/fsad2.0K ./db/gtpcd2.0K ./db/certs/common/key-pair2.0K ./db/certs/common/local2.0K ./db/certs/common/certificate-request2.0K ./db/certs/common/certification-authority2.0K ./db/certs/common/crl 12K ./db/certs/common4.0K ./db/certs/system-key-pair4.0K ./db/certs/system-cert 22K ./db/certs2.0K ./db/commit-queue306K ./db/help2.0K ./db/leases2.0K ./db/scripts/commit2.0K ./db/scripts/event2.0K ./db/scripts/op2.0K ./db/scripts/lib 10K ./db/scripts2.0K ./db/zoneinfo2.0K ./db/persistent_mac2.0K ./db/dhcp_snoop386K ./db4.0K ./etc/filters2.0K ./etc/ssl-/local2.0K ./etc/ssl-/peer2.0K ./etc/ssl-/certificate-authority8.0K ./etc/ssl-2.0K ./etc/ssl/local2.0K ./etc/ssl/peer2.0K ./etc/ssl/certificate-authority8.0K ./etc/ssl2.0K ./etc/ssl+/local2.0K ./etc/ssl+/peer2.0K ./etc/ssl+/certificate-authority8.0K ./etc/ssl+228K ./etc2.0K ./etcroot/ssh 98K ./etcroot2.0K ./home/b4.0K ./home2.0K ./log/ext2.0K ./log/ggsn/gtppcdr4.0K ./log/ggsn2.0K ./log/flowc/failed4.0K ./log/flowc1.2M ./log2.0K ./mfs/var/tmp 82K ./mfs/var/shmipc/if-rtsdb 84K ./mfs/var/shmipc 88K ./mfs/var 90K ./mfs2.0K ./run/ext2.0K ./run/named2.0K ./run/ppp154K ./run2.0K ./sw/pkg4.0K ./sw2.0K ./tmp/.snap2.0K ./tmp/vi.recover2.0K ./tmp/install2.0K ./tmp/pics2.0K ./tmp/gres-tp2.0K ./tmp/rtsdb 14K ./tmp4.0K ./root2.0K ./empty2.0K ./account2.0K ./at/jobs2.0K ./at/spool6.0K ./at2.0K ./backups2.0K ./transfer/config4.0K ./transfer2.0K ./heimdal106K ./jail/etc4.0K ./jail/run2.0K ./jail/tmp/uploads4.0K ./jail/tmp116K ./jail2.0K ./mail2.0K ./msgs2.0K ./named2.0K ./preserve2.0K ./validate2.0K ./logical-systems2.0K ./rwho2.0K ./spool/lock2.0K ./spool/lpd2.0K ./spool/mqueue2.0K ./spool/opielocks2.0K ./spool/output/lpd4.0K ./spool/output 14K ./spool2.0K ./yp2.0K ./rundb/.snap2.0K ./rundb/private 10K ./rundb/sdb/shmem520K ./rundb/sdb 22M ./rundb 24M .
Which doesn't appear to match in terms of size used
Earlier I did have a 230-240Mb httpd.log in /var/log which I deleted - but it doesn't appear to have made a difference in the reported size. Does this require a restart of the switch to update?
Switch is running 12.3R5.7 and has been up for 818 days, so it's probably due a reboot.
Any help would be greatly appreciated
Have you tried request system storage cleanup ?
Yes, tried that.
root@ex4550-sheehan> request system storage cleanupPlease check the list of files to be deleted using the dry-run option. i.e.request system storage cleanup dry-runDo you want to proceed ? [yes,no] (no) yes
fpc0:--------------------------------------------------------------------------error: Rotating files failed: newsyslog: log_trim: fclose: No space left on devi ce
Is the response from this command.
I assume you get the same error without the dry run option attempting the actual clean up.
I would agree that a restart is a next good option to see if the issue clears or at least releases whatever locks are preventing the storage cleanup from working.
I would rather manually delete old logs and then try request system storage cleanup again.
% rm /var/log/*.gz
With full filesystem there is risk that switch will not come up.
if someone still has the same issue try this:
du -sk /var/* | sort -nr | head
It finds files which was not clearly removed.