I'm trying to install a network of SSRs on NFX350 devices, but the storage constraint on those devices for VMs is problematic (19 GB). Within a few days, the VM out grows the available storage that is built into the NFX350 hardware. Has anyone implemented the SSR in a similar limited storage environment? I would assume I need to have a daily cron job running on the SSR to limit (delete) logs and such or prune database/cache files to keep the VM from exceeding about 16 GB. What should I be looking at cleaning daily/hourly?
For reference, a day after install, one device went from 12G available to 4.5G available and searching for larger than 10M files wasn't promising:
# find / -size +10M -ls 2>/dev/null
23 23152 -rw------- 1 root root 23704124 Nov 9 2021 /boot/initramfs-3.10.0-514.el7.x86_64.img
20 49120 -rw------- 1 root root 50298719 Aug 8 23:52 /boot/initramfs-3.10.0-862.el7.x86_64.img
21 45724 -rw------- 1 root root 46820351 Nov 9 2021 /boot/initramfs-0-rescue-92953f1920ad4e5eacafe3317bbce6a5.img
335 13532 -rw------- 1 root root 13854223 Dec 14 17:29 /boot/initramfs-4.18.0-348.20.1.el8_5.x86_64kdump.img
339 54944 -rw------- 1 root root 56262206 Aug 8 23:53 /boot/initramfs-4.18.0-348.20.1.el8_5.x86_64.img
4026532029 0 -r-------- 1 root root 140737477885952 Dec 15 17:23 /proc/kcore
144091 26904 -rwxr-xr-x 1 root root 27548765 Sep 14 2018 /usr/bin/influxd
138081 72676 -rwxr-xr-x 1 root root 74419096 Jul 1 2021 /usr/bin/node
149514 72292 -rwxr-xr-x 1 root root 74024256 Feb 10 2022 /usr/bin/telegraf
144088 15356 -rwxr-xr-x 1 root root 15723706 Sep 14 2018 /usr/bin/influx_inspect
144090 19200 -rwxr-xr-x 1 root root 19658725 Sep 14 2018 /usr/bin/influx_tsm
143853 20300 -rwxr-xr-x 1 root root 20785824 Mar 21 2020 /usr/lib64/libicudata.so.50.2
401948 25048 -rwxr-xr-x 1 root root 25647740 Apr 4 2022 /usr/lib/128T-quickstarter/par/quickstarter.par654306
14540 -rwxr-xr-x 1 root root 14887194 Mar 21 2022 /usr/lib/128T-minion-watchdog/par/minion_watchdog.par
133742 103660 -rw-r--r-- 1 root root 106176928 Aug 8 23:43 /usr/lib/locale/locale-archive
791941 13400 -rwxr-xr-x 1 root root 13721584 Aug 9 2021 /usr/lib/firmware/mrvl/prestera/mvsw_prestera_fw-v3.0.img
791940 13368 -rwxr-xr-x 1 root root 13686596 Aug 9 2021 /usr/lib/firmware/mrvl/prestera/mvsw_prestera_fw-v2.0.img
400763 15156 -rw-r--r-- 1 root root 15515796 Aug 9 2021 /usr/lib/firmware/qcom/sm8250/adsp.mbn
786760 17568 -rwxr-xr-x 1 root root 17985751 Jul 7 2021 /usr/lib/128T-cloud-bootstrapper/par/cloud-bootstrapper.par
1188346 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_run/pypi__36__salt_3002_8/share/man/man7/salt.7
1048099 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/event_publisher/pypi__36__salt_3002_8/share/man/man7/salt.7
1179558 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_key/pypi__36__salt_3002_8/share/man/man7/salt.7
1176645 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_api/pypi__36__salt_3002_8/share/man/man7/salt.7
1052420 14016 -rw-r--r-- 1 root root 14350485 Aug 8 23:47 /usr/lib/128technology/unpar/pcli/com_128technology_i95/python/pcli/pcli/data/consolidatedStatsModel.json
1303837 17460 -rw-r--r-- 1 root root 17878031 Aug 8 23:47 /usr/lib/128technology/unpar/yin-and-friends/com_128technology_i95/python/yinandfriends/yinandfriends/data/scripts/pyang
1185433 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_master/pypi__36__salt_3002_8/share/man/man7/salt.7
1182520 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_main/pypi__36__salt_3002_8/share/man/man7/salt.7
794298 29652 -rwxr-xr-x 1 root root 30362443 Aug 8 21:00 /usr/lib/128technology/par/initialize128t.par
794307 83496 -rwxr-xr-x 1 root root 85498009 Aug 8 21:00 /usr/lib/128technology/par/salt_run.par
794303 83496 -rwxr-xr-x 1 root root 85497839 Aug 8 21:00 /usr/lib/128technology/par/salt_api.par
794309 11932 -rwxr-xr-x 1 root root 12214621 Aug 8 21:00 /usr/lib/128technology/par/t128_analytics.par
794312 32052 -rwxr-xr-x 1 root root 32820459 Aug 8 21:00 /usr/lib/128technology/par/yin-and-friends.par
794302 58416 -rwxr-xr-x 1 root root 59817335 Aug 8 21:00 /usr/lib/128technology/par/pcli.par
794305 83496 -rwxr-xr-x 1 root root 85497981 Aug 8 21:00 /usr/lib/128technology/par/salt_main.par
794304 83496 -rwxr-xr-x 1 root root 85497801 Aug 8 21:00 /usr/lib/128technology/par/salt_key.par
785916 18528 -rwxr-xr-x 1 root root 18972400 Apr 15 2022 /usr/lib/128technology/par/snmp_object_agent.par
794291 90012 -rwxr-xr-x 1 root root 92168784 Aug 8 21:00 /usr/lib/128technology/par/event_publisher.par
794311 14484 -rwxr-xr-x 1 root root 14828116 Aug 8 21:00 /usr/lib/128technology/par/yin-and-enemies.par
794306 83496 -rwxr-xr-x 1 root root 85498017 Aug 8 21:00 /usr/lib/128technology/par/salt_master.par
794294 13540 -rwxr-xr-x 1 root root 13862705 Aug 8 21:00 /usr/lib/128technology/par/generatePasswordHash.par
791128 17672 -rwxr-xr-x 1 root root 18092272 Oct 27 2021 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so
791202 21900 -r--r--r-- 1 root root 22425600 Aug 8 23:45 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/classes.jsa
791184 72120 -rw-r--r-- 1 root root 73848490 Oct 27 2021 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/rt.jar
654309 14348 -rwxr-xr-x 1 root root 14691241 Aug 31 2021 /usr/libexec/128T-rappid-downloader/par/rappid_downloader.par
147478 11196 -rwxr-xr-x 1 root root 11461200 Jul 16 14:10 /usr/libexec/128T-cloud-intel-agent
149530 13724 -rwxr-xr-x 1 root root 14050512 Jul 16 14:10 /usr/libexec/128T-mist-agent
396633 31252 -rwxr-xr-x 1 root root 32000063 Jun 7 2022 /usr/libexec/128T-installer/par/install128t.par
396632 14136 -rwxr-xr-x 1 root root 14472661 Jun 7 2022 /usr/libexec/128T-installer/par/erase128t.par
402268 13544 -rwxr-xr-x 1 root root 13867381 Aug 2 17:39 /usr/libexec/128T-mist-wan-assurance/par/generate_pillar.par
39 77676 -rw-r--r-- 1 root root 79536128 Aug 8 23:52 /var/lib/rpm/Packages
923111 16364 -rw-r--r-- 1 root root 16755213 Jun 7 2022 /var/lib/128T-rappid-data/application-classifications.json
795147 27528 -rw-rwxr-- 1 root root 67108880 Dec 14 22:42 /var/lib/128technology/pds/logs/version-2/log.1795251
21900 -rw-r--r-- 1 root root 67108880 Dec 15 02:07 /var/lib/128technology/pds/logs/version-2/log.17a11
795260 24996 -rw-r--r-- 1 root root 67108880 Dec 15 17:23 /var/lib/128technology/pds/logs/version-2/log.68a37
795254 24132 -rw-r--r-- 1 root root 67108880 Dec 15 05:52 /var/lib/128technology/pds/logs/version-2/log.2a585
787030 27148 -rw-r--r-- 1 root root 67108880 Dec 15 10:07 /var/lib/128technology/pds/logs/version-2/log.3ef78
795258 21612 -rw-r--r-- 1 root root 67108880 Dec 15 13:29 /var/lib/128technology/pds/logs/version-2/log.562b4
660405 103652 -rwxr-xr-x 1 root root 106134674 Dec 15 16:47 /var/lib/128technology/tank/metrics/0/381114086-389712383_1671122870.ilog
660420 103680 -rwxr-xr-x 1 root root 106161405 Dec 15 17:17 /var/lib/128technology/tank/metrics/0/389712384-398311086_1671124675.ilog
660422 18956 -rwxr-xr-x 1 root root 19407404 Dec 15 17:23 /var/lib/128technology/tank/metrics/0/398311087_1671124680.log
924988 10248 -rw-r--r-- 1 root root 10487680 Dec 15 16:45 /var/lib/128technology/influxdb/wal/t128/IntermediateMetrics/13/_00013.wal
924987 10308 -rw-r--r-- 1 root root 10552690 Dec 15 17:21 /var/lib/128technology/influxdb/wal/t128/ShortMetrics/30/_00013.wal
786748 24512 -rw-r----- 1 root root 25100288 Aug 8 23:51 /var/cache/yum/x86_64/7/128t-authenticated-release-deps-centos-7-5/gen/primary.xml.sqlite
786747 21508 -rw-r----- 1 root root 22023867 Aug 4 21:38 /var/cache/yum/x86_64/7/128t-authenticated-release-deps-centos-7-5/gen/primary.xml
2085 92940 -rw-r--r-- 1 root root 95164416 Aug 4 15:39 /var/cache/yum/x86_64/7/updates/gen/primary_db.sqlite
2081 16836 -rw-r--r-- 1 root root 17232987 Aug 4 15:39 /var/cache/yum/x86_64/7/updates/d2ee620b433edae5b49be8b11749510e6288a8efaf2b1da17a2dd8ffa6e8c4dd-primary.sqlite.bz2
391035 30544 -rw-r--r-- 1 root root 31272960 Aug 7 04:45 /var/cache/yum/x86_64/7/epel/gen/primary_db.sqlite
391036 19248 -rw-r--r-- 1 root root 19707427 Aug 7 05:07 /var/cache/yum/x86_64/7/epel/gen/updateinfo.xml786739
54172 -rw-r----- 1 root root 55465984 Aug 3 13:32 /var/cache/yum/x86_64/7/128t-authenticated-legacy-release/gen/primary_db.sqlite
786750 21180 -rw-r----- 1 root root 21682176 Aug 8 23:51 /var/cache/yum/x86_64/7/128t-authenticated-alpha/gen/primary.xml.sqlite
786749 15480 -rw-r----- 1 root root 15851320 Aug 8 23:36 /var/cache/yum/x86_64/7/128t-authenticated-alpha/gen/primary.xml
786752 21592 -rw-r----- 1 root root 22106112 Aug 8 23:51 /var/cache/yum/x86_64/7/128t-authenticated-alpha-deps-centos-7-5/gen/primary.xml.sqlite
786751 18840 -rw-r----- 1 root root 19290993 Aug 8 20:36 /var/cache/yum/x86_64/7/128t-authenticated-alpha-deps-centos-7-5/gen/primary.xml
2083 30880 -rw-r--r-- 1 root root 31614976 Oct 29 2020 /var/cache/yum/x86_64/7/base/gen/primary_db.sqlite
263006 10584 -rw-r----- 1 root root 10833673 Aug 8 23:53 /var/cache/dnf/128t-installer-centos-7-5-epel-c5115b886d38810a/repodata/c6d5cc45a62f4177f681a08f57bdee0bf9f0c12ee2ffcee6bb4e443e1bd6c825-filelists.xml.gz
391124 10980 -rw-r----- 1 root root 11240437 Aug 8 23:53 /var/cache/dnf/128t-installer-centos-7-5-epel-filenames.solvx
107 23152 -rw------- 1 root root 23703067 Dec 14 17:43 /var/log/messages
1755 14012 -rw------- 1 root root 14342705 Dec 11 00:01 /var/log/messages-20221211
924979 10296 -rw-rw-rw- 1 root root 10541582 Dec 15 04:01 /var/log/128technology/influxdb_http.2.log
924990 10928 -rw-rw-rw- 1 root root 11186632 Dec 15 15:01 /var/log/128technology/influxdb_http.1.log
1957 71532 -rw------- 1 root root 73243909 Nov 9 2021 /var/log/anaconda/journal.log
399092 49156 -rw-r----- 1 root systemd-journal 50331648 Dec 15 17:23 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/system.journal
391123 131080 -rw-r-x--- 1 root systemd-journal 134217728 Dec 12 02:28 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/
system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-00000000000035d5-0005ef573c43596c.journal398388 131080 -rw-r-x--- 1 root systemd-journal 134217728 Dec 15 02:50 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/
system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-000000000002ebca-0005ef984250c521.journal390927 16392 -rw-r-x--- 1 root systemd-journal 16777216 Dec 8 20:53 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/
system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-0000000000000001-0005e5c34e5cf778.journal391042 14016 -rw-r--r-- 1 root root 14350485 Dec 14 17:44 /var/model/consolidatedStatsModel.json
152398 4194304 -rw------- 1 root root 4294967296 Dec 14 18:25 /dev/hugepages/nfx-gr-blk-ssr-router1map_2
531581 110368 -rw-r--r-- 1 root root 113045344 Aug 8 23:48 /opt/128technology/lib/locale/locale-archive
Taking that output and finding the largest of the bunch:
# cat ls-large.txt |awk '{print $7 " " $11 }'|sort
92168784 /usr/lib/128technology/par/event_publisher.par
95164416 /var/cache/yum/x86_64/7/updates/gen/primary_db.sqlite
106134674 /var/lib/128technology/tank/metrics/0/381114086-389712383_1671122870.ilog
106161405 /var/lib/128technology/tank/metrics/0/389712384-398311086_1671124675.ilog
106176928 /usr/lib/locale/locale-archive
113045344 /opt/128technology/lib/locale/locale-archive
134217728 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/
system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-00000000000035d5-0005ef573c43596c.journal134217728 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/
system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-000000000002ebca-0005ef984250c521.journalonly 6 files are above 100M and a sampling of those directories doesn't yield a large number of files.
Anyone familiar enough with the inner workings of the SSR to point me in the right direction?
Patrick
------------------------------
PATRICK KELLAHER
------------------------------