SD-WAN

 View Only
last person joined: yesterday 

Ask questions and share experiences with SD-WAN and Session Smart Router (formerly 128T).

SSR On Storage Space Constrained Hypervisor (NFX350)

  • 1.  SSR On Storage Space Constrained Hypervisor (NFX350)

    Posted 12-15-2022 14:30
    I'm trying to install a network of SSRs on NFX350 devices, but the storage constraint on those devices for VMs is problematic (19 GB).   Within a few days, the VM out grows the available storage that is built into the NFX350 hardware.  Has anyone implemented the SSR in a similar limited storage environment?  I would assume I need to have a daily cron job running on the SSR to limit (delete) logs and such or prune database/cache files to keep the VM from exceeding about 16 GB.  What should I be looking at cleaning daily/hourly?

    For reference, a day after install, one device went from 12G available to 4.5G available and searching for larger than 10M files wasn't promising:

    # find / -size +10M -ls 2>/dev/null
    23 23152 -rw------- 1 root root 23704124 Nov 9 2021 /boot/initramfs-3.10.0-514.el7.x86_64.img
    20 49120 -rw------- 1 root root 50298719 Aug 8 23:52 /boot/initramfs-3.10.0-862.el7.x86_64.img
    21 45724 -rw------- 1 root root 46820351 Nov 9 2021 /boot/initramfs-0-rescue-92953f1920ad4e5eacafe3317bbce6a5.img
    335 13532 -rw------- 1 root root 13854223 Dec 14 17:29 /boot/initramfs-4.18.0-348.20.1.el8_5.x86_64kdump.img
    339 54944 -rw------- 1 root root 56262206 Aug 8 23:53 /boot/initramfs-4.18.0-348.20.1.el8_5.x86_64.img
    4026532029 0 -r-------- 1 root root 140737477885952 Dec 15 17:23 /proc/kcore
    144091 26904 -rwxr-xr-x 1 root root 27548765 Sep 14 2018 /usr/bin/influxd
    138081 72676 -rwxr-xr-x 1 root root 74419096 Jul 1 2021 /usr/bin/node
    149514 72292 -rwxr-xr-x 1 root root 74024256 Feb 10 2022 /usr/bin/telegraf
    144088 15356 -rwxr-xr-x 1 root root 15723706 Sep 14 2018 /usr/bin/influx_inspect
    144090 19200 -rwxr-xr-x 1 root root 19658725 Sep 14 2018 /usr/bin/influx_tsm
    143853 20300 -rwxr-xr-x 1 root root 20785824 Mar 21 2020 /usr/lib64/libicudata.so.50.2
    401948 25048 -rwxr-xr-x 1 root root 25647740 Apr 4 2022 /usr/lib/128T-quickstarter/par/quickstarter.par654306
    14540 -rwxr-xr-x 1 root root 14887194 Mar 21 2022 /usr/lib/128T-minion-watchdog/par/minion_watchdog.par
    133742 103660 -rw-r--r-- 1 root root 106176928 Aug 8 23:43 /usr/lib/locale/locale-archive
    791941 13400 -rwxr-xr-x 1 root root 13721584 Aug 9 2021 /usr/lib/firmware/mrvl/prestera/mvsw_prestera_fw-v3.0.img
    791940 13368 -rwxr-xr-x 1 root root 13686596 Aug 9 2021 /usr/lib/firmware/mrvl/prestera/mvsw_prestera_fw-v2.0.img
    400763 15156 -rw-r--r-- 1 root root 15515796 Aug 9 2021 /usr/lib/firmware/qcom/sm8250/adsp.mbn
    786760 17568 -rwxr-xr-x 1 root root 17985751 Jul 7 2021 /usr/lib/128T-cloud-bootstrapper/par/cloud-bootstrapper.par
    1188346 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_run/pypi__36__salt_3002_8/share/man/man7/salt.7
    1048099 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/event_publisher/pypi__36__salt_3002_8/share/man/man7/salt.7
    1179558 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_key/pypi__36__salt_3002_8/share/man/man7/salt.7
    1176645 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_api/pypi__36__salt_3002_8/share/man/man7/salt.7
    1052420 14016 -rw-r--r-- 1 root root 14350485 Aug 8 23:47 /usr/lib/128technology/unpar/pcli/com_128technology_i95/python/pcli/pcli/data/consolidatedStatsModel.json
    1303837 17460 -rw-r--r-- 1 root root 17878031 Aug 8 23:47 /usr/lib/128technology/unpar/yin-and-friends/com_128technology_i95/python/yinandfriends/yinandfriends/data/scripts/pyang
    1185433 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_master/pypi__36__salt_3002_8/share/man/man7/salt.7
    1182520 14412 -rw-r--r-- 1 root root 14755043 Aug 8 23:47 /usr/lib/128technology/unpar/salt_main/pypi__36__salt_3002_8/share/man/man7/salt.7
    794298 29652 -rwxr-xr-x 1 root root 30362443 Aug 8 21:00 /usr/lib/128technology/par/initialize128t.par
    794307 83496 -rwxr-xr-x 1 root root 85498009 Aug 8 21:00 /usr/lib/128technology/par/salt_run.par
    794303 83496 -rwxr-xr-x 1 root root 85497839 Aug 8 21:00 /usr/lib/128technology/par/salt_api.par
    794309 11932 -rwxr-xr-x 1 root root 12214621 Aug 8 21:00 /usr/lib/128technology/par/t128_analytics.par
    794312 32052 -rwxr-xr-x 1 root root 32820459 Aug 8 21:00 /usr/lib/128technology/par/yin-and-friends.par
    794302 58416 -rwxr-xr-x 1 root root 59817335 Aug 8 21:00 /usr/lib/128technology/par/pcli.par
    794305 83496 -rwxr-xr-x 1 root root 85497981 Aug 8 21:00 /usr/lib/128technology/par/salt_main.par
    794304 83496 -rwxr-xr-x 1 root root 85497801 Aug 8 21:00 /usr/lib/128technology/par/salt_key.par
    785916 18528 -rwxr-xr-x 1 root root 18972400 Apr 15 2022 /usr/lib/128technology/par/snmp_object_agent.par
    794291 90012 -rwxr-xr-x 1 root root 92168784 Aug 8 21:00 /usr/lib/128technology/par/event_publisher.par
    794311 14484 -rwxr-xr-x 1 root root 14828116 Aug 8 21:00 /usr/lib/128technology/par/yin-and-enemies.par
    794306 83496 -rwxr-xr-x 1 root root 85498017 Aug 8 21:00 /usr/lib/128technology/par/salt_master.par
    794294 13540 -rwxr-xr-x 1 root root 13862705 Aug 8 21:00 /usr/lib/128technology/par/generatePasswordHash.par
    791128 17672 -rwxr-xr-x 1 root root 18092272 Oct 27 2021 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/libjvm.so
    791202 21900 -r--r--r-- 1 root root 22425600 Aug 8 23:45 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/amd64/server/classes.jsa
    791184 72120 -rw-r--r-- 1 root root 73848490 Oct 27 2021 /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.312.b07-1.el7_9.x86_64/jre/lib/rt.jar
    654309 14348 -rwxr-xr-x 1 root root 14691241 Aug 31 2021 /usr/libexec/128T-rappid-downloader/par/rappid_downloader.par
    147478 11196 -rwxr-xr-x 1 root root 11461200 Jul 16 14:10 /usr/libexec/128T-cloud-intel-agent
    149530 13724 -rwxr-xr-x 1 root root 14050512 Jul 16 14:10 /usr/libexec/128T-mist-agent
    396633 31252 -rwxr-xr-x 1 root root 32000063 Jun 7 2022 /usr/libexec/128T-installer/par/install128t.par
    396632 14136 -rwxr-xr-x 1 root root 14472661 Jun 7 2022 /usr/libexec/128T-installer/par/erase128t.par
    402268 13544 -rwxr-xr-x 1 root root 13867381 Aug 2 17:39 /usr/libexec/128T-mist-wan-assurance/par/generate_pillar.par
    39 77676 -rw-r--r-- 1 root root 79536128 Aug 8 23:52 /var/lib/rpm/Packages
    923111 16364 -rw-r--r-- 1 root root 16755213 Jun 7 2022 /var/lib/128T-rappid-data/application-classifications.json
    795147 27528 -rw-rwxr-- 1 root root 67108880 Dec 14 22:42 /var/lib/128technology/pds/logs/version-2/log.1795251
    21900 -rw-r--r-- 1 root root 67108880 Dec 15 02:07 /var/lib/128technology/pds/logs/version-2/log.17a11
    795260 24996 -rw-r--r-- 1 root root 67108880 Dec 15 17:23 /var/lib/128technology/pds/logs/version-2/log.68a37
    795254 24132 -rw-r--r-- 1 root root 67108880 Dec 15 05:52 /var/lib/128technology/pds/logs/version-2/log.2a585
    787030 27148 -rw-r--r-- 1 root root 67108880 Dec 15 10:07 /var/lib/128technology/pds/logs/version-2/log.3ef78
    795258 21612 -rw-r--r-- 1 root root 67108880 Dec 15 13:29 /var/lib/128technology/pds/logs/version-2/log.562b4
    660405 103652 -rwxr-xr-x 1 root root 106134674 Dec 15 16:47 /var/lib/128technology/tank/metrics/0/381114086-389712383_1671122870.ilog
    660420 103680 -rwxr-xr-x 1 root root 106161405 Dec 15 17:17 /var/lib/128technology/tank/metrics/0/389712384-398311086_1671124675.ilog
    660422 18956 -rwxr-xr-x 1 root root 19407404 Dec 15 17:23 /var/lib/128technology/tank/metrics/0/398311087_1671124680.log
    924988 10248 -rw-r--r-- 1 root root 10487680 Dec 15 16:45 /var/lib/128technology/influxdb/wal/t128/IntermediateMetrics/13/_00013.wal
    924987 10308 -rw-r--r-- 1 root root 10552690 Dec 15 17:21 /var/lib/128technology/influxdb/wal/t128/ShortMetrics/30/_00013.wal
    786748 24512 -rw-r----- 1 root root 25100288 Aug 8 23:51 /var/cache/yum/x86_64/7/128t-authenticated-release-deps-centos-7-5/gen/primary.xml.sqlite
    786747 21508 -rw-r----- 1 root root 22023867 Aug 4 21:38 /var/cache/yum/x86_64/7/128t-authenticated-release-deps-centos-7-5/gen/primary.xml
    2085 92940 -rw-r--r-- 1 root root 95164416 Aug 4 15:39 /var/cache/yum/x86_64/7/updates/gen/primary_db.sqlite
    2081 16836 -rw-r--r-- 1 root root 17232987 Aug 4 15:39 /var/cache/yum/x86_64/7/updates/d2ee620b433edae5b49be8b11749510e6288a8efaf2b1da17a2dd8ffa6e8c4dd-primary.sqlite.bz2
    391035 30544 -rw-r--r-- 1 root root 31272960 Aug 7 04:45 /var/cache/yum/x86_64/7/epel/gen/primary_db.sqlite
    391036 19248 -rw-r--r-- 1 root root 19707427 Aug 7 05:07 /var/cache/yum/x86_64/7/epel/gen/updateinfo.xml786739
    54172 -rw-r----- 1 root root 55465984 Aug 3 13:32 /var/cache/yum/x86_64/7/128t-authenticated-legacy-release/gen/primary_db.sqlite
    786750 21180 -rw-r----- 1 root root 21682176 Aug 8 23:51 /var/cache/yum/x86_64/7/128t-authenticated-alpha/gen/primary.xml.sqlite
    786749 15480 -rw-r----- 1 root root 15851320 Aug 8 23:36 /var/cache/yum/x86_64/7/128t-authenticated-alpha/gen/primary.xml
    786752 21592 -rw-r----- 1 root root 22106112 Aug 8 23:51 /var/cache/yum/x86_64/7/128t-authenticated-alpha-deps-centos-7-5/gen/primary.xml.sqlite
    786751 18840 -rw-r----- 1 root root 19290993 Aug 8 20:36 /var/cache/yum/x86_64/7/128t-authenticated-alpha-deps-centos-7-5/gen/primary.xml
    2083 30880 -rw-r--r-- 1 root root 31614976 Oct 29 2020 /var/cache/yum/x86_64/7/base/gen/primary_db.sqlite
    263006 10584 -rw-r----- 1 root root 10833673 Aug 8 23:53 /var/cache/dnf/128t-installer-centos-7-5-epel-c5115b886d38810a/repodata/c6d5cc45a62f4177f681a08f57bdee0bf9f0c12ee2ffcee6bb4e443e1bd6c825-filelists.xml.gz
    391124 10980 -rw-r----- 1 root root 11240437 Aug 8 23:53 /var/cache/dnf/128t-installer-centos-7-5-epel-filenames.solvx
    107 23152 -rw------- 1 root root 23703067 Dec 14 17:43 /var/log/messages
    1755 14012 -rw------- 1 root root 14342705 Dec 11 00:01 /var/log/messages-20221211
    924979 10296 -rw-rw-rw- 1 root root 10541582 Dec 15 04:01 /var/log/128technology/influxdb_http.2.log
    924990 10928 -rw-rw-rw- 1 root root 11186632 Dec 15 15:01 /var/log/128technology/influxdb_http.1.log
    1957 71532 -rw------- 1 root root 73243909 Nov 9 2021 /var/log/anaconda/journal.log
    399092 49156 -rw-r----- 1 root systemd-journal 50331648 Dec 15 17:23 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/system.journal
    391123 131080 -rw-r-x--- 1 root systemd-journal 134217728 Dec 12 02:28 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-00000000000035d5-0005ef573c43596c.journal
    398388 131080 -rw-r-x--- 1 root systemd-journal 134217728 Dec 15 02:50 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-000000000002ebca-0005ef984250c521.journal
    390927 16392 -rw-r-x--- 1 root systemd-journal 16777216 Dec 8 20:53 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-0000000000000001-0005e5c34e5cf778.journal
    391042 14016 -rw-r--r-- 1 root root 14350485 Dec 14 17:44 /var/model/consolidatedStatsModel.json
    152398 4194304 -rw------- 1 root root 4294967296 Dec 14 18:25 /dev/hugepages/nfx-gr-blk-ssr-router1map_2
    531581 110368 -rw-r--r-- 1 root root 113045344 Aug 8 23:48 /opt/128technology/lib/locale/locale-archive


    Taking that output and finding the largest of the bunch:
    # cat ls-large.txt |awk '{print $7 " " $11 }'|sort
    92168784 /usr/lib/128technology/par/event_publisher.par
    95164416 /var/cache/yum/x86_64/7/updates/gen/primary_db.sqlite
    106134674 /var/lib/128technology/tank/metrics/0/381114086-389712383_1671122870.ilog
    106161405 /var/lib/128technology/tank/metrics/0/389712384-398311086_1671124675.ilog
    106176928 /usr/lib/locale/locale-archive
    113045344 /opt/128technology/lib/locale/locale-archive
    134217728 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-00000000000035d5-0005ef573c43596c.journal
    134217728 /var/log/journal/92953f1920ad4e5eacafe3317bbce6a5/system@f1bfb7ee2e8c48cbb1d9d761af42aa4a-000000000002ebca-0005ef984250c521.journal

    only 6 files are above 100M and a sampling of those directories doesn't yield a large number of files.

    Anyone familiar enough with the inner workings of the SSR to point me in the right direction?

    Patrick

    ------------------------------
    PATRICK KELLAHER
    ------------------------------