SRX

 View Only
last person joined: yesterday 

Ask questions and share experiences about the SRX Series, vSRX, and cSRX.
Expand all | Collapse all

High routing memory utilization

  • 1.  High routing memory utilization

    Posted 04-07-2025 05:54

    Hi all,

    I have a doubt about the behavior of an SRX4100 cluster regarding the use of the routing engine memory.

    Specifically, the memory is never cleaned, reaching almost 100%. In the past, I noticed that when memory usage approaches 100%, the device seems to not work correctly (it does not process policies correctly) and, moreover, it responds very slowly from the CLI.

    Attached you can see the screenshot of the graph related to the use of the routing engine memory (or routing engine buffer pool).

    Does anyone experience the same behavior?

    Regards



    ------------------------------
    Dario Puliero
    ------------------------------


  • 2.  RE: High routing memory utilization

    Posted 04-07-2025 12:21

    The user-identification service seems to go crazy sometimes but who knows. You can collect show system processes extensive every day for a few days and see where the memory is going. Either way, you'll likely have to open a case with Juniper.



    ------------------------------
    Nikolay Semov
    ------------------------------



  • 3.  RE: High routing memory utilization

    Posted 04-08-2025 02:41

    Thank you for your suggestion.

    I have started saving the command output and hope to understand, in a few days, where the memory is being allocated since it increases by 1% every day.

    Thanks



    ------------------------------
    DARIO PULIERO
    ------------------------------



  • 4.  RE: High routing memory utilization

    Posted 04-07-2025 16:24

    I agree this is potentially a memory leak which would be a bug in the version of Junos you are currently running.   What is your current version?

    There is a public bug search site, Juniper calls the PR (problem reports).  If you find the one that describes your symptoms it will list the version of Junos you will need to upgrade for the bug fix.

    https://prsearch.juniper.net/home



    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP - Retired)
    http://puluka.com/home
    ------------------------------



  • 5.  RE: High routing memory utilization

    Posted 04-08-2025 03:09

    Hi Spuluka,

    The currently installed version is 22.4 R3-S2 but the problem also occurred in previous versions. In the past, JTAC asked me to update the devices to this version which they recommended. Every time JTAC has not been able to find the cause of the problem and there seems to be no clue in the logs. The only solution is to turn the devices off and on again.



    ------------------------------
    DARIO PULIERO
    ------------------------------



  • 6.  RE: High routing memory utilization

    Posted 04-08-2025 14:55

    There are 14 PR where memory leaks are associated with RPD.  You could have a look at the details of each and see if the configuration and related processes are applicable to your situation and any of these potential are related.

    https://prsearch.juniper.net/home?stype=affectingthis&ostype=junos&sid=SRXSeries-junos&bv=22.4R3-S2&uv=&qt=memory%20rpd



    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP - Retired)
    http://puluka.com/home
    ------------------------------



  • 7.  RE: High routing memory utilization

    Posted 04-09-2025 02:52

    Thank you, I'm reading the PRs and hope to find the cause of the problem.



    ------------------------------
    DARIO PULIERO
    ------------------------------



  • 8.  RE: High routing memory utilization

    Posted 04-09-2025 12:56

    I don't know if this is already in the list you're going through, but here another memory leak:

    https://supportportal.juniper.net/s/article/2025-04-Security-Bulletin-Junos-OS-SRX-Series-On-devices-with-Anti-Virus-enabled-malicious-server-responses-will-cause-memory-to-leak-ultimately-causing-forwarding-to-stop-CVE-2025-30658



    ------------------------------
    Nikolay Semov
    ------------------------------



  • 9.  RE: High routing memory utilization

    Posted 04-11-2025 04:17

    Hi, 

    I tried to disable the antivirus but unfortunately the buffer continues to grow. 



    ------------------------------
    DARIO PULIERO
    ------------------------------



  • 10.  RE: High routing memory utilization

    Posted 04-10-2025 22:41

    Perhaps

    request system storage cleanup

    https://community.juniper.net/discussion/srx550-is-showing-100-cpu-for-many-days



    ------------------------------
    Adrian Aguinaga
    B.S.C.M. I.T.T. Tech
    (Construction Management)
    A.A.S. I.T.T. Tech
    (Drafting & Design)
    ------------------------------



  • 11.  RE: High routing memory utilization

    Posted 04-11-2025 04:18

    Hi,

    I've already done this command but it cleans the storage but not the buffers



    ------------------------------
    DARIO PULIERO
    ------------------------------



  • 12.  RE: High routing memory utilization

    Posted 16 days ago
    Edited by DARIO PULIERO 16 days ago

    Hi all, 

    the routing engine memory utilization (min 1.3.6.1.4.1.2636.3.1.13.1.11.9.1.0.0) is 91% ...Can anyone help me understand why?

    as in the past, if I don't restart the device, it will get very close to 100% and stop working properly

    Thanks


    ------------------------------
    DARIO PULIERO
    ------------------------------



  • 13.  RE: High routing memory utilization

    Posted 16 days ago

    I inserted some information that might help. I don't know if the second section applies to you but you might be keen to see about routing engine redundancy.

    No reply necessary.

    Understanding Default Routes

    date_range 19-Dec-24

    A default route is the route that takes effect when no other route is available for an IP destination address.

    If a packet is received on a routing device, the device first checks to see if the IP destination address is on one of the device's local subnets. If the destination address is not local, the device checks its routing table. If the remote destination subnet is not listed in the routing table, the packet is forwarded to the next hop toward the destination using the default route. The default route generally has a next-hop address of another routing device, which performs the same process. The process repeats until a packet is delivered to the destination.

    The route evaluation process in each router uses the longest prefix match method to obtain the most specific route. The network with the longest subnet mask that matches the destination IP address is the next-hop network gateway.

    The default route in IPv4 is designated as 0.0.0.0/0 or simply 0/0. Similarly, in IPv6, the default route is specified as ::/0. The subnet mask /0 specifies all networks, and is the shortest match possible. A route lookup that does not match any other route uses this route if it is configured and active in the routing table. To be active, the configured next-hop address must be reachable.

    Administrators generally point the default route toward the routing device that has a connection to a network service provider. Therefore, packets with destinations outside the organization's local area network, typically destinations on the Internet or a wide area network, are forwarded to the routing device with the connection to that provider. The device to which the default route points is often called the default gateway.

    [SRX] Troubleshooting Chassis Cluster Redundancy Group not failing over

    Article ID

    KB20987

    Created

    2011-04-30

    Last Updated

    2024-09-18

    Report a Security Vulnerability

    Description

    This article provides self-troubleshooting steps to determine why a Redundancy Group (RG) in a High Availability Chassis Cluster of SRX services gateway is not failing over.

    This article is part of the Resolution Guide -- SRX Chassis Cluster (High Availability) .

    Symptoms

    If you are expecting a Redundancy Group (RG) to failover due to some reason but it is not failing over, then follow the below mentioned steps to troubleshoot and to find the root cause.

    Solution

    1. On the SRX device, run the command:   show chassis cluster status
      Sample Output:
      > show chassis cluster status

      Cluster ID: 1
      Node                 Priority Status Preempt Manual failover

      Redundancy group: 0 , Failover count: 0
      node0                   150       primary        no           no
      node1                   100   secondary      no          no

      Redundancy group: 1 , Failover count: 0
      node0                   150   primary yes          no
      node1                   100   secondary      yes          no

      Make sure you have properly configured the redundancy groups, specially RG1+: SRX Getting Started - Configure Chassis Cluster (High Availability)

    2. Are you trying to do a Redundancy Group Manual Failover?  For details regarding a Manual Failover, refer to Understanding Chassis Cluster Redundancy Group Manual Failover .

      • Yes : Contine to Step 3

      • No :   Jump to Step 4

    3. Have you done a Redundancy Group Manual Failover before?

    4. Are the Control and Fabric links configured correctly and up?
      Run the following command:
      >show chassis cluster interfaces
      Sample Output for a Branch series SRX services gateway device:
      {primary:node0}
      root@SRX_Branch> show chassis cluster interfaces
      Control link 0 name: fxp1
      Control link status: Up

      Fabric interfaces:
      Name Child-interface Status
      fab0 ge-0/0/2 down
      fab0
      fab1 ge-9/0/2 down
      fab1
      Fabric link status: down
      Sample Output for a High End series SRX services gateway device:
      {primary:node0}
      root@SRX_HighEnd> show chassis cluster interfaces
      Control link 0 name: em0
      Control link 1 name: em1
      Control link status: up

      Fabric interfaces:
      Name Child-interface Status
      fab0 ge-0/0/5 down
      fab0
      Fabric link status: down

      If either one or both links are down, refer to the following articles:
      KB20698 - Troubleshooting Control Link
      KB20687 - Troubleshooting Fabric Link

      If both links are up, continue to Step 5

    5. Have you correctly configured Interface Monitoring or IP Address Monitoring?
      Either Interface Monitoring or IP Address Monitoring is required for RG1+ failover. For detailed explanation on the working of Interface and IP Address Monitoring, refer to the following:
      Monitoring Chassis Cluster Interfaces
      Monitoring IP Addresses on a Chassis Cluster

      Cross check your configuration with the following examples:
      Configuring IP Monitoring and
      Step 6 in the specific SRX models in KB15650

      • Yes the config is correct: Continue to Step 6

    6. What is the priority of each node in the output of >show chassis cluster status ?

    7. If the above steps do not resolve your problem, KB15911 - SRX Getting Started -- Troubleshoot High Availability (HA) is a good reference for failover tips.
      Also, KB21164 - [SRX] Finding out possible reasons for Chassis Cluster failover contains tips on logs to review.
      If still not resolved, refer to  KB21781 - [SRX] Data Collection Checklist - Logs/data to collect for troubleshooting in order to collect the necessary logs from BOTH devices , and open a case with your technical support representative.

    Modification History

    2020-09-26: Article reviewed for accuracy. Article is correct and complete.

    Related Information



    ------------------------------
    Adrian Aguinaga
    B.S.C.M. I.T.T. Tech
    (Construction Management)
    A.A.S. I.T.T. Tech
    (Drafting & Design)
    ------------------------------



  • 14.  RE: High routing memory utilization

    Posted 16 days ago

    Did you ever run show system processes extensive over time to see which process is taking up more and more memory? Having no additional information beyond what you mentioned last time is unlikely to result in any revelations.



    ------------------------------
    Nikolay Semov
    ------------------------------



  • 15.  RE: High routing memory utilization

    Posted 15 days ago
      |   view attached

    Hi Nikolay,

    in attach the output of "run show system processes extensive".

    everything looks okay to me.



    ------------------------------
    DARIO PULIERO
    ------------------------------

    Attachment(s)

    rtf
    output.rtf   34 KB 1 version


  • 16.  RE: High routing memory utilization

    Posted 15 days ago

    Interesting ... 

    So, the total for all processes comes to about 1.8 GB. show system memory may be an even better command for this.

    You do have 2.4 GB wired memory though; that seems excessive. Maybe something's leaking in the kernel, or there's lots of cached I/O. Do you have any active traceoptions statements in your config?

    You can also start shell user root and use vmstat -m to take a peek at some kernel memory allocations to see if anything would pop out there...



    ------------------------------
    Nikolay Semov
    ------------------------------



  • 17.  RE: High routing memory utilization

    Posted 14 days ago

    Hi Nikolay,

    I haven't active traceoptions:

    @srx# show | display set | match traceoptions

    {primary:node0}[edit]

    In attach you can find the output commands of show memory system and vmstat -n.

    Many Thanks



    ------------------------------
    DARIO PULIERO
    ------------------------------

    Attachment(s)

    rtf
    show system memory.rtf   17 KB 1 version
    rtf
    vmstat.rtf   13 KB 1 version


  • 18.  RE: High routing memory utilization

    Posted 14 days ago

    Hi Nikolay,

    I have any active traceoptions:

    @srx# show | display set | match traceoptions
    
    {primary:node0}[edit]

    Below the output of show system memory command:

    @srx> show system memory
    node0:
    --------------------------------------------------------------------------
    System memory usage distribution:
            Total memory: 3193324 Kbytes (100%)
         Reserved memory:   81736 Kbytes (  2%)
            Wired memory: 2513524 Kbytes ( 78%)
           Active memory:   87748 Kbytes (  2%)
         Inactive memory:  203068 Kbytes (  6%)
            Cache memory:       0 Kbytes (  0%)
             Free memory:   65520 Kbytes (  2%)
    Pid     VM-Kbytes(  %  ) Resident(  %  ) Process-name
          0         0(00.00)        0(00.00) [kernel]
          1         0(00.00)        0(00.00) /sbin/init --
          2         0(00.00)        0(00.00) [jfe_job_0_0]
          3         0(00.00)        0(00.00) [jfe_job_1_0]
          4         0(00.00)        0(00.00) [jfe_job_1_1]
          5         0(00.00)        0(00.00) [jfe_job_2_0]
          6         0(00.00)        0(00.00) [jfe_job_3_0]
          7         0(00.00)        0(00.00) [jfe_job_3_1]
          8         0(00.00)        0(00.00) [jfe_job_4_0]
          9         0(00.00)        0(00.00) [jfe_job_4_1]
         10         0(00.00)        0(00.00) [audit]
         11         0(00.00)        0(00.00) [idle]
         12         0(00.00)        0(00.00) [intr]
         13         0(00.00)        0(00.00) [geom]
         14         0(00.00)        0(00.00) [md0]
         15         0(00.00)        0(00.00) [crypto]
         16         0(00.00)        0(00.00) [crypto returns 0]
         17         0(00.00)        0(00.00) [crypto returns 1]
         18         0(00.00)        0(00.00) [md0.uzip]
         19         0(00.00)        0(00.00) [cam]
         20         0(00.00)        0(00.00) [usb]
         21         0(00.00)        0(00.00) [sctp_iterator]
         22         0(00.00)        0(00.00) [rand_harvestq]
         23         0(00.00)        0(00.00) [kern_dump_proc]
         24         0(00.00)        0(00.00) [pagedaemon]
         25         0(00.00)        0(00.00) [vmdaemon]
         26         0(00.00)        0(00.00) [bufdaemon]
         27         0(00.00)        0(00.00) [syncer]
         28         0(00.00)        0(00.00) [vnlru]
         29         0(00.00)        0(00.00) [netdaemon]
         30         0(00.00)        0(00.00) [wkupdaemon]
         31         0(00.00)        0(00.00) [jsr_kkcm]
         32         0(00.00)        0(00.00) [jsr_jsm]
         33         0(00.00)        0(00.00) [dfree_resourced]
         34         0(00.00)        0(00.00) [if_pfe_listen]
         35         0(00.00)        0(00.00) [ifsclientclosed]
         36         0(00.00)        0(00.00) [kern_pir_proc]
         37         0(00.00)        0(00.00) [if_pic_listen0]
         38         0(00.00)        0(00.00) [tnparpproc]
         48         0(00.00)        0(00.00) [soaiod1]
         49         0(00.00)        0(00.00) [soaiod2]
         50         0(00.00)        0(00.00) [soaiod3]
         51         0(00.00)        0(00.00) [soaiod4]
        168         0(00.00)        0(00.00) [md1]
        169         0(00.00)        0(00.00) [md1.uzip]
        503         0(00.00)        0(00.00) [md2]
        504         0(00.00)        0(00.00) [md2.uzip]
        702         0(00.00)        0(00.00) [md3]
        703         0(00.00)        0(00.00) [md3.uzip]
       1226         0(00.00)        0(00.00) [md4]
       1227         0(00.00)        0(00.00) [md4.uzip]
       1284         0(00.00)        0(00.00) [md5]
       1285         0(00.00)        0(00.00) [md5.uzip]
       1345         0(00.00)        0(00.00) [md6]
       1346         0(00.00)        0(00.00) [md6.uzip]
       1404         0(00.00)        0(00.00) [md7]
       1405         0(00.00)        0(00.00) [md7.uzip]
       1465         0(00.00)        0(00.00) [md8]
       1466         0(00.00)        0(00.00) [md8.uzip]
       1526         0(00.00)        0(00.00) [md9]
       1527         0(00.00)        0(00.00) [md9.uzip]
       1627         0(00.00)        0(00.00) [md10]
       1628         0(00.00)        0(00.00) [md10.uzip]
       1715         0(00.00)        0(00.00) [md11]
       1716         0(00.00)        0(00.00) [md11.uzip]
       1774         0(00.00)        0(00.00) [md12]
       1775         0(00.00)        0(00.00) [md12.uzip]
       1833         0(00.00)        0(00.00) [md13]
       1834         0(00.00)        0(00.00) [md13.uzip]
       2379         0(00.00)        0(00.00) [md14]
       2380         0(00.00)        0(00.00) [md14.uzip]
       2487         0(00.00)        0(00.00) [md15]
       2488         0(00.00)        0(00.00) [md15.uzip]
       2622         0(00.00)        0(00.00) [md16]
       2623         0(00.00)        0(00.00) [md16.uzip]
       2722         0(00.00)        0(00.00) [md17]
       2723         0(00.00)        0(00.00) [md17.uzip]
       2788         0(00.00)        0(00.00) [md18]
       2789         0(00.00)        0(00.00) [md18.uzip]
       2859         0(00.00)        0(00.00) [md19]
       2860         0(00.00)        0(00.00) [md19.uzip]
       2918         0(00.00)        0(00.00) [md20]
       2919         0(00.00)        0(00.00) [md20.uzip]
       2977         0(00.00)        0(00.00) [md21]
       2978         0(00.00)        0(00.00) [md21.uzip]
       3036         0(00.00)        0(00.00) [md22]
       3037         0(00.00)        0(00.00) [md22.uzip]
       3095         0(00.00)        0(00.00) [md23]
       3096         0(00.00)        0(00.00) [md23.uzip]
       3154         0(00.00)        0(00.00) [md24]
       3155         0(00.00)        0(00.00) [md24.uzip]
       3215         0(00.00)        0(00.00) [md25]
       3216         0(00.00)        0(00.00) [md25.uzip]
       3276         0(00.00)        0(00.00) [md26]
       3277         0(00.00)        0(00.00) [md26.uzip]
       3337         0(00.00)        0(00.00) [md27]
       3338         0(00.00)        0(00.00) [md27.uzip]
       3398         0(00.00)        0(00.00) [md28]
       3399         0(00.00)        0(00.00) [md28.uzip]
       3460         0(00.00)        0(00.00) [md29]
       3461         0(00.00)        0(00.00) [md29.uzip]
       3521         0(00.00)        0(00.00) [md30]
       3522         0(00.00)        0(00.00) [md30.uzip]
       3582         0(00.00)        0(00.00) [md31]
       3583         0(00.00)        0(00.00) [md31.uzip]
       3643         0(00.00)        0(00.00) [md32]
       3644         0(00.00)        0(00.00) [md32.uzip]
       3704         0(00.00)        0(00.00) [md33]
       3705         0(00.00)        0(00.00) [md33.uzip]
       3808         0(00.00)        0(00.00) [md34]
       3809         0(00.00)        0(00.00) [md34.uzip]
       3869         0(00.00)        0(00.00) [md35]
       3870         0(00.00)        0(00.00) [md35.uzip]
       3930         0(00.00)        0(00.00) [md36]
       3931         0(00.00)        0(00.00) [md36.uzip]
       3989         0(00.00)        0(00.00) [md37]
       3990         0(00.00)        0(00.00) [md37.uzip]
       4048         0(00.00)        0(00.00) [md38]
       4049         0(00.00)        0(00.00) [md38.uzip]
       4109         0(00.00)        0(00.00) [md39]
       4110         0(00.00)        0(00.00) [md39.uzip]
       4170         0(00.00)        0(00.00) [md40]
       4171         0(00.00)        0(00.00) [md40.uzip]
       4231         0(00.00)        0(00.00) [md41]
       4232         0(00.00)        0(00.00) [md41.uzip]
       4292         0(00.00)        0(00.00) [md42]
       4293         0(00.00)        0(00.00) [md42.uzip]
       4359         0(00.00)        0(00.00) [md43]
       4360         0(00.00)        0(00.00) [md43.uzip]
       4420         0(00.00)        0(00.00) [md44]
       4421         0(00.00)        0(00.00) [md44.uzip]
       4481         0(00.00)        0(00.00) [md45]
       4482         0(00.00)        0(00.00) [md45.uzip]
       4542         0(00.00)        0(00.00) [md46]
       4543         0(00.00)        0(00.00) [md46.uzip]
       4691         0(00.00)        0(00.00) [md47]
       4692         0(00.00)        0(00.00) [md47.uzip]
       4808         0(00.00)        0(00.00) [md48]
       4809         0(00.00)        0(00.00) [md48.uzip]
       4875         0(00.00)        0(00.00) [md49]
       4876         0(00.00)        0(00.00) [md49.uzip]
       4942         0(00.00)        0(00.00) [md50]
       4943         0(00.00)        0(00.00) [md50.uzip]
       5012         0(00.00)        0(00.00) [md51]
       5013         0(00.00)        0(00.00) [md51.uzip]
       5079         0(00.00)        0(00.00) [md52]
       5080         0(00.00)        0(00.00) [md52.uzip]
       7319         0(00.00)        0(00.00) [md53]
       7320         0(00.00)        0(00.00) [md53.uzip]
       7519         0(00.00)        0(00.00) [md54]
       7520         0(00.00)        0(00.00) [md54.uzip]
       7591         0(00.00)        0(00.00) [md55]
       7592         0(00.00)        0(00.00) [md55.uzip]
       8648         0(00.00)        0(00.00) [ftcleanup]
      11499      3092(55.17)     3076(00.10) /sbin/jlaunchhelperd
      11500      7476(42.79)     4884(00.15) /sbin/jlaunchd -N
      11515      1884(91.63)     1296(00.04) /usr/libexec/bslockd -mp -N
      11516      6576(71.05)     3808(00.12) /usr/sbin/tnetd -N -l
      11519     20808(89.08)    11988(00.38) /usr/sbin/sdk-vmmd -N
      11520     15596(00.00)     7164(00.22) /usr/sbin/cron -s
      11523     16660(91.96)    10556(00.33) /usr/sbin/craftd -N
      11525     64036(01.53)    42536(01.33) /usr/sbin/mgd -N
      11531     13044(79.61)     6900(00.22) /usr/sbin/inetd -N -w -R 512
      11536     10092(82.16)     5856(00.18) /usr/sbin/shm-rtsdbd -N
      11537      7548(71.72)     4884(00.15) /usr/sbin/gstatd -N
      11541     24064(56.35)    12244(00.38) /usr/sbin/agentd -N
      11542     20380(91.55)    13528(00.42) /usr/sbin/jsrpd -N
      11562     30204(44.02)    12372(00.39) /usr/sbin/jsd -N
      11563     14480(92.65)     6736(00.21) /usr/sbin/xmlproxyd -N
      11564     33316(72.15)    12880(00.40) /usr/sbin/dot1xd -N
      11566     23652(00.56)    12552(00.39) /usr/sbin/utmd -N
      11567     11844(91.96)     6784(00.21) /usr/sbin/sysctlrelayd -N
      11568     28356(50.34)     3312(00.10) /usr/sbin/na-grpcd -c /opt/telemetry/na-grpcd/na-grpc-server.ini
      11569     19108(79.33)    10944(00.34) /usr/sbin/pkid -N
      11642      3096(55.33)     3064(00.10) /usr/libexec/getty.junos 3wire ttyu0
      11646     62772(01.83)    41316(01.29) mgd: (mgd) /packages/mnt/junos-runtime/usr/sbin/mgd (root)
      11650      2080(93.02)     1516(00.05) /sbin/watchdog -t-1
      11734     17508(91.92)    10876(00.34) /usr/sbin/ppmd -N
      11738     21080(81.64)    11432(00.36) /usr/sbin/bfdd -N
      11739     24160(91.02)    12476(00.39) /usr/sbin/lacpd -N
      11740     15616(80.07)     9408(00.29) /usr/sbin/lfmd -N
      11742     20168(53.62)     9008(00.28) /usr/libexec32/smid -N
      11743      6780(42.05)     6728(00.21) /usr/sbin/rpcbind -N -I -h 128.0.0.1
      11744      2628(52.64)     2344(00.07) /bin/sh /etc/srx_sntpd.sh
      11745     20484(34.49)    11488(00.36) /usr/sbin/jsqlsyncd -N
      11748      2916(86.48)     2292(00.07) /usr/sbin/tnp.sntpd -JU __juniper_private4__ -N
      11756      7276(45.10)     7128(00.22) /usr/sbin/na-mqttd -c /opt/telemetry/na-mqttd/na-mqtt.conf
      11763      9416(92.57)     5524(00.17) /usr/sbin/fipsd -N
      11855         0(00.00)        0(00.00) [ppt_0a_82100010]
      11961         0(00.00)        0(00.00) /usr/sbin/license-check -U -M -p 10 -i 10
      11999         0(00.00)        0(00.00) [ppt_0a_81100010]
      27042     36736(00.88)    19028(00.60) /usr/sbin/aamwd -N
      36202     24324(00.00)    11760(00.37) sshd: dario [priv]
      36204     24764(00.00)    11848(00.37) sshd: dario@0
      36205     92060(02.20)    31088(00.97) -cli
      36206     64860(01.55)    42984(01.35) mgd: (mgd) (dario)/dev/pts/0
      54612     21748(00.52)     8376(00.26) /usr/sbin/ntf-agent -N -c /var/etc/ntf-agent.conf
      54769     15632(00.37)     7568(00.24) /usr/sbin/mgd-api -N
      54850     24060(00.57)    13608(00.43) /usr/sbin/appsecured -N
      54917     34696(00.83)    20552(00.64) /usr/sbin/eventd -N -r -s -A
      54952     44160(01.05)    20768(00.65) /usr/sbin/idpd -N
      56460    737304(17.60)     9468(00.30) /usr/sbin/httpd-gk -N
      56469     23280(00.56)    10468(00.33) /usr/sbin/httpd --config /jail/etc/httpd.conf
      56994     21988(00.00)    21984(00.69) /usr/sbin/xntpd -j -N -g -e
      58251     95100(02.27)    65496(02.05) /usr/sbin/appidd -N
      71763     36840(49.21)    12452(00.39) /usr/sbin/l2ald -N
      71764     22556(00.54)    11188(00.35) /usr/sbin/sampled -N
      71765     25880(00.62)    12488(00.39) /usr/sbin/rmopd -N
      71766     34264(81.37)    13360(00.42) /usr/libexec32/cosd
      71770      9448(76.22)     5704(00.18) /usr/sbin/jflowd -N
      71771     10640(76.68)     8124(00.25) /usr/sbin/srrd -N
      71772     16336(94.91)     8296(00.26) /usr/sbin/dood -N
      71773      9816(92.01)     5584(00.17) /usr/sbin/pppd -N
      71774     31924(71.14)    14160(00.44) /usr/sbin/l2cpd -N
      71775     16244(90.47)    10432(00.33) /usr/sbin/oamd -N
      71776     11160(52.62)     6128(00.19) /usr/sbin/mplsoamd -N
      71777     12392(92.84)     7032(00.22) /usr/sbin/sendd -N
      71778     16788(72.68)    10516(00.33) /usr/sbin/wwand -N
      71779      6840(00.00)     6720(00.21) nfsd: master
      71780     13524(77.37)     8008(00.25) /usr/sbin/commit-syncd -N
      71782      6728(40.89)     6680(00.21) /usr/sbin/mountd -N -s -h 128.0.0.1 -p 6666
      71783      9432(76.53)     5556(00.17) /usr/sbin/datapath-traced -N
      71784     21864(71.27)    12068(00.38) /usr/sbin/smihelperd -N
      71785     20280(94.50)     8516(00.27) /usr/sbin/grpc-tunnel -N
      71787      6840(00.00)     6720(00.21) nfsd: server
      71788     13632(91.51)     6768(00.21) /usr/sbin/nstraced -N
      71789     12256(61.10)     6820(00.21) /usr/sbin/gprsd -N
      71790     15340(74.90)     7320(00.23) /usr/sbin/syshmd -N
      71792      9420(76.29)     5556(00.17) /usr/sbin/smtpd -N
      71793     21528(96.33)     2564(00.08) /usr/sbin/wmic -N
      71794     11820(77.34)     6392(00.20) webapid: master process /usr/sbin/webapid -N
      71805     67804(55.64)    16328(00.51) /usr/sbin/chassisd -N
      71806     25180(82.85)    13252(00.41) /usr/sbin/alarmd -N
      71807    744448(17.77)    35332(01.11) /sbin/dcd -N
      71810     17100(94.18)    10932(00.34) /usr/sbin/snmpd -N
      71826     29620(00.71)    14240(00.45) /usr/sbin/mib2d -N
      71828     27080(00.65)    14936(00.47) /usr/libexec32/pfed -N
      71830     29528(74.08)    12500(00.39) /usr/sbin/kmd -N
      71831     29040(60.34)    14884(00.47) /usr/libexec32/dfwd -N
      71832      9524(76.61)     5512(00.17) /usr/sbin/irsd -N
      71833     17216(86.70)     8196(00.26) /usr/sbin/bdbrepd -N
      71834     59820(01.43)    37952(01.19) /usr/sbin/nsd -N
      71835     21148(81.19)    10900(00.34) /usr/sbin/lsysd -N
      71836     20136(74.56)    10988(00.34) /usr/libexec32/repd -N
      71842     18884(71.04)     9744(00.31) /usr/sbin/dyn-filterd -N
      71844     32964(36.58)    14140(00.44) /usr/sbin/ipfd -N
      71845     19404(00.46)    11012(00.34) /usr/sbin/rtlogd -N
      71846     11196(91.59)     6724(00.21) /usr/sbin/jkm -N
      71850     22488(77.85)     8660(00.27) /usr/sbin/sub-brokerd -N
      71851     22488(77.49)     8580(00.27) /usr/sbin/pub-brokerd -N
      71884     11832(77.41)     6400(00.20) webapid: worker process
      71935         0(00.00)        0(00.00) [ppt_07_82100001]
      78368     24324(00.00)    11760(00.37) sshd: dario [priv]
      78376     24328(00.00)    11772(00.37) sshd: dario@1
      78377     92064(02.20)    31008(00.97) -cli
      78378     64404(01.54)    42912(01.34) mgd: (mgd) (dario)/dev/pts/1
      78451    897684(21.42)   107540(03.37) /usr/sbin/rpd -N
      78673     64100(01.53)    42696(01.34) mgd: (mgd) (root)
    
    node1:
    --------------------------------------------------------------------------
    System memory usage distribution:
            Total memory: 3193324 Kbytes (100%)
         Reserved memory:   81736 Kbytes (  2%)
            Wired memory: 2209252 Kbytes ( 69%)
           Active memory:   54220 Kbytes (  1%)
         Inactive memory:  600576 Kbytes ( 18%)
            Cache memory:       0 Kbytes (  0%)
             Free memory:  244860 Kbytes (  7%)
    Pid     VM-Kbytes(  %  ) Resident(  %  ) Process-name
          0         0(00.00)        0(00.00) [kernel]
          1         0(00.00)        0(00.00) /sbin/init --
          2         0(00.00)        0(00.00) [jfe_job_0_0]
          3         0(00.00)        0(00.00) [jfe_job_1_0]
          4         0(00.00)        0(00.00) [jfe_job_1_1]
          5         0(00.00)        0(00.00) [jfe_job_2_0]
          6         0(00.00)        0(00.00) [jfe_job_3_0]
          7         0(00.00)        0(00.00) [jfe_job_3_1]
          8         0(00.00)        0(00.00) [jfe_job_4_0]
          9         0(00.00)        0(00.00) [jfe_job_4_1]
         10         0(00.00)        0(00.00) [audit]
         11         0(00.00)        0(00.00) [idle]
         12         0(00.00)        0(00.00) [intr]
         13         0(00.00)        0(00.00) [geom]
         14         0(00.00)        0(00.00) [md0]
         15         0(00.00)        0(00.00) [crypto]
         16         0(00.00)        0(00.00) [crypto returns 0]
         17         0(00.00)        0(00.00) [crypto returns 1]
         18         0(00.00)        0(00.00) [md0.uzip]
         19         0(00.00)        0(00.00) [cam]
         20         0(00.00)        0(00.00) [usb]
         21         0(00.00)        0(00.00) [sctp_iterator]
         22         0(00.00)        0(00.00) [rand_harvestq]
         23         0(00.00)        0(00.00) [kern_dump_proc]
         24         0(00.00)        0(00.00) [pagedaemon]
    
    {primary:node0}

    And this is the output of vmstat -m:

    # vmstat -m
             Type InUse MemUse HighUse Requests  Size(s)
        CAM queue     6     2K       -       39  16,32,512
              USB    20    23K       -       22  16,128,512,4096
           USBdev    18     4K       -       24  32,64,128,256,512
            vtbuf    24  1968K       -       46  4096
               vt    11     6K       -       11  512
           DEVFS2   231    15K       -      845  16,32,64
           DEVFS3   990   248K       -     1174  256
           DEVFS1   231   116K       -      295  512
            DEVFS    41     2K       -       42  16,128
           DEVFSP     0     0K       -        6  64
     msdosfs_node     7     2K       -      468  256
    msdosfs_mount     1     1K       -        1  512
      msdosfs_fat     1     8K       -        1  8192
      nullfs_hash     1   512K       -        1
      nullfs_node    44     3K       -      225  64
     nullfs_mount     4     1K       -        4  32
        pfs_nodes    35    18K       -       35  512
      pfs_vncache   739    47K       -     2985  64
      tmpfs mount     2     1K       -        2  128
       tmpfs name   492     9K       -     8200  16,32,64,128
             GEOM  1272   178K       -     7295  16,32,64,128,256,512,1024,16384
        geom_uzip   224  3188K       -    72684  128,256,512,1024,2048,4096,8192,16384,32768,65536
           isadev     1     1K       -        1  128
      isofs_mount    56    14K       -       56  256
       isofs_node 12642  3161K       -    12642  256
    CAM dev queue     3     1K       -        3  64
             cdev     4     1K       -        4  256
         filedesc    42   169K       -     3491  16,4096,8192
            sigio     2     1K       -      640  64
         filecaps     0     0K       -      886  16,32,64,128
          kdtrace   688   124K       - 12174792  64,256
             kenv    76    12K       -       78  16,32,64,128,512,1024,8192
           kqueue   318    69K       -  3967386  64,128,256,512,2048
        proc-args   120     6K       -  3751504  16,32,64,128,256
            hhook    16     3K       -       24  16,32,256
          ithread   202    30K       -      202  32,128,256
           prison    12     1K       -       12  32
              KTR     1 11264K       -        1
           KTRACE   100    13K       -      100  128
      ddb_capture     1    64K       -        1  65536
           linker  1276 10481K       -     1775  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
         acpiintr     1     1K       -        1  64
            lockf   372    40K       - 82115451  64,128
       loginclass     4     1K       -     1177  128
           devbuf  4278  3109K       -    45057  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
             temp 14375344 2020898K       - 123160135  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
           acpica   930    99K       -    39673  16,32,64,128,256,512,2048,8192
           module   396    50K       -      396  128
         mtx_pool     2    72K       -        2  8192,65536
         netstack     1     1K       -        1  16
              osd   978    17K       - 23536461  16,32,64,128,256
         pmchooks     1     1K       -        1  128
              pmc     1     1K       -        1  64
             pgrp    13     2K       -   159315  128
          session    10     3K       -   159189  256
             proc     2     2K       -        2  1024
          subproc   550  1189K       -  3967357  512,4096
             cred   139    35K       -  1303168  256
         acpitask     1    64K       -        1  65536
          acpisem    23     3K       -       23  128
          acpidev    21     2K       -       21  64
           plimit    47    12K       -  2385539  256
          uidinfo     4     1K       -      400  128,256
           dumper     1     1K       -        1  512
          CAM SIM     3     1K       -        3  256
           sysctl     1     1K       - 864632618  32,64,128
        sysctloid 10272   530K       -    10348  16,32,64,128
        sysctltmp     0     0K       -  4891235  16,32,64,128,256,512,1024
          CAM XPT    12     1K       -       25  32,128,512,1024
          CAM DEV     3     6K       -       14  2048
          tidhash     1     2K       -        1  2048
          callout     3  1672K       -        3
             umtx   940   118K       -      940  128
         p1003.1b     1     1K       -        1  16
             SWAP     2   132K       -        2  65536
              bus  1832   193K       -     4378  16,32,64,128,256,1024
           bus-sc    66   208K       -     1927  16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
          CAM CCB     0     0K       -       26  2048
          devstat    58   117K       -       58  32,4096
     eventhandler   138    12K       -      138  64,128
       gtaskqueue    18    26K       -       18  16,32,256,8192
             kobj   189   756K       -      550  4096
          Per-cpu     1     1K       -        1  32
         CAM path     3     1K       -       39  32
             rman   150    18K       -      469  16,32,128
             sbuf     1     1K       -    44336  16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
           sglist    21    14K       -       21  32,512,1024
        toponodes     8     1K       -        8  128
       CAM periph     2     1K       -       15  16,32,64,128,256
        taskqueue    54     6K       -       54  16,32,256
         terminal    11     3K       -       11  256
           Unitno    37     3K       - 64743175  32,64
             vmem     3   276K       -       11  2048,4096,8192,16384,32768
         ioctlops     0     0K       -    10939  512,2048
           select   172    22K       -      172  128
              iov     0     0K       - 219075467  16,32,64,128,256,512,1024,2048
              msg     4    30K       -        4  2048,4096,8192,16384
              sem     4   106K       -        4  2048,4096
              shm    21   224K       -     2260  8192,65536
              tty    15    15K       -       79  1024
              pts     2     1K       -       66  256
         mbuf_tag    76     4K       - 16790309525  32,64,128
            shmfd   121    61K       -      345  64,256,1024,8192
           soname   241    28K       - 222948171  16,32,64,128
              pcb   832   762K       - 18294679  16,32,64,128,256,512,1024,2048,4096,8192
              acl     0     0K       -       13  4096
         vfscache     4  1025K       -        4  256,512
       cl_savebuf     0     0K       -     2233  64
         vfs_hash     1   512K       -        1
           vnodes     5     1K       -        8  32,256
            mount  1127    47K       -     1222  16,32,64,128,256,4096
           statfs     0     0K       -  6315993  4096
      vnodemarker     0     0K       - 11813919  512
    chacha20random     1     1K       -        1  512
             zlib    56   448K       -       56  8192
           crypto     2     2K       -        2  256,1024
    audit_evclass   230     8K       -      287  32
         veriexec 10842  1347K       -    11298  16,32,128,4096
          pagedep     4   385K       -     3605  256
         inodedep     6  1538K       -    25622  512
        bmsafemap     4    25K       -    15652  256,8192
           newblk     3  3072K       -   103333  256
         indirdep     0     0K       -     1122  128,32768
         freefrag     0     0K       -     6284  128
         freeblks     0     0K       -    12880  128
         freefile     0     0K       -    10613  64
           diradd     0     0K       -    16968  128
            mkdir     0     0K       -       82  128
           dirrem     3     1K       -    16988  128
        newdirblk     0     0K       -       41  64
         freework     3     1K       -    21475  64,128
            sbdep     0     0K       -     8671  64
         savedino     0     0K       -     3864  128,256
          softdep     3     2K       -        3  512
        ufs_mount     9    46K       -       13  512,2048,4096,8192
        vm_pgdata     2     1K       -        2  128
          UMAHash     5    39K       -       10  512,1024,2048,16384,32768
          memdesc     1     4K       -        1  4096
         pci_link    10     1K       -       10  16,128
         atkbddev     1     1K       -        1  64
          ata_pci     1     1K       -        1  64
           apmdev     1     1K       -        1  128
       madt_table     0     0K       -        2  16,4096
           kbdmux     6    22K       -        6  16,512,1024,2048,16384
          md_disk    56   224K       -       56  4096
          entropy     2     1K       -    14719  16,32,64,128,256,512,2048
             intr     4   220K       -        4  32768,65536
          io_apic     1     2K       -        1  2048
       local_apic     1     2K       -        1  2048
             UART     4    12K       -        4  16,1024,2048,8192
              MCA    10     2K       -       10  128
             cpus     2     1K       -        2  16
              msi    39     5K       -       39  128
         nexusdev     6     1K       -        6  16
        hmac_drbg     0     0K       -        2  128,256
               lr     1     4K       -        1  4096
              pic     7     4K       -        7  32,128,1024
          pfestat   193   111K       - 29126640  16,32,64,4096,8192
           gencfg  1922  3157K       - 839823845  16,32,64,128,512,1024,2048,4096,8192,16384,32768
              jsr     4     1K       -       88  32
              idl    64   980K       - 60916544  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768
           rtsmsg     1     4K       - 709396726  4096,8192,16384,32768
           rtstat   166  5312K       -  8694724  32768
       mesh-group     4     1K       -        4
       ifl_qstats    52   208K       -       56  4096
           sample     9     8K       -        9  32,1024,2048
    vpls_lc_instance     1     8K       -        1  8192
         ifa_list    41     1K       -       51  16
    ifstateSLLNode     0     0K       -      256  16
    lag_list_node    24     1K       -       54  32
    gresstatevarlog     1    96K       -        1
          JFE job     1     1K       -        1  1024
      JFE PROFILE    45    51K       -       45  16,64,2048
    JFE statistics    18  1387K       -       18  8192,32768
    sysctl policing   899    43K       -      899
            jlist     1     1K       -        1
        schedhist     1   704K       -        1
            jlock   433  6185K       -    51580  64,2048
           gtable  1264    71K       -    82864  64
              DDI     6     1K       -        6
       in6ifmulti     3     1K       -        6  64
      ifl_idx_mgr     1     1K       -        1
       in6grentry     6     1K       -       12  64
        iflogical   190   123K       -      205  16,128,1024,2048,4096
         iffamily   204    73K       -      228  16,32,64,2048
        rtnexthop   932   375K       - 644747216  16,256,512,1024,2048
          metrics    16     4K       -    47827
        inifmulti     2     1K       -        3  64
        ingrentry     3     1K       -        5  64
            rnode   995    48K       - 197470595  32,64,256
              STP    73    17K       -       76  16,128,512
           rcache     4    16K       -        4  4096
         syncache     1    16K       -        1  16384
         fragment     0     0K       -       73
           ip6opt     1     1K       -  1477061  32,128
         tlv_stat     0     0K       -  3452035  16384,32768,65536
    nhack_info_entries     4     1K       - 141075022  32
         ifdevice    46    12K       -       64  16,32,64,128,2048
           ifstat   429   427K       -  1097953  16,32,64,1024,4096,8192
           rtdata     1     1K       -        1  64
    shared_bitvector     3     1K       -        3
     rts_osd_slot    57     1K       -      837  16
    Bridge Domain     4     1K       -        4  32
            iflib    16   103K       -       20  16,64,128,512,1024,4096,8192,16384,32768
             ipfw    48    26K       -      148  16,32,64,128,256,512,1024,2048,4096
          ifmaddr   283     9K       -      361  32
           rtable  1040   129K       - 197527015  16,32,128,512,1024
           sysctl     0     0K       -       89  256,512,1024,2048
           ifaddr   112    12K       -      132  32
             mkey   847    27K       - 23484333  32,128
          pfe_ipc     3     1K       - 11626247  16,32,64,4096,8192,16384,32768
          ifstate 10497   796K       - 1213550749  16,32,64,128,512,2048,16384,32768
    virtfs_mount_tag    10     1K       -       10  16,32
        p9_client    10     2K       -       10  128
     virtfs_mount    10     5K       -       10  512
         ifstrace 100129  8337K       - 3968418849  64,128,256
         itable16  1156   427K       - 822710436  256,512
         itable32   551    44K       -      552
         itable64     1     1K       -        1  128
      Export Host     2     2K       -        2  1024
      inpcbpolicy   257     9K       -   553768  32
      ipsecpolicy   514   129K       -  1107536  256
          solaris 397093 20229K       -   397138  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
              fbt 90810 11608K       -    90810  128
              rpc     7     5K       -        7  32,64,256,512,4096
    NFSD V4client     1     1K       -        1  256
     NFSD lckfile     1     1K       -        1  256
     NFSD session     1     1K       -        1  1024
              SDT    58     3K       -       58  16,64

    Do you see anything strange?

    Thanks :)



    ------------------------------
    DARIO PULIERO
    ------------------------------



  • 19.  RE: High routing memory utilization

    Posted 14 days ago

    Yeah ... Now we just have to figure out what this "temp" bucket is and why it's taking up 2 GB of memory...

            temp 14375344 2020898K       - 123160135  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768


    ------------------------------
    Nikolay Semov
    ------------------------------



  • 20.  RE: High routing memory utilization

    Posted 14 days ago

    I do believe this is a memory leak. But who knows where ...

    Without going into the weeds, the easiest thing to do, in my opinion, is to restart components one by one (see restart ? in the CLI) and then check vmstat -m | grep temp from a root shell to see at what point the temp memory bucket gets freed up.

    If you feel like going very deep into the weeds, you could use dtrace to monitor temp memory allocations in real time as well as memory releases and see which process is allocating memory without releasing it. DO NOT DO THIS UNLESS NON-PRODUCTION, OR IN MAINTENANCE WINDOW. Here are some sample commands, all from a root shell:

    // this command will display temp memory allocations; stop it with Ctrl+C after you 
    // get a few lines of output. the number in parenthesis is the address 
    // corresponding to TEMP memory allocations 
    dtrace -n 'fbt::malloc:entry /1 == 1/ { printf("malloc type=%s (%#x) \n", stringof(*(uintptr_t *)(arg1+0x10)), arg1); }' | grep temp
    
    // this will query TEMP memory allocations, showing which process 
    // requested it, how much, when, and at what address the allocated memory is.
    // the number for the arg1 comparison should be written as it appears in the 
    // output of the previous command, starting with 0x
    // the ret value is the allocated memory block address
    dtrace -n 'fbt::malloc:entry /arg1 == THE_NUMBER_FROM_THE_ABOVE_COMMAND_HERE/ { self->size = arg0; self->type = arg1; } fbt::malloc:return /self->size/ { printf("malloc TEMP size %d bytes at %Y by %s ret=%p\n", self->size, walltimestamp, execname, args[1]); }'
    
    // in a separate root shell, you can simultaneously monitor memory releases
    // both should be captured for the same time window
    // the value displayed is the freed memory block address
    dtrace -n 'fbt::free:entry /1 == 1/ { printf("free %p \n", arg0); }'
    
    // stop commands with Ctrl+C after a while 
    // then you can ingest everything into your favorite data processing tools 
    // and then eliminate lines for which the memory block address appears in both
    // the allocated list and the freed list; that will leave you with
    // a list of memory allocations that are not freed.
    // this may not be 100% precise, it's possible that dtrace may miss events 
    // but overall, if you monitor this for long enough, you should notice a process
    // that's allocating memory that it's not releasing



    ------------------------------
    Nikolay Semov
    ------------------------------