SRX

Greetings. We have vSRX 3 VM hosted on VMware Cloud Director. The network productivity seems to be low, so we're investigating what causes it. Throughput is not the problem, vSRX forwards packets at about 2Gbps. On the other hand, packet latency is suspiciously inconsistent when traffic goes through the vSRX. This is how ping looks like on mostly idle VM:

64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=84 ttl=63 time=0.970 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=85 ttl=63 time=38.3 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=86 ttl=63 time=0.934 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=87 ttl=63 time=14.1 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=88 ttl=63 time=227 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=89 ttl=63 time=16.9 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=90 ttl=63 time=0.830 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=91 ttl=63 time=7.39 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=92 ttl=63 time=166 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=93 ttl=63 time=1.07 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=94 ttl=63 time=230 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=95 ttl=63 time=6.100 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=96 ttl=63 time=124 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=97 ttl=63 time=0.807 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=98 ttl=63 time=71.4 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=99 ttl=63 time=1.02 ms
64 bytes from 10.240.8.50 (10.240.8.50): icmp_seq=100 ttl=63 time=88.7 ms

--- 10.240.8.50 ping statistics ---
100 packets transmitted, 100 received, 0% packet loss, time 99620ms
rtt min/avg/max/mdev = 0.688/53.242/234.987/67.991 ms

So far we tried the following:

• Run VM with the minimal number of NICs.
• Run VM with the minimal config.
• Scale hardware: VSRX-2CPU-4G→VSRX-5CPU-8G→VSRX-9CPU-16G→...
• NUMA affinity settings.
• Check logs/console messages - nothing weird.
• Check if there any resource shortage - seems to be ok.
• vSRX version on this hosting is 4 years old - 20.2R1.10 by builder on 2020-06-25 13:14:24 UTC, we're going to update it, so maybe it'll help.

Do you have any ideas what else can we check? Can latency be caused by the physical host's CPU overprovisioning?