Earlier you had mentioned that you see the dips in Security Director as well. With TCP transport SD seems to be working fine. Can you confirm that with UDP transport logs to SD are dropping?
I'm just trying to confirm that SD good SPLUNK bad.
If that's the case, then you'll have to consider what's between the SRX and the SPLUNK ingestion point. From the logs, it would appear that the TCP connection to SPLUNK is established, but at some point TCP doesn't see an ACKnowledgment for 30 seconds and closes out the connection, which is normal TCP behavior dead connections. That, combined with the Splunk stats for the UDP transport suggests the forward traffic from SRX to Splunk is getting lost somewhere, and potentially also any return traffic from Splunk to SRX.
If you can double-check and confirm that Junos Space SD is ingesting logs normally without a problem, then I think we can reasonably exclude the SRX as the source of the problem.
------------------------------
Nikolay Semov
------------------------------
Original Message:
Sent: 08-08-2025 12:10
From: Fawad Saleem
Subject: Intermittent Log dips in SRX4100
The graph which you saw above is directly from splunk. Green bars are the events mapped against the time. The empty spaces in between the green bars are the dips.
your observation is correct about junosspace. I don't observe the dips over there. Dips are only against the splunk despite both are logged via same revenue port.
------------------------------
Fawad Saleem
------------------------------
Original Message:
Sent: 08-08-2025 09:15
From: Nikolay Semov
Subject: Intermittent Log dips in SRX4100
What's interesting about the log snippet is that it doesn't include any "error" lines for junosspaceSD, only SPLUNK. Can you check more logs to confirm that's true? I may have missed something...
As a test, you could deactivate SPLUNK logging altogether and leave just junosspaceSD and see if your monitoring graph will continue to show the same dips, though I'm not sure exactly what you're monitoring there.
------------------------------
Nikolay Semov
Original Message:
Sent: 08-08-2025 02:04
From: Fawad Saleem
Subject: Intermittent Log dips in SRX4100
Hi Nikolay,
Good to hear from you.
Yes, we are using UDP transport. But for the sake of trial, I did change the stream mode transport to TCP, and there is a behaviour change. With UDP, I was losing logs for the duration where logs were dropping. Now, with TCP, I am still experiencing drops, but logs for that duration are being recovered now. Which also helped me to catch the errors due to which logs were dropping. I have added s some additional logs in the main post. Can you take a look at them?
------------------------------
Fawad Saleem
Original Message:
Sent: 08-05-2025 15:32
From: Nikolay Semov
Subject: Intermittent Log dips in SRX4100
Looking at the graph, while the green blocks are not exactly evenly spaced, they do seem to be evenly sized.
I'm wondering if it's maybe some sort of a session timeout. Are you using UDP transport? Have you tried switching the transport protocol?
------------------------------
Nikolay Semov