Hi,
Strange problem I'm seeing. Customer currently runs an IPsec VPN over an MPLS link, but they would like to save some money and move to an internet-based IPsec VPN. Problem is, even though the internet-based VPN comes up fine, some application traffic is failing. Switch traffic back to the MPLS-based IPsec VPN and everything comes good.
We've narrowed the issue to MTU/MSS. For the sake of clarity, the two setups look like this:
Over MPLS:
App client --> SRX1(st0.x) --> VPN --> SRX2 --> F5 --> App server (This works)
Over internet:
App client --> SRX1(st0.y) --> VPN --> ??? (Cisco device I think) --> F5 --> App server (TCP/SSL setup correctly, but some app traffic fails)
I have control over SRX1 and SRX2, but nothing else. SRX1 and SRX2 both have a tunnel interface MTU of 1400 and an IPsec tcp-mss value of 1350. The traffic always originates from the left of the flow above, transiting SRX1 first.
I have been given a packet capture taken from the app client computer and can see:
- When the traffic uses the MPLS link, the SYN/ACK in the TCP handshake has the MSS value set to 1350, which matches the setting on SRX1 and SRX2
- When the traffic uses the internet-based VPN, the SYN/ACK in the TCP handshake keeps the original MSS value set by the app client of 1460
After some reading up on the subject, my understanding is that, with the TCP MSS configuration that we have, SRX1 should ALWAYS replace the MSS value in the SYN packet coming from the app client before sending the SYN to the destination through the VPN tunnel. This would mean that the SYN packet would have an MSS value of 1350 when arriving at the app server. Given what I've seen in the packet capture I mentioned, I am assuming this is working properly when the traffic is using the MPLS-based IPsec VPN, but for some reason not when using the internet-based VPN.
Further:
- Both IPsec VPNs originate on SRX1
- Both VPN configs leave the DF-bit setting as default (ie. clear)
- Both the st0.x and st0.y interfaces are in the same security zone and routing instance
- Nothing changes in the path from "app client" to SRX1 regardless of which VPN is used
- The only change I make to have the app traffic use the internet-based VPN is to add a static route, pointing the traffic at st0.y
- When using the MPLS-based VPN, the route is learned via BGP
- The only difference between the st0.x interface that uses the MPLS link is that is has an IP address, whereas the st0.y interface that uses the internet does not
- Routing, security zones and security policies are all set up correctly. In the internet-based VPN scenario we still get a successful TCP handshake, followed by a successful SSL handshake for the app client to app server connection. It's only once the app client tries to send data that the communication fails. In pcaps, I have seen that the data packets sent by app client are obviously larger when the route is switched to the internet-based VPN. Hence the MSS conclusion
I've done datapath-debug and flow traces on SRX1 and cannot find anything that helps me understand where the problem lies. I could assume that SRX1 isn't changing the MSS value in the TCP SYN when using the internet-based VPN, but that would seem like a bug and I'd rather exhaust all other possibilites first.
Happy to hear thoughts and suggestions 🙂
Ben.
#TCP#mss#vpn#IPSec