SRX

Background of the problem

We run OSPF between our routers over GRE, and we have multiples of them (4 in total)


Router A ----- 4 x GRE tunnels ------ Router B

The problem I'm seeing is that sometimes OSPF is stuck in the Loading State. I have seen it stays that way for an hour, sometimes 30 minutes, but also in less than 5 seconds. And it happens very randomly. I did a packet capture on the received side of Router A and what I found out is that some of the fragments of the LSU packet is not making its way to complete the reassembly. Basically, router A keeps on sending LS request, but because not all fragments arrive on time until the next LS request retransmit ( default of 5 seconds), it is stuck at Loading until it gets "lucky" and all fragments arrive to complete the reassembly. What I do know is a complete LSUpdate packet is around 2800 bytes (three fragments)and our GRE tunnel is configured MTU under family inet at 1300. There is no problem with fragmentation per se because I can rapid ping large packet across the tunnel (tested it at 6000 bytes).

What I also found out is that Router B has implemented per-packet load-balancing. My working theory is that these LSU fragments are somehow "distributed" to the four GRE tunnels, and somehow, we get lucky when all of them end up on the same tunnel. Also keep in mind that we are using loopback addresses as GRE tunnel source and destination (as opposed to physical inet interfaces, someone else's design decision). The admins at Router B are reluctant to make any changes (they have other OSPF peers, Router A is a new site), so removing the per-packet load balancing is a no go, even just for testing.

One option that I found out is the MTU size under OSPF interface


According to here, it should capped the size of any OSPF packet including LSU. I supposed the router would just send more (not fragmented) LSUs instead of a single fragmented one.

I could ask them to configured this just for the GRE tunnels that points to our site, and should not affect any existing tunnels or adjacency they have with other site. To test my theory, I have disabled the other 3 GRE tunnels between Router A and Router B, and sure enough, the Full adjacency comes up in less than a few seconds.

Also, I have confirmed from the FLOW log on the SRX that I'm getting fragment reassembly timeout (FCB timeout on packet 10.1.X.X --> 224.0.0.5) which would make sense as that is an OSPF packet.

One thing that perplexed me still is why would load balancing send IP fragments on different interfaces... Any insight to why would this be?