I learned something new the other day (or at least was reminded of something I'd learned previously but forgotten...) that I wanted to share here in case anyone else runs into a similar circumstance.
I ran into an issue where a 128T had a service-route that looked something like this:
*admin@labsystem1.fiedler (service-route[name=rte_cloudflare-dns])# show
name rte_cloudflare-dns
service-name cloudflare-dns
nat-target 1.1.1.1
next-hop labsystem2 wan0
node-name labsystem2
interface wan0
exit
The intention here was to "catch" all DNS traffic on a network and forcibly send it to Cloudflare's DNS server at 1.1.1.1. However, when turning up this service it wound up blackholing all DNS traffic.
It is important to note that the next-hop above does not have a gateway configured. When you have a next-hop for a service-route that is doing nat-targetting and there is no gateway defined, the system will do a routing lookup on the nat-target address as new sessions arrive, to ensure that the routing table would have it egress using the same interface specified in the next-hop. In this particular situation, a routing lookup produced a result for a "public internet" route (0.0.0.0/0) that was using SVR -- and therefore blackholed via the router's RIB. Because this blackhole didn't jibe with the service-route's egress interface, the DNS traffic was dropped.
To rectify this situation, all we needed to do was add a gateway to the next-hop. (This is NOT required for interfaces that use DHCP... the gateway will be filled in automatically.) We also could've "fixed" this problem by adding a static-route to the router for 1.1.1.1 to leave via wan0, since in this case the routing lookup would've yielded the same result as the next-hop egress interface.
Clear as mud? Good :-) Ask any questions here!
------------------------------
pt.
------------------------------