Layer 3 Virtual Private Network Inter-AS option using SRv6 as underlay transport on MX and ACX7000 routers.
Introduction
This is the 5th blog post in the series of SRv6 blogs. This blog post is co-authored by Krzysztof Szarkowicz and Rajesh M, and discusses the L3VPN (Layer 3 Virtual Private Network) Inter-AS (Inter Autonomous System) Option C, using SRv6 as underlay transport. This could be used by SP (Service Provider) in different deployments where IGP (Interior Gateway Protocol) domains are not merged, for example:
- network is divided into separate AS (Autonomous Systems) as a result of network mergers and migrations
- network is divided into separate AS domains due to logical separation (e.g., different AS for data canter network, and different AS for transport network)
- network is divided into separate AS domains due to operational reasons (network size, different operations team maintaining different parts of the network, etc.)
This blog post is based on the capabilities of Junos 22.3 running on MX Series and ACX7000 routers. Configuration and operational command outputs have been collected on vMX in our labs.
You can test yourself all the concepts described in this article, we created labs in JCL and vLabs:
JCL (Junivators and partners)
vLabs (open to all)
In this blog post, following IP (Internet Protocol) addressing is used:
Transport Infrastructure (P/PE)
- Router-ID: 198.51.100.<XX>
- Loopback: 2001:db8:bad:cafe:<area>00::<XX>/128
- SRv6 locator: fc01:<area>00:<XX>::/48
- Core Links: 2001:db8:beef:<area>00::<XXYY>:<local-ID>/112
PE-CE links:
- IPv4: <VLAN>.<XX>.<YY>.<local-ID>/24
- IPv6: 2001:db8:babe:face:<VLAN>::<XXYY>:<local-ID>/112
VPN (Virtual Private Network) Loopbacks (CE/PE):
- 192.168.<VLAN>.<XX>/32
- 2001:db8:abba:<VLAN>::<XX>/128
Architecture
RFC4364 (BGP/MPLS IP Virtual Private Networks) describes in Section 10 three different ways of providing L3VPN services over the multi-AS network. These three different architectures are commonly referenced as Inter-AS Option 10A, Inter-AS Option 10B, and Inter-AS Option 10C, given the fact, that a), b), c) options are described in Section 10. Or, simply, they could be referenced as Inter-AS Option A, Inter-AS Option B or Inter-As Option C.
This blog post focuses on Inter-AS Option C architecture. However, as opposed to original Inter-AS Option C described in RFC4364, this time the underlay is not MPLS but SRv6. Figure 1 summarizes the example network topology used as the basis for L3VPN Inter-AS Option C discussion.
Figure 1: Inter-AS Topology and L3VPN service prefix distribution
The example network has two domains, each domain with different AS number (AS 65501, and AS 65502, respectively), and each domain with its own IGP (IS-IS L2 area 49.0001 and IS-IS L2 area 49.0002). There is no IGP connectivity between these domains – they are interconnected via BGP only.
The essence of Inter-AS Option C is as follows:
- Services prefixes are exchanged between domains without modification of BGP NEXT_HOP attribute. In the context of SRv6, it also means they are exchanged without modification of SRv6 service SID (i.e., without modification of END.DT4, END.DT6 or END.DT46 SRv6 SIDs). This implies that between PEs in different domains, there must be end-to-end transport tunnel suitable for transporting L3VPN traffic. In the example of this blog post, there is a full mesh of BGP sessions between PEs (within, as well as between, domains), carrying L3VPN prefixes, as outlined in Figure 1. In most deployments, instead of creating full BGP mesh, a RR (Route Reflector) architecture would be used, with PEs within a domain peering with iBGP sessions to RRs of that domain, and L3VPN prefixes exchanged between domains via eBGP sessions between RRs. To simplify this blog post, RR architecture is not used here.
- Transport prefixes (e.g. loopbacks) are exchanged between domains with BGP NEXT_HOP attribute being changed (set to local address, so called “next-hop self” – nhs – action) on domain boundaries, as outlined in Figure 2. In the context of SRv6 (see L3VPN over SRv6 blog post for more details), transport end-points for L3VPN services are not loopbacks, but SRv6 locators, hence nhs action must be executed on SRv6 locators as well. Both loopbacks and SRv6 locators are distributed via IPv6 unicast (AFI/SAFI=2/1).
Figure 2: Inter-AS Topology and transport prefix (loopbacks, SRv6 locators) distribution
In Inter-AS Option C, there is no need to maintain service prefixes on ASBR (autonomous system boundary router), as the end-to-end tunnels between PEs in different domains provides end-to-end transport capability. This tunnel is a hierarchical tunnel, where inter-domain tunnel (tunnel towards SRv6 locator from remote domain) is tunneled in each domain via intra-domain tunnel (tunnel towards SRv6 locator in the same domain). In this blog post, we will discuss details of this architecture.
Base Configuration for Exchanging Transport Prefixes
For reference, Configuration 1 shows base configuration for IS-IS on P1 router. Similar configuration is deployed on all other P and PE routers in the topology, and details of this configuration were discussed in previous SRv6 blog posts. Please note, inter-AS link (ge-0/0/3) is not included in IS-IS.
1 routing-options {
2 source-packet-routing {
3 srv6 {
4 locator SL-000 fc01:100:1::/48;
5 no-reduced-srh;
6 }
7 }
8 resolution {
9 preserve-nexthop-hierarchy;
10 }
11 router-id 198.51.100.1;
12 autonomous-system 65501;
13 ipv6-router-id 2001:db8:bad:cafe:100::1;
14 forwarding-table {
15 export PS-LOAD-BALANCE;
16 }
17 }
18 protocols {
19 isis {
20 apply-groups GR-ISIS;
21 interface ge-0/0/0.0;
22 interface ge-0/0/1.0;
23 interface ge-0/0/2.0;
24 interface lo0.0 {
25 passive;
26 }
27 source-packet-routing {
28 srv6 {
29 locator SL-000 {
30 end-sid fc01:100:1::;
31 }
32 }
33 }
34 level 1 disable;
35 level 2 {
36 wide-metrics-only;
37 }
38 reference-bandwidth 1000g;
39 no-ipv4-routing;
40 }
41 }
Configuration 1: Base IS-IS configuration on P1
There is no IS-IS connectivity between two domains, so before BGP sessions outlined in Figure 1 could be established, connectivity between loopbacks of remote domains must be provided. This is the task of BGP sessions outlined in Figure 2. Therefore, let’s have a look at base BGP configuration for distributing loopbacks (to support establishment of multi-hop eBGP sessions outlined in Figure 1), as well as distributing SRv6 locators (to support creation of end-to-end SRv6 tunnels for L3VPN services), taking as an example PE11 (Configuration 2).
1 policy-options {
2 policy-statement PS-BGP-IPV6-EXP {
3 term TR-LOCAL-LOOPBACK {
4 from {
5 protocol direct;
6 rib inet6.0;
7 interface lo0.0;
8 route-filter 2001:db8:bad:cafe:100::11/128 exact;
9 }
10 then {
11 community add CM-LOOPBACK-65501;
12 accept;
13 }
14 }
15 term TR-LOCAL-LOCATOR {
16 from {
17 rib inet6.0;
18 route-filter fc01:100:11::/48 exact;
19 }
20 then {
21 community add CM-LOCATOR-65501;
22 accept;
23 }
24 }
25 then reject;
26 }
27 community CM-LOCATOR-65501 members 65501:1002;
28 community CM-LOOPBACK-65501 members 65501:1001;
29 }
30 protocols {
31 bgp {
32 (…)
33 group GR-IBGP-IPV6-TRANSPORT {
34 local-address 2001:db8:bad:cafe:100::11;
35 family inet6 {
36 unicast;
37 }
38 export PS-BGP-IPV6-EXP;
39 neighbor 2001:db8:bad:cafe:100::1 {
40 description P1;
41 }
42 neighbor 2001:db8:bad:cafe:100::2 {
43 description P2;
44 }
45 }
46 (…)
47 }
48 }
Configuration 2: BGP distribution of transport IPv6 prefixes on PE11
It is pretty standard configuration advertising local loopback and local SRv6 locator with some geographic community. These communities will be used later in BGP policies.
ASBR routers (P1, P2, P3, P4) have different BGP configuration, as they perform different tasks (Configuration 3).
1 policy-options {
2 policy-statement PS-EBGP-IMP {
3 then accept;
4 }
5 policy-statement PS-EBGP-IPV6-EXP {
6 term TR-IPV6 {
7 from {
8 protocol bgp;
9 rib inet6.0;
10 community CM-LOCAL-AS;
11 }
12 then {
13 next-hop self;
14 accept;
15 }
16 }
17 then reject;
18 }
19 policy-statement PS-IBGP-IPV6-EXP {
20 term TR-LOCAL-AS {
21 from community CM-LOCAL-AS;
22 then reject;
23 }
24 term TR-REMOTE-AS {
25 from {
26 protocol bgp;
27 rib inet6.0;
28 }
29 then {
30 next-hop self;
31 accept;
32 }
33 }
34 then reject;
35 }
36 community CM-LOCAL-AS members 65501:*;
37 }
38 protocols {
39 bgp {
40 (…)
41 group GR-IBGP-IPV6-TRANSPORT {
42 local-address 2001:db8:bad:cafe:100::1;
43 family inet6 {
44 unicast;
45 }
46 export PS-IBGP-IPV6-EXP;
47 neighbor 2001:db8:bad:cafe:100::11 {
48 description PE11;
49 }
50 neighbor 2001:db8:bad:cafe:100::12 {
51 description PE12;
52 }
53 neighbor 2001:db8:bad:cafe:100::2 {
54 description P2;
55 }
56 }
57 group GR-EBGP-IPV6-TRANSPORT {
58 local-address 2001:db8:beef::0104:1;
59 import PS-EBGP-IMP;
60 family inet6 {
61 unicast;
62 }
63 export PS-EBGP-IPV6-EXP;
64 peer-as 65502;
65 neighbor 2001:db8:beef::0104:4 {
66 description P4;
67 }
68 }
69 (…)
70 defaults {
71 ebgp {
72 no-policy {
73 receive reject-always;
74 advertise reject-always;
75 }
76 }
77 }
78 }
Configuration 3: BGP distribution of transport IPv6 prefixes on P1
Again, pretty standard configuration. Both export policies (for iBGP group, and for eBGP group) use next-hop self (lines 13 and 30), as discussed earlier. There is some basic loop prevention mechanism based on AS specific communities (lines 10, 21, and 36), as best common practice. iBGP sessions are multi-hop (established between loopbacks), while eBGP session is single-hop (established between link addresses).
Additionally, as best common practice, based on RFC 8212, eBGP session, by default, should block prefix exchange, for both inbound and outbound direction. RFC 8212 compliant behavior is achieved in Junos with explicit configuration (lines 70-77). Accepting and sending prefixes requires thus explicit import and export policies. Therefore, in addition to explicit export policy (line 63), an explicit import policy is defined as well (lines 2-4, and 59). For simplification, this blog post uses ‘allow all’ policy as import policy. In real deployment, you probably define some more restrictive policy here. Please note this implicit import policy is needed for eBGP session only.
Verification of inter-AS transport path
Now, having the transport prefix distribution in place, let’s verify it! Starting on PE11 (CLI-Output 1).
1 kszarkowicz@PE11> show route advertising-protocol bgp 2001:db8:bad:cafe:100::1 detail
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 * 2001:db8:bad:cafe:100::11/128
(1 entry, 1 announced)
5 BGP group GR-IBGP-IPV6-TRANSPORT type Internal
6 Nexthop: Self
7 Localpref: 100
8 AS path: [65501] I
9 Communities: 65501:1001
10
11 * fc01:100:11::/48
(1 entry, 1 announced)
12 BGP group GR-IBGP-IPV6-TRANSPORT type Internal
13 Nexthop: Self
14 MED: 0
15 Localpref: 100
16 AS path: [65501] I
17 Communities: 65501:1002
CLI-Output 1: Advertising IPv6 Prefixes on PE11
Looks good! PE11 advertises its own loopback (line 4) and own SRv6 locator to P1 (line 11). Also, appropriate communities are attached (line 9 and line 17). Now, let’s check next router, P1 (CLI-Output 2).
1 kszarkowicz@P1> show route advertising-protocol bgp 2001:db8:beef::104:4 detail
2
3 inet6.0: 29 destinations, 37 routes (29 active, 0 holddown, 0 hidden)
4 2001:db8:bad:cafe:100::11/128 (2 entries, 2 announced)
5 BGP group GR-EBGP-IPV6-TRANSPORT type External
6 Nexthop: Self
7 Flags: Nexthop Change
8 AS path: [65501] I
9 Communities: 65501:1001
10
11 2001:db8:bad:cafe:100::12/128 (2 entries, 2 announced)
12 BGP group GR-EBGP-IPV6-TRANSPORT type External
13 Nexthop: Self
14 Flags: Nexthop Change
15 AS path: [65501] I
16 Communities: 65501:1001
17
18 fc01:100:11::/48 (2 entries, 2 announced)
19 BGP group GR-EBGP-IPV6-TRANSPORT type External
20 Nexthop: Self
21 Flags: Nexthop Change
22 AS path: [65501] I
23 Communities: 65501:1002
24
25 fc01:100:12::/48 (2 entries, 2 announced)
26 BGP group GR-EBGP-IPV6-TRANSPORT type External
27 Nexthop: Self
28 Flags: Nexthop Change
29 AS path: [65501] I
30 Communities: 65501:1002
CLI-Output 2: Advertising IPv6 Prefixes over eBGP on P1
Looks also good! Loopbacks and SRv6 locators from PE11 (lines 4 and 18) and PE12 (lines 11 and 12) are sent by P1 to P4 in another AS. Also, as instructed in the configuration, next-hop is changed to self (lines 6-7, 13-14, 20-21, 27-28). Now, let’s check next router, P4 (CLI-Output 3).
1 kszarkowicz@P4> show route advertising-protocol bgp 2001:db8:bad:cafe:200::21 detail
2
3 inet6.0: 29 destinations, 37 routes (29 active, 0 holddown, 0 hidden)
4 * 2001:db8:bad:cafe:100::11/128 (2 entries, 1 announced)
5 BGP group GR-IBGP-IPV6-TRANSPORT type Internal
6 Nexthop: Self
7 Flags: Nexthop Change
8 Localpref: 100
9 AS path: [65502] 65501 I
10 Communities: 65501:1001
11
12 * 2001:db8:bad:cafe:100::12/128 (2 entries, 1 announced)
13 BGP group GR-IBGP-IPV6-TRANSPORT type Internal
14 Nexthop: Self
15 Flags: Nexthop Change
16 Localpref: 100
17 AS path: [65502] 65501 I
18 Communities: 65501:1001
19
20 * fc01:100:11::/48 (2 entries, 1 announced)
21 BGP group GR-IBGP-IPV6-TRANSPORT type Internal
22 Nexthop: Self
23 Flags: Nexthop Change
24 Localpref: 100
25 AS path: [65502] 65501 I
26 Communities: 65501:1002
27
28 * fc01:100:12::/48 (2 entries, 1 announced)
29 BGP group GR-IBGP-IPV6-TRANSPORT type Internal
30 Nexthop: Self
31 Flags: Nexthop Change
32 Localpref: 100
33 AS path: [65502] 65501 I
34 Communities: 65501:1002
CLI-Output 3: Advertising IPv6 Prefixes over iBGP on P4
Again, everything looks fine. Again, when P4 sends the transport prefixes to PE21, next-hop of transport prefixes from remote AS is changed to self, which is what we wanted (lines 6-7, 13-14, 20-21, 27-28). Just last check on PE21 (CLI-Output 4).
1 kszarkowicz@PE21> show route receive-protocol bgp 2001:db8:bad:cafe:200::4 detail
2
3 inet.0: 6 destinations, 6 routes (6 active, 0 holddown, 0 hidden)
4
5 RI-VRF20.inet.0: 12 destinations, 22 routes (7 active, 0 holddown, 10 hidden)
6
7 RI-VRF30.inet.0: 12 destinations, 22 routes (7 active, 0 holddown, 10 hidden)
8
9 iso.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
10
11 mpls.0: 2 destinations, 2 routes (2 active, 0 holddown, 0 hidden)
12
13 bgp.l3vpn.0: 42 destinations, 42 routes (22 active, 0 holddown, 20 hidden)
14
15 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
16 2001:db8:bad:cafe:100::11/128 (2 entries, 1 announced)
17 Accepted MultipathContrib
18 Nexthop: 2001:db8:bad:cafe:200::4
19 Localpref: 100
20 AS path: 65501 I
21 Communities: 65501:1001
22
23 2001:db8:bad:cafe:100::12/128 (2 entries, 1 announced)
24 Accepted MultipathContrib
25 Nexthop: 2001:db8:bad:cafe:200::4
26 Localpref: 100
27 AS path: 65501 I
28 Communities: 65501:1001
29
30 fc01:100:11::/48 (2 entries, 1 announced)
31 Accepted MultipathContrib
32 Nexthop: 2001:db8:bad:cafe:200::4
33 Localpref: 100
34 AS path: 65501 I
35 Communities: 65501:1002
36
37 fc01:100:12::/48 (2 entries, 1 announced)
38 Accepted MultipathContrib
39 Nexthop: 2001:db8:bad:cafe:200::4
40 Localpref: 100
41 AS path: 65501 I
42 Communities: 65501:1002
43
44 inet6.3: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
45
46 RI-VRF20.inet6.0: 15 destinations, 25 routes (10 active, 0 holddown, 10 hidden)
47
48 RI-VRF30.inet6.0: 15 destinations, 25 routes (10 active, 0 holddown, 10 hidden)
49
50 bgp.l3vpn-inet6.0: 44 destinations, 44 routes (24 active, 0 holddown, 20 hidden)
51
52 bgp.rtarget.0: 4 destinations, 10 routes (4 active, 0 holddown, 0 hidden)
CLI-Output 4: Receiving IPv6 Prefixes on PE21
Perfect! PE21 sees remote transport prefixes received from P4 with the BGP NEXT_HOP of P4 loopback (lines 18, 25, 32, 39). All ASBRs change the next-hop to self, so in theory we should be able now to ping between remote PEs (CLI-Output 5).
1 kszarkowicz@PE21> ping 2001:db8:bad:cafe:100::11 count 1
2 PING6(56=40+8+8 bytes) 2001:db8:beef:200::321:21 --> 2001:db8:bad:cafe:100::11
3
4 --- 2001:db8:bad:cafe:100::11 ping6 statistics ---
5 1 packets transmitted, 0 packets received, 100% packet loss
CLI-Output 5: Ping from PE22 to PE11 (first try)
Unfortunately, although prefixes are correctly exchanged (verification in only one direction was shown in this blog post, but other direction is similar), ping doesn’t work (line 5)!
Oops, wait a minute! We exchanged only loopbacks and SRv6 locators. And, our ping is sourced by default from interface address (line 2). So, let’s try a ping sourced from loopback (CLI-Output 6).
1 kszarkowicz@PE21> ping source 2001:db8:bad:cafe:200::21 2001:db8:bad:cafe:100::11 count 1
2 PING6(56=40+8+8 bytes) 2001:db8:bad:cafe:200::21 --> 2001:db8:bad:cafe:100::11
3 16 bytes from 2001:db8:bad:cafe:100::11, icmp_seq=0 hlim=62 time=8.762 ms
4
5 --- 2001:db8:bad:cafe:100::11 ping6 statistics ---
6 1 packets transmitted, 1 packets received, 0% packet loss
7 round-trip min/avg/max/std-dev = 8.762/8.762/8.762/0.000 ms
8
9 kszarkowicz@PE21> ping source 2001:db8:bad:cafe:200::21 fc01:100:11:: count 1
10 PING6(56=40+8+8 bytes) 2001:db8:bad:cafe:200::21 --> fc01:100:11::
11 16 bytes from 2001:db8:beef:100::111:11, icmp_seq=0 hlim=62 time=13.639 ms
12
13 --- fc01:100:11:: ping6 statistics ---
14 1 packets transmitted, 1 packets received, 0% packet loss
15 round-trip min/avg/max/std-dev = 13.639/13.639/13.639/0.000 ms
CLI-Output 6: Ping from PE22 to PE11 (second try)
Fortunately, sourcing the ping from the local loopback solves the issue. Both remote loopback and remote SRv6 locator (or, remote SRv6 END SID, to be more precise – check SRv6 Basics: Locator and End SIDs blog post for more details) are reachable (lines 7 and 14).
Base Configuration for exchanging L3VPN prefixes
It is now time to look for L3VPN prefix distribution. As indicated earlier (Figure 1), in this blog post we exchange L3VPN prefix via full mesh of iBGP/eBGP sessions between PE routers. Let’s look at PE11 for an example BGP configuration (Configuration 4).
1 policy-options {
2 policy-statement PS-EBGP-IMP {
3 then accept;
4 }
5 policy-statement PS-EBGP-L3VPN-EXP {
6 term TR-EBGP {
7 from {
8 protocol bgp;
9 external;
10 }
11 then reject;
12 }
13 term TR-RTC {
14 from rib bgp.rtarget.0;
15 then accept;
16 }
17 term TR-L3VPN {
18 from community RT;
19 then {
20 community delete CM-RTE-TYPE;
21 accept;
22 }
23 }
24 then reject;
25 }
26 community CM-RTE-TYPE members 0x306:*:*;
27 community RT members target:*:*;
28 }
29 protocols {
30 bgp {
31 (…)
32 group GR-IBGP-L3VPN {
33 local-address 2001:db8:bad:cafe:100::11;
34 family inet-vpn {
35 unicast {
36 extended-nexthop;
37 advertise-srv6-service;
38 accept-srv6-service;
39 }
40 }
41 family inet6-vpn {
42 unicast {
43 advertise-srv6-service;
44 accept-srv6-service;
45 }
46 }
47 family route-target;
48 neighbor 2001:db8:bad:cafe:100::12 {
49 description P12;
50 }
51 }
52 group GR-EBGP-L3VPN {
53 multihop {
54 no-nexthop-change;
55 }
56 local-address 2001:db8:bad:cafe:100::11;
57 import PS-EBGP-IMP;
58 family inet-vpn {
59 unicast {
60 extended-nexthop;
61 advertise-srv6-service;
62 accept-srv6-service;
63 }
64 }
65 family inet6-vpn {
66 unicast {
67 advertise-srv6-service;
68 accept-srv6-service;
69 }
70 }
71 family route-target;
72 export PS-EBGP-L3VPN-EXP;
73 peer-as 65502;
74 neighbor 2001:db8:bad:cafe:200::21 {
75 description P21;
76 }
77 neighbor 2001:db8:bad:cafe:200::22 {
78 description P22;
79 }
80 }
81 (…)
82 multipath {
83 list-nexthop;
84 }
85 rfc8950-compliant;
86 (…)
Configuration 4: BGP distribution of L3VPN prefixes on PE11
Similar configuration is deployed on all other PE routers as well. Most of the stanzas were explained in the previous SRv6 blog posts, so we will concentrate here only on a few new aspects of the configuration specific to Inter-AS Option C.
As mentioned earlier, in Inter-AS Option C the L3VPN service prefixes must be sent without changing the NEXT_HOP attribute. This is the default behavior for iBGP, but not for eBGP, where NEXT_HOP attribute is changed to the local address of the BGP session. To prevent this change, dedicated configuration is required (lines 53-54). In the particular example of this blog post, this configuration is, though, not strictly needed, as there are direct eBGP sessions between PE routers. However, if eBGP sessions distributing L3VPN prefixes are established between RRs, as mentioned earlier in the introduction, explicit change of NEXT_HOP attribute on RRs would be required.
Further, as already discussed in the context of exchanging transport prefixes over eBGP sessions, we need some explicit import (line 57) and export (line 72) policies in eBGP group. Import BGP policy (lines 2-4) is the same as already discussed. Export BGP policy (lines 5-23) is some simple policy to advertises RT constraints NLRIs, as well as local L3VPN prefixes. In the test lab, OSPF v3 is used as PE-CE protocol (configuration not shown for brevity), so as we remove OSPF Route Type extended community (line 20), so that information about original OSPF Route Type or OSPF area is lost. In that way, on remote PE the prefixes are imported into OSPF as Type 5 (external) prefixes. Depending on the actual deployment, this BGP export policy might differ significantly between use cases.
Verification of L3VPN prefix distribution
Having this basic configuration in place, let’s have a look, if L3VPN prefix distribution works properly (CLI-Output 7).
1 kszarkowicz@PE11> show bgp summary
2 (…)
3 Peer AS InPkt OutPkt OutQ Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
4 2001:db8:bad:cafe:100::1 65501 9454 9421 0 0 2d 23:33:28 Establ
5 inet6.0: 4/4/4/0
6 2001:db8:bad:cafe:100::2 65501 9454 9421 0 0 2d 23:33:22 Establ
7 inet6.0: 4/4/4/0
8 2001:db8:bad:cafe:100::12 65501 9469 9476 0 0 2d 23:33:14 Establ
9 bgp.rtarget.0: 0/4/4/0
10 bgp.l3vpn.0: 10/10/10/0
11 bgp.l3vpn-inet6.0: 10/10/10/0
12 RI-VRF20.inet.0: 0/5/5/0
13 RI-VRF30.inet.0: 0/5/5/0
14 RI-VRF20.inet6.0: 0/5/5/0
15 RI-VRF30.inet6.0: 0/5/5/0
16 2001:db8:bad:cafe:200::21 65502 9452 9451 0 0 2d 23:33:13 Establ
17 bgp.rtarget.0: 2/2/2/0
18 bgp.l3vpn.0: 0/10/10/0
19 bgp.l3vpn-inet6.0: 0/10/10/0
20 RI-VRF20.inet.0: 0/5/5/0
21 RI-VRF30.inet.0: 0/5/5/0
22 RI-VRF20.inet6.0: 0/5/5/0
23 RI-VRF30.inet6.0: 0/5/5/0
24 2001:db8:bad:cafe:200::22 65502 9467 9421 0 0 2d 23:33:09 Establ
25 bgp.rtarget.0: 2/2/2/0
26 bgp.l3vpn.0: 0/10/10/0
27 bgp.l3vpn-inet6.0: 0/10/10/0
28 RI-VRF20.inet.0: 0/5/5/0
29 RI-VRF30.inet.0: 0/5/5/0
30 RI-VRF20.inet6.0: 0/5/5/0
31 RI-VRF30.inet6.0: 0/5/5/0
CLI-Output 7: BGP session state on PE11
All expected BGP sessions are up. So, it means transport connectivity between remote PEs is OK (and, we verified that previously with ping). For each VRF we are advertising/receiving five IPv4 and five IPv6 prefixes (lines 12-15, 20-23, 28-31) over each BGP session distributing L3VPN prefixes. However, what is suspicious, we don’t accept any prefix (see ‘0’ accepted prefixes)! Neither from iBGP neighbor (lines 8-15), nor from eBGP neighbor (lines 16-31). So, we need a closer look at it (CLI-Output 8).
1 kszarkowicz@PE11> show route receive-protocol bgp 2001:db8:bad:cafe:200::21 table RI-VRF20.inet.0 hidden detail2
3 RI-VRF20.inet.0: 12 destinations, 22 routes (7 active, 0 holddown, 10 hidden)
4 20.21.92.0/24 (2 entries, 0 announced)
5 Import Accepted
6 Route Distinguisher: 198.51.100.21:20
7 VPN Label: 16
8 Nexthop: 2001:db8:bad:cafe:200::21
9 AS path: 65502 I
10 Communities: target:65000:20
11
12 20.22.92.0/24 (2 entries, 0 announced)
13 Import Accepted
14 Route Distinguisher: 198.51.100.21:20
15 VPN Label: 16
16 Nexthop: 2001:db8:bad:cafe:200::21
17 MED: 2
18 AS path: 65502 I
19 Communities: target:65000:20
20
21 192.168.20.21/32 (2 entries, 0 announced)
22 Import Accepted
23 Route Distinguisher: 198.51.100.21:20
24 VPN Label: 16
25 Nexthop: 2001:db8:bad:cafe:200::21
26 AS path: 65502 I
27 Communities: target:65000:20
28
29 192.168.20.22/32 (2 entries, 0 announced)
30 Import Accepted
31 Route Distinguisher: 198.51.100.21:20
32 VPN Label: 16
33 Nexthop: 2001:db8:bad:cafe:200::21
34 MED: 2
35 AS path: 65502 I
36 Communities: target:65000:20
37
38 192.168.20.92/32 (2 entries, 0 announced)
39 Import Accepted
40 Route Distinguisher: 198.51.100.21:20
41 VPN Label: 16
42 Nexthop: 2001:db8:bad:cafe:200::21
43 MED: 1
44 AS path: 65502 I
45 Communities: target:65000:20
CLI-Output 8: L3VPN prefixes received at PE11 from PE21
Hmm. Prefixes received at PE11 from PE21 look pretty OK. Next-hop is set to PE21 loopback (lines 8, 16, 25, 33, 42), there are correct route-targets attached (lines 10, 19, 27, 36, 45). At the first sight, nothing really suspicious. So, let’s look further (CLI-Output 9.)
1 kszarkowicz@PE11> show route 192.168.20.92/32 table RI-VRF20 hidden extensive
2
3 RI-VRF20.inet.0: 12 destinations, 22 routes (7 active, 0 holddown, 10 hidden)
4 192.168.20.92/32 (2 entries, 0 announced)
5 BGP Preference: 170/-101
6 Route Distinguisher: 192.169.2.22:20
7 Next hop type: Unusable, Next hop index: 0
8 Address: 0x7b3f394
9 Next-hop reference count: 80, key opaque handle: 0x0, non-key opaque handle: 0x0
10 Source: 2001:db8:bad:cafe:200::22
11 State: <Secondary Hidden Ext ProtectionCand>
12 Local AS: 65501 Peer AS: 65502
13 Age: 35:59 Metric: 1
14 Validation State: unverified
15 Task: BGP_65502.2001:db8:bad:cafe:200::22
16 AS path: 65502 I
17 Communities: target:65000:20
18 Import Accepted
19 VPN Label: 16
20 Localpref: 100
21 Router ID: 198.51.100.22
22 Primary Routing Table: bgp.l3vpn.0
23 Thread: junos-main
24 Indirect next hops: 1
25 Protocol next hop: 2001:db8:bad:cafe:200::22
26 Label operation: Push 16
27 Label TTL action: prop-ttl
28 Load balance label: Label 16: None;
29 Indirect next hop: 0x0 - INH Session ID: 0
30 (…)
CLI-Output 9: Detailed view of remote L3VPN prefix on PE11
Well, there is something wrong with the protocol next hop (line 7). Protocol next hop is the loopback of remote PE (line 25). So, what is wrong with it? Let’s figure it out (CLI-Output 10).
1 kszarkowicz@PE11> show route 2001:db8:bad:cafe:200::22
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 + = Active Route, - = Last Active, * = Both
5
6 2001:db8:bad:cafe:200::22/128
7 *[BGP/170] 2d 23:57:32, localpref 100, from 2001:db8:bad:cafe:100::1
8 AS path: 65502 I, validation-state: unverified
9 > to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
10 to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
11 [BGP/170] 2d 23:57:32, localpref 100, from 2001:db8:bad:cafe:100::2
12 AS path: 65502 I, validation-state: unverified
13 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
14 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
CLI-Output 10: PE22 loopback visibility on PE11
Well, remote loopback is there. And we can ping remote loopback (as we checked earlier, when discussing transport connectivity between AS-es – CLI-Output 6).
But, looking from L3VPN perspective, the problem is that the remote loopback is only in inet6.0, but not in inet6.3. For L3VPN next-hop resolution, next-hops must be in inet6.3. Saying that, we are now looking at L3VPN over SRv6, so as discussed in SRv6 SID Encoding and Transposition blog post the next-hop resolution happens via SRv6 SID announced together with L3VPN prefix, and not via NEXT_HOP attribute. Therefore, we should rather look for SRv6 SIDs, and not NEXT_HOP attributes, if there is some problem with L3VPN next-hop resolution.
So, let’s see, what SRv6 SIDs we are receiving from remote PE (CLI-Output 8). Ooooops! We don’t receive any (CLI-Output 8)! Does remote PE send at all SRv6 SID (CLI-Output 11)?
1 kszarkowicz@PE22> show route advertising-protocol bgp 2001:db8:bad:cafe:100::11 detail table RI-VRF20.inet.0
2
3 RI-VRF20.inet.0: 7 destinations, 12 routes (7 active, 0 holddown, 0 hidden)
4 * 20.21.92.0/24 (2 entries, 1 announced)
5 BGP group GR-EBGP-L3VPN type External
6 Route Distinguisher: 192.169.2.22:20
7 VPN Label: 16
8 Nexthop: Self
9 Flags: Nexthop Change
10 MED: 2
11 AS path: [65502] I
12 Communities: target:65000:20
13
14 * 20.22.92.0/24 (2 entries, 1 announced)
15 BGP group GR-EBGP-L3VPN type External
16 Route Distinguisher: 192.169.2.22:20
17 VPN Label: 16
18 Nexthop: Self
19 Flags: Nexthop Change
20 AS path: [65502] I
21 Communities: target:65000:20
22
23 * 192.168.20.21/32 (2 entries, 1 announced)
24 BGP group GR-EBGP-L3VPN type External
25 Route Distinguisher: 192.169.2.22:20
26 VPN Label: 16
27 Nexthop: Self
28 Flags: Nexthop Change
29 MED: 2
30 AS path: [65502] I
31 Communities: target:65000:20
32
33 * 192.168.20.22/32 (2 entries, 1 announced)
34 BGP group GR-EBGP-L3VPN type External
35 Route Distinguisher: 192.169.2.22:20
36 VPN Label: 16
37 Nexthop: Self
38 Flags: Nexthop Change
39 AS path: [65502] I
40 Communities: target:65000:20
41
42 * 192.168.20.92/32 (2 entries, 1 announced)
43 BGP group GR-EBGP-L3VPN type External
44 Route Distinguisher: 192.169.2.22:20
45 VPN Label: 16
46 Nexthop: Self
47 Flags: Nexthop Change
48 MED: 1
49 AS path: [65502] I
50 Communities: target:65000:20
CLI-Output 11: PE22 advertisements towards PE11
Well, it doesn’t.
The problem is, that by default, for increased security, BGP Prefix SID (which includes SRv6 SID – please check lines 181-212 in Packet Capture 1 of SRv6 SID Encoding and Transposition blog post) is neither advertised, nor accepted over eBGP session. Advertising/accepting BGP Prefix SID requires explicit configuration, as outlined in Configuration 5.
1 protocols {
2 bgp {
3 group GR-EBGP-L3VPN {
4 advertise-prefix-sid;
5 accept-prefix-sid;
6 }
7 }
8 }
Configuration 5: Enabling advertisements/acceptance of BGP Prefix SID
With this change in place (on all PEs), now we can receive SRv6 SIDs from remote PEs (CLI-Output 12).
1 kszarkowicz@PE11> show route receive-protocol bgp 2001:db8:bad:cafe:200::21 table RI-VRF20.inet.0 hidden detail | match SID
2 SRv6 SID: fc01:200:21:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
3 SRv6 SID: fc01:200:21:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
4 SRv6 SID: fc01:200:21:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
5 SRv6 SID: fc01:200:21:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
6 SRv6 SID: fc01:200:21:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
CLI-Output 12: SRv6 SIDs received from PE21 on PE11
Now, checking again one of the remote prefixes (CLI-Output 13 and CLI-Output 14) we see some changes.
1 kszarkowicz@PE11> show route receive-protocol bgp 2001:db8:bad:cafe:200::22 table RI-VRF20.inet.0 hidden detail 192.168.20.92/32
2
3 RI-VRF20.inet.0: 12 destinations, 22 routes (7 active, 0 holddown, 10 hidden)
4 192.168.20.92/32 (2 entries, 0 announced)
5 Import Accepted MultiNexthop RecvNextHopIgnored
6 Route Distinguisher: 192.169.2.22:20
7 VPN Label: 16896
8 Nexthop: 2001:db8:bad:cafe:200::22
9 MED: 1
10 AS path: 65502 I
11 Communities: target:65000:20
12 SRv6 SID: fc01:200:22:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
CLI-Output 13: Remote L3VPN prefix received from PE
1 kszarkowicz@PE11> show route 192.168.20.92/32 table RI-VRF20 hidden extensive
2
3 RI-VRF20.inet.0: 12 destinations, 22 routes (7 active, 0 holddown, 10 hidden)
4 192.168.20.92/32 (2 entries, 0 announced)
5 BGP Preference: 170/-101
6 Route Distinguisher: 192.169.2.22:20
7 Next hop type: Unusable, Next hop index: 0
8 Address: 0x7b3f394
9 Next-hop reference count: 80, key opaque handle: 0x0, non-key opaque handle: 0x0
10 Source: 2001:db8:bad:cafe:200::22
11 State: <Secondary Hidden Ext ProtectionCand>
12 Local AS: 65501 Peer AS: 65502
13 Age: 5:57 Metric: 1
14 Validation State: unverified
15 Task: BGP_65502.2001:db8:bad:cafe:200::22
16 AS path: 65502 I
17 Communities: target:65000:20
18 Import Accepted MultiNexthop RecvNextHopIgnored
19 SRv6 SID: fc01:200:22:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
20 VPN Label: 16896
21 Localpref: 100
22 Router ID: 198.51.100.22
23 Primary Routing Table: bgp.l3vpn.0
24 Thread: junos-main
25 Indirect next hops: 1
26 Protocol next hop: fc01:200:22::
27 Indirect next hop: 0x0 - INH Session ID: 0
28 (…)
CLI-Output 14: Detailed view of remote L3VPN prefix on PE11
BGP NEXT_HOP attribute is still remote loopback (CLI-Output 13, line 8). However, next-hop resolution is based on SRv6 SID (CLI-Output 13, line 12, and CLI-Output 14, lines 19 and 26), not on the NEXT_HOP attribute. Why it is like that was explained in SRv6 SID Encoding and Transposition blog post. These are good news.
Troubleshooting of L3VPN Next-Hop Resolution
However, the bad news is, the next-hop is still not resolved (CLI-Output 14, line7). So, we need to troubleshoot further (CLI-Output 15).
1 kszarkowicz@PE11> show route fc01:200:22::
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 + = Active Route, - = Last Active, * = Both
5
6 fc01:200:22::/48 *[BGP/170] 3d 02:52:03, localpref 100, from 2001:db8:bad:cafe:100::1
7 AS path: 65502 I, validation-state: unverified
8 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
9 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
10 [BGP/170] 3d 02:52:03, localpref 100, from 2001:db8:bad:cafe:100::2
11 AS path: 65502 I, validation-state: unverified
12 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
13 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
CLI-Output 15: PE22 SRv6 Locator visibility on PE11
Remote PE SRv6 locator is present only in inet6.0 RIB (Routing Information Base). For L3VPN next-hop resolution to work, the next-hop must be present in inet6.3 RIB, inet6.0 is not sufficient. As you probably remember from the very first SRv6 blog post (SRv6 Basics: Locator and End SID), SRv6 from local domain are distributed via IS-IS using two TLVs (Type-Length-Value): IPv6 prefix (IS-IS TLV 236: IPv6 IP Reachability), as well as SRv6 locator (IS-IS TLV 27: SRv6 Locator). First TLV is used to install the SRv6 locator into inet6.0 RIB as “normal” IPv6 prefix, while second TLV is used for inet6.3 RIB installation. With BGP, we are distributing both IPv6 loopbacks and SRv6 locators as “normal” IPv6 prefixes. So, what do we do? Let’s keep remote IPv6 loopbacks in inet6.0 RIB only, while install SRv6 locators into both inet6.0 and inet6.3 with a RIB group (Configuration 6).
1 policy-options {
2 policy-statement PS-IPV6-TRANSPORT {
3 term TR-LOCATOR {
4 from community CM-LOCATOR;
5 then accept;
6 }
7 then reject;
8 }
9 community CM-LOCATOR members *:1002;
10 }
11 routing-options {
12 rib-groups {
13 RG-IPV6-TRANSPORT {
14 import-rib [ inet6.0 inet6.3 ];
15 import-policy PS-IPV6-TRANSPORT;
16 }
17 }
18 }
19 protocols {
20 bgp {
21 group GR-IBGP-IPV6-TRANSPORT {
22 family inet6 {
23 unicast {
24 rib-group RG-IPV6-TRANSPORT;
25 }
26 }
27 }
28 }
29 }
Configuration 6: Installing remote SRv6 locators in inet6.3 RIB
The configuration has a RIB group (lines 12-17) listing two RIBs: inet6.0 (primary import RIB) and inet6.3 (secondary import RIB). This RIB group is attached to BGP IPv6 unicast address family (line 22-26). Thus, all received IPv6 unicast prefixes are undergoing the import route policy used by the RIB group (lines 1-10 and 15). Prefixes rejected by the route policy are installed into primary RIB only (inet6.0), while prefixes accepted by the route policy are installed into both primary and secondary RIBs (both inet6.0 and inet6.3).
So, what is the route policy doing? It accepts prefixes with community CM-LOCATOR. Which are our SRv6 locators, as we are attaching different communities for IPv6 loopbacks and SRv6 locators. Please check Configuration 2, line 21 and 28, as well as CLI-Output 1, line 17 as well.
This configuration is required only on PE routers. P (or ASBR) routers do not host L3VPN services, hence they do not require next-hop resolution via inet6.3 RIB.
Did this configuration change help? (CLI-Output 16).
1 kszarkowicz@PE11> show route fc01:200:22::
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 + = Active Route, - = Last Active, * = Both
5
6 fc01:200:22::/48 *[BGP/170] 3d 03:32:16, localpref 100, from 2001:db8:bad:cafe:100::1
7 AS path: 65502 I, validation-state: unverified
8 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
9 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
10 [BGP/170] 3d 03:32:16, localpref 100, from 2001:db8:bad:cafe:100::2
11 AS path: 65502 I, validation-state: unverified
12 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
13 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
14
15 inet6.3: 5 destinations, 7 routes (5 active, 0 holddown, 0 hidden)
16 + = Active Route, - = Last Active, * = Both
17
18 fc01:200:22::/48 *[BGP/170] 00:31:37, localpref 100, from 2001:db8:bad:cafe:100::1
19 AS path: 65502 I, validation-state: unverified
20 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
21 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
22 [BGP/170] 00:31:37, localpref 100, from 2001:db8:bad:cafe:100::2
23 AS path: 65502 I, validation-state: unverified
24 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
25 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
CLI-Output 16: PE22 SRv6 Locator visibility on PE11
Yes. It did! Remote SRv6 locator is now in both inet6.0 and inet6.3. Just for comparison, let’s check remote IPv6 loopback (CLI-Output 17).
1 kszarkowicz@PE11> show route 2001:db8:bad:cafe:200::22
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 + = Active Route, - = Last Active, * = Both
5
6 2001:db8:bad:cafe:200::22/128
7 *[BGP/170] 3d 03:55:30, localpref 100, from 2001:db8:bad:cafe:100::1
8 AS path: 65502 I, validation-state: unverified
9 > to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
10 to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
11 [BGP/170] 3d 03:55:30, localpref 100, from 2001:db8:bad:cafe:100::2
12 AS path: 65502 I, validation-state: unverified
13 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
14 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
CLI-Output 17: PE22 IPv6 Locator visibility on PE11
It looks, our RIB group works correctly. Remote IPv6 loopbacks are installed only into inet6.0, while remote SRv6 locators are installed into both inet6.0 and inet6.3. So far, so good.
Tunneling remote SRv6 Locators over SRv6 tunnels
Returning to our original problem of next-hop resolution of the L3VPN prefix. Did it help? Let’s check (CLI-Output 18).
1 kszarkowicz@PE11> show route 192.168.20.92/32 table RI-VRF20 hidden extensive
2
3 RI-VRF20.inet.0: 12 destinations, 22 routes (7 active, 0 holddown, 10 hidden)
4 192.168.20.92/32 (2 entries, 0 announced)
5 BGP Preference: 170/-101
6 Route Distinguisher: 192.169.2.22:20
7 Next hop type: Unusable, Next hop index: 0
8 Address: 0x7b3f394
9 Next-hop reference count: 88, key opaque handle: 0x0, non-key opaque handle: 0x0
10 Source: 2001:db8:bad:cafe:200::22
11 State: <Secondary Hidden Ext ProtectionCand>
12 Local AS: 65501 Peer AS: 65502
13 Age: 3:04:34 Metric: 1
14 Validation State: unverified
15 Task: BGP_65502.2001:db8:bad:cafe:200::22
16 AS path: 65502 I
17 Communities: target:65000:20
18 Import Accepted MultiNexthop RecvNextHopIgnored
19 SRv6 SID: fc01:200:22:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
20 VPN Label: 16896
21 Localpref: 100
22 Router ID: 198.51.100.22
23 Primary Routing Table: bgp.l3vpn.0
24 Thread: junos-main
25 Indirect next hops: 1
26 Protocol next hop: fc01:200:22::
27 Indirect next hop: 0x0 - INH Session ID: 0
28 (…)
CLI-Output 18: Detailed view of remote L3VPN prefix on PE11
Surprise, surprise – it didn’t! There is still some problem with the next-hop (lines 7 and 18), although our next-hop (SRv6 locator) is present in the inet6.3 RIB. So, what is still wrong here?
The problem is, despite installing some prefixes in inet6.3 RIB, it doesn’t make them ‘SRv6-tunneling capable’. For comparison, let’s look at some local (not remote) SRv6 locator (CLI-Output 19).
1 kszarkowicz@PE11> show route fc01:100:12::
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 + = Active Route, - = Last Active, * = Both
5
6 fc01:100:12::/48 *[IS-IS/18] 3d 16:42:01, metric 2000
7 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
8 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0
9
10 inet6.3: 5 destinations, 7 routes (5 active, 0 holddown, 0 hidden)
11 + = Active Route, - = Last Active, * = Both
12
13 fc01:100:12::/48 *[SRV6-ISIS/14] 3d 16:42:01, metric 2000
14 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0, SRV6-Tunnel, Dest: fc01:100:12::
15 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0, SRV6-Tunnel, Dest: fc01:100:12::
CLI-Output 19: PE12 SRv6 Locator visibility on PE11
If you compare CLI-Output 16 and CLI-Output 19, you see the major difference in inet6.3. Remote domain SRv6 locator (CLI-Output 16, lines 18-25) currently resolves over plain IPv6 next-hop, while local domain SRv6 locator (CLI-Output 19, lines 13-15) resolves over SRv6 tunnel. For that reason, in the current state in the example network, L3VPN resolution with local domain SRv6 works, while L3VPN resolution with remote domain SRv6 locator fails – we need SRv6 tunnel here.
So, how do we force remote SRv6 locator to use SRv6 tunnel (so that it is eligible as next-hop for L3VPN), and not plain IPv6 next-hop? To answer this question, let’s go back to almost the beginning of this blog post. In Configuration 3, on ASBRs (P1, P2, P3, P4), next-hop self is configured for all IPv6 prefixes (meaning all remote IPv6 loopbacks and remote SRv6 locators) advertised over iBGP sessions (line 30). This is classical Inter-AS Option C configuration. As the result of this action, these IPv6 prefixes (both remote IPv6 loopbacks and remote SRv6 locators) are advertised with ASBR’s loopback as NEXT_HOP attribute (e.g., CLI-Output 4, lines 18, 25, 32, 39). It also means, both remote IPv6 loopbacks and remote SRv6 locators, when finally arriving to a PE, are resolved over IPv6 loopback of ASBR. IPv6 loopbacks are not capable for SRv6 tunneling – we have SRv6 locators for SRv6 tunneling. Installing remote SRv6 locators in inet6.3 doesn’t change the resolution scheme – they are still resolved over non-SRv6-tunneling capable IPv6 loopback of ASBR.
Thus, to change this, we need to change the way SRv6 locators are advertised by ASBRs over iBGP sessions. Instead of using simply ‘next-hop self’, which causes the NEXT_HOP attribute being set to the source address of the iBGP session (i.e., local loopback), for SRv6 locators we need to change the NEXT_HOP attribute not to local loopback, but to local SRv6 locator (Configuration 7).
1 [edit policy-options policy-statement PS-IBGP-IPV6-EXP]
2 term TR-LOCAL-AS { ... }
3 + term TR-LOOPBACK {
4 + from {
5 + rib inet6.0;
6 + community CM-LOOPBACK;
7 + }
8 + then {
9 + next-hop self;
10 + accept;
11 + }
12 + }
13 + term TR-LOCATOR {
14 + from {
15 + protocol bgp;
16 + rib inet6.0;
17 + community CM-LOCATOR;
18 + }
19 + then {
20 + next-hop fc01:100:1::;
21 + accept;
22 + }
23 + }
24 - term TR-REMOTE-AS {
25 - from {
26 - protocol bgp;
27 - rib inet6.0;
28 - }
29 - then {
30 - next-hop self;
31 - accept;
32 - }
33 - }
34 [edit policy-options]
35 + community CM-LOCATOR members *:1002;
36 + community CM-LOOPBACK members *:1001;
37 }
Configuration 7: iBGP export policy on P1
The new iBGP export policy on ASBRs replaces the iBGP export policy used so far (defined in Configuration 3, lines 19-35). The difference is that the term TR-REMOTE-AS is split into two terms: TR-LOOPBACK and TR-LOCATOR. Remote IPv6 loopbacks are readvertised with ‘next-hop self’ action, as previously. However, remote SRv6 locators are readvertised on ASBRs with ‘next-hop <local-SRv6-END-SID>’ action.
Let’s check if that finally helped with L3VPN resolution (CLI-Output 20).
1 kszarkowicz@PE11> show route receive-protocol bgp 2001:db8:bad:cafe:100::1 table inet6
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 Prefix Nexthop MED Lclpref AS path
5 2001:db8:bad:cafe:200::21/128
6 * 2001:db8:bad:cafe:100::1 100 65502 I
7 2001:db8:bad:cafe:200::22/128
8 * 2001:db8:bad:cafe:100::1 100 65502 I
9 * fc01:200:21::/48 fc01:100:1:: 100 65502 I
10 * fc01:200:22::/48 fc01:100:1:: 100 65502 I
11
12 inet6.3: 5 destinations, 7 routes (5 active, 0 holddown, 0 hidden)
13 Prefix Nexthop MED Lclpref AS path
14 * fc01:200:21::/48 fc01:100:1:: 100 65502 I
15 * fc01:200:22::/48 fc01:100:1:: 100 65502 I
CLI-Output 20: Receiving IPv6 prefixes at PE11
Well, we certainly see some changes. Remote IPv6 loopbacks have ASBR’s loopback as NEXT_HOP attribute (lines 6 and 8), while remote SRv6 locators have ASBR’s SRv6 locator as NEXT_HOP attribute (lines 9, 10, 14, 15). Please compare previous similar output on PE21 (CLI-Output 4), where NEXT_HOP attribute for both loopbacks and SRv6 locators was the loopback of ASBR.
Does it make remote SRv6 locators SRv6-tunelling capable? Let’s check (CLI-Output 21).
1 kszarkowicz@PE11> show route fc01:200:22:: active-path
2
3 inet6.0: 27 destinations, 31 routes (27 active, 0 holddown, 0 hidden)
4 + = Active Route, - = Last Active, * = Both
5
6 fc01:200:22::/48 *[BGP/170] 00:29:58, localpref 100, from 2001:db8:bad:cafe:100::1
7 AS path: 65502 I, validation-state: unverified
8 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0, SRV6-Tunnel, Dest: fc01:100:1::
9 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0, SRV6-Tunnel, Dest: fc01:100:2::
10
11 inet6.3: 5 destinations, 7 routes (5 active, 0 holddown, 0 hidden)
12 + = Active Route, - = Last Active, * = Both
13
14 fc01:200:22::/48 *[BGP/170] 00:29:58, localpref 100, from 2001:db8:bad:cafe:100::1
15 AS path: 65502 I, validation-state: unverified
16 to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0, SRV6-Tunnel, Dest: fc01:100:1::
17 > to fe80::5604:dff:fe00:82a6 via ge-0/0/2.0, SRV6-Tunnel, Dest: fc01:100:2::
CLI-Output 21: PE22 SRv6 Locator visibility on PE11
Well. Now looks better (compared to CLI-Output 16), as now remote SRv6 locator is resolved over the SRv6 tunnel towards the SRv6 END SID of ASBR (lines 8, 9, 16, 17).
And, did it eventually resolve the problem with L3VPN resolution? Let’s check that as well (CLI-Output 22 and CLI-Output 23)
1 kszarkowicz@PE11> show route 192.168.20.92/32 table RI-VRF20 active-path
2 (…)
3 192.168.20.92/32 @[BGP/170] 00:37:32, MED 1, localpref 100, from 2001:db8:bad:cafe:200::21
4 AS path: 65502 I, validation-state: unverified
5 > to fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0, SRV6-Tunnel, Dest: fc01:100:1::
CLI-Output 22: View of remote L3VPN prefix on PE11
1 kszarkowicz@PE11> show route 192.168.20.92/32 table RI-VRF20 active-path extensive
2
3 RI-VRF20.inet.0: 12 destinations, 23 routes (12 active, 0 holddown, 0 hidden)
4 192.168.20.92/32 (3 entries, 2 announced)
5 State: <CalcForwarding>
6 TSI:
7 KRT in-kernel 192.168.20.92/32 -> {list:composite(686), composite(690)}
8 OSPF3 realm ipv4-unicast area : 0.0.0.0, LSA ID : 0.0.0.4, LSA type : Extern
9 @BGP Preference: 170/-101
10 Route Distinguisher: 198.51.100.21:20
11 Next hop type: Indirect, Next hop index: 0
12 Address: 0x7b41af4
13 Next-hop reference count: 10, key opaque handle: 0x0, non-key opaque handle: 0x0
14 Source: 2001:db8:bad:cafe:200::21
15 Next hop type: Chain, Next hop index: 687
16 Next hop: via Chain Tunnel Composite, SRv6
17 Next hop: ELNH Address 0x7b40d64, selected
18 SRV6-Tunnel: Reduced-SRH Encap-mode Remove-Last-Sid
19 Src: 2001:db8:bad:cafe:100::11 Dest: fc01:100:1::
20 Segment-list[0] fc01:100:1::
21 Next hop type: Router, Next hop index: 691
22 Address: 0x7b40d64
23 Next-hop reference count: 7, key opaque handle: 0x0, non-key opaque handle: 0x0
24 Next hop: fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0 weight 0x1
25 Protocol next hop: fc01:200:21::
26 Composite next hop: 0x74cbba0 686 INH Session ID: 370
27 Indirect next hop: 0x7e5c3c4 1048580 INH Session ID: 370
28 State: <Secondary Active Ext ProtectionCand>
29 Local AS: 65501 Peer AS: 65502
30 Age: 38:59 Metric: 1 Metric2: 1000
31 Validation State: unverified
32 ORR Generation-ID: 0
33 Task: BGP_65502.2001:db8:bad:cafe:200::21
34 Announcement bits (1): 2-RI-VRF20-OSPF3
35 AS path: 65502 I
36 Communities: target:65000:20
37 Import Accepted MultiNexthop RecvNextHopIgnored
38 SRv6 SID: fc01:200:21:: Behavior: 19 BL: 48 NL: 0 FL: 16 AL: 0 TL: 16 TO: 48
39 VPN Label: 16896
40 Localpref: 100
41 Router ID: 198.51.100.21
42 Primary Routing Table: bgp.l3vpn.0
43 Thread: junos-main
44 Composite next hops: 1
45 Protocol next hop: fc01:200:21:: Metric: 1000
46 Composite next hop: 0x74cbba0 686 INH Session ID: 370
47 Indirect next hop: 0x7e5c3c4 1048580 INH Session ID: 370
48 Indirect path forwarding next hops: 1
49 Next hop type: Chain
50 Next hop: fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
51 fc01:200:21::/48 Originating RIB: inet6.3
52 Metric: 1000 Node path count: 1
53 Indirect next hops: 1
54 Protocol next hop: fc01:100:1:: Metric: 1000
55 Inode flags: 0x204 path flags: 0x80
56 Path fnh link: 0x74c7820 path inh link: 0x7263280
57 Indirect next hop: 0x7e5ab44 1048575 INH Session ID: 367
58 Indirect path forwarding next hops: 1
59 Next hop type: Chain
60 Next hop: fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
61 fc01:100:1::/48 Originating RIB: inet6.3
62 Metric: 1000 Node path count: 1
63 Forwarding nexthops: 1
64 Next hop type: Chain
65 Next hop: fe80::5604:dff:fe00:2ba0 via ge-0/0/1.0
CLI-Output 23: Detailed view of remote L3VPN prefix on PE11
Wow! We did it! Remote L3VPN prefix is no longer hidden. It is resolved via SRv6 chain tunnel composite next-hop (line 16), using SRv6 tunnel sourced from local loopback and destined to ASBR SRv6 locator (lines 18-20).
And, just final check, to verify forwarding (CLI-Output 24).
1 kszarkowicz@CE91> ping 192.168.20.92 routing-instance RI-20 count 1
2 PING 192.168.20.92 (192.168.20.92): 56 data bytes
3 64 bytes from 192.168.20.92: icmp_seq=0 ttl=62 time=6.232 ms
4
5 --- 192.168.20.92 ping statistics ---
6 1 packets transmitted, 1 packets received, 0% packet loss
7 round-trip min/avg/max/stddev = 6.232/6.232/6.232/0.000 ms
CLI-Output 24: CE to CE verification
Uff! Together, we did it! CE-to-CE forwarding in SRv6 Inter-AS Option C finally works.
Inter-AS Option C with pure transit P routers
However, we are still not ready, yet. Let’s modify slightly the topology, by shutting down few links (Figure 3).
Figure 3: Modified topology
With this modification, the path within AS 65501 has one router in PE role (PE11), two pure transit (P role) routers (P2 and PE12) and one router in ASBR role (P1).
Does the ping between CEs still work (CLI-Output 25).
1 kszarkowicz@CE91> ping 192.168.20.92 routing-instance RI-20 count 1
2 PING 192.168.20.92 (192.168.20.92): 56 data bytes
3
4 --- 192.168.20.92 ping statistics ---
5 1 packets transmitted, 0 packets received, 100% packet loss
CLI-Output 25: CE to CE verification
Bad luck. After this change, ping doesn’t work any longer. So, no time to relax, as there is still job to do!
For troubleshooting, let’s start a ping with 10 pps (packets per second), so that we can observe, where the packets are dropped (CLI-Output 26).
1 kszarkowicz@CE91> ping 192.168.20.92 routing-instance RI-20 count 100000 interval 0.1
2 PING 192.168.20.92 (192.168.20.92): 56 data bytes
CLI-Output 26: CE to CE ping with 10 pps
And observe on the path from CE91 to CE92, which router drops the packets using ‘monitor interface traffic’ command (CLI-Output 27).
1 PE11 Seconds: 946 Time: 03:25:55
2
3 Interface Link Input packets (pps) Output packets (pps)
4 ge-0/0/0 Up 7753 (10) 889 (1)
5 lc-0/0/0 Up 0 0
6 pfh-0/0/0 Up 0 0
7 ge-0/0/1 Down 915 (0) 1001 (0)
8 ge-0/0/2 Up 2593 (0) 9588 (10)
9
10
11
12 P2 Seconds: 1009 Time: 03:27:14
13
14 Interface Link Input packets (pps) Output packets (pps)
15 ge-0/0/0 Up 841 (0) 1042 (0)
16 lc-0/0/0 Up 0 0
17 pfh-0/0/0 Up 0 0
18 ge-0/0/1 Up 10489 (10) 2732 (0)
19 ge-0/0/2 Up 2742 (1) 10469 (10)
20
21
22
23 PE12 Seconds: 0 Time: 03:27:54
24
25 Interface Link Input packets (pps) Output packets (pps)
26 ge-0/0/0 Up 361 (0) 920 (0)
27 lc-0/0/0 Up 0 0
28 pfh-0/0/0 Up 0 0
29 ge-0/0/1 Up 3121 (2) 11134 (10)
30 ge-0/0/2 Up 10916 (10) 2810 (1)
31
32
33
34 P1 Seconds: 1045 Time: 03:28:13
35
36 Interface Link Input packets (pps) Output packets (pps)
37 ge-0/0/0 Down 826 (0) 841 (0)
38 lc-0/0/0 Up 0 0
39 pfh-0/0/0 Up 0 0
40 ge-0/0/1 Up 1001 (0) 1144 (0)
41 ge-0/0/2 Up 11329 (10) 3155 (0)
42 ge-0/0/3 Up 720 (0) 832 (0)
CLI-Output 27: Traffic volumes at different routers in AS 65501
On routers PE11, P2 and PE12 we see 10 pps coming in, and 10 pps going out. So, we are good there. However, on router P1, 10 pps is coming, but there is no outgoing traffic. Thus, P1 is dropping. We need to figure out, what is the reason for P1 to drop the traffic.
For that, let’s capture the transit traffic (how to capture transit traffic is subject for another blog post) on couple of links, so that we can see the encapsulation of the original CE-to-CE packet (Figure 4, Figure 5)
Figure 4: Traffic capture on PE11-P2 link
Figure 5: Traffic capture on PE12-P1 link
What we can see (Figure 4) is that PE11 encapsulates the original IPv4 packet with single IPv6 header destined to END.DT4 SID on PE21. So, SRv6 tunnel to P1 SRv6 END SID is effectively not used. If you go back to CLI-Output 23, you can see that “Reduced-SRH Encap-mode Remove-Last-Sid” (line 18) is used. Therefore last SID, fc01:100:1:: (line 20), which is END SID of P1, is removed from the encapsulation before packet is sent out.
Now, packet with destination address fc01:200:21:420:: (END.DT4 SID on PE21) is sent to P2. Let’s check what routing instructions are there on P2 to route such packet (CLI-Output 28).
1 kszarkowicz@P2> show route fc01:200:21:420::
2
3 inet6.0: 28 destinations, 32 routes (28 active, 0 holddown, 0 hidden)
4 + = Active Route, - = Last Active, * = Both
5
6 fc01:200:21::/48 *[BGP/170] 00:27:50, localpref 100, from 2001:db8:bad:cafe:100::1
7 AS path: 65502 I, validation-state: unverified
8 > to fe80::5604:dff:fe00:1060 via ge-0/0/2.0, SRV6-Tunnel, Dest: fc01:100:1::
CLI-Output 28: PE21 SRv6 Locator visibility on P2
The instruction is to send the packet via SRv6 tunnel to the SRv6 END SID of P1 (line 8). This is what can be seen in Figure 5 – router P2 pushes additional header. Why is this instruction? You probably still remember, this is the result of Configuration 7.
Now, such packet arrives to P1. Destination address of the packet is END SID of P1, which means packet is subject to local consumption, and not forwarded further (please check SRv6 Basics: Locator and End SIDs blog post). And exactly this happens on P1 – packet is not forwarded.
So, what can we do? END SIDs can be advertised with different flavors (draft-ietf-lsr-isis-srv6-extensions, RFC8986):
- PSP – Penultimate Segment Pop
- USP – Ultimate Segment Pop
- USD – Ultimate Segment Decapsulation
Flavor can be explicitly configured. Junos SRv6 implementation supports all 3 flavors, so it can support SRv6 deployments requiring these flavors.
For the particular case observed in this blog post, PSP is of particular interest. With support for PSP advertised by P1, the penultimate node (PE12 in our example) will pop the outer segment (header added by P2). With this, packet will arrive to P1 with only one IPv6 header, as originally generated by PE11 (Figure 4).
In general, it is advisable to enable advertisement for support of all three flavors on all routers, so that configuration is prepared for different use cases requiring different flavors (Configuration 8).
1 protocols {
2 isis {
3 source-packet-routing {
4 srv6 {
5 locator SL-000 {
6 end-sid fc01:100:2:: {
7 flavor {
8 psp;
9 usp;
10 usd;
11 }
12 }
13 }
14 }
15 }
16 }
17 }
Configuration 8: Enabling advertisement of support for PSP, USP, USD END SID flavors
As the result, IS-IS advertises the support for configured END SID flavors (CLI-Output 29).
1 kszarkowicz@PE12> show isis database P2 extensive | match "locator| SID"
2 SRv6 Locator: fc01:100:2::/48, Metric: 0, MTID: 0, Flags: 0x0, Algorithm: 0
3 SRv6 SID: fc01:100:2::, Flavor: PSP, USP, USD
CLI-Output 29: IS-IS advertisement of END SID flavors
Now, all routers in the IS-IS domain (including PE12 router) know, that P2 supports PSP flavor (among other flavors) for its END SID (line 3). So, PE12 will can safely remove other header before sending the packet to P1. With this configuration change, finally CE-to-CE data plane connectivity works as well for the use case with transit P routers.
1 kszarkowicz@CE91> ping 192.168.20.92 routing-instance RI-20 count 1
2 PING 192.168.20.92 (192.168.20.92): 56 data bytes
3 64 bytes from 192.168.20.92: icmp_seq=0 ttl=62 time=8.894 ms
4
5 --- 192.168.20.92 ping statistics ---
6 1 packets transmitted, 1 packets received, 0% packet loss
7 round-trip min/avg/max/stddev = 8.894/8.894/8.894/0.000 ms
CLI-Output 30: CE to CE verification
Next steps
In the next blog post we will show an interesting use case, showing guaranteed link slicing with SRv6, where Function field is further divided to carry Slice ID and VPN ID.
Useful links
RFC4364: BGP/MPLS IP Virtual Private Networks
RFC 8212: Default External BGP (EBGP) Route Propagation Behavior without Policies
RFC8986: Segment Routing over IPv6 (SRv6) Network Programming
draft-ietf-lsr-isis-srv6-extensions: IS-IS Extensions to Support Segment Routing over IPv6 Dataplane
SRv6 in Junos: https://www.juniper.net/documentation/us/en/software/junos/is-is/topics/topic-map/infocus-isis-srv6-network-programming.html
TechPost 1: SRv6 Basics Locator and End-SIDs - https://community.juniper.net/blogs/krzysztof-szarkowicz/2022/06/29/srv6-basics-locator-and-end-sids
TechPost 2: L3VPN on SRv6 - https://community.juniper.net/blogs/krzysztof-szarkowicz/2022/08/11/l3vpn-over-srv6
TechPost 3: SRv6 Summarisation - https://community.juniper.net/blogs/krzysztof-szarkowicz/2022/09/28/srv6-summarization
TechPost 4: SRv6 SID Encoding and Transposition - https://community.juniper.net/blogs/krzysztof-szarkowicz/2022/12/02/srv6-sid-encoding-and-transposition
TechPost 5: SRv6 L3VPN Inter-AS Option-C - https://community.juniper.net/blogs/krzysztof-szarkowicz/2023/02/06/srv6-l3vpn-inter-as-option-c
TechPost 6: Link Slicing with MPLS and SRv6 Underlays - https://community.juniper.net/blogs/krzysztof-szarkowicz/2023/05/12/link-slicing-with-mpls-and-srv6-underlays
Glossary
- AS: Autonomous System
- ASBR: Autonomous System Boundary Router
- BGP: Border Gateway Protocol
- CE: Customer Edge
- CLI: Command Line Interface
- eBGP: external Border Gateway Protocol
- iBGP: internal Border Gateway Protocol
- ID: Identifier
- IGP: Interior Gateway Protocol
- Inter-AS: Inter Autonomous System
- IP: Internet Protocol
- IPv4: Internet Protocol version 4
- IPv6: Internet Protocol version 6
- IS-IS: Intermediate System to Intermediate System
- L2: Level 2
- L3VPN: Layer 3 Virtual Private Network
- MPLS: Multiprotocol Label Switching
- NHS: Next Hop Self
- OSPF: Open Shortest Path First
- P: Provider
- PE: Provider Edge
- PSP: Penultimate Segment Pop
- RFC: Request for Comments
- RIB: Routing Information Base
- RR: Route Reflector
- SID: Segment Identifier
- SP: Service Provider
- SRv6: Segment Routing version 6
- TLV: Type Length Value
- USD: Ultimate Segment Decapsulation
- USP: Ultimate Segment Pop
- VPN: Virtual Private Network
Acknowledgements
Many thanks to Anton Elita for his thorough review and suggestions, and Aditya T R for preparing JCL and vLabs topologies.
Feedback
Revision History
| Version |
Author(s) |
Date |
Comments |
| 1 |
Krzysztof Szarkowicz |
Feb 2022 |
Initial publication |
#Routing
#SolutionsandTechnology