Routing

Hi all,

We have a MX-104, with the following OS:

Model: mx104
Junos: 13.3R7.4
JUNOS Base OS boot [13.3R7.4]

In BGP configuration, there was only one BGP group, and inside this group, 2 neighbors (i-BGP, towards route reflectors).

We added a second BGP group, and inside of it, an e-BGP neighbor. When we committed, ALL BGP neighbors went down, and then re-established adjacency.

The log shows:

Jun 1 14:45:54 DEVICE-re0 rpd[1541]: %DAEMON-4: bgp_adv_main_update:8762: NOTIFICATION sent to NEIGHBOR-1 (Internal AS MY AS): code 6 (Cease) subcode 6 (Other Configuration Change), Reason: Configuration change - VPN table advertise
Jun 1 14:45:54 DEVICE-re0 rpd[1541]: %DAEMON-4: bgp_adv_main_update:8762: NOTIFICATION sent to NEIGHBOR-2 (Internal AS MY AS): code 6 (Cease) subcode 6 (Other Configuration Change), Reason: Configuration change - VPN table advertise
Jun 1 14:45:54 DEVICE-re0 rpd[1541]: %DAEMON-4: bgp_set_peer_if: BGP peer NEIGHBOR-3 (External AS ISP AS) interface not found. Leaving peer idled
Jun 1 14:45:57 DEVICE-re0 mgd[88798]: %INTERACT-4-UI_COMMIT_COMPLETED: commit complete
Jun 1 14:45:58 DEVICE-re0 mib2d[1473]: %DAEMON-4-SNMP_TRAP_LINK_DOWN: ifIndex 551, ifAdminStatus down(2), ifOperStatus down(2), ifName xe-2/0/1.950
Jun 1 14:46:00 DEVICE-re0 /kernel: %KERN-4: ae_linkstate_ifd_change: MUP received for interface xe-2/0/1, member of ae1
Jun 1 14:46:28 DEVICE-re0 rpd[1541]: %DAEMON-3: bgp_process_caps: mismatch NLRI with NEIGHBOR-2 (Internal AS MY AS): peer: <inet-vpn-unicast l2vpn>(68) us: <inet-vpn-unicast>(4)
Jun 1 14:46:31 DEVICE-re0 rpd[1541]: %DAEMON-3: bgp_process_caps: mismatch NLRI with NEIGHBOR-1 (Internal AS MY AS): peer: <inet-vpn-unicast l2vpn>(68) us: <inet-vpn-unicast>(4)

So, why is this behaviour? I don't understand what's the reason all bgp neighbors go down/reset when adding another BGP group in BGP configuration. Is this MX-104 specific?

Thanks.

Depending on configuration , but i think this is expected behaviior . Thy to read this

Hi,

On a quick replication i did not see any flaps for default family

[edit]
labroot:r1# set protocols bgp group ebgp neighbor 9.9.121.2 peer-as 200 

[edit]
labroot:r1# commit 
commit complete

[edit]
labroot:r1# run show bgp summary 
Groups: 2 Peers: 3 Down peers: 1
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
9.9.101.2               100          7          6       0       0        1:49 0/0/0/0              0/0/0/0
9.9.111.2               100          6          5       0       0        1:45 0/0/0/0              0/0/0/0
9.9.121.2               200          0          0       0       0           2 Active

I see from your output that a new vpn table is being advertised , were you trying to add a new address family support something similar like below, then you would definitely see a bump

labroot:r1# set protocols bgp family inet-vpn unicast        

[edit]
labroot:r1# set protocols bgp group ebgp neighbor 9.9.121.2 peer-as 200 

[edit]
labroot:r1# commit 
commit complete

[edit]
labroot:r1# run show bgp summary 
Groups: 2 Peers: 3 Down peers: 3
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       0          0          0          0          0          0
bgp.l3vpn.0          
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
9.9.101.2               100          0          0       0       0           2 Active
9.9.111.2               100          0          0       0       0           2 Active
9.9.121.2               200          0          0       0       0           2 Active

HI,

It is expected behaviour.

If you add any family on BGP group global level, it is applicable for all neighbor. during this change all bgp session will re-establish.

to avoid, apply family on neighbor level instead of BGP group global level.

e.g. 

set protocols bgp group ebgp neighbor 9.9.121.2 family inet-vpn unicast

We didn't modify the internal BGP group. We just added another group. So I think this shouldn't be normal. Coming from Cisco, haven't seen this behaviour in any of their platforms.

Reason for the flap is:

"code 6 (Cease) subcode 6 (Other Configuration Change), Reason: Configuration change - VPN table advertise"

It appears a BGP family change was also applied. This is standard BGP behavior and would be same accross vendors I suppose, Cisco included.

Hello,

I'm sorry but I do no see any BGP going down with these logs:

Jun 1 14:45:54 DEVICE-re0 rpd[1541]: %DAEMON-4: bgp_adv_main_update:8762: NOTIFICATION sent to NEIGHBOR-1 (Internal AS MY AS): code 6 (Cease) subcode 6 (Other Configuration Change), Reason: Configuration change - VPN table advertise

This log is a Notification sent to the neighbor.

Jun 1 14:45:54 DEVICE-re0 rpd[1541]: %DAEMON-4: bgp_adv_main_update:8762: NOTIFICATION sent to NEIGHBOR-2 (Internal AS MY AS): code 6 (Cease) subcode 6 (Other Configuration Change), Reason: Configuration change - VPN table advertise

This log says there's an BGP advertise, that's normal as the routing-table has changed, It has to update neighbors

Jun 1 14:45:54 DEVICE-re0 rpd[1541]: %DAEMON-4: bgp_set_peer_if: BGP peer NEIGHBOR-3 (External AS ISP AS) interface not found. Leaving peer idled

This message it's a litle more to worry about, it's seems when the new peer was added, there weren't interface for this neighbor, BGP status went to idle.

Base on this logs, I do not see any neighbor going down, I do not see BGP DOWN neighbor message.

Are you sure the new  neigbor comes up?

Do you see the timers for old BGP neighbors reseted?

Regards,

LordofOceans 🙂

Hi,

As per RFC4486, the Cease code is a notification message to signal BGP reset:

If a BGP speaker decides to administratively reset the peering with a
   neighbor due to a configuration change other than the ones described
   above, then the speaker SHOULD send a NOTIFICATION message with the
   Error Code Cease and the Error Subcode "Other Configuration Change".

https://tools.ietf.org/html/rfc4486

This indicates that the local router has initiated a BGP session reset and has notified its peers about the reset.

Also, Notification messages are sent to signal an error.

The BGP session Uptime can be verified with "show bgp summary".

For the new neighbor,  is it Established or down?

Can you check in the respective routing-table if you see an active direct route towards the peer as the BGP config suggests an eBGP.  Is this a direct or multihop eBGP peering?

Hope this helps.

Cheers

Ashvin

BGP sessions towards internal route reflectors were up and established for almost one month. After adding the new BGP group, ALL sessions went down/reset and are now up and established.

The strange thing is, we don't see this behaviour in MX-960. We have several MX-104 and the issue happened in two of them, were we implemented new BGP groups.

It seems to me that this issue is MX-104 specific. Can anybody try this on a real device? (in lab, please!) The idea is to validate this behaviour.

Thanks.

Hi,

AFAIK, the BGP reset behavior is same accross all JUNOS platforms when changes are applied to BGP family, MX480 and MX960 inclusive.

Can you paste the diff of the changes ['show configuration | compare rollback x'] that caused the BGP reset.

Cheers,

Ashvin