vMX

I have configured ISIS on the interfaces and also the protocol itself, starting at the very basic level as follows (as I will build the network slowly to replicate our live systems):

System 2:

set interfaces ge-0/0/4 unit 0 family inet address 192.168.1.1/30

set interfaces ge-0/0/4 unit 0 family iso

set interfaces lo0 unit 0 family inet address 192.168.200.1/32

set interfaces lo0 unit 0 family iso address 49.0001.1920.0168.2001.00

set protocols isis interface ge-0/0/4.0

set protocols isis interface lo0.0

System 2:

set interfaces ge-0/0/4 unit 0 family inet address 192.168.1.2/30

set interfaces ge-0/0/4 unit 0 family iso

set interfaces lo0 unit 0 family inet address 192.168.200.2/32

set interfaces lo0 unit 0 family iso address 49.0001.1680.0192.2002.00

set protocols isis interface ge-0/0/4.0

set protocols isis interface lo0.0

So, the isis adjacency extensive on system 1 shows "up, down, up, down, up, down" continuously.

The isis adjacency extensive on system 2 shows as initialising.

The MTU on both interfaces is 1514 (default).

This should just come up and work. This is configured on other interfaces and they are behaving the same way. up, down, up, down, up, down.

Any ideas?

Apologies, the adjacency between system 2 and system 3 shows as UP.

So, one is working and the other is not.....

I've spotted the following in a log file, so I will invesitgate a little more:

Dec 10 15:32:08.888829 Designated router mismatch, ours HEX-CORE-02-TEST, remote HEX-CORE-02-TEST
Dec 10 15:32:08.888839     speaks IP
Dec 10 15:32:08.888870     speaks IPV6
Dec 10 15:32:08.888885     IP address 192.168.1.2
Dec 10 15:32:08.888901     area address 49.0001 (3 bytes)
Dec 10 15:32:08.888914     restart flags []
Dec 10 15:32:08.888931     updating neighbor HEX-CORE-02-TEST 

Okay, even stating a DR has not solved the issue.... as follows:

set protocols isis interface ge-0/0/4.0 level 1 priority 100

set protocols isis interface ge-0/0/4.0 level 2 priority 100

It's still flapping. As this is vMX and I had zero problems in the real world with ISIS, I can possibly say that this could be down to numerous reasons:

A: Manually changing MAC addresses.

B: Running the system in "lite" mode.

C: No purchase of a license as yet

I'll move onto to BGP and see if that throws up errors.

Sorry for disturbing you guys.

So, BGP establishes fine. No problem at all. So thought I would move back to ISIS and have made a little progress that someone may have seen before, or know what it is associated with:

I have 4 vMX connected as shown below:

vMX3                                  vMX4

  |                                             |

  |                                             |

vMX1 ------------------------vMX2

If I create an isis adajcency between vMX2 and vMX4 and leave it at that, it stays UP and works as expected.

On the loopback I have created the unique NET addresses as follows:

vMX2 - 49.0001.2000.2020.0002.00

vMX4 - 49.0001.2000.2020.0004.00

vMX1 - 49.0001.2000.2020.0001.00

Forget vMX3 for the moment.

So, all is currently working, but then I add vMX1 to the ISIS config and suddenly it all goes wrong and I get the UP, DOWN, INITIALISING state.

If I remove the vMX2 to vMX4 isis config the other isis adjacency then comes UP and stays UP.

So, it appears that I can get ONE adjacency UP and it stays UP, but the moment I add a second interface to the ISIS configuration, it all fails.....

Any one have any ideas?

Could this because of the "mode-lite"? Lack of a licence?

It's very strange, because in our live environment this works fine.....

Okay, some help would be great......

I have started from scratch and deleted all vMX and re-configured. I have now also added the 60 day trial license....

I am still having the same issue:

vMX2 config:

set interfaces ge-0/0/4 unit 0 family inet address  192.168.20.2/30   ---- To vMX4 in diagram in above post

set interfaces ge-0/0/4 unit 0 family iso

set interfaces ge-0/0/2 unit 0 family inet address 192.168.1.2/30  ---- To vMX1 in diagram in above post
set interfaces ge-0/0/2 unit 0 family iso

set interfaces lo0 unit 0 family inet address 20.20.20.2/32
set interfaces lo0 unit 0 family iso address 49.0001.2000.2020.0002.00
set protocols isis interface ge-0/0/2.0 
set protocols isis interface ge-0/0/4.0
set protocols isis interface lo0.0

vMX1 config:

set interfaces ge-0/0/2 unit 0 family inet address 192.168.1.1/30
set interfaces ge-0/0/2 unit 0 family iso

set interfaces lo0 unit 0 family inet address 20.20.20.1/32
set interfaces lo0 unit 0 family iso address 49.0001.2000.2020.0001.00

set protocols isis interface ge-0/0/2.0
set protocols isis interface lo0.0

On vMX2, if I only include ge-0/0/4 in the isis config then it all stays UP and is great. If I add in ge-0/0/2 it all starts flapping. This flapping is being caused by the lack of a DR election final decision.

All MAC addresses are unique, so isis should choose the highest MAC as the DR for each link, but it is almost as if both networks are being seen as 1 and the whole system cannot work out who the DR should be.

Any ideas?

vMX4 config:

set interfaces ge-0/0/4 unit 0 family inet address 192.168.20.1/30  ----- To vMX2 
set interfaces ge-0/0/4 unit 0 family iso

set interfaces lo0 unit 0 family inet address 20.20.20.4/32
set interfaces lo0 unit 0 family iso address 49.0001.2000.2020.0004.00
set protocols isis interface ge-0/0/4.0 
set protocols isis interface lo0.0

Okay, from the log file I have found what is causing the error, but it is still strange.

All MAC addresses are unique and the IP Configs are okay (/30s)...

Dec 11 17:23:09.778773 ERROR: ISIS ignored a bad packet: IIH with duplicate sysid on interface ge-0/0/4.0
Dec 11 17:23:09.778787 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.778806 ERROR: ISIS ignored a bad packet: IIH with duplicate sysid on interface ge-0/0/4.0
Dec 11 17:23:09.778813 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.778832 Received L1 LAN IIH, source id THW-CORE-01-TEST on ge-0/0/4.0
Dec 11 17:23:09.778853    Address 192.168.1.1 not matching interface ge-0/0/4.0 addresses
Dec 11 17:23:09.778862 ERROR: IIH from THW-CORE-01-TEST without matching addresses, interface ge-0/0/4.0
Dec 11 17:23:09.778868 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.778888 Received L2 LAN IIH, source id THW-CORE-01-TEST on ge-0/0/4.0
Dec 11 17:23:09.778897 ERROR: Nbr THW-CORE-01-TEST thinks we are DR but is not reflecting our circuit-id correctly, ours 2, remote 3
Dec 11 17:23:09.778903 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.778920 Received L1 LAN IIH, source id THW-CORE-01-TEST on ge-0/0/4.0
Dec 11 17:23:09.778935    Address 192.168.1.1 not matching interface ge-0/0/4.0 addresses
Dec 11 17:23:09.778943 ERROR: IIH from THW-CORE-01-TEST without matching addresses, interface ge-0/0/4.0
Dec 11 17:23:09.778950 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779019 Received L2 LAN IIH, source id THW-CORE-01-TEST on ge-0/0/4.0
Dec 11 17:23:09.779044    Address 192.168.1.1 not matching interface ge-0/0/4.0 addresses
Dec 11 17:23:09.779053 ERROR: IIH from THW-CORE-01-TEST without matching addresses, interface ge-0/0/4.0
Dec 11 17:23:09.779060 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779082 Received L1 LAN IIH, source id THW-CORE-01-TEST on ge-0/0/4.0
Dec 11 17:23:09.779091 ERROR: Nbr THW-CORE-01-TEST thinks we are DR but is not reflecting our circuit-id correctly, ours 2, remote 3
Dec 11 17:23:09.779096 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779114 Received L2 LAN IIH, source id THW-CORE-01-TEST on ge-0/0/4.0
Dec 11 17:23:09.779130    Address 192.168.1.1 not matching interface ge-0/0/4.0 addresses
Dec 11 17:23:09.779138 ERROR: IIH from THW-CORE-01-TEST without matching addresses, interface ge-0/0/4.0
Dec 11 17:23:09.779145 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779164 Received L1 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779179    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779187 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779194 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779212 Received L2 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779227    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779235 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779242 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779261 Received L1 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779276    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779284 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779290 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779309 Received L2 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779324    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779332 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779338 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779357 Received L1 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779372    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779379 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779386 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779405 Received L2 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779419    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779427 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779433 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779453 Received L1 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779468    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779476 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779483 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency
Dec 11 17:23:09.779503 Received L2 LAN IIH, source id HEX-LNS-02-TEST on ge-0/0/2.0
Dec 11 17:23:09.779518    Address 192.168.20.1 not matching interface ge-0/0/2.0 addresses
Dec 11 17:23:09.779526 ERROR: IIH from HEX-LNS-02-TEST without matching addresses, interface ge-0/0/2.0
Dec 11 17:23:09.779532 Notification not sent due to 5-second throttle as per rfc4444: isisRejectedAdjacency

Based on the logs, I think there is issue in the vMX NIC configuration at hypervisor level. Ensure that ge-0/0/4 connected to vMX4 and ge-0/0/2 connected to vMX1 are in seperate VMnetwork. Which hypervisor your are using?

Hi Nellikka

I'm running Ubuntu Server 16.04 and Junos 18.3 (as per recommended)....

eno1 is the physical NIC on the box.

Read out of brctl show and sudo virsh list attached.

I was running each isis instance as Level 1 and Level 2 so I have now disabled level 1 acroos all the 4 vMX. This has made no difference.

Attachment(s)

Sudo listing.txt   3 KB 1 version

All vMX interfaces are mapped to same virbr0. That is why you are facing problems when you enable ISIS on more than 2 vMX.

You have to create vitio vindings at ubuntu to enable communication between each vMX or enable vlan tagging on each interfaces at vMX.

Please refer the URL for more details to see how two vMXs are connected each other

http://matt.dinham.net/juniper-vmx-lab-setup-2-vmx-evpn-logical-systems/

----

• br-ext – the external bridge for management traffic
• br-int-vmx1 – the internal bridge for vMX1 RE to PFE traffic
• br-int-vmx2 – the internal bridge for vMX2 RE to PFE traffic
• bridge_vmx12 – to enable the communication between ge-0/0/3 on vMX1 and vMX2
• virbr0 – unused as all vMX interfaces are defined <--------------------------------------------------
• vmx1_link_ls – connects ge-0/0/1 and ge-0/0/2 on vMX1 <----
• vmx2_link_ls – connects ge-0/0/1 and ge-0/0/2 on vMX2 <-------
• vmx_link – connects ge-0/0/0 on vMX1 and vMX2 to eth1 on the host

Thanks Nellikka.....

I'm a little confused by the document though..... he clearly states "What if I want to create more than two vMX on the SAME ethernet segment?"

Well, my vMX are not on the same Ethernet segment. So, I am not too sure where using the same link helps..... My vmx-junosdev.conf looks like this:

- link_name : vmx_link1
mtu : 1500
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/0
endpoint_2 :
- type : host_dev
dev_name : eno2

- link_name : vmx_link2
mtu : 1500
endpoint_1 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/0
endpoint_2 :
- type : bridge_dev
dev_name : bridge1

- link_name : vmx_link3
mtu : 1500
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/1
endpoint_2 :
- type : host_dev
dev_name : eno2

- link_name : vmx_link4
mtu : 1500
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/2
endpoint_2 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/2

- link_name : vmx_link5
mtu : 1500
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/3
endpoint_2 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/3

- link_name : vmx_link6
mtu : 1500
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/4
endpoint_2 :
- type : junos_dev
vm_name : vmx3
dev_name : ge-0/0/4

- link_name : vmx_link7
mtu : 1500
endpoint_1 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/4
endpoint_2 :
- type : junos_dev
vm_name : vmx4
dev_name : ge-0/0/4

So, where would the changes need to be made here please?

Hi Nellika

Okay, I have changed the file to the following:

- link_name : bridge_vmx_12
endpoint_1 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/4
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx_12

- link_name : bridge_vmx_12
endpoint_1 :
- type : junos_dev
vm_name : vmx4
dev_name : ge-0/0/4
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx_12

- link_name : bridge_vmx_13
endpoint_1 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/2
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx_13

- link_name : bridge_vmx_13
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/2
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx_13

I'll see if it works and let you know.

As per your topology, vMX3 and vMX1 are connected each other, so they are in same ethernet segment, same is applicable to other vMXs

vMX3                                  vMX4

  |                                             |

  |                                             |

vMX1 ------------------------vMX2

"brctl show" comand should show these connectivity. In your case, it is not there.

Example from the previous link:

"ge-0/0/3 on vMX1 and vMX2 will be interconnected using a Linux virtio bridge on the host"

bridge_vmx12    8000.fe060a0efff3    no              ge-0.0.3-vmx1 <--- vMX1 and vMX2 are connected each other
                                                     ge-0.0.3-vmx2 <--- vMX2

 OR

You may test ISIS issue by enabling vlan-tagging on interfaces on each vMX and enable ISIS on tagged interfaces

vMX2 - vMX-4:

+++++++++++++++

set interfaces ge-0/0/4 vlan-tagging

set interfaces ge-0/0/4 unit 24 vlan-id 24 family inet address <x.x.x.x>

set interfaces ge-0/0/4 unit 24 vlan-id 24 family iso

set protocols isis interface ge-0/0/4.24

vMX4 - vMX-2:

+++++++++++++

set interfaces ge-0/0/4 vlan-tagging

set interfaces ge-0/0/4 unit 24 vlan-id 24 family inet address <x.x.x.x>

set interfaces ge-0/0/4 unit 24 vlan-id 24 family iso

set protocols isis interface ge-0/0/4.24

likewise on other vMX, use different vlan id

Hi Nellikka,

Okay. We are kind of getting there.... but kind of not.

In our scenario, we do not want any routing-instances. We want 4 separate instances of vMX on the same Ubuntu server.

So, when configuring up the bridge, as mentioned, we found that we could only get two instances available via SSH. This does not mean the other 2 instances were not running, just they were not available via ssh from my desktop. They were available SSH internally from one vMX to another.

This seemed to cure the issue for the ISIS which is good, but, could you please confirm if 2 vMX is all that can be linked to a host-physical-interface? It certainly seems that way and even the vMX day one book appears to point in that direction.

I can get away with that set up for now, but for the future I may have to look at another server and run 2 x vMX on one server and 2 x vMX on another server and connect them together....

Please can you clarify this?

Multiple instances of vMX can share external bridge (host-physcial interface) - Day one guide page 30 SSH section

Hi all,

I want to add this here to show the final resolution to the IS-IS issue.

I rebuilt the system as it was easier to start from scratch. I then configured the 4 vMX under /vmx/config. Make sure you ensure all MACs are unique in each vMX and also the ports.

For the vmx-junosdev.conf file, this is the set up I had for the configuration:

vMX1 ge-0/0/2 <----> ge-0/0/2 vMX2

vMX1 ge-0/0/4 <----> ge-0/0/4 vMX3

vMX3 ge-0/0/4 <----> ge-0/0/4 vMX1

vMX2 ge-0/0/2 <----> ge-0/0/2 vMX1

vMX2 ge-0/0/4 <----> ge-0/0/4 vMX4

vMX4 ge-0/0/4 <----> ge-0/0/4 vMX2

Here is what the vmx-junosdev.conf looks like now:

- link_name : bridge_vmx_12
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/2
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx12

- link_name : bridge_vmx_12
endpoint_1 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/2
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx12

- link_name : bridge_vmx_13
endpoint_1 :
- type : junos_dev
vm_name : vmx1
dev_name : ge-0/0/4
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx13

- link_name : bridge_vmx_13
endpoint_1 :
- type : junos_dev
vm_name : vmx3
dev_name : ge-0/0/4
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx13

- link_name : bridge_vmx_14
endpoint_1 :
- type : junos_dev
vm_name : vmx2
dev_name : ge-0/0/4
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx14

- link_name : bridge_vmx_14
endpoint_1 :
- type : junos_dev
vm_name : vmx4
dev_name : ge-0/0/4
endpoint_2 :
- type : bridge_dev
dev_name : bridge_vmx14

Make sure when you have started the vMX instances that you complete the "sudo ./vmx.sh --bind-dev" command to bring the interfaces up, and just to ensure, complete a "brctl show" command. The output should look like mine:

bridge_vmx12      8000.fe020a0efff2     no     ge-0.0.2-vmx1
                                                                              ge-0.0.2-vmx2
bridge_vmx13      8000.fe030a0efff4     no     ge-0.0.4-vmx1
                                                                              ge-0.0.4-vmx3
bridge_vmx14      8000.fe020a0efff3     no     ge-0.0.4-vmx2
                                                                              ge-0.0.4-vmx4

Then you configure your isis as you normally would on the vMX and it all works perfectly.