Junos OS

Hi all,

I have just been given a remit to configure Q-in-Q Tunneling on our MX240s for the Ethernet Customers. I have not configured Q-in-Q before and cannot seem to find any MX to SRX documentation for this (I can on an EX)....

Could anybody point me in the right directions to:

A: Explain fully - apart from the extra tag part for the tunneling (I understand the basics of it).

B: How to configure Q-in-Q from an MX240 perspective (Pop the S-TAG and place on - I guess) please?

C: How I can apply VoiP CoS to these tunnels?

As an add on, I don't need to configure this on the SRX as the downstream provider will encapsulate the C-TAGs at their Access Exchange, but I do need to "pop" the S-TAG at the MX240. Any help on that end would be greatly appreciated.

Thanks

Actually, there is one part here I am struggiling to understand how to configure (I think I can complete the rest)... Here is the topology (as the SRX340 acts as Layer 2 NTE):

CPE WAN Address (192.168.1.1/30)(VLAN10) ----- VLAN10 (NTE) VLAN10 ------ VLAN10 192.168.1.2/30 MX240 

So, the issue here is that I have 2 VLANs and BOTH of them need a layer 3 address. How can I overcome this issue with the S-Tag on one interface at the MX240?

How about if I configure the following:

S-TAG = 500

VLAN10 - 192.168.10.0/30 - Customer

VLAN99 - 10.10.10.0/30 - NTE Management

CPE - 192.168.10.2/30
NTE - IRB.99 - 10.10.10.2/30

set interfaces xe-1/2/4 flexible-vlan-tagging
set interfaces xe-1/2/4 mtu 2016 --- set by provider
set interfaces xe-1/2/4 encapsulation flexible-ethernet-services
set interfaces xe-1/2/4 unit 500 description test-ethernet-s-tag
set interfaces xe-1/2/4 unit 500 vlan-id 500
set interfaces xe-1/2/4 unit 500 family bridge interface-mode trunk
set interfaces xe-1/2/4 unit 500 family bridge inner-vlan-id-list [10 99]

set interfaces xe-1/2/4 unit 10 vlan-tags outer 500 inner 10
set interfaces xe-1/2/4 unit 10 family inet address 192.168.10.1/30

set interfaces xe-1/2/4 unit 99 vlan-tags outer 500 inner 99
set interfaces xe-1/2/4 unit 99 family inet address 10.10.10.1/30

Then configure the routing to the correct logical interface.

Will this configuration be correct please?

Once the downstream provider has confirmed the S-TAG configuration setup, then I can test anyway. Just thought I would confirm here first. Do I need to configure any bridge information?

Hello,

Judging by Your made-up config, I reckon You want to do the following:

1/ have 2 VLANs (10 and 99) to arrive into MX xe-1/2/4 already double-tagged

2/ terminate VLAN 10 on L3 in MX 

3/ pass VLANs 10 and 99 further on also double-tagged 

Please confirm my understanding is correct so I can write a proper config for You.

Also I need clarification about my point #3 above - are You happy to pass double-tagged VLAN 10 and VLAN 99 onwards inside separate pseudowires/PW or do You want a single PW that forwards both VLAN 10 and 99 inside, double-tagged of course?

And to answer Your last question

---

@adgwytc wrote:

Do I need to configure any bridge information?

---

Yes You need it, otherwise the configuration You made up won't commit.

HTH

Thx
Alex

Hi Alex,

Thank you for the information. 

So that we are clear, please see below the exact route topology to be taken for each vlan:

CPE WAN INT ---> ge-0/0/1 SRX340 ge-0/0/2 --> ISP --> xe-1/2/4.10 MX240 xe-1/2/5 --> Internet (Upstream ISP)

So, from discussions with the downstream ISP (Marked above as "ISP") they will complete the S-TAG on their equipment. So, from an SRX340 perspective nothing is changed, which leaves that out.

So, when the two VLANs arrive at interface xe-1/2/4.10they will have an encapsulated S-TAG around them of 500. This tag needs to be removed and the inner tags shown and then forwarded out of xe-1/2/5 (they have to be routed correctly at xe-1/2/4.10 as they will first query our DNS so cannot have the S-TAG after that point).

The packets need to be encapsulated with the same S-TAG on the return out of interface xe-1/2/4.10 towards the SRX340. The downstream ISP will strip off the S-TAG at egress towards the SRX340.

Hope this explains it a little better.

My guess is, that as I have two VLANs assigned to each customer, I would need to create 3 interfaces per customer for this to work? VLAN99 will always be used for the Management of every customers SRX, hence the reason for this configuration.

Thank you for your help.

Hello,

Here is the MX config for You I tested in my lab:

set interfaces xe-1/2/5 flexible-vlan-tagging
set interfaces xe-1/2/5 encapsulation flexible-ethernet-services
set interfaces xe-1/2/5 unit 10 encapsulation vlan-bridge
set interfaces xe-1/2/5 unit 10 family bridge interface-mode trunk
set interfaces xe-1/2/5 unit 10 family bridge vlan-id-list 10
set interfaces xe-1/2/5 unit 10 family bridge vlan-id-list 99

set interfaces xe-1/2/4 flexible-vlan-tagging
set interfaces xe-1/2/4 encapsulation flexible-ethernet-services
set interfaces xe-1/2/4 unit 10 encapsulation vlan-bridge
set interfaces xe-1/2/4 unit 10 vlan-tags outer 500
set interfaces xe-1/2/4 unit 10 family bridge interface-mode trunk
set interfaces xe-1/2/4 unit 10 family bridge inner-vlan-id-list 10
set interfaces xe-1/2/4 unit 10 family bridge inner-vlan-id-list 99
set interfaces irb unit 10 family inet address 192.168.10.1/30

set routing-instances VS1 instance-type virtual-switch
set routing-instances VS1 interface xe-1/2/4.10
set routing-instances VS1 interface xe-1/2/5.10
set routing-instances VS1 bridge-domains BD-else-no-irb vlan-id-list 99
set routing-instances VS1 bridge-domains BD10 vlan-id 10
set routing-instances VS1 bridge-domains BD10 routing-interface irb.10

HTH

Thx

Alex

Hi Alex,

Thank you for the configuration. However, I am not sure about the xe-1/2/5 interface requirements.

xe-1/2/5 is used for the eBGP peering with the upstream ISP, Cymru for Martians and also blackhole services. I do not think I can configure xe-1/2/5 as flexible-ethernet-services etc.... won't that stop BGP operating?

Will the unit 10 configuration cause any issues at all with regards to that? I maybe missing something here, but I need the S-TAG stripped off at interface xe-1/2/4.10 (this is an example interface as we will have multiple interfaces for each customer and the S-TAG needs to be unique).

I thought the S-TAG would be stripped off at the xe-1/2/4.10 interface and then routed, however, from there?

Okay. I did a test on our non-utilised (currently) MX240.....

Here is the current configuration on the xe-1/2/5 interface:

set interfaces xe-1/2/5 unit 0 description To-GTT-ASNxxx
set interfaces xe-1/2/5 unit 0 family inet tcp-mss 1410
set interfaces xe-1/2/5 unit 0 family inet filter input-list bgpfilter-223
set interfaces xe-1/2/5 unit 0 family inet filter input-list filter-ssh
set interfaces xe-1/2/5 unit 0 family inet filter input-list cos1
set interfaces xe-1/2/5 unit 0 family inet address 94.94.94.1/30
set interfaces xe-1/2/5 unit 0 family inet6 address 1975:668:0:3:ffff:0:4d43:5046/126

And , when I try and add the configuration recommended, I get the Unit 0 error as follows:

re0:
[edit interfaces xe-1/2/5]
'unit 0'
VLAN-ID must be specified on tagged ethernet interfaces
error: configuration check-out failed

This is not viable. I cannot change the eBGP configuration and lose communication.

As I mentioned, I felt sure there must be a way where the S-TAG is stripped and applied at interface xe-1/2/4 ....  I'm not sure why xe-1/2/5 would require any tagging.

My apologies Alex, I appreciate your help greatly, as always, but there must be another way around this?

Hello,

---

@adgwytc wrote:

Hi Alex,

 

Thank you for the configuration. However, I am not sure about the xe-1/2/5 interface requirements.

 

xe-1/2/5 is used for the eBGP peering with the upstream ISP, Cymru for Martians and also blackhole services. I do not think I can configure xe-1/2/5 as flexible-ethernet-services etc.... won't that stop BGP operating?

 

 I see You keep adding requirements. What did stop You mentioning this in Your last post, I wonder?

There IS a way to :

1/ peer over untagged subinterface off xe-1/2/5

2/ pass any number of VLAN tags (albeit You specified 2: 10 and 99) off xe-1/2/5

If You did enumerate all and any of Your requirements in as much detail as poss beforehand, You'd saved Yourself time & trouble.

Please see below ADDITIONAL configuration (additional to what I already shared):

set interfaces xe-1/2/5 unit 0 vlan-id 1
set interfaces xe-1/2/5 native-vlan-id 1

 xe-1/2/5 must have "encapsulation flexible-ethernet-services" and "flexible-vlan-tagging" for the above commands to work.

HTH

Thx
Alex

Hi Alex,

My apologies. I made an assumption that stating "Upstream ISP" would indicate eBGP peering.

As I said, your help is greatly appreciated and trying to explain a complete setup via a message is difficult.

So, to be sure of the exact MX240 configuration:

xe-1/2/4 is the main interface that faces the "downstream ISP" - This is where the S-TAGged VLANs will enter the MX240. This is where, I would think, the outer S-TAG is stripped leaving the inner VLANs. The obvious problem here is that the logical interface will be configured at "Layer 2". So some form of "Layer 3" requirement is needed. This is where I am hitting the stumbling block.

Interface xe-1/2/5 is utilised for the "upstream ISP" and eBGP peering.

The flow of the packet from the ethernet core will be to the xe-1/2/4 logical interface and to the DNS (for queries) then out of the xe-1/2/5 interface to wherever the destination may be (web server - another customer site etc).

Again, apologies for the mix up, I should never just assume something in this business..... apologies Alex

Hello,

Right, I believe Your requirements have changed once again. On 4/11/2018, You said 

---

@adgwytc wrote:

This tag needs to be removed and the inner tags shown and then forwarded out of xe-1/2/5 

The previous wording indicates to me :

1/ only S-TAG needs to be removed 

2/ C-TAG don't need to be removed

3/ packets with C-TAGs need to be forwarded out of xe-1/2/5

This is what my configuration does. I even accomodated Your late request to eBGP-peer out of xe-1/2/5 using untagged subinterface.

Now, I see You are requesting the following:

---

@adgwytc wrote:

 

The flow of the packet from the ethernet core will be to the xe-1/2/4 logical interface and to the DNS (for queries) then out of the xe-1/2/5 interface to wherever the destination may be (web server - another customer site etc).

 

---

 Please provide a detailed diagram of traffic flow for VLAN 10 and VLAN 99 from SRX to "Upstream ISP" clearly showing the expected VLAN tags on each hop and IP addresses so I can understand Your complete requirement.

HTH

Thx

Alex

Hi Alex,

Let me supply that for ypou, if I can add an attachment.

The confusion is in the reading of the words in that paragraph, there is no comma and there should have been. So the wording should say:

"This tag needs to be removed and the inner tags shown, and then forwarded out of the xe-1/2/5 interface."

The missing comma, as can be seen from the above paragraph, is what caused the confusion. I always meant that the xe-1/2/4 interface is where the S-TAG should be stripped.

I have attached the current configuration topology as it works. The problem with this scenario is that the "downstream ISP" wants to charge us each time we ask for VLAN 99.... we are not willing to do this, hence the sudden requirement of "Q-in-Q".

Hopefully the two attached documents are all you need.

One is for the previous 2 VLAN configuration and the other is the current MX240 Q-in-Q.

If you need anything else, then please let me know

Hi Alex,

Please find attached another document that shows the complete flow.

Hello,

Thanks for sharing the diagrams but You did not show the traffic flow I asked, and You did not show IP addressing required.

I am going to ask one last time the following questions:

1/ is there a requirement to strip the C-TAGs?

2/ if [1] is true then where the C-TAGs are stripped?

3/ if [1] is true then what happens with underlying Ethernet frames once C-TAG is stripped:

3a/ is the frame that had its VLAN 10 C-tag stripped going to be examined further and if it contains an IP packet then is this IP packet going to be routed along the best route to destination?

or

3b/ is the frame that had its VLAN 10 C-tag stripped going to be forwarded untagged out of xe-1/2/5 interface?

and

3c/ is the frame that had its VLAN 99 C-tag stripped going to be examined further and if it contains an IP packet then is this IP packet going to be routed along the best route to destination?

or

3d/ is the frame that had its VLAN 99 C-tag stripped going to be forwarded untagged out of xe-1/2/5 interface?

4/ what is the mask length of the subnet associated with VLAN 10?

5/ what is the mask length of the subnet associated with VLAN 99?

HTH

Thx

Alex

Hi Alex,

Again. The attached document shows the flow of the traffic and I have added the IP addressing.

1: Is there a requirement to strip the C-TAGs?

Yes. The routing must take place before exiting the xe-1/2/5 interface. There will be Layer 3 IPv4 addressing requirements on the MX240 for both VLANs.

2: If (1) is yes, where are the C-TAGs stripped?

Given the curernt configuration (document was attached), both VLANs are currently stripped at xe-1/2/4.10 and xe-1/2/4.99 respectively. This means I configure static routes for the return traffic via these interfaces. With QinQ, the interface will be a layer 2 and I somehow need to create Layer 3 interfaces to allow routing.

3: If (1) is true, then what happens with the underlying Ehternet frames once C-TAG is stripped?

3a: Is the frame that had its VLAN10 tag-stripped going to be examined further and if it contains an IP packet then is this IP packet going to be routed along the best route to the destination?

Yes. 100%. This is exactly what should happen. 

3c: Answer: Yes, also 100% true. Our VPN traffic will be routed over this interface for management (VPN DHCP Range). THis is so the NTE devices can only be accessed via our VPN and not from any external source.

4: What is the mask length of VLAN10? 

Answer: On the attached document, but it will be a /30

5: What is the subnet length of VLAN99?

Answer: Currently, as a test, it will be /30. However, once live it will be a /16 with the one (hopefully) gateway address on the MX240. If this is not possible then this /16 will be further subnetted to /30s.

Hi Alex,

Let me share with you the current xe-1/2/4 interface and static routes so you can see what I am trying to achieve with QinQ:

set interfaces xe-1/2/4 hierarchical-scheduler implicit-hierarchy
set interfaces xe-1/2/4 flexible-vlan-tagging
set interfaces xe-1/2/4 mtu 2016
set interfaces xe-1/2/4 encapsulation flexible-ethernet-services
set interfaces xe-1/2/4 unit 10 description clivecom
set interfaces xe-1/2/4 unit 10 vlan-id 10
set interfaces xe-1/2/4 unit 10 family inet address 192.168.23.17/30
set interfaces xe-1/2/4 unit 99 description management-vlan
set interfaces xe-1/2/4 unit 99 vlan-id 99
set interfaces xe-1/2/4 unit 99 family inet address 10.10.10.1/16

set routing-options static route 192.168.23.18/32 next-hop 192.168.23.17

set routing-options static route 10.10.0.0/16 next-hop 10.10.10.1

This configuration works perfectly for our requirements. In fact, it works for CoS, IPv6, IPv4 .....  But we cannot use it due to the QinQ requirement.

Hope this helps Alex.

Thank you

Hello,

Thanks for answering the questions, it helped.

All You need to do  now is to add 1 extra S-tag to existing xe-1/2/4.10 and xe-1/2/4.99 subinterfaces:

delete interfaces xe-1/2/4 unit 10 vlan-id 10
delete interfaces xe-1/2/4 unit 99 vlan-id 99
set interfaces xe-1/2/4 unit 10 vlan-tags outer 500 inner 10

Then as I understand it, the VLAN 99 will be multipoint - because of subnet size. You could use bridge-domain as below:

delete interfaces xe-1/2/4 unit 99
set interfaces xe-1/2/4 unit 99 encapsulation vlan-bridge
set interfaces xe-1/2/4 unit 99 vlan-tags outer 500
set interfaces xe-1/2/4 unit 99 vlan-tags inner 99
set interfaces xe-1/2/4 unit 99 family bridge

set bridge-domains BD99 vlan-id 99
set bridge-domains BD99 no-local-switching
set bridge-domains BD99 interface xe-1/2/4.99
set bridge-domains BD99 routing-interface irb.99
set interfaces irb unit 99 family inet address 10.10.10.1/16 ## could be /30 initially

You can add more QinQ VLANs to BD99 as below. Assuming fake xe-10/10/10 interface with S-TAG 510 and C-TAG 909:

set interfaces xe-10/10/10 unit 99 encapsulation vlan-bridge
set interfaces xe-10/10/10 unit 99 vlan-tags outer 510
set interfaces xe-10/10/10 unit 99 vlan-tags inner 909
set interfaces xe-10/10/10 unit 99 family bridge
set bridge-domains BD99 interface xe-10/10/10.99

And if there is an IP address behind xe-10/10/10.99 that belongs to 10.10/16 subnet, it should answer the ARP and be reachable.  The tag normalization will happen automatically, You don't have to worry about it.

I also recommed to enable processing of all known TPIDs so You are prepared if Your ISP sends You QinQ frame where TPID is not 0x8100:

set interfaces xe-1/2/4 gigether-options ethernet-switch-profile tag-protocol-id [ 0x8100 0x88a8 0x9100 ]

HTH

Thx
Alex

Hi Alex,

Thank you .....

That looks absolutely perfect to me, and simple too, and will save on a lot of extra interfaces or VSs.....

I am waiting for the order of the unique S-TAG, so unfortunately, I cannot currently test. Once that has been ordered and committed by the downstream ISP then I can configure and test.

Again, apologies for any mis-communication and thank you for your help.

Hi Alex, 

One more quesiton please, as I do not see this in the configuration:

Is the address for the customer VLAN still housed on the Logical interface?

As follows:

set interfaces xe-1/2/4 unit 10 description to-customer-cpe
set interfaces xe-1/2/4 unit 10 vlan-tags outer 500 inner 10
set interfaces xe-1/2/4 unit 10 family inet address 192.168.35.1/30

If it is, then the last piece of configuration won't work unless I can put an address there:

set interfaces xe-1/2/4 unit 200 encapsulation vlan-bridge
set interfaces xe-1/2/4 unit 200 vlan-tags outer 50 inner 909
set interfaces xe-1/2/4 unit 200 family bridge

If this is family bridge, where can I put the IP address please?

Hello,

---

@adgwytc wrote:

 

 

If it is, then the last piece of configuration won't work unless I can put an address there:

set interfaces xe-1/2/4 unit 200 encapsulation vlan-bridge
set interfaces xe-1/2/4 unit 200 vlan-tags outer 50 inner 909
set interfaces xe-1/2/4 unit 200 family bridge

 

If this is family bridge, where can I put the IP address please?

 

---

 Step-by-step instructions:

1/ use "family bridge" QinQ subinterfaces ONLY when You need mutlipoint connectivity, otherwise regular L3 subinterfaces with "vlan-tags outer inner" are fine

2/ assuming [1] is true, create a bridge subinterface 

3/ insert it into an existing bridge-domain , or

4/ create a new bridge domain if bridge subinterface created at step [2] needs to be in a different subnet (different to existing subnets)

5/ if [4] is true, add "routing-interface irb.BLAH" to that bridge-domain

6/ if [4] is true add "set interfaces irb.BLAH family inet address blah-blah/blah" 

Hope this makes sense.

HTH

Thx

Alex

Hi Alex,

Thank you. It does make sense and I am sure once I get the S-TAG I can test .......

I think I'll need to complete the following:

set interfaces xe-1/2/4 unit 10 description to-customer-01

set interfaces xe-1/2/4 unit 10 encapsulation vlan-bridge

set interfaces xe-1/2/4 unit 10 vlan-tags outer 500 inner-list [10 99]

set interfaces xe-1/2/4 unit 10 family bridge

set interfaces xe-1/2/4 unit 50 description to-customer-02

set interfaces xe-1/2/4 unit 50 encapsulation vlan-bridge

set interfaces xe-1/2/4 unit 50 vlan-tags outer 250 inner-list [50 99]

set interfaces xe-1/2/4 unit 50 family bridge

set interfaces irb unit 10 family inet address 192.168.50.1/30

set interfaces irb unit 50 family inet address 192.168.100.1/30

set interfaces irb unit 99 family inet address 10.10.10.1/16

set bridge-domains BD99 vlan-id 99

set bridge-domains BD99 no-local-switching

set bridge-domains BD99 interface xe-1/2/4.10

set bridge-domains BD99 interface xe-1/2/4.50

set bridge-domains BD99 routing-instance irb.99

set bridge-domains BD10 vlan-id 10

set bridge-domains BD10 no-local-switching

set bridge-domains BD10 interface xe-1/2/4.10

set bridge-domains BD10 routing-instance irb.10

set bridge-domains BD50 vlan-id 50

set bridge-domains BD50 no-local-switching

set bridge-domains BD50 interface xe-1/2/4.50

set bridge-domains BD50 routing-instance irb.50

Am I allowed to place the same interface into 2 bridge-domains?

 Hello,

You haven't been following my last post with step-by-step instructions.

No worries, happens all the time and usually causes a flurry of deperate calls for help in all directions.

My [1] step is - don't use bridge-domain IF the VLAN is point-to-point. /30 is p2p. 

Next - You won't be able to insert IFL with "inner-vlan-id-list" into single BD.

Finally, same IRB interface cannot be in 2 bridge domains and same Ethernet IFL can be in >1 BD _IF_ this IFL has "inner-vlan-id-list" and is placed into a virtual-switch.

Please rework Your config as per my step-by-step instructions and example given earlier, or seek professional help.

HTH

Thx

Alex

Hi Alex,

I am not trying to be pedantic as I know you are simply helping me resolve this issue..... I think the problem is here:

My [1] step is - don't use bridge-domain IF the VLAN is point-to-point. /30 is p2p

You are quite correct and I understand that. The problem is that VLAN99 is Multi-Point and is on EVERY customer interface. If I have to have a BD for VLAN99 (Multipoint) then that means EVERY interface will become Multipoint and then we have the issue I have described in my config.

Another way around this is to create many /30s from the /16 and use that as P2P too. But then I cannot have 2 IP addresses on the interface for 2 vlans. So, those are the two issues.....

Maybe I am missing something you have said and need to see professional help.... lol

I think you are missing the point that VLAN99 must be on every customer interface.

Hi Alex,

As an add on, to start with, thank you for all the help.

As soon as I get the order commissioned for a test S-TAG I can then test the configs you have supplied and can tweak them a little if they need a slight change. As soon as I have done that and it's working, I'll post the config here for others.

Again, thanks for your help.

Hi Alex,

After seeking professional help 🙂 🙂 I believe my brain may be functioning on the correct technical level now.....

I see where the confusion is......

You were always advising froma two interface scenario whereas I was looking at it from a 1 interface scenario.

I still do not have the S-TAG so cannot confirm, but I'm guessing the following config will complete the task for me:

Let's pretend I have two customers now. here would be their details:

Customer 1:

Customer VLAN-ID: 10

Management VLAN-ID: 99

S-TAG: 500

Customer 2:

Customer VLAN-ID: 20

Management VLAN-ID: 99

S-TAG: 200

Configuration Customer 1:

set interfaces xe-1/2/4 unit 10 description to-customer1-cpe

set interfaces xe-1/2/4 unti 10 vlan-tags outer 500 inner 10

set interfaces xe-1/2/4 unit 10 family inet address 192.168.10.1/30

set interfaces xe-1/2/4 unit 99 description management-customer1

set interfaces xe-1/2/4 unit 99 encapsulation vlan-bridge

set interfaces xe-1/2/4 unit 99 vlan-tags outer 500 inner 99

set interfaces xe-1/2/4 unit 99 family bridge

Configuration Customer 2:

set interfaces xe-1/2/4 unit 20 description to-customer2-cpe

set interfaces xe-1/2/4 unit 20 vlan-tags outer 200 inner 20

set interfaces xe-1/2/4 unit 20 family inet address 192.168.20.1/30

set interfaces xe-1/2/4 unit 98 description management-customer2

set interfaces xe-1/2/4 unit 98 encapsulation vlan-bridge

set interfaces xe-1/2/4 unit 98 vlan-tags outer 200 inner 99

set interfaces xe-1/2/4 unit 98 family bridge

Mangement irb:

set interfaces irb unit 99 family inet address 10.10.99.1/16

Bridge Domain:

set bridge-domains BD99 vlan-id 99

set bridge-domains BD99 no-local-switching

set bridge-domains BD99 interface xe-1/2/4.99

set bridge-domains BD99 interface xe-1/2/4.98

set bridge-domains BD99 routing-interface irb.99

The only downside is that I will need two interfaces per customer, but, that's the way it's gotta be.

As soon as I test this I'll let you know the results.