SD-WAN

Hello All,

Can someone help me understand what are the tshoot steps when receiving an alert like this: "128T Duplicate Reverse Flow has exceeded an acceptable rate"
The alerting lasts for a short time then it clears.

Thanks.

------------------------------
NDS NOC
Schaumburg IL
8476033225
------------------------------

Hi NOC,

This is not a 128T alarm, this is an alarm that your team set up to monitor a specific statistic on the 128T. AFAIK the value your team chose for alerting (and clearing the alert) are somewhat arbitrary. Thus the arrival and departure of the alarm is somewhat subjective and not to be any indicator of a problem window.

That said, the statistic being monitored (stats/service-area/session-install-errors/duplicate-reverse-flow) increments when a new session setup arrives in the "reverse" direction to a session that is already established in the "forward" direction. In your particular deployment it incremented rapidly at the first deployed site when there were some issues on a neighboring device's QoS marking, causing lots of packets to be discarded.

This counter ticking up could be a symptom of congestion on the link. Thus you should troubleshoot it the way you'd troubleshoot any congestion condition: look for link utilization, packet drops, etc. on the path(s) between the 128T devices.

The counter ticking up could be a symptom of asymmetric routing. Look at both ends of the SVR link to ensure that there is path symmetry.

The counter ticking up could be a symptom of packet loss on the intervening network. Look at stats to see packets/bytes transmitted and packets/bytes received at both ends to make sure that they're lining up as you'd expect.

------------------------------
pt.
------------------------------

Original Message:
Sent: 08-13-2020 09:54
From: NDS NOC
Subject: Duplicate Reverse Flow

Hello All,

Can someone help me understand what are the tshoot steps when receiving an alert like this: "128T Duplicate Reverse Flow has exceeded an acceptable rate"
The alerting lasts for a short time then it clears.

Thanks.

------------------------------
NDS NOC
Schaumburg IL
8476033225
------------------------------

Hi Patrick,

Thanks for info.
I looked at script used and I might not be understanding it well, but I cannot identify the interface based on alert.
And this gives too many interfaces to verify.

And this might not be an issue indicator which is great, but we cut tickets based on threshold recommended by your support team.
And even if we need to increase threshold, it's ok, but sooner or later there will be a real issue to tshoot and I still need to get detailed steps on what to look at before opening support case.

Thanks.

------------------------------
NDS NOC
Schaumburg IL
8476033225
------------------------------

Original Message:
Sent: 08-14-2020 08:45
From: Patrick A Timmons
Subject: Duplicate Reverse Flow

Hi NOC,

This is not a 128T alarm, this is an alarm that your team set up to monitor a specific statistic on the 128T. AFAIK the value your team chose for alerting (and clearing the alert) are somewhat arbitrary. Thus the arrival and departure of the alarm is somewhat subjective and not to be any indicator of a problem window.

That said, the statistic being monitored (stats/service-area/session-install-errors/duplicate-reverse-flow) increments when a new session setup arrives in the "reverse" direction to a session that is already established in the "forward" direction. In your particular deployment it incremented rapidly at the first deployed site when there were some issues on a neighboring device's QoS marking, causing lots of packets to be discarded.

This counter ticking up could be a symptom of congestion on the link. Thus you should troubleshoot it the way you'd troubleshoot any congestion condition: look for link utilization, packet drops, etc. on the path(s) between the 128T devices.

The counter ticking up could be a symptom of asymmetric routing. Look at both ends of the SVR link to ensure that there is path symmetry.

The counter ticking up could be a symptom of packet loss on the intervening network. Look at stats to see packets/bytes transmitted and packets/bytes received at both ends to make sure that they're lining up as you'd expect.

------------------------------
pt.

Original Message:
Sent: 08-13-2020 09:54
From: NDS NOC
Subject: Duplicate Reverse Flow

Hello All,

Can someone help me understand what are the tshoot steps when receiving an alert like this: "128T Duplicate Reverse Flow has exceeded an acceptable rate"
The alerting lasts for a short time then it clears.

Thanks.

------------------------------
NDS NOC
Schaumburg IL
8476033225
------------------------------

There are only two WAN interfaces you need to concern yourself with at your deployment. The LAN and fabric interfaces should not cause this counter to increment.

Look on the SVR path between the branch and data center for the symptoms I mentioned in my previous post.

------------------------------
pt.
------------------------------

Original Message:
Sent: 08-14-2020 09:50
From: NDS NOC
Subject: Duplicate Reverse Flow

Hi Patrick,

Thanks for info.
I looked at script used and I might not be understanding it well, but I cannot identify the interface based on alert.
And this gives too many interfaces to verify.

And this might not be an issue indicator which is great, but we cut tickets based on threshold recommended by your support team.
And even if we need to increase threshold, it's ok, but sooner or later there will be a real issue to tshoot and I still need to get detailed steps on what to look at before opening support case.

Thanks.

------------------------------
NDS NOC
Schaumburg IL
8476033225

Original Message:
Sent: 08-14-2020 08:45
From: Patrick A Timmons
Subject: Duplicate Reverse Flow

Hi NOC,

This is not a 128T alarm, this is an alarm that your team set up to monitor a specific statistic on the 128T. AFAIK the value your team chose for alerting (and clearing the alert) are somewhat arbitrary. Thus the arrival and departure of the alarm is somewhat subjective and not to be any indicator of a problem window.

That said, the statistic being monitored (stats/service-area/session-install-errors/duplicate-reverse-flow) increments when a new session setup arrives in the "reverse" direction to a session that is already established in the "forward" direction. In your particular deployment it incremented rapidly at the first deployed site when there were some issues on a neighboring device's QoS marking, causing lots of packets to be discarded.

This counter ticking up could be a symptom of congestion on the link. Thus you should troubleshoot it the way you'd troubleshoot any congestion condition: look for link utilization, packet drops, etc. on the path(s) between the 128T devices.

The counter ticking up could be a symptom of asymmetric routing. Look at both ends of the SVR link to ensure that there is path symmetry.

The counter ticking up could be a symptom of packet loss on the intervening network. Look at stats to see packets/bytes transmitted and packets/bytes received at both ends to make sure that they're lining up as you'd expect.

------------------------------
pt.

Original Message:
Sent: 08-13-2020 09:54
From: NDS NOC
Subject: Duplicate Reverse Flow

Hello All,

Can someone help me understand what are the tshoot steps when receiving an alert like this: "128T Duplicate Reverse Flow has exceeded an acceptable rate"
The alerting lasts for a short time then it clears.

Thanks.

------------------------------
NDS NOC
Schaumburg IL
8476033225
------------------------------