vMX

Hello,

Our objective is to get the telemetry data from juniper vMX device to Server. To achieve that, we followed the below steps:

Device information:
Model: vmx
Junos: 17.4R1.16
JUNOS OS Kernel 64-bit [20171206.f4cad52_builder_stable_11]

1. On the Juniper device, We have configured the all available sensors (for testing) and pointed towards our collector.
a.) we have reachability from vMX to server.

2. We have tried netcat and tcpdump for streaming data verification, no data is flowing through.

nc -ul 0.0.0.0 30000
tcpdump -i ens180

No data is streaming from juniper device though we have bunch of sensors configured. We did ping from juniper vMX to collecting server, we are able to see the ICMP data related to that using tcpdump utility.

3. Though we have enabled the local port on the juniper device, the port not shown in udp (netstat -a | grep udp)
Attached the telemetry configuration and juniper version deatils for your reference. Please let us know, if any additional configuration or troubleshooting steps are needed.

We followed the below link for setting up telemetry.
https://www.juniper.net/documentation/en_US/junos/topics/reference/general/junos-telemetry-interface-decoding-data.html

Thanks for your help.

 

Regards,

 

Rajesh N

Hi Folks,

Just my 2 cents on this…

 

What is the connectivity to the server? Is it via Management interface? If so, it will not work.

 

You should make sure it is exiting out of the FPC interface…

 

 

To add…

 

PFE/linecard generated sensor telemetry is inserted directly in the data plane at the linecard. This means, by architecture, it cannot jump into fpx0, there is no forwarding path between the data plane and fpx0 interfaces. This enhancement will be fixed in future Junos releases 18.1.

 

You could use though gRPC based sensors for that purpose, which are ultimately proxied via the RE and the RE can send the traffic through fpx0.

 

So right now, don’t use fxp0 as output interface to telemetry server. Try to use other physical interfaces on any of FPC to try.

I’ve tried this in lab [mx480] and validate this behavior.

 

 

 

 

Thanks alot,Its really helpful.. When I tried with gRPC it worked out well. 
Few more questions on the same:
1. How can I configure gRPC , on logstash (ELK stack)? ( i have tried with telegraf with influx db, its wokring fine. But our primary interest is on ELK)

2. How to configure triggered based telemetry (not just the periodic) on Juniper vMX?

 

Please provide us the few more pointers on ELK configuration and Triggered based telemetry.  Once agian,thanks for the quick reply.

Hi Folks,

I don’t have any pointers related to logstash on top of mind. However, I have some pointers on gRPC OpenConfig input plugin for OpenNTI.

 

https://github.com/Juniper/nfd14-demo/tree/master/oc-telemetry-telegraf

https://github.com/Juniper/open-nti-input-oc

 

Understanding OpenConfig and gRPC on Junos Telemetry Interface

https://www.juniper.net/documentation/en_US/junos/topics/concept/open-config-grpc-junos-telemetry-interface-understanding.html

 

Agentd traceoptions :

set services analytics traceoptions file agentd.trace files 5 size 10m 

set services analytics traceoptions flag all

 

Native Sensor Configuration:

set services analytics streaming-server ifd remote-address x.y.z.a

set services analytics streaming-server ifd remote-port 2000

set services analytics streaming-server ifl remote-address x.y.z.a

set services analytics streaming-server ifl remote-port 2001

set services analytics streaming-server lsp remote-address x.y.z.a

set services analytics streaming-server lsp remote-port 2002

set services analytics export-profile test local-address a.b.c.d

set services analytics export-profile test reporting-rate 30

set services analytics export-profile test format gpb

set services analytics export-profile test transport udp

set services analytics sensor ifd server-name ifd

set services analytics sensor ifd export-name test

set services analytics sensor ifd resource /junos/system/linecard/interface/

set services analytics sensor ifl server-name ifl

set services analytics sensor ifl export-name test

set services analytics sensor ifl resource /junos/system/linecard/interface/logical/usage/

set services analytics sensor lsp server-name lsp

set services analytics sensor lsp export-name test

set services analytics sensor lsp resource /junos/services/label-switched-path/usage/

Thanks again... Does juniper vMX supports triggered telemetry?

Hi Rajesh,

Could you help me with the Telemetry configuration? Any step-by-step guides available? I'm not sure how to connect the VCP with VFP. I'm using VMware workstation and have created two VM's for VCP & VFP.

I’m trying to get Telemetry data from my MX960 Into Grafana which I believe is using Influxdb and Telegraf. On the Ubuntu server with Grafana/influxdb/telegraf I do see my MX960 Telemetry data using tcpdump, so I know it’s getting there. How can I verify that it’s making it into telegraf and influxDB and then into grafana graphs?

The telemetry data is in gRPC format.  So you will need to find out on the Graphana side if that format is supported for ingesting data.

 

are you sure it's gRPC ?  I have transport set to UDP...i now have data at the telegraf server since the linux guy that I work with told me to change the destination port to 8094, once I did that, we now see data at telegraf service on grafana ubuntu server... but not sure why it's saying "unable to parse incoming packet"

set services analytics streaming-server my-grafana-srvr remote-address 172.22.14.151
set services analytics streaming-server my-grafana-srvr remote-port 8094
set services analytics export-profile my-exprt-prfl local-address 10.101.128.254
set services analytics export-profile my-exprt-prfl local-port 21111
set services analytics export-profile my-exprt-prfl reporting-rate 10
set services analytics export-profile my-exprt-prfl format gpb
set services analytics export-profile my-exprt-prfl transport udp
set services analytics sensor my-sensor-14 server-name my-grafana-srvr
set services analytics sensor my-sensor-14 export-name my-exprt-prfl
set services analytics sensor my-sensor-14 resource /junos/system/linecard/interface/

 

...and, btw, i now have this at the telegraf (grafana) server...

2018-10-08T16:10:37Z E! Error in plugin [inputs.socket_listener]: unable to parse incoming packet: metric parse error: expected tag at offset 137: “\n\x16lab-960:10.101.128.254\x10\x00\“Tmy-sensor-14:/junos/system/linecard/interface/:/junos/system/linecard/interface/:PFE(\x110¤ǣ\xe5,8\x01@\x01\xaa\x06\x97\x03\xe2\xa4\x01\x92\x03\x1a\x8f\x03\n\x8c\x03\n\bet-0/1/0\x10\xb4\x9d\xb2\xd7\x05\x18\xd4\x05\“\x04ae60*$\b\x00\x10\xf1\x88\xfa\x1f\x18\xbbÈ\x8at \x00(\x000\x008\x00@\x00H\x00P\x00X\xb1\x01`\x80\x80\x80\xe4\x01*\“\b\x01\x10ç¶\x01\x18\xa5\xcaÒ\x05 \x00(\x000\x008\x00@\x00H\x00P\x00X\xf2\x02`\x80\x80\x02*!\b\x02\x10\xbf\xbf\xcd\x01\x18\xa9\xf7\x9a\xb8\x05 \x00(\x000\x008\x00@\x00H\x00P\x00X\x00`\x80\x80\x02*\“\b\x03\x10Ч\xd9\t\x18\x86Å\xb1\n \x00(\x000\x008\x00@\x00H\x00P\x00X>`\x80\x80\xf8\v*\x1a\b\x04\x10\x00\x18\x00 \x00(\x000\x008\x00@\x00H\x00P\x00X

Yes telemtry is based on the open sourced grpc project.  You can see the overview here.

 

https://www.juniper.net/documentation/en_US/junos/topics/concept/junos-telemetry-interface-export-format-understanding.html

 

I have not used grafana much but thought this was expecting to connect to database formats.  In googling around I can't find a reference to setting up a streaming interface on grafana for grpc.

 

Thanks Steve, I see the first sentence in that web doc is "The Junos Telemetry Interface supports two ways of exporting data in the protocol buffers (gpb) format:"

I think those 2 ways are said to be ...

1 - UDP "native" via linecard - maybe in-band revenue interfaces

2 - gRPC via Route Engine - maybe out-of-band fxp0

 

(i use the words "i think" and "maybe" for the obvious reasons that I have not made this work yet and am currently trying to learn this via all sorts of web pages and support avenues)

 

So Steve, you seem to be speaking to the 2nd option - gRPC Open Sourced project...
...and this guy here at the following web link seems to be showing the 3-step process of config'ing the juniper router to do the 1st option - udp/native and gives guidance about using the OpenNTI stuff to make that work....

 

https://techmocha.blog/2017/07/21/using-opennti-as-a-collector-for-streaming-telemetry-from-juniper-devices-part-2/

 

https://github.com/Juniper/open-nti

 

Do you understand this to be 2 different ways of accomplishing this?  I'm wondering if I'm tracking correctly.

 

{master}[edit]
agould@lab-960# set services analytics export-profile my-exprt-prfl transport ?
Possible completions:
grpc Use grpc transport
udp Use UDP transport protocol
{master}[edit]
agould@lab-960# set services analytics export-profile my-exprt-prfl format ?
Possible completions:
gpb Use gpb format
gpb-sdm Use gpb self-describing-message format

My understanding is both these streams are originally from google.  But my real point is that searching the Grafana documentation I don't see reference to supporting these formats for ingestion.

 

My impression from the site is that it supports a multitude of database storage models but I don't see where it has references to ingest streaming formats let alone that it supports these.

 

 

Thanks Steve, what do you think about this? (I just realized I sent that link yesterday, but I’m taking a closer look at OpenNTI today... as previously I was unable to boot the OpenNTI docker image... we will try this again on a clean Ubuntu imaged VM)

The following link mentions using OpenNTI as a package containing all the required options for accomplishing this. Myself and my Linux engineer coworker will try this today.

https://techmocha.blog/2017/06/26/using-opennti-as-a-collector-for-streaming-telemetry-from-juniper-devices-part-1/

I got telemetry streaming working using this site ... I have a couple MX960's streaming telemetry to the suite of software provided in this Open-NTI project spoken of on this techmocha blog site. I think my previous problems were related to conflicting installs.... as myself and my coworker had loaded individual items *and* then the open-nti suite (which i understand is a docker container with all the items like grafana, fluentd, chronograf, influxdb, etc).... anyway, we started with a fresh install Ubunto virtual machine and *only* loaded Open-NTI and it works.

I do not know or understand all of the innerworkings of it at this point, but am quickly learning, even while writing this post... I'm currently using Chronograf hosted at port 8888 and browsing the Data Explorer function and seeing some nice graphs. (I'm wondering if Chrongraf is simply an alternative to Grafana gui front end, unsure) There seems to be tons of items to monitor and analyze, and I'm currently only sending the following sensor resource from the MX960 and there are several more that can be sent.... /junos/system/linecard/interface/

I am sending the telemetry from the MX960 using UDP transport and GPB format to port 50000 and source port 21111 (mx960-1) and 21112 (mx960-2). I'm unsure that I had to use unique source ports... as I wonder if the source-ip would have been sufficient to make the streaming sources unique in the Open-NTI server.

Looking at the techmocha pictures, and the "docker ps" command on the linux server, and now this new-found techmocha link (see "deconstructed" below) i understand that FluentD is the TSDB (time series db) that is receiving/ingesting the *Native* streaming form of telemetry from my MX960's on udp port 50000 and looks like fluentd hands off that data to InfluxDB port 8086 (which i think happens internally at that server). (I'm not evening talking about the other form of jti telemetry using openconfig and grpc....I've yet to do that and don't know why I would exactly...which i beleive is ingested using telegraf, unsure)

...the link i followed to deploy open-nti suite....

https://techmocha.blog/2017/06/26/using-opennti-as-a-collector-for-streaming-telemetry-from-juniper-devices-part-1/#comments

...interestingly, i just now found this, which apparently is a way of deploying all the components individually...

https://techmocha.blog/2017/10/31/serving-up-opennti-deconstructed/

 

Hi Python,

Could you help please? any guides available for configuring telemetry with Juniper vMX?

Hi Check if It helps.

https://www.juniper.net/documentation/en_US/junos/information-products/pathway-pages/junos-telemetry-interface/junos-telemetry-interface.pdf

Hi Kingsman,

The guide that you mentioned in the post, doesn't help much. The end-to-end steps are missing. For e.g. I initially started with configuring telemetry over UDP and then I came across this post that it is not possible for Juniper devices to send telemetry data over fpx0 (management port). I tried using VCP & VFP VM's but then again there is no good documentation in order to understand how they have to be setup. Now I'm trying with gRPC based sensors. I have installed the Network Agent (upgrade FreeBSD version) on my vMX17.4R1.16, and done the following configuration using the configure mode:

set system services extension-service request-response grpc clear-text port 50051
set system services extension-service request-response grpc skip-authentication
set system services extension-service notification allow-clients address 0.0.0.0/0

Now my question is, how to subscribe to telemetry data? Which file or command needs to be looked at? Is there a Juniper Telemetry Interface Client which needs to be setup (one link i found - https://github.com/nileshsimaria/jtimon) ?

 

My Juniper configuration look like this:

version 17.4R1.16;
system {
    host-name vmx-junos-telemetry;
    root-authentication {
        encrypted-password "$6$w8tRl2N7$DVwFSsoyT9NHh741AbUMZXs8GxE2CzOabpu6OdhO                                                                                                                     t/C60sxYBIUKfzvEz3/MpphGBMXbUS.aYv8GhSX64JHUs/"; ## SECRET-DATA
    }
    services {
        ftp;
        ssh {
            root-login allow;
        }
        telnet;
        extension-service {
            request-response {
                grpc {
                    clear-text {
                        port 50051;
                    }
                    skip-authentication;
                }
            }
            notification {
                allow-clients {
                    address 0.0.0.0/0;
                }
            }
        }
        netconf {
            ssh;
        }
    }
    syslog {
        user * {
            any emergency;
        }
        file messages {
            any notice;
            authorization info;
        }
        file interactive-commands {
            interactive-commands any;
        }
    }
    processes {
        dhcp-service {
            traceoptions {
                file dhcp_logfile size 10m;
                level all;
                flag all;
            }
        }
    }
}
services {
    analytics {
        streaming-server telemetry-server {
            remote-address 192.168.152.30;
            remote-port 2030;
        }
        export-profile telemetry {
            local-address 192.168.152.40;
            local-port 35000;
            forwarding-class assured-forwarding;
            reporting-rate 60;
            format gpb;
            transport udp;
        }
        sensor cpu {
            server-name telemetry-server;
            export-name telemetry;
            resource /junos/system/linecard/cpu/memory/;
        }
    }
}
interfaces {
    fxp0 {
        unit 0 {
            family inet {
                address 192.168.152.40/24;
            }
        }
    }
}

In my first attempt with streaming telemetry data using UDP, I have configured the streaming server, export-profile, resources using this guide - https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/junos-telemetry-interface-configuring.html

So do I have to delete this configuration in order for the gRPC sensors to work?

 

Thanks,

Vinay