vMX

Expand all | Collapse all

Telemetry data is not streaming from Juniper vMX 17.4R1.16

Jump to Best Answer
  • 1.  Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 01-04-2018 01:33

    Hello,

    Our objective is to get the telemetry data from juniper vMX device to Server. To achieve that, we followed the below steps:

    Device information:
    Model: vmx
    Junos: 17.4R1.16
    JUNOS OS Kernel 64-bit [20171206.f4cad52_builder_stable_11]

    1. On the Juniper device, We have configured the all available sensors (for testing) and pointed towards our collector.
    a.) we have reachability from vMX to server.

    2. We have tried netcat and tcpdump for streaming data verification, no data is flowing through.

    nc -ul 0.0.0.0 30000
    tcpdump -i ens180

    No data is streaming from juniper device though we have bunch of sensors configured. We did ping from juniper vMX to collecting server, we are able to see the ICMP data related to that using tcpdump utility.

    3. Though we have enabled the local port on the juniper device, the port not shown in udp (netstat -a | grep udp)
    Attached the telemetry configuration and juniper version deatils for your reference. Please let us know, if any additional configuration or troubleshooting steps are needed.

    We followed the below link for setting up telemetry.
    https://www.juniper.net/documentation/en_US/junos/topics/reference/general/junos-telemetry-interface-decoding-data.html

    Thanks for your help.

     

    Regards,

     

    Rajesh N

    Attachment(s)



  • 2.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

     
    Posted 01-05-2018 11:57

    Hi Folks,

    Just my 2 cents on this…

     

    What is the connectivity to the server? Is it via Management interface? If so, it will not work.

     

    You should make sure it is exiting out of the FPC interface…

     

     



  • 3.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16
    Best Answer

     
    Posted 01-05-2018 11:59

    To add…

     

    PFE/linecard generated sensor telemetry is inserted directly in the data plane at the linecard. This means, by architecture, it cannot jump into fpx0, there is no forwarding path between the data plane and fpx0 interfaces. This enhancement will be fixed in future Junos releases 18.1.

     

    You could use though gRPC based sensors for that purpose, which are ultimately proxied via the RE and the RE can send the traffic through fpx0.

     

    So right now, don’t use fxp0 as output interface to telemetry server. Try to use other physical interfaces on any of FPC to try.

    I’ve tried this in lab [mx480] and validate this behavior.

     

     

     

     



  • 4.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 01-06-2018 02:33

    Thanks alot,Its really helpful.. When I tried with gRPC it worked out well. 
    Few more questions on the same:
    1. How can I configure gRPC , on logstash (ELK stack)? ( i have tried with telegraf with influx db, its wokring fine. But our primary interest is on ELK)

    2. How to configure triggered based telemetry (not just the periodic) on Juniper vMX?

     

    Please provide us the few more pointers on ELK configuration and Triggered based telemetry.  Once agian,thanks for the quick reply.



  • 5.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

     
    Posted 01-06-2018 04:36

    Hi Folks,

    I don’t have any pointers related to logstash on top of mind. However, I have some pointers on gRPC OpenConfig input plugin for OpenNTI.

     

    https://github.com/Juniper/nfd14-demo/tree/master/oc-telemetry-telegraf

    https://github.com/Juniper/open-nti-input-oc

     

    Understanding OpenConfig and gRPC on Junos Telemetry Interface

    https://www.juniper.net/documentation/en_US/junos/topics/concept/open-config-grpc-junos-telemetry-interface-understanding.html

     

    Agentd traceoptions :

    set services analytics traceoptions file agentd.trace files 5 size 10m 

    set services analytics traceoptions flag all

     

    Native Sensor Configuration:

    set services analytics streaming-server ifd remote-address x.y.z.a

    set services analytics streaming-server ifd remote-port 2000

    set services analytics streaming-server ifl remote-address x.y.z.a

    set services analytics streaming-server ifl remote-port 2001

    set services analytics streaming-server lsp remote-address x.y.z.a

    set services analytics streaming-server lsp remote-port 2002

    set services analytics export-profile test local-address a.b.c.d

    set services analytics export-profile test reporting-rate 30

    set services analytics export-profile test format gpb

    set services analytics export-profile test transport udp

    set services analytics sensor ifd server-name ifd

    set services analytics sensor ifd export-name test

    set services analytics sensor ifd resource /junos/system/linecard/interface/

    set services analytics sensor ifl server-name ifl

    set services analytics sensor ifl export-name test

    set services analytics sensor ifl resource /junos/system/linecard/interface/logical/usage/

    set services analytics sensor lsp server-name lsp

    set services analytics sensor lsp export-name test

    set services analytics sensor lsp resource /junos/services/label-switched-path/usage/



  • 6.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 01-06-2018 08:01

    Thanks again... Does juniper vMX supports triggered telemetry?



  • 7.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 02-14-2018 22:38

    Hi Rajesh,

    Could you help me with the Telemetry configuration? Any step-by-step guides available? I'm not sure how to connect the VCP with VFP. I'm using VMware workstation and have created two VM's for VCP & VFP.



  • 8.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 10-06-2018 19:33
    I’m trying to get Telemetry data from my MX960 Into Grafana which I believe is using Influxdb and Telegraf. On the Ubuntu server with Grafana/influxdb/telegraf I do see my MX960 Telemetry data using tcpdump, so I know it’s getting there. How can I verify that it’s making it into telegraf and influxDB and then into grafana graphs?


  • 9.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

     
    Posted 10-07-2018 10:33

    The telemetry data is in gRPC format.  So you will need to find out on the Graphana side if that format is supported for ingesting data.

     



  • 10.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 10-08-2018 09:15

    are you sure it's gRPC ?  I have transport set to UDP...i now have data at the telegraf server since the linux guy that I work with told me to change the destination port to 8094, once I did that, we now see data at telegraf service on grafana ubuntu server... but not sure why it's saying "unable to parse incoming packet"

    set services analytics streaming-server my-grafana-srvr remote-address 172.22.14.151
    set services analytics streaming-server my-grafana-srvr remote-port 8094
    set services analytics export-profile my-exprt-prfl local-address 10.101.128.254
    set services analytics export-profile my-exprt-prfl local-port 21111
    set services analytics export-profile my-exprt-prfl reporting-rate 10
    set services analytics export-profile my-exprt-prfl format gpb
    set services analytics export-profile my-exprt-prfl transport udp
    set services analytics sensor my-sensor-14 server-name my-grafana-srvr
    set services analytics sensor my-sensor-14 export-name my-exprt-prfl
    set services analytics sensor my-sensor-14 resource /junos/system/linecard/interface/

     

    ...and, btw, i now have this at the telegraf (grafana) server...


    2018-10-08T16:10:37Z E! Error in plugin [inputs.socket_listener]: unable to parse incoming packet: metric parse error: expected tag at offset 137: “\n\x16lab-960:10.101.128.254\x10\x00\“Tmy-sensor-14:/junos/system/linecard/interface/:/junos/system/linecard/interface/:PFE(\x110¤ǣ\xe5,8\x01@\x01\xaa\x06\x97\x03\xe2\xa4\x01\x92\x03\x1a\x8f\x03\n\x8c\x03\n\bet-0/1/0\x10\xb4\x9d\xb2\xd7\x05\x18\xd4\x05\“\x04ae60*$\b\x00\x10\xf1\x88\xfa\x1f\x18\xbbÈ\x8at \x00(\x000\x008\x00@\x00H\x00P\x00X\xb1\x01`\x80\x80\x80\xe4\x01*\“\b\x01\x10ç¶\x01\x18\xa5\xcaÒ\x05 \x00(\x000\x008\x00@\x00H\x00P\x00X\xf2\x02`\x80\x80\x02*!\b\x02\x10\xbf\xbf\xcd\x01\x18\xa9\xf7\x9a\xb8\x05 \x00(\x000\x008\x00@\x00H\x00P\x00X\x00`\x80\x80\x02*\“\b\x03\x10Ч\xd9\t\x18\x86Å\xb1\n \x00(\x000\x008\x00@\x00H\x00P\x00X>`\x80\x80\xf8\v*\x1a\b\x04\x10\x00\x18\x00 \x00(\x000\x008\x00@\x00H\x00P\x00X



  • 11.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

     
    Posted 10-08-2018 14:36

    Yes telemtry is based on the open sourced grpc project.  You can see the overview here.

     

    https://www.juniper.net/documentation/en_US/junos/topics/concept/junos-telemetry-interface-export-format-understanding.html

     

    I have not used grafana much but thought this was expecting to connect to database formats.  In googling around I can't find a reference to setting up a streaming interface on grafana for grpc.

     



  • 12.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 10-09-2018 07:30

    Thanks Steve, I see the first sentence in that web doc is "The Junos Telemetry Interface supports two ways of exporting data in the protocol buffers (gpb) format:"


    I think those 2 ways are said to be ...

    1 - UDP "native" via linecard - maybe in-band revenue interfaces

    2 - gRPC via Route Engine - maybe out-of-band fxp0

     

    (i use the words "i think" and "maybe" for the obvious reasons that I have not made this work yet and am currently trying to learn this via all sorts of web pages and support avenues)

     

    So Steve, you seem to be speaking to the 2nd option - gRPC Open Sourced project...
    ...and this guy here at the following web link seems to be showing the 3-step process of config'ing the juniper router to do the 1st option - udp/native and gives guidance about using the OpenNTI stuff to make that work....

     

    https://techmocha.blog/2017/07/21/using-opennti-as-a-collector-for-streaming-telemetry-from-juniper-devices-part-2/

     

    https://github.com/Juniper/open-nti

     

    Do you understand this to be 2 different ways of accomplishing this?  I'm wondering if I'm tracking correctly.

     

    {master}[edit]
    agould@lab-960# set services analytics export-profile my-exprt-prfl transport ?
    Possible completions:
    grpc Use grpc transport
    udp Use UDP transport protocol
    {master}[edit]
    agould@lab-960# set services analytics export-profile my-exprt-prfl format ?
    Possible completions:
    gpb Use gpb format
    gpb-sdm Use gpb self-describing-message format



  • 13.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

     
    Posted 10-10-2018 04:57

    My understanding is both these streams are originally from google.  But my real point is that searching the Grafana documentation I don't see reference to supporting these formats for ingestion.

     

    My impression from the site is that it supports a multitude of database storage models but I don't see where it has references to ingest streaming formats let alone that it supports these.

     

     



  • 14.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 10-10-2018 06:40
    Thanks Steve, what do you think about this? (I just realized I sent that link yesterday, but I’m taking a closer look at OpenNTI today... as previously I was unable to boot the OpenNTI docker image... we will try this again on a clean Ubuntu imaged VM)

    The following link mentions using OpenNTI as a package containing all the required options for accomplishing this. Myself and my Linux engineer coworker will try this today.

    https://techmocha.blog/2017/06/26/using-opennti-as-a-collector-for-streaming-telemetry-from-juniper-devices-part-1/


  • 15.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 10-11-2018 05:33

    I got telemetry streaming working using this site ... I have a couple MX960's streaming telemetry to the suite of software provided in this Open-NTI project spoken of on this techmocha blog site. I think my previous problems were related to conflicting installs.... as myself and my coworker had loaded individual items *and* then the open-nti suite (which i understand is a docker container with all the items like grafana, fluentd, chronograf, influxdb, etc).... anyway, we started with a fresh install Ubunto virtual machine and *only* loaded Open-NTI and it works.


    I do not know or understand all of the innerworkings of it at this point, but am quickly learning, even while writing this post... I'm currently using Chronograf hosted at port 8888 and browsing the Data Explorer function and seeing some nice graphs. (I'm wondering if Chrongraf is simply an alternative to Grafana gui front end, unsure) There seems to be tons of items to monitor and analyze, and I'm currently only sending the following sensor resource from the MX960 and there are several more that can be sent.... /junos/system/linecard/interface/


    I am sending the telemetry from the MX960 using UDP transport and GPB format to port 50000 and source port 21111 (mx960-1) and 21112 (mx960-2). I'm unsure that I had to use unique source ports... as I wonder if the source-ip would have been sufficient to make the streaming sources unique in the Open-NTI server.


    Looking at the techmocha pictures, and the "docker ps" command on the linux server, and now this new-found techmocha link (see "deconstructed" below) i understand that FluentD is the TSDB (time series db) that is receiving/ingesting the *Native* streaming form of telemetry from my MX960's on udp port 50000 and looks like fluentd hands off that data to InfluxDB port 8086 (which i think happens internally at that server). (I'm not evening talking about the other form of jti telemetry using openconfig and grpc....I've yet to do that and don't know why I would exactly...which i beleive is ingested using telegraf, unsure)


    ...the link i followed to deploy open-nti suite....

    https://techmocha.blog/2017/06/26/using-opennti-as-a-collector-for-streaming-telemetry-from-juniper-devices-part-1/#comments


    ...interestingly, i just now found this, which apparently is a way of deploying all the components individually...

    https://techmocha.blog/2017/10/31/serving-up-opennti-deconstructed/

     



  • 16.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 02-14-2018 22:40

    Hi Python,

    Could you help please? any guides available for configuring telemetry with Juniper vMX?



  • 17.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 02-14-2018 22:51


  • 18.  RE: Telemetry data is not streaming from Juniper vMX 17.4R1.16

    Posted 02-20-2018 22:16

    Hi Kingsman,

    The guide that you mentioned in the post, doesn't help much. The end-to-end steps are missing. For e.g. I initially started with configuring telemetry over UDP and then I came across this post that it is not possible for Juniper devices to send telemetry data over fpx0 (management port). I tried using VCP & VFP VM's but then again there is no good documentation in order to understand how they have to be setup. Now I'm trying with gRPC based sensors. I have installed the Network Agent (upgrade FreeBSD version) on my vMX17.4R1.16, and done the following configuration using the configure mode:

    set system services extension-service request-response grpc clear-text port 50051
    set system services extension-service request-response grpc skip-authentication
    set system services extension-service notification allow-clients address 0.0.0.0/0

    Now my question is, how to subscribe to telemetry data? Which file or command needs to be looked at? Is there a Juniper Telemetry Interface Client which needs to be setup (one link i found - https://github.com/nileshsimaria/jtimon) ?

     

    My Juniper configuration look like this:

    version 17.4R1.16;
    system {
        host-name vmx-junos-telemetry;
        root-authentication {
            encrypted-password "$6$w8tRl2N7$DVwFSsoyT9NHh741AbUMZXs8GxE2CzOabpu6OdhO                                                                                                                     t/C60sxYBIUKfzvEz3/MpphGBMXbUS.aYv8GhSX64JHUs/"; ## SECRET-DATA
        }
        services {
            ftp;
            ssh {
                root-login allow;
            }
            telnet;
            extension-service {
                request-response {
                    grpc {
                        clear-text {
                            port 50051;
                        }
                        skip-authentication;
                    }
                }
                notification {
                    allow-clients {
                        address 0.0.0.0/0;
                    }
                }
            }
            netconf {
                ssh;
            }
        }
        syslog {
            user * {
                any emergency;
            }
            file messages {
                any notice;
                authorization info;
            }
            file interactive-commands {
                interactive-commands any;
            }
        }
        processes {
            dhcp-service {
                traceoptions {
                    file dhcp_logfile size 10m;
                    level all;
                    flag all;
                }
            }
        }
    }
    services {
        analytics {
            streaming-server telemetry-server {
                remote-address 192.168.152.30;
                remote-port 2030;
            }
            export-profile telemetry {
                local-address 192.168.152.40;
                local-port 35000;
                forwarding-class assured-forwarding;
                reporting-rate 60;
                format gpb;
                transport udp;
            }
            sensor cpu {
                server-name telemetry-server;
                export-name telemetry;
                resource /junos/system/linecard/cpu/memory/;
            }
        }
    }
    interfaces {
        fxp0 {
            unit 0 {
                family inet {
                    address 192.168.152.40/24;
                }
            }
        }
    }

    In my first attempt with streaming telemetry data using UDP, I have configured the streaming server, export-profile, resources using this guide - https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/junos-telemetry-interface-configuring.html

    So do I have to delete this configuration in order for the gRPC sensors to work?

     

    Thanks,

    Vinay