Switching

Expand all | Collapse all

NTP process is stuck on EX.

  • 1.  NTP process is stuck on EX.

    Posted 10-29-2014 19:32

    I found the NTP process is stuck on EX.

    I have deactivated NTP by issuing a “ deactivate system ntp “ in the JunOS CLI, which shut down the xntpd process, but there is a tnp.sntpd or tnp.sntpc that is used by NTP, remains stuck:

    root@SW01:RE:0% ps aux | grep -i tnp
    root    1289  0.0  0.0  2264   992  ??  I    10Oct14   0:02.59 /usr/sbin/tnp.sn    <————————— This process is stuck, refuses to quit by itself
    root   46764  0.0  0.1  2392  1064  u0  S+    9:10PM   0:00.01 grep -i tnp
    root@SW01:RE:0%

    Which would explain these log messages:

    Oct 29 21:05:21  SW01 xntpd[46408]: bind() fd 5, family 2, port 123, addr 0.0.0.0, in_classd=0 flags=8 fails: Address already in use
    Oct 29 21:05:22  SW01 xntpd[46408]: setsockopt IP_MULTICAST_TTL fails on address 0.0.0.0: Bad file descriptor
    Oct 29 21:05:22  SW01 xntpd[46408]: sendto(10.1.21.254): Bad file descriptor


    Normally, like a Unix box, you can issue a “ kill 1289 “ to kill off the process. But again, its a switch in production use. I’m uneasy issuing that command…

     

    So,does someone can help me with it. Is there any impact if I use kill command on the EX.

     

    Thanks.

     

     
     


  • 2.  RE: NTP process is stuck on EX.

     
    Posted 10-30-2014 01:01

    Why do you think the logs are related to that process? The logs are generated by process 46408.

     

    tnp.sntp is not going away even after a reboot after ntp was completely disabled:

     

    {master:0}[edit]
    lab@ex4200# show system | display set | match ntp | except server
    set system processes ntp disable
    deactivate system ntp

    {master:0}[edit]
    lab@ex4200#

     

    After reboot:


    root@ex4200:RE:0% ps axu | grep ntp
    root  1305  0.0  0.1  2264   992  ??  S     8:55AM   0:00.01 /usr/sbin/tnp.sntp
    root  1400  0.0  0.1  2388  1056  u0  S+    8:56AM   0:00.01 grep ntp
    root@ex4200:RE:0%

     

     

     

    =====

    If this worked for you please flag my post as an "Accepted Solution" so others can benefit. A kudo would be cool if you think I earned it.



  • 3.  RE: NTP process is stuck on EX.

    Posted 10-30-2014 18:06

     

    I still have this odd issue. Please see my output from switch. The 10.1.21.254 is firewall VIP. I already have rule permit port 123. Do you have any ideals? Thank you.

     

     

     

    root@SW01> show configuration system ntp | display set
    set system ntp server 10.1.21.254

    {master:0}
    root@SW01> show configuration forwarding-options | display set
    set forwarding-options helpers port 123 server 10.1.1.254
    set forwarding-options helpers port 123 interface vlan.256

    {master:0}
    root@SW01> show ntp status                                        
    localhost: timed out, nothing received
    ***Request timed out

    {master:0}
    root@SW01> ping 10.1.21.254
    PING 10.1.21.254 (10.1.21.254): 56 data bytes
    64 bytes from 10.1.21.254: icmp_seq=0 ttl=64 time=1.234 ms
    64 bytes from 10.1.21.254: icmp_seq=1 ttl=64 time=1.240 ms
    64 bytes from 10.1.21.254: icmp_seq=2 ttl=64 time=1.463 ms
    ^C
    --- 10.1.21.254 ping statistics ---
    3 packets transmitted, 3 packets received, 0% packet loss
    round-trip min/avg/max/stddev = 1.234/1.312/1.463/0.107 ms

     


  • 4.  RE: NTP process is stuck on EX.

    Posted 11-01-2014 01:28

    Hi there,

     


    @eric.song wrote:

     

    I still have this odd issue. 

     

    No, You are having a different one.

     


    @eric.song wrote:

     

     The 10.1.21.254 is firewall VIP. I already have rule permit port 123. Do you have any ideals? Thank you.

     

     

     

    root@SW01> show configuration system ntp | display set
    set system ntp server 10.1.21.254


    Did You check this FW VIP actually can serve time via NTP? If it cannot, please choose a different time server. If You are labbing this up and don't have internet access/don't want your lab gear to go out to internet, You can make any Windows PC to become an NTP server, there are plenty of instructions on google.

     

     

     


    @eric.song wrote:

     

     

    {master:0}
    root@SW01> show configuration forwarding-options | display set
    set forwarding-options helpers port 123 server 10.1.1.254
    set forwarding-options helpers port 123 interface vlan.256


     

    This one is for BROADCAST NTP packets from OTHER machines. Locally-generated traffic, broadcast or not, is not captured by IP helper.

     


    @eric.song wrote:

     

     

    {master:0}
    root@SW01> show ntp status                                        
    localhost: timed out, nothing received
    ***Request timed out


     

    Please try:

    1/ disable Your lo0.0 filter if You have any, and

    2/ add this line to the config 

     

    set system static-host-mapping localhost inet 127.0.0.1

     

    3/ run "show ntp status no-resolve"

    - and then report back.

     

    HTH

    Thanks
    Alex



  • 5.  RE: NTP process is stuck on EX.

    Posted 11-01-2014 06:13

     

    Thank you so much for your help.

     

    The firewall VIP is working for sure. I've tested it with my laptop, I can get the time from firewall VIP. Also I've configured other EX switch using NTP server as 10.1.21.254 (VIP). It is getting time from there. I had NTP attacks from ouside router before. The attacks were stoped when I disabled the interface on EX.

     

    I confirmed with Juniper J-TAC, it is means NTP not configured if I get the output as below.

    root@SW01> show ntp status                                        
    localhost: timed out, nothing received
    ***Request timed out

     

    So I don't have any filter on the EX, this is our producation environment. We have l3-interface routing enabled and many things on it. I don't know I will have some risk if I add the command line on the EX.

     

    "set system static-host-mapping localhost inet 127.0.0.1"

     

    If the xntp session is ok for Juniper EX, I have no ideal what is cause and how to solve it.

     

    Thanks again.

     

    Eric

     

     

     

     

     

     

     

     

     

     

     

     


  • 6.  RE: NTP process is stuck on EX.

    Posted 11-01-2014 10:23

    Hello,

     


    @eric.song wrote:

     

     

    I confirmed with Juniper J-TAC, it is means NTP not configured if I get the output as below.

    root@SW01> show ntp status                                        
    localhost: timed out, nothing received
    ***Request timed out

     

     

     

    Eric

     

     


    This is only half of the story. The other half is that this command produces same output when UDP port 123 is blocked for: (a) NTP source-address and (b) 127.0.0.1 if no NTP source-address is configured, in lo0.0 filter.

    That's why I asked You to disable lo0 filter.

     

    These two situations can be discerned with another command:

     

    1/ when NTP is disabled, the following printout applies:

     

    aarseniev@m7i> set date ntp 
    Nov 01 18:13:47
     1 Nov 18:13:47 ntpdate[32415]: no servers can be used, exiting

     2/ when UDP port 123 is blocked in lo0.0 filter

     

    aarseniev@rm7i> set date ntp    
    Nov 01 18:16:19
     1 Nov 18:16:19 ntpdate[32761]: no server suitable for synchronization found

     


    @eric.song wrote:

     

     I don't know I will have some risk if I add the command line on the EX.

     

    "set system static-host-mapping localhost inet 127.0.0.1"

     

     

     

     

     

    The "risk" is that when any JUNOS CLI command referencing "localhost" is run (which is almost any "show ntp..." command), instead of querying Your DNS server for 127.0.0.1, the resolution will use this static mapping. Your DNS server may not have any idea how to resolve "localhost", hence commands previously hanging before will mysteriously start working.

     

    HTH

    Thanks
    Alex



  • 7.  RE: NTP process is stuck on EX.

    Posted 04-14-2019 23:37

    Hi, 

     

    Thanks for your inputs in this discussion.

     

    I am a bit struggling to understand - "This one is for BROADCAST NTP packets from OTHER machines. Locally-generated traffic, broadcast or not, is not captured by IP helper."

     

    Do you mean locally-generated traffic as within the same subnet? OR device specific internal generated traffic? What do you mean, OTHER machines here?

     

    Thank you. 



  • 8.  RE: NTP process is stuck on EX.

    Posted 11-01-2014 01:33

    Hello,

     


    @eric.song wrote:

    I found the NTP process is stuck on EX.

    I have deactivated NTP by issuing a “ deactivate system ntp “ in the JunOS CLI, which shut down the xntpd process, but there is a tnp.sntpd or tnp.sntpc that is used by NTP, remains stuck:


     

     
     

    This process is not stuck, it is used by EX cluster master to serve time to the other cluster members not via IP but via internal Juniper-proprietary transport protocol called TNP.

     


    @eric.song wrote:


    Normally, like a Unix box, you can issue a “ kill 1289 “ to kill off the process. But again, its a switch in production use. I’m uneasy issuing that command…

     

    So,does someone can help me with it. Is there any impact if I use kill command on the EX.

     

    Thanks.

     

     
     

    If You kill this process, it will be re-spawned again. But it has nothing to do with Your syslog messages, as already pointed out by others.

    HTH

    Thanks
    Alex