Hello,
We recently deployed a new MX240 in our network, and have encountered some odd behavior in regards to NETCONF. We manage our other MX240 routers with a couple of NETCONF scripts written in python that do not work on this new MX240. Basically, these scripts will run and successfully establish an SSH connection with the MX240, but then close the session, causing the script to stop running. We've been able to rule out misconfigured firewalls and SSH/known_host keys as potential causes and now suspect either NETCONF or a NETCONF-related process isn't running properly on the MX240. Here is what we're seeing:
To rule out the possibility that it was our python scripts at fault, I tried to establish a NETCONF connection just from a Python shell: If I try to start a NETCONF session from the Python interactive shell on our management server the following way:
root@managementserver:~# python
Python 2.7.17 (default, Jul 20 2020, 15:37:01)
[GCC 7.5.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from ncclient import manager
>>> conn = manager.connect (host='172.16.0.142',
... port=22,
... username='username',
... password='password',
... hostkey_verify=False,)
I get the following result:
No handlers could be found for logger "paramiko.hostkeys"
Traceback (most recent call last):
File "<stdin>", line 5, in <module>
File "/usr/local/lib/python2.7/dist-packages/ncclient/manager.py", line 147, in connect
return connect_ssh(*args, **kwds)
File "/usr/local/lib/python2.7/dist-packages/ncclient/manager.py", line 120, in connect_ssh
raise ex
ncclient.transport.errors.SessionCloseError: Unexpected session close
>>>
Again, we've already ruled out any problems with the SSH keys in our management server's known_hosts file, and the last line in the error output suggests that our new MX240 is the one ending the session, not our server. When I look at our MX240's messages.log:
Oct 5 14:27:15 new_router2 sshd[21255]: Accepted password for username from 192.168.0.100 port 49746 ssh2
Oct 5 14:27:15 new_router2 sshd[21258]: fatal: Failed to set uids to 60000.
So it seems clear that our server is able to establish an SSH connection, but our MX240 closes it immediately after. We then tried configuring traceoptions to log NETCONF activity. Here is how we configured it:
user@new_router2> show configuration system services
netconf {
ssh;
traceoptions {
file netconf.log size 1m files 5;
flag all;
}
}
What's odd is the netconf.log file is never created. Here is what we see in our messages.log immediately after committing the netconf traceoptions config:
Oct 5 14:28:15 new_router2 mgd[21269]: UI_COMMIT: User 'user' requested 'commit' operation (comment: none)
Oct 5 14:28:16 new_router2 mgd[21269]: UI_CHILD_EXITED: Child exited: PID 21283, status 7, command '/usr/sbin/mustd'
Oct 5 14:28:20 new_router2 inetd[17402]: shell/tcp: setsockopt: No such file or directory
If we run " show log ?" on our MX240, the netconf.log file is not listed as an option.
When I did a cursory Problem Report search, I couldn't find any known NETCONF-related issues that matched our current experience. Any thoughts or suggestions on what to try next are extremely appreciated!