As the title describes, i have a big batch of EX2300-C switches that i want to upgrade in batches. I configured a linux server for both DHCP and TFTP and Zero Touch is working fine. The switch boots up out of the box, gets an IP from the DHCP server and then automatically fetches the junos package i included in my ZTP configuration on the server. My problem is with the EX2300-C not having enough space to unpack and install the new Junos. This is a known issue we encountered before and i can overcome that by adding the knobs "force" and "unlink" in "request system software add" when i am upgrading the switch manually.
Im therefore looking for a way to include these knobs when the switches are fetching the package automatically from the server.
This issue is addressed on Junos 18.2R3-S1 or 18.3R3 and higher, please see the following:
For other Junos versions, although cleaning up the device storage can help as a workaround, but that needs to be done prior to ZTP so the workaround may not be a good fit for pure ZTP.
Hope this helps.
If this solves your problem, please mark this post as "Accepted Solution."Kudos are always appreciated :).
I ended up using ZTP to push a SLAX script and event option, the SLAX script would do the install with all the options needed and also did storage cleanup for snapshot recoveries. Im willing to share if you want.
I would definitely appreciate it if you could share the script Kyle!
Ill get it sorted in my Github and post it up tomorrow.
I've been bashing my head aganst this the last few days and would greatly appreciate the slax script. I have 28 EX2300-C sitting on a table plugged up in the lab to do ZTP but cannot deploy because it gets stuck in the out of space loop. Really wanted those ready to go on Monday but here we are.
At least I'm not on this island of ZTP suck alone.
as far as the out os space issue, the only 2 ways I found out of this is ...
it's no slax, but here is the content of an .sh file I sent to all my switches, then executed and went for coffe.
root@vvb-valparadis-01-BYPASS:RE:0% cat GetImageAndUpgrade.shcli -c "file copy ftp://10.10.10.4/ex2300/junos-arm-32-18.2R3-S2.9.tgz /tmp/junos-arm-32-18.2R3-S2.9.tgz"cli -c "request system software add /tmp/junos-arm-32-18.2R3-S2.9.tgz force unlink no-copy reboot"
«hope it helps,
I worked out another option last night that seems to work for me. I boot the switch and do CTRL-C to stop uboot from loading the juniper partition. At the menu I go through the steps to get to the loader> prompt (5, then 5) and enter:
install --format tftp://<ip address>/junos-install-media-net-ex-arm-32-18.2R3.4.tgz
then walk away and let it reinstall using 18.2R3.4 (which seems to fix/ignore the space alert that 18.1 gives). Then on first boot it will go through the normal ZTP procedure to upgrade the switch to 18.2R3-S2.9 and get our template config on it.
The good thing is that loader is at least DHCP aware and as soon as you give it a TFTP path it starts the vme interface and gets an IP from the DHCP server so you don't have to setup the network parameters on each switch.
I think when Juniper starts shipping 2300s with 18.2 as the preload then we will not have to do these work arounds. But at least now I've got it workable solution for these 50 or so I'm getting in the next few weeks.
Thanks for sharing your procedure too; hopefully between the multiple ways here this will help others in the same boat.
that's worth a shot. I'll try that
Im in the middle of trying everything out right now and i will be reporting back with my experience :). But for the life of me, i cannot figure out dhcp options on windows. I had a setup working fine with an ISC DHCP server on linux but i want to use a windows laptop and i tried tftpd with dhcp but it seems options are not going through as i want them to in order for Junos to get the image file.
Michel what kind of dhcp/tftp server are you using on windows?
Michel what kind of dhcp/tftp server are you using on windows?
Denis is managing this. I believe we have a Ubuntu server running dhcp. I just ssh to it and modify the /etc/dhcp/dhcpd.conf file depending on the config and image file I want installed when the new or reseted switch connect to it.
I don't believe we have a tftp server - I just send ftp requests. I do activate ftp on every Juniper switch.
I can forward other questions to Denis if you need.
by the way, I am trying another solution on the swizzle reboot problem. 240 switch have been running at least 18 hours now with no swizzle reboot. Hope it works.
Ooof, Windows Server's DHCP Server does not make setting up the options easy. Can your run a small Linux instance in a VM in something like VirtualBox or maybe even a Docker instance for DHCP and TFTP? Configuring either one of those may be easier than converting the text values needed in Option 43 sub-options for ZTP to the hexadecimal strings required to be encapsulated in for option 43 on windows boxes. I did just find this at GitHub: https://github.com/JNPRAutomate/opt43builder I've not used it but that might be the way to get all those sub-options converted into a hex string for use in Windows DHCP.
Also if you are trying to do this on a Windows 10 install instead of some version of Windows Server maybe http://tftpd32.jounin.net/ is something to look at.
Another option maybe Cygwin running dhcpd or maybe even Windows Services for Linux and the Ubuntu instance from the Windows Store.
I'm a Mac/Linux user so I can't offer much more than the results of some GoogleFu above. For something portable I'd probably grab a RaspberryPi or the like and get dhcpd, tftpd, rsyslog, and a http server running and call it done. Or maybe a inexpensive tiny laptop that would take a Linux distro.
good morning jgrider,
just learned from JTAC that there is a PR related to the space issue.
On EX2300 and EX3400 switches, when you perform multiple CLI upgrades, sometimes the software fails with "insufficient space" error
looking forward to a solve !
Still have to try your trick
probably today: I have to cleran up some switches.
there is no way to introduce those knobs from the ZTP configs on server, there should be a modification for EX2300 and EX3400 to have the unlink option default , could you please check what is the SW version on the device before the upgrade ? what you can do is maybe a python script to access the reachable IPs in DHCP server provided subnet and check the current version and if it is still not upgraded you can upgrade it using the script and utilizing pyez libraries.
If this solves your problem, please mark this post as "Accepted Solution
I am surprised that they do this to you right out of the box. I believe at least the first time I upgraded them using a similar ztp setup it worked fine. But that was a while ago.
I have had to install images on a number of EX2300 , and been confronted with the space issue. Here are the 2 tricks I developped and that work especially well on switches that have been used and are not brand new.
1) I have a bunch of autoboot usb keys with an old Image. I'll line up 10 switches, power them up , connect the vme (mgmt) port to access the dhcp server, plug in the usb keys and just issue a request system reboot usb from the console: the process will go on and put the switch back in a state where it accept the image coming from the DHCP without arguing (I am using 18.2R3S2 - which among other things solve the PR1442376 issue, and it's the recommended JTAC image) . It worked for me every time
2) on a connected switch, I will ftp the image not into the /var/tmp directory as recommended but in the /tmp directory. Then, issuing a request system software add /tmp/junos-arm-32-18.2R3-S2.9.tgz force unlink no-copy reboot . for some reason, I don't get the no space failure. That also worked everytime, and I have 240 of these things !!! I used secureCRT to access my switches, and load an .sh file on each switch that fetched the image and executed it using "cli -c "blablabla" lines. I upgraded 240 switches in an afternoon
once this is done, if I may suggest to disable the miniusb console port : go figure, but I haven't had a single swizzle reboot on the switches I did that ( set system port aux port-type rj45 disable) . You can try it , it will only disable the miniusb port console. If you get message like "getty request...", I strongly suggest you do it.
more on this here
have fun !
Thats exactly the issue we have. I have hundreds of 2300s and i need to get them to 18.2R3-S2.9 due to the PR you mentioned (and a couple of others like the LED issue etc..) !
I will definitely try the autoboot procedure, havent htought of that. I did think of just plugging a bunch of usbs on a bunch of switches and then running some kind of a script to go through the procedure to mount it, copy the file and blah blah but it was still too cumbersome. Your method might do the trick!
Ill go through the thread you posted as well, try your suggested methods and come back.
Thank you very much for the awesome reply, appreciated!
glad I can help. We have had to postpone deploying 240 switches in the field due to 2 issues: the PR1442376, that I affectuously called "the zombie problem", and random reboot diagnosed as "swizzle reboot". Therefore, I have ended up with a big laboratory of ex2300 switches !!!
we will start deploying in january, so my lab will be dismantled. Right now, all my switches are running 18.2R3-S2.9 with the miniusb port disable. The 8 switches in production are running this setup too and I don't hear from them anymore.
I want to get into the slax and writing scripts now. next challenge.
if you have hundreds of 2300, I strongly suggest the disabling of the miniusb console port to prevent swizzle reboot. On my sample of 250 switches, I had about 30 that were susceptible to having it.
Show Chassis routing-engine will show you the last reboot reason of a switch. if a switch swizzle, turn on messages (monitor start messages) wait a minute to see if you get a deluge of getty request. JTAC answered to me at the time that it was related to bad consol usb cabling. Since I had none connected, we decided to simply turn it off, and it seems to fix the issue. They are working on it now and I was told a PR was issued
I had a switch swizzle reboot friday the 13 ... now it's only one in 240, on which the miniusb console port solution had been running successfully for 2 months, but still ...
I am sending the files to JTAC. I'll keep you posted.