Switching

I am trying to minimize issues I've run into when I do large firmware update projects. We have around 350 EX series switches, and I've not had a 100% issue-free upgrade.  

I have checked system storage to make sure there is space on the switch, before this last project. I make a flow chart of the switch connections so I can start with the most edge switches, then fall back to the next layer so I don't have a switch rebooting and a downstream switch loses connection during it's update process. The core switches I always do last. 

During every project I end up with some switches that get the update loaded, I then visit the Mist Dashboard and click the "Reboot Switch" button as part of the notice to "Reboot the switch to finish the update". I'll check the Switch Insights and see how far along it's gotten and wait until it is back online and the version shows the new firmware. Then... occasionally, I get a switch that will display " Switch disconnected " and never get connected again. Those I have to visit with a console cable to determine what state it is in. Sometimes the console session will not provide a login prompt and I have to power cycle it. Most of the time I get a UNIX prompt and I have to tell it to reboot.  The power cycle step always works.  

I do not use the " reboot switch after update " checkbox, as I can not reboot the switches during normal use hours. I let them download the update and then manually reboot them later at night.  That way, they all have the update and should just need to reboot. 

Is there something I am missing? 

Hi John,

Thanks for sharing your feedback.

You can follow the following steps:

1. Ensure NTP port 123 is allowed on the firewall. If that doesn't resolve the issue, or if it is already done,
2. After recovering the switch go to /var/log directory and look into the following log files:
   • messages logs
   • outbound-ssh logs
   • mist-critical logs
3. If by these logs you are not able to identify and rectify the issue, please raise a case with JTAC, they will be able to help you out.

I don't think keeping 'Reboot after image copy' unchecked is causing any issue. 

If you are not getting a prompt after you connect the console cable, it could indicate an issue with the switch itself. Also, if the old configuration is incompatible with the new Junos version, it might go to Amnesiac mode, from where you will have to remove the unsupported configuration to get it up. For these things, we would need more details and a thorough troubleshooting; opening a JTAC case is the best way forward.

I am trying to minimize issues I've run into when I do large firmware update projects. We have around 350 EX series switches, and I've not had a 100% issue-free upgrade.  

I have checked system storage to make sure there is space on the switch, before this last project. I make a flow chart of the switch connections so I can start with the most edge switches, then fall back to the next layer so I don't have a switch rebooting and a downstream switch loses connection during it's update process. The core switches I always do last. 

During every project I end up with some switches that get the update loaded, I then visit the Mist Dashboard and click the "Reboot Switch" button as part of the notice to "Reboot the switch to finish the update". I'll check the Switch Insights and see how far along it's gotten and wait until it is back online and the version shows the new firmware. Then... occasionally, I get a switch that will display " Switch disconnected " and never get connected again. Those I have to visit with a console cable to determine what state it is in. Sometimes the console session will not provide a login prompt and I have to power cycle it. Most of the time I get a UNIX prompt and I have to tell it to reboot.  The power cycle step always works.  

I do not use the " reboot switch after update " checkbox, as I can not reboot the switches during normal use hours. I let them download the update and then manually reboot them later at night.  That way, they all have the update and should just need to reboot. 

Is there something I am missing? 

Thanks for the suggestions, Sougata. 

I suspect that if there was an issue getting to remote time servers, all of the switches would have an issue during the update. We have two time servers on our network, which are listed in the config, so I would hope that would cover all situations. 

I'll check out the logs you mentioned and see if something stands out. 

I hope I never have a switch that can not properly boot up after an upgrade.  That would result in a number of classrooms to even a whole school campus not working, depending on what switch has a serious issue. 

Hi John,

Thanks for sharing your feedback.

You can follow the following steps:

1. Ensure NTP port 123 is allowed on the firewall. If that doesn't resolve the issue, or if it is already done,
2. After recovering the switch go to /var/log directory and look into the following log files:
   • messages logs
   • outbound-ssh logs
   • mist-critical logs
3. If by these logs you are not able to identify and rectify the issue, please raise a case with JTAC, they will be able to help you out.

I don't think keeping 'Reboot after image copy' unchecked is causing any issue. 

If you are not getting a prompt after you connect the console cable, it could indicate an issue with the switch itself. Also, if the old configuration is incompatible with the new Junos version, it might go to Amnesiac mode, from where you will have to remove the unsupported configuration to get it up. For these things, we would need more details and a thorough troubleshooting; opening a JTAC case is the best way forward.

I am trying to minimize issues I've run into when I do large firmware update projects. We have around 350 EX series switches, and I've not had a 100% issue-free upgrade.  

I have checked system storage to make sure there is space on the switch, before this last project. I make a flow chart of the switch connections so I can start with the most edge switches, then fall back to the next layer so I don't have a switch rebooting and a downstream switch loses connection during it's update process. The core switches I always do last. 

During every project I end up with some switches that get the update loaded, I then visit the Mist Dashboard and click the "Reboot Switch" button as part of the notice to "Reboot the switch to finish the update". I'll check the Switch Insights and see how far along it's gotten and wait until it is back online and the version shows the new firmware. Then... occasionally, I get a switch that will display " Switch disconnected " and never get connected again. Those I have to visit with a console cable to determine what state it is in. Sometimes the console session will not provide a login prompt and I have to power cycle it. Most of the time I get a UNIX prompt and I have to tell it to reboot.  The power cycle step always works.  

I do not use the " reboot switch after update " checkbox, as I can not reboot the switches during normal use hours. I let them download the update and then manually reboot them later at night.  That way, they all have the update and should just need to reboot. 

Is there something I am missing? 

I did not see anything interesting in the logs. 

Thanks for the suggestions, Sougata. 

I suspect that if there was an issue getting to remote time servers, all of the switches would have an issue during the update. We have two time servers on our network, which are listed in the config, so I would hope that would cover all situations. 

I'll check out the logs you mentioned and see if something stands out. 

I hope I never have a switch that can not properly boot up after an upgrade.  That would result in a number of classrooms to even a whole school campus not working, depending on what switch has a serious issue. 

Hi John,

Thanks for sharing your feedback.

You can follow the following steps:

1. Ensure NTP port 123 is allowed on the firewall. If that doesn't resolve the issue, or if it is already done,
2. After recovering the switch go to /var/log directory and look into the following log files:
   • messages logs
   • outbound-ssh logs
   • mist-critical logs
3. If by these logs you are not able to identify and rectify the issue, please raise a case with JTAC, they will be able to help you out.

I don't think keeping 'Reboot after image copy' unchecked is causing any issue. 

If you are not getting a prompt after you connect the console cable, it could indicate an issue with the switch itself. Also, if the old configuration is incompatible with the new Junos version, it might go to Amnesiac mode, from where you will have to remove the unsupported configuration to get it up. For these things, we would need more details and a thorough troubleshooting; opening a JTAC case is the best way forward.

I am trying to minimize issues I've run into when I do large firmware update projects. We have around 350 EX series switches, and I've not had a 100% issue-free upgrade.  

I have checked system storage to make sure there is space on the switch, before this last project. I make a flow chart of the switch connections so I can start with the most edge switches, then fall back to the next layer so I don't have a switch rebooting and a downstream switch loses connection during it's update process. The core switches I always do last. 

During every project I end up with some switches that get the update loaded, I then visit the Mist Dashboard and click the "Reboot Switch" button as part of the notice to "Reboot the switch to finish the update". I'll check the Switch Insights and see how far along it's gotten and wait until it is back online and the version shows the new firmware. Then... occasionally, I get a switch that will display " Switch disconnected " and never get connected again. Those I have to visit with a console cable to determine what state it is in. Sometimes the console session will not provide a login prompt and I have to power cycle it. Most of the time I get a UNIX prompt and I have to tell it to reboot.  The power cycle step always works.  

I do not use the " reboot switch after update " checkbox, as I can not reboot the switches during normal use hours. I let them download the update and then manually reboot them later at night.  That way, they all have the update and should just need to reboot. 

Is there something I am missing? 

Hi John,

I understand your predicament. Yeah, I hope it doesn't fail, but if in case it fails you can try these steps. For now, the logs might have been overwritten, we don't know for sure. Without knowing the details of the issue it will be difficult to find out what really happened. Since this is a break/fix kind of a situation, to ensure it gets the best attention, it is advisable to open a TAC case right away.  You can open a support case from the circled '?' option that shows up in the right top corner - the first option in that is 'Support Tickets'. Alternatively, you can ask Marvis CA (chatbot) to give you the link.

https://www.juniper.net/documentation/us/en/software/mist/mist-management/topics/task/support-create-ticket.html

I did not see anything interesting in the logs. 

Thanks for the suggestions, Sougata. 

I suspect that if there was an issue getting to remote time servers, all of the switches would have an issue during the update. We have two time servers on our network, which are listed in the config, so I would hope that would cover all situations. 

I'll check out the logs you mentioned and see if something stands out. 

I hope I never have a switch that can not properly boot up after an upgrade.  That would result in a number of classrooms to even a whole school campus not working, depending on what switch has a serious issue. 

Hi John,

Thanks for sharing your feedback.

You can follow the following steps:

1. Ensure NTP port 123 is allowed on the firewall. If that doesn't resolve the issue, or if it is already done,
2. After recovering the switch go to /var/log directory and look into the following log files:
   • messages logs
   • outbound-ssh logs
   • mist-critical logs
3. If by these logs you are not able to identify and rectify the issue, please raise a case with JTAC, they will be able to help you out.

I don't think keeping 'Reboot after image copy' unchecked is causing any issue. 

If you are not getting a prompt after you connect the console cable, it could indicate an issue with the switch itself. Also, if the old configuration is incompatible with the new Junos version, it might go to Amnesiac mode, from where you will have to remove the unsupported configuration to get it up. For these things, we would need more details and a thorough troubleshooting; opening a JTAC case is the best way forward.

I am trying to minimize issues I've run into when I do large firmware update projects. We have around 350 EX series switches, and I've not had a 100% issue-free upgrade.  

I have checked system storage to make sure there is space on the switch, before this last project. I make a flow chart of the switch connections so I can start with the most edge switches, then fall back to the next layer so I don't have a switch rebooting and a downstream switch loses connection during it's update process. The core switches I always do last. 

During every project I end up with some switches that get the update loaded, I then visit the Mist Dashboard and click the "Reboot Switch" button as part of the notice to "Reboot the switch to finish the update". I'll check the Switch Insights and see how far along it's gotten and wait until it is back online and the version shows the new firmware. Then... occasionally, I get a switch that will display " Switch disconnected " and never get connected again. Those I have to visit with a console cable to determine what state it is in. Sometimes the console session will not provide a login prompt and I have to power cycle it. Most of the time I get a UNIX prompt and I have to tell it to reboot.  The power cycle step always works.  

I do not use the " reboot switch after update " checkbox, as I can not reboot the switches during normal use hours. I let them download the update and then manually reboot them later at night.  That way, they all have the update and should just need to reboot. 

Is there something I am missing? 

Would it be a Mist ticket or a JTAC ticket?

Hi John,

I understand your predicament. Yeah, I hope it doesn't fail, but if in case it fails you can try these steps. For now, the logs might have been overwritten, we don't know for sure. Without knowing the details of the issue it will be difficult to find out what really happened. Since this is a break/fix kind of a situation, to ensure it gets the best attention, it is advisable to open a TAC case right away.  You can open a support case from the circled '?' option that shows up in the right top corner - the first option in that is 'Support Tickets'. Alternatively, you can ask Marvis CA (chatbot) to give you the link.

https://www.juniper.net/documentation/us/en/software/mist/mist-management/topics/task/support-create-ticket.html

I did not see anything interesting in the logs. 

Thanks for the suggestions, Sougata. 

I suspect that if there was an issue getting to remote time servers, all of the switches would have an issue during the update. We have two time servers on our network, which are listed in the config, so I would hope that would cover all situations. 

I'll check out the logs you mentioned and see if something stands out. 

I hope I never have a switch that can not properly boot up after an upgrade.  That would result in a number of classrooms to even a whole school campus not working, depending on what switch has a serious issue. 

Hi John,

Thanks for sharing your feedback.

You can follow the following steps:

1. Ensure NTP port 123 is allowed on the firewall. If that doesn't resolve the issue, or if it is already done,
2. After recovering the switch go to /var/log directory and look into the following log files:
   • messages logs
   • outbound-ssh logs
   • mist-critical logs
3. If by these logs you are not able to identify and rectify the issue, please raise a case with JTAC, they will be able to help you out.

I don't think keeping 'Reboot after image copy' unchecked is causing any issue. 

If you are not getting a prompt after you connect the console cable, it could indicate an issue with the switch itself. Also, if the old configuration is incompatible with the new Junos version, it might go to Amnesiac mode, from where you will have to remove the unsupported configuration to get it up. For these things, we would need more details and a thorough troubleshooting; opening a JTAC case is the best way forward.

I am trying to minimize issues I've run into when I do large firmware update projects. We have around 350 EX series switches, and I've not had a 100% issue-free upgrade.  

I have checked system storage to make sure there is space on the switch, before this last project. I make a flow chart of the switch connections so I can start with the most edge switches, then fall back to the next layer so I don't have a switch rebooting and a downstream switch loses connection during it's update process. The core switches I always do last. 

During every project I end up with some switches that get the update loaded, I then visit the Mist Dashboard and click the "Reboot Switch" button as part of the notice to "Reboot the switch to finish the update". I'll check the Switch Insights and see how far along it's gotten and wait until it is back online and the version shows the new firmware. Then... occasionally, I get a switch that will display " Switch disconnected " and never get connected again. Those I have to visit with a console cable to determine what state it is in. Sometimes the console session will not provide a login prompt and I have to power cycle it. Most of the time I get a UNIX prompt and I have to tell it to reboot.  The power cycle step always works.  

I do not use the " reboot switch after update " checkbox, as I can not reboot the switches during normal use hours. I let them download the update and then manually reboot them later at night.  That way, they all have the update and should just need to reboot. 

Is there something I am missing?