Junos OS

Have some trouble:

After upgarade to 19.1R1.6 Junos on my EX2300-48 switches (have 10 switches) - i have some problem:

1) 3 of 10 switches stop working after  a certain amount of time (one switch a day after the update, the second after a week) and only the reboot helps. Unfortunately, I do not see what is happening with it at the moment, core dumps are empty

2) show chassis enviroment dont show second fan and show chassis alarms - no alarms - its software bug or its really second fan doesnt work? On 18.2R1.9 Junos second fan is visible.

3) and of course jweb not work - but this problem was in release note document.ex2300-48t with Junos 18.2R1.9ex2300-48T with Junos 19.1R1.6

Stops working meaning what? What troubleshooting have you done?

Means that EX2300 from L3 switch mutates into L2 switch. And only way to solve this is rebooted, because you cant connect to switch over ssh, console, mgmt port, telnet. After reboot coredumps was empty. Create crase for JTAC - they are testet ex2300 switch with 19,1R1|R2|R3 3 month and dont see any problems. But only one of my switch ex2300 from 10s was workinhg normally on 19,1 - part number is same. Cant understand whats happening

Good day,

Is it switching off with overheat messsage?

This can be a problem of the "show" command. (Display issue) or can be real problem with FANs.

No, Temp is normal. Switch  just down. L2 domain is working- all other no. Cant view anything from console connection or management port. Only reboot work. On 18.2R1  and 18.2R2 all is ok.

We are seeing the same problem with our EX2300-C units since upgrading to 18.1R3-S4.2 (from 15.1X53-D590.1).  I have a case open and have just escalated since we haven't gotten anywhere in the last two weeks.  My fear is this hitting our production EX3400 virtual chassis installations.

We see the same as described... doesn't respond to management traffic and the console port is unresponive.  It also stops forwarding syslog traffic.  It really looks like the RE is "dead" but it keeps forwarding L2 traffic.

Hi,

I'm seeing the same behavior on an EX2300-24T running 18.3R2.7

Symptoms:
My laptop and a bunch of Juniper MX and QFX mgmt interfaces are connected to the switch in the same VLAN. After a while (few hours), new SSH sessions from my laptop to any other device connected on the switch in the same VLAN fails, while an existing SSH session keeps working until I disconnect.
The switch however is unresponsive for management: no output on the serial console port, and no SSH connection possible. Needed to pull the plug to reboot the switch to get it working again (check the serial console and ssh again, after the reboot they were fine). 
The funky part is that we have 2 EX2300 switches configured almost exactly the same way, one in each of 2 racks.

Did you get any feedback from JTAC regarding the matter?

Regards,
Dante

JTAC closed my case since they couldn't reproduce (having a switch idling).  We went back to 15.1X53-D591.1.

This is probably some obscure bug. When I cleaned up the config (my colleague left a lot of default stuff in it, it's a lab setup), the switch hasn't stalled on me for at least a week.

Hello RJTaylor, 

        we were about to deploy 250 EX2300-C in june but they started exhibiting the behavior you and other in that thread described: 

- no access SSH

-no ping

-no access console

-SPD/DX/EN switch on the right do not respond

-when disconnecting interface cable, leds keep blinking

-the only way to reset the switch was to disconnect and reconnect AC feed.

-seems the trafic on connected interfaces keep flowing. You can't connect a new interface, though. 

the issue affect at minimum image 19.1 and also 18.1R3 . The switches were delivered with 15.1 and we had upgraded them. 

Not sure if going back to 15.1 would resolve the issue.

pened a case in june and escalated it. We have had an image beta to try in july and no switch went zombie with it. Not a prod image yet, though. 

what are you up to ? 

Michel

---

@LapointeMichel wrote:

<snip>

- no access SSH

-no ping

-no access console

-SPD/DX/EN switch on the right do not respond

-when disconnecting interface cable, leds keep blinking

-the only way to reset the switch was to disconnect and reconnect AC feed.

-seems the trafic on connected interfaces keep flowing. You can't connect a new interface, though. 

</snip>

---

This describes the issue I'm seeing exactly. I even had the "privilege" to experience that an existing SSH session kept working when the issue started. Any new connection (physical or logical) fails. It seems the switch keeps working with as-is information, but isn't processing any new information, like connecting/disconnecting an interface, adding a new address in the mac or arp table, etc.

I've got a case running, but not much progress yet. The engineer might have found a related PR, and he's asking to open an new case 8-). Do you have a case number? I'd like to mention it in our case.

Regards,

Dante

Bonjour,

My pleasure:  Service Request 2019-0603-0600 "4 EX2300 Switches went into an unreachable state for no particular reason" was opened on June 3rd 2019. it is now high priority, and was moved to PR sometime in mid june. We were about to deploy them in 245 sites and while configuring them in lab, they kept goign down, roughly 1 /day . It has grown since then to 161 notes and 87 attached files. Highlight happenned on july 1st when I installed a beta image on 60, then 245 ex2300 and those never went into zombie state. I don't have any date for a production release, so we are now planning to go back to the 15.1X53.D591.1  which was released on may 17, 2019. I plan to install it on 100 switches today and wait for failures. I'll keep you posted. 

I tried this weekend 18.1R3S7.1 which was released mid august. I installed it on 109 switches and by monday, I had 4 zombies ! 

By the way, we had 2 switches in production going zombie (they were running 19.1 and 18.3) and trafic kept flowing with no issues from users. But once a switch is in that state, you can't connect a new interface.

---

Not sure if going back to 15.1 would resolve the issue.

pened a case in june and escalated it.

 Salut Michel,

I've been out of the office for a few weeks so I'm just seeing your message now.  I'm glad to hear that you have a working beta.  We're looking forward to its release to production.

To answer your question, we began getting watchdog/swizzle reboots with 15.1X53-D591.1.  We were told this should be fixed somewhere in 18.3R3 but it was unclear whether this includes the fixes for the zombie state we're seeing or just the swizzle reboot.

We're currently running 18.1R3-S6.1 and haven't had a zombie in three weeks.  That said, our sample size is *way* smaller than yours.

Bon mardi, 

before the holiday weekend, I finished downgrading roughly 130 switches to 15.1 591.1  No zombies yet, and that amount of switches was a gaarantee since june to have average one a day going zombie on images 18.n or 19,1 . I have the "chance" of having 245 switches plugged and available for test until I get a reliable image so I can deploy them and be sure I will not have to drive 90mns to unplug the AC to re-establish communication. Deployment has been on hold since june and I can tell you that it is not easy to squat unused spaces with 250 switches !!!

I am trying today to finish the other ones and if I can have 5 days with no zombies, then we'll keep this image. we just add this weekend a 19.1 going zombie and we plan to downgrade it when we are sure our configuration made on 19.1 is accepted by 15.1.

we are tryning this way since I can't get a production date on the test version we have.

I'll keep you posted on the success-failure of the test.  Question for you: I don't think I experienced the sizzle-reboot you mention. How does that happenned and what traces does it leaves ? I can access switches log, and also run Junos Space and Network Director.

---

@LapointeMichel wrote:

Question for you: I don't think I experienced the sizzle-reboot you mention. How does that happenned and what traces does it leaves?

---

We have not found a pattern causing the swizzle reboot.  It's a known bug but there doesn't seem to be anything in our environment that we could change to prevent it.

You can tell you've had a Swizzle (watchdog) reboot by checking "show chassis routing-engine".   The last reboot reason will read: "0x8000:Swizzle reboot".

---

You can tell you've had a Swizzle (watchdog) reboot by checking "show chassis routing-engine".   The last reboot reason will read: "0x8000:Swizzle reboot".

 

thanks RJTaylor. I'll keep an eye open. 

184 switches are running15.1X53 D591.1 (we call it "may 2019 release"). I'll finish the last ones tomorrow.  The minute one goes zombie, that's the end of the test for me. If you don't hear from me, then it means it's stil going. 5 straight days with no failures will reassure me. 

I am a bit concerned about how long Juniper will support 15.1.  But it's going to take a lot to convince me to upgrade.

by the way, if you upgrade , you may want to check an issue we had: no leds going up when connecting the optical interface connectors., even though traffic flows. 15.1 is ok. this is EX2300-C we are talking about.

 

 

---

Hi Michel,

Thanks a lot for sharing!

We have about 15 EX2300-24Ts that will be rolled out in an OOB setup for a customers new (Juniper) MPLS network, gradually during September-October. We've downgraded them all to 15.1X (latest) to avoid the "freezing" issue, which we have encountered several times in the lab setup. There are only 2 switches in that setup and we hit the issue multiple times per week before the downgrade, which we decided on end of last week. Unfortunately we don't have the option to postpone the roll-out and test more. If any show the issue after downgrading, I'll share it in this post. Hopefully they don't, in which case I'll be able to give an (positive) update at the end of the roll-out + some extra time, say mid November.

Regards,

Dante

---

If any show the issue after downgrading, I'll share it in this post.

The control plane doesn't lockup on 15.1 but you may have the occassional watchdog/swizzle reboot.

Hi rjtaylor,

 There is a major PR open and worked by Eng. This is affecting multiple 18 releases. Juniper is working to introduce a fix soon. The PR will be updated once the formal fix is published:

https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1442376

Hi egarro,

Thank you very much for providing the PR.  This is very helpful and gives us something we can watch to know when our defect is fixed!

Best regards,

RJ

---

Thank you very much for providing the PR.  This is very helpful and gives us something we can watch to know when our defect is fixed!

 

---

bonjour all, 

well, well, well, ...  I knew this number rang a bell ... here an excerpt from an email I received last june after opening my service request on the zombie switches  🙂

=============================================================================

Hi Michel,

I opened a PR for this issue. I will work with engineering and get back to you on this.

I’m also working on replicating the issue in JTAC lab. I will set up a lab and get back to you on this.

Here is the PR information for your reference.

1442376 2-CL2: EX2300:19.1R1.6 Switch is not responding to ssh/console and reboot fix the issue

=============================================================================

a first update was written yesterday. no release date yet for a fix. Like I wrote in an earlier post, I spent July testing successfully a beta image on my 250 switches and none went zombie.So there may be light at the end of the tunnel. 

To be clear: switch is out of reach, but traffic keeps going through.  based on this, we juggled with the idea of including a ip power bar connected to the switch to reboot it remotely but it was  laughed off ... 🙂

I switched all my switches back to 15.1X53 but we have pretty much given up on the idea  after experiencing the swizzle reboot. Good news on 15.1 was no zombie switches. Bad news was random swizzle reboot on 30/250 switches, including one in prod. Zombie switch is bad, but random reboot was unacceptable. The case I opened on this subject point to NETCONF request from Junos Space at the time of the reboot, so I am opening one with Junos Space. I'll do it this morning right after opening one on our failures to upgrade Junos Space from 19.1 to 19.3 

I plan to keep this post updated with any information I get from Juniper. 

Michel,

Hey there, Iam in the same boat.

I have 3 EX2300-48t (JunOS 18.1R3-S6.1) in production right now, which get randomly stuck into a zombie state and another 2 EX2300-48t (JunOS 15.1X53-D58.3) which reboot randomly (swizzle reboots).

Can anyone tell me which JunOS has the most stability for these devices? This is driving me crazy.

Best regards,

Julian V.

Bonjour Julian, and welcome to the club.

to make a long story short: I can't recommend anything other than a Beta Version !!!!!

I have an open Service request for swizzle reboot. I'll keep you posted if anything useful comes up. 

As of now, I can't recommend anything: swizzle reboot started affecting us when we decided to go back to 15.1X53. It mayhave existed before that, but it's only then we started notice it. We decided to go back to higher version, but we just had a swizzle reboot this weekend on a 2300 switch running 19.1.  I am scared.

I installed 19.3 (september release)  on some of my test switches and some of them fell in a zombie state. Same thing for 18.2R3S1.7. Some went zombie. 

At least trafic keeps on going in the zombie state. I have one now in production and it handles trafic pefectly. It is just out of reach. I am much more concerned about the swizzle reboot, especially since I had one on a 19.1 switch... I opened a SR on the swizzle reboot last week and the tech wrote back saying the switch behaved like this after answering NETCONF requests. Switch is ok. OK then, fine... I closed this one and opened another with Junos Space, I was told that all these request from JS are normal so it must be a switch behavior problem.  

I was told by our Juniper rep that a new 18.2R3-S2 should be release in october and may or may not solve the zombie issue. I'll test it as soon as I get it. 

I tested a beta  image in july and monitored 250 switches never going zombie for 2 months 🙂 I was extatic :- )   but this was a beta release. 😞 

so I am down to checking daily that PR1442376 will have a note saying the problem has been solved. I juste checked, and "Resolved In" row is still empty. In the meantime I have 250 switches enjoying fresh air in the server room, waiting for a stable image to be deployed. 

keep me posted if anything happens on your side. I'll keep this discussion running if anything comes up. 

Michel

Hello Michel,

thank you very much for the insight and all the much appreciated information you share!

Unfortunately I have no new meaningful information to share regarding this thematic. Currently Iam thinking of configuring some kind of cronjob so the switches will reboot themself in the night when there are no colleagues working. But this is definitely not my solution to this thematic, just some kind of workaround so the zombie state wont occur.

Nevertheless, thank you again Michel!

Best regards,

Julian V.

The built in Junos reboot command allows you to set a scheduled time directly.

https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/request-system-reboot.html

Don't mention  it Julian,  Glad I can help. 

actually we thought of another solution for the zombie state that you might like: since the trafic still goes through, we thought of installing the switch on a IP controlled powerbar, give it one of the 2300 port and then being able to  issue a command to the power bar to restart if the switch goes zombie. 

We discarded it as not feasible for our 250 switches but for a small installation, it might do the trick and buy time until Juniper comes up with something.

Michel

hello Dante et Rjtaylor

so as of today sep 4, 2019:   236 of my EX2300 are running on image 15.1X53 D591.1  (released in may 2019) . 100 have been running no problem for the last 4 days.  I have set the arbitrary target of 5 days with no zombies to call this a success. It's based on my previous experiences since last june. I'll update if a single one goes down. 

wish me luck !!! 

Michel

Hi Michel,

My fingers are crossed! 🙂

Regards,

Dante

Hi everyone,

FYI I have encountered 3 customers that have confirmed that when rolling back the software the issue didnt resurface.

also there seems to be 2 behaviors 1 when only the control plane is affected and the other one where forwarding traffic is also affected. 

---

@dante wrote:

Hi Michel,

My fingers are crossed! 🙂

Regards,

Dante

---

well, the fingers trick seems to work. No zombies on 236   2300-C  running 15.1X53-D591.1 as of friday afternoon. But I have now 2 zombies out of my 8 prod switches runnning on 19.1. Those are in production. Trafic still goes through, which is kind of a credit to Juniper's architecture,  but someone is going to have to go there and unplug the thing. I am planning to change all these next week for switches with 15.1... 

I have not finished testing, but so far the only config command I found that des not downgrade from 18 to 15 is  a  system processes jsd disable  that we tried on JTAC suggestion while investigating extremely low cpu idle associated with 18. and 19.  It did not have effect. Cpu idle swinged from below 50% to 0% , while the prod switches running 15 were always flat above 80% . so the switch to 15.1 may solve that issue too.  And we are going to find back the led display if I connect an optical link. Full/half duplex led still does not work but we'll live with that. 

I'll update the forum next week on how the weekend went. Have a nice one ,Dante, 

Michel

---

@LapointeMichel wrote:

hello Dante et Rjtaylor

so as of today sep 4, 2019:   236 of my EX2300 are running on image 15.1X53 D591.1  (released in may 2019) . 100 have been running no problem for the last 4 days.  I have set the arbitrary target of 5 days with no zombies to call this a success. It's based on my previous experiences since last june. I'll update if a single one goes down. 

---

===================================================================================================

hello again Dante, RJTaylor and all,

so here we are Monday sept 9: 236 EX2300-C have now been running for at least 5 days (for 130, more than 5 days) and not a single one of them has gone zombie on me. Since last june, more than 100 switch running anything else higher (18.1, 18.3, 19.1...) was a sure recipe for an average of one switch a day going down. "going down" meaning losing the ability to connect with the switch: no console, no SSH, impossible to connect anything else than what was already connected. By having some switches in production going down, we also found that trafic was still going through seemingly unaffected. The only way to get back control of the switch was disconnecting AC.

After sending numerous log files and RSI (Req Support Information)    files to JTAC, they said they could not find any indication of cause. I was supplied in early july with a beta Image that, when I installed it, seemed to solve the problem: not a single switch equipped with this image went down in 2 months testing. 

Since this image is not in production, no date for this is available, and even getting an official release means testing to make sure it works,  we plan to go back to 15.1X53-D591.1, which I understand is the latest version of the image the switches were delivered with last year. Next steps include  checking with Juniper how long they plan to support 15.1.  close the case I opened in june with JTAC, and make sure the QinQ  config we designed for the switches using images above 15 are still compatible through the downgrade. So far we found that only a jsd disable command does not downgrade (irrelevant to our purpose) and we have to remember that going above 15 may mean losing ssh root access if system ssh root login allow is not set (or something like that, I have to check my notes ...)

When we were about to deploy 250 switches, I thought it was a good idea to upgrade them to the latest release. I still think it was, but it wasn't ! when I want to scare myself, I think of what would have happened deploying the switches to 250 different sites last june and losing access to them one after the other. 

Please comment and suggest, but that case in now closed for me and I'll get on with my life 🙂 Thanks to Dante and RJTaylor to have opened and commented this issue on the forum

Michel Lapointe

Hello again, Michel,

Here is another thing to consider.  I have a vague recollection that after upgrading from 15.1 to 18.x, I was unable to perform recovery snapshotting.  The device did not have enough space (even after clean up).  In order to having working recovery snapshots, I ended up having to boot from USB and install 18.x that way instead of upgrading.

Just some food for thought.

RJ

---

I ended up having to boot from USB and install 18.x that way instead of upgrading.

 

 

---

with 250 swithes in test, I look forward to any image change as much as to a root-canal. I have been faced with the space issue too, and gave up on trying to figure it out. So the best trick I found was to make a bunch of usb with an old 15 version made with autoinstall, and use them in conjunction with a DHCP setup with Zero Touch provisionning. request system rebbot usb, than autoimage upgrade would take over, communicate with dhcp and download test image and a baseline configuration common to all switches. 

A funny  trick I was given by JTAC worked sometimes: instead of putting the new image in /var/tmp, I was putting it in /tmp and executing it from the with force no-copy unlink reboot options. 

image upgrade using Junos Space was also frustrating: I remember a number of jobs that failed, but the "retry on failed" option went through ! in the end I gave up and relied on usb keys and DHCP.

Michel

Hi RJ,

Just received the release notes for 18.2R3-S1 and I bumped into this fixed PR1439189: "The recovery snapshot cannot be created after system zeroize" and remembered you mentioned an issue resembling something like that.

Release notes: https://kb.juniper.net/InfoCenter/index?page=content&id=TSB17632&actp=METADATA

PR: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1439189 (login required)

Fixed in releases 

Regards,

Dante

Dante,

Thanks for posting that.  While we weren't using zeroize when we experienced the error, that is exactly the error we see.  I'll have to try an upgrade at some point and see whether we can get a snapshot after.

How about that change list in the release notes?!  So much work went into that release.  It's good to see this stuff occurring.  The promise of new hardware is alluring but the first couple years with a new platform can be painful.

RJ

Editing to add, I'm just realizing that the release notes are not EX-specific but cover several products.  Still impressive.

Bonjour Dante and all, 

good news is the switches still hold. so there is no going back to anything above 15.1 for now.

Less good is that  moving config made on 19.1 to 15.1 is not so smooth as I thought. 

here is one problem 

under 19. 1, this basic config was for inserting the switch in a bypass mode to check traffic for futur QinQ. installation. so trafic was just going unaffected between port 0 and port 11, with a logical interface to access the switch through vlan 3999. here are code snipets. 

  ge-0/0/0 {
        description BYPASS;
        vlan-tagging;
        unit 0 {
            family ethernet-switching {
                interface-mode trunk;
                vlan {
                    members Z-Legacy;
                }
====================================
 ge-0/0/11 {
        description GIRAT;
        vlan-tagging;
        unit 0 {
            family ethernet-switching {
                interface-mode trunk;
                vlan {
                    members Z-Legacy;
                }
            }
        }
        unit 3999 {
            vlan-id 3999;
            family ethernet-switching {
========================
    irb {
        unit 0 {
            family inet {
                dhcp {
                    vendor-id Juniper-ex2300-c-12t;
                }
            }
        }
        unit 3999 {
            family inet {
                address 10.10.10.13/23;
            }
        }
    }
==========================================
vlans {
    Management {
        vlan-id 3999;
        interface ge-0/0/11.3999;
        l3-interface irb.3999;
    }
    Z-Legacy {
        vlan-id-list [ 3000-3900 3949 ];
    }

pings go through between switches connected to port 0 and 11, and I ssh the switch on 3999. no problem.

I jts move the 2 cables to a switch running 15.1X53: pings stop, but access to ssh is still ok. I get the traffic back by removing the "vlan-tagging"  line, but I get the "Only unit 0 is valid for this encapsuation " on the ge-0/0/11 unit 3999 interface.  Any suggestion on how to have again my trunk on port 11 unit 0, and my unit 3999 to talk to my switch ?

suggestions are welcome.

Thanks,

Michel Lapointe

You may need to define all the member vlans in a single logical interface.  For example, on ge-0/0/11, delete unit 3999 and configure it all under unit 0 with:

 ge-0/0/11 {
        description GIRAT;
        vlan-tagging;
        unit 0 {
            family ethernet-switching {
                interface-mode trunk;
                vlan {
                    members [ Z-Legacy 3999 ];
                }
            }
        }

it works

pings get through. Still have to remove  vlan-tagging to get it to work 

Merci RJTaylor

---

@LapointeMichel wrote:

Still have to remove  vlan-tagging to get it to work 

---

Ah yes.  If I recall correctly, it'll commit without complaining but doesn't move traffic.  It appears that tagging is implied by using "members" and it all works as expected once you don't ask for tagging.  Go figure!