Junos OS

last person joined: 2 days ago 

Ask questions and share experiences about Junos OS.
Expand all | Collapse all

EX2300-48T Stop working after random time after upgrade to 19.1R1.6

rjtaylor

rjtaylor11-06-2019 05:00

LapointeMichel

LapointeMichel11-07-2019 04:56

LapointeMichel

LapointeMichel11-07-2019 12:02

LapointeMichel

LapointeMichel11-08-2019 12:58

LapointeMichel

LapointeMichel11-11-2019 05:52

LapointeMichel

LapointeMichel11-12-2019 04:54

LapointeMichel

LapointeMichel11-12-2019 13:00

LapointeMichel

LapointeMichel11-13-2019 05:10

LapointeMichel

LapointeMichel11-13-2019 19:07

  • 1.  EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 05-05-2019 06:24

    Have some trouble:

    After upgarade to 19.1R1.6 Junos on my EX2300-48 switches (have 10 switches) - i have some problem:

    1) 3 of 10 switches stop working after  a certain amount of time (one switch a day after the update, the second after a week) and only the reboot helps. Unfortunately, I do not see what is happening with it at the moment, core dumps are empty

    2) show chassis enviroment dont show second fan and show chassis alarms - no alarms - its software bug or its really second fan doesnt work? On 18.2R1.9 Junos second fan is visible.

    3) and of course jweb not work - but this problem was in release note document.ex2300-48t with Junos 18.2R1.9ex2300-48t with Junos 18.2R1.9ex2300-48T with Junos 19.1R1.6ex2300-48T with Junos 19.1R1.6

     


    #EX2300
    #JUNOS
    #Juniper
    #19.1R1.6
    #19.1
    #EX3300


  • 2.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

     
    Posted 05-05-2019 11:57

    Stops working meaning what? What troubleshooting have you done?



  • 3.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 01-04-2020 04:27

    Means that EX2300 from L3 switch mutates into L2 switch. And only way to solve this is rebooted, because you cant connect to switch over ssh, console, mgmt port, telnet. After reboot coredumps was empty. Create crase for JTAC - they are testet ex2300 switch with 19,1R1|R2|R3 3 month and dont see any problems. But only one of my switch ex2300 from 10s was workinhg normally on 19,1 - part number is same. Cant understand whats happening



  • 4.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 05-17-2019 04:02

    Good day,

     

    Is it switching off with overheat messsage?

    This can be a problem of the "show" command. (Display issue) or can be real problem with FANs.



  • 5.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 06-01-2019 04:20

    No, Temp is normal. Switch  just down. L2 domain is working- all other no. Cant view anything from console connection or management port. Only reboot work. On 18.2R1  and 18.2R2 all is ok.



  • 6.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 06-03-2019 06:32

    We are seeing the same problem with our EX2300-C units since upgrading to 18.1R3-S4.2 (from 15.1X53-D590.1).  I have a case open and have just escalated since we haven't gotten anywhere in the last two weeks.  My fear is this hitting our production EX3400 virtual chassis installations.

     

    We see the same as described... doesn't respond to management traffic and the console port is unresponive.  It also stops forwarding syslog traffic.  It really looks like the RE is "dead" but it keeps forwarding L2 traffic.



  • 7.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 07-16-2019 06:25

    Hi,

    I'm seeing the same behavior on an EX2300-24T running 18.3R2.7

    Symptoms:
    My laptop and a bunch of Juniper MX and QFX mgmt interfaces are connected to the switch in the same VLAN. After a while (few hours), new SSH sessions from my laptop to any other device connected on the switch in the same VLAN fails, while an existing SSH session keeps working until I disconnect.
    The switch however is unresponsive for management: no output on the serial console port, and no SSH connection possible. Needed to pull the plug to reboot the switch to get it working again (check the serial console and ssh again, after the reboot they were fine). 
    The funky part is that we have 2 EX2300 switches configured almost exactly the same way, one in each of 2 racks.

    Did you get any feedback from JTAC regarding the matter?

    Regards,
    Dante

     

     



  • 8.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 07-16-2019 06:31

    JTAC closed my case since they couldn't reproduce (having a switch idling).  We went back to 15.1X53-D591.1.



  • 9.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 07-28-2019 00:08

    This is probably some obscure bug. When I cleaned up the config (my colleague left a lot of default stuff in it, it's a lab setup), the switch hasn't stalled on me for at least a week.



  • 10.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 08-26-2019 11:28

    Hello RJTaylor, 

            we were about to deploy 250 EX2300-C in june but they started exhibiting the behavior you and other in that thread described: 

    - no access SSH

    -no ping

    -no access console

    -SPD/DX/EN switch on the right do not respond

    -when disconnecting interface cable, leds keep blinking

    -the only way to reset the switch was to disconnect and reconnect AC feed.

    -seems the trafic on connected interfaces keep flowing. You can't connect a new interface, though. 

    the issue affect at minimum image 19.1 and also 18.1R3 . The switches were delivered with 15.1 and we had upgraded them. 

    Not sure if going back to 15.1 would resolve the issue.

    pened a case in june and escalated it. We have had an image beta to try in july and no switch went zombie with it. Not a prod image yet, though. 

    what are you up to ? 

    Michel

     



  • 11.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 08-27-2019 00:34

    @LapointeMichel wrote:

    <snip>

    - no access SSH

    -no ping

    -no access console

    -SPD/DX/EN switch on the right do not respond

    -when disconnecting interface cable, leds keep blinking

    -the only way to reset the switch was to disconnect and reconnect AC feed.

    -seems the trafic on connected interfaces keep flowing. You can't connect a new interface, though. 

    </snip>


    This describes the issue I'm seeing exactly. I even had the "privilege" to experience that an existing SSH session kept working when the issue started. Any new connection (physical or logical) fails. It seems the switch keeps working with as-is information, but isn't processing any new information, like connecting/disconnecting an interface, adding a new address in the mac or arp table, etc.

    I've got a case running, but not much progress yet. The engineer might have found a related PR, and he's asking to open an new case 8-). Do you have a case number? I'd like to mention it in our case.

     

    Regards,

    Dante

     



  • 12.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 08-27-2019 05:03

    Bonjour,

    My pleasure:  Service Request 2019-0603-0600 "4 EX2300 Switches went into an unreachable state for no particular reason" was opened on June 3rd 2019. it is now high priority, and was moved to PR sometime in mid june. We were about to deploy them in 245 sites and while configuring them in lab, they kept goign down, roughly 1 /day . It has grown since then to 161 notes and 87 attached files. Highlight happenned on july 1st when I installed a beta image on 60, then 245 ex2300 and those never went into zombie state. I don't have any date for a production release, so we are now planning to go back to the 15.1X53.D591.1  which was released on may 17, 2019. I plan to install it on 100 switches today and wait for failures. I'll keep you posted. 

    I tried this weekend 18.1R3S7.1 which was released mid august. I installed it on 109 switches and by monday, I had 4 zombies ! 

    By the way, we had 2 switches in production going zombie (they were running 19.1 and 18.3) and trafic kept flowing with no issues from users. But once a switch is in that state, you can't connect a new interface.



  • 13.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-03-2019 05:19

    Not sure if going back to 15.1 would resolve the issue.

    pened a case in june and escalated it.

     Salut Michel,

     

    I've been out of the office for a few weeks so I'm just seeing your message now.  I'm glad to hear that you have a working beta.  We're looking forward to its release to production.

     

    To answer your question, we began getting watchdog/swizzle reboots with 15.1X53-D591.1.  We were told this should be fixed somewhere in 18.3R3 but it was unclear whether this includes the fixes for the zombie state we're seeing or just the swizzle reboot.

     

    We're currently running 18.1R3-S6.1 and haven't had a zombie in three weeks.  That said, our sample size is *way* smaller than yours.



  • 14.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-03-2019 11:04

    Bon mardi, 

    before the holiday weekend, I finished downgrading roughly 130 switches to 15.1 591.1  No zombies yet, and that amount of switches was a gaarantee since june to have average one a day going zombie on images 18.n or 19,1 . I have the "chance" of having 245 switches plugged and available for test until I get a reliable image so I can deploy them and be sure I will not have to drive 90mns to unplug the AC to re-establish communication. Deployment has been on hold since june and I can tell you that it is not easy to squat unused spaces with 250 switches !!!

    I am trying today to finish the other ones and if I can have 5 days with no zombies, then we'll keep this image. we just add this weekend a 19.1 going zombie and we plan to downgrade it when we are sure our configuration made on 19.1 is accepted by 15.1.

    we are tryning this way since I can't get a production date on the test version we have.

    I'll keep you posted on the success-failure of the test.  Question for you: I don't think I experienced the sizzle-reboot you mention. How does that happenned and what traces does it leaves ? I can access switches log, and also run Junos Space and Network Director.



  • 15.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-03-2019 11:20

    @LapointeMichel wrote:

    Question for you: I don't think I experienced the sizzle-reboot you mention. How does that happenned and what traces does it leaves?


    We have not found a pattern causing the swizzle reboot.  It's a known bug but there doesn't seem to be anything in our environment that we could change to prevent it.

     

    You can tell you've had a Swizzle (watchdog) reboot by checking "show chassis routing-engine".   The last reboot reason will read: "0x8000:Swizzle reboot".



  • 16.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-03-2019 12:49

    You can tell you've had a Swizzle (watchdog) reboot by checking "show chassis routing-engine".   The last reboot reason will read: "0x8000:Swizzle reboot".

     

    thanks RJTaylor. I'll keep an eye open. 

    184 switches are running15.1X53 D591.1 (we call it "may 2019 release"). I'll finish the last ones tomorrow.  The minute one goes zombie, that's the end of the test for me. If you don't hear from me, then it means it's stil going. 5 straight days with no failures will reassure me. 

    I am a bit concerned about how long Juniper will support 15.1.  But it's going to take a lot to convince me to upgrade.

    by the way, if you upgrade , you may want to check an issue we had: no leds going up when connecting the optical interface connectors., even though traffic flows. 15.1 is ok. this is EX2300-C we are talking about.

     

     


     



  • 17.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-03-2019 13:14

    Hi Michel,

    Thanks a lot for sharing!

    We have about 15 EX2300-24Ts that will be rolled out in an OOB setup for a customers new (Juniper) MPLS network, gradually during September-October. We've downgraded them all to 15.1X (latest) to avoid the "freezing" issue, which we have encountered several times in the lab setup. There are only 2 switches in that setup and we hit the issue multiple times per week before the downgrade, which we decided on end of last week. Unfortunately we don't have the option to postpone the roll-out and test more. If any show the issue after downgrading, I'll share it in this post. Hopefully they don't, in which case I'll be able to give an (positive) update at the end of the roll-out + some extra time, say mid November.

    Regards,

    Dante


    #freeze
    #EX2300


  • 18.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-04-2019 04:41

    If any show the issue after downgrading, I'll share it in this post.

    The control plane doesn't lockup on 15.1 but you may have the occassional watchdog/swizzle reboot.



  • 19.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

     
    Posted 10-03-2019 15:53

    Hi rjtaylor,

     

     There is a major PR open and worked by Eng. This is affecting multiple 18 releases. Juniper is working to introduce a fix soon. The PR will be updated once the formal fix is published:

     

    https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1442376



  • 20.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-04-2019 04:58

    Hi egarro,

     

    Thank you very much for providing the PR.  This is very helpful and gives us something we can watch to know when our defect is fixed!

     

    Best regards,

    RJ



  • 21.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-04-2019 06:14

    Thank you very much for providing the PR.  This is very helpful and gives us something we can watch to know when our defect is fixed!

     


    bonjour all, 

    well, well, well, ...  I knew this number rang a bell ... here an excerpt from an email I received last june after opening my service request on the zombie switches  🙂

    =============================================================================

    Hi Michel,

    I opened a PR for this issue. I will work with engineering and get back to you on this.

    I’m also working on replicating the issue in JTAC lab. I will set up a lab and get back to you on this.

    Here is the PR information for your reference.

    1442376 2-CL2: EX2300:19.1R1.6 Switch is not responding to ssh/console and reboot fix the issue

    =============================================================================

    a first update was written yesterday. no release date yet for a fix. Like I wrote in an earlier post, I spent July testing successfully a beta image on my 250 switches and none went zombie.So there may be light at the end of the tunnel. 

    To be clear: switch is out of reach, but traffic keeps going through.  based on this, we juggled with the idea of including a ip power bar connected to the switch to reboot it remotely but it was  laughed off ... 🙂

    I switched all my switches back to 15.1X53 but we have pretty much given up on the idea  after experiencing the swizzle reboot. Good news on 15.1 was no zombie switches. Bad news was random swizzle reboot on 30/250 switches, including one in prod. Zombie switch is bad, but random reboot was unacceptable. The case I opened on this subject point to NETCONF request from Junos Space at the time of the reboot, so I am opening one with Junos Space. I'll do it this morning right after opening one on our failures to upgrade Junos Space from 19.1 to 19.3 

    I plan to keep this post updated with any information I get from Juniper. 

    Michel,

     



  • 22.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-13-2019 23:43

    Hey there, Iam in the same boat.

     

    I have 3 EX2300-48t (JunOS 18.1R3-S6.1) in production right now, which get randomly stuck into a zombie state and another 2 EX2300-48t (JunOS 15.1X53-D58.3) which reboot randomly (swizzle reboots).

     

    Can anyone tell me which JunOS has the most stability for these devices? This is driving me crazy.

     

    Best regards,

    Julian V.



  • 23.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-16-2019 06:02

    Bonjour Julian, and welcome to the club.

    to make a long story short: I can't recommend anything other than a Beta Version !!!!!

    I have an open Service request for swizzle reboot. I'll keep you posted if anything useful comes up. 

    As of now, I can't recommend anything: swizzle reboot started affecting us when we decided to go back to 15.1X53. It mayhave existed before that, but it's only then we started notice it. We decided to go back to higher version, but we just had a swizzle reboot this weekend on a 2300 switch running 19.1.  I am scared.

    I installed 19.3 (september release)  on some of my test switches and some of them fell in a zombie state. Same thing for 18.2R3S1.7. Some went zombie. 

    At least trafic keeps on going in the zombie state. I have one now in production and it handles trafic pefectly. It is just out of reach. I am much more concerned about the swizzle reboot, especially since I had one on a 19.1 switch... I opened a SR on the swizzle reboot last week and the tech wrote back saying the switch behaved like this after answering NETCONF requests. Switch is ok. OK then, fine... I closed this one and opened another with Junos Space, I was told that all these request from JS are normal so it must be a switch behavior problem.  

    I was told by our Juniper rep that a new 18.2R3-S2 should be release in october and may or may not solve the zombie issue. I'll test it as soon as I get it. 

    I tested a beta  image in july and monitored 250 switches never going zombie for 2 months 🙂 I was extatic :- )   but this was a beta release. 😞 

    so I am down to checking daily that PR1442376 will have a note saying the problem has been solved. I juste checked, and "Resolved In" row is still empty. In the meantime I have 250 switches enjoying fresh air in the server room, waiting for a stable image to be deployed. 

    keep me posted if anything happens on your side. I'll keep this discussion running if anything comes up. 

    Michel

     

     

     



  • 24.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-16-2019 23:47

    Hello Michel,

     

    thank you very much for the insight and all the much appreciated information you share!

     

    Unfortunately I have no new meaningful information to share regarding this thematic. Currently Iam thinking of configuring some kind of cronjob so the switches will reboot themself in the night when there are no colleagues working. But this is definitely not my solution to this thematic, just some kind of workaround so the zombie state wont occur.

     

    Nevertheless, thank you again Michel!

     

    Best regards,

    Julian V.



  • 25.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-17-2019 03:02

    The built in Junos reboot command allows you to set a scheduled time directly.

     

    https://www.juniper.net/documentation/en_US/junos/topics/reference/command-summary/request-system-reboot.html

     



  • 26.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-17-2019 07:07

    Don't mention  it Julian,  Glad I can help. 

    actually we thought of another solution for the zombie state that you might like: since the trafic still goes through, we thought of installing the switch on a IP controlled powerbar, give it one of the 2300 port and then being able to  issue a command to the power bar to restart if the switch goes zombie. 

    We discarded it as not feasible for our 250 switches but for a small installation, it might do the trick and buy time until Juniper comes up with something.

    Michel

     



  • 27.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-04-2019 10:43

    hello Dante et Rjtaylor

    so as of today sep 4, 2019:   236 of my EX2300 are running on image 15.1X53 D591.1  (released in may 2019) . 100 have been running no problem for the last 4 days.  I have set the arbitrary target of 5 days with no zombies to call this a success. It's based on my previous experiences since last june. I'll update if a single one goes down. 

    wish me luck !!! 

    Michel

     



  • 28.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-04-2019 12:56

    Hi Michel,

    My fingers are crossed! 🙂

    Regards,

    Dante



  • 29.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-04-2019 13:19

    Hi everyone,

     

    FYI I have encountered 3 customers that have confirmed that when rolling back the software the issue didnt resurface.

     

    also there seems to be 2 behaviors 1 when only the control plane is affected and the other one where forwarding traffic is also affected. 

     



  • 30.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-06-2019 09:47

    @dante wrote:

    Hi Michel,

    My fingers are crossed! 🙂

    Regards,

    Dante


    well, the fingers trick seems to work. No zombies on 236   2300-C  running 15.1X53-D591.1 as of friday afternoon. But I have now 2 zombies out of my 8 prod switches runnning on 19.1. Those are in production. Trafic still goes through, which is kind of a credit to Juniper's architecture,  but someone is going to have to go there and unplug the thing. I am planning to change all these next week for switches with 15.1... 

    I have not finished testing, but so far the only config command I found that des not downgrade from 18 to 15 is  a  system processes jsd disable  that we tried on JTAC suggestion while investigating extremely low cpu idle associated with 18. and 19.  It did not have effect. Cpu idle swinged from below 50% to 0% , while the prod switches running 15 were always flat above 80% . so the switch to 15.1 may solve that issue too.  And we are going to find back the led display if I connect an optical link. Full/half duplex led still does not work but we'll live with that. 

    I'll update the forum next week on how the weekend went. Have a nice one ,Dante, 

    Michel



  • 31.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-09-2019 05:12

    @LapointeMichel wrote:

    hello Dante et Rjtaylor

    so as of today sep 4, 2019:   236 of my EX2300 are running on image 15.1X53 D591.1  (released in may 2019) . 100 have been running no problem for the last 4 days.  I have set the arbitrary target of 5 days with no zombies to call this a success. It's based on my previous experiences since last june. I'll update if a single one goes down. 


    ===================================================================================================

    hello again Dante, RJTaylor and all,

    so here we are Monday sept 9: 236 EX2300-C have now been running for at least 5 days (for 130, more than 5 days) and not a single one of them has gone zombie on me. Since last june, more than 100 switch running anything else higher (18.1, 18.3, 19.1...) was a sure recipe for an average of one switch a day going down. "going down" meaning losing the ability to connect with the switch: no console, no SSH, impossible to connect anything else than what was already connected. By having some switches in production going down, we also found that trafic was still going through seemingly unaffected. The only way to get back control of the switch was disconnecting AC.

    After sending numerous log files and RSI (Req Support Information)    files to JTAC, they said they could not find any indication of cause. I was supplied in early july with a beta Image that, when I installed it, seemed to solve the problem: not a single switch equipped with this image went down in 2 months testing. 

    Since this image is not in production, no date for this is available, and even getting an official release means testing to make sure it works,  we plan to go back to 15.1X53-D591.1, which I understand is the latest version of the image the switches were delivered with last year. Next steps include  checking with Juniper how long they plan to support 15.1.  close the case I opened in june with JTAC, and make sure the QinQ  config we designed for the switches using images above 15 are still compatible through the downgrade. So far we found that only a jsd disable command does not downgrade (irrelevant to our purpose) and we have to remember that going above 15 may mean losing ssh root access if system ssh root login allow is not set (or something like that, I have to check my notes ...)

    When we were about to deploy 250 switches, I thought it was a good idea to upgrade them to the latest release. I still think it was, but it wasn't ! when I want to scare myself, I think of what would have happened deploying the switches to 250 different sites last june and losing access to them one after the other. 

    Please comment and suggest, but that case in now closed for me and I'll get on with my life 🙂 Thanks to Dante and RJTaylor to have opened and commented this issue on the forum

    Michel Lapointe

     



  • 32.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-09-2019 05:58

    Hello again, Michel,

     

    Here is another thing to consider.  I have a vague recollection that after upgrading from 15.1 to 18.x, I was unable to perform recovery snapshotting.  The device did not have enough space (even after clean up).  In order to having working recovery snapshots, I ended up having to boot from USB and install 18.x that way instead of upgrading.

     

    Just some food for thought.

     

     

    RJ



  • 33.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-09-2019 06:37

    I ended up having to boot from USB and install 18.x that way instead of upgrading.

     

     


    with 250 swithes in test, I look forward to any image change as much as to a root-canal. I have been faced with the space issue too, and gave up on trying to figure it out. So the best trick I found was to make a bunch of usb with an old 15 version made with autoinstall, and use them in conjunction with a DHCP setup with Zero Touch provisionning. request system rebbot usb, than autoimage upgrade would take over, communicate with dhcp and download test image and a baseline configuration common to all switches. 

    A funny  trick I was given by JTAC worked sometimes: instead of putting the new image in /var/tmp, I was putting it in /tmp and executing it from the with force no-copy unlink reboot options. 

    image upgrade using Junos Space was also frustrating: I remember a number of jobs that failed, but the "retry on failed" option went through ! in the end I gave up and relied on usb keys and DHCP.

    Michel



  • 34.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-09-2019 23:58

    Hi RJ,

     

    Just received the release notes for 18.2R3-S1 and I bumped into this fixed PR1439189: "The recovery snapshot cannot be created after system zeroize" and remembered you mentioned an issue resembling something like that.

    Release notes: https://kb.juniper.net/InfoCenter/index?page=content&id=TSB17632&actp=METADATA

    PR: https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1439189 (login required)

    Fixed in releases 

    18.4R3
    18.2R3-S1
    19.2R2
    19.1R2

     

    Regards,

    Dante



  • 35.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-10-2019 06:05

    Dante,

     

    Thanks for posting that.  While we weren't using zeroize when we experienced the error, that is exactly the error we see.  I'll have to try an upgrade at some point and see whether we can get a snapshot after.

     

    How about that change list in the release notes?!  So much work went into that release.  It's good to see this stuff occurring.  The promise of new hardware is alluring but the first couple years with a new platform can be painful.

     

    RJ

     

    Editing to add, I'm just realizing that the release notes are not EX-specific but cover several products.  Still impressive.



  • 36.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-12-2019 11:58

    Bonjour Dante and all, 

    good news is the switches still hold. so there is no going back to anything above 15.1 for now.

    Less good is that  moving config made on 19.1 to 15.1 is not so smooth as I thought. 

    here is one problem 

    under 19. 1, this basic config was for inserting the switch in a bypass mode to check traffic for futur QinQ. installation. so trafic was just going unaffected between port 0 and port 11, with a logical interface to access the switch through vlan 3999. here are code snipets. 

     

      ge-0/0/0 {
            description BYPASS;
            vlan-tagging;
            unit 0 {
                family ethernet-switching {
                    interface-mode trunk;
                    vlan {
                        members Z-Legacy;
                    }
    ====================================
     ge-0/0/11 {
            description GIRAT;
            vlan-tagging;
            unit 0 {
                family ethernet-switching {
                    interface-mode trunk;
                    vlan {
                        members Z-Legacy;
                    }
                }
            }
            unit 3999 {
                vlan-id 3999;
                family ethernet-switching {
    ========================
        irb {
            unit 0 {
                family inet {
                    dhcp {
                        vendor-id Juniper-ex2300-c-12t;
                    }
                }
            }
            unit 3999 {
                family inet {
                    address 10.10.10.13/23;
                }
            }
        }
    ==========================================
    vlans {
        Management {
            vlan-id 3999;
            interface ge-0/0/11.3999;
            l3-interface irb.3999;
        }
        Z-Legacy {
            vlan-id-list [ 3000-3900 3949 ];
        }
    
    

    pings go through between switches connected to port 0 and 11, and I ssh the switch on 3999. no problem.

    I jts move the 2 cables to a switch running 15.1X53: pings stop, but access to ssh is still ok. I get the traffic back by removing the "vlan-tagging"  line, but I get the "Only unit 0 is valid for this encapsuation " on the ge-0/0/11 unit 3999 interface.  Any suggestion on how to have again my trunk on port 11 unit 0, and my unit 3999 to talk to my switch ?

    suggestions are welcome.

    Thanks,

    Michel Lapointe

     



  • 37.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-12-2019 12:09

    You may need to define all the member vlans in a single logical interface.  For example, on ge-0/0/11, delete unit 3999 and configure it all under unit 0 with:

     ge-0/0/11 {
            description GIRAT;
            vlan-tagging;
            unit 0 {
                family ethernet-switching {
                    interface-mode trunk;
                    vlan {
                        members [ Z-Legacy 3999 ];
                    }
                }
            }

     



  • 38.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-12-2019 12:23

    it works

    pings get through. Still have to remove  vlan-tagging to get it to work 

    Merci RJTaylor

     



  • 39.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-12-2019 12:34

    @LapointeMichel wrote:

    Still have to remove  vlan-tagging to get it to work 


    Ah yes.  If I recall correctly, it'll commit without complaining but doesn't move traffic.  It appears that tagging is implied by using "members" and it all works as expected once you don't ask for tagging.  Go figure!



  • 40.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 01-04-2020 05:06

    As i know on 15.1 software you cant use irb interface in your config. on 15.1 you need use l3-interface vlan.3999



  • 41.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

     
    Posted 02-06-2020 09:06

    Hi,

     

    I also got hit by ex2300 zombi state where 2 48 port switch in virtual chassis, Only power cycle reboot solved the issue I am still in Junos: 18.2R3-S2.9

    fpc0:
    --------------------------------------------------------------------------
    Hostname: kannurcore
    Model: ex2300-48t
    Junos: 18.2R3-S2.9
    JUNOS OS Kernel 32-bit [20191022.14c2ad5_builder_stable_11]
    JUNOS OS libs [20191022.14c2ad5_builder_stable_11]
    JUNOS OS runtime [20191022.14c2ad5_builder_stable_11]
    JUNOS OS time zone information [20191022.14c2ad5_builder_stable_11]
    JUNOS py extensions [20191115.190104_builder_junos_182_r3_s2]
    JUNOS py base [20191115.190104_builder_junos_182_r3_s2]
    JUNOS OS crypto [20191022.14c2ad5_builder_stable_11]
    JUNOS network stack and utilities [20191115.190104_builder_junos_182_r3_s2]
    JUNOS libs [20191115.190104_builder_junos_182_r3_s2]
    JUNOS runtime [20191115.190104_builder_junos_182_r3_s2]
    JUNOS Web Management Platform Package [20191115.190104_builder_junos_182_r3_s2]
    JUNOS ex libs [20191115.190104_builder_junos_182_r3_s2]
    JUNOS ex runtime [20191115.190104_builder_junos_182_r3_s2]
    JUNOS ex platform support [20191115.190104_builder_junos_182_r3_s2]
    JUNOS dcp network modules [20191115.190104_builder_junos_182_r3_s2]
    JUNOS modules [20191115.190104_builder_junos_182_r3_s2]
    JUNOS ex modules [20191115.190104_builder_junos_182_r3_s2]
    JUNOS ex Data Plane Crypto Support [20191115.190104_builder_junos_182_r3_s2]
    JUNOS daemons [20191115.190104_builder_junos_182_r3_s2]
    JUNOS SDN Software Suite [20191115.190104_builder_junos_182_r3_s2]
    JUNOS Extension Toolkit [20191115.190104_builder_junos_182_r3_s2]
    JUNOS Phone-home [20191115.190104_builder_junos_182_r3_s2]
    JUNOS Packet Forwarding Engine Support (EX34XX) [20191115.190104_builder_junos_182_r3_s2]
    JUNOS jdocs ex [20191115.190104_builder_junos_182_r3_s2]
    JUNOS jail runtime [20191022.14c2ad5_builder_stable_11]
    JUNOS FIPS mode utilities [20191115.190104_builder_junos_182_r3_s2]

    fpc1:
    --------------------------------------------------------------------------
    Hostname: kannurcore
    Model: ex2300-48t
    Junos: 18.2R3.4
    JUNOS OS Kernel 32-bit [20190605.30b921f_builder_stable_11]
    JUNOS OS libs [20190605.30b921f_builder_stable_11]
    JUNOS OS runtime [20190605.30b921f_builder_stable_11]
    JUNOS OS time zone information [20190605.30b921f_builder_stable_11]
    JUNOS py extensions [20190620.215446_builder_junos_182_r3]
    JUNOS py base [20190620.215446_builder_junos_182_r3]
    JUNOS OS crypto [20190605.30b921f_builder_stable_11]
    JUNOS network stack and utilities [20190620.215446_builder_junos_182_r3]
    JUNOS libs [20190620.215446_builder_junos_182_r3]
    JUNOS runtime [20190620.215446_builder_junos_182_r3]
    JUNOS Web Management Platform Package [20190620.215446_builder_junos_182_r3]
    JUNOS Web Management Application package [18.2A1]
    JUNOS ex libs [20190620.215446_builder_junos_182_r3]
    JUNOS ex runtime [20190620.215446_builder_junos_182_r3]
    JUNOS ex platform support [20190620.215446_builder_junos_182_r3]
    JUNOS dcp network modules [20190620.215446_builder_junos_182_r3]
    JUNOS modules [20190620.215446_builder_junos_182_r3]
    JUNOS ex modules [20190620.215446_builder_junos_182_r3]
    JUNOS ex Data Plane Crypto Support [20190620.215446_builder_junos_182_r3]
    JUNOS daemons [20190620.215446_builder_junos_182_r3]
    JUNOS SDN Software Suite [20190620.215446_builder_junos_182_r3]
    JUNOS Extension Toolkit [20190620.215446_builder_junos_182_r3]
    JUNOS Phone-home [20190620.215446_builder_junos_182_r3]
    JUNOS Packet Forwarding Engine Support (EX34XX) [20190620.215446_builder_junos_182_r3]
    JUNOS jdocs ex [20190620.215446_builder_junos_182_r3]
    JUNOS jail runtime [20190605.30b921f_builder_stable_11]
    JUNOS FIPS mode utilities [20190620.215446_builder_junos_182_r3]

     

     

    and the curious thing is one switch automatically rolled back to problematic version Junos: 18.2R3.4 can you any suggest me the 15.1 software train where this issue is not there, I can't take the risk I narrowly escaped from production outage.



  • 42.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 02-07-2020 11:03

    hi, your problem its not because your build virtual chassis. its freebsd kernel error in some version wich results is memory leak and your switch mutate to stupid L2.
    Restore your version to 18.1R1/18.2R1 or from PR and check results. best regards



  • 43.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 08-28-2020 00:05

    In my case, even though I am using 18.2R3-S2, there is still happening like a stop working some of switch. Is there any other cause or solution?



  • 44.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-09-2019 06:19

    Hi Michel,

    FYI: we're currently using the same version (15.1X53-D591), without an issue so far (on a limited number of switches though).

    As for the root-login, the default behavior has changed since 17.4R1 on MX routers from implicit default allow to deny. The same has been applied to other Junos devices running 17.4R1 and above, from what I can tell.

    https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/root-login-edit-system.html

    In any case, it's a good practice to define login classes/users (or use central authentication through RADIUS or TACACS+) and not to use/rely on root to log in remotely ;-).

    Regards,

    Dante



  • 45.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-30-2019 12:39

    @rjtaylor wrote:

    @LapointeMichel wrote:

    Question for you: I don't think I experienced the sizzle-reboot you mention. How does that happenned and what traces does it leaves?


     


    well, it was just a matter of waiting long enough:  we had one this morning at 6...   I don,t remember random reboot when we were struggling with 18.1 and 19.1.  So it seems I have to chose between random reboot or zombie state !    

    do you experience this issue very often ????

    thanks 

    Michel



  • 46.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-01-2019 04:59

    @LapointeMichel wrote:

    So it seems I have to chose between random reboot or zombie state !   do you experience this issue very often ????

     

    So you got swizzled, huh?  That's the watchdog being overzealous (I guess).  And you are correct in that you get to pick from reboots or zombies.  We saw the 15.1 reboot frequently enough that we took what JTAC considered our best choice, which was 18.1R3-S6.1.

     

    Unlike you, I only have a handful of the EX2300 but we've been running without crash or reboot since we installed 18.1R3-S6.1.  Uptime:

    fpc0:
    --------------------------------------------------------------------------
    Current time: 2019-10-01 07:54:04 EDT
    Time Source:  NTP CLOCK
    System booted: 2019-08-20 14:39:12 EDT (5w6d 17:14 ago)
    Protocols started: 2019-08-20 14:43:37 EDT (5w6d 17:10 ago)
    Last configured: 2019-08-12 12:51:56 EDT (7w0d 19:02 ago) by root
     7:54AM  up 41 days, 17:15, 1 users, load averages: 0.99, 0.86, 0.76
    

     



  • 47.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-01-2019 13:01

    @rjtaylor wrote:

     

    So you got swizzled, huh?  That's the watchdog being overzealous (I guess).  And you are correct in that you get to pick from reboots or zombies.

    hello ,

       the whole thing unfolded when by sheer luck, one of our 8 production switches rebooted at 6am yesterday. After determining that it was a swizzle thing, I checked my other switches and out of 236, I had 30 that had a sizzle reboot. So I now have to choose between an occasionnal reboot, that the client is sure to feel, or a zombie state that let the traffic goe through with only me knowing it ...    I just talked this morning to Juniper and an image 18.2R3-S2 should be available in october that MAY solve the zombie thing. .. or not.



  • 48.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-02-2019 05:32

    @LapointeMichel wrote:
    I just talked this morning to Juniper and an image 18.2R3-S2 should be available in october that MAY solve the zombie thing.

     

    Cool, maybe our switches will work in October... or maybe not.  We were told to look for 18.3R3 but not given any release estimate.



  • 49.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-22-2019 10:08

    hello rjtaylor, 

        I have started to monitor my fine collection of ex2300  switches for swizzle reboot issues. Out of my 250, I have 24 that experienced the swizzle reboot. What I am concerned with is that 2 of them were running images higher than 15.1.  One was running 19.1, the other 19.3.  I installed 19.3 on a couple of switches for fun and one of them went Swizzle. 

    Any thoughts on the subject ?

    we have decided to move out of 15.1 and wait for the fix PR1442376. configuration was tricky to adapt, and we had more than 20 of the switches with 15.1 going up swizzle. 

    thanks, 

    Michel



  • 50.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-22-2019 10:53

    Hi Michel,

     

    So 19.x gets swizzle too, huh?  I only played with 18.2 since that was JTAC's recommendation (along with 15.1).  I didn't see the swizzle with 18.2 but you might if you had enough units running it.  We haven't heard anything from Juniper whether 18.2 is affected with that.

     

    We're still at 18.1R3-S6.1 but ready to test -S7 if that should zombie.  That's been least painful for us but we're really hoping that 18.2R3-S2 with the fix for PR1442376 shows up soon.

     

    Yours in swizzle mayhem,

    RJ



  • 51.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-25-2019 07:38

    I installed 18.3R2-S2.1 on 236 switches (EX2300-C) in test setup this wedsneday oct 23

    in 48 hours ...

    - 6 went zombie (PR1442376): no surprise there

    - 2 went swizzle reboot.: I am a bit disappointed

    I am more concerned about the swizzle reboot issue now. I have a case opened with Juniper. 

    I juggled with the idea of reinstalling the test image I had in July to see if it had any effect on the swizzle. 

    Michel



  • 52.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-28-2019 05:48

    Pursuing the Swizzle Reboot issue, I may (or not...) have stumbled into something ...

    last friday, I went to all my 236 switches and put them in "Monitor start message ", just to have a look at the messages being issued in real time. I found that 20 of them were vomiting the same stuff every second ...

    Oct 28 08:41:41  cslt-bearn-01-BYPASS init: getty repeating too quickly on port /dev/ttyu1, sleeping 30 secs
    Oct 28 08:42:12  cslt-bearn-01-BYPASS getty[48492]: tcsetattr /dev/ttyu1: Invalid argument
    Oct 28 08:42:12  cslt-bearn-01-BYPASS getty[48671]: tcsetattr /dev/ttyu1: Invalid argument
    Oct 28 08:42:12  cslt-bearn-01-BYPASS getty[48674]: tcsetattr /dev/ttyu1: Invalid argument
    Oct 28 08:42:13  cslt-bearn-01-BYPASS getty[48679]: tcsetattr /dev/ttyu1: Invalid argument
    Oct 28 08:42:13  cslt-bearn-01-BYPASS getty[48680]: tcsetattr /dev/ttyu1: Invalid argument
    Oct 28 08:42:13  cslt-bearn-01-BYPASS init: getty repeating too quickly on port /dev/ttyu1, sleeping 30 secs

    and this morning, 2 had gone in swizzle reboot during the weekend. 

    when I mentionned it to JTAC (and to this forum, in another post "this can not be good ...") I was directed to poor console cable connection. Except no console connection exist on any switch ! only port 11 is ethernet connected.   

    What causes this ? 

    is this a symptom of a coming swizzle reboot ? 

    keep you posted !

    Michel

     



  • 53.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-28-2019 05:29

    Hello rjtaylor and all, 

    Good news is PR 1442376 ("The zombie switch PR") was updated yesterday. However ...

    1. suggested workaround is 
      Downgrade Junos OS to a version with a kernel build before 2019-02-12.
      how can I find wich images this statement describes ?
    2. for your information, problem item 3 indicates that traffic is not going through. I beg to differ. we had at least 3 production switches going zombie on us for days without any trafic reported by customers.  1 case may be related to slow traffic, but the jury is still out on this one.  This pretty much is the list of symproms I sent when opening the case last june, when I stated by mistake that trafic was not going through.  I left a not on the PR. 
    3. in triggers section, 19.3 is not included. It should. I tried.

    still checking daily for a resolve before deploying my switches. 

    Michel



  • 54.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-28-2019 05:46

    @LapointeMichel wrote:

    Hello rjtaylor and all, 

    Good news is PR 1442376 ("The zombie switch PR") was updated yesterday. However ...

    1. suggested workaround is 
      Downgrade Junos OS to a version with a kernel build before 2019-02-12.

    The file date on 15.1X53-D590 is 21Nov2018 but I'm guessing it suffers from swizzle reboot?

     

    It's funny that you wrote this morning, I came in to a zombied lab switch running 18.1R3-S6.  Guess we'll go to 18.1R3-S7 while we wait for 18.2R3-S2.



  • 55.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-29-2019 05:48

    well welll well,....

    PR1442376 was updated this morning and they show 2 images in the "resolved in" item: 18.4R3 et 19.2R2

     

    I just checked the download page and can't see them. 

    Can't wait to try them. I'll keep you posted. 

    Michel

     



  • 56.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-31-2019 06:28

    Bonjour LapointeMichel!

    I'm eagerly waiting for ANY feedback.

    Also, does anyone have any clue/idea/resource/someone to ask - what the hell was with this EX2300s?!



  • 57.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-31-2019 13:05

    Hello Ivu,  everyone,

     

    As of now the issue is well identified however we cannot share the specific details, I can assure you though the fixed versions have been extensively tested internally many of the ones that have replied to this thread have been closely related to the PR.

     

    We appreciate the responses and constant feedback on this thread, we also are eagerly waiting for the versions to be released so you guys can deploy them and give us the "real world" feedback that only you, our contributors can provide.

     

    -CC

     



  • 58.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-31-2019 10:06

    bonjour rjtaylor, 

    my 236 switches now have a ping sensor on PRTG. this way,  I see a zombie switche as 1 red email , and a swizzle reboot switch as 2 emails (a red and a green) 5 minutes apart. Convenient. 

    While waiting for J to make available the 19.2R2 or 18.4R3 images, I have made some interesting progress on the swizzle reboot issue. I may have a band-aid. Not a fix, but a very good bandaid

    I started a thread called "this can not be good..." about some of my 236 test switches vomiting getty requests on ttuy1(miniusb console port) . To make a very  long story short, after being told by J that this was related to bad console connection and answering back that none of those switches have console connection (!!!) , we decided to simply disable the miniusb port. simple way to do it is set system auxiliray port-type rj45 disable. why rj45 ? does not matter since it seems the auxilirary is the miniusb console port . whatever you do on auxiliary only affect the miniusb console/auxiliary/whatever port .

    Now have I set this on my 236 switches after identifying 15 that were sick. I did it tuesday and monitor them daily like a doctor: 

    • no vomiting getty requests symptom
    • if I reactivate miniusb, symptom comes back on the 15 switches, plus the sickness developped on others (1, so far) 
    • symptom stop when miniusb disabled
    • but more importantly: no swizzle reboot for the pas 2 days .

    Monitoring is done by accessing the switches, enabling messages by monitor start messages and sipping coffee while running a script on SecureCRT showing the sessions one by one every second .

    . now I will not stick my head out and say it's fixed, but the coincidence is troubling, expecially after witnessing sick switches doing swizzle reboot and then being cured of vomiting getty request. The quantity of switches I happen to have in test makes the difference in seeing a potential pattern here. 

    your thoughts ? 

    I check HOURLY for the release of the promised fix for PT1442376. I'll try it as soon as it is available and keep you posted. 

    Michel

     



  • 59.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-31-2019 10:41

    @LapointeMichel wrote:

    set system auxiliray port-type rj45 disable.

    [snip]

    • no vomiting getty requests symptom
    • if I reactivate miniusb, symptom comes back on the 15 switches, plus the sickness developped on others (1, so far) 
    • symptom stop when miniusb disabled
    • but more importantly: no swizzle reboot for the pas 2 days .

     

    This is really interesting!  If it also cures swizzle on 15.1, one could run that somewhat reliably.  Have you witnessed 15.1 going zombie?  I haven't it.  All I saw was swizzle.

     

    I've been watching for the new software too.  Let's hope relief is coming!

     

     

    RJ



  • 60.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-31-2019 11:42

    Have you witnessed 15.1 going zombie?  I haven't it.  All I saw was swizzle.

     


    I went 15.1X53 in early september as a fix to zombie problem and it worked. I dont have a record of 15.1 switch going zombie, just the happy memory of not having to reboot a switch manually for a month. I was so happy. but after a month, we gave it up for 2 reasons

    1.  problem changing our config files designed under 18 and 19. Ithe config file were more complicated. 
    2. the apparition of swizzle reboot

    to be fair, I never monitored swizzle reboot before and we probably had some switches doing it. The cat came out of the bag when we downgraded some production switches  to 15.1 and one of them swizzle reboot at 6:00 AM.  so the light at the end of the tunnel turned out to be a train, literally. 

    so if they fix the zombie issue, and if I band aid the swizzle reboot, I will be able to deploy using a18.4R3 or 19.2R2. 

    on my SR on the miniusb thing, I have just been informed that they may revive a closed PR since it looks definitely like there is a link between getty request and swizzle reboot they were not able to duplicate. Seems my lab is bigger than theirs 🙂 

    On your side, though, if you stick to 15 and the disable mini-usb trick works, by all means try it. But they  announce end of support for 15 for sept 2020, 

    Michel



  • 61.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-01-2019 08:09

    hello rjtaylor, 

     

    day 3 of no swizzle on my switches. 

    roughly 2 switches/day goes zombie, but no swizzle. 

     

    Since the miniusb port is disable, I don't see any messages when i monitor start messages. If I turn the miniusb on again, I see the getty request from the switches that werre sick before, and also a new one adding everyday. I then disable again the miniusb port and everything looks cool. 

     

    next verification will take place monday.

     

    by the way, still nothing on the image supppose to fix PR1442376.

    Michel

     



  • 62.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-01-2019 08:15

    Salut Michel!

     

    This latest update of yours really has me considering going back to 15.1 along with your miniusb fix.  Then we could just wait for Juniper's releases.  Maybe we'll give it another week to see if they release.

     

     

    RJ



  • 63.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-01-2019 09:23

    This latest update of yours really has me considering going back to 15.1 along with your miniusb fix. 

     

    if there is still no swizzle next monday, we are considering turning the miniusb back on again and wait for a swizzle .On my sample of 236  switches, that should take no more than 48 jours. 

     

    have a good weekend



  • 64.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-04-2019 07:47

    @LapointeMichel wrote:

    This latest update of yours really has me considering going back to 15.1 along with your miniusb fix. 


    bonjour rjtaylor, 

    I checked again this morning:

    1. none of the 236 switches on which miniusb console port was disabled experienced a swizzle reboot in the pas 6 days. 
    2. of these, about 15 still issues the getty request when I enable the miniusb for a quick check 
    3. I still have switches going zombie all the time (6 this weekend) 

    If there is a link between miniusb and swizzle reboot, then the next test would be to enable back the miniusb port on all of them and expect a swizzle reboot to happen, This i precisely what I just did  and wil keep the forum posted.

     

    by the way: since we are here dealing with 2 distinct problems (Zombie switch and Swizzle reboot)  I will update the other thread I started on this particular  problem and keep this thread on the zombie issue. More on this later today..

    The swizzle reboot thread is  called "this can not be good ..." 

    https://forums.juniper.net/t5/Junos/this-can-not-be-good-getty-tcsetattr-dev-ttyu1-Invalid-argument/td-p/469660

    see you there !

    Michel

     



  • 65.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-04-2019 07:54

    I check HOURLY for the release of the promised fix for PT1442376. I'll try it as soon as it is available and keep you posted. 

    Michel

     


    regarding the zombie issue, things are not looking as good as last week: I hate to write it, but the updates of PR1442376 that 18.4R3 and 19.2R2 were fixes ("resolved In ...") looks like vaporware. They took out 19.2R2 mention of the PR, and they should do the same for 18.4R3 since it is nowhere to be found in the downloads.  

    I have been promised last friday access to a beta of an image that should solve the problem. Still waiting. I'll try it when I get it, even if it screws up the miniusb/swizzle reboot test. 

    Michel

     



  • 66.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-04-2019 19:30

     

     


    @LapointeMichel wrote:

    I check HOURLY for the release of the promised fix for PT1442376. I'll try it as soon as it is available 


    OK my bad: I have been told by JTAC that the "resolved In" information does not directly relate to an available version. I'll stop complaining about it. 



  • 67.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-05-2019 06:57

    as of nov 5, 10:00, I have 236 switches running with a beta image from Juniper that may solve PR1442376, and I have disabled the miniusb console port on all of them. So it is possible that in the coming days, I have no zombie switches neither swizzle reboot. Basically, I expect the "alarms" on my outlook folder that receive PRTG ping sensor warnings to be empty
    How about a daily post to show how things are going ???
    Michel



  • 68.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-05-2019 07:00

    @LapointeMichel wrote:

    How about a daily post to show how things are going ???


     

    I know we'll be watching for sure!



  • 69.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-05-2019 12:38

    1 alarm this morning  😞     building power failure ...

    so 24 hours with no zombies or swizzle reboot .... 

    it's much too soon to say "mission accomplished", but it's a start .

    à demain ,

    Michel

     



  • 70.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-06-2019 04:34

    36 hours ...

    can't believe it. 

    as a comparison, last weekend, with 238 switches running 18.3R2, I had 6 zombies.

    Michel



  • 71.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-06-2019 05:00

    Is this beta build based on 18.3?



  • 72.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-06-2019 11:52

    @rjtaylor wrote:

    Is this beta build based on 18.3?


    the exact number of the one they gave me to try is 18.2R3-S2.  it is a big relief for me because going back to 15 was a rewriting of my config files (the switches will be used for QinQ) 

    the PR1442376 annouce the issue as resolved in now 18.4R3 and 17.4R3. 

    all I know so far is this:  

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         2 days, 17 minutes, 3 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    ... I keep staring at it for minutes at a time, and refresh it, and stares again  ... 🙂

    Michel



  • 73.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-06-2019 12:30

    18.2R3-S2 makes sense.  They told me long ago that was the target for this fix.



  • 74.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-07-2019 04:56
        Start time                     2019-11-04 14:04:20 EST
        Uptime                         2 days, 17 hours, 47 minutes, 48 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    2.5 days and counting ...

    received a swizzle reboot notification yesterday from one of our 8 production switches.  I believe at will disable the miniusb ports on all of them today. just in case ...

    Miche



  • 75.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-07-2019 12:02
        Start time                     2019-11-04 14:04:20 EST
        Uptime                         3 days, 25 minutes, 46 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    3 days 🙂    can't take my eyes of that display ....

    Miche



  • 76.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-07-2019 12:32

    @LapointeMichel wrote:
        Start time                     2019-11-04 14:04:20 EST
        Uptime                         3 days, 25 minutes, 46 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    3 days 🙂 


     

     

    That's like, two years with all your switches contributing 😉



  • 77.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-08-2019 12:58
        Start time                     2019-11-04 14:04:20 EST
        Uptime                         4 days, 1 hour, 49 minutes, 27 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    I know I will check from time to time this weekend. 

    by the way, all 238 lab switches are like that, give or take an hour or so. 

    I am still dealing with JTAC on 2 service request: the swizzle reboot issue, and a minor alarm on high  partition usage that started on most switches. But I am in a better position now that I have been in months. 

    I'll have a nice weekend 🙂

    Michel



  • 78.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-11-2019 05:52

     

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         6 days, 17 hours, 39 minutes, 55 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    6.5 days on.....    things are going well. 

     

    so well that the  alarms I received this weekend were for actual problems unrelated to PR1442376 or swizzle reboot. I lost communication with one switch, but it turned out not to be PR1442376/zombie. It looks like a genuine tech problem: all ports led went down, and RJ45 console keep showing never seen before error messages.

     

    da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 0f a0 a8 00 00 08 00 
    (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
    (da0:umass-sim0:0:0:0): Retrying command
    (da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 0f a0 a8 00 00 08 00 
    (da0:umass-sim0:0:0:0): CAM status: CCB request completed with an error
    (da0:umass-sim0:0:0:0): Error 5, Retries exhausted
    g_vfs_done():gpt/junos[WRITE(offset=11730944, length=4096)]error = 5
    g_vfs_done():gpt/junos[WRITE(offset=739377152, length=20480)]error = 5
    g_vfs_done():gpt/junos[WRITE(offset=739573760, length=32768)]error = 5
    g_vfs_done():gpt/junos[WRITE(offset=739606528, length=32768)]error = 5
    g_vfs_done():gpt/junos[WRITE(offset=739639296, length=32768)]error = 5

     

    I thought of trying the miniusb console, but I turned it off last week,  so I am stuck here. I'll reset it using AC. 

    So test will go on until I get a legit version ... that will go through testing again. But things are looking up 🙂

    .Michel

    Michel



  • 79.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-11-2019 06:50

    Great news about your uptime!

     

    I recognize the CAM/VFS errors from FreeBSD.  That's related to storage.  I wonder if the flash chip is failing?  Let us know if it comes back after power cycling.



  • 80.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-11-2019 06:59

    it did came back. Does not show any alarms so far. 

    oddest thing was to see all leds from the ports turned off. 

    I'll open a Service Request just in case

    Michel

     

     



  • 81.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-12-2019 04:54
        Start time                     2019-11-04 14:04:20 EST
        Uptime                         7 days, 17 hours, 38 minutes
        Last reboot reason             Router rebooted after a normal shutdown.

    7.5 days up, the trust is slowly building back. 

    I'll do another check today of enabling back the miniusb console and see the state of the getty requests sickness.

    set system ports auxiliary port-type rj45
    commit
    quit
    monitor start messages

    last week, it took 4 days after the new image installation for 34/236 to catch a cold (cold = issuing getty requests) .

    so  the disabling of the miniusb looks like a very efficient bandaid, but a bandaid nonetheless: like we say in french: "le ver est dans la pomme". 

    never forget this is a pre-version. 



  • 82.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-12-2019 13:00
        Start time                     2019-11-04 15:43:29 EST
        Uptime                         8 days, 14 minutes, 46 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    8 days and counting, 

    I just re-enabled the miniusb console port on the 34 switches that still issue getty request. My bet is that some of them will swizzle reboot eventually. Let's see what happens. 

    Michel



  • 83.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-12-2019 21:48

    How is 18.2R3-S1 still labelled as a recommended release for this switch?



  • 84.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-13-2019 05:03

    @ablanken wrote:

    How is 18.2R3-S1 still labelled as a recommended release for this switch?


    bonjour ablanken,

    That would be up to Juniper to answer. What I am running is a pre-release with strict order not to install it in production, even though it looks stable like a rock. I suppose 18.2R3-S2 will be the recommended version once it is made available. 

    Juniper added somw details in PR1442376. Interesting. 

    This issue has been fixed in 18.2R3-S2 and fix will be available on all upcoming Junos software releases. 
    Issue has been fixed from FreeBSD kernel version 11 build date of 2019-10-22. Junos releases built after this date would contain the fix.

    https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1442376&actp=SUBSCRIPTION 

     

    by the way, 8.5 days with no zombies.... 

    Michel



  • 85.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-13-2019 05:10
        Start time                     2019-11-04 15:19:31 EST
        Uptime                         8 days, 16 hours, 33 minutes, 28 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    8.5 days now. Since yesterday, I reactivated the miniusb consle port on the 34 switches that developped the getty request sickness last week. I expect one of them to swizzle reboot, but the night was uneventful. 

    "uneventful"... what a beautiful english word. French Canadian equivalent would be "plate".  :-).  

    miniusb console port will stay enabled for a while. other than vomiting getty request and clogging the messages files, nothing seems wrong. 

    Nov 13 08:05:19  mrcrn-evain-01-BYPASS getty[32914]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:20  mrcrn-evain-01-BYPASS getty[32918]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:20  mrcrn-evain-01-BYPASS getty[32919]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:20  mrcrn-evain-01-BYPASS getty[32920]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:21  mrcrn-evain-01-BYPASS getty[32921]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:21  mrcrn-evain-01-BYPASS init: getty repeating too quickly on port /dev/ttyu1, sleeping 30 secs
    Nov 13 08:05:51  mrcrn-evain-01-BYPASS getty[32922]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:51  mrcrn-evain-01-BYPASS getty[32923]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:52  mrcrn-evain-01-BYPASS getty[32924]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:52  mrcrn-evain-01-BYPASS getty[32925]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:52  mrcrn-evain-01-BYPASS getty[32926]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:05:52  mrcrn-evain-01-BYPASS init: getty repeating too quickly on port /dev/ttyu1, sleeping 30 secs
    Nov 13 08:06:23  mrcrn-evain-01-BYPASS getty[32927]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:06:23  mrcrn-evain-01-BYPASS getty[32928]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:06:23  mrcrn-evain-01-BYPASS getty[32929]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:06:24  mrcrn-evain-01-BYPASS getty[32930]: tcsetattr /dev/ttyu1: Invalid argument
    Nov 13 08:06:24  mrcrn-evain-01-BYPASS getty[32931]: tcsetattr /dev/ttyu1: Invalid argument

    Michel



  • 86.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-13-2019 19:07

    @LapointeMichel wrote:
        Start time                     2019-11-04 15:19:31 EST
        Uptime                         8 days, 16 hours, 33 minutes, 28 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    8.5 days now. Since yesterday, I reactivated the miniusb consle port on the 34 switches that developped the getty request sickness last week. I expect one of them to swizzle reboot, but the night was uneventful. 


    well, this one wasn't.  One of the switch I reactivated the miniusb port and who was sending getty request at ttyo1 ended up doing a swizzle reboot, just as I had predicted 🙂  

    now it,s far from being a scientific proof, but I see a correlation between a switch that emits getty request , the minusb console port  and the dreaded swizzle reboot action.  I may prevent the swizzle reboot and the clogging of the message files by disabling the miniusb console port, but there is still something brewing in the switch and it,s up to JTAC to figure it out. I already have a Service Request open and I'll update it first thing in the morning.

    ... never thought I'd be happy to see a switch rebooting 🙂 

    Michel

     



  • 87.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-14-2019 05:00

    This is especially interesting if those folks hanging onto 15.1 can eliminate the reboot by disabling mini-usb.  At that point, they'd have a stable switch.



  • 88.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-14-2019 05:53

    bon matin rjtaylor, 

    well guess what ? I came in this morning and I am up to 4 switches that went swizzle reboot. Just as I predicted. Here is some highlights...

    1. they were part of 236 switches on which I installed 18.2R3 S2.7 monday nov 4 AND disabled the miniusb console port
    2. they are part of 34 switches that I identified as "sick" on nov 8 aftre reactivating the port and issuing a monitor start message that showed only those 34 were issuing getty ttyu1 requests.
    3. on nov 12, I left the miniusb activated on those 34 switches and predicted some of them would swizzle reboot
    4. this morning, I am up to 4 of the 34 switches that did just that.

    Now I am no JTAC expert, but like you said, if I was running 15.x, I would go ahead today and disable the miniusb console port. The exact command I used was set system ports auxiliary port-type rj45 disable,  if you have followed this saga from the start, and also its spinoff    this-can-not-be-good-getty-tcsetattr-dev-ttyu1-Invalid-argument/ you will have learned in previous episodes that any system port aux command will affect the miniusb console port. RJ45 console port was still working on my switches after this command.  The switch may still get sick, but for some unknown reason, you may save yourself annoying swizzle reboot.  Now since I have absolutely no clue as to what happens here, this procedure may develop adverse effects . So far, I have not seen any but we all know that absence of proof is not proof of absence 🙂

    so if you try it , you're  on your own. But in the good company of my 236 switches that seems to enjoy what I do to them.

    I sent this with more details to my friend at JTAC working on my Service Request. Hope it helps. 

    Michel



  • 89.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-14-2019 08:48

    What can we do and is there any place we can downlaod 18.2R3 S2.7 if we have this issue? I have a customer in the financial sector who is now askign me to rip out these new switches due to this issue and the impact is causes. Swizzel would be MUCH worse, so I have not triex 15.X. I have havd great dificulty getting my distributer to open a ticket directly with Juniper. That was finally done but it looks like it may take days if they even choose to give me the software...

     

    Until then I am left with a crippled product that my customer is now askign I replace...



  • 90.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-14-2019 09:49

    hi Dave, 

    the good news are that the fix seems to work, and  that I was told repeatedly the official release should be available this month. I got my version by dealing with my Juniper rep, and maybe for the dubious reason that I have been dealing with this problem since last may  and kept bugging them since then. 

    Getting that pre-release  image and installing it  not a route you want to follow. Can't speak for others who got it, but I had to put in writing that I would not install it on production equipement and they clearly wrote back to me not to expect support on this pre-release. So I too have to wait. 

    I only have 8 switches running 15.1 and 19.1 in production and I am a bit scared every day that one will reboot or freeze, and they are reliable on that 🙂   After a reboot last week, we went ahead and disable the miniusb on all prod switches. As far a freezing, we put one running 19.1 that is located at an  hour drive on a ethernet power bar, so we can reach it and reset the switch from the power bar relying on the fact that the traffic still holds even when PR1442376 happens ! ! give a whole new meaning to the word "bootstrap", now doesn'it ? 

     

    Now I state again that I am not a JTAC or Juniper tech, but the route suggested by rjtaylor (and me) of using 15 to get around the PR1442376 issue and disabling the miniusb port to circumvent whatever cause the swizzle reboot may be a way to get something stable until 18.2R3-S2 available release. 

    good luck, Dave.

    Michel

     

     



  • 91.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

     
    Posted 11-14-2019 11:49

    Hello Dave,

     

    I will reach out to you via the JTAC case. I have been working with Michel who has been updating this post as well.

     

    -Alper



  • 92.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-15-2019 04:59

    well bonjour  Alper, 

    had I known you read this forum, I would have been more careful in my writing 🙂

     

    another of my 34 sick switches went into swizzle reboot during the night. Never thought I would be looking forward to a switch rebooting !!!    so leaving the miniusb console port open seems to be an invitation toward swizzle reboot IF the switch is emitting getty request. 

    So I have roughly 200 switches that seems immune to swizzle reboot (mini usb disabled) . the other 34 (miniusb enabled)  are slowly swizzle rebooting one by one. 

    next verification is: on my 200, I will turn the port on and message on to see if any developped the emitting getty request sickness since the moment I identified the 34 sick ones. That should be interesting ...

    Still puzzled that turning the miniusb off has the combined effect of stopping the getty request and preventing the swizzle reboot, but hey ... 

    Michel



  • 93.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-15-2019 05:28

    @LapointeMichel wrote:

    Still puzzled that turning the miniusb off has the combined effect of stopping the getty request and preventing the swizzle reboot, but hey ... 


     

     

    The getty daemon's rapid respwan must be eating enough resources on the control plane that something the watchdog is checking appears unresponsive.  Or maybe the watchdog sees many PIDs being generated from a single parent?  Then the watchdog does its thing (swizzle).

     

    Thanks to all your efforts, Michel, I've seen enough to go back to 15.1 and disable mini-usb on my tiny population of EX2300-C.  Then I'll wait comfortably for the golden code to be delivered.



  • 94.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-15-2019 06:36

      bon matin rjtaylor, 

    you may remember I considered this route in september, but adapting the QinQ configuration we developped under 18 and 19 proved very cumbersome, and the swizzle reboot was the last straw. 

    Your explanation make sense: something eats ressources and  triggers the watchdog. 

    but ...

    I just finished re-checking my beloved switches. I reactivated the miniusb and monitor start messages on all of them. Then all I have to do is wait 60 secondes for the getty request to be issued if a switch is affected. I then run a short script on SecureCRT  that goes from tab to tab stopping 4 seconds on each for me to see the messages - and I can drink coffe while doing it 🙂 

    so here is the results..

    • all of the 34 switches that were identified last week as issuing getty request still do EXCEPT the ones (5, so far) that went into swizzle reboot in the past 72 hours. 
    • None of the switches that did not issued getty request before have started doing so.

    I keep the miniusb console port enabled only on the 34 that were issuing getty request, so I expect them to slowly swizzle reboot one by one. 

    Not sure what to make of that result:

    • if the trigger was caused by something in the switch, then the 5 that swizzled reboot should have started emitting getty requests again (they have not) 
    • if the trigger is truly random and can happen on any switch, I would have expected some of those that never emitted getty request to have start doing it sometimes (not one did)

    so that's it for now. The main result is still ...

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         10 days, 19 hours, 26 minutes, 42 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    can't wait for the official release. And I know I am not the only one ...

    Michel

     

     



  • 95.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-15-2019 07:09

    @LapointeMichel wrote:

      bon matin rjtaylor, 

    you may remember I considered this route in september, but adapting the QinQ configuration we developped under 18 and 19 proved very cumbersome, and the swizzle reboot was the last straw.


     

     

    Yeah, I had to change a QoS config that applies in 18 but not 15.  With the swizzle in check, it was worth it for me.  Zombies aren't any good for produciton.

     

    I suspect that, given enough time, your five swizzle-rebooted switches would end up exhibiting the problem again.  The mini-usb bandaid has proven 100% effective in your lab.  For me, that's the sure bet.  My first switch is now reverted to 15.1X53-D592.1.  I'm off to do a few more now.

     

    For anybody going backwards to 15.1, sort out your config to make sure it'll commit when you get back there.  I didn't want to attempt to downgrade via "request system software add" so here's what I've done:

     

    1. Add this to the config:
      system {
          ports {
              /* disabled mini-usb to prevent swizzle reboot */
              auxiliary {
                  port-type rj45;
                  disable;
              }
          }
      }
    2. Format a USB stick with FAT32.
    3. Save config (/config/juniper.conf.gz) to that USB stick.
    4. Get 15.1 install image from Juniper.  It's a .gz that you can make into bootable USB.
    5. Expand .gz to .img with 7-Zip or gunzip.
    6. Write .img to a different USB stick (not the one with your config).  Rufus or dd.
    7. Plug bootable stick into switch.
    8. Connect to console.
    9. Reboot switch.
    10. Interrupt boot and choose [B] for boot menu.
    11. Boot from USB.
    12. When install is done and switch is at login, login as root.
    13. Plug in your USB stick with config.
    14. mount -t msdosfs /dev/da1 /mnt
    15. cp /mnt/yourconfig.gz /tmp
    16. umount /mnt
    17. cli
    18. configure
    19. delete
    20. load replace /tmp/yourconfig.gz
    21. commit and-quit


  • 96.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-15-2019 10:06

    My money is also that it's a matter of time before the 34 all go through swizzle reboot. I'll keep you posted on monday.

     

    Gee , thanks for the procedure: I wish I could have read that last march. It took me a while to figure it.

     

    I still use the USB key sometimes like that. It works fine. It may happen though that after plugging it, their name differ from "da1". I have ran into "da1s1", for exemple. ls /dev  will show the name that was given and should be used. 

     

    My favorite way now is  going through the console port and use "load override terminal", paste the text file with mouse right click and then ctrl-d to end the process. if you got a text file of set commands,  use "load set terminal " .the same way . I have Excel generating text files of set commands for individuals files configuration and "l s t"  them on each new switch

     

    I am still looking for a way to copy a file through the console port, though. There must be a way but I gave up searching .When I have to do it, I still rely on the usb key. 

    Michel



  • 97.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-15-2019 10:29

    For a collection as large as yours, Michel, I think I'd use autoinstallation via DHCP/TFTP.  I really only have a few of these little guys.

     

    Understanding Autoinstallation of Configuration Files 



  • 98.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-15-2019 12:31

    @rjtaylor wrote:

    For a collection as large as yours, Michel, I think I'd use autoinstallation via DHCP/TFTP.  I really only have a few of these little guys.

     

    Understanding Autoinstallation of Configuration Files 


    let me check that next week. Thanks. Have a nice weekend, 

    latest update:  

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         11 days, 1 hour, 24 minutes, 11 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    I fully expect some swizzle reboot this weekend and, why not, some of my 5 swizzled switch getting sick again. see you monday 

    Michel

     



  • 99.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 05:47

    I fully expect some swizzle reboot this weekend and, why not, some of my 5 swizzled switch getting sick again. see you monday 


    well, at least, I am getting good at predicting the future ....

    so from the group of 34 sick switches, 5 more went into swizzle reboot, which is kind of expected. 

    oops - make it 6: one more sick switch just went swizzle !

     

    but out of the now 10 that went into swizzle reboot, I have 2 of them that started doing getty requests again !  coincidence ???? 

    I can't explain what is happening, but once I get the official release of 18.2R3-S2,

    • the first thing I'll do is install it, 
    • the second thing I'll do is disable miniusb console port. 

    other than that,the image keeps the zombies away ....

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         13 days, 18 hours, 38 minutes, 5 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    Michel

     

     



  • 100.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 06:31

    @LapointeMichel wrote:

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         13 days, 18 hours, 38 minutes, 5 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

     

     

     

    Glorious!



  • 101.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 07:17

    what's worse than a swizzle reboot ? a swizzle reboot that get stuck ...

    The swizzle reboot itself is very annoying, and in our case where the switch has noredundancy, has the potential to stop traffic for 6 mns. 

    But i just discovered this morning that it can be worst ...

    I had one switch that went down this weekend but not up: turn out it's one of my 34 sick switch that tried to swizzle reboot but got stuck in the reboot process: the sys led was flashing, no console com, no ip access, no nothing. AC reset was the only option. 

    I don't know what is causing the swizzle reboot and the link to miniusb, but this has the potential to really put you in trouble. 

    Michel

     



  • 102.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 13:02

    Hey rjtaylor, 

    for what it's worth, can you tell me the hw serial number of a switch that went swizzle reboot on you ???

    HW0218250nnn or HW0218230nnn ????

    Michel



  • 103.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 13:30

    Mine are EX2300-C.  These two swizzled:

    HV0217520nnn

    HV0217520nnn

     



  • 104.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 13:51

    Thanks everyone for your help. Juniper was able to get back to me and did offer me the pre-release software with the same warning Michel got. They have been great. The only real delay was getting my distributer to open a case on my behalf. I have a cheaper support option so I cant open cases directly. May rethink that in the future. 

     

    While this is a SERIOUS issue, its so great to see progress made that doesnt involve a hardware swap and its a testiment to the design that it still somewhat functions (including forwarding and vlan seperation)  completely braindead (zombie switch)

     

    For anyone wondering, my customer has a production and a cold spare switch. I loaded pre-release on the cold spare and issued "set system ports auxiliary port-type rj45 disable" on both of them. Now they have the option along with the tidbit that the final release may be coming before Thanksgiving (or soon after)



  • 105.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 17:37

    Bonsoir Dave, 

    very happy to hear you could get help. The combination of 18.2R3S2 for the zombie issue  and turning off the miniusb console port for the swizzle reboot  might be the winning solution for now. I know Juniper is working on the swizzle reboot.  That's the combination I plan to use in january for my deployment.  mini usb is off on our 8 prod switches and no swizzle for the pas 3 weeks. 

    I was afraid of having a zombie switch this morning: turns out it was a swizzle reboot that got stuck !!! but it was on one of the 34 switches that I leave miniusb on for test and so far, the reliably go into swizzle reboot. being able to predict the problem is reassuring. The symptom is the issuing of getty request. 

    Michel

     



  • 106.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-20-2019 05:01

    so 15 days now of running 18.2R3-S2.7 and no PR1442376. Can,t wait for the official version .

    In the meantime, I still have the swizzle reboot issue. I have posted this morning  an update on this thread 

    thread "this-can-not-be-good-getty-tcsetattr-dev-ttyu1-Invalid-argument" 

    basically, I am up to 13 /34 switches that I left the miniusb on that have swizzled reboot. The other 200 have their miniusb port off for 15 days now and none swizzled. those 34 were the ones still issuing getty requests on ttyu1. So I am turning the miniusb port off on those last 34 and keeping it this way from now on So my beloved switches should not be goigng zombies or swizzle reboot anymore. 

    since this thread was for the PR1442376 that seems under control and the swizzle reboot issue is not directly related to this thread, I'll keep updating the other one. 

    See you there !

    Michel

     



  • 107.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-25-2019 06:05

    good news is:  tests switches running 18.2R3S2.7 with disabled miniusb console still holds steady

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         20 days, 18 hours, 42 minutes, 57 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    the bad news is that one of the prod switches still running 19.1 seems to have gone zombie: no communication. So I hope that the dec 4 date for the official release still holds, Nice to be reminded from time to time of the urgency of the situation and that this PR deserves it's "Major" severity status. 

     

    Since I don't have physical access, I'll have to check with a technician over there  (1.5 hour drive) if it has th other symptoms (richt side sel switch dead. No console access.) .  Contrary to what is still displayed on the PR1442376, the traffic still goes through when the switch lose communication. At least , this is my experience of the situation, and I had my share.  I once kept a switch in a zombie state for more than 3 weeks: traffic still going, no customer complaints. That statement in the PR that traffic is stopped may come from the first exchanges I had in june with JTAC when I tried to connect ports on a zombie switch and could not. What I believe happens is that the existing connections on a zombie switche wil keep on carrying traffic, but you can't connect anything else. 

    Or do I go ahead and update my prod switches with the pre-release I promised not to use ???? like said the old Macbeth: "that is the question" ...



  • 108.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-28-2019 05:36

    Guess what I found on the download site  this morning ...

    EX2300(-C) and EX3400 Series	18.2R3-S2	28 Nov 2019	tgz   (313.01MB)
    Checksums
    Limited - EX2300(-C) and EX3400 Series	18.2R3-S2	28 Nov 2019	tgz   (313.01MB)
    Checksums

    has the zombie switch saga reached it's end ?  I'll try and install it on my switches before the weekend, and  plan to install it on the production switches early next week. 

    so far, here is the result of 18.2R3-S2.7: 

        Start time                     2019-11-04 14:04:20 EST
        Uptime                         23 days, 18 hours, 30 minutes, 27 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    let's see if the official release is up to it !

    I'll keep you posted. 

    Michel



  • 109.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-28-2019 05:48

    oops ...  🙂

     

    Alert
    The Software file that you are trying to access is unavailable. We apologize for the inconvenience.

     



  • 110.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-29-2019 05:27

    @LapointeMichel wrote:

    oops ...  🙂

     

    Alert
    The Software file that you are trying to access is unavailable. We apologize for the inconvenience.

     


    OK now ! got it 🙂   let's try it...



  • 111.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-30-2019 06:27

    Here in the States it was a Thanksgiving gift!  Cannot wait to get this onto our gear!



  • 112.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-01-2019 23:23

    Hey Michel,

     

    I downloaded the JunOS 18.2R3-S2.9 and installed it on two EX2300-48t. So far so good, but you have way more devices to test it.

     

    Do you already have some results to share 🙂 ??

     

    Cheers,

    Julian



  • 113.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-03-2019 08:17

    bonjour !

          so I downloaded and installed yesterday 18.2R3S2 on 237 Juniper Ex2300-C that I have in test, following the official release last friday.  so ...

    --- JUNOS 18.2R3-S2.9 Kernel 32-bit  JNPR-11.0-20191022.14c2ad5_buil
    
    root@vvb-valparadis-01-BYPASS> show chassis routing-engine 
    Routing Engine status:
    ...
        Start time                     2019-12-02 09:10:04 EST
        Uptime                         1 day, 1 hour, 27 minutes, 34 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

     remember that on all those switches, the miniusb console port is disabled. So I expect the combination of the new image and the miniusb disabling to protect my lab from the zombi switches AND the swizzle reboot. if everything hold until after the holdays, then I am on to disassembling the lab and deploy the switches.

    so it's been only a day - nothing special to report. 

    well, just 2 things ...

    1) a month ago, when I made the pre-release installation, I was confronted with a majority of switches giving me a minor alarm of   Minor RE 0 /var partition usage is high.  I was surprised because most of the switches showed it. SR to JTAC and they advised me to do a request system storage clean-up. I ended up doing it on all my switches before installing the new image and now the alarms are gone. I am watching for alarms, but all my yellow leds are off 🙂 

    2) my procedure to image upgrade is to ftp the image on the /tmp directory of the switch, and do a software add with the options force unlink no-copy reboot. I tried a couple of times to copy the image on /var/tmp directory as the procedure suggest 

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB11409&actp=METADATA 

    Copy the jinstall package to the router. We recommend that you copy it to the /var/tmp directory, which is a large file system on the hard disk.

    ... but I kept running into space issues that no storage spasce cleanup could solve. I then went back to this trick I learned las july of using the /tmp directory. 

     

    I'll keep an eye open and post from time to time where I am exactly in my test. even good news are worth posting !  I also long to get the release document to read with my own eyes that PR1442376 is solved . I'll probably frame it. 

     

    by the way, if you are curious as to how I upgrade 237 switches, I have all their names and IP adress in Excel. I generate 237 small batch files that I execute with a click and that ftp a .sh file on each switch.  I then use SecureCRT to open a bunch of ssh, issue a common storage space cleanup, and execute the .sh file that issue 2 cli commands: one is to get the image from ftp, the second to do the software add with the options. It worked the first time on all my switches !!!   

     I will try again with Junos Space, but I gave up using it for this last summer: for exemple, staging the image was alaways done in the /var/tmp directory, and asking to install it failed most of the times. 

    Michel

     

     



  • 114.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-04-2019 07:34

    everything still runs ok.

        Start time                     2019-12-02 09:10:04 EST
        Uptime                         2 days, 1 hour, 21 minutes, 13 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

     

    I noticed this morning that 18.2R3-S2 is now the recommended EX2300 image . My next concern is adressing the swizzle reboot issue. 

    Juniper recommended Images 

    EX2300	Junos 15.1X53-D591 / 18.2R3-S2	3 Dec 2019
    EX2300-C	Junos 15.1X53-D591 / 18.2R3-S2	3 Dec 2019


  • 115.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-06-2019 06:45

    We've been seeing this same issue on multiple EX2300's running 18.2R3.4 and one we have currently on 19.3R1.8 for testing purposes. We've also had a similar issue on 2 switches that are members of 2 different virtual chassis but they are not passing L2 traffic when this happens, has anybody seen that? I'm wondering if the direct attach cables we're using between SFP slots are failing as part of this "zombie state" as well?



  • 116.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-06-2019 10:36

    bonjour 3F

    couple of things:

    1. I tried pretty much all available EX2300 images and only 18.2R3-S2.9 so far is free from the PR1442376 zombie state.  anyhing else than -S2 is at risk
    2. the particularity of the zombie state I experienced is that the traffic flows through the copper ethernet ports connected before the switch get sick. Never tried the optical ports. 
    3. ... which reminds me I have another test I forgot to do on the new image: is the unlit led problem solved ? on some (not all)  images, I was able to plug my sfp between 2 EX2300. Traffic flows, but no leds were on. JTAC recognized and duplicated the problem, which is a pain when I will ask a technician to plug it, and tell him  "don,t worry if the lights dont come up "  🙂 I'll try that this afternoon. 
    4. basically, I strongly suggest you move to 18.2R3S2 AND disable the miniusb console port (set system port aux rj45 disable) to prevent the swizzle reboot for which a PR is being open. 

    Michel

    still can't get used to watching this ...

        Start time                     2019-12-02 09:10:04 EST
        Uptime                         4 days, 4 hours, 24 minutes, 26 seconds
        Last reboot reason             Router rebooted after a normal shutdown.


  • 117.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-10-2019 12:41

    so we are now 8 days just like they should be: boring ...  🙂

     

        Start time                     2019-12-02 09:10:04 EST
        Uptime                         8 days, 6 hours, 20 minutes, 49 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    the documentation is also available for the 18.2R3S2,  

    Software Release Notification for JUNOS Software Version 18.2R3-S2 

    and PR1442376  is now under the classification 

    18.2R3-S2 - List of Fixed issues

    1442376
    EX2300 platforms might stop forwarding traffic or responding to console
    On EX2300/EX2300-C platforms, if Junos software is with FreeBSD kernel version 11 with the build date on or after 2019-02-12, the switch may stop forwarding traffic or responding to console. Power cycle of switch would recover the issue.

    good !!!!

     

     

     



  • 118.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-10-2019 13:03

    Same here on my small collection of EX-2300C:

     

    Start time 2019-11-30 14:21:35 EST
    Uptime 10 days, 1 hour, 39 minutes, 59 seconds
    Last reboot reason Router rebooted after a normal shutdown.


  • 119.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-17-2019 06:14

    bonjour rjtaylor, 

    so I plan this as my last entry on this topic. Seems Juniper / JTAC nailed it with the new 18.2R3-S2. If I put the pre-release and the official release together, I have a documented 40 days without having the PR1442376 problem coming up. It is also being showed as "solved" in the release documentation. 

        Start time                     2019-12-02 09:10:04 EST
        Uptime                         14 days, 23 hours, 43 minutes, 38 seconds
        Last reboot reason             Router rebooted after a normal shutdown.

    so we on to the next problem: the swizzle reboot....

    remember I thought I had found a way around it by disabling the miniusbconsole port ? well I had a bad surprise friday 13 (i am not making this up...)  when a switch with the new os and the disabled port went swizzle reboot on me. it is one of 240, and the fix worked for more than 40 days but still ...

    I trust the trick make a difference: I re-activated the miniusb yesterday on all switches for test: i identified around 30/240 emitting getty request , and 2 of them swizzle rebooted in 24 hours ! 

    there is another thread  dedicated to this particular problem :   https://forums.juniper.net/t5/Junos/this-can-not-be-good-getty-tcsetattr-dev-ttyu1-Invalid-argument/m-p/471437#M16208   this is were I will follow up on it.  

    Michel



  • 120.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 12-17-2019 13:12

    Thanks for sharing Michel!



  • 121.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 11-18-2019 17:40

    @rjtaylor wrote:

    Mine are EX2300-C.  These two swizzled:

    HV0217520nnn

    HV0217520nnn

     


    well, so much for that idea ...    but the issuing of getty request is definitely affecting some of my switche and not the others. 

    Michel



  • 122.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-19-2019 07:53

    Hi,

     

    I have Ex2300 switch that hangs completley every two weeks or so. It started after upgrade to JUNOS 18.4R2.7 Kernel 32-bit JNPR-11.0-20190517.f0321c3_buil.

    After upgrade i now have only 145MB free space after system cleanup. No space to upgrade or downgrade.
    Can you advise - how did you downgrade?
    Using USB?

    (I am small user an have only 4 of these switches. Only one has this "hang" problem. I have no central managment system to control these switches.)


    Any advice?



  • 123.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-19-2019 08:31

    Hi ivugrinec,

    We ran into the same issue when upgrading the EX3400 and EX2300 series at a customer earlier this year: upgrades failed due to lack of enough free storage. We tried out several options together with Juniper support. In the end, they updated the following KB, containing the possible workarounds: https://kb.juniper.net/InfoCenter/index?page=content&id=KB31198.

    Hope it helps.

    Regards,

    Dante



  • 124.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-19-2019 08:48

    Hi Dante,

     

    Thanks for your help, but KB31198. does not help as i am unable to even download the new (or old) image to EX300. Regarding the the PR1442376 (the hang issue) - while we wait for a solution - if and when the solution arrives - what can i do?
    Any knowledge - is there any files on the system that can be deleted to accomodate some space for the new (or old image)?
    Any other way to upgrade/downgrade in this case?



  • 125.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-19-2019 15:28

    Hi ivugrinec,

    As stated in the KB:

    "If the above methods do not work, you can recover the switch using the method identified in this article “ KB31265 - How to format install on EX2300 / EX3400”. Please note that this procedure will format the switch and all configurations will be lost."

    I wouldn't recommend deleting files in the shell without any manual or KB stating it's ok to do so.

    AFAIK, your only "safe"/supported option is a format install from the loader.

    Regards,

    Dante



  • 126.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 10-21-2019 06:14

    Bonjour Dante, 

    Bonjour Ivugrinec, 

          here some of the steps I go through when upgrade/dowgrade images on my EX2300 switches.: By the way, they are still not deployed: zombie issue is not solved, and I don't want to call customers and ask to go and pull the plug to reset the switch. 

    • you may still have the image file in the var/tmp directory . (Junos-arm32-blablabla.tgz) I always erase this one to begin with. this will give you more space.
    • FreeBSD is rm /var/tmp/Junos-blablabla,  or CLI can do it with file delete /var/tmp/Junos-blablabla
    • when you are faced with a first fail, don't hesitate to try a second one. I have seen situation when the add command is issued, deletes some files and fails to run. Up arrow and enter, by some magic, then succeed. Go figure. 
    • after multiple trys, here is the upgrade command combination I use 
    • cli -c "request system software add /tmp/junos-arm-32-19.3R1.8.tgz force unlink no-copy reboot"
    1. the last item in the command make sure the switch reboot istself after installation. If not, you have to check it and reboot it manually:-(
    2. IMPORTANT :you will notice the cli -c at the beginning: it means i am running it from the command prompt and not the cli. This line is in a file I created called add .sh file so that I can ssh to the switch and invoke it by using sh add.sh directly. When I upgrade 250 switches, I need to save all the typing I can
    3. MORE IMPORTANT you will also have noticed that the image file is located in the /tmp directory and not the /var/tmp directory. This one is trickier to justify: last july, when I was faced with storage space issues, I got this suggestion from a JTAC tech that should remains nameless: move the image file from /var/tmp to /tmp. I did a couple of test when image in /var/tmp failed, but just moving the image to /tmp and retry it succeed. So I keep doing it and have not found any issue yet. Again, go figure, and please tell me if you do, 
    4. that's pretty much it. getting the file on the switch I usually do from a ftp server on the same vlan   
    5. cli -c "file copy ftp://xx.xx.xx.xx/ex2300/junos-arm-32-19.3R1.8.tgz /tmp/junos-arm-32-19.3R1.8.tgz"
    6. if you want to do a request system storage cleanup, do it before putting the image or the command will wipe it up and you'll have to re-copy it 😞 
    7. I have about 10 usb keys configure with an image and automatic boot. If I want to clean a switch, I will insert it in the usb connector and ask for request system reboot USB. once this is rebooted, it is wiped clean, and ask for dhcp where a image file and a basic config wait to be downloaded. 
    8.  standing in front of the switch and cursing it does not work. I tried ...

    good luck !! I use to dread image upgrades  like a root canal. Now i am loking forward to any new image that will solve the zombie issue. Bring it on !!

    Michel



  • 127.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 06-03-2019 05:19

    Good Day,

     

    According to the next document, EX2300 Switches without PoE Capability have only one fan.

    https://www.juniper.net/documentation/en_US/release-independent/junos/topics/reference/general/cooling-system-ex2300.html

     

    Basically, the fact that second fan is visible on 18.2R1.9 is a bug, fixed in public PR# 1361696

    https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1361696

    PR# 1361696 is fixed in 19.1R1 JunOS version, but not in 18.2R1.9



  • 128.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 08-22-2019 09:02

    I've seen the same problem on a EX-2300-C-PoE running 19.2R1.8. It's new, running for a month now and I've seen it twice.



  • 129.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 08-27-2019 12:36

    hello everyone,  

     

    as some of you have stated this is an ongoing PR investigation, the PR has not been published yet so the details I cannot share here but you should ask your JTAC or account team, this is known to affect 2300s only not 3400.

     

    hope that helps in some way.



  • 130.  RE: EX2300-48T Stop working after random time after upgrade to 19.1R1.6

    Posted 09-03-2019 05:11
    this is known to affect 2300s only not 3400.

    Carlos, 

    Thanks for posting this!  It's a big relief for us.  We've been entirely unsure what to do with our EX3400s since May.

     

    RJ