Routing

I have 4 new MX104 routers in the network and all of them generate the above alarm.

Any one ever encountered this same issue?

integrator@VP_mcRNC_MX-2> show log chassisd | last 50
Mar 18 17:12:09 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:12:14 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:13:14 send: red alarm clear, device PEM 0, reason PEM 0 Not OK
Mar 18 17:13:19 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply OK (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 2)
Mar 18 17:14:19 tray 0 fan 0 current speed 0x0 requested speed 0x4
Mar 18 17:24:19 tray 0 fan 0 current speed 0x4 requested speed 0x0
Mar 18 17:26:59 send: red alarm set, device PEM 0, reason PEM 0 Not OK
Mar 18 17:26:59 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:27:04 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:27:05 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply failed (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 6)
Mar 18 17:27:09 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:27:14 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:28:09 send: red alarm clear, device PEM 0, reason PEM 0 Not OK
Mar 18 17:28:20 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply OK (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 2)
Mar 18 17:31:59 send: red alarm set, device PEM 0, reason PEM 0 Not OK
Mar 18 17:31:59 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:32:04 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:32:05 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply failed (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 6)
Mar 18 17:32:09 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:32:14 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:33:14 send: red alarm clear, device PEM 0, reason PEM 0 Not OK
Mar 18 17:33:20 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply OK (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 2)
Mar 18 17:34:19 tray 0 fan 0 current speed 0x0 requested speed 0x4
Mar 18 17:44:19 tray 0 fan 0 current speed 0x4 requested speed 0x0
Mar 18 17:46:59 send: red alarm set, device PEM 0, reason PEM 0 Not OK
Mar 18 17:46:59 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:47:04 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:47:05 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply failed (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 6)
Mar 18 17:47:09 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:47:14 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:48:09 send: red alarm clear, device PEM 0, reason PEM 0 Not OK
Mar 18 17:48:20 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply OK (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 2)
Mar 18 17:52:04 send: red alarm set, device PEM 0, reason PEM 0 Not OK
Mar 18 17:52:04 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:52:05 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply failed (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 6)
Mar 18 17:52:09 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:52:14 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:52:19 CHASSISD_PEM_INPUT_BAD: status failure for power supply 0 (status bits: 0x6); check circuit breaker
Mar 18 17:53:19 send: red alarm clear, device PEM 0, reason PEM 0 Not OK
Mar 18 17:53:20 CHASSISD_SNMP_TRAP6: SNMP trap generated: Power Supply OK (jnxContentsContainerIndex 2, jnxContentsL1Index 1, jnxContentsL2Index 0, jnxContentsL3Index 0, jnxContentsDescr PEM 0, jnxOperatingState/Temp 2)
Mar 18 17:54:19 tray 0 fan 0 current speed 0x0 requested speed 0x4

Kind regards,

Kip.
 

If they are using DC power supplies this is a known issue on PR 1064039 where the fans do not detect the temperature correctly and fail to cool the power supply causing the shutdown to occur.

This is fixed in Junos 13.3R7

If you are not in a position to upgrade, JTAC can give you a SLAX script to periodically test the temperature and run the fans.  You may need to adjust the temperature thresholds in this script for your environment to prevent the power supply failures.  We found the settings too high initially but the script does work for systems not yet upgraded.

Hi Steve, Thanks a lot for your feedback. Actually we have version 15.1R2.9 running (Domestic version).

Is it only resolved on 13.3R7 or on all versions after that including version 15.1? Does domestic version / export version matter?

Kind regards,

Kip.

Looks like they show it resolved in your version.  So yours must be a different issue.

https://prsearch.juniper.net/InfoCenter/index?page=prcontent&id=PR1064039

---

@spuluka wrote:

If they are using DC power supplies this is a known issue on PR 1064039 where the fans do not detect the temperature correctly and fail to cool the power supply causing the shutdown to occur.

 

This is fixed in Junos 13.3R7

 

If you are not in a position to upgrade, JTAC can give you a SLAX script to periodically test the temperature and run the fans.  You may need to adjust the temperature thresholds in this script for your environment to prevent the power supply failures.  We found the settings too high initially but the script does work for systems not yet upgraded.

---

Steve,

What were your preferred temperature thresholds for this script?

Chris

I adjusted them experimentally to prevent the PEM error logs.  In the worst case I had to drop the script values by 20 degrees.  This was an enclosed rack setup with front to back cooling while the MX104 is of course side vented.

Hi Chris,

It all depends on adequate air flow and ambient temperature conditions. If you read the PR 1064039, it says following:

NOTE: In some cases, we may have to use the script after the upgrade. Please make sure to change the thresholds before applying the new script:
tmp-full: 50
tmp-int: 45
tmp-nor: 35

Hope this helps.

Thanks

Sarathi