Switching

 View Only
last person joined: 2 days ago 

Ask questions and share experiences about EX and QFX portfolios and all switching solutions across your data center, campus, and branch locations.
Expand all | Collapse all

EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

  • 1.  EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-06-2020 07:18

    Hello,

     

    I'm working with a few EX3400-48P that are exhibiting some odd behavior around PoE. Every few hours, one of the poe devices (there doesn't look to be any pattern to which device, it seems evenly distributed) will reboot indicating a momentary loss of PoE. At the same time, the following message appears in the syslog:

     

    fpc0 PoE: Failed to read voltage measurements

    My searching hasn't found what that message means so I'm at a bit of a loss to where to look for a resolution. This is happening on multiple switches in the facility. I have already updated the PoE controller. Every port/switch is well below the PoE budget.

     

    root@switch1> show chassis firmware detail 
    FPC 0                     
       PoE firmware            2.1.1.19.3
       Boot Firmware          
          U-Boot               U-Boot 2016.01-rc1 (Sep 01 2016 - 16:00:13 -0700)  1.3.0
       Boot Firmware          
          loader                       FreeBSD/armv6 U-Boot loader 1.2
    FPC 1                     
       Boot Firmware          
          U-Boot               U-Boot 2016.01-rc1 (Sep 01 2016 - 16:00:13 -0700)  1.3.0
       Boot Firmware          
          loader                       FreeBSD/armv6 U-Boot loader 1.2
    
    root@switch1> show version 
    fpc0:
    --------------------------------------------------------------------------
    Hostname: switch1
    Model: ex3400-48p
    Junos: 18.2R3.4
    JUNOS OS Kernel 32-bit  [20190605.30b921f_builder_stable_11]
    JUNOS OS libs [20190605.30b921f_builder_stable_11]
    JUNOS OS runtime [20190605.30b921f_builder_stable_11]
    JUNOS OS time zone information [20190605.30b921f_builder_stable_11]
    JUNOS py extensions [20190620.215446_builder_junos_182_r3]
    JUNOS py base [20190620.215446_builder_junos_182_r3]
    JUNOS OS crypto [20190605.30b921f_builder_stable_11]
    JUNOS network stack and utilities [20190620.215446_builder_junos_182_r3]
    JUNOS libs [20190620.215446_builder_junos_182_r3]
    JUNOS runtime [20190620.215446_builder_junos_182_r3]
    JUNOS Web Management Platform Package [20190620.215446_builder_junos_182_r3]
    JUNOS ex libs [20190620.215446_builder_junos_182_r3]
    JUNOS ex runtime [20190620.215446_builder_junos_182_r3]
    JUNOS ex platform support [20190620.215446_builder_junos_182_r3]
    JUNOS dcp network modules [20190620.215446_builder_junos_182_r3]
    JUNOS modules [20190620.215446_builder_junos_182_r3]
    JUNOS ex modules [20190620.215446_builder_junos_182_r3]
    JUNOS ex Data Plane Crypto Support [20190620.215446_builder_junos_182_r3]
    JUNOS daemons [20190620.215446_builder_junos_182_r3]
    JUNOS SDN Software Suite [20190620.215446_builder_junos_182_r3]
    JUNOS Extension Toolkit [20190620.215446_builder_junos_182_r3]
    JUNOS Phone-home [20190620.215446_builder_junos_182_r3]
    JUNOS Packet Forwarding Engine Support (EX34XX) [20190620.215446_builder_junos_182_r3]
    JUNOS jdocs ex [20190620.215446_builder_junos_182_r3]
    JUNOS jail runtime [20190605.30b921f_builder_stable_11]
    JUNOS FIPS mode utilities [20190620.215446_builder_junos_182_r3]
    
    fpc1:
    --------------------------------------------------------------------------
    Hostname: switch1
    Model: ex3400-48t
    Junos: 18.2R3.4
    JUNOS OS Kernel 32-bit  [20190605.30b921f_builder_stable_11]
    JUNOS OS libs [20190605.30b921f_builder_stable_11]
    JUNOS OS runtime [20190605.30b921f_builder_stable_11]
    JUNOS OS time zone information [20190605.30b921f_builder_stable_11]
    JUNOS py extensions [20190620.215446_builder_junos_182_r3]
    JUNOS py base [20190620.215446_builder_junos_182_r3]
    JUNOS OS crypto [20190605.30b921f_builder_stable_11]
    JUNOS network stack and utilities [20190620.215446_builder_junos_182_r3]
    JUNOS libs [20190620.215446_builder_junos_182_r3]
    JUNOS runtime [20190620.215446_builder_junos_182_r3]
    JUNOS Web Management Platform Package [20190620.215446_builder_junos_182_r3]
    JUNOS ex libs [20190620.215446_builder_junos_182_r3]
    JUNOS ex runtime [20190620.215446_builder_junos_182_r3]
    JUNOS ex platform support [20190620.215446_builder_junos_182_r3]
    JUNOS dcp network modules [20190620.215446_builder_junos_182_r3]
    JUNOS modules [20190620.215446_builder_junos_182_r3]
    JUNOS ex modules [20190620.215446_builder_junos_182_r3]
    JUNOS ex Data Plane Crypto Support [20190620.215446_builder_junos_182_r3]
    JUNOS daemons [20190620.215446_builder_junos_182_r3]
    JUNOS SDN Software Suite [20190620.215446_builder_junos_182_r3]
    JUNOS Extension Toolkit [20190620.215446_builder_junos_182_r3]
    JUNOS Phone-home [20190620.215446_builder_junos_182_r3]
    JUNOS Packet Forwarding Engine Support (EX34XX) [20190620.215446_builder_junos_182_r3]
    JUNOS jdocs ex [20190620.215446_builder_junos_182_r3]
    JUNOS jail runtime [20190605.30b921f_builder_stable_11]
    JUNOS FIPS mode utilities [20190620.215446_builder_junos_182_r3]
    
    root@switch1> show poe controller 
    Controller  Maximum   Power         Guard    Management   Status        Lldp
    index       power     consumption   band                                Priority 
       0        740W      67.00W          0W     Class        AT_MODE       Disabled

    Any insight into this would be greatly appreciated.



  • 2.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-06-2020 07:44

    Are you connecting your switches (which have failed) using the POE ports by the way? That's a usual issue where they both try to power each other and it causes a crash.

    KR
    Adam



  • 3.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

     
    Posted 01-06-2020 08:03

    How to troubleshoot when POE is input to a POE enabled port (back to back POE) is called out in the KB below:

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB31588&cat=EX2300&actp=LIST



  • 4.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-06-2020 08:37

    Each switch is only connected to other PoE switches via DAC or fiber. I'll verify once more to be sure.



  • 5.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-06-2020 08:40

    None of the ports are indicating the status 36 so I have high confidence that there isn't any mis-patch anywhere:

     

    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/0 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/1 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/2 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/3 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/4 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/5 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/6 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/7 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/8 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/9 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/10 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/11 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/12 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/13 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/14 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/15 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/16 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/17 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/18 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/19 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/20 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/21 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/22 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/23 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/24 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/25 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/26 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/27 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/28 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/29 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/30 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/31 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/32 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/33 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/34 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/35 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/36 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/37 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/38 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/39 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/40 status 1
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/41 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/42 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/43 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/44 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/45 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/46 status 27
    Jan  5 18:27:30  switch1 chassisd[5155]: PoE port name ge-0/0/47 status 27


  • 6.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-06-2020 09:26

    Strange...

     

    It might be a firmware issue with the 3rd party POE code.

    When you ran 'show poe controller' was there a notice in the CLI response to update the POE Firmware?

     

    Generally I'd say now open a ticket with JTAC though and confirm with them it isnt a JUNOS issue but if that isn't an issue it's likely be the POE firmware and you can update via CLI using 'request system firmware upgrade poe fpc-slot 0'.

    That's a last resort one though.

    KR
    Adam



  • 7.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-06-2020 11:27

    I've already updated the PoE firmware as it was one of the few things that I could do. This happened across multiple different brands of devices which have never seen this problem before on either Juniper or other switch vendors.

     

    I'll see about opening JTAC but I'm not hopeful that it'll be too fruitful.



  • 8.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

     
    Posted 01-06-2020 12:45

    I would think a TAC case where they can tell you meaning of Status = 1 and probably more importantly Status = 27.  I might 'guess' 1 is good, and 27 (like 36) may indicate what is going on.

     

    Just IMHO.



  • 9.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-07-2020 03:51

    @RCCPGM 1 is POE off and 27 is POE on flag codes. Normal Codes all is fine codes.

     

    What is discerning is the fact this isn't a vendor specific issue. This is making me thing theres a power issue into these boxes and not the boxes themselves (just my opinion).



  • 10.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-07-2020 05:22

    That's exactly my thinking and mostly what I was hoping someone would know what "Failed to read voltage measurements" meant. It's too much of a concidence for the power to be dropped at the same time as those log entries. And even more strange that it's not all of the power, just a port or two at a time.

     

    I'll see about moving some connections around and adjusting the poe enable/disable to see if that helps.

     

    And thank you all for the replies. Very welcoming for my first post here.



  • 11.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 01-07-2020 05:30

    No problem, it's why the community is great and filled with some great people. We all can't know everything.

    Please do update the thread when you find out what the issue was though haha! Sparked my curiousity!

    KR
    Adam



  • 12.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 03-25-2020 01:38

    hello Guys,

    someone found solution to this error? i have the same issue with ex3400



  • 13.  RE: EX PoE drop with error "fpc0 PoE: Failed to read voltage measurements"

    Posted 08-19-2022 10:18
    This is an interesting debug page: https://bryanward.net/wp/2020/04/09/advanced-juniper-poe-debugging/

    ------------------------------
    Dan Popescu
    ------------------------------