Switching

Expand all | Collapse all

QFX5100 to Cisco UCS 40GB QFSP+ Wont Come Up?!?!?! HELP

Jump to Best Answer
  • 1.  QFX5100 to Cisco UCS 40GB QFSP+ Wont Come Up?!?!?! HELP

    Posted 09-23-2020 15:23

    I need the communities help desperately.  I have a QFX5100 connected directly to a Cisco UCS Fabric Interconnect 6332 via 40gb QFSP+ cable.  I have no link light, zilch, nadda but the QFX sees the gbic and it shows up in show interface terse.  If I loop the cable back on the Cisco, it works like a champ.  Same with the Juniper, if its looped back on itself it works fine.  I tried turning off auto-negotiation but no joy.  According to the docs I read, it doesn't appear that one can set port duplex, speed, etc.  If I use a 40gb to (4) 10gb break out cable, that works but I really wanted native 40gb and not a port-channel.  Can anyone point me in the right direction?  I read where somewhere posted about padding the light by 5db on the QFX, but I haven't a clue how to do that.  Unfortunately, the post had commands on how to do that.  Any ideas? 



  • 2.  RE: QFX5100 to Cisco UCS 40GB QFSP+ Wont Come Up?!?!?! HELP

    Posted 09-23-2020 15:55

    Hi Csnow,

     

    Sounds weird, we should maybe check the code and confirm support, also what is the QFX5100 model you are using and which ports are you using to try to bring up the 40gig QSFP ?

     

    could you share the '> show chassis hardware' also the config under [chassis] stanza. also check for the '> show interface extensive et-x/x/x ' and cehck if you can spot any issue.

     

    also get the 'show interfaces terse', get the commands with the links connected

     

    maybe this helps:

     

    https://www.juniper.net/documentation/en_US/junos/topics/topic-map/switches-interface-port.html

     

    Cheers,

     

    Benjamin



  • 3.  RE: QFX5100 to Cisco UCS 40GB QFSP+ Wont Come Up?!?!?! HELP

    Posted 09-23-2020 17:49

    Thanks Benjamin.  Here is the chassis hardware output.  The cables are Cisco brand but I am using the same cable for the VC interface.  I also tried a different brand as well with same result.  I am running the latest version of code.

     

    telecom@Austin_DC_Core> show chassis hardware
    Hardware inventory:
    Item Version Part number Serial number Description
    Chassis TA3717440427 Virtual Chassis
    Routing Engine 0 BUILTIN BUILTIN RE-QFX5100-48S-6Q
    Routing Engine 1 BUILTIN BUILTIN RE-QFX5100-48S-6Q
    FPC 0 REV 18 650-064381 TA3717440427 QFX5100-48S-6Q
    CPU BUILTIN BUILTIN FPC CPU
    PIC 0 BUILTIN BUILTIN 48x10G-6x40G
    Xcvr 0 NON-JNPR TED1834L190-CH1 SFP+-10G-CU3M
    Xcvr 1 NON-JNPR TED1834L190-CH2 SFP+-10G-CU3M
    Xcvr 2 NON-JNPR TED1841L571-CH3 SFP+-10G-CU3M
    Xcvr 3 NON-JNPR TED1841L571-CH4 SFP+-10G-CU3M
    Xcvr 4 REV 01 740-021308 07T511100506 SFP+-10G-SR
    Xcvr 46 REV 01 740-021308 07T511100901 SFP+-10G-SR
    Xcvr 48 NON-JNPR JPC22010155-B QSFP+-40G-CU1M
    Xcvr 53 NON-JNPR JPC22170EHE-B QSFP+-40G-CU1M
    Power Supply 0 REV 04 740-044332 1GA17350566 JPSU-650W-AC-AFI
    Power Supply 1 REV 04 740-044332 1GA17352451 JPSU-650W-AC-AFI
    Fan Tray 0 fan-ctrl-0 0, Back to Front Airflow - AFI
    Fan Tray 1 fan-ctrl-0 1, Back to Front Airflow - AFI
    Fan Tray 2 fan-ctrl-1 2, Back to Front Airflow - AFI
    Fan Tray 3 fan-ctrl-1 3, Back to Front Airflow - AFI
    Fan Tray 4 fan-ctrl-2 4, Back to Front Airflow - AFI
    FPC 1 REV 18 650-064381 TA3717440368 QFX5100-48S-6Q
    CPU BUILTIN BUILTIN FPC CPU
    PIC 0 BUILTIN BUILTIN 48x10G-6x40G
    Xcvr 0 NON-JNPR TED1834L190-CH4 SFP+-10G-CU3M
    Xcvr 1 NON-JNPR TED1834L190-CH3 SFP+-10G-CU3M
    Xcvr 2 NON-JNPR TED1841L571-CH2 SFP+-10G-CU3M
    Xcvr 3 NON-JNPR TED1841L571-CH1 SFP+-10G-CU3M
    Xcvr 4 REV 01 740-021308 07T511100966 SFP+-10G-SR
    Xcvr 47 REV 02 740-013111 E213238 SFP-T
    Xcvr 53 NON-JNPR JPC22170EHE-A QSFP+-40G-CU1M
    Power Supply 0 REV 04 740-044332 1GA17350567 JPSU-650W-AC-AFI
    Power Supply 1 REV 04 740-044332 1GA17120036 JPSU-650W-AC-AFI
    Fan Tray 0 fan-ctrl-0 0, Back to Front Airflow - AFI
    Fan Tray 1 fan-ctrl-0 1, Back to Front Airflow - AFI
    Fan Tray 2 fan-ctrl-1 2, Back to Front Airflow - AFI
    Fan Tray 3 fan-ctrl-1 3, Back to Front Airflow - AFI
    Fan Tray 4 fan-ctrl-2 4, Back to Front Airflow - AFI

     

    telecom@Austin_DC_Core> show interfaces et-0/0/48 extensive
    Physical interface: et-0/0/48, Enabled, Physical link is Down
    Interface index: 663, SNMP ifIndex: 572, Generation: 156
    Link-level type: Ethernet, MTU: 1514, LAN-PHY mode, Speed: 40Gbps, BPDU Error: None, Loop Detect PDU Error: None,
    Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled, Flow control: Disabled,
    Media type: Copper
    Device flags : Present Running Down
    Interface flags: Hardware-Down SNMP-Traps Internal: 0x4000
    Link flags : None
    CoS queues : 12 supported, 12 maximum usable queues
    Hold-times : Up 0 ms, Down 0 ms
    Current address: c0:42:d0:11:1c:33, Hardware address: c0:42:d0:11:1c:33
    Last flapped : 2020-09-23 21:42:05 UTC (03:05:53 ago)
    Statistics last cleared: Never
    Traffic statistics:
    Input bytes : 0 0 bps
    Output bytes : 0 0 bps
    Input packets: 0 0 pps
    Output packets: 0 0 pps
    IPv6 transit statistics:
    Input bytes : 0
    Output bytes : 0
    Input packets: 0
    Output packets: 0
    Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0,
    L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0
    Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0,
    Resource errors: 0
    Egress queues: 12 supported, 5 in use
    Queue counters: Queued packets Transmitted packets Dropped packets
    0 0 0 0
    3 0 0 0
    4 0 0 0
    7 0 0 0
    8 0 0 0
    Queue number: Mapped forwarding classes
    0 best-effort
    3 fcoe
    4 no-loss
    7 network-control
    8 mcast
    Active alarms : LINK
    Active defects : LINK, LOCAL-FAULT
    PCS statistics Seconds
    Bit errors 0
    Errored blocks 0
    Ethernet FEC statistics Errors
    FEC Corrected Errors 0
    FEC Uncorrected Errors 0
    FEC Corrected Errors Rate 0
    FEC Uncorrected Errors Rate 0
    MAC statistics: Receive Transmit
    Total octets 0 0
    Total packets 0 0
    Unicast packets 0 0
    Broadcast packets 0 0
    Multicast packets 0 0
    CRC/Align errors 0 0
    FIFO errors 0 0
    MAC control frames 0 0
    MAC pause frames 0 0
    Oversized frames 0
    Jabber frames 0
    Fragment frames 0
    VLAN tagged frames 0
    Code violations 0
    MAC Priority Flow Control Statistics:
    Priority : 0 0 0
    Priority : 1 0 0
    Priority : 2 0 0
    Priority : 3 0 0
    Priority : 4 0 0
    Priority : 5 0 0
    Priority : 6 0 0
    Priority : 7 0 0
    PRBS Statistics : Disabled
    Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
    CoS information:
    Direction : Output
    CoS transmit queue Bandwidth Buffer Priority Limit
    % bps % usec
    0 best-effort 5 2000000000 5 0 low none
    3 fcoe 35 14000000000 35 0 low none
    4 no-loss 35 14000000000 35 0 low none
    7 network-control 5 2000000000 5 0 low none
    8 mcast 20 8000000000 20 0 low none
    Interface transmit statistics: Disabled
    MACSec statistics:
    Output
    Secure Channel Transmitted
    Protected Packets : 0
    Encrypted Packets : 0
    Protected Bytes : 0
    Encrypted Bytes : 0
    Input
    Secure Channel Received
    Accepted Packets : 0
    Validated Bytes : 0
    Decrypted Bytes : 0

    Logical interface et-0/0/48.0 (Index 570) (SNMP ifIndex 576) (Generation 159)
    Flags: Device-Down SNMP-Traps 0x24024000 Encapsulation: Ethernet-Bridge
    Traffic statistics:
    Input bytes : 0
    Output bytes : 0
    Input packets: 0
    Output packets: 0
    Local statistics:
    Input bytes : 0
    Output bytes : 0
    Input packets: 0
    Output packets: 0
    Transit statistics:
    Input bytes : 0 0 bps
    Output bytes : 0 0 bps
    Input packets: 0 0 pps
    Output packets: 0 0 pps
    Protocol eth-switch, MTU: 1514, Generation: 185, Route table: 8

     

     

     



  • 4.  RE: QFX5100 to Cisco UCS 40GB QFSP+ Wont Come Up?!?!?! HELP

     
    Posted 09-23-2020 22:10

    Is the port configured as a virtual-chassis uplink?



  • 5.  RE: QFX5100 to Cisco UCS 40GB QFSP+ Wont Come Up?!?!?! HELP

    Posted 09-23-2020 23:04

    Hi csnow,  

     

    I'd like to try 2 things if you don't mind.

     

    To start with. and, based on documentation we better disable channelized for this interface, in case you haven't done it.

    By default, the 40-Gbps QSFP+ ports on EX4600 and QFX5100 switches are channelized automatically (auto-channelized) if any of the four channels on a 40-Gbps QSFP+ port receive data, unless you have configured channelization either at the chassis level or at the port level.  

     

    # set chassis fpc 0 pic 0 port 48 channel-speed disable-auto-speed-detection  

    # commit 

     

    second:  

    Please disable FEC on both platforms, since the QFX5100 doesn’t support FEC for QSFP40g please try disable it on the cisco router.  

     

    Cisco:

    Configure the Forward Error Correction feature for the interface range.

    switch(config-if-range)# fec {auto | cl91 | off}

     

    auto—Enables the FEC feature based on the transceiver type.

    cl91—Enables clause 91 for 100 Gigabit interface.

    off—Disables FEC for the interface range.

     

    keep us posted!  

     

    if this resolved your issue please mark as solved!  

     

    thanks, 

    esmontes



  • 6.  RE: QFX5100 to Cisco UCS 40GB QFSP+ Wont Come Up?!?!?! HELP
    Best Answer

    Posted 09-29-2020 09:11

    Hi everyone,

     

    Just an update and resolution in case anyone else runs into this.  The only way I could get this to come up was with using Juniper and Cisco branded gbics on each with separate fiber. The prefabbed cables with built in gbics would not work regardless of what I tried.  With the separate gbics, I still needed to disable flow control and auto-negotiation on the Juniper side.  The Cisco 6332 Fabric Interconnects would not allow me to disable either within the Nexus OS CLI or from the gui via a link profile.  Cisco provides pathetic configurations options on the 40gb fabric interconnects. 

     

    Thanks for everyone's help.  It is finally working 🙂