Switching

Expand all | Collapse all

EX4300: Framing error with macsec enabled

Jump to Best Answer
  • 1.  EX4300: Framing error with macsec enabled

    Posted 04-19-2020 01:34

    Dear experts,
    I've an EX4300 (Junos 17.3R3-S3.3) which have a constant Framing error counter increase also if the traffic is very low.
    Interface is connected to a WAN link from a carrier and bw is 1 Gbs but traffic max is actually 100 Mbs and on average 10 Mbs.
    On this interface I've enabled macsec, if I disable macsec the issue is not in place but unfortunately macsec is mandatory to be kept enabled.

    I cannot sniff since the packet is encrypted but to me it seems that traffic is not lost, if I have 100 Mbs inside from WAN I see 100 Mbs outside to DC.

    Due to the fact that monitoring system contantly raise an alert, I'd like to know how to fix it or at least let the EX4300 do not raise the counter increase.

    I've opened a JTAC case but they found a PR which is currently related to a Broadcom chipset raising framing errors during spikes (ie 70% of the interface bandwidth).

     

    https://kb.juniper.net/InfoCenter/index?page=content&id=KB32264&actp=METADATA

     

    Also enabling flow-control as described in the KB do not change the behaviour.

    I'm wondering if there could the option we're receiving some sort on "unknown protocol" from the carrier (I remeber Cisco has something like that) or could be an harware issue..

    On the other side of the link, the other EX4300 (side B) do not experience the same issue but the traffic is mostly from side B to side A.

    Here an example of the output, statistics cleared and after 1 minute I get 12 framing errors with 2 Mbs running on the WAN link:

     

    @EX4300-A> show interfaces ae0 extensive
    Physical interface: ae0, Enabled, Physical link is Up
    Interface index: 220, SNMP ifIndex: 549, Generation: 131
    Description: xxx
    Link-level type: Ethernet, MTU: 9192, Speed: 1Gbps, BPDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None,
    Loopback: Disabled, Source filtering: Disabled, Flow control: Enabled, Minimum links needed: 1, Minimum bandwidth needed: 1bps
    Device flags : Present Running
    Interface flags: SNMP-Traps Internal: 0x0
    Current address: cc:e5:94:11:43:23, Hardware address: cc:e5:94:11:43:23
    Last flapped : 2020-04-19 02:05:05 CEST (06:50:45 ago)
    Statistics last cleared: 2020-04-19 09:11:22 CEST (00:01:00 ago)
    Traffic statistics:
    Input bytes : 10014863 2205456 bps
    Output bytes : 4095720 582456 bps
    Input packets: 33292 624 pps
    Output packets: 33023 568 pps
    IPv6 transit statistics:
    Input bytes : 0
    Output bytes : 0
    Input packets: 0
    Output packets: 0
    Input errors:
    Errors: 12, Drops: 0, Framing errors: 12, Runts: 0, Giants: 0, Policed discards: 0, Resource errors: 0
    Output errors:
    Carrier transitions: 0, Errors: 0, Drops: 0, MTU errors: 0, Resource errors: 0
    Egress queues: 12 supported, 11 in use


    @EX4300-A> show interfaces ge-0/0/0 extensive
    Physical interface: ge-0/0/0, Enabled, Physical link is Up
    Interface index: 649, SNMP ifIndex: 509, Generation: 140
    Description: WAN link
    Link-level type: Ethernet, MTU: 9192, LAN-PHY mode, Link-mode: Full-duplex, Speed: 1000mbps, BPDU Error: None, Loop Detect PDU Error: None,
    Ethernet-Switching Error: None, Source filtering: Disabled
    Ethernet-Switching Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Flow control: Enabled, Auto-negotiation: Enabled,
    Remote fault: Online, Media type: Copper, IEEE 802.3az Energy Efficient Ethernet: Disabled, Auto-MDIX: Enabled
    Device flags : Present Running
    Interface flags: SNMP-Traps Internal: 0x0
    Link flags : None
    CoS queues : 12 supported, 12 maximum usable queues
    Hold-times : Up 0 ms, Down 0 ms
    Current address: cc:e5:94:11:43:23, Hardware address: cc:e5:94:11:43:23
    Last flapped : 2020-03-28 18:43:04 CET (3w0d 13:30 ago)
    Statistics last cleared: 2020-04-19 09:11:18 CEST (00:02:18 ago)
    Traffic statistics:
    Input bytes : 21782579 932296 bps
    Output bytes : 17898068 498704 bps
    Input packets: 76844 569 pps
    Output packets: 82594 590 pps
    IPv6 transit statistics:
    Input bytes : 0
    Output bytes : 0
    Input packets: 0
    Output packets: 0
    Input errors:
    Errors: 28, Drops: 0, Framing errors: 28, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0,
    L2 mismatch timeouts: 0, FIFO errors: 0, Resource errors: 0


    Here part of the config:

    @EX4300-A> show configuration interfaces ge-0/0/0 | display set
    set interfaces ge-0/0/0 ether-options auto-negotiation
    set interfaces ge-0/0/0 ether-options flow-control
    set interfaces ge-0/0/0 ether-options 802.3ad ae0


    @EX4300-A> show configuration interfaces ae0 | display set
    set interfaces ae0 mtu 9192
    set interfaces ae0 aggregated-ether-options flow-control
    set interfaces ae0 aggregated-ether-options lacp active
    set interfaces ae0 aggregated-ether-options lacp periodic fast
    set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
    set interfaces ae0 unit 0 family ethernet-switching vlan members 2228
    set interfaces ae0 unit 0 family ethernet-switching vlan members 2552-2553
    set interfaces ae0 unit 0 family ethernet-switching filter input QOS


    @EX4300-A> show configuration security macsec | display set
    set security macsec connectivity-association MAC security-mode static-cak
    set security macsec connectivity-association MAC pre-shared-key ckn xxxx
    set security macsec connectivity-association MAC pre-shared-key cak "tttttvvvv"
    set security macsec connectivity-association MAC exclude-protocol lldp
    set security macsec connectivity-association MAC exclude-protocol lacp
    set security macsec interfaces ge-0/0/0 connectivity-association MAC


    Dear all, an help is appreciated and welcomme, please let me thank in advance anyone will give an hint.

    Cheers


    #macsec
    #framingerror


  • 2.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-19-2020 05:18
    Hi Rambo,

    Framing errors in general are physical layer (like CRC errors) however if you're not seeing the errors without Macsec, I'm certain this is because the switch will drop packets coming in without the Macsec header. You might still be able to confirm this with a sniffer, to find why and which packets are coming in without the header. As I believe it's the ethernet data that's encrypted and it might not encrypt the header as such.

    Also, although I don't think this is related to your case but will mention just in case you need to rule it out before or after the above is checked: https://kb.juniper.net/InfoCenter/index?page=content&id=TSB17538

    It'll benefit the reader if you do come back and mention which of the above two resolves.

    Hope this helps.

    Regards,
    -r.

    --------------------------------------------------

    If this solves your problem, please mark this post as "Accepted Solution."
    Kudos are always appreciated :).


  • 3.  RE: EX4300: Framing error with macsec enabled

    Posted 04-19-2020 07:48

    Hi mryaz,

    for the second point my gear is EX4300-24T hence I've no sfp connected to the link which is delivered with native copper interface.

     

    Regarding first point, you suspect that if I go to sniff I will see non macsec frames which are currently the ones letting the counter to increase, do I understand in the right way ?

     

    Thanks in advance and regards



  • 4.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-19-2020 08:47
    @rambo wrote:

    Hi mryaz,

    for the second point my gear is EX4300-24T hence I've no sfp connected to the link which is delivered with native copper interface.

     

    Regarding first point, you suspect that if I go to sniff I will see non macsec frames which are currently the ones letting the counter to increase, do I understand in the right way ?

    [ANS] Yes your understanding is correct. 

     

    Thanks in advance and regards

     

    Hope this helps.

     

    Regards,
    -r.

    --------------------------------------------------

    If this solves your problem, please mark this post as "Accepted Solution."
    Kudos are always appreciated :).



  • 5.  RE: EX4300: Framing error with macsec enabled

    Posted 04-19-2020 09:40

    According to monitoring traffic it seems that only EAPOL traffic and LLDP (which is out of macsec) is receveid:

     

    @EX4300-A# run monitor traffic interface ge-0/0/0 brief
    verbose output suppressed, use <detail> or <extensive> for full protocol decode
    Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay.
    Address resolution timeout is 4s.
    Listening on ge-0/0/0, capture size 96 bytes

    18:33:35.709180 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:35.714528 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:37.770524 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:37.775166 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:39.809132 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:39.830914 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:41.838233 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:41.897531 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:43.858173 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:43.984499 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:45.886173 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:46.013486 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:47.910151 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:48.052481 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:49.934144 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:50.094498 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:51.965410 In LLDP, name EX4300-B, length 60
    [|LLDP]
    18:33:51.992193 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:52.136473 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:54.057460 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:54.181461 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:56.081183 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:56.213456 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:58.099074 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:33:58.262459 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:34:00.162127 In cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    18:34:00.302447 Out cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:



  • 6.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-19-2020 14:06

    If you run the same test on the Side B (no errors) do you see both EAPOL and LLDP?  I would have no idea why your SP would be sending you EAPOL/801.2x frames, . . .  LLDP fine, the other ???



  • 7.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-19-2020 05:26

    Framing errors are also caused by physical issues with the link as well.  These are the most common starting points to verify it is not physical layer.

    • both sides correctly showt the link as the gig full duplex
    • swap the interconnect cable
    • move the physical port on each side one at a time

    If any of these changes clears the issue that physical component would be at fault.

     

    Your counts are very low so what ever the issue is present it is subtle.  Typically with framing errors I tend to see numbers in the hundreds and thousands pretty quickly on active links.

     

    Once you are sure it is not physical, are there any differences in the config between the two sides?

    You don't seem to be hitting the Broadcom PR to me based on the traffic requirement.  So if this is a software PR other data would be needed to see the trigger.

     

    Is the partner device on the other side of the link the same make and model?

    There may be some subtle issue between the carrier device and the switch if the models are not the same operating system on both sides.  That difference could play into the error rate.

     



  • 8.  RE: EX4300: Framing error with macsec enabled

    Posted 04-19-2020 08:06

    Hi Puluka,

    the two macsec devices are EX4300-24T with same junos version on both sides, but in the middle there are carrier metroethernet equipments.

    Both EX4300-A and EX4300-B in both side are configured equivalent but I've no clue about the carrier devices.

    EX4300-B do not report issues, EX4300-A reports around 35-40 framing errors per minute, as far as I understood on side B the carrier have different local provider from side A and they peer with an NNI somewhere in the middle, also this can be the difference.

     

    I'm quite sure it's something cosmetic, but I still cannot demostrate...

     

    We've already changed the cables and port (ge-0/0/1 instead of ge-0/0/0) but the trend is the same...

     

    Thanks for your help and kind regards 



  • 9.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-19-2020 21:36

    Hi rambo,

     

    I'd still insist this could be due to packets coming in out side of "macsec".  Could you please correlate the framing error count with the packets outside of Macsec (EAPOL etc.).  To track that live for correlation, you can use the CLI "show interfaces ge-0/0/0 extensive | grep Framing | refresh 1" in one SSH session to the device while capturing "monitor traffic interface <> no-resolve" on another parallel SSH session.

     

    And agree with @rccpgm, EAPOL frames aren't expected here.  Please contrast if the working EX4300 side sees EAPOLs like that, really doubt it though.

     

    Hope this helps.

    Regards,
    -r.

    --------------------------------------------------

    If this solves your problem, please mark this post as "Accepted Solution."
    Kudos are always appreciated :).



  • 10.  RE: EX4300: Framing error with macsec enabled

    Posted 04-20-2020 00:09

    Hi all
    here the same monitoring on the other switch-B:

     

    @EX4300-B> monitor traffic interface ge-0/0/0 brief no-resolve
    verbose output suppressed, use <detail> or <extensive> for full protocol decode
    Address resolution is OFF.
    Listening on ge-0/0/0, capture size 96 bytes
    08:56:58.574586 In cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:56:58.614521 Out cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:00.595588 In cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:00.677481 Out cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:02.637024 In cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:02.736449 Out cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:04.690531 In cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:04.764485 Out cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:04.840322 In LLDP, name EX4300-A, length 60
    [|LLDP]
    08:57:06.709532 In cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:06.837481 Out cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:08.775553 In cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:08.897468 Out cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:09.440767 Out LLDP, name EX4300-B, length 60
    [|LLDP]
    08:57:10.820513 In cc:e1:94:d2:43:23 > 01:80:c2:00:00:03, EAPOL, length 74:
    08:57:10.933484 Out cc:e1:94:d2:9a:e3 > 01:80:c2:00:00:03, EAPOL, length 74:

     

    I'm wondering why your're surprised about EAPOL, which are macsec pdu ?

     

    I've only LLDP and LACP outside of macsec as depicted in original post:

     

    @EX4300-A> show configuration security macsec | display set
    set security macsec connectivity-association MAC security-mode static-cak
    set security macsec connectivity-association MAC pre-shared-key ckn xxxx
    set security macsec connectivity-association MAC pre-shared-key cak "tttttvvvv"
    set security macsec connectivity-association MAC exclude-protocol lldp
    set security macsec connectivity-association MAC exclude-protocol lacp
    set security macsec interfaces ge-0/0/0 connectivity-association MAC

     

    I'd like to try to filter CDP on ingress (maybe the carrier is sending me?), I need to find out the best way to do it but I'm not sure if the filter will act before the counter to increase...

     

    Thanks for your help
    Cheers



  • 11.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-20-2020 03:58

    Hello rambo,

     

    This will likely need some collaboration with your Carrier/Provider.  Good thought to filter out suspected packets if you have an idea of the header fields (src/dst mac etc.), try to apply the filter as input at the interface level.  Yea I doubt if it'll still catch it before the "framing counters" are populated.  Ideally sniffing the packets and monitor the counters so you can compare/match the diff/counters for a fixed time is more likely to help nail this along with the Carrier/Provider.

     

    Indeed EAPOL is the macsec pdu, please ignore comments about that assumed to be the 802.1x packets, sorry about that.

     

    Hope this helps.

     

    Regards,
    -r.

    --------------------------------------------------

    If this solves your problem, please mark this post as "Accepted Solution."
    Kudos are always appreciated :).

     



  • 12.  RE: EX4300: Framing error with macsec enabled

    Posted 04-20-2020 13:08

    Unfortunately if I go to sniff the traffic with a port analyzer config I suprisely see traffic already decrypted and nothing particular which hits every 2-3 seconds.... I attach the pcap of around 45 seconds...

     

    Thanks



  • 13.  RE: EX4300: Framing error with macsec enabled

    Posted 04-20-2020 13:14
      |   view attached

    pcap attached

    Attachment(s)

    zip
    MACSEC-ae0.zip   745K 1 version


  • 14.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-21-2020 06:18

    @rambo - I assume you saw the response from Chuck A on Juniper-NSP about turning off LLDP.  I think you want to run both LLDP and LACP outside the MACSEC tunnel, not inside.

     

    I should have thought about that earlier, . . .

     https://www.juniper.net/documentation/en_US/junos/topics/task/configuration/macsec.html

     

    (Optional) Exclude a protocol from MACsec:

    [edit security macsec connectivity-association connectivity-association-name]
    user@switch# set exclude-protocol protocol-name

    For instance, if you did not want Link Level Discovery Protocol (LLDP) to be secured using MACsec:

    [edit security macsec connectivity-association ca-dynamic1]
    user@switch# set exclude-protocol lldp

    When this option is enabled, MACsec is disabled for all packets of the specified protocol—in this case, LLDP—that are sent or received on the link.

     
    BEST PRACTICE

    We recommend that any protocol other than MACsec being used on the MACsec connection, such as LLDP, LACP, STP, or layer 3 routing protocols, should be excluded and moved outside of the MACsec tunnel.



  • 15.  RE: EX4300: Framing error with macsec enabled

    Posted 04-23-2020 04:58

    Hi

    this is actually our case, lldp and lacp are out of macsec encryption.

     

    Thanks



  • 16.  RE: EX4300: Framing error with macsec enabled

    Posted 04-22-2020 19:10

    Hello,

     

    I'm from ATAC, I did some quick search in our database and found a PR that is currently under investigation related to framing errors when mac sec is enable. As per the description, the problem is seeing when there is high utilization in the interface around 70% of the bw.

     

    I would suggest you to open a case to confirm if it is the same problem.

     

    If this solves your problem, please mark this post as "Accepted Solution".

     

     

     

     



  • 17.  RE: EX4300: Framing error with macsec enabled

    Posted 04-23-2020 05:02

    Hi

    facing our configuration if I go to span the family ether-switch port (ae0) I obtain to see traffic after decryption, I was expecting to see traffic before encryption and this was useful to see if I get some traffic of control protocol recevied by the carrier.

    Our issue is not related to the mentioned PR since also with 1% of the traffic on the link we hit continuos framing error increment (1 every 4-6 seconds).

    Is there a way to dump on the EX4300 the traffic before decryption ?

    Thanks for your help



  • 18.  RE: EX4300: Framing error with macsec enabled

     
    Posted 04-23-2020 05:28

    @rambo - did you try and filter out LLDP from the ISP or get them to stop?

     

    For capture traffic prior to hitting EX4300 I think only method would be to add in a tap or a switch/hub, and then mirror that traffic prior to it getting to EX4300.

     

    Otherwise turn of MACSEC (both ends) mirror and then re-enable MACSEC, . . .



  • 19.  RE: EX4300: Framing error with macsec enabled

    Posted 04-24-2020 04:13

    It seems I was able to find some spurious traffic coming from 00:02:ab:0e:28:91 broadcast destinated and with unknown 

    ethertype 0x1021 which is Private, Protocol unavailable.

     

    Macaddress owner seems to be CTC Union Technologies Co.

     

    I'm not sure how I can filter it, I guess only the carrier can do it.

    Thanks



  • 20.  RE: EX4300: Framing error with macsec enabled
    Best Answer

    Posted 04-26-2020 00:58
    For info, the issue was related to a carrier mediaconverter sending frames with a private unknown ethertype not decoded by EX4300.

    Thanks for your help all

    Cheers


  • 21.  RE: EX4300: Framing error with macsec enabled

    Posted 06-04-2020 10:28

    Can you tell me what type of media converter it was and what had to be done to remove those packets?  I have a customer having a similar issue, but we are unable to get the link running using MACSec.  As soon as they enable MACSec the carrier starts seeing errors (as well as our switches on both sides) and packets get dropped to the point that OSPF can't come up between routers behind each of the switches at the sites.



  • 22.  RE: EX4300: Framing error with macsec enabled

    Posted 06-04-2020 11:55
    It is a fiber to copper from Accedian.
    Anyway on us macsec is working properly. I suggest to check if the carrier tunnels all the needed ethertype.

    Regards