Data Center

 View Only
last person joined: 10 days ago 

Ask questions and share experiences about Data Center Architecture and approaches.

  • 1.  proxy-macip-advertisement: What is the purpose?

    Posted 12-15-2024 00:10

    Hi everyone,

    I am trying to understand what problem  proxy-macip-advertisement  knob solves.  I have read several juniper links but  none of them make any sense to me.

    I built a small CRB topology, where  EX2/EX3 are layer 2 gateways while EX1 is a spine with RR enabled.

    Can anyone explain what is the point of using proxy-macip advertisement in below set up?

    Much appreciated!!

    Configuration:

    E1(S1):

    E2(L2):

    L3 ( E3):

    W



    ------------------------------
    Be kind!!
    ------------------------------


  • 2.  RE: proxy-macip-advertisement: What is the purpose?

    Posted 12-15-2024 00:30

    It looks like  this feature is useful in muri homed site as per below link, I do not understand why juniper recommends using it for CRB case 

    https://www.ietf.org/archive/id/draft-rbickhart-evpn-ip-mac-proxy-adv-02.html



    ------------------------------
    Be kind!!
    ------------------------------

    Original Message


  • 3.  RE: proxy-macip-advertisement: What is the purpose?
    Best Answer

    Posted 12-19-2024 01:17

    That's right, it's recommended to use for CRB Designs when the Spines serve as the Layer 3 gateway.  Basically, it's a way for the spines to synchronize their ARP tables and make sure they have the same EVPN Type-2 MAC+IP Routes.

    Let's say you have a basic two leaf and two spine multihomed design with one host behind each leaf (h1 and h2).  Both spines are running virtual-gateway across.  

    If h1 and h2 are in different subnets, then h1 will ARP for the default gateway when trying to communicate with h2.  Leaf1 hashes this traffic and forwards it across the link to spine1.  The leaf1, running Layer 2 only, has advertised an EVPN Type 2 MAC only route to spine1.  Traffic hits spine1 then goes to h2.  Now when h2 responds, it could take the path to spine2.  Without the proxy-macip-advertisement feature enabled, spine2 would have to re-ARP back to h1 causing a small delay in traffic.

    What happens when it is enabled, is that spine1 now knows that h1 lies behind leaf1 and generates a Type 2 MAC+IP route on behalf of leaf1 pointing to leaf1's VTEP.  So now spine2 has the MAC+IP route and can forward directly to leaf1 without having to re-ARP to learn h1's IP Address.  Although layer 2 only leafs can snoop ARP packets and create ARP entries plus generate a Type 2 MAC+IP route even when running only layer 2.


    Original Message


  • 4.  RE: proxy-macip-advertisement: What is the purpose?

    Posted 12-20-2024 17:40

    thanks jsullivan!!

    I have to read it real slow as it contains a lot of useful info.



    ------------------------------
    Be kind!!
    ------------------------------

    Original Message


  • 5.  RE: proxy-macip-advertisement: What is the purpose?

    Posted 12-20-2024 23:27
    Edited by LEEBAHI 12-20-2024 23:27

    Hi Jsullivan,

    Thanks so much for responding. Below I have quoted you for easy reference:

    f h1 and h2 are in different subnets, then h1 will ARP for the default gateway when trying to communicate with h2.  Leaf1 hashes this traffic and forwards it across the link to spine1.  The leaf1, running Layer 2 only, has advertised an EVPN Type 2 MAC only route to spine1.  Traffic hits spine1 then goes to h2.  Now when h2 responds, it could take the path to spine2.  Without the proxy-macip-advertisement feature enabled, spine2 would have to re-ARP back to h1 causing a small delay in traffic.

    But question becomes why L1 is able to  send BGP MAC/IP ( h1 mac/ip) to S1 but not to S2? In the lab testing , I do not see that. What I see L1 sends BGP MAC/IP update to S1 and S2 ( via RR),  both S1 and S2 are able to create arp caches using MAC/IP without using this proxy mac/ip advertisement.

    Much appreciated!!



    ------------------------------
    Be kind!!
    ------------------------------

    Original Message


  • 6.  RE: proxy-macip-advertisement: What is the purpose?

    Posted 12-21-2024 10:25

    L1 does send to both Spine 1 and Spine 2.  I just included Spine 1 because in my example, that is the Spine that answered the ARP request for the default gateway.

    Also in my scenario,  the leaf only sends a MAC Type 2 route to both spines, not a MAC+IP Type 2 route since the leaf is running layer 2 only.  So even if Spine 2 has the MAC Type 2 route, it doesn't have the IP address associated with it; hence the re-ARP that happens when the reply traffic from h2 is hashed toward Spine 2.

    This example assumes that the Layer 2 only leafs cannot send MAC+IP Type 2 routes when the IRBs are moved to the spines.  When the proxy-macip-advertisement option was first introduced, that was the original use case.  However, in time as more features were added we now know that the Leafs can snoop the ARP packets toward the Spines and build/advertise MAC+IP Type 2 routes even when running in Layer 2.  There are still more nuanced use cases though for the command, but they are more convoluted.

    If you're interested in really getting to the nuts and bolts of all this, a great book on Juniper and EVPN VXLAN is Deploying Juniper Data Centers with EVPN VXLAN by Aninda Chatterjee.  It goes into a great level of detail for all the typical deployments, and the labs are fairly easy to setup and follow along.  It also goes over all these extra knobs that aren't really explained in the Juniper KB.  Once I got about halfway through the book, everything just started clicking.


    Original Message


  • 7.  RE: proxy-macip-advertisement: What is the purpose?

    Posted 12-25-2024 19:20
    Edited by LEEBAHI 12-31-2024 22:46

    I just got the book and it is very great book. But it did not address the why Juniper still recommends using  Proxy MAC/IP in CRB design.

    (With all respect and credit to ChatterJee  the author of this great book)

    Chapter 5 page #255 Section:

     Historical (and Present Day) Relevance of proxy-macip-advertisement:

     With time and new enhancements, these L2-only leafs were able to snoop ARP packets inbound and create an ARP entry in 
    the micro-kernel of the PFE. This allowed such leafs to generate EVPN Type-2 MAC+IP routes directly without having the 
    need for a Layer 3–capable device to proxy for them. This rendered the use of the proxy-macip-advertisement configura
    tion option obsolete, for this particular use case.

    Then author explains a use case where we can still use this knob as explained below:

    Since these L2-only leafs are snooping ARP packets, they maintain a MAC-to-IP binding in the micro-kernel as well, which 
    has an aging timer associated to it. Once this timer expires, these leafs re-ARP for the destination. This is not a problem in 
    and of itself. The problem is that for re-ARP, the leafs use a source MAC address that belongs to an interface called pip0, and 
    this MAC address is unique to every device.

     This ARP reply can get hashed toward leaf2 or leaf3. If it gets hashed toward leaf3, this is a problem—since this pip0 inter
    face MAC address is unique per device, leaf3 does not have this MAC address in its table, which leads to a destination MAC 
    miss for this ARP reply, as shown in Figure 7-15

     Since this is a MAC address miss in leaf3’s MAC address table, the ARP reply gets unknown unicast flooded in the fabric, 
    using ingress replication, as shown in Figure 7-16. At a small scale, this is irrelevant, but when you consider large-scale data 
    centers with hundreds of thousands of hosts, this is lot of unnecessary flooding of traffic, potentially impacting the network 
    negatively.

     To get past this issue, the proxy-macip-advertisement configuration option continues to be a recommendation on CRB spines. 
    With this configuration in place, the spines associate an aging timer against the ARP entries that are installed via EVPN 
    Type-2 MAC+IP routes as well. Once the timer expires, the spines re-ARP for the destination, and when the destination host 
    responds with an ARP reply, the leafs refresh their timer as well, circumventing this problem.

    ####################################################################

    The whole point is to  mitigate ingress replication when arp entry expires on  Leaves  in multihomed case as explained by the author above

    But it  does not help mitigate at all:

    When arp entry on  spine do age aout, it forces spine to do ingress replication for arp request so we still end up with ingress replication when arp entry ages out on spine, so the problem is shifted form leaves to spine using this knob.

    I appreciate anyone from Juniper to share their thoughts.

    Much appreciated!!



    ------------------------------
    Be kind!!
    ------------------------------

    Original Message


  • 8.  RE: proxy-macip-advertisement: What is the purpose?

    Posted 03-13-2025 12:21

    I'm going to take this straight out of "Deploying juniper Data Centers with EVPN VXLAN"  by ANINDA CHATTERJEE

    The configuration option proxy-macip-advertisement has some interesting history, which is important to understand because
    its use has changed over time, and the reason it is used today differs from why it was used earlier. Let's go back in time,
    shall we?
    This command was first released in 2015–2016, and its use was to fix a simple problem that most centralized routing solutions have: syncing state between redundant gateways. Back then, there was no ARP suppression, and the L2-only leafs could
    generate only EVPN Type-2 MAC routes (no EVPN Type-2 MAC+IP routes were generated). The EVPN Type-2 MAC+IP
    route is significant because it allows Layer 3–capable VXLAN devices to build their ARP tables and prepopulate MAC+IP
    bindings. Lack thereof means that the ARP cache can only be built using actual inbound ARP packets in the data plane.
    Let's consider an example of host h1, connected to leaf1, in a CRB fabric, as shown in Figure 7-9. Once the host comes
    online, leaf1 learns its MAC address (from a packet like GARP), installs it in the MAC address table against the interface
    and VLAN on which the learn occurred, and then generates an EVPN Type-2 MAC route that is sent to both spines via
    BGP EVPN. 

    The locally learned MAC address in the MAC address table of leaf1 is shown in Example 7-38, while the BGP EVPN
    advertisement from the leaf to both spines is shown in Example 7-39, using the show route table bgp.evpn.0 advertisingprotocol bgp [neighbor-address] operational mode command.
    Both spines install this address in their respective MAC address tables, with leaf1 as the destination VTEP behind which this
    host resides. Their ARP caches, however, remain empty because there is no information about which IP address is bound to
    this MAC address (due to the lack of an EVPN Type-2 MAC+IP route).
    (Notice we have the VTEP address but not the host address as author is pointing out)
    When host h1 wants to communicate with host h2, it ARPs for its default gateway, which is the virtual IP address,
    10.100.100.254, configured on both spines. This ARP packet will be hashed to one of the two available links-toward
    either spine1 or spine2. If the link toward spine1 is chosen, spine1 will receive this ARP request, as shown in Figure 7-10.
    Once it is processed, spine1 will respond with an ARP reply. At the same time, it will add this MAC-to-IP binding in its
    ARP table. 
    Since spine2 never received this ARP request, it has no knowledge of the MAC-to-IP binding for host h1, and thus its ARP
    cache does not have this entry, as confirmed in Example 7-41, using the show arp hostname [ip-address] operational mode
    command.
    While in this state, it is possible for return traffic from host h2 to get hashed to spine2. Since spine2 does not have an ARP
    entry for host h1, it must resolve it by generating an ARP request. This causes a small loss in traffic forwarding while the
    ARP process completes. This problem can be fixed by syncing ARP state between the spines, and this is exactly why the configuration option proxy-macip-advertisement was introduced.
    Considering the same situation as before, when spine1 receives this ARP request and builds its ARP cache for host h1, with
    proxy-macip-advertisement configured, it generates an EVPN Type-2 MAC+IP route on behalf of the original VTEP behind
    which the host resides, which is leaf1 in this case. This EVPN Type-2 MAC+IP route is shown in Example 7-42.
    (My note:  notice the 10.100.100.1 in the route-distinguisher is now present)
    When spine2 receives this route, it processes it and installs a MAC-to-IP binding in its ARP table for host h1, pointing to
    leaf1 as the VTEP, as shown in Example 7-44.
    With time and new enhancements, these L2-only leafs were able to snoop ARP packets inbound and create an ARP entry in
    the micro-kernel of the PFE. This allowed such leafs to generate EVPN Type-2 MAC+IP routes directly without having the
    need for a Layer 3–capable device to proxy for them. This rendered the use of the proxy-macip-advertisement configuration option obsolete, for this particular use case.
    However, there is another use case in which this command is needed, and thus it continues to be a recommended configuration option on CRB-enabled spines. Consider a multihoming scenario in which host h2 is multihomed to leaf2 and leaf3 via
    EVPN multihoming (ESI LAG), as shown in Figure 7-12.
    Since these L2-only leafs are snooping ARP packets, they maintain a MAC-to-IP binding in the micro-kernel as well, which
    has an aging timer associated to it. Once this timer expires, these leafs re-ARP for the destination. This is not a problem in
    and of itself. The problem is that for re-ARP, the leafs use a source MAC address that belongs to an interface called pip0, and
    this MAC address is unique to every device.
    The packet capture presented in Figure 7-13 shows a re-ARP originated from such a L2-only leaf (leaf2, in this case).
    When host h2 responds, the ARP reply is a unicast packet with a destination MAC address of the pip0 interface, as shown in
    This ARP reply can get hashed toward leaf2 or leaf3. If it gets hashed toward leaf3, this is a problem-since this pip0 interface MAC address is unique per device, leaf3 does not have this MAC address in its table, which leads to a destination MAC
    miss for this ARP reply
    Since this is a MAC address miss in leaf3's MAC address table, the ARP reply gets unknown unicast flooded in the fabric,
    using ingress replication, as shown in Figure 7-16. At a small scale, this is irrelevant, but when you consider large-scale data
    centers with hundreds of thousands of hosts, this is lot of unnecessary flooding of traffic, potentially impacting the network
    negatively.
    (Side Note here is the important part)
    To get past this issue, the proxy-macip-advertisement configuration option continues to be a recommendation on CRB spines.
    With this configuration in place, the spines associate an aging timer against the ARP entries that are installed via EVPN
    Type-2 MAC+IP routes as well. Once the timer expires, the spines re-ARP for the destination, and when the destination host
    responds with an ARP reply, the leafs refresh their timer as well, circumventing this problem.
    Side note: This is saying with this knob enabled leaf3 will have the pip0 mac synchronized of leaf2. hence this knob seems to be relevant in multihomed layer 2 only leaf designs, with centrally routed b ridigng.


    ------------------------------
    HARES FAKOOR
    ------------------------------

    Original Message


  • 9.  RE: proxy-macip-advertisement: What is the purpose?

    Posted 03-13-2025 12:21

    this knob is used to synchronize mac addresses.

    Il old juniper software 2015-2016 ip address not in Type2 route advertisements. With this knob we had an host ip address associated with host mac address advertised.

    However newer software this is not a problem.

    Why use it?

    In Multi homed layer 2 leaf designs with centrally routed bridging

    host2 is connected to leaf 2 and leaf3

    leaf2 ARP's to host2. Host2 can reply to leaf2 or leaf3.

    if it replies to leaf3, the leaf3 doesn't have the source interface ip address (pip0 software address)of leaf2original arp in its database.

    With the proxy knob host3 now has this in its database and this avoids flooding to look for leaf2s source interface of the original arp



    ------------------------------
    HARES FAKOOR
    ------------------------------

    Original Message