SRX

 View Only
last person joined: yesterday 

Ask questions and share experiences about the SRX Series, vSRX, and cSRX.
Expand all | Collapse all

DHCP for Windows & Kubernetes Clusters

  • 1.  DHCP for Windows & Kubernetes Clusters

    Posted 02-01-2021 00:07
    Edited by Roelf Zomerman 02-01-2021 00:08
    Hi all, 

    I'm running a dhcp-local-server on my SRX340, and while it works for my clients (Windows, Linux, IOT etc..) it doesn't seem to work for Windows Clusters and Kubernetes clusters - the DHCP request itself times out. 

    When installing a Windows DHCP on the same subnet, everything works, but then I'd have 2 DHCP servers on my network which I'd rather not have.. 

    Is there any filter or option I have to enable to allow Windows Clusters to be able to receive a DHCP as well? (perhaps it's like dual MAC protection or something??) 

    I know - clusters should have fixed IP's, but Azure Kubernetes Services on Azure Stack HCI uses DHCP mostly.. 

    Roelf

    ------------------------------
    Roelf Zomerman
    ------------------------------


  • 2.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-01-2021 06:15
    On the SRX side you could enable dhcp trace options and see if there are helpful messages.
    https://kb.juniper.net/InfoCenter/index?page=content&id=KB26748

    On the windows server if you could do a packet capture on the working windows dhcp transaction.  Then see if they are using one of the dhcp option codes that could be added to the SRX configuration.  Perhaps the Windows dhcp server has some default response config that can be duplicated.

    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
    http://puluka.com/home
    ------------------------------



  • 3.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-02-2021 01:37
    Hi, 

    I did a wireshark (throug port monitor) on the traffic while I disabled the Windows DHCP - I can see the DHCP Discover, but never a reply. 

    <With SRX only enabled>
    253 3.664839 172.16.5.42 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xc99e6a6c
    416 6.787367 172.16.5.42 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xc99e6a6c
    576 9.914060 172.16.5.42 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xc99e6a6c
    1171 17.168538 172.16.5.42 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xc99e6a6c
    2146 32.669655 172.16.5.42 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0xc99e6a6c

    compared to when Windows DHCP is enabled:
    93 1.435756 172.16.5.209 255.255.255.255 DHCP 290 DHCP Inform - Transaction ID 0x659a0493
    173 3.438161 172.16.5.209 255.255.255.255 DHCP 290 DHCP Inform - Transaction ID 0x659a0493
    463 11.532693 172.16.5.42 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x991d0672
    585 14.645256 172.16.5.42 255.255.255.255 DHCP 342 DHCP Discover - Transaction ID 0x991d0672
    588 14.840335 172.16.5.209 255.255.255.255 DHCP 343 DHCP Offer - Transaction ID 0x991d0672
    589 14.840941 172.16.5.42 255.255.255.255 DHCP 354 DHCP Request - Transaction ID 0x991d0672
    590 14.843037 172.16.5.209 255.255.255.255 DHCP 348 DHCP ACK - Transaction ID 0x991d0672

    I tried enabling the DHCP trace options, but the command is not recognized ... do you have other options to get the DHCP logs?
    For the DHCP logs, I tried this:
    set system processes dhcp-service traceoptions file JDHCPDEBUG
    set system processes dhcp-service traceoptions file size 20m
    set system processes dhcp-service traceoptions file files 5
    set system processes dhcp-service traceoptions flag all

    but ended up with 
    root@GW2> show log JDHCPDEBUG
    Feb 2 10:30:31.268010 [MSTR][ERROR] jdhcpd_persistent_db_set_filename: Unable to open file (/var/preserve/jdhcp_client_data)
    Feb 2 10:30:35.858449 [MSTR][ERROR] jdhcpd_persistent_db_init: File open(/var/preserve/jdhcp_client_data) operation failed
    Feb 2 10:30:35.858527 [MSTR][ERROR] main: jdhcpd_persistent_db_init failed
    Feb 2 10:30:35.859210 [MSTR][ERROR] jdhcpd_persistent_restore: Unable to restore jdhcpd bindings as file(/var/preserve/jdhcp_client_data) is not present
    Feb 2 10:30:35.859251 [MSTR][ERROR] jdhcpd_persistent_restore_enable: jdhcpd_persistent_restore failed

    Thanks so far!

    ------------------------------
    Roelf Zomerman
    ------------------------------



  • 4.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-02-2021 05:39
    Ive ran into ODD things with SRX dhcp and posted a ER. 
    Try this set system services dhcp-local-server overrides process-inform.

    I to only had odd issues with dhcp issues on my srx . My lab linux dhcp worked 100% my office Infoblox 100%, remote sites running Palo Altos.
    So try what i posted and I hope it helps you out.  FYI I have posted a ER but got nothing back as of yet .   So if anyone reading that  can do anything about ERs..   
    Please request to have this option set  on by default  in jdhcp ! If they say no please ask them to explain why. If there is a good reason id like to know.
    Personally Id recommend to have all your customers enable this if they are running dhcp on their Juniper gear.

    My tshoot story/notes  if you care

    The odd thing about the access point. It would get its IP from my SRX just fine, Extreme ap boot up in a discovery mode WING and Identify
    1st it goes into WiNG mode if its not able to connect to its controller via dhcp options 190, 191 .. Then it sends out a dhcp inform with src 169.254.39.40 but inside the dhcp inform it posted its Client add it got on bootup . Took me a few days to get the right capture as I used a filter on the SRX ONLY looking for the ip address the AP had via dhcp 10.3.208.0/24 .. not the 169.254.39.40 ..
    I keep thinking it was a vpn / polciy issue were something was getting alg and dropped but not getting logged.


  • 5.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-02-2021 06:43
    Yeah it seems like the request is coming in from one the primary nodes running the cluster - a machine that already has an IP from the DHCP 

    the machine itself does get an IP from the Juniper DHCP - but the resources inside the cluster on that host will not.. 

    really strange.. do i need to put additional values after the override? without it it does not solve the problem..

    ------------------------------
    Roelf Zomerman
    ------------------------------



  • 6.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-02-2021 07:26
    Thats what resolved my issue.. but  here is my full   overrides just to show you.
    set system services dhcp-local-server overrides process-inform
    set system services dhcp-local-server overrides include-option-82 nak
    set system services dhcp-local-server group jdhcp-group interface irb.0
    set system services dhcp-local-server group jdhcp-phone interface irb.3000

     Im sure its a knob to adjust  somewhere..  Please take no offense to this question. Did you commit  , and did the commit not roll back ?
    also in your trace  add
    set system processes dhcp-service traceoptions level all

    That should give way more data in your trace..**if you care I wrote an update on how to get traces off box to a remote syslog server and posted to community*.

    Can you do a full pcap on the device not working via srx dhcp  and one that works when you run off other dhcp server ?
    Since i was rabbit holing myself  last time. Ive learned even when tshooting dhcp always get full pcap and filter out the display ** 2 bad days of hurt on my own time





  • 7.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-02-2021 09:09
    Edited by Roelf Zomerman 02-02-2021 09:21
    Finally managed to get the logs from Juniper. 


    Feb 2 17:59:16.979283 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP from == 172.16.5.42, port == 68 ]--
    Feb 2 17:59:16.979332 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP size == 300, op == 1 ]--
    Feb 2 17:59:16.979370 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP flags == 8000 ]--
    Feb 2 17:59:16.979409 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP htype == 1, hlen == 16 ]--
    Feb 2 17:59:16.980218 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP hops == 0, xid == edd51852 ]--
    Feb 2 17:59:16.980274 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP secs == 768, flags == 8000 ]--
    Feb 2 17:59:16.980318 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP ciaddr == 0.0.0.0 ]--
    Feb 2 17:59:16.980358 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP yiaddr == 0.0.0.0 ]--
    Feb 2 17:59:16.980398 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP siaddr == 0.0.0.0 ]--
    Feb 2 17:59:16.980437 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP giaddr == 0.0.0.0 ]--
    Feb 2 17:59:16.980495 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP chaddr == ab 05 57 f4 ba ba cb 47 b0 c7 5f fc 40 b3 86 56 ]--
    Feb 2 17:59:16.980533 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP sname == ]--
    Feb 2 17:59:16.980570 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ DHCP/BOOTP file == ]--
    Feb 2 17:59:16.980619 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ OPTION code 53, len 1, data DHCP-DISCOVER ]--
    Feb 2 17:59:16.980665 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ OPTION code 116, len 1, data 01 ]--
    Feb 2 17:59:16.980726 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ OPTION code 61, len 17, data 01 ab 05 57 f4 ba ba cb 47 b0 c7 5f fc 40 b3 86 56 ]--
    Feb 2 17:59:16.980773 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ OPTION code 12, len 4, data 42 4c 55 45 ]--
    Feb 2 17:59:16.980820 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ OPTION code 60, len 8, data 4d 53 46 54 20 35 2e 30 ]--
    Feb 2 17:59:16.980875 [MSTR][DEBUG][default:default][SVR][INET][ge-0/0/1.0] --[ OPTION code 55, len 14, data 01 03 06 0f 1f 21 2b 2c 2e 2f 77 79 f9 fc ]--
    Feb 2 17:59:16.980915 [MSTR][INFO] [default:default][SVR][INET][ge-0/0/1.0] --[ OPTION code 255, len 0 ]--
    Feb 2 17:59:16.980958 [MSTR][NOTE] [default:default][SVR][INET][ge-0/0/1.0] jdhcpd_packet_decode: packet has an invalid hardware address
    Feb 2 17:59:16.981001 [MSTR][NOTE] [default:default][SVR][INET][ge-0/0/1.0] jdhcpd_packet_handle: Packet decode failed

    When looking at the Wireshark trace, I also see: 

    Client hardware address: ab0557f4babacb47b0c75ffc40b38656

    compared to the Log
    DHCP/BOOTP chaddr == ab 05 57 f4 ba ba cb 47 b0 c7 5f fc 40 b3 86 56 

    Ethernet II, Src: Mellanox_7c:1f:72 (f4:52:14:7c:1f:72), Dst: Broadcast (ff:ff:ff:ff:ff:ff)

    So, the mac address used by the Windows Cluster resources doesn't even look like a MAC address.. hmmm

    ------------------------------
    Roelf Zomerman
    ------------------------------



  • 8.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-03-2021 02:43
    so it seems that the Widows Cluster is using a multicast MAC address to get a Unicast IP address.. something that is blocked by the DHCP of SRX... still looking for the solution, but that is the problem..

    ------------------------------
    Roelf Zomerman
    ------------------------------



  • 9.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-03-2021 05:36
    You could report it to Microsoft in their ticket system as a defect not following the standards so it might eventually be patched.

    And if you report it to Juniper support as a feature request pointing out that other vendors ignore that part of the standard and maybe they should too.

    ------------------------------
    Steve Puluka BSEET - Juniper Ambassador
    IP Architect - DQE Communications Pittsburgh, PA (Metro Ethernet & ISP)
    http://puluka.com/home
    ------------------------------



  • 10.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-03-2021 05:45
    I talked to the product group within Microsoft - and they claim its by design and is not changeable.. it seems only Juniper DHCP's have a problem with it .. 

    not sure how to report it to Juniper.. Its a home lab environment and I have no support contract

    ------------------------------
    Roelf Zomerman
    ------------------------------



  • 11.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-03-2021 07:36
    Looks like you have the how, where, what, when ..   If your contact at MS can tell you why that would be great ..

    Can you test your environment with an ISC dhcp server ? (aka install a some docker as a ISC dhcp server or spin up a vm in your home lab.)
    Just wanting to see it works with a vanilla isc dhcp server. Which im sure it can or there would be tons of links about this kinda stuff.

    Kubernetes is not in my bag of tricks  or id gladly work on this with you.

    One last request.. Would you post your access address-assignment pool config ?


  • 12.  RE: DHCP for Windows & Kubernetes Clusters

    Posted 02-04-2021 03:57
    Edited by Roelf Zomerman 02-04-2021 06:24
    I checked with ISC-dhcp on a Debian VM - and IP addressing worked for clustered resources.. so it seems to be limited to SRX

    [access]
    root@GW2# show access
    address-assignment {
        pool 172-16-5-1 {
            family inet {
                network 172.16.5.0/24;
                range r1 {
                    low 172.16.5.20;
                    high 172.16.5.150;
                }
                dhcp-attributes {
                    domain-name forestroot.local;
                    name-server {
                        172.16.5.1;
                    }
                    router {
                        172.16.5.1;
                    }
                }
            }
        }
    ​

    and the proof that the lease was given to a multicast address

    lease 172.16.5.140 {
      starts 4 2021/02/04 08:53:54;
      ends 4 2021/02/04 09:03:54;
      cltt 4 2021/02/04 08:53:54;
      binding state active;
      next binding state free;
      rewind binding state free;
      hardware ethernet ab:05:57:f4:ba:ba:cb:47:b0:c7:5f:fc:40:b3:86:56;
      uid "\001\253\005W\364\272\272\313G\260\307_\374@\263\206V";
      set vendor-class-identifier = "MSFT 5.0";
      client-hostname "BLUE";
    }
    ​

    ------------------------------
    Roelf Zomerman
    ------------------------------