A weird thing occured yesterday. Our ISP (Bredbandsbolaget) has a switch in our basement which obviously has had problems for a while. Due to these problems we’ve been offline from time to time. The fact that a ISP has problems now and then isn’t the weird thing, what is the weird thing is that when the switch in the basement has been rebooted and our connection should be up and running again my FreeBSD (pfSense installation) can’t get a new DHCP lease.

To give you a clearer view of what our infrastructure looks like I’ll give you a brief description:
We have a physical machine in our wardrobe which runs a slimmed down version of Windows XP (much like TinyXP, but I’ve built it myself using nLite). Apart from the regular stuff this machine also has an extra network card, this card is connected to our ISP and the other is connected to our internal switch. Installed on this machine is VMware Server 2 and one of the virtual machines running within it is a pfSense installation (pre-built for VMware – nice!) which we use as our network gateway and firewall. This machine is configured to use the ISP facing network card as its external interface and the internally facing network card as its internal interface (d’uh). This setup has been working nicely for quite some time now (about a month or so) and we’re getting pretty good throughput as well, I’ve measured it up to 95 Mbps!

So, now to the weird stuff. When our ISP has been offline and it goes back online our gateway won’t or can’t get a DHCP lease from the ISPs DHCP server, or atleast that’s what it looks like. Changing the MAC address in the VMware configuration of our gateway and then rebooting the gateway doesn’t solve it. Running

dhclient em0

in order to interactivelly register a DHCP lease doesn’t help either, it just times out and then states that no lease could be obtained. The only thing that seems to help is to boot up another virtual machine configured to use the same external network card (in my case this happens to be a Fedora installation) and have it use the exact same MAC address as the gateway machine. The fedora machine will successfully receive a DHCP lease and will also able to communicate with other machines on the internet. After shutting down the fedora machine and then starting the gateway the gateway will receive the same DHCP lease as the fedora machine just had and then everything works!

I haven’t had the time to look at this with a packet sniffer, it might reveal something interesting. Right now I can’t really deduce what the real problem here is:

• Is it a problem with the fact that I’m running the gateway as a virtual machine within VMware? If that would be the cause the fedora machine shouldn’t be able to get a DHCP lease either – but it does.
• Or is it some sort of problem with the combination of FreeBSD/pfSense and Bredbandsbolagets DHCP server? This is the most reasonable explanation yet, since the fedora installation has no problems at all obtaining a new DHCP lease.

This is isn’t such a big problem right now, but I imagine that when we move and our server (with all the virtual servers in it) is moved to our friends internet connection (also Bredbandsbolaget) for the duration of our relocation this might be a greater PITA than it is now because our traveltime to the server increases drastically.

Any input would of course be greatly appreciated. =)