Update 2 Aug 2010 09:26: replaced two Wikipedia links.
… and how to avoid a pitfall on some Dell PowerConnect equipment. On that note, I haven’t yet managed to get this working in VMWare (Server 1.0.5, and Workstation 6.5 Beta, I believe) on Linux, using some host-only interfaces. It seems the switching mechanism gets things mixed up a bit.
So: in order to add redundancy to (say) firewalls, one could use such a protocol as VRRP. Unfortunately, Cisco has claimed patents on it, leading the OpenBSD developers to implement a similar system, called CARP.
The basics are relatively simple: instead of a primary and a secondary firewall to use their own MAC addresses for routing, we create virtual interfaces on each firewall, with corresponding MAC addresses that are used on both firewalls.
In the simple setup, only one firewall would be active at any time, meaning that you could bring down a firewall, upgrade it, and bring it back up, all without anyone noticing. Done right, the traffic is transparently routed through any of the two firewalls, and downtime is measured in seconds at the most.
Imagine a small example, where we want to replace one firewall with two network cards, with a redundant set of two similar ones; let’s say we have an external network (the Internet), and our internal network. We reserve some IP addresses for the new redundant set:
External addresses: (physical card: em0)
100.0.0.1 = Routing (CARP)
100.0.0.2 = Firewall 1 (master)
100.0.0.3 = Firewall 2 (backup)
Internal addresses: (physical card: em1)
200.0.0.1 = Routing (CARP)
200.0.0.2 = Firewall 1 (master)
200.0.0.3 = Firewall 2 (backup)
(In practice you may not need individual addresses on all interfaces with CARP, but let’s work with this for now.)
Addresses for pfsync: (assuming a crossover cable here, so basically any unique addresses can be used)
10.0.1.2 = Firewall 1
10.0.1.3 = Firewall 2
Note: pfsync does not keep track of the rules in pf.conf, only the actual user sessions registered on the firewalls.
We’ll need to edit numerous files in /etc on each firewall:
pf.conf – add:
pass quick on { em2 } proto pfsync keep state (no-sync)
pass on { em0 em1 } proto carp keep state
sysctl.conf – uncomment:
net.inet.carp.preempt=1
This ensures that if even one interface on a firewall fails, the second firewall (if intact) will take over.
hostname.em0 – add:
inet 100.0.0.2 255.255.255.0
to the first firewall, and:
inet 100.0.0.3 255.255.255.0
to the second; similarly for hostname.em1 use 200.0.0.2 and 200.0.0.3.
hostname.em2 – add:
inet 10.0.1.2 255.255.255.0
to the first firewall, and:
inet 10.0.1.3 255.255.255.0
to the second.
hostname.carp0 – add:
inet 100.0.0.1 255.255.255.0 vhid 1
to the first firewall, and:
inet 100.0.0.1 255.255.255.0 vhid 1 advskew 100
to the second; similarly for hostname.carp1 on both machines, but with 200.0.0.1 and vhid 2 instead. It is interesting to note that we don’t have to say that carp0 corresponds to em0, and carp1 to em1; OpenBSD can figure this out from the IP addresses and netmasks.
hostname.pfsync0 – add:
up syncif em2
on both machines. We can also get this running live without rebooting. Let’s kick in the backup node first: (it is assumed here that pf is already enabled and has rules loaded into memory, and also packet forwarding is in use on both firewalls)
firewall-2 # sysctl -w net.inet.carp.preempt=1
firewall-2 # sh /etc/netstart
firewall-2 # ifconfig carp
You should see that all CARP interfaces display MASTER, since firewall-1 isn’t up yet.
firewall-1 # sysctl -w net.inet.carp.preempt=1
firewall-1 # sh /etc/netstart
firewall-1 # ifconfig carp
Again, you should see that all CARP interfaces display MASTER (and, obviously, BACKUP on firewall-2), since it’s advskew is lower (0) than on the backup node (100). Note: setting the advskew to higher values than 240 will probably not work. You’ll see what I mean. 😎
Fun thing to try on another machine, when failing over from one firewall to another:
watch traceroute 100.0.0.1
There are some other things I could go into, but let me just make a note on Dell equipment:
As I mentioned, fail-over from one firewall to another is supposed to take seconds as the most (depending on advbase (default: 1) and advskew), even if you disconnect cables randomly from one firewall, cut the power to the machine, or attack it with a raygun. As soon as one firewall stops talking to the other on one of the interfaces, the second one will prepare to act in its stead.
Why, then, doesn’t it work on some Dell PowerConnect equipment, for instance? Well, it seems the RSTP mechanism gets confused by the shared MAC address, resulting in having (say) two master nodes, and the situation may take as much as 30 seconds to stabilize, if not more (or if at all).
The answer is that you should be able to turn it off for the ports where the firewalls are connected. Considering a case where both firewalls are connected to the same switch (using a different VLAN for each interface, for instance), it should be safe to do so.
Happy fail-overs!
IT and politics 