Buidling a Backup WAN with Tinc

First let's talk about our requirements.  If we were going to go through all this trouble we wanted something that was as functional, yet simple as possible.  We use Cisco 2800 series routers with WAAS (WAN Acceleration) modules in the routers at the remote sites.  If the trouble was simply a down WAN link we didn't want to loose WAN acceleration, so the secondary route would be via the VPN but still through the Router/WAAS combo.  But what if the router itself had malfunctioned?  We wanted to cover this issue as well so we wanted a tertiary route directly to the local layer 3 switch.

Since we wanted a full-mesh technology we had to find a mesh VPN solution.  We looked at both CloudVPN and Tinc, which are both Layer-2 SSL type VPN's that offer any-to-any connectivity, and we think either would have probably worked fine.  In the end we selected Tinc because the website seemed to show it as a more of a project with long term goals and a plan, and quite a few useful examples and complete documentation and it looked easy to setup while CloudVPN looked a little more like one guy building his own cool tool and doing with it what he wanted.  There's certainly nothing wrong with that, but we found the documentation confusing and, when we found that Tinc packages already existed in Dag's RHEL repository, we decided to go with it.

So, after picking our VPN technology we started talking about how to deploy it.  We considered deploying it on our existing edge firewall/IDS server as we had with OpenVPN, but we decided it would be better to keep it completely separate.  We use an almost 100% virtual infrastructure so we thought a small virtual machine that could be copied and deployed at the other sites with minimal changes would be a good way to do it, basically a little virtual VPN router.  We considered using a "small" distro, but in the in decided that our experience with RHEL made simply using it the easiest choice.  We performed a minimal RHEL5 install, added quagga, yum, and tinc, and ended up with a install just under 900MB.  Pretty big by small distro standards, but small enough for us.

Now we needed to build our VPN.  We deployed our new "virtual VPN router" to each of the four remote sites that utilize VPN backup.  The VM's each have two network connections, one connected to our "VPN Backup VLAN" an the other connected to the same VLAN as the internal interface on our firewall.  Both the local router and switch also have an interface on the "VPN Backup VLAN".  When we're done, sites with a Tinc VPN backup server look somthing like this:

Whenever a primary MPLS link fails (green paths) the router will use the next best path (yellow paths).  If a site experiences a router failure the local switch will use it's best path (orange paths).  Since Tinc is full mesh, even a complete loss of a site still leaves backup connectivity working for the remote sites so, for example, if you datacenter experiences a catastrophic failure, the remote sites still have backup WAN connectivity to the secondary datacenter without doing anything, even if the MPLS WAN fails at the same time (hopefully unlikely).

In our real network we have more tinc sites, as well as several sites that still rely on ISDN BRI backup.  We are able to shut every primary interface and still have full connectivity to all sites via Tinc and ISDN.  So far the setup has proven to be very robust.  We hope to add some Mobile Broadband backup links to replace the ISDN BRI backups at several remote sites in the near future.

Let me know what you think, and what methods you use for backup connectivity.