Two circuits, one router: dual-cloud DMVPN that uses both links

Most branch sites with two WAN circuits use one of them. The second sits as a backup, paid for every month and exercised only when the first one fails, which is also the first time anyone finds out whether failover really works. With DMVPN and BGP you can run both circuits all the time, fail over between them without thinking, and prove both are carrying traffic. Here is the method.

A quick refresher on DMVPN

We run BGP over DMVPN (mGRE, NHRP, IPsec), so adding or losing a site is a routing event rather than a reconfiguration.

The common design for redundancy is two hubs on one cloud. That protects against a hub failing. It does nothing for a spoke with two circuits, because the spoke's single tunnel interface sources from one of them.

The dual-cloud design

The fix is a second, independent DMVPN cloud, sourced from the second circuit. The usual objection is that it means new public addresses, new firewall rules and new hub configuration. It does not have to, if you separate the clouds with a tunnel key. GRE carries the key in its header, so the hub can terminate two multipoint tunnels on the same physical interface and the same public address and still tell them apart.

On the hub, the second cloud is a copy of the first with its own tunnel subnet, NHRP network ID and key:

interface Tunnel1
 ip address 10.10.1.1 255.255.255.0
 ip nhrp network-id 1
 ip nhrp authentication <secret>
 ip nhrp redirect
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 1
 tunnel protection ipsec profile DMVPN-PROF shared
!
interface Tunnel2
 ip address 10.10.2.1 255.255.255.0
 ip nhrp network-id 2
 ip nhrp authentication <secret>
 ip nhrp redirect
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 2
 tunnel protection ipsec profile DMVPN-PROF shared

The shared keyword on the protection profile matters when two tunnels share a source: without it the IPsec security associations of the two tunnels collide.

On the spoke, Tunnel1 sources from the first circuit and Tunnel2 from the second, each pointing at the same hub address with its own key. Nothing changes for any other spoke.

Make the routing use both

Two tunnels give you two BGP sessions to the hub, and therefore two paths to every destination. By default BGP still picks one. Two settings change that:

router bgp 65010
 address-family ipv4
  maximum-paths 2
  maximum-paths ibgp 2

Both sessions should land in the same routing table so the paths are genuinely equal. Then make sure the forwarding hash spreads flows rather than whole source and destination pairs. A hash that includes layer 4 ports puts parallel flows between the same two hosts on different circuits, which is what you want for backups and replication:

ip cef load-sharing algorithm include-ports source destination

Prove it, do not assume it

show ip route listing two next hops proves that multipath is configured. It does not prove traffic is using both circuits. The only acceptable proof is a throughput test with several parallel flows that achieves more in each direction than either circuit could carry on its own, while you watch the counters on both physical interfaces climb together:

iperf3 -c <far-end> -P 8 -t 60
iperf3 -c <far-end> -P 8 -t 60 -R
show interfaces GigabitEthernet0/0/1 | include rate
show interfaces GigabitEthernet0/0/2 | include rate

If one interface does all the work, look at the hash and at whether both BGP paths are really installed.

Size from the crypto engine, not the licence

Every router has a licensed throughput figure and a real encryption ceiling, and the two are not the same. The ceiling depends on the crypto hardware, packet sizes and how much else the router is doing. Measure it with realistic traffic, not the best-case packet size from a datasheet, and size against what you measured.

On our backbone that measured figure is about 11 Gbit/s of encrypted traffic per hub. When a customer needs more between sites, we do not wait for a bigger box: we spread traffic across several hubs with equal-cost multipath, which takes the encrypted capacity to 30 to 40 Gbit/s. The same principle that makes a branch use both its circuits makes a backbone use all its hubs.

We use virtual routers for quiet sites and hardware routers for busy ones.

Redundant hubs that stay out of the way

A standby hub should be quiet until it is needed. Give it its own BGP AS or a deliberate outbound de-preference so spokes prefer the primary, and use BFD on the tunnels so a dead primary is detected in seconds rather than at the BGP hold timer. Spokes then fail over by routing alone. Nobody touches a branch router because a hub changed.

A checklist for your own WAN

• Does every site with two circuits use both, and can you show it with a throughput test?
• Are the two clouds separated by tunnel key, with shared IPsec protection?
• Is BGP multipath enabled and is the forwarding hash including ports?
• Is BFD running on every tunnel?
• Do you know your measured encryption ceiling per hub, not the licensed one?
• Would a hub failure require you to touch a single spoke?

We run this design across our datacenters in Athens, Amsterdam and Toronto, with redundant hubs in each, and build the same thing for customers who want their own offices joined to their platform. If you would like help designing yours, we are happy to talk.

Talk to an engineer

Related: Multi-site encrypted backbone: the DMVPN mesh · Private connectivity: VPN gateways and leased lines · Network monitoring and evidence