Multi-site encrypted backbone: the DMVPN mesh

Our datacenters in Athens, Amsterdam and Toronto, our management hub in Cyprus and our regional points of presence are joined by a Cisco DMVPN overlay: encrypted tunnels with dynamic spoke-to-spoke paths and BGP routing on top. It carries management, monitoring, backup and inter-site traffic, and the same technology is used to build dedicated site-to-site networks for customers who need their own offices connected to their hosted platform.

How we build it

Hub routers in each datacenter terminate multipoint GRE tunnels protected by IPsec (AES-256, SHA-256), with NHRP authentication and BFD for fast failure detection. BGP, not a static table, runs over the tunnels, so a hub or circuit failure converges by route withdrawal with no reconfiguration at the spokes. Every datacenter has a redundant hub: standby hubs sit in their own autonomous system with an outbound de-preference, so they take over only when the primary is genuinely gone and hand back when it returns.

Sites with two WAN circuits run a second DMVPN cloud on the same router. The tunnel key separates the two overlays on one interface, so the second circuit needs no new public address, no new firewall rule and no change on the hubs. Both clouds share one inner routing instance, and BGP multipath spreads flows across both circuits with a hash that includes ports, so parallel flows to the same destination genuinely split instead of piling onto one link.

Spokes are Cisco hardware routers wherever the traffic justifies it; measured side by side on identical traffic, a hardware router does an order of magnitude more forwarding work per unit of CPU than a virtual one. We size against the crypto engine’s real ceiling, about 11 Gbit/s of encrypted throughput per hub, and when a customer needs more we add hubs and let ECMP carry the site to 30 or 40 Gbit/s.

Every change to the mesh follows the same discipline as the rest of our network: configuration backed up on the device and off it before the change, one command at a time, one half of a redundant pair at a time, verified from the far end before the other half is touched.

What it protects you from

  • A single WAN circuit or carrier failure isolating a site: dual-circuit ECMP with automatic BGP convergence keeps the site reachable.
  • Management sessions, monitoring and backup copies crossing the public internet unencrypted.
  • A “flaky VPN” blamed on the wrong party: we instrument the path and can show where the loss is, hop by hop.
  • A management-plane problem becoming a customer problem: customer traffic never depends on the management overlay.

In practice

A customer’s entire site-to-site estate moved between countries ahead of a hard deadline. A legal-services firm’s hub, its branch-office routers and its remote-user access were migrated by cloning the hub into the new country and bringing it up in parallel with the old one. Every branch re-registered on its own, with no change at the branch end, and the old hub was retired once the new one carried the traffic.

Four virtual routers replaced by one hardware router, live. A hardware router took over a live DMVPN spoke identity and a live transit session in two phases, without touching the sibling circuit that carried production the whole time. Latency before and after was identical and nothing went down in either phase.

Both circuits carrying traffic, proven. On a dual-circuit site, a throughput test over the dual-cloud design measured more than either physical port could carry alone: the proof that the second circuit is a working path, not a standby that fails when it is finally needed.

For your own network

The same design is available as a product: a dedicated encrypted mesh between your offices and your hosted platform, with your own routing, your own hubs in our datacenters, redundant circuits where you need them, and our engineers watching it. Private connectivity for customers is covered in its own page; this is the technology underneath it.

Talk to an engineer