How to test a restore, not just a backup

Every backup console in the world can show you a column of green ticks. Not one of them can tell you whether your business will be running again by lunchtime after a bad night. Only a restore can tell you that, and the only restore you can trust is one you have already done. This is how to test restores so the result means something.

What a green job actually proves

A successful backup job proves that a job ran, read some data and wrote it somewhere without an error the software recognised. It does not prove that the data is complete, that the machine you care about was in the job, that the copy can be read back, or that the application will start from it.

There are three things to test: coverage (is everything that should be backed up actually backed up), recoverability (can the copy be read and started) and time (how long a recovery really takes). Most organisations test none of them on a schedule.

Step one: check coverage against reality

Backup jobs are defined at a point in time. Infrastructure keeps changing. A new virtual machine created next to the old ones, a database moved to a new host, a server renamed: each one can fall outside every job without a single error being raised.

So compare the list of what exists with the list of what is protected. Export the inventory from the place that knows what is running, the hypervisor, the cloud platform or the DCIM, and the list of protected objects from the backup system, then compare them:

sort -u running-inventory.txt > running.sorted
sort -u protected-objects.txt > protected.sorted
comm -23 running.sorted protected.sorted   # running, but not protected
comm -13 running.sorted protected.sorted   # protected, but no longer running

The first list is the dangerous one. The second is clutter that hides it. Every entry on the first list needs a decision: add it to a job, or write down that it deliberately has no backup and why.

Step two: restore at more than one level

Recoverability has levels, and each level catches problems the one below it misses.

File level. Restore a sample of files to an alternate location and compare them with checksums against the source or a known good copy. This catches unreadable media and broken chains of incremental backups.

Machine level. Boot a whole server or virtual machine from backup. This catches missing boot volumes, drivers and the disk that was quietly excluded from the job. Most enterprise backup platforms, Veeam among them, can start a machine straight from the backup repository into an isolated lab for exactly this purpose.

Application level. Prove the application works from the restored copy. For a database, that means it starts, passes its own consistency check, and contains the newest records you expect for the time of the backup:

# MySQL or MariaDB
mysqlcheck --all-databases --check
mysql -e "SELECT MAX(created_at) FROM orders;"

# PostgreSQL
pg_amcheck --all
psql -c "SELECT max(created_at) FROM orders;"

For file-level backup tools there are integrity checks that read the stored data back, for example borg check --verify-data on a Borg repository.

Step three: restore into a lab, not into production

A restored machine is a copy of production, with production's addresses, credentials, scheduled jobs and outgoing mail. Bring it up on the real network and it can answer requests meant for the live server, send customers a batch of duplicate emails, or run last night's billing job a second time.

Restore tests therefore belong on an isolated network with no route to production and no route to the internet. Before you start any application in that lab, disable the scheduler and outbound mail on the restored copy. Make it a written step in the test, so nobody has to remember it.

Step four: time it

The recovery time on paper is an estimate. The one that counts is measured: from the moment someone decides to restore, through finding the right restore point, moving the data, starting the services and checking the application, to the first successful real transaction. Large restores are often limited by throughput nobody measured, or by a step that only one person knows how to do. Timing the whole sequence finds both.

Step five: write it down

Keep a short entry for each one:

• Date, and who ran it.
• What was restored, from which restore point.
• Which level was tested: file, machine or application.
• Time taken, end to end.
• Anything that failed, surprised you or needed knowledge that is not written down.
• What was changed as a result.

The design underneath matters too

Testing proves the backups exist and work today. The design decides whether they will still exist after a bad day. The rule that matters most is that backups are pulled. The backup system reaches into production and takes what it needs; production has no network path to the backup system and no credentials for it. Then an intruder who owns a production server cannot reach that server's own backups to delete them. Append-only access for file-level tools adds a second lock, and history on the repository means a copy from before the problem still exists.

How we do it

Our backups are pull-only by design and enforced on the firewall appliances, with only the management network able to reach backup and recovery systems. We use Veeam for virtual machines, physical servers and SQL servers, and a deduplicating backup hub that pulls Linux file and database data using restricted, append-only keys. Coverage is checked against the live inventory, schedules are set from measured run times, and full restores are run as often as weekly.

If you have never timed a full restore of your most important system, that is the one to start with, and we are glad to help you plan it.

Talk to an engineer

Related: Disaster recovery and cross-site replicas · Storage replication between Athens and Amsterdam · NAS file services, NFS and SMB