Pull-only backup: why production should never reach its own backups

Most backup designs are built for the failure everyone pictures: a dead disk, a deleted table, a broken upgrade. The failure that actually ends companies is different. Someone gets onto a production server, finds the backup credentials sitting right there, and deletes the backups before encrypting everything else. This article explains how to design backups that survive that day, using the pattern we build to: the backup side pulls, production never pushes, and the keys that create backups cannot destroy them.

The problem with push

The simplest backup is a cron job on the server that copies data to a backup target. It is easy to set up, and it carries a structural flaw. To push, the production server needs a route to the backup target and credentials that can write to it. Very often those credentials can also delete, overwrite or prune, because retention has to happen somewhere and the client is the obvious place.

So the server you are protecting holds the keys to its own protection. Anyone who owns that server owns its backups. Ransomware operators know this, and the first thing a competent intrusion does is look for backup agents, mounted backup shares and SSH keys pointing at storage hosts.

Mounting a backup share on the production host is the worst form of this. A share that is writable from production is, from the attacker's point of view, just more disk to encrypt.

Turn the arrow around

The fix is to reverse the direction of trust. The backup system initiates every connection. It reaches into production, takes what it needs and leaves. Production has no network path to the backup infrastructure and holds no credentials for it.

Enforce this where it cannot be undone from a compromised host: on the firewall appliance between the two zones, not in the host firewall of the backup server. The rule set is short:

permit ip <management-subnet> host <backup-host>   ! admin access only
deny   ip any host <backup-host>                    ! production and everyone else
permit ip any any                                   ! leave other traffic alone

The backup host itself needs outbound access to production on the ports it uses to collect data, and nothing inbound except from management. If a connection from a production host still succeeds, the design is not in force, whatever the diagram says.

This is a data-diode pattern in spirit, and it maps cleanly to the controls auditors ask about: segregation of backup infrastructure, least privilege and boundary protection in ISO/IEC 27001, NIS2 and NIST SP 800-53.

Least privilege on the production side

Pulling still means the backup host logs in to production. That login must be useless for anything except taking a backup. With SSH this is straightforward: give the backup host's key a forced command and the restrict option in authorized_keys on the production server.

command="rrsync -ro /srv/www",restrict ssh-ed25519 AAAA... backup-files
command="/usr/local/bin/backup-dump",restrict ssh-ed25519 AAAA... backup-db

The first key can only read the web root through rrsync in read-only mode. The second can only run a dump wrapper you wrote, which takes a consistent database dump (for MySQL or MariaDB, from a replica or a node not taking writes, with --single-transaction or a physical tool such as Percona XtraBackup) and streams it to standard output. Neither key gives a shell, port forwarding or agent forwarding. If the backup host itself were compromised, the attacker would gain read access to one customer's data, not control of production.

Use separate keys for separate jobs, and separate keys for separate customers. Blast radius should be one job on one customer.

Make the repository append-only

The second half of the design is what happens on the backup side. With BorgBackup, the deduplicating, compressing, encrypting archiver, a repository can be accessed in append-only mode: new archives can be written, but existing data cannot be deleted or rewritten through that access path.

When Borg itself has to run on the source (for example, because a web root holds millions of small files and you want deduplication to happen before data crosses the network), you can still keep the hub in charge. The hub opens the SSH session to the production host and provides a reverse tunnel back to itself, and the account at the far end of that tunnel is locked down on the hub:

command="borg serve --append-only --restrict-to-repository /srv/backup/tenants/<customer>/borg",restrict ssh-ed25519 AAAA... customer-node

The production host never initiates anything; it only talks back through the tunnel the hub opened, and only to its own repository, in append-only mode.

Retention then becomes a separate, local job on the hub:

borg prune --keep-daily 7 --keep-weekly 4 --keep-monthly 6 /srv/backup/tenants/<customer>/borg
borg compact /srv/backup/tenants/<customer>/borg

Run it as a privileged job on the backup host itself, never through the path production can see. An attacker who encrypts a customer's server can, at worst, write more archives. They cannot remove the old ones.

One repository, one key, one customer

On a multi-customer backup platform, give every customer its own repository and its own passphrase, with encryption enabled when the repository is created:

borg init --encryption=repokey-blake2 /srv/backup/tenants/<customer>/borg

Isolation then holds at the cryptographic layer, not only in file permissions: one customer's data cannot be read with another customer's key, even by someone with a shell on the hub. Deduplication still works well inside a customer, and across a customer's own near-identical web nodes the second node costs almost nothing.

Prove it, do not trust it

A green backup job tells you the job ran. It does not tell you that every machine that matters was covered, or that the data comes back. Three habits close that gap.

1. Check coverage against the live inventory. List what is running, list what has a fresh backup on disk, and compare. Anything running without a recent restore point is a finding.
2. Watch freshness, not status. Alarm on "hours since the last good archive" per customer, so a job that silently stops running is noticed quickly.
3. Restore on a schedule. Extract files, restore a database into a scratch instance and check it, regularly. Borg makes this cheap:

borg list /srv/backup/tenants/<customer>/borg
borg extract --dry-run /srv/backup/tenants/<customer>/borg::<archive>
borg check --verify-data /srv/backup/tenants/<customer>/borg

Restores also follow the pull-only rule. The backup host, or an engineer, delivers restored data to production. Production is never given access to the repository, not even "just for the restore".

What to check on your own platform

• Can any production server open a connection to your backup storage? Try it.
• Do any credentials on a production server allow deletion or pruning of backups?
• Is there a backup share mounted, writable, on any production host?
• Are backup keys restricted to a single command, or do they give a shell?
• Is each customer or system in its own encrypted repository with its own key?
• When did you last restore something, and did you check the result?

If any answer makes you uncomfortable, start with the firewall rule. It is the cheapest control on this list and it removes the most dangerous path first.

If you would like a second pair of eyes on your backup design, our engineers are happy to help. Talk to an engineer

Related: Managed backup on Veeam and Borg, pull-only · Disaster recovery and cross-site replicas