A GitLab backup you cannot restore is just a large file

Most self-hosted GitLab instances have a nightly backup job, and most of the people who set it up have never restored from it. That is a problem, because the archive GitLab produces is deliberately incomplete. Restore it on a fresh server without the files it leaves out, and your repositories come back while your CI variables, two-factor settings and runner registrations turn into unreadable data. Here is what a complete GitLab backup contains, how to move it without weakening your security, and how to prove it works.

What the backup task includes, and what it leaves out

On a package (Omnibus) install, the backup task is:

sudo gitlab-backup create STRATEGY=copy

It writes a single tar archive to the backup directory, usually /var/opt/gitlab/backups. That archive holds the database, the Git repositories, uploads, LFS objects, CI job artifacts, pages, packages and, if enabled, the container registry. STRATEGY=copy copies data before archiving it, which avoids errors when files change during a long backup on a busy instance.

What it does not hold, on purpose:

• /etc/gitlab/gitlab-secrets.json, the keys used to encrypt sensitive columns in the database.
• /etc/gitlab/gitlab.rb, your configuration.
• TLS certificates and keys under /etc/gitlab/ssl, if you keep them there.
• The SSH host keys in /etc/ssh, which your developers' clients have already recorded.

GitLab leaves these out so that a single stolen archive does not contain both the encrypted data and the key that decrypts it. That is sound reasoning, and it puts the job of backing them up separately on you.

Why the secrets file matters so much

CI/CD variables, runner registration tokens, two-factor authentication seeds, integration passwords and several other values are encrypted in the database with keys from gitlab-secrets.json. Restore the database without the matching secrets file and GitLab starts, the repositories are there, and then pipelines fail, users with two-factor authentication cannot sign in, and runners need registering again.

Back the configuration directory up as its own artifact:

sudo gitlab-ctl backup-etc --backup-path /var/opt/gitlab/config-backups

This produces a tar of /etc/gitlab, including the secrets file. Keep it in a different place from the data archive, under stricter access, and keep matching pairs together in your retention policy. A data archive from Monday and a secrets file from Thursday may not agree with each other if anything was rotated in between.

For container installs the commands are the same, run inside the container, and /etc/gitlab is whichever volume you mapped to it. Make sure that volume is part of what you back up; it is easy to back up only the data volume.

Pull the backup, do not push it

The usual pattern is a cron job on the GitLab server that copies the archive to a backup host. That means the GitLab server holds credentials that can write to your backups. If GitLab is compromised, so is the way to reach and delete the backups.

Turn the direction around. The backup host connects to GitLab and fetches the files. GitLab holds no credentials for the backup host and has no network path to it. On the GitLab side, limit what the backup host's key can do with a forced command in authorized_keys:

command="/usr/local/bin/backup-send",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAA... backup-puller

where backup-send is a small script that does nothing except stream the newest data archive and its matching configuration archive. On the backup host, the job fetches both, checks their sizes and checksums, and alerts if either is missing or older than expected.

Then enforce the direction at the firewall: the backup host may open connections to GitLab, nothing on GitLab's network may open connections to the backup host.

Restore rules worth knowing before you need them

A GitLab backup restores only onto the same GitLab version and edition that created it. Write the version into the archive name or a sidecar file at backup time:

sudo gitlab-rake gitlab:env:info > gitlab-env-$(date +%F).txt

If you lose the server and the only installer you can find is newer, you will first install the old version, restore, and then upgrade along the supported path. Knowing that in advance turns a long night into a planned procedure.

The restore itself, on a fresh install of the matching version:

sudo gitlab-ctl stop puma
sudo gitlab-ctl stop sidekiq
# put gitlab-secrets.json and gitlab.rb back into /etc/gitlab first
sudo gitlab-ctl reconfigure
sudo gitlab-backup restore BACKUP=<archive-id>
sudo gitlab-ctl restart
sudo gitlab-rake gitlab:check SANITIZE=true
sudo gitlab-rake gitlab:doctor:secrets

The last command is the one that matters. gitlab:doctor:secrets walks through the encrypted columns and reports any it cannot decrypt. If it reports failures, your secrets file and your data archive do not belong together.

Prove it on a schedule

A restore you have not tested is a hope. Make it routine:

1. Build a disposable VM on an isolated network with no route to production.
2. Install the GitLab version recorded with the backup.
3. Restore the newest configuration and data pair.
4. Run gitlab:check and gitlab:doctor:secrets.
5. Clone a known repository, compare its latest commit hash with production, and open a project's CI variables page.
6. Record the result and the time it took, then destroy the VM.

Time it, too. The duration of the restore is your real recovery time, and it is usually longer than people guess on a large instance.

A second, independent copy of the code

The full backup protects the whole application. For the repositories themselves, a simple mirror gives you a second copy that does not depend on GitLab's tooling at all:

git clone --mirror git@gitlab.example.com:group/project.git
git -C project.git remote update --prune

Run that from the backup side for the repositories you cannot afford to lose. It will not restore issues or merge requests, but it means your source history survives even a backup archive you cannot read.

The short version

• gitlab-backup create is half a backup. gitlab-ctl backup-etc is the other half.
• Keep secrets and data in matching pairs, stored apart, with stricter access on the secrets.
• The backup side pulls; GitLab has no path to its own backups.
• Record the GitLab version with every archive.
• Restore on a schedule, and run gitlab:doctor:secrets every time.

This is how we protect the GitLab instances we run for customers in our own datacenters. If you want your own restore path checked, we are happy to help.

Talk to an engineer

Related: Managed source control on GitLab · Managed office documents on Nextcloud