m4nifest0-tech

Search the site

Storage and backup systems

Backup and disaster recovery strategy

Backup plan based on the 3-2-1 rule, with Proxmox Backup Server, a second copy on a NAS, an off-site copy, and periodic test restores.
ActiveUpdated on

Objective

Be able to restore virtual machines, containers, and data after a failure, a human error, or ransomware, with recovery objectives defined per service class.

Architecture

Workloads (VMs and containers) are backed up to a local Proxmox Backup Server, which is the first copy. A second copy lives on a NAS, so on a different medium. The third copy is sent to an off-site destination, preferably encrypted. Retention, data verification, and test restores are scheduled.

3-2-1 backup with Proxmox Backup ServerVirtual machines and LXC containers are backed up by scheduled jobs to a local Proxmox Backup Server, which is the first copy. The data is then copied to a NAS, on a different medium, and finally synchronized to an off-site or cloud destination, shown as a dashed line because it is the off-site copy.ProductionCopy 1Copy 2Copy 3Virtual machineLXC containerProxmox Backup Serverlocal and deduplicatedNASon a different mediumOff-site destinationoff-site, encryptedbackup jobbackup jobsecond copyoff-site sync
  • Virtual machine is connected to Proxmox Backup Server (backup job)
  • LXC container is connected to Proxmox Backup Server (backup job)
  • Proxmox Backup Server is connected to NAS (second copy)
  • NAS is connected to Off-site destination (off-site sync), logical link
3-2-1 backup with Proxmox Backup Server. Virtual machines and LXC containers are backed up by scheduled jobs to a local Proxmox Backup Server, which is the first copy. The data is then copied to a NAS, on a different medium, and finally synchronized to an off-site or cloud destination, shown as a dashed line because it is the off-site copy.Solid line: direct connection. Dashed line: logical or secondary connection.

Details

Principles

  • 3-2-1 rule: at least three copies of the data, on two different media, one of them off-site. Here: Proxmox Backup Server as the local copy, the NAS as the other medium, and an off-site destination.
  • Recovery objectives: for each service class I define how much data can be lost (RPO) and how long I can wait (RTO); frequency and retention follow from these.
  • Test restores: a backup that has never been tested is only a hypothesis, so test restores are scheduled and logged.
  • Separate credentials: access to the backups is distinct from production access, to limit the damage of a compromise.
  • Disaster recovery: a written procedure sets out the restore order, from the basic dependencies (network, DNS, authentication) to the application services.

The reference design is in 3-2-1 backup strategy with Proxmox Backup Server and a NAS.

Current setup

  • Critical services (e.g. VPN gateway, internal DNS): RPO ~20 minutes thanks to frequent ZFS replication, RTO on the order of minutes through the cluster's HA manager.
  • Standard services: daily backup to Proxmox Backup Server (RPO 24 hours), RTO of about half an hour for a full VM restore.
  • Non-critical data: weekly RPO, restore without tight time constraints.
  • Off-site destination: an encrypted copy, synchronized periodically to storage kept away from the main site.
  • Retention: daily backups for 14 days, weekly for 2 months, monthly for 12 months (rotating "grandfather-father-son" scheme).
  • Test restores: one test restore per month, chosen on rotation among the services, with the result logged.