Problems faced
- A single host was a single point of failure: a hardware problem stopped every virtual machine.
- Updates and reboots required stopping the whole environment, with maintenance windows that were hard to schedule.
- Without shared storage, virtual disks were not available on a second node in case of failure.
- Without well-defined quorum and fencing, losing the link between nodes could cause the same VM to start in two places (split-brain).
Implemented solution
The cluster consists of three Proxmox VE nodes connected by a network dedicated to cluster services, separate from virtual machine traffic. Each node uses a local ZFS pool and the critical VMs are replicated asynchronously to the other nodes. The HA manager watches the registered resources and, if a node fails, restarts them on a surviving node from the replicated disks. Majority quorum and watchdog-based fencing prevent a VM from being active on two nodes at the same time. Live migration makes it possible to drain a node before maintenance.
Results
- A node failure no longer requires urgent intervention: the protected VMs restart automatically on another node.
- Node updates are done on a rotating basis, moving the VMs first with live migration and without stopping services.
- The maximum amount of data lost after a failover is known and manageable, because it depends on the replication interval chosen for each VM.
- Recovery procedures for losing a node or quorum are written and tested, not improvised.
Technical description
This project describes the three-node virtualization cluster in my homelab, designed to tolerate the failure of a single server without immediate manual intervention.
Architecture
The cluster consists of three Proxmox VE nodes with equivalent hardware. Each node has two distinct networks: one for management and virtual machine traffic, connected to the managed switch, and one dedicated to cluster services and replication. With three nodes, majority quorum remains valid even when one stops: the two remaining nodes keep governing the cluster.
Configurations are distributed by the Proxmox VE cluster file system, while the VM disks live in local ZFS pools. Asynchronous replication copies the disks of the protected machines to the other nodes at regular intervals.
Components and choices
- Corosync handles cluster membership and quorum. It is sensitive to latency: if the same network also carries replication and migrations, it is worth adding a second backup link and limiting the replication bandwidth.
- ZFS with replication was preferred over shared or distributed storage to keep complexity and hardware requirements down. The trade-off is that after a failover, the data written since the last replication is missing.
- HA manager registers the resources to protect and restarts them on a surviving node. Watchdog-based fencing prevents the same VM from staying active on an isolated node.
ha-manager add vm:100 --state started
pvesr create-local-job 100-0 nodo2 --schedule "*/15"Note
ZFS replication is asynchronous: the interval chosen for each VM determines how much recent data can be lost after a failover. Workloads that cannot tolerate any loss require shared or distributed storage.
Testing and acceptance
Before considering the project done, I tested the following: an abrupt power-off of a node, checking that the protected VMs restart automatically; temporary loss of the cluster network; live migration to drain a node before an update; the node rejoining and replication resuming. The recovery procedures for losing quorum are written down, along with periodic backups of the node configurations.
Possible next steps
Possible developments include: an external vote (QDevice) for two-node environments, distributed storage such as Ceph to eliminate data loss on failover, integration with Proxmox Backup Server, and automation of VM creation. How quorum works is explored further in Three-node Proxmox VE cluster: setup, quorum, and first checks.