A Proxmox VE cluster brings several nodes under a single management layer: shared configuration, virtual machine migration, replication and high availability. At the base sit Corosync, which keeps the list of reachable nodes, and the quorum, which decides which part of the cluster may keep making decisions. The procedure applies to Proxmox VE 8 (Debian 12) and 9 (Debian 13) and uses generic names and addresses reserved for documentation.
Prerequisites
A cluster is hard to fix after it has been created, so get the basics right before the first command.
- A network dedicated to Corosync. Corosync is sensitive to delays: the official documentation calls for latency below 5 ms between nodes and recommends at least one physical NIC reserved for the main link. If the network is shared with backups, storage or VM traffic, a spike can cost you the quorum. UDP 5405-5412 and TCP 22 must be open between the nodes.
- Synchronized clocks. On every node
timedatectlmust report the clock as synchronized. - Stable names and addresses. Set the final hostnames and IPs before creating the cluster: the documentation rules out changing them afterwards.
- Matching versions. All nodes should run the same Proxmox VE version.
- No guests on a node when it joins. A joining node cannot hold VMs or containers: the IDs could conflict, and the configuration in
/etc/pveis replaced by the cluster's. Guests you want to keep are saved withvzdumpand restored after the join with a different ID.
Tip
A VLAN or a physical network reserved for Corosync avoids competition with data traffic. The reference design is in Segmenting the network with VLANs.
Creating the cluster
On the first node
The cluster name is unique and cannot be changed later. The --link0 option gives the address of nodo1 on the Corosync network.
pvecm create lab-cluster --link0 198.51.100.11On the other nodes
Each node joins by giving the address of an existing member and its own address on the Corosync network. The command asks for the root password of the existing node and, on the first connection, for confirmation of the certificate fingerprint. It is best to go one node at a time, waiting for the previous one to show up in the cluster.
pvecm add 192.0.2.11 --link0 198.51.100.12pvecm add 192.0.2.11 --link0 198.51.100.13First checks
On any node pvecm status shows the quorum state and pvecm nodes lists the members.
pvecm status
pvecm nodesCluster information
-------------------
Name: lab-cluster
Config Version: 3
Transport: knet
Votequorum information
----------------------
Expected votes: 3
Total votes: 3
Quorum: 2
Flags: Quorate
Membership information
----------------------
Nodeid Votes Name
0x00000001 1 198.51.100.11 (local)
0x00000002 1 198.51.100.12
0x00000003 1 198.51.100.13The cluster is healthy when Total votes matches the number of nodes, Quorate appears among the flags and all members are listed. If not, journalctl -b -u corosync and systemctl status corosync usually point to a network or clock problem.
Quorum: why you need three votes
Each node has one vote. Operations that change the cluster state (starting a guest, changing the configuration, migrating) are allowed only to the partition that holds the majority, meaning more than half of the votes. With two nodes you cannot tell a failed peer from a broken link: if both kept working, the configuration would diverge (split-brain) and a guest on shared storage could start twice. The majority rule prevents two partitions from operating at the same time.
| Nodes | Votes for quorum | Nodes you can lose |
|---|---|---|
| 2 | 2 | 0 |
| 3 | 2 | 1 |
| 4 | 3 | 1 |
| 5 | 3 | 2 |
Three is therefore the minimum that tolerates a failure, and a fourth node does not increase the tolerance. Without quorum /etc/pve becomes read-only: guests that are already running keep running, but you cannot start others or change configurations. The exception is resources managed by high availability, where the isolated node can be rebooted by the watchdog (fencing).
QDevice for two-node clusters
With two nodes, losing one blocks the cluster. A QDevice adds an external vote, provided by corosync-qnetd on an independent host: with three votes in total the cluster tolerates the failure of one node. The Proxmox VE documentation recommends it for two-node clusters and advises against it with an odd number of nodes, so it is not needed in the three-node cluster of this guide.
apt install corosync-qnetdapt install corosync-qdevicepvecm qdevice setup 203.0.113.10You need root SSH access to the external host (the setup copies the cluster key there; password access can stay enabled only for the duration of the operation) and TCP port 5403 must be reachable. The host must not depend on the cluster it arbitrates: a VM hosted on the same nodes would defeat the purpose. Before adding or removing nodes the QDevice must be removed with pvecm qdevice remove; after the setup, pvecm status shows the Qdevice flag and three total votes.
Emergency recovery with pvecm expected
If fewer nodes remain than needed, for example two out of three lost permanently, the survivor loses quorum and no longer lets you manage guests. The following command sets the expected votes to 1 and makes the local node operational.
pvecm expected 1Warning
The command disables the protection against split-brain. Use it only after verifying, physically or through out-of-band management, that the other nodes are powered off. If they were running and merely isolated from the network, two partitions would change the configuration independently and the same guest could start twice on the same storage, risking data corruption.
The emergency ends in one of two ways: the lost nodes come back and the quorum is restored, or you remove them permanently with pvecm delnode nodo3, after verifying that they are powered off and cannot reappear on the network with the old configuration. A removed node is reinstalled before it is reused.
What not to do
- Rename a node or change its address after joining. The name and IP appear in the Corosync configuration, in certificates and in SSH keys, and the official documentation rules the operation out. The safe alternative is to remove the node, reinstall it with the correct data and join it again.
- Join a node that hosts guests that have not been saved. Its local configuration is overwritten.
- Edit
/etc/pve/corosync.confby hand. If you must, work on a copy, incrementconfig_version, keep a backup and apply the file only once the change has been checked. - Shut down two nodes out of three for maintenance. The cluster loses quorum.
- Use
pvecm expectedas a shortcut. It is an emergency procedure, not a maintenance feature.
Before enlarging or changing the cluster, make sure you have verified backups, as described in 3-2-1 backup strategy with Proxmox Backup Server, and monitoring that flags the loss of quorum, for example with Prometheus and Grafana in an LXC container.