Cluster Guide

Build a high-availability cluster with ovios-cluster: a primary, one or more mirrored secondaries, and a floating IP that follows the primary
OviOS Linux v6 ovios-cluster ZFS Send/Receive · mTLS Heartbeats
← Home ← Back to Docs

1 How the Cluster Works

ovios-cluster builds an active/passive cluster on ZFS. One node is the primary: it serves your data and holds the floating virtual IP (VIP). The other nodes are replicas, which follow the primary and hold a mirror of its data. A replica can sit in the same rack or at your disaster-recovery site.

Primary
10.0.0.1
serves data · holds VIP 10.0.0.100
zfs send · recv
Replica
10.0.0.2
mirror of the primary
  • Replication is asynchronous, using zfs send -w | zfs recv, every 10 seconds by default.
  • Liveness is tracked with heartbeats over mutually authenticated TLS.
  • Witnesses are optional vote-only nodes. They hold no data, are never promoted and need no fencing, but they let the cluster tell a dead node from a network partition.
  • Failover moves the primary role, and the VIP, to a replica. It is manual by default and free. Automatic failover is a licensed feature.
ℹ️

HA and replication are separate tools. ovios-cluster mirrors a primary to secondaries and fails over. retadm replicates individual volumes and LUNs for backup, and works independently of a cluster.

What is free and what is licensed

Feature Free Licensed
Replication, retention, resume, status, metrics Yes
Fencing, quorum, epoch checks, receive guard, self-demotion Yes
Any number of nodes, witnesses, tiebreaker Yes
Manual failover (ovios-cluster promote) Yes
Automatic failover Subscription

A license unlocks the feature, not nodes, so adding or replacing nodes never needs a new license. Nothing that protects your data consults the license: it only decides whether a replica may promote itself automatically.

2 How It Keeps Data Safe

Failover is only useful if it never makes things worse. These are the protections built into the cluster.

Risk Protection
Two primaries (split brain) Every promotion raises a cluster epoch, saved before anything is exposed. The higher epoch always wins, so a returning old primary demotes itself instead of serving.
Old primary keeps writing Promotion fences it first (for example an IPMI power-off). A primary that loses quorum for lease_seconds also steps down by itself, and replicas wait longer than that before promoting.
A cut-off replica promotes wrongly It needs a strict majority of votes, and no reachable voter may still see the primary.
Replica too far behind Promotion is refused if the newest replicated snapshot is older than max_replica_lag_seconds.
Old primary overwrites the new one The receiving node refuses streams from a sender with an older epoch, an unknown sender, or for an excluded dataset.
Destructive "self-healing" A failed transfer is retried and resumed, but never destroys snapshots or forces a receive. Divergence is reported as DIVERGED and fixed only by an explicit resync.
Spoofed peers Heartbeats use mutual TLS with a per-cluster CA, and SSH host keys are pinned. A changed host key is a hard error.
⚠️

Rehearse before you rely on it. Build a staging pair first and test a failover, including pulling the primary's power, before using the cluster for production data.

3 Prerequisites

Requirement Details
Two or more OviOS nodes At least a primary and a replica, on IPv4 addresses and able to reach each other.
A free VIP An unused address on the same network as the nodes, optional but normally what clients connect to.
Root SSH key access The primary must reach every node as root without a password. See Passwordless SSH in the Replication Guide. The first contact pins the node's host key.
ZFS pools with the same names Each replica needs the destination pool for every replicated dataset. Replication fails with destination pool does not exist otherwise. Witnesses need no pools.
ovios-cluster on every node Installed at /usr/bin/ovios-cluster. add checks this, because the receiving side runs it over SSH as a guard.
For automatic failover A quorum of three or more voters (a witness node, or a tiebreaker IP), a fence_cmd on every data node, and a license.
💡

Two nodes cannot tell a dead peer from a partition. Add a witness (add --witness) or set a --tiebreaker IP such as your default gateway. Without one, automatic failover is rejected by validation, and a lone primary keeps serving when its peer dies.

ℹ️

On a stateless (live) node the cluster configuration is part of the node configuration saved to your pools, so it comes back at every boot. See the Live-OS guide.

4 Step 1 — Create the Cluster

All commands in this guide are run in the ovios-shell, where oc is a shorthand for ovios-cluster. On the node that should start as primary, run ovios-cluster init. It creates the cluster CA, a node identity and the configuration, and makes this node the primary.

PRIMARY — 10.0.0.1
ovios-shell ➤ ovios-cluster init --vip 10.0.0.100 --tiebreaker 10.0.0.254

Cluster initialized. Node 5d1f0c3a-... (10.0.0.1) is primary at epoch 1.
Cluster ID: <id> (needed to request a license for automatic failover)
Next: start the cluster daemon with 'options cluster.enable on' (or 'service ovios-cluster enable') in ovios-shell,
      then 'ovios-cluster add <ip>' for each other node.

Then start the cluster daemon with the cluster.enable option, or the equivalent service command, and check that it is running:

ovios-shell ➤ options cluster.enable on
option cluster.enable  on
ovios-shell ➤ service ovios-cluster status
ℹ️

service ovios-cluster enable does the same as options cluster.enable on: it starts the daemon and keeps it enabled, including across stateless reboots. The flag is saved in the node configuration, so config sync carries it to the other nodes.

Option Meaning
--vip The floating IP that follows the primary. Optional.
--interface Interface for the VIP. Default: the default-route interface.
--tiebreaker An IP (for example the gateway) that counts as an extra vote, so a two-node cluster can tell a partition from a dead peer.
--hbport Heartbeat port. Default 9000.
--force Overwrite an existing configuration, creating a new CA and identity.

Note the Cluster ID. A license is bound to it, not to hardware.

5 Step 2 — Add Nodes

Run ovios-cluster add on the primary for every other node. It pins the node's SSH host key, checks that the address really belongs to that host, checks that ovios-cluster (and, for data nodes, ZFS) is installed, issues the node a certificate and starts its daemon.

PRIMARY — 10.0.0.1
# a data node (replica)
ovios-shell ➤ ovios-cluster add 10.0.0.2
Node 5d1f0c3a-... (10.0.0.2) added as replica.

# a witness: votes only, no data, never promoted
ovios-shell ➤ ovios-cluster add --witness 10.0.0.3
⚠️

Cloned machines. If a node was cloned from another, add refuses a duplicate node ID. Remove /etc/ovios-cluster/node-id on the new machine and try again.

6 Step 3 — Configure and Sync

The configuration lives in /etc/ovios-cluster/ovios-cluster.yaml. Edit it on the primary only, then run ovios-cluster sync. Sync validates the file and pushes it to every node, and running daemons pick the change up automatically. An invalid file is rejected and the previous configuration stays in force.

The settings you will most often change:

# services started (in order) on promotion, stopped (in reverse) on demotion
services:
  - ovios-nfs.service
  - ovios-smb.service
  - ovios-iscsi.service

# every data node that may be fenced needs a command that really powers it off
nodes:
  - id: 5d1f0c3a-...
    ip: 10.0.0.1
    fence_cmd: ipmitool -I lanplus -H 10.0.1.1 -U admin -P secret chassis power off
  - id: 5d1f0c3a-...
    ip: 10.0.0.2
    fence_cmd: ipmitool -I lanplus -H 10.0.1.2 -U admin -P secret chassis power off
ovios-shell ➤ ovios-cluster sync

Keep the OviOS configuration identical on every node

The cluster replicates data and dataset properties: sharesmb, sharenfs, mountpoint, quota and the LUN properties (type, mapping, id, target group, provisioning). The iSCSI target definitions and other OviOS settings travel with the shell's configuration sync, and ovios-cluster push-archive <file.tar.gz> distributes a configuration archive from the primary to every other data node. See Config Sync in the Replication Guide.

ℹ️

Datasets of the operating system (rpool, bpool and their children) are excluded from replication by default. Add more under datasets.exclude if some data should not be mirrored.

7 Step 4 — Check Status

ovios-cluster status shows the whole cluster from any node (add --json for scripts).

ovios-shell ➤ ovios-cluster status
Cluster:   ovios-cluster
This node: e303ee93-c225-4e42-9e3d-068dfeecd858 (PRIMARY, applied: primary)
Primary:   e303ee93-c225-4e42-9e3d-068dfeecd858  epoch 21
Votes:     3/3  quorum
Failover:  automatic=true (licensed)
License:   expiring - license expires on 2026-11-12 (28 day(s) left): renew to keep automatic failover
Cluster ID: bcd94e2b-a6f6-432b-948d-77455f04fab7
VIP:       10.0.0.160 present locally=true
Last:      none: primary with valid lease

PEER                                  IP         TYPE     ALIVE  LAST    EPOCH  PRIMARY
3997fac1-40fa-4981-ad03-da5b7e18d981  10.0.0.25  data     true   0s ago  21     e303ee93-c225-4e42-9e3d-068dfeecd858
00c81f6a-30f5-4121-b54f-23d8469928a7  10.0.0.98  witness  true   0s ago  21     e303ee93-c225-4e42-9e3d-068dfeecd858

REPLICATION                           DATASET     LAG  STATE
3997fac1-40fa-4981-ad03-da5b7e18d981  data/lun01  14s  ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/lun02  1s   ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/lun03  14s  ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/vol01  14s  ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/vol02  13s  ok

This is a three-node cluster: the primary (this node), a data node holding the replicas, and a witness that only votes.

Replication state Meaning
ok Replicating normally. LAG is the age of the newest replicated snapshot.
error: … A transfer failed. It is retried automatically and resumed from a token.
DIVERGED The replica holds changes the primary does not have. Needs an explicit resync.
FENCED The receiving node refused the stream (older epoch, unknown sender or excluded dataset).
PROPERTIES NOT SYNCED Data is fine but a property could not be applied (for example a quota below current usage). The next cycle retries.

Metrics for Prometheus are available when metrics_listen is set: epoch, role, quorum votes, peer state, and per-dataset lag, divergence and errors.

8 Manual Failover

Manual failover is free and needs no license. Run ovios-cluster promote on the replica that should take over.

ovios-shell ➤ ovios-cluster promote

Promotion runs as a sequence with rollback: safety checks, the pre_promote hook, fencing of the old primary, saving the new epoch, then mounting datasets, bringing up the VIP (with an ARP announcement), starting the configured services and running the post_promote hook. If a step fails, the local steps are undone and the epoch is never reused.

Without --force, promotion is refused while:

  • the primary is still reachable,
  • there is no quorum,
  • another node still sees the primary, or
  • the replicated data is older than max_replica_lag_seconds.
🛑

--force skips those checks and continues even if fencing fails. Use it only after you have confirmed that the old primary is really down.

ℹ️

Planned maintenance. There is no switchover command yet, because promote is meant for a down primary. For maintenance, stop the primary's services, wait for one final replication cycle, then run promote --force on the target.

9 Automatic Failover

With a license, a replica promotes itself when the primary goes down, with no manual step, and only when it can be done safely: it needs a majority of votes, no voter may still see the primary, the old primary is fenced, and the data is recent enough. Only the first live candidate in configuration order promotes.

  1. Show the cluster ID and the license state: 
    ovios-shell ➤ ovios-cluster license
  2. Email the cluster ID to ovi@ovios.org and ask for a free 30-day test license. You receive a license file.
  3. On the primary, install it. It is verified, installed and pushed to every node, and takes effect without a restart: 
    ovios-shell ➤ ovios-cluster license install acme.license
  4. On the primary, set automatic_failover: true in the configuration and sync. This needs a quorum (a witness or tiebreaker) and a fence_cmd on every data node, and is refused without a valid license: 
    ovios-shell ➤ ovios-cluster sync
  5. Confirm with ovios-cluster status: the Failover line should read automatic=true (licensed).

If a license expires

An expiring license never stops a running primary or replication.

  1. 30 days before expiry: status and the log warn. Everything keeps working.
  2. After expiry, during the grace period (7 days by default, set per license): still works, with a louder warning.
  3. After the grace period: automatic failover turns off and the cluster behaves as manual failover. status shows configured but NOT ACTIVE, and promote works as usual. Installing a renewal re-enables it immediately.

Licenses are signed documents verified on your own systems, so no connection to us is needed. License terms are in the Terms & License.

10 After a Failover

Replication is asynchronous, so the old primary may hold writes that were never replicated. When it comes back it demotes itself, and replication reports it as DIVERGED. Datasets are never overwritten without an explicit decision from you.

  1. Run ovios-cluster status on the new primary and find the diverged replica.
  2. Check what is on the old primary and decide whether any of that data should be kept.
  3. Roll the old primary back and re-sync it from the new primary. This discards its divergent changes: 
    ovios-shell ➤ ovios-cluster resync 10.0.0.1 --discard-divergent-data
    resync accepts a node's ID or its IP address.

Restarting the daemon never changes a node's role and never drops the VIP.

11 Command Reference

Command Where Description
init [--vip] [--tiebreaker] First node Create a cluster; this node becomes the primary.
add <ip> [--witness] Primary Enrol a data node, or a vote-only witness.
sync Primary Validate the configuration and push it to every node.
status [--json] Any node Show roles, quorum, peers, license and replication state.
promote [--force] The replica to take over Manual failover.
resync <node> --discard-divergent-data Primary Roll a diverged replica back and re-sync it. Discards its changes.
push-archive <file.tar.gz> Primary Extract a configuration archive on every other data node.
license / license install <file> Primary Show the cluster ID and license state, or install a license (it is pushed to every node).
remove <node-id-or-ip> Primary Remove a node from the cluster.
destroy --yes [--all] Any node Dismantle the cluster configuration on this node (with --all, every node). Your datasets are kept.

12 Key Options

Set in /etc/ovios-cluster/ovios-cluster.yaml on the primary, then ovios-cluster sync.

Option Default Description
automatic_failover false Licensed. Needs quorum and a fence_cmd on every data node.
heartbeat_interval 2 s Seconds between heartbeats.
lease_seconds 15 A primary without quorum steps down after this. A replica waits this plus half again before promoting.
max_replica_lag_seconds 300 Promotion is refused if the newest replicated snapshot is older.
replication.interval 10 s Seconds between replication cycles.
replication.retention.keep_last 12 Newest snapshots kept. Snapshots a replica still needs are never removed.
services none Units started on promotion and stopped on demotion.
hooks none pre_promote (a failure aborts), post_promote, pre_demote, post_demote.
metrics_listen off For example 127.0.0.1:9100 for Prometheus.

13 Limits to Know

  • Asynchronous replication. A failover can lose up to the replication interval plus the cycle time of writes, bounded by max_replica_lag_seconds.
  • Active/passive. One primary serves data at a time.
  • IPv4 only for node addresses.
  • Node certificates are valid for 5 years and are not rotated automatically.
  • Root SSH. Nodes authenticate to each other over SSH with root's key.
  • Clocks. License dates are checked against the node's clock, so keep time in sync.

14 Troubleshooting

Symptom Cause and fix
add: "not installed on …" /usr/bin/ovios-cluster is missing on the new node. Install it there first.
add: "zfs is not installed" The node is not a data node. Use --witness if it should only vote.
add: "does not appear to be a local address" The IP you gave is not that host's own address. Check for NAT or a wrong address.
"destination pool … does not exist" Create the pool, with the same name, on the replica.
sync refuses the configuration Automatic failover needs a valid license, a quorum (witness or tiebreaker) and a fence_cmd on every data node.
promote refuses The primary is reachable, there is no quorum, another node still sees the primary, or the data is too old. Fix the cause, or use --force only after confirming the old primary is down.
Status shows "quorum not possible" Add a witness or set a tiebreaker.
Status shows "Daemon is NOT running" service ovios-cluster enable (or options cluster.enable on). Status falls back to the last known state from disk.
Status shows "configured but NOT ACTIVE" The license is missing, or expired past its grace period. Install or renew it.
💡

Want to see automatic failover work? Email ovi@ovios.org for a free 30-day test license.