Cluster Guide

1 How the Cluster Works

ovios-cluster builds an active/passive cluster on ZFS. One node is the primary: it serves your data and holds the floating virtual IP (VIP). The other nodes are replicas, which follow the primary and hold a mirror of its data. A replica can sit in the same rack or at your disaster-recovery site.

• Replication is asynchronous, using zfs send -w | zfs recv, every 10 seconds by default.
• Liveness is tracked with heartbeats over mutually authenticated TLS.
• Witnesses are optional vote-only nodes. They hold no data, are never promoted and need no fencing, but they let the cluster tell a dead node from a network partition.
• Failover moves the primary role, and the VIP, to a replica. It is manual by default and free. Automatic failover is a licensed feature.

What is free and what is licensed

A license unlocks the feature, not nodes, so adding or replacing nodes never needs a new license. Nothing that protects your data consults the license: it only decides whether a replica may promote itself automatically.

2 How It Keeps Data Safe

Failover is only useful if it never makes things worse. These are the protections built into the cluster.

3 Prerequisites

4 Step 1 — Create the Cluster

All commands in this guide are run in the ovios-shell, where oc is a shorthand for ovios-cluster. On the node that should start as primary, run ovios-cluster init. It creates the cluster CA, a node identity and the configuration, and makes this node the primary.

ovios-shell ➤ ovios-cluster init --vip 10.0.0.100 --tiebreaker 10.0.0.254

Cluster initialized. Node 5d1f0c3a-... (10.0.0.1) is primary at epoch 1.
Cluster ID: <id> (needed to request a license for automatic failover)
Next: start the cluster daemon with 'options cluster.enable on' (or 'service ovios-cluster enable') in ovios-shell,
      then 'ovios-cluster add <ip>' for each other node.

Then start the cluster daemon with the cluster.enable option, or the equivalent service command, and check that it is running:

ovios-shell ➤ options cluster.enable on
option cluster.enable  on
ovios-shell ➤ service ovios-cluster status

Note the Cluster ID. A license is bound to it, not to hardware.

5 Step 2 — Add Nodes

Run ovios-cluster add on the primary for every other node. It pins the node's SSH host key, checks that the address really belongs to that host, checks that ovios-cluster (and, for data nodes, ZFS) is installed, issues the node a certificate and starts its daemon.

# a data node (replica)
ovios-shell ➤ ovios-cluster add 10.0.0.2
Node 5d1f0c3a-... (10.0.0.2) added as replica.

# a witness: votes only, no data, never promoted
ovios-shell ➤ ovios-cluster add --witness 10.0.0.3

6 Step 3 — Configure and Sync

The configuration lives in /etc/ovios-cluster/ovios-cluster.yaml. Edit it on the primary only, then run ovios-cluster sync. Sync validates the file and pushes it to every node, and running daemons pick the change up automatically. An invalid file is rejected and the previous configuration stays in force.

The settings you will most often change:

# services started (in order) on promotion, stopped (in reverse) on demotion
services:
  - ovios-nfs.service
  - ovios-smb.service
  - ovios-iscsi.service

# every data node that may be fenced needs a command that really powers it off
nodes:
  - id: 5d1f0c3a-...
    ip: 10.0.0.1
    fence_cmd: ipmitool -I lanplus -H 10.0.1.1 -U admin -P secret chassis power off
  - id: 5d1f0c3a-...
    ip: 10.0.0.2
    fence_cmd: ipmitool -I lanplus -H 10.0.1.2 -U admin -P secret chassis power off

ovios-shell ➤ ovios-cluster sync

Editing the configuration safely

On the primary, service ovios-cluster edit opens the file in $VISUAL / $EDITOR (nano or vi if unset) on a private copy. When you close the editor the copy is checked with ovios-cluster sync --check --file; an invalid file is never saved and you can edit again or discard. A valid file replaces the live one (the old one is kept as ovios-cluster.yaml.bak) and you are asked whether to push it to the other nodes. If the live file changed while you were editing, it is not overwritten and your edit is kept in a temporary file.

ovios-shell ➤ service ovios-cluster edit

The check rejects bad YAML, misspelt (unknown) keys, inconsistent settings and automatic failover without a license. You can run it yourself on any file without pushing anything:

ovios-shell ➤ ovios-cluster sync --check --file /root/new.yaml

Configuration is synced automatically — including passwords

ovios-cluster sync above only pushes the cluster configuration file. The OviOS configuration (LUNs, volumes, shares, iSCSI targets, users and options) is a separate sync, and it keeps the replicas ready to take over:

The cluster itself replicates data and dataset properties: sharesmb, sharenfs, mountpoint, quota and the LUN properties (type, mapping, id, target group, provisioning). ovios-cluster push-archive <file.tar.gz> distributes a configuration archive from the primary to every other data node.

7 Step 4 — Check Status

ovios-cluster status shows the whole cluster from any node (add --json for scripts).

ovios-shell ➤ ovios-cluster status
Cluster:   ovios-cluster
This node: e303ee93-c225-4e42-9e3d-068dfeecd858 (PRIMARY, applied: primary)
Primary:   e303ee93-c225-4e42-9e3d-068dfeecd858  epoch 21
Votes:     3/3  quorum
Failover:  automatic=true (licensed)
License:   expiring - license expires on 2026-11-12 (28 day(s) left): renew to keep automatic failover
Cluster ID: bcd94e2b-a6f6-432b-948d-77455f04fab7
VIP:       10.0.0.160 present locally=true
Last:      none: primary with valid lease

PEER                                  IP         TYPE     ALIVE  LAST    EPOCH  PRIMARY
3997fac1-40fa-4981-ad03-da5b7e18d981  10.0.0.25  data     true   0s ago  21     e303ee93-c225-4e42-9e3d-068dfeecd858
00c81f6a-30f5-4121-b54f-23d8469928a7  10.0.0.98  witness  true   0s ago  21     e303ee93-c225-4e42-9e3d-068dfeecd858

REPLICATION                           DATASET     LAG  STATE
3997fac1-40fa-4981-ad03-da5b7e18d981  data/lun01  14s  ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/lun02  1s   ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/lun03  14s  ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/vol01  14s  ok
3997fac1-40fa-4981-ad03-da5b7e18d981  data/vol02  13s  ok

This is a three-node cluster: the primary (this node), a data node holding the replicas, and a witness that only votes.

Cluster Stats page and prompt marker

On the primary only, the web dashboard shows a Cluster Stats button. It opens a page with the cluster status, votes, VIP and failover state, every node with its heartbeat age, the replication state of each dataset (lag, diverged, errors) and the last lines of the cluster log. It refreshes by itself, and on a replica or a node outside a cluster the page does not exist.

In the shell, the prompt shows the node's role — [PRIMARY] in red or [replica] in green — so you do not run a command on the wrong node. See shell.prompt in the Options Guide.

Metrics for Prometheus are available when metrics_listen is set: epoch, role, quorum votes, peer state, and per-dataset lag, divergence and errors.

8 Planned Maintenance (Switchover)

To do maintenance on the current primary, hand the primary role to a replica first with ovios-cluster switchover. It is loss-free: the primary stops serving, sends the replica its final state, and the replica takes over the VIP and the services. Nothing is fenced or powered off, and no write is lost. Switchover is free and needs no license.

# 1. rehearse: runs every check and the pre-sync, but does not hand over
ovios-shell ➤ ovios-cluster switchover --check

# 2. do it (it asks for confirmation; -y skips it)
ovios-shell ➤ ovios-cluster switchover

# or choose the replica yourself (node ID or IP)
ovios-shell ➤ ovios-cluster switchover --host 10.0.0.2

Without --host, the cluster picks the healthiest replica: the one with the smallest replication lag, with the configuration order breaking ties.

What it checks first

The replica must be reachable, agree on who the primary is, replicate cleanly (no divergence, no errors, within max_replica_lag_seconds), and confirm over SSH that it can take over.

What it does

1. Waits for replication. It blocks the next replication cycle and never interrupts a transfer in flight. Every ten seconds it says what is sending, what is queued and what replication is doing right now.
2. Pre-sync while this node still serves, so that most data is already on the replica, and a replica that cannot receive is found before anything is stopped. If it fails, nothing has changed.
3. Stops serving: services, VIP and datasets (read-only), step by step. This is the point of no return.
4. Final sync of the now-final state to the replicas. It is short, thanks to the pre-sync.
5. Hand-over. The replica takes over at the next epoch once it holds the final snapshot, and its own steps are shown while the old primary waits.

If something goes wrong, or you interrupt it

• If anything fails before the replica has taken over, the old primary resumes as primary.
• If it cannot be known whether the replica promoted (no answer, no heartbeat), the old primary stays down on purpose and says so. ovios-cluster promote on it resumes it.
• Ctrl-C during steps 1 and 2 gives up: nothing is changed and replication continues. If your terminal swallows Ctrl-C (the OviOS shell does), run ovios-cluster switchover --abort from another session.
• From the line "Point of no return" on, Ctrl-C is ignored. Stopping halfway could leave the cluster without a primary, so the switchover runs to the end, or undoes itself if something fails, even if the command is closed or the SSH session drops. Follow it with ovios-cluster status: the Last: line shows the step in progress.

When the maintenance is done, you can switch back the same way, from the new primary.

9 Manual Failover

Manual failover is free and needs no license. Run ovios-cluster promote on the replica that should take over.

ovios-shell ➤ ovios-cluster promote

Promotion runs as a sequence with rollback: safety checks, the pre_promote hook, fencing of the old primary, saving the new epoch, then mounting datasets, bringing up the VIP (with an ARP announcement), starting the configured services and running the post_promote hook. If a step fails, the local steps are undone and the epoch is never reused.

Without --force, promotion is refused while:

• the primary is still reachable,
• there is no quorum,
• another node still sees the primary, or
• the replicated data is older than max_replica_lag_seconds.

10 Manual Demotion

Normally the cluster demotes a node by itself. If it did not — for example a former primary that came back with writable datasets or the VIP still up — run ovios-cluster demote on that node. It stops replication and the configured services, removes the VIP, and unmounts the datasets and sets them readonly.

ovios-shell ➤ ovios-cluster demote

11 Automatic Failover

With a license, a replica promotes itself when the primary goes down, with no manual step, and only when it can be done safely: it needs a majority of votes, no voter may still see the primary, the old primary is fenced, and the data is recent enough. Only the first live candidate in configuration order promotes.

1. Show the cluster ID and the license state:

   ovios-shell ➤ ovios-cluster license

2. Email the cluster ID to ovi@ovios.org and ask for a free 30-day test license. You receive a license file.
3. On the primary, install it. It is verified, installed and pushed to every node, and takes effect without a restart:

   ovios-shell ➤ ovios-cluster license install acme.license

4. On the primary, set automatic_failover: true in the configuration and sync. This needs a quorum (a witness or tiebreaker) and a fence_cmd on every data node, and is refused without a valid license:

   ovios-shell ➤ ovios-cluster sync

5. Confirm with ovios-cluster status: the Failover line should read automatic=true (licensed).

If a license expires

An expiring license never stops a running primary or replication.

1. 30 days before expiry: status and the log warn. Everything keeps working.
2. After expiry, during the grace period (7 days by default, set per license): still works, with a louder warning.
3. After the grace period: automatic failover turns off and the cluster behaves as manual failover. status shows configured but NOT ACTIVE, and promote works as usual. Installing a renewal re-enables it immediately.

Licenses are signed documents verified on your own systems, so no connection to us is needed. License terms are in the Terms & License.

12 After a Failover

Replication is asynchronous, so the old primary may hold writes that were never replicated. When it comes back it demotes itself, and replication reports it as DIVERGED. Datasets are never overwritten without an explicit decision from you.

1. Run ovios-cluster status on the new primary and find the diverged replica.
2. Check what is on the old primary and decide whether any of that data should be kept.
3. Roll the old primary back and re-sync it from the new primary. This discards its divergent changes:

   ovios-shell ➤ ovios-cluster resync 10.0.0.1 --discard-divergent-data

   resync accepts a node's ID or its IP address.

Restarting the daemon never changes a node's role and never drops the VIP.

13 Command Reference

14 Key Options

Set in /etc/ovios-cluster/ovios-cluster.yaml on the primary, then run ovios-cluster sync. Unknown keys are an error, so a misspelt option is not silently ignored. To check a file without pushing it, run ovios-cluster sync --check.

15 Alerting and Metrics

Every node can expose its cluster state as Prometheus metrics, so you can graph it and get alerted before a problem reaches your clients. Metrics are free. All of them are gauges, and each carries the label node="<this node's ID>".

Turn on the endpoint

Set metrics_listen in the cluster configuration on the primary, sync it, and restart the daemon (a change of this option needs a restart). The endpoint is then http://<node>:9100/metrics.

# /etc/ovios-cluster/ovios-cluster.yaml
metrics_listen: ":9100"

ovios-shell ➤ ovios-cluster sync
ovios-shell ➤ service ovios-cluster restart    # on every node

Which metrics are exposed

This node

VIP

License

Per peer

With the extra label peer="<id>": one series per other node, witness included.

Replication

With the extra labels target="<node id>" and dataset="<name>". Meaningful on the primary.

Set up Prometheus

OviOS can run the Prometheus service on the node itself: options prometheus.enable on starts it now and at every boot (port 9090, manage it with service prometheus, configuration in /etc/prometheus, which ships with a prometheus.yml). You can equally use a Prometheus server elsewhere. Either way, it has to reach port 9100 of every node, witnesses included.

1. Edit the /etc/prometheus/prometheus.yml that ships with OviOS: add the nodes as a scrape job (under scrape_configs), and point rule_files at your rules:

   scrape_configs:
     - job_name: ovios-cluster
       scrape_interval: 15s
       static_configs:
         - targets: ['10.0.0.1:9100', '10.0.0.2:9100', '10.0.0.3:9100']
   
   rule_files:
     - /etc/prometheus/rules/ovios-cluster.yml

2. Save the alert rules below as /etc/prometheus/rules/ovios-cluster.yml, inside the persisted /etc/prometheus directory.
3. To be notified (email, chat, paging), point Prometheus at an Alertmanager. Without one, firing alerts are still shown on the Alerts page of Prometheus:

   alerting:
     alertmanagers:
       - static_configs:
           - targets: ['alertmanager.example.com:9093']

4. Restart Prometheus and check that all targets are UP under Status → Targets:

   ovios-shell ➤ service prometheus restart

Example alert rules

A starting point that covers quorum, roles, the VIP, peers, replication and the license. Adjust the for: durations and thresholds to your environment. In particular, set the replication lag threshold to match your max_replica_lag_seconds, and drop the license alerts if you run without a license.

groups:
  - name: ovios_cluster_alerts
    rules:

      # --- CLUSTER HEALTH & QUORUM ---
      - alert: OviosClusterQuorumLost
        expr: ovios_cluster_quorum == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Cluster quorum lost on node {{ $labels.node }}"
          description: "A strict majority of voters is no longer reachable. Failover and write actions may fail."

      - alert: OviosClusterMissingVoters
        expr: ovios_cluster_votes_have < ovios_cluster_votes_total
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Cluster is missing active voters"
          description: "Node {{ $labels.node }} reaches only {{ $value }} of its voters."

      - alert: OviosClusterNoSinglePrimary
        expr: sum(ovios_cluster_is_primary) != 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "The cluster has no primary, or more than one"
          description: "{{ $value }} nodes report that they are the primary."

      - alert: OviosClusterNodeDegraded
        expr: ovios_cluster_degraded == 1
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Node {{ $labels.node }} reports a degraded condition"
          description: "Check ovios-cluster status and the daemon log on the node."

      - alert: OviosClusterRoleMismatch
        expr: ovios_cluster_role_applied == 0
        for: 2m
        labels:
          severity: major
        annotations:
          summary: "Node role mismatch on {{ $labels.node }}"
          description: "The local machine does not match the role assigned in the cluster state."

      - alert: OviosClusterPrimarySuspended
        expr: ovios_cluster_suspended == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Primary suspended on {{ $labels.node }}"
          description: "This primary has stepped down after losing quorum."

      - alert: OviosClusterVipUnhealthy
        expr: ovios_cluster_vip_healthy == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Node {{ $labels.node }} cannot serve clients on the VIP"
          description: "The VIP is not assigned, its link has no carrier, or a checked port does not answer."

      # --- PEER & CONNECTIVITY ---
      - alert: OviosClusterPeerDown
        expr: ovios_cluster_peer_up == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Cluster peer {{ $labels.peer }} is down"
          description: "Node {{ $labels.node }} has lost heartbeat connectivity to peer {{ $labels.peer }}."

      - alert: OviosClusterPeerSSHDown
        expr: ovios_cluster_peer_ssh_up == 0
        for: 2m
        labels:
          severity: major
        annotations:
          summary: "Cluster peer {{ $labels.peer }} SSH unreachable"
          description: "SSH to peer {{ $labels.peer }} failed from node {{ $labels.node }}. Replication and switchover will fail."

      # --- REPLICATION & STORAGE ---
      - alert: OviosReplicationError
        expr: ovios_cluster_replication_error == 1
        for: 5m
        labels:
          severity: major
        annotations:
          summary: "Replication error on dataset {{ $labels.dataset }}"
          description: "Replication from {{ $labels.node }} to target {{ $labels.target }} is failing for dataset {{ $labels.dataset }}."

      - alert: OviosReplicationDiverged
        expr: ovios_cluster_replication_diverged == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Replication diverged for dataset {{ $labels.dataset }}"
          description: "{{ $labels.target }} holds data that {{ $labels.node }} does not. It needs resync."

      - alert: OviosReplicationHighLag
        expr: ovios_cluster_replication_lag_seconds > 3600
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High replication lag on dataset {{ $labels.dataset }}"
          description: "Replication lag has exceeded 1 hour (currently {{ $value }} seconds)."

      - alert: OviosReplicationNeverSucceeded
        expr: ovios_cluster_replication_lag_seconds == -1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Replication of {{ $labels.dataset }} has never succeeded"
          description: "No successful replication of {{ $labels.dataset }} to {{ $labels.target }} yet."

      # --- LICENSE ---
      # Only useful if you hold a license: a free cluster always reports 0.
      - alert: OviosLicenseNoAutoFailover
        expr: ovios_cluster_license_autofailover == 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Automatic failover is not permitted by the license"
          description: "The installed license, or the lack of one, does not permit automatic failover."

      - alert: OviosLicenseExpiringSoon
        expr: (ovios_cluster_license_expiry_timestamp_seconds - time()) < (30 * 24 * 3600)
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "OviOS cluster license expires within 30 days"
          description: "The license seen by node {{ $labels.node }} expires in less than 30 days. Please renew."

16 Limits to Know

• Asynchronous replication. A failover can lose up to the replication interval plus the cycle time of writes, bounded by max_replica_lag_seconds.
• Active/passive. One primary serves data at a time.
• IPv4 only for node addresses.
• Node certificates are valid for 5 years and are not rotated automatically.
• Root SSH. Nodes authenticate to each other over SSH with root's key.
• Clocks. License dates are checked against the node's clock, so keep time in sync.

17 Troubleshooting