LiveOS & Disaster Recovery

1 The Concept: Total Reproducibility

OviOS is architected for extreme portability. Unlike traditional operating systems where configuration is scattered across the filesystem, OviOS centralizes all system settings within its metadata-driven design. This means a full system rebuild — or a pivot from a disk-based install to a RAM-based Live OS — takes seconds.

Key Architecture Principles

2 First Run: Initial Setup

When booting the appliance for the first time, refer to the Admin Manual for initial pool creation.

Why is skip.import Enabled by Default?

On the first run, the option skip.import is set to 1 (on). This prevents the system from attempting to auto-import ZFS pools before the user has defined the environment, ensuring a clean state for initial configuration.

Once your pools are created and you want automatic import on subsequent boots, set it to off:

options skip.import 0

3 Stateless Operation (OviOS 6.2)

OviOS 6.2 can run stateless: the node never needs a configured boot disk. Boot it from a USB stick, an ISO or the network (PXE), and it restores its own configuration from your storage pools at every boot. Replace the USB stick, boot a newer image, or boot hundreds of identical nodes from one PXE server — each node comes back exactly as it was.

3.1 Two ways to be stateless

The kernel line wins if both are present. The version command shows which one is in effect.

3.2 How the kernel-line method works

Saving. Every configuration change is copied to a hidden dataset on each data pool, one per node: <pool>/.ovios-system/<node-id>. The node id is the machine's DMI system UUID, so several nodes can share the same pools without overwriting each other. Each save becomes a ZFS snapshot (a generation); the newest 10 are kept. Saves happen automatically after options changes, iSCSI target, LUN and CHAP changes, pool create, the SMB/NFS export, user, AD-join and network tools, passwd, hostnamectl, configuration restores, and at shutdown.

Restoring. Early at boot, before networking and before any service starts, OviOS imports the pools read-only, restores the newest generation saved by this node, exports the pools again, and enables the services that the restored ovios.conf asks for. The normal boot then imports your pools and starts everything.

3.3 What is saved

Pool data, LUN properties and share properties live on the pools themselves and need no copying.

3.4 SMB passwords and the Active Directory join: smb.secrets.restore

Samba keeps SMB passwords and the domain machine account in its own databases, separate from the Unix passwords. Decide whether those secrets may be stored on your pools:

options smb.secrets.restore on    # keep SMB passwords and the AD join across reboots
options smb.secrets.restore off   # default: do not store them

3.5 Commands

sync-config persist          # save the configuration to all pools now
sync-config persist-status   # saved generations per pool, and pools that were skipped and why
sync-config persist-restore 168  # apply generation 168 on the running node

persist-restore asks for confirmation, writes the chosen generation back (files created after it are kept), restarts cron so restored crontabs load, and saves the result as a new generation, so the next boot restores this state rather than the newer one. Then apply it with restore all and service reset.

Saves skip the system pools (rpool, bpool), pools excluded with exclude.pools, and pools that are faulted or imported read-only.

3.6 Boot options

# normal stateless node
... ovios.autorestore=1
# first boot after replacing a node's hardware
... ovios.autorestore=1 ovios.autorestore.anynode=1 ovios.autorestore.force=1

3.7 Upgrades, rollback and troubleshooting

Rolling upgrades can be done live, on installed systems and on nodes running the live OS: upgrade one node at a time while the rest keep serving. On a live node, an upgrade applied in RAM lasts until the next reboot. To make it permanent, boot the newer image, and boot the previous image to roll back. The configuration follows either way. Each generation records the OviOS release that wrote it, and the restore logs a note when it comes from an older or newer release.

journalctl -b -u ovios-autorestore   # what was restored at this boot
cat /var/log/ovios/persist.log          # every save and restore

If sync-config persist-status reports a pool as still imported read-only by ovios-autorestore, the export at boot failed; fix it with zpool export <pool> followed by service zfs start from the system shell.

4 Backing Up Configuration

Before a failure occurs, you must maintain a current backup of your system state. OviOS configuration backups are compact .tar.gz archives that capture everything needed to fully rebuild the system.

Configuration Tuning

• Files to Sync: Backup archives and configuration sync do not include the network configuration by default, so a replication partner keeps its own network setup. To include extra files or directories, add their paths to options files.to.sync, for example options files.to.sync /etc/systemd/network. (Stateless saves to the pools always include the network configuration.)
• Target Pool: Ensure system.backup.pool is defined when using sync-config so backups are written to the correct pool.

options system.backup.pool       # check which pool backups go to
options system.backup.pool osb  # set backup pool to 'osb'

Backup Methods

Creating and Restoring a Backup via CLI

# Verify your backup pool is set
ovios-shell ➤ options system.backup.pool
option system.backup.pool  osb

# Create a backup archive
ovios-shell ➤ sync-config bo
Backup archive created: /ovios/osb/osbackup/osbackup-20260506_205325.tar.gz

# Restore from the latest backup
ovios-shell ➤ sync-config restore
Verifying archive: /ovios/osb/osbackup/osbackup-20260506_205325.tar.gz
[RESTORING] Applying system configuration...
[SUCCESS] Restore finished. Reloading systemd...
Configuration restored from backup successfully.

5 Restoring Configuration

Use these steps for a new installation, a system disk replacement, or when restarting in RAM mode. The process is the same in all cases.

Pre-Restoration: Mount the Backup Pool

If your backup resides on a storage pool (rather than external media), you must manually mount the pool before restoring:

options skip.import 0   # allow ZFS pool import
service zfs start        # import and mount pools

Restoration via CLI

Restoration via API or Web UI

For uploading a backup file to a fresh RAM instance from a remote machine:

1. Enable web access on the new instance:

   options ovios-web.enable on

2. Upload backup via CURL:

   curl -k -XPOST https://oviosadm:ovios@ovios-storage:8443/api/v1/sys/restore \
     -F config_file=@/path/to/backup.tar.gz

3. Upload via Web UI: Navigate to OviOS Web Monitoring → use the Restore Config button to upload your local .tar.gz file.

6 Finalization & Verification

After restoration, apply the pool metadata and restart the full service stack to bring the system to its desired state.

The Finalize Commands

Verifying the Restored System

ovios hc   # full health check — verify all services and pools are online

Troubleshooting: iSCSI Portal

If the iSCSI service fails to start after a restore, check your portal binding. If options iscsi.address is bound to an IP address that no longer exists on the current hardware or instance, reset it to the current valid IP:

options iscsi.address                      # check current portal binding
options iscsi.address 192.168.10.20     # update to current interface IP

7 Success & Next Steps

At this stage, your LUNs, NFS shares, and SMB shares should be fully accessible. The system is now running in its desired state — regardless of whether it was booted from a disk or entirely from RAM.

Recommended Ongoing Practices

• Check sync-config persist-status to confirm every pool has a recent saved generation
• Schedule sync-config bo in cron for periodic explicit snapshots of system state
• Store a copy of your latest backup archive on external media or a separate pool
• Run ovios hc after any configuration change to confirm system health
• Document your iscsi.address IP and backup pool name — you'll need them during recovery