Unplanned failover.
A pause, not an outage.

1. Executive summary

This report documents unplanned-failover testing of a three-node OviOS Linux HA storage cluster. For each protocol, the network cable was physically pulled from the active node while a Linux client ran a sustained 4 KiB random read/write workload against it.

Key finding: across iSCSI, SMB 3.1.1 and NFS v4.2, the replica took over automatically and the client application saw zero I/O errors. Client I/O was held during the takeover, about 35 seconds for iSCSI and SMB and about 60 seconds for NFS, then resumed without any administrator action.

2. Test environment, protocols and mount configurations

Test environment

Protocols and mounts

Each protocol was tested on its own, with its own cable pull, from a Linux client using standard kernel clients and no special software.

On the cluster, the NFSv4 grace and lease time were set to 15 seconds instead of the Linux default of 90 (options nfs.grace.time 15, options nfs.lease.time 15). SMB required SMB3 as the minimum protocol, with multichannel on.

3. Failover performance summary

The figure that matters to an application is the longest single I/O pause: how long one request waited while the replica took over. Every request then completed normally.

Outside the failover cycle, the slowest I/O stayed in the sub-second to low-second range on every protocol. The cable pull shows as one spike, and the next cycle is back to normal.

All three protocols paused once, at the cable pull, and resumed with zero errors

Slowest single I/O in each 30-second cycle, seconds (log scale)

4. Detailed protocol breakdown

4.1 iSCSI block target

The cable was pulled about 5 seconds into cycle 3 (16:44:46). The Linux iSCSI initiator held its 64 outstanding requests instead of failing them. The longest waited 35.6 s, and the 30-second cycle completed in 55.8 s with 0 errors. Writes succeeded in every following cycle, so the ext4 filesystem never switched to read-only. A few I/Os in the first cycle on the replica took up to 3 s as it settled; by the next cycle the slowest was about 1 s.

4.2 SMB 3.1.1 share

The cable was pulled a few seconds into cycle 5 (22:14:40). With the hard and echo_interval=5 mount options, the client detected the lost server quickly, held its requests and reconnected to the replica on the same address. The longest request waited 35.0 s. The cycle completed in 58.1 s with 0 errors, and all four workers kept their open files. By two cycles later the slowest I/O was under 1 s.

4.3 NFS v4.2 share

The cable was pulled about 1 second into cycle 3 (19:29:05). The hard mount kept the client retrying. The longest request waited 59.8 s, and the cycle completed in 72.5 s with 0 errors and no stale file handles. The following cycles ran normally, with no I/O slower than 0.2 s.

NFS pauses longer than iSCSI and SMB because of the NFSv4 grace period: after a takeover, the server gives clients a window to reclaim their open files before it accepts new ones. With the Linux default of 90 seconds the same test paused for about 139 s. Setting grace and lease to 15 seconds removed about 78 s of that.

5. Stress-cycle metrics matrix

Every 30-second cycle of every test, from the client-side fio output on 6 November 2026. The failover cycle stretches past 30 s because its requests were held, not failed. In each test the final cycle was stopped by hand after a few seconds, with 0 errors, and is not listed.

6. High-availability architecture

An OviOS HA cluster runs three roles, so it can tell a failed node from a broken link and recover with nobody at the console.

When the primary drops off the network:

1. The replica and the witness agree the primary is gone.
2. The replica takes the active role and brings the storage, the service address, the LUNs and the shares online.
3. The isolated primary detects that it has lost the cluster, and its watchdog fences it: the node reboots rather than keep serving data. This happened as designed in every test.
4. Clients reconnect to the same address and continue. Nothing is reconfigured on the client side.

Self-fencing is what makes this safe. Two nodes writing to the same data at once (“split brain”) is the classic way HA storage corrupts data; the watchdog removes the losing node before that can happen. The Advanced Cluster Administrator Guide describes fencing, the watchdog and the decision rules in detail.

7. Recommended client and cluster configuration

These are the settings used in this test, with the SMB password moved into a credentials file. The one that matters most is hard: it makes the client wait for the replica instead of reporting an error.

Linux clients

# SMB 3.1.1 share
mount -t cifs //<service-address>/<share> /mnt/smb \
  -o vers=3.1.1,credentials=/etc/smb-credentials,cache=none,actimeo=0,hard,echo_interval=5,nobrl

# NFS v4.2 share
mount -t nfs4 <service-address>:/ /mnt/nfs \
  -o vers=4.2,hard,proto=tcp,timeo=600,retrans=2

# iSCSI LUN: log in with open-iscsi, then mount as usual
iscsiadm -m node -T <target-iqn> -p <service-address> --login
mount /dev/<lun-partition> /mnt/iscsi

OviOS cluster

Set once, with the OviOS options command:

options nfs.grace.time 15   # NFSv4 grace period, seconds (Linux default: 90)
options nfs.lease.time 15   # NFSv4 lease time, seconds (Linux default: 90)

The SMB service ran with smb.min.protocol SMB3 and smb.multichannel on; iSCSI ran on the standard port 3260.

Planning notes

• Always point clients at the cluster service address, never at an individual node.
• Applications and hypervisors have their own I/O timeouts. Allow at least about 60 seconds so a failover is a pause, not an error.

8. Conclusion

An OviOS Linux HA cluster turns a hard failure of the active node into a pause, not an outage. With the network cable pulled mid-workload, iSCSI, SMB and NFS clients all kept their connections, saw zero I/O errors and carried on once the replica took over: about 35 seconds for block and SMB storage, about 60 seconds for NFS. The failed node fenced itself every time, so the data was only ever served from one node.

Test scope

• One unplanned failure per protocol in the final configuration, injected by pulling the primary’s network cable under load.
• During tuning, NFS paused for about 139 s with the default 90-second grace period.
• Linux clients only. Windows SMB clients and VMware or Hyper-V iSCSI initiators were not part of this round; Windows handles open files across a server change differently.
• The test measured service continuity and I/O errors. It did not checksum the data written before and after the failover.
• Failback to the original primary was not part of this test.