Why Does a ZFS Send Resume Token Stop Working After the Destination Reboots?

A destination reboot should not normally invalidate a ZFS resume token unless the saved receive state, dataset, or required source history changed.

The token is an opaque description of a specific interrupted receive, not a reusable bookmark for any later replication attempt. It belongs to the destination filesystem or volume that preserved partial state with a resumable receive. After reboot, a task may read the wrong dataset, import the pool differently, clear the partial state, modify the destination, lose a required source snapshot, or generate an incompatible stream. Verify the token at both ends before restarting a full transfer.

Confirm the Partial Receive State Survived the Reboot

On the destination, read the receive_resume_token property from the exact filesystem or volume used by the interrupted receive. Record the pool name, dataset path, token value, and partial-state space use.

The FreeBSD handbook explains that resumable receives preserve partial state and keep an opaque token on the receiving dataset until the transfer completes or the state is explicitly abandoned.

If the property is empty after reboot, the receive was not saved with the resumable option, the partial state was completed or aborted, the wrong dataset is being queried, or an automated cleanup removed it. Do not reuse a token copied from an earlier transfer.

Verify the Token Belongs to the Exact Destination Dataset

Check whether the destination pool imported under the expected name and whether replication still targets the same dataset path. Pay attention to alternate roots, renamed pools, parent-path changes, and task options that add or remove path components.

Ubuntu’s ZFS properties reference defines receive_resume_token as a dataset property, which means the token must be read from the filesystem or volume holding that specific saved state.

A token retrieved from backup/pool/data cannot safely be assumed to resume a receive now targeting backup/data. Correct the task path before changing snapshots or destroying the partial receive.

Check Whether the Destination Was Modified or Rolled Back

Inspect commands, scheduled tasks, replication software, snapshot retention, and administrator activity since the interruption. Look for rollback, clone promotion, dataset rename, receive abort, or a new receive into the same target.

Klara Systems notes that replication tooling manages destination state around ZFS send and receive, so an orchestration task can invalidate the original recovery path by cleaning or replacing the saved state.

Do not write ordinary files into a replication destination while investigating. Even when the token still exists, destination changes can block the stream or force a rollback that destroys newer destination data.

-15% OFF

ZimaBoard 2 - Mini Home Server for Your Big Idea

Single board computer zimaboard2

$339.00 - $519.00

Buy Now

Verify the Source Still Has the Snapshot or Bookmark Chain

Identify the source dataset and the snapshots or bookmarks encoded by the interrupted transfer. Compare them with current retention and any snapshot renames or deletions since the transfer stopped.

The FreeBSD zfs-send manual specifies that zfs send -t generates a stream from the receive resume token, tying the new stream to the interrupted receive rather than to an arbitrary current snapshot.

If retention deleted the required source history, the token cannot reconstruct data that no longer exists. Preserve the remaining destination partial state until you have decided whether another source copy or a fresh full send is required.

Check Pool Features and Stream Options on Both Systems

Record ZFS versions, enabled pool features, encryption state, and the original stream options such as raw, compressed, embedded, or large-block sends. Compare them after any software or pool upgrade.

Oracle’s resumable replication documentation describes resuming an interrupted transfer as a coordinated send-and-receive operation, so compatibility and the original transfer context still matter after a reboot.

A reboot alone does not change feature flags, but an upgrade performed during the outage can. Reproduce the resume command manually with verbose output before assuming the token itself is corrupt.

Verify Replication Services and SSH After Destination Startup

Confirm the destination pool is imported, required encrypted datasets are unlocked, SSH is running, the replication user can execute ZFS commands, and the task starts only after storage is ready.

TrueNAS remote-replication guidance requires the destination SSH and dataset prerequisites to be available after restart, otherwise the automation can fail before it even attempts the saved token.

Test authentication and a read-only property query before launching the resumed stream. A network or permission failure can look like a token failure in a high-level task log.

Resume Once or Abort the Partial Receive Deliberately

Generate one resumed send stream using the current token and feed it into a resumable receive on the same destination. Save the full error output and avoid launching parallel replication tasks.

The ZimaSpace NAS data migration guide provides the adjacent safety rule: preserve the source and rollback path until the destination has been verified.

If the token is unusable and the partial state is no longer valuable, abort it with the supported receive-abort command only after confirming that a fresh full or incremental transfer can be generated. Aborting frees the saved partial state and cannot be undone.

Frequently Asked Questions

Does a destination reboot always invalidate a ZFS resume token?

No. A saved partial receive is designed to survive interruption, including an unclean shutdown. Failure after reboot usually means the task is reading a different dataset, the partial state was cleared, required source history changed, or startup dependencies are missing.

Can a new resume token be generated from the source alone?

No. The opaque token comes from the destination dataset’s saved partial receive state. The source uses that token to generate a continuation stream, but it cannot recreate a deleted destination state by itself.

When should the partial receive be aborted?

Abort only when the resume path has been proven unusable, the source can generate a replacement transfer, and the destination partial state is no longer needed for recovery. Preserve logs and available snapshots before removing it.