How to Recover files from a stable and working ZFS pool when snapshots are unavailable

When dealing with corruption beyond ZFS ecosystem (i.e. file deletion by accident, removal by malicious script, ransomware activity), we can employ rolling back transactions to help recover files. However, rolling back directly can lead to inconsistencies, especially if writes were partially completed. Instead, this guide will show how to safely inspect, recover, and extract critical data by manually selecting a stable transaction group (TXG) and recovering files before committing any changes.

1. Immediate Pool Safeguarding

To minimize further damage to the pool:

Set the pool to read-only mode during recovery.
Unmount the pool if possible.
Export the pool to lock it for administrative actions.

Commands to secure the pool:

zpool export <pool>
zpool import -N -o readonly=on -f -R /mnt <pool>

If the pool contains the root filesystem, boot into a ZFS-capable recovery OS (e.g., FreeBSD, Ubuntu with ZFS, Debian with ZFS, or GParted Live) to perform these operations.

2. Inspecting the VDEVs and Uberblocks

To find a stable transaction group (TXG), you must examine the pool’s vdevs and their associated uberblocks.

Identify the Pool's VDEVs

List all vdevs in the pool:

zpool status <pool>

Locate the physical device(s) corresponding to the pool.

Examine the Uberblocks

Use zdb to inspect the uberblocks for each vdev. This will show a list of TXGs:

zdb -ul <vdev>

For example:

------------------------------------
LABEL 0 
------------------------------------
    version: 5000
    name: 'xxx'
    state: 0
    txg: 12577
    pool_guid: 23567
    errata: 0
    hostname: '...'
    top_guid: 23569
    guid: 23569
    vdev_children: 1
    vdev_tree:
        type: 'disk'
        id: 0
        guid: 23569
        path: '/dev/diskid/...'
        whole_disk: 1
        metaslab_array: 256
        metaslab_shift: 33
        ashift: 9
        asize: 4849857349
        is_log: 0
        DTL: 130769
        create_txg: 4
    features_for_read:
        com.delphix:hole_birth
        com.delphix:embedded_data
    labels = 0 1 2 3 
    Uberblock[0]
        magic = 409385340
        version = 5000
        txg = 180766725
        guid_sum = 0487235087340875
        timestamp = 1733409369 UTC = Thu Dec  5 15:36:09 2024
        mmp_magic = 4875943785348
        mmp_delay = 0
        mmp_valid = 0
        checkpoint_txg = 0
        labels = 0 1 2 3 
    Uberblock[1]
        magic = 409385340
        version = 5000
        txg = 180761615
        guid_sum = 0487235087340875
        timestamp = 1733406769 UTC = Thu Dec  5 14:52:49 2024
        mmp_magic = 4875943785348
        mmp_delay = 0
        mmp_valid = 0
        checkpoint_txg = 0
        labels = 0 1 2 3

Find the oldest stable TXG from the list. Note the txg and associated timestamp for use in the next steps.

3. Import the Pool Using a Specific TXG

Using the oldest stable TXG, import the pool in read-only mode and avoid making any changes.

Command to import with a specific TXG:

zpool export <pool>
zpool import -N -o readonly=on -f -R /mnt -F -T <txg> <pool>

Here:

-N prevents mounting the pool automatically.
-R /mnt sets the root directory for the pool.
-T <txg> specifies the transaction group to roll back to.

4. Mount and Recover Files

Once the pool is imported, manually mount its datasets to a temporary directory (e.g., /mnt) and copy the necessary files to a different storage location (preferably on another pool or filesystem).

Commands to mount and copy:

zfs mount <dataset>
mount -t zfs <pool>/<dataset> /mnt
cd /mnt

Inspect the files and copy them

cp -r /mnt/<path_to_files> /<path_to_backup>

Ensure you back up all important data before proceeding further as the transaction group will most likely be overwritten on recovering copying. You can copy the whole structure, please be advised snapshots won't work because the pool is readonly.

5. Export and Re-import the Pool

Once the necessary files are safely backed up, export the pool and attempt to re-import it in read-write mode. This reverts the pool to its current state, preserving consistency.

Commands to export and re-import:

zpool export <pool>
zpool import -f <pool>

After importing the pool, return the recovered files to their original location.

cp -r /<path_to_backup> /<pool_mount>/<path_to_files>

6. Verify Pool Health

Perform a scrub to check the integrity of the pool and resolve any remaining issues:

zpool scrub <pool>

Monitor the status of the scrub:

zpool status -v

7. Prevent Future Issues

To avoid data loss and pool corruption in the future:

Maintain regular snapshotting and consistency checking.
Use the standard 3-2-1 backup strategy with testing. ZFS supports snapshotting over network via send/receive to provide cloning and testing in one.
Maintain regular backups on a separate pool or filesystem.
Schedule routine scrubs to detect errors early.
Ensure the pool always has sufficient free space for operations.

This method provides a safe approach to recovering files from a ZFS pool while minimizing the risk of data inconsistencies. By manually selecting a stable TXG and recovering files before making changes, you can try to recover at least some of the files safely.