Every night, I have a system making a snapshot in my machines and transfering the snapshot to my backup server.

I recently noticed that for one dataset the snapshot transfer failed:

ssh machine "zfs send -c rpool/data@2021-11-26_00-10-01" | pv -L 10000000 | zfs recv -F backups/machine/data
warning: cannot send 'rpool/data@2021-11-26_00-10-01': Input/output error
1.17GiB 0:02:06 [9.54MiB/s]
cannot receive new filesystem stream: incomplete stream

In the machine, I scrubbed the pool and check the status:

zpool scrub rpool
zpool status rpool
  pool: rpool
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub in progress since Fri Nov 26 10:26:03 2021
    105G scanned at 8.11G/s, 964K issued at 74.2K/s, 124G total
    0B repaired, 0.00% done, no estimated completion time
config:

    NAME                                             STATE     READ WRITE CKSUM
    rpool                                            DEGRADED     0     0     0
      ata-WDC_WDS100T1B0A-00H9H0_164710800985-part4  DEGRADED     0     0     1  too many errors

errors: 8 data errors, use '-v' for a list

The status shows there are errors in the pool and the scan section shows that scrub is progress.

The command status -v shows where are the errors:

zpool status rpool -v

errors: Permanent errors have been detected in the following files:

        rpool/data:<0x0>
        /data/git/tmp/rust/src/test/ui/conditional-compilation
        rpool/data@2021-11-26_00-10-01:/.cargo/registry/index/github.com-1ecc6299db9ec823/.git/objects/pack/pack-d89989aaadc7c20739f2cb81d1257112d0cddd67.pack

To fix the errors, I restored the files from backup and destroyed the snapshot with errors or when zpool clear rpool when errors don't affect applications. To be able to create a new snapshot without errors, the pool has to be scrubbed again.

# destroy the snapshot with errors
zfs destroy rpool/data@2021-11-26_00-10-01
# scrub again
zpool scrub rpool
# create a new snapshot
zfs snapshot rpool/data@2021-11-26_00-10-01
# transfer to the backup machine
ssh machine "zfs send -c rpool/data@2021-11-26_00-10-01" | pv -L 10000000 | zfs recv -F backups/machine/data

hashtags: #zfs