System Drive Backup and Recovery

The system drive (NVMe ext4) is rsynced daily to a ZFS mirror (tank/backup/system) and snapshotted, retaining 30 copies. This article covers the backup mechanism, daily checks, three-level recovery scenarios (single file → dataset rollback → bare-metal full system recovery), and lists items not included in the backup at the end. Scheduling is covered in timers-and-crons.md.

Overview

The initial full sync was executed on 2026-07-09, with the source being approximately 500G. Why not use block-level imaging: vg_sys space is fully allocated with no LVM snapshot capacity remaining; file-level backup allows direct browsing and single-file recovery, while ZFS snapshots add versioning, eliminating the need for a block-level solution.

Backup Mechanism

The script performs three tasks daily:

1. rsync -aHAXx --delete --numeric-ids / → $DEST/root/, then sync /boot/ → $DEST/boot/
2. zfs snapshot tank/backup/system@<timestamp>
3. Clean up based on creation time, keeping only the most recent 30 snapshots

Key parameters: -x prevents crossing filesystem boundaries (automatically skipping /proc, /sys, /tank, etc., leaving mount points as empty directories); -HAX preserves hard links/ACLs/xattrs, ensuring a complete system upon restoration; exit code 24 (file disappeared during copy) is considered normal—this is inevitable with online backups.

Exclusions (only content is excluded, directories themselves are preserved):

Docker data (/var/lib/docker) is included in the backup scope.

Consistency: rsync online backup is crash-consistent. There are no databases on the system drive (immich postgres is on tank), so the risk is acceptable; if stateful services are run on the system drive in the future, a dump step should be added before backup.

Daily Checks

systemctl list-timers backup-system.timer      # Next trigger time
systemctl is-active backup-system              # activating=running inactive=idle
journalctl -u backup-system -n 50              # Run logs
zfs list -t snapshot tank/backup/system        # Snapshot list (one should be added daily)
zfs list -o used,avail tank/backup/system      # Space usage

Long-term: There is currently no alert for backup failures. You can add an OnFailure= hook to the service to integrate with Bark push notifications (see monitoring.md).

Known Pitfalls: Non-UTF-8 Filenames

The tank pool has utf8only=on (pool creation attribute, immutable). Non-UTF-8 filenames will be rejected by ZFS, causing rsync to error with code 23, failing the entire backup and skipping the snapshot. This was encountered during the initial full sync (2026-07-09): GBK-encoded directories/shortcut names installed by Tencent games in Wine, where mkdir reported Invalid or incomplete multibyte or wide character.

Troubleshooting and Fix (convert to UTF-8 rather than excluding, for a permanent solution):

# Scan entire disk for non-UTF-8 filenames (-x is mandatory, otherwise empty matches pass)
find / -xdev \( -path /proc -o -path /sys \) -prune -o -print 2>/dev/null | grep -avx '.*'

# Rename one by one (GBK→UTF-8), in the problematic directory:
for f in *; do n=$(printf '%s' "$f" | iconv -f GBK -t UTF-8 2>/dev/null) || continue; [ -n "$n" ] && [ "$n" != "$f" ] && mv -n "$f" "$n"; done

The trigger source is running Wine Chinese game installers under a GBK locale. It is worth running a scan after installing new games; the backup log (journalctl -u backup-system) will directly point out the problematic path with recv_generator: mkdir ... failed.

Scenario 1: Accidental Deletion/Corruption of a Single File

Snapshots are readable online as hidden directories; simply copy back:

ls /tank/backup/system/.zfs/snapshot/                # List all time points
cp -a /tank/backup/system/.zfs/snapshot/20260709-0400/root/etc/foo.conf /etc/foo.conf

.zfs is not visible in ls -a, but you can cd into it directly. The latest backup (current state before snapshot) can be read directly from /tank/backup/system/root/.

Scenario 2: Bad State Synced into Backup

If accidental deletion/poisoning occurs and the backup runs before discovery, the current backup tree is already bad—no need to rollback (which would destroy newer snapshots); simply copy back from an earlier snapshot directory as in Scenario 1. Only when you confirm you want to revert the entire state:

zfs rollback -r tank/backup/system@<earlier_snapshot>   # -r destroys all subsequent snapshots, use with caution

Scenario 3: Bare-Metal Full System Recovery (System Drive Failure/Replacement)

Requires a LiveUSB with ZFS support (SystemRescue is sufficient). The entire process takes about 1–2 hours, with the bulk being HDD read-back of 400G+.

1. Rebuild Partitions and LVM (fstab uses device names, VG/LV names remain vg_sys/lv_root, no UUID dependency):

parted /dev/nvme0n1 -- mklabel gpt \
  mkpart ESP fat32 1MiB 1GiB set 1 esp on \
  mkpart lvm 1GiB 100%
mkfs.vfat -F32 /dev/nvme0n1p1
pvcreate /dev/nvme0n1p2
vgcreate vg_sys /dev/nvme0n1p2
lvcreate -L 32G -n lv_swap vg_sys && mkswap /dev/vg_sys/lv_swap
lvcreate -l 100%FREE -n lv_root vg_sys && mkfs.ext4 /dev/vg_sys/lv_root

2. Import Pool and Restore:

zpool import -o readonly=on tank        # Read-only import to protect backup
mount /dev/vg_sys/lv_root /mnt/root
rsync -aHAX /tank/backup/system/root/ /mnt/root/
mount /dev/nvme0n1p1 /mnt/root/boot
rsync -a /tank/backup/system/boot/ /mnt/root/boot/

3. Chroot and Reinstall Bootloader (GRUB EFI, /boot/EFI/Gentoo):

for d in dev proc sys; do mount --rbind /$d /mnt/root/$d; done
chroot /mnt/root /bin/bash
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=Gentoo
grub-mkconfig -o /boot/grub/grub.cfg
exit

4. Cleanup:

umount -R /mnt/root
zpool export tank
reboot

After reboot, confirm: systemctl --failed shows no anomalies, zpool status is ONLINE after zpool import tank, and the backup timer is present in systemctl list-timers.

Not Included in Backup

Long-term: Irreplaceable data like photos exists only as a single pool mirror. It is recommended to add offsite backup (restic → cloud object storage, or offsite machine zfs send).

Related Documentation

• timers-and-crons.md — Global view of scheduled tasks
• monitoring.md — Alert system (backup failure alerts pending integration)
• server-lifecycle.md — Server lifecycle