---
title: System Drive Backup and Recovery
url: https://doc.liz6.com/en/homelab/system-backup
locale: en
area: homelab
tags:
- homelab
date: 2026-07-09
modified: 2026-07-16
description: Daily rsync of the system drive (NVMe ext4) to a ZFS mirror (tank/backup/system) with snapshots, retaining 30 copies. This article covers the backup mechanism, daily checks, three-level recovery scenarios (single file → dataset rollback → bare-metal full system recovery), and lists items not included in the backup at the end. Scheduling is covered in timers-and-crons.md.
---

# System Drive Backup and Recovery

> The system drive (NVMe ext4) is rsynced daily to a ZFS mirror (`tank/backup/system`) and snapshotted, retaining 30 copies. This article covers the backup mechanism, daily checks, three-level recovery scenarios (single file → dataset rollback → bare-metal full system recovery), and lists items not included in the backup at the end. Scheduling is covered in [timers-and-crons.md](timers-and-crons.md).

## Overview

| Item | Value |
|----|-----|
| Source | `/`(ext4,`vg_sys/lv_root`,898.5G)+ `/boot`(vfat,`nvme0n1p1`) |
| Target | `/tank/backup/system`(2×4T WD Red ZFS mirror, lz4 compression) |
| Method | rsync file-level sync (full initially, incremental thereafter) + `zfs snapshot` after each completion |
| Retention | Keep the most recent 30 snapshots; excess are automatically destroyed |
| Schedule | systemd timer, daily at 04:00, `Persistent=true` + idle IO priority |
| Script | `/usr/local/sbin/backup-system.sh` |

The initial full sync was executed on 2026-07-09, with the source being approximately 500G. Why not use block-level imaging: `vg_sys` space is fully allocated with no LVM snapshot capacity remaining; file-level backup allows direct browsing and single-file recovery, while ZFS snapshots add versioning, eliminating the need for a block-level solution.

## Backup Mechanism

The script performs three tasks daily:

1. `rsync -aHAXx --delete --numeric-ids / → $DEST/root/`, then sync `/boot/ → $DEST/boot/`
2. `zfs snapshot tank/backup/system@<timestamp>`
3. Clean up based on creation time, keeping only the most recent 30 snapshots

Key parameters: `-x` prevents crossing filesystem boundaries (automatically skipping /proc, /sys, /tank, etc., leaving mount points as empty directories); `-HAX` preserves hard links/ACLs/xattrs, ensuring a complete system upon restoration; exit code 24 (file disappeared during copy) is considered normal—this is inevitable with online backups.

**Exclusions** (only content is excluded, directories themselves are preserved):

| Path | Reason |
|------|------|
| `/tmp/*`, `/var/tmp/*` | Temporary files, including portage build directories |
| `/var/cache/distfiles/*`, `/var/cache/binpkgs/*` | Gentoo source/binary package caches, can be re-downloaded and recompiled |
| `/lost+found` | ext4 metadata |

Docker data (`/var/lib/docker`) **is included** in the backup scope.

**Consistency**: rsync online backup is crash-consistent. There are no databases on the system drive (immich postgres is on tank), so the risk is acceptable; if stateful services are run on the system drive in the future, a dump step should be added before backup.

## Daily Checks

```bash
systemctl list-timers backup-system.timer      # Next trigger time
systemctl is-active backup-system              # activating=running inactive=idle
journalctl -u backup-system -n 50              # Run logs
zfs list -t snapshot tank/backup/system        # Snapshot list (one should be added daily)
zfs list -o used,avail tank/backup/system      # Space usage
```

> **Long-term**: There is currently no alert for backup failures. You can add an `OnFailure=` hook to the service to integrate with Bark push notifications (see [monitoring.md](monitoring.md)).

## Known Pitfalls: Non-UTF-8 Filenames

The `tank` pool has `utf8only=on` (pool creation attribute, immutable). Non-UTF-8 filenames will be rejected by ZFS, causing rsync to error with code 23, **failing the entire backup and skipping the snapshot**. This was encountered during the initial full sync (2026-07-09): GBK-encoded directories/shortcut names installed by Tencent games in Wine, where `mkdir` reported `Invalid or incomplete multibyte or wide character`.

**Troubleshooting and Fix** (convert to UTF-8 rather than excluding, for a permanent solution):

```bash
# Scan entire disk for non-UTF-8 filenames (-x is mandatory, otherwise empty matches pass)
find / -xdev \( -path /proc -o -path /sys \) -prune -o -print 2>/dev/null | grep -avx '.*'

# Rename one by one (GBK→UTF-8), in the problematic directory:
for f in *; do n=$(printf '%s' "$f" | iconv -f GBK -t UTF-8 2>/dev/null) || continue; [ -n "$n" ] && [ "$n" != "$f" ] && mv -n "$f" "$n"; done
```

> The trigger source is running Wine Chinese game installers under a GBK locale. It is worth running a scan after installing new games; the backup log (`journalctl -u backup-system`) will directly point out the problematic path with `recv_generator: mkdir ... failed`.

## Scenario 1: Accidental Deletion/Corruption of a Single File

Snapshots are readable online as hidden directories; simply copy back:

```bash
ls /tank/backup/system/.zfs/snapshot/                # List all time points
cp -a /tank/backup/system/.zfs/snapshot/20260709-0400/root/etc/foo.conf /etc/foo.conf
```

> `.zfs` is not visible in `ls -a`, but you can `cd` into it directly. The latest backup (current state before snapshot) can be read directly from `/tank/backup/system/root/`.

## Scenario 2: Bad State Synced into Backup

If accidental deletion/poisoning occurs and the backup runs before discovery, the current backup tree is already bad—no need to rollback (which would destroy newer snapshots); simply copy back from an earlier snapshot directory as in Scenario 1. Only when you confirm you want to revert the entire state:

```bash
zfs rollback -r tank/backup/system@<earlier_snapshot>   # -r destroys all subsequent snapshots, use with caution
```

## Scenario 3: Bare-Metal Full System Recovery (System Drive Failure/Replacement)

Requires a LiveUSB with ZFS support (SystemRescue is sufficient). The entire process takes about 1–2 hours, with the bulk being HDD read-back of 400G+.

**1. Rebuild Partitions and LVM** (fstab uses device names, VG/LV names remain `vg_sys`/`lv_root`, no UUID dependency):

```bash
parted /dev/nvme0n1 -- mklabel gpt \
  mkpart ESP fat32 1MiB 1GiB set 1 esp on \
  mkpart lvm 1GiB 100%
mkfs.vfat -F32 /dev/nvme0n1p1
pvcreate /dev/nvme0n1p2
vgcreate vg_sys /dev/nvme0n1p2
lvcreate -L 32G -n lv_swap vg_sys && mkswap /dev/vg_sys/lv_swap
lvcreate -l 100%FREE -n lv_root vg_sys && mkfs.ext4 /dev/vg_sys/lv_root
```

**2. Import Pool and Restore**:

```bash
zpool import -o readonly=on tank        # Read-only import to protect backup
mount /dev/vg_sys/lv_root /mnt/root
rsync -aHAX /tank/backup/system/root/ /mnt/root/
mount /dev/nvme0n1p1 /mnt/root/boot
rsync -a /tank/backup/system/boot/ /mnt/root/boot/
```

**3. Chroot and Reinstall Bootloader** (GRUB EFI, `/boot/EFI/Gentoo`):

```bash
for d in dev proc sys; do mount --rbind /$d /mnt/root/$d; done
chroot /mnt/root /bin/bash
grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=Gentoo
grub-mkconfig -o /boot/grub/grub.cfg
exit
```

**4. Cleanup**:

```bash
umount -R /mnt/root
zpool export tank
reboot
```

After reboot, confirm: `systemctl --failed` shows no anomalies, `zpool status` is ONLINE after `zpool import tank`, and the backup timer is present in `systemctl list-timers`.

## Not Included in Backup

| Content | Status |
|------|------|
| tank pool data (photos/media/immich) | Mirror redundancy only, **no second copy/offsite backup**, lost if all drives fail or are accidentally deleted |
| Excluded items (/tmp, portage cache, etc.) | Rebuildable; first `emerge` after recovery will re-download |
| swap LV | No backup needed; rebuild with `mkswap` during recovery |

> **Long-term**: Irreplaceable data like photos exists only as a single pool mirror. It is recommended to add offsite backup (restic → cloud object storage, or offsite machine `zfs send`).

## Related Documentation

- [timers-and-crons.md](timers-and-crons.md) — Global view of scheduled tasks
- [monitoring.md](monitoring.md) — Alert system (backup failure alerts pending integration)
- [server-lifecycle.md](server-lifecycle.md) — Server lifecycle