Storage Changes and Recovery Manual

This guide covers partition, LVM, filesystem, mount and ZFS changes. Identify the real device with disk diagnostics before planning changes. /dev/vdb, vg_data/lv_data and /srv/data are placeholders, not discovery commands. Practice on disposable storage, not the only data copy.

Task	Start here
Capacity unchanged after growth	Grow each layer
New disk or mount	Create and verify mounts
Filesystem damage	Offline checks and recovery
Maintain the recorded tank pool	Local deployment record

Each layer must expose capacity upward

Identify the filesystem and backing layers

# Identify the device and filesystem serving the directory
findmnt -T /srv/data

Record SOURCE/FSTYPE/TARGET to select ext4/XFS/Btrfs steps. An unmounted directory may actually use the root disk.

# Trace filesystem, LV, PV, partition, and disk
lsblk -f

Encryption, thin, and RAID layers require their own capacity/health checks.

# Inspect PV capacity and VG membership
sudo pvs -o pv_name,pv_size,pv_free,vg_name

Confirm /dev/vdb1 is the intended PV using identity and stable paths, not name alone.

# Check allocatable VG free space
sudo vgs -o vg_name,vg_size,vg_free

If VFree suffices, skip PV growth and plan the LV increase.

# Inspect LV type, size, and thin data/metadata usage
sudo lvs -a -o lv_name,vg_name,lv_size,segtype,data_percent,metadata_percent

Linear LVs and thin pools need different capacity reasoning. Record the filesystem baseline with df -hT /srv/data.

Preparing the visual

The model shows a conventional linear LV, not thin pools, RAID or encryption. Grow the backing disk/partition before pvresize; if VG free is sufficient, start at the LV. A larger block device does not imply a larger filesystem.

Resize the PV only after its backing device grows

This covers a linear LV. If the PV is partition-backed, enlarge the same partition and confirm the kernel sees its new boundary before pvresize.

# Let the target PV use the enlarged backing space
sudo pvresize /dev/vdb1

Recheck pvs/vgs for PSize/VFree growth. If absent, inspect partitions/mappings before extending the LV.

# Increase the verified LV by 10 GiB
sudo lvextend -L +10G /dev/vg_data/lv_data

+10G is an increment; 10G without plus is an absolute size. Verify VFree first, then LSize afterward. Grow the filesystem separately next.

Run pvresize only after confirming that partition is the grown PV. +10G is an increment and requires free extents. Then choose the matching filesystem command: ext4 takes the device, XFS the mounted path. Verify both lvs and df. lvextend -r delegates resizing but still needs support and result checks.

Choose one growth command by FSTYPE

# ext4 only: grow into the enlarged LV
sudo resize2fs /dev/vg_data/lv_data

The argument is a block device. Online growth requires support. Inspect failures; lvreduce is not a generic rollback.

# XFS only: grow the mounted filesystem
sudo xfs_growfs /srv/data

The argument is a mountpoint. This is an alternative to ext4; verify the XFS mount with findmnt first.

# Verify filesystem capacity actually increased
df -hT /srv/data

Compare the baseline and lvs, then test application writes. If only lower layers grew, another layer remains.

Shrinking is not reversed lvextend: where supported, shrink the filesystem offline before the LV/partition; wrong boundaries truncate data. Do not apply ext4 shrink procedures to XFS. Migration to a smaller new filesystem is often more controllable. Thin pools require both metadata and data monitoring.

Creation, mounting and boot configuration are separate

Inspect signatures and existing use

# List recognized disk signatures without writing
sudo wipefs -n /dev/vdb

-n prevents writes. Empty output alone does not prove no data; compare partitions, PVs, RAID, and inventory.

# Read partition layout, size, and boundaries
sudo fdisk -l /dev/vdb

Back up an existing partition table. Enter fdisk only for a confirmed writable target; inspect p before writing and q to cancel. Keep partitioning separate from formatting.

# Read filesystem UUIDs and types for mounting
sudo blkid

fstab needs the new filesystem UUID, not an old disk or PV UUID.

wipefs -n only inspects signatures; mkfs, pvcreate and partition writes alter storage. Confirm empty media, backup and layout before interactive creation. fdisk supports GPT too. New LVM storage follows pvcreate → vgcreate/vgextend → lvcreate → mkfs, verifying the target at every boundary.

Creation example: a confirmed empty, disposable partition, using a new VG and ext4. Inspect first and verify it is not mounted, swap, an existing PV or RAID member. Run creation commands individually and stop on any failure; do not continue to mkfs. vg_newdata must be new, never an existing data LV.

New LVM: identify the empty partition first

# Set the verified empty partition’s stable path
ops_part=/dev/disk/by-id/REPLACE_WITH_CONFIRMED_EMPTY_PARTITION

Replace this value and keep the same shell for this workflow. Use the intended partition, not an entire disk or existing data partition.

# Inspect the target’s layers and filesystem use
lsblk -f "${ops_part:?Set the verified partition}"

Also inspect wipefs -n, pvs, and swapon --show, plus RAID/encryption membership. Stop on uncertainty about existing use.

# Initialize the partition as a PV only after confirming overwrite intent
sudo pvcreate "${ops_part:?Set the verified partition}"

Verify with pvs before proceeding. This writes metadata; deleting later LVM objects cannot restore previous disk contents.

# Create a new VG from this PV
sudo vgcreate vg_newdata "${ops_part:?Set the verified partition}"

Check vgs capacity/free space. If the name exists, stop rather than modifying existing storage unintentionally.

# Allocate a 10 GiB LV in the new VG
sudo lvcreate -L 10G -n lv_data vg_newdata

Confirm free space, then verify the new name and size with lvs. Format only this newly created LV.

# Create ext4 only on the new LV
sudo mkfs.ext4 /dev/vg_newdata/lv_data

This writes filesystem structures; it is not a check of an existing filesystem. Obtain the mount identity next.

# Record the new filesystem UUID and type
sudo blkid /dev/vg_newdata/lv_data

Confirm TYPE=ext4 and use this UUID in the mount workflow; set appropriate application ownership/permissions.

Create the partition with the appropriate tool and verify its stable path first, not historical sda/sdb names. Put the resulting UUID in the intended fstab entry, then mount and verify the device, capacity and application access using the workflow below. The 10G allocation requires enough free VG space.

UUID=REPLACE_WITH_REAL_UUID /srv/data ext4 defaults 0 2

Inspect the directory before mounting

# Prepare the mount directory
sudo mkdir -p /srv/data

Inspect existing contents; mounting hides rather than migrates them. Add the adapted fstab entry while retaining a backup and other entries.

# Validate fstab and mount metadata
sudo findmnt --verify --verbose

Fix failures first. Passing does not grant application permissions.

# Refresh generated mount units on systemd hosts
sudo systemctl daemon-reload

This reloads manager configuration, not the mount itself.

# Activate only the reviewed fstab mount
sudo mount /srv/data

Verify SOURCE/type/options with findmnt, then df and application-user access.

# Identify users when unmounting is blocked
sudo fuser -vm /srv/data

Stop the owning application and finish writes before sudo umount. Do not confuse fuser’s kill option with inspection.

The fstab example is for that ext4 case; design other filesystem options separately. Mounting hides existing directory contents, not moves them. Test one target before activating unrelated entries. Inspect busy users with fuser -vm; lazy unmount detaches a namespace path without proving references are gone or formatting is safe.

Checks, snapshots and backups have different boundaries

Goal	Method and precondition
ext4	Use e2fsck -fn on an unmounted device for assessment; back up before repair writes
XFS	Assess unmounted storage with xfs_repair -n; handle dirty logs deliberately, not default -L
Btrfs	Scrub verifies checksums and may repair from redundancy; check handles structure, --repair is not routine
LVM snapshot	Classic COW snapshots need monitored reserve space and can invalidate when full; thin snapshots differ
Snapshot ≠ backup	Pool failure can lose both live data and snapshots; keep and restore-test an independent copy

Confirm the filesystem is unmounted before assessment

Stop applications, preserve readable data, inspect hardware, and verify no mount with findmnt -S DEVICE. Use rescue media for root filesystems. ext4/XFS checks are alternatives.

# ext4 only: assess the unmounted target without writes
sudo e2fsck -fn /dev/vg_data/lv_data

-f forces checking and -n declines repairs. Interpret nonzero results; decide repair separately after preserving a copy, not by automatically switching to -y.

# XFS only: assess the unmounted filesystem read-only
sudo xfs_repair -n /dev/vg_data/lv_data

Follow diagnostics for dirty logs; -L discards logs and can lose data, so it is not a default next step.

Inspect Btrfs capacity, subvolumes, and scrub separately

# Btrfs only: read data/metadata allocation and free space
sudo btrfs filesystem usage /srv/data

Metadata pressure is not captured by df free space alone; consider redundancy profiles.

# List subvolumes in this Btrfs filesystem
sudo btrfs subvolume list /srv/data

Distinguish subvolumes from ordinary directories; snapshots do not recursively copy nested subvolume contents.

# Inspect current scrub state and errors first
sudo btrfs scrub status /srv/data

Start a scrub separately only when needed, schedule I/O load, and inspect final status. This differs from structural check --repair.

Swap: inspect use before changing it

# Inspect swap devices, sizes, use, and priority
swapon --show

Compare free -h before disabling swap; swapoff moves data back to RAM and may create pressure.

# Read the current swap-versus-file reclaim preference
sysctl vm.swappiness

This is a reclaim preference, not a memory-use threshold. Record baselines and validate workload effects against kernel documentation.

Run Btrfs commands only there. Swapfiles have hole/COW restrictions; use the filesystem-supported Btrfs creation method. swapoff needs memory to absorb pages. Swappiness is not restricted to 0–100; assess memory, swap medium and workload.

tank: retained local deployment record

Recorded by-id values (verify GUIDs and physical slots before maintenance):

/dev/disk/by-id/ata-WDC_WD40EFZZ-68CPAN0_WD-WX12DA5H2PCN
/dev/disk/by-id/ata-WDC_WD40EFZZ-68CPAN0_WD-WX12DA5H234S

This is the 2026-07-02 deployment record, not a live inventory. Reconcile it with zpool status, zfs get and installed units. Local alert thresholds are not universal recommendations; historical sda/sdb names are not replacement identifiers.

Item	Record
Pool	tank · 2-way mirror · WD40EFZZ 4TB × 2 · D4-320 USB3
ashift / ARC	12 / 32GB (/etc/modprobe.d/zfs.conf)
Mount	/tank

Dataset	recordsize	compression	Purpose / share
tank	128K	lz4	Root
tank/media	1M	lz4	Movies / SMB + NFS
tank/photos	128K	lz4	Photos / SMB + NFS
tank/data	128K	lz4	Data / SMB + NFS
tank/backup	128K	lz4	Backup parent / private
tank/backup/system	1M	lz4	restic system backup / private

Verify tank device identity and health

# Inspect GUIDs, resolved paths, and detailed errors
zpool status -gLPv tank

-g shows GUIDs, -L resolves links, -P shows full paths, and -v details data errors. Map them to stable IDs and slots before replacement.

# Read pool/vdev size and allocation
zpool list -v tank

Pool capacity differs from dataset availability because of quotas/reservations and other constraints.

# Inspect dataset hierarchy, usage, and mountpoints
zfs list -r tank

Inspect USED/AVAIL/REFER/MOUNTPOINT. Snapshots and children affect usage beyond du.

# Inspect values and local/inherited property sources
zfs get -r mountpoint,compression,recordsize tank

Record sources before changes; restoring inheritance differs from setting a value. Historical tables are not instructions to overwrite current settings.

Events, throughput, and cache provide different evidence

# Inspect global ZFS events and correlate pool/time
zpool events -v

This covers all pools; tank is not a positional argument here.

# Compare per-vdev I/O continuously
zpool iostat -v tank 1

The first report is generally cumulative; later reports are interval samples. Inspect imbalance, then Ctrl-C. Query ARC separately with arc_summary.

zpool events lists global events; tank is not its positional pool argument. recordsize controls file block sizing and does not rewrite existing files. The ARC cap is local policy; distinguish cold starts and working sets when interpreting hit rates.

Scrub and snapshots are separate tasks

# Start pool verification in a maintenance window
sudo zpool scrub tank

Check no scrub/resilver is already running. Observe progress and final errors with status; command return is not completion.

# Create a uniquely named pre-change snapshot
sudo zfs snapshot tank/data@before-reorg

Coordinate application consistency first. Filesystem snapshots do not guarantee cross-service transactions; this targets one dataset, not an implicit recursive set.

# Verify snapshot existence and referenced space
zfs list -t snapshot -r tank/data

Match the name/time to this change. A same-pool snapshot does not protect against pool loss.

# Compare snapshot paths with the current dataset
sudo zfs diff tank/data@before-reorg

This reports path changes, not content patches. Copy a file from the snapshot directory to a separate review location, verify content/permissions, then replace only the intended file. Avoid dataset-wide rollback for a single-file recovery.

Schedule scrub load and inspect completion; a mirror may repair from a healthy replica but is not backup. Rollback discards later changes. Prefer snapshot browsing or a separate clone for individual-file recovery. Snapshot destruction also affects recovery and incremental-send bases.

Service	unit / path
SMB / NFS	smb.service / nfs-server.service
ZFS mount / events	zfs-mount.service / zfs-zed.service
Metrics / monthly scrub	zfs-prometheus.timer / zfs-scrub-monthly@tank.timer
Grafana dashboard UID	host-li-home-0-zfs
Grafana alert file	/etc/grafana/provisioning/alerting/zfs-storage.yaml

Recorded shares cover media/photos/data; NFS clients use 192.168.31.0/24. Historical alerts use 80%/90% capacity and 50% ARC hit rate; reassess them. For share failures, verify the actual dataset mount before testparm, exportfs -v and network checks. showmount alone does not prove NFSv4 health.

Recover the identified failure scope

Failure	Action / completion evidence
One mirror member failed	Record GUID/by-id/slot and verify survivor/backup; replace the identified member, then verify resilver and errors
Import fails	Inspect devices and import candidates first; establish exclusive pool ownership before targeted import, not blanket -f/-a
USB disconnect	Check cable/power/kernel logs, then stable identifiers and pool status; do not blindly rescan buses or online all disks

LVM lvextend · Btrfs administration · OpenZFS manuals · Kernel swappiness