Storage Operations Manual

Storage layer operations including LVM, file systems, mounting, and swap. For disk viewing/SMART, see disk-ops.md.

LVM (Logical Volume Management)

LVM three layers: PV (Physical Volume, disk/partition) → VG (Volume Group, PV pool) → LV (Logical Volume, block device that can be formatted).

# === PV Layer ===
pvdisplay                                 # Details of all PVs
pvs                                       # Concise view
pvcreate /dev/sdb1                        # Create PV
pvremove /dev/sdb1                        # Remove (ensure it has been removed from VG first)
pvresize /dev/sdb1                        # Refresh PV size after partition expansion

# === VG Layer ===
vgdisplay                                 # Details of all VGs
vgs                                       # Concise view
vgcreate vg0 /dev/sdb1                    # Create VG, add a PV
vgextend vg0 /dev/sdc1                    # Add new PV to VG (expand pool)
vgreduce vg0 /dev/sdc1                    # Remove PV from VG (need to pvmove data first)
vgremove vg0                              # Delete VG (! Dangerous)

# === LV Layer ===
lvdisplay                                 # Details of all LVs
lvs                                       # Concise view
lvcreate -L 50G -n data vg0               # Create 50G LV
lvcreate -l 100%FREE -n data vg0          # Use all remaining space
lvcreate -L 50G -n data vg0 --type raid1  # Mirror LV (requires 2 PVs)
lvremove vg0/data                         # Delete LV (! Dangerous)
lvextend -L +10G vg0/data                 # Expand by +10G
lvextend -l +100%FREE vg0/data            # Expand to all free space
lvreduce -L -10G vg0/data                 # Shrink (! Use with caution, XFS does not support shrinking)

# After expansion, extend the file system:
resize2fs /dev/vg0/data                   # ext4
xfs_growfs /mnt/data                      # XFS (must be mounted)

# Rename
lvrename vg0 oldname newname

# Snapshots (LVM thin recommended)
lvcreate -L 5G -s -n snap_data vg0/data   # Traditional snapshot
lvcreate -L 5G -s -n snap_data vg0/data --permission r  # Read-only snapshot
lvconvert --merge vg0/snap_data           # Restore snapshot (snapshot is automatically deleted after restore)
lvs -a                                    # View snapshots (including hidden LVs)

# Scan/Activate
vgscan --mknodes                          # Rescan VGs and create device nodes
vgchange -ay vg0                          # Activate all LVs in VG
vgchange -an vg0                          # Deactivate VG

File Systems

# Create
mkfs.ext4 -L mylabel /dev/sdb1            # ext4 with label
mkfs.ext4 -L mylabel -m 0 /dev/sdb1       # -m 0 = do not reserve root space
mkfs.xfs -L mylabel /dev/vg0/data         # XFS
mkfs.btrfs -L mylabel /dev/sdb1           # btrfs
mkfs.vfat -F32 /dev/sdb1                  # FAT32 (EFI/USB drive)
mkfs.exfat /dev/sdb1                      # exFAT (large file USB drive)
mkswap -L swap /dev/vg0/swap              # swap

# Labels
e2label /dev/sdb1 mylabel                 # Change ext label
xfs_admin -L mylabel /dev/vg0/data        # Change XFS label
btrfs filesystem label /dev/sdb1 mylabel  # Change btrfs label

# View
blkid                                      # UUID + FS type
findfs UUID=abc123                         # UUID → device path
dumpe2fs /dev/sdb1 | head -40             # ext superblock info
xfs_info /mnt/data                         # XFS info
btrfs filesystem show                      # btrfs overview
btrfs filesystem df /mnt                   # btrfs space usage

# Repair (must umount first)
fsck /dev/sdb1                             # ext (auto-repair non-severe issues)
fsck -y /dev/sdb1                          # Auto-yes for all
fsck -n /dev/sdb1                          # Check only, no repair
xfs_repair /dev/vg0/data                   # XFS repair (faster than fsck)
xfs_repair -n /dev/vg0/data                # XFS check only
btrfs check /dev/sdb1                      # btrfs check (read-only)
btrfs check --repair /dev/sdb1             # btrfs repair (! Use with caution)

# Force check (on next reboot)
tune2fs -c 1 /dev/sdb1                     # fsck on next mount
touch /forcefsck                           # Force check for root partition (systemd)

Mounting (mount / fstab)

# Temporary mount
mount /dev/sdb1 /mnt/data
mount -t ext4 /dev/sdb1 /mnt/data          # Specify type
mount -o ro,noexec /dev/sdb1 /mnt/data     # Read-only + no execute
mount -o remount,rw /                      # Remount as read-write
mount -o remount,ro /                      # Remount as read-only
umount /mnt/data                           # Unmount
umount -l /mnt/data                        # Lazy unmount (unmount when no longer in use)
umount -f /mnt/data                        # Force unmount (when NFS etc. is unresponsive)
fuser -mv /mnt/data                        # Who is using this mount point

# Persistence: /etc/fstab
# UUID=abc123    /mnt/data    ext4    defaults,noatime    0 2
# /dev/vg0/swap  none         swap    sw                  0 0
# //server/share /mnt/smb    cifs    credentials=/etc/smb.cred,uid=1000  0 0

# Column meanings: device  mountpoint  type  options  dump  fsck_order
# dump: 0=do not backup, 1=backup
# fsck_order: 0=do not check, 1=root (first), 2=others

# Common fstab options:
# defaults = rw,suid,dev,exec,auto,nouser,async
# noatime     Do not update access time (recommended for SSDs, reduces writes)
# relatime    Update only when atime < mtime (compromise)
# nodiratime  Do not update directory access time
# discard     TRIM (for NVMe, kernel built-in discard is better)
# x-systemd.automount  Mount only when accessed (lazy mount)
# x-systemd.device-timeout=10  Wait for device up to 10s

# Test fstab
mount -a                                   # Mount all unmounted items in fstab
findmnt --verify                           # Check fstab syntax

# View mounts
findmnt                                    # Mount tree
findmnt -t ext4,xfs                        # Filter by type
findmnt /home                              # View mount source and options for /home

Swap

# View
swapon --show                              # Current swap status (device/size/usage)
free -h                                    # Includes swap usage
cat /proc/swaps                            # Kernel perspective

# Manage
swapon /dev/vg0/swap                       # Enable
swapoff /dev/vg0/swap                      # Disable
mkswap -L swap /dev/vg0/swap               # Create swap signature
swapon -a                                  # Enable all swaps in fstab

# Swap file (alternative)
dd if=/dev/zero of=/swapfile bs=1M count=4096  # Create 4G
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
# fstab: /swapfile none swap sw 0 0

# Swap priority (when multiple swaps exist)
# swapon -p 10 /dev/sda1       # 0-32767, higher is higher priority
# fstab: ... sw pri=10 0 0

# Tuning
sysctl vm.swappiness                        # Current swappiness (0-100, lower = less swap usage)
# Desktop recommended 1-10, server depends on situation

Common btrfs Operations

# Subvolumes
btrfs subvolume list /                      # List subvolumes
btrfs subvolume create /mnt/@data           # Create subvolume
btrfs subvolume delete /mnt/@data           # Delete subvolume
btrfs subvolume snapshot /mnt/@data /mnt/@data-snap  # Snapshot

# Compression (mount option)
# mount -o compress=zstd /dev/sdb1 /mnt
# fstab: UUID=... /mnt btrfs compress=zstd,noatime 0 0

# Scrub (data integrity check)
btrfs scrub start /mnt
btrfs scrub status /mnt
btrfs scrub cancel /mnt

# Device management
btrfs device add /dev/sdc1 /mnt             # Add device (expand)
btrfs device delete /dev/sdc1 /mnt          # Remove device
btrfs balance start /mnt                    # Rebalance data distribution

# RAID (btrfs native, do not overlay mdadm)
# mkfs.btrfs -m raid1 -d raid1 /dev/sdb1 /dev/sdc1

Partition Operations

# View
fdisk -l                                    # MBR
gdisk -l                                    # GPT (recommended)
parted -l                                   # Supports both

# Interactive partitioning
gdisk /dev/sdb                              # GPT
# n → new, d → delete, w → write, q → quit without saving

# Non-interactive
parted /dev/sdb mklabel gpt                 # Create GPT partition table
parted /dev/sdb mkpart primary ext4 0% 100% # One partition takes up all space
parted /dev/sdb mkpart primary ext4 1MiB 51201MiB  # Exact size

# Partition expansion (after partitioning with unallocated space)
growpart /dev/sdb 1                         # Expand partition 1 to max (commonly used in cloud-init)
# Then: resize2fs /dev/sdb1 or xfs_growfs /mnt

# Partition table backup/restore
sgdisk --backup=table.bak /dev/sdb          # Backup GPT
sgdisk --load-backup=table.bak /dev/sdb     # Restore

ZFS (Zettabyte File System)

Pool: tank

# Device identifiers (by-id, stable)
/dev/disk/by-id/ata-WDC_WD40EFZZ-68CPAN0_WD-WX12DA5H2PCN  → sda
/dev/disk/by-id/ata-WDC_WD40EFZZ-68CPAN0_WD-WX12DA5H234S  → sdb

Dataset Layout

Common Operations

# Status and Health
zpool status tank                         # Pool + VDEV status
zpool list -v tank                        # Pool size/usage
zfs list -r tank                          # Dataset tree
zpool events tank -v                      # Event history

# Scrub (data integrity check, automatically executed monthly)
zpool scrub tank                          # Manually trigger
zpool scrub -s tank                       # Stop
zpool status tank | grep scan             # View progress

# Snapshots
zfs snapshot tank/documents@before-reorg  # Create
zfs list -t snapshot                      # List all snapshots
zfs rollback tank/documents@before-reorg  # Rollback (discards changes made after!)
zfs destroy tank/documents@old-snap       # Delete snapshot

# Disk Replacement (when mirror is degraded)
zpool offline tank <faulty_disk_ID>
# After physical disk replacement:
zpool replace tank <old_disk_ID> <new_disk_ID>
zpool status -v tank                      # Monitor resilver

# Import/Export
zpool export tank                         # Safe unmount (before migration)
zpool import -d /dev/disk/by-id/ tank     # Import

# Performance Monitoring
arc_summary                               # ARC cache detailed statistics
zpool iostat -v 1                         # Real-time IO (per second)

Service List

Grafana Monitoring

• Dashboard: Host Status / li-home-0 · ZFS Pool (uid: host-li-home-0-zfs)
• Alert Rules: /etc/grafana/provisioning/alerting/zfs-storage.yaml
  • pool DEGRADED/FAULTED → critical
  • Capacity >80% → warning, >90% → critical
  • Single VDEV failure → critical
  • ARC hit rate <50% → warning

Recovery Procedures

Single Disk Failure (mirror survives)

1. zpool status tank to confirm faulty disk ID
2. Physically replace the faulty disk
3. zpool replace tank <old> <new>
4. zpool status -v tank to monitor resilver completion

Pool Cannot Be Imported

1. ls /dev/disk/by-id/ata-* to confirm disks are visible
2. zpool import -f -d /dev/disk/by-id/ tank to force import
3. If already exported: zpool import -d /dev/disk/by-id/ -a

Samba Unavailable

1. systemctl status smb
2. zfs mount to confirm /tank is mounted
3. testparm && systemctl restart smb

NFS Unavailable

1. systemctl status nfs-server
2. showmount -e localhost
3. exportfs -ra

USB Disk Disconnection (D4-320)

1. Check if USB cable is loose
2. dmesg | tail -30 to check for USB disconnect
3. echo "0224" | sudo -S modprobe -r usb-storage uas && echo "0224" | sudo -S modprobe uas usb-storage
4. zpool online tank <disk> (if it went offline automatically)