--- title: Storage Operations Manual url: https://doc.liz6.com/en/devops/storage-ops locale: en area: devops tags: - devops date: 2026-06-30 modified: 2026-07-16 description: Storage Operations Manual Storage layer operations including LVM, file systems, mounting, and swap. For disk viewing/SMART, see disk-ops.md. LVM (Logical Volume… --- # Storage Operations Manual Storage layer operations including LVM, file systems, mounting, and swap. For disk viewing/SMART, see `disk-ops.md`. ## LVM (Logical Volume Management) LVM three layers: PV (Physical Volume, disk/partition) → VG (Volume Group, PV pool) → LV (Logical Volume, block device that can be formatted). ```bash # === PV Layer === pvdisplay # Details of all PVs pvs # Concise view pvcreate /dev/sdb1 # Create PV pvremove /dev/sdb1 # Remove (ensure it has been removed from VG first) pvresize /dev/sdb1 # Refresh PV size after partition expansion # === VG Layer === vgdisplay # Details of all VGs vgs # Concise view vgcreate vg0 /dev/sdb1 # Create VG, add a PV vgextend vg0 /dev/sdc1 # Add new PV to VG (expand pool) vgreduce vg0 /dev/sdc1 # Remove PV from VG (need to pvmove data first) vgremove vg0 # Delete VG (! Dangerous) # === LV Layer === lvdisplay # Details of all LVs lvs # Concise view lvcreate -L 50G -n data vg0 # Create 50G LV lvcreate -l 100%FREE -n data vg0 # Use all remaining space lvcreate -L 50G -n data vg0 --type raid1 # Mirror LV (requires 2 PVs) lvremove vg0/data # Delete LV (! Dangerous) lvextend -L +10G vg0/data # Expand by +10G lvextend -l +100%FREE vg0/data # Expand to all free space lvreduce -L -10G vg0/data # Shrink (! Use with caution, XFS does not support shrinking) # After expansion, extend the file system: resize2fs /dev/vg0/data # ext4 xfs_growfs /mnt/data # XFS (must be mounted) # Rename lvrename vg0 oldname newname # Snapshots (LVM thin recommended) lvcreate -L 5G -s -n snap_data vg0/data # Traditional snapshot lvcreate -L 5G -s -n snap_data vg0/data --permission r # Read-only snapshot lvconvert --merge vg0/snap_data # Restore snapshot (snapshot is automatically deleted after restore) lvs -a # View snapshots (including hidden LVs) # Scan/Activate vgscan --mknodes # Rescan VGs and create device nodes vgchange -ay vg0 # Activate all LVs in VG vgchange -an vg0 # Deactivate VG ``` ## File Systems ```bash # Create mkfs.ext4 -L mylabel /dev/sdb1 # ext4 with label mkfs.ext4 -L mylabel -m 0 /dev/sdb1 # -m 0 = do not reserve root space mkfs.xfs -L mylabel /dev/vg0/data # XFS mkfs.btrfs -L mylabel /dev/sdb1 # btrfs mkfs.vfat -F32 /dev/sdb1 # FAT32 (EFI/USB drive) mkfs.exfat /dev/sdb1 # exFAT (large file USB drive) mkswap -L swap /dev/vg0/swap # swap # Labels e2label /dev/sdb1 mylabel # Change ext label xfs_admin -L mylabel /dev/vg0/data # Change XFS label btrfs filesystem label /dev/sdb1 mylabel # Change btrfs label # View blkid # UUID + FS type findfs UUID=abc123 # UUID → device path dumpe2fs /dev/sdb1 | head -40 # ext superblock info xfs_info /mnt/data # XFS info btrfs filesystem show # btrfs overview btrfs filesystem df /mnt # btrfs space usage # Repair (must umount first) fsck /dev/sdb1 # ext (auto-repair non-severe issues) fsck -y /dev/sdb1 # Auto-yes for all fsck -n /dev/sdb1 # Check only, no repair xfs_repair /dev/vg0/data # XFS repair (faster than fsck) xfs_repair -n /dev/vg0/data # XFS check only btrfs check /dev/sdb1 # btrfs check (read-only) btrfs check --repair /dev/sdb1 # btrfs repair (! Use with caution) # Force check (on next reboot) tune2fs -c 1 /dev/sdb1 # fsck on next mount touch /forcefsck # Force check for root partition (systemd) ``` ## Mounting (mount / fstab) ```bash # Temporary mount mount /dev/sdb1 /mnt/data mount -t ext4 /dev/sdb1 /mnt/data # Specify type mount -o ro,noexec /dev/sdb1 /mnt/data # Read-only + no execute mount -o remount,rw / # Remount as read-write mount -o remount,ro / # Remount as read-only umount /mnt/data # Unmount umount -l /mnt/data # Lazy unmount (unmount when no longer in use) umount -f /mnt/data # Force unmount (when NFS etc. is unresponsive) fuser -mv /mnt/data # Who is using this mount point # Persistence: /etc/fstab # UUID=abc123 /mnt/data ext4 defaults,noatime 0 2 # /dev/vg0/swap none swap sw 0 0 # //server/share /mnt/smb cifs credentials=/etc/smb.cred,uid=1000 0 0 # Column meanings: device mountpoint type options dump fsck_order # dump: 0=do not backup, 1=backup # fsck_order: 0=do not check, 1=root (first), 2=others # Common fstab options: # defaults = rw,suid,dev,exec,auto,nouser,async # noatime Do not update access time (recommended for SSDs, reduces writes) # relatime Update only when atime < mtime (compromise) # nodiratime Do not update directory access time # discard TRIM (for NVMe, kernel built-in discard is better) # x-systemd.automount Mount only when accessed (lazy mount) # x-systemd.device-timeout=10 Wait for device up to 10s # Test fstab mount -a # Mount all unmounted items in fstab findmnt --verify # Check fstab syntax # View mounts findmnt # Mount tree findmnt -t ext4,xfs # Filter by type findmnt /home # View mount source and options for /home ``` ## Swap ```bash # View swapon --show # Current swap status (device/size/usage) free -h # Includes swap usage cat /proc/swaps # Kernel perspective # Manage swapon /dev/vg0/swap # Enable swapoff /dev/vg0/swap # Disable mkswap -L swap /dev/vg0/swap # Create swap signature swapon -a # Enable all swaps in fstab # Swap file (alternative) dd if=/dev/zero of=/swapfile bs=1M count=4096 # Create 4G chmod 600 /swapfile mkswap /swapfile swapon /swapfile # fstab: /swapfile none swap sw 0 0 # Swap priority (when multiple swaps exist) # swapon -p 10 /dev/sda1 # 0-32767, higher is higher priority # fstab: ... sw pri=10 0 0 # Tuning sysctl vm.swappiness # Current swappiness (0-100, lower = less swap usage) # Desktop recommended 1-10, server depends on situation ``` ## Common btrfs Operations ```bash # Subvolumes btrfs subvolume list / # List subvolumes btrfs subvolume create /mnt/@data # Create subvolume btrfs subvolume delete /mnt/@data # Delete subvolume btrfs subvolume snapshot /mnt/@data /mnt/@data-snap # Snapshot # Compression (mount option) # mount -o compress=zstd /dev/sdb1 /mnt # fstab: UUID=... /mnt btrfs compress=zstd,noatime 0 0 # Scrub (data integrity check) btrfs scrub start /mnt btrfs scrub status /mnt btrfs scrub cancel /mnt # Device management btrfs device add /dev/sdc1 /mnt # Add device (expand) btrfs device delete /dev/sdc1 /mnt # Remove device btrfs balance start /mnt # Rebalance data distribution # RAID (btrfs native, do not overlay mdadm) # mkfs.btrfs -m raid1 -d raid1 /dev/sdb1 /dev/sdc1 ``` ## Partition Operations ```bash # View fdisk -l # MBR gdisk -l # GPT (recommended) parted -l # Supports both # Interactive partitioning gdisk /dev/sdb # GPT # n → new, d → delete, w → write, q → quit without saving # Non-interactive parted /dev/sdb mklabel gpt # Create GPT partition table parted /dev/sdb mkpart primary ext4 0% 100% # One partition takes up all space parted /dev/sdb mkpart primary ext4 1MiB 51201MiB # Exact size # Partition expansion (after partitioning with unallocated space) growpart /dev/sdb 1 # Expand partition 1 to max (commonly used in cloud-init) # Then: resize2fs /dev/sdb1 or xfs_growfs /mnt # Partition table backup/restore sgdisk --backup=table.bak /dev/sdb # Backup GPT sgdisk --load-backup=table.bak /dev/sdb # Restore ``` --- ## ZFS (Zettabyte File System) ### Pool: tank | Property | Value | |------|-----| | Type | Mirror (2-way) | | Disks | WDC WD40EFZZ 4TB ×2 (D4-320 USB3) | | ashift | 12 (4K) | | ARC Limit | 32GB (/etc/modprobe.d/zfs.conf) | | Mount Point | /tank | | Created | 2026-07-02 | ```bash # Device identifiers (by-id, stable) /dev/disk/by-id/ata-WDC_WD40EFZZ-68CPAN0_WD-WX12DA5H2PCN → sda /dev/disk/by-id/ata-WDC_WD40EFZZ-68CPAN0_WD-WX12DA5H234S → sdb ``` ### Dataset Layout | Dataset | recordsize | compression | Purpose | Shared | |---------|------------|-------------|------|------| | tank | 128K | lz4 | Root | - | | tank/media | 1M | lz4 | Home movie library | SMB + NFS | | tank/photos | 128K | lz4 | Home photo album | SMB + NFS | | tank/data | 128K | lz4 | Additional storage | SMB + NFS | | tank/backup | 128K | lz4 | Backup container | Not shared | | tank/backup/system | 1M | lz4 | System disk backup (restic) | Not shared | ### Common Operations ```bash # Status and Health zpool status tank # Pool + VDEV status zpool list -v tank # Pool size/usage zfs list -r tank # Dataset tree zpool events tank -v # Event history # Scrub (data integrity check, automatically executed monthly) zpool scrub tank # Manually trigger zpool scrub -s tank # Stop zpool status tank | grep scan # View progress # Snapshots zfs snapshot tank/documents@before-reorg # Create zfs list -t snapshot # List all snapshots zfs rollback tank/documents@before-reorg # Rollback (discards changes made after!) zfs destroy tank/documents@old-snap # Delete snapshot # Disk Replacement (when mirror is degraded) zpool offline tank # After physical disk replacement: zpool replace tank zpool status -v tank # Monitor resilver # Import/Export zpool export tank # Safe unmount (before migration) zpool import -d /dev/disk/by-id/ tank # Import # Performance Monitoring arc_summary # ARC cache detailed statistics zpool iostat -v 1 # Real-time IO (per second) ``` ### Service List | Service | systemd unit | Description | |------|-------------|------| | Samba | smb.service | Guest share /tank/{media,photos,data} | | NFS | nfs-server.service | Export to 192.168.31.0/24 | | ZFS Mount | zfs-mount.service | Auto-mount datasets on boot | | ZFS Events | zfs-zed.service | Disk failure email notification (root) | | ZFS Metrics | zfs-prometheus.timer | Collect every 5 minutes → Prometheus | | Monthly Scrub | zfs-scrub-monthly@tank.timer | Monthly data integrity check | ### Grafana Monitoring - Dashboard: **Host Status / li-home-0 · ZFS Pool** (uid: host-li-home-0-zfs) - Alert Rules: `/etc/grafana/provisioning/alerting/zfs-storage.yaml` - pool DEGRADED/FAULTED → critical - Capacity >80% → warning, >90% → critical - Single VDEV failure → critical - ARC hit rate <50% → warning ### Recovery Procedures #### Single Disk Failure (mirror survives) 1. `zpool status tank` to confirm faulty disk ID 2. Physically replace the faulty disk 3. `zpool replace tank ` 4. `zpool status -v tank` to monitor resilver completion #### Pool Cannot Be Imported 1. `ls /dev/disk/by-id/ata-*` to confirm disks are visible 2. `zpool import -f -d /dev/disk/by-id/ tank` to force import 3. If already exported: `zpool import -d /dev/disk/by-id/ -a` #### Samba Unavailable 1. `systemctl status smb` 2. `zfs mount` to confirm /tank is mounted 3. `testparm && systemctl restart smb` #### NFS Unavailable 1. `systemctl status nfs-server` 2. `showmount -e localhost` 3. `exportfs -ra` #### USB Disk Disconnection (D4-320) 1. Check if USB cable is loose 2. `dmesg | tail -30` to check for USB disconnect 3. `echo "0224" | sudo -S modprobe -r usb-storage uas && echo "0224" | sudo -S modprobe uas usb-storage` 4. `zpool online tank ` (if it went offline automatically)