Disk Diagnostics Manual
On this page
Start by separating space, inodes, device health and I/O latency. This guide diagnoses them; partition, LVM and filesystem changes live in the storage manual. Replace device names and paths with verified local values.
| Task | Start here |
|---|---|
| Disk full | Reconciling df and du |
| Slow writes or stalls | Latency and load |
| I/O errors or read-only | Health and evidence |
Map the path to its device
# Identify the mount containing the directory
-T accepts a path. Read SOURCE/FSTYPE/TARGET before selecting tools; a data-looking directory may still reside on the root filesystem.
# Map the mount source down through block layers
Inspect layer types such as part/lvm/crypt/raid; SIZE is layer capacity, not free filesystem space.
# Match stable IDs to current device names
For maintenance verify model, serial, capacity, and slot together, not sdX alone.
# Read filesystem block usage and type
Near-100% Use% calls for space diagnosis; -h uses readable units and -T shows type.
# Check inode exhaustion separately
High IUse% may indicate many small files; inode allocation and reporting differ by filesystem.
Follow TYPE and parent relationships: the path may use LVM, encryption, container overlay or network storage. /dev/sdb can change across enumeration. Verify by-id, model, size and mounts together. df reports a filesystem, not free capacity across the whole disk.
Compare the same path and mount scope
Find usage within the same mount scope
# Summarize immediate directories without crossing filesystems
Drill into the largest relevant directory. -x stays on one filesystem; permission errors undercount, so read stderr. The scan itself generates I/O.
# Find deleted files still held open
Inspect COMMAND/PID/SIZE/OFF/NAME and match the device. Release handles through supported rotation or graceful restart, not arbitrary fd truncation.
# Inspect nested mounts when the path is a mountpoint
For an ordinary directory, get TARGET with -T and apply -R to that mountpoint. Nested mounts can hide underlying files and change comparison scope.
# Measure journal-specific usage
Use the systemd manual for rotation/retention rather than deleting active logs by directory size.
GNU du syntax: passing the directory includes hidden entries, unlike a shell * glob. -x stays on one filesystem. lsof +L1 finds unlinked open files; identify the owner and rotation policy, then release handles normally. Do not truncate unknown /proc/PID/fd entries.
| Observation | Interpretation and next check |
|---|---|
| df: blocks 100%, inodes available | Inspect large directories, unlinked open files, snapshots and reserved space |
| df -i: IUse% 100% | Many small files exhausted inodes; locate their directory and retention owner |
| df >> du | Rule out permissions, mount scope and hidden files, then handles/snapshots; accounting differs |
| Deleting files did not free space | Open handles, snapshots, trash or concurrent growth; diagnose before more deletion |
Use ncdu for interactive browsing and find/fd for discovery. fd respects ignore rules and hides dotfiles by default; use -HI when that scope is intended. Follow Docker lifecycle rules rather than deleting its large data directory directly.
Correlate device latency with process load
# Observe device throughput, queues, and latency continuously
sysstat provides it. -y skips the since-boot report, -x expands fields, -z hides idle devices, and 1 is seconds. Correlate await/aqu-sz/throughput with the incident; Ctrl-C stops.
# Inspect accumulated per-process I/O for active objects
-o filters activity, -P groups processes, -a accumulates. Large totals do not imply high current rates; inspect accounting/permission errors.
# Sample process reads, writes, and I/O delay
Correlate with iostat at the same time; page cache means process and device writes need not match each second.
# Read task stall pressure caused by I/O
some/full avg10/60/300 are window percentages; total is cumulative microseconds. This measures stalls, not which drive is faulty.
iostat is in sysstat; -y skips the since-boot first report. Correlate rates, await, queues and application latency over time. High %util is not a universal NVMe/RAID throughput limit; low throughput with high latency can reflect sync writes, queueing or errors. iotop requires permissions and kernel accounting.
Keep evidence before repair
Select a health query for the device type
# Read detailed health data from the verified SATA/SAS device
Replace the device. For USB bridges, inspect --scan-open and supported -d types. This is a read task, not firmware update or long-test initiation.
# Read NVMe controller health data
This is the NVMe alternative. Track critical_warning/media_errors/temperature and wear changes, not just historical totals.
# Correlate device resets, I/O errors, and read-only transitions
Preserve surrounding logs. For hardware read errors, prioritize recoverable copies over repeated forced repairs on the source.
Inspect TRIM support and scheduling
# Inspect reported discard capabilities across layers
Fields such as DISC-MAX indicate capability, not completed TRIM.
# Check periodic TRIM scheduling
Inspect fstrim.service logs for execution results. Review existing schedules before adding another or enabling continuous discard.
Choose the commands for your device. USB bridges may need a supported smartctl -d protocol. SMART PASSED does not rule out cable, power, controller or future failures. Track counter changes, temperature, media errors and resets. Self-tests add load and have device-specific durations.
For I/O errors or a read-only remount, stop unnecessary writes and confirm backup/recovery access before deciding between hardware and filesystem faults. Do not start with remount,rw or fsck -y. See storage for offline checks. TRIM requires stack support; inspect timer policy before enabling continuous discard.