ZFS: Storage Pools, Transaction Groups, and Synchronous Write Recovery

A synchronous write has succeeded, yet the latest TXG has not been committed; deleting the current file does not immediately return all space. These two phenomena in ZFS are explained by recovery guarantees and shared references, respectively, and cannot be simply summarized as "there is caching."

1. Hierarchy of pool, vdev, and dataset

Assume the pool consists of two top-level vdevs, each being a dual-disk mirror. First, assess faults by group:

Failing two disks yields different results depending on their positions. RAIDZ also requires analysis based on group layout, actual errors, and remaining copies. See Pool and vdev concepts.

2. The software layer is not a pipeline where all requests pass sequentially

ZPL connects POSIX file operations to the Linux VFS; DMU manages objects, data buffers, and transactions; the SPA and vdev-related layers handle storage pool allocation and low-level I/O. ARC is a caching mechanism, and ZIL is a synchronous intent log; these two should not be drawn as mandatory serial layers between DMU and SPA.

OpenZFS provides out-of-tree kernel modules on Linux, which can be installed using DKMS or distribution pre-compiled modules. SPL primarily adapts the required kernel environment and abstractions; it is not a legal conversion layer that automatically validates any license combination. OpenZFS uses the CDDL; for release and packaging, consult its License Statement and the target distribution's policies; the claim that it "can never be merged" is an assertion beyond current facts and should not be treated as a technical conclusion.

The interaction between ARC and the Linux mapping/page cache path is affected by the implementation version, so it is incorrect to simply assert that ZFS does not involve the Linux page cache at all. Regular files, mmap, zvol, and direct I/O should not be forced into the same read/write function chain.

3. TXG: Committing a group of consistent block references

Modifications first enter the object state in memory and are added to a transaction group. When syncing a TXG, data and the affected metadata paths are written out, finally establishing a recognizable valid root state. Recovery relies on mechanisms such as block checksums, ordering, and valid uberblocks; it is not the case that every block pointer write is naturally atomic, nor that arbitrary sector tearing is insignificant.

TXG aggregates multiple modifications to amortize costs; stages such as open, quiescing, and syncing allow work from different groups to overlap. Timeout parameters are only one trigger condition; dirty data pressure also drives synchronization. One cannot map a single write to "exactly one atomic transaction five seconds later." Filesystem transaction boundaries do not automatically correspond to multi-step business boundaries in applications.

Synchronous requests typically use ZIL to establish a replayable basis in case of a crash, without needing to wait for the full synchronization of every large TXG. Subsequently, the corresponding data is still written to regular storage via the normal TXG path; during recovery, logs that have not yet been overwritten by committed TXGs are replayed as needed. ZIL records may contain written data or reference already written data blocks; it is incorrect to say it "only stores operation names and no data."

Placing TXG and ZIL on the Same Recovery Timeline

Assume TXG 100 has been stably committed, with file content in a certain range being O. The application performs a synchronous modification W₁, changing it to N. Subsequently, another range's ordinary asynchronous modification W₂ is executed, but no sync request is made for W₂. Assume sync=standard, and the storage device honors write and sync guarantees.

Select range A or B and write a new version synchronously or asynchronously. Commit a TXG or crash, then compare the durable root, log, and recovery.

Revisiting Asynchronous W₂: Not Guaranteed Does Not Mean Definitely Lost

W₂'s fate cannot be inferred solely from "it happened after W₁": if it has not been covered by a valid TXG and has no sync guarantee, it may be lost; if it has been stably committed with a TXG, it may also be retained. Asynchronous means the application did not receive the same durability guarantee at that return point, not that the kernel guarantees it will definitely be lost.

This timeline illustrates what COW and ZIL each provide. COW's valid old root allows the system to return to a set of structurally consistent block references; ZIL fills in synchronous operations that have been confirmed to the application but have not yet entered the latest regular tree. The low latency of the log device affects the speed of establishing the latter type of recovery basis, but it cannot permanently exempt the main storage from writing out modifications.

Which Recovery Basis Fails When Log Is Lost

If the only log copy that needs to be replayed is lost simultaneously, the proof in the second row no longer holds. The risk set is determined by "which confirmed operations still rely on that log," which is neither a fixed wall-clock five seconds nor determined solely by a parameter name. Redundant logs and device power-loss protection ensure that recovery bases still exist under designed fault conditions.

4. ZIL, SLOG, L2ARC, and special vdev

SLOG cannot turn arbitrary asynchronous large file writes into sustained SSD speeds; it primarily changes the latency path for related synchronous requests, while regular storage still handles TXG writes. If the backend consistently lacks sufficient write speed, the system will eventually throttle. Device latency must also be evaluated based on real-world performance under flush and power-loss protection, not just nominal sequential bandwidth.

Losing the log device and encountering a crash again may corrupt modifications that were confirmed to the application but still require log recovery; one cannot guarantee "only the last fixed five seconds are lost, and async writes are unaffected." The outcome changes depending on whether the system is running, during import, whether the log has redundancy, and whether the corresponding TXG has been committed. See OpenZFS Hardware Recommendations for related device responsibilities.

sync=standard honors application sync requests, sync=always expands sync requirements, and sync=disabled ignores normal durability requirements for such requests. Directly comparing the performance of disabled with normal fsync performance is equivalent to comparing different guarantees, not proving faster speeds under the same safety conditions. See zfsprops(7) for property semantics.

5. Snapshots, Record Size, and Space Retention

Snapshots retain old block references, and clones can create writable branches based on snapshots. Assume a file originally has blocks A and B. After creating a snapshot, the current version changes A to A′, while the old A is still referenced by the snapshot; deleting the current file does not necessarily reclaim A and B. Pool usage and the logical size of files in a directory differ due to sharing, compression, snapshots, and metadata.

Space Accounting: Why a 512 KiB File Retains 640 KiB

Limit the model: the file consists of four 128 KiB records A, B, C, and D, ignoring metadata, compression, padding, and extra copies. After creating snapshot S, the two views share these four records; the current file then rewrites A as A′.

The new usage comes from the historical A still having references, not from the file's logical length increasing.

At this point, deleting the current file does not reclaim old A, B, C, and D as long as S exists; deleting S but keeping the current file allows old A to be released after corresponding reclamation is complete. Actual statistics include other overheads and sharing relationships, so this number cannot be applied directly, but the reasoning method for reference accounting is the same: file logical length statistics view content, pool usage statistics actual storage that must be retained. This also explains why snapshots are cheap to create initially but may retain many historical blocks after long-term frequent modifications.

Record Size: Layout Attribute Does Not Equal Every Device Request Size

recordsize is a filesystem data record size-related attribute, while volblocksize is used for zvol. They do not mean "every application write will necessarily issue a device request of the same size." Small-range random writes, compression, COW, and existing file layout affect amplification; modifying the attribute does not simply mean immediately rewriting all old files. Before tuning, one should first clarify the read/write scope and sync frequency of the workload.

Cache Observation: Look Beyond Hit Rate to Sync Tail Latency

ARC adjusts caching strategies based on recent/frequent access and ghost history information; ghost records do not save another complete copy of the data. High ARC hit rates may mask tail latencies of a few synchronous writes, so they must be observed separately according to the workload. L2ARC has index, padding, and memory overheads; expanding it does not automatically fix all performance issues outside the hot working set.

6. Checksums, Scrub, Resilver, and Backup

Content passing checksums does not mean the business meaning of the application's write is correct, nor does it undo normal execution of erroneous deletions.

Filesystem snapshots provide a storage-layer point-in-time view, but they do not automatically commit database memory state together with multiple external systems. Application-consistent recovery points require database backup interfaces, checkpoints, or explicit coordination protocols.

send/receive transfers snapshot state or differences, allowing the construction of a replica on another pool or host; incremental chains require matching baselines and retention policies. Being able to generate a send stream does not mean a recoverable backup is complete; the availability of the receiver and the recovery process still need verification.

7. Read-Only ARC Access, Without Modifying the Pool

from pathlib import Path

path = Path("/proc/spl/kstat/zfs/arcstats")
if not path.exists():
    print("ARC statistics not exposed; OpenZFS may not be loaded")
else:
    values = {}
    for line in path.read_text().splitlines():
        columns = line.split()
        if len(columns) == 3 and columns[2].isdigit():
            values[columns[0]] = int(columns[2])
    for name in ("size", "c", "c_min", "c_max", "hits", "misses"):
        print(name, values.get(name, "Not Provided"))
    hits, misses = values.get("hits", 0), values.get("misses", 0)
    if hits + misses:
        print("Cumulative Hit Ratio:", hits / (hits + misses))

This is the cumulative ratio and cannot replace differential analysis for a specified workload time period. zpool status is used to understand topology and errors; zpool iostat is used for device/pool activity; zfs get is used to verify actual attributes and inheritance sources. First, combine topology, attributes, and business workload to explain observations, then decide if configuration changes are needed.