Page Cache and I/O: Caching, Writeback, and Persistence

Writing a few seconds is fast, but sustained writing slows down; even after write succeeds, fsync may still report an error. The Page Cache not only stores content but also coordinates concurrent filling, dirty page budget, and completion boundaries.

1. Cache is organized by file content, not copied per fd

The inode of a regular file is typically associated with an address_space via i_mapping; i_pages uses an XArray to index the cached content. Hard links pointing to the same inode with different names, as well as multiple fds obtained from independent open calls, usually share this content cache. address_space is not the process's virtual address space mm_struct.

Manual calculation: Which index does file offset 10000 fall into?

Assume PAGE_SIZE is 4096 bytes. 10000 = 2 × 4096 + 1808: the cache index is 2, and the page offset is 1808.

The index unit is the base page size, not the filesystem block, device logical sector, or number of folios. The base page size still depends on the architecture and kernel configuration.

Why introduce folio

A folio represents a group of physically contiguous base pages managed as a whole. Its size is a power-of-two multiple of the base page size and can be just one page. Explicitly stating "here we operate on a whole group of pages" reduces ambiguity where the trailing page of a compound page might be mistaken for an independent object, and it also aids in batch management of the cache.

Large folios do not mean all files use 2 MiB pages, nor does it mean the filesystem doesn't need to care about size. Allocation capability, alignment, filesystem support, read/write effective ranges, reclaim, and splitting all affect the actual choice. A folio spanning multiple base pages still covers the corresponding range according to the file page index; you cannot recalculate the index as "the Nth folio."

XArray supports multi-index entries, tags, and concurrent lookups, but the found entry is not necessarily a regular pointer; it might also be a shadow/swap value entry. Its internal synchronization does not automatically resolve the lifetime or file-range locking of the stored objects; page cache code should use the appropriate filemap/folio interfaces. You cannot simply xa_store() a pointer and assume the insertion, reference, and accounting are complete. See the XArray documentation.

2. Read path: Existing in cache does not mean data is available

A buffered read requires locating the requested range, obtaining a valid reference, and confirming that the corresponding content is ready before copying it to the user buffer. If a folio is currently being filled by another request, the reader may have to wait; thus, a cache hit does not guarantee zero wait time. The filesystem may track partial validity status by block, and the uptodate status of a whole folio must be distinguished from the validity of a specific range.

When multiple requests read the same uncached range, they need to coordinate around the same filling state:

1. Establish candidate. Allocate and attempt to insert a locked folio, handling concurrent insertion races.
2. Advance filling. Initiate backend reads; other requests can wait for the object being filled.
3. Publish results. After I/O completes, update validity status, unlock, and wake up waiters.

If each request reads independently outside the cache and then unconditionally inserts at the end, duplicate filling becomes hard to avoid. The actual path also handles truncation and I/O errors.

The backend does not necessarily produce a bio: local block filesystems may go through the block layer, NFS uses network requests, and tmpfs has different backing mechanisms. file_accessed(file) is related to access time and should not be written as a cache access marker function that receives a folio and updates the LRU.

Readahead is a bet on the next access

Sequential access is suitable for reading ahead into subsequent ranges, allowing backend I/O to overlap with application computation. When jumping randomly or the working set is much larger than memory, excessive readahead may waste bandwidth and evict useful cache.

"Synchronous readahead" usually refers to the readahead path triggered by a current cache miss; "asynchronous readahead" often advances the window via trigger markers consumed from the readahead window. The naming here does not imply that the synchronous path must wait for the entire readahead window to complete, nor does it equal a fixed 128 KiB readahead. The window is influenced by access history, file boundaries, device settings, and implementation strategies. See the Page Cache and Readahead API.

3. Write path: Ensure local content correctness first, then manage dirty data

Buffered writing obtains the corresponding cache range, copies user data into it, and updates the size and dirty state. When partially overwriting existing file blocks, it may be necessary to first read the uncovered old content; if the old content is already in the cache, or the range is newly allocated, a hole, and should be zero, there is no need to mechanically read from disk first. Different filesystems may use paths such as buffer-head or iomap; you cannot treat a fixed function chain as applicable to all implementations.

dirty indicates modifications that still need to be written back; writeback indicates that a writeback is currently in progress. After starting writeback, the application may modify the same folio again, so both states can appear simultaneously. The completion of this round of I/O does not automatically clear new dirty data generated afterwards. Writeback code needs to track ranges, states, and completion timing; see the iomap design for detailed contracts.

Delayed allocation allows the filesystem to decide on physical blocks during subsequent writeback, which is beneficial for layout aggregation, but it also means errors such as insufficient space may occur later than the initial write. The kernel uses a writeback error recording mechanism to return errors to relevant file descriptors; applications cannot ignore subsequent fsync failures. Even if close returns success, this does not guarantee persistence.

4. Why writers are throttled by dirty pages

First, calculate the cache backlog speed, assuming constant input and output rates:

For every additional second, approximately 824 MiB of difference needs to be accommodated. The cache can absorb bursts, but it cannot maintain this speed indefinitely; under pressure on the dirty page budget, writers need to throttle. A short test of fast write may just be consuming this budget.

Ratio and bytes are paired alternative configuration methods; writing to one clears the corresponding other; actual control also involves estimable dirty memory, share of each writeback domain, and cgroup support. Current values should be read from /proc/sys/vm/; do not treat a set of default values as facts for all machines. Background work is also not permanently "one disk corresponds to one exclusive flusher thread." See the VM sysctl documentation.

5. Three completion semantics

fdatasync() still synchronizes metadata necessary for data recovery, such as file size affecting reads; it does not mean no metadata is written. To ensure a name persists after creation or renaming, you usually also need to sync the relevant parent directory; specific failure protocols should be designed according to filesystem capabilities. Sync failures must be reported or enter an explicit recovery process. See fsync(2).

Deliberately write again during a writeback in progress, then complete the old request. Below, we preserve the cache, device volatile state, and stable content respectively, and use power loss results to check which version each completion point actually saved.

6. What mmap and direct I/O change respectively

MAP_SHARED mapping of a regular file is typically connected to the same page cache, and writing can form dirty pages; MAP_PRIVATE copy-on-write forms private modifications, which are not written back to the original file. The mapping can survive after closing the original fd; accessing pages beyond the new end after the file is concurrently truncated may result in SIGBUS. msync(MS_SYNC) can sync the shared mapping range, but it does not replace cross-file transactions or directory sync. See mmap(2).

Direct I/O: Bypassing the cache path does not expand persistence promises

O_DIRECT usually causes file data transfer to bypass the cache path of buffered I/O, but it is not equivalent to O_SYNC, nor does it automatically promise zero-copy, asynchronous execution, or power-loss persistence. Buffer address, length, and file offset alignment requirements vary by filesystem and device; when supported, they can be queried via STATX_DIOALIGN of statx(), and cannot be uniformly hardcoded to 512 or 4096 bytes. If constraints are not met, an error may be returned, and some paths may fall back.

When mixing paths, still coordinate cache and buffer lifetimes

When direct I/O is mixed with buffered writing and shared mapping, the kernel may still need to write back or invalidate the cache to maintain consistency, making performance and concurrency reasoning more complex. User buffers must remain valid and satisfy access constraints before asynchronous requests complete; buffers involving fork and private mappings must also comply with additional restrictions. Interface restrictions see the O_DIRECT description in open(2).

7. Small experiment: Shared mapping and private copy

Below, we only create a temporary file, verify visible content, and do not test throughput or power loss recovery:

import mmap
import os
import tempfile

with tempfile.TemporaryFile() as file:
    file.write(b"A" * mmap.PAGESIZE)
    file.flush()                    # Clear Python user-space buffer first
    with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_WRITE) as shared:
        shared[0:1] = b"S"
        shared.flush()              # Sync shared mapping on Linux
    os.fsync(file.fileno())
    assert os.pread(file.fileno(), 1, 0) == b"S"
    with mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_COPY) as private:
        private[0:1] = b"P"
        assert private[0:1] == b"P"
        assert os.pread(file.fileno(), 1, 0) == b"S"
    assert os.pread(file.fileno(), 1, 0) == b"S"
    print("Shared mapping write visible; private modification not written back to file: Passed")

file.flush(), mapping flush, and fsync() span different buffer/sync boundaries; a Python method having the same name does not mean the work done is the same.

For read-only troubleshooting, you can check Cached, Dirty, Writeback in /proc/meminfo and relevant counters in /proc/vmstat. Cached should not be directly treated as ordinary disk file cache that can be released without cost at any time; tmpfs/shmem etc. also affect statistics. The resident status reported by mincore() is a snapshot and does not guarantee the latency of the next read. drop_caches is a system-wide operation that will disturb the cache status of other processes and should not be placed in daily observation steps as a side-effect-free "flush".

Further reading on cache reads along mm/filemap.c, readahead along mm/readahead.c, and throttling and writeback along mm/page-writeback.c and fs/fs-writeback.c. File layout and logging mechanisms connect to Filesystem Implementation.