io_uring: Submission and Completion Model

io_uring decouples request preparation, submission, execution, and result retrieval via shared Submission Queue (SQ) and Completion Queue (CQ). It allows amortizing overhead through batching and unifies handling of various asynchronous operations; whether it is actually faster depends on the workload, device, queue depth, and implementation.

The basic framework was introduced in Linux 5.1, with subsequent additions of operations and features. Do not assume that all opcodes, flag combinations, and filesystem paths are supported just because the kernel version is higher than 5.1. Practical applications should probe capabilities and handle failures during creation or request submission.

SQ and CQ Carry Control Information

The application prepares SQEs, filling in opcode, fd, buffer address, length, offset, and user_data, among other fields; the kernel consumes these to execute them and returns results such as user_data, res, and flags via CQEs. Sharing ring queues reduces the copying of control information back and forth, but it does not mean that data payloads automatically become zero-copy.

Userspace and the kernel advance their respective production and consumption indices, requiring correct memory barriers. This is typically managed by liburing to avoid manually manipulating head/tail with plain integers. When multiple application threads operate on the same ring concurrently, they must also adhere to the concurrency contracts of the library and the mode used. io_uring(7)

Four Results That Must Not Be Confused

When io_uring_wait_cqe returns 0, you should check cqe->res again; if res is -ENOENT, the error comes from that specific operation and cannot be inferred from the current thread's errno. A read operation res of 0 usually indicates EOF, while a positive value is the actual number of bytes read. wait_cqe, prep_read

A Complete Single-Shot Read Program

Save as uring_read.c. The program reads at most 4096 bytes from the specified file starting at offset 0, outputting the actual byte count; it is not a loop to read the entire file. Fatal errors during initialization or waiting will terminate this demo process; long-running services need to implement independent cancellation and reclamation paths.

#define _GNU_SOURCE
#include <errno.h>
#include <fcntl.h>
#include <liburing.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

static void fail_uring(const char *what, int result) {
    fprintf(stderr, "%s: %s\n", what, strerror(-result));
    exit(1);
}

int main(int argc, char **argv) {
    if (argc != 2) {
        fprintf(stderr, "usage: %s FILE\n", argv[0]);
        return 2;
    }
    int fd = open(argv[1], O_RDONLY);
    if (fd == -1) { perror("open"); return 1; }

    struct io_uring ring;
    int rc = io_uring_queue_init(8, &ring, 0);
    if (rc < 0) fail_uring("queue_init", rc);

    unsigned char buffer[4096];
    struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
    if (sqe == NULL) {
        fprintf(stderr, "no SQE available\n");
        exit(1);
    }
    io_uring_prep_read(sqe, fd, buffer, sizeof buffer, 0);
    sqe->user_data = 42;

    do { rc = io_uring_submit(&ring); } while (rc == -EINTR);
    if (rc < 0) fail_uring("submit", rc);
    if (rc != 1) {
        fprintf(stderr, "expected one submitted request, got %d\n", rc);
        exit(1);
    }

    struct io_uring_cqe *cqe;
    do { rc = io_uring_wait_cqe(&ring, &cqe); } while (rc == -EINTR);
    if (rc < 0) fail_uring("wait_cqe", rc);
    int result = cqe->res;
    unsigned long long tag = cqe->user_data;
    io_uring_cqe_seen(&ring, cqe);

    int failed = result < 0 || tag != 42;
    if (result < 0) fprintf(stderr, "read: %s\n", strerror(-result));
    else printf("read=%d, user_data=%llu\n", result, tag);
    io_uring_queue_exit(&ring);
    if (close(fd) == -1) failed = 1;
    return failed;
}

cc -Wall -Wextra uring_read.c -luring -o uring_read
./uring_read input.bin

The buffer remains alive after submission until the CQE for this single-shot request arrives. After cqe_seen, the original CQE slot can be reused, so the program saves res and user_data first. The tag is an application-associated key; the kernel does not increment reference counts for objects pointed to by tags.

Explicit file offsets avoid contention on implicit file positions among multiple concurrent requests. To read a file completely, you should adjust the offset and remaining length based on the actual returned length, and handle EOF, errors, and cancellation; do not treat every short read as a failure.

SQ is empty; can buffers be reused?

Prepare and submit two requests: SQ empties. Prepare a third and all buffers now have owners. Complete R2 before R1; CQ follows completion order, while IDs retain association. The model returns buffers only after completion consumption, not when SQ slots become free.

Completion Order and Dependencies

Standard independent requests may complete out of order; you cannot directly match them to business objects by submission queue index. user_data usually encodes request ID, type, or object generation to prevent old completions from landing on reused objects.

When sequential dependencies are needed, use mechanisms like link and drain according to semantics, but they are not database transactions. Side effects already produced by a previous write will not automatically roll back just because a subsequent request fails. For write persistence, you also need to submit appropriate synchronization operations and observe their results.

SQPOLL and IOPOLL Are Two Different Directions

SQPOLL has a kernel thread poll the submission queue. The application still needs to publish prepared SQEs to the queue; when using liburing, you still call the appropriate submit path, and cannot just get_sqe/prep and do nothing. When the polling thread sleeps and sets NEED_WAKEUP, the library wakes it up as needed. io_uring_submit

IOPOLL, on the other hand, polls for completion of supported I/O, depending on device, filesystem, and access mode conditions. The two can be discussed separately; do not treat "polling submission" and "polling device completion" as the same feature.

Continuous polling can significantly consume CPU, not just a fixed 5%–10% of one core. DEFER_TASKRUN defers the corresponding work to the specified path for entering the kernel to collect events, requiring constraints like SINGLE_ISSUER and correct event advancement methods; it is not simply "disabling interrupts to become faster." Advanced flags should be checked against version compatibility item by item. liburing setup docs

Registered Resources: Prepaying Some Management Costs

Registering buffers can reduce the work of repeatedly pinning user pages in eligible paths, but consumes resources and incurs registration/unregistration costs. When using them, you must select the corresponding fixed-buffer operations and indices; registering once does not mean all standard reads automatically enjoy the same path. register_buffers

Registering files establishes a file table for the ring; when using IOSQE_FIXED_FILE, the corresponding value in the SQE is an index within this table, not a standard process fd. Updates, unregistration, and their relationship with in-flight requests must be handled according to the interface protocol. register_files

Buffers, descriptor objects, and request states all have lifecycles. You cannot free(buffer) just because submission succeeded, nor can you immediately reuse an old request_id just because a cancellation request has been submitted.

Cancellation and Timeouts: Observe the Termination of the Original Request

A cancellation operation is another request and will have its own completion result. It may succeed, find that the target has already completed, or fail to cancel immediately; the completion record of the original operation still needs to be handled according to the interface. Applications should track both the "cancellation requested" and "original request terminated" states. prep_cancel

link timeout can limit the wait time for associated operations, but it does not automatically establish timeout isolation for the entire ring. An independent slow request usually does not naturally block all other independent requests from completing; whether there is a dependency chain is determined by the submission method.

Standard one-off operations often correspond to one CQE, but multishot accept or recv can produce multiple; you should check IORING_CQE_F_MORE to determine if the request will continue. Some zero-copy sends require additional notifications to determine when data buffers can be reused; do not treat all CQEs as the final point for resource reclamation. Multishot accept

Cancellation Races: Record Both Results Separately

The following table is limited to canceling one standard single-shot request by unique user_data, without using flags to cancel all matching items. C is the cancellation request, R is the original request; cancellation itself does not roll back external side effects already completed by R. Cancellation Results

If the business timeout means you simply no longer need the result, you can mark R as "result discard," but its I/O lifecycle still needs to complete. Business state end and kernel access buffer end are not the same thing; this is also why asynchronous interfaces must clearly define ownership relationships.

Multishot: Separate Individual Results from Request Termination

Below is the offline CQE model, where MORE represents the boolean value corresponding to IORING_CQE_F_MORE, not reading the actual ring. Two successful accepts yield two fds, and the final cancellation completion ends this multishot request. The example only records numbers, not creating or closing real sockets.

import errno
active = True
accepted = []
terminal = None
cqes = [(101, True), (102, True), (-errno.ECANCELED, False)]
for result, more in cqes:
    assert active
    if result >= 0:
        accepted.append(result)
    if not more:
        active = False
        terminal = result
assert accepted == [101, 102]
assert not active and terminal == -errno.ECANCELED
# Successful results can also be the last one; do not use result >= 0 to judge if still active.
last_success, more = 103, False
assert last_success >= 0 and not more

The successful result of a real accept is the connection resource already obtained, which needs to be handed over to connection management or closed; request termination does not automatically close previously accepted connections. To judge whether to continue waiting for subsequent CQEs of this request, check MORE, not just the sign of res. Multishot accept

Cancel completion is not original completion

Consume only C: retain the buffer. R settles the original request. If read wins, its four-byte result is not undone by later cancellation. Consumption order illustrates independent accounting, not guaranteed CQE ordering on every kernel.

Batching and Backpressure

When no SQE can be obtained, you need to submit prepared items and continue advancing based on the number of in-flight requests; CQ also needs to be collected and consumption position advanced in time. Waiting for multiple completions and actually traversing batches are different actions; do not interpret the name wait_cqes as having already consumed all CQEs for the application.

Queue depth controls concurrent in-flight work. Too small may fail to fill the device, while too large may increase queuing delay, memory usage, and tail latency; there is no universal "256–512 is enough." First estimate concurrent needs using completion time and target throughput, then measure saturation points and tail latency.

Estimate Quantity Using Throughput and Latency, Plus Buffer Accounting

Assume the target throughput is 20,000 requests per second, and the average in-flight time is 2 ms; the average number of in-flight requests in a stable state is approximately 40. This is a capacity estimate of throughput multiplied by average time, not a tail latency guarantee, nor does it mean the kernel must configure SQ depth to 40.

rate = 20_000
mean_seconds = 0.002
average_inflight = rate * mean_seconds
assert average_inflight == 40
buffer_size = 64 * 1024
limit = 64
buffer_budget = limit * buffer_size
assert buffer_budget == 4 * 1024 * 1024

If each standard read request exclusively occupies a 64 KiB buffer, an in-flight limit of 64 corresponds to at least 4 MiB of payload capacity, not to mention request objects, rings, alignment, and other queues. SQ empty slots are only submission capacity: after an SQE has been consumed by the kernel, the request may still be executing, so you cannot treat "getting another SQE" as "in-flight buffer budget automatically decreasing."

Applications should independently maintain the number of in-flight requests and the available buffer pool; when the limit is reached, collect completions first or pause upstream. For multishot, also account for the number of connections/messages already accepted but not yet processed; a single active SQE can continuously generate business resources, and their accumulation cannot be limited solely by SQ depth.

When comparing with libaio or POSIX AIO, fix buffered/direct modes, block sizes, devices, threads, and synchronization guarantees. The asynchrony of Linux native AIO is limited in some buffered paths, which does not mean that syntactically submitting buffered file I/O is absolutely impossible; POSIX AIO is an interface specification, and the thread implementation of Linux libc is just one specific implementation.

The request API provides an asynchronous model, which does not mean that every operation is executed asynchronously by device hardware; some work may be completed instantly or handed over to kernel worker threads. Prioritize verifying behavior, resource reclamation, and backpressure before discussing how many system calls are reduced.

Keywords: io_uring, SQE, CQE, liburing, completion, buffer lifetime, SQPOLL, IOPOLL, fixed buffers, cancellation, multishot, queue depth