io_uring: Revolutionary Asynchronous IO

Coverage: SQ/CQ ring → fixed buffers → polled IO → comparison with libaio/posix aio → performance considerations Applicable: Linux 5.1+

Overview

io_uring is an asynchronous IO framework introduced in Linux 5.1 that has completely changed how Linux IO works. It passes IO requests and completion notifications between user space and kernel space using two shared memory rings (SQ/CQ)—allowing IO submission and completion retrieval without system calls. In optimal cases, io_uring can achieve ~6M IOPS (per core), whereas traditional aio only reaches ~1M.

Ring Buffer Model

flowchart LR
    subgraph USER["User Space"]
        APP["App<br/>Writes SQ / Reads CQ"]
    end

    subgraph SHARED["Shared Memory (mmap, zero-copy)"]
        SQ["Submission Queue<br/>IO Request Queue"]
        CQ["Completion Queue<br/>IO Completion Queue"]
    end

    subgraph KERNEL["Kernel Space"]
        K["Kernel<br/>Reads SQ / Writes CQ"]
    end

    APP -->|"① Write SQE"| SQ
    SQ -->|"② Kernel reads"| K
    K -->|"③ Write CQE"| CQ
    CQ -->|"④ App harvests"| APP

    classDef user fill:#e3f2fd,stroke:#1565c0
    classDef shared fill:#fff3e0,stroke:#ef6c00
    classDef kernel fill:#f3e5f5,stroke:#7b1fa2
    class APP user
    class SQ,CQ shared
    class K kernel

Basic Usage

#include <liburing.h>  // liburing helper library

struct io_uring ring;
io_uring_queue_init(256, &ring, 0);  // 256 entries in SQ

// Prepare a read request:
struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
io_uring_prep_read(sqe, fd, buf, size, offset);  // offset: fixed position
sqe->user_data = 42;  // User tag (returned in CQE)

// Submit request:
io_uring_submit(&ring);

// Harvest completions:
struct io_uring_cqe *cqe;
io_uring_wait_cqe(&ring, &cqe);  // Block waiting for 1 completion
printf("res=%d, user_data=%lu\n", cqe->res, cqe->user_data);
io_uring_cqe_seen(&ring, cqe);

Advanced Features

SQ Polling (Kernel-side Busy Polling)

// io_uring_setup(..., IORING_SETUP_SQPOLL)
// Kernel starts a kthread → continuously polls SQ → processes requests immediately
// → No need for io_uring_submit() → Zero system calls!
// → io_uring_enter(ring_fd, 0, 0, IORING_ENTER_GETEVENTS, NULL)
//    Only harvests completions; submission is handled automatically by the SQ poller

// Cost: SQ poller thread continuously consumes CPU (5-10% of one core)
// Suitable for: Scenarios with continuous IO (databases, file servers)

Fixed Buffers / Fixed Files

// Pre-register buffers and fds:
io_uring_register_buffers(&ring, iovecs, nr_iovs);  // Fixed buffers
io_uring_register_files(&ring, fds, nr_fds);        // Fixed files

// Subsequently use registered buffers → avoids get_user_pages for each IO
// Subsequently use fixed fds → avoids fget/fput for each IO
// → Further reduces latency (reduces ~30-50% kernel overhead)

IORING_SETUP_DEFER_TASKRUN

// Default: CQE → Kernel marks completion in interrupt context
// DEFER_TASKRUN: Kernel defers IO completion processing → batches processing during next io_uring_enter
// → Reduces interrupts + batch processing → Higher throughput

Comparison with aio

Performance Pitfalls

1. CQE Harvesting: io_uring_wait_cqe harvests only 1 at a time → Use io_uring_wait_cqes to harvest multiple
2. SQ Full: io_uring_get_sqe returns NULL → Submit first, then retry
3. Large SQ Depth: 256-512 is sufficient → Larger depth just wastes memory
4. Link Timeout: Add timeouts to each request → IORING_OP_LINK_TIMEOUT
   → Prevents slow IO from blocking the entire ring

References

• liburing: https://github.com/axboe/liburing (examples + tests)
• Kernel Source: fs/io_uring.c (~20,000 lines)
• LWN: "The io_uring API", "io_uring performance improvements"

Keywords: io_uring, SQ, CQ, SQE, CQE, fixed buffers, SQPOLL, DEFER_TASKRUN, libaio