epoll and Event-Driven Programming

epoll maintains a set of interested file descriptors and their ready events, allowing a program to wait centrally for multiple I/O sources. It returns hints that "read/write can be attempted now," not that I/O has been completed on behalf of the application, nor does it guarantee that one event corresponds to one business message.

Why It Suits Many Idle Connections

select/poll submit a set or array of file descriptors and scan them in each iteration; epoll retains the interest set and maintains a candidate list through underlying ready notifications, typically reducing the repetitive processing of a large number of idle descriptors.

This does not mean that any epoll_wait is O(1): returning k events incurs at least the cost of processing and copying them. Registration modifications, callbacks, lock contention, and a large number of active connections also have costs. The number of descriptors for poll and epoll is limited by resources and cannot be written as unlimited; glibc's select also has a fixed FD_SETSIZE limit.

Three Types of Operations

int epfd = epoll_create1(EPOLL_CLOEXEC);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = client_fd };
int rc = epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);

This is an API snippet requiring <sys/epoll.h>; you must check epfd and rc. ADD registers, MOD modifies interest and user data, and DEL deletes. A positive return value from epoll_wait is the number of events in this batch, 0 indicates a timeout, and -1 indicates an error; EINTR should not be mistaken for "no connections" or "all connections closed."

.data is association information saved by the application and returned verbatim; it can use an fd, a pointer, or an ID with a generation counter. The kernel does not manage the lifecycle of the pointed-to object on behalf of the application.

Event Bits Are Not Mutually Exclusive Enums

IN, HUP, ERR, etc., can appear simultaneously. ONESHOT does not automatically execute DEL, nor does it close the fd. epoll_ctl(2)

LT vs. ET: When to Wait Again

LT (Level-Triggered) is the default mode; it continues to report if the condition remains ready. The application can process some data and return to waiting, but if it never consumes data or always subscribes to writable events, the loop will continue to be woken up.

ET (Edge-Triggered) does not guarantee that unconsumed ready states will be repeatedly notified. The common practice is to use non-blocking I/O and only consider the notification reliable after processing until EAGAIN is returned. Do not simplify ET to strictly reporting a binary state flip for all fds; specific wake-ups come from underlying events, and the application should not rely on extra notifications to fix missed processing. epoll(7)

Below is a reading snippet for a stream socket. consume_bytes must process quickly or be enqueued to the application queue; it must not block in the event thread. Returning 0 means EAGAIN was reached, 1 means EOF on the input direction, and -1 means other errors:

#include <errno.h>
#include <stddef.h>
#include <sys/socket.h>

void consume_bytes(const unsigned char *data, size_t size);

int drain_socket(int fd) {
    unsigned char buf[4096];
    for (;;) {
        ssize_t n = recv(fd, buf, sizeof buf, 0);
        if (n > 0) {
            consume_bytes(buf, (size_t)n);
            continue;
        }
        if (n == 0) return 1;
        if (errno == EINTR) continue;
        if (errno == EAGAIN || errno == EWOULDBLOCK) return 0;
        return -1;
    }
}

EOF only ends the input direction from the peer to this end; you may still need to send responses or drain the output. Whether to close the entire connection should be determined by protocol state, not by immediately discarding the object upon any HUP.

There is still data, why didn't epoll_wait return new events?

Under ET, executing "arrive → wait → read → wait" can result in the second wait returning 0, while two bytes remain in the buffer. The pending processing marker retained by the application allows the remaining work to continue scheduling; it cannot be deleted just because there are no new events in this round. Reading exactly two bytes does not automatically prove the buffer is drained; you must attempt to read again and get EAGAIN to confirm this boundary.

Write Backpressure and Fairness

When the application has data to send, it attempts non-blocking send first; if only part is written or EAGAIN is returned, save the remaining range and watch for EPOLLOUT. Once the send queue is empty, it is usually best to cancel writable interest; otherwise, LT mode may spin idly. Socket writing also requires clear policies for SIGPIPE or MSG_NOSIGNAL.

Serving a single high-throughput connection until EAGAIN may starve other connections. You can set byte or time budgets per connection; under ET, if the budget is exhausted but the buffer is not drained, the connection must remain in the application's ready queue and continue scheduling, rather than returning directly to epoll_wait hoping for another notification.

Listening sockets should also loop accept until EAGAIN; accept4 can atomically specify NONBLOCK/CLOEXEC for new connections. You must also handle errors such as fd exhaustion to avoid busy loops caused by persistent failures when the listener is ready.

Fair scheduling requires explicitly saving "still has work"

Assume A currently has 10 units readable and B has 2 units readable, with a per-round budget of 4. After A uses its first round of budget but still has work, placing it at the end of the queue allows B to finish first. If A is forgotten directly under ET, the remaining bytes may wait indefinitely for a notification that is not guaranteed to come again.

from collections import deque
remaining = {"A": 10, "B": 2}
ready = deque(["A", "B"])
queued = set(ready)
turns = []
while ready:
    connection = ready.popleft()
    queued.remove(connection)
    count = min(4, remaining[connection])
    remaining[connection] -= count
    turns.append((connection, count))
    if remaining[connection] and connection not in queued:
        ready.append(connection)
        queued.add(connection)
assert turns == [("A", 4), ("B", 2), ("A", 4), ("A", 2)]
assert all(n == 0 for n in remaining.values())

This is an offline model with known workload. Real sockets cannot know remaining in advance; use EAGAIN, EOF, and errors from non-blocking read to update state, and keep connections that have exhausted their budget but not yet drained as application-runnable. If parsing a single message is itself time-consuming, limiting read bytes is still insufficient to guarantee fairness; you must also limit parsing/business execution time.

Multiple Worker Threads and ONESHOT

ONESHOT is often used to let one handler temporarily take over a connection. After processing, re-enable interest via EPOLL_CTL_MOD; this does not automatically make all connection fields thread-safe, nor does it replace memory synchronization during task handover.

When resetting, publish the connection state and interest set first, and consider that other threads may receive events immediately after reset. If unread ready data remains, re-enabling may deliver events immediately again. Locks, ownership, or serial executors should clearly specify who can modify the output queue and close connections.

Using eventfd to See ONESHOT and Rearming

Save as oneshot-demo.c. eventfd is initialized with a value of 3; after the first epoll_wait receives a readable event, intentionally do not read; the count remains, but ONESHOT has disabled subsequent notifications. After re-MODding, the existing readable condition causes the event to return again.

#include <stdint.h>
#include <stdio.h>
#include <sys/epoll.h>
#include <sys/eventfd.h>
#include <unistd.h>
int main(void) {
    int fd = eventfd(3, EFD_NONBLOCK | EFD_CLOEXEC);
    if (fd == -1) return 1;
    int ep = epoll_create1(EPOLL_CLOEXEC);
    if (ep == -1) { close(fd); return 1; }
    int failed = 1;
    struct epoll_event interest = { .events = EPOLLIN | EPOLLONESHOT };
    interest.data.u64 = 42;
    struct epoll_event event;
    if (epoll_ctl(ep, EPOLL_CTL_ADD, fd, &interest) == -1) goto done;
    if (epoll_wait(ep, &event, 1, 0) != 1 || event.data.u64 != 42) goto done;
    if (!(event.events & EPOLLIN)) goto done;
    if (epoll_wait(ep, &event, 1, 0) != 0) goto done;
    if (epoll_ctl(ep, EPOLL_CTL_MOD, fd, &interest) == -1) goto done;
    if (epoll_wait(ep, &event, 1, 0) != 1 || event.data.u64 != 42) goto done;
    uint64_t count;
    if (read(fd, &count, sizeof count) != sizeof count || count != 3) goto done;
    if (epoll_ctl(ep, EPOLL_CTL_MOD, fd, &interest) == -1) goto done;
    if (epoll_wait(ep, &event, 1, 0) != 0) goto done;
    puts("notified, disabled, rearmed; count=3; drained");
    failed = 0;
done:
    if (close(ep) == -1) failed = 1;
    if (close(fd) == -1) failed = 1;
    return failed;
}

Run cc -Wall -Wextra oneshot-demo.c -o oneshot-demo then execute ./oneshot-demo. All waits use zero timeout, checking already existing ready conditions; in this example, there are no other signal sources, so wait failures terminate directly; general loops still need to handle EINTR. ONESHOT

Here read returns 8 bytes, and the read count is 3; the two have different meanings. Without EFD_SEMAPHORE, a successful read retrieves the current accumulated value and clears it; eventfd acts as a notification counter and does not store three separate business messages. eventfd(2)

Closing and fd Number Reuse

fds are reusable integers. A batch may already contain events from an old connection, which is then closed by a callback, and a new connection obtains the same fd. Directly looking up the current object by this number may incorrectly apply old events to the new connection; object generations, stable IDs, deferred reclamation, or checksums can solve such issues.

Duplicated descriptors may still share open file descriptions; closing one fd does not necessarily make all related registrations disappear immediately. For deterministic revocation, execute DEL while the object lifecycle is still valid, and clean up according to the rules for events already in the application queue. The kernel removing a registration cannot revoke events already copied to user space.

How an old event can be misdelivered to a new connection

Assume the current batch saved (slot=7, generation=4). The previous callback closed that connection and released the slot; the new connection reuses slot=7, but the generation becomes 5. When dispatching remaining events, you must check the generation; you cannot just find a non-empty object by slot or fd and continue.

current = {7: {"generation": 5, "name": "new connection"}}
old_event = (7, 4)
new_event = (7, 5)
def resolve(event):
    slot, generation = event
    item = current.get(slot)
    if item is None or item["generation"] != generation:
        return None
    return item
assert resolve(old_event) is None
assert resolve(new_event)["name"] == "new connection"

Generations need sufficient range and must define wrap-around and reuse strategies; they do not replace concurrent memory reclamation. If data.ptr is already a freed pointer, dereferencing it to read the generation is still unsafe. You can use integer IDs to query a still-valid registry, or have the event batch hold object references until dispatch ends.

timerfd, signalfd, and eventfd

timerfd needs timerfd_settime to start after creation; reading usually gets the accumulated expiration count, not meaning only one expiration per readable event. eventfd reads the notification count. Both should be read according to the record size specified by the interface, and handle non-blocking empty reads.

signalfd requires blocking the target signal in the relevant thread first. Below is a snippet for program initialization, requiring <signal.h>, <pthread.h>, <sys/signalfd.h>:

sigset_t mask;
sigemptyset(&mask);
sigaddset(&mask, SIGINT);
sigaddset(&mask, SIGTERM);
int err = pthread_sigmask(SIG_BLOCK, &mask, NULL);
if (err != 0) return err;
int sfd = signalfd(-1, &mask, SFD_NONBLOCK | SFD_CLOEXEC);
if (sfd == -1) return errno;

You should establish the mask you want worker threads to inherit before creating them, or coordinate per-thread; otherwise, signals may be received by other threads that did not block them. Forgetting sigemptyset leaves the mask contents uninitialized. SIGKILL and SIGSTOP cannot be taken over by signalfd. signalfd(2)

Real Boundaries of Event-Driven Programming

Executing blocking DNS, disk reads, long computations, or waiting for regular locks inside a ready callback will still block the entire event thread. Offload time-consuming tasks to appropriate execution mechanisms and pass results back via queues, eventfd, or other notifications.

epoll suits the readiness model; io_uring uses request completion as its primary interface. Both require application state machines, backpressure, cancellation, and resource management; interface names do not automatically solve these problems.

Keywords: epoll, readiness, LT, ET, EPOLLONESHOT, half close, backpressure, fairness, fd reuse, timerfd, signalfd, eventfd