PCIe and USB Driver Models

Both PCIe and USB can enumerate devices, but they provide different control mechanisms for drivers. PCI drivers typically operate BAR registers directly and set up DMA queues; USB interface drivers submit requests to the USB core, which executes bus transactions via the host controller. A common challenge for both is that devices and requests can disappear asynchronously, and resources must be released only after the last user has exited.

1. PCI: Configuration Space, BARs, and DMA are Three Separate Things

The configuration space provides device identification, capabilities, and resource window information. The system scans the topology and reserves or allocates available resources; firmware may still describe platform information such as root bridges and configuration space windows. A complete address like 0000:03:00.0 cannot be uniquely identified by writing only bus:device.function when spanning multiple domains.

Standard endpoint configuration headers have six BAR slots; a 64-bit BAR occupies two adjacent slots, and bridge layouts differ. Therefore, you should not pretend that the kernel's pci_dev.resource is a fixed array of six items and iterate over all resources based on that. Interpret resources using APIs such as pci_resource_start/len/flags(), and use appropriate mapping and I/O accessors.

__iomem addresses returned by mapping functions like pci_iomap() cannot be dereferenced arbitrarily or used with memcpy as if they were normal RAM. Use access methods that match the device protocol, such as ioread32() / iowrite32(); consult hardware and platform rules for MMIO ordering and posted write completion requirements.

2. Resource Dependencies in Probe

Typical initialization includes enabling the device, requesting required BARs, setting the DMA mask actually supported by the hardware, mapping registers, allocating queues and interrupts, and then registering external interfaces. The specific order is affected by hardware reset, firmware loading, and interrupt masking requirements; you cannot treat a generic list of calls as a runnable driver.

Enabling bus mastering allows the device to initiate transactions; this should be handled when buffers and device state are ready. Before publishing interfaces externally, the error return path must be able to reverse the order in which resources were acquired. New drivers usually express power management using dev_pm_ops rather than copying the suspend/resume fields from older pci_driver structures.

The PCI Driver Documentation explains initialization and resource interfaces; when writing code, refer to the target kernel headers and ensure that the release responsibilities for managed/unmanaged resources are consistent.

3. MSI/MSI-X: The Requested Upper Bound Is Not the Returned Quantity

Vector Budget: Build Subsequent Resources Based on the Actual Returned Quantity

MSI/MSI-X uses messages sent by the device to trigger interrupts. The MSI-X table supports up to 2048 entries, but the device implementation, platform, and available resources limit the actual quantity. Requesting 1 to 16 vectors might yield only 1, 4, or 8.

The following is a partial example from the initialization phase, not a complete driver:

int nvec = pci_alloc_irq_vectors(pdev, 1, 16, PCI_IRQ_MSIX);
if (nvec < 0)
    return nvec;

/* Iterate only over [0, nvec), then install handlers one by one and check for errors. */
for (int i = 0; i < nvec; ++i) {
    int irq = pci_irq_vector(pdev, i);
    dev_dbg(&pdev->dev, "vector %d uses IRQ %d\n", i, irq);
}

i is the device's vector index, and irq is the Linux IRQ number; the two are not interchangeable. The number of vectors does not equal the number of queues or CPUs: multiple queues can share an interrupt, and affinity determines routing rather than permanently binding to a specific worker thread.

During teardown, first prevent the device from generating further related interrupts, synchronize the handlers correctly, and then release the handlers and vectors according to the conventions of the chosen API. Some managed PCI interfaces have automatic cleanup behavior and should not be unconditionally released manually at the same time. See the MSI Driver Guide.

How to Allocate When You Have Eight Queues but Only Three Vectors

Assume the device allows multiple completion queues to be routed to a single MSI-X vector. We adopt a clear driver strategy: when there are more than one vector, reserve index 0 for management events, and have the remaining vectors take turns handling data queues; when there is only one vector, management events and all data queues share index 0. Whether this routing is supported must be confirmed by the hardware; it cannot be inferred from the number returned by the operating system.

def route_queues(queue_count, linux_irqs):
    if queue_count < 0 or not linux_irqs or len(set(linux_irqs)) != len(linux_irqs):
        raise ValueError("Requires valid queue count and non-duplicate vector IRQ mapping")
    nvec = len(linux_irqs)
    indices = [0 if nvec == 1 else 1 + q % (nvec - 1)
               for q in range(queue_count)]
    groups = {i: [] for i in range(nvec)}
    for q, index in enumerate(indices):
        groups[index].append(q)
    return indices, groups

indices, groups = route_queues(8, [48, 113, 207])
assert indices == [1, 2, 1, 2, 1, 2, 1, 2]
assert groups == {0: [], 1: [0, 2, 4, 6], 2: [1, 3, 5, 7]}
for nvec in (1, 3, 5):
    for count in (0, 1, 8, 17):
        indices, groups = route_queues(count, list(range(100, 100 + nvec)))
        assert sorted(q for group in groups.values() for q in group) == list(range(count))
        assert all(0 <= i < nvec for i in indices)
assert route_queues(8, [48])[0] == [0] * 8

The handler for IRQ 113 must check the completion status of queues 0, 2, 4, and 6; you cannot substitute "received one interrupt" for "exactly one request completed." Interrupt coalescing, multiple completion entries, and queues continuing to generate work all require advancing the consumption position according to the device queue protocol. Sharing a single device vector is not equivalent to the shared line mode of traditional INTx; the two types of "sharing" should be understood separately.

Initialization failures must also roll back based on the resources actually established. If three vectors were allocated but the third request_irq() fails, release only the first two installed handlers, then release the vectors; queues and DMA buffers are still handled in their own acquisition order. After entering normal operation, in addition to synchronizing hard interrupts, you must drain work items already spawned by the handlers; otherwise, releasing the IRQ does not prove that work items no longer access the queues.

4. USB: Drivers Often Bind to Interfaces

A USB device can provide multiple configurations; the selected configuration contains several interfaces; interfaces also have alternate settings, each describing a set of endpoints. A camera's audio and video functions can therefore be handled by different interface drivers.

Common usb_driver.probe() receives a usb_interface *. The driver should check the endpoints of the currently set interface rather than assuming "the first endpoint is always bulk IN." Endpoint zero is the device's default control endpoint and should not be applied as an endpoint unique to each interface.

Here, "interrupt" does not mean the peripheral directly interrupts the CPU at any time. USB bus transfers are scheduled by the host; the HCD's CPU interrupt is a separate mechanism. Port connection detection varies with USB speed and protocol and cannot be uniformly interpreted as a D+ level change.

5. URBs are Asynchronous Requests, Not Proof That Data Has Been Sent

Prepare the endpoint, buffer, and callback context before submission. Successful submission does not mean the device has finished using the data.

A successfully submitted request may complete quickly; the callback may even occur before usb_submit_urb() returns. Therefore, the context used by the callback must be prepared before calling the submission interface. In the diagram, "accepting the request" occurs during the submission call; do not treat the function return as a barrier for the callback start.

Obtain Callback Reference Before Submitting Request

The driver's private request object and the URB itself are two separate objects. The USB core's management of the URB does not automatically acquire a reference for any arbitrary urb->context pointer. The following illustrates the pairing for a single request: the caller first holds a reference to the private object, then reserves one for the completion path before submission. Upon successful submission, the callback releases this reference; upon submission failure (which does not produce this completion callback), the caller revokes the reservation.

class Request:
    def __init__(self):
        self.refs = 1
        self.ready = True
        self.callback_owned = False
        self.completions = 0
        self.freed = False

    def put(self):
        assert self.refs > 0 and not self.freed
        self.refs -= 1
        self.freed = self.refs == 0

    def reserve_callback(self):
        assert self.ready and not self.freed and not self.callback_owned
        self.refs += 1
        self.callback_owned = True

    def finish(self):
        assert self.ready and self.callback_owned and not self.freed
        self.completions += 1
        self.callback_owned = False
        self.put()

    def rollback_failed_submit(self):
        assert self.callback_owned and self.completions == 0
        self.callback_owned = False
        self.put()


def one_request(mode):
    r = Request()
    r.reserve_callback()                 # Must be done before entering submit
    if mode == "failure":
        result = -1                      # Failed submission has no completion callback for this instance
    else:
        result = 0
        if mode == "early":
            r.finish()                   # Callback occurs before the submit function returns
    if result:
        r.rollback_failed_submit()
    assert not r.freed                    # Caller's own reference is still valid
    r.put()                              # Caller finishes this access
    if mode == "late":
        assert not r.freed                # Callback reference continues to protect the object
        r.finish()
    assert r.freed and r.refs == 0
    assert r.completions == (mode != "failure")

for mode in ("failure", "early", "late"):
    one_request(mode)

This model only accounts for the lifetime of the private object for a single submission; real concurrent code also requires appropriate atomic references, state synchronization, and independent URB/buffer management. If the callback resubmits, a complete handover must be established for the new request; you cannot mistake the completion of the old request for the end of the entire session. Saving submitted = true separately and incrementing the reference only after the submission returns is already too late for the early completion path.

Status Success, Byte Completeness, and Business Completeness Must Be Judged Separately

The following discusses non-isochronous transfers such as bulk IN. If a request reads 4096 bytes, a successful return of 512 bytes can be a valid short read; if the upper-layer protocol requires a fixed 4096-byte record, this USB success still does not mean the record is complete. Conversely, a single URB may carry multiple business records, and record boundaries must be maintained by the protocol parser.

Non-isochronous reads can use URB_SHORT_NOT_OK to report short reads as errors, but it will not pad the remaining bytes nor verify business records. For isochronous transfers, you must check the status and length of each iso_frame_desc item; you cannot infer that all frames succeeded based solely on the overall URB status. See the USB Host Side API for field meanings.

Check in order upon completion:

1. Use urb->status to determine the result.
2. Use actual_length to determine the actual length; success does not guarantee it equals the requested length.
3. If resubmission is needed, first confirm that you have not entered a disconnected, closed, or suspended state.

The callback runs in a non-sleepable context; complex blocking work should be handed off to appropriate work mechanisms; otherwise, transmissions may be restarted during the exit process.

• usb_unlink_urb() requests asynchronous cancellation; you cannot immediately release objects that may still be accessed by the callback based on this.
• usb_kill_urb() waits for the request to exit and the completion callback to finish; the call site must allow sleeping and must not hold locks needed by the callback.
• USB anchors can track a group of URBs, facilitating the stopping of in-flight requests in groups, but you still need to manage the driver's own open files, work items, and references.

Interface rules and cancellation semantics are detailed in the URB Documentation.

Try freeing the buffer while a request or callback is active. Then hold a lock needed by the callback and examine the synchronous wait.

6. Hot-Plug and Power Management Both Require Stopping Requests First

disconnect/remove should first mark the device as offline and prevent new requests, then revoke asynchronous requests, wait for callbacks, unregister functional interfaces, and release resources. Users may still hold device files; subsequent operations should see appropriate errors rather than accessing freed private data. The physical device has already been unplugged, which also means you cannot rely on "writing to a register again will definitely succeed" to complete cleanup.

Submission References: Handle Idle Suspend and System Suspend Separately

Runtime PM handles suspend and resume during idle periods, while system suspend coordinates the state of the entire machine. Obtain the correct PM usage reference before submitting I/O and handle the return value on failure; release the reference as agreed upon after transmission completion. USB auto-suspend requires cooperation between the interface driver and the core; PCI may also reconstruct device state after resume. See Power Management.

Continue Reading

• Device Model: Discovery, matching, and object references.
• DMA API: Address and buffer handover behind PCI queues.
• Kernel source entry points: drivers/pci/, drivers/usb/core/, include/linux/usb.h; actual driver error rollback and remove paths should be read together with probe.