Tower and Middleware

Tower abstracts an asynchronous request into a Service and the configuration of "wrapping a service" into a Layer. Timeouts, concurrency limits, buffering, and retries can be composed around this same interface. It does not mandate that requests be HTTP, nor does it automatically resolve business idempotency or all forms of backpressure.

This article uses the Tower 0.5 API as an example. When reading middleware, first determine whether it controls request admission, waiting queues, response Futures, or the subsequent flow of response bodies; these time ranges are often different.

Service has two stages: Admission and Call

The shape of the core interface is:

use std::future::Future;
use std::task::{Context, Poll};

pub trait Service<Request> {
    type Response;
    type Error;
    type Future: Future<Output = Result<Self::Response, Self::Error>>;

    fn poll_ready(&mut self, cx: &mut Context<'_>)
        -> Poll<Result<(), Self::Error>>;
    fn call(&mut self, request: Request) -> Self::Future;
}

The receiver of poll_ready is also &mut self. It may need to reserve permits or update internal state, not just "look at the current load" in a read-only manner. When Pending, it should schedule a wake-up after the state change; the caller waits and then polls again. Only Ready(Ok(())) indicates that a call can be made this time, while Ready(Err(e)) indicates a service error.

Callers must adhere to the readiness contract and cannot treat a single success as an infinite license to call. Some services will panic if called when not ready. You also cannot wait for ready on instance A, clone it to B, and then assume B inherits the reserved capacity of A. Service Documentation

The mutable borrow in call is used to create the associated Future, it does not mean borrowing the service until that Future completes. A service can produce multiple independent in-flight Futures; whether to accept more requests is determined by readiness and implementation. Therefore, Buffer is not a necessity to "bypass Rust's inherent serial limitation."

Observe state changes with a single permit

The following complete program uses the Cargo dependencies listed later in this page. The inner Future is always Pending, so state changes are entirely determined by holding and dropping permits, without needing to wait for timers.

use std::{convert::Infallible, future::pending, task::{Context, Poll, Waker}};
use tower::{Service, ServiceBuilder, service_fn};
fn main() {
    let inner = service_fn(|(): ()| pending::<Result<(), Infallible>>());
    let mut service = ServiceBuilder::new().concurrency_limit(1).service(inner);
    let mut cx = Context::from_waker(Waker::noop());
    assert!(matches!(service.poll_ready(&mut cx), Poll::Ready(Ok(()))));
    let first = service.call(());
    assert!(service.poll_ready(&mut cx).is_pending());
    drop(first);
    assert!(matches!(service.poll_ready(&mut cx), Poll::Ready(Ok(()))));
}

Even if first is never polled, it already holds the permit obtained from readiness. After dropping it, the permit is released, and the next poll_ready becomes ready again. This shows that the starting point of admission resources may be earlier than the first execution of internal business logic; creating a large number of response Futures and storing them may also occupy service capacity.

Waker::noop is used here only to manually check immediately observable states, with the program actively polling again after each state change. Real waiting code should hand off to the executor and ready().await; busy loops cannot be written using invalid wakers. Service Readiness Contract

Layer is the interface for constructing wrappers

Layer::layer receives an inner service and returns a new service. ServiceBuilder places layers added earlier on the outside in the order declared: A, B, S form A(B(S)). Requests pass through A first, then B; completed results return to the caller in the opposite direction. ServiceBuilder

This order changes semantics. Suppose A is retry and B is timeout: Retry(Timeout(S)) allows each attempt to have its own timeout; Timeout(Retry(S)) constrains the response Future containing multiple attempts. Both also require interpreting whether the time spent waiting for readiness is included in the budget.

Runnable example: ready first, then call

Use in an empty Cargo binary project:

[package]
name = "tower-demo"
version = "0.1.0"
edition = "2024"

[dependencies]
tokio = { version = "1", features = ["macros", "rt", "time"] }
tower = { version = "0.5", features = ["util", "limit", "timeout"] }

src/main.rs:

use std::convert::Infallible;
use std::time::Duration;
use tower::{Service, ServiceBuilder, ServiceExt, service_fn};

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), tower::BoxError> {
    let inner = service_fn(|delay_ms: u64| async move {
        tokio::time::sleep(Duration::from_millis(delay_ms)).await;
        Ok::<_, Infallible>(delay_ms)
    });
    let mut service = ServiceBuilder::new()
        .timeout(Duration::from_millis(50))
        .concurrency_limit(1)
        .service(inner);

    let value = service.ready().await?.call(0).await?;
    println!("completed: {value}");

    let result = service.ready().await?.call(500).await;
    match result {
        Ok(value) => println!("completed: {value}"),
        Err(error) => println!("failed: {error}"),
    }
    Ok(())
}

Run cargo run. Usually the first item completes, and the second times out. When the machine is severely blocked, scheduling affects wall-clock observations, so this program cannot be used to calibrate precise delays.

Here, Timeout is outside ConcurrencyLimit, but ready().await happens before call, so the 50 ms does not include that waiting period. If an end-to-end budget is needed, the caller can wrap the entire async process of ready plus call with an absolute deadline, and pass the remaining budget downstream; retries should also consume the same total budget.

Place the absolute deadline outside ready and call

use std::{convert::Infallible, future::pending, time::Duration};
use tower::{Service, ServiceBuilder, ServiceExt, service_fn};
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), tower::BoxError> {
    let inner = service_fn(|(): ()| pending::<Result<(), Infallible>>());
    let mut service = ServiceBuilder::new().concurrency_limit(1).service(inner);
    let first = service.ready().await?.call(());
    let deadline = tokio::time::Instant::now() + Duration::from_millis(1);
    let second = tokio::time::timeout_at(deadline, async {
        service.ready().await?.call(()).await
    }).await;
    assert!(second.is_err()); // Waiting for admission already consumes the budget, before the second call is even issued
    drop(first);
    let ready = service.ready().await?;
    drop(ready.call(()));
    Ok(())
}

This example deliberately leaves the first request unfinished to verify that waiting for a permit is also constrained by the outer deadline; 1 ms is just a short test budget and does not guarantee thread scheduling at that precision. Production chains need to select a single request deadline, and each downstream wait or retry should calculate the remaining time, rather than getting the full budget anew for each layer.

If the inner layer continues to read the body after obtaining the response object, the above deadline only covers the phase of obtaining the Response. For a full-stream deadline, you should continue to use the original deadline for waiting on the next item of the stream; for "idle timeout," update another timer based on the last valid progress. These two types of timeouts solve different problems.

Where does the timeout budget begin?

Separate admission waiting from the response Future. Timeout bounds the Future returned by call, so 40 ms waiting plus 30 ms response can succeed after 70 ms overall. A 50 ms budget from entry needs one absolute deadline around ready and call; exact-boundary races depend on runtime behavior.

Backpressure requires cooperation across the entire call chain

ConcurrencyLimit uses permits to control in-flight response Futures. When the limit is reached, readiness waits; if the caller instead infinitely creates waiting tasks, the system may still consume a large amount of memory. Backpressure only forms overall capacity control when it propagates further upstream, or when combined with bounded queues and rejection policies.

Buffer forwards requests via bounded channels and background worker tasks, providing a cloneable entry point without needing to create dedicated OS threads. It isolates readiness on both sides but introduces queue waiting. The ready obtained by the upstream may reserve a queue position; you should call promptly to avoid holding unused permits. Buffer

RateLimit constrains the number of requests that can be processed within a certain time, which is not equal to a concurrency limit; it cannot be called a generic token bucket that smoothly replenishes tokens without checking the implementation. LoadShed converts temporarily unavailable services into errors, suitable for boundaries where explicit rejection is preferred over continued queuing. RateLimit

What does Timeout cancel?

After Timeout ends the returned response Future, the caller usually discards it, and its inner Future is released accordingly. For normal asynchronous logic that advances via polling, this can stop subsequent advancement; but already sent network requests, committed database transactions, and independently spawned tasks will not be automatically revoked as a result.

Therefore, you cannot write "timeouts never stop internal execution," nor can you write "timeouts revoke all side effects." Whether cancellation is safe should be judged in combination with the Future's Drop, resource ownership, and underlying operation semantics. A timeout error does not prove that the remote business did not succeed.

Two boundaries for retries and streaming responses

Retry determines whether to try again based on the strategy and requires the ability to rebuild or clone the request. Retriability comes from business effects and failure stages: queries are often easy to retry, writes can rely on idempotent keys and persistent deduplication; you cannot rely solely on HTTP method names to guarantee safety. You should also limit the number of attempts, backoff, and total time to avoid amplifying the load further during failures.

The Response of a Service may contain a stream that has not yet been consumed. After the response Future completes, it does not mean the stream has ended: the lifecycle covered by standard Timeout or ConcurrencyLimit may have ended, while the client is still continuously receiving messages. To limit the complete stream, you need to extend permits, timeouts, and cleanup policies to the stream lifecycle.

Use a Response with deferred consumption to see where permits are released

use std::convert::Infallible;
use tower::{Service, ServiceBuilder, ServiceExt, service_fn};
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), tower::BoxError> {
    let inner = service_fn(|(): ()| async {
        Ok::<_, Infallible>(vec![1, 2, 3].into_iter())
    });
    let mut service = ServiceBuilder::new().concurrency_limit(1).service(inner);
    let mut response = service.ready().await?.call(()).await?;
    // The first Response has not been consumed yet, but the next call can be admitted.
    let second = service.ready().await?.call(()).await?;
    assert_eq!(response.next(), Some(1));
    assert_eq!(second.collect::<Vec<_>>(), [1, 2, 3]);
    Ok(())
}

Here, a standard iterator is used to demonstrate the ownership relationship where "it can still be consumed after the object is returned," not simulating a network Stream. Tower's generic limiters cannot guess when a stream is truly finished from an arbitrary Response type. If the restricted resource belongs to the stream, the stream wrapper should hold the permit, with clear release paths for normal completion, errors, and Drop; if you only want to limit the work of generating response headers, the current scope might be just right.

Record queuing and processing times by layer

Queuing, timeouts, and retries should share a logical request identifier, while retaining records for each attempt. Otherwise, a request retried three times might simultaneously inflate call volume and "success rate," masking the user's actual wait time and additional downstream load.

Does the stream retain a permit after Response?

Response is a value that may own a stream consumed later. At “Response returned,” a normal response-Future permit has been released while Body remains alive. To limit complete live streams, their owning wrapper must retain the permit until termination, error or Drop.

Check types and error mapping before integrating with tonic

tonic uses Tower at the transport and generated service layers, but it is not "every gRPC method can directly wrap any Layer." The constraints of middleware such as Request, Response, Error, Clone, Send must match the insertion point; HTTP layer errors also need to be correctly mapped to gRPC Status.

HTTP request tracing usually uses tower-http's TraceLayer, and it is a mistake to assume all traces come from the tower core. Interceptor is suitable for short synchronous metadata processing; asynchronous authentication and response body observation usually require more complete middleware, see tonic.