Tower and Middleware

Tower builds an async middleware stack using two layers of abstraction: the Service trait (handles requests and returns a Future) and Layer (wraps a Service to produce a new Service). Middlewares can be stacked (timeout → retry → rate limiting → business logic), and backpressure is propagated upstream by waiting for readiness. This is the most elegant onion architecture implementation in the Rust async ecosystem.

The Problem Tower Solves

When building network services, there are several cross-cutting concerns: timeouts, rate limiting, concurrency control, retries, and logging/metrics. If you hardcode this logic into every service, it leads to code duplication and poor composability—your gRPC handler would need to handle both business logic and timeouts, and your HTTP handler would require the same rate-limiting logic.

Tower's answer: abstract services as the Service trait and use Layer to compose these concerns via the decorator pattern. Each middleware is responsible for only one thing, and the Layer stack combines them into a complete service.

Service trait: Async request → response

pub trait Service<Request> {
    type Response;
    type Error;
    type Future: Future<Output = Result<Self::Response, Self::Error>>;

    fn poll_ready(&self, cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>>;
    fn call(&mut self, req: Request) -> Self::Future;
}

The two methods of this trait have clear responsibilities:

poll_ready: Checks if the service can accept new requests. A rate limiter checks remaining quotas, and a concurrency limiter checks for available slots. If the service cannot yet receive requests, it returns Poll::Pending—the caller must wait to be woken up and try again. This is Tower's built-in backpressure mechanism—pressure on downstream services is propagated upstream via poll_ready, without requiring any additional message channels.

call: Executes the request and returns a Future. The &mut self signature of call means that calls to the same Service instance are naturally serialized—if you need to call in parallel on the same instance, use the Buffer middleware to make it Clone.

The separation of these two phases is the core difference between Tower and most other middleware frameworks (such as Express/actix middleware)—they typically only have a single fn handle(&self, req) -> Future, lacking the semantics of backpressure.

Layer: Functional Representation of the Decorator Pattern

pub trait Layer<S> {
    type Service;
    fn layer(&self, inner: S) -> Self::Service;
}

A Layer is not a Service—it is a factory for Services. Given an inner service, a Layer returns a wrapper service that encapsulates the inner one. The key advantage: Layers can be composed:

let svc = ServiceBuilder::new()
    .layer(TimeoutLayer::new(Duration::from_secs(5)))
    .layer(ConcurrencyLimitLayer::new(100))
    .layer(RateLimitLayer::new(10, Duration::from_secs(1)))
    .service(my_service);

ServiceBuilder::new().layer(A).layer(B).service(S) actually constructs:

A::layer(B::layer(S))

i.e., A << B << S—each layer wraps the inner one. When a real request arrives, the order is outside-in: request → A → B → S → B → A → response.

Understanding Backpressure

Without backpressure, concurrent requests hit the service directly, potentially overwhelming it. Tower's poll_ready allows middlewares to reject requests before accepting them:

This means when ConcurrencyLimitLayer reaches its concurrency limit, new poll_ready calls return Pending—not a panic, not a rejection, but an elegant deferral. The caller's Future is woken up once a slot becomes free, and processing continues. The entire process is naturally expressed within the Future poll model, without needing additional semaphores or channels.

Common Middlewares

• Timeout: If the Future returned by call is not Ready after a specified time → return a timeout error. Note: this does not stop internal execution (in tokio, you cannot forcibly cancel the underlying task unless using tokio::select! or JoinHandle::abort)
• ConcurrencyLimit: Limits the number of concurrently processed requests via an internal counter—poll_ready returns Pending when the counter reaches the limit
• RateLimit: Limits request rate using a token bucket algorithm—poll_ready returns Pending when tokens are insufficient
• Buffer: Solves the problem that call(&mut self) cannot be shared—internally uses a channel to forward requests to worker threads, exposing a Clone interface externally
• Retry: Automatically retries failed requests—configurable retry conditions and exponential backoff. Note idempotency: retries are only safe for idempotent operations (GET, PUT)
• Trace: Automatically records the duration of each request—no need to manually insert timing code in business logic

tonic is a Native Consumer of Tower

tonic::transport::Server::builder()
    .layer(TimeoutLayer::new(Duration::from_secs(5)))
    .add_service(GreeterServer::new(MyGreeter))
    .serve(addr).await?;

Each gRPC method automatically becomes a Service—tonic's codegen generates a Service implementation for your handler. This means all Tower middlewares can be applied to any gRPC service without modification.

References

• tower: docs.rs/tower (the README contains explanations of the design philosophy)
• tonic: github.com/hyperium/tonic

Keywords: tower, Service, Layer, middleware, tonic, poll_ready, backpressure