Backpressure — Senior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Adaptive Concurrency: AIMD
5. Vegas-Style Adaptive Limits
6. Token Buckets and Leaky Buckets
7. Queue Theory in Practice
8. Little's Law and Capacity Planning
9. Latency Budgets and Tail Tolerance
10. Cross-Service Backpressure Protocols
11. HTTP/2 and gRPC Flow Control
12. Kafka and Message-Queue Backpressure
13. Database Backpressure
14. Hedged Requests and Speculative Execution
15. Designing for Graceful Degradation
16. Observability at Scale
17. Code Examples
18. Coding Patterns
19. Architecture Patterns
20. Common Mistakes
21. Tricky Points
22. Test
23. Cheat Sheet
24. Summary
25. What You Can Build
26. Further Reading
27. Related Topics
28. Diagrams & Visual Aids

Introduction¶

Focus: "How do you design backpressure across multiple services, processes, and tiers — and how do you make the concurrency limit adapt to the system's real capacity?"

At the senior level, backpressure stops being a property of a single channel and starts being a property of an entire distributed system. You stop hard-coding "100 concurrent requests" and start measuring the real limit and adapting to it.

This page covers:

• AIMD (Additive Increase, Multiplicative Decrease) — the canonical adaptive concurrency algorithm.
• Vegas-style limits — using latency, not failure, as the adaptation signal.
• Token buckets and leaky buckets — the two classical shapes for rate limiting and rate smoothing.
• Queue theory — M/M/1, M/M/c, and Little's law applied to capacity planning.
• Cross-service backpressure — propagating "I am full" across network boundaries.
• HTTP/2 and gRPC flow control — what the transport gives you and where you must add more.
• Kafka consumer lag — backpressure in a persistent, replayable queue.
• Database backpressure — connection pools, lock waits, and PgBouncer.
• Hedged requests — sending duplicates to reduce tail latency.
• Graceful degradation — what your service returns when it cannot keep up.
• Observability — SLOs, error budgets, and what dashboards a senior on-call expects.

You should be comfortable with everything in junior.md and middle.md. You have probably operated a production system through at least one overload incident.

Prerequisites¶

• Built and operated a Go service in production for at least a year.
• Comfortable reading Go runtime internals (runtime/proc.go, runtime/chan.go).
• Have shipped a feature that required adaptive concurrency, rate limiting, or load shedding.
• Have done capacity planning with measured data.
• Understand TCP flow control conceptually.

Glossary¶

Adaptive Concurrency: AIMD¶

The static concurrency limit problem: you hard-code "8 workers" or "100 slots" based on a benchmark. The real capacity depends on the workload mix, the hardware, the GC, and what other processes are doing. Hard-coded numbers are wrong as soon as the environment shifts.

AIMD adapts. Start with a small limit; increase on success; decrease on failure.

type AIMD struct {
    mu       sync.Mutex
    limit    int
    inFlight int
    succ     int
    failed   int
    minLimit int
    maxLimit int
    incEvery int
    decRatio float64
}

func NewAIMD(initial, min, max int) *AIMD {
    return &AIMD{
        limit:    initial,
        minLimit: min,
        maxLimit: max,
        incEvery: 20,
        decRatio: 0.5,
    }
}

func (a *AIMD) Acquire() bool {
    a.mu.Lock()
    defer a.mu.Unlock()
    if a.inFlight >= a.limit {
        return false
    }
    a.inFlight++
    return true
}

func (a *AIMD) Release(ok bool) {
    a.mu.Lock()
    defer a.mu.Unlock()
    a.inFlight--
    if ok {
        a.succ++
        if a.succ >= a.incEvery {
            a.succ = 0
            if a.limit < a.maxLimit {
                a.limit++
            }
        }
    } else {
        a.failed++
        a.limit = max(a.minLimit, int(float64(a.limit)*a.decRatio))
        a.succ = 0
    }
}

func (a *AIMD) Limit() int {
    a.mu.Lock()
    defer a.mu.Unlock()
    return a.limit
}

The behaviour is a sawtooth: the limit climbs linearly on the success path, then halves on each failure. Over time, the limit oscillates around the system's true capacity.

Key parameters:

• incEvery controls how aggressively the limit grows. Smaller = faster growth, more probing.
• decRatio controls the back-off depth. 0.5 is classic; some systems prefer 0.7 for less aggressive cuts.
• minLimit prevents the limit from collapsing to zero during a bad spell.
• maxLimit caps growth (defensive against malformed feedback).

What counts as "failure" for AIMD?¶

The signal you back off on matters. Options:

• Latency exceeds SLO. Catches slowness before failure.
• Server returns 5xx. Common; can be too late.
• Timeout. Pure congestion signal.
• Queue depth high. Pre-emptive.

A robust adaptive limiter uses multiple signals: any one triggers backoff. Otherwise the limiter can climb past the latency cliff before failures begin.

When AIMD shines¶

• Variable workload mixes — easy/hard requests share a path.
• Variable hardware — different replicas with different capacity.
• Variable downstream — DB load changes throughout the day.

When AIMD is overkill¶

• Stable workload with known capacity. A static limit is simpler.
• Hard latency contract. AIMD's probing causes occasional overshoots.
• Tiny variance — adaptation buys little.

Vegas-Style Adaptive Limits¶

AIMD is loss-based: it grows until something fails, then backs off. Vegas-style limiters are latency-based: they grow until latency rises, then back off — before failures begin.

The Vegas algorithm:

1. Compute baseline RTT under low load (smallest observed RTT).
2. Track current RTT as a moving average.
3. Compute the "expected" inflight: expected = limit * baseline / current_rtt.
4. If inflight - expected < alpha, the system is underloaded; increase limit.
5. If inflight - expected > beta, the system is overloaded; decrease limit.

For Go, Netflix's concurrency-limits (Java) has a Go port (github.com/platinummonkey/go-metrics, and others). The core idea is small:

type Vegas struct {
    mu       sync.Mutex
    limit    int
    minRTT   time.Duration
    estRTT   time.Duration
    inFlight int
    alpha    int
    beta     int
}

func (v *Vegas) Update(rtt time.Duration) {
    v.mu.Lock()
    defer v.mu.Unlock()
    if rtt < v.minRTT || v.minRTT == 0 {
        v.minRTT = rtt
    }
    v.estRTT = v.estRTT*9/10 + rtt/10 // EWMA
    expected := float64(v.limit) * float64(v.minRTT) / float64(v.estRTT)
    diff := float64(v.inFlight) - expected
    switch {
    case diff < float64(v.alpha):
        v.limit++
    case diff > float64(v.beta):
        v.limit--
    }
}

This is much more responsive than AIMD because it reacts before failure. The cost is sensitivity to RTT noise — a noisy upstream can cause oscillation.

Choosing between AIMD and Vegas¶

• AIMD is robust and simple. Use for backend services where occasional failures are acceptable.
• Vegas is responsive and pre-emptive. Use for latency-sensitive paths.
• Both implementations are < 100 lines. Try both, measure under your workload.

Token Buckets and Leaky Buckets¶

These two algorithms govern rate, not concurrency. They are upstream of backpressure: they cap arrival, not in-flight work.

Token Bucket¶

A bucket holds up to N tokens. Tokens refill at rate R per second. Each request consumes one token. When the bucket is empty, requests block or are rejected.

type TokenBucket struct {
    mu       sync.Mutex
    tokens   float64
    cap      float64
    rate     float64
    last     time.Time
}

func NewTokenBucket(cap, rate float64) *TokenBucket {
    return &TokenBucket{cap: cap, rate: rate, tokens: cap, last: time.Now()}
}

func (b *TokenBucket) Allow() bool {
    b.mu.Lock()
    defer b.mu.Unlock()
    now := time.Now()
    elapsed := now.Sub(b.last).Seconds()
    b.last = now
    b.tokens = min(b.cap, b.tokens + elapsed*b.rate)
    if b.tokens >= 1 {
        b.tokens--
        return true
    }
    return false
}

This is the algorithm Go's standard golang.org/x/time/rate uses. Properties:

• Bursts up to cap are allowed.
• Long-term rate is bounded by rate.
• Implementation is lock-free if you use atomics; the version above is simple-locking.

Leaky Bucket¶

A queue (the bucket) drains at a fixed rate. Arrivals enqueue; over-arrival is dropped or queued. The output is a smoothed stream at the drain rate.

type LeakyBucket struct {
    queue chan struct{}
    drain time.Duration
}

func NewLeakyBucket(cap int, rate time.Duration) *LeakyBucket {
    b := &LeakyBucket{queue: make(chan struct{}, cap), drain: rate}
    go func() {
        t := time.NewTicker(rate)
        defer t.Stop()
        for range t.C {
            select {
            case <-b.queue:
            default:
            }
        }
    }()
    return b
}

func (b *LeakyBucket) Add() bool {
    select {
    case b.queue <- struct{}{}:
        return true
    default:
        return false
    }
}

Properties:

• Output is rate-limited and smooth (no bursts).
• Bursts queue up to cap then drop.
• Useful for downstream protection: "I will not send more than X RPS to the downstream, period."

When to use which¶

Both are about rate. Combine with concurrency limit for full coverage.

Queue Theory in Practice¶

Queue theory gives you the math behind capacity planning. Two formulas are enough for most decisions.

M/M/1: single server, exponential arrivals and service times¶

Define: - λ = arrival rate (requests per second) - μ = service rate (requests per second) - ρ = λ/μ (utilisation; must be < 1)

Then: - Average queue length: Lq = ρ²/(1-ρ) - Average wait time: Wq = ρ/(μ(1-ρ)) - Average response time: W = 1/(μ-λ)

The killer is (1-ρ) in the denominator. At ρ = 0.5, queue length is 0.5. At ρ = 0.9, queue length is 8.1. At ρ = 0.99, queue length is 98.

Practical implication: do not target high utilisation. Aim for ρ around 0.5–0.7 in latency-sensitive services. The "wasted" capacity is your buffer against bursts.

M/M/c: c servers¶

For c workers: - ρ = λ/(c·μ) - Same intuition: queueing grows nonlinearly as ρ approaches 1.

With more servers, you can run at slightly higher utilisation safely. A 100-worker pool at ρ=0.9 has less queueing than a 1-worker pool at ρ=0.9. This is why "scale horizontally" works.

Applying it: choosing worker count¶

Given expected λ and per-request service time 1/μ, the required worker count for ρ ≤ 0.7 is:

c >= λ / (0.7 × μ)

Example: λ = 1000 RPS, service time = 50 ms (μ = 20 RPS per worker). Then c ≥ 1000 / (0.7 × 20) = 71.4. Round up to 80 workers (with headroom).

If you can only afford 50 workers, you will run at ρ = 1000/(50×20) = 1.0 — saturation. Queueing will be unbounded; backpressure must shed.

Applying it: choosing buffer size¶

Given target tail wait (p99) and worker count, the buffer should not exceed tail × μ × c.

Example: p99 wait target = 100 ms; c = 80; μ = 20 RPS per worker; tail × μ × c = 0.1 × 20 × 80 = 160.

Round to 128 buffer. Beyond this, items spend more than 100 ms in queue — violating the SLO.

This is where the "buffer = 2× worker count" heuristic breaks down. Use the math for serious sizing.

Little's Law and Capacity Planning¶

Little's law: L = λ × W.

In English: the average number of items in a system equals the arrival rate times the average wait time. Surprisingly general — works for any stable queue, regardless of arrival distribution.

Practical uses:

1. Verify a system is stable. If λ is constant and L is climbing, W is climbing — the queue is growing. Something is wrong.
2. Predict capacity. If you want to handle λ = 1000 RPS with W = 100 ms, you need L = 100 in-flight on average. Provision accordingly.
3. Tune buffer size. L should be small relative to capacity. If L grows past capacity, you are saturated.

A backpressure-aware service's dashboard has L (in-flight), λ (arrival rate), and W (latency) as three of its primary signals. Little's law connects them; an inconsistency reveals a bug.

Latency Budgets and Tail Tolerance¶

A senior question: "What's the latency budget for queueing?" If your SLO is p99 < 200 ms and the service time is 50 ms, you have 150 ms of headroom — but not all for queueing. You also need budget for:

• Network in/out: 5–10 ms each side.
• Downstream calls: usually the biggest.
• Serialisation, GC, kernel scheduling: 10–20 ms.

Realistic queueing budget might be only 50–80 ms of the SLO. That bounds: - Buffer size (per the M/M/c math above). - Submit timeout (SubmitCtx deadline). - Admission policy.

For p99 SLOs, the rule of thumb: buffer size in slots × per-slot time should be less than 1/3 of the latency budget.

Cross-Service Backpressure Protocols¶

When backpressure must cross a network boundary, you need a protocol both sides understand. The common ones:

HTTP 503 / 429¶

503 ("Service Unavailable") and 429 ("Too Many Requests") are the two HTTP signals for overload. Subtleties:

• Retry-After header can tell the client how long to wait. If you have a known recovery time, include it.
• Retry-After: 1 (seconds) vs Retry-After: <date>. Both are valid; seconds is more common for backpressure.
• Idempotency. Clients must only auto-retry idempotent requests. POST without an idempotency key should not be auto-retried.

w.Header().Set("Retry-After", "1")
http.Error(w, "busy", http.StatusServiceUnavailable)

gRPC ResourceExhausted¶

The gRPC equivalent is codes.ResourceExhausted. Clients can be configured to retry on this code with backoff. The grpc-retry plugin in the standard Go gRPC library does this.

return status.Error(codes.ResourceExhausted, "pool busy")

Reactive streams (request-N protocol)¶

A subscriber sends "request N items" to a publisher. The publisher sends at most N items, then waits for the next request. Used in Reactive Streams (Java), RxJava, and some Go libraries (less common in Go).

The mechanism is "explicit credit" — the consumer hands out credits, the producer spends them. Cleaner than HTTP for streaming; more invasive to implement.

Kafka consumer lag¶

In Kafka, the producer writes to a partition; consumers read at their own rate. Backpressure is implicit: a slow consumer falls behind (lag grows). The producer is unaffected (Kafka is the buffer).

This works only if Kafka has enough disk to absorb the lag and if the lag eventually catches up. If lag grows unboundedly, you have a real capacity problem; no protocol fixes it.

Monitoring lag (kafka_consumer_lag) is the equivalent of monitoring queue depth.

HTTP/2 and gRPC Flow Control¶

HTTP/2 has built-in flow control via "windows." Each stream and each connection has a window size that the receiver advertises. The sender can send at most window bytes before waiting for the receiver to ACK.

Default windows are small (64 KB). For high-throughput streaming, tune:

server := grpc.NewServer(
    grpc.InitialWindowSize(1 << 20),       // 1 MB per-stream
    grpc.InitialConnWindowSize(2 << 20),   // 2 MB per-conn
)

Beware: large windows mean the consumer can be far behind without backpressure showing — which can hide bugs. Tune for the workload.

For gRPC streaming, the client sends to the server via a stream; the server is the consumer. If the server is slow, the stream's window does not refill, the client cannot send. This is automatic backpressure across the TCP boundary.

The application-level layer (stream.Send on the server side) blocks if the network is congested. Add admission control on top: only admit a streaming request if there is server capacity to consume it.

Kafka and Message-Queue Backpressure¶

Kafka shifts backpressure from in-process to disk-and-time. The producer never feels the consumer directly; the broker absorbs the offset gap.

Implications:

• Producer backpressure is rare. Only happens when the broker is overloaded.
• Consumer lag is the metric. Increases when the consumer is slower than the producer.
• At-least-once delivery means duplicate work. The consumer must be idempotent.
• Resharding / repartitioning is operational. Adding partitions and consumers is how you scale.

For Go consumers, the standard libraries (segmentio/kafka-go, confluent-kafka-go) have prefetch settings. A small prefetch keeps the consumer's local buffer small; a large prefetch buffers more but increases memory.

reader := kafka.NewReader(kafka.ReaderConfig{
    Brokers: []string{"..."},
    Topic:   "events",
    GroupID: "service-x",
    MinBytes: 10e3,
    MaxBytes: 10e6,
    QueueCapacity: 100, // local prefetch buffer
})

Set QueueCapacity to match your processing rate; too large and a crash loses more work; too small and you starve the worker pool.

Database Backpressure¶

Database connection pools are themselves backpressure mechanisms. When all connections are in use, db.Query blocks (or fails with deadline).

For Postgres specifically:

• MaxOpenConns caps concurrent in-flight DB queries.
• PgBouncer (transaction-pooling) multiplexes many Go connections onto fewer Postgres connections.
• statement_timeout server-side prevents runaway queries.

Senior-level design: every DB call has a context deadline. The deadline should be a fraction of the request's overall deadline (say, half). Within the connection pool's wait, the deadline expires — caller sees a clean timeout, not a hang.

db.SetMaxOpenConns(50)
db.SetMaxIdleConns(20)
db.SetConnMaxLifetime(time.Hour)

ctx, cancel := context.WithTimeout(reqCtx, reqCtx.Deadline().Sub(time.Now())/2)
defer cancel()
rows, err := db.QueryContext(ctx, q)

When the DB is the bottleneck, the connection pool is the signal. Watch pq_pool_wait_count and similar metrics.

Hedged Requests and Speculative Execution¶

Tail latency is dominated by stragglers — a small percentage of requests that take much longer than average. Hedged requests reduce tail by sending duplicates.

Algorithm:

1. Send the request.
2. After T (e.g., p95 of the response time), send a duplicate to another replica.
3. Use the first response; cancel the other.

This adds at most one extra call per request (in the p5 case where the first is slow), so worst-case overhead is 5%.

func hedge(ctx context.Context, fn func(context.Context) error, delay time.Duration) error {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()
    done := make(chan error, 2)
    go func() { done <- fn(ctx) }()
    select {
    case <-time.After(delay):
        go func() { done <- fn(ctx) }()
    case err := <-done:
        return err
    }
    return <-done
}

Hedged requests interact with backpressure: a hedged duplicate adds load. If your downstream is overloaded, hedging makes it worse. Combine with circuit breakers or disable hedging when downstream rejection rate is high.

Use hedging on idempotent reads (e.g., cache lookups). Do not hedge on writes without idempotency.

Designing for Graceful Degradation¶

A backpressure-aware service should not just "return 503." It should degrade — serve a cheaper response when the full one is not possible. Examples:

• Search returns top 10 cached results instead of computing full ranking.
• Recommendation engine returns generic popular items instead of personalised ones.
• Image API returns a placeholder for huge resizes.
• Feed shows posts without like-counts when the like-service is degraded.
• Login uses cached session without re-validating when the auth-service is slow.

The pattern is fallback + circuit breaker + backpressure:

func (s *Service) Get(ctx context.Context, id string) (Result, error) {
    if s.breaker.Allow() {
        if r, err := s.primary(ctx, id); err == nil {
            return r, nil
        }
        s.breaker.RecordFailure()
    }
    return s.fallback(ctx, id) // cheap, always available
}

When primary is overloaded (high reject rate trips the breaker), the service silently falls back. Users get a slightly degraded experience instead of an outage.

This requires a cheap fallback. Building one is half the design work; the other half is wiring it correctly.

Observability at Scale¶

For a service handling 10K+ RPS, dashboards and alerts must be opinionated. Some patterns:

USE method (Brendan Gregg)¶

For each resource: Utilisation, Saturation, Errors. Apply to:

• CPU (utilisation = %, saturation = runqueue length, errors = throttling).
• Memory (utilisation = used, saturation = swap, errors = OOM).
• Concurrency pool (utilisation = in-flight/limit, saturation = submit-block rate, errors = rejection rate).

RED method (Tom Wilkie)¶

For each service: Rate, Errors, Duration. Three metrics per endpoint. The minimum viable dashboard.

Golden signals (Google SRE)¶

Traffic, errors, latency, saturation. Four signals; saturation is the backpressure one.

Service-specific signals¶

• Queue depth (gauge, per-pool).
• Submit wait time (histogram).
• In-flight count (gauge).
• Drop rate (counter).
• Reject rate (counter).
• Adaptive limit (gauge — if using AIMD/Vegas).
• Downstream call duration (histogram, per-downstream).

For each, define: - Normal range. - Alert threshold. - Runbook action.

A dashboard without runbooks is decoration. Every alert must have a known response.

Code Examples¶

Example 1 — A Vegas-style adaptive limiter¶

package vegas

import (
    "context"
    "sync"
    "time"
)

type Limiter struct {
    mu       sync.Mutex
    limit    int
    inFlight int
    minRTT   time.Duration
    estRTT   time.Duration
    alpha    int
    beta     int
}

func New() *Limiter {
    return &Limiter{
        limit: 10, alpha: 3, beta: 6,
    }
}

func (l *Limiter) Acquire(ctx context.Context) (release func(), err error) {
    start := time.Now()
    l.mu.Lock()
    if l.inFlight >= l.limit {
        l.mu.Unlock()
        return nil, ErrLimited
    }
    l.inFlight++
    l.mu.Unlock()
    return func() {
        rtt := time.Since(start)
        l.observe(rtt)
    }, nil
}

func (l *Limiter) observe(rtt time.Duration) {
    l.mu.Lock()
    defer l.mu.Unlock()
    l.inFlight--
    if rtt < l.minRTT || l.minRTT == 0 { l.minRTT = rtt }
    if l.estRTT == 0 { l.estRTT = rtt } else { l.estRTT = l.estRTT*9/10 + rtt/10 }
    expected := int(float64(l.limit) * float64(l.minRTT) / float64(l.estRTT))
    diff := l.inFlight - expected
    switch {
    case diff < l.alpha:
        l.limit++
    case diff > l.beta:
        if l.limit > 1 { l.limit-- }
    }
}

This adapts the limit based on observed latency. When the system is fast (RTT near minimum), the limit grows; when slow, it shrinks.

Example 2 — Cross-service backpressure protocol¶

type Client struct {
    base    string
    breaker *Breaker
}

func (c *Client) Do(ctx context.Context, req *Request) (*Response, error) {
    if !c.breaker.Allow() {
        return nil, ErrCircuitOpen
    }
    resp, err := c.do(ctx, req)
    switch {
    case err != nil:
        c.breaker.RecordFailure()
    case resp.StatusCode == 503 || resp.StatusCode == 429:
        c.breaker.RecordFailure()
        if ra := resp.Header.Get("Retry-After"); ra != "" {
            if d, err := strconv.Atoi(ra); err == nil {
                c.breaker.PauseFor(time.Duration(d) * time.Second)
            }
        }
    default:
        c.breaker.RecordSuccess()
    }
    return resp, err
}

The client honours 503/429 with Retry-After and trips the circuit breaker. The breaker pauses further calls until recovery time has passed. Combined with server-side rejection, this is the loop that keeps a cascading outage from forming.

Example 3 — Hedged read¶

func HedgedRead(ctx context.Context, replicas []func(context.Context) ([]byte, error), delay time.Duration) ([]byte, error) {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()
    type result struct { data []byte; err error }
    ch := make(chan result, len(replicas))
    started := 0
    var fire func()
    fire = func() {
        if started >= len(replicas) { return }
        i := started; started++
        go func() {
            d, err := replicas[i](ctx)
            ch <- result{d, err}
        }()
    }
    fire()
    timer := time.NewTimer(delay)
    defer timer.Stop()
    for {
        select {
        case <-timer.C:
            fire()
            timer.Reset(delay)
        case r := <-ch:
            if r.err == nil {
                return r.data, nil
            }
            if started >= len(replicas) {
                return nil, r.err
            }
        case <-ctx.Done():
            return nil, ctx.Err()
        }
    }
}

Reads with successive hedges. Each delay launches another replica. First successful one wins.

Example 4 — Bounded outbound rate to a downstream¶

type SafeDownstream struct {
    rate    *rate.Limiter
    breaker *Breaker
    sem     *semaphore.Weighted
}

func (s *SafeDownstream) Call(ctx context.Context, req *Request) (*Response, error) {
    if err := s.rate.Wait(ctx); err != nil { return nil, err }
    if !s.breaker.Allow() { return nil, ErrCircuit }
    if err := s.sem.Acquire(ctx, 1); err != nil { return nil, err }
    defer s.sem.Release(1)
    resp, err := s.client.Do(ctx, req)
    s.breaker.Record(err)
    return resp, err
}

Three layers: rate (token bucket), concurrency (semaphore), failure protection (breaker). Together they form a defensive wrapper that protects your service from a misbehaving downstream — and protects the downstream from your service.

Coding Patterns¶

Pattern 1 — Adaptive limiter wrapping a bounded pool¶

type AdaptivePool struct {
    pool    *Pool
    limiter *AIMD
}

func (a *AdaptivePool) Submit(ctx context.Context, j Job) error {
    if !a.limiter.Acquire() {
        return ErrLimit
    }
    err := a.pool.SubmitCtx(ctx, j)
    a.limiter.Release(err == nil)
    return err
}

The bounded pool sets a hard ceiling; AIMD sets the soft, adaptive limit underneath.

Pattern 2 — Two-tier shedding¶

When mildly overloaded, drop low-priority work first. Only drop high-priority when severely overloaded.

type Shedder struct {
    low, hi int // queue depth thresholds
}

func (s *Shedder) Allow(priority int, depth int) bool {
    switch {
    case depth < s.low:    return true
    case depth < s.hi:     return priority >= 1
    default:               return priority >= 2
    }
}

A senior service distinguishes between "drop the analytics call" and "drop the payment authorisation."

Pattern 3 — Out-of-band metrics for queue depth¶

If the queue is in another process (Kafka, Redis), check its depth via the broker's API and incorporate into local admission decisions.

go func() {
    t := time.NewTicker(5 * time.Second)
    defer t.Stop()
    for range t.C {
        lag := kafkaConsumerLag()
        if lag > threshold {
            admissionMode.Store(int32(ShedAggressively))
        } else {
            admissionMode.Store(int32(Normal))
        }
    }
}()

The service self-throttles when its downstream queue grows. This is "feedback loop" backpressure — common in event-driven architectures.

Architecture Patterns¶

Pattern 1 — Edge-only admission¶

Apply admission control at the edge (load balancer, ingress). Internal services trust each other. Simple, but vulnerable to internal traffic surprises.

Pattern 2 — Defense in depth¶

Every hop has its own admission. More resilient; more complex. Common in large services.

Pattern 3 — Capacity broker¶

A central service that allocates "tokens" or capacity to other services. Used at very high scale (e.g., per-region quotas). Complex.

Pattern 4 — Reactive scaling¶

Scale workers (Kubernetes HPA) based on queue depth or saturation. Combines with backpressure to handle transient bursts (shed) and sustained load (scale).

Pattern 5 — Async ingest + sync compute¶

Accept requests asynchronously into a durable queue. Process synchronously from the queue with bounded concurrency. The queue is the buffer; the compute layer applies backpressure on its own input.

[client] ──(POST /ingest)──► [accept service] ──► [Kafka] ──► [compute pool] ──► [DB]
                              never overloaded     buffer        bounded         downstream

Used for write-heavy paths where occasional delays are acceptable. The accept service is trivially fast; the compute pool absorbs load.

Common Mistakes¶

1. Using a static concurrency limit when workload is highly variable.
2. Adapting without measuring — picking AIMD parameters by guess.
3. Trusting a single metric. Use queue depth + latency + error rate together.
4. Ignoring downstream signals. Your client should honor Retry-After.
5. Hedging on writes without idempotency.
6. Setting connection-pool size larger than the DB can handle.
7. Not draining on shutdown.
8. Not testing the overload path.

Tricky Points¶

• AIMD oscillates. This is feature, not bug. The probe-and-back-off is what makes it adaptive.
• Vegas requires baseline RTT. During cold start the baseline is unknown; the algorithm may overshoot.
• Hedging doubles load. Be careful in already-overloaded systems.
• Circuit breakers can flap. Use a "half-open" state to test recovery without committing.
• Kafka producer is rarely backpressured. Lag grows on disk; backpressure surfaces on the consumer side.

Test¶

func TestAIMDIncreasesOnSuccess(t *testing.T) {
    a := NewAIMD(10, 1, 100)
    for i := 0; i < 200; i++ {
        if !a.Acquire() { t.Fatal("unexpected limit") }
        a.Release(true)
    }
    if a.Limit() <= 10 { t.Fatal("limit did not grow") }
}

func TestAIMDDecreasesOnFailure(t *testing.T) {
    a := NewAIMD(20, 1, 100)
    a.Acquire()
    a.Release(false)
    if a.Limit() >= 20 { t.Fatal("limit did not shrink") }
}

Cheat Sheet¶

Summary¶

Senior-level backpressure is system design, not coding. The concurrency limit adapts to real capacity (AIMD/Vegas). The rate is shaped by buckets. Capacity is planned with queue theory. Tail latency is managed with hedges. Cross-service overload propagates via standardised signals (HTTP 503, gRPC ResourceExhausted, Kafka lag). Database limits are explicit in the pool. Graceful degradation provides a fallback. All of it is observable with the USE/RED/golden-signals frameworks.

A senior engineer builds services that survive Black Friday, deployment churn, and downstream chaos — not because they are faster, but because they bend cleanly under load and recover automatically.

What You Can Build¶

• A worker pool with AIMD/Vegas-driven dynamic concurrency.
• A client library that honours 503/429 with intelligent retry.
• A service with priority-tiered shedding and graceful fallbacks.
• A monitoring dashboard for backpressure with USE, RED, and golden signals.
• A capacity-planning spreadsheet driven by Little's law and M/M/c.

Further Reading¶

• Site Reliability Engineering (Google), chapter on overload.
• Netflix Tech Blog on adaptive concurrency.
• "Performance is a Feature" — Eric Brewer.
• "Queueing Theory in Action" — Adrian Cockcroft.
• TCP Vegas paper (Brakmo & Peterson, 1994).
• golang.org/x/time/rate — official rate-limiter doc.
• HTTP/2 RFC 7540, section 5.2 (Flow Control).

Related Topics¶

• Drain pattern
• Dynamic worker scaling
• Batching
• Rate limiting (token, leaky)
• Circuit breakers
• Reactive systems

Diagrams & Visual Aids¶

AIMD limit over time:

       /\        /\        /\
      /  \      /  \      /  \
     /    \    /    \    /    \
    /      \__/      \__/      \__
   ↑       ↑         ↑         ↑
   limit    failure   failure   failure
   grows    halve     halve     halve

Backpressure across services:

  Client ──► Service A ──► Service B ──► DB
                ▲              ▲           │
                │              │           │
        503/Retry-After   gRPC ResourceExhausted
                │              │
                └─── backoff   └─── circuit breaker

Defense in depth:

         [load balancer rate limit]
                  ↓
         [edge service admission]
                  ↓
         [internal service pool]
                  ↓
         [DB connection pool]
                  ↓
              [DB]

Deep Dive: An Adaptive Concurrency Library¶

Let us build a self-contained adaptive concurrency library that combines AIMD ideas with realistic production-grade features: minimum limit, hysteresis, history-aware decisions, and observability hooks.

package adaptive

import (
    "context"
    "errors"
    "math"
    "sync"
    "sync/atomic"
    "time"
)

var ErrLimit = errors.New("adaptive: concurrency limit reached")

type Observer interface {
    OnAcquire(limit, inFlight int)
    OnRelease(rtt time.Duration, err error)
    OnLimitChange(old, new int, reason string)
}

type Limiter struct {
    mu        sync.Mutex
    limit     int
    inFlight  int
    minLimit  int
    maxLimit  int
    minRTT    time.Duration
    estRTT    time.Duration
    rttAlpha  float64
    successes int
    growEvery int
    backoff   float64

    obs Observer
    stats Stats
}

type Stats struct {
    Acquires atomic.Uint64
    Rejects  atomic.Uint64
    Growths  atomic.Uint64
    Shrinks  atomic.Uint64
    LastRTT  atomic.Uint64
    Limit    atomic.Int64
}

func New(initial, min, max int) *Limiter {
    return &Limiter{
        limit:     initial,
        minLimit:  min,
        maxLimit:  max,
        growEvery: 20,
        backoff:   0.5,
        rttAlpha:  0.125,
    }
}

func (l *Limiter) Acquire(ctx context.Context) (release func(error), err error) {
    l.mu.Lock()
    if l.inFlight >= l.limit {
        l.mu.Unlock()
        l.stats.Rejects.Add(1)
        return nil, ErrLimit
    }
    l.inFlight++
    inFlight, limit := l.inFlight, l.limit
    l.mu.Unlock()
    l.stats.Acquires.Add(1)
    if l.obs != nil {
        l.obs.OnAcquire(limit, inFlight)
    }
    start := time.Now()
    return func(opErr error) {
        rtt := time.Since(start)
        l.feedback(rtt, opErr)
        if l.obs != nil {
            l.obs.OnRelease(rtt, opErr)
        }
        l.stats.LastRTT.Store(uint64(rtt))
    }, nil
}

func (l *Limiter) feedback(rtt time.Duration, opErr error) {
    l.mu.Lock()
    defer l.mu.Unlock()
    l.inFlight--
    if opErr != nil {
        l.shrink("error")
        return
    }
    // Update RTT EWMAs.
    if l.minRTT == 0 || rtt < l.minRTT {
        l.minRTT = rtt
    }
    if l.estRTT == 0 {
        l.estRTT = rtt
    } else {
        l.estRTT = time.Duration(float64(l.estRTT)*(1-l.rttAlpha) + float64(rtt)*l.rttAlpha)
    }
    // If estimated RTT is much larger than minimum, we are seeing queueing.
    if l.estRTT > l.minRTT*2 {
        l.shrink("rtt-grew")
        return
    }
    l.successes++
    if l.successes >= l.growEvery {
        l.successes = 0
        l.grow("steady-success")
    }
}

func (l *Limiter) grow(reason string) {
    old := l.limit
    if l.limit < l.maxLimit {
        l.limit++
        l.stats.Growths.Add(1)
        l.stats.Limit.Store(int64(l.limit))
        if l.obs != nil {
            l.obs.OnLimitChange(old, l.limit, reason)
        }
    }
}

func (l *Limiter) shrink(reason string) {
    old := l.limit
    n := int(math.Max(float64(l.minLimit), float64(l.limit)*l.backoff))
    if n < l.limit {
        l.limit = n
        l.successes = 0
        l.stats.Shrinks.Add(1)
        l.stats.Limit.Store(int64(l.limit))
        if l.obs != nil {
            l.obs.OnLimitChange(old, l.limit, reason)
        }
    }
}

func (l *Limiter) SetObserver(o Observer) { l.obs = o }
func (l *Limiter) Limit() int             { l.mu.Lock(); defer l.mu.Unlock(); return l.limit }
func (l *Limiter) InFlight() int          { l.mu.Lock(); defer l.mu.Unlock(); return l.inFlight }
func (l *Limiter) Stats() *Stats          { return &l.stats }

Notes:

• Failures and latency growth both trigger backoff. This catches degradation before failure.
• The Observer interface enables both Prometheus metrics and pluggable logging.
• Stats counters are atomic so callers can read them without locking.
• The shrink ratio (0.5) is configurable; tune for your workload.

Now wire it into a handler:

func handler(w http.ResponseWriter, r *http.Request) {
    release, err := limiter.Acquire(r.Context())
    if err != nil {
        http.Error(w, "busy", 503)
        return
    }
    err = doWork(r.Context())
    release(err)
    if err != nil { /* ... */ }
}

The limit naturally adapts to load. If you trace limiter.Limit() over time, you will see a sawtooth pattern hugging the system's real capacity.

Deep Dive: A Token-Bucket Rate Limiter with Distributed Coordination¶

When you run N replicas, each has its own local token bucket. The aggregate rate is N × per-replica rate, which is rarely what you want. For a true distributed rate limit, you need shared state.

The simplest approach: Redis.

package rate

import (
    "context"
    "errors"
    "time"

    "github.com/redis/go-redis/v9"
)

const luaScript = `
local key = KEYS[1]
local rate = tonumber(ARGV[1])
local cap = tonumber(ARGV[2])
local now = tonumber(ARGV[3])

local tokens = tonumber(redis.call("HGET", key, "tokens") or cap)
local last = tonumber(redis.call("HGET", key, "last") or now)

local elapsed = math.max(0, now - last)
tokens = math.min(cap, tokens + elapsed * rate)
if tokens >= 1 then
    tokens = tokens - 1
    redis.call("HSET", key, "tokens", tokens, "last", now)
    redis.call("EXPIRE", key, 60)
    return 1
end
redis.call("HSET", key, "tokens", tokens, "last", now)
redis.call("EXPIRE", key, 60)
return 0
`

type DistributedLimiter struct {
    client *redis.Client
    rate   float64
    cap    float64
    script *redis.Script
}

func New(client *redis.Client, rate, cap float64) *DistributedLimiter {
    return &DistributedLimiter{
        client: client, rate: rate, cap: cap,
        script: redis.NewScript(luaScript),
    }
}

func (l *DistributedLimiter) Allow(ctx context.Context, key string) (bool, error) {
    res, err := l.script.Run(ctx, l.client, []string{key},
        l.rate, l.cap, float64(time.Now().UnixNano())/1e9).Int64()
    if err != nil { return false, err }
    return res == 1, nil
}

The Lua script ensures atomic check-and-decrement on Redis. Replicas share the bucket; the aggregate rate is constant regardless of replica count.

Latency cost: ~1 ms per Redis call. Acceptable for many use cases; too expensive for very high-RPS paths. For those, use local limiting with synchronized refresh from a central source.

Deep Dive: Distributed Backpressure Across a Microservice Mesh¶

In a microservice architecture, every service is a producer and consumer relative to its neighbours. Backpressure must propagate through every hop.

A ── calls ──► B ── calls ──► C ── reads ──► DB

If DB is slow, C waits. If C waits, B's calls to C take longer. If B waits, A's calls to B take longer. Without backpressure at each hop, the slowness propagates as growing latency, then growing memory, then OOM somewhere.

With backpressure at each hop:

• C admits only what its CPU and DB can handle. Excess gets 503.
• B's call to C respects deadlines. On 503, B returns 503 to A.
• A handles 503 by retrying (with backoff) or returning to its client.
• The client (browser, mobile app) sees a brief error and recovers.

Result: the slow DB causes a short blip of 503s and recoveries, not a cascade.

This pattern is implemented in several ways across the industry:

• Istio / Envoy sidecars apply admission and rate limiting at the network layer. The application is unaware; the proxy returns 503 when overloaded.
• Hystrix-like libraries wrap each downstream call with breaker + bulkhead. Failures are isolated; the application sees clean errors.
• Application-level admission in each service, as described in this page. More control; more code.

The right mix depends on the team and the platform.

Deep Dive: Backpressure in Event-Driven Systems¶

In an event-driven system, services communicate via persistent queues (Kafka, NATS, AWS SQS). The queue is the buffer; the producer's "backpressure" is whether the queue can absorb writes.

For Kafka:

• Producer side: writes are buffered locally (Kafka producer client). When the broker is slow, the local buffer fills, and the next call to Produce blocks or returns an error.
• Consumer side: a slow consumer falls behind. Lag grows. Memory does not (Kafka is on disk).

Monitoring:

• kafka_producer_buffer_bytes: how full the producer's local buffer is. If it grows, the producer is being slowed by the broker.
• kafka_consumer_lag: how far behind the consumer is. The most important metric for event-driven services.

When lag grows persistently, you have a capacity problem. Options:

• Add more consumers (if partitioned).
• Make each consumer faster.
• Drop messages (rare; usually only for telemetry).

The choice is: more compute, faster compute, or less work. There is no other lever.

Idempotency¶

In any at-least-once system, consumers may see duplicates. Idempotent processing is mandatory. The typical pattern:

func (c *Consumer) Process(ctx context.Context, msg *Message) error {
    if seen, err := c.cache.Check(ctx, msg.ID); err != nil || seen {
        return err // skip duplicate
    }
    if err := c.handle(ctx, msg); err != nil { return err }
    return c.cache.Mark(ctx, msg.ID)
}

Backpressure interacts with idempotency: if you drop a message, you must drop it definitively, not partially. The state machine must be visible to operators.

Deep Dive: Designing a Stream-Processing Pipeline¶

A stream pipeline often has multiple stages with different rates. Each stage needs its own admission and shedding. A typical layout:

[ingest] → [Kafka topic] → [enrich worker pool] → [Kafka topic] → [persist] → [DB]

Backpressure manifests as:

• Lag on the first Kafka topic if ingest is slow.
• Lag on the second topic if enrichment is slow.
• DB connection pool waits if persistence is slow.

Each component needs:

• Bounded local memory (no unbounded slices).
• Context-aware operations.
• A way to surface its bottleneck (metric).
• Idempotent processing (since retries and duplicates are likely).

The senior responsibility is to ensure every link in the chain has bounded resources. One unbounded buffer somewhere defeats the whole design.

Deep Dive: Capacity Planning Workshop¶

A worked capacity-planning problem.

Scenario¶

You are building a service that: - Accepts user uploads (max 5 MB each). - Runs image OCR on each. - Stores the result in Postgres.

Targets: - 500 RPS peak. - p99 latency < 500 ms. - 99.9% availability.

Step 1: Measure per-request work¶

A single OCR run on a sample takes: - 30 ms reading body (5 MB at gigabit ≈ 40 ms wire time, optimistically). - 200 ms OCR (CPU-bound). - 10 ms Postgres insert.

Total ≈ 240 ms per request.

Step 2: Per-replica capacity¶

If each replica has 8 cores and OCR is CPU-bound at 1 core per request, throughput per replica = 8 / 0.24 ≈ 33 RPS.

For 500 RPS, need 500/33 ≈ 16 replicas. Add headroom: 20 replicas.

Step 3: Queue sizing¶

Per-replica concurrency = 8 (one per core). For p99 < 500 ms and per-request time 240 ms, queueing budget = 500 - 240 = 260 ms.

Max queue depth = 260 ms × 8 / 240 ms ≈ 8 slots.

So each replica's admission semaphore = 8, queue = 8.

Step 4: DB capacity¶

20 replicas × 8 concurrent = 160 concurrent DB inserts at peak. Postgres handles ~1000 connections; per-replica pool of 10 = 200 connections total. Comfortable.

Step 5: Backpressure¶

If sustained > 33 RPS per replica, queue fills, 503s begin. Client sees 503 — gracefully degrade or retry with backoff.

Step 6: Scaling¶

Add Kubernetes HPA based on queue depth: scale up if depth > 4 sustained for 1 min; scale down if depth < 2 sustained for 5 min.

Step 7: Failure handling¶

Postgres outage: connection acquire times out → 503 to client. OCR engine outage: handler short-circuits to cached error or fallback.

Capacity result¶

20 replicas, 8 cores each, 8 concurrent slots per replica, 8 queue slots per replica. Total fleet capacity = 660 RPS (with 30% headroom for the 500 RPS target). p99 latency budget met. DB has comfortable margin.

This is the kind of analysis a senior should be able to do on a napkin. Adjust as the workload mix or hardware changes.

Deep Dive: Black Friday Survival Guide¶

A retail service has 50× normal traffic during a sale. Generic patterns to survive:

1. Static admission limit set well in advance. Do not introduce new code on the day; you cannot validate it.
2. Aggressive load shedding policies. Better to serve 60% of users perfectly than 100% poorly.
3. Cached responses. Stale catalogue is acceptable; broken catalogue is not.
4. Queue-based ingest for write paths. Accept the order, process later. Customer sees "we got your order"; you process within minutes.
5. Disable expensive features. No personalised recommendations during peak; serve generic.
6. Pre-scale. Add capacity hours in advance, not minutes.
7. Runbook ready. Every alert has an action; every action is tested.

Backpressure is the engine that lets you implement "serve 60% perfectly." Without it, you serve 100% poorly, which usually means none of them complete.

Deep Dive: Postmortems Through a Backpressure Lens¶

When a Go service fails under load, the usual question is "what did we miss?" A backpressure-lens checklist:

• Was every queue bounded? Often the answer is "no, there's an unbounded slice somewhere."
• Did we measure queue depth? If not, we cannot say when the queue filled.
• Did rejections appear in time to act? If 503 only began at 95% queue full, you have no warning margin.
• Did clients have retry-with-backoff? Or did 503 → instant retry → 10× amplification?
• Did downstream see our pressure? Or did we hammer them while shedding ourselves?
• Did shutdown drain? Or did the kill-9 lose in-flight work?

A backpressure-aware service has answers to all of these. A non-aware service tends to have post-mortems that conclude "we need a bigger queue."

Deep Dive: Real-World Code Smells in Senior Code Reviews¶

Even senior engineers ship backpressure mistakes. Common ones to flag:

1. Unbounded retry loops. A retry that has no maximum becomes a slow infinite loop.
2. Retries without idempotency check. Doubled writes; data corruption.
3. No deadline on outbound calls. Stuck calls leak resources.
4. Fan-out without per-target rate limit. One bad target slows the entire fan-out.
5. Goroutine spawned per item without limit. Looks innocent; produces unbounded goroutine count.
6. Drop counter without alert. Drops happen invisibly until the queue is empty for the wrong reason.
7. sync.Pool for items with unbounded size. Pool stays large after a spike; same memory bug as unbounded queue.
8. Heartbeat without timeout. A consumer that never stops trying to reach a dead service.

If you see these in a PR, push back. The fix is usually small but high-leverage.

Deep Dive: Backpressure for Streaming APIs¶

Server-Sent Events (SSE), gRPC streaming, and WebSocket all involve a long-lived connection where the server pushes data to clients over time. Backpressure here is per-connection.

For SSE: HTTP/2 windows apply. If the client is slow, the server's writes block on the network.

For WebSocket: each connection has a write buffer in gorilla/websocket or similar. Writes block when the buffer is full. The server must decide: block (slow the producer), drop (skip messages), or disconnect (the client cannot keep up).

type Client struct {
    conn *websocket.Conn
    send chan []byte
}

func (c *Client) writeLoop() {
    for msg := range c.send {
        c.conn.WriteMessage(websocket.TextMessage, msg)
    }
}

func (c *Client) Push(msg []byte) bool {
    select {
    case c.send <- msg:
        return true
    default:
        return false // client is slow; drop
    }
}

For very slow clients, prefer disconnect. A laggy client tying up memory is worse than a brief reconnect.

Deep Dive: Backpressure in CLI Tools¶

Sometimes backpressure shows up in command-line tools. xargs -P for example caps parallelism; find | head is a producer-consumer where head exits and SIGPIPE stops find. These are operating-system-level backpressure.

In your own Go CLI tools, the same applies: if your tool reads input and processes concurrently, bound the parallelism. A for line := range file { go process(line) } is unbounded — it might OOM on a 10 GB input.

sem := make(chan struct{}, 16)
for line := range file {
    sem <- struct{}{}
    go func(l string) {
        defer func() { <-sem }()
        process(l)
    }(line)
}

This is the same pattern as a server; only the duration is shorter.

Deep Dive: Backpressure and Garbage Collection¶

Go's GC interacts subtly with backpressure. Large in-flight queues mean more heap, more GC pressure, longer GC pauses. The GC pause itself can cause a queue to fill (workers paused).

For latency-sensitive systems:

• Smaller queues → less heap → shorter GC pauses.
• Use runtime/debug.SetGCPercent to tune; default 100 means heap doubles before GC.
• Use GOMEMLIMIT (Go 1.19+) to cap total memory.

A 10 GB heap with a 100 ms GC pause means up to 100 ms × λ items can pile up during GC. The buffer should accommodate this or the system rejects during every GC.

This is one reason "small buffer + adaptive limit" beats "big buffer." Big buffers turn into big heaps turn into big GC pauses turn into more backpressure.

Deep Dive: Backpressure Under Memory Pressure¶

When memory is tight, the right response is to reject harder. New requests should fail fast, not pile up in queues.

var memThreshold uint64 = 4 << 30 // 4 GB

func handler(w http.ResponseWriter, r *http.Request) {
    var ms runtime.MemStats
    runtime.ReadMemStats(&ms)
    if ms.HeapAlloc > memThreshold {
        http.Error(w, "memory pressure", 503)
        return
    }
    // ...
}

Reading MemStats is cheap (single syscall) but has some allocation cost. Cache it (refresh every 100 ms or so) for hot paths.

For Go 1.19+, GOMEMLIMIT makes the GC more aggressive when approaching the limit. Combined with explicit memory-based admission, this provides graceful behaviour under memory pressure rather than OOM kills.

Deep Dive: Backpressure and Multi-Tenancy at Scale¶

Per-tenant queues (covered in middle.md) handle a few tenants. At scale (10K+ tenants), per-tenant queues become a memory problem. Patterns:

1. Shared queue, per-tenant rate limit. Each tenant has a quota; excess is dropped per-tenant.
2. Tiered classes. Group tenants by tier (free, paid, enterprise) with separate pools per tier.
3. Weighted fair queueing. Internally rotate among tenants by weight.

A typical large multi-tenant service has: - Per-tenant rate limit (DDoS protection). - Tier-based pool (resource isolation between tiers). - Global concurrency limit (overall service protection).

Three layers, three different concerns. Each is a kind of backpressure.

Deep Dive: Building a Reactive Service¶

Some services adopt a "reactive" model where every layer pulls from the layer above on demand. This is a different shape of backpressure: instead of pushing from producer to consumer, the consumer pulls.

// Pull-based reader.
type Pull interface {
    Pull(ctx context.Context, n int) ([]Item, error)
}

func (s *Service) ProcessLoop(ctx context.Context, src Pull) {
    for {
        items, err := src.Pull(ctx, 10) // ask for up to 10
        if err != nil || len(items) == 0 { return }
        for _, it := range items {
            process(it)
        }
    }
}

The consumer controls the rate. The producer never sends more than requested. There is no buffer because there is no excess production.

Pure reactive is rare in Go (more common in Java/Scala). But the idea is useful: when designing a system, ask "is the consumer pulling or is the producer pushing?" Pull is harder to mis-design.

Deep Dive: SLO-Driven Backpressure¶

Some services derive admission policy from their SLO. If the SLO is "p99 < 200 ms," the admission limit is whatever lets p99 stay below 200 ms. The system continuously measures p99 and adapts:

type SLODriven struct {
    sloTarget time.Duration
    limit     atomic.Int64
    p99       *p99Estimator
}

func (s *SLODriven) Tick() {
    p99 := s.p99.Get()
    cur := s.limit.Load()
    switch {
    case p99 > s.sloTarget:
        s.limit.Store(int64(float64(cur) * 0.9)) // reduce 10%
    case p99 < s.sloTarget*9/10:
        s.limit.Store(cur + 1)
    }
}

This is the most direct adaptation: aim at the SLO, adjust accordingly. Useful when SLO compliance is the primary metric (consumer-facing services).

Deep Dive: The Cost of Wrong Backpressure¶

A miscalibrated backpressure policy can be worse than none. Examples:

• Buffer too large → memory bloat under spikes → OOM → service down.
• Buffer too small → spurious 503s on normal jitter → SLO miss → unhappy users.
• Drop threshold too aggressive → high reject rate → revenue loss.
• Drop threshold too lenient → slow drift to overload → eventual incident.
• Wrong shrink ratio in AIMD → either over-correction (drops to 1, slow recovery) or under-correction (never escapes overload).
• Missing observability → operators cannot tell whether shedding is happening or normal traffic.

Backpressure is engineering. Engineering means measure, design, deploy, measure, adjust. There is no setting that works without measurement.

Deep Dive: Reading Production Goroutine Dumps¶

When a Go service hangs, send SIGQUIT (or call runtime.Stack). The dump reveals everything.

Backpressure-related patterns to look for:

• Thousands of goroutines in chan send → producer outrunning consumer; queue full.
• Thousands in chan receive → consumer outrunning producer; queue empty (probably normal).
• Goroutines in select with context.Done() → waiting for deadline; usually normal.
• Goroutines in sync.(*Mutex).Lock → lock contention; investigate.
• One goroutine holding a lock + many waiting → the holder is slow or dead.

For huge dumps, use tools like pprof to aggregate by stack:

curl http://localhost:6060/debug/pprof/goroutine?debug=1 > dump.txt
grep -A1 "goroutine" dump.txt | grep "chan send" | wc -l

This counts goroutines stuck on chan send — a direct indicator of backpressure-related blocking.

Deep Dive: Capacity vs Reliability Tradeoffs¶

A senior often has to choose between capacity (handle more) and reliability (handle better). Backpressure forces the choice.

Examples: - Higher concurrency limit → more capacity, more risk of overload. - Larger buffer → smoother handling of jitter, more memory at risk. - Aggressive shed threshold → safer, fewer requests served. - Per-tenant queues → fairer, more memory.

There is no universal answer. The right tradeoff depends on: - Cost of an outage (revenue, reputation, SLA penalty). - Cost of rejections (lost requests, user friction). - Operational maturity (can the team respond to alerts at 3 AM?). - Hardware/capacity headroom (can we scale out quickly?).

Document the chosen tradeoff. Operators and future engineers should understand why a given limit is set where it is.

Deep Dive: Backpressure and Cost¶

Every backpressure decision has a cost dimension. Larger pools cost more in cloud bills. Smaller pools cost more in lost requests. The right answer is the one that minimises total cost over a window.

Simple model:

Cost = compute_cost + revenue_lost_to_rejects + outage_risk

A senior service maintains a measurable estimate of revenue per accepted request and uses it to set thresholds. If a 1% reject rate costs $1000/day in lost revenue and a 1% capacity increase costs $300/day in compute, scaling is cheap; lean on capacity. If a 1% increase costs $5000/day, lean on shedding.

Most teams do this informally. Few do it explicitly. The explicit calculation is a senior-level instrument.

Closing Senior Mantra¶

Backpressure at the senior level is system design. The mechanisms — AIMD, Vegas, token buckets, hedges, circuit breakers — are tools in a larger toolkit. The discipline is: measure capacity, adapt to it, propagate signals across services, document tradeoffs, observe everything, run drills.

A service designed with senior-level backpressure can:

• Run at 90%+ steady-state utilisation without surprises.
• Survive 10×–50× spikes by shedding cleanly.
• Recover from downstream outages without operator action.
• Roll out new features without capacity panic.
• Be operated by a small team because incidents are rare and short.

Professional level will go further: distributed flow control protocols, Kafka consumer group strategies, deep queue theory, multi-tier global rate limits, and the operational practices that keep all of it running in production. The mechanisms grow more sophisticated, but the discipline is constant: the consumer dictates, the producer responds, the system stays honest.

Deep Dive: Worked Real-World Failures and Their Backpressure Diagnosis¶

This section walks through several anonymised production failures and explains them through a backpressure lens.

Failure 1 — The unbounded retry storm¶

A payments service called a downstream fraud-check service. The fraud-check service had an internal bug and started returning 500s for 1% of requests. The payments service used retry-on-failure with no maximum retry count. Each retry-on-500 was retried 10 times within 100 ms.

Effect: the 1% bad responses turned into ~10% repeat traffic. The fraud-check service started seeing 1.1× normal load, which triggered its own slow-mode, which made more 500s, which triggered more retries. Within 90 seconds the fraud-check service was at 5× normal load and almost completely failing.

Diagnosis: missing backpressure in two places. (a) The payments service should have honoured 503 / 5xx with backoff. (b) The fraud-check service had no admission control to reject excess traffic.

Fix: cap retries at 3 with exponential backoff and jitter; add admission control on fraud-check that returns 503 when in-flight exceeds capacity.

Failure 2 — The slow queue consumer¶

An analytics pipeline consumed events from Kafka and wrote them to a data warehouse. The warehouse upgrade reduced write throughput by 30%. The consumer fell behind, lag started growing, but the consumer was healthy — it just could not keep up.

After 3 hours, lag was 30 minutes. After 6 hours, lag was 90 minutes. Stakeholders noticed when end-of-day reports were missing 90 minutes of data.

Diagnosis: backpressure existed (consumer lag), but no alerting was wired to it. The signal was visible but not actionable.

Fix: alert on kafka_consumer_lag > 5 minutes; runbook says "page on-call, investigate downstream slowness or scale consumers."

Failure 3 — The OOM that wasn't a leak¶

A service started OOM-ing every few days. Heap profiles showed no obvious leak. The pattern: memory grew linearly with time, then spiked, then OOM.

Investigation: the service had a chan Event with capacity 1,000,000. Under normal load, only 50 items were ever in the channel. Under peak load (rare), 500,000+ items were sometimes queued. The buffer worked correctly — until the peak coincided with a downstream slowdown, when the channel filled completely and the heap blew up.

Diagnosis: "buffer too large" disguised as "lots of headroom." A buffer that is only used during a multi-failure scenario is the buffer that takes down the system when both failures happen.

Fix: reduce buffer to 10,000; add explicit shed policy when > 90% full; add metrics on channel depth.

Failure 4 — The deploy-time stampede¶

A deployment rolled out a new version of a service. New replicas booted with empty caches; first requests took 5× normal time. The old replicas were drained quickly; new replicas could not absorb the full load. The load balancer kept routing traffic; new replicas fell behind; old ones drained; the brief moment of overload became a full outage.

Diagnosis: deployment caused a temporary capacity crunch. No backpressure mechanism warmed up cold replicas.

Fix: (a) readiness checks gated on cache warmth — a new replica reports ready only after 100 successful test requests. (b) Load balancer warm-up — gradually ramp traffic to new replicas over 30 seconds. (c) AIMD-style adaptive concurrency limit per replica.

Failure 5 — The runaway integration test¶

A CI pipeline that ran integration tests against a staging environment hammered the service with 1000 parallel test workers. The staging service was sized for 50 RPS, not 1000 RPS. Tests started failing; the CI pipeline blamed the service. The service blamed the tests. Both were wrong; the missing piece was backpressure between test runner and service.

Diagnosis: a tester is also a client. Without rate limiting, it can DoS its own service.

Fix: rate-limit test runners; tests fail fast on 503 rather than retry forever.

Deep Dive: Designing Service Mesh Policies¶

When using Envoy or Istio, much backpressure is configured declaratively. Some senior-level patterns:

Local rate limit¶

filters:
  - name: envoy.filters.http.local_ratelimit
    typed_config:
      stat_prefix: http_local
      token_bucket:
        max_tokens: 100
        tokens_per_fill: 100
        fill_interval: 1s
      filter_enabled:
        runtime_key: local_rate_limit_enabled
        default_value:
          numerator: 100
          denominator: HUNDRED

This caps incoming RPS at 100 per pod. Each pod gets its own bucket; aggregate scales with pod count.

Circuit breaker¶

circuit_breakers:
  thresholds:
    - priority: DEFAULT
      max_connections: 1024
      max_pending_requests: 1024
      max_requests: 1024
      max_retries: 3

Limits outbound connections, pending requests, and retries. When exceeded, requests fail fast — they never reach the destination.

Outlier detection¶

When a target replica returns too many 5xx, Envoy ejects it from the load-balancer pool temporarily. This is automatic circuit-breaking at the network layer.

outlier_detection:
  consecutive_5xx: 3
  interval: 30s
  base_ejection_time: 30s
  max_ejection_percent: 50

These features are not unique to Envoy — Linkerd, Consul Connect, and other meshes have similar capabilities. Senior architects choose between "code-level backpressure" and "mesh-level backpressure" based on team capability and platform.

Deep Dive: Bidirectional Streaming and Backpressure¶

In gRPC bidirectional streaming, both sides can send. Backpressure becomes more nuanced:

• The server's stream.Recv() blocks when the client is slow to send.
• The client's stream.Send() blocks when the server is slow to read.
• Internal application queues on both sides need bounded sizes.

A common pattern: a worker pool on each side, with bounded channels mediating between the network read/write and the application logic.

// Server side
func (s *Server) Sync(stream pb.Service_SyncServer) error {
    in := make(chan *pb.Request, 16)
    out := make(chan *pb.Response, 16)

    go func() {
        defer close(in)
        for {
            req, err := stream.Recv()
            if err != nil { return }
            in <- req // blocks if app is slow
        }
    }()

    go func() {
        for resp := range out {
            if err := stream.Send(resp); err != nil { return }
        }
    }()

    for req := range in {
        resp := s.process(req)
        out <- resp // blocks if network is slow
    }
    close(out)
    return nil
}