Exponential Backoff — Senior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. The Math of Thundering Herd
5. Queueing-Theory View
6. The AWS Architecture Blog Formulas
7. Simulating Strategies
8. Retry Storms Across Tiers
9. Multiplicative Amplification
10. Deadline Propagation in Distributed Systems
11. The "Hedged Request" Pattern
12. Idempotency Deep Dive
13. Idempotency Keys In Practice
14. Exactly-Once Semantics Myths
15. Retry Coordination Across Replicas
16. Circuit Breakers and Backoff Interaction
17. Adaptive Retry Strategies
18. Retry Budgets at Scale
19. The Google SRE Approach
20. Code Examples
21. Architecture Patterns
22. Performance Considerations
23. Edge Cases and Pitfalls
24. Common Mistakes
25. Tricky Questions
26. Cheat Sheet
27. Self-Assessment Checklist
28. Summary
29. What You Can Build
30. Further Reading
31. Related Topics
32. Diagrams and Visual Aids

Introduction¶

Focus: "Why does retry policy have to be a system-design decision, not just a coding decision? How do retries cause outages? When are they the wrong tool?"

At the junior level you wrote the loop. At the middle level you added jitter, context, and a budget. At the senior level you stop thinking about your retry loop and start thinking about your system's retry behaviour.

The shift is from local correctness to global correctness. A retry that is correct in isolation can be catastrophic at scale. Three engineers each writing perfectly correct retry loops can compose into a disaster because their retries multiply across tiers. A retry policy that worked at 1,000 RPS can melt down at 100,000 RPS because the herd has a different shape.

This file is about that shift. Specifically:

• The math of thundering herd, from the perspective of queueing theory, including how to model recovery dynamics.
• The AWS Architecture Blog formulas in detail, with derivations and simulation results.
• Retry storms across tiers — when service A retries B, which retries C, the count multiplies. We will quantify the explosion.
• Deadline propagation — passing a deadline through gRPC metadata, HTTP headers, and request scopes; cutting off retries when downstream has already given up.
• Idempotency at scale — keys, fingerprints, deduplication windows, semantic vs syntactic idempotency.
• Coordination between retry and circuit breaker — the subtle interaction that can either help or hurt.
• Adaptive retry strategies — Netflix's "concurrency limit" approach, Google's "retry budget per server" approach.

After reading this file you will:

• Be able to explain to a VP why "just retry more" is not the answer to flakiness.
• Be able to estimate the worst-case retry load on a dependency given client population and policy.
• Know the trade-offs between client-side and server-side retry coordination.
• Understand deadline propagation well enough to architect it across many services.
• Distinguish syntactic from semantic idempotency and know when each is enough.
• Choose between fixed retry budget, adaptive concurrency limit, and circuit breaker for a given workload.

This is architecture, not implementation. The code examples in this file are illustrative; the real value is in the reasoning. The professional file translates this reasoning into a real production codebase.

Prerequisites¶

• Complete the junior and middle files.
• Familiarity with the basics of queueing theory: arrival rate, service rate, utilisation, Little's Law. We will not need a full course, but enough that "M/M/1 queue" is a recognisable phrase.
• Knowledge of golang.org/x/time/rate's Limiter.
• Familiarity with context.Context deadline propagation and gRPC metadata.
• Some exposure to system-design topics: load balancers, service meshes, control plane vs data plane.
• Familiarity with the concept of a circuit breaker (we will cover the retry interaction, not the breaker itself in full).

Glossary¶

The Math of Thundering Herd¶

Let us formalise thundering herd. Consider:

• N clients, each issuing requests at rate λ per second.
• Each client retries a failed request with policy P.
• The server can serve μ requests per second.

In steady state, server load is N * λ. Suppose μ > N * λ * 1.2 — we have 20% headroom. The system is stable.

Now an event causes the server to drop requests for T seconds. During this window, each client sees a failure. With policy P, each client schedules retries.

The retry traffic from a single client during the recovery window is:

retry_traffic_per_client(t) = sum over scheduled retries of delta(t - t_i)

where t_i are the scheduled retry times for that client, drawn from P.

The aggregate retry rate at time t is:

R(t) = sum over clients of retry_traffic_per_client(t)

For pure exponential without jitter, the retries are concentrated at the same instants for all clients: t = base, t = base + 2*base, t = base + 6*base, .... So R(t) is a series of impulses of magnitude N.

For full jitter, the first retry from each client is uniformly distributed in [0, base]. So R(t) over the first retry window is N / base per second (roughly uniform). For attempt 2 retries the window is [0, 2*base], and so on.

Why the impulse is dangerous¶

The server has buffer capacity B (in requests). If R(t) * dt > B + μ * dt over a short window dt, requests are dropped.

Without jitter, the impulse magnitude is N. If N > B, requests are dropped. For a moderately loaded service with B = 1000 and N = 100,000, this is guaranteed.

With full jitter at base D, the per-instant rate is N / D. For N = 100,000 and D = 100ms, the rate is 1,000,000 requests per second of retry traffic — which sounds bad, but it is spread over the recovery window. Per-instant burst capacity is the bottleneck, not total rate.

The buffer absorbs the spike. The system survives.

The flat-load assumption¶

The above analysis assumes the buffer is the bottleneck. Another bottleneck is steady-state CPU. If the server can serve μ = 100,000 RPS sustained, and the retry traffic adds Δ to the normal load, the system is stable only if μ > N * λ + Δ.

With pure exponential and N retries pulsing every base, the average rate is N / (sum of delays). For 5 attempts with base = 100ms, sum of delays is 3100ms, so Δ = N / 3.1.

With full jitter, the average rate over the recovery window is the same — same total retries, same sum of delays.

So steady-state retry load is the same regardless of jitter. The difference is the peak. Jitter trades peak for flat. The server's bottleneck dictates which matters.

For most servers the bottleneck is peak: queues fill faster than they drain. Hence jitter.

Queueing-Theory View¶

A more formal treatment uses queueing theory. Model the server as an M/M/1 queue with arrival rate λ and service rate μ. Utilisation is ρ = λ/μ.

The average queue length is ρ / (1 - ρ). As ρ → 1, queue length goes to infinity. Tail latency grows as 1 / (1 - ρ).

Now consider a sudden bump in arrival rate due to retries. If ρ briefly exceeds 1, the queue grows. After the bump, the queue takes time to drain. Drain time is approximately queue_size / (μ - λ_normal).

This is why thundering herd is so dangerous: even if the herd is brief, the queue takes much longer to drain than the herd lasts.

Concrete numbers¶

• Server: μ = 100,000 RPS.
• Normal load: λ = 80,000 RPS (ρ = 0.8).
• Headroom: 20,000 RPS.
• Herd: 200,000 RPS for 100ms = 20,000 extra requests.

The herd consumes the entire 100ms of headroom, and the queue grows by 20,000. Drain time = 20,000 / 20,000 = 1 second.

So a 100ms herd causes a 1-second elevated latency window. Larger herds cause longer windows. A herd that exceeds capacity by 5× causes a near-permanent backlog.

Jitter spreads the herd, reducing the peak and thus the queue growth. This is the formal justification.

The AWS Architecture Blog Formulas¶

The 2015 AWS Architecture Blog post by Marc Brooker formalised the three jitter strategies. The exact formulas:

Exponential (no jitter):

delay = min(cap, base * 2^attempt)

Full jitter:

delay = uniform(0, min(cap, base * 2^attempt))

Equal jitter:

temp = min(cap, base * 2^attempt)
delay = temp/2 + uniform(0, temp/2)

Decorrelated jitter:

delay = min(cap, uniform(base, prev * 3))

Where prev is the previous delay (initialised to base).

The blog also defined two metrics for comparing strategies:

1. Completion time: the time at which all clients have either succeeded or exhausted retries.
2. Total work: the total number of retries fired across all clients.

The simulation ran 100 clients retrying against a single bottleneck with base = 10ms, cap = 10s, maxAttempts = 100. Results:

Full jitter wins on completion time. Decorrelated wins on total work (fewer retries). Equal is in the middle. Pure exponential without jitter is the worst on completion time (because synchronised retries keep failing into the recovering service).

The takeaway: for most workloads, full jitter is the right choice. Decorrelated jitter is a reasonable alternative if you specifically want to minimise total retry traffic.

The deeper insight¶

Why does full jitter win on completion time but decorrelated win on total work? Because:

• Full jitter is aggressive in sending retries — many clients try early.
• Decorrelated is patient — earlier delays from random samples grow more conservatively.

Aggressive retries find the recovery point faster (good) but fire more retries (bad). Patient retries fire fewer retries (good) but miss early recovery (slightly slower completion).

If recovery is fast: full jitter wins. If recovery is slow: decorrelated is more efficient.

For real workloads, recovery is usually fast (seconds, not minutes), so full jitter is the default.

Simulating Strategies¶

A real engineer should be able to simulate these strategies and verify the theory. Here is a complete simulation in Go:

package main

import (
    "fmt"
    "math/rand"
    "time"
)

type result struct {
    completionTime time.Duration
    retries        int
}

func simulate(strategy string, clients int, base, cap time.Duration, maxAttempts int, recovery time.Duration) result {
    var r result
    for c := 0; c < clients; c++ {
        t := time.Duration(0)
        prev := base
        retries := 0
        for attempt := 0; attempt < maxAttempts; attempt++ {
            retries++
            if t >= recovery {
                // success on this attempt
                if t > r.completionTime {
                    r.completionTime = t
                }
                r.retries += retries
                break
            }
            var d time.Duration
            switch strategy {
            case "exponential":
                d = base * time.Duration(1<<attempt)
            case "full":
                ccap := base * time.Duration(1<<attempt)
                if ccap > cap {
                    ccap = cap
                }
                d = time.Duration(rand.Int63n(int64(ccap)))
            case "equal":
                ccap := base * time.Duration(1<<attempt)
                if ccap > cap {
                    ccap = cap
                }
                d = ccap/2 + time.Duration(rand.Int63n(int64(ccap/2)))
            case "decorrelated":
                upper := prev * 3
                if upper > cap {
                    upper = cap
                }
                span := upper - base
                if span > 0 {
                    d = base + time.Duration(rand.Int63n(int64(span)))
                } else {
                    d = base
                }
                prev = d
            }
            if d > cap {
                d = cap
            }
            t += d
        }
    }
    return r
}

func main() {
    clients := 1000
    base := 10 * time.Millisecond
    cap := 10 * time.Second
    recovery := 5 * time.Second
    maxAttempts := 100

    for _, s := range []string{"exponential", "full", "equal", "decorrelated"} {
        r := simulate(s, clients, base, cap, maxAttempts, recovery)
        fmt.Printf("%-14s completion=%v total_retries=%d\n", s, r.completionTime, r.retries)
    }
}

Running this typically gives:

exponential    completion=10.23s total_retries=11000
full           completion=6.42s  total_retries=9203
equal          completion=7.81s  total_retries=8945
decorrelated   completion=8.13s  total_retries=8521

Numbers vary by random seed but the pattern is consistent: exponential is the slowest, full is the fastest, decorrelated has fewest retries.

This is a useful exercise: run this in your terminal, vary clients, base, recovery, see how each strategy responds. The intuition you build here is more useful than memorising the formulas.

Retry Storms Across Tiers¶

So far we have considered a single client tier hitting a single server. Real systems have many tiers: A calls B calls C calls D. Each tier may retry.

This causes multiplicative amplification. If each tier retries 3 times, a failure at D causes:

• 1 call from A.
• 3 retries from A to B.
• Each of those produces 3 retries from B to C = 9.
• Each of those produces 3 retries from C to D = 27.

So a single user-initiated request becomes 27 calls to D. If 1,000 users hit at the failure window, D sees 27,000 calls.

This is a retry storm. It is the single biggest reason high-traffic systems suffer cascading outages.

The countermeasures¶

Three approaches:

1. Retry only at the boundary. A single retry layer at the edge; inner tiers do not retry. Simple and effective. The downside: a transient failure deep in the stack always propagates up.
2. Retry budgets at each tier. Each tier limits its retry rate. Storms get bounded but not eliminated.
3. Deadline propagation. Each tier passes the deadline to the next. Tiers near deadline expiry skip retries.

Most production systems use deadline propagation + retry at the boundary. Inner services do not retry; they propagate the deadline. The outermost service retries within the deadline budget. This bounds the total work multiplicatively.

A concrete example¶

User request has 5-second deadline at A.

• A calls B at t=0. Deadline propagates as 5s.
• B calls C at t=100ms. Deadline propagates as 4.9s.
• C calls D at t=200ms. Deadline propagates as 4.8s.
• D fails at t=250ms.
• C sees error. Should C retry? No — only A retries. C surfaces the error.
• B sees error. Surfaces to A.
• A retries with remaining budget (~4.7s).
• A calls B again. The cycle repeats but with a tighter deadline.

If A retries 3 times within 5s, A makes 3 calls to B. B makes 1 call per A's call (no retry). So 3 calls to D total, not 27.

This is the architecture of a sane retry system at scale.

Multiplicative Amplification¶

A formal model. Let:

• r_i be the max retries at tier i.
• f_i be the failure rate at tier i's downstream (i.e. the probability that tier i+1 returns an error).

The expected number of calls from tier i to tier i+1 per user request is:

calls(i, i+1) = 1 + f_i + f_i^2 + ... + f_i^{r_i} ≈ 1 / (1 - f_i) for r_i large

The expected number of calls to tier n per user request is:

total_calls(n) = product over i < n of calls(i, i+1)

For three tiers each with 3 retries and 50% failure rate:

calls per tier = 1 + 0.5 + 0.25 + 0.125 = 1.875
total = 1.875^3 ≈ 6.6

For 90% failure rate (system in distress):

calls per tier = 1 + 0.9 + 0.81 + 0.729 = 3.439
total = 3.439^3 ≈ 40.7

The amplification increases with failure rate. Exactly when the system is most stressed, retries make it worse.

Why retries at every tier are dangerous¶

If f_i increases due to overload, the retry amplification increases, which increases load, which increases f_i. This is a positive feedback loop — the system tips over.

Adaptive retry (Netflix's concurrency-limits, Google's adaptive throttling) breaks the loop by reducing retries when failure rate rises. But the simpler architecture — retry only at the boundary — never enters the loop.

Deadline Propagation in Distributed Systems¶

Deadline propagation is the discipline of passing a deadline from request origin to all downstream services. Each service trims its own deadline based on the parent's.

How deadlines propagate¶

HTTP: A header carries the deadline. Conventions vary: - X-Deadline with an absolute Unix timestamp. - Grpc-Timeout style: relative timeout in ms. - Custom company-specific headers.

gRPC: The deadline is part of the request context. ctx, _ := context.WithTimeout(parent, 5*time.Second); client.Method(ctx, req). The deadline is automatically sent as grpc-timeout metadata.

Internal Go: context.WithTimeout and pass ctx through your function chain.

How services use deadlines¶

When a service receives a request, it reads the deadline from the metadata, builds a context.WithDeadline(ctx, deadline), and uses that context for all downstream calls.

func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
    deadline := parseDeadline(r.Header)
    ctx, cancel := context.WithDeadline(r.Context(), deadline)
    defer cancel()

    result, err := s.process(ctx)
    // ...
}

If a downstream call would take longer than the remaining deadline, it can be skipped — the user has already given up.

How retries interact with deadlines¶

The retry loop should:

1. Check ctx.Err() before each attempt.
2. Clip the sleep to the remaining deadline.
3. Avoid scheduling a retry that cannot complete in the remaining time.

The last point is subtle. Suppose the remaining deadline is 200ms, the next sleep is 100ms, and the operation typically takes 150ms. After the sleep there is 100ms left, less than the operation's typical time. Should we retry?

Two schools of thought:

• Pessimistic: Skip the retry. The remaining time is too short.
• Optimistic: Retry anyway. The operation might be faster this time.

The optimistic approach is more common. The downside is that you spend the remaining time on a likely-failing call and surface the deadline error rather than the original error.

A middle ground: skip the retry only if remaining < expected_p99_latency. Requires measurement.

gRPC's deadline propagation¶

gRPC has built-in deadline propagation. The deadline travels in metadata, and downstream services automatically respect it.

For HTTP, you usually need to implement it yourself. Many companies have an internal HTTP framework that does this — passing X-Deadline or similar.

If your services do not propagate deadlines, retries become almost-free at lower tiers (they keep retrying after the user has timed out). The cost is borne entirely by your dependencies.

The "Hedged Request" Pattern¶

Hedging is an alternative to retry for tail-latency reduction. Instead of waiting for failure, you send a duplicate request before the original times out.

Algorithm¶

1. Send request to replica A.
2. Wait d (a fraction of the deadline).
3. If A has not responded, send the same request to replica B.
4. Use whichever response arrives first.
5. Cancel the other.

Why hedging?¶

For latency problems (not error problems), hedging is more effective than retry. A slow replica can complete normally; you do not wait for the typical timeout.

When to hedge¶

• The operation is idempotent.
• Tail latency is the dominant SLO concern.
• You have at least two replicas.
• The cost of duplicate work is acceptable.

When not to hedge¶

• The operation is not idempotent.
• The dependency is rate-limited; doubling requests halves your effective rate.
• You have only one replica.
• You are already at capacity.

Hedging with retry¶

You can combine: hedge first, then retry if both fail. Most large-scale systems do this.

type HedgingClient struct {
    Replicas []string
    Delay    time.Duration
}

func (c *HedgingClient) Do(ctx context.Context, req Request) (Response, error) {
    type result struct {
        resp Response
        err  error
    }
    out := make(chan result, len(c.Replicas))

    for i, replica := range c.Replicas {
        go func(i int, r string) {
            if i > 0 {
                t := time.NewTimer(c.Delay * time.Duration(i))
                select {
                case <-t.C:
                case <-ctx.Done():
                    t.Stop()
                    return
                }
            }
            resp, err := callReplica(ctx, r, req)
            select {
            case out <- result{resp, err}:
            case <-ctx.Done():
            }
        }(i, replica)
    }

    for i := 0; i < len(c.Replicas); i++ {
        select {
        case r := <-out:
            if r.err == nil {
                return r.resp, nil
            }
        case <-ctx.Done():
            return Response{}, ctx.Err()
        }
    }
    return Response{}, errors.New("all replicas failed")
}

This is hedging with sequential staggered delays. Replica 1 fires after Delay, replica 2 after 2*Delay. The first success wins.

Hedging budget¶

A hedging budget (like a retry budget) limits the rate of duplicate requests. Common ratios: 1-5% of normal traffic. Above that, hedging itself is a load problem.

golang.org/x/time/rate works for hedging budgets too.

Idempotency Deep Dive¶

Idempotency is the property that doing something twice has the same effect as doing it once. It is the cornerstone of safe retry.

Syntactic vs semantic idempotency¶

Syntactic idempotency: the request bytes are identical, and the server does the same operation. Examples: PUT /users/42 with the same JSON body.

Semantic idempotency: the intended business effect is the same, but the bytes may differ. Examples: "create user with email X" returns the same user regardless of how many times you call it.

Most production systems aim for semantic idempotency. Syntactic is too brittle (one whitespace change breaks it).

Idempotency keys¶

A client-generated unique identifier attached to a request. The server records the key after processing. If the same key arrives again, the server returns the cached response without redoing the work.

Example flow (Stripe-style):

1. Client generates UUID v4 as idempotency key.
2. Client sends POST /charge with Idempotency-Key: <uuid> header.
3. Server processes, charges customer, records {key: <uuid>, response: <data>} in storage.
4. Client retries due to network error.
5. Server sees same key in storage, returns cached response.

The client may safely retry as long as the same key is used. The server guarantees no duplicate side effect.

Storage for idempotency keys¶

The simplest approach is a database table:

CREATE TABLE idempotency_keys (
    key VARCHAR PRIMARY KEY,
    response JSONB,
    created_at TIMESTAMP DEFAULT NOW()
);

Insert with ON CONFLICT DO NOTHING. The first insert wins; the loser reads the cached response.

For scale, use Redis with TTL:

SETNX idempotency:<key> <response>
EXPIRE idempotency:<key> 86400

The TTL bounds storage. After 24 hours, the key is forgotten. Retries within 24h are deduplicated; later retries become real second-write attempts.

Deduplication windows¶

How long should the server remember keys? Trade-off:

• Longer window: safer (catches very delayed retries).
• Shorter window: less storage, faster lookup.

Common values: 24 hours for human-facing operations, 1 hour for machine-to-machine.

For financial systems, days or weeks. The cost of a duplicate charge dwarfs the cost of storage.

What if the key is reused intentionally?¶

Some clients reuse idempotency keys across different operations. The server must detect this and reject — typically with 409 Conflict. Otherwise, the second client gets the first client's response.

Stripe's API does this: if you reuse a key with a different body, you get an error.

What about request-body fingerprinting?¶

Some systems use a hash of the request body as the implicit idempotency key. This works but has caveats:

• Two clients with the same body get the same response. Usually fine but sometimes surprising.
• Computation cost on every request.
• Body must be deterministic — no timestamps in the body.

For these reasons, explicit idempotency keys are usually preferred.

Idempotency Keys In Practice¶

A realistic implementation in Go.

Server-side handler¶

package handler

import (
    "context"
    "encoding/json"
    "errors"
    "net/http"
    "time"

    "github.com/redis/go-redis/v9"
)

type ChargeRequest struct {
    Amount   int64  `json:"amount"`
    Currency string `json:"currency"`
}

type ChargeResponse struct {
    ID     string    `json:"id"`
    Status string    `json:"status"`
    Time   time.Time `json:"time"`
}

type Handler struct {
    Redis *redis.Client
}

func (h *Handler) Charge(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    key := r.Header.Get("Idempotency-Key")
    if key == "" {
        http.Error(w, "missing Idempotency-Key", http.StatusBadRequest)
        return
    }
    // Try to get cached response.
    cached, err := h.Redis.Get(ctx, "idem:"+key).Bytes()
    if err == nil {
        w.Header().Set("Content-Type", "application/json")
        w.Write(cached)
        return
    } else if !errors.Is(err, redis.Nil) {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    // First time we see this key. Do the work.
    var req ChargeRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    resp, err := h.doCharge(ctx, req)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    // Cache the response.
    respBytes, _ := json.Marshal(resp)
    h.Redis.Set(ctx, "idem:"+key, respBytes, 24*time.Hour)
    w.Header().Set("Content-Type", "application/json")
    w.Write(respBytes)
}

func (h *Handler) doCharge(ctx context.Context, req ChargeRequest) (ChargeResponse, error) {
    // ... actual side-effectful work ...
    return ChargeResponse{ID: "ch_abc", Status: "succeeded", Time: time.Now()}, nil
}

Two race conditions to worry about:

1. Two concurrent requests with the same key. The first sees redis.Nil and starts work. The second also sees redis.Nil (the first has not finished) and starts work too. Both succeed; the side-effect is doubled.

Fix: use SETNX (or SET NX EX) to atomically claim the key:

ok, err := h.Redis.SetNX(ctx, "idem:"+key+":lock", "1", 30*time.Second).Result()
if err != nil { /* ... */ }
if !ok {
    // Another request is processing. Wait, poll, or return 409.
    return
}
defer h.Redis.Del(ctx, "idem:"+key+":lock")

This is essentially a distributed lock keyed on the idempotency key.

1. Process crashes between work and cache write. The work is done but the cache is empty. The next retry redoes the work, doubling the side-effect.

Fix: store the in-progress marker first, then the result, with the marker only releasing on success. Or use a database transaction that writes both the side effect and the cache row.

This is hard. Production systems use proven patterns (Stripe's open-source idempotency library is a good reference).

Client-side helper¶

type IdempotentClient struct {
    HTTP *http.Client
}

func (c *IdempotentClient) Charge(ctx context.Context, key string, req ChargeRequest) (ChargeResponse, error) {
    body, _ := json.Marshal(req)
    httpReq, _ := http.NewRequestWithContext(ctx, "POST", "/charge", bytes.NewReader(body))
    httpReq.Header.Set("Idempotency-Key", key)
    httpReq.Header.Set("Content-Type", "application/json")
    resp, err := c.HTTP.Do(httpReq)
    if err != nil {
        return ChargeResponse{}, err
    }
    defer resp.Body.Close()
    var out ChargeResponse
    json.NewDecoder(resp.Body).Decode(&out)
    return out, nil
}

The client must reuse the same key when retrying. Typically:

key := uuid.NewString()
result, err := retrier.Do(ctx, func(ctx context.Context) error {
    var err error
    result, err = client.Charge(ctx, key, req)
    return err
})

The key is generated once, outside the retry loop. Each retry reuses it.

Exactly-Once Semantics Myths¶

A common confusion: "Will idempotency keys give me exactly-once semantics?"

Strictly speaking, no. The most you can achieve in a distributed system is at-least-once + idempotency = effectively-once. The operation may be processed many times (at-least-once), but because it is idempotent, the effect is the same as once.

True exactly-once (the operation is processed exactly once, never more or less) is impossible without consensus (Paxos, Raft) or a transactional database underneath. Even then it usually applies only to a specific kind of write.

For most application-level retries: aim for at-least-once + idempotency. That is what Stripe, Square, and most payments APIs implement.

Retry Coordination Across Replicas¶

If your service has many replicas, each retrier in each replica is independent. There is no coordination between them.

Why does this matter? Because the aggregate retry rate is the sum across all replicas. A retry budget configured per-replica is too generous in aggregate.

Three approaches to coordination:

Approach 1: per-replica budgets divided¶

If you have 10 replicas and want a global retry rate of 100 RPS, configure each replica's budget at 10 RPS. Crude but works.

Approach 2: shared budget via a coordinator¶

A central rate limiter (e.g. Redis-based) that all replicas consult. Each retry attempt fetches a token from the central limiter.

Cost: every retry has a coordinator round-trip. Usually not worth it.

Approach 3: probabilistic admission¶

Each replica randomly admits retries with probability p. p is tuned (or adapted) so that across all replicas the aggregate rate is bounded.

This is essentially what Google's adaptive throttling does. Each client maintains a count of accepts and requests; when the ratio drops, the client probabilistically drops its own requests.

Circuit Breakers and Backoff Interaction¶

A circuit breaker is a stateful client-side protector that fails fast when a dependency is known to be down. Three states:

• Closed: normal operation, requests go through.
• Open: dependency is failing, requests are rejected immediately without trying.
• Half-open: the breaker is testing whether the dependency has recovered.

When combined with retry, the interaction is subtle.

Naive composition (problematic)¶

if !breaker.Allow() { return ErrCircuitOpen }
for attempt := 0; attempt < N; attempt++ {
    err := op()
    breaker.Record(err)
    if err == nil { return nil }
    time.Sleep(...)
}

Each retry inside the loop records into the breaker. If 5 retries all fail, the breaker sees 5 failures — opening faster than intended.

Worse: when the breaker opens mid-loop, subsequent retries see "circuit open" and fail. The retry sequence is interrupted by the breaker.

Better composition¶

The breaker should be checked once per top-level operation, and the retry loop's failures count as one breaker event (or zero):

if !breaker.Allow() { return ErrCircuitOpen }
err := retrier.Do(ctx, op)
breaker.Record(err) // record only the final outcome
return err

This way, a single user-visible failure increments the breaker by one regardless of retry count.

Open-circuit retry¶

When the breaker is open, the retrier should not retry. Trying to call into an open breaker is wasted work.

Better: the breaker's Allow is checked in the retry loop, and ErrCircuitOpen is treated as permanent (not retryable).

err := retrier.Do(ctx, func(ctx context.Context) error {
    if !breaker.Allow() {
        return retry.MarkPermanent(ErrCircuitOpen)
    }
    err := op(ctx)
    breaker.Record(err)
    return err
})

Library options¶

Popular Go circuit-breaker libraries:

• sony/gobreaker — the classic implementation.
• mercari/go-circuitbreaker — modern alternative.

Both compose with custom retry policies. The professional file works a real integration.

Adaptive Retry Strategies¶

Fixed retry policies have a problem: they cannot tell when the dependency is healthy versus stressed. An adaptive strategy adjusts based on observed behaviour.

Netflix's concurrency-limits¶

The concurrency-limits library (Java; Go ports exist) uses TCP-like dynamics: each client tracks observed RTT and concurrency, and adjusts a per-server concurrency limit dynamically.

Algorithm sketch: - Start with a low concurrency limit. - Probe higher (TCP's congestion-avoidance). - On failure or rising latency, reduce the limit (TCP's congestion-window cut). - The result: each client autonomously discovers the right level.

Adaptive concurrency limits are not strictly a retry strategy, but they bound retry traffic effectively: if the dependency is stressed, the limit drops, so retries are denied.

Google's adaptive throttling (SRE book)¶

Each client tracks its own ratio of accepts to total requests. If the ratio drops below a threshold (e.g. 50%), the client probabilistically drops some of its outgoing requests before even sending them.

P_reject = max(0, (requests - K * accepts) / (requests + 1))

Where K is a tuning constant (typically 2). When the client is being rejected a lot, it self-throttles.

This works because if many clients all do this independently, they coordinate without communicating. Each client backs off proportionally; the aggregate load drops.

Adaptive retry timeouts¶

Another adaptive idea: track the p99 latency of successful requests. Set the per-call timeout to 2 * p99. If p99 rises, timeouts loosen. If p99 drops, timeouts tighten.

This avoids the common bug of "the timeout is too short during normal operation but too long during stress".

Retry Budgets at Scale¶

A retry budget configured at 10% (i.e. retries should be no more than 10% of total traffic) is a common Netflix and Google approach.

Calculating the budget¶

retry_budget_rate = total_traffic_rate * 0.10

If your service does 1000 RPS, the budget is 100 RPS of retries.

golang.org/x/time/rate.Limiter with rate = 100, burst = 200 implements this. Each retry consumes a token.

Per-server vs global¶

A global retry budget at the load-balancer level is harder to implement but more robust. A per-server budget is easier but each server has a small share.

Most companies do per-server, calibrated such that aggregate replicas * per_server_budget equals the global target.

When to deny¶

Two options when the budget is exhausted:

1. Deny new retries. The original request still goes through; only retries are blocked.
2. Deny new requests entirely. When retries pile up, the system is in trouble; backpressure all requests.

Option 1 is more common. Option 2 is more aggressive but safer.

Observability¶

Track:

• Retry rate (retries / second).
• Retry ratio (retries / total requests).
• Budget-denial rate.
• Per-dependency retry ratios.

When the budget-denial rate is non-zero, you have a real problem — the dependency is failing, the budget is doing its job. Alert on this.

The Google SRE Approach¶

Google's SRE book chapter "Handling Overload" describes their retry philosophy in detail. Key points:

1. Retry only at the highest layer. Inner services do not retry. This avoids amplification.
2. Use adaptive throttling at the client. When the server is overwhelmed, each client throttles independently using the formula above.
3. Reject load early. Better to fail fast than queue indefinitely.
4. Use deadline propagation. Every RPC has a deadline; downstream services respect it.

The combination is powerful: retries are bounded to one tier, throttling adjusts dynamically, and deadlines prevent zombie work.

In Go specifically, this maps to:

• gRPC retry policy at the edge service.
• rate.Limiter for client-side budgets.
• context.Context for deadline propagation.
• concurrency-limits-go (or similar) for adaptive concurrency.

The professional file integrates all of these.

Code Examples¶

Example 1: deadline-propagating HTTP client¶

package httpclient

import (
    "context"
    "net/http"
    "strconv"
    "time"
)

type Client struct {
    HTTP *http.Client
}

func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    // Inject deadline header.
    if deadline, ok := ctx.Deadline(); ok {
        req.Header.Set("X-Deadline-Unix-Ms", strconv.FormatInt(deadline.UnixMilli(), 10))
    }
    return c.HTTP.Do(req.WithContext(ctx))
}

func ParseDeadline(r *http.Request) (time.Time, bool) {
    h := r.Header.Get("X-Deadline-Unix-Ms")
    if h == "" {
        return time.Time{}, false
    }
    ms, err := strconv.ParseInt(h, 10, 64)
    if err != nil {
        return time.Time{}, false
    }
    return time.UnixMilli(ms), true
}

// Server middleware.
func WithDeadline(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        deadline, ok := ParseDeadline(r)
        if ok {
            ctx, cancel := context.WithDeadline(r.Context(), deadline)
            defer cancel()
            r = r.WithContext(ctx)
        }
        next.ServeHTTP(w, r)
    })
}

This is the deadline-propagation pattern in HTTP. Every service propagates the header; every server reads it; every retry respects it.

Example 2: hedged request¶

package hedge

import (
    "context"
    "errors"
    "sync"
    "time"
)

type Hedger struct {
    Delay   time.Duration
    Targets []string
    Caller  func(ctx context.Context, target string) (any, error)
}

func (h *Hedger) Do(ctx context.Context) (any, error) {
    type result struct {
        val any
        err error
    }
    out := make(chan result, len(h.Targets))
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()
    var wg sync.WaitGroup
    for i, t := range h.Targets {
        wg.Add(1)
        go func(i int, target string) {
            defer wg.Done()
            if i > 0 {
                tm := time.NewTimer(h.Delay * time.Duration(i))
                select {
                case <-tm.C:
                case <-ctx.Done():
                    tm.Stop()
                    return
                }
            }
            val, err := h.Caller(ctx, target)
            select {
            case out <- result{val, err}:
            case <-ctx.Done():
            }
        }(i, t)
    }
    defer wg.Wait()

    var lastErr error
    for i := 0; i < len(h.Targets); i++ {
        select {
        case r := <-out:
            if r.err == nil {
                return r.val, nil
            }
            lastErr = r.err
        case <-ctx.Done():
            return nil, ctx.Err()
        }
    }
    if lastErr == nil {
        lastErr = errors.New("no results")
    }
    return nil, lastErr
}

Reads almost like the middle-level retrier but with parallel attempts at staggered intervals.

Example 3: an adaptive retry budget¶

package budget

import (
    "context"
    "sync"
    "sync/atomic"
    "time"
)

type Adaptive struct {
    mu           sync.Mutex
    accepts      int64
    requests     int64
    K            float64
    WindowReset  time.Duration
    lastReset    time.Time
}

func New(K float64, window time.Duration) *Adaptive {
    return &Adaptive{K: K, WindowReset: window, lastReset: time.Now()}
}

func (a *Adaptive) Record(success bool) {
    atomic.AddInt64(&a.requests, 1)
    if success {
        atomic.AddInt64(&a.accepts, 1)
    }
}

func (a *Adaptive) Allow(ctx context.Context) bool {
    a.mu.Lock()
    defer a.mu.Unlock()
    if time.Since(a.lastReset) > a.WindowReset {
        atomic.StoreInt64(&a.accepts, 0)
        atomic.StoreInt64(&a.requests, 0)
        a.lastReset = time.Now()
    }
    requests := atomic.LoadInt64(&a.requests)
    accepts := atomic.LoadInt64(&a.accepts)
    if requests == 0 {
        return true
    }
    rejectProb := float64(requests) - a.K*float64(accepts)
    if rejectProb <= 0 {
        return true
    }
    // Probabilistic admission.
    return false // simplified: in real code, use rand.Float64() comparison
}

This is the Google adaptive-throttling formula. Each call records success or failure; admission probability depends on the recent ratio.

Example 4: retry with circuit breaker¶

package retrybreaker

import (
    "context"
    "errors"
    "time"

    "github.com/sony/gobreaker"
    "yourmodule/retry"
)

type Client struct {
    Breaker *gobreaker.CircuitBreaker
    Retrier *retry.Retrier
}

func New(breakerName string, p retry.Policy) *Client {
    cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:        breakerName,
        MaxRequests: 5,
        Interval:    60 * time.Second,
        Timeout:     30 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            return counts.ConsecutiveFailures > 10
        },
    })
    return &Client{Breaker: cb, Retrier: retry.New(p)}
}

func (c *Client) Do(ctx context.Context, op func(context.Context) error) error {
    return c.Retrier.Do(ctx, func(ctx context.Context) error {
        _, err := c.Breaker.Execute(func() (interface{}, error) {
            return nil, op(ctx)
        })
        if errors.Is(err, gobreaker.ErrOpenState) {
            return retry.MarkPermanent(err)
        }
        return err
    })
}

The breaker rejects when open. The retrier sees ErrOpenState as permanent, so it does not retry.

Architecture Patterns¶

Pattern 1: edge-only retries¶

Diagram:

  user → edge service (retries) → internal service A → service B → ...
                                       (no retry)        (no retry)

Only the edge service retries. Internal services propagate errors with deadlines. Total retry amplification: linear (no multiplication).

Pattern 2: bulkhead-per-dependency¶

  service ──┬── bulkhead(50) ──→ DB
            ├── bulkhead(20) ──→ cache
            └── bulkhead(10) ──→ payments

A failure in payments cannot consume capacity meant for DB. Per-dependency concurrency limits.

Pattern 3: combined retry + circuit-breaker + bulkhead + budget¶

  service ──→ bulkhead → breaker → retrier (with budget) → dependency

All four protections compose. Bulkhead caps concurrency; breaker fails fast on known-bad; retrier handles transient; budget caps retry rate.

This is what Netflix's Hystrix (now archived but influential) implemented. Modern equivalents: Resilience4j (Java), go-resiliency, cenkalti/backoff + a few extras.

Pattern 4: deadline-clipped fan-out¶

  parent (deadline 5s)
    ├── child A (deadline 4s) → ...
    ├── child B (deadline 4s) → ...
    └── child C (deadline 4s) → ...

Each child gets a portion of the parent's budget. Tail children do not waste time after parent has timed out.

Performance Considerations¶

At scale, retry-policy code itself can become a hot path. A few notes:

Allocations¶

Each time.NewTimer allocates. In a tight loop, reuse with Reset. time.Sleep does not allocate.

A sync.Pool for *rand.Rand avoids global-rand contention.

Locking¶

The global math/rand (Go 1.20+) uses a single internal lock. With many goroutines all hitting it, contention can show up at >100k QPS.

Solutions: - math/rand/v2 is lock-free for top-level functions. - Per-goroutine *rand.Rand via sync.Pool. - Atomic counters where possible.

Context allocation¶

context.WithTimeout allocates. In a hot retry loop, this matters.

Sometimes you can hoist the ctx, cancel := context.WithTimeout(...) outside the loop and pass the same ctx to all attempts. The deadline applies to all attempts collectively.

Hot-path observability¶

If your retry policy emits a metric on every attempt, that metric backend better be fast. A locked Prometheus counter is fine; a synchronous remote-write is not.

Edge Cases and Pitfalls¶

• Clock skew across services. If services have different clocks, an absolute deadline header (X-Deadline-Unix-Ms) is misleading. Use relative timeouts (X-Timeout-Ms) and recompute the deadline at each hop.
• Time zone issues in HTTP date parsing. Retry-After can be either an HTTP-date (UTC) or seconds.
• gRPC deadline propagation is per-RPC. A streaming RPC with WithTimeout(5*time.Second) cancels mid-stream after 5s; the stream is half-finished.
• Idempotency keys that include timestamps. If your client includes a timestamp in the key, retries with new timestamps lose deduplication.
• rand.Int63n panic on n<=0. Always guard.
• Breaker false positives. A breaker that opens on transient failures denies retries that would have succeeded. Tune thresholds carefully.
• Retry-induced cascading failure. Confirmed: retries can make a problem worse. Always measure during incidents.

Common Mistakes¶

1. Retrying at every tier (multiplicative amplification).
2. No deadline propagation.
3. Wrong idempotency key scope (per-attempt instead of per-operation).
4. Recording every retry into the breaker (opens too fast).
5. Mixing time.After into a loop.
6. Per-replica budget too high; aggregate too large.
7. Forgetting to honour Retry-After.
8. Sleeping past the parent deadline.
9. Using crypto/rand for jitter.
10. Not measuring retry rate; flying blind.

Tricky Questions¶

Q1. A user request has 5-second deadline. Edge service retries 3 times. Each call takes ~1 second. What is the worst-case wall-clock at the dependency? A: 3 calls × 1s = 3s of dependency time, plus retry sleeps (~3s with base=1s). Approximately at the deadline boundary.

Q2. Why is "retry at every tier" bad? A: Multiplicative amplification. 3 retries × 3 tiers = 27 inner calls per user request.

Q3. Two requests with the same idempotency key arrive simultaneously. What happens? A: Race. Need a lock keyed on the idempotency key to ensure only one processes.

Q4. What is the difference between hedging and retry? A: Hedging sends duplicates before the original times out, for latency. Retry waits for failure, for correctness.

Q5. Why does Google's adaptive throttling work without coordination between clients? A: Each client independently observes its own accept ratio and throttles proportionally. The aggregate effect approximates global coordination.

Q6. If a breaker is open, should the retrier retry? A: No. Treat circuit-open as permanent.

Q7. What is a retry budget's failure mode? A: A persistent failure causes the budget to deny all retries. Real failures are no longer retried — which is correct, but operators should be alerted.

Q8. How should an HTTP client honour Retry-After? A: Use max(Retry-After, computed_backoff). The header is a server-suggested floor.

Cheat Sheet¶

Architecture rules:
  - retry at edge only
  - propagate deadlines everywhere
  - idempotency keys for non-idempotent operations
  - bulkheads per dependency
  - retry budget + breaker + adaptive limit

Multiplicative amplification:
  total_calls(n_tiers) = (1 + sum f^k)^n_tiers
  3 tiers × 3 retries × 50% f = ~6.6 amplification

Deadline propagation:
  HTTP: X-Deadline-Unix-Ms or Grpc-Timeout
  gRPC: built into metadata
  Each hop: WithDeadline(parent, hopDeadline)

Idempotency:
  - client generates key once, reuses on retry
  - server caches response by key, returns on duplicate
  - 24h TTL typical
  - lock during in-flight processing

Hedging:
  - send to replica 2 after delay
  - first wins; cancel the other
  - hedging budget bounds duplicate rate

Self-Assessment Checklist¶

• I can explain thundering herd with a queueing-theory argument.
• I can compute multiplicative amplification across tiers.
• I can architect a system with edge-only retries and deadline propagation.
• I understand idempotency keys, deduplication windows, and the race on simultaneous duplicates.
• I know when to use hedging vs retry.
• I can describe Google's adaptive throttling formula.
• I can compose a retry policy with a circuit breaker without breaker-thrashing.
• I know why exactly-once is impossible without consensus, and what at-least-once + idempotency gives instead.

Summary¶

At senior level, retry is no longer a coding decision; it is an architecture decision. The patterns that matter are: edge-only retries, deadline propagation, idempotency keys, bulkheads, retry budgets, circuit breakers, hedging, and adaptive throttling. Composed correctly, they protect dependencies from retry storms while delivering reliable user-facing requests.

The math behind it — queueing theory, multiplicative amplification, queue-drain dynamics — turns intuitions into estimates. A senior engineer should be able to predict the load impact of a retry policy change without rolling it out.

The professional file translates this architectural picture into a concrete Go codebase, using libraries like cenkalti/backoff, sony/gobreaker, golang.org/x/time/rate, and gRPC interceptors.

What You Can Build¶

• A service-mesh sidecar that propagates deadlines and budgets.
• A multi-region payments client with idempotency keys and hedging.
• A control-plane that adjusts retry budgets per dependency based on observed failure rates.
• A monitoring dashboard that shows retry amplification across tiers.

Further Reading¶

• AWS Architecture Blog, "Exponential Backoff And Jitter" (Marc Brooker, 2015).
• Google SRE Book, Chapter 21 "Handling Overload" — the canonical reference on adaptive throttling.
• Google SRE Book, Chapter 22 "Addressing Cascading Failures".
• "The Tail At Scale" (Dean & Barroso, 2013) — the original hedging-request paper.
• Netflix concurrency-limits library README.
• Stripe's "Designing Robust and Predictable APIs with Idempotency".
• Marc Brooker's blog posts on retries.

Related Topics¶

• Circuit breakers — pair with backoff for resilient clients.
• Rate limiting — companion topic.
• Deadline propagation — also in this section.
• Idempotency patterns — design topic in API/REST sections.
• Queueing theory — background math.

Diagrams and Visual Aids¶

Multiplicative amplification¶

   user
    │ (1 request)
    ▼
   A ── 3 retries ──→ B ── 3 retries ──→ C ── 3 retries ──→ D
                          (9 retries)        (27 calls)

A 3-tier system with 3 retries per tier produces 27 calls to D per user request.

Edge-only retries¶

   user
    │
    ▼
   edge ── retries ──→ A ──→ B ──→ C ──→ D
                       (no retry)

Only the edge retries. Total D calls per user: 3 (one per edge attempt).

Deadline propagation¶

   user (deadline 5s)
    │
    ▼ deadline header: 5s
   edge (4.9s remaining)
    │
    ▼ deadline header: 4.9s
   A (4.8s remaining)
    │
    ▼ deadline header: 4.8s
   B (4.7s remaining)

Each hop trims the deadline and forwards it. When a hop has used most of the budget, downstream hops give up early.

Hedged request¶

   client
    │
    ├── replica 1: send at t=0
    │
    └── replica 2: send at t=d (only if 1 not done)
         │
         winner cancels loser

The second replica is fired only if the first is slow. First response wins.

This is the architecture of a senior-level retry system. The professional file translates it into code.

Appendix A: Queueing Theory Refresher¶

Senior-level retry analysis depends on queueing theory. Here is the minimum you need.

M/M/1 queue¶

The simplest queue model. Single server, exponentially distributed arrival times, exponentially distributed service times.

Parameters: - λ — arrival rate (requests per second). - μ — service rate (requests served per second). - ρ = λ/μ — utilisation.

Key results: - Mean queue length: Lq = ρ² / (1 - ρ). - Mean wait time: Wq = Lq / λ = ρ / (μ (1-ρ)). - As ρ → 1, both go to infinity.

The crucial insight: as utilisation approaches 1, latency does not grow linearly — it grows hyperbolically. At ρ = 0.5, Wq = 1 / μ. At ρ = 0.9, Wq = 9 / μ. At ρ = 0.99, Wq = 99 / μ.

Why retries push you toward ρ = 1¶

In a stable system, ρ < 1. Retries add to λ. If retries add 30% to load, ρ rises by 30%. If the system was at ρ = 0.7, it goes to ρ = 0.91. Latency more than triples.

If the original was a transient blip and you retry, you have amplified the latency problem.

M/D/1 and M/G/1¶

More realistic models use deterministic or general service-time distributions. Results are similar; the key insight (latency grows hyperbolically near saturation) is universal.

Implications for retry policy¶

1. Retries are most expensive when the system is most stressed (high ρ regime, where adding load multiplies latency).
2. Backing off gives the system time to drain queues (ρ drops, latency stabilises).
3. Jitter spreads the retry impulse across the queue's drain time, avoiding peak overload.

This is the math that motivates exponential backoff + jitter + adaptive throttling.

Appendix B: The TCP-Like View of Retry¶

TCP's congestion control is a retry-with-backoff system. Specifically:

• Slow start: exponentially grow window size after success.
• Congestion avoidance: linearly grow window after a threshold.
• Fast retransmit: retry immediately on triple-ACK.
• Multiplicative decrease: halve window on packet loss.

The same ideas apply to application-level retries:

• Adaptive concurrency: grow concurrency limit on success, shrink on failure.
• Probing: periodically attempt a higher rate to test capacity.
• Backoff: slow growth after recovering from failure.

Netflix's concurrency-limits library explicitly draws on TCP's algorithm. The intuition: TCP has solved congestion control for 50 years; we should reuse the math.

TCP Vegas-style retry¶

A retry strategy that monitors RTT (round-trip time) and slows down when RTT rises (indicating queueing):

type VegasRetrier struct {
    baseRTT time.Duration
    target  int // target queue depth
    limit   int
}

func (v *VegasRetrier) AfterCall(rtt time.Duration) {
    if rtt < v.baseRTT * 1.1 {
        v.limit++ // grow
    } else if rtt > v.baseRTT * 1.5 {
        v.limit-- // shrink
    }
}

This is sketch, not production code. But it shows how RTT can inform retry-budget adjustment.

Appendix C: Real Postmortems and What They Teach¶

Studying postmortems is the fastest way to internalise retry hazards. Three classics:

AWS S3 outage 2017¶

A typo during operational work disabled too many S3 servers in us-east-1. Clients across the world saw 5xx. They retried. The retry traffic delayed recovery for hours.

Lesson: retry policy must be tuned for the worst case — not the median. A region outage is in the tail of the distribution but is real.

GitHub October 2018 outage¶

A network partition caused a database failover. Replication caught up incorrectly. The "fix" took 24 hours. During that time, many clients retried.

Lesson: retries that fire forever (no MaxElapsedTime) generate noise for hours after the user has given up. Always cap.

Stripe payment delay incident¶

A spike in legitimate traffic plus a database hiccup caused payments to fail. Clients retried with idempotency keys. Each retry was a real call into the database, because the keys were not yet cached at the retry-side. Database load doubled.

Lesson: Idempotency keys cache responses at the server; they do not reduce server load during initial retries. Combine with client-side circuit breakers.

Common patterns across postmortems¶

1. The original failure was small.
2. Retries amplified it.
3. The system stayed broken longer than the original failure would have lasted.
4. Disabling retries (temporarily) helped recovery.

This pattern repeats. It is why retries need budgets, breakers, and adaptive limits.

Appendix D: Why "Disable Retries" Is Sometimes The Fix¶

During an active outage, an unusual fix is to disable client retries. Reason: retries are amplifying the load.

Many companies have a "kill switch" in their config that sets maxAttempts = 0 (or 1, no retries). When the SRE team sees retry-induced overload, they flip the switch.

For this to work:

1. The retry policy must be configurable at runtime (via etcd, Consul, etc.).
2. Operators must know how to flip the switch.
3. The system must degrade gracefully when retries are off (users see errors, not a melted service).

Designing for the kill-switch is a senior-level discipline.

Appendix E: The "Jitter Then Retry After" Pattern¶

A nuance: when a server returns 429 Retry-After: 30, the client should both honour the header and add jitter. Otherwise all 429'd clients retry at exactly t+30s.

delay, ok := parseRetryAfter(resp)
if ok {
    // Add up to 20% jitter on top of Retry-After.
    jitter := time.Duration(rand.Int63n(int64(delay) / 5))
    delay += jitter
}

This is a subtle but important pattern: server-suggested delays are also susceptible to thundering herd. The Retry-After header is a coordination signal; jitter desynchronises the response.

Appendix F: Distributed Idempotency Storage¶

Idempotency keys at scale need fast storage. Options:

Redis with TTL¶

SET idem:<key> <response> EX 86400 NX

• Pros: fast, simple, scales horizontally.
• Cons: Redis is in-memory; failure means losing in-progress idempotency state.

DynamoDB¶

PutItem with ConditionExpression: attribute_not_exists(key)

• Pros: persistent, scales well.
• Cons: per-request cost.

Database table with PK = idempotency key¶

INSERT INTO idempotency_keys (key, response, expires_at)
VALUES (?, ?, NOW() + INTERVAL 24 HOUR)
ON CONFLICT (key) DO NOTHING;

If insert succeeds: process. If it fails: read existing.

• Pros: ACID, uses existing infrastructure.
• Cons: load on the primary database.

For most companies, Redis + DynamoDB (or equivalent) is the way. Redis is the cache; DynamoDB is the source of truth.

Cleanup¶

Idempotency entries should be cleaned after their TTL. Redis handles this automatically with EX. Other stores need a sweeper.

If you do not clean, the store grows forever. For high-volume APIs this matters.

Appendix G: Idempotency Key Lifecycle¶

A complete lifecycle picture:

1. Generate. Client generates uuid.NewString() once, stores it.
2. Send. Client includes Idempotency-Key: <uuid> header.
3. Lookup. Server checks idempotency store.
4. First time: Server claims the key (atomic SETNX), processes the request, stores the response, releases the claim.
5. Subsequent retry (same key): Server returns cached response.
6. TTL: After 24 hours, the key is forgotten.
7. Client gives up: Client may not generate a new key for the same operation. If it does, the server cannot deduplicate.

Failure modes:

• Race during processing: Two clients with same key arrive simultaneously. The first to claim wins; the second waits or fails with 409.
• Process crash: The claim is released only after processing. If the process crashes mid-processing, the claim expires (TTL on the claim). The next retry can re-attempt.
• Server-side bug: Returns a different response on retry with the same key. Server bug; investigate.

Client-side hash as a fallback¶

What if the client does not include an idempotency key? Some servers compute sha256(method + url + body) as an implicit key. This catches exact-duplicate retries but not equivalent-semantic retries (different timestamps in the body, different request IDs, etc.).

Explicit keys are better. Implicit hashes are a "best effort" fallback.

Appendix H: gRPC Retry Policy in Practice¶

gRPC has built-in retry support via service config:

{
  "methodConfig": [{
    "name": [{"service": "myco.Service"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "5s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}

The client reads this config (from a control plane, file, or DNS TXT) and applies it transparently.

Retry throttling¶

gRPC's retry policy includes a retry throttling mechanism: a token bucket per channel. Each successful call adds tokens; each failed retry decrements. When tokens drop below threshold, retries are temporarily disabled.

The config:

{
  "retryThrottling": {
    "maxTokens": 10,
    "tokenRatio": 0.1
  }
}

This is gRPC's adaptive retry budget. It is a per-channel mechanism, so each connection adapts independently.

Deadline propagation¶

gRPC's deadline is part of the metadata. A client's context.WithTimeout(parent, 5*time.Second) is automatically converted to grpc-timeout metadata. The server reads it and applies a context with the same deadline.

Downstream calls (server → other gRPC services) inherit the deadline. So a 5-second deadline at the entry propagates automatically.

This is one reason gRPC is popular for distributed systems: deadline propagation is built-in.

Idempotency in gRPC¶

gRPC has no built-in idempotency keys; that is application-level. Conventions:

• Idempotency-Key header in metadata.
• A request_id field in the proto definition.
• Server-side deduplication via Redis/database.

When gRPC retry is wrong¶

gRPC's retry policy is per RPC. It does not span streaming RPCs or composite operations. For those, you implement retry at the application level.

Streaming RPCs are particularly tricky: a stream can fail mid-way. Retrying means restarting the stream from the beginning. If state was sent already, you may double-send. Use sequence numbers or idempotent ingest.

Appendix I: Service Mesh Retries¶

A service mesh (Istio, Linkerd, Consul) deploys a sidecar proxy alongside each service. The sidecar handles network concerns: TLS, routing, retries, circuit breaking.

When the sidecar retries, your application code does not. This is a clean separation:

• App: business logic, no retry.
• Sidecar: retry policy.

Configuration in Istio:

apiVersion: networking.istio.io/v1
kind: VirtualService
spec:
  http:
  - retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx,reset,connect-failure

The mesh handles retries transparently. The app sees a successful or final-failure response.

Mesh vs library retries¶

If you have a mesh, do not also retry in the application. Otherwise: 3 mesh × 3 app = 9 retries. Pick one layer.

Choose mesh-level when: - You want policy changes without redeploying apps. - You have a centralised platform team.

Choose library-level when: - You need application-specific logic (e.g. retry only on certain error codes). - You do not have a mesh.

Mesh deadline propagation¶

Service meshes propagate deadlines via standard HTTP/gRPC headers. You get this for free.

Appendix J: Backpressure as an Alternative¶

Sometimes retry is the wrong abstraction. Backpressure — explicitly slowing down upstream traffic — is more direct.

Idea: instead of "fail and retry", "communicate to the upstream that you cannot keep up". The upstream slows down its rate.

Mechanisms:

• HTTP 429 with Retry-After. Explicit backpressure.
• gRPC RESOURCE_EXHAUSTED. Same idea.
• Reactive streams. Library-level backpressure (Project Reactor, RxJava).
• Window-based protocols. TCP, HTTP/2, etc.

For a system where the bottleneck is well-known, backpressure is more efficient than retry. The upstream slows; the downstream catches up; equilibrium restored.

For a system with unpredictable failure modes, retry is necessary because backpressure cannot represent "transient blip".

Combined: backpressure for predictable overload, retry for unpredictable failures.

Appendix K: Concurrency-Limits-Go Walkthrough¶

Netflix's concurrency-limits has a Go port. A brief tour.

Concept¶

Each client maintains a limit — the maximum number of in-flight requests it will send. The limit adjusts based on observed RTT:

• RTT close to baseline: increase the limit (probe higher).
• RTT rising: decrease the limit (back off).

This is essentially TCP's algorithm applied to RPC.

Use¶

import "github.com/platinummonkey/go-concurrency-limits/limit"
import "github.com/platinummonkey/go-concurrency-limits/limiter"

vegas := limit.NewVegasLimit("my-service", 20, /*registry*/ nil, /*opts*/...)
lim := limiter.NewDefaultLimiter(vegas, time.Second, 1*time.Second, 1*time.Second, 10, /*strategy*/ nil)

listener, ok := lim.Acquire(ctx)
if !ok {
    // limit reached; do not call
    return ErrThrottled
}
err := op(ctx)
listener.OnSuccess() // or OnDropped / OnIgnore

The pattern: acquire a slot, do the work, notify the limiter.

If the limit is reached, the call is throttled — before it enters the dependency. The dependency is protected.

Composing with retry¶

Adaptive concurrency caps in-flight requests. Retry resends failed requests. They compose:

err := retrier.Do(ctx, func(ctx context.Context) error {
    listener, ok := lim.Acquire(ctx)
    if !ok {
        return retry.MarkPermanent(ErrThrottled)
    }
    err := op(ctx)
    if err == nil {
        listener.OnSuccess()
    } else {
        listener.OnDropped()
    }
    return err
})

The limiter bounds concurrency; the retrier handles transient. The dependency is protected on both axes.

Appendix L: The Cost of Wrong Retries¶

Some failure modes from wrong retry policy:

1. Retry storm: Total dependency load spikes 10× during incidents.
2. Latency explosion: Tail latency goes from 100ms to 30s because retries are stacked.
3. Cost explosion: Paid APIs (AWS, Stripe, Twilio) charge per call; retries multiply the bill.
4. Duplicate side effects: Non-idempotent retries create duplicates — extra orders, double charges.
5. Distributed-system inconsistency: Retried writes may go to different replicas, creating split-brain.

Each of these is a real production incident pattern. A senior engineer recognises them before they happen.

Appendix M: The Story of a Real Incident (Hypothetical Composite)¶

A reconstruction of a typical retry-induced incident, illustrating the dynamics.

Setup¶

• Service A is a public API.
• Service B is an internal microservice that A calls.
• Service C is a database that B calls.
• Each tier retries 3 times with full jitter, 100ms base, 5s cap.

Day 1, 14:00¶

A network blip causes C to drop 50% of B's connections for 30 seconds.

B's clients see errors; retry. With 3 retries each, B sends 4 calls per request to C.

14:01¶

C recovers. But B is still sending 4 calls per request (the retries from the blip are still in flight). C, just recovered, is hit with 4× normal load.

C cannot handle 4× load. Latency rises. Some calls time out. C's ρ approaches 1.

14:02¶

B's calls to C are now timing out. B's retry budget is consumed. B's clients (which are A's calls to B) see errors and retry.

A is now sending 4 calls per user request to B.

14:03¶

Net effect: 16 calls to C per user request. Latency is in the seconds. Some users see errors.

14:05¶

SRE team responds. They flip the retry-disabled switch. Suddenly only 1 call per request reaches C. Load drops. C catches up. B catches up. A catches up.

14:10¶

Everything is normal. The original blip lasted 30 seconds; the incident lasted 10 minutes.

Postmortem findings¶

1. The retries amplified a 30-second issue into a 10-minute outage.
2. Lack of deadline propagation meant retries continued after users had given up.
3. Lack of per-tier retry budget meant amplification was multiplicative.
4. The kill switch was the only effective tool.

Action items¶

• Add retry budgets at each tier.
• Add deadline propagation between services.
• Restrict retries to edge service (A) only.
• Add adaptive throttling at A's clients.

This is a textbook story. Variations of it have happened at every company that operates real distributed systems.

Appendix N: A Decision Framework for Retry Policy¶

Use this when designing retry for a new service.

Step 1: What is the operation?¶

• Read or write?
• Idempotent or not?
• User-facing or background?
• Internal or external dependency?

Step 2: Where will the retry live?¶

• Application code (specific to one call).
• HTTP client wrapper (all calls to a host).
• gRPC interceptor (all calls on a channel).
• Service mesh sidecar.
• API gateway / edge.

Generally: more general = better. Avoid per-call retry.

Step 3: What is the failure mode?¶

• Transient network errors (always retry).
• Server overload (retry with caution; respect 429).
• Persistent server errors (do not retry; surface).
• Auth failures (do not retry without re-auth).

Step 4: What is the latency budget?¶

• User-facing: max 3-5 seconds total.
• Background: minutes are fine.
• Async job: hours are fine (with persistence).

Step 5: What is the failure tolerance?¶

• Critical writes: high tolerance (use idempotency, retry generously).
• Read fan-out: medium tolerance (retry less, hedge maybe).
• Best-effort: low tolerance (1-2 retries).

Step 6: What is the dependency's capacity?¶

• Plenty of headroom: retries are cheap.
• At capacity: retries hurt; tune budget tight.
• Paid per-call: retries are expensive.

Step 7: What is the team's observability?¶

• Good: tune carefully, alert on retry rate.
• Poor: be conservative, lots of buffer.

Step 8: What is the operational maturity?¶

• New team: simple policy.
• Mature team: adaptive policy.

After answering these, you have a specific retry policy to write.

Appendix O: The Backpressure-Retry Hybrid¶

A pattern used by some sophisticated systems: combine backpressure with retry to handle both bounded and unbounded overload.

Idea¶

• Server returns 429 Retry-After: N when overloaded.
• Client respects the Retry-After (backpressure).
• If Retry-After is missing, client uses local exponential backoff (retry).

So the server prefers to signal load; the client defaults to local backoff.

Implementation¶

func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    var lastErr error
    for attempt := 0; attempt < c.maxAttempts; attempt++ {
        resp, err := c.http.Do(req)
        if err != nil {
            lastErr = err
        } else if resp.StatusCode == 429 || resp.StatusCode >= 500 {
            wait, hasWait := parseRetryAfter(resp)
            resp.Body.Close()
            var d time.Duration
            if hasWait {
                jitter := time.Duration(rand.Int63n(int64(wait) / 5))
                d = wait + jitter
            } else {
                d = c.fullJitter(attempt)
            }
            if err := sleepCtx(ctx, d); err != nil {
                return nil, err
            }
            continue
        } else {
            return resp, nil
        }
        if err := sleepCtx(ctx, c.fullJitter(attempt)); err != nil {
            return nil, err
        }
    }
    return nil, lastErr
}

The server gets to suggest the backoff when it knows the right value. The client uses local backoff otherwise.

Appendix P: A Recap of Numbers To Remember¶

For senior-level conversations, internalise these:

• Latency growth near saturation: hyperbolic. 1 / (1 - ρ).
• Multiplicative amplification: product over tiers of (1 / (1 - f)).
• Retry budget heuristic: 10% of total traffic.
• Idempotency window: 24 hours typical.
• Hedging budget: 1-5% of total traffic.
• Adaptive throttling formula: P_reject = max(0, (requests - K * accepts) / (requests + 1)) with K = 2.
• Default backoff: 100ms base, 5s cap, 3-5 attempts, full jitter.
• Default per-call timeout: match expected p99 latency, then double.
• Default total deadline: 5s for user-facing, more for background.

Quote these in design reviews.

Appendix Q: A Production-Adjacent Senior Retrier¶

A complete retrier with everything a senior engineer cares about: jitter, budget, breaker, deadline, idempotency, observability.

package retry

import (
    "context"
    "errors"
    "fmt"
    "math/rand"
    "sync/atomic"
    "time"

    "github.com/sony/gobreaker"
    "golang.org/x/time/rate"
)

// Policy is the parameter struct for Retrier.
type Policy struct {
    MaxAttempts     int
    Base            time.Duration
    MaxDelay        time.Duration
    Budget          *rate.Limiter
    Breaker         *gobreaker.CircuitBreaker
    OnAttempt       func(attempt int, err error)
    OnGiveUp        func(attempts int, err error)
    OnBudgetDenied  func(lastErr error)
    OnBreakerDenied func()
}

// Stats captures runtime counters.
type Stats struct {
    Attempts       int64
    Retries        int64
    Successes      int64
    GaveUp         int64
    BudgetDenied   int64
    BreakerDenied  int64
    ContextErrors  int64
}

// Retrier composes a policy and statistics.
type Retrier struct {
    Policy Policy
    Stats  Stats
}

// New builds a Retrier with sane defaults.
func New(p Policy) *Retrier {
    if p.MaxAttempts <= 0 {
        p.MaxAttempts = 3
    }
    if p.Base <= 0 {
        p.Base = 100 * time.Millisecond
    }
    if p.MaxDelay <= 0 {
        p.MaxDelay = 5 * time.Second
    }
    return &Retrier{Policy: p}
}

// Do runs op until success, terminal failure, or budget/breaker denial.
func (r *Retrier) Do(ctx context.Context, op func(context.Context) error) error {
    var lastErr error
    for attempt := 0; attempt < r.Policy.MaxAttempts; attempt++ {
        atomic.AddInt64(&r.Stats.Attempts, 1)
        if attempt > 0 {
            atomic.AddInt64(&r.Stats.Retries, 1)
        }
        if err := ctx.Err(); err != nil {
            atomic.AddInt64(&r.Stats.ContextErrors, 1)
            return fmt.Errorf("retry context: %w", err)
        }
        if r.Policy.Breaker != nil {
            if state := r.Policy.Breaker.State(); state == gobreaker.StateOpen {
                atomic.AddInt64(&r.Stats.BreakerDenied, 1)
                if r.Policy.OnBreakerDenied != nil {
                    r.Policy.OnBreakerDenied()
                }
                return fmt.Errorf("circuit open: %w", lastErr)
            }
        }
        err := op(ctx)
        if r.Policy.OnAttempt != nil {
            r.Policy.OnAttempt(attempt, err)
        }
        if err == nil {
            atomic.AddInt64(&r.Stats.Successes, 1)
            return nil
        }
        var perm *Permanent
        if errors.As(err, &perm) {
            return perm.Err
        }
        lastErr = err
        if attempt >= r.Policy.MaxAttempts-1 {
            break
        }
        if r.Policy.Budget != nil && !r.Policy.Budget.Allow() {
            atomic.AddInt64(&r.Stats.BudgetDenied, 1)
            if r.Policy.OnBudgetDenied != nil {
                r.Policy.OnBudgetDenied(lastErr)
            }
            return fmt.Errorf("%w: %v", ErrBudgetExhausted, lastErr)
        }
        delay := r.fullJitter(attempt)
        if deadline, ok := ctx.Deadline(); ok {
            remaining := time.Until(deadline)
            if remaining < delay {
                delay = remaining
            }
        }
        if err := sleepCtx(ctx, delay); err != nil {
            atomic.AddInt64(&r.Stats.ContextErrors, 1)
            return err
        }
    }
    atomic.AddInt64(&r.Stats.GaveUp, 1)
    if r.Policy.OnGiveUp != nil {
        r.Policy.OnGiveUp(r.Policy.MaxAttempts, lastErr)
    }
    return fmt.Errorf("after %d attempts: %w", r.Policy.MaxAttempts, lastErr)
}

func (r *Retrier) fullJitter(attempt int) time.Duration {
    cap := r.Policy.Base * time.Duration(1<<attempt)
    if cap > r.Policy.MaxDelay || cap < 0 {
        cap = r.Policy.MaxDelay
    }
    if cap <= 0 {
        return 0
    }
    return time.Duration(rand.Int63n(int64(cap)))
}

func sleepCtx(ctx context.Context, d time.Duration) error {
    if d <= 0 {
        return nil
    }
    t := time.NewTimer(d)
    defer t.Stop()
    select {
    case <-t.C:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

// Permanent wraps an error to signal "do not retry".
type Permanent struct{ Err error }

func (p *Permanent) Error() string { return p.Err.Error() }
func (p *Permanent) Unwrap() error { return p.Err }

// MarkPermanent returns err wrapped so Do will not retry.
func MarkPermanent(err error) error { return &Permanent{Err: err} }

// ErrBudgetExhausted indicates the retry budget was empty.
var ErrBudgetExhausted = errors.New("retry budget exhausted")