Exponential Backoff — Middle Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. The Thundering Herd Problem
5. The Three Jitter Strategies
6. Full Jitter
7. Equal Jitter
8. Decorrelated Jitter
9. Choosing Among the Three
10. Implementation in Go
11. Math/Rand vs Crypto/Rand
12. Sharing a Random Source Safely
13. Context Cancellation
14. Context-Aware Sleep
15. Deadline Propagation
16. The Retry Budget Idea
17. Token Bucket as a Retry Budget
18. Composable Policies
19. Mental Models
20. Real-World Analogies
21. Pros and Cons of Jitter
22. Use Cases
23. Code Examples
24. Coding Patterns
25. Clean Code
26. Error Handling
27. Performance Tips
28. Best Practices
29. Edge Cases and Pitfalls
30. Common Mistakes
31. Common Misconceptions
32. Tricky Points
33. Test
34. Tricky Questions
35. Cheat Sheet
36. Self-Assessment Checklist
37. Summary
38. What You Can Build
39. Further Reading
40. Related Topics
41. Diagrams and Visual Aids

Introduction¶

Focus: "Why must I randomise the delays? How do I cancel a retry mid-sleep? What is a retry budget and why do I need one?"

At the junior level you wrote a working retry loop with a doubling delay. It is correct enough for low-volume code. At any meaningful scale it has three serious problems that this file addresses:

1. All your clients retry in lockstep. When a service has a 200-ms blip, ten thousand clients each see the failure at roughly the same instant and each schedule a retry exactly 100 ms later. The recovering service is then hit by a synchronised pulse of ten thousand requests. This is the thundering herd, and exponential backoff alone does not prevent it. The fix is jitter — random variation of the delay so the retries spread out.
2. Sleeping cannot be cancelled. Your user clicked "cancel" while you are inside time.Sleep(5 * time.Second). The user gets to wait another 4.7 seconds for nothing. Worse, if the request handler is itself bounded by a deadline, your retry loop may exceed it. The fix is a context-aware sleep that wakes early when ctx.Done() fires.
3. There is no system-wide cap on retry traffic. Each individual loop is bounded by maxAttempts, but if a million users all hit a failure at once, you produce a million retries. The fix is a retry budget: a system-wide token bucket that throttles the total retry rate.

This file gives you all three. By the end you will have a context-aware, jittered, budgeted retry helper that is suitable for moderate production use. The senior file goes deeper into the math and architecture; the professional file takes it to integration with real libraries and observability.

After reading this file you will:

• Understand the thundering-herd failure mode and why it kills services that pure-exponential clients use.
• Know the three jitter strategies — full, equal, decorrelated — and when to use each.
• Be able to implement each in Go from memory.
• Know the difference between math/rand and crypto/rand for jitter (spoiler: use math/rand, but be careful about seeding and sharing).
• Be able to write a retry loop that wakes early on context cancellation.
• Understand deadline propagation: a 5-second deadline at the top means at most 5 seconds across all retries.
• Know what a retry budget is and how to implement it with golang.org/x/time/rate.
• Be able to compose policies — a JitteredPolicy wrapping a BasePolicy wrapping a ContextPolicy.

You do not need to know about distributed tracing, gRPC interceptors, or circuit breakers yet. Those are the professional level.

Prerequisites¶

• Required: Comfort with the junior file. You should be able to write a retry helper in 5 minutes.
• Required: Familiarity with context.Context: context.Background, context.WithCancel, context.WithTimeout, <-ctx.Done(), ctx.Err().
• Required: Familiarity with select over channels.
• Required: Knowledge of math/rand basics: rand.Float64, rand.Intn, rand.New(rand.NewSource(seed)).
• Helpful: Familiarity with the golang.org/x/time/rate token-bucket limiter, or willingness to read a small bit of its API.
• Helpful: Some exposure to sync.Mutex and the idea that math/rand's global functions are not safe for concurrent use without locking (until Go 1.20+, anyway — we discuss this).

Glossary¶

The Thundering Herd Problem¶

Imagine you run a popular API with 100,000 active clients. Each client calls your API once a second. Most calls succeed.

One day, your database fails over. For 2 seconds, the API returns 503 Service Unavailable. Every client sees the failure. Every client implements exponential backoff with no jitter: delay = 100ms * 2^attempt.

What happens?

The recovering database — which has cold caches, an empty query plan cache, and a barely-warmed connection pool — receives a synchronised pulse of 100,000 retries at t=3.7. That is a spike, not a steady load. The service, which could normally handle 200k req/s of steady traffic, falls over because the spike exceeds the burst capacity.

Now the service is broken again. Every client schedules another retry. The cycle repeats. This is the retry storm form of thundering herd.

Exponential backoff did not prevent it. Why? Because all the clients still synchronised their retries. Doubling the delay shifts the spike to later, but does not spread it across time.

Jitter is the antidote. If each client adds a random amount to its delay, the spike becomes a plateau — the same number of retries spread over a window of time. The recovering service sees a gentle ramp rather than a cliff.

The picture:

Without jitter:        With jitter:

   |  |                  ___________
   |  |   ←  spike  →   /           \   ← plateau →
   |  |                /             \
   |  |               /               \
___|__|___           /_________________\___

Same total number of retries; very different load profile.

The Three Jitter Strategies¶

There are three well-known jitter strategies, named in a 2015 AWS Architecture Blog post by Marc Brooker. All three have been studied empirically and are used in production at AWS, Google, Stripe, and elsewhere.

1. Full jitter — delay = uniform(0, cap), where cap = min(maxDelay, base * 2^attempt).
2. Equal jitter — delay = cap/2 + uniform(0, cap/2).
3. Decorrelated jitter — delay = min(maxDelay, uniform(base, prev * 3)), where prev is the previous delay.

The AWS post showed that all three are better than no jitter, and that full jitter is best in most cases — both for total time to completion and for load on the failing service. Decorrelated jitter is the second-best and has the property that consecutive delays are less correlated, which can be useful in pathological cases.

We discuss each in detail.

Full Jitter¶

delay = uniform(0, min(maxDelay, base * 2^attempt))

The "cap" is the exponentially-growing value, but the actual delay is drawn uniformly from [0, cap). So early delays are short on average, late delays are longer on average, but each delay could be anywhere in the range.

Example¶

base = 100ms, maxDelay = 30s:

Expected delay at attempt n is min(maxDelay, base * 2^n) / 2. That is half the no-jitter value. Concretely, the average total wait is half what no-jitter would produce — and that is a feature, not a bug. The randomness gives some clients short delays (good for fast recovery) and some clients long delays (good for spreading load).

Implementation¶

func fullJitter(attempt int, base, maxDelay time.Duration, r *rand.Rand) time.Duration {
    cap := base * time.Duration(1<<attempt)
    if cap > maxDelay || cap < 0 {
        cap = maxDelay
    }
    return time.Duration(r.Int63n(int64(cap)))
}

r.Int63n(n) returns a uniformly distributed integer in [0, n). We cast to time.Duration.

If cap is zero — e.g. base = 0 — r.Int63n panics. Guard with if cap <= 0 { return 0 }.

Properties¶

• Expected delay: cap / 2.
• Variance: cap² / 12 (the variance of a uniform distribution on [0, cap]).
• Worst case: cap.
• Best case: 0 (a client may retry immediately).

The "best case" is sometimes surprising. A client could draw a delay of 0 ns and retry instantly. This is fine — it does not produce a herd because other clients drew larger delays.

Why this is the AWS recommendation¶

The AWS blog tested all three strategies in a simulation of 100 clients retrying against a single bottleneck server. Full jitter produced:

• The shortest total time to complete all client requests.
• The most uniform load on the server during recovery.
• The smallest standard deviation of completion times across clients.

The reason is intuitive: full jitter has the highest variance, so client retries are most spread out. A high-variance distribution covers the recovery window most uniformly.

For the vast majority of cases, full jitter is the right default.

Equal Jitter¶

delay = cap/2 + uniform(0, cap/2)

Half-exponential plus half-uniform. The delay is always between cap/2 and cap. Compared to full jitter, equal jitter has lower variance.

Example¶

base = 100ms, attempt 3, cap = 800ms. Equal jitter draws delays from [400, 800]ms. Full jitter draws from [0, 800]ms. Equal jitter is "guaranteed to wait at least half the exponential", which makes it more polite to the failing service in the short term.

Implementation¶

func equalJitter(attempt int, base, maxDelay time.Duration, r *rand.Rand) time.Duration {
    cap := base * time.Duration(1<<attempt)
    if cap > maxDelay || cap < 0 {
        cap = maxDelay
    }
    half := cap / 2
    return half + time.Duration(r.Int63n(int64(half)))
}

When to use equal jitter¶

If you specifically want to avoid the "client retries immediately after a 503" case — for example, if your service tracks retries by client and penalises clients who retry too aggressively — equal jitter guarantees at least cap/2 wait. Full jitter could draw a 0 ms wait.

Otherwise full jitter is preferable.

Decorrelated Jitter¶

delay = min(maxDelay, uniform(base, prev * 3))

Where prev is the delay used in the previous retry (initially base).

Unlike full and equal jitter, decorrelated jitter does not directly depend on attempt. It evolves: each delay is sampled from a window that grows based on the previous delay.

Walk-through¶

base = 100ms, maxDelay = 30s. Starting prev = base = 100ms.

• Attempt 0: range is [100ms, 300ms]. Sample 200ms. prev = 200ms.
• Attempt 1: range is [100ms, 600ms]. Sample 350ms. prev = 350ms.
• Attempt 2: range is [100ms, 1050ms]. Sample 700ms. prev = 700ms.
• Attempt 3: range is [100ms, 2100ms]. Sample 1500ms. prev = 1500ms.
• Attempt 4: range is [100ms, 4500ms]. Sample 3200ms. prev = 3200ms.
• Attempt 5: range is [100ms, 9600ms]. Sample 7500ms. prev = 7500ms.

The delay grows on average like a power law, but each individual delay is uniformly distributed over a window. The growth factor is 3, which gives roughly the same expected growth as a factor = 2 no-jitter exponential (because the expected value of uniform(a, b) is (a+b)/2).

Implementation¶

type DecorrelatedJitter struct {
    Base     time.Duration
    MaxDelay time.Duration
    Prev     time.Duration
    Rand     *rand.Rand
}

func (d *DecorrelatedJitter) Next() time.Duration {
    if d.Prev == 0 {
        d.Prev = d.Base
    }
    upper := d.Prev * 3
    if upper > d.MaxDelay || upper < 0 {
        upper = d.MaxDelay
    }
    span := upper - d.Base
    if span <= 0 {
        d.Prev = d.Base
        return d.Base
    }
    delay := d.Base + time.Duration(d.Rand.Int63n(int64(span)))
    d.Prev = delay
    return delay
}

Note that the function is stateful — it needs Prev between calls. This is the main reason decorrelated jitter is less commonly used; it requires a struct rather than a pure function.

Why "decorrelated"?¶

With full or equal jitter, the expected delay at attempt N+1 is fixed (it depends only on N). So if you look at the sequence of delays across many retries, they have a strong "growing on average" pattern.

With decorrelated jitter, each delay depends on the previous random value. The sequence is less correlated with attempt number. This can be useful in scenarios where you want to avoid any predictable schedule that an attacker could exploit — but those scenarios are rare.

Empirically, the AWS blog showed that decorrelated jitter performs almost as well as full jitter for thundering-herd mitigation, and better in some pathological cases. It is a reasonable choice, but slightly more complex to implement.

Choosing Among the Three¶

Default: full jitter. Use this unless you have a specific reason not to.

Use equal jitter when: - You want a guaranteed minimum delay between retries (e.g. to comply with a rate-limit contract). - Your monitoring is easier to interpret with predictable lower-bounds on delays.

Use decorrelated jitter when: - You need the lowest possible correlation between successive delays. - You are implementing a specific algorithm (some AWS SDKs use decorrelated jitter internally).

Avoid no-jitter: - Always. The thundering-herd risk is too high for any non-trivial system.

The AWS blog post's overall recommendation, repeated by many large-scale engineering teams: if in doubt, use full jitter.

Implementation in Go¶

A clean, composable implementation of all three:

package backoff

import (
    "math/rand"
    "sync"
    "time"
)

// Jitterer computes a single delay for a given attempt.
type Jitterer interface {
    Delay(attempt int) time.Duration
}

type FullJitter struct {
    Base     time.Duration
    MaxDelay time.Duration
    r        *rand.Rand
    mu       sync.Mutex
}

func (j *FullJitter) Delay(attempt int) time.Duration {
    cap := j.Base * time.Duration(1<<attempt)
    if cap > j.MaxDelay || cap < 0 {
        cap = j.MaxDelay
    }
    if cap <= 0 {
        return 0
    }
    j.mu.Lock()
    defer j.mu.Unlock()
    return time.Duration(j.r.Int63n(int64(cap)))
}

type EqualJitter struct {
    Base     time.Duration
    MaxDelay time.Duration
    r        *rand.Rand
    mu       sync.Mutex
}

func (j *EqualJitter) Delay(attempt int) time.Duration {
    cap := j.Base * time.Duration(1<<attempt)
    if cap > j.MaxDelay || cap < 0 {
        cap = j.MaxDelay
    }
    half := cap / 2
    if half <= 0 {
        return 0
    }
    j.mu.Lock()
    defer j.mu.Unlock()
    return half + time.Duration(j.r.Int63n(int64(half)))
}

type DecorrelatedJitter struct {
    Base     time.Duration
    MaxDelay time.Duration
    r        *rand.Rand
    mu       sync.Mutex
    prev     time.Duration
}

func (j *DecorrelatedJitter) Delay(_ int) time.Duration {
    j.mu.Lock()
    defer j.mu.Unlock()
    if j.prev == 0 {
        j.prev = j.Base
    }
    upper := j.prev * 3
    if upper > j.MaxDelay || upper < 0 {
        upper = j.MaxDelay
    }
    span := upper - j.Base
    if span <= 0 {
        j.prev = j.Base
        return j.Base
    }
    delay := j.Base + time.Duration(j.r.Int63n(int64(span)))
    j.prev = delay
    return delay
}

// New returns a Jitterer of the given strategy, sharing a *rand.Rand.
func New(strategy string, base, maxDelay time.Duration, r *rand.Rand) Jitterer {
    switch strategy {
    case "full":
        return &FullJitter{Base: base, MaxDelay: maxDelay, r: r}
    case "equal":
        return &EqualJitter{Base: base, MaxDelay: maxDelay, r: r}
    case "decorrelated":
        return &DecorrelatedJitter{Base: base, MaxDelay: maxDelay, r: r}
    }
    return &FullJitter{Base: base, MaxDelay: maxDelay, r: r}
}

Notice the mutex per jitterer. rand.Rand is not safe for concurrent use; each call to Int63n mutates the source's internal state. We discuss the trade-offs of locking next.

Math/Rand vs Crypto/Rand¶

For jitter, you want a pseudo-random number generator. Two choices in Go:

• math/rand — fast, deterministic given a seed, not cryptographically secure.
• crypto/rand — slow, cryptographically secure, uses the OS's entropy source.

For jitter, use math/rand. Reasons:

1. Jitter does not need cryptographic randomness. An attacker who can predict your jitter has bigger problems.
2. math/rand is ~50× faster than crypto/rand.
3. math/rand is deterministic, which makes tests reproducible.

The exception is when jitter is used in a security-sensitive context — e.g. timing-attack mitigation. There, crypto/rand makes sense.

math/rand/v2 (Go 1.22+)¶

Go 1.22 introduced math/rand/v2 with a cleaner API and better default generator. The recommended idiom:

import "math/rand/v2"

delay := rand.Int64N(int64(cap))

rand/v2's top-level functions are safe for concurrent use without locking. The underlying generator is the PCG family, which is fast and high-quality. If you are on Go 1.22+, prefer math/rand/v2 for new code.

For Go 1.20-1.21, the global math/rand is also concurrent-safe, but the API is older. Either is fine.

For Go 1.19 and earlier, the global math/rand was not concurrent-safe, and you had to lock or use rand.New(rand.NewSource(...)) per goroutine. Most of the patterns in this section accommodate the older convention because libraries you read may still use it.

Seeding¶

In Go 1.20+, the global math/rand is auto-seeded with a random value at program start. You no longer need rand.Seed(time.Now().UnixNano()). The rand.Seed function is now deprecated.

If you use rand.New(rand.NewSource(seed)), choose a seed that varies per process: time.Now().UnixNano() is fine.

A common gotcha¶

// Before Go 1.20:
src := rand.NewSource(time.Now().UnixNano())
r := rand.New(src)

This is safe in one goroutine. If you share r across goroutines without locking, you get data races. In Go 1.20+ the global functions are concurrent-safe, but a *rand.Rand you create yourself with rand.New is not.

The lesson: either use the global math/rand (or math/rand/v2) functions, or wrap your *rand.Rand with a sync.Mutex.

Sharing a Random Source Safely¶

If you have many goroutines doing jittered retries, they all need random numbers. Three strategies:

Strategy 1: global rand (Go 1.20+)¶

func fullJitter(attempt int, base, maxDelay time.Duration) time.Duration {
    cap := base * time.Duration(1<<attempt)
    if cap > maxDelay { cap = maxDelay }
    return time.Duration(rand.Int63n(int64(cap)))
}

The global functions in math/rand use an internal lock. Concurrent calls are serialised. On hot paths with many goroutines this can be a contention point — but for jitter, which fires at most once per retry, the contention is negligible.

Strategy 2: per-goroutine *rand.Rand¶

var randPool = sync.Pool{
    New: func() any {
        return rand.New(rand.NewSource(time.Now().UnixNano()))
    },
}

func fullJitter(attempt int, base, maxDelay time.Duration) time.Duration {
    cap := base * time.Duration(1<<attempt)
    if cap > maxDelay { cap = maxDelay }
    r := randPool.Get().(*rand.Rand)
    delay := time.Duration(r.Int63n(int64(cap)))
    randPool.Put(r)
    return delay
}

sync.Pool lets each goroutine grab its own *rand.Rand. No locking is needed because each *rand.Rand is owned by exactly one goroutine at a time.

This pattern is overkill for jitter but useful if you do a lot of random sampling.

Strategy 3: wrap with a mutex¶

type LockedRand struct {
    mu sync.Mutex
    r  *rand.Rand
}

func (l *LockedRand) Int63n(n int64) int64 {
    l.mu.Lock()
    defer l.mu.Unlock()
    return l.r.Int63n(n)
}

Same as the global rand internally, but you control the source.

Recommendation¶

For Go 1.20+, just use the global math/rand (or math/rand/v2) functions. They are concurrent-safe and fast enough.

For older versions, wrap with a mutex.

Do not use crypto/rand for jitter unless you have a specific reason.

Context Cancellation¶

context.Context is Go's idiomatic way to signal cancellation and propagate deadlines. Every request-scoped operation should accept a context.Context as its first argument.

A retry loop without context support is broken in three ways:

1. The user cannot cancel. If the user closes their browser or hits Ctrl+C, the retry keeps going.
2. A deadline cannot be enforced. A 5-second SLO cannot be honoured if the loop is mid-sleep.
3. Resources leak. A goroutine running a retry loop after the rest of the request has been cancelled is a leak.

The fix is straightforward: accept ctx context.Context, check ctx.Err() before each attempt, and replace time.Sleep with a context-aware sleep.

func Do(ctx context.Context, op func(context.Context) error, p Policy) error {
    var lastErr error
    for attempt := 0; attempt < p.MaxAttempts; attempt++ {
        if err := ctx.Err(); err != nil {
            return fmt.Errorf("retry cancelled: %w", err)
        }
        err := op(ctx)
        if err == nil {
            return nil
        }
        if isPermanent(err) {
            return err
        }
        lastErr = err
        if attempt < p.MaxAttempts-1 {
            if err := sleepCtx(ctx, p.Delay(attempt)); err != nil {
                return fmt.Errorf("retry cancelled while sleeping: %w", err)
            }
        }
    }
    return fmt.Errorf("after %d attempts: %w", p.MaxAttempts, lastErr)
}

The key changes from the junior version:

1. The signature now takes ctx.
2. op itself accepts ctx — so the operation can use it for its own timeouts.
3. Before each attempt we check ctx.Err(). If cancelled, return immediately.
4. The sleep is via sleepCtx, which respects cancellation.

Context-Aware Sleep¶

func sleepCtx(ctx context.Context, d time.Duration) error {
    if d <= 0 {
        return nil
    }
    t := time.NewTimer(d)
    defer t.Stop()
    select {
    case <-t.C:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

This is the canonical pattern. Three points to notice:

1. time.NewTimer not time.After. time.After(d) creates a timer that is not garbage-collected until it fires. In a tight loop this leaks timers. time.NewTimer + defer t.Stop() is the leak-free version.
2. defer t.Stop(). Even though t.Stop() returns false if the timer already fired, calling it does not harm. Always defer Stop after NewTimer.
3. Return ctx.Err(). If ctx.Done() fires, we return the context's error — context.Canceled or context.DeadlineExceeded. The caller can errors.Is(err, context.Canceled) to distinguish.

A common variation reuses the timer across iterations:

t := time.NewTimer(0) // initial delay 0
defer t.Stop()
if !t.Stop() { <-t.C } // drain
for attempt := 0; attempt < N; attempt++ {
    // ...
    if attempt < N-1 {
        t.Reset(delay)
        select {
        case <-t.C:
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

This avoids the per-iteration allocation of time.NewTimer. For very high-frequency retry loops, it matters. For typical applications, the simpler form is fine.

Deadline Propagation¶

A context.Context can carry a deadline. The pattern:

ctx, cancel := context.WithTimeout(parentCtx, 5*time.Second)
defer cancel()

err := retry.Do(ctx, doIt, policy)

The retry loop must respect the deadline. If the deadline is at t=5s and the loop is at t=4.5s about to sleep 2s, it should not sleep 2s — it should sleep at most 0.5s, then return context.DeadlineExceeded.

The sleepCtx we wrote already handles this: <-ctx.Done() fires when the deadline arrives.

But the loop should also be smart about the per-attempt sleep:

func nextSleep(ctx context.Context, requested time.Duration) time.Duration {
    deadline, ok := ctx.Deadline()
    if !ok {
        return requested
    }
    remaining := time.Until(deadline)
    if requested > remaining {
        return remaining
    }
    return requested
}

If the parent deadline is closer than requested, sleep only until the deadline. Then the next call to op(ctx) will see ctx.Err() != nil and short-circuit.

This avoids the situation where you sleep most of the remaining time and then attempt a call with a tiny budget. Better to give the operation as much budget as possible.

Propagation downstream¶

Deadline propagation matters because the operation itself takes time:

err := op(ctx) // op should respect ctx.Deadline()

If op is http.Client.Do, it must use the context. Go's http.NewRequestWithContext plus client.Do does this; the request will be cancelled when the context is.

Likewise for gRPC, database libraries (db.QueryContext), and your own functions. All of them should accept and respect a context.Context.

Code smell: a function that does not accept context.Context. In modern Go, every I/O-bound function should accept it.

The Retry Budget Idea¶

The retry-budget concept is a system-wide cap on retry traffic, separate from the per-call attempt cap. It is the answer to "what if a million users all hit a failure at once".

Without a budget:

• 1,000,000 users hit a failing API.
• Each user retries up to 5 times.
• Maximum total traffic: 5,000,000 calls per failure-window.

With a budget:

• 1,000,000 users hit a failing API.
• The retry budget allows 10% retry traffic (or some other fraction).
• Only 100,000 retries fire; the rest are denied locally.

The budget is enforced at the client side — your retry helper consults a counter before retrying, and if the counter is exhausted, it surfaces the error without retrying.

Why on the client side? Because the purpose of the budget is to protect the failing service. If every client retries unlimited, the service is overwhelmed. If clients voluntarily limit retries, the service has headroom to recover.

What does the budget protect?¶

The budget protects against correlated failures. When one failure causes all clients to retry, the retry traffic itself is the second-order failure.

A bare maxAttempts per call does not solve this. Each individual call is bounded. The aggregate is not.

Token Bucket as a Retry Budget¶

The canonical implementation uses a token bucket: tokens accumulate at a configured rate; each retry consumes a token; if the bucket is empty, retries are denied.

Go has a standard implementation: golang.org/x/time/rate.Limiter.

import "golang.org/x/time/rate"

type Retrier struct {
    Policy  Policy
    Budget  *rate.Limiter
}

func (r *Retrier) Do(ctx context.Context, op func(context.Context) error) error {
    var lastErr error
    for attempt := 0; attempt < r.Policy.MaxAttempts; attempt++ {
        if attempt > 0 {
            // require a budget token before retrying
            if !r.Budget.Allow() {
                return fmt.Errorf("retry budget exhausted: %w", lastErr)
            }
        }
        err := op(ctx)
        if err == nil {
            return nil
        }
        lastErr = err
        if attempt < r.Policy.MaxAttempts-1 {
            if err := sleepCtx(ctx, r.Policy.Delay(attempt)); err != nil {
                return err
            }
        }
    }
    return fmt.Errorf("after %d attempts: %w", r.Policy.MaxAttempts, lastErr)
}

Construction:

// Allow 100 retries/sec system-wide, with a burst of 200.
budget := rate.NewLimiter(100, 200)
retrier := &Retrier{Policy: policy, Budget: budget}

The limiter is shared across all goroutines using the same retrier. When traffic spikes, the bucket empties and retries are denied until tokens replenish.

Tuning the budget¶

The choice of rate is workload-specific. A common heuristic:

• Allow retries equal to 0.1 * normal_request_rate. (10% retry budget.)
• Burst = 2× the rate.

Google's SRE book has more sophisticated formulas; we discuss them in the senior file.

Per-route budgets¶

A single global budget is the simplest. For more sophistication, per-route budgets:

budgets := map[string]*rate.Limiter{
    "users":   rate.NewLimiter(100, 200),
    "payments": rate.NewLimiter(10, 20),
}

Then retries for users and payments are bounded independently. If payments is broken, users keeps working.

Composable Policies¶

You now have several concepts: a backoff schedule, a jitter strategy, a max-attempts cap, a max-delay cap, a context, a budget. A retry library needs to compose them cleanly. The idiomatic Go pattern is interfaces.

type Policy interface {
    // NextDelay returns the duration to wait before the (attempt+1)th attempt,
    // or (0, false) if no more retries should occur.
    NextDelay(attempt int) (time.Duration, bool)
}

type ExponentialPolicy struct {
    Base        time.Duration
    MaxDelay    time.Duration
    MaxAttempts int
}

func (p ExponentialPolicy) NextDelay(attempt int) (time.Duration, bool) {
    if attempt >= p.MaxAttempts {
        return 0, false
    }
    d := p.Base * time.Duration(1<<attempt)
    if d > p.MaxDelay || d < 0 {
        d = p.MaxDelay
    }
    return d, true
}

// Jittered wraps another policy with full jitter.
type Jittered struct {
    Inner Policy
    Rand  *rand.Rand
    mu    sync.Mutex
}

func (j *Jittered) NextDelay(attempt int) (time.Duration, bool) {
    d, ok := j.Inner.NextDelay(attempt)
    if !ok {
        return 0, false
    }
    j.mu.Lock()
    defer j.mu.Unlock()
    return time.Duration(j.Rand.Int63n(int64(d))), true
}

// Budgeted wraps a policy with a retry budget.
type Budgeted struct {
    Inner  Policy
    Budget *rate.Limiter
}

func (b *Budgeted) NextDelay(attempt int) (time.Duration, bool) {
    if !b.Budget.Allow() {
        return 0, false
    }
    return b.Inner.NextDelay(attempt)
}

Now you can compose:

policy := &Budgeted{
    Inner: &Jittered{
        Inner: ExponentialPolicy{Base: 100 * time.Millisecond, MaxDelay: 5 * time.Second, MaxAttempts: 5},
        Rand:  rand.New(rand.NewSource(time.Now().UnixNano())),
    },
    Budget: rate.NewLimiter(100, 200),
}

Read it inside-out: exponential schedule, jittered, budgeted.

A single retry loop calls policy.NextDelay(attempt):

func Do(ctx context.Context, op func(context.Context) error, policy Policy) error {
    var lastErr error
    for attempt := 0; ; attempt++ {
        err := op(ctx)
        if err == nil {
            return nil
        }
        lastErr = err
        delay, ok := policy.NextDelay(attempt)
        if !ok {
            return fmt.Errorf("retry exhausted: %w", lastErr)
        }
        if err := sleepCtx(ctx, delay); err != nil {
            return err
        }
    }
}

Composability is the point. You can plug different policies into the same loop without changing the loop.

Mental Models¶

Model 1: jitter is a "spreader"¶

Without jitter, every client has a delta function at base * 2^n. Jitter convolves it with a uniform distribution, spreading it into a plateau. The same total area, less peak.

Model 2: context is a "deadline contract"¶

A context.Context is a promise: "I will give up by this time". The retry loop's job is to deliver on that promise. Every time.Sleep is a potential violation; replace it with sleepCtx.

Model 3: budget is a "shared throttle"¶

The retry budget is what makes individual retry decisions globally coherent. Without it, each call's maxAttempts = 5 aggregates to N×5 retries during an outage. With it, the aggregate is bounded.

Model 4: composition is the way¶

You will encounter many retry features: rate-limited delays, deadline-aware delays, attempt caps, budget caps, conditional retry. The idiomatic Go design is a Policy interface that you compose by wrapping. Resist building a monolithic retry function with 12 parameters.

Real-World Analogies¶

Jitter: Imagine a thousand people leaving a stadium at the same instant. The corridor is jammed for an hour. Now imagine they each wait a random 0–30 minutes before leaving. The corridor is busy for 30 minutes but no one is jammed.

Context cancellation: You are on hold with customer service. You hear "your call will be answered in 5 minutes". A timer ticks. If you hang up before then, the queue forgets you. That is ctx.Done().

Retry budget: A water system has a daily supply. Each household has unlimited taps but the city has a daily cap. When the supply runs low, distribution stops. That is the token bucket.

Decorrelated jitter: Imagine a dice game where each roll's range depends on the previous roll. Big rolls beget big rolls; small rolls beget small rolls — but with enough randomness that you do not get stuck.

Pros and Cons of Jitter¶

Pros:

• Eliminates synchronised retry pulses (thundering herd).
• Spreads load on the recovering service.
• Empirically reduces total time-to-recovery in simulations.
• Simple to implement: one rand.Int63n per delay.

Cons:

• Slightly slower average case for individual clients (a client could draw a short delay).
• Requires a random source (and thread-safety considerations).
• Makes debugging harder — the schedule is not deterministic.
• Test fixtures must seed the source or accept variability.

The cons are minor compared to the thundering-herd risk. Use jitter.

Use Cases¶

• Any client calling a shared service (your microservice calling another microservice).
• Any client doing high-frequency reconnect attempts (websocket clients).
• Any background worker processing a queue with potentially-correlated failures.
• Any system where multiple processes coordinate via a third party.

The rule: if your code might be running in many copies simultaneously, jitter. The only no-jitter case is a single-process tool retrying against a private dependency — and even there, the cost of jitter is nearly zero.

Code Examples¶

Example 1: full-jitter HTTP client¶

package main

import (
    "context"
    "fmt"
    "io"
    "math/rand"
    "net/http"
    "time"
)

type Client struct {
    http       *http.Client
    maxAttempts int
    base       time.Duration
    maxDelay   time.Duration
}

func (c *Client) Get(ctx context.Context, url string) ([]byte, error) {
    var lastErr error
    for attempt := 0; attempt < c.maxAttempts; attempt++ {
        if err := ctx.Err(); err != nil {
            return nil, err
        }
        body, err, retry := c.attempt(ctx, url)
        if err == nil {
            return body, nil
        }
        if !retry {
            return nil, err
        }
        lastErr = err
        if attempt < c.maxAttempts-1 {
            delay := c.fullJitter(attempt)
            if err := sleepCtx(ctx, delay); err != nil {
                return nil, err
            }
        }
    }
    return nil, fmt.Errorf("after %d attempts: %w", c.maxAttempts, lastErr)
}

func (c *Client) fullJitter(attempt int) time.Duration {
    cap := c.base * time.Duration(1<<attempt)
    if cap > c.maxDelay || cap < 0 {
        cap = c.maxDelay
    }
    if cap <= 0 {
        return 0
    }
    return time.Duration(rand.Int63n(int64(cap)))
}

func (c *Client) attempt(ctx context.Context, url string) ([]byte, error, bool) {
    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
    if err != nil {
        return nil, err, false
    }
    resp, err := c.http.Do(req)
    if err != nil {
        return nil, err, true
    }
    defer resp.Body.Close()
    if resp.StatusCode == 429 || (resp.StatusCode >= 500 && resp.StatusCode <= 599) {
        return nil, fmt.Errorf("status %d", resp.StatusCode), true
    }
    if resp.StatusCode >= 400 {
        return nil, fmt.Errorf("status %d", resp.StatusCode), false
    }
    body, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, err, true
    }
    return body, nil, false
}

func sleepCtx(ctx context.Context, d time.Duration) error {
    if d <= 0 {
        return nil
    }
    t := time.NewTimer(d)
    defer t.Stop()
    select {
    case <-t.C:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

Example 2: decorrelated jitter with a generic helper¶

package retry

import (
    "context"
    "errors"
    "fmt"
    "math/rand"
    "sync"
    "time"
)

type DecorrelatedJitter struct {
    Base     time.Duration
    MaxDelay time.Duration
    prev     time.Duration
    mu       sync.Mutex
}

func (d *DecorrelatedJitter) Next() time.Duration {
    d.mu.Lock()
    defer d.mu.Unlock()
    if d.prev == 0 {
        d.prev = d.Base
    }
    upper := d.prev * 3
    if upper > d.MaxDelay || upper < 0 {
        upper = d.MaxDelay
    }
    span := upper - d.Base
    if span <= 0 {
        d.prev = d.Base
        return d.Base
    }
    delay := d.Base + time.Duration(rand.Int63n(int64(span)))
    d.prev = delay
    return delay
}

func DoDecorrelated(ctx context.Context, op func(context.Context) error, base, maxDelay time.Duration, maxAttempts int) error {
    jit := &DecorrelatedJitter{Base: base, MaxDelay: maxDelay}
    var lastErr error
    for attempt := 0; attempt < maxAttempts; attempt++ {
        if err := ctx.Err(); err != nil {
            return err
        }
        err := op(ctx)
        if err == nil {
            return nil
        }
        lastErr = err
        if attempt < maxAttempts-1 {
            if err := sleepCtx(ctx, jit.Next()); err != nil {
                return err
            }
        }
    }
    return fmt.Errorf("after %d attempts: %w", maxAttempts, lastErr)
}

func sleepCtx(ctx context.Context, d time.Duration) error {
    if d <= 0 {
        return nil
    }
    t := time.NewTimer(d)
    defer t.Stop()
    select {
    case <-t.C:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

var ErrPermanent = errors.New("permanent")

Example 3: a budget-aware retrier¶

package retry

import (
    "context"
    "fmt"
    "time"

    "golang.org/x/time/rate"
)

type Retrier struct {
    MaxAttempts int
    Base        time.Duration
    MaxDelay    time.Duration
    Budget      *rate.Limiter
}

func (r *Retrier) Do(ctx context.Context, op func(context.Context) error) error {
    var lastErr error
    for attempt := 0; attempt < r.MaxAttempts; attempt++ {
        if err := ctx.Err(); err != nil {
            return err
        }
        err := op(ctx)
        if err == nil {
            return nil
        }
        lastErr = err
        if attempt < r.MaxAttempts-1 {
            // consume a token before scheduling retry
            if r.Budget != nil {
                if err := r.Budget.Wait(ctx); err != nil {
                    return fmt.Errorf("retry budget: %w", err)
                }
            }
            delay := r.fullJitter(attempt)
            if err := sleepCtx(ctx, delay); err != nil {
                return err
            }
        }
    }
    return fmt.Errorf("after %d attempts: %w", r.MaxAttempts, lastErr)
}

func (r *Retrier) fullJitter(attempt int) time.Duration {
    cap := r.Base * time.Duration(1<<attempt)
    if cap > r.MaxDelay || cap < 0 {
        cap = r.MaxDelay
    }
    if cap <= 0 {
        return 0
    }
    // global math/rand is concurrent-safe in Go 1.20+
    return time.Duration(rand.Int63n(int64(cap)))
}

Note r.Budget.Wait(ctx) — this blocks until a token is available or the context is cancelled. If the budget is exhausted, the wait could be long.

If you want "deny rather than wait", use r.Budget.Allow():

if !r.Budget.Allow() {
    return fmt.Errorf("retry budget exhausted: %w", lastErr)
}

Both are valid. Wait is more polite (you eventually retry), Allow is more decisive (you give up immediately).

Example 4: testing with seeded jitter¶

To make tests reproducible:

func TestFullJitter(t *testing.T) {
    src := rand.NewSource(42)
    r := rand.New(src)
    j := &FullJitter{Base: 100 * time.Millisecond, MaxDelay: 5 * time.Second, r: r}

    delays := []time.Duration{}
    for i := 0; i < 5; i++ {
        delays = append(delays, j.Delay(i))
    }
    // delays is now deterministic, can assert exact values
}

Seeding rand.NewSource(42) produces the same sequence on every run.

Example 5: comparing strategies in a simulation¶

package main

import (
    "fmt"
    "math/rand"
    "time"
)

func main() {
    const attempts = 6
    const base = 100 * time.Millisecond
    const maxDelay = 5 * time.Second
    const clients = 1000

    for _, strategy := range []string{"none", "full", "equal", "decorrelated"} {
        buckets := make(map[int]int)
        for c := 0; c < clients; c++ {
            elapsed := time.Duration(0)
            prev := time.Duration(0)
            for a := 0; a < attempts; a++ {
                var d time.Duration
                switch strategy {
                case "none":
                    d = base * time.Duration(1<<a)
                case "full":
                    cap := base * time.Duration(1<<a)
                    if cap > maxDelay { cap = maxDelay }
                    d = time.Duration(rand.Int63n(int64(cap)))
                case "equal":
                    cap := base * time.Duration(1<<a)
                    if cap > maxDelay { cap = maxDelay }
                    d = cap/2 + time.Duration(rand.Int63n(int64(cap/2)))
                case "decorrelated":
                    if prev == 0 { prev = base }
                    upper := prev * 3
                    if upper > maxDelay { upper = maxDelay }
                    span := upper - base
                    if span > 0 {
                        d = base + time.Duration(rand.Int63n(int64(span)))
                    } else {
                        d = base
                    }
                    prev = d
                }
                if d > maxDelay { d = maxDelay }
                elapsed += d
                bucket := int(elapsed / (100 * time.Millisecond))
                buckets[bucket]++
            }
        }
        fmt.Printf("strategy=%s peak=%d\n", strategy, peakBucket(buckets))
    }
}

func peakBucket(b map[int]int) int {
    peak := 0
    for _, v := range b {
        if v > peak { peak = v }
    }
    return peak
}

Running this gives you concrete numbers for how much each strategy reduces the peak retry rate. The "none" strategy produces the worst peak (all clients retry at the same instant); the jittered strategies produce ~1/10th the peak.

Coding Patterns¶

Pattern A: pass the policy in¶

type Doer interface {
    Do(ctx context.Context, op func(context.Context) error) error
}

Code that calls Do does not need to know which policy is in use. This lets you swap full jitter for equal jitter at the configuration level.

Pattern B: separate "next delay" from "sleep"¶

The NextDelay method computes a duration. The retry loop calls sleepCtx(ctx, delay). By separating, you can unit-test NextDelay without sleeping. Schedule tests run in microseconds.

Pattern C: structure the policy struct¶

type Policy struct {
    MaxAttempts int
    Base        time.Duration
    MaxDelay    time.Duration
    Strategy    string // "full", "equal", "decorrelated", "none"
    Budget      *rate.Limiter // optional
}

A single struct that captures all knobs. Configurable via YAML/JSON.

Pattern D: instrument every retry¶

type Stats struct {
    Attempts      int
    Retries       int
    BudgetDenied  int
    ContextCancel int
}

Counters incremented at each branch. You will use these in professional.md for Prometheus metrics.

Clean Code¶

• Name jitter strategies by string in config but by interface in code. "full" is a config value; FullJitter{} is a Go type.
• Document the policy with a comment giving an example: "// full jitter: delay ~ U[0, base*2^attempt]".
• Test each jitter strategy with a seeded random source for reproducibility.
• Hide *rand.Rand behind a Rand interface so tests can inject a deterministic implementation.

Error Handling¶

Three new error-handling cases at the middle level:

1. context.Canceled. The caller has given up. Return immediately. Wrap as fmt.Errorf("retry cancelled: %w", ctx.Err()) so the caller can errors.Is(err, context.Canceled).
2. context.DeadlineExceeded. The deadline has passed. Same as above; the err type tells the caller why.
3. Retry budget exhausted. A specific error type. Make it distinguishable.

var ErrBudgetExhausted = errors.New("retry budget exhausted")

// In Do:
if !budget.Allow() {
    return fmt.Errorf("%w: %v", ErrBudgetExhausted, lastErr)
}

The caller can errors.Is(err, ErrBudgetExhausted) to handle this case specially — for example, fall back to a degraded mode.

Performance Tips¶

• Avoid time.After in loops. Use time.NewTimer + Stop.
• Reuse a single *time.Timer across iterations of a hot retry loop.
• Use math/rand's global functions (or math/rand/v2) rather than a sync.Mutex-protected source.
• Compute cap once and reuse it.
• Avoid math.Pow for power-of-two delays; use 1 << attempt.
• Cap attempt at 30 before the shift to avoid overflow.

For sub-microsecond retry loops these matter. For typical application retries (millisecond-scale), they do not.

Best Practices¶

1. Always jitter. No-jitter exponential backoff is an antipattern in shared services.
2. Use full jitter by default. Switch to equal or decorrelated only with a reason.
3. Accept context.Context and respect it. Never time.Sleep in a retry that the caller might cancel.
4. Propagate deadlines. Cap the remaining sleep by the remaining deadline budget.
5. Add a retry budget for any retrier that may run at scale.
6. Test with seeded random sources so behaviour is reproducible.
7. Separate "compute delay" from "sleep" so the schedule is unit-testable.
8. Wrap errors with %w so errors.Is/errors.As work.

Edge Cases and Pitfalls¶

• Negative durations. rand.Int63n(int64(cap)) panics if cap <= 0. Guard.
• Zero cap at attempt 0. If base = 0, every delay is zero. Loop spins. Validate config.
• time.After leaks. Use time.NewTimer.
• *rand.Rand is not concurrent-safe. Either wrap with mutex or use global functions.
• Budget.Wait can block indefinitely if no tokens and no deadline. Always pass a context.
• The deadline check is racy. Even if you check ctx.Err() before op(), the deadline may pass during op(). That is fine; the operation will see ctx.Err() and bail.
• Cancellation during sleep. sleepCtx handles this; ensure your loop returns the error.
• Jitter making the average wait too short. Full jitter halves the expected wait. If you wanted the unjittered values, switch to equal jitter.

Common Mistakes¶

1. Forgetting to seed the random source (pre-Go 1.20).
2. Sharing a *rand.Rand across goroutines without locking.
3. Using crypto/rand for jitter ("for security"). Slow, unnecessary.
4. Using time.After in a hot loop.
5. Not propagating the deadline; the retry loop runs past the parent's deadline.
6. Not checking ctx.Err() at the start of each iteration.
7. Calling op without passing it the context.
8. Mixing the budget with the per-call cap; the budget should be additional, not a replacement.
9. Tuning the budget too tight, denying retries that would have succeeded.
10. Not surfacing ErrBudgetExhausted distinctly; callers cannot react.

Common Misconceptions¶

• "Jitter makes the wait longer." No. Full jitter halves the expected wait. Equal jitter has the same expected wait as no-jitter.
• "Context-aware sleep is slower." No. The overhead of select is nanoseconds.
• "Retry budget is the same as maxAttempts." No. maxAttempts is per-call; budget is system-wide.
• "math/rand is unsafe for any concurrent use." Not since Go 1.20+. The global functions are safe; per-instance *rand.Rand still is not.
• "Decorrelated jitter is always better than full jitter." AWS's data shows full jitter is best in most simulations. Decorrelated is close, not better.

Tricky Points¶

• The naming "full jitter" is confusing — it sounds like "maximum jitter" but is actually "uniformly distributed over the full range up to the cap". Synonyms include "0-jitter" or "random-jitter".
• Limiter.Allow() is non-blocking. Limiter.Wait(ctx) blocks. Limiter.Reserve() is lower-level. Pick the right one.
• A rate.Limiter with Limit(0) allows nothing — including the first call. If you want to allow the first call but throttle retries, do the budget check only when attempt > 0.
• The deadline propagated via context.WithTimeout is absolute, computed from the wall clock at the time of the WithTimeout call. If you call time.Sleep for 100s and then check ctx.Done(), the deadline may already have passed.
• In math/rand/v2, the function names changed: Int63n → Int64N. Be careful when porting.

Test¶

Tests at the middle level should cover:

1. The retry loop terminates on ctx.Done.
2. The sleep respects context cancellation.
3. Each jitter strategy produces delays in the documented range.
4. The retry budget denies after enough retries.
5. Permanent errors short-circuit.
6. The deadline is honoured.

func TestSleepCtxCancelled(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    go func() {
        time.Sleep(10 * time.Millisecond)
        cancel()
    }()
    start := time.Now()
    err := sleepCtx(ctx, 1*time.Second)
    elapsed := time.Since(start)
    if err == nil {
        t.Fatal("expected error")
    }
    if elapsed > 100*time.Millisecond {
        t.Fatalf("sleep did not wake on cancel; elapsed=%v", elapsed)
    }
}

func TestFullJitterRange(t *testing.T) {
    src := rand.NewSource(1)
    r := rand.New(src)
    j := &FullJitter{Base: 100*time.Millisecond, MaxDelay: 5*time.Second, r: r}
    for i := 0; i < 1000; i++ {
        d := j.Delay(3)
        // attempt 3 => cap = min(5s, 100ms * 8) = 800ms
        if d < 0 || d > 800*time.Millisecond {
            t.Errorf("delay %v out of [0, 800ms]", d)
        }
    }
}

func TestBudgetExhausted(t *testing.T) {
    budget := rate.NewLimiter(rate.Inf, 0) // no tokens ever
    r := &Retrier{MaxAttempts: 5, Base: 1*time.Millisecond, MaxDelay: 10*time.Millisecond, Budget: budget}
    callCount := 0
    err := r.Do(context.Background(), func(_ context.Context) error {
        callCount++
        return errors.New("transient")
    })
    if err == nil {
        t.Fatal("expected error")
    }
    if callCount != 1 {
        t.Fatalf("expected 1 call (no budget for retry), got %d", callCount)
    }
}