Exponential Backoff — Professional Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Cenkalti/Backoff Walkthrough
5. Hashicorp Go-Retryablehttp
6. AWS SDK V2 Retry Internals
7. Integration With net/http
8. Integration With gRPC
9. Service Config and Retry
10. gRPC Interceptors For Retry
11. Circuit Breaker Integration
12. Sony/Gobreaker In Depth
13. Resilience4j-Style Composition
14. Observability — Metrics
15. Observability — Tracing
16. Observability — Logging
17. Postmortem Case Studies
18. Operational Runbook
19. Performance Tuning
20. End-To-End Example
21. Best Practices
22. Edge Cases and Pitfalls
23. Common Mistakes
24. Tricky Questions
25. Cheat Sheet
26. Summary
27. Further Reading
28. Related Topics

Introduction¶

Focus: "Take everything from junior, middle, and senior and turn it into a production Go codebase using real libraries, with real observability, that survives real incidents."

This file is the bridge from theory to running production. The junior, middle, and senior files built the conceptual scaffolding. This file uses real libraries, real integration points, real monitoring, and real failure modes you will encounter in a Go service that handles millions of requests per day.

We will cover, in depth:

• cenkalti/backoff v4 — the most popular Go retry library, from its API down to its internal state machine.
• hashicorp/go-retryablehttp — wraps net/http with retry semantics.
• AWS SDK v2 retry — full-jitter, classifier, rate limiter, all in production code.
• gRPC retry — service config, retry throttling, deadline propagation.
• Circuit breaker composition with sony/gobreaker.
• OpenTelemetry tracing — spans per attempt, retry-aware traces.
• Prometheus metrics — what to emit, what to alert on, how to build dashboards.
• Structured logging with log/slog and how to balance noise vs signal.
• Real postmortem case studies — three anonymised incidents and what changed afterwards.
• An end-to-end example — a payments client built using all of the above.

After reading this file you will be able to:

• Drop a battle-tested retry library into a Go project and configure it correctly.
• Wire retry, breaker, budget, and metrics together in a single client.
• Read library source code and predict failure modes.
• Diagnose retry-related incidents using metrics and traces.
• Tune retry policies based on observed behaviour.
• Write production-quality Go retry code that you would put your name on.

This is the level expected of staff engineers and tech leads. The skills here distinguish "knows about retries" from "can architect a resilient distributed system in Go".

Prerequisites¶

• Complete the junior, middle, and senior files.
• Familiarity with Go modules, package structure, go.mod/go.sum.
• Familiarity with net/http client and server.
• Some exposure to gRPC, even if just the "hello world".
• Familiarity with Prometheus and basic metric types (counter, histogram, gauge).
• Familiarity with OpenTelemetry concepts (span, trace, attributes).
• Familiarity with log/slog or a structured logger like zap.

If you have not used these libraries before, plan to spend a couple of hours reading their docs. This file references them frequently.

Glossary¶

Cenkalti/Backoff Walkthrough¶

github.com/cenkalti/backoff/v4 is the most popular Go retry library. It is small (a few hundred lines), well-tested, and the API has been stable for years. Let us walk through it in depth.

Installation¶

go get github.com/cenkalti/backoff/v4

Import:

import "github.com/cenkalti/backoff/v4"

The simplest usage¶

operation := func() error {
    return doSomething()
}
err := backoff.Retry(operation, backoff.NewExponentialBackOff())

That is the whole API for trivial use cases. Retry keeps calling operation until it succeeds, or the backoff schedule signals "stop".

What NewExponentialBackOff returns¶

type ExponentialBackOff struct {
    InitialInterval     time.Duration  // 500ms by default
    RandomizationFactor float64         // 0.5 by default
    Multiplier          float64         // 1.5 by default
    MaxInterval         time.Duration  // 60s by default
    MaxElapsedTime      time.Duration  // 15 minutes by default
    Stop                time.Duration  // -1 (no early stop)
    Clock               Clock          // real clock by default

    currentInterval time.Duration
    startTime       time.Time
    random          *rand.Rand
}

The defaults: - Initial interval: 500ms. - Randomisation factor: 0.5 (i.e. uniform on [currentInterval * 0.5, currentInterval * 1.5]). - Multiplier: 1.5 (each interval is 1.5× the previous). - Max interval: 60 seconds. - Max elapsed time: 15 minutes (after this, give up).

These defaults are conservative. For most user-facing code you want shorter values:

b := backoff.NewExponentialBackOff()
b.InitialInterval = 100 * time.Millisecond
b.MaxInterval = 5 * time.Second
b.MaxElapsedTime = 30 * time.Second

The schedule¶

Each call to b.NextBackOff() returns the next delay. The schedule (no randomisation, for clarity):

0:  500ms
1:  750ms
2:  1.125s
3:  1.687s
4:  2.531s
5:  3.797s
6:  5.696s
...

With randomisation factor 0.5, each actual delay is in [interval * 0.5, interval * 1.5].

After 15 minutes of total elapsed time, NextBackOff returns backoff.Stop (a sentinel -1). The retry loop terminates.

Retry internals¶

The source of backoff.Retry:

func Retry(o Operation, b BackOff) error {
    return RetryNotify(o, b, nil)
}

func RetryNotify(operation Operation, b BackOff, notify Notify) error {
    return RetryNotifyWithTimer(operation, b, notify, nil)
}

func RetryNotifyWithTimer(operation Operation, b BackOff, notify Notify, t Timer) error {
    var err error
    var next time.Duration
    if t == nil {
        t = &defaultTimer{}
    }
    defer t.Stop()

    ctx := getContext(b)

    b.Reset()
    for {
        if err = operation(); err == nil {
            return nil
        }
        var permanent *PermanentError
        if errors.As(err, &permanent) {
            return permanent.Err
        }
        if next = b.NextBackOff(); next == Stop {
            if cerr := ctx.Err(); cerr != nil {
                return cerr
            }
            return err
        }
        if notify != nil {
            notify(err, next)
        }
        t.Start(next)
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-t.C():
        }
    }
}

Reading this carefully:

1. The loop is unbounded; termination depends on NextBackOff returning Stop.
2. The operation is called, then evaluated.
3. errors.As(err, &permanent) checks if the caller used backoff.Permanent(err) to signal "do not retry".
4. b.NextBackOff() returns the next delay or Stop.
5. The notify callback (if set) lets you log or metric each retry.
6. A select over the timer and context.

The context integration is via BackOffContext:

ctx, cancel := context.WithTimeout(parent, 30*time.Second)
defer cancel()

bWithCtx := backoff.WithContext(backoff.NewExponentialBackOff(), ctx)
err := backoff.Retry(operation, bWithCtx)

Now the loop respects ctx.Done().

backoff.Permanent¶

err := backoff.Retry(func() error {
    resp, err := http.Get(url)
    if err != nil {
        return err // transient
    }
    if resp.StatusCode >= 400 && resp.StatusCode < 500 && resp.StatusCode != 429 {
        return backoff.Permanent(fmt.Errorf("status %d", resp.StatusCode))
    }
    return nil
}, b)

The wrapping causes Retry to surface the error immediately.

RetryNotify for metrics¶

notify := func(err error, d time.Duration) {
    log.Printf("retry after %v: %v", d, err)
    retryCounter.Inc()
}
err := backoff.RetryNotify(operation, b, notify)

Each retry triggers the callback. Convenient for logging and metrics.

ConstantBackOff¶

For testing or specific use cases:

b := backoff.NewConstantBackOff(100 * time.Millisecond)
err := backoff.Retry(op, b)

Each delay is the same. Not exponential; mostly useful for tests.

WithMaxRetries¶

b := backoff.WithMaxRetries(backoff.NewExponentialBackOff(), 5)

Wraps another BackOff with a max-retries decorator. After 5 attempts, returns Stop.

BackOffContext¶

b := backoff.WithContext(backoff.NewExponentialBackOff(), ctx)

Returns a BackOff that returns Stop if the context is done.

Composing¶

You can compose the wrappers:

b := backoff.NewExponentialBackOff()
b.InitialInterval = 100 * time.Millisecond
b.MaxInterval = 5 * time.Second
bWithMax := backoff.WithMaxRetries(b, 5)
bWithCtx := backoff.WithContext(bWithMax, ctx)

err := backoff.RetryNotify(operation, bWithCtx, notify)

This is "exponential, capped at 5s, max 5 retries, context-aware, with notify callback for metrics". A production-grade retry in 4 lines.

Caveats¶

• The randomisation factor is symmetric: delay = interval ± interval * factor. So with factor 0.5, the delay is uniform on [0.5 * interval, 1.5 * interval]. This is closer to "equal jitter" than "full jitter".
• The default MaxElapsedTime of 15 minutes is far too long for user-facing operations. Set it explicitly.
• backoff.Retry does not pass the context to the operation. You must capture it in a closure.
• The defaultTimer uses time.NewTimer and time.Stop, properly. No leak.

Recommended setup¶

func NewBackoff() backoff.BackOff {
    b := backoff.NewExponentialBackOff()
    b.InitialInterval = 100 * time.Millisecond
    b.MaxInterval = 5 * time.Second
    b.MaxElapsedTime = 30 * time.Second
    return b
}

func Retry(ctx context.Context, op func() error) error {
    return backoff.RetryNotify(
        op,
        backoff.WithContext(NewBackoff(), ctx),
        func(err error, d time.Duration) {
            slog.Debug("retrying", "err", err, "delay", d)
            retryCounter.Inc()
        },
    )
}

Drop this into your codebase. It is production-ready.

Hashicorp Go-Retryablehttp¶

github.com/hashicorp/go-retryablehttp wraps net/http.Client with retry. It is HashiCorp's standard for client-side retry.

Installation¶

go get github.com/hashicorp/go-retryablehttp

Basic usage¶

import "github.com/hashicorp/go-retryablehttp"

client := retryablehttp.NewClient()
client.RetryMax = 5
client.RetryWaitMin = 100 * time.Millisecond
client.RetryWaitMax = 5 * time.Second

resp, err := client.Get("https://example.com")

That is it. The returned *http.Response is a normal Go HTTP response.

What it retries on¶

By default: - Network errors. - HTTP 500 and above. - HTTP 429.

You can override with client.CheckRetry:

client.CheckRetry = func(ctx context.Context, resp *http.Response, err error) (bool, error) {
    if err != nil {
        return true, nil // network error
    }
    if resp.StatusCode == 429 || (resp.StatusCode >= 500 && resp.StatusCode <= 599) {
        return true, nil
    }
    return false, nil
}

The backoff function¶

By default, DefaultBackoff does exponential with jitter:

func DefaultBackoff(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration {
    if resp != nil {
        if resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode == http.StatusServiceUnavailable {
            if s, ok := resp.Header["Retry-After"]; ok {
                if sleep, err := strconv.ParseInt(s[0], 10, 64); err == nil {
                    return time.Second * time.Duration(sleep)
                }
            }
        }
    }
    mult := math.Pow(2, float64(attemptNum)) * float64(min)
    sleep := time.Duration(mult)
    if float64(sleep) != mult || sleep > max {
        sleep = max
    }
    return sleep
}

Note that this is min * 2^attemptNum without jitter (the standard exponential). For jitter, you can replace with:

client.Backoff = func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration {
    if resp != nil && resp.Header.Get("Retry-After") != "" {
        if secs, err := strconv.ParseInt(resp.Header.Get("Retry-After"), 10, 64); err == nil {
            return time.Duration(secs) * time.Second
        }
    }
    cap := min * time.Duration(1<<attemptNum)
    if cap > max {
        cap = max
    }
    return time.Duration(rand.Int63n(int64(cap))) // full jitter
}

Request body re-reading¶

retryablehttp automatically wraps the body so it can be re-read on retry. You can pass io.Reader, []byte, or a *retryablehttp.ReaderFunc:

body := []byte(`{"key": "value"}`)
req, err := retryablehttp.NewRequest("POST", url, body)

Internally, the library reads the body once and replays it on each attempt. No body-already-consumed bug.

Logger¶

client.Logger = log.New(os.Stderr, "", log.LstdFlags)

Logs each attempt.

Conclusion¶

hashicorp/go-retryablehttp is a drop-in for http.Client with retry. If you do not want to roll your own, this is the recommended choice.

AWS SDK V2 Retry Internals¶

The AWS SDK has a sophisticated retry implementation. Worth studying because it represents industry best practice from a team that operates massive infrastructure.

Configuration¶

import (
    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/aws/retry"
    "github.com/aws/aws-sdk-go-v2/config"
)

cfg, err := config.LoadDefaultConfig(ctx,
    config.WithRetryer(func() aws.Retryer {
        return retry.NewStandard(func(o *retry.StandardOptions) {
            o.MaxAttempts = 5
            o.MaxBackoff = 5 * time.Second
        })
    }),
)

retry.Standard¶

The Standard retryer:

• Uses full jitter.
• Has a retry token bucket: 5 tokens per failed call, 1 token per retry. Empty bucket = no retry.
• Configurable max attempts (default 3).
• Configurable max backoff (default 20s).
• Configurable retryables (errors and status codes that trigger retry).

Source code highlights¶

The token bucket is at aws/retry/retryable_error.go:

type tokenRetryQuota interface {
    HasRetryToken() bool
    RetrieveToken() error
    ReleaseToken(...) 
}

Each retry attempts to retrieve a token. If empty, retry is skipped.

The token system protects against retry storms when many calls fail simultaneously.

The standard retryer's algorithm¶

func (s *Standard) RetryDelay(attempt int, err error) (time.Duration, error) {
    return s.backoff.BackoffDelay(attempt, err)
}

Where backoff is NewExponentialJitterBackoff from aws-sdk-go-v2/aws/retry. The formula:

func (j *ExponentialJitterBackoff) BackoffDelay(attempt int, err error) (time.Duration, error) {
    delay := time.Duration(1<<uint(attempt)) * j.MaxBackoff
    if delay > j.MaxBackoff {
        delay = j.MaxBackoff
    }
    jitterDelay, err := j.rand.Int63n(int64(delay))
    if err != nil {
        return 0, err
    }
    return time.Duration(jitterDelay), nil
}

This is full jitter, with the cap being MaxBackoff. Notice the rand.Int63n — same primitive as our hand-rolled jitter.

Classifier¶

The classifier decides which errors are retryable:

type Retryables []IsErrorRetryable

func (rs Retryables) IsErrorRetryable(err error) aws.Ternary {
    for _, r := range rs {
        if v := r.IsErrorRetryable(err); v != aws.UnknownTernary {
            return v
        }
    }
    return aws.UnknownTernary
}

Standard retryables include RetryableConnectionError, RetryableHTTPStatusCode, RetryableErrorCode. Each is a struct with rules.

Custom classification:

o.Retryables = append(o.Retryables, retry.IsErrorRetryableFunc(func(err error) aws.Ternary {
    if strings.Contains(err.Error(), "my custom transient") {
        return aws.TrueTernary
    }
    return aws.UnknownTernary
}))

Why study AWS's retry?¶

Two reasons:

1. It is the largest production deployment of retry in the Go ecosystem.
2. It illustrates the patterns we have discussed: full jitter, token bucket, classifier, max attempts. All in one library.

If you ever wonder "is this retry design right?", check what AWS does.

Integration With net/http¶

You have two paths: use hashicorp/go-retryablehttp or wrap your own.

Approach 1: hashicorp library (recommended)¶

import "github.com/hashicorp/go-retryablehttp"

client := retryablehttp.NewClient()
client.RetryMax = 3
client.RetryWaitMin = 100 * time.Millisecond
client.RetryWaitMax = 5 * time.Second
client.Logger = nil // disable noisy logging

You now have an HTTP client with retry. Use it like *http.Client:

resp, err := client.Get(url)
// or
req, _ := retryablehttp.NewRequest("POST", url, body)
resp, err := client.Do(req)

Approach 2: custom wrapper¶

If you need more control:

package httpclient

import (
    "bytes"
    "context"
    "io"
    "net/http"
    "time"

    "github.com/cenkalti/backoff/v4"
)

type Client struct {
    HTTP *http.Client
    NewBackoff func() backoff.BackOff
}

func New() *Client {
    return &Client{
        HTTP: &http.Client{Timeout: 10 * time.Second},
        NewBackoff: func() backoff.BackOff {
            b := backoff.NewExponentialBackOff()
            b.InitialInterval = 100 * time.Millisecond
            b.MaxInterval = 5 * time.Second
            b.MaxElapsedTime = 30 * time.Second
            return b
        },
    }
}

func (c *Client) Do(ctx context.Context, req *http.Request) (*http.Response, error) {
    var body []byte
    if req.Body != nil {
        var err error
        body, err = io.ReadAll(req.Body)
        if err != nil {
            return nil, err
        }
        req.Body.Close()
    }

    var resp *http.Response
    b := backoff.WithContext(c.NewBackoff(), ctx)
    err := backoff.Retry(func() error {
        var bodyReader io.Reader
        if body != nil {
            bodyReader = bytes.NewReader(body)
        }
        attemptReq, err := http.NewRequestWithContext(ctx, req.Method, req.URL.String(), bodyReader)
        if err != nil {
            return backoff.Permanent(err)
        }
        attemptReq.Header = req.Header.Clone()

        r, err := c.HTTP.Do(attemptReq)
        if err != nil {
            return err // transient
        }
        if r.StatusCode == 429 || (r.StatusCode >= 500 && r.StatusCode <= 599) {
            r.Body.Close()
            return fmt.Errorf("status %d", r.StatusCode)
        }
        if r.StatusCode >= 400 {
            return backoff.Permanent(fmt.Errorf("status %d", r.StatusCode))
        }
        resp = r
        return nil
    }, b)
    return resp, err
}

This is similar to what hashicorp/go-retryablehttp does internally. The advantage of rolling your own is fitting your specific conventions (logging, metrics, etc.).

Request body considerations¶

Both approaches need to handle the body. bytes.NewReader is seekable; for arbitrary io.Reader, you must read it all into memory first. For very large bodies, retry is impractical.

Best practice: keep bodies under 1 MB and read them into memory. For larger bodies, design the API to be resumable or accept that retry is impossible.

Connection pool considerations¶

net/http.Transport keeps connections alive. A retried request may use a different underlying connection. This is usually fine but can confuse debugging (the original request used connection A; the retry used connection B; both connections look healthy in netstat).

For HTTPS, connection reuse is important — TLS handshakes are expensive. Do not disable keep-alives unless you have specific reasons.

Integration With gRPC¶

gRPC has built-in retry support. Configure via service config.

Service config JSON¶

{
  "methodConfig": [{
    "name": [{"service": "myapp.Service"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "5s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE", "DEADLINE_EXCEEDED"]
    }
  }]
}

Loading the config in Go¶

import "google.golang.org/grpc"

svc := `{
  "methodConfig": [{
    "name": [{"service": "myapp.Service"}],
    "retryPolicy": {
      "maxAttempts": 4,
      "initialBackoff": "0.1s",
      "maxBackoff": "5s",
      "backoffMultiplier": 2,
      "retryableStatusCodes": ["UNAVAILABLE"]
    }
  }]
}`

conn, err := grpc.NewClient("dns:///service",
    grpc.WithDefaultServiceConfig(svc),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

That is all. gRPC retries any matching call automatically.

Throttling¶

Add retry throttling:

{
  "retryThrottling": {
    "maxTokens": 10,
    "tokenRatio": 0.1
  },
  "methodConfig": [...]
}

Each successful call adds 0.1 tokens. Each failed retry costs 1 token. When tokens drop below 5, retries are denied.

Per-method config¶

You can have different policies per method:

{
  "methodConfig": [
    {
      "name": [{"service": "myapp.Reads", "method": "Get"}],
      "retryPolicy": { ... }
    },
    {
      "name": [{"service": "myapp.Writes", "method": "Create"}],
      "retryPolicy": null  // no retry for writes
    }
  ]
}

Often you want reads retried aggressively and writes only with idempotency keys.

Deadline propagation¶

ctx, cancel := context.WithTimeout(parent, 5*time.Second)
defer cancel()
resp, err := client.MyMethod(ctx, req)

The 5-second deadline is sent as grpc-timeout metadata. The server reads it. Downstream gRPC calls automatically inherit.

If retries are happening, the total time including retries is bounded by the deadline.

Retryable status codes¶

DEADLINE_EXCEEDED is tricky: a server-side deadline-exceeded is retryable (the server might be faster next time), but a client-side deadline-exceeded is not (the caller has given up).

Service Config and Retry¶

A deeper look at gRPC's retry config.

How gRPC retries internally¶

1. Client calls client.MyMethod(ctx, req).
2. gRPC stub checks the retry policy from service config.
3. Stub calls the server. If the call fails with a retryable status code, gRPC waits per the policy and retries.
4. Each retry decrements the retry-throttling token.
5. After maxAttempts total attempts (including the first), gRPC returns the final error.

The client does not see individual attempts. From the client's perspective, the call either succeeds or returns the final error after all attempts.

Idempotent semantics in gRPC¶

gRPC retries assume idempotency. If your method is not idempotent, configure maxAttempts: 1 (no retry) or accept the risk.

Some services use a request-level idempotency key in the proto:

message ChargeRequest {
  string idempotency_key = 1;
  // ...
}

The server deduplicates based on the key.

Hedging vs retry in gRPC¶

gRPC supports both. Hedging is configured separately:

{
  "methodConfig": [{
    "name": [{"service": "myapp.Reads"}],
    "hedgingPolicy": {
      "maxAttempts": 3,
      "hedgingDelay": "0.1s",
      "nonFatalStatusCodes": []
    }
  }]
}

Hedging sends duplicates after the delay. First success wins. The two patterns are mutually exclusive per method.

Tuning service config¶

Start conservative. Measure. Tune. Common pattern:

1. Deploy with maxAttempts: 1 (no retry). Observe failure rates.
2. Add maxAttempts: 3 with full retry policy. Observe retry rate.
3. If retry rate is low and success-after-retry is high, increase max attempts.
4. If retry rate is high, lower the threshold for retry throttling.

This is empirical tuning. There is no universal right answer.

gRPC Interceptors For Retry¶

If you need behaviour beyond what service config provides — custom backoff, custom classifiers, integration with your retry library — write an interceptor.

Unary interceptor¶

import (
    "context"
    "google.golang.org/grpc"
    "github.com/cenkalti/backoff/v4"
)

func RetryUnaryInterceptor(b backoff.BackOff) grpc.UnaryClientInterceptor {
    return func(ctx context.Context, method string, req, reply interface{}, cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
        bCtx := backoff.WithContext(b, ctx)
        return backoff.Retry(func() error {
            err := invoker(ctx, method, req, reply, cc, opts...)
            if err == nil {
                return nil
            }
            if !isRetryable(err) {
                return backoff.Permanent(err)
            }
            return err
        }, bCtx)
    }
}

func isRetryable(err error) bool {
    st, ok := status.FromError(err)
    if !ok {
        return false
    }
    switch st.Code() {
    case codes.Unavailable, codes.DeadlineExceeded, codes.Internal:
        return true
    }
    return false
}

Use:

b := backoff.NewExponentialBackOff()
b.InitialInterval = 100 * time.Millisecond
b.MaxInterval = 5 * time.Second
b.MaxElapsedTime = 30 * time.Second

conn, err := grpc.NewClient("dns:///service",
    grpc.WithUnaryInterceptor(RetryUnaryInterceptor(b)),
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

Now every unary call on this connection is retried.

Stream interceptors¶

Streaming RPCs are harder. The retry needs to know how to replay the client side of the stream. If the client has already sent N messages, the retry must resend them. This is application-specific.

Best practice: do not retry streaming RPCs in the interceptor. Build retry into the application logic where you have the context.

Composing interceptors¶

Multiple interceptors compose:

conn, err := grpc.NewClient("dns:///service",
    grpc.WithChainUnaryInterceptor(
        TracingUnaryInterceptor(),
        MetricsUnaryInterceptor(),
        RetryUnaryInterceptor(b),
    ),
)

Order matters. Tracing should be outermost (so it captures retries as one logical span). Retry should be innermost relative to tracing but outer relative to anything that should run per-attempt.

Circuit Breaker Integration¶

A circuit breaker complements retry. Retry handles transient; breaker handles persistent.

Two-layer composition¶

type Client struct {
    Retrier *retry.Retrier
    Breaker *gobreaker.CircuitBreaker
}

func (c *Client) Do(ctx context.Context, op func(context.Context) error) error {
    return c.Retrier.Do(ctx, func(ctx context.Context) error {
        _, err := c.Breaker.Execute(func() (interface{}, error) {
            return nil, op(ctx)
        })
        if errors.Is(err, gobreaker.ErrOpenState) {
            return retry.MarkPermanent(err)
        }
        return err
    })
}

The breaker is inside the retrier. If the breaker opens, the retrier sees it as permanent and stops.

One failure event per retry sequence¶

A subtle point: the breaker should not record every retry as a separate failure. If 5 retries all fail, that is one user-visible failure.

The simplest way: only record the final outcome.

func (c *Client) Do(ctx context.Context, op func(context.Context) error) error {
    if state := c.Breaker.State(); state == gobreaker.StateOpen {
        return retry.MarkPermanent(gobreaker.ErrOpenState)
    }
    err := c.Retrier.Do(ctx, op)
    c.Breaker.Counts() // or a custom record method
    return err
}

But gobreaker records via Execute. Workaround: wrap the retrier in Execute:

func (c *Client) Do(ctx context.Context, op func(context.Context) error) error {
    _, err := c.Breaker.Execute(func() (interface{}, error) {
        return nil, c.Retrier.Do(ctx, op)
    })
    return err
}

Now the breaker sees the retrier's final outcome — success or failure — as one event.

Trade-offs¶

• Breaker outside retrier: one event per user request. Breaker opens slowly.
• Breaker inside retrier: one event per attempt. Breaker opens faster.

The "outside" composition is usually right. The "inside" composition is useful if you want the breaker to react quickly to a series of failures within one user request — which usually means the breaker thresholds are too low.

Sony/Gobreaker In Depth¶

github.com/sony/gobreaker is the standard Go circuit breaker. ~400 lines, well-tested, simple API.

States¶

• Closed: requests go through.
• Open: requests are rejected with ErrOpenState.
• Half-open: a few requests go through; success closes; failure re-opens.

Settings¶

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "stripe",
    MaxRequests: 5,                // in half-open state, how many to allow
    Interval:    60 * time.Second, // window for counting failures in closed state
    Timeout:     30 * time.Second, // time in open state before going half-open
    ReadyToTrip: func(c gobreaker.Counts) bool {
        return c.Requests >= 20 && float64(c.TotalFailures)/float64(c.Requests) > 0.5
    },
})

The ReadyToTrip callback decides when to open. The argument is the current count of requests, total failures, consecutive failures, etc.

A common pattern: open if at least 20 requests in the window and failure rate exceeds 50%.

Execute¶

result, err := cb.Execute(func() (interface{}, error) {
    return doSomething()
})
if err != nil {
    if errors.Is(err, gobreaker.ErrOpenState) {
        // breaker is open
    } else {
        // operation failed
    }
}

Execute records success/failure automatically and may transition states.

State observability¶

state := cb.State() // Closed, HalfOpen, or Open
counts := cb.Counts() // current counters

Useful for metrics.

State transitions¶

You can hook into transitions:

cb := gobreaker.NewCircuitBreaker(gobreaker.Settings{
    OnStateChange: func(name string, from, to gobreaker.State) {
        slog.Info("breaker state change", "name", name, "from", from, "to", to)
        breakerStateChangeCounter.WithLabelValues(name, to.String()).Inc()
    },
})

Alert when breaker opens. Critical operational signal.

Picking thresholds¶

The threshold is workload-specific:

• High-volume, low-failure-rate service: 50% failure over 20 requests is a sharp drop.
• Low-volume service: may need a smaller request count (e.g. 5 requests) for the breaker to trigger.

Start conservative (high threshold), tune based on observation.

Open duration¶

The Timeout is how long the breaker stays open. After this, it goes to half-open and probes.

• Too short: thrashing. Open, close, open, close.
• Too long: takes too long to recover after the dependency comes back.

30 seconds is a common default. Adjust based on the dependency's MTTR.

Half-open behaviour¶

In half-open, the breaker allows MaxRequests probes. If all succeed, the breaker closes. If any fail, it re-opens.

MaxRequests = 5 is a reasonable default. Too low and you cannot reliably detect recovery; too high and you let too many requests through during testing.

Per-dependency breakers¶

You should have one breaker per dependency, not a single global breaker:

type Breakers struct {
    Stripe   *gobreaker.CircuitBreaker
    Internal *gobreaker.CircuitBreaker
    Db       *gobreaker.CircuitBreaker
}

A Stripe outage should not stop your DB calls.

Resilience4j-Style Composition¶

Resilience4j (Java library) popularised "decorator stack" composition. The same approach works in Go.

The composition stack¶

A typical stack:

fallback(
  cache(
    retry(
      breaker(
        bulkhead(
          rate_limiter(
            timeout(call)
          )
        )
      )
    )
  )
)

Reading outside-in: fallback gives a default; cache short-circuits; retry handles transient; breaker fails fast; bulkhead limits concurrency; rate limiter caps RPS; timeout bounds per-call.

Go implementation¶

package resilience

import (
    "context"
    "time"

    "github.com/cenkalti/backoff/v4"
    "github.com/sony/gobreaker"
    "golang.org/x/time/rate"
    "golang.org/x/sync/semaphore"
)

type Chain struct {
    Timeout    time.Duration
    RateLimit  *rate.Limiter
    Bulkhead   *semaphore.Weighted
    Breaker    *gobreaker.CircuitBreaker
    BackOffNew func() backoff.BackOff
    Cache      func(ctx context.Context, key string) (interface{}, bool)
    Fallback   func(ctx context.Context, key string) (interface{}, error)
}

func (c *Chain) Do(ctx context.Context, key string, op func(context.Context) (interface{}, error)) (interface{}, error) {
    // Cache
    if c.Cache != nil {
        if v, ok := c.Cache(ctx, key); ok {
            return v, nil
        }
    }

    // Rate limit
    if c.RateLimit != nil {
        if err := c.RateLimit.Wait(ctx); err != nil {
            if c.Fallback != nil {
                return c.Fallback(ctx, key)
            }
            return nil, err
        }
    }

    // Bulkhead
    if c.Bulkhead != nil {
        if err := c.Bulkhead.Acquire(ctx, 1); err != nil {
            if c.Fallback != nil {
                return c.Fallback(ctx, key)
            }
            return nil, err
        }
        defer c.Bulkhead.Release(1)
    }

    // Timeout per attempt
    var result interface{}
    err := backoff.Retry(func() error {
        attemptCtx, cancel := context.WithTimeout(ctx, c.Timeout)
        defer cancel()

        // Breaker
        var err error
        result, err = c.Breaker.Execute(func() (interface{}, error) {
            return op(attemptCtx)
        })
        return err
    }, backoff.WithContext(c.BackOffNew(), ctx))

    if err != nil && c.Fallback != nil {
        return c.Fallback(ctx, key)
    }
    return result, err
}

This is a heavyweight composition; not needed for every call. Use for critical-path dependencies.

Decorator vs builder¶

Resilience4j uses decorators (function composition). Go's syntactic ceremony makes this awkward. The struct-based approach above is more idiomatic Go.

You could also use middleware-style:

type Middleware func(next OpFunc) OpFunc

retry := func(next OpFunc) OpFunc {
    return func(ctx context.Context) (interface{}, error) {
        return backoff.Retry(...)
    }
}
breaker := func(next OpFunc) OpFunc { ... }

op = retry(breaker(timeout(call)))

This is closer to Resilience4j. Use whichever feels natural.

Observability — Metrics¶

Production retry requires metrics. Without them, you cannot tell when retries are happening or whether they are helping.

What to measure¶

Prometheus client¶

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    attempts = promauto.NewCounterVec(
        prometheus.CounterOpts{Name: "retry_attempts_total"},
        []string{"client", "op", "outcome"},
    )
    attemptAtSuccess = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "retry_attempt_at_success",
            Buckets: []float64{1, 2, 3, 5, 10},
        },
        []string{"client", "op"},
    )
)

func (r *Retrier) Do(ctx context.Context, op func(context.Context) error) error {
    client := r.Name
    operation := getOpName(ctx)
    var lastErr error
    for attempt := 0; attempt < r.MaxAttempts; attempt++ {
        err := op(ctx)
        if err == nil {
            attempts.WithLabelValues(client, operation, "success").Inc()
            attemptAtSuccess.WithLabelValues(client, operation).Observe(float64(attempt + 1))
            return nil
        }
        attempts.WithLabelValues(client, operation, "failure").Inc()
        lastErr = err
        // ... rest of retry logic
    }
    return lastErr
}

Building dashboards¶

A retry dashboard should show:

1. Retry rate. sum(rate(retry_attempts_total{outcome="failure"}[5m])) by (client).
2. Retry ratio. sum(rate(retry_attempts_total[5m])) / sum(rate(retry_successes_total[5m])).
3. Success-at-attempt histogram. What fraction of requests succeed at attempt 1, 2, 3, etc.
4. Budget denial rate. sum(rate(retry_budget_denied_total[5m])).
5. Breaker state. breaker_state over time.

Each panel should have alerts:

• Retry rate > 5% — warn.
• Budget denial > 0 — page.
• Breaker open > 60s — page.
• Avg attempts > 1.5 — warn.

Cardinality¶

Be careful with labels. op should be a method name (e.g. Charge), not a URL with IDs. Otherwise cardinality explodes.

A op label with ~50 distinct values is fine. A op label with 50,000 values is a memory bomb.

Observability — Tracing¶

OpenTelemetry traces let you see retries in context.

Setup¶

import "go.opentelemetry.io/otel"

tracer := otel.Tracer("retry")

Per-retry spans¶

func (r *Retrier) Do(ctx context.Context, op func(context.Context) error) error {
    ctx, parentSpan := tracer.Start(ctx, "retry.Do",
        trace.WithAttributes(
            attribute.String("client", r.Name),
            attribute.Int("max_attempts", r.MaxAttempts),
        ),
    )
    defer parentSpan.End()

    for attempt := 0; attempt < r.MaxAttempts; attempt++ {
        attemptCtx, attemptSpan := tracer.Start(ctx, fmt.Sprintf("retry.attempt.%d", attempt+1))
        err := op(attemptCtx)
        if err != nil {
            attemptSpan.RecordError(err)
            attemptSpan.SetStatus(codes.Error, err.Error())
        }
        attemptSpan.End()
        if err == nil {
            parentSpan.SetAttributes(attribute.Int("successful_attempt", attempt+1))
            return nil
        }
        // ... retry logic ...
    }
    return nil
}

In your tracing UI you see:

[──────── retry.Do ─────────]
   [─ attempt 1 ─]
              [─ attempt 2 ─]
                        [─ attempt 3 ─] (success)

Each attempt's sub-spans (HTTP request, DB query, etc.) appear under the attempt span.

Linking retries¶

Modern tracing (OTel 1.0+) supports span links. The parent span gathers attributes; each attempt is a child. The retry mechanism is visible.

Sampling¶

In high-volume systems, you cannot trace every request. Sample 1% (or less) for traces. Retries are still recorded in metrics for 100% of requests.

For incident response, you may temporarily enable 100% tracing on a specific endpoint.

Tracing across services¶

If you call another service via gRPC or HTTP, the tracing context propagates automatically (via traceparent header). The downstream service's spans appear in the same trace.

Observability — Logging¶

Structured logs complement metrics and traces.

Use log/slog¶

import "log/slog"

logger := slog.Default()

Log levels¶

• DEBUG: every attempt.
• INFO: every retry-then-success.
• WARN: every give-up.
• ERROR: every breaker-open transition.

Structured fields¶

logger.Debug("retry attempt",
    slog.String("client", r.Name),
    slog.String("op", opName),
    slog.Int("attempt", attempt+1),
    slog.Any("err", err),
)

Correlation¶

Include the trace ID in every log:

span := trace.SpanFromContext(ctx)
logger := slog.With(
    slog.String("trace_id", span.SpanContext().TraceID().String()),
)

Now you can grep logs by trace ID and see all retry attempts for one user request.

Don't log secrets¶

The request body may have sensitive data (passwords, tokens). Never log it directly. Use a slog.Attr that redacts:

func RedactedRequest(req Request) slog.Attr {
    return slog.Group("request",
        slog.String("method", req.Method),
        slog.String("url", req.URL),
        slog.Int("body_size", len(req.Body)),
        // do not include req.Body
    )
}

Log volume¶

Even at DEBUG, logging every attempt is too much for high-volume systems. Sample:

if attempt > 0 && rand.Float64() < 0.01 {
    logger.Info("retry attempt", ...)
}

Or use a different log level for verbose retries.

Postmortem Case Studies¶

Three anonymised case studies. Each illustrates a specific lesson.

Case 1: The Idempotency Window Mismatch¶

Setup: A payments service used Stripe-style idempotency keys. The keys had a 1-hour TTL.

Incident: A network outage between the service and Stripe lasted 90 minutes. During the outage, requests retried (with the same key). Some retries arrived after Stripe processed the original but before the response got back. New retries arrived 70 minutes after the original — past the idempotency window. Stripe processed them as new charges.

Result: ~50 duplicate charges. Refunds issued. Customer trust damaged.

Lesson: Idempotency-key TTL must be longer than the worst-case retry duration plus any out-of-order processing window. After this incident, TTL increased to 24 hours.

Case 2: The Cascading Retry Storm¶

Setup: Service A → B → C → D, each with 3 retries on transient errors. No deadline propagation, no budget.

Incident: D's database had a 30-second hiccup. Each tier amplified: A made 3× requests to B; B made 3× requests to C; C made 3× requests to D. Effective load: 27× normal. D crashed under retry load.

Cascade: D was down. C retried, exhausting its retry budget. B retried, exhausting its budget. A retried, exhausting its budget. Service was effectively down for 20 minutes after the initial 30-second hiccup.

Lesson: Retries multiply across tiers. After this incident: - B and C disabled retries. - A added deadline propagation. - A added a retry budget. - All services added circuit breakers.

Case 3: The Silent Retry Eating The Budget¶

Setup: A monitoring service polled 10,000 endpoints every minute. Each poll had retry with 3 attempts, 100ms base, full jitter, 5s cap.

Incident: A network blip caused 30% of polls to fail. Retries kicked in. Each retry took ~500ms additional. The poller fell behind. New polls queued. The queue grew unbounded.

Result: The monitoring system stopped producing alerts. The team did not know systems were failing. Hours later, customer reports surfaced the issue.

Lesson: Retries can cause silent latency degradation that bypasses your alerts. After this incident: - The monitoring system added a polls_in_progress metric. - Alert when polls_in_progress > threshold. - The retry budget was tightened. - The poll latency p99 became an SLO with alerting.

Operational Runbook¶

When retry-related alerts fire, what do you do?

Alert: Retry rate spike¶

Diagnose: 1. Check the metric. Which client, which op? 2. Check dependency health: status page, dashboards. 3. Check error rates: are calls failing or just slow?

Mitigate: - If dependency is degraded but recovering: monitor. - If dependency is broken: consider failover to alternative. - If self-inflicted (e.g. recent deploy broke a client): rollback.

Resolve: - Document the cause in incident notes. - Update dashboards to make this easier next time.

Alert: Retry budget denied¶

Diagnose: 1. Confirm: is the dependency actually failing? 2. If yes: retry budget is doing its job. Continue. 3. If no: budget is too tight; legitimate retries are being denied.

Mitigate: - If budget is too tight: increase it. - If dependency is failing: kill switch to disable retries entirely, allowing the dependency to recover.

Resolve: - Tune budget based on observed retry rate.

Alert: Breaker open¶

Diagnose: 1. Confirm: which dependency? How long open? 2. Check dependency: is it actually down?

Mitigate: - If dependency is down: monitor; breaker will half-open in Timeout seconds. - If dependency is up but breaker is mis-configured: investigate ReadyToTrip logic.

Resolve: - Adjust breaker thresholds if mis-configured. - Document for postmortem.

Alert: Tail latency spike¶

Diagnose: 1. Is the tail latency from retries or from base call latency? 2. Check retry_attempt_at_success histogram. Are more requests succeeding on attempt 2+?

Mitigate: - If retries are dominant: tighten retry policy (fewer attempts, smaller cap). - If base latency is the problem: retries are masking; address upstream.

Kill switch¶

type Switch struct {
    mu       sync.RWMutex
    disabled atomic.Bool
}

func (s *Switch) Disable() { s.disabled.Store(true) }
func (s *Switch) Enable()  { s.disabled.Store(false) }
func (s *Switch) IsOn() bool { return !s.disabled.Load() }

Wired into the retrier:

if !retrySwitch.IsOn() {
    return op(ctx) // no retry; surface error immediately
}

The switch is set via runtime config (etcd, Consul, feature flags). Flip during incidents.

Performance Tuning¶

For high-RPS systems, retry code can become a hot path. Tuning:

Avoid allocations in the loop¶

Pre-allocate the timer:

t := time.NewTimer(0)
defer t.Stop()
if !t.Stop() { <-t.C }
for attempt := 0; attempt < N; attempt++ {
    // ...
    t.Reset(delay)
    select {
    case <-t.C:
    case <-ctx.Done():
        return ctx.Err()
    }
}

Saves one time.NewTimer allocation per attempt.

Avoid lock contention on jitter¶

If you have many concurrent retries on the same *rand.Rand, contention shows up. Solutions:

• Use math/rand/v2 (Go 1.22+) — lock-free per-goroutine state.
• Use sync.Pool of *rand.Rand.

Avoid time.After in loops¶

We have covered this. Always use time.NewTimer + Stop.

Batch metrics¶

If you emit metrics on every attempt, the metric backend becomes the bottleneck. Use buffered counters (atomic increments) and flush periodically.

Avoid context.WithTimeout per attempt if possible¶

// expensive
for attempt := 0; attempt < N; attempt++ {
    ctx2, cancel := context.WithTimeout(ctx, perAttempt)
    err := op(ctx2)
    cancel()
}

If perAttempt does not need to vary per attempt, hoist the context out. Or use deadline propagation.

Hot path measurement¶

Benchmark your retry helper: