Cancellation Propagation — Senior Level¶

Table of Contents¶

1. Architecture View
2. Structured Concurrency in Go
3. Supervisor Trees and Restart Policies
4. The Shape of Real Production Pipelines
5. Cancellation Flow: Up, Down, Sideways
6. Drain Semantics in Detail
7. Backpressure and Cancellation Interplay
8. Cancellation Across Process Boundaries
9. Lifecycle Modelling and State Machines
10. Designing for Predictable Cancellation Latency
11. Cancellation in Streaming Systems
12. Race-Free Shutdown Protocols
13. Cancellation and Resource Lifetime
14. Choosing the Right Abstraction
15. Testing Strategies for Production-Grade Cancellation
16. Patterns from the Standard Library
17. Anti-Patterns at Scale
18. Cancellation Bugs in the Wild
19. Diagrams
20. Cheat Sheet
21. Summary
22. Further Reading

Architecture View¶

At the senior level, cancellation propagation stops being a per-stage concern and becomes a property of the system. The right question is no longer "does this stage exit on cancel?" but "what is the cancellation topology of the whole service, and where are its weakest links?"

A typical Go service has a multi-layered cancellation graph:

• A root context cancelled by SIGTERM, SIGINT, or an admin-triggered drain.
• One or more per-listener contexts (HTTP, gRPC, message-queue subscriber).
• One per-request context for each incoming call.
• One per-pipeline context inside each request.
• One per-call context for each outbound dependency.
• One per-task context for each background job, periodic worker, or scheduled task.

Every context inherits from one above. Cancelling any node cascades to its descendants. The shape is a forest, with context.Background() as the implicit root of every tree.

The architecture-level goals:

• Liveness. Every cancellation reaches every goroutine that depends on it, within a known time bound.
• Safety. No data loss without explicit acceptance; no double-close; no use-after-cancel of resources that require active state.
• Observability. Every cancellation has a recorded reason; latency from cancel-trigger to fully-stopped is measurable.
• Composability. New pipelines, workers, and listeners plug into the cancellation graph without manual rewiring.

The remainder of this file unpacks each goal and the patterns that achieve them.

A deeper look at the architectural goals¶

Each of the four goals — liveness, safety, observability, composability — has concrete patterns that achieve it. Let me unpack each.

Liveness in detail¶

Liveness is "the system makes progress and eventually stops when told." For cancellation, two sub-properties:

• Eventual cancellation: every goroutine that should stop, does stop, given enough time after cancel().
• Bounded latency: "enough time" is a known constant, ideally well under the shutdown SLA.

Eventual cancellation requires every blocking operation to have a path to cancellation. The minimum: every select has a <-ctx.Done() case. The maximum: every external I/O call uses a context-aware variant, every long inner loop polls ctx.Err().

Bounded latency requires measurement. You cannot promise a 30-second shutdown SLA if you have never measured the cancellation latency under realistic load. The discipline is:

1. Define the SLA (e.g. "fully stopped within 30 seconds of SIGTERM under any conditions").
2. Identify the slowest component (often per-item work or external I/O).
3. Either reduce its latency or accept it as the bottleneck and set the SLA accordingly.
4. Test under load that exercises the worst case.

Safety in detail¶

Safety is "nothing bad happens during cancellation." The threats:

• Double-close: closing a channel twice panics. Solution: single closer per channel.
• Use-after-close: sending to a closed channel panics. Solution: producer closes its own output; nobody else writes after close.
• Resource leaks: a goroutine that holds a connection, file, or lock and does not release it on cancellation. Solution: defer releases for everything held.
• Data loss: in-flight items dropped on cancellation. Solution: design to either tolerate drops or implement an ack/commit protocol.
• Partial state: a state machine left in a transient state (e.g. "half-flushed") on cancellation. Solution: explicit recovery on restart, idempotent operations.

Senior-level designs identify these threats up-front and engineer them away. The patterns are familiar — defer, WaitGroup, ack channels — but the discipline is the architectural commitment.

Observability in detail¶

You cannot fix what you cannot see. For cancellation:

• Cancellation reason: every cancel carries a cause. context.WithCancelCause is the primitive; logging and tracing carry it forward.
• Cancellation latency: histogram of time from cancel-trigger to last-goroutine-exited.
• Goroutine count: gauge of runtime.NumGoroutine, with alerts on drift.
• Stage-level metrics: counters for "cancelled mid-process" per stage.
• Per-goroutine traces: optional, but valuable for debugging — each goroutine logs its enter/exit with timestamps.

Together these form a "cancellation observability stack" that turns shutdown from a black box into a visible, debuggable process.

Composability in detail¶

A new pipeline should plug into the existing graph without manual wiring. The pattern: every component takes ctx context.Context as its first parameter, derives child contexts as needed, and respects cancellation. New components inherit the same discipline.

The shape-by-default: a function or struct that holds state and goroutines exposes a ctx context.Context argument in its constructor (and possibly a separate Run(ctx) method). The implementation derives an internal context for its own work but cancels it on parent cancel.

type Pool struct {
    ctx    context.Context
    cancel context.CancelFunc
    // ...
}

func NewPool(parent context.Context) *Pool {
    ctx, cancel := context.WithCancel(parent)
    return &Pool{ctx: ctx, cancel: cancel}
}

func (p *Pool) Close() { p.cancel() }

The pool's internal context derives from the parent; cancelling the parent cancels the pool. Close is also available for explicit shutdown. The pool composes with other components automatically.

Structured Concurrency in Go¶

Structured concurrency is the discipline of constraining goroutine lifetimes to lexical scopes. A function that spawns goroutines must wait for them before returning. The function's scope is the "structure" that contains the concurrency.

Go does not enforce structured concurrency at the language level. There is no await that forces a join, no with block that closes goroutines. But you can achieve it by convention, using errgroup, sync.WaitGroup, and defer cancel(). The patterns become muscle memory.

The structured idiom¶

func operation(parent context.Context) error {
    g, ctx := errgroup.WithContext(parent)
    g.Go(func() error { return stageA(ctx) })
    g.Go(func() error { return stageB(ctx) })
    return g.Wait() // every goroutine has exited before this returns
}

Wait guarantees that when operation returns, every goroutine it spawned has also returned. No leak, no race. The cancellation is automatic on any error.

Violations of structure¶

Spawning a goroutine that outlives the function:

func operation(ctx context.Context) {
    go forever(ctx) // never joined
    return
}

forever is now orphaned. If ctx is cancelled, it exits — but who waits for it? Nobody. The function returns; the goroutine continues until ctx cancels. This is acceptable only for true fire-and-forget tasks (logging, metrics emission), and even then the long-lived context should be explicit.

Promotion: escaping the scope deliberately¶

Sometimes a sub-operation should outlive the function. The principled way: separate the cancellation scope from the function scope.

func handler(parent context.Context, longLived context.Context) {
    if err := work(parent); err != nil {
        // log the failure even if parent's request is over
        go logFailure(longLived, err)
    }
}

The fire-and-forget logger uses longLived — a context tied to the application lifetime. It will outlive the handler but will still die on shutdown. The pattern is explicit: anyone reading the code sees that logFailure is detached.

context.WithoutCancel (Go 1.21+) formalises this: detached := context.WithoutCancel(parent) returns a context with parent's values but no parent cancellation.

Why structure matters at scale¶

A single unstructured goroutine in a request handler is usually harmless. Multiplied by 10 000 RPS and 30 days of uptime, it becomes a leak. Structured concurrency is the discipline that prevents accumulation. Every goroutine has a join point; every join point waits.

The cost is verbosity: you cannot write go work() and forget. The benefit is that the goroutine count of your service is bounded by the active operations, not by the cumulative history.

Comparison with other languages¶

• Kotlin coroutines enforce structure via CoroutineScope. Every coroutine belongs to a scope; cancelling the scope cancels all children.
• Trio in Python has "nurseries" — a context manager that owns its tasks.
• Java's ExecutorService has shutdown and awaitTermination — informal structure.
• Rust's tokio::spawn is unstructured by default; JoinHandle lets you join, but it is optional.

Go is closer to Rust than to Kotlin: structure is a convention, not a language feature. The patterns in this file are how you achieve it.

Structured concurrency: case studies¶

Case study A: an in-memory job runner¶

A function that runs a set of jobs in parallel, waits for all, returns the first error. Structured concurrency makes this trivial:

func runJobs(ctx context.Context, jobs []Job) error {
    g, ctx := errgroup.WithContext(ctx)
    g.SetLimit(8)
    for _, j := range jobs {
        j := j
        g.Go(func() error {
            return runJob(ctx, j)
        })
    }
    return g.Wait()
}

When runJobs returns, every job goroutine has returned. No leaks. Cancellation cascades naturally. This is the gold standard for structured concurrency in Go.

Case study B: a long-running daemon¶

A daemon that exposes a status API while running a background workload. The structure is more complex but still bounded:

func runDaemon(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)

    g.Go(func() error {
        return runStatusAPI(ctx)
    })

    g.Go(func() error {
        return runBackgroundWork(ctx)
    })

    return g.Wait()
}

Two goroutines, joined by g.Wait. If either returns an error, the other cancels. If ctx is cancelled externally, both stop. The daemon's caller knows that when runDaemon returns, the daemon is fully stopped.

Case study C: a request handler¶

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    g, ctx := errgroup.WithContext(ctx)

    var partA, partB Result
    g.Go(func() error {
        var err error
        partA, err = fetchA(ctx)
        return err
    })
    g.Go(func() error {
        var err error
        partB, err = fetchB(ctx)
        return err
    })

    if err := g.Wait(); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    json.NewEncoder(w).Encode(combine(partA, partB))
}

Fetch two parts in parallel, combine, respond. Structured concurrency: when g.Wait() returns, both fetches have returned. Cancellation cascades from client disconnect.

The data race on partA and partB is fine because g.Wait() synchronises — after Wait, the assignments in g.Go-spawned goroutines are visible to the caller.

Where structured concurrency breaks¶

The structured pattern requires "wait for every goroutine before returning." This conflicts with fire-and-forget tasks. Patterns to handle the conflict:

1. Promote the goroutine to a longer-lived scope. Use context.WithoutCancel(parent) or pass a long-lived context directly. The goroutine then has a join point in the long-lived scope (e.g. main).
2. Track the goroutine in a registry. A service-level "background tasks" registry that tracks every spawned task. Shutdown waits on all of them.
3. Accept the trade-off. For truly fire-and-forget tasks (a log line, a metric increment) the goroutine is short-lived and the leak is bounded. Document and move on.

A typical service has a few "background tasks" (metrics emitter, health checker) that outlive request handlers but live within the service lifetime. They are structured against the service, not against any request.

Supervisor Trees and Restart Policies¶

A long-lived service has goroutines that should be restarted on failure, not just cancelled. The supervisor pattern handles this.

Basic supervisor¶

func supervise(ctx context.Context, name string, fn func(context.Context) error) {
    for {
        select {
        case <-ctx.Done():
            return
        default:
        }
        if err := fn(ctx); err != nil {
            log.Printf("%s: %v, restarting", name, err)
        }
    }
}

A goroutine that runs fn, restarts on error, and exits cleanly on cancellation. Useful for background workers that should be resilient to transient failures.

With backoff¶

func superviseWithBackoff(ctx context.Context, name string, fn func(context.Context) error) {
    backoff := time.Second
    for {
        select {
        case <-ctx.Done():
            return
        default:
        }
        start := time.Now()
        if err := fn(ctx); err != nil {
            log.Printf("%s: %v", name, err)
        }
        if time.Since(start) > 10*time.Second {
            backoff = time.Second // reset if the task ran for a while
        } else {
            backoff *= 2
            if backoff > time.Minute {
                backoff = time.Minute
            }
        }
        select {
        case <-ctx.Done():
            return
        case <-time.After(backoff):
        }
    }
}

Crash-loop protection. The backoff grows on rapid failures and resets after a stable interval.

Tree of supervisors¶

A supervisor that supervises multiple supervisors:

func runService(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)
    g.Go(func() error {
        superviseWithBackoff(ctx, "queue-consumer", runConsumer)
        return ctx.Err()
    })
    g.Go(func() error {
        superviseWithBackoff(ctx, "scheduler", runScheduler)
        return ctx.Err()
    })
    g.Go(func() error {
        superviseWithBackoff(ctx, "http-server", runServer)
        return ctx.Err()
    })
    return g.Wait()
}

Three supervised tasks; each restarts independently. Cancellation from ctx shuts them all down.

One-for-one vs one-for-all¶

In Erlang OTP, supervisors choose a restart strategy: one-for-one (restart only the failed child), one-for-all (restart everyone), or rest-for-one (restart the failed child and everything that depends on it).

Go does not have these primitives built in, but you can implement them. The simplest mapping:

• One-for-one — each child has its own supervise goroutine. Default behaviour.
• One-for-all — when any child fails, cancel the parent context (which cancels all children), then start fresh:

func oneForAll(parent context.Context, children []func(context.Context) error) {
    for {
        select {
        case <-parent.Done():
            return
        default:
        }
        ctx, cancel := context.WithCancel(parent)
        g, ctx := errgroup.WithContext(ctx)
        for _, c := range children {
            c := c
            g.Go(func() error { return c(ctx) })
        }
        _ = g.Wait()
        cancel()
        time.Sleep(time.Second)
    }
}

Any child returning an error cancels the inner errgroup; all children stop; the supervisor sleeps and starts a new errgroup with all children. Crude but functional.

When to use restarts¶

Restarts are appropriate when:

• The work is idempotent (rerunning is safe).
• Failures are transient (network blips, momentary backend unavailability).
• The task is long-lived (a queue consumer, a scheduler).

Restarts are not appropriate when:

• The work has side effects that should not repeat.
• Failures are permanent (misconfiguration, missing dependency).
• The task should fail fast and let the orchestrator (Kubernetes, systemd) restart the whole process.

For most cloud-native Go services, "let the orchestrator restart the process" is the right answer. Internal supervisors are for goroutines that should outlive transient errors but cannot afford a full process restart.

Supervisor: detecting unhealthy restarts¶

A supervisor that restarts a failing task is useful, but a task that fails repeatedly (a crash loop) wastes resources and masks the bug. Add health monitoring:

type Health struct {
    Failures    int
    LastFailure time.Time
    LastSuccess time.Time
}

func superviseWithHealth(ctx context.Context, name string, fn func(context.Context) error, health *Health) {
    backoff := time.Second
    for {
        select {
        case <-ctx.Done():
            return
        default:
        }
        start := time.Now()
        err := fn(ctx)
        if err == nil {
            health.LastSuccess = time.Now()
            health.Failures = 0
            backoff = time.Second
        } else {
            health.LastFailure = time.Now()
            health.Failures++
            log.Printf("%s: %v (failures=%d)", name, err, health.Failures)
            if health.Failures > 10 {
                log.Printf("%s: too many failures, escalating", name)
                // signal alerting system, possibly exit
            }
        }
        if time.Since(start) > 30*time.Second {
            backoff = time.Second
        }
        select {
        case <-ctx.Done():
            return
        case <-time.After(backoff):
            backoff *= 2
            if backoff > time.Minute {
                backoff = time.Minute
            }
        }
    }
}

After 10 consecutive failures, the supervisor escalates. Options: trigger an alert, exit the process so the orchestrator restarts it, or just log loudly. The choice depends on operational preferences.

The lesson: a supervisor is more than a for loop. It is a small state machine that tracks health, applies backoff, and reports issues. Production code often externalises this into a library.

Reasoning about cancellation latency, by component¶

A breakdown of where latency typically lives.

select latency¶

A goroutine in select { case <-ctx.Done(): ... case <-other: ... } wakes within microseconds of ctx.Done() closing. The exact number depends on:

• How many goroutines are runnable at the time (scheduler congestion).
• Whether the goroutine was sleeping in a system call (slightly slower wake).
• GOMAXPROCS and the load on the running threads.

Typical: 1-10 microseconds. Worst case under heavy contention: 100 microseconds. This is usually negligible compared to other components.

Per-item work¶

If a stage processes one item in 1 ms, the average cancellation latency for that stage is 0.5 ms (uniformly distributed within the item). With ctx.Err() polling inside the work, this drops to the polling interval — typically tens of microseconds.

For 100 ms per item, the average is 50 ms; this can be reduced to single-digit milliseconds with polling.

For 1 second per item, the average is 500 ms; this is usually unacceptable for a strict shutdown SLA and you must either reduce the per-item work, add polling, or restructure.

External I/O¶

The slowest component. A blocking I/O call without context support can hold a goroutine for seconds or minutes.

• TCP read on a slow connection: until the connection times out.
• Database query without QueryContext: until the query completes.
• HTTP request without NewRequestWithContext: until the response or socket timeout.

Mitigations:

• Use the context-aware variant of every blocking call.
• Set explicit deadlines (SetReadDeadline, WriteTimeout).
• Have a watcher goroutine that closes the underlying resource on cancel.

A common pattern: every external call has a 5-second timeout via context.WithTimeout, and the call propagates the timeout to the protocol layer. After 5 seconds, the call returns with context.DeadlineExceeded and the goroutine exits.

Buffer flush¶

A buffered channel may hold pending items at the time of cancellation. The consumer may continue to drain (if the cancellation policy is "deliver pending") or skip (if "drop pending"). The drain time is buffer_size * per_item_consumer_time.

For small buffers (1-10) the time is negligible. For large buffers (1000+) it dominates. Design buffers small enough that drain time is acceptable.

Cumulative latency in a multi-stage pipeline¶

For a pipeline of N stages with cancellation latency L per stage, the total cancellation time is approximately L * N (if stages are sequential) or max(L_i) (if stages cancel in parallel).

In errgroup-driven pipelines, the cancel is broadcast to all stages simultaneously; each stage exits independently. Total latency is max(L_i), dominated by the slowest stage.

In sequential cleanup (stage 1 closes its output, stage 2 sees and exits, etc.), latency is the sum. Avoid this by using a shared context rather than serial close-propagation.

The Shape of Real Production Pipelines¶

Real production pipelines are not three stages on one machine. They are:

• A frontend (HTTP, gRPC, or queue subscriber).
• An internal queue or pipeline buffer.
• A pool of workers.
• One or more backend dependencies (databases, RPCs, cloud APIs).
• A result sink (another queue, a database, a response).
• A metrics emitter.
• A logger.
• A health-check endpoint.

Each of these is a participant in the cancellation graph. The challenge is wiring them so that one cancel signal flows everywhere, with predictable latency, no leaks, and no data loss.

A reference architecture¶

type Service struct {
    rootCtx    context.Context
    rootCancel context.CancelCauseFunc

    httpSrv *http.Server
    pool    *WorkerPool
    metrics *MetricsEmitter
    health  *HealthChecker
    db      *DBPool
}

func NewService(parent context.Context, cfg Config) (*Service, error) {
    rootCtx, rootCancel := context.WithCancelCause(parent)

    db, err := NewDBPool(rootCtx, cfg.DB)
    if err != nil {
        rootCancel(err)
        return nil, err
    }

    pool := NewWorkerPool(rootCtx, cfg.Workers, db)
    metrics := NewMetricsEmitter(rootCtx, cfg.Metrics)
    health := NewHealthChecker(rootCtx, pool)

    httpSrv := &http.Server{
        Addr:    cfg.Addr,
        Handler: buildHandler(pool, health),
        BaseContext: func(net.Listener) context.Context {
            return rootCtx
        },
    }

    return &Service{
        rootCtx:    rootCtx,
        rootCancel: rootCancel,
        httpSrv:    httpSrv,
        pool:       pool,
        metrics:    metrics,
        health:     health,
        db:         db,
    }, nil
}

func (s *Service) Run() error {
    g, _ := errgroup.WithContext(s.rootCtx)
    g.Go(func() error {
        if err := s.httpSrv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            return err
        }
        return nil
    })
    return g.Wait()
}

func (s *Service) Shutdown(deadline time.Duration) error {
    shutdownCtx, cancel := context.WithTimeout(context.Background(), deadline)
    defer cancel()

    // Stop accepting new
    if err := s.httpSrv.Shutdown(shutdownCtx); err != nil {
        log.Printf("server shutdown: %v", err)
    }

    // Drain workers
    s.pool.Drain()

    // Cancel root context
    s.rootCancel(errors.New("shutdown"))

    // Wait for background goroutines
    s.metrics.Wait()
    s.health.Wait()

    return s.db.Close()
}

The cancellation graph:

rootCtx (cancel cause: "shutdown" or initial error)
   ├── http server (BaseContext: rootCtx)
   │     └── per-request handlers (r.Context())
   │           └── pool.Submit -> worker -> db.QueryContext
   ├── worker pool
   ├── metrics emitter
   ├── health checker
   └── db pool (internal connection management)

Shutdown orchestrates:

1. HTTP server stops accepting; in-flight handlers see rootCtx cancel via r.Context().
2. Pool drains; workers finish in-flight jobs.
3. rootCancel cancels the metrics emitter, health checker, and any other background tasks.
4. The DB pool closes after all consumers are done.

Order matters. Closing the DB first would error out workers mid-query. Cancelling rootCtx first might race with the HTTP server's drain. Following the strict outside-in ordering (listener → workers → background → resources) avoids these races.

Per-component lifecycle: the Run/Stop interface¶

A common shape for components in a service:

type Component interface {
    Run(ctx context.Context) error
    Stop(ctx context.Context) error
}

Run blocks until ctx cancels or an error occurs; it returns the cancellation reason or the error. Stop is an optional graceful shutdown trigger that may signal Run to drain.

Implementations:

type Pool struct {
    parent  context.Context
    cancel  context.CancelFunc
    drained chan struct{}
}

func (p *Pool) Run(ctx context.Context) error {
    p.parent, p.cancel = context.WithCancel(ctx)
    p.drained = make(chan struct{})
    defer close(p.drained)
    // ... main loop, blocks until p.parent.Done() ...
    return p.parent.Err()
}

func (p *Pool) Stop(ctx context.Context) error {
    p.cancel()
    select {
    case <-p.drained:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

Run is invoked once when the service starts; it blocks until Stop (or the parent context) triggers cancellation. Stop is invoked during shutdown and waits for Run to finish, with its own deadline.

This pattern composes: a service has many components, each with Run/Stop; the service's main is:

func main() {
    rootCtx, rootCancel := signal.NotifyContext(context.Background(), os.Interrupt)
    defer rootCancel()

    components := []Component{
        NewPool(...),
        NewServer(...),
        NewMetrics(...),
    }

    g, runCtx := errgroup.WithContext(rootCtx)
    for _, c := range components {
        c := c
        g.Go(func() error { return c.Run(runCtx) })
    }

    err := g.Wait()
    log.Println("service stopped:", err)

    // graceful Stop with deadline (best effort)
    stopCtx, stopCancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer stopCancel()
    for _, c := range components {
        if err := c.Stop(stopCtx); err != nil {
            log.Println("stop:", err)
        }
    }
}

Each component runs concurrently; the errgroup joins them; on shutdown each is stopped with a deadline. This is the standard service skeleton in production Go.

Cancellation Flow: Up, Down, Sideways¶

At senior level, the directions of cancellation flow deserve careful analysis.

Downstream propagation¶

The most common case: an upstream cancellation flows to downstream consumers. Examples:

• SIGTERM cancels rootCtx; everything downstream cancels.
• Request deadline expires; the request's pipeline cancels.
• An upstream stage encounters EOF; downstream stages see the input channel close.

The mechanism is ctx.Done() (a shared close) or channel close (per-channel signal). Both broadcast.

Upstream propagation¶

A downstream stage decides to stop the pipeline. Examples:

• The consumer found what it wanted (first(N) pattern).
• The consumer encountered an unrecoverable error (cancel-on-error).
• The consumer is overloaded and signals back-pressure (rare; usually handled by channel buffering).

The mechanism: the consumer calls cancel() on a shared context. Upstream stages see ctx.Done() and stop producing. This is the same channel close, just initiated from the other end.

Sideways propagation¶

Sibling goroutines in the same group cancel each other on the first error. This is the errgroup model: any goroutine returning a non-nil error triggers cancellation that reaches every sibling.

g, ctx := errgroup.WithContext(parent)
g.Go(func() error { return stageA(ctx) }) // can cancel B
g.Go(func() error { return stageB(ctx) }) // can cancel A

If A errors, B sees ctx.Done(). If B errors first, A sees it. The cancellation is symmetric.

Diagonal propagation¶

A nested errgroup: child errors cancel only the child group; parent group continues unless explicitly told.

g, gctx := errgroup.WithContext(parent)
g.Go(func() error {
    sub, subctx := errgroup.WithContext(gctx)
    sub.Go(func() error { return work(subctx) })
    return sub.Wait() // sub-cancel does not affect parent siblings
})
g.Go(func() error { return otherWork(gctx) })
return g.Wait()

Sub-errors are isolated to the sub-group. Only when the outer g.Go returns the sub's error does the outer group cancel.

This isolation is what makes nested errgroups valuable: you can cleanly retry the sub-operation without affecting the rest of the parent pipeline.

Choosing the right propagation model¶

The choice depends on the failure semantics of the system:

• All-or-nothing. Single errgroup; first error kills everyone. Right for transactional pipelines where partial results are useless.
• Best-effort. Per-task contexts; each task fails independently. Right for fan-out where partial results are valuable.
• Hierarchical. Nested errgroups; sub-failures contained. Right for systems with logical groupings of related tasks.

Senior-level pipeline design is largely about choosing the right propagation model for each layer.

The narrow case of mid-pipeline mutation¶

A subtle situation: a pipeline stage mutates a slice or map that is shared with downstream. Cancellation interacts with the mutation.

func enricher(ctx context.Context, in <-chan *Item) <-chan *Item {
    out := make(chan *Item)
    go func() {
        defer close(out)
        for item := range in {
            item.Enriched = computeEnrichment(ctx, item)
            select {
            case out <- item:
            case <-ctx.Done():
                return
            }
        }
    }()
    return out
}

The enricher mutates item.Enriched in place. If the downstream consumer reads item.Enriched after cancellation, what does it see?

• If the enricher set the field before the select, the field is set.
• If the enricher was cancelled before setting it, the field is the zero value.

The downstream cannot distinguish. This is a small example of a larger issue: cancellation interrupts work in progress, and partial mutations are visible.

The fix: do not mutate; return new values. Functional style avoids this entirely:

result := &Item{...item, Enriched: computeEnrichment(ctx, item)}
select {
case out <- result:
case <-ctx.Done():
    return
}

Now the downstream gets a fully-formed item or nothing. Cancellation cannot leak partial mutations.

Pull-mode vs push-mode pipelines¶

Two design styles for pipelines:

• Push-mode: producer sends, consumer receives. The producer drives the rate.
• Pull-mode: consumer requests, producer responds. The consumer drives the rate.

Go channels naturally express push-mode (out <- v). Pull-mode requires more wiring but offers different cancellation properties.

Push-mode cancellation¶

The producer's out <- v is select-guarded; cancellation interrupts the send. Standard pattern.

Pull-mode cancellation¶

type Producer struct {
    in  chan request
    out chan response
}

func (p *Producer) Request(ctx context.Context) (Response, error) {
    select {
    case p.in <- request{}:
    case <-ctx.Done():
        return Response{}, ctx.Err()
    }
    select {
    case r := <-p.out:
        return r, nil
    case <-ctx.Done():
        return Response{}, ctx.Err()
    }
}

The consumer sends a request and waits for a response. Cancellation interrupts either the request or the wait. The producer is a long-lived goroutine that handles requests one at a time.

Pull-mode is useful when:

• The consumer should control the work rate (e.g. avoid producing items the consumer cannot keep up with).
• The work is on-demand rather than streaming.
• Each item has a specific consumer (not just any).

Combined: pull with batching¶

A pull-mode producer can batch internally:

func (p *Producer) Run(ctx context.Context) {
    for {
        select {
        case <-ctx.Done():
            return
        case <-p.in:
            // produce a batch and send each response
            batch := p.produceBatch()
            for _, item := range batch {
                select {
                case p.out <- item:
                case <-ctx.Done():
                    return
                }
            }
        }
    }
}

The consumer requests once and receives a batch. The producer's internal batching is opaque to the consumer. Cancellation aborts mid-batch.

Most Go pipelines are push-mode by default. Pull-mode is a valuable variation for specific scenarios.

Drain Semantics in Detail¶

Drain is what makes cancellation graceful. The high-level rule: when cancelling, ensure that pending work has somewhere to go (or is explicitly discarded). The low-level rule: every producer's last send must complete, either by a consumer reading or by the producer noticing cancellation and aborting.

The minimal drain¶

After cancelling, read the remaining values from each channel:

cancel()
for range out {
}

This unblocks producers stuck on out <-. They see ctx.Done() on the next iteration and return.

Drain with deadline¶

A drain that does not block forever:

cancel()
drainCtx, drainCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer drainCancel()
drained := make(chan struct{})
go func() {
    for range out {
    }
    close(drained)
}()
select {
case <-drained:
case <-drainCtx.Done():
    log.Println("drain timed out; some goroutines may have leaked")
}

If the drain does not complete within 5 seconds, log and proceed. This is the realistic shape for production shutdown — you do not want to wait forever.

Drain ordering across stages¶

In a multi-stage pipeline, the drain order matters. Drain the output of the pipeline; the cancellation cascades naturally backward. If you drain a middle channel, the producers upstream may still be blocked on the channel before it (because the middle stage cannot push into the drained-but-then-cancelled channel).

The orchestrator should:

1. Cancel the shared context.
2. Drain the final output channel.
3. Wait for all stages to exit (via WaitGroup or errgroup).

Manual mid-pipeline drains are usually unnecessary because each stage's select on ctx.Done() lets it exit without waiting for its output to be consumed (the send case loses to the cancel case).

What to do with in-flight values¶

When draining, the values you read are no longer useful — by definition, you have cancelled. Two options:

• Discard them silently (for range out {}).
• Process them with reduced effort (for v := range out { logDropped(v) }).

The choice depends on whether the values are observable or recoverable. For idempotent reads, discard. For pending writes, you may need to commit them (a special case requiring careful design).

Drain in fan-in stages¶

A fan-in stage forwards from N inputs to one output. On cancellation, each forwarder must drain its own input (to release upstream producers) and stop forwarding (to let the output close).

go func(in <-chan T) {
    defer wg.Done()
    for {
        select {
        case <-ctx.Done():
            // drain remaining
            for range in {
            }
            return
        case v, ok := <-in:
            if !ok {
                return
            }
            select {
            case out <- v:
            case <-ctx.Done():
                // drain remaining
                for range in {
                }
                return
            }
        }
    }
}(in)

The drain inside the forwarder ensures the upstream producer can exit. Without it, the producer remains stuck on its send.

When to skip the drain¶

If you know the producer also respects ctx, it will exit on ctx.Done() regardless of whether you drain. The drain is necessary only when the producer's send is blocking on you.

In errgroup-driven pipelines where every stage uses the shared context, drain is often unnecessary; the cancellation reaches everyone independently. The drain becomes important in mixed pipelines (some stages context-aware, others not) and in finite-data pipelines where the producer closes its output on EOF.

Idempotent vs at-most-once vs at-least-once delivery¶

A pipeline carries data from upstream to downstream. Cancellation can interrupt mid-delivery. The semantics of that interruption fall into three categories.

At-most-once¶

Each item is delivered zero or one times. On cancellation, in-flight items are discarded. No duplicates.

select {
case out <- v:
case <-ctx.Done():
    return // v is silently dropped
}

The simplest model. Suitable when items can be lost without consequence (metrics, logs, idempotent reads).

At-least-once¶

Each item is delivered one or more times. The producer retains the item until acknowledged. On cancellation, unacknowledged items may be redelivered next run.

for _, item := range items {
    if delivered(item) {
        continue
    }
    select {
    case out <- item:
        markDelivered(item)
    case <-ctx.Done():
        return // item is not marked delivered; next run will retry
    }
}

Requires durable state (e.g. a checkpoint). Suitable when duplicates are tolerable but loss is not (most messaging systems).

Exactly-once¶

Each item is delivered exactly once. Requires coordination: an ack protocol, transactional delivery, or idempotent consumers with deduplication.

for _, item := range items {
    ackCh := make(chan struct{})
    select {
    case out <- AckItem{Value: item, Ack: ackCh}:
    case <-ctx.Done():
        return // item not sent
    }
    select {
    case <-ackCh:
        markDelivered(item)
    case <-ctx.Done():
        return // item sent but not acked; retry next time
    }
}

The consumer must idempotently handle the case where it receives the same item twice (e.g. deduplicate by ID). The producer waits for the ack before advancing.

The choice of delivery semantics is an architectural decision. Cancellation behaviour follows from it.

Pragmatic trade-offs¶

In practice, "exactly-once" is rare and expensive. Most production systems use "at-least-once with idempotent consumers" — accepting that a cancellation may cause duplicate processing, designing consumers to handle it.

Designing for at-least-once means:

• Each item has a unique ID.
• The consumer tracks "seen" IDs (in a database or LRU cache).
• On receiving a duplicate, the consumer silently drops it.

The cancellation contract is: "If you cancel mid-delivery, an item may be processed twice; this is safe because the consumer handles duplicates."

Backpressure and Cancellation Interplay¶

Backpressure is how a slow consumer slows down a fast producer. In Go pipelines, unbuffered or small-buffered channels provide back-pressure automatically: the producer's send blocks when the buffer is full, slowing it to the consumer's pace.

Cancellation interacts with back-pressure in interesting ways:

Cancellation under back-pressure¶

The producer is blocked on out <- v because the consumer is slow. cancel() fires. The producer's select { case out <- v: case <-ctx.Done(): } picks the cancel case. The producer returns. The buffered value v is not delivered.

This is the correct behaviour: cancellation overrides back-pressure. The consumer was slow; cancellation says "we no longer care."

Backpressure during shutdown¶

If the consumer is mid-task, it cannot read the channel. The producer is wedged. Cancellation arrives; the producer's select notices and the producer exits. Good.

But what if the consumer is reading slowly because it is doing valuable per-item work? On cancel, in-flight items are dropped. If those items represent committed writes, you have lost data. The fix: design the producer to not consider items "delivered" until the consumer acknowledges.

// Producer
for v := range source {
    ack := make(chan struct{})
    select {
    case out <- WorkItem{Value: v, Ack: ack}:
    case <-ctx.Done():
        return
    }
    select {
    case <-ack:
        // item processed, safe to advance
    case <-ctx.Done():
        return
    }
}

Each item carries an ack channel; the consumer closes it after processing. The producer waits for the ack before sending the next item. This is a "stop-and-wait" protocol that integrates cancellation cleanly.

Buffered channels as back-pressure dial¶

A buffer of N decouples producer and consumer up to N items. Smaller buffer means tighter coupling and faster propagation of consumer slowness. Larger buffer means more memory and longer cancellation latency.

A practical heuristic: buffers are 0, 1, or "sufficient for one batch of work." Buffers of "make the producer fast" usually indicate a design problem — the consumer should be made faster or the producer slower, not buffered.

Bounded queues as a control point¶

Instead of a raw channel buffer, use a queue object with a Push(ctx, v) method that blocks (with select on ctx.Done()) when full. This makes the back-pressure point explicit and measurable.

type Queue struct {
    ch chan T
}

func (q *Queue) Push(ctx context.Context, v T) error {
    select {
    case q.ch <- v:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

func (q *Queue) Pop(ctx context.Context) (T, error) {
    select {
    case v := <-q.ch:
        return v, nil
    case <-ctx.Done():
        var zero T
        return zero, ctx.Err()
    }
}

The queue is just a channel with a typed API. The benefit is observability: you can instrument Push to measure back-pressure (how often does it block? for how long?).

Backpressure across services¶

Across an HTTP or gRPC boundary, back-pressure is the connection itself: the client's Write blocks when the kernel send buffer fills, which happens when the server is slow. Cancellation is the connection close: client cancels, connection closes, server's Read errors out.

The result: cross-service back-pressure and cancellation are physically the same mechanism (TCP flow control and FIN/RST). Designing for cancellation across the wire means designing for clean connection lifecycle.

Backpressure design patterns¶

Pattern: bounded queue with timeout¶

A queue that drops items if Push waits too long:

func (q *Queue) PushWithTimeout(ctx context.Context, v T, d time.Duration) error {
    pushCtx, cancel := context.WithTimeout(ctx, d)
    defer cancel()
    select {
    case q.ch <- v:
        return nil
    case <-pushCtx.Done():
        return pushCtx.Err()
    }
}

Bounded waiting; the caller decides whether the timeout means "retry," "drop," or "fail." This is the closest Go offers to an explicit back-pressure policy.

Pattern: shedding load¶

When the queue is full, drop the new item rather than block:

select {
case q.ch <- v:
case <-ctx.Done():
    return ctx.Err()
default:
    // queue full; drop or signal load shedding
    metrics.IncDropped()
    return ErrLoadShed
}

The default case in select makes this non-blocking. If the queue cannot accept immediately, we drop.

Load shedding is a sophisticated topic; the key cancellation-related insight is that shedding is not the same as cancellation. Shedding is "I am too busy to accept this work right now"; cancellation is "stop the work you have already accepted." Both protect the system from overload, but at different stages.

Pattern: adaptive concurrency¶

The number of workers adjusts based on observed latency:

func adaptive(ctx context.Context, items <-chan Item) error {
    workers := 10
    var wg sync.WaitGroup
    for {
        select {
        case <-ctx.Done():
            wg.Wait()
            return ctx.Err()
        default:
        }
        // ... adjust workers based on latency observations ...
    }
}

A more advanced pattern; usually implemented via libraries (e.g. Netflix Concurrency Limits). Cancellation cleanly stops the adjustment loop along with the workers.

Cancellation Across Process Boundaries¶

A pipeline that crosses a process boundary (RPC, HTTP, queue) loses the in-process cancellation primitive. The signal must be reconstructed on the remote side.

gRPC¶

gRPC has first-class cancellation support:

• The client's Context carries the deadline as a "grpc-timeout" header.
• The server reconstructs a Context with the same deadline.
• On client cancel, gRPC sends RST_STREAM; the server sees its Context cancel.
• On server-side cancellation, the response is closed; the client sees an error.

This is the cleanest cross-process cancellation in Go's ecosystem. Use it freely.

HTTP/1.1¶

HTTP/1.1 has no header for deadlines or explicit cancellation. The signal is:

• Client cancellation: close the TCP connection. The server sees the next Read return.
• Server cancellation: write a response and close. The client sees EOF on Read.

Without deadlines in metadata, the server cannot enforce the client's intended deadline. The client's http.NewRequestWithContext with a timeout closes the connection on timeout, but the server may have already started a slow operation that continues to completion (consuming server resources).

Mitigation: enforce server-side timeouts independently (http.Server.WriteTimeout, per-handler context.WithTimeout). Do not rely on the client-deadline-as-server-deadline pattern.

HTTP/2 and HTTP/3¶

Both have stream cancellation primitives:

• HTTP/2: RST_STREAM frame closes a single stream without closing the connection.
• HTTP/3: stream reset over QUIC.

Modern Go HTTP servers detect these and cancel the request context appropriately.

Message queues¶

Cancellation across a message queue is asymmetric: the consumer can stop consuming, but it cannot "cancel" a message that has been delivered.

• The consumer's local cancellation aborts its processing of the current message.
• Depending on the queue (Kafka commit, SQS ack), the message may be redelivered.
• The producer cannot recall a message that has been published.

For pipelines that include message queues, design idempotent consumers. Cancellation may cause a message to be processed twice (once partially, once retried) — the system must tolerate that.

Database transactions¶

A database transaction is its own micro-cancellation domain:

• db.BeginTx(ctx, ...) starts a transaction tied to ctx.
• If ctx cancels mid-transaction, the driver issues a rollback.
• Already-committed work is durable, regardless of subsequent cancellation.

The cancellation latency depends on the driver's responsiveness to the cancel signal. pgx (Postgres) is well-instrumented; some MySQL drivers are less so.

Worker queues across processes¶

For pipelines that span multiple processes (a producer in one, a worker in another), cancellation is by convention:

• The producer publishes a "cancel" message.
• The worker subscribes to the same channel and processes the cancel.

This is heavyweight; in practice, most multi-process pipelines rely on per-task deadlines rather than explicit cancel. If a task takes too long, it is aborted independently rather than coordinated.

A study: cancellation in long-poll vs streaming¶

Two related patterns with different cancellation profiles.

Long-poll: one response per request¶

func longPoll(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
    defer cancel()
    select {
    case ev := <-events:
        json.NewEncoder(w).Encode(ev)
    case <-ctx.Done():
        w.WriteHeader(204)
    }
}

The handler waits for one event or a deadline. Cancellation arrives via either source. The handler returns once.

Server-Sent Events: many responses per request¶

func sse(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    flusher := w.(http.Flusher)
    for {
        select {
        case <-ctx.Done():
            return
        case ev := <-events:
            fmt.Fprintf(w, "data: %s\n\n", ev.JSON())
            flusher.Flush()
        }
    }
}

The handler streams events. Cancellation arrives when the client disconnects. The handler exits the loop and returns.

The difference: long-poll has a fixed end (response or timeout); SSE is open-ended. SSE relies entirely on cancellation to stop. A SSE handler that does not check ctx.Done() is a permanent goroutine leak.

Cross-process cancellation: case studies¶

Case study: gRPC chained calls¶

A gRPC server receives a call, then makes a downstream gRPC call. The context is forwarded:

func (s *server) Handle(ctx context.Context, req *Req) (*Resp, error) {
    downstream, err := s.client.Other(ctx, &OtherReq{})
    if err != nil {
        return nil, err
    }
    return &Resp{Data: downstream.Result}, nil
}

If the original caller cancels, ctx cancels in this server. The s.client.Other(ctx, ...) call sends RST_STREAM downstream; the downstream server's context cancels. The whole chain unwinds.

Deadlines propagate similarly: the original deadline travels in grpc-timeout metadata; each hop reconstructs the same deadline.

Case study: HTTP-to-Kafka¶

A handler receives an HTTP request and publishes to Kafka:

func handler(w http.ResponseWriter, r *http.Request) {
    if err := publish(r.Context(), msg); err != nil {
        http.Error(w, err.Error(), 500)
        return
    }
}

func publish(ctx context.Context, msg Message) error {
    select {
    case <-ctx.Done():
        return ctx.Err()
    case producer.SendChan() <- msg:
        return nil
    }
}

The handler cancels if the client disconnects. The publish call respects ctx and returns. Kafka itself does not "see" the cancellation — once a message is in flight to the broker, it cannot be retracted.

This is the asymmetry: in-process, cancellation is two-way; into Kafka, cancellation only stops the producer's intent to publish further messages.

Case study: distributed sagas¶

A saga is a sequence of compensable steps across services. Cancellation in the middle of a saga must trigger compensations (undo the completed steps).

func runSaga(ctx context.Context, steps []Step) error {
    completed := []Step{}
    defer func() {
        if ctx.Err() != nil {
            for i := len(completed) - 1; i >= 0; i-- {
                completed[i].Compensate(context.Background()) // use detached ctx
            }
        }
    }()
    for _, s := range steps {
        if err := s.Execute(ctx); err != nil {
            return err
        }
        completed = append(completed, s)
    }
    return nil
}

The compensation uses context.Background() because we want it to run even after the saga's ctx cancelled. Or use context.WithoutCancel(ctx) to preserve values without cancellation.

Sagas are a deep topic; the relevant cancellation insight is that "graceful shutdown" of a saga is not just "stop running"; it is "stop running and undo what was started."

Lifecycle Modelling and State Machines¶

A complex pipeline has states: Initialising, Running, Draining, Stopped. Modeling them explicitly clarifies cancellation behaviour.

Explicit state¶

type State int

const (
    StateInit State = iota
    StateRunning
    StateDraining
    StateStopped
)

type Service struct {
    mu    sync.Mutex
    state State
    // ...
}

func (s *Service) transition(to State) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    if !valid(s.state, to) {
        return fmt.Errorf("invalid transition: %d -> %d", s.state, to)
    }
    s.state = to
    return nil
}

Transitions are validated. Init -> Running -> Draining -> Stopped is the happy path; Running -> Stopped (skip drain) is forceful.

Reading state from goroutines¶

func (s *Service) acceptWork(j Job) error {
    s.mu.Lock()
    state := s.state
    s.mu.Unlock()
    if state != StateRunning {
        return ErrNotRunning
    }
    return s.pool.Submit(j)
}

Acceptance checks the state under the mutex. After Draining, new work is rejected.

Coordinating state and context¶

The state machine drives the cancellation:

func (s *Service) Drain() {
    s.transition(StateDraining)
    s.pool.StopAccepting()
    s.pool.Drain() // waits for workers
    s.transition(StateStopped)
    s.cancel(errors.New("drained"))
}

Workers respect the pool's "drain" signal; once they have finished in-flight jobs, they exit. The root cancel fires after, cleaning up background tasks.

When to model state explicitly¶

Small services: do not bother. The implicit state from context.Context and channel close is sufficient.

Large services with complex lifecycle (multiple shutdown phases, hot config reload, graceful upgrade): model state explicitly. The clarity is worth the code.

Lifecycle examples: rolling updates and hot reload¶

Production services often need to update without dropping in-flight work. Two patterns.

Rolling update via two processes¶

The orchestrator (Kubernetes, systemd) starts a new process before stopping the old. Both run concurrently for a short window. The old process drains while the new one accepts new traffic.

In Go, the old process responds to SIGTERM by stopping its listener (e.g. srv.Shutdown) and draining its work. The new process is already up.

The cancellation flow in the old process:

1. SIGTERM arrives; rootCtx cancels.
2. HTTP listener stops accepting; Shutdown waits for handlers.
3. Background workers see rootCtx.Done(); they drain in-flight work.
4. After all goroutines exit, the old process terminates.

The orchestrator (not Go code) handles the dual-running window. The new process is just a regular boot.

Hot reload via a single process¶

A single process reloads configuration without restarting. Cancellation is more interesting:

• Old configuration in use; goroutines hold pointers to it.
• New configuration arrives.
• Old goroutines continue with the old config (or are cancelled).
• New goroutines pick up the new config.

A common pattern: each component has a Restart(newConfig) method that creates a new internal context and replaces the old goroutines.

func (p *Pool) Restart(newConfig Config) {
    p.mu.Lock()
    oldCancel := p.cancel
    p.ctx, p.cancel = context.WithCancel(p.parent)
    p.config = newConfig
    p.startWorkers()
    p.mu.Unlock()
    oldCancel() // old workers see cancel and exit
}

The new workers start before the old are cancelled; the changeover is brief. In-flight items in the old workers are either drained or dropped depending on policy.

Hot reload is hard to get right; many services prefer rolling updates instead.

Designing for Predictable Cancellation Latency¶

Cancellation latency is the time from cancel() to the last goroutine exiting. For production pipelines, this should be measurable, bounded, and tested.

Components of latency¶

• Per-stage select latency. The time between cancel() and the stage's next select iteration. Usually sub-microsecond if the stage selects often.
• Per-item work time. If a stage takes 1 second per item with no internal cancellation checks, that is 1 second of latency.
• Channel buffer flush. A buffer of 100 with 10 ms per item adds up to 1 second of buffered work.
• External I/O. A blocking I/O call that does not respect context can add seconds or minutes.

Bounding each component¶

• Select latency: keep select in every loop; cancellation interruption is sub-millisecond.
• Per-item work: insert ctx.Err() checks in long inner loops, every 1ms or so.
• Channel buffer: keep buffers small (0, 1, or "one batch") unless there is a specific reason.
• External I/O: always pass ctx to context-aware variants (db.QueryContext, http.NewRequestWithContext).

Measuring latency in tests¶

func TestCancellationLatency(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    done := make(chan struct{})
    go func() {
        runPipeline(ctx)
        close(done)
    }()
    time.Sleep(100 * time.Millisecond) // let it run
    start := time.Now()
    cancel()
    <-done
    latency := time.Since(start)
    if latency > 200*time.Millisecond {
        t.Errorf("cancellation took %v", latency)
    }
}

A simple measurement test. Production pipelines should have these and a Prometheus histogram for the same metric.

Measuring latency in production¶

start := time.Now()
<-ctx.Done()
cancelObserved := time.Since(start) // when did this goroutine see cancel
// ... continue cleanup ...
totalLatency := time.Since(start)
metrics.CancelLatency.Observe(totalLatency.Seconds())

A histogram of per-goroutine cancellation latency tells you which stages are slow to exit. Combined with a stack-trace dump, you can pinpoint the offending blocking call.

The 99th percentile matters¶

For shutdown SLAs, you care about the worst case, not the average. A pipeline that exits in 10 ms average but 5 seconds 99th percentile will sometimes blow the SLA. Either improve the worst case (refactor the slow path) or shorten the SLA budget.

A note on select semantics revisited¶

At senior level it pays to revisit select precisely. Some properties that affect cancellation design:

• Fairness: when multiple cases are ready, Go picks at random. Over many iterations, all cases are picked roughly equally.
• No partial readiness: a case is either fully ready or not. A send to a buffered channel is "ready" only when the buffer has room; a receive is ready when the channel has a value (or is closed).
• default makes select non-blocking: with a default, select returns immediately if no other case is ready. This is the "try-send" idiom.
• nil channel cases are never ready: setting a channel to nil effectively disables its case. This is a clever trick for state-machine-like selects:

var out chan T
if hasOutput {
    out = realChannel
}
select {
case out <- value:
case <-ctx.Done():
}

If out is nil, the send case is disabled; the select waits on ctx.Done() only.

• select with one case is the same as a plain operation: select { case v := <-ch: ... } is just v := <-ch with extra syntax. The compiler may optimise.

These properties combine: a multi-state stage may use nil-channel disabling to express which transitions are valid in each state, while keeping <-ctx.Done() always enabled.

Worst-case cancellation latency: a calculation¶

Suppose a service has:

• An HTTP listener (Shutdown latency: bounded by WriteTimeout).
• A worker pool of 16 workers, each processing items in 500 ms.
• A background metrics emitter ticking every 10 seconds.
• A DB connection pool.

Worst-case cancellation latency:

• HTTP Shutdown: up to WriteTimeout (say 5 seconds).
• Pool drain: each worker finishes current item, up to 500 ms; in parallel, so 500 ms total.
• Metrics emitter: up to 10 seconds (next tick), unless the ticker is cancellable. With select { case <-ctx.Done(): case <-ticker.C: }, latency is 1 microsecond.
• DB pool close: depends on number of in-use connections; near-zero after workers finish.

Total: max(5s, 500ms, 1us, ~0) = 5 seconds. Within a 30-second SLA, comfortable.

If the worker per-item time were 30 seconds instead of 500 ms, the drain alone would blow the SLA. Mitigations: shorter per-item work, cancellation polling, or accept a longer SLA.

The discipline: enumerate the components, identify worst-case per-component latency, sum (or take max), compare to SLA.

Cancellation in Streaming Systems¶

Streaming systems — ones where the data flows continuously rather than in discrete requests — have specific cancellation challenges.

Endless streams¶

A Kafka consumer, a server-sent events stream, a WebSocket connection: all of these produce data indefinitely. Cancellation is the only way to stop them.

func consumeKafka(ctx context.Context, topic string) error {
    consumer := newConsumer(topic)
    defer consumer.Close()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }
        msg, err := consumer.Poll(ctx, 100*time.Millisecond)
        if err != nil {
            if errors.Is(err, context.Canceled) {
                return err
            }
            log.Println("poll:", err)
            continue
        }
        if msg == nil {
            continue // poll timeout
        }
        if err := process(ctx, msg); err != nil {
            return err
        }
    }
}

The poll-based design lets the consumer check ctx regularly. A push-based design would need a watcher goroutine to cancel via consumer.Close().

Stream multiplexing¶

A service that handles multiple streams (e.g. a chat server with many connected clients) needs per-stream cancellation. The architecture:

• One context per stream.
• Each stream's goroutine selects on its own ctx.Done().
• A shutdown cancels the root context, cascading to every stream.

Cancellation latency is one select-cycle per stream, in parallel. For 10 000 streams, full shutdown is sub-millisecond if every stream selects frequently.

Stream merging¶

When multiple streams feed into one output (e.g. multiple message queues into one processor), each input has its own cancellation. The merged output cancellation fires when the parent context cancels.

g, ctx := errgroup.WithContext(parent)
merged := make(chan Event, 64)
for _, src := range sources {
    src := src
    g.Go(func() error {
        for {
            select {
            case <-ctx.Done():
                return ctx.Err()
            case ev := <-src.Receive():
                select {
                case merged <- ev:
                case <-ctx.Done():
                    return ctx.Err()
                }
            }
        }
    })
}
// closer
go func() {
    _ = g.Wait()
    close(merged)
}()

Each source has its own goroutine; the merged channel is closed after all sources exit. Cancellation reaches every source via ctx.Done().

Streaming systems: detailed patterns¶

Pattern: cursor-based pagination with cancellation¶

A pipeline reads from a paginated source (e.g. a DB cursor or a paginated API). Each page is a batch.

func paginate(ctx context.Context, src Source) <-chan []Row {
    out := make(chan []Row)
    go func() {
        defer close(out)
        cursor := ""
        for {
            select {
            case <-ctx.Done():
                return
            default:
            }
            page, nextCursor, err := src.Fetch(ctx, cursor)
            if err != nil {
                return
            }
            if len(page) == 0 {
                return
            }
            select {
            case out <- page:
            case <-ctx.Done():
                return
            }
            if nextCursor == "" {
                return
            }
            cursor = nextCursor
        }
    }()
    return out
}

Each page is fetched with the context (so the fetch is cancellable). The send is select-guarded. The cursor logic carries forward; on cancellation, the cursor is lost — next run starts over (or resumes from a checkpoint if you store one).

Pattern: checkpointed streaming¶

The pipeline periodically writes its progress to a durable store, so restart can resume from the checkpoint.

func checkpointed(ctx context.Context, src Source, ckpt Checkpoint) error {
    cursor, _ := ckpt.Load(ctx)
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        default:
        }
        page, next, err := src.Fetch(ctx, cursor)
        if err != nil {
            return err
        }
        if len(page) == 0 {
            return nil
        }
        if err := process(ctx, page); err != nil {
            return err
        }
        if err := ckpt.Save(ctx, next); err != nil {
            return err
        }
        cursor = next
    }
}