Drain Pattern — Senior Level¶

Table of Contents¶

1. Introduction
2. The Architecture View
3. Two-Phase Shutdown
4. Supervisor-Driven Drain
5. Drain DAG and Topological Order
6. Drain Across Service Boundaries
7. Cluster-Aware Drain
8. Drain and Leader Election
9. Drain and Distributed Transactions
10. Quiesce in Stateful Systems
11. Drain Telemetry at Scale
12. Drain Anti-Patterns at Senior Level
13. Designing For Drainability
14. Drain Testing Strategy
15. Drain Performance Budgets
16. Drain and Capacity Planning
17. Hot Path Cost of Drain Tracking
18. Drain in Polyglot Stacks
19. Drain Incidents — A Case Study
20. Drain and Sidecars
21. Drain and Long-Running Jobs
22. Mentoring Through Drain
23. Self-Assessment Checklist
24. Summary

Introduction¶

Focus: "How do I design a system so drain is automatic? How do I lead a team to write drainable services by default?"

At the junior level you learn the recipe. At the middle level you apply the recipe across components. At the senior level you stop thinking about the recipe — drain is a property of the system architecture, baked in from the first lines of code. Your job is to design systems that drain naturally, to mentor engineers in writing drainable code without ceremony, to debug drain failures in production from a goroutine dump and a metric graph, and to push drain into the same first-class status as health checks and logging.

This page is less about new code and more about judgement. The decisions you make at the senior level: which subsystems share a deadline, which can run drains in parallel, when to break the recipe, how to teach drainability without slowing the team down.

The Architecture View¶

Drain at the senior level is not a feature; it is a property. A system either has it or does not. Designing for drain means:

• Every long-lived goroutine has an owner.
• Every owner has a Drain method with a context parameter.
• The dependency graph of components is explicit (no surprise back-edges).
• The orchestrator's grace period is a known input to every design decision.
• The drain budget is allocated across components based on measured P99s, not guesses.

A system without these is a system that might drain — it works in dev, sometimes works in prod, fails on every deploy with a corner case. A system with these is a system that drains by construction.

The drain-friendly architecture diagram¶

                      ┌───────────────────────────┐
                      │       Process Boundary    │
                      └────────────┬──────────────┘
                                   │
       ┌───────────────────────────┼───────────────────────────┐
       │                           │                           │
┌──────▼──────┐             ┌──────▼──────┐             ┌──────▼──────┐
│  Ingress     │            │  Workers     │            │  Schedulers  │
│  (HTTP/gRPC) │            │  (Pools)     │            │  (Cron)      │
└──────┬──────┘             └──────┬──────┘             └──────┬──────┘
       │                           │                           │
       └────────────┬──────────────┴───────────────────────────┘
                    │
              ┌─────▼─────┐
              │  Producers │
              │  (Kafka)   │
              └─────┬─────┘
                    │
              ┌─────▼─────┐
              │  Stores    │
              │  (DB/Redis)│
              └───────────┘

Drain travels top to bottom. Ingress drains first; stores drain last. Each layer's Drain(ctx) blocks until in-flight at that layer is zero.

Drainability invariants¶

Treat these as architectural invariants:

• Invariant 1. Every goroutine is reachable from a Drain method.
• Invariant 2. Every Drain method takes a context and returns an error.
• Invariant 3. The Drain dependency graph is a DAG (no cycles).
• Invariant 4. Total drain time across all components fits in the grace period.

A code review that catches invariant violations catches drain bugs at design time.

Two-Phase Shutdown¶

Single-phase drain is "stop intake, wait, close." Two-phase drain adds a quiesce step:

• Phase A — quiesce. Tell the system "you are going down soon, but keep running." The system uses this hint to bias toward completion: stop accepting long-running work, slow down throughput to drain in-flight faster.
• Phase B — drain. The standard drain.

Two-phase is useful for:

• Systems where in-flight work is many-stage. Quiesce stops new sagas; drain finishes existing ones.
• Distributed leader election. Quiesce releases leadership before drain.
• Long-running batch jobs. Quiesce stops new jobs; drain waits for current ones.

type Server struct {
    quiescing atomic.Bool
    draining  atomic.Bool
}

func (s *Server) Quiesce() {
    s.quiescing.Store(true)
    // release leadership, stop scheduling
}

func (s *Server) Drain(ctx context.Context) error {
    s.draining.Store(true)
    return s.wait(ctx)
}

// main:
s.Quiesce()
time.Sleep(quiescePeriod)
_ = s.Drain(drainCtx)

quiescePeriod is typically 5–15 seconds. Long enough for outstanding sagas to finish a stage, short enough to fit in the grace period.

Supervisor-Driven Drain¶

For a service with many components, a supervisor coordinates start, runtime monitoring, and drain. Erlang/OTP pioneered this; in Go, we build it with errgroup and context.

type Component interface {
    Name() string
    Start(ctx context.Context) error
    Drain(ctx context.Context) error
}

type Supervisor struct {
    components []Component
    logger     *slog.Logger
}

func (s *Supervisor) Run(ctx context.Context) error {
    g, gCtx := errgroup.WithContext(ctx)
    for _, c := range s.components {
        c := c
        g.Go(func() error {
            s.logger.Info("starting", "component", c.Name())
            err := c.Start(gCtx)
            s.logger.Info("stopped", "component", c.Name(), "err", err)
            return err
        })
    }
    return g.Wait()
}

func (s *Supervisor) Drain(ctx context.Context) error {
    // Reverse order
    for i := len(s.components) - 1; i >= 0; i-- {
        c := s.components[i]
        s.logger.Info("draining", "component", c.Name())
        start := time.Now()
        err := c.Drain(ctx)
        s.logger.Info("drained", "component", c.Name(),
            "duration", time.Since(start), "err", err)
    }
    return nil
}

A supervisor centralises logging, sequencing, and error handling. New components are added with one line.

Supervisor strategies¶

Inspired by Erlang:

• One-for-one. One component crashes; restart only that component. Drain on supervisor exit.
• One-for-all. One component crashes; drain everyone, exit.
• Rest-for-one. One component crashes; drain it and everything that depends on it.

For Go services, the most common is one-for-all: any crash triggers a service-wide drain and exit. This is what errgroup gives you naturally.

Drain DAG and Topological Order¶

A real service has a dependency graph, not just a flat list. The HTTP server depends on the worker pool, which depends on Kafka, which depends on the connection layer. Drain order is the topological sort of this graph.

type Node struct {
    Name     string
    Drain    func(context.Context) error
    DependsOn []string
}

type Graph struct {
    nodes map[string]*Node
}

func (g *Graph) DrainOrder() ([]*Node, error) {
    // Kahn's algorithm
    indeg := map[string]int{}
    deps := map[string][]string{} // reverse: deps[a] = nodes that depend on a
    for _, n := range g.nodes {
        indeg[n.Name] = 0
    }
    for _, n := range g.nodes {
        for _, d := range n.DependsOn {
            indeg[n.Name]++
            deps[d] = append(deps[d], n.Name)
        }
    }
    var queue []*Node
    for _, n := range g.nodes {
        if indeg[n.Name] == 0 {
            queue = append(queue, n)
        }
    }
    var order []*Node
    for len(queue) > 0 {
        n := queue[0]
        queue = queue[1:]
        order = append(order, n)
        for _, dep := range deps[n.Name] {
            indeg[dep]--
            if indeg[dep] == 0 {
                queue = append(queue, g.nodes[dep])
            }
        }
    }
    if len(order) != len(g.nodes) {
        return nil, errors.New("cycle")
    }
    // Reverse for drain order
    for i, j := 0, len(order)-1; i < j; i, j = i+1, j-1 {
        order[i], order[j] = order[j], order[i]
    }
    return order, nil
}

The algorithm: build a topo order of "starts" (dependencies before dependents), then reverse for drain order (dependents before dependencies).

Parallel drain¶

Components without a direct dependency can drain in parallel. For a graph with N nodes, drain is at most the height of the DAG, not the size.

func (g *Graph) DrainParallel(ctx context.Context) error {
    // Group nodes by depth.
    depth := g.depths()
    maxDepth := 0
    for _, d := range depth {
        if d > maxDepth {
            maxDepth = d
        }
    }
    for d := maxDepth; d >= 0; d-- {
        var grp errgroup.Group
        for name, dd := range depth {
            if dd != d {
                continue
            }
            n := g.nodes[name]
            grp.Go(func() error { return n.Drain(ctx) })
        }
        if err := grp.Wait(); err != nil {
            return err
        }
    }
    return nil
}

Each "wave" runs in parallel. The drain takes time proportional to the depth, not the total number of components.

Drain Across Service Boundaries¶

A microservice does not drain in isolation. Its upstreams and downstreams matter.

Upstream drain notification¶

When pod A drains, its callers should know. Options:

• Health check. Caller polls /ready; sees 503; takes pod A out of rotation.
• Push notification. Pod A publishes a "draining" event on a control bus.
• Connection close. Pod A closes its server; callers see EOF.

Health check is the most common. Push notification is faster but adds complexity. Connection close is implicit but unreliable (the caller may not retry).

Downstream drain coordination¶

When pod A drains, its callees may still be running. Pod A should:

• Not start new requests to callees in the last seconds of drain.
• Let in-flight downstream calls finish (within the drain deadline).
• Close its connection pool last.

Drain hand-off in a saga¶

A saga is a multi-step distributed transaction. Pod A may be in step 3 of 7 when drain starts. Options:

1. Finish the saga locally. Continue to step 7 within the drain budget.
2. Hand off. Persist saga state; let pod B (or A's replacement) resume.
3. Compensate. Roll back steps 1-3.

The choice depends on the saga's idempotency and the budget. Hand-off requires storage; finish-locally requires time; compensate requires backward steps.

Cluster-Aware Drain¶

In a cluster, drain of one pod must consider:

• Are there enough other pods to handle the load?
• Is a leader election needed?
• Are any other pods also draining?

Quorum-aware drain¶

For a cluster of N pods needing quorum M, never let more than N-M drain at once. Implement this in the orchestrator:

maxUnavailable: 1

Kubernetes' PodDisruptionBudget enforces this for voluntary disruptions.

Drain throttling¶

For large clusters, throttle drain rate to avoid mass session migration:

maxSurge: 1
maxUnavailable: 0

One new pod up, then one old pod drains. Sessions migrate gradually.

Cross-region drain¶

For multi-region services, drain in one region should not affect others. Ensure each region's drain is independent. Common pitfall: a shared database or message bus that becomes a serial bottleneck.

Drain and Leader Election¶

A service with leader election (etcd, Zookeeper, consensus libraries) must release leadership during drain.

type LeaderService struct {
    election *etcd.Election
}

func (s *LeaderService) Drain(ctx context.Context) error {
    // Step 1: release leadership FIRST.
    if err := s.election.Resign(ctx); err != nil {
        // continue draining anyway
    }
    // Step 2: let the new leader be elected.
    // Step 3: drain own work.
    return s.drainWork(ctx)
}

Why first? Because resigning leadership allows another pod to take over. If you drain work first, you stay leader for the duration — and no other pod can take leader-only actions during that time.

The downside: tasks that require leader privilege cannot run after resignation. If your drain needs to write a checkpoint as leader, do it before resignation.

Drain and Distributed Transactions¶

Distributed transactions (2PC, Sagas) interact poorly with drain. The classic problems:

• Drain in the middle of prepare phase. Hand off without commit decision: the transaction hangs.
• Drain in the middle of commit phase. Some participants commit; others don't.
• Drain of the coordinator. Replacement coordinator must resume.

Best practices¶

1. Persist transaction state at every step. Drain or crash, the next node resumes.
2. Idempotent participants. A replay does not double-effect.
3. Bound prepare/commit time. If prepare takes longer than drain budget, the design is broken.

A common pattern: short transactions (< 1 second each), persistent log, idempotent steps. Drain is then a non-event for transactions — at worst, one is replayed.

Quiesce in Stateful Systems¶

Stateful systems (databases, caches, search indices) drain differently. They must:

1. Stop accepting writes.
2. Flush WAL / write-ahead log.
3. Persist any in-memory state.
4. Disconnect followers / replicas cleanly.
5. Close storage handles.

A short example: an in-memory cache with periodic snapshotting.

type Cache struct {
    mu       sync.RWMutex
    data     map[string][]byte
    dirty    atomic.Bool
    snapshot func(ctx context.Context, data map[string][]byte) error
}

func (c *Cache) Set(k string, v []byte) {
    c.mu.Lock()
    c.data[k] = v
    c.mu.Unlock()
    c.dirty.Store(true)
}

func (c *Cache) Drain(ctx context.Context) error {
    c.mu.Lock()
    defer c.mu.Unlock()
    if !c.dirty.Load() {
        return nil
    }
    return c.snapshot(ctx, c.data)
}

Snapshot during drain captures the final state. Next start reads from the snapshot. No data loss.

Stateful drain order¶

Within a stateful component, drain order matters:

1. Close listeners (no new clients).
2. Wait for in-flight writes.
3. Flush WAL.
4. Snapshot in-memory state.
5. Close replication channels.
6. Close storage.

Each step has its own budget. Skipping or reordering breaks consistency.

Drain Telemetry at Scale¶

In a fleet of 100 pods, telemetry per-drain is essential.

Metrics that matter at scale¶

• Drain duration histogram. P50, P95, P99 across all pods.
• Force-cancelled count. Anything > 0 is a signal.
• In-flight at drain start. Trends suggest leak.
• Goroutines at drain start. Same.
• Memory at drain start. Same.

Alerts¶

• drain_duration P99 > grace_period * 0.8 for 5 min — drains approaching the limit.
• drain_force_cancelled > 0 in any pod — incident.
• drain_count{result="failure"} > 0 — incident.

Dashboards¶

Every service should have a "deploy health" dashboard with:

• Drain duration per deploy.
• Force-cancellations per deploy.
• 5xx rate during deploy windows.
• Goroutine count change post-deploy.

Looking at these once per week catches drift before it becomes incident.

Drain Anti-Patterns at Senior Level¶

Anti-pattern: Drain in init()¶

func init() {
    signal.Notify(...)
}

init is the wrong place for signal handling. It runs in package import order; you do not control it. Put signal.NotifyContext in main.

Anti-pattern: Global drain functions¶

var drainFns []func()

func RegisterDrain(f func()) {
    drainFns = append(drainFns, f)
}

Globals make ordering hard to reason about. Use explicit lifecycle management.

Anti-pattern: Drain that holds locks¶

A drain that holds a top-level mutex serialises the entire process. Use fine-grained locks and atomic flags.

Anti-pattern: Component owning its own context¶

type Server struct {
    ctx context.Context // wrong
}

A stored context is a hidden coupling. Pass context as a method parameter.

Anti-pattern: Drain inside a goroutine's main loop¶

go func() {
    for {
        if shouldDrain() {
            drain()
            return
        }
        work()
    }
}()

Mixes concerns. The goroutine does both work and drain. A select on context cancel is cleaner.

Anti-pattern: Drain that reaches into other components¶

func (s *Server) Drain(ctx context.Context) error {
    s.workerPool.queue = nil // poking internals
}

Each component drains itself. The supervisor coordinates.

Designing For Drainability¶

When designing a new service, ask:

1. What are the long-lived goroutines? Each must have an owner.
2. What are the lifecycle hooks? Start, Drain, Close minimum.
3. What is the dependency graph? Sketch it before coding.
4. What is the budget per component? Sum to grace period.
5. What is the test strategy? Drain test in CI minimum.

Designs that answer all five drain easily. Designs that skip any of them lead to fragile shutdown.

Code review checklist¶

• Every long-lived goroutine has a Drain or equivalent.
• Drain takes a context, returns an error.
• Drain order is documented in code.
• No os.Exit outside main.
• Drain test exists.
• Drain metric is emitted.

Adopt this checklist for every PR. The cost is two minutes; the benefit is years of clean deploys.

Drain Testing Strategy¶

At the senior level, drain testing goes beyond unit tests.

Levels of drain test¶

1. Unit. Each component's Drain method, in isolation.
2. Integration. Multiple components together with simulated load.
3. Soak. Run for hours under traffic; trigger drain; verify clean.
4. Chaos. Random kill -TERM under load; verify no data loss.
5. Production canary. Deploy a single pod; observe metrics.

Each level catches different bugs. Skipping the integration level is the most common mistake.

Drain test as deploy gate¶

A drain test in CI that fails blocks the deploy. The test:

1. Starts the binary.
2. Sends synthetic load.
3. Sends SIGTERM.
4. Asserts: clean exit code, drain duration under threshold, zero 5xx.

./service &
PID=$!
./loadgen --duration 30s --rps 100 &
LPID=$!
sleep 5
kill -TERM $PID
wait $PID
EXIT_CODE=$?
wait $LPID
if [ $EXIT_CODE -ne 0 ]; then
    echo "service did not exit cleanly"
    exit 1
fi

Chaos drain test¶

Inject SIGTERM at random times during a sustained workload. Run for an hour. Count anomalies. Tools like chaosmesh or simple bash loops work.

Drain Performance Budgets¶

A drain budget is the time you allow each phase to take. Sum to grace period.

Budgeting from measurement¶

1. Measure each component's drain duration at P50, P95, P99 in production.
2. Allocate budget = P99 + safety margin (typically 1.5x).
3. Sum the budgets. Compare to grace period.
4. If sum > grace, reduce: parallelise where possible, optimise slow components.

Example budget¶

If the sum exceeds the grace period, the team has work to do. The budget is the forcing function.

Drain and Capacity Planning¶

Drain takes capacity from the cluster — one pod is offline for drain_duration + restart_duration. Across rolling deploys:

• N pods, drain D, restart R, deploy parallelism P.
• Effective capacity loss during deploy: P * (D + R) / N pods worth of capacity.
• Total deploy time: N / P * (D + R).

For N=100, P=10, D=20s, R=10s: 10% capacity loss during a 5-minute deploy.

If your service runs at 80% capacity, a 10% loss is fine. At 95% capacity, it is an outage.

The senior engineer thinks about this at capacity-planning time.

Hot Path Cost of Drain Tracking¶

In-flight tracking has runtime cost. Mostly small, but at scale it matters.

Cost of WaitGroup.Add(1) ; defer wg.Done()¶

• 2 atomic operations (Add up, Add down).
• 1 deferred function call.
• ~20-50 ns per request.

For 100k RPS: ~5 ms/s of CPU. Negligible.

Cost of an atomic counter¶

• 2 atomic operations.
• No defer.
• ~10-20 ns per request.

Slightly cheaper. For ultra-hot paths.

Cost of a sharded counter¶

• 2 atomic operations, but on different cache lines per CPU.
• Eliminates contention.
• ~5-10 ns per request.

Used by tracing libraries. For drain tracking, usually overkill.

Optimisation: only track during drain¶

type Tracker struct {
    active atomic.Bool
    count  atomic.Int64
}

func (t *Tracker) Begin() {
    if t.active.Load() {
        t.count.Add(1)
    }
}

active is set true at drain start. Before that, no tracking. Saves the atomic on the hot path.

The trade-off: at drain start, you don't know the count. You can only measure new in-flight from that moment on. For services where drain is brief, this is acceptable.

Drain in Polyglot Stacks¶

In a polyglot service (Go front-end, Python ML backend, Rust database), drain semantics differ. Each language has its own primitives. Coordination patterns:

Shared signal source¶

All processes respond to the same SIGTERM. Each drains in its own way. Orchestrator's grace period covers the slowest.

Coordinated drain message¶

A control plane broadcasts "drain now" via a queue. Each service handles it. Useful when drain timing matters across services.

Hierarchical drain¶

A parent service tells its children to drain via API call. Children drain themselves. Parent waits.

The pattern depends on the topology. Senior engineers design the topology to make drain simple.

Drain Incidents — A Case Study¶

A real incident (anonymised). A team running 50 pods, drain time 25s, grace 30s. Last week's deploy: 8 pods exited with SIGKILL. Customer-facing 5xx spike at 0.3%. Pager goes off.

Investigation:

1. Pull goroutine dumps from a SIGKILL-ed pod via pre-stop hook.
2. Find: 100+ goroutines stuck in db.QueryContext(rootCtx, ...).
3. Root cause: rootCtx was cancelled at drain start, but the query was already past the cancellation check — it had sent the SQL and was waiting for the response.
4. The Postgres driver's QueryContext has a 60-second default statement timeout. The drain deadline of 25s is shorter.
5. The drain waits for these queries; the deadline fires; force-cancel; but the driver does not interrupt the wire-level read fast enough.

Fix:

1. Set Postgres statement_timeout to a value less than drain budget (e.g., 10s).
2. Add a worker-pool-level deadline on each query.
3. Log queries that exceed deadline so we know which to optimise.

Lesson: drain budget must be coordinated with all downstream timeouts. A query timeout longer than drain deadline is a drain bug, not a query bug.

This is the kind of incident review a senior engineer leads. The fix is small; the root cause analysis is the value.

Drain and Sidecars¶

In a service mesh with sidecars, drain is multi-process. Common issues:

Sidecar exits first¶

The sidecar receives SIGTERM and exits immediately. The app cannot reach the network. Drain fails because the app cannot flush.

Fix: configure the sidecar to drain after the app, or to wait for the app to exit.

App exits first¶

The app drains in 5s; sidecar continues for 25s; the orchestrator waits for both. Total deploy time is dominated by the sidecar.

Fix: align sidecar drain budget with app drain.

Sidecar drops connections during drain¶

The sidecar may close idle connections aggressively, breaking the app's in-flight requests.

Fix: configure sidecar terminationDrainDuration to a sane value.

Sidecar fails to drain¶

A bug in the sidecar leads to SIGKILL. Drain logs show clean app exit but the deploy is slow.

Fix: file a bug with the sidecar maintainers; in the meantime, lower the sidecar's drain time.

A senior engineer treats the sidecar as a peer service with its own drain semantics, not as a black box.

Drain and Long-Running Jobs¶

Some workloads (video transcoding, ML batch inference, large ZIP unpacks) cannot fit in a 25-second drain. Strategies:

Strategy: Different pod types¶

Web pods drain in 25s; worker pods get 5-minute grace. The orchestrator handles them differently. Long jobs go to worker pods.

Strategy: Checkpoint and resume¶

Job state is persisted every minute. On drain, the partial result is saved. Another worker resumes from the checkpoint.

type Job struct {
    ID         string
    Progress   int
    State      []byte
}

func (j *Job) Run(ctx context.Context) error {
    for {
        if err := ctx.Err(); err != nil {
            j.Save(context.Background())
            return err
        }
        j.Progress++
        j.State = compute(j.State)
        if j.Progress%100 == 0 {
            _ = j.Save(ctx)
        }
        if j.Done() {
            return nil
        }
    }
}

The drain becomes a forced checkpoint.

Strategy: External execution¶

The Go service submits the long job to a job runner (Argo, Airflow, etc.). The runner is separately managed. Drain of the submitter does not affect the running job.

The trade-off: more infrastructure, but cleaner drain semantics.

Mentoring Through Drain¶

A senior engineer teaches drainability without taking over. Tactics:

1. PR comments with patterns, not solutions. "What happens if SIGTERM arrives now?" Let the author answer.
2. Code review checklists. The drainability checklist (above) is a forcing function.
3. Pair-program one drain test. Then the engineer can write the rest.
4. Incident reviews that focus on root cause. Drain bugs almost always have a teaching moment.
5. Templates in the org's repo. A service-template repo with drain built in saves dozens of hours per new service.
6. Lunch-and-learn on drain. 30 minutes; live coding; question time. Reaches more engineers than 1:1s.

The goal: drain becomes implicit knowledge across the team. Newcomers absorb it from existing code without needing a special class.

Self-Assessment Checklist¶

• I can sketch a drain dependency graph for any service in 5 minutes.
• I lead drain code reviews and catch invariant violations.
• I have debugged at least one production drain incident from a goroutine dump.
• I budget drain time based on measured P99s, not guesses.
• I integrate drain with leader election, sagas, and stateful systems.
• I mentor engineers in drainability without explicit teaching.
• I treat drain as a first-class architectural property.
• I write drain tests at unit, integration, soak, and chaos levels.
• I align drain budget with downstream timeouts (DB, RPC, etc.).
• I track drain metrics fleet-wide and alert on regressions.

Summary¶

At the senior level, drain stops being a recipe and starts being a system property. You design for it from the first commit. You measure it, budget it, test it across levels, and mentor others into the habit. The result is a team that ships fearlessly: every deploy is a non-event because every service drains cleanly by default.

This is also the level where drain interacts with the broader system: leader election, sagas, sidecars, polyglot stacks, long-running jobs. The mid-level patterns become tools in a larger toolkit. The judgement of when to use which tool is what distinguishes senior from middle.

Once you can think in invariants and budgets, the patterns generalise. The next page, professional.md, goes deeper into Kafka rebalances, exactly-once semantics, partition revocation, and production war stories.

Extended Section: Drain Patterns Catalogue¶

Pattern: Two-phase quiesce-drain¶

type Service struct {
    quiescing atomic.Bool
    draining  atomic.Bool
}

func (s *Service) Quiesce() {
    s.quiescing.Store(true)
}

func (s *Service) Drain(ctx context.Context) error {
    s.draining.Store(true)
    return s.wait(ctx)
}

Pattern: Drain registry¶

type Drainable interface{ Drain(ctx context.Context) error }

type Registry struct {
    mu sync.Mutex
    items []Drainable
}

func (r *Registry) Register(d Drainable) {
    r.mu.Lock()
    defer r.mu.Unlock()
    r.items = append(r.items, d)
}

func (r *Registry) DrainAll(ctx context.Context) error {
    r.mu.Lock()
    items := append([]Drainable{}, r.items...)
    r.mu.Unlock()
    for i := len(items) - 1; i >= 0; i-- {
        _ = items[i].Drain(ctx)
    }
    return nil
}

Pattern: Drain by group¶

type Group struct {
    items []Drainable
}

func (g *Group) Drain(ctx context.Context) error {
    var eg errgroup.Group
    for _, it := range g.items {
        it := it
        eg.Go(func() error { return it.Drain(ctx) })
    }
    return eg.Wait()
}

Components in the same group drain in parallel.

Pattern: Drain with backpressure¶

type RateLimitedQueue struct {
    in chan Item
    rate atomic.Int64
}

func (q *RateLimitedQueue) Drain(ctx context.Context) error {
    q.rate.Store(0) // backpressure to zero
    for len(q.in) > 0 {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(10 * time.Millisecond):
        }
    }
    return nil
}

Pattern: Drain with snapshot¶

type Stateful struct {
    state map[string][]byte
}

func (s *Stateful) Drain(ctx context.Context) error {
    return s.snapshot.Save(ctx, s.state)
}

Extended Section: A Day in the Life of a Senior Drain Engineer¶

8:00 — Glance at the deploy dashboard. Yesterday's deploys: 12 in production, drain P99 = 4.2s. No alerts. Fine.

8:30 — Stand-up. A teammate mentions drain duration crept up last week. Add to my list.

9:00 — Pull the time series. Drain P99 went from 3.1s to 4.2s over 10 days. Correlates with a new feature that adds a Kafka consumer.

9:30 — Review the Kafka consumer's drain. It uses Reader.Close but does not commit pending offsets. Open a PR fix.

10:30 — Code review. New service with no Drain method. Comment: "How do we drain this?" Author replies: "Oh, I forgot." Fix in 20 minutes.

11:00 — Design review for a new feature: a long-running batch job. Discuss drain strategy. Settle on checkpoint-and-resume.

13:00 — Lunch.

14:00 — Pair with a junior on writing a drain test. They get it working in 30 minutes.

15:00 — Incident: a pod hit drain deadline. Pull goroutine dump. Find a hung HTTP request to a flaky downstream. The downstream's timeout was 30s; drain budget was 25s. Fix: lower the downstream timeout to 15s.

16:00 — Write up the incident. Add a check to CI: any HTTP client with a timeout > drain budget fails the build.

17:00 — Update the team wiki with the new check. EOD.

This is what senior drain work looks like: a mix of monitoring, mentoring, incident response, and tooling improvements. The role is preventive more than reactive.

Extended Section: Drain Interview Mock — Senior¶

Interviewer: Walk me through how you'd add drain to an existing 50k-line Go service that has none.

You: "First, I'd audit. List all goroutines: search for go func, go method, go funcLit. For each, identify the owner and the exit path. Most of them should be in a long-lived struct with a Run method. The rest are leaks — fix those first.

Then I'd add a Drainable interface. Implement it on each owner. Wire them through a Supervisor that drains in reverse construction order.

Then main: replace whatever signal handling exists with signal.NotifyContext. Add the drain deadline derived from grace period minus safety margin.

Add the readiness flip and propagation sleep. Add metrics: drain_duration_seconds, drain_force_cancelled_total. Hook them up.

Write drain tests: empty, in-flight, hung, double-call. Run with -race.

Roll out to one canary pod. Watch metrics for a week. Then full rollout. Total time: two weeks for a 50k-line service."

Interviewer: What's the riskiest part?

You: "The drain test in CI. If it passes locally but fails in CI, the team disables it. Then drain breaks silently for months. Need to make the test stable and fast — under 30 seconds — so people respect it."

Interviewer: How do you measure success?

You: "Three metrics: deploy frequency goes up (people trust it), 5xx rate during deploys stays flat, drain P99 stays well under the grace period. If all three trend right for a quarter, drain is healthy."

Closing Thoughts At Senior Level¶

The senior view of drain is that it is infrastructure. Like logging, metrics, and config, drain is what every service has by default. Engineering effort goes into making it cheap, observable, and uniform across the org. Once you reach that state, drain stops being a topic of conversation — it just works, and the team focuses on features.

The professional page expands on the most demanding scenario: drain in Kafka consumers with exactly-once semantics. That is the senior view stretched into a real, hairy, production system.

Appendix A: A Detailed Drain Architecture Reference¶

This appendix walks through a senior-level drain architecture in detail.

The Drainable contract¶

// Drainable is implemented by anything with a lifecycle of running goroutines
// or owned resources that must be released cleanly before process exit.
//
// Contract:
//
//   - Drain must be safe to call exactly once. A second call returns nil
//     immediately (or, if your implementation prefers, an error).
//
//   - Drain must block until either:
//       * all in-flight work has completed, OR
//       * the provided context has expired.
//
//   - Drain must not panic on already-closed channels, double-cancelled
//     contexts, or concurrent calls.
//
//   - Drain returns the context error on deadline expiry, nil on clean
//     completion, or another error for transient failures.
//
//   - After Drain returns, no further work should be accepted by the
//     component. Submit-like methods should reject with a sentinel error.
type Drainable interface {
    Drain(ctx context.Context) error
}

This contract is short but exact. Every implementation should be auditable against it in three minutes.

The Lifecycle owner¶

// Lifecycle owns a sequence of (start, drain) pairs and runs them in order.
// It is the standard wiring at the top of main.
type Lifecycle struct {
    mu     sync.Mutex
    starts []func(context.Context) error
    drains []func(context.Context) error
    names  []string
    logger *slog.Logger
}

func New(logger *slog.Logger) *Lifecycle {
    return &Lifecycle{logger: logger}
}

func (l *Lifecycle) Add(name string,
    start func(context.Context) error,
    drain func(context.Context) error) {
    l.mu.Lock()
    defer l.mu.Unlock()
    l.starts = append(l.starts, start)
    l.drains = append(l.drains, drain)
    l.names = append(l.names, name)
}

func (l *Lifecycle) Start(ctx context.Context) error {
    for i, s := range l.starts {
        l.logger.Info("starting", "component", l.names[i])
        if err := s(ctx); err != nil {
            return fmt.Errorf("start %s: %w", l.names[i], err)
        }
    }
    return nil
}

func (l *Lifecycle) Drain(ctx context.Context) {
    for i := len(l.drains) - 1; i >= 0; i-- {
        start := time.Now()
        err := l.drains[i](ctx)
        l.logger.Info("drained",
            "component", l.names[i],
            "duration", time.Since(start),
            "err", err,
        )
    }
}

A Lifecycle is the smallest piece that orchestrates drain. Components register themselves; main calls Start and Drain.

The Supervisor¶

For long-running services with multiple goroutines, a supervisor wraps errgroup with lifecycle awareness.

type Supervisor struct {
    lc  *Lifecycle
    g   *errgroup.Group
    ctx context.Context
}

func NewSupervisor(ctx context.Context, lc *Lifecycle) *Supervisor {
    g, gCtx := errgroup.WithContext(ctx)
    return &Supervisor{lc: lc, g: g, ctx: gCtx}
}

func (s *Supervisor) Spawn(name string, fn func(ctx context.Context) error) {
    s.g.Go(func() error {
        err := fn(s.ctx)
        return err
    })
}

func (s *Supervisor) Wait() error {
    return s.g.Wait()
}

func (s *Supervisor) Drain(ctx context.Context) {
    s.lc.Drain(ctx)
}

Supervisor is the runtime; Lifecycle is the configuration. The two collaborate.

Appendix B: Detailed Two-Phase Shutdown¶

Two-phase shutdown is sometimes called "quiesce + drain." The principle is to bias the system toward completion before fully stopping intake.

When two-phase is worth it¶

• The system has multi-stage workflows (sagas, pipelines).
• Some workflows can be cancelled cheaply; others cannot.
• The quiesce hint allows the system to prioritise completing the expensive ones.

Implementation outline¶

type Service struct {
    quiescing atomic.Bool
    draining  atomic.Bool
}

// Handler logic:
func (s *Service) StartWork(ctx context.Context, kind Kind) error {
    if s.draining.Load() {
        return errors.New("draining")
    }
    if s.quiescing.Load() && kind.LongRunning() {
        return errors.New("quiescing")
    }
    // proceed
    return nil
}

// Shutdown sequence:
func (s *Service) Shutdown(ctx context.Context) error {
    // Phase 1: quiesce.
    s.quiescing.Store(true)
    time.Sleep(s.quiescePeriod)
    // Phase 2: drain.
    s.draining.Store(true)
    return s.waitInflight(ctx)
}

StartWork refuses long-running work during quiesce, but allows short-running work to proceed. This biases the in-flight set toward completion.

Quiesce period¶

Tune to your workload. Typical values: 5-15 seconds. Long enough for short-running tasks to finish; short enough to leave drain budget.

Quiesce and metrics¶

Emit a metric for quiesce_started_total and quiesce_duration_seconds. Helps tune the period over time.

Appendix C: Supervisor Patterns From OTP¶

Erlang/OTP supervisor strategies translate well to Go. A quick tour:

one-for-one¶

If component A crashes, restart only A. Other components continue.

type OneForOne struct {
    components map[string]Component
}

func (s *OneForOne) Run(ctx context.Context) error {
    g, gCtx := errgroup.WithContext(ctx)
    for name, c := range s.components {
        name, c := name, c
        g.Go(func() error {
            for {
                err := c.Start(gCtx)
                if err == nil || errors.Is(err, context.Canceled) {
                    return err
                }
                log.Printf("%s crashed: %v, restarting", name, err)
                select {
                case <-gCtx.Done():
                    return gCtx.Err()
                case <-time.After(time.Second):
                }
            }
        })
    }
    return g.Wait()
}

one-for-all¶

If any component crashes, drain everyone, exit.

This is what errgroup gives by default — when one goroutine returns an error, the group context cancels, others see it.

rest-for-one¶

If component A crashes, restart A and everything downstream of A.

Requires a dependency graph. Restart in topological order.

Choice¶

For most Go services, one-for-all is simplest and safest. Crashes are surfaced; the operator decides whether to restart. Restart loops can hide problems.

Appendix D: Drain in Service Mesh Sidecars¶

A service mesh sidecar (Envoy via Istio, Linkerd proxy) intercepts traffic. Drain ordering between app and sidecar matters.

Recommended sequence¶

1. Orchestrator sends SIGTERM to all containers in the pod.
2. App receives it; flips readiness off.
3. Sidecar receives it; configured to enter "drain" mode (no new connections accepted, existing ones continue).
4. App finishes in-flight; exits.
5. Sidecar's drain period expires; sidecar exits.
6. Pod terminates.

Configuration¶

Istio: set terminationDrainDuration on the proxy. Linkerd: similar setting in Linkerd2-Proxy.

The sidecar's drain duration should equal the app's grace period minus a buffer.

A bug to know¶

If the sidecar exits before the app, the app cannot reach the network. Outbound calls fail. Drain might be interrupted.

Mitigation: ensure sidecar drain duration >= app drain time. Some platforms have a preStop hook that sleeps to enforce this.

Appendix E: Drain in Stateful Sets¶

A StatefulSet in Kubernetes runs ordered, persistent pods (databases, queues). Drain is more delicate:

• Each pod has a stable identity.
• Pods are drained one at a time, in reverse order.
• Each pod's drain must complete before the next starts.

For application code, this means:

• Drain is single-threaded across the cluster.
• The orchestrator's per-pod grace period bounds each drain, not the total.
• Replication catch-up may delay drain.

A typical stateful-set drain:

1. Pod N (highest index) receives SIGTERM.
2. Pod N transfers any leader role / shards to others.
3. Pod N flushes WAL and snapshots state.
4. Pod N closes replication channels cleanly.
5. Pod N exits.
6. Replacement pod N starts, syncs from disk, joins cluster.

The drain function must handle each step explicitly. Stateful services have larger drain codebases.

Appendix F: Drain in Distributed Locks¶

A service holding distributed locks (Redis, etcd, Consul) must release them on drain. Otherwise, lock holders block other pods for the lock's TTL.

type LockHolder struct {
    locks sync.Map // map[string]*Lock
}

func (lh *LockHolder) Drain(ctx context.Context) error {
    var wg sync.WaitGroup
    lh.locks.Range(func(_, v any) bool {
        l := v.(*Lock)
        wg.Add(1)
        go func() {
            defer wg.Done()
            _ = l.Release(ctx)
        }()
        return true
    })
    done := make(chan struct{})
    go func() { wg.Wait(); close(done) }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

Release all locks in parallel. Each release is an RPC; total wall-clock is roughly one RPC.

If release fails, the lock will expire on its own (via TTL). That is the back-stop — drain should try, but the TTL is the guarantee.

Appendix G: Drain and Hot-Reload¶

Some services support config hot-reload via SIGHUP. The interaction with drain:

• SIGHUP triggers reload, not drain.
• SIGTERM triggers drain.
• The two should not be confused.

ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer cancel()

hup := make(chan os.Signal, 1)
signal.Notify(hup, syscall.SIGHUP)
go func() {
    for range hup {
        _ = svc.Reload()
    }
}()

SIGHUP and SIGTERM are handled separately. Reload is in-process; drain leads to exit.

Appendix H: Drain on Crash¶

If the process is about to crash (panic, OOM warning), should it drain?

For panic: deferred functions run, but the process exits with the panic. A panic in a worker goroutine does not naturally drain the rest. You can use recover to log and continue, but at that point you have no guarantees.

For OOM: too late. The kernel has decided to kill you.

For OOM warnings (high memory): proactively drain. Some services monitor their own memory and drain when above a threshold.

go func() {
    for range time.NewTicker(time.Second).C {
        var m runtime.MemStats
        runtime.ReadMemStats(&m)
        if m.HeapInuse > 0.9 * memoryLimit {
            log.Println("memory pressure, draining")
            cancel() // triggers drain
            return
        }
    }
}()

Pre-emptive drain prevents OOM-kill, which is unceremonious.

Appendix I: Drain Discipline Across A Codebase¶

A 50k-line codebase has many places where drain can break. A discipline matrix:

Automate as much as possible. Manual review is unreliable across a team.

Appendix J: Concurrency Patterns That Make Drain Harder¶

Some Go concurrency patterns are powerful but make drain harder:

Pattern: pipeline with chained channels¶

out1 := stage1(in)
out2 := stage2(out1)
out3 := stage3(out2)

Drain requires closing in, then each stage closes its output after exhausting its input. Errors midway are hard to surface.

Mitigation: use errgroup per stage; pass context; close output channels via defer.

Pattern: fan-in from many sources¶

merged := merge(src1, src2, src3)

merge typically launches a goroutine per source. Drain requires draining each goroutine; the merge goroutine closes the output after all sources finish.

Mitigation: explicit WaitGroup inside merge; close output via defer wg.Wait(); close(out).

Pattern: goroutine per request¶

Each request spawns a goroutine. The server tracks them with a WaitGroup. Simple, but the wait group becomes a contention point at high RPS.

Mitigation: sharded counter; or use HTTP middleware (which already tracks).

Pattern: pub-sub with broker goroutine¶

A broker goroutine receives messages and distributes to subscribers. Drain: cancel context, broker exits, subscribers exit.

Mitigation: explicit Stop method on broker; subscribers see channel close.

Appendix K: Drain Friction Audit¶

Track "drain friction" — places in the codebase where adding drain is non-trivial. A friction audit:

1. Pick 5 random goroutines.
2. For each, trace the path from spawn to exit.
3. Count the number of steps.
4. Identify which steps lack a drain-aware mechanism.

A score: average steps × (1 - drain-aware steps / total steps). Lower is better.

Friction predicts incident rate. High-friction code drains by accident; low-friction code drains by design.

Aim for friction below a target (say, 3.0 average steps with > 90% drain-aware). Anything above is a refactoring target.

Appendix L: The Senior Code Review¶

A senior code review of a drain change focuses on these questions:

1. What changes if SIGTERM arrives now? Walk through the new code.
2. What is the worst-case drain time of this change? Multiply P99 by 1.5x.
3. Does this introduce a new long-lived goroutine? If yes, where is its Drain?
4. Does this change touch the drain order? If yes, document it.
5. Is there a test for the drain behaviour? If not, request one.
6. Does this share a deadline with other drains? Verify the budgets sum.
7. Are there any hidden timeouts that exceed the drain budget? (DB queries, HTTP clients.)

Five-minute review; catches 80% of drain bugs.

Appendix M: Drain Across Versions¶

A rolling deploy has v1 pods and v2 pods running simultaneously. Drain interacts with version skew:

• v1 may have different drain semantics than v2.
• Cross-version protocol changes during drain are dangerous.
• State migrations during drain may race.

Mitigations:

• Backwards-compatible API changes.
• Two-step migrations (write v1+v2, read v2 only, write v2 only).
• Feature flags to enable new drain logic gradually.

The senior engineer thinks about version skew when introducing new drain mechanics. "How does this interact with the previous version draining nearby?"

Appendix N: Drain Logging In Production¶

Three levels of log:

Level 1 — start and end¶

INFO drain started ts=2026-05-15T03:14:15Z budget=25s
INFO drain complete ts=2026-05-15T03:14:18Z duration=3.2s

Always emit these. They are the spine of every drain-related investigation.

Level 2 — phases¶

INFO drain phase=readiness ts=...
INFO drain phase=http elapsed=2.1s err=
INFO drain phase=workers elapsed=0.8s err=
INFO drain phase=producers elapsed=0.3s err=

Per-phase logs let you find the bottleneck.

Level 3 — events¶

DEBUG drain forced-cancel goroutine=worker-3 reason=deadline
DEBUG drain remaining in_flight=2 elapsed=24s

Optional. Useful for debugging but noisy.

Use structured logging. Make drain=true a tag so all drain logs can be filtered.

Appendix O: Drain And Capacity¶

Recap: when a pod drains, total cluster capacity drops by 1/N. If N is small or load is high, this matters.

A capacity-aware deploy:

• maxUnavailable=1 — never drop more than one pod.
• maxSurge=2 — bring up two new ones before draining old.
• Wait for new pods to be ready before continuing.

For 100 pods running at 70% capacity, this is fine. For 10 pods at 95%, you need different math.

Senior engineers run the math.

Appendix P: Drain And Cost¶

Drain costs:

• Engineering time — initial implementation and ongoing maintenance.
• Deploy duration — drain time × number of pods.
• Runtime overhead — tracking in-flight has a small CPU cost.

Costs are well-bounded. The benefit (clean deploys, no customer-visible 5xx) is large. The ratio is excellent.

The cost where teams over-invest: making drain "perfect." Diminishing returns kick in fast. A 99% clean drain is 100x easier than a 99.99% clean one. Aim for 99%; alert on outliers.

Appendix Q: Long Worked Example — Drain a Multi-Tenant Service¶

Imagine a service that serves multiple tenants from one binary: each tenant has its own queue, worker pool, and connection pool. Drain must finish each tenant cleanly.

type Tenant struct {
    id       string
    queue    chan Task
    wg       sync.WaitGroup
    draining atomic.Bool
}

func (t *Tenant) Start(ctx context.Context) {
    for i := 0; i < 4; i++ {
        t.wg.Add(1)
        go t.run(ctx)
    }
}

func (t *Tenant) run(ctx context.Context) {
    defer t.wg.Done()
    for {
        select {
        case <-ctx.Done():
            return
        case task, ok := <-t.queue:
            if !ok {
                return
            }
            t.process(ctx, task)
        }
    }
}

func (t *Tenant) Drain(ctx context.Context) error {
    if !t.draining.CompareAndSwap(false, true) {
        return nil // already drained
    }
    close(t.queue)
    done := make(chan struct{})
    go func() { t.wg.Wait(); close(done) }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

type Service struct {
    mu      sync.RWMutex
    tenants map[string]*Tenant
}

func (s *Service) Drain(ctx context.Context) error {
    s.mu.RLock()
    tenants := make([]*Tenant, 0, len(s.tenants))
    for _, t := range s.tenants {
        tenants = append(tenants, t)
    }
    s.mu.RUnlock()

    // Drain tenants in parallel, but each bounded by the same context.
    var eg errgroup.Group
    for _, t := range tenants {
        t := t
        eg.Go(func() error {
            if err := t.Drain(ctx); err != nil {
                log.Printf("tenant %s drain: %v", t.id, err)
            }
            return nil
        })
    }
    return eg.Wait()
}

The pattern: each tenant is a Drainable; the service Drain runs all tenants in parallel under one context. Total wall-clock is bounded by the slowest tenant.

Variations¶

• Sequential drain. Replace errgroup with a for-loop. Useful when tenants share a resource and parallel drain causes contention.
• Throttled drain. Drain tenants in batches of K to limit concurrency.
• Priority drain. Drain high-priority tenants first; lower priority may be cancelled.

Each variation has uses; the senior engineer chooses based on the workload.

Appendix R: Drain Failure Modes¶

A catalogue of failure modes seen in production:

Failure mode 1: Deadlock during drain¶

Drain holds lock A; a worker is waiting on lock A; the wait group never reaches zero. Cause: holding locks during drain. Fix: release before blocking on drain.

Failure mode 2: Stuck on closed channel¶

A worker did select { case ch <- v: ... } where ch was closed by drain. Sends on closed channels panic. Cause: producer did not check drain state. Fix: gate sends with the drain flag.

Failure mode 3: Drain context cascaded too aggressively¶

A sub-component derived its context from a context that already expired. Drain returns DeadlineExceeded instantly. Cause: wrong context choice. Fix: derive from context.Background with fresh timeout.

Failure mode 4: Wait group counter wrong¶

Add(1) was paired with two Done() calls (or zero). The wait group either over-decrements (panic) or never reaches zero. Cause: bug in worker. Fix: every Add(1) paired with exactly one Done().

Failure mode 5: Goroutine leak¶

A goroutine has no exit path. Drain finishes, but the goroutine lives on. Cause: missing <-ctx.Done case. Fix: add the case.

Failure mode 6: Premature health flip¶

Readiness flipped after listener closed. LB sends traffic; pod refuses. Brief 5xx. Cause: wrong order. Fix: readiness first, listener second.

Failure mode 7: Drain budget too short¶

Drain deadline shorter than P99 handler. Some requests cancelled. Cause: misconfigured. Fix: raise budget.

Failure mode 8: Drain budget longer than grace period¶

Drain takes 35s, grace is 30s, SIGKILL arrives mid-drain. Cause: misconfigured. Fix: budget < grace.

Failure mode 9: Async producer drops messages¶

Buffered producer not flushed before close. Cause: missing Flush. Fix: add Flush(ctx) before Close().

Failure mode 10: Cascade cancellation¶

One component's drain triggers another's, recursively. Cause: components calling each other's drain. Fix: only the supervisor calls drain.

Each failure mode has a name and a known fix. Build a library of these in your team's wiki.

Appendix S: Drain Audit Worksheet¶

Use this for a 30-minute drain audit of any service:

1. List all components (HTTP server, workers, producers, consumers, cron, etc.).
2. For each, locate the Drain (or equivalent) method.
3. For each, check: context parameter, deadline honoured, no panic on double call.
4. List the drain order. Verify it matches dependency reverse.
5. Find the signal handler. Verify SIGTERM is handled.
6. Find the readiness handler. Verify it reflects drain state.
7. Find the propagation sleep. Verify it is present.
8. Find the drain metrics. Verify duration and force-cancelled are emitted.
9. Find the drain tests. Verify empty + hung + double-call cases.
10. Check downstream timeouts. Verify all < drain budget.

A pass on all 10 is a healthy service. Any miss is fixable in under an hour.

Appendix T: Drain Across Generations Of A Codebase¶

A team's drain code evolves over time:

• Generation 1. No drain. os.Exit everywhere. Production runs okay because deploys are rare.
• Generation 2. Ad-hoc drain. Each service has its own approach. Inconsistent. Maintainable by the original author only.
• Generation 3. Shared drain helpers. A pkg/drain library in the monorepo. Most services use it.
• Generation 4. Drain is invisible. The framework handles it. Engineers do not think about drain except to add a new component.

Aim for generation 4. The framework looks like:

fw := framework.New(myConfig)
fw.RegisterHTTP(myHandler)
fw.RegisterWorker(myWorkerFn)
fw.RegisterProducer(myProducer)
fw.Run() // handles signals, drain, exit

Engineers register; the framework drains. Drain bugs are framework bugs, not application bugs.

Appendix U: Building The Framework¶

If you build the framework yourself, the architecture:

type Framework struct {
    lc       *Lifecycle
    graceTime time.Duration
}

func New(cfg Config) *Framework {
    return &Framework{
        lc: NewLifecycle(),
        graceTime: cfg.GracePeriod - 5*time.Second,
    }
}

func (f *Framework) RegisterHTTP(h http.Handler) {
    srv := &http.Server{Addr: ":8080", Handler: h}
    f.lc.Add("http",
        func(ctx context.Context) error {
            go srv.ListenAndServe()
            return nil
        },
        func(ctx context.Context) error {
            return srv.Shutdown(ctx)
        },
    )
}

func (f *Framework) Run() error {
    ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
    defer cancel()
    if err := f.lc.Start(ctx); err != nil {
        return err
    }
    <-ctx.Done()
    dctx, dcancel := context.WithTimeout(context.Background(), f.graceTime)
    defer dcancel()
    f.lc.Drain(dctx)
    return nil
}

Add RegisterWorker, RegisterProducer, etc. Each follows the same shape.

A senior engineer can build this in a day. The team benefits for years.

Appendix V: Real-World Drain Trade-offs¶

Drain decisions in production are almost always trade-offs:

• Drain longer → cleaner exit. But longer deploys.
• Drain shorter → faster deploys. But more force-cancellations.
• More phases → finer control. But more code.
• Sequential drain → easier to reason. But slower wall-clock.
• Parallel drain → faster wall-clock. But harder ordering.

There is no globally correct answer. The senior engineer makes the trade-off explicit, documents it, and revisits as the service evolves.

Appendix W: A Drain Code Smell Checklist¶

Smells specific to drain code:

• A function called Drain that does not take a context.Context.
• A function called Shutdown that returns without an error.
• A Stop method with no documentation on semantics.
• A goroutine spawn without a documented exit path.
• A for { ... } without a select with context.
• A time.Sleep longer than 100ms without context awareness.
• A wg.Wait() without a select and deadline.
• A <-ch without a select and context.

Any of these in a code review should trigger discussion.

Appendix X: Closing Remarks¶

Senior-level drain is mostly judgement: which patterns to use, which to skip, which to mentor on. The patterns themselves are taught at junior and middle. The judgement is built by years of incident reviews, code reviews, and design discussions.

The good news: this judgement is teachable. Junior engineers under good mentorship learn it in a year or two. The patterns above are the curriculum.

A team where drain is invisible — where new services have it by default and old services have been retrofitted — is a team that ships fast and sleeps well. That is the goal.

Move on to professional.md for the deepest drain scenario: Kafka consumer rebalances with exactly-once semantics, partition revocation, and the full ceremony of production-grade messaging.

Appendix Y: Drain Across Process Restarts¶

A pod that crashes and restarts goes through:

1. Old process death (clean drain or forced kill).
2. Restart delay (Kubernetes pod restart policy).
3. New process start.
4. Steady state resumes.

The interaction with drain:

• A clean drain leaves no orphaned state.
• A forced kill may leave: half-flushed Kafka offsets, uncommitted DB transactions, held distributed locks (until TTL).
• The new process must handle this gracefully — it cannot assume clean state.

A drain-aware design assumes the previous instance may have died ungracefully. Recovery code accepts:

• Duplicate messages.
• Stale leases.
• Half-applied state.

The drain pattern reduces these but does not eliminate them. The recovery code is the back-stop.

Appendix Z: Drain And Idempotency¶

Idempotent operations make drain easier. If process(msg) is idempotent, force-cancellation followed by retry is safe.

func process(ctx context.Context, msg Message) error {
    if alreadyProcessed(msg.ID) {
        return nil
    }
    if err := apply(ctx, msg); err != nil {
        return err
    }
    return markProcessed(msg.ID)
}

The order matters:

• Check alreadyProcessed first to avoid duplicates.
• apply then markProcessed — if the process dies between, the next attempt retries; idempotent.

Idempotency is the strongest invariant for distributed-system drain. Invest in it before investing in drain finesse.

Appendix AA: Drain Under Resource Constraints¶

A pod that is memory-constrained must drain carefully:

• Drain in-flight may increase memory short-term (more requests held).
• An OOM during drain corrupts state.
• Pre-emptive drain on memory pressure is one technique (see earlier).

Strategies:

1. Bounded in-flight. Reject new requests if in-flight count is high; gives breathing room during drain.
2. Memory budget per request. A request that allocates too much is rejected; aggregate memory stays bounded.
3. Backpressure at the queue. Slow producers if the pipeline is full.

A pod that often OOMs during drain has a deeper memory bug. Drain is the symptom; the fix is upstream.

Appendix BB: Drain And Connection Reuse¶

HTTP keep-alive connections persist after a request completes. On drain:

• New requests on existing keep-alive connections may arrive.
• Server.Shutdown closes idle connections, then waits for active ones.

If a client opens a keep-alive connection and sends slowly, the server might keep it active for the full drain budget.

Mitigations:

• Set Server.ReadTimeout and Server.IdleTimeout to bound how long a keep-alive can sit idle.
• Use HTTP/2 graceful close (GOAWAY frame) to tell clients to reconnect.

For gRPC: the server sends GOAWAY on GracefulStop. Clients see it and route new RPCs elsewhere.

Appendix CC: Drain And Connection Multiplexing¶

HTTP/2 multiplexes many streams over one TCP connection. Drain interacts:

• Closing the connection abruptly cancels all streams.
• GOAWAY signals "no new streams; existing streams may complete."

net/http HTTP/2 server handles GOAWAY during Shutdown automatically. For custom implementations, send GOAWAY before closing.

Appendix DD: Drain And Streaming Uploads¶

A streaming upload (chunked transfer) may take minutes. Drain interacts:

• The handler holds the request body open.
• Server.Shutdown waits for the handler.
• If the upload outlasts the drain budget, the request is cancelled mid-stream.

Mitigations:

• Set a per-request timeout via r.Context().Deadline().
• Reject new uploads during drain.
• Accept partial uploads at the storage layer (resumable uploads).

A senior engineer designs the API for resumability. Upload tokens, ranges, etc. Drain becomes a non-event for clients.

Appendix EE: Drain And External Services¶

A service A that calls service B during drain must consider:

• B might also be draining.
• A's request to B might fail with 503.
• A's retry logic might compound the problem.

Best practices:

• Short timeouts on outbound calls during drain.
• Do not retry "draining" responses during drain.
• Mark outbound calls as drain-sensitive so middleware can drop them.

ctx := context.WithValue(ctx, drainKey, true)
// outbound client checks the key

This is plumbing, but it is the kind of plumbing that separates "works in prod" from "fails during deploy windows."

Appendix FF: Drain In A Single-Pod Service¶

Even a single-pod service benefits from drain. Why?

• The orchestrator may restart it for many reasons (OOM, image update, node drain).
• A restart drops in-flight work.
• Customers see 5xx for the restart duration.

The single-pod drain follows the same pattern as the multi-pod one. The grace period is the orchestrator's bound.

Common mistake: skipping drain for "small" services. The cost is low; the benefit is uniform.

Appendix GG: Drain In CLIs And Batch Jobs¶

A CLI that processes a batch of items should drain on Ctrl+C:

ctx, cancel := signal.NotifyContext(context.Background(), os.Interrupt)
defer cancel()

for _, item := range items {
    if ctx.Err() != nil {
        break
    }
    processItem(ctx, item)
}
log.Printf("processed %d of %d items", processed, len(items))

A graceful CLI exit at Ctrl+C:

• Stops at the next item.
• Logs progress.
• Returns a non-zero exit code (1, conventional for user interrupt).

A batch job runner (e.g., a cron job) should:

• Drain on SIGTERM.
• Mark the partial result so the next run can resume.
• Exit cleanly.

Appendix HH: Drain Cargo Cult¶

Some teams add drain code without understanding it. Signs of cargo cult drain:

• Drain functions that return immediately.
• Signal handlers that just call os.Exit.
• WaitGroups with no actual goroutines.
• Context deadlines on operations that never check the context.

Why it happens: copy-paste from another codebase, no testing under drain.

Cure: a drain test in CI. Cargo cult drain fails the test.

Appendix II: Drain In Library Code¶

If you write a library that internally manages goroutines, expose drain:

type Client struct{ /* ... */ }

func NewClient(opts ...Option) *Client { /* no goroutines yet */ }
func (c *Client) Start(ctx context.Context) error { /* spawn */ }
func (c *Client) Drain(ctx context.Context) error { /* drain */ }

Document semantics in package doc comments. Specify thread-safety, contract for repeated calls, what happens on context expiry.

A library that hides goroutines without drain is a library that cannot be used in production safely.

Appendix JJ: Drain Edge Cases In Real Libraries¶

Cases worth knowing:

• database/sql: db.Close() blocks until all in-use connections are returned. Combined with hung queries, this can hang forever. Mitigation: kill long queries first (SET statement_timeout).
• cloud.google.com/go/pubsub: subscription.Receive() returns when the context expires. Make sure the handler also respects the context.
• aws-sdk-go-v2: most clients accept context.Context. Pass the drain context for outbound calls.
• redis/go-redis: client.Close() is non-blocking; connections in use will error on next use. Watch your error handling.

Spend time learning the drain semantics of every library in your stack. It pays off.

Appendix KK: Drain In gRPC Streaming¶

gRPC streaming RPCs are the trickiest to drain. The handler is a long-lived function reading and writing to a stream. Drain:

func (s *Server) BigStream(in pb.Service_BigStreamServer) error {
    ctx := in.Context()
    for {
        select {
        case <-ctx.Done():
            return status.FromContextError(ctx.Err()).Err()
        default:
        }
        msg, err := in.Recv()
        if err == io.EOF {
            return nil
        }
        if err != nil {
            return err
        }
        if err := s.process(ctx, msg); err != nil {
            return err
        }
    }
}

On GracefulStop, the stream context is cancelled. The handler sees ctx.Done and returns. Client receives the error.

For sending streams:

func (s *Server) Stream(req *pb.Req, out pb.Service_StreamServer) error {
    ctx := out.Context()
    for evt := range s.events(ctx) {
        select {
        case <-ctx.Done():
            return status.FromContextError(ctx.Err()).Err()
        default:
        }
        if err := out.Send(evt); err != nil {
            return err
        }
    }
    return nil
}

The events channel must also respect cancellation. Otherwise the goroutine producing events leaks.

Appendix LL: Drain Under Test¶

Some go test integration tests share the binary's drain logic. Patterns:

func TestServer(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    srv := NewServer()
    go srv.Run(ctx)

    t.Cleanup(func() {
        dctx, dcancel := context.WithTimeout(context.Background(), time.Second)
        defer dcancel()
        _ = srv.Drain(dctx)
    })

    // ... run test ...
}