Cleanup Ordering — Senior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Core Concepts
5. Real-World Analogies
6. Mental Models
7. Pros & Cons
8. Use Cases
9. Code Examples
10. Architecture Patterns
11. Clean Code
12. Production Considerations
13. Error Handling
14. Security Considerations
15. Performance Engineering
16. Best Practices
17. Edge Cases & Pitfalls
18. Common Mistakes
19. Common Misconceptions
20. Tricky Points
21. Test
22. Tricky Questions
23. Cheat Sheet
24. Self-Assessment Checklist
25. Summary
26. What You Can Build
27. Further Reading
28. Related Topics
29. Diagrams & Visual Aids

Introduction¶

Focus: "How does cleanup ordering scale across a large codebase? How do I design resource hierarchies that compose? How do defer, AfterFunc, and shutdown choreography interact at the architecture level?"

A junior writes defer f.Close() and feels clever. A middle-level engineer writes the closure that propagates the close error and pairs AfterFunc with stop(). A senior engineer designs the system around these primitives. They decide which package owns which resource, which goroutine cleans up which state, how a 500-line shutdown sequence stays correct as teams add new components year over year.

At the senior level, cleanup is no longer a per-function concern. It is an architectural one. You think about:

• Resource hierarchies that cross package boundaries. Who closes the shared metrics exporter? Whose responsibility is the connection pool?
• Shutdown choreography in services with dozens of components. The order matters for correctness. How do you make it readable, testable, and resilient to growth?
• Cleanup that interacts with panic recovery, supervisor patterns, and crash-only design. When does panic-during-cleanup matter? When does it not?
• The interaction between context.AfterFunc, the runtime's defer machinery, and the Go scheduler. You know enough to read the source.
• Design discipline that makes cleanup composable. Every new component plugs into the same recipe.

This file assumes you internalised the junior and middle files. You should already be writing defer cancel(), defer stop(), named returns, and cancel-drain-close shutdown without thinking. The senior level builds on top.

Prerequisites¶

• Required: Full comfort with the junior and middle files of this sub-topic.
• Required: Experience building or maintaining a Go service with at least five components (HTTP server, database, cache, logger, metrics).
• Required: Awareness of supervisor patterns, structured concurrency, and graceful shutdown SLOs.
• Required: Comfort with errgroup, signal handling, and the slog structured logger.
• Helpful: Experience reading the Go standard library's net/http, database/sql, context, and runtime packages.
• Helpful: Familiarity with go vet, staticcheck, and errcheck linting passes.

If you have ever spent a week debugging "the service hangs on SIGTERM," you have the right level of scar tissue.

Glossary¶

Core Concepts¶

Concept 1: Cleanup as a contract¶

At the senior level, cleanup is part of every component's contract. When you design a new component, you ask:

• What resources does it own?
• What is its Close/Stop/Shutdown method's signature?
• Is it idempotent?
• Does it block until cleanup completes, or return immediately?
• Does it accept a context for deadline control?
• Does it return errors? What kinds?
• Can it be called from a goroutine other than the one that created it?

Codify these decisions. Document them. Test them. The contract is what lets your component compose with others.

A typical senior-level contract:

// Service is a long-lived component.
//
// Lifecycle:
//   - NewService(cfg) creates a new instance.
//   - Start(ctx) begins background work; returns when ready or on failure.
//   - Stop(ctx) initiates shutdown; returns when shutdown is complete or ctx expires.
//
// Stop is idempotent and safe to call from any goroutine.
// Stop returns any errors encountered during shutdown, joined with errors.Join.
// Stop is non-blocking; the actual shutdown work happens in the goroutine that called Start.
type Service struct { /* ... */ }

This contract is rigorous enough that any caller can use the service without reading its source.

Concept 2: The owner-releases rule¶

A core principle of cleanup design: the goroutine (or component) that creates a resource is the one that releases it. Passing a resource to another goroutine usually means transferring ownership; the receiver now owns the release.

Violations create ambiguity:

// AMBIGUOUS: who closes f?
func handler(f *os.File) {
    go process(f)
    // does handler close f, or does process?
}

Make the contract explicit:

// process owns f; it must close f before returning.
func process(f *os.File) {
    defer f.Close()
    // ...
}

In code comments and in API names, encode ownership. consumes/takes ownership of are useful phrases. The standard library does this: bufio.NewReader(r) uses r but does not own it; io.MultiReader(rs...) does not own its inputs; tar.NewReader(r) does not own r.

Concept 3: The composition rule for shutdown¶

Imagine you have three components: A, B, C. A depends on B; B depends on C. Their Shutdown order is A → B → C.

Now you compose them into a parent component, S. S's shutdown should not re-implement the order — it should delegate:

func (s *S) Shutdown(ctx context.Context) error {
    var errs []error
    if err := s.a.Shutdown(ctx); err != nil { errs = append(errs, err) }
    if err := s.b.Shutdown(ctx); err != nil { errs = append(errs, err) }
    if err := s.c.Shutdown(ctx); err != nil { errs = append(errs, err) }
    return errors.Join(errs...)
}

Each component's Shutdown is responsible for releasing its own resources, not its dependencies'. This composition keeps the code linear, even as you add more components.

Concept 4: The shutdown context¶

Always pass a fresh context to shutdown methods, with a deadline. Not the cancelled signal context.

// good
shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
s.Shutdown(shutdownCtx)

// bad: shutdown gets immediately-cancelled context
ctx, _ := signal.NotifyContext(context.Background(), syscall.SIGTERM)
<-ctx.Done()
s.Shutdown(ctx)

Why? Because the shutdown context is the budget for graceful shutdown — the window during which we try to drain in-flight work. The signal context is already cancelled by the time you're shutting down. Using it would tell every component "you have zero time to drain."

Some components, like http.Server.Shutdown, treat a cancelled context as "abort immediately, drop in-flight requests." Others might block forever waiting for non-cancellable work. Either way, give them a fresh deadline.

Concept 5: Cleanup that survives panics¶

In Go, a panic propagates up the goroutine's call stack, running deferred calls. An unrecovered panic terminates the program — including all other goroutines. For long-running services, this is too brittle. Senior design wraps every goroutine in a top-level recover:

func supervise(name string, run func() error) error {
    defer func() {
        if r := recover(); r != nil {
            slog.Error("panic", "worker", name, "value", r, "stack", debug.Stack())
        }
    }()
    return run()
}

go func() {
    for {
        if err := supervise("worker-1", workerLoop); err != nil {
            slog.Error("worker exited", "err", err)
        }
        // optionally restart, or exit
    }
}()

The recover catches anything the worker panics on, including panics from the worker's own deferred cleanups. Without this, a single panic in any defer can kill the process.

But beware: recover catches the first panic in flight. If the deferred cleanup itself panics, the recover catches its panic, losing the original. For diagnostic purposes, log both:

defer func() {
    if r := recover(); r != nil {
        slog.Error("recovered", "value", r, "stack", debug.Stack())
    }
}()

defer func() {
    if err := closer.Close(); err != nil {
        slog.Error("close failed", "err", err)
    }
}()

These two defers are independent. If closer.Close panics, the recover catches it. The order matters: the recover must be registered first (so it pops last, after the close).

Concept 6: Choreographed shutdown across goroutines¶

Real services have many goroutines: HTTP handlers, worker pools, background timers, leader-election heartbeats, metric flushers. Each has its own lifecycle. Coordinating their shutdown is a design problem, not a code problem.

The standard pattern is:

1. Signal. A SIGTERM closes the root context.
2. Propagate. Components observe the closed context via their own derived contexts.
3. Drain. Each component stops accepting new work; finishes in-flight; signals done.
4. Wait. A central coordinator (wg.Wait) blocks until every component has signalled done.
5. Close. Resources are released in dependency order.

The senior-level question is: where does the coordinator live, and how does it know about every component?

Two approaches:

• Hierarchical. A top-level Service owns sub-components. Each sub-component has its own internal coordinator. The top-level's Shutdown calls each sub-component's Shutdown in order. Errors propagate up.
• Flat registry. A LifecycleManager knows about every component. Components register themselves at startup. The manager's Shutdown runs them in LIFO of registration. Simpler but less type-safe.

Both work. Choose based on your codebase's culture.

Concept 7: AfterFunc at scale¶

For services with thousands of contexts, AfterFunc has performance implications.

• Each registration allocates a small struct and adds to the context's callback list.
• On cancel, each callback starts a new goroutine.
• Long-lived parent contexts accumulate callbacks; if you register without calling stop, they linger until the parent dies.

Senior-level use:

• Register AfterFunc only when you need its specific semantics (cleanup that outlives the function).
• Always call stop() (via defer) when you no longer need the callback.
• For high-frequency cancellation, profile: AfterFunc startup can dominate.
• Consider a single AfterFunc that fans out to many cleanups, rather than many AfterFuncs.

Concept 8: The role of runtime.SetFinalizer¶

Finalizers run when an object is garbage-collected. They are not a replacement for Close. Their proper use:

• Debugging aid. Set a finalizer that panics if Close was not called. Catches missing-close bugs in tests. Remove the finalizer in production code (it has GC cost and can mask bugs).
• Last-resort safety net. For libraries with a long history of users forgetting to close. The Go standard library uses finalizers on *os.File to close the file descriptor at GC time if Close was forgotten. This prevents FD leaks but is not a substitute for the user calling Close.

The big caveat: finalizers run at unpredictable times during GC. They cannot reliably run cleanup that has timing constraints (e.g., flushing a buffer before the program exits).

Real-World Analogies¶

Decommissioning a power plant¶

Closing a power plant is a multi-week choreographed shutdown. Steps run in dependency order: stop generating electricity (signal), let the turbines spin down (drain), cool the reactor (close in stages). Skipping a step can cause physical damage. Software shutdown lacks the physical risk but otherwise mirrors the discipline.

Closing down a hospital wing¶

A hospital wing closes in phases: stop admitting patients (cancel), transfer current patients (drain), shut down equipment in inverse dependency order (close — life support last, decorative lights first). A service's shutdown has the same phases. The patients are in-flight requests.

A symphony's coda¶

The end of a symphony is choreographed: instruments fade in reverse of how they entered. The conductor cues each section to stop. A senior engineer designs shutdown the same way: each component takes its cue at the right moment.

A research lab closing for the holidays¶

The lab does not just turn off the lights. Equipment is parked, samples are stored, the freezer is checked, the lab notebook is closed. The order is dictated by what depends on what: samples before freezer power, notebook before lights. Service shutdown is the lab closing — every component has its parking procedure.

Mental Models¶

Model 1: The dependency DAG¶

Every component in your service depends on zero or more others. Drawing the DAG (directed acyclic graph) makes the release order obvious: topological sort, with most-dependent first.

        ┌────────────────┐
        │   HTTP server  │  most dependent
        └─────┬──────────┘
              │
        ┌─────▼──────────┐
        │   handler      │
        └─────┬──────────┘
              │
   ┌──────────┼─────────┐
   │          │         │
┌──▼───┐ ┌────▼──┐ ┌────▼──┐
│  db  │ │ cache │ │ queue │
└──┬───┘ └────┬──┘ └────┬──┘
   │          │         │
   └─────┬────┴─────────┘
         │
    ┌────▼───────────┐
    │ logger / metrics │  least dependent
    └─────────────────┘

Release order: HTTP → handler → db/cache/queue → logger/metrics

In a real service, the DAG has many more nodes. The principle scales.

Model 2: Phases and gates¶

Imagine shutdown as a series of gates. Each gate has a deadline. Components must pass through their gate within the deadline; if they do not, they are forcefully closed.

Gate 1: Stop accepting new work    (deadline: instant)
Gate 2: Drain in-flight work       (deadline: 25s)
Gate 3: Close stateful resources   (deadline: 5s)

If a component fails to pass gate 2 by its deadline, the orchestrator moves on. Resources leak, but the service shuts down. The cost of the leak is bounded; the cost of hanging is not.

Model 3: A finite state machine per component¶

Each component has a state machine:

created ──[Start]──► running ──[Stop signal]──► draining ──[done]──► stopped
                         │                          │
                         └──[panic]──► crashed      └──[timeout]──► force-stopped

The senior engineer designs the state machine deliberately. Each transition has a contract. The cleanup runs at one specific transition (running → draining → stopped). Other transitions (crash, force-stop) have their own cleanup paths.

Model 4: Cleanup as a tree, not a list¶

Defers form a stack (LIFO), but a multi-component service has a tree of cleanups: each component owns a sub-tree. The top-level shutdown traverses the tree post-order: visit children first, then visit the node.

root.Shutdown:
   for each child:
      child.Shutdown      // post-order: children first
   close my own resources // then myself

The tree's shape mirrors the component hierarchy. The traversal is post-order. The result is correct release order without any centralised list.

Model 5: Cleanup as transactional rollback¶

Every operation a component does can be paired with a compensating operation that undoes it. Cleanup is the suffix of compensating operations for all completed steps. Defer's LIFO order naturally implements this: each defer is the compensating action for the work that preceded it.

For systems that span processes (distributed sagas), the same principle applies at a larger scale. Within a single process, defers and AfterFunc are the local saga compensators.

Pros & Cons¶

Pros of design-level cleanup discipline¶

• Predictable shutdown. A service that shuts down in 30 seconds every time is operationally sound. One that sometimes hangs is not.
• No leaks. Goroutines, FDs, connections, all released. Long-running services do not slowly degrade.
• Easier debugging. When something goes wrong, you know the shutdown order. You can find the offending component.
• Composability. New components plug into the existing recipe. No surprises.
• SLO-friendly. Bounded shutdown time. Bounded request loss. Bounded data loss.

Cons of strict cleanup design¶

• More code. Each component has Start, Stop, doc comments, tests. More lines than a "just leak it" approach.
• More state. Each component tracks its lifecycle. Bugs in the state machine are real.
• Harder to test. Lifecycle tests are inherently slower than unit tests (they involve actual start/stop). They are still essential.
• Cognitive load. Junior engineers may not immediately grasp why every component needs the same structure. Education and consistency help.

The cost is worth paying for any service that runs for more than a few minutes at a time.

Use Cases¶

Senior-level cleanup ordering applies whenever:

• Your service has more than a handful of components.
• Components have dependency relationships.
• The service must shut down gracefully under SIGTERM.
• The service has SLOs around shutdown time or request loss.
• The service runs in Kubernetes / a managed environment with a terminationGracePeriod.
• The service has long-lived state (caches, queues, connections) that needs explicit close.
• You have on-call rotations and want shutdown logs to be diagnostic.

If your service is a five-minute batch job that exits cleanly, you do not need this level of design. If it is a production service that runs for weeks, you do.

Code Examples¶

Example 1: A LifecycleManager¶

package lifecycle

import (
    "context"
    "errors"
    "fmt"
    "sync"
)

type Component interface {
    Name() string
    Start(ctx context.Context) error
    Stop(ctx context.Context) error
}

type Manager struct {
    mu         sync.Mutex
    components []Component
    started    bool
}

func (m *Manager) Add(c Component) {
    m.mu.Lock()
    defer m.mu.Unlock()
    if m.started {
        panic("cannot add after Start")
    }
    m.components = append(m.components, c)
}

func (m *Manager) Start(ctx context.Context) error {
    m.mu.Lock()
    m.started = true
    components := m.components
    m.mu.Unlock()

    for i, c := range components {
        if err := c.Start(ctx); err != nil {
            // unwind already-started components
            stopCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
            defer cancel()
            for j := i - 1; j >= 0; j-- {
                _ = components[j].Stop(stopCtx)
            }
            return fmt.Errorf("start %s: %w", c.Name(), err)
        }
    }
    return nil
}

func (m *Manager) Stop(ctx context.Context) error {
    m.mu.Lock()
    components := m.components
    m.mu.Unlock()

    var errs []error
    for i := len(components) - 1; i >= 0; i-- {
        if err := components[i].Stop(ctx); err != nil {
            errs = append(errs, fmt.Errorf("stop %s: %w", components[i].Name(), err))
        }
    }
    return errors.Join(errs...)
}

This is a complete, generic lifecycle manager. Components register; Start runs them in order; on partial failure, unwinds in reverse; Stop runs them in reverse of registration. errors.Join accumulates failures.

In a real service, you would extend it: parallel start where possible, configurable timeouts per component, instrumentation, etc.

Example 2: A panic-safe goroutine launcher¶

package supervisor

import (
    "context"
    "fmt"
    "log/slog"
    "runtime/debug"
)

func Go(ctx context.Context, name string, run func(context.Context) error) <-chan error {
    done := make(chan error, 1)
    go func() {
        defer close(done)
        defer func() {
            if r := recover(); r != nil {
                slog.Error("goroutine panic",
                    "name", name,
                    "value", r,
                    "stack", string(debug.Stack()))
                done <- fmt.Errorf("%s: panic: %v", name, r)
            }
        }()
        if err := run(ctx); err != nil {
            done <- fmt.Errorf("%s: %w", name, err)
        }
    }()
    return done
}

A spawned goroutine that: - Logs and converts panics into errors - Closes its done channel on exit (so callers can select) - Carries the goroutine's name in error wrapping for diagnostics

A real supervisor would add restart-on-error, exponential backoff, jittered retries. The structure above is the foundation.

Example 3: Parallel shutdown with errgroup¶

package main

import (
    "context"
    "fmt"
    "time"

    "golang.org/x/sync/errgroup"
)

type Closer interface {
    Close(ctx context.Context) error
}

func parallelShutdown(ctx context.Context, closers []Closer) error {
    g, ctx := errgroup.WithContext(ctx)
    for _, c := range closers {
        c := c
        g.Go(func() error { return c.Close(ctx) })
    }
    return g.Wait()
}

type fakeCloser struct {
    name string
    d    time.Duration
}

func (f *fakeCloser) Close(ctx context.Context) error {
    select {
    case <-time.After(f.d):
        fmt.Println("closed", f.name)
        return nil
    case <-ctx.Done():
        return fmt.Errorf("%s: %w", f.name, ctx.Err())
    }
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
    defer cancel()
    closers := []Closer{
        &fakeCloser{name: "a", d: 30 * time.Millisecond},
        &fakeCloser{name: "b", d: 50 * time.Millisecond},
        &fakeCloser{name: "c", d: 200 * time.Millisecond},
    }
    if err := parallelShutdown(ctx, closers); err != nil {
        fmt.Println("error:", err)
    }
}

When components do not depend on each other, parallel shutdown saves time. The errgroup cancels the context on first error and waits for all. c is the only one that times out.

Note: parallel shutdown is only safe for independent components. If a depends on b, sequential shutdown is required.

Example 4: A component with full lifecycle¶

package worker

import (
    "context"
    "errors"
    "fmt"
    "sync"
)

type Worker struct {
    name    string
    in      chan string
    done    chan struct{}
    cancel  context.CancelFunc
    wg      sync.WaitGroup
    once    sync.Once
    stopErr error
}

func New(name string, buffer int) *Worker {
    return &Worker{
        name: name,
        in:   make(chan string, buffer),
        done: make(chan struct{}),
    }
}

func (w *Worker) Name() string { return w.name }

func (w *Worker) Start(ctx context.Context) error {
    ctx, cancel := context.WithCancel(ctx)
    w.cancel = cancel
    w.wg.Add(1)
    go w.loop(ctx)
    return nil
}

func (w *Worker) loop(ctx context.Context) {
    defer w.wg.Done()
    defer close(w.done)
    for {
        select {
        case <-ctx.Done():
            // drain
            for {
                select {
                case m := <-w.in:
                    fmt.Println(w.name, "drain:", m)
                default:
                    return
                }
            }
        case m := <-w.in:
            fmt.Println(w.name, "process:", m)
        }
    }
}

func (w *Worker) Submit(s string) error {
    select {
    case w.in <- s:
        return nil
    case <-w.done:
        return errors.New("worker stopped")
    }
}

func (w *Worker) Stop(ctx context.Context) error {
    w.once.Do(func() {
        w.cancel()
        select {
        case <-w.done:
            w.stopErr = nil
        case <-ctx.Done():
            w.stopErr = fmt.Errorf("%s: %w", w.name, ctx.Err())
        }
    })
    return w.stopErr
}

A full-featured worker: - Idempotent Stop via sync.Once - Honors the shutdown context's deadline - Drains in-flight work on cancel - Submit fails after stop (channel send would otherwise block forever) - Records the stop error for repeated Stop calls to return

This is what a senior-level component looks like. Every behaviour is intentional.

Example 5: Wiring components into a service¶

package main

import (
    "context"
    "fmt"
    "os"
    "os/signal"
    "syscall"
    "time"

    "myapp/lifecycle"
    "myapp/worker"
)

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer stop()

    var m lifecycle.Manager
    m.Add(worker.New("worker-1", 10))
    m.Add(worker.New("worker-2", 10))
    m.Add(worker.New("worker-3", 10))

    startCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()
    if err := m.Start(startCtx); err != nil {
        fmt.Println("start failed:", err)
        os.Exit(1)
    }

    <-ctx.Done()
    fmt.Println("shutdown signal")

    stopCtx, cancel2 := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel2()
    if err := m.Stop(stopCtx); err != nil {
        fmt.Println("stop errors:", err)
    } else {
        fmt.Println("clean shutdown")
    }
}

main is short. It wires components into the manager, starts, waits for SIGTERM, stops. Adding a new component is one m.Add call. The cleanup ordering is encoded in the manager, not in main.

Example 6: A test that validates shutdown¶

package worker_test

import (
    "context"
    "runtime"
    "testing"
    "time"

    "myapp/worker"
)

func TestWorkerCleanShutdown(t *testing.T) {
    before := runtime.NumGoroutine()

    w := worker.New("test", 10)
    if err := w.Start(context.Background()); err != nil {
        t.Fatal(err)
    }
    for i := 0; i < 10; i++ {
        w.Submit("item")
    }

    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
    defer cancel()
    if err := w.Stop(ctx); err != nil {
        t.Fatal(err)
    }

    time.Sleep(10 * time.Millisecond) // give the runtime a moment

    after := runtime.NumGoroutine()
    if after > before {
        t.Errorf("goroutine leak: before=%d after=%d", before, after)
    }
}

The test asserts that after Stop, the goroutine count returns to baseline. A worker that leaks fails the test. CI catches it. This kind of test is non-negotiable for production components.

Example 7: AfterFunc-driven cleanup with proper synchronisation¶

package main

import (
    "context"
    "fmt"
    "sync"
    "time"
)

type Connection struct {
    closed chan struct{}
    once   sync.Once
}

func NewConnection() *Connection {
    return &Connection{closed: make(chan struct{})}
}

func (c *Connection) Close() {
    c.once.Do(func() { close(c.closed) })
}

func (c *Connection) IsClosed() bool {
    select {
    case <-c.closed:
        return true
    default:
        return false
    }
}

func use(ctx context.Context, conn *Connection) {
    cleanupDone := make(chan struct{})
    stop := context.AfterFunc(ctx, func() {
        defer close(cleanupDone)
        fmt.Println("cleanup: closing connection")
        conn.Close()
    })
    defer func() {
        if stop() {
            // the AfterFunc was deregistered before firing
            close(cleanupDone)
        }
        <-cleanupDone // wait for cleanup to finish, either way
    }()

    select {
    case <-time.After(100 * time.Millisecond):
        fmt.Println("work completed")
    case <-ctx.Done():
        fmt.Println("work cancelled")
    }
}

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
    defer cancel()
    conn := NewConnection()
    use(ctx, conn)
    fmt.Println("conn closed?", conn.IsClosed())
}

This example shows the careful dance to wait for the AfterFunc callback. The deferred stop() returns true if the callback never ran (so we close cleanupDone ourselves). Otherwise the callback closes it. Either way, <-cleanupDone synchronises before the function returns. This is the canonical pattern when you need both cleanup-on-cancel and cleanup-is-complete-before-return.

Example 8: A staged shutdown coordinator¶

package main

import (
    "context"
    "errors"
    "fmt"
    "log/slog"
    "time"
)

type Stage struct {
    Name string
    Run  func(ctx context.Context) error
}

type Coordinator struct {
    stages []Stage
}

func (c *Coordinator) Stage(name string, run func(ctx context.Context) error) {
    c.stages = append(c.stages, Stage{Name: name, Run: run})
}

func (c *Coordinator) Shutdown(ctx context.Context) error {
    var errs []error
    for _, s := range c.stages {
        start := time.Now()
        if err := s.Run(ctx); err != nil {
            slog.Error("stage failed", "stage", s.Name, "elapsed", time.Since(start), "err", err)
            errs = append(errs, fmt.Errorf("%s: %w", s.Name, err))
        } else {
            slog.Info("stage done", "stage", s.Name, "elapsed", time.Since(start))
        }
    }
    return errors.Join(errs...)
}

func main() {
    var c Coordinator
    c.Stage("stop-listener", func(ctx context.Context) error { time.Sleep(10 * time.Millisecond); return nil })
    c.Stage("drain-requests", func(ctx context.Context) error { time.Sleep(50 * time.Millisecond); return nil })
    c.Stage("close-db", func(ctx context.Context) error { time.Sleep(20 * time.Millisecond); return nil })
    c.Stage("flush-tracer", func(ctx context.Context) error { time.Sleep(15 * time.Millisecond); return nil })

    ctx, cancel := context.WithTimeout(context.Background(), 200*time.Millisecond)
    defer cancel()
    if err := c.Shutdown(ctx); err != nil {
        fmt.Println("errors:", err)
    } else {
        fmt.Println("clean")
    }
}

A simple staged coordinator. Each stage is named and timed. Logs go to slog. Errors are joined. The shutdown context is shared across all stages — if it expires mid-stage, the stage's Run should observe and abort.

Example 9: Cleanup with context.WithCancelCause for error reporting¶

package main

import (
    "context"
    "errors"
    "fmt"
)

var (
    errDiskFull = errors.New("disk full")
    errAborted  = errors.New("user aborted")
)

func work(ctx context.Context) error {
    select {
    case <-ctx.Done():
        cause := context.Cause(ctx)
        switch {
        case errors.Is(cause, errDiskFull):
            return fmt.Errorf("cleanup: disk-full path: %w", cause)
        case errors.Is(cause, errAborted):
            return fmt.Errorf("cleanup: abort path: %w", cause)
        default:
            return fmt.Errorf("cleanup: generic: %w", ctx.Err())
        }
    }
}

func main() {
    ctx, cancel := context.WithCancelCause(context.Background())
    go func() { cancel(errDiskFull) }()
    fmt.Println(work(ctx))
}

The cleanup branches on the cause of cancellation. Different cleanup paths for different errors. Without WithCancelCause, you would lose this information after cancel.

Example 10: A Service type with all the trimmings¶

package main

import (
    "context"
    "errors"
    "fmt"
    "log/slog"
    "sync"
    "time"
)

type Service struct {
    name    string
    mu      sync.Mutex
    started bool
    stopped bool
    stopOnce sync.Once
    stopErr  error
    cancel   context.CancelFunc
    done     chan struct{}
}

func New(name string) *Service { return &Service{name: name, done: make(chan struct{})} }

func (s *Service) Name() string { return s.name }

func (s *Service) Start(ctx context.Context) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    if s.started {
        return errors.New("already started")
    }
    s.started = true
    ctx, cancel := context.WithCancel(ctx)
    s.cancel = cancel
    go s.run(ctx)
    return nil
}

func (s *Service) run(ctx context.Context) {
    defer close(s.done)
    defer func() {
        if r := recover(); r != nil {
            slog.Error("service panic", "name", s.name, "value", r)
        }
    }()

    t := time.NewTicker(50 * time.Millisecond)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            slog.Info("service stopping", "name", s.name)
            return
        case <-t.C:
            // do periodic work
        }
    }
}

func (s *Service) Stop(ctx context.Context) error {
    s.stopOnce.Do(func() {
        s.mu.Lock()
        if !s.started {
            s.mu.Unlock()
            s.stopErr = errors.New("not started")
            return
        }
        cancel := s.cancel
        s.stopped = true
        s.mu.Unlock()

        cancel()
        select {
        case <-s.done:
        case <-ctx.Done():
            s.stopErr = fmt.Errorf("%s: timeout: %w", s.name, ctx.Err())
        }
    })
    return s.stopErr
}

func main() {
    s := New("test")
    if err := s.Start(context.Background()); err != nil {
        fmt.Println(err); return
    }
    time.Sleep(120 * time.Millisecond)
    ctx, cancel := context.WithTimeout(context.Background(), time.Second)
    defer cancel()
    if err := s.Stop(ctx); err != nil {
        fmt.Println(err); return
    }
    fmt.Println("clean")
    // double-stop is safe and returns the same result
    if err := s.Stop(ctx); err != nil {
        fmt.Println("second stop:", err)
    }
}

Every behaviour is deliberate: - Start is one-shot (returns error on second call). - Stop is idempotent via sync.Once; subsequent calls return the cached stopErr. - run has a top-level recover for panic safety. - The ticker is properly stopped via defer. - Stop honors the shutdown context's deadline.

This is the senior-level template. Copy it. Modify it for your specific component. The structure is the contract.

Architecture Patterns¶

Pattern 1: Hierarchical component tree¶

Services are trees of components. Each component has children. Shutdown is post-order traversal of the tree.

Service (root)
├── HTTP Server
│   └── Handler
│       ├── AuthMiddleware
│       └── RateLimitMiddleware
├── WorkerPool
│   └── Worker [×N]
└── Infrastructure
    ├── Database
    ├── Cache
    ├── Logger
    └── Metrics

Each node's Shutdown calls Shutdown on its children (post-order), then releases its own resources. The root's Shutdown triggers the whole tree.

Implementation: each component owns a slice of child components and a Shutdown(ctx) method that iterates them. No central registry needed.

Pattern 2: Flat registry¶

Alternative to the tree: every component registers itself with a central manager. The manager owns the shutdown order.

Manager.Register(component)
Manager.Shutdown(ctx) → run all components in LIFO of registration

Simpler than the tree. Harder to enforce dependency order: components must register in the right order (which is, conveniently, the order they are created if you wire dependencies before registering).

Pattern 3: Two-phase shutdown¶

Phase 1: signal-only. All components get a "prepare to shut down" call. They stop accepting new work. They drain in-flight.

Phase 2: close. All components get a "close" call. They release resources.

This pattern separates "stop work" from "release resources." Useful when work-stopping is fast but release is slow, or when you want all components to stop together before any begins to close.

for _, c := range components { c.Drain(ctx) }
for _, c := range components { c.Close(ctx) }

Pattern 4: Per-component supervisor¶

Each component is supervised by a goroutine that restarts it on crash. The supervisor itself has a shutdown method that stops restarting.

type Supervisor struct {
    create func() Runnable
    cancel context.CancelFunc
    done   chan struct{}
}

func (s *Supervisor) Run(ctx context.Context) {
    ctx, s.cancel = context.WithCancel(ctx)
    defer close(s.done)
    for {
        r := s.create()
        if err := r.Run(ctx); err != nil {
            if errors.Is(err, context.Canceled) {
                return
            }
            slog.Error("supervised crashed; restarting", "err", err)
        }
    }
}

func (s *Supervisor) Stop(ctx context.Context) error {
    s.cancel()
    select {
    case <-s.done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

The supervisor's Run loop creates a new Runnable on each iteration, runs it, and either returns (on context cancel) or restarts (on any other error). Stop cancels the context.

Pattern 5: Cleanup as event handlers¶

Treat cleanup as event-driven. Components register handlers for events like "context cancelled" or "service shutting down." A dispatcher fires them in order.

type Dispatcher struct {
    handlers []func()
}

func (d *Dispatcher) On(event Event, fn func()) {
    d.handlers = append(d.handlers, fn)
}

func (d *Dispatcher) Emit() {
    for i := len(d.handlers) - 1; i >= 0; i-- {
        d.handlers[i]()
    }
}

Similar to context.AfterFunc but allows multiple events and explicit ordering.

Pattern 6: Cleanup co-located with acquisition¶

Each acquisition site immediately registers its cleanup. The cleanup may be a defer, an AfterFunc, or a registration with a manager.

func openDB(ctx context.Context, mgr *Manager) *DB {
    db := connect()
    mgr.Register("db", func(ctx context.Context) error { return db.Close() })
    return db
}

The caller does not need to remember to close. The manager owns the order.

This is sometimes called "resource scoping." It is the senior-level evolution of defer Close() — same locality, but cross-scope.

Pattern 7: Tree of contexts mirrors tree of components¶

Each component has its own context, derived from its parent. Cancellation flows top-down via the context tree. Cleanup flows bottom-up via the component tree.

ctx (root)
├── ctx-http
│   └── ctx-handler
└── ctx-workers
    ├── ctx-worker-1
    ├── ctx-worker-2

When ctx is cancelled, all descendants are too. Each component watches its own context and reacts. Cleanup runs in each component's own Stop method.

Clean Code¶

1. One responsibility per Stop method¶

A Stop that does ten things is hard to test and hard to debug. Each component's Stop does its own cleanup. Composition does the rest.

2. Explicit ordering in code, not in implicit defer¶

When ordering matters for correctness, write it explicitly. Do not rely on the reader to count defers and unwind them mentally. A four-line block with named steps reads better than four defers.

// good: explicit
if err := server.Stop(ctx); err != nil { errs = append(errs, err) }
if err := workers.Stop(ctx); err != nil { errs = append(errs, err) }
if err := db.Close(); err != nil { errs = append(errs, err) }

vs

// fine for one function, but not for a 30-step shutdown
defer db.Close()
defer workers.Stop(ctx)
defer server.Stop(ctx)

The former is more maintainable at scale.

3. Document the shutdown order¶

Every service's main.go or its lifecycle manager should have a comment that lists the shutdown order and the reasoning.

4. Single source of truth for graceful period¶

Define a single constant or config value for "how long do we allow for graceful shutdown?" Reference it from every component. Avoid a maze of magic numbers.

5. Tests are part of the contract¶

Every component with a Stop method should have at least one test that asserts: - After Stop, no goroutines leak. - After Stop, repeat Stop is safe and returns the same error. - Stop honors its context deadline.

These tests are non-negotiable for production.

Production Considerations¶

Kubernetes and terminationGracePeriodSeconds¶

Kubernetes sends SIGTERM to your pod, then waits for terminationGracePeriodSeconds (default 30s) before SIGKILL. Your shutdown SLO must fit inside that window — minus some margin for the kubelet's own work.

A typical setup: - terminationGracePeriodSeconds: 60 - Graceful shutdown budget: 30s - Margin: 30s

The 30s budget is wired into the shutdown context's timeout. If your shutdown takes longer, you get SIGKILL'd and lose state.

Health-check coordination¶

When SIGTERM arrives: 1. Mark the service as "not ready" (so Kubernetes stops routing traffic). 2. Wait a short period (5s) for the load balancer to notice. 3. Begin actual shutdown.

Step 1 is a common mistake: people skip it and lose requests during the "in-flight" window.

Logs during shutdown¶

Shutdown is the period when ops engineers most need clear logs. Every component should log: - Start of its Stop - End of its Stop - Any errors

Use slog with structured fields: component name, elapsed time, error.

Metrics during shutdown¶

Emit metrics: - shutdown_duration_seconds{component="db"} - shutdown_errors_total{component="db"} - shutdown_inflight_at_signal{}

These metrics let you tune the graceful period over time.

Crash dump on cleanup failure¶

If shutdown fails (timeout or error), consider writing a crash dump (runtime.Stack) before exiting. Lets you diagnose stuck shutdowns post-mortem.

Replayable shutdown tests¶

In CI, run a "soak test" that starts the service, sends some load, sends SIGTERM, and asserts: - Service exited within the graceful period. - No request loss. - All resources released (check FDs with /proc/PID/fd or equivalent).

These tests catch regressions that unit tests miss.

Error Handling¶

Error categories in cleanup¶

• Recoverable. A retry, even within the same shutdown, might succeed. Use retry.Do with backoff.
• Fatal-for-this-component. This component cannot shut down cleanly. Log and move on.
• Fatal-for-shutdown. A component crucial for the rest of shutdown failed. Skip the rest, log loudly, exit non-zero.

In practice, most cleanup errors are category 2: log and move on. Avoid category 1 (retries during shutdown burn the time budget). Category 3 is rare but important.

Error wrapping¶

Always wrap shutdown errors with the component name:

return fmt.Errorf("shutdown %s: %w", c.Name(), err)

The caller can errors.Is the underlying cause and still see which component failed.

errors.Join for multi-component shutdown¶

When shutdown of multiple components produces multiple errors, join them:

var errs []error
for _, c := range components {
    if err := c.Stop(ctx); err != nil {
        errs = append(errs, err)
    }
}
return errors.Join(errs...)

The caller can iterate the joined error with a custom function, or call errors.Is to check for specific causes.

Panic during shutdown¶

A panic in a Stop method must not crash the entire shutdown. Wrap each Stop call:

func safeStop(c Component, ctx context.Context) (err error) {
    defer func() {
        if r := recover(); r != nil {
            err = fmt.Errorf("stop %s: panic: %v", c.Name(), r)
        }
    }()
    return c.Stop(ctx)
}

Then iterate components calling safeStop. The panic is converted to an error; shutdown continues.

Security Considerations¶

Audit logs at shutdown¶

Log every shutdown event with timestamps. Ops teams use these to investigate "service went down at 14:32 UTC" reports.

Secret zeroing¶

If a service holds secrets in memory, zero them during shutdown:

func (s *Service) Stop(ctx context.Context) error {
    defer s.zeroSecrets()
    // ... usual shutdown ...
}

func (s *Service) zeroSecrets() {
    for i := range s.apiKeys {
        for j := range s.apiKeys[i] {
            s.apiKeys[i][j] = 0
        }
    }
}

Why? In case the process is core-dumped or its memory is inspected after exit. Defense in depth.

Sensitive cleanup blocks¶

Some cleanup must run regardless of panics. Use defer inside a defer recover to ensure it runs:

defer func() {
    defer s.eraseSecrets() // outer defer; runs even if inner panics
    if r := recover(); r != nil {
        slog.Error("recover", "value", r)
    }
}()

The eraseSecrets is the outermost guard; it runs after the recover handles any panic.

Shutdown as an attack surface¶

A poorly-designed shutdown can be a denial-of-service vector. If an attacker can trigger SIGTERM (e.g., via a control endpoint), they can take the service down. Restrict shutdown signals to trusted sources. Use signed control plane requests if cluster orchestration is not enough.

Half-open connections¶

During the drain phase, connections that are in-flight may be left in an inconsistent state if the peer is malicious (e.g., never sends end-of-request). Use deadlines aggressively: every in-flight read/write should have one.

Performance Engineering¶

Defer cost at scale¶

Open-coded defers are nearly free (Go ≥ 1.14). Heap-allocated defers cost a small allocation (~64 bytes) and a linked-list traversal. For most services, the cost is negligible.

If profiling shows defer cost as significant: - Make sure the compiler is open-coding (no defers in loops, ≤ 8 defers per function). - Move defers out of inner loops. - Use manual cleanup if absolutely necessary.

AfterFunc cost¶

Each AfterFunc registration is a small struct allocation and an atomic operation. The callback firing is one goroutine creation. Cost: a few μs.

For services that create thousands of contexts per second, profile. If AfterFunc dominates, batch cleanups: one AfterFunc per context that fans out to multiple cleanup actions.

Goroutine cost during shutdown¶

Cancel-storm scenarios (10,000 contexts cancelled at once) can spawn 10,000 AfterFunc goroutines. If each does I/O, the system can fall over.

Options: - Use a worker pool for cleanup work (the AfterFunc just enqueues into a channel). - Rate-limit cleanup operations. - Coalesce cleanups: one AfterFunc on the parent context, not per-child.

Memory churn during shutdown¶

A graceful shutdown that allocates aggressively (e.g., for error wrapping) can hit the GC at the wrong moment. Pre-allocate shutdown buffers when possible. Avoid expensive operations during shutdown.

Lock contention during shutdown¶

Components that hold a global lock and try to release it during shutdown can serialise everything. Profile lock contention; reduce critical sections; consider per-shard locks for components that need them.

Profiling shutdown¶

Use runtime/pprof to profile the shutdown sequence in test environments. The shutdown profile is different from steady-state — different code paths, different allocations. Treat it as its own performance target.

Best Practices¶

• One Stop method per component, with a well-defined contract.
• Idempotent Stop via sync.Once.
• Pass a shutdown context with a deadline, not the cancelled trigger context.
• Document the shutdown order explicitly.
• Test shutdown paths in CI (goroutine leak assertions, soak tests).
• Use errors.Join to surface all cleanup failures.
• Wrap each Stop call with panic recovery during multi-component shutdown.
• Log structured shutdown events (slog with component name and elapsed time).
• Emit shutdown metrics for ops tuning.
• Mark the service as not-ready before beginning shutdown.
• Coordinate with Kubernetes' terminationGracePeriodSeconds.
• Use signal.NotifyContext for SIGTERM handling.
• Use context.AfterFunc only when defer cannot do the job.
• Compose components hierarchically; let each manage its children.

Edge Cases & Pitfalls¶

Pitfall 1: Components that ignore the shutdown context¶

If your component's Stop method does work that does not respect the passed context, the deadline is meaningless. Always plumb the context through every blocking operation.

Pitfall 2: Cyclic shutdown dependencies¶

A depends on B; B depends on A. Cannot shut down either first. Either break the cycle (refactor) or use a two-phase shutdown.

Pitfall 3: Goroutines spawned during shutdown¶

A component that spawns a goroutine during its Stop method needs to wait for that goroutine before returning. Otherwise the goroutine outlives the shutdown.

Pitfall 4: Stop called before Start¶

Common in tests. Decide your contract: error, no-op, or panic. Document it.

Pitfall 5: Two callers to Stop simultaneously¶

If Stop is not designed for concurrent invocation, two callers can race. Use sync.Once to make it safe.

Pitfall 6: Late-arriving SIGTERM during shutdown¶

If the user presses Ctrl-C twice, you get two SIGINTs. The first triggers graceful shutdown; the second should force exit. signal.NotifyContext does not handle this directly; build your own:

sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGINT, syscall.SIGTERM)

<-sigs
slog.Info("graceful shutdown")
go func() {
    <-sigs
    slog.Warn("forced exit")
    os.Exit(1)
}()

// proceed with graceful shutdown

Pitfall 7: AfterFunc registered on a long-lived context¶

Each registration accumulates. Long-lived parent contexts can hold thousands of callbacks. Always call stop() when no longer needed.

Pitfall 8: Recursive Stop calls¶

A component's Stop that internally calls parent.Stop (which then calls child.Stop again) creates a recursion. Use the sync.Once to break the cycle.

Pitfall 9: Shutdown that depends on external services¶

If your shutdown calls an external API (e.g., deregister from a service registry), it might hang. Wrap with a deadline. Failure to deregister is usually acceptable; failure to shut down is not.

Pitfall 10: Test setup leaks¶

Tests that start components but do not stop them leak goroutines. Use t.Cleanup(func() { svc.Stop(ctx) }) to ensure cleanup. The goroutine leak detector catches the rest.

Common Mistakes¶

Mistake 1: Re-using the signal context for shutdown¶

// BUG
ctx, _ := signal.NotifyContext(context.Background(), syscall.SIGTERM)
<-ctx.Done()
service.Stop(ctx) // ctx is already cancelled!

The context is cancelled by SIGTERM. Passing it to Stop means "you have zero time." Use a fresh context with a deadline.

Mistake 2: Forgetting to call Stop on sub-components¶

A composite component owns sub-components. Its Stop must call each sub's Stop. Easy to forget when adding a new sub.

Mistake 3: Holding a lock during slow shutdown¶

// BUG: serialises all shutdown
func (s *S) Stop(ctx context.Context) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    return s.slowShutdown(ctx)
}

The lock is held for the entire shutdown. If two goroutines try Stop, they serialise. Use sync.Once instead.

Mistake 4: Not handling Stop errors¶

// BUG
defer s.Stop(context.Background())

The error is dropped. Shutdown failures are silent. Log them at minimum.

Mistake 5: Stop that returns immediately without waiting¶

// BUG
func (s *S) Stop(ctx context.Context) error {
    s.cancel()
    return nil // does not wait for the worker goroutine to exit
}

The contract says "Stop returns when shutdown is complete." Wait for the goroutine. Otherwise the caller's defer wg.Wait() (or equivalent) is bypassed.

Mistake 6: Shutting down infrastructure before workers¶

// BUG
db.Close()
workers.Stop(ctx) // workers try to query DB, panic

Order matters. Workers depend on DB. Stop workers first.

Mistake 7: AfterFunc that races with Stop¶

// BUG
stop := context.AfterFunc(ctx, func() {
    s.Close() // races with the explicit Stop path
})
defer stop()
defer s.Stop(ctx)

If both Close and Stop run, they race. Make Close idempotent, or pick one.

Common Misconceptions¶

"Shutdown should be fast."

Sometimes. For services with no state, yes. For services with cached data, in-flight requests, or pending writes, the shutdown must be correct first, fast second. A 30-second graceful shutdown that saves 10MB of data is better than a 1-second crash.

"If I use defer, I do not need a Stop method."

For long-lived services, defer alone is insufficient. A defer fires when its function exits — which is the moment the program is about to die. By then it is too late to do proper drain. You need a Stop method that runs before the moment of exit.

"errgroup handles cleanup."

errgroup handles goroutine lifecycle. Cleanup of resources owned by those goroutines is still your job. errgroup helps coordinate but does not free files, close connections, or flush buffers.

"I do not need to test shutdown."

You do. Shutdown is where most operational pain lives. Untested shutdown is shipping a known-broken feature.

"AfterFunc is a panacea for context-based cleanup."

AfterFunc is one tool. It has specific semantics (new goroutine, no waiting, one-shot). Use it where it fits; defer or explicit Stop is often clearer.

Tricky Points¶

• context.AfterFunc vs context.WithCancelCause. AfterFunc reads ctx.Done(); cause is separate metadata. Cleanup can use context.Cause(ctx) to read the cause inside the callback.
• Defer in a return-and-then statement. return fn() evaluates fn() before defers run. If fn panics, the defer still runs.
• runtime.Goexit vs panic. Goexit ends the goroutine cleanly, running defers. Panic is more dramatic but also runs defers (until recovered or until program terminates).
• os.Exit from a panic handler. If your top-level recover decides to exit, defers below it do not run. Be deliberate.
• sync.Pool and finalisers. A pooled object can have a finaliser, but it runs at GC, not at Put. Do not use finalisers for resource release on Pool objects.
• Cleanup order vs initialization order. Cleanup is reverse of initialization. Encode both in the same place to make symmetry obvious.

Test¶

Predict the output. Then run it.

package main

import (
    "context"
    "fmt"
    "sync"
    "time"
)

func main() {
    var wg sync.WaitGroup
    defer wg.Wait()

    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    wg.Add(1)
    go func() {
        defer wg.Done()
        defer fmt.Println("g1 cleanup")
        <-ctx.Done()
    }()

    wg.Add(1)
    go func() {
        defer wg.Done()
        defer fmt.Println("g2 cleanup")
        time.Sleep(20 * time.Millisecond)
    }()

    time.Sleep(10 * time.Millisecond)
    fmt.Println("main: ready to exit")
}