Channel Close Violations — Professional Level¶

Table of Contents¶

1. Introduction
2. Operating Production Systems with Channel Closes
3. Shutdown Protocols Across Services
4. Graceful Drain of Pipelines
5. Context Cancellation Versus Close
6. Observability of Channel State
7. Framework-Level Patterns: errgroup
8. Framework-Level Patterns: Worker Pools
9. Runtime Anatomy of Close
10. The hchan Structure
11. Panic Stack Trace Forensics
12. Race Detector Output on Close Violations
13. Production Incident Playbook
14. Kubernetes-Scale Shutdown Patterns
15. Pod Termination and Channel Drain
16. Signal Handling and Close Triggers
17. Distributed Shutdown Coordination
18. Health Checks and Pre-Close Behaviour
19. Backpressure and Close
20. Performance Cost of Safe-Close Patterns
21. Lock-Free Idempotent Close
22. Memory Model Implications
23. Telemetry: Metrics, Logs, Traces
24. Chaos Engineering for Close
25. Post-Incident Review Patterns
26. Self-Assessment
27. Summary

Introduction¶

The senior level covered the design of close protocols. The professional level covers the operation of systems built on those protocols.

In production:

• Close failures cause incidents, not just unit-test failures.
• A graceful shutdown that takes 30 seconds when the SLO is 5 seconds is a real bug.
• Close panics in the field are a paging event and a public post-mortem.
• Observability of close events is required for capacity planning and incident response.
• Patterns must scale to thousands of nodes, millions of QPS, and dozens of microservices interleaved.

This file is the operator's handbook. Topics include the runtime mechanics that produce close-related panics, the patterns that make production-grade shutdown possible, and the tools (metrics, traces, logs, profiling) that surface close issues before they wake you at 3 AM.

It assumes you have internalised the senior-level material. We will spend less time on the why and more time on the operational details.

Operating Production Systems with Channel Closes¶

A production system distinguishes itself from a development prototype by how it handles edge cases. For channels, the edge cases are:

1. Cold start. Channels are created, but no senders are running yet. Closing is a no-op for receivers.
2. Steady state. Normal operation. Close should not happen.
3. Graceful shutdown. Initiated by SIGTERM, a deploy, a scale-down. Close cascades through the system.
4. Hard shutdown. Initiated by SIGKILL or a panic. Goroutines die mid-stride; channels are abandoned.
5. Restart after crash. A new process comes up; old channels are gone with the old process.

Production discipline: every channel must have a documented behaviour in each of these states.

Cold start¶

Channels created during initialisation should not be exposed to senders until they are ready to receive. A common mistake: spawn the receiver goroutine after exposing the channel.

// BAD
type Svc struct { events chan Event }
func New() *Svc { return &Svc{events: make(chan Event)} }
func (s *Svc) Start() { go s.consume() }

// caller does:
svc := New()
svc.SubmitEvent(e) // blocks; receiver not running
svc.Start()

The fix is to start the receiver in the constructor, or to defer event submission until after Start. The convention "Start before Submit" must be enforced — usually by making events private and exposing a SubmitEvent method that checks state.

Graceful shutdown¶

The canonical sequence:

1. Signal: SIGTERM or admin endpoint.
2. Stop accepting new work (close listener, refuse Submit).
3. Cancel context to signal in-flight work.
4. Wait for in-flight work to finish or timeout.
5. Close internal channels (after senders have stopped).
6. Close external resources (DB, log file, etc.).
7. Exit process.

Each step has potential close issues. A misordered step (close before drain, drain before stop-accepting) causes panic or data loss.

Hard shutdown¶

A panic in any goroutine terminates the process. If a defer-based close was scheduled, it may or may not run (the deferred function in the panicking goroutine runs; other goroutines' defers do not). Channels in an inconsistent state may panic on subsequent operations.

Production-grade systems often have a top-level recover-and-log that captures panics, logs context, then re-panics (to terminate the process cleanly). The point: do not try to recover into normal operation after an unintended panic in a critical goroutine.

Shutdown Protocols Across Services¶

Most production systems are not single processes. A microservice ecosystem has interdependencies: service A depends on service B, which depends on service C. Shutdown must respect these dependencies.

Independent shutdowns¶

If services are truly independent — no synchronous calls between them — each shuts down on its own schedule. Channels within each service close per the senior-level patterns. No cross-service coordination.

In practice, even "independent" services share message queues or databases. Shutdown of those shared resources is a different problem (database connection drains, queue acknowledgements, etc.).

Dependent shutdowns¶

If service A calls service B synchronously, and B shuts down first, A's in-flight requests fail. Two strategies:

1. Drain protocol. B announces intent to shut down. A stops sending new requests. A drains its in-flight requests. B confirms drain. B finishes shutdown.
2. Retry-and-tolerate. B shuts down abruptly. A's in-flight requests fail; retry against a different B replica. A's open connections to B drop and reconnect.

Drain is gentler but slower. Retry is faster but causes blip in latency. Most production systems use retry; drain is reserved for resource-intensive operations (large file transfers).

How channels fit in¶

Within each service, channels coordinate goroutines. The "shutdown" of a service is the cascade of channel closes:

1. Listener closes accept loop.
2. Accept loop's exit closes its connection registry.
3. Connection handlers' exits close their per-connection state.
4. State holders close their persistence channels.

Each level is one application of the senior-level patterns. The professional skill is sequencing them.

Graceful Drain of Pipelines¶

A pipeline (source → stages → sink) under shutdown must drain: finish processing in-flight items before exiting. Done wrong, items are lost.

Drain algorithm¶

1. Signal source: stop emitting new items.
2. Source exits when current item is sent (or aborted on timeout).
3. Source closes its output channel.
4. Stage 1 ranges over source; range exits when source closes.
5. Stage 1 finishes its current item, sends downstream, closes its output.
6. ... cascade through all stages ...
7. Sink ranges over last stage's output, finishes when it closes.
8. All goroutines have exited. Service exits.

This is quiescent shutdown. No items lost (assuming all goroutines cooperate).

Implementing drain with errgroup¶

func RunPipeline(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)

    src := make(chan Item, 32)
    mid := make(chan Item, 32)

    g.Go(func() error {
        defer close(src)
        return source(ctx, src)
    })

    g.Go(func() error {
        defer close(mid)
        for it := range src {
            out, err := stage1(ctx, it)
            if err != nil { return err }
            select {
            case <-ctx.Done(): return ctx.Err()
            case mid <- out:
            }
        }
        return nil
    })

    g.Go(func() error {
        for it := range mid {
            if err := sink(ctx, it); err != nil { return err }
        }
        return nil
    })

    return g.Wait()
}

On context cancellation, source observes ctx.Done() and exits. defer close(src) fires. Stage1 ranges exit; its defer close(mid) fires. Sink ranges exit. errgroup's Wait returns. Total drain time = source's response time + stage1's processing time + sink's processing time.

The drain is bounded by the slowest in-flight item, not the total queue size — because items already in mid have been processed by source.

Wait — items in the buffer of src are not yet processed by stage1. Those drain through stage1 as stage1 reads them. Items in the buffer of mid are not yet processed by sink. So total drain time depends on buffer sizes: 32 items in src buffer + 32 in mid buffer + the in-flight item per stage.

If you need stricter bounds, reduce buffer sizes or use unbuffered channels — at the cost of throughput.

Drain with timeout¶

If drain exceeds the SLO, you must abandon and lose data. The pattern:

func GracefulShutdown(ctx context.Context, p *Pipeline) error {
    cancelCtx, cancel := context.WithCancel(context.Background())
    defer cancel()
    drainDone := make(chan error, 1)
    go func() { drainDone <- p.Run(cancelCtx) }()

    cancel() // signal pipeline to drain
    select {
    case err := <-drainDone:
        return err
    case <-ctx.Done():
        return ctx.Err() // shutdown timeout
    }
}

Caller provides ctx with a deadline. If drain exceeds the deadline, shutdown returns with the deadline error. The pipeline may still be running (its goroutines will eventually finish or leak). Hard shutdown follows (process exit).

What drain cannot do¶

Drain is graceful for in-process queues. It cannot drain:

• External queues (Kafka, RabbitMQ): drain there means stop fetching, which is a separate protocol.
• Network connections to clients: drain means stop accepting, finish responding to existing, then close listener.
• Database transactions: drain means commit in-flight transactions and refuse new ones.

These each have their own shutdown protocols. Channel-based drain is just one layer.

Context Cancellation Versus Close¶

The professional-level distinction:

• context.Context.Done() is a signal. It says "stop, please". The receiver decides how to comply.
• close(ch) is a fact. It says "no more data will arrive". The receiver must handle this state.

A pipeline that uses only context cancellation can leak items: cancellation tells everyone to stop, but in-flight items are not delivered. A pipeline that uses only channel close can hang on errors: there is no way to abort.

Production pipelines use both. Context is the abort signal; close is the natural-end signal. Both must be handled in every loop.

The five places to check ctx.Done¶

In a pipeline stage:

for {
    select {
    case <-ctx.Done(): return ctx.Err() // 1. waiting for input
    case it, ok := <-in:
        if !ok { return nil }              // 2. input channel closed
        out, err := process(ctx, it)       // 3. inside processing (passed ctx)
        if err != nil { return err }
        select {
        case <-ctx.Done(): return ctx.Err() // 4. waiting to send
        case sink <- out:
        }
    }
}
// 5. before return, defer close(sink) // if this stage owns sink

Five places. Miss any one, and cancellation does not propagate through that arm.

In practice, the top-level select catches most cases. But the send arm (place 4) is the most frequently missed: developers write sink <- out directly, assuming the receiver is fast enough that it never blocks. Under load, it can block; cancellation never reaches the goroutine.

Anti-pattern: bare send after Receive¶

for it := range in {
    out := process(it)
    sink <- out  // can block forever if sink consumer gone
}

Cancellation cannot reach this goroutine while it is stuck on sink <-. Always wrap sends in select { case sink <-out: case <-ctx.Done(): }.

Anti-pattern: ctx-only without close¶

for {
    select {
    case <-ctx.Done(): return
    default:
    }
    out := produce()
    sink <- out
}

The receiver does not know when production has ended. It must rely on ctx.Done() somehow. But if the receiver does:

for {
    select {
    case v := <-sink: process(v)
    case <-ctx.Done(): return
    }
}

Then on cancellation, the receiver may exit before draining the buffered items in sink. Lost data.

Fix: close sink when the producer exits. Receiver ranges; range exits when sink closes; drain is complete.

The pattern is always: close-on-natural-end + ctx-for-abort, observed by both ends.

Observability of Channel State¶

In production, you want to know:

• How many channels are currently allocated.
• How occupied each channel's buffer is (over time).
• How long sends and receives block (latency distributions).
• When channels close (events).
• Which channels block longest (for capacity planning).

Go does not expose these directly. You instrument them.

Instrumented channel¶

type InstrCh[T any] struct {
    name string
    ch   chan T
    cap  int
}

func NewInstrCh[T any](name string, buf int) *InstrCh[T] {
    return &InstrCh[T]{name: name, ch: make(chan T, buf), cap: buf}
}

func (c *InstrCh[T]) Send(v T) {
    start := time.Now()
    c.ch <- v
    sendLatency.WithLabelValues(c.name).Observe(time.Since(start).Seconds())
    occupancy.WithLabelValues(c.name).Set(float64(len(c.ch)))
}

func (c *InstrCh[T]) Recv() (T, bool) {
    start := time.Now()
    v, ok := <-c.ch
    recvLatency.WithLabelValues(c.name).Observe(time.Since(start).Seconds())
    occupancy.WithLabelValues(c.name).Set(float64(len(c.ch)))
    return v, ok
}

func (c *InstrCh[T]) Close() {
    closeCount.WithLabelValues(c.name).Inc()
    close(c.ch)
}

Each operation updates metrics. The overhead is small (a few atomic operations on the Prometheus counters); for high-QPS channels, sample at 1/N.

Occupancy sampling¶

Polling len(ch) periodically is cheaper than per-operation:

go func() {
    ticker := time.NewTicker(time.Second)
    defer ticker.Stop()
    for range ticker.C {
        for _, c := range channels {
            occupancy.WithLabelValues(c.name).Set(float64(len(c.ch)))
        }
    }
}()

A 1Hz sample is enough for capacity planning. Higher rates only matter if you are debugging a specific issue.

Tracking close events¶

A trace span per close:

func (c *InstrCh[T]) Close() {
    span := trace.SpanFromContext(c.parentCtx) // or start a new span
    span.AddEvent("channel.close",
        trace.WithAttributes(
            attribute.String("name", c.name),
            attribute.Int("len_at_close", len(c.ch)),
        ),
    )
    close(c.ch)
}

In a trace visualisation (Jaeger, Tempo), you can see exactly when each channel closed during a shutdown. Invaluable for debugging slow drains.

Framework-Level Patterns: errgroup¶

golang.org/x/sync/errgroup is the canonical framework for structured concurrency in Go. Its close protocol is the gold standard.

How errgroup handles close¶

g, ctx := errgroup.WithContext(parentCtx)
g.Go(f1)
g.Go(f2)
err := g.Wait()

Internally:

1. errgroup holds a context derived from parentCtx via WithCancel.
2. Each g.Go(fn) wraps fn in a goroutine that calls fn, captures the return error, and if non-nil and first-error, cancels the context via the captured cancel func.
3. g.Wait() waits for all goroutines, then returns the first error (or nil).

The "first error cancels context" is the key. It means every other goroutine sees ctx.Done() and can abort.

When errgroup is the right tool¶

• Fixed set of parallel tasks; first error aborts all.
• Each task takes context, respects cancellation.
• You want all tasks to finish (or abort) before continuing.

When errgroup is not the right tool¶

• Indefinite/long-running goroutines (services, listeners). errgroup's Wait blocks indefinitely.
• Goroutines that don't take context.
• Patterns that want all tasks to continue even if some fail.

For the second and third, write your own goroutine management or use a different library.

Channels within errgroup¶

A common pattern: each task sends to a shared channel; the main goroutine collects from the channel.

g, ctx := errgroup.WithContext(ctx)
out := make(chan Result, 16)

for i := 0; i < 10; i++ {
    i := i
    g.Go(func() error {
        r, err := compute(ctx, i)
        if err != nil { return err }
        select {
        case <-ctx.Done(): return ctx.Err()
        case out <- r:
        }
        return nil
    })
}

go func() { _ = g.Wait(); close(out) }()

for r := range out { handle(r) }

The producer goroutines are managed by errgroup. After all return, a separate goroutine closes out. The consumer ranges and exits.

This is the standard "errgroup + channel" pattern. Note the separate goroutine for the close — you cannot do g.Wait() followed by close(out) in the main thread because Wait() blocks until all goroutines including the consumer-side are done, and the consumer's for range cannot exit until close.

The separate close-after-wait goroutine breaks the chicken-and-egg.

Pitfall: errgroup with non-error returning goroutines¶

g.Go(func() error {
    for {
        select {
        case <-ctx.Done(): return nil
        case msg := <-input:
            // process
        }
    }
})

This goroutine never returns. g.Wait() blocks forever. Even if you cancel ctx, the goroutine returns nil (no error), so the errgroup sees "first error was nil" and waits for all other goroutines.

Fix: ensure all goroutines have a termination condition that matches your shutdown protocol. For service goroutines, errgroup is not the right fit. Use sync.WaitGroup directly.

Framework-Level Patterns: Worker Pools¶

A production worker pool has many failure modes related to close.

Production worker pool: full implementation¶

package pool

import (
    "context"
    "errors"
    "sync"
    "sync/atomic"
    "time"
)

var ErrPoolClosed = errors.New("pool: closed")

type Job interface {
    Run(ctx context.Context) error
}

type Pool struct {
    workers  int
    queueLen int

    jobs chan Job
    done chan struct{}
    wg   sync.WaitGroup
    once sync.Once

    submitted atomic.Int64
    completed atomic.Int64
    failed    atomic.Int64
}

func New(workers, queueLen int) *Pool {
    p := &Pool{
        workers:  workers,
        queueLen: queueLen,
        jobs:     make(chan Job, queueLen),
        done:     make(chan struct{}),
    }
    p.start()
    return p
}

func (p *Pool) start() {
    p.wg.Add(p.workers)
    for i := 0; i < p.workers; i++ {
        go p.worker(i)
    }
}

func (p *Pool) worker(id int) {
    defer p.wg.Done()
    for {
        select {
        case <-p.done:
            // Drain remaining jobs without blocking.
            for {
                select {
                case j := <-p.jobs:
                    p.runJob(id, j)
                default:
                    return
                }
            }
        case j := <-p.jobs:
            p.runJob(id, j)
        }
    }
}

func (p *Pool) runJob(workerID int, j Job) {
    defer func() {
        if r := recover(); r != nil {
            p.failed.Add(1)
            // log r, possibly increment metrics
        }
    }()
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    if err := j.Run(ctx); err != nil {
        p.failed.Add(1)
    } else {
        p.completed.Add(1)
    }
}

func (p *Pool) Submit(j Job) error {
    select {
    case <-p.done:
        return ErrPoolClosed
    default:
    }
    select {
    case <-p.done:
        return ErrPoolClosed
    case p.jobs <- j:
        p.submitted.Add(1)
        return nil
    }
}

func (p *Pool) SubmitOrTimeout(j Job, timeout time.Duration) error {
    select {
    case <-p.done:
        return ErrPoolClosed
    default:
    }
    timer := time.NewTimer(timeout)
    defer timer.Stop()
    select {
    case <-p.done:
        return ErrPoolClosed
    case p.jobs <- j:
        p.submitted.Add(1)
        return nil
    case <-timer.C:
        return errors.New("pool: timeout")
    }
}

func (p *Pool) Shutdown(ctx context.Context) error {
    p.once.Do(func() {
        close(p.done)
    })
    done := make(chan struct{})
    go func() {
        p.wg.Wait()
        close(done)
    }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

func (p *Pool) Stats() (submitted, completed, failed int64) {
    return p.submitted.Load(), p.completed.Load(), p.failed.Load()
}

Properties:

• Workers are supervised via WaitGroup.
• Close uses sync.Once for idempotency.
• Close signals via p.done; workers drain p.jobs and exit.
• p.jobs is never closed. This eliminates send-on-closed races.
• Submit checks p.done twice: pre-check for fast path, select-with-cancel for the actual send.
• Workers recover panics in user code and increment a failure counter.
• Shutdown has a deadline; if it exceeds, return the deadline error.

This is the production template. Test it; ship it. We will scrutinise the close logic in more detail.

Why two <-p.done checks in Submit¶

select { case <-p.done: return; default: }
// what if Shutdown fires here?
select { case <-p.done: return; case p.jobs <- j: }

Without the first check: if jobs has buffer space, the second select might choose p.jobs <- j even after Shutdown closed done, because select picks pseudo-randomly when multiple cases are ready.

With the first check: we eliminate the race for the common case (Shutdown was already fired before Submit started). The second select handles the case where Shutdown fires while Submit is in progress.

There is still a residual race: a Submit that passes the first check, enters the second select, picks the p.jobs <- j arm, and starts to send — concurrently, Shutdown fires. The send completes. The worker has already exited (drained p.jobs and returned).

Wait — the worker's drain loop is for { select { case j := <-p.jobs: ... default: return } }. If the Submit happens after the worker's default: return, the job is in the buffer but no one is reading. Submit succeeded, but the job is never run.

Is this acceptable? It depends on the contract. Most pools accept this as "best-effort during shutdown". If you need stronger guarantees, you must reject Submits earlier — perhaps by adding a separate "stopping" state that rejects all Submits before any worker exits.

State machine for graceful shutdown¶

const (
    stateRunning int32 = iota
    stateStopping
    stateStopped
)

type Pool struct {
    state atomic.Int32
    // ...
}

func (p *Pool) Submit(j Job) error {
    if p.state.Load() != stateRunning {
        return ErrPoolClosed
    }
    // ... existing logic ...
}

func (p *Pool) Shutdown(ctx context.Context) error {
    if !p.state.CompareAndSwap(stateRunning, stateStopping) {
        return ErrPoolClosed
    }
    close(p.done) // signal workers to drain
    // ... wait for workers ...
    p.state.Store(stateStopped)
    return nil
}

Submit checks the state atomically; the state transitions happen-before the close. This eliminates the residual race because Submit returns ErrPoolClosed once stateStopping is set.

But still a race: a Submit that loaded stateRunning before the Shutdown CAS proceeds. The check-then-act sequence is not atomic. We've narrowed the window but not eliminated it.

To fully eliminate, you need to either:

1. Hold a mutex during Submit and Shutdown (heavy).
2. Use the done-channel pattern as the source of truth, not the atomic state (the existing code).
3. Accept the residual window and document it.

In practice, option 3 is fine. The window is microseconds wide; the failure mode (a job submitted but not run) is benign for most pools.

Runtime Anatomy of Close¶

To debug close-related issues at the professional level, you need to understand what the Go runtime does inside close(ch).

The function is runtime.closechan(c *hchan) in src/runtime/chan.go. Approximately:

func closechan(c *hchan) {
    if c == nil {
        panic(plainError("close of nil channel"))
    }
    lock(&c.lock)
    if c.closed != 0 {
        unlock(&c.lock)
        panic(plainError("close of closed channel"))
    }
    // ... possibly record the close for race detection ...
    c.closed = 1

    // Release all readers.
    var glist gList
    for {
        sg := c.recvq.dequeue()
        if sg == nil { break }
        if sg.elem != nil {
            typedmemclr(c.elemtype, sg.elem) // zero-out
            sg.elem = nil
        }
        // mark this sudog as "channel closed"
        sg.closed = true
        gp := sg.g
        gp.param = unsafe.Pointer(sg)
        glist.push(gp)
    }
    // Release all writers — they will panic.
    for {
        sg := c.sendq.dequeue()
        if sg == nil { break }
        gp := sg.g
        gp.param = nil
        glist.push(gp) // these will panic on resume
    }
    unlock(&c.lock)
    // Wake up everyone.
    for !glist.empty() {
        gp := glist.pop()
        gp.schedlink = 0
        goready(gp, 3)
    }
}

(Pseudocode for clarity; the real function has more bookkeeping.)

Key insights:

1. The lock is taken first. All operations are serialised.
2. The closed-flag check raises the double-close panic.
3. Readers in the recv queue are dequeued, marked, and woken up. They will return zero-value + ok=false.
4. Writers in the send queue are dequeued and woken up. They will panic on resume.
5. The lock is released before the wakeups. This means another goroutine entering closechan immediately after will see closed=1 and panic.

The whole operation is O(N) where N is the number of waiting goroutines. For low-fanout, it's microseconds; for high-fanout (a broadcast close with thousands of waiters), it can be milliseconds.

What the panic looks like in stack traces¶

A "close of closed channel" panic:

panic: close of closed channel

goroutine 1 [running]:
main.cleanup(...)
        /path/to/file.go:42
main.main()
        /path/to/main.go:88 +0x...

The frame at the top of the goroutine 1 stack is the caller of close(ch). The runtime's closechan is not shown by default in the user-visible trace because it is in runtime/chan.go which is filtered.

To see the runtime frames, set GOTRACEBACK=all or GOTRACEBACK=system.

What the panic looks like for send-on-closed¶

panic: send on closed channel

goroutine 5 [running]:
main.producer(...)
        /path/to/producer.go:23
created by main.main
        /path/to/main.go:14 +0x...

The "created by" line tells you which goroutine started this one. Useful for tracing back to the origin: who started the producer that is now sending on a closed channel?

What "close of nil channel" looks like¶

panic: close of nil channel

Rare in practice; usually indicates a constructor that forgot to call make. The fix is always at the constructor, not at the close site.

The hchan Structure¶

The runtime's hchan struct (in src/runtime/chan.go):

type hchan struct {
    qcount   uint           // total data in the queue
    dataqsiz uint           // size of the circular queue (capacity)
    buf      unsafe.Pointer // points to an array of dataqsiz elements
    elemsize uint16
    closed   uint32         // close flag
    elemtype *_type
    sendx    uint           // send index (circular)
    recvx    uint           // receive index (circular)
    recvq    waitq          // list of recv waiters
    sendq    waitq          // list of send waiters
    lock     mutex
}

Close interacts mainly with three fields:

1. closed. A uint32 (could be a bool, but uint32 for alignment). 0 = open, 1 = closed. Read under c.lock.
2. recvq. Goroutines blocked on <-ch. On close, all are released with zero value.
3. sendq. Goroutines blocked on ch <-. On close, all are released and will panic on resume.

The lock serialises all of these. There is no lock-free close path; even the most optimised channel operations take the lock.

Implication: close is not free¶

A close that wakes up N waiters takes O(N) under the lock. For high-fanout cases (broadcasts to thousands of subscribers), this can be a hotspot. If you observe long close times in profiles, look for high-fanout patterns.

Implication: read-after-close is fast¶

A receive on a closed empty channel:

v, ok := <-ch // ch closed, empty

The runtime acquires the lock, sees closed=1, qcount=0, returns zero value with ok=false, releases the lock. Constant time, no waiters touched.

This makes close-as-broadcast efficient on the receiver side: thousands of receivers, each doing <-done, all wake up at close, and each does an O(1) operation.

Implication: buffered close preserves data¶

ch := make(chan int, 3)
ch <- 1; ch <- 2; ch <- 3
close(ch)
for v := range ch { fmt.Println(v) } // 1, 2, 3, exits

The hchan's buf still contains 1, 2, 3. close just sets closed=1. Subsequent receives drain the buf, decrementing qcount. When qcount=0 and closed=1, the range loop terminates.

Panic Stack Trace Forensics¶

Reading a panic stack trace from production to find the close-violation bug.

Example 1: close of closed channel¶

panic: close of closed channel

goroutine 152 [running]:
github.com/example/svc.(*Worker).Stop(0xc000123450)
        /src/worker.go:88 +0x55
github.com/example/svc.(*Manager).Shutdown.func1(0xc000123450)
        /src/manager.go:42 +0x33
sync.(*WaitGroup).Wait(0xc000089060)
        /usr/local/go/src/sync/waitgroup.go:117 +0x6e
github.com/example/svc.(*Manager).Shutdown(0xc000089000)
        /src/manager.go:55 +0x...

Read top-down:

• The panic happened in Worker.Stop at worker.go:88.
• That was called from Manager.Shutdown.func1 (an anonymous function inside Shutdown).
• The Shutdown caller is at manager.go:55.

Look at worker.go:88. Likely close(w.ch) without an idempotency guard. Look at manager.go:42 — likely a loop calling Stop on each worker.

Now: who called the first Stop? The double-close means two code paths reached the close. Search for callers of Worker.Stop. Likely the worker itself has a self-stop on its done channel, and Manager.Shutdown also calls Stop.

The fix: add sync.Once to Worker.Stop:

func (w *Worker) Stop() {
    w.once.Do(func() { close(w.ch) })
}

Example 2: send on closed channel¶

panic: send on closed channel

goroutine 893 [running]:
github.com/example/svc.(*Producer).produce(0xc000abcdef)
        /src/producer.go:42 +0x88
created by github.com/example/svc.NewProducer
        /src/producer.go:15 +0x55

Send is at producer.go:42. The producer goroutine was created by NewProducer at line 15.

Look at line 42. It's p.out <- v. Who closed p.out? Search for close(.*\.out). Likely the Manager closed p.out when it shut down, but did not wait for the producer to exit first.

The fix: do not close p.out externally; let the producer close on its own when it exits. The Manager should signal via a done channel, not by closing the data channel.

Example 3: goroutine dump shows hung close¶

Sometimes the problem is not a panic but a deadlock: the close has not happened, and someone is waiting for it.

goroutine 5 [chan receive, 60 minutes]:
main.consume(0xc000...)
        /src/consumer.go:23 +0x...

The consumer has been waiting on a channel receive for 60 minutes. Find the channel; find the producer; ask why the producer is not sending or closing.

Common cause: producer is stuck on a different channel that the consumer is supposed to read from but cannot, because it is waiting on the first channel. Circular dependency.

Send-side analogous: producer stuck on chan send, no consumer reading. Look at the consumer; why is it not consuming?

Stack trace tools¶

• go tool pprof goroutine.prof — analyse a goroutine profile.
• kill -SIGQUIT $PID (or kill -ABRT) — dumps all goroutines on stderr.
• runtime/pprof HTTP endpoint at /debug/pprof/goroutine?debug=2 — full stack of every goroutine.

Production binaries should expose /debug/pprof/goroutine (behind authorisation). When close-related deadlocks or panics are reported, fetch the goroutine dump and find the channel operation in question.

Race Detector Output on Close Violations¶

Run with -race. When a close-violation occurs concurrently with another op, the detector logs:

==================
WARNING: DATA RACE
Write at 0x00c000123450 by goroutine 7:
  runtime.closechan()
      /usr/local/go/src/runtime/chan.go:357 +0x39
  main.cleanup()
      /home/user/example.go:42 +0x33

Previous read at 0x00c000123450 by goroutine 8:
  runtime.chansend()
      /usr/local/go/src/runtime/chan.go:160 +0x...
  main.producer()
      /home/user/example.go:25 +0x...

Goroutine 7 (running) created at:
  main.main()
      /home/user/example.go:78 +0x...

Goroutine 8 (running) created at:
  main.main()
      /home/user/example.go:82 +0x...
==================

Interpretation:

• Goroutine 7 (closer) wrote to the channel state at example.go:42.
• Goroutine 8 (sender) read the channel state at example.go:25.
• They overlapped — a data race.

In a few microseconds, one of them will probably trigger the "send on closed channel" panic. The race detector caught the cause before the panic surfaces.

Use this to find close bugs that have not yet manifested. Run go test -race -count=10 -timeout=120s against your packages regularly.

Limitations¶

The race detector only finds races that actually happen during the test run. It cannot find races in code paths that the tests do not exercise. Code coverage of concurrent paths is the limiting factor.

For close violations specifically, the race detector finds:

• Send-on-closed if the send and close are concurrent.
• Write to a non-channel-internal field that should be protected.

It does not find:

• Double-close where both closes are sequential.
• Send-after-close where the close completes before the send (no overlap; still panics, but not a race).

For these, you need stress tests that drive the patterns under load.

Production Incident Playbook¶

When a close-related panic hits production:

1. Confirm the panic¶

Check logs for panic: close of closed channel, panic: send on closed channel, or panic: close of nil channel.

2. Identify the goroutine¶

Find the line number in the stack trace. The first frame after the runtime is the user code that did the wrong thing.

3. Find the second site¶

For close-of-closed, two sites called close(ch). Find both.

For send-on-closed, find every close(ch) and every ch <- v. The bug is at the intersection: close happened, then send, on a channel that should not have been closed yet.

4. Reconstruct the timeline¶

Did the close happen during a shutdown? At a specific point in handler logic? Check the request that triggered it. Logs around the panic may show the trigger.

5. Identify the design defect¶

Once you find the two sites, ask: why are both sites authorised to close (or send-after-close)? The design has a defect; identify it.

Common defects:

• Receiver closes the channel (Rule 4 violation).
• Multiple producers each defer close(ch).
• Close in error path duplicates close in success path.
• A Close method called both by explicit invocation and by some other lifecycle hook.

6. Hotfix¶

If the bug is "two senders both close", add a coordinator goroutine. If it's "receiver closes", remove that close and use a done-channel. If it's "double-close", add sync.Once.

Deploy the hotfix; observe panic rate.

7. Root cause analysis¶

After stabilisation, write a post-mortem. The relevant questions:

• What was the design assumption that proved wrong?
• Why did tests not catch it? (Insufficient concurrency? No race detector?)
• What linter could have caught it?
• What documentation should change?

8. Preventive measures¶

• Add a unit test that reproduces the bug.
• Add stress tests with -race for the affected component.
• Add a linter or vet check for similar patterns.
• Update the team's close-pattern style guide.

Kubernetes-Scale Shutdown Patterns¶

A service running on Kubernetes may be terminated at any time. The lifecycle:

1. SIGTERM sent to PID 1.
2. Pod grace period begins (default 30 seconds).
3. Pod IP removed from service endpoints (so new traffic does not arrive).
4. Liveness/readiness probes can still hit the pod.
5. Container exits. Either by graceful shutdown completing, or by SIGKILL after grace period.

Channels participate in step 2: the application processes the SIGTERM, signals shutdown, drains in-flight work, exits.

Reading SIGTERM into a channel¶

sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
go func() {
    sig := <-sigCh
    log.Printf("received %v; shutting down", sig)
    shutdown()
}()

The signal handler is a small goroutine; it converts the signal into a shutdown call. The shutdown function does the rest.

Signal handler must be idempotent¶

Two SIGTERMs (e.g., user impatient and runs kill twice) should not cause double shutdown:

var shutdownOnce sync.Once
shutdown := func() {
    shutdownOnce.Do(func() { realShutdown() })
}

sync.Once ensures realShutdown runs at most once, regardless of how many signals fire.

Grace period bounded close¶

If the application takes 60 seconds to drain but Kubernetes gives 30 seconds, the container will be SIGKILLed mid-drain. In-flight work is lost.

Options:

1. Increase Kubernetes grace period. terminationGracePeriodSeconds: 60.
2. Shrink drain time. Reduce buffer sizes; cancel slow operations early.
3. Externalise the queue. If work is in a durable queue (Kafka), drainage is the queue's problem, not the pod's. The pod just stops accepting new work and exits.

In production, option 3 is most common. The application never has more than a few seconds of in-flight work; the queue persists the rest.

Pre-stop hook for graceful drain¶

Kubernetes supports a pre-stop lifecycle hook: a command (or HTTP call) runs before SIGTERM is sent. Use it to flip the readiness probe to "not ready" before the application is told to shut down. This gives the load balancer time to drop the pod from its rotation before traffic ceases.

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "/app/notify-shutdown; sleep 10"]

The 10-second sleep gives the load balancer time to converge. During this window, the application keeps serving but reports "not ready".

Channels are not involved in this step, but the overall protocol is the same: signal, drain, exit.

Pod Termination and Channel Drain¶

For applications with significant in-flight work, the drain must be coordinated with Kubernetes.

State machine¶

running → not-ready → draining → stopping → terminated

• running: healthy, accepting work.
• not-ready: readiness probe fails; new requests do not arrive; in-flight requests proceed.
• draining: new submissions rejected; in-flight work continues.
• stopping: all goroutines exit; channels close.
• terminated: process exits.

The transitions are driven by:

• running → not-ready: pre-stop hook (or explicit admin call).
• not-ready → draining: SIGTERM received.
• draining → stopping: in-flight work finished.
• stopping → terminated: cleanup complete.

Each transition is a state change; concurrent observers (handlers, workers) react to it.

Implementation¶

type ServiceState int32

const (
    StateRunning ServiceState = iota
    StateNotReady
    StateDraining
    StateStopping
    StateTerminated
)

type Service struct {
    state atomic.Int32
    done  chan struct{}
    once  sync.Once
}

func (s *Service) State() ServiceState {
    return ServiceState(s.state.Load())
}

func (s *Service) Ready() bool {
    return s.State() == StateRunning
}

func (s *Service) BeginNotReady() {
    s.state.CompareAndSwap(int32(StateRunning), int32(StateNotReady))
}

func (s *Service) BeginDrain() {
    s.state.CompareAndSwap(int32(StateNotReady), int32(StateDraining))
    s.state.CompareAndSwap(int32(StateRunning), int32(StateDraining))
    s.once.Do(func() { close(s.done) })
}

Readiness probe handler:

func (s *Service) Health(w http.ResponseWriter, r *http.Request) {
    if s.Ready() {
        w.WriteHeader(200)
        return
    }
    w.WriteHeader(503)
}

Pre-stop endpoint:

func (s *Service) PreStop(w http.ResponseWriter, r *http.Request) {
    s.BeginNotReady()
    w.WriteHeader(200)
}

Signal handler:

sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, syscall.SIGTERM)
go func() {
    <-sigCh
    svc.BeginDrain()
}()

The svc.done channel closes once on BeginDrain. Workers and handlers observe it; they drain in-flight and exit.

Signal Handling and Close Triggers¶

Signals (SIGTERM, SIGINT) trigger shutdown. The signal handler must convert them into a clean shutdown sequence.

Best practice: only one shutdown initiator¶

The signal handler should be the only source of "start shutdown". Other code paths should request shutdown via the same channel:

type App struct {
    shutdown chan struct{}
    once     sync.Once
}

func (a *App) Shutdown() {
    a.once.Do(func() { close(a.shutdown) })
}

// Signal handler:
go func() {
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGTERM, syscall.SIGINT)
    <-sigCh
    a.Shutdown()
}()

// Admin endpoint:
http.HandleFunc("/admin/shutdown", func(w http.ResponseWriter, r *http.Request) {
    a.Shutdown()
})

// On fatal error:
func (a *App) onFatalError(err error) {
    log.Printf("fatal: %v", err)
    a.Shutdown()
}

All sources converge on a.Shutdown(), which is idempotent via sync.Once. The application sees one shutdown signal.

Signal handler must not block¶

The signal handler should be tiny and non-blocking. If you do real work inside the handler, the signal delivery may queue (depending on the runtime), and the application may not respond promptly.

// BAD: signal handler does real work
go func() {
    <-sigCh
    drainQueue() // takes minutes
    cleanupResources()
    os.Exit(0)
}()

Better:

go func() {
    <-sigCh
    a.Shutdown() // returns immediately
}()

// Main goroutine waits for shutdown to complete.
<-a.done
drainQueue()
cleanupResources()
os.Exit(0)

The signal handler signals; the main goroutine does the work. This separation makes the signal handler robust.

Handling SIGUSR1, SIGHUP for in-process reload¶

Some applications use SIGUSR1 or SIGHUP to trigger reload (re-read config, rotate logs). This is not shutdown; do not converge it with the shutdown channel.

go func() {
    sigCh := make(chan os.Signal, 1)
    signal.Notify(sigCh, syscall.SIGUSR1)
    for range sigCh {
        a.Reload()
    }
}()

Reload uses its own mechanism. Shutdown is reserved for actual termination.

Distributed Shutdown Coordination¶

In a distributed system, a node's shutdown is one event in a larger coordinated dance.

Single-leader shutdown¶

If the cluster has a leader, the leader's shutdown initiates a re-election. New leader takes over. Old leader drains its in-flight work and exits.

The close coordination is local to the old leader: drain → close internal channels → exit. Coordination with the new leader is via external state (etcd, ZooKeeper, etc.), not via channels.

Leaderless shutdown¶

A node going down notifies peers (e.g., via gossip). Peers stop routing requests to it. The node drains and exits.

Within the node, channels handle the drain. The "notify peers" step is a network operation, not a channel close.

Centralised orchestrator¶

An orchestrator (Kubernetes, Nomad) decides when to terminate. The node receives a signal; the close protocol begins.

The orchestrator may impose a deadline. The node must complete shutdown within it or be forcibly killed.

Implication for channel close patterns¶

Distributed coordination changes nothing about channel close inside a process. The close patterns are the same. What changes is:

• The signal that triggers shutdown.
• The deadline by which shutdown must complete.
• The acceptable level of data loss.

For most services, these are externally-imposed parameters. Design close protocols to respect them.

Health Checks and Pre-Close Behaviour¶

Health checks (liveness, readiness) interact with close.

Liveness probe during shutdown¶

Liveness probes ask "is the process alive?" If it fails, Kubernetes kills the pod.

During shutdown, the process is still alive (responding) but is in a transitional state. Should liveness still return 200?

Convention: yes. Liveness should fail only if the process is wedged. Shutdown is a controlled state; the process is responsive.

If liveness fails during shutdown, Kubernetes kills the pod, abandoning the drain. Bad.

Readiness probe during shutdown¶

Readiness asks "should new traffic be routed here?" During shutdown, the answer is no.

func ReadinessHandler(s *Service) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        if s.State() == StateRunning {
            w.WriteHeader(200)
        } else {
            w.WriteHeader(503)
        }
    }
}

Flipping readiness to fail is one of the earliest steps in shutdown. It causes Kubernetes to remove the pod from the service endpoints, draining traffic from the load balancer's perspective.

Drift between readiness and in-flight work¶

If readiness fails at T=0 and the load balancer takes 5 seconds to converge, there are 5 seconds of new connections arriving even though we are not ready. These must still be handled.

The pre-stop hook can sleep for this period before SIGTERM:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "wget -q -O- localhost:8080/notready; sleep 10"]

The application sees /notready (or similar), flips state, returns 200. Then sleeps 10 seconds. Then exits the pre-stop script, allowing Kubernetes to send SIGTERM.

During the 10-second window, the application keeps serving in-flight work but reports not-ready. The load balancer drains. After the window, SIGTERM triggers the actual shutdown.

This is the production-grade pattern. The channel close happens at SIGTERM, when traffic has already been drained externally.

Backpressure and Close¶

When the consumer is slow, the producer's send blocks. Close interacts with backpressure.

Backpressure absorbs short bursts¶

A buffered channel absorbs bursts: producer sends N items, buffer fills, producer blocks. Consumer catches up; producer unblocks.

Close during backpressure: producer is blocked on ch <- v. Close fires. Producer's send panics (send on closed).

To avoid this, the producer must select on a done channel:

select {
case ch <- v: // succeeded
case <-done:  // cancellation
    return
}

Now under backpressure, if close fires, the producer sees done and returns. No panic.

Backpressure across services¶

In a multi-service system, backpressure surfaces as latency. Service A calls B; B is slow; A waits. A's caller (C) waits on A. C's caller (D) waits.

If any of these times out, the timeout fires; the call returns an error. No close happens.

But: if the system is failing, you may want to start dropping load. Two strategies:

1. Reject early. Service A rejects requests if its queue is full. Caller gets an error, can retry against another instance.
2. Tail drop. Service A accepts requests but drops the oldest in its queue when new ones arrive. Maintains throughput but with random latency tails.

Neither involves channel close directly, but both interact with the channel-based queues at each service.

Performance Cost of Safe-Close Patterns¶

The patterns we have discussed have measurable costs. Understanding the costs helps when deciding whether to apply them in hot paths.

Cost of bare close¶

close(ch)

About 50-100 nanoseconds for a channel with no waiters. Linear in number of waiters: each wakeup is 100-200 nanoseconds.

Cost of sync.Once.Do¶

Uncontended fast path: one atomic load (~1 ns), one branch (~1 ns). Effectively free.

Contended slow path: mutex lock + function call (~100-500 ns). Runs only once in the channel's lifetime.

Cost of done-channel select¶

select {
case ch <- v:
case <-done:
}

About 100-300 nanoseconds, depending on which case wins. The select machinery has a fixed overhead for locking and polling.

Cost of mutex+flag¶

m.Lock()
if !closed {
    ch <- v
}
m.Unlock()

Lock: ~20 ns uncontended, much more contended. Plus the channel send. For a hot path with many senders, the mutex contention dominates.

Comparison for a "hot send"¶

For most production workloads, all of these are sub-microsecond and irrelevant compared to the work the goroutine does (database queries, RPC calls, etc.).

For ultra-hot paths (a million sends per second per CPU), profile. The done-channel pattern is usually within 10% of bare send.

Lock-Free Idempotent Close¶

For the rare case where sync.Once is too expensive:

type FastClose struct {
    done   chan struct{}
    closed atomic.Uint32
}

func New() *FastClose {
    return &FastClose{done: make(chan struct{})}
}

func (f *FastClose) Close() {
    if f.closed.CompareAndSwap(0, 1) {
        close(f.done)
    }
}

func (f *FastClose) Done() <-chan struct{} {
    return f.done
}

The CAS is a single x86 CMPXCHG instruction, about 5-10 nanoseconds. sync.Once is comparable on the fast path (it uses an atomic load + branch).

The real savings of pure-atomic over sync.Once: no allocation. sync.Once is two fields (uint32 + mutex); a manually managed atomic is one field (uint32). For data structures with many of these, the savings add up.

For most uses, prefer sync.Once for readability. Use the atomic version only when profiling shows it matters.

Memory Model Implications¶

Go's memory model says:

The closing of a channel happens before a receive that returns because the channel is closed.

This guarantees that any write the closer made before close(ch) is visible to a receiver after the close.

Practical use¶

var state Config

go func() {
    state = loadConfig()
    close(ready)
}()

<-ready
fmt.Println(state) // guaranteed to see loadConfig's value

The write to state is sequenced before the close. The receive observes the close. Therefore the receive observes the write.

This is the canonical "init-and-broadcast" pattern. Initialize, close ready, all observers see initialized state.

Subtle case: writes after close¶

go func() {
    close(ready)
    state = loadConfig() // ORDER WRONG
}()

<-ready
fmt.Println(state) // may NOT see loadConfig's value

The write is after the close. The memory model does not guarantee that observers of the close see writes that happened after it.

Always sequence writes before the close.

Compositional close-ordering¶

If you close two channels in sequence:

close(a)
close(b)

A receiver of <-b sees writes before close(a), but not necessarily writes between close(a) and close(b). The memory model relates a close to a receive of that channel; transitivity through closes is not guaranteed.

In practice, you rarely care; sequencing is implicit through program order in a single goroutine.

Channel close and atomics¶

If you mix atomic operations with channel close:

atomic.StoreInt32(&flag, 1)
close(done)

A receiver:

<-done
v := atomic.LoadInt32(&flag) // sees 1?

Yes, the atomic store is sequenced before the close. The receiver sees the close, which is happens-after the store. By transitivity, the load sees the store.

The same logic applies to any happens-before relationship through close.

Telemetry: Metrics, Logs, Traces¶

Production observability for close events.

Metrics¶

Useful Prometheus metrics:

• channel_send_total{name} — count of sends.
• channel_recv_total{name} — count of receives.
• channel_close_total{name} — count of closes (should be 0 or 1 for most channels).
• channel_occupancy{name} — current buffer occupancy.
• channel_send_latency_seconds{name} — histogram of send wait time.
• channel_recv_latency_seconds{name} — histogram of receive wait time.

Track these for the top-level channels in your application. Buffer occupancy and send latency are particularly useful: rising occupancy indicates a slow consumer; rising send latency indicates backpressure.

Logs¶

Log every close event with context:

log.WithFields(log.Fields{
    "channel": name,
    "occupancy": len(ch),
    "capacity": cap(ch),
    "trigger": triggerSource,
}).Info("channel closing")

In post-mortems, these logs let you reconstruct the close sequence.

Traces¶

Span every channel close as part of a shutdown trace:

ctx, span := tracer.Start(ctx, "channel.close",
    trace.WithAttributes(
        attribute.String("name", "events"),
        attribute.Int("occupancy", len(ch)),
    ))
defer span.End()
close(ch)

In a shutdown trace, you see the order and timing of each close. A drain that's stuck on one channel is visible as a long span.

Continuous profiling¶

Enable continuous profiling with goroutine, mutex, and block profiles. Look for:

• Goroutine count rising over time: channels not closing, goroutines leaking.
• Mutex contention on channel internals (hchan.lock): close storms or high-fanout broadcasts.
• Block profile entries on channel sends/receives: backpressure.

These point to close-related issues even when no panic has occurred.

Chaos Engineering for Close¶

A practitioner technique: inject failures into close paths to verify the system handles them.

Chaos test: random close mid-pipeline¶

go func() {
    delay := time.Duration(rand.Intn(100)) * time.Millisecond
    time.Sleep(delay)
    pipeline.Cancel()
}()

Run the pipeline under random cancellations. If any run panics or hangs, you have a bug.

Chaos test: slow consumer¶

go func() {
    for v := range pipeline.Out() {
        time.Sleep(10 * time.Millisecond) // simulate slow
        _ = v
    }
}()

Pipeline must apply backpressure or drop without panicking on close.

Chaos test: kill mid-drain¶

Send SIGKILL while shutdown is in progress (in a test). Restart. Verify no data corruption or duplicated work.

This tests the durability of your queues, not just the close protocol, but they are linked.

Chaos test: concurrent close¶

Multiple shutdown triggers (admin endpoint, signal, lifecycle event) arrive simultaneously. The system must converge on a single shutdown.

go func() { svc.Shutdown() }()
go func() { svc.Shutdown() }()
go func() { svc.Shutdown() }()

If any panics or hangs, sync.Once is missing or misused.

Post-Incident Review Patterns¶

After a close-related production incident, a structured review.

Timeline¶

• T+0: anomaly first detected (alert, customer complaint).
• T+x: paged engineer identifies the panic.
• T+y: hotfix deployed.
• T+z: stability confirmed.

Include the times for each step. Identify gaps (slow detection, slow diagnosis, slow fix).

Five whys¶

• Why did the panic occur? (Send on closed channel.)
• Why was the channel closed while a sender was in flight? (Two paths reached close.)
• Why were two paths authorised to close? (Cleanup code in two places.)
• Why was that not caught in code review? (The two paths were in different files.)
• Why was that not caught in tests? (Tests do not exercise the concurrent close path.)

Each "why" points to a process improvement.

Action items¶

• Add unit test that reproduces the panic.
• Add lint rule for "close in deferred cleanup".
• Update onboarding docs with the close-pattern style guide.
• Add stress tests for any module that exposes a Close method.

Make these concrete, assigned, and time-bound.

Practical: Building a Production-Grade Close Library¶

Putting it all together: a reusable close-helper library.

package closer

import (
    "context"
    "errors"
    "sync"
    "sync/atomic"
    "time"
)

var (
    ErrClosed   = errors.New("closer: already closed")
    ErrTimeout  = errors.New("closer: timeout")
)

// Closer is a reusable shutdown coordinator. It is safe to call Close
// from multiple goroutines; only the first triggers shutdown. Subsequent
// calls observe the existing shutdown.
type Closer struct {
    done   chan struct{}
    once   sync.Once
    fn     []func(context.Context) error
    timeout time.Duration

    closing atomic.Bool
    closed  atomic.Bool
    err     error
    errOnce sync.Once
}

// New returns a Closer that runs the given cleanup functions on Close.
// Functions are run in order; first error stops the chain.
func New(timeout time.Duration, fns ...func(context.Context) error) *Closer {
    return &Closer{
        done: make(chan struct{}),
        fn:   fns,
        timeout: timeout,
    }
}

// Closing returns true once Close has been called.
func (c *Closer) Closing() bool { return c.closing.Load() }

// Closed returns true once the shutdown is complete.
func (c *Closer) Closed() bool { return c.closed.Load() }

// Done returns a channel that is closed when shutdown completes.
func (c *Closer) Done() <-chan struct{} { return c.done }

// Err returns the first error from any cleanup function, or nil.
func (c *Closer) Err() error { return c.err }

// Close triggers shutdown. Multiple calls are safe; only the first runs
// cleanup. Returns when cleanup completes or the timeout elapses.
func (c *Closer) Close() error {
    c.once.Do(func() {
        c.closing.Store(true)
        ctx, cancel := context.WithTimeout(context.Background(), c.timeout)
        defer cancel()
        for _, fn := range c.fn {
            if err := fn(ctx); err != nil {
                c.errOnce.Do(func() { c.err = err })
                break
            }
        }
        c.closed.Store(true)
        close(c.done)
    })
    <-c.done
    return c.err
}

This is a small, focused library. It enforces:

• Single shutdown (sync.Once).
• Idempotent Close (returns same error every time).
• Bounded duration (timeout context).
• Observable state (Closing, Closed, Done, Err).

Use it as the basis for service-level shutdown.

Closed Channel as Boolean¶

A neat idiom: a closed channel of struct{} is a boolean that flips once.

type Latch struct {
    ch chan struct{}
}

func NewLatch() *Latch { return &Latch{ch: make(chan struct{})} }
func (l *Latch) Trigger() {
    // Safe-close
    select { case <-l.ch: default: close(l.ch) }
}
func (l *Latch) Wait() { <-l.ch }
func (l *Latch) Triggered() bool {
    select { case <-l.ch: return true; default: return false }
}

The Trigger has a race (two goroutines both pass default, both call close). Fix with sync.Once.

This is the basis of context.Context.Done() and many "one-shot" signals. A latch is a higher-level name for the pattern.

When You Cannot Use Channels¶

Sometimes the close protocol is infeasible. Channels are not appropriate when:

• The number of waiters is huge (millions). Each waiter is a goroutine; the close storm is expensive.
• The signal must be observable without blocking (poll-only). Channels are designed for blocking observation.
• The signal must be reusable. Channels close once.

For these cases, use sync.Cond, atomic.Bool, or eventfd/futex (in cgo).

A reusable signal:

type Reusable struct {
    mu   sync.Mutex
    cond *sync.Cond
    set  bool
}

func New() *Reusable {
    r := &Reusable{}
    r.cond = sync.NewCond(&r.mu)
    return r
}

func (r *Reusable) Set() {
    r.mu.Lock()
    r.set = true
    r.cond.Broadcast()
    r.mu.Unlock()
}

func (r *Reusable) Wait() {
    r.mu.Lock()
    for !r.set { r.cond.Wait() }
    r.mu.Unlock()
}

func (r *Reusable) Reset() {
    r.mu.Lock()
    r.set = false
    r.mu.Unlock()
}

Set and Reset can be called repeatedly. Broadcast wakes all waiters. No close needed; the cond is reusable.

The trade-off: cond is more code and less self-documenting than a channel. Use only when you need reusability or extreme fanout.

Cross-Cutting Concerns: Observability and Close¶

A service's observability infrastructure (metrics, logs, traces) often outlives the service's primary workload. Shutdown should not stop observability prematurely.

Order of shutdown:

1. Stop accepting new work.
2. Drain in-flight work, emitting telemetry as usual.
3. Flush telemetry buffers (metrics push, log flush, trace flush).
4. Close telemetry connections.
5. Exit.

If you close telemetry first, you lose visibility into the rest of shutdown. Bugs in shutdown become invisible.

The close protocol must respect this order. Telemetry is one of the last things to shut down, not one of the first.

Network Service Close¶

A network service has:

• A listener accepting connections.
• Per-connection goroutines.
• Per-connection channels for request/response.
• Shared resources (DB pool, cache).

Shutdown order:

1. Close listener (no new connections).
2. Signal per-connection goroutines to drain (close per-connection done channel).
3. Wait for per-connection goroutines to exit.
4. Close shared resources.
5. Exit.

Implementing step 2: each connection goroutine holds a reference to a global done channel. When done closes, the goroutine wraps up its current request and exits.

func handleConn(c net.Conn, done <-chan struct{}) {
    defer c.Close()
    for {
        select {
        case <-done:
            return
        default:
        }
        c.SetReadDeadline(time.Now().Add(time.Second))
        req, err := readRequest(c)
        if err != nil {
            if isTimeout(err) { continue }
            return
        }
        resp := handleRequest(req)
        if err := writeResponse(c, resp); err != nil {
            return
        }
    }
}

The connection goroutine polls done on each iteration; the read deadline ensures it returns to the poll every second even if the client is idle.

This is the pattern of the standard library's http.Server.Shutdown.

Workers, Producers, and Consumers: A Unified View¶

A unified mental model: any goroutine is either a producer, a consumer, or both. Each has close obligations.

• Producer: sends to a channel. May close it when done. Must observe cancellation.
• Consumer: receives from a channel. Detects close via range or comma-ok. May trigger cancellation.
• Worker: both producer (to its output) and consumer (from its input).

For each goroutine in your system:

• Identify whether it is producer, consumer, or worker.
• Identify the channels it interacts with.
• For each channel, identify its closer.
• For each channel send, ensure cancellation is observable.
• For each channel receive, ensure close is handled (range exits, comma-ok checked).

This is a code-review checklist for any concurrent module.

Real-World Pattern: HTTP Server Graceful Shutdown¶

The standard library's http.Server.Shutdown is a reference implementation worth studying.

// (Simplified from net/http)
func (s *Server) Shutdown(ctx context.Context) error {
    s.inShutdown.Store(true)
    err := s.closeListenersLocked()
    s.closeDoneChanLocked()
    for _, cancel := range s.onShutdown {
        go cancel()
    }
    pollInterval := 500 * time.Millisecond
    timer := time.NewTimer(pollInterval)
    defer timer.Stop()
    for {
        if s.closeIdleConns() && s.numListeners() == 0 {
            return err
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
            timer.Reset(pollInterval)
        }
    }
}