Wait-for-Empty-Channel — Senior Level¶

Table of Contents¶

1. Introduction
2. Ownership as the Root Cause
3. Synchronisation Primitives, Side by Side
4. Channel Send/Receive Happens-Before Edges
5. Mutex and RWMutex Edges
6. Atomic Edges
7. sync.Once and Idempotent Initialisation
8. sync.Cond as a Predicate Wait
9. Semaphore Patterns
10. Token Bucket Patterns
11. Drain Protocols: The Five Variants
12. Race-Free Completion Signalling
13. Structured Concurrency
14. Ownership Patterns Per Channel
15. Designing APIs Without Polling
16. The <-chan and chan<- Discipline
17. Goroutine Lifecycle Diagrams
18. Hierarchical Cancellation
19. Backpressure Without Polling
20. Supervisors and Restart Strategies
21. Saga-Style Coordinations
22. Generic Worker Pool API
23. The "Token Return" Drain Pattern
24. Pipelines with Backpressure and Cancellation
25. Diff-Driven Refactor of a Legacy System
26. Production Failure Modes That Trace Back to Polling
27. Common Variants of the Anti-Pattern
28. Composable Helpers You Should Have
29. Testing Strategies for Race-Free Code
30. Code Review Discipline
31. Migration Across a Codebase
32. Self-Assessment
33. Summary

Introduction¶

The senior level question is no longer "is len(ch) wrong?" (it is) and no longer "what should I use instead?" (you know the catalogue). It is how do you design a system so that the polling impulse never even arises? That question takes you into ownership rules, structured concurrency, drain protocols, and API design.

This file is dense. It assumes you have read junior.md and middle.md. It assumes you have implemented worker pools and pipelines in production, fixed at least one race condition that involved goroutine lifetime, and read the Go memory model end to end.

The goal is to walk you from "uses the right primitives" to "designs systems where the wrong primitives cannot be used." The two are very different jobs.

After reading you should:

• Articulate ownership rules for every channel in a system, in a sentence per channel.
• Choose between WaitGroup, errgroup, chan struct{}, context, sync.Cond, sync.Once, atomic, and Mutex for any synchronisation requirement, and justify the choice.
• Implement five drain protocols and pick the right one for a given system.
• Build a worker pool whose API makes the polling anti-pattern impossible.
• Audit a codebase for goroutine lifetime issues and produce a migration plan.

Ownership as the Root Cause¶

Every time you see len(ch) used for synchronisation, the underlying problem is unclear ownership. Two questions reveal it:

1. Who owns the channel?
2. What event marks the channel's end of life?

If the answer to either is "I'm not sure," the polling will arrive. The polling is the symptom; the unclear ownership is the disease.

What ownership means¶

Ownership in Go's channel context is the answer to: who is allowed to close this channel? Who is allowed to send to it? Who reads from it?

The canonical rules:

• One owner. A single goroutine (or coordinator function) is the owner. Everyone else is a borrower.
• Sender closes. If there is one sender, that sender owns and closes. If there are many senders, a coordinator waits for all of them and closes.
• Receivers do not close. Closing on the receiver side risks send on closed channel panics.
• The owner is documented. In code comments. In package doc. In team standards.

When ownership is clear, the question "when is the channel done?" has an answer: the moment the owner closes it. The polling never arises.

When ownership is unclear¶

A typical symptom: a channel is created in main, passed to four goroutines, none of which know who closes. The channel never closes, so receivers never observe completion. Someone adds for len(ch) > 0 { time.Sleep(...) } as the work-around. The bug ships.

Fix the ownership first. The synchronisation fix is then trivial.

Ownership patterns to memorise¶

PATTERN A: producer-owned
    producer ──send──► channel ──recv──► consumer
                                          consumer ranges
                                          producer closes when done

PATTERN B: coordinator-owned with N producers
    producers ──send──► channel ──recv──► consumer
                       (WaitGroup)
                                          consumer ranges
                                          coordinator waits for all
                                          producers, then closes

PATTERN C: lifetime-bound
    parent goroutine creates channel,
    parent owns the goroutine that uses it,
    parent closes the channel when the goroutine exits

PATTERN D: broadcast (done channel)
    one closer (often parent),
    many waiters,
    close is the broadcast

Most production code fits one of these four. If your code does not, simplify.

The "two senders" trap¶

The single most common cause of polling in real codebases: a channel with two senders and no coordinator.

ch := make(chan int)
go producer1(ch)
go producer2(ch)
// who closes ch?

If producer1 closes, producer2 may panic. If producer2 closes, producer1 may panic. If neither closes, the consumer hangs. So the engineer writes for len(ch) > 0 and pretends it works.

The fix: introduce a coordinator.

ch := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); producer1(ch) }()
go func() { defer wg.Done(); producer2(ch) }()
go func() { wg.Wait(); close(ch) }()

for v := range ch {
    handle(v)
}

The coordinator owns the close. Both producers can send safely. The consumer ranges.

Synchronisation Primitives, Side by Side¶

Go ships seven core synchronisation primitives. Each solves a specific problem; together they cover every concurrent need. The wait-for-empty-channel anti-pattern is what happens when you try to make one primitive do the job of another.

Table of properties:

Each primitive provides a specific kind of happens-before edge. None of them provides a happens-before edge from "send" to "future read of len," which is why the anti-pattern is unfixable with any primitive — it must be replaced.

How to choose¶

Walk the requirements top-to-bottom:

1. Is the goal communication of a value? Channel.
2. Is the goal "wait for N goroutines to finish"? WaitGroup or errgroup.
3. Is the goal "mutual exclusion over a critical section"? Mutex or RWMutex.
4. Is the goal "run this once across many callers"? Once.
5. Is the goal "wait for a predicate over shared state to become true"? Cond (rarely; channels usually win).
6. Is the goal a counter or boolean flag? Atomic.
7. Is the goal cancellation or deadline? Context.

If the goal is "wait until the channel has no more items," the answer is also a channel — close-and-range. Not len.

Channel Send/Receive Happens-Before Edges¶

The specification states, in its current form:

The k'th receive on a channel with capacity C is synchronized before the (k+C)'th send from that channel completes.

This is the backpressure edge. For an unbuffered channel (C=0), every send waits for a receive: recv[k] synchronizes before send[k]. For a buffered channel with C=64, the 64th receive synchronizes before the 128th send.

And the other two edges:

A send on a channel is synchronized before the completion of the corresponding receive. The closing of a channel is synchronized before a receive that returns because the channel is closed.

These three edges are the only ones that channel operations create. Every other "channel does X" intuition is wrong. In particular:

• A receive does not synchronize the send that put the next value into the buffer. It only synchronizes the one it received.
• A len(ch) call does not create any edge. It reads a counter under an internal lock but the read is non-synchronizing in the memory-model sense.
• A cap(ch) call is a compile-time-known property of the channel; reading it creates no edge.
• A nil channel send or receive blocks forever; this is sometimes useful in a select to disable a branch.

The senior engineer reasons about channel-based code in terms of these three edges. The polling pattern fails because no edge exists at the polled site.

Using the third edge for backpressure¶

The k'th receive synchronizes before the (k+C)'th send. This is the source of the standard "buffered channel as a semaphore" pattern:

sem := make(chan struct{}, N) // capacity N

// acquire
sem <- struct{}{}
// critical section, up to N concurrent
<-sem
// release

The send acquires, the receive releases. The third edge guarantees the (k+N)'th send waits for the k'th receive — i.e., the (N+1)'th acquirer waits for the 1st releaser. The semaphore is correct.

A done channel uses the second edge. A range over a closed channel uses the second edge. A producer-consumer pipeline uses the first edge. Senior code uses these edges deliberately and names them in comments when the design is subtle.

Mutex and RWMutex Edges¶

sync.Mutex.Unlock synchronizes before the next Lock. sync.RWMutex.Unlock synchronizes before the next Lock or RLock; RUnlock synchronizes before the next Lock.

This means: any write made under the lock is visible to any subsequent reader who takes the lock. The lock is the publication mechanism.

var mu sync.Mutex
var x int

// writer
mu.Lock()
x = 7
mu.Unlock()

// reader
mu.Lock()
v := x // guaranteed to see 7 if the writer's Unlock happened first
mu.Unlock()

You cannot use a mutex to publish a channel's length. The channel state is owned by the channel; your mutex protects whatever variables it protects, no more.

This is why "wrap len(ch) in a mutex" does not fix the race. The mutex publishes mutations to whatever shared variables you write under it. It does not publish the channel's internal counter, because that counter is not protected by your mutex.

Picking Mutex vs Channel¶

A useful heuristic: if the data you are sharing is a fact (a value), prefer channels. If the data you are sharing is a state (a mutable object), prefer mutex.

• "The job is done" — fact — channel close.
• "The result of the computation is 42" — fact — channel value.
• "The cache currently has 7 entries" — state — mutex over the cache.
• "The HTTP server is shutting down" — fact — channel close.

The wait-for-empty-channel anti-pattern is what happens when you treat a fact ("the work is done") as a state ("the buffer is empty"). The state is the wrong variable to observe.

Pitfalls in mutex use¶

• Locking around a channel op. The lock can serialise unrelated work, hurting throughput. Channels are already thread-safe; do not double-lock.
• Recursive locking. sync.Mutex is not re-entrant. Calling Lock twice from the same goroutine deadlocks.
• Forgetting Unlock. defer mu.Unlock() immediately after mu.Lock() is the safest idiom.
• Locking the wrong granularity. A single global mutex serialises everything; many fine-grained mutexes increase complexity. Use one mutex per coherent state group.

Atomic Edges¶

sync/atomic.LoadXxx and StoreXxx create a happens-before edge: a store synchronizes before any load that observes its value. They are useful for lock-free counters, boolean flags, and one-shot configuration.

var stopped int32

go func() {
    for atomic.LoadInt32(&stopped) == 0 {
        work()
    }
}()

// signal stop:
atomic.StoreInt32(&stopped, 1)

This is a correct way to signal "stop now" to a goroutine. It is not a substitute for channels in most cases — atomic flags do not compose with select and do not carry data — but for a hot inner loop where channel selection would be expensive, an atomic flag is sometimes preferred.

Since Go 1.19 the preferred API is the typed atomics: atomic.Bool, atomic.Int32, etc.

var stopped atomic.Bool

go func() {
    for !stopped.Load() {
        work()
    }
}()

stopped.Store(true)

This is cleaner. Use it.

When to prefer atomic over channel¶

• The hot loop runs a million iterations per second; channel ops would dominate.
• The signal is "just one boolean" — no payload, no select-compose.
• You explicitly want non-blocking check.

When to prefer channel over atomic¶

• The signal needs to compose with other events via select.
• The signal carries a value.
• The consumer should block until the signal arrives.
• The producer's writes-before-store must be visible — channels and atomics both provide this, but channels are more idiomatic.

sync.Once and Idempotent Initialisation¶

sync.Once.Do(f) runs f at most once, even if Do is called from many goroutines. It synchronizes the completion of f before the return of any subsequent call.

Common uses:

var (
    config     *Config
    configOnce sync.Once
)

func GetConfig() *Config {
    configOnce.Do(func() {
        config = loadConfig()
    })
    return config
}

Or for "close once":

var (
    once    sync.Once
    done    = make(chan struct{})
)

func Finish() {
    once.Do(func() { close(done) })
}

The "close once" idiom is invaluable when multiple goroutines might trigger completion. It prevents the double-close panic without requiring explicit coordination.

Once is not a substitute for waiting¶

Once tells you "the initialisation finished" but blocks only the first caller. Subsequent calls return immediately. If you need to wait for the initialisation to finish from another goroutine, the standard pattern is:

var initDone = make(chan struct{})
var once sync.Once

func ensureInit() {
    once.Do(func() {
        initialise()
        close(initDone)
    })
}

func waitForInit() {
    <-initDone
}

Once.Do is the executor; the channel close is the broadcast.

sync.Cond as a Predicate Wait¶

sync.Cond exists for one specific use case: waiting for a predicate over shared state to become true. The classical example is a bounded queue:

type BoundedQueue struct {
    mu   sync.Mutex
    cond *sync.Cond
    buf  []int
    cap  int
}

func New(cap int) *BoundedQueue {
    q := &BoundedQueue{cap: cap}
    q.cond = sync.NewCond(&q.mu)
    return q
}

func (q *BoundedQueue) Put(x int) {
    q.mu.Lock()
    defer q.mu.Unlock()
    for len(q.buf) == q.cap {
        q.cond.Wait()
    }
    q.buf = append(q.buf, x)
    q.cond.Broadcast()
}

func (q *BoundedQueue) Get() int {
    q.mu.Lock()
    defer q.mu.Unlock()
    for len(q.buf) == 0 {
        q.cond.Wait()
    }
    x := q.buf[0]
    q.buf = q.buf[1:]
    q.cond.Broadcast()
    return x
}

The len(q.buf) calls here are not the anti-pattern: they are under the mutex, in the loop condition of a cond.Wait(), and they re-check after every wake. This is the correct shape of predicate-wait.

When Cond is right¶

• The state has a non-trivial predicate (more than just "non-zero").
• Multiple goroutines wait on different predicates over the same state.
• You need to broadcast a state change to many waiters.

Why Cond is rare¶

• It does not compose with select.
• It requires manual mutex management.
• Most use cases are better served by a channel.

For a "wait until the queue is non-empty" need, a channel-based queue is simpler. For a "wait until reservation count is ≤ N" need, a channel-based semaphore is simpler. Cond shines only when no channel-based equivalent expresses the predicate naturally.

Cond pitfalls¶

• Using if instead of for. Spurious wake-ups exist. Always for predicate { cond.Wait() }.
• Calling Wait without holding the lock. Panics.
• Forgetting to Broadcast. Waiters hang. Always Signal or Broadcast after mutating the state.
• Mismatched Mutex. Cond's mutex must be the same one held by the mutating code. Two Conds with two Mutexes will not coordinate.

Semaphore Patterns¶

A semaphore bounds concurrent access to a resource. In Go the standard implementation is a buffered channel.

sem := make(chan struct{}, N)

func use() {
    sem <- struct{}{} // acquire
    defer func() { <-sem }() // release
    work()
}

The capacity N is the maximum concurrent users. Acquire blocks when N are already inside.

Weighted semaphore¶

When different operations cost different amounts of resource, weight matters. The standard library does not include a weighted semaphore; use golang.org/x/sync/semaphore:

import "golang.org/x/sync/semaphore"

sem := semaphore.NewWeighted(100)

func use(ctx context.Context, weight int64) error {
    if err := sem.Acquire(ctx, int64(weight)); err != nil {
        return err
    }
    defer sem.Release(int64(weight))
    return work()
}

The weighted semaphore respects context cancellation, which the channel-based one does not natively. For shutdown-aware code, prefer the weighted version even if all weights are 1.

When the semaphore wins over the worker pool¶

A worker pool has N long-lived goroutines. A semaphore has 0 long-lived goroutines and spawns ephemeral ones up to N concurrent. The trade-off:

• Pool: cheaper per-task (no goroutine spawn), better for many small tasks.
• Semaphore: simpler API, no need to manage worker lifecycle, better for irregular workloads.

For a request handler that fans out to a small bounded set of operations, the semaphore is usually right. For a long-running service that processes a stream, the worker pool is usually right.

Combining semaphore with errgroup¶

A common shape:

g, ctx := errgroup.WithContext(ctx)
sem := semaphore.NewWeighted(8)
for _, item := range items {
    item := item
    if err := sem.Acquire(ctx, 1); err != nil {
        break
    }
    g.Go(func() error {
        defer sem.Release(1)
        return process(ctx, item)
    })
}
return g.Wait()

The semaphore bounds concurrency; errgroup propagates errors and cancels on the first failure. Together they form the canonical "bounded concurrent processing" idiom.

Token Bucket Patterns¶

A token bucket is a rate-limiter pattern: tokens accrue at a fixed rate, each operation consumes one. golang.org/x/time/rate is the standard implementation.

limiter := rate.NewLimiter(10, 1) // 10 per second, burst of 1

for _, item := range items {
    if err := limiter.Wait(ctx); err != nil {
        return err
    }
    handle(item)
}

The token bucket is not directly related to the wait-for-empty-channel anti-pattern, but it is mentioned here because engineers sometimes confuse "limit the rate" with "limit the concurrency" and reach for the wrong tool.

• Rate = events per second. Use a rate.Limiter.
• Concurrency = number in flight at once. Use a semaphore.

A polling loop in production may be there because someone tried to use one as the other.

Drain Protocols: The Five Variants¶

When a service shuts down, in-flight work must finish (or be abandoned) before the process exits. "Drain" is the act of completing that work. There are five distinct drain protocols, each appropriate for different needs.

Protocol 1: hard stop¶

cancel()
// no wait; process exits

The context cancels; goroutines are expected to exit; the process terminates without confirming. Use only when correctness does not depend on completion (e.g., stateless cache).

Protocol 2: best-effort drain with deadline¶

cancel()
done := make(chan struct{})
go func() { wg.Wait(); close(done) }()
select {
case <-done:
case <-time.After(30 * time.Second):
    log.Println("timed out waiting for shutdown")
}

Cancel triggers shutdown; the WaitGroup confirms completion; the deadline bounds the wait. After the deadline, abandon. The standard shape for HTTP servers.

Protocol 3: full drain¶

// stop accepting new work
close(inputs)
// wait for all workers to finish
wg.Wait()

No deadline; the shutdown blocks until every in-flight item completes. Use only when latency on shutdown is acceptable and correctness depends on it (e.g., financial batch jobs).

Protocol 4: graceful with reject¶

close(stopAcceptingNew)
// existing goroutines keep working
// new submissions return ErrShuttingDown
wg.Wait()

A two-phase shutdown: stop accepting, then drain. The system rejects new work cleanly while finishing what is in flight. The standard shape for queue consumers.

Protocol 5: graceful with priority¶

cancel(lowPriorityCtx)
wg.Wait()
cancel(highPriorityCtx)

Different work has different priorities. Lower-priority work is cancelled first; higher-priority work continues. Useful in mixed workloads where some operations must finish (writes) and others can be abandoned (reads).

Which to pick¶

The polling anti-pattern often arises in services that want Protocol 3 but lack the synchronisation. The right move is to add a WaitGroup, not a polling loop.

Race-Free Completion Signalling¶

The crux of avoiding the anti-pattern is making completion observable. Every long-running goroutine should emit a completion signal. The patterns:

Pattern: <-done¶

type Worker struct {
    done chan struct{}
}

func (w *Worker) Run(ctx context.Context) {
    defer close(w.done)
    for {
        select {
        case <-ctx.Done():
            return
        case j := <-w.jobs:
            handle(j)
        }
    }
}

func (w *Worker) Wait() {
    <-w.done
}

The worker exposes Wait(). The caller does not need to know anything about goroutines.

Pattern: errgroup return¶

g, ctx := errgroup.WithContext(parent)
g.Go(worker1)
g.Go(worker2)
return g.Wait()

Wait returns when every Go has returned, with the first error if any. No external completion signal needed.

Pattern: result channel¶

results := make(chan Result, len(items))
for _, item := range items {
    go func(item Item) {
        results <- process(item)
    }(item)
}
for i := 0; i < len(items); i++ {
    handle(<-results)
}

Consume exactly N results. The loop knows the count up front. No len(results) check needed; the for-loop counter is the synchronisation.

Pattern: structured close¶

results := make(chan Result)
go func() {
    defer close(results)
    for _, item := range items {
        results <- process(item)
    }
}()
for r := range results {
    handle(r)
}

Producer closes; consumer ranges. The two patterns above (count-based and close-based) are the only two ways to terminate a result-consumption loop without polling.

Structured Concurrency¶

The structured-concurrency principle: every goroutine has an enclosing scope; the scope does not return until every goroutine has finished. Go does not enforce this at the language level (unlike Kotlin or Java's Loom-with-StructuredTaskScope), but the idiomatic patterns approach it.

The principle in code:

func parent(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)
    g.Go(func() error { return childA(ctx) })
    g.Go(func() error { return childB(ctx) })
    return g.Wait() // parent does not return until both children exit
}

parent is the scope. The goroutines spawned inside it cannot outlive it. The wait is explicit; the cancellation cascades; the errors propagate.

When this principle is honoured, the wait-for-empty-channel anti-pattern becomes impossible: every goroutine has a Wait to call.

When this principle is violated, the anti-pattern thrives: someone needs to know when work is done but has no Wait to call, so they poll.

Anti-patterns that violate structured concurrency¶

• "Fire and forget" goroutines: go doStuff() with no associated wait.
• Long-lived background goroutines started from request handlers.
• Goroutines that start other goroutines without Add-ing to the parent's WaitGroup.
• Goroutines that store themselves in a global registry without a lifetime.

Senior code avoids all four. The rule: if you cannot point to the Wait that joins this goroutine, you have a leak.

Ownership Patterns Per Channel¶

For every channel in your system, you should be able to answer four questions:

1. Who creates it?
2. Who sends to it?
3. Who receives from it?
4. Who closes it?

Document the answers. In a code comment, in package doc, or in an architecture decision record. Here are the canonical patterns.

Per-request channels¶

type Request struct {
    Payload   []byte
    Response  chan Response // created by caller, sent by handler
}

• Created by: the caller.
• Sent by: the handler.
• Received by: the caller.
• Closed by: nobody (the channel is GC'd after the response).

For a single response, capacity 1 means the handler never blocks. For multiple responses, the handler closes after the last one.

Long-lived service channels¶

type Service struct {
    jobs chan Job // created in New, sent by callers, received by workers, closed in Close
}

• Created by: New.
• Sent by: callers via Submit.
• Received by: workers in their loop.
• Closed by: Close.

The ownership is Service. Close is the single closer.

Pipeline stage channels¶

func stage(in <-chan T) <-chan T {
    out := make(chan T)
    go func() {
        defer close(out)
        for v := range in {
            out <- transform(v)
        }
    }()
    return out
}

• Created by: the stage function.
• Sent by: the stage's goroutine.
• Received by: the next stage.
• Closed by: the stage's goroutine (on input close).

This is the cleanest pattern in Go. Each stage owns its output channel.

Broadcast channels¶

type Broadcaster struct {
    sub chan chan<- Message
}

• Created by: subscribers, who pass their receive channel to the broadcaster.
• Sent by: the broadcaster.
• Received by: subscribers.
• Closed by: subscriber (when unsubscribing) or broadcaster (when terminating).

More complex than the others; needs careful documentation.

Designing APIs Without Polling¶

The best defence against the anti-pattern is an API that prevents it. Eight design rules.

Rule 1: do not expose channels¶

Public methods return values or accept callbacks. Channels are an internal implementation detail.

// Bad
func (s *Service) Jobs() <-chan Job

// Good
func (s *Service) ForEachJob(ctx context.Context, fn func(Job) error) error

Rule 2: every long-running operation returns a wait handle¶

// Bad
func (s *Service) Start() // who waits?

// Good
func (s *Service) Start(ctx context.Context) error // blocks until done or ctx cancels

Rule 3: every cancellable operation accepts a context¶

The context is the cancellation. Callers pass ctx, the operation respects ctx.Done().

Rule 4: errors are returned, not logged¶

A polling loop's signature is func() {}. A correct loop's signature is func() error. The latter has nowhere to hide the error.

Rule 5: every constructor that spawns has a Close¶

If New starts a goroutine, then *T has a Close() error method that waits for the goroutine to exit. No exceptions.

Rule 6: hide the WaitGroup¶

Wrap the WaitGroup in a method. Callers should not need to know it exists.

type Pool struct {
    wg sync.WaitGroup
    // ...
}

func (p *Pool) Wait() { p.wg.Wait() }

Rule 7: prefer iterators to channels for public APIs¶

In Go 1.23+, range-over-function lets you expose iteration without exposing channels:

func (p *Pool) Results() iter.Seq[Result] {
    return func(yield func(Result) bool) {
        for r := range p.results {
            if !yield(r) {
                return
            }
        }
    }
}

Callers for r := range p.Results() without seeing a channel.

Rule 8: document ownership in package doc¶

// Pool processes jobs in N worker goroutines.
//
// Ownership:
//   - p.jobs is owned by Pool; Submit sends, workers receive,
//     Close closes.
//   - p.results is owned by Pool; workers send,
//     callers receive via Results(); Close closes after all
//     workers exit.
//   - Workers are owned by Pool; Close waits for them to exit.

The next engineer reads this paragraph and knows where every synchronisation lives.

The <-chan and chan<- Discipline¶

A small but powerful discipline: use directional channel types in function signatures. They document intent and prevent misuse.

// Producer: writes only
func produce(out chan<- Job) {
    for _, j := range source {
        out <- j
    }
    close(out)
}

// Consumer: reads only
func consume(in <-chan Job) {
    for j := range in {
        handle(j)
    }
}

The compiler enforces:

• chan<- T cannot be received from.
• <-chan T cannot be sent to.
• <-chan T cannot be closed.

That last rule is critical: it prevents the "consumer closes the channel" anti-pattern at compile time. If you write func consume(in <-chan Job) and someone tries close(in), the compiler rejects it.

Directional channels at boundaries¶

In function parameters, always declare direction. In struct fields, sometimes declare direction:

type Pipeline struct {
    in  chan<- Job  // external writes here
    out <-chan Job  // external reads here
}

The internal worker has the bidirectional view:

func (p *Pipeline) worker(jobs chan Job) { ... }

This idiom localises the bidirectional channel to the single goroutine that needs it.

Goroutine Lifecycle Diagrams¶

A goroutine's lifecycle is: spawn → run → exit. The senior question is: who triggers the exit, and how is it observed?

Diagram A: parent-owned worker¶

spawn:    go w.run(ctx, &wg)
                ▼
             ┌──────────┐
             │ running  │
             └────┬─────┘
                  │ ctx.Done() or jobs closed
                  ▼
             ┌──────────┐
             │  exit    │ ── wg.Done()
             └──────────┘
                  ▼
            (joined by parent's wg.Wait)

The parent owns the lifecycle. The exit is observed via WaitGroup.

Diagram B: self-terminating consumer¶

spawn:    go consume(in)
                ▼
             ┌──────────┐
             │ running  │
             └────┬─────┘
                  │ in closed by producer
                  ▼
             ┌──────────┐
             │  exit    │ ── close(done)
             └──────────┘
                  ▼
            (joined by parent's <-done)

The producer's close drives the consumer's exit. The done channel reports completion.

Diagram C: cancellable broadcast¶

spawn:    go subscriber(ctx)  // N subscribers
                ▼
             ┌──────────┐
             │ running  │
             └────┬─────┘
                  │ ctx.Done() (all see it)
                  ▼
             ┌──────────┐
             │  exit    │ ── wg.Done()
             └──────────┘
                  ▼
            (joined by g.Wait or wg.Wait)

All subscribers exit simultaneously when the context cancels.

When the diagram is unclear¶

If you cannot draw the diagram for a goroutine, the design is wrong. The diagram is the documentation. Every long-running goroutine in your system should have a clearly defined lifecycle, drawable on a napkin.

Hierarchical Cancellation¶

Contexts compose. A child context cancels when its parent cancels, when its own cancel is invoked, or when its deadline fires.

ctx := context.Background()
ctx1, cancel1 := context.WithCancel(ctx)
ctx2, cancel2 := context.WithTimeout(ctx1, 5*time.Second)
ctx3, cancel3 := context.WithDeadline(ctx2, deadline)
defer cancel1()
defer cancel2()
defer cancel3()

ctx3 cancels when any of: cancel3 invoked, deadline reached, cancel2 invoked, 5 seconds elapsed, cancel1 invoked, the parent (context.Background) cancels. (The last is impossible; Background never cancels.)

This composition lets you build hierarchical cancellation that mirrors your goroutine tree. The senior pattern: every level of the tree gets its own context, derived from its parent.

func server(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)
    for i := 0; i < 4; i++ {
        g.Go(func() error {
            return handler(ctx)
        })
    }
    return g.Wait()
}

func handler(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)
    for j := 0; j < 8; j++ {
        g.Go(func() error {
            return worker(ctx)
        })
    }
    return g.Wait()
}

func worker(ctx context.Context) error {
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case j := <-jobs:
            if err := process(ctx, j); err != nil {
                return err
            }
        }
    }
}

When the top-level ctx cancels, every leaf worker observes ctx.Done() within microseconds. The cancellation cascades.

The wait-for-empty-channel anti-pattern is the absence of this cascade: a goroutine that does not respect any cancellation, so the caller must invent one — and the only observable they invent is len(ch).

Backpressure Without Polling¶

Backpressure is the feedback from a slow consumer to a fast producer: "slow down, I cannot keep up." The wrong way to implement it:

for {
    if len(ch) > threshold {
        time.Sleep(time.Millisecond)
        continue
    }
    ch <- next()
}

The right way: let the channel block.

for _, item := range source {
    ch <- item // blocks until the consumer is ready
}

If you want to do something else while waiting, use a select:

for _, item := range source {
    select {
    case ch <- item:
    case <-ctx.Done():
        return ctx.Err()
    }
}

If you want to drop on backpressure rather than wait:

for _, item := range source {
    select {
    case ch <- item:
    default:
        metrics.Inc("dropped")
    }
}

In none of these does the producer poll len(ch). The channel's own semantics provide the backpressure. The producer's behaviour adapts via select, not via a sleep loop.

Tunable backpressure¶

A common need: "block up to 100 ms, then drop."

select {
case ch <- item:
case <-time.After(100 * time.Millisecond):
    metrics.Inc("timed_out")
}

Or with context:

ctx, cancel := context.WithTimeout(parent, 100*time.Millisecond)
defer cancel()
select {
case ch <- item:
case <-ctx.Done():
    metrics.Inc("timed_out")
}

The context variant composes with the rest of the system's cancellation.

Supervisors and Restart Strategies¶

Long-running services often have supervisor goroutines that restart failed workers. The naive supervisor:

for {
    if len(workers) == 0 {
        for i := 0; i < N; i++ {
            go worker(...)
            workers = append(workers, 1)
        }
    }
    time.Sleep(time.Second)
}

Two anti-patterns at once: polling a length, sleeping in the loop. The right shape:

type Supervisor struct {
    spawn func(ctx context.Context) error
    n     int
}

func (s *Supervisor) Run(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)
    failures := make(chan struct{}, s.n)
    for i := 0; i < s.n; i++ {
        g.Go(s.runOne(ctx, failures))
    }
    return g.Wait()
}

func (s *Supervisor) runOne(ctx context.Context, fail chan<- struct{}) func() error {
    return func() error {
        for {
            err := s.spawn(ctx)
            if err == nil || ctx.Err() != nil {
                return ctx.Err()
            }
            log.Println("worker failed:", err)
            select {
            case <-ctx.Done():
                return ctx.Err()
            case <-time.After(restartBackoff):
            }
        }
    }
}

Each worker has its own restart loop. The supervisor is just the errgroup. No polling, no shared length.

Backoff strategies¶

The supervisor's restart delay should grow with consecutive failures. A simple exponential backoff:

backoff := time.Second
for {
    err := s.spawn(ctx)
    if err == nil || ctx.Err() != nil {
        return ctx.Err()
    }
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(backoff):
    }
    backoff = min(2*backoff, 30*time.Second)
}

After several seconds of consecutive failures, the supervisor backs off rather than tight-looping. The time.After channel composes with ctx.Done() so shutdown remains fast.

Saga-Style Coordinations¶

Sagas coordinate multi-step workflows where each step may need compensation. The naive saga that uses polling:

for len(pending) > 0 {
    time.Sleep(time.Millisecond)
}
if success {
    commit()
} else {
    rollback()
}

The correct saga uses a coordinator with explicit state transitions:

type Saga struct {
    steps []Step
}

func (s *Saga) Run(ctx context.Context) error {
    var compensations []func(context.Context) error
    for _, step := range s.steps {
        if err := step.Do(ctx); err != nil {
            for i := len(compensations) - 1; i >= 0; i-- {
                _ = compensations[i](ctx)
            }
            return err
        }
        compensations = append(compensations, step.Undo)
    }
    return nil
}

The state machine is explicit. No polling. Compensation runs in reverse order if any step fails.

Generic Worker Pool API¶

A production-quality worker pool API. Walk through the design end to end.

package pool

import (
    "context"
    "errors"
    "sync"
)

// Pool is a fixed-size worker pool that processes tasks of type T using
// the configured handler. The zero Pool is not usable; call New.
type Pool[T any] struct {
    handler func(context.Context, T) error
    in      chan T
    g       *errgroupX
    closeOnce sync.Once
    closed  chan struct{}
}

type errgroupX struct {
    wg      sync.WaitGroup
    cancel  func(error)
    errOnce sync.Once
    err     error
}

// New returns a started pool with `workers` goroutines, each running `handler`.
// The pool processes jobs until ctx is cancelled or Close is called.
func New[T any](ctx context.Context, workers int, handler func(context.Context, T) error) *Pool[T] {
    ctx, cancel := context.WithCancelCause(ctx)
    p := &Pool[T]{
        handler: handler,
        in:      make(chan T),
        g:       &errgroupX{cancel: cancel},
        closed:  make(chan struct{}),
    }
    for i := 0; i < workers; i++ {
        p.g.wg.Add(1)
        go p.run(ctx)
    }
    return p
}

func (p *Pool[T]) run(ctx context.Context) {
    defer p.g.wg.Done()
    for {
        select {
        case <-ctx.Done():
            return
        case job, ok := <-p.in:
            if !ok {
                return
            }
            if err := p.handler(ctx, job); err != nil {
                p.g.errOnce.Do(func() {
                    p.g.err = err
                    p.g.cancel(err)
                })
                return
            }
        }
    }
}

// Submit blocks until a worker accepts the job, ctx cancels, or the pool closes.
func (p *Pool[T]) Submit(ctx context.Context, job T) error {
    select {
    case <-p.closed:
        return ErrPoolClosed
    case <-ctx.Done():
        return ctx.Err()
    case p.in <- job:
        return nil
    }
}

// Close stops accepting new submissions, waits for in-flight jobs to finish,
// and returns the first handler error, if any.
func (p *Pool[T]) Close() error {
    p.closeOnce.Do(func() {
        close(p.closed)
        close(p.in)
    })
    p.g.wg.Wait()
    return p.g.err
}

var ErrPoolClosed = errors.New("pool: closed")

What this API gets right:

• Generic over T — type-safe job payload.
• Constructor starts the workers; no separate Start to forget.
• Submit respects both the caller's context and the pool's lifecycle.
• Close is idempotent (via sync.Once).
• Closing the input channel cleanly terminates workers; Close joins them.
• Errors propagate via the embedded errgroup-like mechanism.
• No len(ch). No polling. No sleeps.

The caller never has the opportunity to poll because the channel is hidden behind methods.

The "Token Return" Drain Pattern¶

A sophisticated drain pattern for systems where in-flight work must complete but new work must be rejected.

type Service struct {
    tokens chan struct{} // capacity = max concurrency
    state  atomic.Int32  // 0=open, 1=draining, 2=closed
    drain  chan struct{}
}

func New(maxConcurrency int) *Service {
    s := &Service{
        tokens: make(chan struct{}, maxConcurrency),
        drain:  make(chan struct{}),
    }
    for i := 0; i < maxConcurrency; i++ {
        s.tokens <- struct{}{}
    }
    return s
}

func (s *Service) Do(ctx context.Context, fn func() error) error {
    if s.state.Load() >= 1 {
        return ErrDraining
    }
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-s.tokens:
        defer func() { s.tokens <- struct{}{} }()
        return fn()
    }
}

func (s *Service) Shutdown(ctx context.Context) error {
    s.state.Store(1) // draining
    // wait for all tokens to be returned
    for i := 0; i < cap(s.tokens); i++ {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-s.tokens:
        }
    }
    s.state.Store(2) // closed
    close(s.drain)
    return nil
}

The token pool is the concurrency limiter and the drain signal. Each in-flight operation holds a token; Shutdown drains every token before returning. No len(s.tokens) polling — the for-loop receives all tokens explicitly.

Why this works¶

• The token count is known (it equals the capacity). The drain receives that exact count.
• Each receive is a synchronisation point; the held happens-before guarantees that the worker has finished before the token returns.
• The atomic state prevents new admissions during drain.
• The context provides bounded waiting.

This pattern is gold for connection pools, rate limiters, and any service with a known concurrency cap.

Pipelines with Backpressure and Cancellation¶

A full pipeline with three stages, backpressure, cancellation, and proper completion.

package pipeline

import (
    "context"
    "sync"
)

func Source(ctx context.Context, items []Item) <-chan Item {
    out := make(chan Item)
    go func() {
        defer close(out)
        for _, item := range items {
            select {
            case <-ctx.Done():
                return
            case out <- item:
            }
        }
    }()
    return out
}

func Transform(ctx context.Context, in <-chan Item, workers int) <-chan Result {
    out := make(chan Result)
    var wg sync.WaitGroup
    wg.Add(workers)
    for i := 0; i < workers; i++ {
        go func() {
            defer wg.Done()
            for item := range in {
                r := compute(item)
                select {
                case <-ctx.Done():
                    return
                case out <- r:
                }
            }
        }()
    }
    go func() {
        wg.Wait()
        close(out)
    }()
    return out
}

func Sink(ctx context.Context, in <-chan Result) error {
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case r, ok := <-in:
            if !ok {
                return nil
            }
            if err := persist(r); err != nil {
                return err
            }
        }
    }
}

The pipeline:

• Source closes out when the input slice is exhausted or context cancels.
• Transform has N workers; the WaitGroup tracks them; close-after-wait converts to a closed channel.
• Sink ranges via select to allow cancellation mid-receive.

Backpressure: if Sink is slow, Transform's sends block; if Transform's sends block, the workers stall; if workers stall, Source's sends block. The pressure cascades upstream automatically.

Cancellation: a ctx.Done() causes every stage to exit. Closes cascade downstream; cancellation cascades immediately.

No len. No polling. No sleeps.

Diff-Driven Refactor of a Legacy System¶

A real-world scenario: a 600-line legacy worker file with seven len(ch) checks, three time.Sleep loops, and zero WaitGroups. The migration plan.

Step 1: characterise the current state¶

Read every function. List every channel, every goroutine, every wait. Tabulate ownership:

ch jobs:    created in NewService
            sent by Submit, internal feedJobs goroutine
            received by 8 workers
            closed by ??? (never, currently)

ch results: created in NewService
            sent by 8 workers
            received by aggregator goroutine
            closed by ??? (never)

ch stop:    created in NewService
            received by feedJobs, aggregator
            closed by Shutdown

The "closed by ???" rows are the bugs.

Step 2: pick the new architecture¶

Decision: use errgroup. Hide channels behind methods. Make Close the single shutdown entry point.

Step 3: rewrite, one routine at a time¶

Start with feedJobs: refactor it to range over a pending channel and exit when closed. Then worker: range over jobs, return on close. Then aggregator: range over results. Then Close: close pending, then g.Wait(), then close results.

Step 4: add tests¶

For each refactored function, a test under -race -count=200. For the whole service, a stress test that submits 10000 jobs, shuts down, and asserts that exactly 10000 results emerged.

Step 5: replace in production¶

Behind a feature flag. Roll out to 1% of pods. Watch error rates and latency tails. Roll forward at 1% per hour for a day.

This is migration work. There is no shortcut. The senior engineer plans it carefully and executes incrementally.

Production Failure Modes That Trace Back to Polling¶

A taxonomy of incidents I have seen.

Failure 1: silent data loss¶

The polling loop exits while a worker is mid-process. The consumer "moves on" to the next batch, dropping the unprocessed item. Symptom: data missing in downstream system, with no error logged.

Failure 2: phantom duplicates¶

Two consumers both see len > 0, both try to receive, one succeeds, one retries. If the retry hits a different but identical item, the system processes it twice.

Failure 3: shutdown stuck¶

The shutdown waits for len == 0 while a slow consumer continues to consume. Items move from the channel into the consumer's local state but the channel buffer goes up and down. The polling sees occasional zero and exits, even though work is still in flight.

Failure 4: CPU saturation under low load¶

The polling loop has no sleep. When the system is idle (no items to process), the polling goroutine spins at 100% CPU. A four-pod deployment has four idle cores pegged forever.

Failure 5: high latency tails¶

Even with a sleep, the polling adds latency at every transition. A request that should complete in 50 ms instead waits up to 10 ms for the next poll cycle. P99 latency exceeds SLA.

Failure 6: cascading timeouts¶

The polling loop has a deadline. Under load, the deadline expires before the queue drains. The system returns "timed out" responses even though it is technically functioning. Cascades upstream.

Failure 7: lock contention¶

The polling loop reads len(ch), which acquires the channel's internal mutex. Under high contention, the mutex becomes a hotspot, slowing every channel operation in the program.

Every one of these has been observed in production code. Every one is fixable by removing the polling.

Common Variants of the Anti-Pattern¶

The pattern hides in many shapes. A senior engineer recognises them all.

Variant 1: classic len(ch)¶

for len(ch) > 0 { time.Sleep(...) }

The textbook shape.

Variant 2: cap(ch) polling¶

for len(ch) == cap(ch) { time.Sleep(...) } // wait for room

Same race, different observation.

Variant 3: select/default busy-loop¶

for {
    select {
    case x := <-ch:
        handle(x)
    default:
        if condition { return }
    }
}

Looks like a select-based loop but pegs CPU because of default.

Variant 4: counter polling¶

for atomic.LoadInt32(&pending) > 0 { time.Sleep(...) }

Same race over an atomic instead of a channel length.

Variant 5: timestamp polling¶

for time.Since(last) < 100 * time.Millisecond { time.Sleep(...) }

"Wait until the last event was more than 100 ms ago." Racy because last updates concurrently.

Variant 6: state-flag polling¶

for !service.IsDone() { time.Sleep(...) }

Hidden behind a method, same shape.

Variant 7: composite polling¶

for len(in) > 0 || len(out) > 0 || atomic.LoadInt32(&pending) > 0 {
    time.Sleep(...)
}

Polling multiple observables. Even more races.

Recognise the family resemblance. All seven are the same anti-pattern. All seven have the same fix: replace with event-driven synchronisation.

Composable Helpers You Should Have¶

A senior team maintains an internal/syncx (or similar) package with helpers that compose. Some of mine:

Wait with context¶

package syncx

import (
    "context"
    "sync"
)

// Wait returns nil when wg becomes zero, or ctx.Err() if ctx cancels first.
func Wait(ctx context.Context, wg *sync.WaitGroup) error {
    done := make(chan struct{})
    go func() {
        wg.Wait()
        close(done)
    }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

OneShot¶

type OneShot struct {
    once sync.Once
    done chan struct{}
}

func NewOneShot() *OneShot {
    return &OneShot{done: make(chan struct{})}
}

func (o *OneShot) Fire() {
    o.once.Do(func() { close(o.done) })
}

func (o *OneShot) Done() <-chan struct{} { return o.done }

A safe single-close primitive.

Latch¶

type Latch struct {
    n    atomic.Int32
    done chan struct{}
}

func NewLatch(n int) *Latch {
    return &Latch{
        n:    atomic.Int32{},
        done: make(chan struct{}),
    }
}

func (l *Latch) CountDown() {
    if l.n.Add(-1) == 0 {
        close(l.done)
    }
}

func (l *Latch) Wait() <-chan struct{} { return l.done }

A countdown latch — N callers must CountDown before Wait unblocks. Different from WaitGroup in that the count is fixed at construction.

Bounded¶

type Bounded struct {
    sem chan struct{}
}

func NewBounded(n int) *Bounded {
    return &Bounded{sem: make(chan struct{}, n)}
}

func (b *Bounded) Acquire(ctx context.Context) error {
    select {
    case b.sem <- struct{}{}:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

func (b *Bounded) Release() { <-b.sem }

A context-aware semaphore.

These four primitives, plus the standard library, cover every synchronisation need I have encountered in production. None of them use polling.

Testing Strategies for Race-Free Code¶

Tests are the proof that the refactor worked. Senior tests do four things.

1. Run with -race¶

Always. In CI, in pre-commit, in your editor's test runner. The race detector is cheap insurance.

2. Run with -count=N¶

go test -race -count=200 -timeout=60s ./...

Concurrency bugs are probabilistic. A single run is not evidence. Two hundred runs without a failure is moderate evidence; 10000 is strong evidence.

3. Use goleak¶

import "go.uber.org/goleak"

func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}

Asserts no goroutine outlives the test. Catches the most common refactor mistake: forgetting to wait.

4. Inject delays at synchronisation points¶

For "flaky-test-finder" duties, use runtime.Gosched() between every line of the system under test in a copy of the function. Or use a chaos library to inject random delays. If the test still passes, the synchronisation is sound; if it starts failing, you found a race.

5. Property-based tests¶

Generate random inputs and assert invariants:

func TestPoolProcessesAllInputs(t *testing.T) {
    f := func(items []int) bool {
        result, err := process(context.Background(), items)
        if err != nil {
            return false
        }
        return len(result) == len(items)
    }
    if err := quick.Check(f, &quick.Config{MaxCount: 1000}); err != nil {
        t.Fatal(err)
    }
}

If the original len-polling code had dropped occasionally, quick.Check would find it within a few iterations.

Code Review Discipline¶

In review, the senior engineer applies a structured pass for concurrency.

Pass 1: identify every goroutine¶

Grep for go (with a trailing space) and errgroup.Group.Go. For each, ask:

• Who waits for it?
• How does it exit?

Pass 2: identify every channel¶

Grep for make(chan. For each, ask:

• Who sends?
• Who receives?
• Who closes?
• Is the direction declared in function parameters?

Pass 3: identify every wait¶

Grep for Wait, <-done, <-ctx.Done(), time.Sleep. For each, ask:

• What event triggers the wake-up?
• Is the wait deterministic, or probabilistic?

Pass 4: identify every len(ch)¶

Grep for len(. For each that operates on a channel, ask:

• Is this for synchronisation or for diagnostics?
• If synchronisation, refactor.

Pass 5: identify every cap(ch)¶

Same as above. cap is almost never useful in production code outside of constructor parameters.

If all five passes are clean, the code is concurrency-safe at the structural level. (Logic bugs may remain; the race-detector catches some; tests catch the rest.)

Migration Across a Codebase¶

When a codebase has hundreds of instances of the anti-pattern, you cannot refactor all at once. The phased plan.

Phase 1: add lints¶

Add a CI check that fails on new len(ch) polling. Use a custom go/analysis pass or a semgrep rule. New code stops introducing the pattern.

Phase 2: triage existing instances¶

For each existing instance, classify:

• Hot path: refactor first; production impact.
• Cold path: lower priority but should be fixed.
• Diagnostics: tag with a comment to exempt from lint.

Phase 3: refactor in batches¶

One package per week. Each refactor:

• Replace polling with appropriate primitive.
• Add tests under -race -count=200.
• Roll out behind a feature flag if the package is hot.

Phase 4: enforce¶

After all instances are fixed, raise the lint to error. Block new instances at PR review.