Wait-for-Empty-Channel — Professional Level¶

Table of Contents¶

1. Introduction
2. Production Realities
3. Graceful Shutdown: A Complete Treatment
4. Drain Protocols for Live Traffic
5. Queue Draining in Workers
6. Observability for In-Flight Work
7. Tracing Concurrent Lifecycles
8. Coordinator Patterns
9. Cluster-Wide Coordination
10. Kubernetes Pod Lifecycle
11. PreStop and Shutdown Hooks
12. Connection Draining
13. Load Balancer Coordination
14. Message Queue Drain Semantics
15. Idempotency and Replay
16. Migration Without Downtime
17. Feature Flags for Concurrency Refactors
18. Production Telemetry: Key Metrics
19. Alerting on the Anti-Pattern
20. Capacity Planning
21. Cost of Polling in Cloud Bills
22. Latency Tails: How Polling Hurts P99
23. CPU Profile Reading
24. Trace Reading at Scale
25. Continuous Profiling
26. Postmortem Patterns
27. Incident Playbook
28. Audit Process for Legacy Codebases
29. Migration Playbook
30. Rollout Strategy
31. Rollback Strategy
32. Team Training Plan
33. Self-Assessment
34. Summary

Introduction¶

At the professional level the wait-for-empty-channel anti-pattern is no longer a code-correctness problem; it is a systems problem. The code is wrong, yes — but the consequences are felt operationally: shutdowns hang, deployments drop traffic, alerts fire from elevated CPU, latency tails grow, and customer-facing failures occur. The professional engineer's job is to manage the consequences and migrate the system off the pattern without disruption.

This file walks the operational dimensions. It assumes you have read junior, middle, and senior. The synchronisation theory is settled; here we cover:

• How to shut down a service gracefully under live traffic.
• How to observe in-flight work without polling.
• How to coordinate shutdown across a fleet.
• How to migrate a large codebase off the pattern without downtime.
• How to alert on the pattern's symptoms before customers notice.
• How to read profiles, traces, and dashboards to find the pattern in production.

The patterns here apply to any concurrent service. The Go-specific bits are about the standard library and ecosystem, but most of the discussion translates to any language with green threads or async runtimes.

Production Realities¶

Operations is different from development. Three realities to internalise.

Reality 1: traffic does not pause for you¶

You do not get to "stop the world" while you shut down. Pods are killed; new requests still arrive at the load balancer; existing requests are mid-flight. Your shutdown logic runs concurrently with all of that.

Reality 2: shutdowns have hard deadlines¶

Kubernetes gives you terminationGracePeriodSeconds (default 30s, often shorter in production). After that, SIGKILL. Whatever in-flight work you had is gone unless persisted. The polling pattern often exceeds the deadline because it never confirms completion, only that the queue is briefly empty.

Reality 3: errors compound¶

A polling loop's CPU usage is one cost; the latency it adds is another; the requests it drops during shutdown are a third. Each is a separate metric in your dashboards. Each can trigger an alert. Each contributes to a customer-visible outage. The compound effect is more than the sum.

These realities frame everything in this file.

Graceful Shutdown: A Complete Treatment¶

The reference implementation of graceful shutdown in a Go HTTP service.

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "syscall"
    "time"
)

func main() {
    ctx, cancel := signal.NotifyContext(
        context.Background(),
        syscall.SIGINT, syscall.SIGTERM,
    )
    defer cancel()

    srv := &http.Server{
        Addr:    ":8080",
        Handler: newHandler(),
    }

    serverErr := make(chan error, 1)
    go func() {
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            serverErr <- err
        }
    }()

    select {
    case <-ctx.Done():
        log.Println("signal received; shutting down")
    case err := <-serverErr:
        log.Printf("server error: %v", err)
    }

    shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancelShutdown()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("graceful shutdown failed: %v", err)
        return
    }
    log.Println("shutdown complete")
}

What this does right:

• Uses signal.NotifyContext to convert OS signals into a cancellable context.
• The server runs in a goroutine; the main goroutine waits on either the signal or a server error.
• On shutdown, http.Server.Shutdown is called with a deadline-bounded context.
• Shutdown blocks until all active requests complete or the deadline expires. It uses internal WaitGroup-like coordination, not polling.
• After Shutdown returns, the program exits.

No len. No polling. The 25-second deadline gives Kubernetes' default 30-second window enough room.

Add background workers¶

A real service has background workers too. They must shut down with the same discipline.

type App struct {
    srv    *http.Server
    pool   *worker.Pool
    cancel context.CancelFunc
}

func (a *App) Run(ctx context.Context) error {
    ctx, a.cancel = context.WithCancel(ctx)
    defer a.cancel()

    g, ctx := errgroup.WithContext(ctx)

    g.Go(func() error {
        return a.runServer(ctx)
    })

    g.Go(func() error {
        return a.pool.Run(ctx)
    })

    return g.Wait()
}

func (a *App) Shutdown(ctx context.Context) error {
    g, gctx := errgroup.WithContext(ctx)
    g.Go(func() error { return a.srv.Shutdown(gctx) })
    g.Go(func() error { return a.pool.Close() })
    return g.Wait()
}

Both components have their own Shutdown. They run concurrently. The errgroup waits for both.

The shutdown order¶

For complex services, shutdown order matters. The general pattern:

1. Stop accepting new traffic at the edge (server.Shutdown stops accepting new connections).
2. Drain in-flight work (server.Shutdown waits for active requests; worker pools wait for active jobs).
3. Close downstream resources (database connections, file handles).
4. Final flush of metrics, logs.
5. Exit.

Reversing the order (closing the DB before draining requests) causes the in-flight requests to fail. Get the order right.

Drain Protocols for Live Traffic¶

A web service receives requests continuously. Shutdown must:

• Stop accepting new requests promptly (within ~100 ms).
• Let in-flight requests finish (up to a bounded time).
• Reject net-new requests cleanly (or rely on the load balancer to redirect them).

Pattern: readiness-first shutdown¶

type Server struct {
    ready atomic.Bool
}

func (s *Server) ReadyHandler(w http.ResponseWriter, r *http.Request) {
    if s.ready.Load() {
        w.WriteHeader(200)
        return
    }
    w.WriteHeader(503)
}

func (s *Server) Shutdown(ctx context.Context) error {
    s.ready.Store(false) // fail readiness probe
    time.Sleep(s.gracePeriod) // give LB time to drain
    return s.srv.Shutdown(ctx)
}

The readiness probe goes red first. The load balancer notices and stops sending new traffic. After a grace period (typically 5-15s, matching the LB poll interval), the HTTP server drains in-flight requests.

This pattern requires careful coordination with the load balancer's health check frequency. Document it.

Pattern: connection draining¶

If the LB does HTTP/2 or keepalive connections, in-flight connections must drain. The HTTP/2 GOAWAY frame tells clients "use a different connection." http.Server.Shutdown sends GOAWAY automatically.

For raw TCP services, you implement this yourself: close the listener (so no new accepts), set a deadline on existing connections, wait.

listener.Close()
var wg sync.WaitGroup
for _, conn := range activeConns {
    conn := conn
    wg.Add(1)
    go func() {
        defer wg.Done()
        conn.SetDeadline(time.Now().Add(30 * time.Second))
        // existing reads/writes will return with timeout
    }()
}
wg.Wait()

No polling. The deadline ensures bounded wait; the WaitGroup confirms completion.

Queue Draining in Workers¶

A worker that consumes from a queue (Kafka, SQS, in-memory) has its own drain semantics.

Drain semantics by queue type¶

Kafka:

• Stop pulling new messages.
• Finish processing the current batch.
• Commit offsets.
• Disconnect.

SQS:

• Stop pulling new messages.
• Finish processing in-flight messages.
• Delete completed messages (so they are not retried).
• Disconnect.

In-memory channel:

• Stop accepting new submissions.
• Drain the channel.
• Wait for workers to finish processing.

In each case, the drain has two phases: stop accepting, then wait for completion. The polling pattern attempts to short-circuit by checking "is the queue empty?" — which conflates the two phases incorrectly.

Reference implementation¶

type Worker struct {
    queue   Queue
    g       *errgroup.Group
    workers int
}

func (w *Worker) Start(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)
    w.g = g
    for i := 0; i < w.workers; i++ {
        g.Go(func() error {
            for {
                select {
                case <-ctx.Done():
                    return nil
                default:
                }
                msg, err := w.queue.Receive(ctx, 30*time.Second)
                if err != nil {
                    if errors.Is(err, context.Canceled) {
                        return nil
                    }
                    return err
                }
                if msg == nil {
                    continue // poll returned empty; loop and check ctx
                }
                if err := w.process(ctx, msg); err != nil {
                    log.Println("process error:", err)
                    continue // do not return; one bad message should not kill the worker
                }
                if err := msg.Ack(ctx); err != nil {
                    return err
                }
            }
        })
    }
    return nil
}

func (w *Worker) Shutdown(ctx context.Context) error {
    // The errgroup's context cancels when ctx cancels (assuming Start was called with ctx).
    // Workers observe and return.
    return w.g.Wait()
}

The worker loop respects ctx.Done(), processes one message at a time, acks on success. No len, no polling, no sleeps.

The Receive(ctx, 30*time.Second) is a long-poll. If no message arrives within 30 seconds, it returns nil and the loop iterates. The 30-second poll is the queue's API; the worker's ctx.Done() check ensures shutdown happens promptly.

Observability for In-Flight Work¶

You cannot manage what you cannot measure. Observability for in-flight work has three pillars: metrics, logs, traces.

Metric: in-flight count¶

The number of operations currently in flight. Easy to track with atomics or a metric registry:

var inFlight atomic.Int64

func (s *Service) Handle(r *Request) {
    inFlight.Add(1)
    defer inFlight.Add(-1)
    // ...
}

// Periodically:
metrics.Gauge("inflight").Set(float64(inFlight.Load()))

This is a legitimate use of state inspection. The result is informational, not a synchronisation primitive.

Metric: queue depth¶

The current number of items in a queue. Use len(ch) for this — it is legitimate.

metrics.Gauge("queue.depth").Set(float64(len(jobs)))

Metric: processing rate¶

Counter of items processed per second:

metrics.Counter("processed").Inc()

Rate calculation happens in the metrics backend (Prometheus rate(), Datadog rate, etc.).

Metric: shutdown duration¶

How long shutdown takes. Useful for confirming drain works.

start := time.Now()
defer func() {
    metrics.Histogram("shutdown.duration").Observe(time.Since(start).Seconds())
}()

Log: shutdown events¶

Logs document the sequence:

log.Println("shutdown: received signal, stopping new connections")
log.Println("shutdown: draining; in-flight =", inFlight.Load())
log.Println("shutdown: in-flight =", inFlight.Load(), "elapsed =", time.Since(start))
log.Println("shutdown: complete")

If shutdown hangs, the logs show where.

Trace: per-request lifecycle¶

OpenTelemetry traces show every operation, its duration, and its relationship to others. A shutdown anomaly shows up as a span with unusually long duration or a sub-span that never completes.

Tracing Concurrent Lifecycles¶

For complex services, distributed tracing is the most valuable observability tool.

Span hierarchy¶

serve_request
  ├── validate
  ├── enqueue_job
  │     └── (later, in worker)
  │          process_job
  │            ├── db_query
  │            ├── external_api
  │            └── persist_result
  └── respond

The trace links the request to the eventual job processing, even across async boundaries. If the polling anti-pattern is present, you see weird shapes: a span called "wait_for_drain" that spans seconds while a worker span next to it finishes in milliseconds — clear evidence of polling versus event.

Propagating context for traces¶

ctx, span := tracer.Start(ctx, "serve_request")
defer span.End()

// pass ctx into the worker queue
job := Job{
    Payload: payload,
    Span:    span.SpanContext(),
}
queue <- job

// in worker:
ctx = trace.ContextWithSpanContext(workerCtx, job.Span)
ctx, span := tracer.Start(ctx, "process_job")
defer span.End()

The job carries the trace context. The worker resumes the trace. The request span and the worker span are linked even though they are in different goroutines.

What to look for in traces¶

• Spans much longer than the work they describe — usually polling.
• Sub-spans that never finish — usually a leaked goroutine.
• Spans whose start times exceed their parent's end — usually misordered cleanup.
• Spans that take exactly N ms multiple of polling interval — usually a polling loop.

A senior SRE pattern: in trace analysis, build histograms of span duration modulo common polling intervals. Spikes at exact multiples of 10 ms or 100 ms indicate polling.

Coordinator Patterns¶

A coordinator is a goroutine (or process) that orchestrates other goroutines (or processes). Its job is to make the lifecycle observable.

Single-process coordinator¶

type Coordinator struct {
    children []Component
    ready    chan struct{}
    done     chan struct{}
}

func (c *Coordinator) Run(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)
    for _, child := range c.children {
        child := child
        g.Go(func() error {
            return child.Run(ctx)
        })
    }
    close(c.ready)
    err := g.Wait()
    close(c.done)
    return err
}

The coordinator exposes ready (all children started) and done (all children exited). Operators observe both.

Distributed coordinator¶

For multi-process systems, a distributed coordinator (etcd, ZooKeeper, Consul) replaces the in-process goroutine. The same patterns apply:

• A ready indicator (leader election succeeded; all replicas reporting healthy).
• A done indicator (all replicas drained and shut down).

The polling anti-pattern can appear at any layer. In a distributed context, "poll the cluster's queue depth" is the same bad shape. Use events: each node publishes its state to the coordinator; the coordinator's view updates as events arrive.

Cluster-Wide Coordination¶

When deploying a new version, the cluster goes through a rolling update. Old pods drain; new pods come up. The wait-for-empty-channel anti-pattern can hide at this layer.

The problem¶

A pod is killed (SIGTERM); it has 30 seconds to shut down. Inside the pod, the worker polls a Kafka topic. The polling cycle wakes every minute. The shutdown signal is received; the pod waits for the worker to finish. The worker's poll cycle has not completed yet, so it has not seen the signal. The 30 seconds elapses; SIGKILL.

The fix¶

The worker's poll must accept a context.

msg, err := reader.ReadMessage(ctx)

With Kafka's kafka-go library or confluent-kafka-go, the read accepts a context. On ctx.Done(), the read returns promptly. The worker observes the shutdown signal and exits.

Verification¶

Check every long-running blocking call in your service. Each must accept a context. If any does not, the shutdown deadline will be exceeded.

# Quick audit:
grep -nR ".Read(\|.Recv(\|.Get(\|.Wait(" --include="*.go" | grep -v ctx

Suspect anything that does not have ctx as a parameter.

Kubernetes Pod Lifecycle¶

A Kubernetes pod's lifecycle has phases. Each phase has implications for the wait-for-empty-channel discussion.

Phase 1: Pending¶

The pod is scheduled but not yet running. No traffic. No work.

Phase 2: Running¶

The container is up. Readiness probes determine if traffic flows. Liveness probes detect hangs.

Phase 3: Terminating¶

SIGTERM is sent. The pod has terminationGracePeriodSeconds to exit cleanly. After that, SIGKILL.

What happens in Terminating¶

1. Kubernetes removes the pod from the Service's endpoint list (approximately — this is eventually consistent).
2. The load balancer's next poll notices the change and stops sending traffic to the pod.
3. The pod receives SIGTERM.
4. The pod's process handles the signal and begins shutdown.

The race: step 1-2 takes time (typically 5-15s). During that window, the pod is receiving SIGTERM but also still receiving new traffic. The shutdown logic must accept new connections during this window or the load balancer will see failures.

The canonical fix¶

// PreStop hook in the pod spec:
preStop:
  exec:
    command:
      - sleep
      - "15"

// Or in code:
func (s *Server) handleSIGTERM(ctx context.Context) {
    log.Println("SIGTERM received; failing readiness for 15s")
    s.ready.Store(false)
    time.Sleep(15 * time.Second)
    log.Println("draining")
    s.srv.Shutdown(ctx)
}

The pod fails its readiness probe immediately, then sleeps for 15s. During that time, the load balancer notices and stops sending new traffic. After 15s, the pod calls Shutdown and drains existing connections.

No polling of len(ch). The mechanism is: readiness probe → load balancer → traffic cessation. The pod waits for this to propagate via a time-based delay, which is acceptable because the LB's polling interval is known.

PreStop and Shutdown Hooks¶

preStop hooks run before SIGTERM. They give you a way to coordinate the LB drain without coding it inside the application.

spec:
  containers:
  - name: app
    lifecycle:
      preStop:
        exec:
          command: ["sh", "-c", "curl -X POST localhost:8080/admin/drain && sleep 15"]

The hook calls the app's /admin/drain endpoint, which flips a flag to fail readiness. Then it sleeps for 15 seconds, during which the LB notices. Then the hook exits, Kubernetes sends SIGTERM, and the app's shutdown logic runs.

This pattern separates concerns: the hook handles "tell the LB to stop"; the app's signal handler handles "drain and exit." Both are testable in isolation.

Connection Draining¶

For services that hold persistent connections (gRPC streams, WebSockets, raw TCP), draining the connections is an explicit step.

gRPC¶

grpc.Server.GracefulStop() blocks new connections and waits for existing RPCs to finish.

srv := grpc.NewServer()
go func() {
    if err := srv.Serve(listener); err != nil {
        log.Println(err)
    }
}()

<-ctx.Done()
done := make(chan struct{})
go func() {
    srv.GracefulStop()
    close(done)
}()
select {
case <-done:
case <-time.After(25 * time.Second):
    srv.Stop() // force
}

GracefulStop is the structured wait. Stop is the force.

WebSocket¶

For WebSocket connections, you cannot interrupt them mid-stream gracefully. The pattern is:

1. Mark new connections as rejected (readiness fails).
2. Send a "close" frame to each existing connection.
3. Wait for the client to acknowledge.
4. Close.

for _, conn := range activeConns {
    conn := conn
    go func() {
        conn.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(1001, "shutting down"))
        conn.Close()
    }()
}

Track active connections in a map under a mutex. Iterate during shutdown. Use a WaitGroup to wait for goroutines that handle each connection.

Raw TCP¶

Set a deadline on every connection at shutdown time:

for _, conn := range activeConns {
    conn.SetDeadline(time.Now().Add(30 * time.Second))
}

Existing reads/writes return with timeout. Handler goroutines clean up. Wait via WaitGroup.

Load Balancer Coordination¶

The load balancer's polling cycle is a known quantity. Your shutdown sequence must accommodate it.

AWS ALB¶

Default deregistration delay: 300s. Configurable. After SIGTERM, the ALB stops sending traffic to your pod within this window.

GCP Internal Load Balancer¶

The drain period is configurable per backend service. Default 0s (immediate drain). Set to match your shutdown time.

Kubernetes Service¶

The Endpoints object is updated when a pod terminates. The kube-proxy's IPVS rules are updated subsequently. There is no fixed delay; it depends on the cluster's reaction time, typically 1-3 seconds.

Implication for shutdown¶

Your pod's preStop sleep should be at least as long as the LB's polling interval plus a margin. For ALB with default settings, that means 30-60s. For internal services, 10-15s is usually enough.

What the polling anti-pattern does here¶

If your shutdown logic polls "is the queue empty?", it doesn't know about the LB's drain. It happily reports "queue empty" while the LB is still routing requests to this pod. The pod accepts a request, then exits, dropping the request.

The fix is to integrate with the LB drain via readiness or preStop hooks. The synchronisation is between the LB's polling cycle and your pod's shutdown — that synchronisation is time-based and acceptable. The synchronisation between your pod's internal goroutines must be event-based.

Message Queue Drain Semantics¶

Each message queue has its own drain semantics. Understanding them is operational, not theoretical.

Kafka¶

• Messages are not "consumed" until offsets are committed.
• Stopping a consumer without committing causes the messages to be redelivered to other consumers.
• Stopping a consumer after committing means the messages are durably acknowledged.

For drain, the consumer should:

1. Finish processing all in-flight messages.
2. Commit offsets.
3. Disconnect cleanly (close the consumer; it sends LeaveGroup to the broker).

Failure modes:

• If the consumer crashes before commit, messages are redelivered. Idempotency is essential.
• If the consumer commits but crashes before disconnect, the broker waits for the session timeout before rebalancing. New consumers take over.

SQS¶

• Messages have a visibility timeout. While processing, the consumer can extend the timeout.
• Failure to delete the message before timeout causes redelivery.

For drain:

1. Stop receiving.
2. Finish processing in-flight messages.
3. Delete completed messages.

RabbitMQ¶

• Messages can be auto-ack (acked on delivery) or manual-ack (acked after processing).
• Manual-ack is the norm in production.

For drain:

1. Cancel the consumer (channel.basic_cancel).
2. Finish processing in-flight messages.
3. Ack completed messages.
4. Close the channel.

Common pattern¶

In every case, drain has two phases: stop receiving, then wait for in-flight completion. The polling anti-pattern fails because it does not separate the phases.

Idempotency and Replay¶

A consequence of drain failures: messages may be replayed. Idempotency is the protection.

Idempotent processors¶

A processor is idempotent if processing the same message twice produces the same outcome as processing it once.

func process(msg Message) error {
    if seen(msg.ID) {
        return nil // already processed
    }
    if err := do(msg); err != nil {
        return err
    }
    return markSeen(msg.ID)
}

The seen/markSeen pair forms an idempotency check. Implementations vary: a Redis set, a Postgres row, an in-memory map.

When idempotency matters¶

For at-least-once delivery systems (Kafka, SQS), every consumer must be idempotent. Failures during drain will cause replay.

When polling fails idempotency¶

If a polling consumer fails to commit before shutdown, the message is replayed. If the processor is not idempotent, the side effect happens twice. The polling pattern itself does not break idempotency, but it makes shutdown less reliable, which exposes idempotency gaps.

Migration Without Downtime¶

Refactoring a polling loop in production code is risk. The risk must be managed.

Strategy: blue-green¶

Run both versions side by side. Route 1% of traffic to the new version. Compare metrics. Increase percentage as confidence grows.

Strategy: feature flag¶

Wrap the refactor behind a flag:

if featureflag.Enabled("worker.event-driven") {
    return s.runEventDriven(ctx)
}
return s.runPolling(ctx)

The flag can be toggled in real time. Rollback is a config change, not a deploy.

Strategy: shadow traffic¶

Run the new code in parallel with the old. The new code's output is discarded. Compare what it would have produced to what the old code did.

if featureflag.Enabled("worker.shadow") {
    go func() {
        result := s.runEventDriven(ctx) // hypothetical run
        compare(result, oldResult)
    }()
}
return s.runPolling(ctx)

After a week of shadow traffic with no discrepancies, promote.

Strategy: incremental refactor¶

Refactor one function at a time. Each refactor is small, reviewable, and reversible. Avoid big-bang rewrites; they fail.

Feature Flags for Concurrency Refactors¶

Feature flags are the canonical tool for safe rollouts. For concurrency refactors specifically:

Flag granularity¶

• Service-wide flag: simple, all-or-nothing.
• Per-route flag: refactor one endpoint at a time.
• Per-customer flag: refactor for one customer at a time, useful for canary.
• Percentage rollout: gradually shift traffic.

Flag observability¶

Each flag emits metrics:

metrics.Counter("flag.worker_event_driven.enabled").Inc()
metrics.Counter("flag.worker_event_driven.disabled").Inc()

Dashboards show the rollout percentage and the metric divergence between the two code paths.

Flag lifecycle¶

• Phase 1: flag created, default off, code in production.
• Phase 2: flag enabled for canary (1%); observe.
• Phase 3: gradual rollout to 100%.
• Phase 4: flag default on; code path removed for default-off path.
• Phase 5: flag removed; code is unconditional.

The lifecycle takes weeks. Plan accordingly.

Production Telemetry: Key Metrics¶

What to monitor when refactoring off the anti-pattern.

Metric 1: CPU usage¶

The polling loop's signature is high baseline CPU. After refactor, baseline drops.

worker.cpu_usage_percent: was 18%, after refactor: 0.5%

Metric 2: shutdown duration¶

Before refactor, shutdown often hangs or hits the deadline. After, it completes in milliseconds.

shutdown.duration_seconds.p99: was 28s, after refactor: 0.2s

Metric 3: dropped messages on shutdown¶

Before refactor, drain failures cause drops. After, zero.

shutdown.dropped_messages: was 50/day, after refactor: 0

Metric 4: P99 request latency¶

Polling loops add latency. After refactor, P99 improves.

request.duration_seconds.p99: was 250ms, after refactor: 140ms

Metric 5: goroutine count¶

Polling goroutines may leak. After refactor, count drops.

runtime.goroutines: was 5000, after refactor: 1200

Track each metric. Alert on regressions.

Alerting on the Anti-Pattern¶

You can build alerts that find the anti-pattern in production.

Alert 1: high baseline CPU on a worker¶

If a worker pod's CPU is consistently above 5% with no work, polling is likely. Alert.

Alert 2: shutdown duration near the deadline¶

If shutdown.duration is close to terminationGracePeriodSeconds, your shutdown is unreliable. Alert.

Alert 3: goroutine count growth¶

If runtime.goroutines grows without bound, goroutine leak is happening. Alert.

Alert 4: queue depth oscillation¶

If queue depth alternates between zero and N (a polling consumer not keeping up), alert.

Alert 5: response latency tied to polling interval¶

If P99 latency is a multiple of your polling interval (e.g., latency 100 ms when poll is 100 ms), polling is in the path. Alert.

Each alert has a runbook. The runbook says: "Search for polling. Likely in . Replace with ."

Capacity Planning¶

The polling pattern affects capacity:

• Each polling goroutine consumes ~0.5-1% of a core when sleeping with time.Sleep(time.Millisecond).
• A service with 100 polling loops uses 50-100% of a core just for polling.
• That core is unavailable for real work.

Calculation¶

polling_cpu = N_polling_loops * 0.005 * cores_per_pod
unused_cpu = total_cores - polling_cpu
effective_capacity = unused_cpu / cpu_per_request

A pod with 4 cores and 100 polling loops effectively has 3.5 cores for work. Capacity is 87.5% of nameplate.

After refactor¶

polling_cpu = 0
unused_cpu = total_cores
effective_capacity = 4 / cpu_per_request

Capacity returns to 100%. For a fleet of 1000 pods, that is 125 pods' worth of effective capacity reclaimed.

Cost in cloud bills¶

At AWS m5.xlarge ($0.192/hour), that is $24/hour saved, or $17,520/year. At GCP n2-standard-4 ($0.194/hour), similar. The migration's cost (engineering time) typically pays back within 2-6 months.

Cost of Polling in Cloud Bills¶

A concrete example. A service with 200 pods, each running 50 polling loops at 1ms intervals.

Per pod: - Polling CPU: 50 * 0.7% = 35% of a core. - Reserved for polling: 0.35 cores * 4 cores/pod = 1.4 cores wasted (incorrect: 0.35 of a core total, not per-core).

Across the fleet: - 200 pods * 0.35 cores wasted = 70 cores wasted. - At m5.xlarge (4 cores per pod, $0.192/hour), 70 cores = 17.5 m5.xlarge instances. - 17.5 * $0.192 = $3.36/hour. - $3.36 * 24 * 365 = $29,433/year.

The migration to event-driven primitives reclaims this. For a fleet of 1000 pods, the savings scale linearly to $147,000/year.

Numbers vary by workload but the qualitative pattern holds: polling is a measurable line item in the cloud bill.

Latency Tails: How Polling Hurts P99¶

P99 latency is the latency experienced by the slowest 1% of requests. For a polling system:

• P50 latency: dominated by actual work; polling adds little.
• P99 latency: includes worst-case polling delay (the full polling interval).

Example: a service with a 10 ms polling cycle. Request work itself takes 20 ms.

• P50: 20 ms (work + no poll wait).
• P99: 30 ms (work + worst-case poll wait of 10 ms).

After refactor:

• P50: 20 ms (no change).
• P99: 20 ms (no polling delay).

P99 improves by 33%. Customer-perceived latency tails shrink.

Bigger impact at higher polling intervals¶

For a 100 ms polling cycle:

• P50: 20 ms.
• P99: 120 ms (work + worst-case poll wait).

After refactor: 20 ms. P99 improves by 6x.

This is why polling is poison for latency-sensitive workloads. The mean is fine; the tail is catastrophic.

CPU Profile Reading¶

How to find polling in a production CPU profile.

Step 1: capture a profile¶

import _ "net/http/pprof"

go func() { http.ListenAndServe(":6060", nil) }()

curl http://service:6060/debug/pprof/profile?seconds=60 > cpu.out

Step 2: analyse¶

go tool pprof cpu.out
(pprof) top10

Look for:

• time.Sleep high in the profile — a polling loop's sleep.
• runtime.chanlen in the top — len(ch) calls dominating.
• runtime.semasleep with high count but short duration — many small waits, characteristic of polling.

Step 3: find the source¶

(pprof) list polling_function_name

The annotated source shows exactly which lines burn CPU.

Step 4: refactor¶

Identify the polling, replace with proper primitive, redeploy, capture profile again. Compare.

Trace Reading at Scale¶

Distributed tracing at scale (millions of spans per day) requires sampling and analysis tools.

Tail-based sampling¶

Keep traces that are anomalous: long duration, high error rate, weird shapes. Discard the rest.

sampler:
  long_duration: keep all > p99
  error: keep all
  shape_anomaly: keep all with > 50 sub-spans

The polling anti-pattern usually produces a "long duration" trace. Tail sampling catches it.

Trace analysis¶

Tools like Jaeger, Honeycomb, Tempo aggregate traces. Search for traces with spans containing "wait" or "drain" in the name and high duration. Polling is visible.

Anomaly detection¶

ML-based tools (Honeycomb's BubbleUp, Datadog's Watchdog) find anomalous traces automatically. The polling pattern's signature — long waits without corresponding child spans — is the kind of anomaly these tools detect.

Continuous Profiling¶

Continuous profiling captures profiles continuously and stores them for retroactive analysis.

Tools¶

• Pyroscope (open source, language-agnostic).
• Polar Signals (managed).
• Grafana Pyroscope (now bundled).
• Datadog Continuous Profiler (managed).

What you can ask¶

"Show me CPU profile flame graphs from yesterday between 14:00 and 15:00 UTC, filtered to the order-processor service."

The graphs reveal polling in production over time. If a deploy introduces polling, the next day's profile shows it immediately.

Workflow¶

Continuous profiling becomes a development tool: before merging a PR, review the profile in staging. After deploying, compare production profiles.

This catches the anti-pattern as a regression signal, not as an incident signal.

Postmortem Patterns¶

After an incident traced to the anti-pattern, the postmortem covers:

Section: timeline¶

When did the incident start? When was it detected? When was it mitigated?

Section: impact¶

How many customers affected? How much revenue lost? What SLA was breached?

Section: root cause¶

The polling pattern in file:line. Explanation of the race. Why it manifested under load.

Section: detection¶

How was the incident detected? (Metric alert, customer report, reconciliation.) Could it have been detected sooner?

Section: response¶

What actions did the on-call take? Were they effective? How long did each step take?

Section: prevention¶

What changes prevent recurrence?

• Refactor the specific polling instance.
• Add a lint rule to prevent new instances.
• Add a stress test that exercises the race window.
• Add an alert for the symptom (high baseline CPU, shutdown duration near deadline).

Section: action items¶

Concrete, owned, dated.

The postmortem is the institutional memory. A team that writes good postmortems for anti-pattern incidents gradually eliminates the pattern.

Incident Playbook¶

When on-call detects symptoms of the anti-pattern, the playbook.

Symptom: shutdown hanging¶

1. Check the logs for the last "shutdown:" line. Where did it stop?
2. SSH into the pod (or kubectl exec). Run kill -ABRT <pid> to dump goroutine stacks.
3. Identify which goroutine is stuck and where.
4. If polling: roll back to the previous version; file a ticket for refactor.

Symptom: dropped messages¶

1. Check the reconciliation gap. Quantify the loss.
2. Identify which service has the gap.
3. Read the service's code for polling.
4. If polling: refactor (priority).
5. Manually replay the dropped messages from the source of truth.

Symptom: high baseline CPU¶

1. Run a CPU profile during the high-CPU window.
2. Identify polling functions in the top of the profile.
3. Refactor.
4. Deploy.

Symptom: P99 latency spike¶

1. Examine traces for the slow requests.
2. Look for spans with names containing "wait" or "drain".
3. Identify polling in the path.
4. Refactor.

Each playbook has the same shape: identify, isolate, refactor.

Audit Process for Legacy Codebases¶

A structured audit for a large legacy codebase.

Step 1: automated discovery¶

grep -nR "for len(" --include="*.go" .
grep -nR "if len(" --include="*.go" . | grep -v "_test"
grep -nR "select.*default" --include="*.go" -A 5 . | grep -i "sleep"
grep -nR "time\.Sleep" --include="*.go" . | grep -v "_test"

Each match is a candidate. Triage by:

• Hot path or cold path?
• Production or test?
• Easy refactor or complex?

Step 2: prioritise¶

Hot path + production + complex = highest priority.

Step 3: refactor¶

Each instance gets its own PR. Reviewed by at least two engineers. Tested under -race -count=200.

Step 4: deploy¶

Behind feature flag. Canary roll. Observe metrics.

Step 5: cleanup¶

After 100% rollout for two weeks, remove the old code path. Remove the flag.

Step 6: prevent regression¶

Add lint. Document the pattern. Train new engineers.

For a 500K LOC codebase with ~200 instances, this is 6-12 months of work for one engineer. Often parallelised across a team.

Migration Playbook¶

A specific migration: replace for len(ch) > 0 { time.Sleep } with wg.Wait().

Step 1: identify the goroutines that should be tracked¶

Read the producer code. Each go func() { ch <- ... }() is a goroutine that should be tracked.

Step 2: introduce a WaitGroup¶

var wg sync.WaitGroup
for ... {
    wg.Add(1)
    go func() {
        defer wg.Done()
        ch <- ...
    }()
}

Step 3: convert the wait¶

// before:
for len(ch) > 0 { time.Sleep(time.Millisecond) }

// after:
go func() {
    wg.Wait()
    close(ch)
}()
for v := range ch {
    handle(v)
}

Step 4: verify¶

Run tests under -race -count=200. Capture a CPU profile before and after; compare.

Step 5: deploy¶

Behind a flag. Canary. Monitor.

This pattern repeats for every instance. After 100 such migrations, the muscle memory is automatic.

Rollout Strategy¶

For each refactor:

Phase 1: staging deploy¶

Deploy to staging. Run integration tests. Verify metrics behave as expected.

Phase 2: canary¶

Deploy to 1% of pods (or 1% of traffic). Monitor for 24-72 hours. Compare metrics against the rest of the fleet.

Phase 3: gradual rollout¶

10%, 50%, 100%. Pause at each step if metrics degrade.

Phase 4: bake¶

Wait 1-2 weeks at 100%. Watch for delayed issues (e.g., a weekly batch that exercises a code path).

Phase 5: cleanup¶

Remove the old code path and the flag.

Each phase has a clear go/no-go. Document them.

Rollback Strategy¶

If a refactor causes issues:

Fast rollback: flag¶

If the refactor is behind a feature flag, flip the flag. Old code path resumes immediately.

Slow rollback: deploy¶

If the flag is removed and the new code is unconditional, deploy the previous version. This takes minutes.

Forensic rollback: hot patch¶

If neither is possible (rare), patch the offending code path in place. Risky; reserve for emergencies.

Document the rollback path before the rollout¶

Every PR for a concurrency refactor includes a "rollback plan" section in its description. The reviewer verifies the plan works before approving.

Team Training Plan¶

To prevent the anti-pattern in a team, training is essential.

Week 1: theory¶

Read junior.md and middle.md. Quiz on key concepts.

Week 2: practice¶

Refactor 5 instances of the pattern in legacy code. Pair review.

Week 3: review¶

Each engineer reviews 3 PRs with a focus on concurrency. Apply the senior-level checklist.

Week 4: design¶

Each engineer designs a small service from scratch, avoiding the anti-pattern. Group review.

Ongoing¶

Weekly study group: read one Go memory model or concurrency paper per week. Discuss.

Monthly anti-pattern hunt: pick a service; audit it for the anti-pattern; refactor as a team exercise.

Annual revisiting: re-read senior.md and professional.md. New best practices may have emerged.

The pattern does not stay gone unless the team's discipline holds. Training is the discipline.

Self-Assessment¶

Without re-reading:

1. Describe the shutdown sequence for a Kubernetes pod running a Go HTTP server.
2. Explain readiness-first shutdown and why it requires a sleep.
3. Compare drain semantics for Kafka, SQS, and RabbitMQ.
4. List five metrics you would track when refactoring a polling loop.
5. Design an alert that detects the anti-pattern in production CPU usage.
6. Walk through the audit process for a 500K LOC codebase.
7. Describe a feature-flagged rollout for a concurrency refactor.
8. Outline a four-week team training plan.
9. Calculate the annual cloud cost of 200 polling loops at 1 ms intervals.
10. Plan an incident response for "shutdown hanging" symptoms.

If any answer is shaky, re-read the relevant section.

Summary¶

The wait-for-empty-channel anti-pattern in production is more than a code-correctness issue: it is an operational liability. It causes shutdown failures, dropped messages, elevated CPU, latency tails, and incidents. The professional engineer's job is to:

• Build graceful shutdown that integrates with Kubernetes lifecycle, load balancers, and message queues.
• Observe in-flight work through metrics, logs, and traces.
• Coordinate shutdown across fleets and clusters.
• Migrate legacy code through feature flags and canary rollouts.
• Alert on the symptoms before customers notice.
• Train the team to prevent regression.

The migration is large but mechanical. The savings — fewer incidents, lower CPU, reclaimed capacity — are real and measurable. A team that finishes the migration has a more reliable, cheaper, faster service.

The specification file follows, capturing the formal semantics. The interview, tasks, find-bug, and optimize files give you the tools to teach others and to test understanding.

Appendix A: A Production Worker Reference Implementation¶

A complete, runnable reference for a production-grade worker service. Pulls jobs from a queue, processes them with bounded concurrency, observes its own state, and shuts down gracefully.

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "sync/atomic"
    "syscall"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "golang.org/x/sync/errgroup"
    "golang.org/x/sync/semaphore"
)

var (
    metricsProcessed = prometheus.NewCounter(prometheus.CounterOpts{
        Name: "worker_processed_total",
        Help: "Total messages processed.",
    })
    metricsFailed = prometheus.NewCounter(prometheus.CounterOpts{
        Name: "worker_failed_total",
        Help: "Total messages failed.",
    })
    metricsInFlight = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "worker_in_flight",
        Help: "Current in-flight messages.",
    })
    metricsShutdownDuration = prometheus.NewHistogram(prometheus.HistogramOpts{
        Name:    "worker_shutdown_duration_seconds",
        Help:    "Shutdown duration in seconds.",
        Buckets: prometheus.LinearBuckets(0, 1, 30),
    })
)

func init() {
    prometheus.MustRegister(metricsProcessed)
    prometheus.MustRegister(metricsFailed)
    prometheus.MustRegister(metricsInFlight)
    prometheus.MustRegister(metricsShutdownDuration)
}

type Queue interface {
    Receive(ctx context.Context) (Message, error)
    Ack(ctx context.Context, m Message) error
}

type Message struct {
    ID   string
    Body []byte
}

type Worker struct {
    queue       Queue
    handler     func(context.Context, Message) error
    concurrency int
    inFlight    atomic.Int64
    ready       atomic.Bool
}

func New(q Queue, h func(context.Context, Message) error, concurrency int) *Worker {
    return &Worker{
        queue:       q,
        handler:     h,
        concurrency: concurrency,
    }
}

func (w *Worker) Run(ctx context.Context) error {
    w.ready.Store(true)
    defer w.ready.Store(false)

    g, ctx := errgroup.WithContext(ctx)
    sem := semaphore.NewWeighted(int64(w.concurrency))

    g.Go(func() error {
        for {
            if err := sem.Acquire(ctx, 1); err != nil {
                return nil
            }
            m, err := w.queue.Receive(ctx)
            if err != nil {
                sem.Release(1)
                if errors.Is(err, context.Canceled) {
                    return nil
                }
                return err
            }
            g.Go(func() error {
                defer sem.Release(1)
                w.inFlight.Add(1)
                metricsInFlight.Inc()
                defer func() {
                    w.inFlight.Add(-1)
                    metricsInFlight.Dec()
                }()
                if err := w.handler(ctx, m); err != nil {
                    metricsFailed.Inc()
                    return nil // do not propagate; one failure does not kill the worker
                }
                metricsProcessed.Inc()
                return w.queue.Ack(ctx, m)
            })
        }
    })

    return g.Wait()
}

func (w *Worker) Ready() bool {
    return w.ready.Load()
}

func (w *Worker) InFlight() int64 {
    return w.inFlight.Load()
}

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer cancel()

    queue := connectQueue()
    worker := New(queue, processMessage, 8)

    // metrics + admin server
    mux := http.NewServeMux()
    mux.Handle("/metrics", promhttp.Handler())
    mux.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
        if worker.Ready() {
            w.WriteHeader(200)
        } else {
            w.WriteHeader(503)
        }
    })
    mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(200)
    })

    adminSrv := &http.Server{Addr: ":8080", Handler: mux}
    go func() {
        if err := adminSrv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            log.Printf("admin server error: %v", err)
        }
    }()

    g, gctx := errgroup.WithContext(ctx)
    g.Go(func() error {
        return worker.Run(gctx)
    })

    err := g.Wait()
    if err != nil && !errors.Is(err, context.Canceled) {
        log.Printf("worker error: %v", err)
    }

    log.Println("shutting down")
    start := time.Now()
    defer func() {
        metricsShutdownDuration.Observe(time.Since(start).Seconds())
    }()

    shutdownCtx, cancelShutdown := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancelShutdown()

    var wg sync.WaitGroup
    wg.Add(1)
    go func() {
        defer wg.Done()
        if err := adminSrv.Shutdown(shutdownCtx); err != nil {
            log.Printf("admin shutdown error: %v", err)
        }
    }()
    wg.Wait()

    log.Println("shutdown complete in", time.Since(start))
    if err != nil {
        os.Exit(1)
    }
}

This implementation:

• Uses errgroup for structured concurrency.
• Uses semaphore.Weighted for bounded concurrency (respects context).
• Tracks in-flight count for observability.
• Exposes Prometheus metrics.
• Has a readiness probe endpoint for Kubernetes.
• Shuts down gracefully with a bounded deadline.
• Records shutdown duration as a histogram for SLO tracking.

No len(ch). No polling. No sleep loops. Every wait is event-driven.

Appendix B: Anti-Pattern Detection in CI¶

A CI step that detects the anti-pattern.

# .github/workflows/concurrency.yml
name: Concurrency Lint

on: [push, pull_request]

jobs:
  semgrep:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: returntocorp/semgrep-action@v1
        with:
          config: .semgrep/concurrency.yml

  race:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with:
          go-version: '1.23'
      - run: go test -race -count=10 ./...

With a semgrep rule:

# .semgrep/concurrency.yml
rules:
  - id: poll-len-channel
    message: |
      Polling len(ch) is a race condition. Use sync.WaitGroup, errgroup,
      a done channel, or context.Done() instead.
    severity: ERROR
    languages: [go]
    patterns:
      - pattern: for len($CH) > 0 { ... }
      - pattern: for len($CH) != 0 { ... }
      - pattern: for len($CH) == 0 { ... }

  - id: select-default-sleep
    message: |
      select/default with sleep is a polling loop. Replace with a proper
      blocking select.
    severity: WARNING
    languages: [go]
    patterns:
      - pattern: |
          select {
          ...
          default:
            ...
            time.Sleep(...)
            ...
          }

Run on every PR. Block merges on errors. Surface warnings for human review.

Appendix C: Custom Lint Rule for Polling Loops¶

A more sophisticated lint using go/analysis:

package pollinglint

import (
    "go/ast"

    "golang.org/x/tools/go/analysis"
    "golang.org/x/tools/go/analysis/passes/inspect"
    "golang.org/x/tools/go/ast/inspector"
)

var Analyzer = &analysis.Analyzer{
    Name:     "pollinglint",
    Doc:      "detects polling loops on channel length",
    Requires: []*analysis.Analyzer{inspect.Analyzer},
    Run:      run,
}

func run(pass *analysis.Pass) (interface{}, error) {
    insp := pass.ResultOf[inspect.Analyzer].(*inspector.Inspector)
    nodeFilter := []ast.Node{
        (*ast.ForStmt)(nil),
    }
    insp.Preorder(nodeFilter, func(n ast.Node) {
        fs := n.(*ast.ForStmt)
        if fs.Cond == nil {
            return
        }
        if be, ok := fs.Cond.(*ast.BinaryExpr); ok {
            if call, ok := be.X.(*ast.CallExpr); ok {
                if id, ok := call.Fun.(*ast.Ident); ok && id.Name == "len" {
                    if len(call.Args) == 1 {
                        argType := pass.TypesInfo.TypeOf(call.Args[0])
                        if argType == nil {
                            return
                        }
                        // Check if argType is a channel type
                        if _, isChan := argType.Underlying().(*types.Chan); isChan {
                            pass.Reportf(fs.Pos(), "polling len(channel) is a race condition")
                        }
                    }
                }
            }
        }
    })
    return nil, nil
}

This pass walks the AST, finds for statements with a condition that compares len(channel) to something, and reports them. Integrate into your go vet setup or your CI.

Appendix D: Comparing Drain Strategies¶

Side-by-side comparison of drain strategies for an HTTP worker service.

The recommendation: combine readiness-first with WaitGroup drain or library shutdown (http.Server.Shutdown, grpc.Server.GracefulStop). This is the production sweet spot.

Appendix E: Coordinating With External Systems on Shutdown¶

External systems often have their own drain semantics. The coordination patterns.

Database¶

Close connection pools after in-flight queries complete. sql.DB.Close() is idempotent and waits.

Cache¶

Caches usually don't need draining; clearing them on shutdown is optional. If you do clear:

cache.Flush(ctx)

Distributed locks¶

Release any held locks. Use a defer release() pattern.

File handles¶

Close files. defer f.Close().

Network connections¶

Close listeners; let in-flight connections drain via the deadline mechanism.

The shutdown sequence: stop the producers, drain the consumers, close the external connections, exit.

Appendix F: A Long Comparison Table¶

A reference table comparing implementations of "wait for N goroutines to finish."

The first two rows are wrong. The rest are correct. Pick based on requirements.

Appendix G: Production War Stories (Compressed)¶

Brief vignettes from real production incidents traced to the anti-pattern.

Story 1. Payment service drops 0.5% of transactions during peak Friday traffic for six months. Root cause: handler polls internal channel; under load the polling races. Fix: per-request completion channel. Impact: $250K monthly revenue recovered.

Story 2. Search indexer takes 90 seconds to shut down; Kubernetes kills it after 60. Some indexing work is lost. Root cause: shutdown polls a queue. Fix: errgroup-based coordination. Impact: zero data loss, shutdowns in 1-2s.

Story 3. Worker pod CPU is 22% baseline with no load. Cost: $400/month per pod across 200 pods = $80K/year. Root cause: 40 polling loops per pod, each at 1ms interval. Fix: replace with event-driven workers. Impact: baseline drops to 0.8%, $76K/year saved.

Story 4. A flaky test fails 0.5% of the time, blocking deploys randomly. For six months engineers retry the test. Root cause: test polls a queue. Fix: replace with done channel. Impact: zero flakes in subsequent year.

Story 5. A migration to a new message broker fails because the legacy code expects polling semantics; the new broker has push semantics. Fix: rewrite the consumers to use push (which is event-driven). The migration improves CPU by 60% as a side effect.

Story 6. A cache invalidator misses 0.2% of invalidations under load. Root cause: polling for queue empty. Fix: settled-after-N pattern. Impact: zero misses; data freshness improves.

These stories are not exceptional. They are the typical impact of the anti-pattern. Read them as case studies in your team meetings.

Appendix H: Observability Stack for Concurrency¶

A reference stack for observing concurrent systems.

Metrics¶

Prometheus + Grafana. Track:

• in-flight count (gauge)
• queue depth (gauge)
• processing rate (counter, rate'd)
• error rate (counter, rate'd)
• shutdown duration (histogram)
• goroutine count (gauge, from runtime.NumGoroutine)

Logs¶

Structured logs (JSON) shipped to a central system (Loki, Elasticsearch).

• Shutdown sequence logged step by step.
• Errors logged with goroutine context (if available).
• Lifecycle events (start, ready, drain, exit).

Traces¶

OpenTelemetry, exported to Jaeger, Tempo, or commercial platforms.

• Every request traced end-to-end.
• Async boundaries propagate context.
• Span names follow a convention (service.operation).

Profiles¶

Continuous profiling (Pyroscope) capturing CPU and memory.

• Compare current to baseline daily.
• Alert on regressions.

This stack is industry standard. With it, you can detect the anti-pattern at the symptom level and trace it to the root cause within minutes.

Appendix I: Closing the Loop¶

The full operational lifecycle:

1. New code is written. Lints prevent the anti-pattern.
2. Code is reviewed. The senior-level checklist applies.
3. Code is tested. -race -count=N runs in CI.
4. Code is deployed. Behind a flag, in stages.
5. Code is observed. Metrics, logs, traces capture behaviour.
6. Code is profiled. Continuous profiling detects regressions.
7. Incidents are triaged. Postmortems trace causes.
8. Patterns are documented. The team learns.
9. New code is written. Lints prevent the anti-pattern.

The loop closes. Each cycle the system gets tighter, the pattern gets rarer, the incidents get smaller.

The professional engineer is the person who builds this loop and tends it. The anti-pattern is one of many things the loop guards against, but it is a vivid example because the cost is so easily measured.

Final Word¶

The wait-for-empty-channel anti-pattern, viewed from production, is a systems failure whose symptoms appear in metrics, traces, profiles, and incidents. The fix is operational discipline as much as code-level discipline: graceful shutdown, observability, coordinated rollout, postmortem-driven learning.

A team that masters this anti-pattern masters the broader skill of running concurrent systems in production. The principles transfer: ownership, lifecycles, event-driven coordination, structured shutdown, observability. These principles apply to every concurrent service, not just the Go ones.

Move next to specification.md for the formal language semantics and standard library reference. Then interview.md, tasks.md, find-bug.md, and optimize.md for testing your knowledge and teaching it to others.

Appendix J: Sample Dashboards for Concurrency Health¶

A Grafana dashboard JSON describing the panels for concurrency health.

Panel: in-flight requests over time¶

sum(rate(worker_in_flight[1m]))

A spike indicates traffic; a slow decline during shutdown indicates a drain in progress; a flat line during shutdown indicates a stuck worker.

Panel: goroutine count¶

go_goroutines{service="worker"}

Steady-state growth indicates a leak. Sudden drops indicate restarts.

Panel: shutdown duration histogram¶

histogram_quantile(0.99, sum(rate(worker_shutdown_duration_seconds_bucket[1h])) by (le))

The P99 shutdown duration. Should be well below terminationGracePeriodSeconds. If it approaches that limit, shutdowns are too slow.

Panel: CPU profile delta¶

A panel showing the difference in CPU usage between two profile windows. After a deploy, a positive delta in time.Sleep or runtime.chanlen indicates polling was introduced.

Panel: drop counter¶

sum(rate(worker_dropped_total[5m]))

The rate of message drops. Should be zero in healthy systems. A non-zero rate is an immediate alert.

Panel: queue depth¶

worker_queue_depth

The current queue depth. Rising indicates the consumer is falling behind. Persistent rising is a capacity issue, not a polling issue, but the two are sometimes confused.

This dashboard is the central operations view for concurrent services. Every team running such a service should have one.

Appendix K: Building a Polling-Free Library Catalogue¶

A reference list of libraries that are explicitly polling-free, suitable as building blocks.

golang.org/x/sync/errgroup¶

Discussed extensively. Use for "wait for N tasks; first error cancels."

golang.org/x/sync/semaphore¶

Weighted semaphore with context support. Use for bounded concurrency.

golang.org/x/sync/singleflight¶

Deduplicates concurrent calls for the same key. Use for cache-fill or rate-limited APIs.

golang.org/x/time/rate¶

Token-bucket rate limiter. Use for limiting events per second.

github.com/cenkalti/backoff/v4¶

Exponential backoff with context support. Use for retry loops.

github.com/sourcegraph/conc¶

A modern structured concurrency library. Use as an alternative to errgroup for richer abstractions.

go.uber.org/goleak¶

Detects goroutine leaks in tests. Required for any concurrent codebase.

google.golang.org/grpc¶

gRPC client and server. GracefulStop is the canonical polling-free shutdown.

github.com/segmentio/kafka-go¶

Kafka client with context support throughout. ReadMessage(ctx) is event-driven.

github.com/redis/go-redis/v9¶

Redis client with context throughout. BLPop(ctx, ...) is event-driven.

Build your service stack from these. Each one respects context, supports cancellation, and avoids polling internally. Composition of polling-free libraries yields a polling-free service.

Appendix L: An Operational Runbook Template¶

A template for runbooks responding to anti-pattern incidents.

Title¶

[Anti-pattern] Worker shutdown hanging

Symptoms¶

• Kubernetes shows pod stuck in Terminating state for longer than terminationGracePeriodSeconds.
• After SIGKILL, lost messages reported by downstream system.
• Alert fired: shutdown.duration_seconds.p99 > 25.

Investigation¶

1. Identify the service. Look at the alert's labels.
2. Find a stuck pod: kubectl get pods | grep Terminating.
3. Get goroutine dump: kubectl exec <pod> -- /bin/sh -c "kill -ABRT 1". Capture stderr.
4. Read the dump. Look for goroutines blocked in time.Sleep or runtime.chanlen.
5. Identify the source file and line.

Mitigation¶

1. Immediate: roll back the deploy to the previous version (if recent).
2. Short-term: patch the polling loop to use WaitGroup or similar.
3. Long-term: add lint rule to prevent regression.

Postmortem¶

• Was the polling loop introduced recently?
• Why did the lint not catch it?
• Why did the tests not catch it?
• What metrics could have alerted us earlier?

Owner¶

[Team name]

Related runbooks¶

• [Anti-pattern] High baseline CPU
• [Anti-pattern] Latency tail growth

The runbook is concrete, actionable, and revisable. Maintain it as a living document.

Appendix M: Disaster Recovery for Drain Failures¶

When a drain fails and messages are lost, recovery is operational.

Step 1: quantify the loss¶

How many messages were affected? Use the message queue's metrics: input rate during the drain window minus the output rate equals the lost count.

Step 2: identify the lost messages¶

If the queue has DLQ semantics, the messages are in the DLQ. Otherwise, replay from the source of truth.

Step 3: replay¶

Submit the lost messages back through the system. Ensure the processors are idempotent so the replay does not double-process.

Step 4: notify stakeholders¶

If customer data is affected, notify per your compliance requirements.

Step 5: prevent recurrence¶

Fix the drain. Add tests. Add alerts.

This recovery path is not new — it is standard disaster recovery. The connection to the anti-pattern is that drain failures from polling are common enough to warrant a specific runbook.

Appendix N: A Migration Worksheet¶

A worksheet for tracking a refactoring migration across a large codebase.

Each row is one instance. Each instance is tracked from discovery to deployment.

Maintain in a spreadsheet, a project board, or a database. The point is visibility: at any time, you know how many instances remain and what blocks them.

Appendix O: Sample SLOs for Concurrent Services¶

Service Level Objectives that constrain anti-pattern behaviour.

SLO 1: shutdown duration¶

• Objective: 99% of shutdowns complete within 10 seconds.
• Measurement: histogram of shutdown duration.
• Alert: P99 > 15s.

A polling-driven shutdown will violate this SLO under load. Tracking it surfaces the anti-pattern.

SLO 2: drop rate¶

• Objective: 99.99% of submitted work is processed successfully.
• Measurement: counter of dropped vs total.
• Alert: drop rate > 0.01% in any 5-minute window.

Polling-driven drains drop messages. Tracking this surfaces them.

SLO 3: baseline CPU¶

• Objective: 95% of pods have baseline CPU < 5%.
• Measurement: percentile of CPU during low-traffic windows.
• Alert: median baseline CPU > 10%.

Polling causes high baseline. SLO catches it.

SLO 4: P99 latency¶

• Objective: P99 latency < SLA.
• Measurement: histogram of request latency.
• Alert: P99 > SLA for 5 minutes.

Polling adds latency to tails.

SLO 5: goroutine stability¶

• Objective: goroutine count stable over 1 hour.
• Measurement: variance of goroutine count.
• Alert: variance > 10% per hour.

Leaked polling goroutines cause growth.

SLOs codify expectations. The anti-pattern violates them; tracking the violations finds the anti-pattern.

Appendix P: Working With Cloud Providers¶

Each cloud provider has its own concurrency-related gotchas.

AWS¶

• Lambda functions cannot hold long-running goroutines after the handler returns. Polling inside a Lambda is paid CPU.
• ECS tasks respect SIGTERM with a configurable grace period.
• ALB has connection draining (default 300s). Configure to match your service's drain.

GCP¶

• Cloud Run also has the "no long goroutines after request" rule.
• GKE follows standard Kubernetes lifecycle.
• Internal load balancer drain is configurable per backend.

Azure¶

• App Service has its own deploy-time draining.
• AKS follows standard Kubernetes lifecycle.

In each, the polling anti-pattern manifests as high cost (paid CPU when idle), failed shutdowns (functions killed before draining), and dropped requests. The fix is the same: replace polling with event-driven primitives.

Appendix Q: Anti-Pattern in Serverless¶

In serverless platforms (Lambda, Cloud Functions), the anti-pattern is especially harmful because you pay for execution time. A polling loop running for 30 seconds during a Lambda's lifetime is 30 seconds of billable CPU.

Example: polling inside a Lambda¶

// Bad Lambda
func Handler(ctx context.Context) error {
    queue := startWorkQueue()
    submitJobs(queue)
    for len(queue) > 0 {
        time.Sleep(time.Millisecond)
    }
    return nil
}

The polling adds milliseconds of billable time per invocation. At a million invocations per day, that is significant cost.

Fix¶

func Handler(ctx context.Context) error {
    var wg sync.WaitGroup
    jobs := getJobs()
    for _, j := range jobs {
        j := j
        wg.Add(1)
        go func() {
            defer wg.Done()
            process(ctx, j)
        }()
    }
    wg.Wait()
    return nil
}

WaitGroup-based wait. Lambda returns as soon as all goroutines finish.

The savings: depending on workload, 10-30% reduction in Lambda execution time. At scale, significant.

Appendix R: Anti-Pattern in Cron Jobs¶

Cron jobs (CronJob in Kubernetes, scheduled Lambda) often run polling loops because the engineer thinks "this only runs every hour, who cares about CPU."

That logic is wrong for two reasons:

1. The job's container reserves the requested CPU even when idle. Polling does not change the reservation; it changes the utilization. But many workloads bill for utilization separately (cloud functions especially).
2. Cron jobs have time bounds. A polling loop that exceeds its time bound is killed mid-work.

Example¶

func nightlyBatch() {
    jobs := loadJobs()
    queue := make(chan Job, len(jobs))
    for _, j := range jobs {
        queue <- j
    }
    for i := 0; i < 8; i++ {
        go worker(queue)
    }
    for len(queue) > 0 {
        time.Sleep(time.Second)
    }
}

The polling adds seconds to every batch. If the batch is bounded by a 10-minute SLA, the polling consumes 10% of the budget.

Fix¶

Standard errgroup or WaitGroup pattern. Saves minutes per batch.

Appendix S: Anti-Pattern in Streaming¶

Streaming systems (Kafka Streams, Apache Beam, Flink) have their own coordination primitives. Mixing them with polling is common but wrong.

Example: Kafka consumer¶

for {
    msg, _ := reader.ReadMessage(ctx)
    queue <- msg
    for len(queue) > 100 {
        time.Sleep(time.Millisecond)
    }
}

The polling is a backpressure mechanism. But it polls; the channel itself provides backpressure for free.

Fix¶

for {
    select {
    case <-ctx.Done():
        return ctx.Err()
    default:
    }
    msg, err := reader.ReadMessage(ctx)
    if err != nil {
        return err
    }
    select {
    case <-ctx.Done():
        return ctx.Err()
    case queue <- msg:
    }
}

The send blocks when the queue is full. That is the backpressure. No polling needed.

Appendix T: Anti-Pattern in Testing Frameworks¶

Even testing frameworks have the anti-pattern. Common shape:

func TestEventuallyTrue(t *testing.T) {
    for !condition() {
        time.Sleep(time.Millisecond)
    }
}

This is assert.Eventually from testify written by hand. The framework's version is no better: it polls. Both fail under load.

The fix in tests: expose the event you are waiting for.

func TestSystemReady(t *testing.T) {
    done := make(chan struct{})
    sys := NewSystem(func() { close(done) })
    sys.Run()
    select {
    case <-done:
    case <-time.After(2 * time.Second):
        t.Fatal("timed out")
    }
}

The test waits on a real event. The bounded timeout exists for safety, not as a polling interval.

Appendix U: A Cross-Functional Story¶

A story in three departments: how the anti-pattern affects each.

Engineering¶

The engineering team writes the polling loop. They notice CPU is a bit high but the tests pass. They ship it. Months later, an incident traces back to the polling.

Operations¶

The ops team sees alerts firing. Shutdown duration exceeds SLO. P99 latency spikes. They escalate to engineering.

Finance¶

The finance team notices cloud spend rising. Engineering investigates and finds the polling overhead. The refactor reduces spend.

Customer¶

The customer experiences slow shutdowns during deploys: their requests time out. They complain. Customer success escalates.

Lessons¶

The anti-pattern's impact cuts across departments. Communicating it requires translating between perspectives:

• Engineering: "polling loop is a race condition."
• Operations: "shutdown is unreliable."
• Finance: "polling costs $X/year."
• Customer: "deploys interrupt my requests."

A senior professional speaks all four languages.

Appendix V: How to Brief Leadership on the Migration¶

When you propose a large refactoring effort to leadership, frame it in their terms.

Slide 1: the problem¶

"We have N instances of a known concurrency anti-pattern that causes drops, slow shutdowns, and elevated CPU. The cost is approximately $Y/year in cloud spend and Z incidents per quarter."

Slide 2: the fix¶

"We refactor each instance to use proper synchronisation primitives. The refactor is mechanical and incremental."

Slide 3: the plan¶

"Six months, one engineer at 40% allocation. Phased rollout with feature flags. Zero downtime."

Slide 4: the benefits¶

"Eliminate Z incidents. Save $Y. Improve P99 latency by W%. Make the codebase teachable for new hires."

Slide 5: the risks¶

"Each refactor is small but could introduce bugs. Mitigation: tests under -race -count=N, feature flags, gradual rollout."

Slide 6: the ask¶

"Approval to allocate the engineer. Quarterly review of progress."

Leadership cares about cost, risk, customer impact. The technical detail is in the appendix. Lead with the impact.

Appendix W: Long-Term Maintenance¶

After the migration completes, maintenance is ongoing.

Quarterly review¶

Audit a sample of services. Confirm no new instances of the pattern. Update lints if new variants emerge.

Annual training¶

Re-introduce the pattern to the team. New hires need the same training. Veterans benefit from refreshers.

Tool review¶

Are the lints catching everything? Are the metrics still meaningful? Are the runbooks up to date? Update annually.

Documentation¶

Keep this file (and the rest of the section) current. As Go evolves, the best practices evolve.

The anti-pattern does not stay gone unless someone tends the garden. Make that part of the team's regular work.

Appendix X: A Reference Migration Plan¶

A specific, detailed migration plan you can adapt.

Phase 0: discovery (weeks 1-2)¶

• Inventory every instance of the anti-pattern.
• Classify by severity (hot path vs cold path).
• Estimate effort per instance.

Phase 1: tooling (weeks 3-4)¶

• Build lint rules.
• Build dashboards.
• Build runbooks.

Phase 2: training (weeks 5-6)¶

• Team training sessions.
• Pair-refactor a few instances as a learning exercise.

Phase 3: refactor hot path (weeks 7-14)¶

• One hot-path instance per week.
• Each behind a feature flag.
• Each rolled out gradually.

Phase 4: refactor cold path (weeks 15-22)¶

• Batch refactors in batches of 5.
• Less ceremony per batch (cold path can be slower-rolled).

Phase 5: cleanup (weeks 23-26)¶

• Remove feature flags.
• Remove old code paths.
• Update documentation.

Phase 6: prevention (week 27+)¶

• Lints in CI, blocking.
• Annual training.
• Quarterly audit.

Total: 27 weeks of one engineer at 40% allocation. Most weeks are not full-time. The discipline matters more than the speed.

Appendix Y: When the Refactor Is Not Worth It¶

There are cases where refactoring is not the right call.

Case 1: code is being deprecated¶

If a service is being replaced by a new version in 6 months, refactoring its anti-patterns is pointless. Focus on the replacement.

Case 2: very low traffic¶

A cron job that runs once a week, polling for 30 seconds, costs almost nothing. Refactor only if you have spare cycles.

Case 3: third-party code¶

If the polling is in a dependency, refactoring requires upstream cooperation. Wrap with a bridge as in senior.md Appendix G; do not fork the library unless necessary.

Case 4: regulatory constraints¶

If the code is part of a regulated certification (financial, medical), changing it requires re-certification. The cost may exceed the benefit. Weigh carefully.

In each case, document the decision. A future engineer should know why the refactor was skipped, not assume oversight.

Appendix Z: One Year Later¶

A reflection on what changes after a successful migration.

Incidents¶

The number of concurrency-related incidents drops to near zero. The team's on-call burden shrinks. Sleep improves.

Cost¶

Cloud bills drop by 5-15% depending on workload. The savings continue indefinitely.

Code quality¶

Engineers reason about concurrency more carefully. PRs get tighter. Reviews catch issues earlier.

Hiring¶

The team's bar rises. New engineers see clean code and write clean code. The codebase becomes a teaching tool.

Reputation¶

The service's reputation improves: fewer outages, faster deploys, better latency. Customer satisfaction rises.

These benefits compound. The migration's one-time cost is paid back many times over the following years.

Final Word (Professional)¶

The wait-for-empty-channel anti-pattern, viewed from the professional level, is a capital-N engineering challenge: it intersects code, operations, observability, cost, customer experience, and team culture. Solving it requires every senior skill — and once solved, it raises the bar across the board.

If you are reading this as a senior or staff engineer in an organisation that has this anti-pattern, the path is clear:

1. Quantify the cost. Numbers in dollars and incidents.
2. Build the tooling. Lints, dashboards, runbooks.
3. Train the team. Theory then practice.
4. Migrate methodically. Phased, observed, rollback-ready.
5. Prevent regression. Lints, training, audits.

This is multi-quarter work. It is also some of the highest-impact engineering work you can do. The codebase that emerges is more reliable, cheaper, faster, and more pleasant to work on.

Read the other files in this section for tactical depth: specification.md, interview.md, tasks.md, find-bug.md, optimize.md.

Appendix AA: Sample Migration Communication¶

A reference for the kind of communication that sustains a migration over months.

Initial announcement (week 1)¶

Subject: Starting the concurrency anti-pattern migration

Team,

We have 200+ instances of the "wait for empty channel" anti-pattern across our codebase. This pattern causes:

- Slow shutdowns (some pods hit terminationGracePeriodSeconds)
- Dropped messages (estimated 0.5% under load)
- Elevated CPU (~15% baseline on worker pods)
- P99 latency spikes (50ms tail attributable to polling)

Estimated cost: $120K/year in cloud spend, 4 incidents/quarter.

Plan: 6-month migration, owned by [team], reviewed by [leadership]. Phase 0 is discovery; we'll share the inventory by [date].

If you encounter the pattern in your team's code, please add it to [tracker]. If you have questions, drop them in [channel].

Thanks,
[name]

Monthly progress update¶

Subject: Concurrency migration — month 3 update

- 47 of 200 instances refactored
- Zero incidents attributable to the migration
- CPU savings to date: $8K/month run-rate
- P99 latency improvement: 12% on refactored services
- On track for completion by [date]

Next month: refactor [list of services]. Owner: [name].

[link to dashboard with detailed metrics]

Final announcement¶

Subject: Concurrency migration complete

The migration to event-driven concurrency primitives is complete.

Summary:
- 213 instances refactored across [N] services
- $115K/year cloud savings (validated against billing)
- Zero incidents from the migration
- 4 lint rules added to prevent regression
- 12 team members trained

Special thanks to [list].

Forward: quarterly audits, annual training, ongoing lint maintenance.

This communication discipline keeps stakeholders informed and the migration on track.

Appendix BB: A Final Code Sample¶

A small but complete service that exemplifies every principle in this file.

package main

import (
    "context"
    "errors"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "sync/atomic"
    "syscall"
    "time"

    "golang.org/x/sync/errgroup"
)

type Service struct {
    addr     string
    workers  int
    inFlight atomic.Int64
    ready    atomic.Bool
    queue    chan Task
}

type Task struct {
    ID      string
    Payload []byte
    Done    chan error
}

func New(addr string, workers int) *Service {
    return &Service{
        addr:    addr,
        workers: workers,
        queue:   make(chan Task, 1024),
    }
}

func (s *Service) Run(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)

    // workers
    for i := 0; i < s.workers; i++ {
        g.Go(func() error {
            for task := range s.queue {
                s.inFlight.Add(1)
                err := s.process(ctx, task)
                s.inFlight.Add(-1)
                task.Done <- err
            }
            return nil
        })
    }

    // http server
    mux := http.NewServeMux()
    mux.HandleFunc("/submit", s.handleSubmit)
    mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(200)
    })
    mux.HandleFunc("/ready", func(w http.ResponseWriter, r *http.Request) {
        if s.ready.Load() {
            w.WriteHeader(200)
        } else {
            w.WriteHeader(503)
        }
    })
    srv := &http.Server{Addr: s.addr, Handler: mux}

    g.Go(func() error {
        s.ready.Store(true)
        if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
            return err
        }
        return nil
    })

    g.Go(func() error {
        <-ctx.Done()
        log.Println("ctx cancelled; starting shutdown")
        s.ready.Store(false)
        time.Sleep(15 * time.Second) // wait for LB to notice
        log.Println("shutting down http server")
        sctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
        defer cancel()
        if err := srv.Shutdown(sctx); err != nil {
            return err
        }
        log.Println("closing queue")
        close(s.queue)
        return nil
    })

    return g.Wait()
}

func (s *Service) handleSubmit(w http.ResponseWriter, r *http.Request) {
    payload, err := io.ReadAll(r.Body)
    if err != nil {
        http.Error(w, err.Error(), 400)
        return
    }
    task := Task{
        ID:      r.URL.Query().Get("id"),
        Payload: payload,
        Done:    make(chan error, 1),
    }
    select {
    case <-r.Context().Done():
        http.Error(w, "request cancelled", 499)
        return
    case s.queue <- task:
    }
    select {
    case <-r.Context().Done():
        http.Error(w, "request cancelled", 499)
    case err := <-task.Done:
        if err != nil {
            http.Error(w, err.Error(), 500)
            return
        }
        w.WriteHeader(200)
    }
}

func (s *Service) process(ctx context.Context, t Task) error {
    // simulate work
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(50 * time.Millisecond):
        return nil
    }
}

func main() {
    ctx, cancel := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
    defer cancel()

    svc := New(":8080", 8)
    if err := svc.Run(ctx); err != nil {
        log.Fatal(err)
    }
}