gammazero/workerpool — Professional Level¶

Table of Contents¶

1. Introduction
2. What Professional Means
3. Pool Sizing in Production
4. Matching Pool Size to Workload Type
5. Sizing Heuristics That Actually Work
6. Observability Foundations
7. Metrics You Must Have
8. Logging Strategy
9. Tracing Integration
10. Alerting Strategy
11. Context Integration Patterns
12. Draining vs Hard Stop
13. Kubernetes Lifecycle Integration
14. Capacity Planning
15. Adaptive Sizing
16. Production Wrapper Anatomy
17. Real-World Incident 1: The Hung Webhook
18. Real-World Incident 2: The 4 GB Queue
19. Real-World Incident 3: The Cascading Slowdown
20. Real-World Incident 4: The Reaping Surprise
21. Production Anti-Patterns
22. Migration Stories
23. Failure Modes and Defences
24. Pool as Part of System Architecture
25. Multi-Tenant Isolation
26. Cost Accounting
27. Performance Engineering
28. Tricky Production Points
29. Test
30. Cheat Sheet
31. Self-Assessment Checklist
32. Summary
33. What You Can Build
34. Further Reading
35. Related Topics
36. Diagrams and Visual Aids

Introduction¶

The professional file is about running workerpool in production. Not learning the API, not reading the source — operating it on systems that handle real traffic with real consequences.

You already know the API (junior), the operational nuances (middle), and the internals (senior). What this file adds:

• How to pick maxWorkers and queue caps deliberately.
• How to observe a pool in flight, in alerting-and-dashboards terms.
• How to survive a slow downstream, a runaway producer, a process restart.
• How to evolve the pool's configuration as load grows.
• Stories from incidents — not contrived, but representative of what actually breaks.

This file is longer than the others because production wisdom is dense. Read it once linearly, then return to specific sections as your work demands. A code review checklist at the end summarises everything.

What Professional Means¶

Senior means understanding the code. Professional means understanding the system the code runs in.

A professional engineer asks:

• "What happens to this pool at 100x load?"
• "What does the on-call see if this pool gets stuck?"
• "How long does a deploy take with this pool? What does shutdown look like?"
• "If the downstream service is degraded, does this pool make it worse or better?"
• "What's the budget — CPU, memory, goroutines, dollars — for this pool?"

You should not be able to answer these by reading source. You answer them by:

• Running load tests.
• Watching metrics during real traffic.
• Reading incident retrospectives.
• Talking to other operators.
• Sometimes, just paying attention as you ship and observe.

The professional file is your guide to those activities applied to workerpool-based code.

Pool Sizing in Production¶

The single most important decision when adopting a pool is maxWorkers. Get this wrong and nothing else matters. Get it right and most other questions become tractable.

The wrong way: pick a "nice round number"¶

pool := workerpool.New(100) // because 100 is reassuring

100 is not based on anything. Maybe it's right; maybe it's 10x too high; maybe it's 10x too low. Without measurement, you do not know.

The right way: derive from workload characteristics¶

You need three pieces of information:

1. Per-task latency. How long does one task take, on average and at p99?
2. Target throughput. Tasks per second you need to sustain.
3. Downstream concurrency capacity. How many concurrent calls can your downstream tolerate?

With these:

• workers_for_throughput = target_throughput * avg_latency
• workers_max = min(workers_for_throughput, downstream_capacity)

Example: target 200 RPS, average latency 0.5s, downstream allows 50 concurrent calls.

• workers_for_throughput = 200 * 0.5 = 100
• workers_max = min(100, 50) = 50

So workerpool.New(50). With 50 workers and 0.5s tasks, you sustain 100 RPS — not the 200 you targeted, because the downstream is the limit. Either:

• Negotiate higher downstream limits.
• Reduce per-task latency.
• Drop target throughput.

The point is: you now know why 50, not "it felt right".

The hot-reload temptation¶

A common request: "can the pool size change at runtime?"

workerpool does not support this. To approximate it, you either:

• Use a wrapper (Appendix in senior file) that swaps pools on resize.
• Use ants with Tune().
• Accept that pool size requires a restart.

For most services, restart-on-config-change is fine. For services where restart is costly (long startup, sticky sessions), implement the wrapper.

Matching Pool Size to Workload Type¶

A taxonomy of workload types and how to size for each.

Pure CPU-bound (no I/O)¶

Examples: image resizing, video encoding, compression, JSON marshalling.

maxWorkers = GOMAXPROCS. More workers than cores wastes context switches. Less than cores leaves cores idle.

pool := workerpool.New(runtime.GOMAXPROCS(0))

Pure I/O-bound (network or disk)¶

Examples: HTTP fetching, database queries, file uploads.

maxWorkers is much higher — typically 10–100x cores. Workers spend most of their time blocked on I/O, so the runtime can schedule many of them on a few threads.

How high exactly? Limited by:

• Downstream concurrency (don't overwhelm).
• Local file descriptors (ulimit -n).
• Local memory (each goroutine has a stack).
• Goroutine accounting (millions are fine but not free).

A good starting point: 50-100 workers per core. So an 8-core machine: 400-800 workers for pure I/O.

Mixed CPU and I/O¶

Examples: parsing JSON from HTTP responses, transforming data fetched from a DB.

If the work is mostly I/O with a small CPU step, treat as I/O-bound. If it's mostly CPU with a small I/O wait, treat as CPU-bound. If it's 50/50, size to the I/O dimension (CPU work will share cores fine).

For real precision, profile: how much wall-clock time does a task spend in each phase? Size for the longer phase.

Database-bound¶

Special case of I/O-bound. The constraint is usually the database's concurrent connection limit:

maxWorkers <= db_max_connections - other_consumers

If your DB allows 100 connections and other processes use 30, set maxWorkers = 70 (or fewer, to leave headroom).

Connection pooling at the DB driver level matters: each task takes a connection from the driver's pool. If the driver's pool is smaller than maxWorkers, tasks block waiting for a connection — wasted worker slots.

Sync your driver pool size to your maxWorkers.

External API-bound¶

Constrained by the API's rate limit and concurrent connection limit. Read their docs:

"50 requests per second, 10 concurrent"

Use a rate limiter (rate.NewLimiter(50, 50)) for the RPS bound, and workerpool.New(10) for the concurrency bound. Both work together: the limiter ensures you don't exceed 50/s, the pool ensures no more than 10 are in flight.

Sizing Heuristics That Actually Work¶

A grab-bag of rules of thumb, in the order you should apply them.

Rule 1: Start small¶

Begin with maxWorkers = 2 or 4. Measure. Increase only when you see queue depth growing under load. Resist the temptation to start "high enough to handle anything" — that's how you accidentally hammer a downstream.

Rule 2: Use NumCPU as a floor for CPU-bound¶

For CPU work: maxWorkers >= runtime.NumCPU() (so all cores can be used).

Rule 3: Use Latency * RPS for I/O-bound¶

maxWorkers = avg_task_seconds * target_rps. Adjust for p99 latency if tail-sensitive.

Rule 4: Bound by downstream¶

Whatever you compute, do not exceed the downstream's tolerated concurrency.

Rule 5: Add 20% headroom¶

maxWorkers = computed_value * 1.2. Handles modest traffic spikes without queueing.

Rule 6: Test with synthetic load¶

Before deploying a new pool size, run a load test that exceeds expected traffic by 2x. Watch queue depth, latency, error rate.

Rule 7: Document the math¶

In code:

// maxWorkers = 50: chosen for downstream limit (DB connection cap = 60, 10 reserved for other services)
// expected steady-state: 30-40 workers active under normal load
// expected peak: 50 workers (saturated) under traffic spikes
const dbPoolWorkers = 50
var dbPool = workerpool.New(dbPoolWorkers)

Future maintainers (or future you) will thank you.

Rule 8: Revisit on every deploy¶

Workload changes, hardware changes, downstream changes. If your pool size has not been reviewed in six months, it is probably wrong.

Rule 9: Be especially careful around 1 and NumCPU¶

maxWorkers = 1 is a serialised queue — sometimes intended, often a bug. maxWorkers = NumCPU() is standard for CPU-bound — but on different machines NumCPU differs, so the same code runs at different parallelism. Be explicit.

Rule 10: When in doubt, smaller¶

A pool that is too small overflows its queue noticeably (you can see and fix it). A pool that is too large can silently kill a downstream. Bias toward smaller.

Observability Foundations¶

A pool without observability is a black box. In production, that means trouble.

You need three layers of observability:

1. Metrics — counters and gauges, for trending and alerting.
2. Logging — discrete events, for forensic analysis.
3. Tracing — per-request flows, for understanding cross-service interactions.

Each layer answers different questions:

• Metrics: "is the pool healthy right now?"
• Logging: "what happened during this incident?"
• Tracing: "where did this specific task go?"

A production-grade pool wrapper instruments all three.

Why instrumentation at the wrapper, not the library?¶

The library is generic; your instrumentation is service-specific. Adding metrics/logging/tracing inside the library would be invasive and opinionated. Adding it in a thin wrapper around the library is clean and gives you full control.

This is the same pattern used by every other production-grade library wrapper in Go: net/http with Prometheus middleware, sql.DB with OpenTelemetry, etc.

Metrics You Must Have¶

The minimum metrics for any production pool:

1. Submitted counter¶

poolSubmitted := prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: "workerpool_submitted_total",
    Help: "Total tasks submitted to the pool",
}, []string{"pool"})

Increment on every Submit. Tells you the input rate.

2. Completed counter¶

poolCompleted := prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: "workerpool_completed_total",
    Help: "Total tasks that completed (including panicked)",
}, []string{"pool", "result"})

Increment with result="ok" on success, result="panic" on panic, result="error" on returned error. Tells you output rate and breakdown.

3. Queue depth gauge¶

poolQueueDepth := prometheus.NewGaugeVec(prometheus.GaugeOpts{
    Name: "workerpool_queue_depth",
    Help: "Current waiting queue depth",
}, []string{"pool"})

Set periodically (every 5-10 seconds) from pool.WaitingQueueSize(). Tells you backpressure status.

4. Active workers gauge (best-effort)¶

The library does not expose this directly. Approximate via submitted - completed (in-flight) minus queue depth (those queued but not running):

inflight := submitted - completed
running := inflight - queueDepth

running is an approximation but usually accurate.

5. Task duration histogram¶

poolDuration := prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name: "workerpool_task_duration_seconds",
    Help: "Task execution time",
    Buckets: prometheus.ExponentialBuckets(0.001, 2, 14), // 1ms to 8s
}, []string{"pool"})

Observe time.Since(start) at the end of each task. Tells you per-task latency distribution.

6. Queue dwell histogram¶

The time a task spends queued before running:

poolDwell := prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name: "workerpool_queue_dwell_seconds",
    Help: "Time in queue before execution",
}, []string{"pool"})

Observe runStart - submitStart. Tells you if the pool is keeping up.

7. Drop counter (if you implement bounded queue)¶

poolDropped := prometheus.NewCounterVec(prometheus.CounterOpts{
    Name: "workerpool_dropped_total",
    Help: "Tasks rejected because queue was full",
}, []string{"pool"})

8. Stopped indicator¶

poolStopped := prometheus.NewGaugeVec(prometheus.GaugeOpts{
    Name: "workerpool_stopped",
    Help: "1 if pool is stopped, 0 otherwise",
}, []string{"pool"})

Periodically set from pool.Stopped(). Tells you if the pool is shutting down.

Putting them together¶

A wrapper that emits all eight metrics looks like this:

type InstrumentedPool struct {
    name string
    pool *workerpool.WorkerPool
}

func (ip *InstrumentedPool) Submit(taskName string, f func()) {
    submitTime := time.Now()
    poolSubmitted.WithLabelValues(ip.name).Inc()
    ip.pool.Submit(func() {
        runStart := time.Now()
        poolDwell.WithLabelValues(ip.name).Observe(runStart.Sub(submitTime).Seconds())

        result := "ok"
        defer func() {
            if r := recover(); r != nil {
                result = "panic"
                log.Printf("pool=%s task=%s panic=%v", ip.name, taskName, r)
            }
            poolCompleted.WithLabelValues(ip.name, result).Inc()
            poolDuration.WithLabelValues(ip.name).Observe(time.Since(runStart).Seconds())
        }()

        f()
    })
}

In a separate goroutine, update gauges periodically:

go func() {
    ticker := time.NewTicker(5 * time.Second)
    for range ticker.C {
        poolQueueDepth.WithLabelValues(ip.name).Set(float64(ip.pool.WaitingQueueSize()))
        if ip.pool.Stopped() {
            poolStopped.WithLabelValues(ip.name).Set(1)
        } else {
            poolStopped.WithLabelValues(ip.name).Set(0)
        }
    }
}()

Logging Strategy¶

Metrics give you trends; logs give you specifics. The right log volume:

• Per-task logs: only in debug mode. At production scale they swamp.
• Slow-task logs: yes, with a threshold. "Task X took 5s" is actionable.
• Panic logs: always. With stack traces.
• Drop logs: rate-limited. "Dropped 100 tasks in the last minute" is useful; logging every drop is noise.
• Pool lifecycle: at INFO level. "Pool created", "Pool drained", "Pool stopped with N tasks dropped".

A pattern:

ip.pool.Submit(func() {
    start := time.Now()
    defer func() {
        elapsed := time.Since(start)
        if r := recover(); r != nil {
            log.Error("task panic",
                "pool", ip.name,
                "task", taskName,
                "panic", r,
                "stack", string(debug.Stack()),
                "elapsed", elapsed)
            return
        }
        if elapsed > 500*time.Millisecond {
            log.Warn("slow task",
                "pool", ip.name,
                "task", taskName,
                "elapsed", elapsed)
        }
    }()
    f()
})

Structured logging (slog, zap, zerolog) is essential. Plain text logs are unsearchable at scale.

What to log on slow tasks¶

The minimum: pool name, task name, elapsed time. Add the request ID if available. Avoid the task arguments (they may be huge, sensitive, or both).

What to log on shutdown¶

INFO pool=webhook draining started, queue_depth=1234
INFO pool=webhook draining progress, remaining=623, elapsed=10s
WARN pool=webhook drain deadline exceeded, calling hard stop
INFO pool=webhook stopped, dropped=120

These four-line stories tell the on-call exactly what happened during shutdown.

Tracing Integration¶

For services with distributed tracing (OpenTelemetry, Jaeger), pool tasks should appear as spans in the trace.

ip.pool.Submit(func() {
    ctx, span := tracer.Start(ctx, "workerpool.task",
        trace.WithAttributes(
            attribute.String("pool", ip.name),
            attribute.String("task", taskName),
        ))
    defer span.End()

    if err := f(ctx); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
    }
})

The span captures:

• The pool name.
• The task name.
• Start and end times.
• Any errors.

In the trace viewer, you see: parent request → handler span → pool task span. Crucially, you see how long the task waited in the queue if you record the submit-time as a span event.

This is how production engineers diagnose "why did this request take 5 seconds?" — they trace it, find the pool task span, see the dwell time was 4 seconds, conclude the pool was overloaded.

Alerting Strategy¶

Alerts are metrics with thresholds and on-call notification. The minimum alerts for a production pool:

Alert 1: Queue depth high¶

alert: PoolQueueGrowing
expr: workerpool_queue_depth > 1000
for: 5m
severity: warning
summary: "Pool {{ $labels.pool }} queue is over 1000 for 5 minutes"

If the queue is over 1000 for 5 minutes, you have backpressure issues. Either size up, slow producers, or investigate the downstream.

Alert 2: Pool stopped unexpectedly¶

alert: PoolStopped
expr: workerpool_stopped == 1
for: 1m
severity: page
summary: "Pool {{ $labels.pool }} is stopped"

If the pool stopped but the service is supposed to be running, page. This usually means a deploy or a crash.

Alert 3: Panic rate elevated¶

alert: PoolPanicRate
expr: rate(workerpool_completed_total{result="panic"}[5m]) > 0.1
for: 5m
severity: page
summary: "Pool {{ $labels.pool }} panicking >0.1/s"

Panics in tasks are bugs. A bursty panic might be a bad input; a sustained panic is a broken deployment.

Alert 4: Drop rate high¶

alert: PoolDropping
expr: rate(workerpool_dropped_total[5m]) > 0
for: 5m
severity: warning
summary: "Pool {{ $labels.pool }} dropping tasks"

If you have bounded queues and they are dropping, you are losing work. Investigate.

Alert 5: Task duration p99 spike¶

alert: PoolSlowTasks
expr: histogram_quantile(0.99, rate(workerpool_task_duration_seconds_bucket[5m])) > 5
for: 10m
severity: warning
summary: "Pool {{ $labels.pool }} p99 task duration > 5s"

Sustained slow tasks usually mean a downstream issue.

These five alerts cover ~90% of pool-related incidents. Tune the thresholds to your service's normal levels.

Context Integration Patterns¶

Production pools must respect cancellation. The library does not enforce this; you do.

Pattern: parent context cancels everything¶

type Service struct {
    pool *workerpool.WorkerPool
    ctx  context.Context
}

func (s *Service) Process(item Item) {
    s.pool.Submit(func() {
        s.handleItem(s.ctx, item)
    })
}

func (s *Service) Shutdown() {
    s.pool.StopWait()
}

The service has a context that lives as long as the service. Cancelling it (via shutdown signal) propagates to every task. Tasks check s.ctx.Done() and bail.

Pattern: per-request context propagation¶

In an HTTP server, each request has its own context. To propagate it to a pool task:

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()
    pool.Submit(func() {
        if err := ctx.Err(); err != nil {
            return // client disconnected before we started
        }
        process(ctx, ...)
    })
    fmt.Fprintln(w, "accepted")
}

If the client disconnects, the context is cancelled. The task notices and bails early. This is good citizenship — no work is wasted on a request that no longer matters.

Caveat: HTTP request contexts are cancelled when the response is fully written. If you fmt.Fprintln(w, "accepted") and return, the response is complete and the context cancels. The pool task then sees a cancelled context. To detach, use context.Background() instead:

func handler(w http.ResponseWriter, r *http.Request) {
    pool.Submit(func() {
        process(context.Background(), ...) // continues even after response
    })
    fmt.Fprintln(w, "accepted")
}

Pick deliberately. "Cancel when client gives up" vs "keep going regardless" is a design choice.

Pattern: per-task timeout¶

pool.Submit(func() {
    ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
    defer cancel()
    process(ctx, ...)
})

Every task has a 10-second deadline. After 10s, the context is cancelled and the task should bail. This bounds worker hold time.

Critical: actually check ctx.Done() inside process. The context does nothing on its own; the code must respect it.

Pattern: deadline from upstream¶

In a chained service:

func handler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithDeadline(r.Context(), time.Now().Add(5*time.Second))
    defer cancel()
    pool.Submit(func() {
        process(ctx, ...)
    })
}

The deadline propagates from request to pool to downstream calls. If the request has 5s budget, no task can run beyond that. Composable and clean.

Draining vs Hard Stop¶

Shutdown is where pool bugs hide. Let's make this explicit.

When to drain¶

• The work in the queue is valuable.
• You have time to wait (maintenance window, scheduled deploy).
• The downstream can handle continued requests.

Use pool.StopWait().

When to hard-stop¶

• You are out of time (signal received with deadline ending).
• The work in the queue is no longer valuable (cancelled, stale).
• Downstream is dead and continued requests just fail.

Use pool.Stop().

When to combine¶

Most often: drain with a deadline, then hard-stop if exceeded.

drained := make(chan struct{})
go func() {
    pool.StopWait()
    close(drained)
}()

select {
case <-drained:
    log.Info("drained cleanly")
case <-time.After(deadline):
    log.Warn("drain deadline; hard stop")
    pool.Stop()
    <-drained
}

Adjust deadline to your service's SLA. For Kubernetes, default terminationGracePeriodSeconds is 30s; the deadline should be ~25s (leave a 5s buffer for the rest of shutdown).

Counting dropped tasks¶

When you hard-stop, some tasks are dropped. Count them so you know the impact:

before := pool.WaitingQueueSize()
pool.Stop()
<-drained
log.Info("hard stop", "dropped", before)

This is approximate (the queue may have changed between WaitingQueueSize and Stop), but close enough for accounting.

Stopping producers first¶

The shutdown sequence must be:

1. Stop accepting new requests (HTTP server shutdown).
2. Stop any internal producers (close channels, signal goroutines).
3. Drain the pool.

If you reverse 2 and 3, the pool drains, then producers add more work, and you never finish draining.

A common bug: forgetting step 2. Background tickers keep submitting; the pool never drains.

// correct order
close(myInternalProducerStopCh)
producerWg.Wait()
pool.StopWait()

Kubernetes Lifecycle Integration¶

If your service runs on Kubernetes, you must handle SIGTERM correctly.

The default behaviour¶

K8s sends SIGTERM, waits terminationGracePeriodSeconds (default 30s), then SIGKILL. If your service doesn't drain in time, it dies abruptly and any in-flight tasks are lost.

A robust signal handler¶

func main() {
    pool := workerpool.New(50)
    srv := &http.Server{...}

    sigs := make(chan os.Signal, 1)
    signal.Notify(sigs, syscall.SIGTERM, syscall.SIGINT)

    go func() {
        srv.ListenAndServe()
    }()

    sig := <-sigs
    log.Info("signal received", "signal", sig)

    // Phase 1: stop accepting new HTTP requests
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Error("server shutdown", "err", err)
    }
    log.Info("http server stopped, draining pool")

    // Phase 2: drain pool with remaining time budget
    drained := make(chan struct{})
    go func() {
        pool.StopWait()
        close(drained)
    }()
    select {
    case <-drained:
        log.Info("pool drained cleanly")
    case <-shutdownCtx.Done():
        log.Warn("pool drain deadline, hard stop")
        pool.Stop()
        <-drained
    }

    log.Info("shutdown complete")
}

Key points:

• One shutdownCtx with the total budget (30s).
• HTTP server shutdown uses some of that budget.
• Pool drain uses the rest.
• If we run out of time, hard-stop and accept the loss.

preStop hook¶

K8s lets you run a script before SIGTERM. Use it to remove the pod from the load balancer earlier:

lifecycle:
  preStop:
    exec:
      command: ["/bin/sh", "-c", "sleep 5"]

This delays SIGTERM by 5 seconds, giving the LB time to stop routing to this pod. Your draining then happens with the pod already out of rotation — no new requests arrive.

Liveness vs readiness¶

• Liveness probe: fails → pod restart. Should NOT fail during shutdown (that triggers restart loops).
• Readiness probe: fails → pod removed from LB. SHOULD fail during shutdown.

Implement readiness to return 503 once pool.Stopped() is true. K8s pulls you from the LB; remaining in-flight requests finish; pool drains; exit.

http.HandleFunc("/readyz", func(w http.ResponseWriter, r *http.Request) {
    if pool.Stopped() {
        http.Error(w, "shutting down", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusOK)
})

Capacity Planning¶

A more rigorous take than "Rule 5: Add 20% headroom".

Establishing the baseline¶

For a stable workload, run for a week and observe:

• Mean RPS.
• Peak RPS (max over 1-minute windows).
• Mean task duration.
• p99 task duration.
• Mean queue depth.
• Peak queue depth.

These six numbers characterise your pool's normal operation.

Projecting growth¶

If you expect 50% growth in traffic over 6 months:

• Peak RPS will be 1.5x current.
• Mean queue depth may be much higher than 1.5x (queues grow non-linearly with load).
• You will need more workers and/or more capacity downstream.

Plot expected queue depth vs maxWorkers for the projected load. Pick the point where queue depth stays manageable.

Headroom for spikes¶

Real traffic has spikes. A standard recommendation:

• Size for peak_rps * 1.5.
• Or: size so queue depth stays under 1000 even at peak.

Adjust to your service's tail-tolerance.

Cost considerations¶

Each worker is roughly 4 KB of memory at idle plus whatever the task allocates. 1000 workers ~= 4 MB minimum. Negligible.

Each worker holds a downstream resource (a DB connection, an HTTP connection). At 1000 workers, you have 1000 connections in use during peak. That may be expensive — connections to RDS, paid API calls, etc.

So pool size affects:

• CPU (negligible unless you saturate cores).
• Memory (negligible).
• Downstream resources (significant).
• Per-request cost (if downstream is paid).

Pick based on the constraining resource, not raw worker count.

Adaptive Sizing¶

For services with widely varying load (a 10x daily curve, for example), a static pool size is suboptimal. Adaptive sizing improves utilisation.

The simplest adaptation¶

Two pool sizes: "day" and "night". Switch on a schedule.

type AdaptivePool struct {
    mu sync.RWMutex
    inner *workerpool.WorkerPool
}

func (ap *AdaptivePool) Switch(target int) {
    ap.mu.Lock()
    defer ap.mu.Unlock()
    old := ap.inner
    ap.inner = workerpool.New(target)
    go old.StopWait()
}

func (ap *AdaptivePool) Submit(f func()) {
    ap.mu.RLock()
    p := ap.inner
    ap.mu.RUnlock()
    p.Submit(f)
}

func adapt(ap *AdaptivePool) {
    ticker := time.NewTicker(time.Hour)
    for range ticker.C {
        h := time.Now().Hour()
        if h >= 9 && h <= 18 {
            ap.Switch(100) // day
        } else {
            ap.Switch(20) // night
        }
    }
}

The drawback: pool swap is disruptive. Inflight tasks on the old pool finish; new tasks go to the new pool. There's a brief period of dual-pool state.

Better: target-utilisation feedback¶

Compute current utilisation; resize to keep it around a target (say 70%):

func adaptUtilization(ap *AdaptivePool, target float64) {
    ticker := time.NewTicker(time.Minute)
    for range ticker.C {
        u := utilization(ap) // your metric of "how full is the pool"
        if u > target+0.1 {
            ap.Switch(ap.size + 10)
        } else if u < target-0.1 {
            ap.Switch(ap.size - 10)
        }
    }
}

Damping the changes (only resize if outside a band) prevents oscillation.

Using ants for resize¶

If adaptive sizing is important, just use ants and its Tune() method:

pool, _ := ants.NewPool(50, ants.WithExpiryDuration(30*time.Second))
// later
pool.Tune(75)

Cheaper and cleaner than the workerpool wrapper.

The choice between "wrap workerpool" and "switch to ants" is mostly cultural. If your codebase already standardised on workerpool, the wrapper is fine. If you are starting fresh and need resize, ants is the simpler path.

Production Wrapper Anatomy¶

The reference wrapper for a production-grade pool. This is what you should aim to have for any pool that handles real traffic.

package prodpool

import (
    "context"
    "errors"
    "fmt"
    "log/slog"
    "runtime/debug"
    "sync"
    "sync/atomic"
    "time"

    "github.com/gammazero/workerpool"
    "github.com/prometheus/client_golang/prometheus"
)

// Pool wraps gammazero/workerpool with:
//   - bounded queue (semaphore)
//   - context-aware submit
//   - panic recovery with attribution
//   - prometheus metrics
//   - structured logging
//   - graceful + bounded shutdown
type Pool struct {
    name   string
    inner  *workerpool.WorkerPool
    sem    chan struct{}
    log    *slog.Logger

    // metrics
    submitted prometheus.Counter
    completed *prometheus.CounterVec
    duration  prometheus.Histogram
    dwell     prometheus.Histogram
    qDepth    prometheus.Gauge
    dropped   prometheus.Counter

    // state
    stopped   atomic.Bool
    stopOnce  sync.Once
    pendingWG sync.WaitGroup
}

// Options configures a Pool at creation.
type Options struct {
    Name          string
    MaxWorkers    int
    QueueCap      int
    Logger        *slog.Logger
    Registerer    prometheus.Registerer
}

// New creates a production-ready pool.
func New(opts Options) *Pool {
    if opts.MaxWorkers < 1 {
        opts.MaxWorkers = 1
    }
    if opts.QueueCap < 1 {
        opts.QueueCap = opts.MaxWorkers * 10
    }
    if opts.Logger == nil {
        opts.Logger = slog.Default()
    }
    if opts.Registerer == nil {
        opts.Registerer = prometheus.DefaultRegisterer
    }

    p := &Pool{
        name:  opts.Name,
        inner: workerpool.New(opts.MaxWorkers),
        sem:   make(chan struct{}, opts.QueueCap),
        log:   opts.Logger,
    }

    labels := prometheus.Labels{"pool": opts.Name}
    p.submitted = prometheus.NewCounter(prometheus.CounterOpts{
        Name: "workerpool_submitted_total", ConstLabels: labels,
    })
    p.completed = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "workerpool_completed_total", ConstLabels: labels,
    }, []string{"result"})
    p.duration = prometheus.NewHistogram(prometheus.HistogramOpts{
        Name: "workerpool_task_duration_seconds", ConstLabels: labels,
        Buckets: prometheus.ExponentialBuckets(0.001, 2, 14),
    })
    p.dwell = prometheus.NewHistogram(prometheus.HistogramOpts{
        Name: "workerpool_queue_dwell_seconds", ConstLabels: labels,
        Buckets: prometheus.ExponentialBuckets(0.001, 2, 14),
    })
    p.qDepth = prometheus.NewGauge(prometheus.GaugeOpts{
        Name: "workerpool_queue_depth", ConstLabels: labels,
    })
    p.dropped = prometheus.NewCounter(prometheus.CounterOpts{
        Name: "workerpool_dropped_total", ConstLabels: labels,
    })

    opts.Registerer.MustRegister(p.submitted, p.completed, p.duration, p.dwell, p.qDepth, p.dropped)

    go p.gaugeLoop()
    return p
}

// Submit schedules a task. Returns error if the pool is closed or full.
func (p *Pool) Submit(ctx context.Context, taskName string, f func(context.Context) error) error {
    if p.stopped.Load() {
        return errors.New("pool stopped")
    }
    submitTime := time.Now()

    select {
    case p.sem <- struct{}{}:
    case <-ctx.Done():
        p.dropped.Inc()
        return ctx.Err()
    default:
        p.dropped.Inc()
        return fmt.Errorf("pool %q full", p.name)
    }

    if p.stopped.Load() {
        <-p.sem
        p.dropped.Inc()
        return errors.New("pool stopped")
    }

    p.submitted.Inc()
    p.pendingWG.Add(1)
    p.inner.Submit(func() {
        runStart := time.Now()
        p.dwell.Observe(runStart.Sub(submitTime).Seconds())
        result := "ok"
        defer func() {
            <-p.sem
            p.pendingWG.Done()
            if r := recover(); r != nil {
                result = "panic"
                p.log.Error("task panic",
                    "pool", p.name,
                    "task", taskName,
                    "panic", r,
                    "stack", string(debug.Stack()))
            }
            p.completed.WithLabelValues(result).Inc()
            p.duration.Observe(time.Since(runStart).Seconds())
        }()
        if err := f(ctx); err != nil {
            result = "error"
            p.log.Warn("task error",
                "pool", p.name,
                "task", taskName,
                "err", err)
        }
    })
    return nil
}

func (p *Pool) gaugeLoop() {
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()
    for range ticker.C {
        if p.stopped.Load() {
            return
        }
        p.qDepth.Set(float64(p.inner.WaitingQueueSize()))
    }
}

// Stop drains the pool with a deadline. After the deadline, performs a hard stop.
func (p *Pool) Stop(deadline time.Duration) (dropped int64) {
    p.stopOnce.Do(func() {
        p.stopped.Store(true)
        p.log.Info("pool draining", "pool", p.name, "queue", p.inner.WaitingQueueSize())

        done := make(chan struct{})
        go func() {
            p.inner.StopWait()
            close(done)
        }()

        select {
        case <-done:
            p.log.Info("pool drained cleanly", "pool", p.name)
        case <-time.After(deadline):
            dropped = int64(p.inner.WaitingQueueSize())
            p.log.Warn("pool drain deadline; hard stop",
                "pool", p.name, "dropped", dropped)
            p.inner.Stop()
            <-done
        }
    })
    return dropped
}

This is ~150 lines and packs in:

• Configurable construction.
• Bounded queue.
• Context-aware drop on overload.
• Panic recovery with stack and attribution.
• Six Prometheus metrics.
• Structured logging.
• Graceful shutdown with deadline.
• Stop idempotency via sync.Once.

You can drop this into a service and ship. It is the kind of code that survives audits.

Real-World Incident 1: The Hung Webhook¶

Setup¶

A service delivers webhooks to customer-configured URLs. Pool size 50. Worked for two years without incident.

Incident¶

One customer's URL started hanging — connections accepted, no response, no close. Tasks delivering to that URL waited on the response. With 50 workers and a high enough rate of webhooks for that customer, all 50 eventually held hung connections. New webhooks queued unbounded.

Memory grew from 200 MB to 4 GB over 10 minutes. Liveness probe failed (metrics endpoint slow under GC pressure). Pod restarted. New pod hit the same hung URL. Restart loop.

Investigation¶

Goroutine dump from the dying pod showed:

50 workers in net/http.(*Transport).RoundTrip
all blocked in chan receive

The HTTP client had no timeout. The downstream never closed the connection. Workers were stuck forever.

Fix¶

Per-call timeout:

client := &http.Client{Timeout: 30 * time.Second}

That single line, applied across the codebase, fixed the immediate issue. The webhook to the hung URL now failed after 30 seconds; workers freed up; the queue drained.

Additional defences added later:

• Bounded queue (1,000 tasks max).
• Per-customer concurrency limit.
• Dead-letter queue for repeatedly failing webhooks.

Lessons¶

1. Network timeouts are mandatory. Always. Default Go HTTP client has no timeout — it will hang forever.
2. Unbounded queues turn small downstream issues into outages. A single bad URL took down a whole service.
3. Liveness probes need to be cheap. If they share resources with the main work, they can fail under load and trigger restarts that make things worse.
4. Per-customer isolation prevents cross-tenant impact. One bad customer should not take out the others.

Real-World Incident 2: The 4 GB Queue¶

Setup¶

A batch processor pulled records from a queue and submitted each to a workerpool for transformation. Pool size 16. Queue was the system's input; transformation involved an external API call.

Incident¶

The external API had a partial outage — requests succeeded but took 30 seconds instead of 200 ms. The batch processor kept pulling at full speed (it didn't know about the slowdown). The pool's queue grew from ~100 to ~500,000 entries over 15 minutes.

Each queue entry held a closure with ~2 KB of captured state. Memory: 500,000 × 2 KB = 1 GB just for closures. Plus the records themselves: another 3 GB. Total: 4 GB and growing.

Pod hit OOM, restarted. New pod started pulling records again. Same outcome.

Investigation¶

heap profile showed:

ROOT
  github.com/gammazero/workerpool.(*WorkerPool).Submit
    main.(*Processor).submit
      main.(*Processor).buildClosure
        ... 3.5 GB ...

The closures captured large records by value. With 500,000 in the queue, that was the lion's share of memory.

Fix¶

Two changes:

1. Bound the queue at 1,000 entries via semaphore. Producers block when full.
2. Slow the producer when API latency rises. A simple feedback loop.

After the fix, the pool's queue stayed under 1,000 even during the outage. The producer paused when the API was slow. Memory stayed flat.

Lessons¶

1. Unbounded queues are a memory bug. Even if you don't OOM today, you might tomorrow.
2. Closures with large captures multiply the problem. Audit what your task closures hold.
3. Producers should react to consumer health. If the downstream is slow, the producer should slow down. Don't blindly fill the queue.
4. Profile heap during outages. It tells you exactly what's eating memory.

Real-World Incident 3: The Cascading Slowdown¶

Setup¶

A pipeline of three pools: parse (4 workers), enrich (8 workers), write (16 workers). Each stage submitted to the next via channels.

Incident¶

A new "enrich" rule was deployed that hit a third-party API. The third-party API was rate-limited. Enrich tasks started taking 5 seconds instead of 50 ms.

The enrich pool's queue grew. Parse kept producing (its workers were fine). The enrich-side channel (between parse and enrich pools) filled to its capacity (10,000). Parse workers blocked on ch <- item. Parse stopped processing.

Now nothing was being read from the input queue. The input queue grew. Soon the upstream service (which fed the input queue) was throttled. The whole pipeline stalled.

Investigation¶

runtime.Stack showed:

• Parse workers: blocked on enrichCh <- item.
• Enrich workers: blocked on third-party API.
• Write workers: idle.

Trace showed third-party API latency had jumped from 50ms p99 to 5s p99.

Fix¶

Multiple changes:

1. Bounded retries on the third-party API with circuit breaker.
2. Drop enrichment when the API is unhealthy (fall back to "no enrichment" mode).
3. Resize enrich pool from 8 to 32 to handle the slower latency.
4. Convert the inter-pool channels to bounded with overflow → log + drop.

After the fix, third-party API slowdowns no longer cascaded. The pipeline kept moving (with degraded enrichment quality, by design).

Lessons¶

1. Pipelines transmit pressure. A slow stage blocks the upstream.
2. Circuit breakers are essential for external dependencies. Otherwise one downstream's incident becomes yours.
3. Have a degraded mode. If enrichment is unhealthy, prefer "unenriched data" to "no data".
4. Pool sizes must be set per-stage based on each stage's latency. Inheriting sizes between stages is wrong.

Real-World Incident 4: The Reaping Surprise¶

Setup¶

A long-running service used workerpool for background tasks that arrived every 5 seconds (one task every five seconds). Pool size 4.

Incident¶

CPU usage was 5% during normal operation but spiked to 30% every 5 seconds in a sawtooth pattern. The team thought it was a memory allocation issue and spent two days profiling allocations. Nothing.

Investigation¶

A goroutine count graph showed:

0s: 1 (just dispatcher)
5s: 2 (dispatcher + 1 worker)
7s: 1 (worker reaped after 2s idle)
10s: 2 (new task, new worker)
12s: 1
...

Every 5-second task spawned a new worker (because the previous one was reaped at the 2-second mark). The CPU spike was the goroutine creation + scheduling overhead. Tiny, but visible.

Fix¶

Two options:

1. Submit a heartbeat task every 1.9 seconds to keep workers warm.
2. Just accept the cost (it was 30% of 5%, so 1.5% effective).

The team chose option 2 after measuring. The CPU spike was real but had no user impact. They added a graph annotation explaining the pattern and moved on.

Lessons¶

1. Idle reaping has visible cost on bursty workloads. Not always a bug; just a thing to understand.
2. Look at goroutine counts, not just allocation profiles. Worker churn isn't an allocation issue.
3. Sometimes the right fix is "accept it". Not every observation requires action.
4. Library defaults are tuned for the common case. Yours might not match.

Production Anti-Patterns¶

A consolidated list of "do not".

1. Untimed network calls in tasks¶

No http.Client.Timeout, no context.WithTimeout. The task hangs forever. The worker is stuck. Eventually the pool is stuck. Always set a timeout.

2. Unbounded queue with untrusted producers¶

A queue accepting input from the network must be bounded. Use a semaphore. Drop tasks rather than OOM.

3. Pool per request¶

func handler(w http.ResponseWriter, r *http.Request) {
    pool := workerpool.New(8)  // wrong
    defer pool.StopWait()
    // ... submit tasks ...
}

Each request pays the dispatcher startup cost. Hoist the pool to package scope.

4. Forgotten StopWait¶

Tests pass because GC reclaims, production leaks because the process doesn't restart often.

5. Cross-pool synchronous calls¶

poolA submits to poolB, blocks waiting. If poolB is saturated, poolA is stuck. Use async hand-off (channels, fire-and-forget).

6. Panicky tasks without recovery¶

The library recovers, but the panic value and stack are lost. Wrap each task with defer recover().

7. Shared state without locks¶

Two workers write to a map without a mutex. The race detector finds it eventually. Production doesn't get the chance.

8. No metrics¶

Pools without observability are bugs waiting to happen. At minimum: submitted, completed, queue depth.

9. Pool size based on optimism¶

maxWorkers = 1000 because "we might need it". You don't. Size from data.

10. Restart on every config change¶

If your maxWorkers is in a YAML file and you have to redeploy to change it, you can't react to incidents quickly. Build hot-reload (or use ants).

11. Single pool for mixed workloads¶

CPU and I/O workloads in one pool starve each other. Split.

12. Ignoring queue depth in capacity planning¶

The queue is part of your memory footprint. A 100,000-entry queue with 2 KB closures is 200 MB. Plan for it.

13. No drain deadline¶

pool.StopWait() blocks until done. If a task hangs, you hang. K8s SIGKILLs you. Use a deadline.

14. Submitting before init complete¶

Module-level var pool = workerpool.New(N) runs during package init. If a sibling package's init also submits, ordering matters. Prefer lazy init or explicit construction in main.

15. Pool inside a goroutine that exits¶

go func() {
    pool := workerpool.New(8)
    defer pool.StopWait()
    // ...
}()

When the goroutine exits, the pool is GC'd. Tasks may not have run. Always make pool lifetime explicit.

Migration Stories¶

From a hand-rolled pool to workerpool¶

Common path. A team has 600 lines of pool code accumulated over years, full of edge cases. They replace it with workerpool and a 100-line wrapper. Net: -500 lines, fewer bugs, more familiar code. Win.

From workerpool to ants¶

Triggered by needing runtime resize, typed args, or higher throughput. Migration is mechanical (Appendix W in senior file). Effort: a day. Gain: configurability.

From workerpool to tunny¶

Triggered by needing per-worker state. Migration is significant — the API shape is different. Effort: a week. Gain: stateful workers.

From workerpool to errgroup.SetLimit¶

Triggered by realizing you didn't actually need a pool — just bounded concurrency for one batch. Migration is trivial. Effort: an hour. Gain: less infrastructure.

From ants to workerpool¶

Less common but happens. Usually because the team realised they were using only the simple part of ants's API and wanted simpler. Effort: a day. Gain: smaller surface area.

The lesson: migrations are easy when libraries share semantics. Hard when they differ. Stay flexible about which library you use; what you write inside the tasks is what matters.

Failure Modes and Defences¶

A defensive checklist for any production pool.

Each row corresponds to an incident pattern we have seen. Implementing every defence may be overkill for your service; implementing none is reckless. Pick based on stakes.

Pool as Part of System Architecture¶

A pool is one component in a larger system. Its place in the architecture affects how you treat it.

Pool inside a request hot path¶

Used for background fan-out from a synchronous handler. The handler responds before the pool tasks complete.

Design:

• Pool is shared, package-scope.
• Submission is fire-and-forget.
• Tasks are idempotent (retry-safe).
• A separate process or job catches up on failed tasks.

Pool for batch processing¶

Used for periodic work — a cron-like job that processes a queue.

Design:

• Pool is constructed at job start, drained at job end.
• Tasks include their own error handling.
• A retry layer wraps failed tasks for the next run.

Pool inside a pipeline¶

Used as one stage among several.

Design:

• Each stage has its own pool.
• Inter-stage hand-off via bounded channels (so each stage has backpressure to upstream).
• Drain stages in order during shutdown.

Pool inside a library¶

Used by a library to manage its own concurrency without exposing the pool to callers.

Design:

• The library should accept a pool or pool size as a configuration parameter.
• Default to a sensible size based on common workloads.
• Document the resource usage so callers can plan.

Multi-Tenant Isolation¶

A pool serving multiple tenants is a target. One noisy tenant can:

• Saturate the pool with their tasks.
• Hold workers via slow downstreams.
• Eat queue capacity.

Mitigations:

1. Per-tenant pool¶

A separate workerpool.WorkerPool per tenant. Total resource use scales with tenant count; only practical for tens of tenants.

2. Per-tenant token bucket¶

One shared pool, but each tenant has an in-flight cap.

type TenantTokens struct {
    mu     sync.Mutex
    bucket map[string]int
    cap    int
}

func (tt *TenantTokens) Acquire(tenant string) bool {
    tt.mu.Lock()
    defer tt.mu.Unlock()
    if tt.bucket[tenant] >= tt.cap {
        return false
    }
    tt.bucket[tenant]++
    return true
}

func (tt *TenantTokens) Release(tenant string) {
    tt.mu.Lock()
    defer tt.mu.Unlock()
    tt.bucket[tenant]--
}

Submit only if Acquire succeeds.

3. Per-tenant queue cap¶

Track the number of queued tasks per tenant. Reject new submits for tenants over their cap.

4. Pool sharding by tenant hash¶

tenantID % shards selects the pool. Total pools = shards. Tenants share a pool but cannot stalk each other (they get different shards via hashing).

This bounds the worst case: a noisy tenant only affects others on the same shard.

Choosing among these¶

For 10 tenants: per-tenant pool. For 1,000 tenants: per-tenant token bucket + shared pool. For 10,000 tenants: pool sharding + per-tenant tokens within each shard. For 100,000+ tenants: don't run them in one process. Shard at the service level.

Cost Accounting¶

Pools consume resources. Production engineers should be able to answer "how much does this pool cost us?"

Memory cost¶

• Per-pool: ~5-50 KB.
• Per-worker: ~2-8 KB stack.
• Per-queue-entry: variable (closure size).
• Total: 5 KB + maxWorkers * 4 KB + queueSize * closureSize.

For typical values (50 workers, 100 closures of 200 bytes): 5 + 200 + 20 = 225 KB. Negligible.

For pathological values (50 workers, 100,000 closures of 2 KB): 5 + 200 + 200,000 = 200 MB. Real.

CPU cost¶

• Per submit: ~200 ns.
• Per task: dominated by the task itself.
• Dispatcher: idle CPU (just parked in select).

For a pool doing 10,000 submits/second: 10,000 * 200ns = 2 ms/second = 0.2% of one core. Negligible.

For a pool doing 1,000,000 submits/second: 20% of one core. Significant. Consider sharding or ants.

Downstream cost¶

If each task makes a downstream call:

• 50 workers × 1 call per task × per-call cost.
• For a paid API at $0.001/call and a million tasks/day: $1,000/day. Significant.

Optimisation: cache, batch, deduplicate.

Engineering cost¶

The wrapper code. Metrics dashboards. Alerts. Documentation. Code reviews.

Per pool, expect 100-300 lines of wrapper + ~50 lines of dashboard YAML + ~50 lines of alert rules + 1 day of engineer time per quarter to maintain.

For a service with 5 pools, that's a tangible engineering investment. Worth it for production-grade services.

Performance Engineering¶

When a pool's performance becomes a bottleneck, here are the steps.

Step 1: Measure¶

go test -bench=Submit -benchmem
go test -bench=Submit -cpuprofile cpu.prof
go test -bench=Submit -memprofile mem.prof

Identify what is slow: submit overhead, task execution, GC pressure?

Step 2: Reduce closure allocation¶

Per-submit allocation is the most common scalable cost. Reduce by:

• Smaller captured state (pass references not values).
• Pre-allocate captured state outside the loop.
• Use ants.PoolWithFunc to avoid closures entirely.

Step 3: Reduce per-task overhead¶

For very small tasks (sub-microsecond), the dispatcher hop dominates. Options:

• Batch tasks: submit a closure that does N items.
• Inline: don't use a pool for trivial tasks.
• Move to ants (single channel hop instead of two).

Step 4: Reduce contention¶

If many goroutines submit simultaneously:

• Shard pools (multiple smaller pools, hash to shard).
• Buffer submissions in a per-goroutine local channel, batch-drain to pool.

Step 5: Reduce GC pressure¶

If GC takes >10% of CPU:

• Audit closure capture sizes.
• Use sync.Pool for reused objects.
• Tune GOGC.

Step 6: Benchmark again¶

before: 200 ns/op  16 B/op  1 alloc
after:  100 ns/op   0 B/op  0 alloc

Document the win. Add a benchmark to your test suite so regressions are caught.

When not to performance-engineer¶

If your pool does 100 RPS and tasks take 100 ms each, the pool's overhead is 0.0002% of total. Optimising the pool is a waste; optimise the task.

90% of pool performance problems are misallocated effort. Measure first.

Tricky Production Points¶

Pool size and GOMAXPROCS¶

If a container is run with GOMAXPROCS=1 (some sidecar configurations do this), a CPU-bound pool with maxWorkers > 1 gains nothing. Always log runtime.GOMAXPROCS(0) at startup.

Pool size and Kubernetes CPU limits¶

K8s resources.limits.cpu doesn't change GOMAXPROCS automatically (unless you use the automaxprocs library). A pod with limits.cpu: 2 and GOMAXPROCS=32 will be throttled. Use go.uber.org/automaxprocs:

import _ "go.uber.org/automaxprocs"

This sets GOMAXPROCS to match the CPU limit.

Stale workers after deploy¶

If you deploy a new version while old workers are running tasks, the old tasks continue running on old code. Tasks must be backward-compatible across deploys.

Memory leaks from forgotten pools¶

pprof goroutine is the diagnostic. If you see many workerpool.dispatch goroutines, you have leaked pools.

Pool inside test code¶

func TestSomething(t *testing.T) {
    pool := workerpool.New(4)
    defer pool.StopWait()
    // ...
}

defer runs at the end of the test. But if t.Parallel() is involved, the test may not actually finish before the next test starts. Use t.Cleanup for explicit ordering.

Race detector slowness with pools¶

-race adds ~10x latency to channel operations. A pool that handles 1M tasks/sec in production might do 100K under race. Tests that depend on rate may need adjustment.

Pool that survives a panic in main¶

If main() panics and defer pool.StopWait() was set up, the defer runs. Pool drains. But the panic still propagates. Useful for crash logging — the drain happens, then the panic continues.

Multiple panics in tasks¶

Each task panic is recovered independently. The library does not aggregate or sample. If many tasks panic, you get many log entries. Consider a panic-rate metric to alert on bursts.

Task that calls os.Exit¶

Bypasses recover. Pool, program, everything dies. Don't call os.Exit from inside a pool task unless that's exactly your intent.

Task that calls runtime.LockOSThread¶

The worker goroutine now owns an OS thread until UnlockOSThread. Other workers are unaffected, but the worker cannot be reaped while locked (it must return from the task to be reaped). Be careful: long-running locked tasks pin a thread per task.

Test¶

A production-style test for a pool wrapper.

func TestPool_Submit_NormalFlow(t *testing.T) {
    p := New(Options{
        Name: "test", MaxWorkers: 4, QueueCap: 100,
        Logger: slog.Default(),
        Registerer: prometheus.NewRegistry(),
    })
    defer p.Stop(5 * time.Second)

    var counter int64
    for i := 0; i < 100; i++ {
        err := p.Submit(context.Background(), "increment", func(ctx context.Context) error {
            atomic.AddInt64(&counter, 1)
            return nil
        })
        if err != nil {
            t.Fatalf("submit: %v", err)
        }
    }
    p.Stop(5 * time.Second)
    if atomic.LoadInt64(&counter) != 100 {
        t.Fatalf("counter = %d, want 100", counter)
    }
}

func TestPool_FullQueue(t *testing.T) {
    p := New(Options{
        Name: "test", MaxWorkers: 1, QueueCap: 5,
        Logger: slog.Default(),
        Registerer: prometheus.NewRegistry(),
    })
    defer p.Stop(5 * time.Second)

    // Block the only worker.
    block := make(chan struct{})
    err := p.Submit(context.Background(), "blocker", func(ctx context.Context) error {
        <-block
        return nil
    })
    if err != nil {
        t.Fatal(err)
    }

    // Fill the queue cap.
    for i := 0; i < 5; i++ {
        err := p.Submit(context.Background(), "task", func(ctx context.Context) error {
            return nil
        })
        if err != nil {
            t.Fatalf("submit %d: %v", i, err)
        }
    }

    // Next submit should be rejected.
    err = p.Submit(context.Background(), "overflow", func(ctx context.Context) error {
        return nil
    })
    if err == nil {
        t.Fatal("expected overflow rejection")
    }

    close(block)
}

func TestPool_Panic(t *testing.T) {
    var loggedPanic atomic.Bool
    logger := slog.New(testLogHandler(func(rec slog.Record) {
        if rec.Message == "task panic" {
            loggedPanic.Store(true)
        }
    }))

    p := New(Options{
        Name: "test", MaxWorkers: 1, QueueCap: 10,
        Logger: logger,
        Registerer: prometheus.NewRegistry(),
    })
    defer p.Stop(5 * time.Second)

    p.Submit(context.Background(), "panicker", func(ctx context.Context) error {
        panic("boom")
    })
    p.Stop(5 * time.Second)
    if !loggedPanic.Load() {
        t.Fatal("panic not logged")
    }
}

func TestPool_StopDeadline(t *testing.T) {
    p := New(Options{
        Name: "test", MaxWorkers: 1, QueueCap: 100,
        Logger: slog.Default(),
        Registerer: prometheus.NewRegistry(),
    })

    block := make(chan struct{})
    p.Submit(context.Background(), "slow", func(ctx context.Context) error {
        <-block
        return nil
    })

    start := time.Now()
    dropped := p.Stop(100 * time.Millisecond)
    elapsed := time.Since(start)

    if elapsed > 200*time.Millisecond {
        t.Fatalf("Stop didn't honour deadline: took %s", elapsed)
    }
    _ = dropped
    close(block)
}

These tests exercise normal flow, full queue, panic recovery, and shutdown deadline — the four bedrock contracts of a production pool wrapper.

Cheat Sheet¶

Sizing:
  CPU-bound:    maxWorkers = NumCPU
  I/O-bound:    maxWorkers = avg_latency * target_rps, capped by downstream
  Mixed:        size to I/O dimension

Always have:
  - Bounded queue (semaphore)
  - Per-task timeout (context)
  - Panic recovery (defer recover)
  - Shutdown deadline (drain in goroutine, select on timer)

Metrics (minimum 6):
  - submitted_total (counter)
  - completed_total (counter, labels: ok/panic/error)
  - queue_depth (gauge)
  - task_duration_seconds (histogram)
  - queue_dwell_seconds (histogram)
  - dropped_total (counter)

Alerts (minimum 5):
  - queue_depth high for 5m
  - panic rate elevated
  - drop rate non-zero
  - p99 task duration spike
  - pool stopped unexpectedly

K8s lifecycle:
  - preStop: sleep 5s (LB drain)
  - SIGTERM: server.Shutdown(ctx) then pool.Stop(deadline)
  - Readiness: 503 once pool.Stopped()

Cost:
  - Memory: ~5 KB + maxWorkers*4 KB + queueSize*closureSize
  - CPU per submit: ~200 ns
  - Dispatcher: idle

Memorise this; refer back as needed.

Self-Assessment Checklist¶

• I can derive maxWorkers from workload latency and target throughput.
• I can list at least 6 metrics every production pool should emit.
• I can write a graceful shutdown with a deadline + hard-stop fallback.
• I can describe the K8s lifecycle integration for a pool.
• I can spot the four classic incident patterns (hung downstream, unbounded queue, cascading slowdown, reaping surprise) from symptoms.
• I can build a production wrapper with metrics, logging, recovery, and bounded queue.
• I can plan capacity from observed metrics (mean, peak, latency).
• I can explain when to drain vs hard-stop.
• I can identify pool anti-patterns in PR review.
• I can decide when to wrap workerpool vs migrate to ants.

If all yes, you have professional-level fluency with workerpool and pool-based architectures in general.

Summary¶

gammazero/workerpool is a small library; using it in production is a substantial discipline. The library itself does almost nothing wrong; almost every pool-related production incident comes from how it was wrapped, sized, observed, or shut down.

The professional engineer's job is to bring guards, observability, and operational wisdom around a small, trustable primitive. Six items to take from this file:

1. Size deliberately. Math, not vibes.
2. Bound the queue. Always, if the producer can outrun the consumer.
3. Wrap with metrics. A pool you cannot see is a pool you cannot operate.
4. Drain with deadline. Bound your shutdown time.
5. Recover panics with attribution. Silent panics are silent bugs.
6. Learn from incidents. Each story above is a real lesson; collect your own.

The next files — specification, interview, tasks, find-bug, optimize — are practical. They sharpen specific skills. The professional file is the philosophy underlying them.

What You Can Build¶

At professional level you can build:

• A production-grade background-task service for any company.
• A pool wrapper that survives audits.
• Capacity plans, alerts, dashboards.
• Migrations between pool libraries.
• Pool-aware shutdown stories that meet SLAs.

You can also lead reviews of teammates' pool code with authority.

Further Reading¶

• The library's source (Senior file).
• Brendan Gregg, "Systems Performance" — for capacity planning.
• "Site Reliability Engineering" by Google — incident response and post-mortems.
• "The Twelve-Factor App" — the "concurrency" factor specifically.
• Real post-mortems on GitHub for "panjf2000/ants" or "workerpool" issues — lessons from others.

Related Topics¶

• Rate limiting (golang.org/x/time/rate) — companion to pools.
• Circuit breakers (sony/gobreaker) — protect against bad downstreams.
• Graceful shutdown patterns — broadly applicable.
• Backpressure theory — Little's Law and friends.
• Observability stacks — Prometheus, OpenTelemetry, Grafana.

Diagrams and Visual Aids¶

A complete production architecture¶

                    HTTP request
                          │
                          ▼
                  ┌──────────────┐
                  │ http server  │
                  └──────┬───────┘
                         │
              ┌──────────┴──────────┐
              │                     │
              ▼                     ▼
        respond OK             pool.Submit(ctx, ...)
                                      │
                                      ▼
                             ┌─────────────────┐
                             │  Pool wrapper   │
                             │ - sem cap 1000  │
                             │ - metrics       │
                             │ - panic recover │
                             └────────┬────────┘
                                      │
                                      ▼
                            ┌─────────────────┐
                            │  workerpool     │
                            │  maxW = 50      │
                            └────────┬────────┘
                                      │
                                      ▼
                            ┌─────────────────┐
                            │ worker goroutine│
                            │ - rate limit    │
                            │ - context check │
                            │ - downstream call│
                            └────────┬────────┘
                                      │
                                      ▼
                                 downstream

Observe: there is a lot more around the pool than the pool itself. Production engineering is mostly the surrounding code.

Drain timeline¶

T=0    SIGTERM received
T=1s   k8s preStop completes, LB stopped routing
T=1s   server.Shutdown() called
T=4s   server done (existing connections finished)
T=4s   pool.Stop(30s) called
T=4-32s   pool draining
T=32s  if not done, hard stop
T=33s  goroutine count returns to baseline
T=33s  os.Exit(0)

Total budget: ~30-35 seconds. Within K8s's default terminationGracePeriodSeconds: 30. Tune your defaults.

Queue depth diagnostic flowchart¶

queue_depth > threshold?
  └─ no → healthy
  └─ yes:
       is it growing?
         └─ no → temporary backpressure, monitor
         └─ yes:
              is downstream healthy?
                └─ no → fix downstream, pool will catch up
                └─ yes:
                     are workers maxed out?
                       └─ yes → size up maxWorkers
                       └─ no → investigate worker stuck states

This kind of flowchart, written down and shared with the team, makes on-call faster.

Multi-pool architecture¶

┌────────────────────────────────────┐
│              Service               │
│                                    │
│  ┌──────────┐  ┌──────────┐       │
│  │ CPU pool │  │ I/O pool │       │
│  │   N=8    │  │  N=128   │       │
│  └────┬─────┘  └────┬─────┘       │
│       │             │              │
│  ┌────▼─────────────▼─────┐       │
│  │ shared infrastructure  │       │
│  │ - logger              │       │
│  │ - metrics registry    │       │
│  │ - shutdown coordinator│       │
│  └────────────────────────┘       │
└────────────────────────────────────┘

Two pools, shared infrastructure, coordinated shutdown. Standard pattern for mixed-workload services.

These visualisations should accompany code in your real systems' documentation. Spending an hour on diagrams pays back tenfold in on-call clarity.

Appendix A: Pool Performance Tuning Cookbook¶

A step-by-step recipe for optimising a pool in production.

Phase 1: Establish a baseline¶

Run the service under normal load for at least 24 hours. Collect:

• Pool metrics (submitted/sec, completed/sec, queue depth, task duration).
• Resource metrics (CPU, memory, goroutine count).
• Downstream metrics (latency, error rate).

Note the steady-state values. These are your "normal".

Phase 2: Identify the bottleneck¶

Two questions:

1. Are queue depth and dwell time growing over time? → Producer outrunning consumer. Either:
2. Increase maxWorkers if downstream allows.

3. Slow producer (rate limit, drop).

4. Is queue stable but tasks are slow? → Task itself is the bottleneck. Profile the task code.

Phase 3: Apply a targeted fix¶

For producer-outrunning-consumer:

• Try increasing maxWorkers in steps (e.g., 50 → 75 → 100). Watch:
• Queue depth (should decrease).
• Downstream latency (should not degrade significantly).

• Downstream error rate (should not increase).

• If downstream degrades, stop increasing. You found the ceiling.

For slow tasks:

• Profile the task. CPU? I/O? Allocations?
• Optimise the hot path.
• Re-measure.

Phase 4: Validate¶

After the change, run for another 24 hours. Compare new metrics to baseline. Verify:

• Queue depth lower or stable.
• No new errors.
• Memory usage stable.
• Downstream healthy.

Phase 5: Document¶

Add a comment near the pool's New call:

// maxWorkers = 75: increased from 50 on 2024-03-15 after queue depth grew during peaks.
// Validated against downstream limit (100 concurrent allowed).
const taskPoolSize = 75

The next engineer (you, in 6 months) will thank you.

Appendix B: Pool-Related On-Call Runbook¶

A typical incident runbook for pool issues.

Symptom: "Service is slow"¶

1. Check pool metrics dashboard.
2. Is queue depth elevated?
3. Yes → backpressure issue. Continue.

4. No → not a pool issue; look elsewhere.

5. Is dwell time elevated?

6. Yes → tasks are queued for long. Capacity issue.

7. No → tasks are running but tasks themselves are slow. Profile.

8. Is downstream latency elevated?

9. Yes → root cause is downstream. Escalate to downstream team.

10. No → continue.

11. Is worker count maxed?

12. Yes → consider increasing maxWorkers (hot fix: ramp up via config).
13. No → investigate why workers aren't busy (deadlock? blocked?).

Symptom: "Memory growing"¶

1. Check pool metrics. Queue depth?
2. Growing → unbounded queue issue. Hot fix: restart, then add bound.

3. Stable → not pool memory; look at heap profile.

4. Heap profile: is workerpool or deque in top allocators?

5. Yes → pool memory issue.
6. No → other allocator; look elsewhere.

Symptom: "Service won't shut down"¶

1. Look at goroutine dump.
2. Are workers stuck on I/O?
3. Yes → tasks have no timeout. Hot fix: SIGKILL. Long fix: add timeouts.

4. No → continue.

5. Are workers stuck on locks?

6. Yes → deadlock in user code.

7. No → continue.

8. Is dispatcher exited?

9. Yes → workers should follow shortly.
10. No → dispatcher is hung; library bug or pause not cancelled.

Symptom: "Tasks dropped"¶

1. Check dropped_total metric.
2. Was queue cap hit?
3. Yes → undersized queue or producer too aggressive.

4. No → tasks dropped at Stop time.

5. Is this acceptable?

6. Yes → adjust monitoring threshold.
7. No → increase queue cap or implement durable retry.

A printed runbook by the on-call's desk pays for itself.

Appendix C: Sizing Calculator¶

Here is a small CLI tool that helps size a pool. Run it with workload parameters.

package main

import (
    "flag"
    "fmt"
)

func main() {
    targetRPS := flag.Float64("rps", 100, "target RPS")
    avgLatency := flag.Float64("latency", 0.1, "avg task latency in seconds")
    p99Latency := flag.Float64("p99", 0.5, "p99 task latency in seconds")
    downstreamCap := flag.Int("downstream", 100, "downstream concurrency cap")
    headroom := flag.Float64("headroom", 0.2, "headroom multiplier (0.2 = 20%)")
    flag.Parse()

    // Based on average
    fromAvg := *targetRPS * *avgLatency

    // Based on p99 (more conservative)
    fromP99 := *targetRPS * *p99Latency

    // Bound by downstream
    bounded := min(fromAvg, float64(*downstreamCap))
    boundedConservative := min(fromP99, float64(*downstreamCap))

    fmt.Printf("Recommendations:\n")
    fmt.Printf("  Aggressive (avg latency, no headroom):     %.0f workers\n", fromAvg)
    fmt.Printf("  Aggressive (avg latency, with headroom):   %.0f workers\n", fromAvg*(1+*headroom))
    fmt.Printf("  Conservative (p99 latency, no headroom):   %.0f workers\n", fromP99)
    fmt.Printf("  Conservative (p99 latency, with headroom): %.0f workers\n", fromP99*(1+*headroom))
    fmt.Printf("\nBounded by downstream cap (%d):\n", *downstreamCap)
    fmt.Printf("  Aggressive bounded:    %.0f\n", bounded)
    fmt.Printf("  Conservative bounded:  %.0f\n", boundedConservative)
    fmt.Printf("\nRecommendation: %d workers\n", int(boundedConservative*(1+*headroom)))
}

func min(a, b float64) float64 {
    if a < b {
        return a
    }
    return b
}

Run:

$ go run ./size-calc -rps 200 -latency 0.05 -p99 0.5 -downstream 100
Recommendations:
  Aggressive (avg latency, no headroom):     10 workers
  Aggressive (avg latency, with headroom):   12 workers
  Conservative (p99 latency, no headroom):   100 workers
  Conservative (p99 latency, with headroom): 120 workers

Bounded by downstream cap (100):
  Aggressive bounded:    10
  Conservative bounded:  100

Recommendation: 120 workers