ants — Professional Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Capacity Planning
5. Multi-Tenant Pools
6. Observability
7. Error & Panic Reporting
8. Integration with errgroup
9. Integration with context
10. Graceful Shutdown
11. Load Shedding & Backpressure
12. Rate Limiting in Front of the Pool
13. Tracing & Distributed Context
14. Resilience Patterns
15. Real-World Case Studies
16. Production Deployment Checklist
17. Testing Strategies
18. SLOs & Pool Metrics
19. Cost & Efficiency
20. Migration Stories
21. Common Mistakes at Scale
22. Anti-Patterns
23. Self-Assessment Checklist
24. Summary
25. Further Reading
26. Related Topics

Introduction¶

Focus: "I know how ants works. Now I need to ship it. How do I size pools in production, observe them, integrate them with errgroup, shut them down cleanly, and survive when things go wrong?"

You've made it through the API (junior.md), the configuration (middle.md), and the internals (senior.md). The remaining gap is the messiest one: real production. Real production cares about things the docs don't cover:

• A pool size that worked in load test doesn't work under real customer traffic.
• A panic in production is a P1 page, not a stderr line.
• Graceful shutdown must drain a pool, but also: respect service mesh deregistration, signal Kubernetes readiness, complete in-flight tasks, and not exceed the kubelet's grace period.
• Multi-tenant systems need fair sharing. One bad customer must not starve all others.
• Observability is non-negotiable. If you can't see the pool from Grafana, you can't operate it.

This file covers the operational reality. The examples are skeletons of real services. The numbers are realistic. The trade-offs are the ones you'll actually face.

By the end you will:

• Know how to size a pool for a specific workload, with measurement.
• Be able to design a multi-tenant pool topology with isolation.
• Know what metrics to export and what alerts to set.
• Know how to integrate ants with errgroup, context, and OpenTelemetry.
• Know how to perform a graceful shutdown that survives Kubernetes.
• Be able to handle load shedding, rate limiting, and circuit breaking.
• Know the production deployment checklist by heart.
• Recognise the antipatterns that get caught only at scale.

This is the longest file in the subsection. Some sections are long-form; others are concise checklists. Use it as a reference.

Prerequisites¶

• Comfortable with everything in junior.md, middle.md, senior.md.
• Familiar with production Go services — HTTP servers, gRPC, database access, monitoring.
• Familiar with Kubernetes pod lifecycle (signals, grace period, readiness probes).
• Familiar with observability stacks (Prometheus, Grafana, OpenTelemetry).
• Familiar with context.Context, errgroup, signal.Notify.
• Familiar with Go's pprof and runtime diagnostics.

If any of these is shaky, fix that first. The patterns here assume you can read Go code and Kubernetes manifests with fluency.

Glossary¶

Capacity Planning¶

The first job of running ants in production: pick the right Cap. Get this wrong and nothing else matters.

Step 1 — Identify the bottleneck¶

A pool caps concurrency. Concurrency is bounded by the slowest downstream you depend on. Examples:

• HTTP client calling a backend with a 100-concurrent-request limit → cap ≤ 100.
• Database with 50 connections in its pool → cap ≤ 50.
• CPU-bound work → cap ≈ GOMAXPROCS.
• Memory-bound work → cap = available_memory / per_task_memory.

Pick the most restrictive limit. Pool cap should be at or below that.

Step 2 — Calculate the throughput formula¶

Throughput = Cap / AverageTaskDuration.

If you target 1000 tasks/sec and task duration is 50 ms:

Cap = 1000 * 0.05 = 50

50-worker pool is the theoretical minimum. Real life adds jitter.

Step 3 — Add headroom¶

For burst absorption and jitter: 1.5x to 2x the minimum.

Cap = 50 * 1.5 = 75

Headroom isn't waste — it's insurance against tail latency spikes from slow tasks.

Step 4 — Validate with load test¶

Run a load test at your target rate. Observe:

• Running should hover around Cap * utilisation_target.
• Free should stay positive most of the time.
• p99 latency should meet SLO.

If Running == Cap constantly and latency exceeds SLO, the pool is too small. If Running is always ≪ Cap, the pool is over-sized.

Step 5 — Set autoscaling triggers¶

For Kubernetes HPA, scale pods (not pool cap) on CPU or queue depth. Pool cap stays fixed per pod. The number of pods scales.

For pool-level autoscaling (rare): poll Free and call Tune based on the ratio. We've covered this in middle.md.

Example — Capacity planning for an email service¶

Service receives email send requests. Each request triggers a SendGrid API call. SendGrid allows 600 req/min per key (10 req/sec).

• Bottleneck: SendGrid rate limit → cap ≤ 10 concurrent at theoretical max throughput.
• Average call duration: 200 ms.
• Throughput at cap 10: 10 / 0.2 = 50 ops/sec. Above SendGrid's 10/sec — wait, we're capped by their rate, not our cap.
• Re-evaluate: max throughput = 10 ops/sec.
• Required cap = 10 * 0.2 = 2 to saturate the rate limit.
• Plus headroom for slow calls: cap 4-8.

So: NewPool(8) with a separate rate limiter at 10 req/sec.

Example — Capacity planning for a CPU-bound batch job¶

Service processes 100 GB of CSVs daily. Each row needs a CPU-bound transformation.

• Bottleneck: CPU cores. Container has 4 cores.
• GOMAXPROCS = 4.
• Cap = 4. Higher wastes memory and increases contention.

So: NewPool(4) with single pod. Or multiple pods, each with NewPool(4), scaled horizontally.

Example — Capacity planning for a chat broadcast service¶

Service fans out a message to N online users via WebSocket. Per-user send takes ~5 ms.

• Bottleneck: WebSocket library's Write performance.
• Burst size: up to 100k recipients per message.
• p99 latency target: 200 ms total.

For 100k users at 5 ms each, sequentially: 500 seconds. We need concurrency.

If cap = 200, throughput = 200 / 0.005 = 40000 sends/sec. 100k users / 40000 = 2.5 seconds. Above SLO.

If cap = 2000, throughput = 400000 sends/sec. 100k / 400000 = 0.25 sec. Within SLO.

So: NewPool(2000). Memory cost: ~4 MB stack. Acceptable.

But watch FD limits — 2000 simultaneous Writes means 2000 sockets. Tune your OS.

Multi-Tenant Pools¶

If your service serves many customers, one customer's bad behaviour should not affect others. This is the bulkhead pattern.

Pattern 1 — One pool per tenant¶

type Service struct {
    mu    sync.RWMutex
    pools map[string]*ants.Pool
}

func (s *Service) Submit(tenant string, task func()) error {
    s.mu.RLock()
    p, ok := s.pools[tenant]
    s.mu.RUnlock()
    if !ok {
        s.mu.Lock()
        p, ok = s.pools[tenant]
        if !ok {
            p, _ = ants.NewPool(50, ants.WithNonblocking(true))
            s.pools[tenant] = p
        }
        s.mu.Unlock()
    }
    return p.Submit(task)
}

Pros: complete isolation. One slow tenant doesn't slow others.

Cons: many pools = many janitors. Per-tenant memory overhead.

For thousands of tenants, this is expensive. Use Pattern 2.

Pattern 2 — Tiered pools¶

Group tenants into tiers (free, paid, enterprise). One pool per tier.

type Service struct {
    free       *ants.Pool
    paid       *ants.Pool
    enterprise *ants.Pool
}

func (s *Service) Submit(tier Tier, task func()) error {
    switch tier {
    case Enterprise: return s.enterprise.Submit(task)
    case Paid:       return s.paid.Submit(task)
    default:         return s.free.Submit(task)
    }
}

Pros: fewer pools. Fair tier allocation.

Cons: not fair within a tier — a single free user can saturate the free pool.

For fair-within-tier, add a per-tenant rate limiter (golang.org/x/time/rate).

Pattern 3 — Sharded pools¶

Use MultiPool with custom load balancing that hashes by tenant:

type TenantStrategy struct{}

func (TenantStrategy) Pick(pools []*ants.Pool, hint interface{}) int {
    tenant, _ := hint.(string)
    h := fnv.New32a()
    h.Write([]byte(tenant))
    return int(h.Sum32()) % len(pools)
}

Each tenant always lands on the same shard. Tenants share shards but in a deterministic way.

Pros: O(1) tenant-to-pool mapping. Easy.

Cons: not strictly isolated — tenants on the same shard share fate.

Pattern 4 — Dynamic pool reclamation¶

For long-tail tenants, pools should be reclaimed when idle:

func (s *Service) janitor() {
    t := time.NewTicker(5 * time.Minute)
    for range t.C {
        s.mu.Lock()
        for tenant, p := range s.pools {
            if p.Running() == 0 && time.Since(lastUsed[tenant]) > 30*time.Minute {
                go p.ReleaseTimeout(10 * time.Second)
                delete(s.pools, tenant)
            }
        }
        s.mu.Unlock()
    }
}

Reclaim pools for inactive tenants. Saves memory in long-tail scenarios.

When to use which¶

• Few high-value tenants → one pool per tenant.
• Many tenants in a few tiers → tiered pools.
• Very many tenants, all small → sharded pools.
• Long tail of inactive tenants → dynamic reclamation.

Observability¶

If you cannot graph it, you cannot operate it. ants doesn't ship metrics — you wire them up.

Metrics to export¶

For each pool in your service:

• pool_cap (gauge) — current cap. Spike means Tune happened.
• pool_running (gauge) — current running. Track utilisation.
• pool_free (gauge) — current free slots.
• pool_waiting (gauge) — blocked submitters. Spike means saturation.
• pool_submit_total{result} (counter) — labelled by ok / overload / closed.
• pool_panic_total (counter) — panic count.
• pool_task_duration_seconds (histogram) — per-task duration. Optional but useful.

Prometheus example¶

import (
    "github.com/prometheus/client_golang/prometheus"
)

var (
    poolCap = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "ants_pool_cap",
        Help: "Pool capacity.",
    }, []string{"name"})

    poolRunning = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "ants_pool_running",
        Help: "Workers currently running tasks.",
    }, []string{"name"})

    poolFree = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "ants_pool_free",
        Help: "Workers free.",
    }, []string{"name"})

    poolWaiting = prometheus.NewGaugeVec(prometheus.GaugeOpts{
        Name: "ants_pool_waiting",
        Help: "Blocked submitters.",
    }, []string{"name"})

    poolSubmitTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "ants_pool_submit_total",
        Help: "Total submits.",
    }, []string{"name", "result"})

    poolPanicTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "ants_pool_panic_total",
        Help: "Total panics in pool tasks.",
    }, []string{"name"})
)

func init() {
    prometheus.MustRegister(poolCap, poolRunning, poolFree, poolWaiting, poolSubmitTotal, poolPanicTotal)
}

Polling loop¶

func (s *Service) startMetricsLoop(ctx context.Context) {
    t := time.NewTicker(5 * time.Second)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done(): return
        case <-t.C:
            poolCap.WithLabelValues("notify").Set(float64(s.pool.Cap()))
            poolRunning.WithLabelValues("notify").Set(float64(s.pool.Running()))
            poolFree.WithLabelValues("notify").Set(float64(s.pool.Free()))
            poolWaiting.WithLabelValues("notify").Set(float64(s.pool.Waiting()))
        }
    }
}

Wrapper for submit metrics¶

func (s *Service) Submit(task func()) error {
    err := s.pool.Submit(task)
    switch {
    case err == nil:
        poolSubmitTotal.WithLabelValues("notify", "ok").Inc()
    case errors.Is(err, ants.ErrPoolOverload):
        poolSubmitTotal.WithLabelValues("notify", "overload").Inc()
    case errors.Is(err, ants.ErrPoolClosed):
        poolSubmitTotal.WithLabelValues("notify", "closed").Inc()
    }
    return err
}

Panic handler metric¶

func panicHandler(p interface{}) {
    poolPanicTotal.WithLabelValues("notify").Inc()
    log.Errorf("pool panic: %v\n%s", p, debug.Stack())
}

Grafana dashboard sketch¶

For each pool:

• Utilisation: pool_running / pool_cap as a percentage. Alert if >90% for 5 min.
• Saturation: pool_waiting. Alert if >0 sustained.
• Submit rate: rate(pool_submit_total[1m]). Alert if drops to zero unexpectedly.
• Overload rate: rate(pool_submit_total{result="overload"}[1m]). Alert if non-zero.
• Panic rate: rate(pool_panic_total[5m]). Alert if >0.

Alerts¶

• PoolSaturated: pool_running == pool_cap for >5 min.
• PoolWaiting: pool_waiting > 0 for >2 min.
• PoolOverloaded: rate(pool_submit_total{result="overload"}[5m]) > 0.
• PoolPanic: increase(pool_panic_total[5m]) > 0.

Each alert has a clear meaning and runbook.

Error & Panic Reporting¶

Panics in production are events to be detected, logged, deduplicated, and alerted on.

Capturing panics¶

In the panic handler:

func panicHandler(p interface{}) {
    stack := debug.Stack()

    // Increment metric.
    poolPanicTotal.WithLabelValues("notify").Inc()

    // Capture to Sentry / Honeybadger / etc.
    sentry.CaptureException(fmt.Errorf("pool panic: %v", p))

    // Log with structured logging.
    log.WithFields(log.Fields{
        "event": "pool.panic",
        "value": fmt.Sprintf("%v", p),
        "stack": string(stack),
    }).Error("pool panic")
}

Deduplication¶

Panics with the same stack are usually the same bug. Use Sentry's fingerprinting or your error tracker's grouping.

Severity¶

• One-off panic: Likely a transient input bug. Log, count, move on.
• Sustained panic rate: Real bug. Page on-call.

Set thresholds:

panic_rate > 1/sec for 5 min -> page
panic_rate > 0.1/sec for 30 min -> ticket

Don't crash on panic¶

The pool's recover prevents crashes. Don't undo that by calling os.Exit in the handler — except for truly unrecoverable conditions like OOM.

Integration with errgroup¶

errgroup provides first-error short-circuit and context cancellation. ants provides worker reuse. They compose well.

Pattern 1 — errgroup over ants¶

g, ctx := errgroup.WithContext(parent)
for _, item := range items {
    item := item
    g.Go(func() error {
        done := make(chan error, 1)
        err := pool.Submit(func() {
            done <- process(ctx, item)
        })
        if err != nil { return err }
        return <-done
    })
}
return g.Wait()

Each g.Go spawns a goroutine that submits to the pool and waits for the result via a channel. The errgroup provides cancellation.

Trade-offs: - Pro: errgroup semantics (cancel-all-on-error). - Pro: bounded by errgroup's SetLimit if you want it. - Con: one extra goroutine per task (the g.Go wrapper).

For high task rate, the extra goroutine adds 10-50% overhead. For lower rate, irrelevant.

Pattern 2 — errgroup as concurrency cap, ants as worker farm¶

g, ctx := errgroup.WithContext(parent)
g.SetLimit(50) // cap submitters
for _, item := range items {
    item := item
    g.Go(func() error {
        // errgroup limits to 50 concurrent g.Go bodies
        errCh := make(chan error, 1)
        err := pool.Submit(func() {
            errCh <- process(ctx, item)
        })
        if err != nil { return err }
        return <-errCh
    })
}
return g.Wait()

errgroup.SetLimit(50) ensures at most 50 producers wait on the pool. The pool handles the actual work. Useful when you don't want unbounded submitter goroutines.

Pattern 3 — Direct submit with collected errors¶

errs := make([]error, len(items))
var wg sync.WaitGroup
for i, item := range items {
    i, item := i, item
    wg.Add(1)
    _ = pool.Submit(func() {
        defer wg.Done()
        errs[i] = process(ctx, item)
    })
}
wg.Wait()
return errors.Join(errs...)

No errgroup, no extra goroutine per task. Trade-off: no early-cancel on first error. All tasks run to completion.

When to choose which¶

• Need first-error cancellation: errgroup (pattern 1 or 2).
• Need all errors: direct (pattern 3).
• Performance-critical with low task count: errgroup pattern 1.
• Performance-critical with high task count: direct pattern 3 or non-errgroup with manual cancel.

Integration with context¶

ants.Pool does not accept context.Context. You add it.

Pattern 1 — Context in the task¶

_ = pool.Submit(func() {
    if ctx.Err() != nil { return }
    process(ctx, work)
})

Task checks context on entry. If already cancelled, return immediately.

Pattern 2 — Context wrapper¶

type ContextPool struct {
    p *ants.Pool
}

func (c *ContextPool) Submit(ctx context.Context, task func(context.Context)) error {
    return c.p.Submit(func() {
        if ctx.Err() != nil { return }
        task(ctx)
    })
}

Caller passes context naturally.

Pattern 3 — Deadline-aware submit¶

func submitWithDeadline(ctx context.Context, p *ants.Pool, task func(context.Context)) error {
    // Check if context is already done before submitting.
    if err := ctx.Err(); err != nil { return err }
    return p.Submit(func() {
        // Inside, also derive a tighter context if needed.
        taskCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
        defer cancel()
        task(taskCtx)
    })
}

Pattern 4 — Cancel chain¶

// Service-level cancel.
sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGTERM)
go func() {
    <-sigs
    serviceCancel()
}()

// Request-level cancel derived from service.
ctx, cancel := context.WithTimeout(s.serviceCtx, 10*time.Second)
defer cancel()
return pool.Submit(func() {
    if ctx.Err() != nil { return }
    doWork(ctx)
})

The task's context is chained from service-level (SIGTERM cancels) to request-level (timeout cancels). Either cancels the task.

Anti-pattern — Background context inside tasks¶

_ = pool.Submit(func() {
    doWork(context.Background()) // ignores all cancellation
})

Tasks should not start with context.Background(). Pass it in from outside.

Graceful Shutdown¶

Shutting down a service that uses ants correctly is harder than it looks.

The goal¶

When the service receives SIGTERM:

1. Stop accepting new requests (HTTP server stops listening).
2. Drain in-flight requests (their tasks complete or time out).
3. Release pools.
4. Exit cleanly.

All within the Kubernetes grace period (default 30 seconds).

Naive shutdown¶

sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGTERM)
<-sigs
pool.Release()
os.Exit(0)

Problems:

• Release returns immediately; in-flight tasks may still run.
• os.Exit(0) doesn't run deferred functions.
• HTTP server may have requests in flight.

Better shutdown¶

sigs := make(chan os.Signal, 1)
signal.Notify(sigs, syscall.SIGTERM, syscall.SIGINT)
<-sigs

// 1. Stop accepting new traffic (drain HTTP).
shutdownCtx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
defer cancel()
if err := httpServer.Shutdown(shutdownCtx); err != nil {
    log.Printf("http shutdown: %v", err)
}

// 2. Cancel service-level context to stop in-flight tasks.
serviceCancel()

// 3. Drain pool.
if err := pool.ReleaseTimeout(20 * time.Second); err != nil {
    log.Printf("pool release: %v", err)
}

log.Println("shutdown complete")

Order matters:

1. HTTP first — no new requests.
2. Service context cancel — in-flight requests are notified.
3. Pool drain — wait for tasks to complete.

Total budget: 25 + 20 = 45 seconds, exceeds 30-second grace. Tighten:

shutdownCtx, _ := context.WithTimeout(context.Background(), 15*time.Second)
httpServer.Shutdown(shutdownCtx)
pool.ReleaseTimeout(10 * time.Second)

Or run them in parallel:

var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); httpServer.Shutdown(ctx) }()
go func() { defer wg.Done(); pool.ReleaseTimeout(20*time.Second) }()
wg.Wait()

Kubernetes considerations¶

• Preserve readiness: Before SIGTERM, k8s removes the pod from service endpoints. Use a preStop hook to drain even before the signal.
• Grace period: terminationGracePeriodSeconds: 30. If you need more, increase.
• PostStart / preStop: Use preStop to mark unready, sleep a few seconds for the iptables update, then send SIGTERM.

preStop hook example¶

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]

Five seconds for the iptables update to propagate. Then your process sees SIGTERM and shuts down.

Health checks during shutdown¶

After SIGTERM, return 503 from health check:

var shuttingDown atomic.Bool

http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
    if shuttingDown.Load() {
        http.Error(w, "shutting down", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(200)
})

// On SIGTERM:
shuttingDown.Store(true)

This way, the load balancer marks the pod unhealthy quickly.

Load Shedding & Backpressure¶

When demand exceeds capacity, you must shed load — refuse some requests to preserve quality for the rest.

Mechanism 1 — Pool non-blocking + 503¶

err := pool.Submit(task)
if errors.Is(err, ants.ErrPoolOverload) {
    w.WriteHeader(http.StatusServiceUnavailable)
    return
}

Direct: pool full → request rejected. Client retries with backoff.

Mechanism 2 — Token bucket in front¶

limiter := rate.NewLimiter(100, 50) // 100 ops/sec, burst 50

if !limiter.Allow() {
    w.WriteHeader(http.StatusTooManyRequests)
    return
}
err := pool.Submit(task)

Rate limiter before the pool. Predictable rate regardless of pool state.

Mechanism 3 — Adaptive concurrency¶

// Monitor pool latency. If p99 latency exceeds threshold, reduce admission rate.
if observedP99 > threshold {
    limiter.SetLimit(rate.Limit(limiter.Limit() * 0.8))
}

Self-tuning. Complex to get right; reach for libraries like aws-go-sdk's service package or Netflix's concurrency-limits port.

Mechanism 4 — Priority shedding¶

Shed low-priority work first:

if pool.Free() == 0 && priority == Low {
    return ErrTooBusy
}
err := pool.Submit(task)

Or use separate pools per priority.

Mechanism 5 — Queue-aware shedding¶

If you have a separate queue in front of the pool:

if queueDepth > threshold {
    return ErrTooBusy
}

The queue depth is your signal.

Don't let producers retry blindly¶

If the pool is shedding, callers should:

• Wait before retrying (exponential backoff).
• Honour Retry-After header.
• Eventually give up.

Without backoff, retries create a thundering herd that makes shedding worse.

Rate Limiting in Front of the Pool¶

ants caps concurrency, not rate. For rate limiting, add another layer.

Token bucket¶

import "golang.org/x/time/rate"

limiter := rate.NewLimiter(rate.Limit(100), 200) // 100 ops/sec, burst 200

if err := limiter.Wait(ctx); err != nil {
    return err
}
_ = pool.Submit(task)

limiter.Wait blocks until a token is available (or context cancels). Pairs with blocking pool.

Sliding window¶

For per-customer rate limits, use a sliding window counter (Redis with INCR + expiry). Check before submit.

Combining rate and concurrency¶

Two layers:

• Rate limiter (token bucket) enforces ops/sec.
• Pool (ants) enforces concurrency.

Sized together: rate * avg_duration ≈ concurrency. If rate is 100 ops/sec and tasks take 0.5 s, concurrency converges to ~50.

Don't size the pool for less than rate * avg_duration — the limiter will be the bottleneck and the pool will be underutilised. Don't size for much more — wasted capacity.

Tracing & Distributed Context¶

Production services use distributed tracing (OpenTelemetry). The trace context must propagate through ants.

Pattern — Tracing through Submit¶

func submitTraced(ctx context.Context, p *ants.Pool, name string, task func(context.Context)) error {
    tracer := otel.Tracer("notify-service")
    return p.Submit(func() {
        ctx, span := tracer.Start(ctx, name)
        defer span.End()
        task(ctx)
    })
}

The trace ID propagates from ctx to the task. The span starts when the worker picks up the task (not when Submit is called).

Span attributes¶

span.SetAttributes(
    attribute.String("pool.name", "notify"),
    attribute.Int("pool.cap", p.Cap()),
    attribute.Int("pool.running", p.Running()),
)

These help correlate slow tasks with pool saturation.

Tracing the wait¶

If your task waits in Submit (blocking mode), you can capture that:

ctx, submitSpan := tracer.Start(ctx, "pool.submit.wait")
err := p.Submit(func() { ... })
submitSpan.End()

If submitSpan duration is large, pool is saturated.

Async vs sync semantics¶

Submit is async — the task runs on a worker, possibly later. Your trace shows:

parent_span (HTTP request)
  ├── child_span (Submit returns immediately)
  └── child_span (worker runs task — possibly milliseconds later)

The async child span is detached from the parent's wall-clock duration. Some tracing systems handle this; others don't. Test your stack.

Linking spans¶

OpenTelemetry has "Span Links" for async work. Use them to correlate the submitter and the task:

link := trace.LinkFromContext(ctx)
go func() {
    ctx2, span := tracer.Start(context.Background(), "task", trace.WithLinks(link))
    defer span.End()
}()

Resilience Patterns¶

Beyond capacity and shutdown, resilience patterns protect against partial failures.

Pattern 1 — Bulkhead¶

Already covered (multi-tenant pools). One bulkhead per dependency or per tenant.

Pattern 2 — Circuit breaker¶

type CircuitBreaker struct {
    failures   int64
    threshold  int64
    resetAfter time.Duration
    openUntil  atomic.Value // time.Time
}

func (cb *CircuitBreaker) Call(f func() error) error {
    if until := cb.openUntil.Load(); until != nil {
        if t := until.(time.Time); time.Now().Before(t) {
            return errors.New("circuit open")
        }
    }
    err := f()
    if err != nil {
        if atomic.AddInt64(&cb.failures, 1) > cb.threshold {
            cb.openUntil.Store(time.Now().Add(cb.resetAfter))
            atomic.StoreInt64(&cb.failures, 0)
        }
    } else {
        atomic.StoreInt64(&cb.failures, 0)
    }
    return err
}

Wrap Submit (or the task itself) with the breaker. After N failures, the circuit opens and rejects calls for resetAfter duration.

For real production, use sony/gobreaker or similar — it handles half-open state, sliding windows, etc.

Pattern 3 — Timeout¶

Every task should have a timeout. Without one, a stuck task occupies a worker forever.

func runWithTimeout(ctx context.Context, d time.Duration, fn func(context.Context) error) error {
    ctx, cancel := context.WithTimeout(ctx, d)
    defer cancel()
    return fn(ctx)
}

Use inside the task:

_ = pool.Submit(func() {
    if err := runWithTimeout(ctx, 5*time.Second, doWork); err != nil {
        log.Warn(err)
    }
})

Pattern 4 — Retry with backoff¶

import "github.com/cenkalti/backoff/v4"

err := backoff.Retry(func() error {
    return submitAndWait(ctx, pool, task)
}, backoff.WithMaxRetries(backoff.NewExponentialBackOff(), 3))

Retry transient errors (ErrPoolOverload, network timeouts). Don't retry permanent errors (ErrPoolClosed, validation errors).

Pattern 5 — Hedging¶

For latency-sensitive reads, submit duplicate requests and use the first to respond:

result := make(chan int, 2)
_ = pool.Submit(func() { result <- backendA(req) })
_ = pool.Submit(func() { result <- backendB(req) })
return <-result // first wins

Doubles pool consumption. Use only for critical reads.

Pattern 6 — Fallback¶

If primary fails, try secondary:

err := pool.Submit(func() { primary() })
if err != nil {
    err = pool.Submit(func() { fallback() })
}

Or fall back inline if pool is overloaded:

if err := pool.Submit(task); err != nil {
    task() // run on caller
}

Inline fallback removes the pool's protection. Use sparingly.

Real-World Case Studies¶

Case Study 1 — Notification Service at a SaaS¶

Service: send push notifications, emails, SMS.

Architecture: - API endpoints accept requests. - Each notification type has its own pool. - Push: cap 1000 (matches APNS HTTP/2 concurrency). - Email: cap 100 (matches SES rate limit). - SMS: cap 50 (matches Twilio). - Each pool has WithNonblocking(true) and overflow to Redis queue.

Lessons learned: - Initial sizing was too small. Doubled after first month of load. - WithExpiryDuration(5*time.Minute) to keep workers warm — connection setup cost was high. - Panic handler integrated with Sentry. Caught early bugs in Twilio response parsing.

Case Study 2 — Image Processing at a Media Company¶

Service: thumbnail generation, format conversion.

Architecture: - Single pool, cap = 2 * GOMAXPROCS for image transforms. - CPU-bound, memory-heavy. - WithDisablePurge(true) — workers always busy. - Tasks read from S3, transform, write back.

Lessons learned: - Pool size much larger than GOMAXPROCS (2x) because tasks have I/O wait. Pure-CPU would have been 1x. - Memory cap (per task: 200 MB) forced careful sizing. 4 GB container = max 20 concurrent. Cap = 16 with headroom. - WithPanicHandler saved during a bad image causing libvips to panic.

Case Study 3 — Event Stream Processor¶

Service: consume Kafka events, transform, write to BigQuery.

Architecture: - MultiPool of 8 sub-pools, cap 100 each. - LeastTasks strategy. - Each task: transform one event, batch into BQ stream.

Lessons learned: - Single Pool was lock-contended. MultiPool reduced contention dramatically. - BQ streaming has its own rate limits. Pool cap matched to total quota / 8 shards. - Graceful shutdown: cancel Kafka consumer first, drain pool, then commit offsets.

Case Study 4 — WebSocket Hub¶

Service: broadcast messages to thousands of WebSocket clients.

Architecture: - One pool per channel/room. - Cap per pool = max recipients per room. - Inside each pool: one task per recipient.

Lessons learned: - Many small pools was costly (each has a janitor). Switched to one large pool with per-room rate limiting. - PoolWithFunc for the hot path. Argument: pointer to message + recipient ID. - Custom panic handler logged the message ID for replay.

Case Study 5 — Database Migration Runner¶

Service: run thousands of small migrations in parallel.

Architecture: - Pool cap = DB connection pool size - safety margin. - WithNonblocking(false) (default) — slow but reliable. - Tasks idempotent.

Lessons learned: - DB connection exhaustion was the killer. Pool cap was too high initially. - Tune down during heavy reads from the app to prioritise user traffic. - Tasks logged progress to a separate table for visibility.

Production Deployment Checklist¶

Before shipping ants to production:

Code¶

• defer pool.Release() (or ReleaseTimeout) on every NewPool.
• Submit errors always handled (logged or propagated).
• WithPanicHandler installed.
• Tasks accept and honour context.Context.
• Closures capture loop variables correctly.

Configuration¶

• Pool capacity is config-driven (not hard-coded).
• Default capacity has been load-tested.
• Tier-based pools for multi-tenant.
• Non-blocking + overflow strategy chosen.

Observability¶

• Metrics exported for Running, Free, Waiting, Cap.
• Submit success/failure counters.
• Panic counter.
• Dashboards in Grafana.
• Alerts on saturation and panic rate.

Resilience¶

• Tasks have per-task timeout.
• Downstream calls have circuit breakers.
• Retries with backoff for transient errors.
• Load shedding on overload.

Shutdown¶

• Signal handler installed.
• HTTP server drained before pool.
• ReleaseTimeout with grace period.
• preStop hook in k8s manifest.
• Readiness probe returns 503 during shutdown.

Testing¶

• Unit tests for tasks.
• Load tests against pool.
• Chaos tests: panic injection, pool release mid-flight.
• Profiling under load.

Operations¶

• Runbook for "pool saturated" alert.
• Runbook for "panic detected" alert.
• Documentation of pool sizes and rationale.
• On-call escalation path.

If you can tick all of these, you're production-ready.

Testing Strategies¶

Unit tests for tasks¶

The task is just a function. Test it directly, without the pool:

func TestProcessTask(t *testing.T) {
    err := processTask(ctx, input)
    require.NoError(t, err)
    // assert side effects
}

The pool itself is ants's responsibility. Don't test their library.

Integration tests for the wiring¶

Test that your service correctly uses the pool:

func TestServiceSubmitsToPool(t *testing.T) {
    svc := NewService(2)
    defer svc.Close(5 * time.Second)

    var processed int64
    for i := 0; i < 100; i++ {
        svc.Submit(func() { atomic.AddInt64(&processed, 1) })
    }
    require.Eventually(t, func() bool {
        return atomic.LoadInt64(&processed) == 100
    }, 10*time.Second, 10*time.Millisecond)
}

Load tests¶

Use a tool like hey, vegeta, or write a custom generator:

func TestPoolUnderLoad(t *testing.T) {
    svc := NewService(50)
    defer svc.Close(30 * time.Second)

    start := time.Now()
    var wg sync.WaitGroup
    for i := 0; i < 10000; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            svc.Submit(realisticTask)
        }()
    }
    wg.Wait()
    t.Logf("10000 ops in %v", time.Since(start))
}

Chaos tests¶

Inject failures during operation:

func TestChaosPanic(t *testing.T) {
    svc := NewService(10)
    defer svc.Close(5 * time.Second)

    // Submit panicking tasks
    for i := 0; i < 100; i++ {
        svc.Submit(func() { panic("test") })
    }

    // Service should still accept normal tasks
    var ok int64
    svc.Submit(func() { atomic.AddInt64(&ok, 1) })
    require.Eventually(t, func() bool {
        return atomic.LoadInt64(&ok) == 1
    }, time.Second, 10*time.Millisecond)
}

Benchmark¶

func BenchmarkSubmit(b *testing.B) {
    pool, _ := ants.NewPool(50)
    defer pool.Release()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _ = pool.Submit(func() {})
    }
}

Run regularly to catch regressions.

SLOs & Pool Metrics¶

If your service has SLOs, the pool is a key signal.

SLO: latency¶

Target: p99 latency < 200 ms.

Pool's contribution: - Time spent blocked in Submit if pool is full. - Time spent on the worker (task execution).

Mitigation: - Size pool so Waiting is rarely > 0. - Optimise task duration.

SLO: availability¶

Target: 99.9% of submits succeed.

Pool's contribution: - ErrPoolOverload reduces availability. - ErrPoolClosed during shutdowns reduces availability.

Mitigation: - Headroom in cap. - Graceful shutdown that completes within grace period.

SLO: throughput¶

Target: 1000 ops/sec sustained.

Pool's contribution: - Cap / AvgDuration is the upper bound.

Mitigation: - Increase cap or decrease task duration.

Error budget¶

If you have 99.9% availability SLO, that's 0.1% errors = 0.001 error rate. Over 1M ops, 1000 errors. Spend wisely: - Some during shutdowns. - Some on legitimate overload. - Some on bugs.

Track error budget burn via Prometheus.

Cost & Efficiency¶

CPU cost¶

Pool overhead per submit: ~100 ns. At 1M submits/sec: ~10% of a core. Worth it for the benefits.

Memory cost¶

Per worker: 2 KB minimum, grown to task's needs. Cap 1000 → ~2-50 MB stack. Plus per-pool struct ~5 KB.

GC pressure¶

Closures allocate. At high submit rate, GC pressure grows. PoolWithFunc mitigates.

Container sizing¶

Allocate memory for: - Heap (your data). - Stack (per goroutine). - Pool internal (~80 KB per pool). - Runtime (~30 MB baseline).

A 256 MB container with 1000 workers needs ~50 MB for stacks alone. Plan accordingly.

Bin packing¶

Multiple services per pod is risky — pool memory adds up. Prefer single-service pods unless you've measured the joint footprint.

Migration Stories¶

Teams migrating to ants typically come from:

From naked goroutines¶

for _, x := range items {
    go func(x int) {
        work(x)
    }(x)
}

To:

for _, x := range items {
    x := x
    _ = pool.Submit(func() {
        work(x)
    })
}

Migration is straightforward. Wins: bounded concurrency, recovery, observability.

From golang.org/x/sync/errgroup¶

g, _ := errgroup.WithContext(ctx)
g.SetLimit(50)
for _, x := range items {
    x := x
    g.Go(func() error { return work(x) })
}
return g.Wait()

To:

g, _ := errgroup.WithContext(ctx)
g.SetLimit(50)
for _, x := range items {
    x := x
    g.Go(func() error {
        errCh := make(chan error, 1)
        _ = pool.Submit(func() { errCh <- work(x) })
        return <-errCh
    })
}
return g.Wait()

More boilerplate but worker reuse. Win only at high task rate. Often, plain errgroup is fine.

From hand-rolled channel-based pools¶

Often the hand-rolled pool grows into a mess of options that mimic ants. Migrating to ants reduces code and gives you community-tested behaviour.

From other libraries (tunny, workerpool)¶

Similar shape, different APIs. Usually a 1-2 day migration including tests.

Common Mistakes at Scale¶

Mistakes that only show up at production scale.

Mistake 1 — Insufficient stress testing¶

Local tests pass. Production fails. Always stress-test at 2-3x expected load.

Mistake 2 — Forgetting downstream limits¶

Your pool cap is 1000 but downstream allows 100. You'll get hammered with rate-limit errors. Cap at the downstream's limit.

Mistake 3 — Shared pool across services¶

Service A's slow tasks starve Service B. Always isolate.

Mistake 4 — Misunderstanding "concurrent" vs "parallel"¶

A pool of 1000 doesn't mean 1000 things actually happening at once on the CPU. CPU parallelism is bounded by GOMAXPROCS. The other 996 are interleaved.

Mistake 5 — No backpressure to callers¶

Your service has unlimited blocking submitters. Callers don't know to back off. They retry harder, your pool gets worse. Use non-blocking mode and return 503.

Mistake 6 — Pool size = container CPU¶

For I/O work, this is way too small. For CPU work, this is right but rarely the bottleneck.

Mistake 7 — Ignoring Waiting¶

You watch Running and Free but not Waiting. Latency complaints arrive because submitters are queued — pool is full but you don't know.

Mistake 8 — Single pool for multi-tenant¶

One bad tenant DoSes everyone. Use per-tenant pools or rate limiters.

Mistake 9 — Pool never Released¶

Service restarts hourly, each instance leaks its pool. Memory creeps. Always defer release.

Mistake 10 — Tune in production without testing¶

Live tuning is dangerous. Always test in staging. Tune is atomic but the effect (workers spawning/dying) takes time.

Anti-Patterns¶

A consolidated list of anti-patterns at the professional level.

Anti 1 — Pool per request¶

Each request creates a new pool. Pool creation overhead dominates. Use a shared, long-lived pool.

Anti 2 — Pool for serialisation¶

Pool of cap 1 to "serialise tasks." That's a mutex. Use a mutex.

Anti 3 — Pool inside a tight CPU loop¶

Adding a pool inside a CPU-bound loop slows it down. Pools are for I/O concurrency or coarse-grained CPU parallelism.

Anti 4 — Pool for everything¶

One global pool for HTTP, DB, file I/O, image processing. Each contends with others. Split per workload.

Anti 5 — Ignoring shutdown signals¶

Service runs until SIGKILL. In-flight tasks lost. Pool leaks. Always have a signal handler.

Anti 6 — No metrics¶

You can't see the pool. You can't operate it. Always export metrics.

Anti 7 — Hard-coded cap¶

Cap baked into source. Can't tune without redeploy. Make it config-driven.

Anti 8 — Premature optimisation¶

Reaching for PoolWithFunc or MultiPool before measuring. Default Pool is fast enough for almost everyone.

Anti 9 — Naked panic recovery¶

ants.WithPanicHandler(func(p interface{}) {})

Empty handler. Hides bugs. Always at least log.

Anti 10 — Releasing without draining¶

pool.Release()
return

In-flight tasks may not complete. Use ReleaseTimeout.

Self-Assessment Checklist¶

You are professional-level when you can:

• Size a pool for a new workload, defending the chosen cap.
• Design a multi-tenant pool topology for your service.
• Wire up Prometheus metrics for every pool.
• Configure alerts that catch saturation, overload, and panics.
• Implement graceful shutdown that survives Kubernetes.
• Combine ants with errgroup and context.
• Identify the right load-shedding strategy for a given service.
• Plumb tracing through Submit.
• Stress-test a pool under realistic conditions.
• Read a service's pool config and predict its behaviour.
• Write a runbook for "pool saturated" alert.
• Defend the decision to use ants vs alternatives in a design review.

Summary¶

Production ants is more than a library; it's a discipline:

• Capacity is sized to the downstream, validated with load tests, made config-driven.
• Tenancy is isolated via bulkheads — per-customer pools, tiered pools, or sharded pools.
• Observability is non-negotiable — metrics, dashboards, alerts.
• Shutdown is orchestrated — HTTP drain, context cancel, pool drain, all within grace period.
• Resilience is layered — circuit breakers, timeouts, retries, fallbacks.
• Costs are accounted — CPU, memory, GC.

The library does the heavy lifting of worker reuse and concurrency capping. The hard work — knowing the right cap, integrating with everything else, surviving production — is yours.

If you've made it through all four levels (junior, middle, senior, professional), you can deploy ants with confidence. The remaining files (specification.md, interview.md, tasks.md, find-bug.md, optimize.md) are references and exercises.

Further Reading¶

• The Tencent gnet source (uses ants).
• "Site Reliability Engineering" by Google — chapters on overload and shedding.
• "Release It!" by Michael Nygard — stability patterns.
• The golang.org/x/sync package docs — errgroup, semaphore, singleflight.
• Brendan Gregg's blog on systems performance — relevant for load testing and profiling.
• OpenTelemetry Go documentation — for tracing.

Related Topics¶

• 12-graceful-shutdown — signal handling, draining, k8s lifecycle.
• 13-runtime-scheduler — GMP and how it interacts with pools.
• 18-errgroup — first-error semantics and concurrency cap.
• 19-semaphore — a lighter alternative for bounded concurrency.
• 20-singleflight — request coalescing.
• 25-observability — metrics, traces, logs.
• 30-production-go — broader production patterns.

Extended Section: Designing a Pool-Backed Service¶

To bring everything together, let's design a service from scratch.

Specification¶

A "Webhook Dispatch" service. Receive webhook events from internal systems; deliver them to customers' HTTP endpoints. Customers configure URL, retry policy, secret for signing. Volume: ~10k events/sec across all customers.

Capacity sizing¶

Each delivery is an outbound HTTP POST. Average duration: 200 ms (mostly network).

To handle 10k events/sec at 200 ms each: concurrency = 10000 * 0.2 = 2000.

Add 50% headroom: cap = 3000. But we need to consider: - File descriptor limits (default 1024 on Linux, need to raise). - Memory: ~3000 workers × ~10 KB stack = 30 MB. Fine. - Each task uses an HTTP client connection; need to size Transport.MaxIdleConnsPerHost.

Decision: pool cap 3000, ulimit -n 65535, transport tuned.

Tenancy¶

Customers can be free, paid, or enterprise. Free customers should not slurp all capacity.

Decision: three pools. - Free: cap 500, WithNonblocking(true). - Paid: cap 1500, blocking with WithMaxBlockingTasks(1000). - Enterprise: cap 1000, blocking with no max.

Overflow from free → Redis queue with hourly retry. Paid overflow → return 503 to the caller.

Code skeleton¶

package webhook

import (
    "context"
    "errors"
    "log"
    "net/http"
    "runtime/debug"
    "sync/atomic"
    "time"

    "github.com/panjf2000/ants/v2"
    "github.com/prometheus/client_golang/prometheus"
)

type Tier int

const (
    Free Tier = iota
    Paid
    Enterprise
)

type Service struct {
    freePool       *ants.Pool
    paidPool       *ants.Pool
    enterprisePool *ants.Pool

    client *http.Client
    ctx    context.Context
    cancel context.CancelFunc

    overflowQueue OverflowQueue

    // Metrics
    submits    *prometheus.CounterVec
    durations  *prometheus.HistogramVec
    pool stats *prometheus.GaugeVec
}

func New(ctx context.Context, q OverflowQueue) (*Service, error) {
    sCtx, cancel := context.WithCancel(ctx)
    s := &Service{
        ctx:           sCtx,
        cancel:        cancel,
        overflowQueue: q,
        client: &http.Client{
            Timeout: 30 * time.Second,
            Transport: &http.Transport{
                MaxIdleConns:        3000,
                MaxIdleConnsPerHost: 100,
                IdleConnTimeout:     90 * time.Second,
            },
        },
    }

    var err error
    s.freePool, err = ants.NewPool(500,
        ants.WithExpiryDuration(60*time.Second),
        ants.WithNonblocking(true),
        ants.WithPanicHandler(s.panicHandler("free")),
    )
    if err != nil { return nil, err }

    s.paidPool, err = ants.NewPool(1500,
        ants.WithExpiryDuration(60*time.Second),
        ants.WithMaxBlockingTasks(1000),
        ants.WithPanicHandler(s.panicHandler("paid")),
    )
    if err != nil { return nil, err }

    s.enterprisePool, err = ants.NewPool(1000,
        ants.WithExpiryDuration(60*time.Second),
        ants.WithPanicHandler(s.panicHandler("enterprise")),
    )
    if err != nil { return nil, err }

    s.startMetricsLoop()
    return s, nil
}

func (s *Service) panicHandler(tier string) func(interface{}) {
    return func(p interface{}) {
        log.Printf("[%s] panic: %v\n%s", tier, p, debug.Stack())
        // metrics.Panics.WithLabelValues(tier).Inc()
    }
}

func (s *Service) Dispatch(tier Tier, event Event) error {
    pool := s.poolForTier(tier)
    err := pool.Submit(func() {
        s.deliver(s.ctx, event)
    })
    if err != nil {
        if errors.Is(err, ants.ErrPoolOverload) {
            if tier == Free {
                return s.overflowQueue.Push(event)
            }
            return errors.New("service unavailable")
        }
        return err
    }
    return nil
}

func (s *Service) poolForTier(t Tier) *ants.Pool {
    switch t {
    case Enterprise: return s.enterprisePool
    case Paid:       return s.paidPool
    default:         return s.freePool
    }
}

func (s *Service) deliver(ctx context.Context, event Event) {
    if ctx.Err() != nil { return }
    req, err := http.NewRequestWithContext(ctx, "POST", event.URL, event.Body)
    if err != nil { return }
    resp, err := s.client.Do(req)
    if err != nil { return }
    resp.Body.Close()
}

func (s *Service) startMetricsLoop() { /* ... */ }

func (s *Service) Close(timeout time.Duration) error {
    s.cancel()
    deadline := time.Now().Add(timeout)
    for _, p := range []*ants.Pool{s.enterprisePool, s.paidPool, s.freePool} {
        left := time.Until(deadline)
        if left <= 0 { break }
        _ = p.ReleaseTimeout(left)
    }
    return nil
}

type Event struct {
    URL  string
    Body interface{ /* ... */ }
}

type OverflowQueue interface {
    Push(Event) error
}

Wiring with HTTP server¶

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    svc, _ := webhook.New(ctx, redisOverflowQueue)

    server := &http.Server{
        Addr: ":8080",
        Handler: webhookHandler(svc),
    }

    sigs := make(chan os.Signal, 1)
    signal.Notify(sigs, syscall.SIGTERM, syscall.SIGINT)

    go func() {
        <-sigs
        log.Println("shutdown initiated")
        shutdownCtx, _ := context.WithTimeout(context.Background(), 25*time.Second)
        _ = server.Shutdown(shutdownCtx)
        cancel()
        _ = svc.Close(20 * time.Second)
    }()

    if err := server.ListenAndServe(); err != http.ErrServerClosed {
        log.Fatal(err)
    }
}

Operational details¶

• Metrics endpoint at /metrics for Prometheus.
• Readiness endpoint at /ready returns 503 during shutdown.
• Logging structured (zap or zerolog).
• Tracing spans on Dispatch and inside delivery.

Why three pools?¶

The bulkhead pattern. Free tenants causing high pool pressure don't affect paid or enterprise.

Why these caps?¶

Sized to expected steady-state load. Total 3000 across pools means the service can sustain 15k ops/sec (3000 / 0.2 avg). Headroom for 5x peaks.

What if free is always saturated?¶

That's their experience problem. Overflow to Redis is the SLA-conforming behaviour. Internally, you may upsell or rate-limit at the API layer.

Extended Section: Common Operational Scenarios¶

Scenario 1 — Service is suddenly slow¶

Symptom: p99 latency jumps from 50 ms to 5 s.

Pool to inspect: see Waiting metric. If > 0 and growing, pool is saturated.

Action: 1. Check downstream: is the service we call slow? 2. Check Running: is it at cap? 3. If both yes: downstream is slowed, pool can't keep up. 4. Mitigation: tune pool down (forces more rejections), or fix downstream.

Scenario 2 — Pod can't start¶

Symptom: new deployment crashes immediately.

Pool relevance: capacity from config. Check value.

Action: 1. Read pool config from environment / config map. 2. Verify cap is positive. 3. Verify ulimit -n is high enough for expected FDs.

Scenario 3 — Memory growth¶

Symptom: pod OOM-killed after a few hours.

Pool relevance: stack growth, task closures retaining refs.

Action: 1. Heap profile: go tool pprof http://pod:6060/debug/pprof/heap. 2. Look for retained references in task closures. 3. Look for slow-growing pool struct (unlikely but possible). 4. Look for blocked submitter goroutines (runtime.NumGoroutine).

Scenario 4 — Pool seems to "freeze"¶

Symptom: Running stays high, Free stays low, no progress.

Pool relevance: workers stuck.

Action: 1. Goroutine dump (SIGQUIT or runtime/pprof). 2. Find workers' stacks. Are they all parked at the same line? 3. Likely a downstream deadlock, slow query, or blocking I/O. 4. Add timeouts inside tasks.

Scenario 5 — Frequent panic alerts¶

Symptom: pool_panic_total rate > 0.

Pool relevance: tasks panicking.

Action: 1. Check panic handler logs for the panic value. 2. Find the bug in the task. 3. Fix.

Don't disable the panic handler. The pool's resilience is by design — your job is to fix the bug.

Scenario 6 — Submit failures during deploy¶

Symptom: ErrPoolClosed errors spike during deployments.

Pool relevance: graceful shutdown.

Action: 1. Verify SIGTERM is sent before SIGKILL. 2. Verify HTTP server is drained before pool is released. 3. Increase grace period if needed.

Scenario 7 — Throughput drop after upgrade¶

Symptom: rate(pool_submit_total{result="ok"}[1m]) drops after deploying new version.

Pool relevance: regression in task duration or pool overhead.

Action: 1. Compare task duration before and after. 2. Check for added allocations. 3. If pool overhead changed, profile pool internals.

Scenario 8 — Multi-tenant noisy neighbour¶

Symptom: one customer's slowness affects all.

Pool relevance: shared pool, no isolation.

Action: 1. Verify tenants are routed to separate pools. 2. If shared, plan migration to bulkheaded pools. 3. As stopgap: rate-limit the noisy tenant at the API layer.

Scenario 9 — Tune cap up but nothing changes¶

Symptom: after Tune(2*cap), throughput stays the same.

Pool relevance: pool wasn't the bottleneck.

Action: 1. Look at Running after tune. Did it grow to fill new cap? 2. If no: downstream or producer is the bottleneck, not the pool. 3. Find the real bottleneck (profile, traces, logs).

Scenario 10 — Pool's CPU profile shows mostly runtime.lock¶

Symptom: pprof CPU is 60% in sync.(*Mutex).Lock inside ants.Submit.

Pool relevance: lock contention.

Action: 1. Migrate from Pool to MultiPool with 4-8 shards. 2. Or batch submits (one task processes 10 items).

Extended Section: Building a Pool Wrapper¶

Production teams typically wrap ants.Pool in a service-specific abstraction. This section shows what a good wrapper looks like.

Goals of the wrapper¶

• Hide pool details from callers.
• Inject metrics automatically.
• Standardise panic handling.
• Make tests easy.
• Provide context-aware Submit.

The wrapper¶

package poolwrap

import (
    "context"
    "errors"
    "log"
    "runtime/debug"
    "time"

    "github.com/panjf2000/ants/v2"
    "github.com/prometheus/client_golang/prometheus"
)

type Pool struct {
    name   string
    pool   *ants.Pool
    hooks  Hooks
    ctx    context.Context
    cancel context.CancelFunc
}

type Hooks struct {
    OnSubmit     func(name string, err error)
    OnTaskStart  func(name string)
    OnTaskEnd    func(name string, panicked bool, duration time.Duration)
    OnPanic      func(name string, p interface{})
}

type Config struct {
    Name        string
    Cap         int
    Expiry      time.Duration
    NonBlocking bool
    MaxBlocking int
    Hooks       Hooks
}

func New(ctx context.Context, cfg Config) (*Pool, error) {
    pCtx, cancel := context.WithCancel(ctx)
    p := &Pool{
        name:   cfg.Name,
        hooks:  cfg.Hooks,
        ctx:    pCtx,
        cancel: cancel,
    }

    pool, err := ants.NewPool(cfg.Cap,
        ants.WithExpiryDuration(cfg.Expiry),
        ants.WithNonblocking(cfg.NonBlocking),
        ants.WithMaxBlockingTasks(cfg.MaxBlocking),
        ants.WithPanicHandler(func(panicVal interface{}) {
            if p.hooks.OnPanic != nil {
                p.hooks.OnPanic(p.name, panicVal)
            } else {
                log.Printf("[%s] panic: %v\n%s", p.name, panicVal, debug.Stack())
            }
        }),
    )
    if err != nil { cancel(); return nil, err }
    p.pool = pool
    return p, nil
}

func (p *Pool) Submit(ctx context.Context, task func(context.Context)) error {
    if err := ctx.Err(); err != nil {
        if p.hooks.OnSubmit != nil { p.hooks.OnSubmit(p.name, err) }
        return err
    }
    err := p.pool.Submit(func() {
        start := time.Now()
        if p.hooks.OnTaskStart != nil { p.hooks.OnTaskStart(p.name) }
        defer func() {
            if p.hooks.OnTaskEnd != nil {
                p.hooks.OnTaskEnd(p.name, recover() != nil, time.Since(start))
            }
        }()
        if ctx.Err() != nil { return }
        task(ctx)
    })
    if p.hooks.OnSubmit != nil { p.hooks.OnSubmit(p.name, err) }
    return err
}

func (p *Pool) Stats() (running, free, cap, waiting int) {
    return p.pool.Running(), p.pool.Free(), p.pool.Cap(), p.pool.Waiting()
}

func (p *Pool) Close(timeout time.Duration) error {
    p.cancel()
    return p.pool.ReleaseTimeout(timeout)
}

// PrometheusHooks builds Hooks that emit Prometheus metrics.
func PrometheusHooks(submits *prometheus.CounterVec,
    durations *prometheus.HistogramVec,
    panics *prometheus.CounterVec) Hooks {
    return Hooks{
        OnSubmit: func(name string, err error) {
            result := "ok"
            if errors.Is(err, ants.ErrPoolOverload) {
                result = "overload"
            } else if errors.Is(err, ants.ErrPoolClosed) {
                result = "closed"
            } else if err != nil {
                result = "error"
            }
            submits.WithLabelValues(name, result).Inc()
        },
        OnTaskEnd: func(name string, panicked bool, d time.Duration) {
            durations.WithLabelValues(name).Observe(d.Seconds())
        },
        OnPanic: func(name string, p interface{}) {
            panics.WithLabelValues(name).Inc()
            log.Printf("[%s] panic: %v\n%s", name, p, debug.Stack())
        },
    }
}