Dynamic Worker Scaling — Professional Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Inside ants v2: a Line-by-Line Tour
4. Inside tunny: Stateful Worker Model
5. Inside pond: Modern Ergonomics
6. Comparing Pool Library Internals
7. Distributed Pool Coordination
8. Capacity Planning Math
9. Queueing Theory Beyond Little's Law
10. Production Failure Modes in Detail
11. Building a Tunable Autoscaler Framework
12. Working at Massive Scale
13. Multi-Region and Multi-Cluster Considerations
14. Performance Engineering for Autoscalers
15. Coding Patterns
16. Clean Code
17. Error Handling
18. Performance Tips
19. Best Practices
20. Edge Cases
21. Common Mistakes
22. Common Misconceptions
23. Tricky Points
24. Test
25. Tricky Questions
26. Cheat Sheet
27. Self-Assessment Checklist
28. Summary
29. What You Can Build
30. Further Reading
31. Related Topics
32. Diagrams

Introduction¶

Focus: "What is happening inside ants and friends? How do I design dynamic scaling at very large scale?"

At professional level, you understand the abstractions deeply. You can read pool library source and explain every decision. You can design scaling for systems with millions of requests per second across global infrastructure. You think about capacity planning, queueing theory, and operational excellence as parts of one whole.

After this chapter you should be able to:

• Read and explain the ants v2 source code in detail
• Compare ants, tunny, pond, and identify trade-offs
• Design distributed pool coordination across instances and regions
• Apply queueing theory beyond Little's Law (M/M/c, M/G/k models)
• Recognize and prevent classic production failure modes
• Build a tunable autoscaler framework usable by other teams
• Engineer for performance at the scale of "every nanosecond counts"

Prerequisites¶

• All of junior, middle, senior chapters
• You have shipped multiple dynamic pools in production
• You have read at least one pool library's source
• You are comfortable with queueing theory basics (M/M/1, M/M/c)
• You have experience with capacity planning, SLO design, and incident response
• You can read and debug code that uses runtime internals (runtime.procPin, etc.)

Inside ants v2: a Line-by-Line Tour¶

ants is the most widely-deployed Go pool library. Let us read its source carefully.

Core types¶

// Pool is the core type.
type Pool struct {
    capacity int32        // max number of in-flight goroutines, atomic
    running  int32        // number of currently-running, atomic
    state    int32        // 0=open, 1=closed, atomic
    lock     sync.Locker  // protects workers and cond
    workers  workerQueue  // free list of available workers
    cond     *sync.Cond   // signal for waiting submitters
    once     *sync.Once   // ensure single close
    options  *Options
    allDone  chan struct{}
}

// goWorker is one worker.
type goWorker struct {
    pool     *Pool
    task     chan func()   // per-worker task channel
    recycleTime time.Time   // for idle expiry
}

Each goWorker is a goroutine that loops on its own task channel. The pool maintains a queue of free workers; when a task is submitted, the pool either pops a worker from the queue (existing free one) or spawns a new one (if under capacity).

Submission path¶

func (p *Pool) Submit(task func()) error {
    if p.IsClosed() {
        return ErrPoolClosed
    }
    var w *goWorker
    if w = p.retrieveWorker(); w != nil {
        w.task <- task
        return nil
    }
    return ErrPoolOverload
}

func (p *Pool) retrieveWorker() (w *goWorker) {
    spawnWorker := func() {
        w = workerCachePool.Get().(*goWorker)
        w.run()
    }

    p.lock.Lock()
    w = p.workers.detach()
    if w != nil {
        p.lock.Unlock()
        return
    }

    if capacity := p.Cap(); capacity == -1 || capacity > int(atomic.LoadInt32(&p.running)) {
        p.lock.Unlock()
        spawnWorker()
        return
    }

    // blocking submission (if not Nonblocking)
    if p.options.Nonblocking {
        p.lock.Unlock()
        return nil
    }

    for {
        if p.options.MaxBlockingTasks != 0 && p.blockingNum >= p.options.MaxBlockingTasks {
            p.lock.Unlock()
            return nil
        }
        p.blockingNum++
        p.cond.Wait()
        p.blockingNum--
        if p.IsClosed() {
            p.lock.Unlock()
            return nil
        }
        var nw int
        if nw = p.workers.len(); nw == 0 {
            if capacity := p.Cap(); capacity == -1 || capacity > int(atomic.LoadInt32(&p.running)) {
                p.lock.Unlock()
                spawnWorker()
                return
            }
            continue
        }
        if w = p.workers.detach(); w == nil {
            if nw == 0 {
                continue
            }
            p.lock.Unlock()
            return nil
        }
        p.lock.Unlock()
        return
    }
}

Let us unpack retrieveWorker:

1. Try to pop a worker from the free list (p.workers.detach()). If success, return it.
2. If no free worker and capacity allows, spawn a new one.
3. If at capacity and Nonblocking is true, return nil (caller gets ErrPoolOverload).
4. Otherwise, wait on cond until either a worker frees up or the pool closes.

The cond-based blocking is classic. Submitters park; workers wake them when returning to the free list.

Worker loop¶

func (w *goWorker) run() {
    w.pool.addRunning(1)
    go func() {
        defer func() {
            if w.pool.addRunning(-1) == 0 && w.pool.IsClosed() {
                w.pool.once.Do(func() { close(w.pool.allDone) })
            }
            w.pool.workerCache.Put(w)
            if p := recover(); p != nil {
                if ph := w.pool.options.PanicHandler; ph != nil {
                    ph(p)
                }
            }
            w.pool.cond.Signal()
        }()

        for f := range w.task {
            if f == nil {
                return
            }
            f()
            if ok := w.pool.revertWorker(w); !ok {
                return
            }
        }
    }()
}

The worker: 1. Loops reading from its task channel. 2. If task is nil, exits (sentinel for shutdown or idle expiry). 3. Runs the task. 4. Calls revertWorker to put itself back on the free list. 5. If revertWorker returns false (pool shrunk), worker exits.

The defer chain handles: - Decrement running count - Recover from panic, call handler - Put goWorker struct back to cache (memory reuse) - Signal cond (wake one waiting submitter) - If pool is closing and this was the last running worker, close allDone

revertWorker¶

func (p *Pool) revertWorker(w *goWorker) bool {
    if capacity := p.Cap(); (capacity > 0 && p.Running() > capacity) || p.IsClosed() {
        p.cond.Broadcast()
        return false
    }
    w.recycleTime = time.Now()
    p.lock.Lock()
    if p.IsClosed() {
        p.lock.Unlock()
        return false
    }
    err := p.workers.insert(w)
    if err != nil {
        p.lock.Unlock()
        return false
    }
    p.cond.Signal()
    p.lock.Unlock()
    return true
}

Three checks: 1. Has the pool shrunk? Running() > Cap() means yes; refuse to revert; worker exits. 2. Is the pool closed? Refuse to revert; worker exits. 3. Insert into free list; signal cond; success.

This is opportunistic shrink in action. A worker that has just finished a task checks the pool's state and decides whether to keep going.

purgeStaleWorkers (idle expiry)¶

func (p *Pool) purgeStaleWorkers(ctx context.Context) {
    ticker := time.NewTicker(p.options.ExpiryDuration)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            if p.IsClosed() {
                return
            }
            p.lock.Lock()
            staleWorkers := p.workers.refresh(p.options.ExpiryDuration)
            p.lock.Unlock()
            for i, w := range staleWorkers {
                w.task <- nil   // sentinel exit signal
                staleWorkers[i] = nil
            }
        }
    }
}

A separate goroutine runs on the expiry ticker. It walks the free list, identifies workers idle longer than ExpiryDuration, and signals them to exit by sending nil to their task channel.

This is the decentralized idle-expiry shrink we covered at middle level. ants's implementation is clean: identify stale workers under lock, signal them outside the lock.

Tune(n)¶

func (p *Pool) Tune(size int) {
    capacity := p.Cap()
    if capacity == -1 || size <= 0 || size == capacity || p.options.PreAlloc {
        return
    }
    atomic.StoreInt32(&p.capacity, int32(size))
    if size > capacity {
        if size - capacity == 1 {
            p.cond.Signal()
        } else {
            p.cond.Broadcast()
        }
    }
}

Tune is short: 1. Validate. 2. Atomically update capacity. 3. If growing, wake submitters waiting on cond (one or many).

That's it. Tune does not spawn workers; the next submission does. Tune does not kill workers; revertWorker handles that.

This is the elegance of ants: by separating cap from live count, Tune is O(1) and concurrency-safe with minimal locking.

worker queue implementations¶

ants has two free-list implementations:

1. workerStack: FIFO stack of workers. Default.
2. workerLoopQueue: ring buffer; pre-allocated for PreAlloc=true.

The choice is small but affects performance:

• Stack: hot workers (recently used) get reused. Good for cache locality.
• Loop queue: round-robin. Distributes work more evenly across worker instances.

For most workloads, stack is fine. Loop queue helps when workers have warm caches and you want to use them.

Sync primitives¶

ants uses: - atomic.LoadInt32/StoreInt32 for cap, running, state - sync.Locker (Mutex) for worker queue protection - sync.Cond for submission blocking - sync.Pool (workerCachePool) for goWorker struct reuse

Each is the right tool for the job. No clever tricks; battle-tested primitives.

Inside tunny: Stateful Worker Model¶

tunny is smaller than ants and has a different design philosophy.

Core types¶

type Worker interface {
    Process(payload interface{}) interface{}
    BlockUntilReady()
    Interrupt()
    Terminate()
}

type workerWrapper struct {
    worker        Worker
    interruptChan chan struct{}
    reqChan       chan workRequest
    closeChan     chan struct{}
    closedChan    chan struct{}
}

type Pool struct {
    queuedJobs int64

    ctor    func() Worker
    workers []*workerWrapper
    reqChan chan workRequest

    workerMut sync.Mutex
}

Each worker is an explicit Worker interface. The user implements it. Each workerWrapper is one goroutine that calls the user's Process method.

Submission¶

func (p *Pool) Process(payload interface{}) interface{} {
    atomic.AddInt64(&p.queuedJobs, 1)
    request, open := <-p.reqChan
    if !open {
        panic("attempted to process when pool is closed")
    }
    request.jobChan <- payload
    payload, open = <-request.retChan
    atomic.AddInt64(&p.queuedJobs, -1)
    if !open {
        panic("worker failed to send back response")
    }
    return payload
}

Submit is synchronous: it sends a payload and waits for a return. The worker's Process method receives the payload, returns a result, and the result flows back.

This is fundamentally different from ants. ants treats tasks as fire-and-forget; tunny treats them as request-response. Tunny is better for "process this and tell me the answer" patterns.

SetSize¶

func (p *Pool) SetSize(n int) {
    p.workerMut.Lock()
    defer p.workerMut.Unlock()

    lWorkers := len(p.workers)
    if lWorkers == n {
        return
    }

    if lWorkers < n {
        for i := lWorkers; i < n; i++ {
            p.workers = append(p.workers, newWorkerWrapper(p.reqChan, p.ctor()))
        }
        return
    }

    // shrinking: pick the last n workers, terminate the rest
    for i := n; i < lWorkers; i++ {
        p.workers[i].stop()
    }
    for i := n; i < lWorkers; i++ {
        p.workers[i].join()
    }
    p.workers = p.workers[:n]
}

Tunny's SetSize is more aggressive than ants's Tune: 1. If growing: spawn new workers immediately. 2. If shrinking: stop the excess workers, wait for them to terminate, remove from the slice.

This means shrink in tunny is immediate. The Worker interface's Interrupt is called; the worker is given a chance to cancel its current operation; then it terminates.

Tunny is harder to use correctly (you must implement Worker carefully with Interrupt handling) but gives precise control.

Worker lifecycle¶

func (w *workerWrapper) run() {
    jobChan, retChan := make(chan interface{}), make(chan interface{})
    defer func() {
        w.worker.Terminate()
        close(retChan)
        close(w.closedChan)
    }()

    for {
        w.worker.BlockUntilReady()
        select {
        case w.reqChan <- workRequest{jobChan: jobChan, retChan: retChan, interruptFunc: w.interrupt}:
            select {
            case payload := <-jobChan:
                result := w.worker.Process(payload)
                select {
                case retChan <- result:
                case <-w.interruptChan:
                    w.interruptChan = make(chan struct{})
                }
            case _, _ = <-w.interruptChan:
                w.interruptChan = make(chan struct{})
            }
        case <-w.closeChan:
            return
        }
    }
}

Each iteration: 1. Worker signals readiness (BlockUntilReady allows stateful warmup). 2. Worker offers itself to the pool's request channel. 3. When a request arrives, run Process, return the result. 4. Listen for interrupt to allow shrink during processing.

Tunny's worker is more involved than ants's because it supports request-response and per-worker state.

When to choose tunny¶

• Workers need persistent state (DB connection, large buffer, ML model)
• Tasks are request-response (you need the return value)
• You want clean state-management hooks (BlockUntilReady, Terminate)

When to choose ants: - Tasks are fire-and-forget - Workers are stateless - You need higher throughput

Inside pond: Modern Ergonomics¶

pond (alitto/pond) is the newer entry. Smaller adoption but cleaner API.

Core types¶

type WorkerPool struct {
    workerCount      int32
    idleWorkerCount  int32
    maxWorkers       int
    maxCapacity      int
    tasks            chan func()
    purgerQuit       chan struct{}
    stopCtx          context.Context
    stopCancel       context.CancelFunc
    waitGroup        sync.WaitGroup
}

Simpler than ants. No free-list of worker structs. The pool is just a channel and an atomic count.

Submission¶

func (p *WorkerPool) Submit(task func()) {
    p.submit(task, false)
}

func (p *WorkerPool) submit(task func(), nonblocking bool) bool {
    if task == nil { return false }
    select {
    case p.tasks <- task:
        return true
    default:
        if p.maxCapacity > 0 && atomic.LoadInt32(&p.workerCount) >= int32(p.maxWorkers) {
            if nonblocking { return false }
            p.tasks <- task   // blocking send
            return true
        }
        // spawn new worker
        p.startWorker()
        p.tasks <- task
        return true
    }
}

The submit logic: 1. Try non-blocking send to task channel. 2. If channel is full and pool can grow, spawn a worker. 3. If can't grow, either return false (Nonblocking) or block.

Each worker is just a goroutine reading from the shared tasks channel. No per-worker channel like ants or tunny.

Worker loop¶

func (p *WorkerPool) worker() {
    defer p.waitGroup.Done()
    atomic.AddInt32(&p.workerCount, 1)
    defer atomic.AddInt32(&p.workerCount, -1)

    for {
        select {
        case task, ok := <-p.tasks:
            if !ok { return }
            p.runTask(task)
        case <-p.stopCtx.Done():
            return
        }
    }
}

Pure simplicity. Read from shared channel; run task; loop.

No Tune¶

pond at the time of writing does not expose a Tune method. Capacity is fixed at construction. Workers spawn on demand up to maxWorkers; they exit on idle.

This makes pond a softly dynamic pool: it grows on demand but does not actively right-size. For workloads where capacity is bounded by maxWorkers and growth is acceptable, pond works.

For workloads needing active downsizing or live tuning, pond would need an external mechanism (fork the lib, or wrap).

Task groups¶

pond's killer feature: task groups.

group, ctx := pool.GroupContext(ctx)
for _, item := range items {
    item := item
    group.Submit(func() error {
        return process(ctx, item)
    })
}
err := group.Wait()

Group is like errgroup: tracks errors, waits for all tasks. Built on top of the pool.

For batch operations, this is cleaner than rolling your own with WaitGroup + channels.

When to choose pond¶

• You want clean ergonomics, including task groups
• You don't need active resize
• You want a small dependency footprint

Comparing Pool Library Internals¶

Each library has a sweet spot:

• ants: high throughput, dynamic resize, panic safety. The default.
• tunny: stateful workers, request-response. Used for connection pools.
• pond: clean ergonomics, task groups, fixed capacity. Used for batch jobs.

A production codebase may use all three for different needs.

Distributed Pool Coordination¶

In a cluster of N instances, each with its own pool, total capacity is N × poolSize. Coordinating this is hard.

Pattern: independent autoscaling¶

Easiest. Each instance autoscales independently. Aggregate behavior emerges.

Pros: simple, robust to network partitions. Cons: collective overcommit possible (each thinks it should grow; combined > host capacity).

Pattern: gossip-based coordination¶

Instances gossip their pool size to peers. Each instance's autoscaler considers cluster total.

type DistributedAutoscaler struct {
    Local      *Pool
    PeerSizes  *PeerCache  // cached sizes of other instances
    Bounds     Bounds
}

func (a *DistributedAutoscaler) ClusterSize() int {
    total := a.Local.Size()
    for _, peer := range a.PeerSizes.All() {
        total += peer
    }
    return total
}

func (a *DistributedAutoscaler) tick() {
    clusterCap := a.Bounds.Max  // total across cluster
    if a.ClusterSize() >= clusterCap {
        return  // don't grow
    }
    // ... normal decide ...
}

Gossip needs: - Heartbeats between peers - Stale data tolerance (peer can crash) - Bandwidth budget (don't flood network)

For 10s of instances, gossip works. For 1000s, hierarchical (sub-clusters gossiping within, leaders gossiping across).

Pattern: central coordinator¶

A central service (e.g., a Kubernetes operator or a custom Go service) decides each instance's pool size.

Pros: tight control, optimal global allocation. Cons: single point of failure; latency overhead; complexity.

Implementations: AWS Auto Scaling Group's target tracking, Kubernetes HPA controller. Both apply the central-coordinator pattern at cluster scale.

Pattern: distributed lease¶

Each pool holds a lease for N workers. Lease has a TTL. Renew periodically. On crash, lease expires, capacity is freed.

Implementations: etcd's lease, Redis-based leases (SET NX EX).

func (a *Autoscaler) acquire(n int) bool {
    return a.lease.Lock(fmt.Sprintf("workers:%d", n), 30*time.Second)
}

Strict bounds; survives crashes. Adds dependency on lease service.

Pattern: hierarchical autoscaling¶

Levels of autoscaling: - Within a pod (in-process pool) - Across pods (HPA, custom controller) - Across regions (Multi-Cluster Federation)

Each level has its own time scale. Lower levels react faster. Higher levels react slower.

This is how big production systems scale dynamically. We covered it at senior level; here we go deeper.

Capacity Planning Math¶

Beyond Little's Law, more queueing models apply.

M/M/1 — single server¶

• Poisson arrivals at rate λ
• Exponential service time with mean 1/μ
• Single server

Average queue length: Lq = ρ² / (1 - ρ) where ρ = λ/μ. Average wait time: Wq = Lq / λ = ρ / (μ(1-ρ)).

Note: as ρ → 1, queue length → ∞. Operating above 80% utilization is dangerous.

M/M/c — c servers¶

• Poisson arrivals at rate λ
• Exponential service time with mean 1/μ
• c servers

This is the model for a worker pool.

Average queue length: complicated formula involving Erlang C. Average wait: Wq = (Erlang C * 1/μ) / c(1-ρ) where ρ = λ / (cμ).

The Erlang C formula gives the probability of queuing (P(W>0)). Higher c reduces queuing dramatically.

In practice: doubling c gives more than 2× headroom because queuing probability drops nonlinearly.

M/G/k — general service time¶

Real workloads aren't exponentially distributed. M/G/k uses general service time. Formulas (Pollaczek-Khinchine) involve the variance of service time:

Wq = (λ * E[S²]) / (2 * (1 - ρ))

where E[S²] is the second moment of service time. High variance = longer waits.

This is why bimodal workloads (50% fast, 50% slow) suffer. The variance is large; queuing time is long.

Applying to autoscaler bounds¶

Suppose your SLO is "p99 wait < 100 ms" and service time variance is high.

By M/G/k math, for fixed utilization: - ρ = 0.5: p99 wait < 100ms easily - ρ = 0.7: p99 wait approaches 100ms - ρ = 0.9: p99 wait > 200ms

So at ρ = 0.7, you are at the SLO limit. Set utilization set-point to 0.7 (not 0.9).

This is queueing theory feeding into autoscaler design. Most engineers skip this; senior+ engineers use it for sanity checks.

Tools¶

• numpy.queuing (Python)
• Free queueing calculators online
• Practical: simulation. Build a load generator, observe queue behavior, fit model.

For most services, a rough estimate from Little's Law + a safety margin (50%) covers planning. Detailed queueing analysis is for precision-critical workloads.

Queueing Theory Beyond Little's Law¶

A few more concepts that show up in autoscaler design.

Bottleneck analysis¶

In a multi-stage pipeline, the slowest stage determines throughput. Adding workers at non-bottleneck stages does nothing.

input → [pool A] → [pool B] → [pool C] → output
        100/sec    50/sec     200/sec
                   ↑
                   bottleneck

Pool A submits faster than Pool B can drain. Queue between A and B grows. Pool B is the bottleneck.

Solution: scale Pool B, not Pool A. Pool A's autoscaler should monitor downstream queue health and stop growing when downstream is the bottleneck.

Coupled systems¶

When pools share a downstream, scaling one affects another's experience.

pool A ─┐
        ├─→ shared downstream
pool B ─┘

If A doubles, downstream sees 2x load from A's tasks. B's tasks now experience downstream slowdown. B's autoscaler sees high latency, grows B. Now downstream sees more load from B too. Eventually downstream is overwhelmed.

Solution: pools sharing a downstream must coordinate. Either share a budget or use a circuit breaker that limits total concurrency to the shared downstream.

Queueing networks¶

Multiple queues feeding each other. The math gets complex. Tools like SimPy (Python) or Go's discrete-event simulation libraries let you model.

In practice, for designing autoscalers: identify the critical path; scale the bottleneck; everything else follows.

Self-similar traffic¶

Internet traffic is often "self-similar" — bursty at all time scales. Aggregating over longer windows does not smooth the bursts.

Implications: - Don't assume Poisson arrivals. - Tail behavior is heavier than Poisson predicts. - Provision for the largest bursts, not the average.

Autoscaler design implication: be eager to grow; reactive autoscaling alone is often not enough for self-similar traffic. Combine with prediction or oversize provision.

Production Failure Modes in Detail¶

Let us catalog production failures specific to dynamic pool autoscaling.

Failure: thundering herd on grow¶

Pool grows from 10 to 50. All 40 new workers spawn simultaneously. They all wake up and read the same task channel. The first 40 tasks are dispatched. Channel sender's lock contention spikes briefly.

Usually harmless but can show up as latency spikes during fast grows.

Defense: stagger spawns. Spread the new worker spawns over a few ticks.

Failure: idle storm on shrink¶

Pool shrinks from 50 to 10. 40 workers exit. All idle timeouts fire near-simultaneously. GC sees a burst of work (stack freeing). Brief CPU spike.

Defense: stagger exits. Or accept; usually negligible.

Failure: signal source corruption¶

A bug in the metric collection causes the signal to spike to a high value momentarily. Autoscaler grows. Then signal returns to normal. Autoscaler shrinks. Pool flaps.

Defense: smooth signals. Clamp outliers. Alert on signal anomalies.

Failure: cascading retry storm¶

Downstream is slow. Workers experience long waits. Autoscaler grows. More workers hit slow downstream. Downstream rate-limits. Workers see errors. They retry. More load on downstream. Downstream collapses.

Defense: circuit breaker, exponential backoff in retries, error-rate veto in autoscaler.

Failure: clock skew¶

A multi-instance system uses time.Now() for cooldown tracking. Clock skew between instances means autoscalers disagree on whether cooldown has elapsed.

Defense: use monotonic clock (time.Since(t) rather than time.Now().Sub(t) for cooldown). Same instance's clock is fine; cross-instance time comparisons are unreliable.

Failure: leaked resize goroutines¶

A bug spawns an autoscaler goroutine on every config reload. Old goroutines never exit. After many reloads, hundreds of autoscalers fight.

Defense: track all spawned autoscalers; cancel old ones via context before spawning new.

Failure: incorrect floor on warm-up¶

Floor is set to 4 but during warm-up, the pool starts at 0 and the autoscaler runs immediately. Autoscaler can't grow because cooldown isn't yet active and signals are noisy.

Defense: prime the pool to floor before starting autoscaler. Or initial size = floor in pool config.

Failure: shrink during deploy¶

Old version draining; new version starting. Autoscaler on old version sees no traffic (drain), shrinks to floor. Then old version exits. New version is at floor. First traffic hits floor-sized pool. Latency spike.

Defense: pause autoscaler during drain; don't shrink while draining.

Failure: configuration drift¶

Ops keeps tweaking thresholds. Over months, config moves far from original design. New incidents arise from accumulated changes.

Defense: version control configs. Periodic review. Lint rule for "config not changed in 6 months."

Failure: noisy neighbor in multi-tenant pool¶

One tenant submits more or slower tasks. Shared autoscaler grows. Other tenants pay (latency on shared pool, sometimes cost).

Defense: per-tenant pools, or fair scheduling within a shared pool.

Failure: ChunkSize=0 in batches¶

A batch processor uses a pool. Default chunk size is 0 (each task is one item). High overhead. Pool overgrows trying to handle the per-item rate.

Defense: tune batch chunk size. Pool should handle work units of "useful size."

Failure: zone failure cascading to pools¶

In a multi-AZ deployment, one AZ goes down. Traffic shifts to remaining AZs. Their pools see 50% more load. Autoscaler in surviving AZs grows. Now they're at 1.5x capacity. AZ recovers. Traffic spreads. Pools shrink. Brief over-provisioning.

Defense: accept brief overshoot. Better than under-provisioning during recovery.

Building a Tunable Autoscaler Framework¶

If your organization runs many services, build a shared framework. Let us sketch one.

Architecture¶

+----------------------------+
| Autoscaler framework        |
|                            |
|  +----------------------+   |
|  | Pool abstractions    |   |
|  | (Pool interface,     |   |
|  |  ants adapter, etc.) |   |
|  +----------------------+   |
|                            |
|  +----------------------+   |
|  | Signal sources       |   |
|  | (wait, util, depth,  |   |
|  |  prometheus, custom) |   |
|  +----------------------+   |
|                            |
|  +----------------------+   |
|  | Deciders              |   |
|  | (threshold, AIMD,    |   |
|  |  PID, composite)     |   |
|  +----------------------+   |
|                            |
|  +----------------------+   |
|  | Coordination         |   |
|  | (budget, lease, fed) |   |
|  +----------------------+   |
|                            |
|  +----------------------+   |
|  | Observability        |   |
|  | (metrics, logging,   |   |
|  |  events)             |   |
|  +----------------------+   |
+----------------------------+

Each layer is independently testable, swappable, configurable.

Config¶

# autoscaler.yaml
service: api
pool:
  type: ants
  initial: 16
  options:
    expiry_duration: 60s
    nonblocking: true

signals:
  - type: wait_p99
    name: wait
  - type: utilization
    name: util

decider:
  type: composite
  parts:
    - type: threshold
      signal: wait
      operator: gt
      value: 500ms
      action: grow
      step: 2
    - type: aimd
      signal: util
      setpoint: 0.7
      grow_step: 1
      shrink_factor: 0.25

cooldown:
  up: 3s
  down: 60s

bounds:
  min: 4
  max: 128

coordination:
  type: budget
  global_max: 1024

Loaded at startup. Hot-reloadable via SIGHUP or API.

Plug points¶

type Framework struct {
    Builders map[string]Builder
    Registry map[string]*Autoscaler
}

type Builder interface {
    BuildSignal(config map[string]interface{}) (Signal, error)
    BuildDecider(config map[string]interface{}) (Decider, error)
    // ...
}

Teams register custom signals or deciders. Most use defaults.

Observability¶

All autoscalers emit: - Resize events (counter with labels) - Pool size (gauge) - Signal values (gauges, one per signal) - Decision reasons (counter with reason label)

Central log of decisions for forensic analysis.

Self-monitoring¶

The framework monitors itself: - Number of registered autoscalers - Last-tick time per autoscaler - Panics caught - Config reload events

Alerts on framework health, not just pool health.

Why a framework?¶

In a 100-service organization, every team building its own autoscaler is wasteful. Shared abstractions, central improvements, consistent observability. The platform team owns the framework; service teams plug in.

This is how big tech companies do it. The framework is the productized version of all the patterns from junior, middle, senior.

Working at Massive Scale¶

When you have 10,000 services, each with its own pool, scaling considerations change.

Resource governance¶

Total compute is the cluster. Workers are units of compute. With 10k services, total worker count can reach 100k+. Cluster has finite memory and CPU.

Governance: - Each service has a quota. - The platform allocates within quotas. - Excess requests trigger alerts; never silently exceed.

Cost attribution¶

Each worker has a cost. Each service's autoscaler decisions translate to cost.

Cost reports per service. Holding service teams accountable for autoscaler tuning.

Standardization¶

At scale, you cannot tolerate divergent autoscaler implementations. The framework enforces patterns. Custom autoscalers are rare and reviewed.

Capacity planning¶

Per-service capacity plans roll up to a cluster plan. The cluster plan informs hardware procurement.

Quarterly review: which services are growing? Which are at ceiling often? Which have over-provisioning?

Incident response¶

Pages tagged by service. On-call follows runbook. Runbooks are pre-written for autoscaler issues.

When the autoscaler is the root cause of an outage, framework team is consulted. Improvements propagate.

Operational excellence¶

Metrics on metrics. Number of resize events per cluster per day. Average pool utilization across services. Number of SLO breaches attributable to autoscaler.

Continuous improvement: the framework's goal is to make autoscaling boring.

Multi-Region and Multi-Cluster Considerations¶

Spreading load across regions adds dimensions.

Region-local autoscaling¶

Each region has its own service deployment. Each has its own autoscaler.

Pros: simple, isolated failures. Cons: no cross-region balancing.

Cross-region orchestration¶

A global controller observes all regions, allocates capacity:

                  Global Controller
                      / | \
                     /  |  \
              Region A Region B Region C
              autoscaler ...

The global controller adjusts per-region targets based on global load.

Failover handling¶

When a region fails, surviving regions take load. Their autoscalers should react fast.

Pre-warm capacity for failover: each region's pool has headroom for 2x normal load (assuming one region can absorb another's).

Costs¶

Cross-region traffic is expensive. Latency too. Most autoscaling stays local.

Global controller intervenes only for sustained imbalances.

CAP considerations¶

Distributed coordination is bounded by CAP. Choose: - Consistency: strict budget, slow. - Availability: each region autonomous, possible overcommit.

For most worker pools, availability wins. Regional autonomy with loose global coordination.

Performance Engineering for Autoscalers¶

At very high request rates, the autoscaler itself is a hot path.

Bottleneck: signal collection¶

If every task records a wait-time sample, the lock contention on the wait tracker becomes the bottleneck.

Mitigation: - Sample (1-in-N) - Sharded trackers (one per CPU) - Lock-free histograms (Prometheus's native)

Bottleneck: tick rate¶

Fast ticks (100ms) waste CPU on samples that didn't change. Slow ticks (5s) react slowly.

Tune per workload. Adaptive ticks (tick faster when signal is volatile) are an option.

Bottleneck: Resize overhead¶

Spawning many workers in one tick takes time. With ants, spawning is microseconds per worker. 1000 workers = 1ms. Acceptable.

For very fast resize, batch the spawns:

for i := 0; i < toAdd; i += batchSize {
    end := min(i+batchSize, toAdd)
    go func() {
        for j := i; j < end; j++ {
            spawnWorker()
        }
    }()
}

Parallel spawning. Faster but adds complexity.

Bottleneck: mutex contention¶

Single autoscaler is fine. Multiple are bad. Stick to one autoscaler per pool.

For shared coordination (budget), the budget's mutex is the bottleneck. Sharded budget (one budget per service category) reduces contention.

Bottleneck: GC pressure¶

Continually allocating closures, structs, sample slices creates GC pressure.

Mitigations: - Reuse buffers (sync.Pool) - Pre-allocate (PreAlloc option in ants) - Tune GOGC

For 100k req/s+ pools, GC tuning matters.

Profiling¶

Run go tool pprof on the autoscaler service. Look for: - CPU hotspots - Allocations - Lock contention (go test -mutexprofile)

Optimize iteratively. Most autoscalers are not bottlenecks; verify before optimizing.

Coding Patterns¶

Pattern: domain types¶

Don't pass float64 everywhere. Use domain types:

type Utilization float64
type WaitTime time.Duration
type QueueDepthRatio float64

func (u Utilization) IsHigh() bool { return u > 0.85 }

Compiler-enforced units. Easier to read.

Pattern: phantom types for safety¶

type Pool[T TaskType] struct { /* ... */ }
type ImageTask struct{}
type EmailTask struct{}

imagePool := NewPool[ImageTask](...)
emailPool := NewPool[EmailTask](...)
imagePool.Submit(EmailTask{}) // compile error!

Useful when you have many pools that should not be mixed up.

Pattern: deferred config¶

type Pool struct {
    config atomic.Pointer[Config]
}

func (p *Pool) Reload(c *Config) {
    p.config.Store(c)
}

Atomic swap of config. Hot reload without locks.

Pattern: typed event log¶

type Event interface{ event() }

type ResizeEvent struct{ /* fields */ }
type VetoEvent struct{ /* fields */ }
type ErrorEvent struct{ /* fields */ }

func (ResizeEvent) event() {}
func (VetoEvent) event() {}
func (ErrorEvent) event() {}

Type-safe event channel. Consumers can switch on type.

Pattern: builder with validation¶

type Builder struct {
    errs []error
}

func (b *Builder) WithFloor(n int) *Builder {
    if n < 0 { b.errs = append(b.errs, errors.New("floor must be >= 0")) }
    // ...
    return b
}

func (b *Builder) Build() (*Autoscaler, error) {
    if len(b.errs) > 0 { return nil, errors.Join(b.errs...) }
    // ...
}

Accumulate errors. Single validation at Build time.

Pattern: prom-style histograms¶

type Histogram struct {
    buckets []int64  // atomic
    bounds  []float64
}

func (h *Histogram) Observe(v float64) {
    i := sort.SearchFloat64s(h.bounds, v)
    atomic.AddInt64(&h.buckets[i], 1)
}

Lock-free, atomic. Fast for hot paths.

Clean Code¶

• Comment why, not what. The code shows what; comments should explain non-obvious decisions.
• Group related code. Put autoscaler, pool, signal in separate files.
• Constants at the top of the file. Easy to scan.
• Tests next to code. autoscaler.go and autoscaler_test.go.
• Documentation: every exported type, function, and package has a doc comment.
• Examples: package-level examples (ExampleAutoscaler_Run) for documentation.
• Versioning: if you publish the framework, semver. Breaking changes are rare and explicit.

Error Handling¶

Resize failure due to memory¶

err := p.Resize(target)
if errors.Is(err, ErrOOM) {
    // alert; stay at current size
    log.Warn("resize failed: out of memory")
    return
}

Cluster coordination failure¶

budget, err := lease.Acquire(n)
if err != nil {
    // network partition; act locally
    return a.localDecide()
}

Configuration error at startup¶

config, err := loadConfig()
if err != nil {
    log.Fatal("invalid config", err)
}
// fail fast at startup; never run with bad config

Panics in user code¶

Always recover in workers; never in the autoscaler loop (panics there are programming bugs).

func (a *Autoscaler) Run(ctx context.Context) {
    defer func() {
        if r := recover(); r != nil {
            log.Error("autoscaler panicked", r, debug.Stack())
            // alert; restart
        }
    }()
    // ...
}

Performance Tips¶

• Cache runtime.NumCPU(); it does a syscall.
• Use atomic.Int32 (Go 1.19+) for cleaner code.
• Avoid runtime.NumGoroutine() in hot paths; it scans all goroutines.
• Profile your autoscaler under realistic load.
• Use sync.Pool for short-lived objects.
• Use time.NewTicker, not time.After (avoids allocation per tick).
• Histograms over sorts for percentiles.
• Sample, don't measure every event.

Best Practices¶

1. Read pool library source. Understanding ants makes you a better engineer.
2. Use a framework if you have many pools. Don't reinvent.
3. Track decisions, not just metrics. Event log enables forensics.
4. Cluster coordination by lease or gossip. Not by trust.
5. Test stress scenarios before production.
6. Capacity plan quarterly.
7. Use queueing theory for sanity checks.
8. Defend against cascading failures (breakers, vetoes).
9. Document policies. Future engineers will tune.
10. Periodically revisit whether dynamic is still the right choice.

Edge Cases¶

Tune to a value the pool can't reach¶

If memory is full, ants can't spawn. Tune(N) is accepted but live count never reaches N.

Detection: alert on "Cap > Running for sustained period without queue building".

Cooldown spans deploy¶

Pool was scaling up before deploy. Cooldown was active. Deploy starts. Old pool drains. New pool starts. Its autoscaler doesn't have the cooldown state. Behaves differently than expected.

Mitigation: persistent cooldown state (rare; usually accept brief deploy-time anomaly).

Negative sizes¶

Bug in arithmetic produces target = -1. Always clamp to floor.

Resize during goroutine spawn¶

Tune(20) then immediately Tune(0). First call wants to spawn many; second cancels. ants handles this; rolled-your-own might not.

Negative running count¶

If addRunning(-1) is called more than addRunning(1), count goes negative. Indicates a bug. Should alert.

if newRunning < 0 {
    log.Error("running count negative", newRunning)
}

Idle expiry during high churn¶

Workers idle for milliseconds, exit. New workers spawn moments later. Churn dominates throughput.

Defense: lengthen idle expiry. Or set a minimum lifetime.

Common Mistakes¶

1. Not reading the library source.
2. Trusting library defaults blindly.
3. Skipping capacity planning.
4. Over-tuning. Most defaults are good.
5. Adding signals without removing old ones.
6. Confusing different time scales (autoscaler tick vs HPA cycle).
7. Mixing autoscaler concerns (signal collection, decision, actuation in one function).
8. Distributed coordination without considering CAP.
9. No event log for forensics.
10. Performance optimization before profiling.

Common Misconceptions¶

• "More signals = better decisions." Often worse: more knobs, more failure modes.
• "Distributed coordination is mandatory." For most workloads, regional autonomy is fine.
• "Capacity planning is just math." It is math plus operational judgment.
• "Queueing theory is academic." It is operational; use it.
• "Framework is overengineering." For one service, yes; for an org with 100, no.

Tricky Points¶

• ants's revertWorker returns false when shrinking; that's how workers exit.
• ants's Tune does not actuate immediately; future submissions and revertWorker do.
• ants's idle expiry sends nil to worker's task channel as the exit sentinel.
• tunny's SetSize is eager; ants's Tune is lazy.
• pond does not support Tune; capacity is fixed.
• M/M/c queue waits grow non-linearly with utilization; 80% is the practical ceiling.

Test¶

Production-grade tests:

func TestAntsTuneOpportunisticShrink(t *testing.T) {
    p, _ := ants.NewPool(50)
    defer p.Release()
    // submit slow tasks
    for i := 0; i < 50; i++ {
        p.Submit(func() { time.Sleep(100 * time.Millisecond) })
    }
    // tune down while tasks in flight
    p.Tune(10)
    // count goroutines later
    time.Sleep(200 * time.Millisecond)
    if p.Running() > 10 {
        t.Errorf("expected at most 10 running, got %d", p.Running())
    }
}

func TestDistributedBudgetNoOvercommit(t *testing.T) {
    budget := NewBudget(100)
    const N = 1000
    var wg sync.WaitGroup
    granted := int64(0)
    for i := 0; i < N; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            g := budget.Request(1)
            atomic.AddInt64(&granted, int64(g))
        }()
    }
    wg.Wait()
    if granted > 100 {
        t.Errorf("budget overcommit: granted=%d, max=100", granted)
    }
}

Tricky Questions¶

1. Why does ants use Cond rather than channels for blocking submission? Cond is more efficient when many goroutines need to wait. Channels would require N goroutines to wait on the same channel, which is fine but loses ordering control.

2. Why does tunny shrink eagerly while ants shrinks lazily? Tunny supports stateful workers; eager shrink calls Terminate which can clean up state. Ants's stateless workers can simply not be reverted.

3. Why does pond not have Tune? Design choice. Pond focuses on ergonomics; dynamic tuning is left to wrappers.

4. What is the M/M/c implication for autoscaling? Operating at high utilization gives short waits but is fragile. Doubling capacity gives more than 2x headroom because queueing probability drops nonlinearly.

5. How does distributed coordination interact with CAP? Strict coordination requires synchronization which costs availability. Loose coordination (gossip) tolerates partitions but allows overcommit. Choose based on workload tolerance.

6. What does ants's MultiPool give over Pool? Sharded free list. Reduces lock contention at very high throughput. For most workloads, plain Pool is enough.

7. When does pre-allocation matter? When startup-time spawning would cause a brief unavailability. Pre-alloc trades steady-state memory for startup smoothness.

8. Why is HPA slower than in-process autoscaling? HPA decisions go through a control loop with multi-minute periods. In-process is single-process, single-millisecond. Designed for different time scales.

Cheat Sheet¶

// Ants internals
ants.NewPool(n, opts...)
p.Tune(n)        // lazy resize
p.Running()      // busy count
p.Cap()          // capacity
p.Free()         // cap - running
p.Submit(task)   // submission
p.Release()      // shutdown

// Tunny internals
tunny.New(n, ctor)
p.SetSize(n)     // eager resize
p.Process(payload)  // sync request-response
p.Close()

// Pond
pond.New(n, cap)
p.Submit(task)
p.GroupContext(ctx) // task groups
p.StopAndWait()

// Queueing models
M/M/1:  Lq = ρ²/(1-ρ)
M/M/c:  Erlang C formula
M/G/k:  Pollaczek-Khinchine

// Distributed coordination
- gossip: peer heartbeats
- central: HPA, custom controller
- lease: etcd, Redis
- hierarchy: pod-pool-cluster-region

Self-Assessment Checklist¶

• I can read and explain ants v2 source
• I can compare ants, tunny, pond design trade-offs
• I can apply M/M/c queueing math to sizing
• I can design gossip-based distributed coordination
• I can build an autoscaler framework with pluggable parts
• I can diagnose production failure modes (cascading, thundering herd, etc.)
• I can performance-tune an autoscaler (sampling, sharding, lock-free)
• I can design multi-region pool coordination
• I can apply capacity planning math at organization scale
• I can teach other engineers these patterns

Summary¶

Professional level brings depth: reading source, understanding queueing, designing distributed coordination, operating at scale.

The themes: - Read library source: ants and friends. Their patterns are the patterns. - Queueing theory: Little's Law, M/M/c, M/G/k for sizing. - Distributed coordination: gossip, central, lease, hierarchy. - Production failure modes: cascading, thundering herd, configuration drift. - Framework thinking: at organization scale, build once, use everywhere. - Performance engineering: profiling, sampling, lock-free.

Mastery here means: you can take any dynamic-pool problem, design the solution, implement it correctly, deploy it safely, and operate it for years. That is professional-level capability.

What You Can Build¶

• An autoscaler framework for an organization
• A custom pool library tuned for a specific workload (e.g., ML inference)
• A distributed pool coordinator (Kubernetes operator)
• A capacity planning tool that combines queueing models with historical data
• A production-grade autoscaler with formal verification of bounds

Further Reading¶

• ants source: read it cover to cover
• tunny source: simpler, also worth reading
• pond source: modern Go style
• Brendan Burns, "Designing Distributed Systems"
• Kleppmann, "Designing Data-Intensive Applications"
• Murray, "Distributed Algorithms"
• Operations Research textbooks on queueing
• AWS Auto Scaling internals (blog posts and patents)
• Kubernetes HPA design docs
• Netflix's autoscaler papers and blogs

Related Topics¶

• Backpressure (sibling subsection)
• Graceful shutdown
• Circuit breaker patterns
• Capacity planning
• Distributed systems coordination
• Queueing theory

Deep Dive: ants's workerStack vs workerLoopQueue¶

We mentioned ants has two free-list implementations. Let us examine both.

workerStack¶

type workerStack struct {
    items  []*goWorker
    expiry []*goWorker  // staging for stale workers
}

func (ws *workerStack) detach() *goWorker {
    n := len(ws.items)
    if n == 0 { return nil }
    w := ws.items[n-1]
    ws.items[n-1] = nil
    ws.items = ws.items[:n-1]
    return w
}

func (ws *workerStack) insert(w *goWorker) error {
    ws.items = append(ws.items, w)
    return nil
}

LIFO. Most recently freed worker is reused first. Pros: cache locality (hot worker has hot stack and CPU caches). Cons: workers near the bottom may stagnate (no longer used; still allocated).

The refresh method walks from the bottom, finding stale workers (idle longer than expiry):

func (ws *workerStack) refresh(duration time.Duration) []*goWorker {
    expiryTime := time.Now().Add(-duration)
    n := len(ws.items)
    if n == 0 { return nil }

    var i int
    l := 0
    r := n - 1
    for l <= r {
        mid := l + (r-l)/2
        if expiryTime.Before(ws.items[mid].recycleTime) {
            r = mid - 1
        } else {
            l = mid + 1
        }
    }
    i = l

    ws.expiry = ws.expiry[:0]
    if i > 0 {
        ws.expiry = append(ws.expiry, ws.items[:i]...)
        m := copy(ws.items, ws.items[i:])
        for i := m; i < n; i++ {
            ws.items[i] = nil
        }
        ws.items = ws.items[:m]
    }
    return ws.expiry
}

Binary search for the oldest non-stale worker. Slice off the stale ones. Efficient.

workerLoopQueue¶

A ring buffer. Pre-allocated with capacity slots.

type workerLoopQueue struct {
    items   []*goWorker
    expiry  []*goWorker
    head    int
    tail    int
    size    int
    isFull  bool
}

func (wq *workerLoopQueue) detach() *goWorker {
    if wq.isEmpty() { return nil }
    w := wq.items[wq.head]
    wq.items[wq.head] = nil
    wq.head = (wq.head + 1) % wq.size
    if wq.isFull { wq.isFull = false }
    return w
}

func (wq *workerLoopQueue) insert(w *goWorker) error {
    if wq.isFull { return errQueueIsFull }
    wq.items[wq.tail] = w
    wq.tail = (wq.tail + 1) % wq.size
    if wq.tail == wq.head { wq.isFull = true }
    return nil
}

FIFO. Round-robin allocation. Pros: deterministic memory usage (pre-allocated); spreads work across workers; predictable for benchmarks. Cons: more complex; loses LIFO cache benefits.

Choosing¶

Default in ants is workerStack. Better for general use.

Set WithPreAlloc(true) to use workerLoopQueue. Best when memory is constrained and you want predictable allocation.

Why both?¶

ants is used in many environments — TiDB (Go-on-server), CDN edges (Go-on-the-edge), embedded (Go-on-things). Each has different memory characteristics. Both queue types serve real needs.

Most engineers never need to choose; the default is right.

Deep Dive: ants Pool vs PoolWithFunc Performance¶

ants has two pool types: - Pool: each Submit takes a closure - PoolWithFunc: each Invoke takes an argument; the function is bound once

Performance difference?

Pool¶

p, _ := ants.NewPool(8)
for i := 0; i < N; i++ {
    arg := i
    p.Submit(func() {
        process(arg)
    })
}

Each Submit allocates a closure (capturing arg). The closure is small (~32 bytes) but has GC cost.

PoolWithFunc¶

p, _ := ants.NewPoolWithFunc(8, func(arg interface{}) {
    i := arg.(int)
    process(i)
})
for i := 0; i < N; i++ {
    p.Invoke(i)
}

Each Invoke sends just the argument. No closure allocation. The function was bound at pool creation.

Benchmarks¶

ants's benchmarks (on a recent Mac):

• Pool, Submit: ~150 ns/op, allocates ~32 bytes
• PoolWithFunc, Invoke: ~110 ns/op, allocates ~16 bytes
• Direct goroutine: ~1000 ns/op, allocates ~2KB stack

PoolWithFunc is ~30% faster than Pool. Both crush direct goroutine spawning by 10x.

At 1M req/s, that 40ns difference is 40 ms of CPU per second. Real.

When to choose¶

• All tasks call the same function: PoolWithFunc.
• Tasks vary: Pool.

A common pattern: have one PoolWithFunc per "type" of task. Image resize, email send, webhook deliver — each a separate pool with its own function.

Deep Dive: Hand-Rolling vs Using Libraries¶

When should you write your own pool?

Use ants when¶

• You need a battle-tested pool
• You want Tune(n) for runtime resize
• You want idle expiry built in
• You don't have a special requirement

This is the default. Don't write your own pool when ants works.

Use tunny when¶

• Workers have meaningful state
• Tasks are request-response
• You need explicit Worker interface

Use pond when¶

• You want clean ergonomics
• Task groups are useful
• No dynamic tuning needed

Hand-roll when¶

• Tasks have very specific shape (e.g., always batched of size N)
• You need integration with an unusual scheduler
• You are building a library others will use; minimal dependencies matter
• Education: build one to understand pools deeply

For most production code: don't hand-roll. Use a library. The cost of a wrong custom pool (bugs, perf issues, missing features) exceeds the benefit.

A hand-rolled minimal example¶

If you do hand-roll, ~150 lines suffice:

type Pool struct {
    jobs      chan func()
    quit      chan struct{}
    target    int32
    live      int32
    wg        sync.WaitGroup
    mu        sync.Mutex
    closed    bool
}

func New(initial int, queueSize int) *Pool {
    p := &Pool{
        jobs: make(chan func(), queueSize),
        quit: make(chan struct{}),
    }
    p.Resize(initial)
    return p
}

func (p *Pool) Submit(task func()) bool {
    select {
    case p.jobs <- task:
        return true
    default:
        return false
    }
}

func (p *Pool) Resize(target int) {
    p.mu.Lock()
    defer p.mu.Unlock()
    if p.closed { return }
    old := atomic.LoadInt32(&p.live)
    atomic.StoreInt32(&p.target, int32(target))
    if int32(target) > old {
        for i := old; i < int32(target); i++ {
            atomic.AddInt32(&p.live, 1)
            p.wg.Add(1)
            go p.worker()
        }
    }
}

func (p *Pool) worker() {
    defer p.wg.Done()
    for {
        if atomic.LoadInt32(&p.live) > atomic.LoadInt32(&p.target) {
            atomic.AddInt32(&p.live, -1)
            return
        }
        select {
        case task, ok := <-p.jobs:
            if !ok {
                atomic.AddInt32(&p.live, -1)
                return
            }
            p.run(task)
        case <-p.quit:
            atomic.AddInt32(&p.live, -1)
            return
        }
    }
}

func (p *Pool) run(task func()) {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("pool worker panic: %v", r)
        }
    }()
    task()
}

func (p *Pool) Close() {
    p.mu.Lock()
    p.closed = true
    close(p.quit)
    p.mu.Unlock()
    p.wg.Wait()
}

func (p *Pool) Size() int { return int(atomic.LoadInt32(&p.live)) }

Production-grade in spirit but missing features ants has (idle expiry, panic handler, metrics, multi-pool). For ~80% of cases, this is enough. For the other 20%, use ants.

Deep Dive: Designing for Observability from Day 1¶

When you build a dynamic pool, observability is not an add-on. It is core. Here is how to design for it.

Identify metrics¶

What questions will you ask in an incident?

• "Is the pool overloaded?" → size, queue, busy
• "Is autoscaler reacting?" → resize events, signals
• "Are tasks completing?" → completed rate, error rate
• "Are tasks slow?" → process time, wait time

Each question maps to a metric. Implement them all.

Instrument at source¶

Don't add metrics later. Add them while writing the pool.

func (p *Pool) Submit(task func()) bool {
    p.metrics.Submitted.Inc()
    submitted := time.Now()
    select {
    case p.jobs <- &Job{Task: task, Submitted: submitted}:
        return true
    default:
        p.metrics.Dropped.Inc()
        return false
    }
}

func (p *Pool) worker() {
    for job := range p.jobs {
        wait := time.Since(job.Submitted)
        p.metrics.Wait.Observe(wait.Seconds())
        p.metrics.Busy.Inc()
        start := time.Now()
        job.Task()
        p.metrics.Process.Observe(time.Since(start).Seconds())
        p.metrics.Busy.Dec()
        p.metrics.Completed.Inc()
    }
}