Dynamic Worker Scaling — Junior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Core Concepts
5. Real-World Analogies
6. Mental Models
7. Why a Static Pool Hurts
8. The First Dynamic Pool
9. Scale-Up Triggers
10. Scale-Down Triggers
11. Using ants.Tune
12. Code Examples
13. Coding Patterns
14. Clean Code
15. Product Use / Feature
16. Error Handling
17. Performance Tips
18. Best Practices
19. Edge Cases & Pitfalls
20. Common Mistakes
21. Common Misconceptions
22. Tricky Points
23. Test
24. Tricky Questions
25. Cheat Sheet
26. Self-Assessment Checklist
27. Summary
28. What You Can Build
29. Further Reading
30. Related Topics
31. Diagrams & Visual Aids

Introduction¶

Focus: "Why is a fixed number of workers wrong? How do I add and remove workers at runtime?"

A worker pool is a fixed-size set of goroutines, each pulling tasks from a shared channel. The classic shape is:

jobs := make(chan Job, 64)
for i := 0; i < 8; i++ {
    go worker(jobs)
}

That 8 is a guess. Sometimes it is right. Most of the time, in production, it is wrong:

• At 03:00 your service is idle; eight workers sleep on the channel, costing memory and a permanent place in the scheduler's run queues.
• At 09:00 traffic doubles; the queue fills; eight workers cannot keep up; jobs sit in the channel for minutes; downstream timeouts cascade.
• At noon a downstream API gets slow; each worker is parked on a network call; you have plenty of CPU but no workers free to take new jobs.

Dynamic worker scaling is the practice of changing the size of the pool while it is running. You start small. You watch one or more signals — typically queue depth or processing latency. When a signal crosses a threshold, you add workers. When the signal returns to normal, you remove them.

After reading this file you will:

• Understand why a fixed pool size is a guess, not a design
• Be able to write a pool that starts new workers when the queue is full
• Be able to write a pool that lets idle workers exit
• Know the basic ants.Tune(n int) API from the most common Go pool library
• Have a feel for scale-up triggers (queue depth, processing latency, utilization)
• Have a feel for scale-down triggers (idle time, low utilization, cooldown)
• Understand the simplest pitfalls: leaks when shrinking, oscillation, races on resize

You do not need to know AIMD, PID, Little's Law, or distributed coordination yet. Those come at the middle and senior levels. This file gets you to the first working dynamic pool.

Prerequisites¶

• Required: Comfortable with goroutines and channels. You can write a worker pool from scratch.
• Required: Familiarity with sync.WaitGroup, context.Context, and time.Ticker.
• Required: You know that close(ch) ends a for x := range ch loop and that sending on a closed channel panics.
• Helpful: Some exposure to backpressure — what happens when a channel buffer fills.
• Helpful: Awareness of runtime.NumCPU() and runtime.GOMAXPROCS.

If you can write a 50-line worker pool with a WaitGroup and clean shutdown, you are ready.

Glossary¶

Core Concepts¶

A pool is two numbers: target size and live size¶

A dynamic pool has a target size (what you want) and a live size (what currently exists). Resizing means making the live size approach the target size.

target = 16   ← what the autoscaler wants
live   = 12   ← what the pool currently has
                → spawn 4 more workers

If target < live, you must tell some workers to exit. This is the harder direction.

Scaling up is easy¶

To grow the pool, spawn new workers. They start running and pulling from the same channel as the existing ones.

for i := 0; i < toAdd; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for job := range jobs {
            process(job)
        }
    }()
}

That is it. The new workers compete with the old ones for jobs. No coordination needed because channels are MPMC (multi-producer multi-consumer) safe by construction.

Scaling down is hard¶

To shrink, you must convince a worker to exit its loop. You cannot kill a goroutine from outside. You must send it a signal.

Three common signals:

1. A "die" sentinel on the task channel. Workers that receive it return instead of processing.
2. A separate "shutdown" channel the worker selects on alongside the job channel.
3. An exit count the worker decrements with atomic.AddInt32; the first N workers to read a non-zero count return.

We will see (2) and (3) in the code examples below.

Triggers come from signals¶

A signal is a measurable property of the running pool. Common ones:

• Queue depth. len(jobs). If the buffer is near full, you are falling behind. Scale up.
• Average wait time. Time from submit to start processing. Rising wait time means the pool cannot keep up.
• Worker utilization. Fraction of workers currently processing vs idle. Near 100% means add workers; near 0% means remove them.
• Tail latency. p99 of processing time. If p99 spikes without code change, something downstream is slow — adding workers may help (more parallelism through slow downstream) or hurt (you create thundering herd on the slow downstream).

The simplest dynamic pool uses one signal — usually queue depth.

Real-World Analogies¶

Supermarket checkout lanes. When the queue at each lane grows beyond two carts, the manager opens a new lane. When the queue drops to zero and stays empty for ten minutes, the manager closes a lane. The manager is the autoscaler; the lanes are the workers; the carts in line are the queue depth.

A restaurant kitchen. During lunch rush, the head chef calls in two more line cooks. During mid-afternoon, he sends one home. The decision is based on how many tickets are on the rail (queue depth) and how long the oldest ticket has been sitting (wait time).

Cloud auto-scaling groups. Exactly the same problem at a different scale. An AWS Auto Scaling Group watches CPU; when CPU exceeds 70% for five minutes, it launches a new EC2 instance; when CPU drops below 30% for ten minutes, it terminates one. The 70/30 split is hysteresis — different thresholds for up and down. The five/ten-minute waits are cooldowns.

The math at each scale is the same. The vocabulary is the same. Once you understand a dynamic worker pool in Go, you understand the inside of a Kubernetes HPA.

Mental Models¶

Model 1: The pool is a tank with two valves¶

The inlet is task submissions. The outlet is workers processing tasks. If the inlet rate exceeds the outlet rate, the tank fills. If the tank gets near full, you must either:

• Open the outlet more (add workers)
• Close the inlet (apply backpressure to the submitters)

The two strategies are complementary. Dynamic scaling opens the outlet. Backpressure closes the inlet. Production systems usually need both.

Model 2: A pool is a control system¶

Think of the autoscaler as a thermostat. The temperature is the queue depth. The target temperature is, say, "queue 25% full." The heater is "add a worker." The AC is "let a worker exit."

A real thermostat does not flip on and off the instant temperature crosses 21°C. It has a deadband: turn heat on at 20.5°C, off at 21.5°C. Without the deadband, you'd hear the click-click-click of the relay every few seconds. The same is true for worker pools: without a deadband (hysteresis), you'll see spawn-exit-spawn-exit on every tick.

Model 3: Little's Law¶

For a stable queue:

L = λ × W

• L = average number of items in the system (workers busy + queue depth)
• λ = arrival rate (jobs per second)
• W = average time in the system (queue wait + processing)

If your jobs each take 100 ms and 200 arrive per second, you need at least 200 × 0.1 = 20 workers busy on average just to keep up. You should provision a bit more headroom: 25 to 30.

This is the formula behind most "how many workers should I have?" answers. We come back to it at senior level.

Why a Static Pool Hurts¶

Symptom 1: Idle cost¶

You set workers = 200 because peak hour needs them. At 03:00 you have 200 goroutines sleeping on a channel. The stacks alone are ~400 KB; the runtime keeps them in its scheduling tables. It is not catastrophic, but it is waste.

Symptom 2: Burst overload¶

You set workers = 8 because that's runtime.NumCPU(). Then a batch of 10000 jobs lands. The channel buffer of 64 fills in 6 ms. After that, every submit blocks (or, if you used a buffered channel without backpressure, the buffer grew unbounded and you OOM'd). Either way, the system is in trouble for the next several seconds.

Symptom 3: Latency cliff¶

For each job, downstream latency is 50 ms. With 8 workers, you can handle 8 / 0.05 = 160 req/s. At 161 req/s, the queue grows without bound. Latency goes from 50 ms to several seconds in under a minute.

A static pool sized for the average load cannot absorb a 2× burst. A static pool sized for peak load is mostly idle.

Symptom 4: Heterogeneous downstreams¶

Imagine half your jobs hit a fast cache (1 ms) and half hit a slow database (200 ms). The same eight workers handle both. The cache jobs are starved waiting for DB-bound workers to free up. You need either separate pools or a pool that can grow when DB workers are saturated.

The First Dynamic Pool¶

Here is the smallest pool that adjusts its size at runtime.

package main

import (
    "context"
    "fmt"
    "sync"
    "sync/atomic"
    "time"
)

type Pool struct {
    jobs        chan func()
    quit        chan struct{}
    targetSize  int32 // atomic
    liveSize    int32 // atomic
    wg          sync.WaitGroup
    mu          sync.Mutex
    closing     bool
}

func NewPool(initial int) *Pool {
    p := &Pool{
        jobs: make(chan func(), 1024),
        quit: make(chan struct{}),
    }
    p.Resize(initial)
    return p
}

func (p *Pool) Submit(task func()) {
    p.jobs <- task
}

func (p *Pool) Resize(target int) {
    p.mu.Lock()
    defer p.mu.Unlock()
    if p.closing {
        return
    }
    old := atomic.LoadInt32(&p.liveSize)
    atomic.StoreInt32(&p.targetSize, int32(target))
    if int32(target) > old {
        // Scale up: spawn (target - old) new workers.
        for i := old; i < int32(target); i++ {
            atomic.AddInt32(&p.liveSize, 1)
            p.wg.Add(1)
            go p.worker()
        }
    }
    // Scale down: workers notice that liveSize > targetSize and exit
    // on their own. We do not interrupt them mid-task.
}

func (p *Pool) worker() {
    defer p.wg.Done()
    defer atomic.AddInt32(&p.liveSize, -1)
    for {
        // Should I exit because the pool has shrunk?
        if atomic.LoadInt32(&p.liveSize) > atomic.LoadInt32(&p.targetSize) {
            return
        }
        select {
        case <-p.quit:
            return
        case task, ok := <-p.jobs:
            if !ok {
                return
            }
            task()
        }
    }
}

func (p *Pool) Close() {
    p.mu.Lock()
    p.closing = true
    close(p.quit)
    p.mu.Unlock()
    p.wg.Wait()
}

func main() {
    p := NewPool(2)
    var done int64
    for i := 0; i < 100; i++ {
        p.Submit(func() {
            time.Sleep(20 * time.Millisecond)
            atomic.AddInt64(&done, 1)
        })
    }
    // After a moment, scale up:
    time.Sleep(50 * time.Millisecond)
    p.Resize(16)
    // Then scale down once the burst passes:
    time.Sleep(300 * time.Millisecond)
    p.Resize(2)
    // Wait for everything to drain:
    for atomic.LoadInt64(&done) < 100 {
        time.Sleep(10 * time.Millisecond)
    }
    p.Close()
    fmt.Println("done:", done, "remaining workers:", atomic.LoadInt32(&p.liveSize))
}

The trick is in worker(): every loop iteration, before pulling a job, the worker checks whether the pool has shrunk. If liveSize > targetSize, it exits. The first worker to notice the shrink is the one that exits — there is no need to choose which worker dies; the race is harmless because all workers are interchangeable.

This is the opportunistic shrink: workers exit on their own when they next become idle. We do not interrupt in-flight tasks. This is the right default. Killing a task mid-processing is almost never acceptable in production.

Scale-Up Triggers¶

The simplest scale-up trigger is queue depth:

func (p *Pool) autoscaleByQueueDepth(ctx context.Context) {
    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            depth := len(p.jobs)
            cap := cap(p.jobs)
            usage := float64(depth) / float64(cap)
            current := int(atomic.LoadInt32(&p.targetSize))
            switch {
            case usage > 0.75 && current < 64:
                p.Resize(current + 2)
            case usage < 0.10 && current > 2:
                p.Resize(current - 1)
            }
        }
    }
}

Every 500 ms we look at the queue. If it is more than 75% full and we have fewer than 64 workers, add two. If it is less than 10% full and we have more than two workers, drop one.

The 75/10 thresholds form a deadband. We never resize when depth is between 10% and 75%. This is hysteresis: it prevents flapping.

Other scale-up signals:

• Wait time. Measure how long each task sat in the channel before a worker picked it up. Scale up when the moving average crosses, say, 50 ms.
• CPU. If CPU is < 60% and the queue is non-empty, you have headroom; add workers.
• External signal. A Kubernetes operator pushes a "scale to N" command in response to cluster-level metrics.

We will see a wait-time autoscaler at middle level.

Scale-Down Triggers¶

Scale-down is the dangerous direction. If you over-shrink, the next burst hits a small pool and you take a latency hit while you re-grow.

Conservative rules:

1. Wait longer to scale down than to scale up. A 5-second cooldown after a scale-up; a 30-second cooldown after a scale-down. Equivalently: be eager to add, slow to remove.
2. Use a higher upper bound and a lower lower bound. Add workers at queue 75% full; remove at queue 10% full. The gap in the middle is where you sit at steady state.
3. Never go below a floor. Always keep at least runtime.NumCPU() or some safe minimum. Going to zero means the first job pays a cold-start penalty.
4. Remove one at a time. Even if the signal says "you can drop ten," remove them gradually. A spike between ticks would catch you flat.

case usage < 0.10:
    last := p.lastShrink.Load()
    if time.Since(last.(time.Time)) > 30*time.Second && current > p.floor {
        p.Resize(current - 1)
        p.lastShrink.Store(time.Now())
    }

The p.lastShrink is an atomic.Value holding the time of the last shrink. We only shrink if at least 30 seconds have passed since the last one.

Using ants.Tune¶

The most popular Go pool library is panjf2000/ants. It supports runtime resize:

import "github.com/panjf2000/ants/v2"

p, err := ants.NewPool(8)
if err != nil { panic(err) }
defer p.Release()

for i := 0; i < 1000; i++ {
    _ = p.Submit(func() {
        time.Sleep(20 * time.Millisecond)
    })
}

// Later, in your autoscaler:
p.Tune(32) // grow to 32
// later still:
p.Tune(4)  // shrink to 4

Tune changes the maximum number of in-flight goroutines. If you tune up, future submissions can spawn more workers. If you tune down, ants does not kill in-flight goroutines; they finish their current task, return to ants's free list, and from there they exit on the next idle pass.

The library handles the bookkeeping for you. The job of your code is just to decide when to call Tune. That decision — the autoscaler — is the topic of this whole subsection.

ants is heavily benchmarked and used in production at large scale (TiDB, several CDN edges, Bilibili). For most projects, you will not roll your own pool; you will use ants or tunny or pond. But you should understand how they work — that is exactly what we are building above.

Code Examples¶

Example 1: Hand-rolled pool with Tune API¶

We saw this above. Re-read the structure: target size atomic, live size atomic, opportunistic shrink in worker().

Example 2: ants pool with queue-depth autoscaler¶

package main

import (
    "context"
    "log"
    "sync/atomic"
    "time"

    "github.com/panjf2000/ants/v2"
)

func main() {
    p, err := ants.NewPool(4, ants.WithNonblocking(false))
    if err != nil {
        log.Fatal(err)
    }
    defer p.Release()

    var submitted int64
    var done int64

    // Producer.
    go func() {
        for {
            err := p.Submit(func() {
                time.Sleep(30 * time.Millisecond)
                atomic.AddInt64(&done, 1)
            })
            if err == nil {
                atomic.AddInt64(&submitted, 1)
            }
            time.Sleep(2 * time.Millisecond)
        }
    }()

    // Autoscaler.
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    autoscale(ctx, p)
}

func autoscale(ctx context.Context, p *ants.Pool) {
    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()
    const minCap, maxCap = 2, 128
    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            running := p.Running()
            free := p.Free()
            cap := p.Cap()
            utilization := float64(running) / float64(cap)
            switch {
            case utilization > 0.80 && cap < maxCap:
                p.Tune(min(cap*2, maxCap))
            case utilization < 0.20 && cap > minCap && free > cap/2:
                p.Tune(max(cap/2, minCap))
            }
        }
    }
}

func min(a, b int) int { if a < b { return a }; return b }
func max(a, b int) int { if a > b { return a }; return b }

The autoscaler runs once every 500 ms. It looks at running, free, and capacity, computes utilization, and doubles or halves the pool. Doubling and halving is multiplicative — fast to react but coarse. We will see additive-increase later (AIMD).

Example 3: Pool with wait-time as the signal¶

type Pool struct {
    jobs        chan Job
    waitMu      sync.Mutex
    waitSamples []time.Duration
    // ...
}

type Job struct {
    Task       func()
    submitted  time.Time
}

func (p *Pool) Submit(task func()) {
    p.jobs <- Job{Task: task, submitted: time.Now()}
}

func (p *Pool) worker() {
    for job := range p.jobs {
        wait := time.Since(job.submitted)
        p.recordWait(wait)
        job.Task()
    }
}

func (p *Pool) recordWait(d time.Duration) {
    p.waitMu.Lock()
    p.waitSamples = append(p.waitSamples, d)
    if len(p.waitSamples) > 100 {
        p.waitSamples = p.waitSamples[1:]
    }
    p.waitMu.Unlock()
}

func (p *Pool) avgWait() time.Duration {
    p.waitMu.Lock()
    defer p.waitMu.Unlock()
    if len(p.waitSamples) == 0 {
        return 0
    }
    var sum time.Duration
    for _, s := range p.waitSamples {
        sum += s
    }
    return sum / time.Duration(len(p.waitSamples))
}

The autoscaler can now look at avgWait() instead of len(jobs). Wait time is a better signal because it is in the same units as your SLO ("p99 wait < 100 ms"). Queue length is easier to read but harder to translate into SLO commitments.

Example 4: Shrink-on-idle workers¶

A different model: each worker decides to exit if it has been idle for some duration.

func (p *Pool) worker() {
    idleTimer := time.NewTimer(p.idleTimeout)
    defer idleTimer.Stop()
    for {
        select {
        case task, ok := <-p.jobs:
            if !ok {
                return
            }
            if !idleTimer.Stop() {
                <-idleTimer.C
            }
            task()
            idleTimer.Reset(p.idleTimeout)
        case <-idleTimer.C:
            if atomic.LoadInt32(&p.liveSize) > p.floor {
                atomic.AddInt32(&p.liveSize, -1)
                return
            }
            idleTimer.Reset(p.idleTimeout)
        case <-p.quit:
            return
        }
    }
}

If a worker is idle for p.idleTimeout and the pool is above its floor, the worker exits. This is MaxIdleTime in database/sql, the worker-KeepAlive in net/http.Server, and the MaxIdleWorkers in many job queue libraries. It is a beautifully decentralized form of autoscaling — each worker decides on its own, no external loop needed.

Combine it with an external scale-up loop (which only adds workers; never removes) and you get the most common production shape: external scale-up, decentralized scale-down.

Coding Patterns¶

Pattern 1: Separate signal collection from decision¶

The function that measures the signal should not also be the function that decides to resize. Mixing them makes both untestable and creates threading messes.

type Signal interface {
    Sample() float64
}

type Decision func(current int, signal float64) (newSize int, changed bool)

func RunAutoscaler(ctx context.Context, p Resizer, s Signal, d Decision, every time.Duration) {
    t := time.NewTicker(every)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done(): return
        case <-t.C:
            v := s.Sample()
            new, changed := d(p.Size(), v)
            if changed { p.Resize(new) }
        }
    }
}

Now Signal and Decision are both pure values you can unit-test trivially. You can swap one decision rule for another (depth-based, wait-based, hybrid) without rewriting the loop.

Pattern 2: Floor and ceiling¶

Always have a hard min and max:

type Bounds struct{ Min, Max int }

func (b Bounds) Clamp(n int) int {
    if n < b.Min { return b.Min }
    if n > b.Max { return b.Max }
    return n
}

A buggy autoscaler that wants to resize to 0 or 100000 will be silently corrected.

Pattern 3: Cooldown between resizes¶

Pair every resize with a timestamp:

type Cooldown struct {
    UpAfter   time.Duration
    DownAfter time.Duration
    lastUp    time.Time
    lastDown  time.Time
}

func (c *Cooldown) AllowUp(now time.Time) bool   { return now.Sub(c.lastUp) >= c.UpAfter }
func (c *Cooldown) AllowDown(now time.Time) bool { return now.Sub(c.lastDown) >= c.DownAfter }

Typical settings: UpAfter = 5s, DownAfter = 30s. Scale up fast, scale down slow.

Pattern 4: Expose metrics¶

A pool that does not export metrics is a pool you cannot tune. At minimum:

• Current size (target and live)
• Queue depth and queue capacity
• Tasks submitted, completed, dropped
• p50, p99 wait time
• p50, p99 processing time

Use Prometheus client library and gauge/counter/histogram. Without these you are flying blind.

Clean Code¶

• Name your knobs. MaxWorkers, MinWorkers, ScaleUpThreshold, ScaleDownThreshold, ScaleUpCooldown, ScaleDownCooldown, IdleTimeout. Don't sprinkle magic numbers.
• Make resize idempotent. Calling Resize(20) twice in a row should be a no-op the second time. Use atomic comparisons.
• Atomic operations for size counters. int32 with sync/atomic. Avoid mixing atomic and non-atomic access.
• Keep workers stateless. The pool may shrink under their feet; a worker should not own any state that survives its goroutine's exit.
• Don't share *time.Timer between goroutines. Each worker has its own idle timer. Sharing timers across workers causes deeply confusing bugs.

Product Use / Feature¶

Where you'd actually want a dynamic pool:

• Email/SMS sending. Off-peak: 4 workers. During a marketing blast: 80 workers. Then back to 4.
• Image thumbnail generation. A user upload burst doubles the queue for a few minutes. Add CPU-bound workers; remove them after.
• Outbound webhook deliveries. Each is an HTTP call with variable latency. Autoscale by wait time, not depth — depth is unreliable when each task is slow.
• PDF/report generation. Some reports take 2 seconds; some take 30. Without scaling, the 30-second jobs block the 2-second jobs. Scale by queue wait time.
• Background data sync. A worker per partition; you may want to add workers when one partition gets behind.

In each case, the same template works: collect signal, decide, resize, repeat.

Error Handling¶

Three failure modes to anticipate:

1. Scale-up blocked by upstream¶

You decide to add 10 workers. The runtime cannot allocate stacks (out of memory). Should the autoscaler crash? No — log a warning, stay at the current size, try again next tick. Autoscalers should be best-effort: they cannot make hardware appear.

2. Shrink during in-flight task¶

A worker that is mid-task does not respond to "exit." The right pattern is: workers check the shrink signal only at the top of their loop, between tasks. Never interrupt a task in flight. If you must interrupt (the task is hung), use context.WithTimeout per task — the task code itself is responsible for cancellation.

3. Panic in a worker¶

If a worker panics, the whole pool can die. Wrap each task in a recover:

func (p *Pool) safeRun(task func()) {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("pool worker recovered from panic: %v", r)
        }
    }()
    task()
}

Without this, one buggy submitted function takes down the program. ants does this for you by default.

Performance Tips¶

• Don't poll faster than you can react. A 100 ms autoscaler tick is overkill if your workers take 200 ms per task. Use 500 ms or 1 s.
• Use atomic counters, not channels for signals. A channel send per task to "count" is overhead; an atomic.AddInt64 is essentially free.
• Prefer adding to halving (or doubling). Multiplicative growth lets you handle bursts; additive shrink prevents over-correction. This is AIMD; we cover it at senior level.
• Cache runtime.NumCPU(). It does a syscall. Compute it once at startup.
• Avoid lock contention in hot paths. A pool that takes a mutex on every submit will not outscale a static pool. Use channels and atomics.

Best Practices¶

1. Start with a static pool. Add dynamic resize only when you have evidence it would help — measure queue depth and processing latency for a week first.
2. Define a min, a max, and a cooldown before writing any code. Without these three, you will write a bug.
3. Make the autoscaler observable. Export a gauge for pool size, a counter for resize events, and a histogram for wait time.
4. Test with synthetic bursts. Hit your pool with 10× normal load for 5 seconds and watch the size graph.
5. Be eager to add, slow to remove. Cost of an extra worker is bytes; cost of a missing worker during a burst is missed SLO.
6. Always have a floor. A pool of size 0 cannot start immediately.
7. Don't react to single samples. Use a moving average over the last N ticks.

Edge Cases & Pitfalls¶

Pitfall 1: Sending on a channel after the last worker exits¶

If your pool shrinks to zero workers, any pending Submit blocks forever. Defense: keep a floor >= 1, or have Submit time out.

Pitfall 2: Tune(n) does not kill in-flight goroutines¶

In ants, calling Tune(2) when 50 are running does not terminate any of them. They finish their current task and return to the pool's idle pool; only then is the cap enforced. If you need immediate shrink for cost reasons, the only safe path is to cancel in-flight tasks via their context.Context.

Pitfall 3: Counting the wrong number¶

p.Running() in ants is the number of busy workers. p.Cap() is the maximum allowed. p.Free() is Cap - Running. Mixing these up gives you a backwards autoscaler — one that scales up when it should scale down.

Pitfall 4: Resize while closing¶

If you call Resize(N) after Close(), you may try to spawn workers on a closed pool. Guard with a closing flag, or use sync.Once on Close.

Pitfall 5: Stale signals after a resize¶

Just after you double the pool, queue depth drops fast — but it took some ticks to drain. If you re-evaluate too soon, you'll see "queue still high" and double again. Use a cooldown to avoid this stutter.

Pitfall 6: Per-tick cost vs steady-state cost¶

A 100 ms tick that spends 5 ms collecting samples costs you 5% of one CPU. On a 32-core machine that may be fine. On a 1-core function-as-a-service container it is not.

Common Mistakes¶

1. No cooldown. Pool oscillates between 8 and 32 every tick. Tail latency spikes match each cycle.
2. No floor. Pool drops to 0 during off-peak. First request after off-peak waits seconds for a cold start.
3. Using len(ch) racily. len(ch) is well-defined but the value is stale by the time you use it. Average over time.
4. Killing workers mid-task. Goroutines cannot be killed in Go. Trying to (with goroutine-IDs, third-party libs, etc.) is a code smell.
5. Ignoring the producer side. A growing queue might mean the consumer is slow or the producer is too fast. Adding workers doesn't help if the producer is misbehaving.
6. Scaling on instant signals. Single-sample resizes (one tick of high depth → 2× the pool) overshoot. Average over 3-5 samples.
7. Multiple autoscalers fighting. If you run two autoscaler goroutines, they will fight. Use a single goroutine with a mutex on Resize.
8. Forgetting recover in workers. One panic and the worker count drops; soon the pool is dead.
9. Submitting from inside a worker. Causes deadlock if the queue is full and all workers are blocked on submit.
10. Treating runtime.NumGoroutine() as the worker count. It includes every goroutine in the program, not just pool workers. Track your own counter.

Common Misconceptions¶

• "Dynamic = better." Not for low-volume, predictable workloads. A static pool is simpler, cheaper, and easier to reason about. Dynamic is a tool, not a virtue.
• "More workers = more throughput." Only up to the bottleneck. After that, more workers just queue at the bottleneck.
• "The pool size should equal NumCPU." Only for CPU-bound work. For I/O-bound it can be 10× or 100× NumCPU.
• "Tune(n) shrinks instantly." ants and most libraries shrink opportunistically. Workers finish their task first.
• "You can interrupt a goroutine." Not in Go. You must cooperate via context or channel signals.
• "Channel buffer is the queue." It is the most-buffered queue, but real wait time depends on how fast workers drain it. Two pools with buffer 1024 can have very different wait times.

Tricky Points¶

• len(ch) is a snapshot, not a transaction. Between reading it and deciding, the value changes. That's fine for autoscaling, where signals are inherently fuzzy. Just be aware.
• time.Ticker drifts. Over hours, a 500 ms ticker may produce ticks slightly off; the drift is bounded but nonzero. Don't rely on exact timing.
• Atomic loads can be reordered with other loads. liveSize and targetSize may be observed inconsistently by a worker. The result is at worst one extra or missing exit; the system corrects itself on the next iteration.
• Tune(0) in ants is allowed and immediately stalls submissions. Surprising. Test it.
• Idle timeout interacts with the floor. If idleTimeout = 30s and floor = 4, the first four idle workers will never exit, but the fifth and beyond will.
• A pool that has shrunk leaves "ghost" goroutine count for a few ticks. This is the gap between "decided to exit" and "actually exited." Don't panic when you see it.

Test¶

How do you test a dynamic pool?

1. Unit test the decision function. It is pure: given current size and signal value, returns new size. Trivially testable.
2. Test resize with a slow task generator. Submit at a known rate; verify pool size converges within N seconds.
3. Test under burst. Submit 1000 tasks at once; verify pool grows, then check it shrinks afterwards.
4. Test the shutdown path. Make sure all workers exit, no leak. Use goleak.
5. Race detector. go test -race reveals concurrent map writes, missing atomics, etc.

func TestPoolGrowsUnderLoad(t *testing.T) {
    p := NewPool(2)
    defer p.Close()
    var done int64
    for i := 0; i < 500; i++ {
        p.Submit(func() {
            time.Sleep(10 * time.Millisecond)
            atomic.AddInt64(&done, 1)
        })
    }
    // Autoscaler should have grown the pool.
    time.Sleep(200 * time.Millisecond)
    if atomic.LoadInt32(&p.liveSize) < 8 {
        t.Errorf("expected pool to grow, got %d", p.liveSize)
    }
}

Tricky Questions¶

1. If len(ch) is racy, why is it OK for autoscaling? Because we average and we resize gradually. A 5% wrong reading does not affect the long-term resize trend. We are not making a financial decision; we are nudging a control system.

2. Can a pool ever shrink while a worker is blocked on a downstream HTTP call? The worker continues; only when it returns to the top of its loop will it exit. The pool's "live" count drops only then. The autoscaler should account for this lag.

3. Why scale up faster than scale down? Asymmetric costs. A missing worker during a spike causes a missed SLO. An extra worker during steady state costs a few KB.

4. Why is one autoscaler goroutine enough? Resize is rare (milliseconds per minute). One goroutine ticking every 500 ms is plenty. Two would just fight.

5. What happens if every worker panics? The pool's live count drops to zero. New submissions block forever (if the channel is unbuffered) or fill the buffer and then block. The autoscaler may or may not respawn workers — depends on your design. Production: respawn with backoff; alert.

Cheat Sheet¶

target = 16     # desired
live   = 12     # actual
queue  = 80%    # signal

if queue > 75% and live < max:      live += 2     # scale up
if queue < 10% and live > min:      live -= 1     # scale down
respect cooldowns: up=5s, down=30s

// Pool API to memorise:
p.Resize(n)         // change target
p.Submit(task)      // enqueue
p.Size()            // current
p.Close()           // drain and exit

// ants:
p, _ := ants.NewPool(8)
p.Tune(32)          // grow
p.Tune(4)           // shrink (opportunistic)
p.Running()         // busy workers
p.Free()            // idle slots
p.Cap()             // max
p.Release()         // close

Self-Assessment Checklist¶

• I can explain why a static pool is a guess
• I can write a pool with Resize(n) from scratch
• I can write a queue-depth autoscaler with hysteresis
• I can use ants.Tune to grow and shrink a pool
• I can list five common mistakes (no cooldown, no floor, etc.)
• I can describe the asymmetry: fast up, slow down
• I can write a wait-time signal collector
• I can write a worker that shrinks itself on idle timeout
• I understand why scale-down is harder than scale-up
• I can think of one workload where dynamic scaling is the wrong choice

If you have nine checked, move on to the middle level.

Summary¶

A worker pool is two numbers: target size and live size. Static pools fix both. Dynamic pools change the target at runtime and let workers converge.

Scale up by spawning. Scale down opportunistically — workers check between tasks. Use hysteresis (different up/down thresholds) and cooldown (minimum interval between resizes) to prevent oscillation. Choose a signal that maps to your SLO: queue depth is easy, wait time is more meaningful. Use ants Tune(n) for the heavy lifting; the autoscaler is the part you write.

Three rules: have a floor. Be eager to add, slow to remove. Measure before you tune.

What You Can Build¶

• An email sender that auto-scales workers between 4 and 64 based on outbox depth
• A thumbnail service that uses ants and Tune based on queue wait time
• A job queue with per-tenant pools, each with its own autoscaler
• A webhook delivery worker pool with idle-timeout shrink and queue-depth grow

Further Reading¶

• The panjf2000/ants source — pool.go and worker_loop_queue.go show exactly how Tune works
• "Little's Law for Software Engineers" — Brendan Gregg
• TCP congestion control basics — the AIMD pattern in RFC 5681
• Go memory model — for understanding atomic semantics

Related Topics¶

• Backpressure — what happens when the queue is full and the pool cannot keep up
• Graceful shutdown — how to close a dynamic pool without losing in-flight tasks
• Worker pool patterns — the static baseline this whole topic improves on
• Context cancellation — the primitive for per-task interruption

Deep Dive: How Resize Actually Happens¶

The whole skill of dynamic scaling is in the careful choreography of one tiny window of time: the moment a Resize(n) call is executing while workers are simultaneously taking work from the channel and finishing previous tasks. Most of the bugs you will ever write in this topic happen in this window. Let us slow it down and study it frame by frame.

Frame 0: steady state¶

Live = 4, Target = 4. Queue has 60 of 1024 capacity used. Four goroutines are each parked on <-jobs, waking up roughly every 5 ms to take a task.

Frame 1: autoscaler decides¶

A scaler goroutine ticks. It samples len(jobs) = 950 (queue is almost full). It computes utilization ratio 950/1024 = 0.93. The decision rule says: utilization > 0.75 and current < max, so resize from 4 to 8. The autoscaler calls pool.Resize(8).

Frame 2: Resize takes the mutex¶

Resize acquires p.mu. Inside the critical section it reads liveSize = 4, sets targetSize = 8, and loops for i := 4; i < 8; i++. On each iteration it calls atomic.AddInt32(&p.liveSize, 1) first (so the live count is consistent before the goroutine starts), then wg.Add(1), then go p.worker().

At this point, between the atomic add and the go, an existing worker could observe liveSize = 5 even though only four goroutines exist. This is a benign race: the inconsistency is tiny and never used for correctness, only for scaling decisions. The next autoscaler tick will see the right number.

Frame 3: new workers race to the channel¶

The four new goroutines are scheduled. The first one wakes up, calls select on jobs and quit, picks a task, runs it. The second does the same. Within a few microseconds, all four new workers have processed tasks. Queue depth is dropping fast because eight workers are draining instead of four.

Frame 4: queue depth observed by next scaler tick¶

500 ms later, the scaler ticks again. len(jobs) is now 150. Utilization is 0.15. The decision rule: utilization < 0.10 is false (we are at 0.15), so do nothing. We have crossed the lower band but not by enough — the deadband prevents a panic-shrink.

Frame 5: producer eases off¶

Another tick. Queue drops to 80 (utilization 0.08). Below the 0.10 threshold. The scaler proposes a resize. But — cooldown check — only 1 s has passed since the last scale up. Down-cooldown is 30 s. The scaler logs "would scale down but cooldown active" and waits.

Frame 6: cooldown elapses¶

29 seconds later (so 30 s after the scale-up), the scaler ticks. Queue is steadily at 50 (utilization 0.05). The scaler calls Resize(7). Now targetSize = 7, liveSize = 8. No goroutines are spawned. Workers continue running.

Frame 7: one worker shrinks itself¶

The next time any worker finishes a task and loops back to the top, it reads liveSize = 8, targetSize = 7. It exits, decrementing liveSize to 7. Now liveSize == targetSize. Other workers loop, find the equality, and continue running.

This is the opportunistic shrink in detail. Notice three properties:

1. There is no central "kill this worker" decision. Workers cooperate.
2. Shrink is asynchronous. The autoscaler does not wait for the shrink to complete.
3. The shrink may take any amount of time — until the next task completes — depending on processing latency. If tasks take 10 minutes, shrink takes up to 10 minutes.

If your workers have very long tasks, opportunistic shrink is slow. For those, you may need cooperative cancellation: workers periodically check ctx.Done() and exit even mid-task. Most pools should not do this — they exist for short tasks where opportunistic is fine.

Frame 8: pool reaches steady state again¶

Live = 7, Target = 7. The autoscaler keeps polling but does not change anything until the next signal change. The system is at rest.

This entire dance — frames 0 through 8 — happens hundreds of times per day in a busy service. Most of the work in writing a dynamic pool is making sure each frame is correct, in isolation, under all orderings of other goroutines.

Anatomy of a Resize Call¶

Here is the minimum correct Resize method, annotated:

func (p *Pool) Resize(target int) {
    // Clamp first. A buggy caller asking for 100000 should not crash us.
    if target < p.floor {
        target = p.floor
    }
    if target > p.ceiling {
        target = p.ceiling
    }

    // Take the mutex. Resize is rare and serialised; this is fine.
    p.mu.Lock()
    defer p.mu.Unlock()

    // Are we mid-shutdown? Refuse new spawns.
    if p.closing {
        return
    }

    old := atomic.LoadInt32(&p.liveSize)
    atomic.StoreInt32(&p.targetSize, int32(target))

    // Are we growing?
    if int32(target) > old {
        toAdd := int32(target) - old
        for i := int32(0); i < toAdd; i++ {
            // Increment liveSize BEFORE the go statement.
            // If we did it after, a fast tick could read the wrong value.
            atomic.AddInt32(&p.liveSize, 1)
            p.wg.Add(1)
            go p.worker()
        }
    }
    // If shrinking, do nothing here. Workers handle their own exit.
}

Three details are easy to get wrong:

• Order of atomic increment vs go. Increment first; otherwise the autoscaler may observe live < real for a brief window and compound the error.
• wg.Add(1) before go. The classic WaitGroup rule.
• Lock held across go. Acceptable here because go is fast (microseconds). For a pool with thousands of growing workers, you would want to release the lock and spawn outside of it.

Two Common Worker Shapes¶

There are two natural shapes for a worker goroutine in a dynamic pool. Both are correct; they differ in cost and behaviour.

Shape A: Check on every iteration¶

func (p *Pool) workerA() {
    defer p.wg.Done()
    for {
        if atomic.LoadInt32(&p.liveSize) > atomic.LoadInt32(&p.targetSize) {
            atomic.AddInt32(&p.liveSize, -1)
            return
        }
        select {
        case <-p.quit:
            atomic.AddInt32(&p.liveSize, -1)
            return
        case task, ok := <-p.jobs:
            if !ok {
                atomic.AddInt32(&p.liveSize, -1)
                return
            }
            task()
        }
    }
}

• Pro: shrink reacts within one task's duration
• Pro: simple to reason about
• Con: two atomic loads on every loop iteration (cheap but not free)

Shape B: Signal-driven shrink¶

func (p *Pool) workerB() {
    defer p.wg.Done()
    for {
        select {
        case <-p.shrink:
            atomic.AddInt32(&p.liveSize, -1)
            return
        case <-p.quit:
            atomic.AddInt32(&p.liveSize, -1)
            return
        case task, ok := <-p.jobs:
            if !ok {
                atomic.AddInt32(&p.liveSize, -1)
                return
            }
            task()
        }
    }
}

Here p.shrink is a separate channel. Resize(target) when shrinking sends (old - target) empty values to p.shrink. Each send wakes one worker which exits.

• Pro: no per-iteration overhead
• Pro: tight control over exact count
• Con: a worker mid-task does not hear the shrink until the task finishes; you still get the same lag as Shape A
• Con: a bug in Resize can over-send or under-send to p.shrink

Shape A is more common because the bookkeeping is centralised in Resize. Shape B is what ants does internally.

What ants Does Under the Hood¶

We have called ants.Tune(n) a few times. Let's look at what really happens, simplified from the v2 source.

When you create a pool:

p, _ := ants.NewPool(8)

ants allocates:

• A circular slice of *goWorker references, the "free list"
• A sync.Cond for blocking submissions when the pool is full
• A capacity int32 (atomic) — the pool's current cap

When you submit:

p.Submit(task)

ants does roughly:

w := p.retrieveWorker()
w.task <- task

retrieveWorker() either pops a worker from the free list, or — if the live count is below capacity — spawns a new one. Each worker is a goroutine that loops on a per-worker task chan. After running a task, the worker puts itself back on the free list and waits for the next one.

When you call Tune(n):

p.Tune(32)

ants atomically sets capacity = 32. It does nothing else. The growth happens lazily: the next submission that finds the free list empty and the live count below 32 spawns a new worker.

When you call Tune(2) after running with 50 workers:

p.Tune(2)

capacity = 2 is set. But there are 50 goroutines alive. ants does not touch them. As each finishes its current task and tries to put itself back on the free list, it checks: is live > capacity? If yes, the worker exits instead of going back to the free list.

So ants implements Shape B with a slight twist: shrink is checked at the moment a worker would re-enqueue itself, not at the top of its loop. The effect is the same: opportunistic shrink, bounded by current task duration.

The free list is bounded by capacity; when shrinking, the excess workers eventually exit and the bookkeeping consistency restores itself.

Implications¶

1. Tune(n) is O(1) in time. It never blocks.
2. After Tune(n) returns, the pool is eventually at size n, not immediately.
3. Calling Tune rapidly is safe but mostly useless: only the last value matters because shrink is opportunistic.
4. ants is designed for high-throughput, low-latency workloads. Tuning is rarely the bottleneck.

Designing the Autoscaler Loop¶

You will write some variation of this loop ten times in your career. Let us look at the canonical shape:

func Autoscale(ctx context.Context, p Pool, opts AutoscaleOpts) {
    ticker := time.NewTicker(opts.Interval)
    defer ticker.Stop()

    var lastUp, lastDown time.Time

    for {
        select {
        case <-ctx.Done():
            return
        case now := <-ticker.C:
            signal := opts.Signal()
            current := p.Size()
            decision := opts.Decide(current, signal)

            switch {
            case decision > current:
                if now.Sub(lastUp) < opts.UpCooldown {
                    continue
                }
                p.Resize(decision)
                lastUp = now
            case decision < current:
                if now.Sub(lastDown) < opts.DownCooldown {
                    continue
                }
                p.Resize(decision)
                lastDown = now
            }
        }
    }
}

The AutoscaleOpts type:

type AutoscaleOpts struct {
    Interval     time.Duration
    UpCooldown   time.Duration
    DownCooldown time.Duration
    Signal       func() float64
    Decide       func(current int, signal float64) int
}

Three pieces are pluggable:

1. Signal. How do you measure current load? Queue depth? Wait time? Utilization?
2. Decide. Given a signal and current size, what is the new size?
3. Cooldowns. How long must pass between resizes of each direction?

A unit test for the loop wires up mock signal and decide:

func TestAutoscaler(t *testing.T) {
    var pool fakePool
    pool.size = 8
    signal := 0.9
    sig := func() float64 { return signal }
    dec := func(cur int, v float64) int {
        if v > 0.75 { return cur + 1 }
        return cur
    }
    ctx, cancel := context.WithCancel(context.Background())
    go Autoscale(ctx, &pool, AutoscaleOpts{
        Interval: 5 * time.Millisecond,
        Signal: sig,
        Decide: dec,
        UpCooldown: 0,
        DownCooldown: 0,
    })
    time.Sleep(50 * time.Millisecond)
    cancel()
    if pool.size <= 8 {
        t.Errorf("expected scaling up, got %d", pool.size)
    }
}

Notice how signal and decide are pure functions of their inputs. They have no globals. The loop has no business logic. This is the right shape because each piece is independently testable.

Three Concrete Workloads, Three Different Decisions¶

The right autoscaler shape depends on what your service does. Let us walk through three real workloads.

Workload A: Outbound email¶

• Each email send: SMTP handshake + body upload, ~150 ms median, 600 ms p99
• I/O-bound (network)
• Bursts during marketing campaigns (10× normal for 5–10 minutes)

Signal: queue depth. Bursts fill the queue in seconds.

Decision: if depth > 60% capacity, add 4 workers; if < 20%, remove 2.

Cooldowns: up=2s, down=60s.

Floor: 4. Ceiling: 200.

Reasoning: email is I/O-bound; we can afford 200 workers (400 KB stacks). Bursts need fast reaction. Shrink slowly because campaigns often come in waves.

Workload B: Image resizing¶

• Each image: 50–200 ms, mostly CPU
• CPU-bound; one worker per available core gets full throughput
• Variable burst pattern (user uploads)

Signal: queue depth.

Decision: if depth > 50% and live < NumCPU * 1.5, add 1; if depth = 0 and idle for 30s, remove 1.

Cooldowns: up=5s, down=120s.

Floor: 2. Ceiling: NumCPU * 1.5.

Reasoning: CPU-bound work cannot benefit from more workers than cores + a small overlap. The ceiling reflects this. Shrink very slowly because spinning up new workers costs cache warm-up.

Workload C: Webhook delivery¶

• Each delivery: outbound HTTP, 10 ms to 10 s, very high variance
• I/O-bound but with extreme variance
• Steady stream of webhooks, occasional spikes

Signal: wait time. Queue depth is misleading because slow webhooks make the queue grow even without a true overload.

Decision: if avg wait > 200 ms over last 10 samples, add 5; if avg wait < 20 ms, remove 1.

Cooldowns: up=3s, down=30s.

Floor: 8. Ceiling: 500.

Reasoning: variance is so high that depth-based scaling overreacts. Wait time integrates over many tasks and is more stable.

The Resize Decision Function in Detail¶

The most important function in the whole topic is Decide. Let us look at three versions, increasingly sophisticated.

Version 1: Step¶

func DecideStep(cur int, sig float64) int {
    switch {
    case sig > 0.75:
        return cur + 1
    case sig < 0.10:
        return cur - 1
    default:
        return cur
    }
}

Add one, remove one. Simple. Slow to react during a 10× burst (would need many ticks). Tail-latency-friendly. The right default for many workloads.

Version 2: Multiplicative¶

func DecideMultiplicative(cur int, sig float64) int {
    switch {
    case sig > 0.75:
        return cur * 2
    case sig < 0.10:
        return cur / 2
    default:
        return cur
    }
}

Doubles and halves. Fast reaction to bursts. Coarse. Wastes capacity briefly during the overshoot. The right choice for very bursty workloads (background batch jobs).

Version 3: AIMD¶

func DecideAIMD(cur int, sig float64) int {
    switch {
    case sig > 0.75:
        return cur + 1            // additive increase
    case sig < 0.10:
        return cur - cur/4        // multiplicative decrease (-25%)
    default:
        return cur
    }
}

Additive increase, multiplicative decrease. Borrowed from TCP. Tail latency is good because growth is gradual. Idle capacity vanishes fast because shrink is aggressive. The right choice for cost-sensitive cloud deployments.

There are also AIAD (additive both), MIAD (multiplicative up, additive down) — they exist in the literature but are rarely used because their behavioural properties are worse.

We cover AIMD with care at senior level. For now, internalise: there is no universal best. The decision rule depends on cost asymmetry.

Hands-on: Build a Pool from Scratch in 100 Lines¶

Let us assemble a complete, runnable dynamic pool. Save as pool.go:

package main

import (
    "context"
    "fmt"
    "log"
    "sync"
    "sync/atomic"
    "time"
)

type Pool struct {
    jobs       chan func()
    quit       chan struct{}
    targetSize int32
    liveSize   int32
    wg         sync.WaitGroup
    mu         sync.Mutex
    closing    bool
    floor      int32
    ceiling    int32
}

func NewPool(initial, floor, ceiling int) *Pool {
    p := &Pool{
        jobs:    make(chan func(), 1024),
        quit:    make(chan struct{}),
        floor:   int32(floor),
        ceiling: int32(ceiling),
    }
    p.Resize(initial)
    return p
}

func (p *Pool) Submit(task func()) bool {
    select {
    case p.jobs <- task:
        return true
    default:
        return false
    }
}

func (p *Pool) Resize(target int) {
    if target < int(p.floor) {
        target = int(p.floor)
    }
    if target > int(p.ceiling) {
        target = int(p.ceiling)
    }
    p.mu.Lock()
    defer p.mu.Unlock()
    if p.closing {
        return
    }
    old := atomic.LoadInt32(&p.liveSize)
    atomic.StoreInt32(&p.targetSize, int32(target))
    if int32(target) > old {
        for i := old; i < int32(target); i++ {
            atomic.AddInt32(&p.liveSize, 1)
            p.wg.Add(1)
            go p.worker()
        }
    }
}

func (p *Pool) Size() int { return int(atomic.LoadInt32(&p.liveSize)) }

func (p *Pool) worker() {
    defer p.wg.Done()
    for {
        if atomic.LoadInt32(&p.liveSize) > atomic.LoadInt32(&p.targetSize) {
            atomic.AddInt32(&p.liveSize, -1)
            return
        }
        select {
        case <-p.quit:
            atomic.AddInt32(&p.liveSize, -1)
            return
        case task, ok := <-p.jobs:
            if !ok {
                atomic.AddInt32(&p.liveSize, -1)
                return
            }
            p.safeRun(task)
        }
    }
}

func (p *Pool) safeRun(task func()) {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("pool: recovered: %v", r)
        }
    }()
    task()
}

func (p *Pool) Close() {
    p.mu.Lock()
    p.closing = true
    close(p.quit)
    p.mu.Unlock()
    p.wg.Wait()
    close(p.jobs)
}

func (p *Pool) Autoscale(ctx context.Context) {
    ticker := time.NewTicker(500 * time.Millisecond)
    defer ticker.Stop()
    var lastUp, lastDown time.Time
    for {
        select {
        case <-ctx.Done():
            return
        case now := <-ticker.C:
            depth := len(p.jobs)
            cap := cap(p.jobs)
            usage := float64(depth) / float64(cap)
            cur := p.Size()
            switch {
            case usage > 0.75 && now.Sub(lastUp) > 2*time.Second:
                p.Resize(cur + 2)
                lastUp = now
            case usage < 0.10 && now.Sub(lastDown) > 30*time.Second:
                p.Resize(cur - 1)
                lastDown = now
            }
        }
    }
}

func main() {
    p := NewPool(2, 2, 64)
    ctx, cancel := context.WithCancel(context.Background())
    go p.Autoscale(ctx)

    var done int64
    for i := 0; i < 1000; i++ {
        for !p.Submit(func() {
            time.Sleep(15 * time.Millisecond)
            atomic.AddInt64(&done, 1)
        }) {
            time.Sleep(time.Millisecond)
        }
    }
    for atomic.LoadInt64(&done) < 1000 {
        time.Sleep(50 * time.Millisecond)
        fmt.Println("pool size:", p.Size(), "done:", atomic.LoadInt64(&done))
    }
    cancel()
    p.Close()
}

Run it. You will see the pool start at 2, grow up to perhaps 20 as the queue fills, and process all 1000 tasks. The output gives you the size trajectory.

This is 130 lines of real code that you can deploy. Production versions add metrics, configurable signals, retries, panic-rollback, dead-letter queues, but the core is here.

Reading Your Pool's Behaviour from Metrics¶

Once you deploy a dynamic pool, you must observe it. The five charts that matter:

1. Pool size over time. Should be reactive to load but not flapping. If it changes every tick, your cooldown is too short.
2. Queue depth over time. Should stay in the deadband most of the time. Spikes should resolve within seconds.
3. Task wait time (p50, p99). The customer-facing metric. p99 should match your SLO.
4. Task processing time (p50, p99). Should be roughly independent of pool size. If it grows with size, your downstream is overwhelmed.
5. Resize events per minute. Should be < 5/min in steady state. > 30/min means oscillation.

A Grafana dashboard with these five panels is what you build first when adopting a dynamic pool. Without them, every claim about its behaviour is a guess.

// Metrics with prometheus client_golang:
var (
    poolSize = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "pool_size", Help: "current worker count",
    })
    queueDepth = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "pool_queue_depth", Help: "queued tasks",
    })
    waitDuration = promauto.NewHistogram(prometheus.HistogramOpts{
        Name: "pool_task_wait_seconds",
        Buckets: prometheus.ExponentialBuckets(0.001, 2, 12),
    })
    processDuration = promauto.NewHistogram(prometheus.HistogramOpts{
        Name: "pool_task_process_seconds",
        Buckets: prometheus.ExponentialBuckets(0.001, 2, 12),
    })
    resizes = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "pool_resizes_total",
    }, []string{"direction"})
)

Call poolSize.Set(float64(p.Size())) after every resize and every minute on a heartbeat. queueDepth.Set(float64(len(p.jobs))) on each autoscaler tick. waitDuration.Observe(...) in the worker. resizes.WithLabelValues("up").Inc() after each scale-up.

What Goes Wrong in Production¶

Real-world failures from dynamic pools, sanitised from real incident reports:

Incident A: Cold start at midnight¶

A reporting service had a dynamic pool with floor=0. Every midnight, traffic was zero, the pool shrank to zero. At 00:00:05, the daily report job submitted 2000 tasks. The first task waited 800 ms while the pool re-grew. The 2000th waited 4 seconds. SLO breach. Fix: floor=4.

Incident B: Flapping on a slow downstream¶

A search service used queue-depth scaling. When the search backend got slow (200 ms → 800 ms), queue grew, pool scaled up, more requests hit the backend, backend got slower. Result: positive feedback loop, p99 latency 30s. Fix: scale on wait time, not depth; add circuit breaker to limit downstream concurrency.

Incident C: Resize storm during a deploy¶

After a deploy, the new container had an autoscaler that started with a default of 0 workers and grew based on traffic. During the first 10 seconds, before any request hit it, the autoscaler ticked 20 times and did nothing. Then traffic hit hard, the autoscaler doubled-and-doubled, and the container OOM'd at 8 GB. Fix: floor at NumCPU, ceiling tuned to memory, no doubling above 100.

Incident D: Forgotten recover¶

A library upgrade introduced a nil pointer dereference in 0.1% of tasks. Workers panicked. Live count dropped by 1 each panic. Within an hour, all workers were dead. New submissions blocked forever. Fix: wrap each task in defer recover(), also add an alert on "live size suddenly dropped."

Incident E: Atomic bug¶

A pool had a bug where Resize did liveSize := old + N instead of atomic.AddInt32(&liveSize, N). Under concurrent resizes (from two autoscaler goroutines — a separate bug), the count drifted. Eventually liveSize was negative. Workers behaved wrongly. Fix: use atomic throughout; single autoscaler goroutine.

Each of these was a few-hour outage in a real system. The lessons are now baked into this guide as best practices and pitfalls.

Building Up Intuition with Small Experiments¶

You will not understand dynamic scaling deeply by reading. Build the following five micro-experiments and observe their behaviour.

Experiment 1: No-cooldown oscillator¶

Build the simplest pool with no cooldown. Set thresholds at 50% (up) and 49% (down). Generate steady traffic that produces queue depth at exactly 49.5%. Watch the pool ping-pong every tick. Now widen the deadband to 75/10 and observe stability.

Experiment 2: Cold start¶

Set floor=0. Stop submitting for 5 seconds. Then submit 100 tasks at once. Measure the wait time of the 1st task vs the 50th. The 1st waited for the pool to wake up; the 50th had a full pool by then.

Experiment 3: Slow downstream¶

Make tasks call time.Sleep(d) where d varies. Plot queue depth and wait time as d increases from 10 ms to 1 s. Notice queue depth grows non-linearly with d. Now scale on wait time and observe how scaling behaviour differs.

Experiment 4: Bursty load¶

Generate 500 tasks/s for 60 s, then 0 for 60 s, repeating. Compare three pool configs: static at 50; AIMD-style dynamic; multiplicative dynamic. Plot tail latency. AIMD usually wins.

Experiment 5: Crash test¶

Inject a 5% panic rate in tasks. Without recover in the worker, watch the pool die. Add recover and observe stable behaviour. Add a panic-rate metric and observe.

Each experiment takes 10 minutes to code. The intuition you gain is irreplaceable.

A Walkthrough: The Day You Add Dynamic Scaling to a Real Service¶

Let us simulate the workflow a junior engineer goes through when given the task "add dynamic scaling to the report-generation service."

Day 1: read existing code¶

The service has:

var (
    pool = make(chan struct{}, 16)   // semaphore
    wg   sync.WaitGroup
)

func GenerateReport(req Request) {
    pool <- struct{}{}
    wg.Add(1)
    go func() {
        defer wg.Done()
        defer func() { <-pool }()
        doReport(req)
    }()
}