Dynamic Worker Scaling — Senior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Architecture: Control Loops in Production
4. AIMD: Additive Increase, Multiplicative Decrease
5. PID Controllers for Worker Pools
6. Hysteresis as a Stability Mechanism
7. Little's Law Driven Sizing
8. Integration with Backpressure
9. Integration with Circuit Breakers
10. Multi-Pool Coordination
11. Predictive Scaling
12. Workload Modeling
13. Cost-Aware Scaling
14. Pluggable Autoscaler Architectures
15. Coding Patterns
16. Clean Code
17. Error Handling and Recovery
18. Performance Tips
19. Best Practices
20. Edge Cases and Pitfalls
21. Common Mistakes
22. Common Misconceptions
23. Tricky Points
24. Test
25. Tricky Questions
26. Cheat Sheet
27. Self-Assessment Checklist
28. Summary
29. What You Can Build
30. Further Reading
31. Related Topics
32. Diagrams

Introduction¶

Focus: "How do I design an autoscaler that is stable, predictable, and integrates with backpressure, circuit breakers, and cost controls?"

At middle level you wrote a multi-signal autoscaler with hysteresis. Senior level brings the math and the systems integration. The big themes:

• AIMD — the algorithm behind TCP congestion control, applicable to worker pools
• PID — the control-theory technique for keeping a signal at a set-point
• Little's Law — the analytical baseline for sizing
• Integration — autoscaling does not live alone; it works with backpressure, breakers, rate limits, cost controls
• Stability — when autoscaling causes instability (oscillation, feedback loops, race conditions), how to diagnose and prevent
• Architecture — building autoscalers as composable, testable, swappable components

After this chapter you should be able to:

• Implement AIMD and reason about its convergence properties
• Implement a PID controller for utilization targeting
• Apply Little's Law to estimate baseline pool size and detect saturation
• Wire autoscaling with backpressure, circuit breakers, and rate limiters into a coherent system
• Architect autoscaler components for testability and composability
• Diagnose autoscaler-induced instabilities (oscillation, cascading failures, retry storms)
• Design cost-aware autoscalers that respect budgets

This is the level where you design the autoscaling for a large service, not just write the loop. You make decisions that the team will live with for years.

Prerequisites¶

• You have shipped at least one dynamic pool in production
• You have read and internalized junior and middle chapters
• You are comfortable with basic control theory concepts (feedback, set-point, stability)
• You have used Prometheus, Grafana, distributed tracing
• You have built or maintained systems with circuit breakers and rate limiters
• You can read complex Go code (sync, context, interfaces, generics)

Architecture: Control Loops in Production¶

A worker pool autoscaler is a feedback control loop. Control theory has well-developed vocabulary for this:

• Plant: the system being controlled. Here: the worker pool + downstream system.
• Set-point: the desired value of the controlled variable.
• Process variable: the measured value of the controlled variable.
• Error: difference between set-point and process variable.
• Controller: the function that maps error to control action (resize amount).
• Actuator: the mechanism that applies the control action. Here: Resize / Tune.

The "thermostat" analogy is direct:

Good control loops:

• Are stable — small disturbances do not cause big oscillations
• Are responsive — react quickly to changes
• Have low overshoot — do not exceed the set-point by much
• Settle — the system reaches steady state

Bad control loops oscillate, overshoot, fail to settle, or chase noise.

Most worker-pool autoscalers are discrete-time bang-bang controllers: at each tick, they take a binary action (grow / shrink / hold). Bang-bang controllers work but can oscillate around the set-point. PID controllers are smoother — we cover them below.

Stability fundamentals¶

A system is stable when small disturbances decay over time. Three properties contribute:

1. Negative feedback dominates positive feedback. Growing the pool reduces queue depth (negative feedback). If growing also somehow increases queue depth (e.g., by overloading downstream — positive feedback), the system is unstable.

2. Loop gain < 1. If a single tick's action overshoots the necessary correction, you oscillate. Loop gain depends on step size, cooldown, and how strongly signals correlate with size.

3. Delay matters. Long feedback delays (signal collection lag + autoscaler tick + resize-takes-effect lag) reduce stability margin. Slower loops have more headroom.

These principles guide all autoscaler design.

AIMD: Additive Increase, Multiplicative Decrease¶

AIMD is the algorithm behind TCP congestion control. It is provably stable under network-like conditions and has nice properties for worker pools too.

Definition¶

• On positive evidence (room to grow): add a small constant N to current size.
• On negative evidence (need to shrink): multiply current size by a factor F < 1.

func AIMD(cur int, sig Signals) int {
    switch {
    case shouldGrow(sig):
        return cur + 1     // additive
    case shouldShrink(sig):
        return cur - cur/4  // multiplicative: -25%
    default:
        return cur
    }
}

Why does it work?¶

AIMD has two convergence properties:

1. Efficiency: the system tends toward maximum utilization (does not under-provision indefinitely).
2. Fairness (across multiple AIMD pools sharing a resource): they converge to equal shares.

In a worker pool context, AIMD means:

• Growth is gentle. You add one worker at a time, easing into more load.
• Shrink is aggressive. When you decide capacity is excess, you shrink by 25% (or 50%, or 75%).

The aggressive shrink prevents over-provisioning from persisting. The gentle growth prevents overshoot.

Why not the other way?¶

Multiplicative increase + additive decrease (MIAD): grow fast, shrink slowly. Result: overshooting capacity, slow to react when load drops.

Additive + additive (AIAD): symmetric. Stable but slow in both directions.

Multiplicative + multiplicative (MIMD): violent in both directions. Unstable.

AIMD is the sweet spot for systems with asymmetric cost: cost of over-provisioning > cost of under-provisioning (because we can ramp up quickly when needed).

Wait — in pools, isn't under-provisioning worse?¶

In worker pools, under-provisioning causes latency spikes — sometimes worse than over-provisioning's cost. So you might wonder: should we use MIAD for pools?

In practice, AIMD still works because:

• The autoscaler runs constantly. Additive grow gets you to the right size in seconds.
• Multiplicative shrink only triggers when sustained low signals are observed. Brief dips don't cause shrink.
• The pool re-grows fast on next burst (additive but eager).

If your workload has very sudden, severe bursts (load spikes 10x in milliseconds), then yes — consider MIAD or even MIMD for the rare emergency-grow case. But these are exceptions.

Implementation¶

type AIMDController struct {
    GrowStep     int     // typically 1 or 2
    ShrinkFactor float64 // 0.25 = -25%, 0.5 = -50%
    GrowAfter    Predicate
    ShrinkAfter  Predicate
}

func (a *AIMDController) Decide(cur int, signals Signals) int {
    switch {
    case a.GrowAfter(signals):
        return cur + a.GrowStep
    case a.ShrinkAfter(signals):
        shrink := int(float64(cur) * a.ShrinkFactor)
        if shrink < 1 { shrink = 1 }
        return cur - shrink
    default:
        return cur
    }
}

Predicate is func(Signals) bool. Inject your conditions. Test in isolation.

Variants¶

• AIMD with floor/ceiling. Clamp results.
• AIMD with cooldown. Same as before.
• AIMD with dampened multiplicative. Shrink by 25% if util very low, by 10% if moderately low.
• MIMD-fallback under saturation. Normal AIMD, except when signal indicates emergency (e.g., wait > 10x SLO), multiplicative grow once.

These variations are common in production. The pure AIMD is a useful baseline.

Convergence example¶

Imagine a workload where the "right" pool size is 30. AIMD starts at 8.

• Tick 1: signals say grow. 8 → 9.
• Tick 2: grow. 9 → 10.
• ...
• Tick 23: grow. 30 → 31.
• Tick 24: util drops slightly (we are at right size). Hold at 31.
• Tick 25: low util signal. Shrink: 31 → 31 - 7 = 24.
• Tick 26: util goes up again (we shrank too much). Grow: 24 → 25.
• ...
• Tick 32: grow. 30 → 31.
• ...

Result: pool oscillates around 30, gradually approaching. Stable, slow, predictable.

Compare to pure additive (no multiplicative shrink): each oscillation is +/-1. Tighter oscillation but less responsive to permanent load drops.

Compare to multiplicative both ways: huge swings, hard to predict.

PID Controllers for Worker Pools¶

PID = Proportional + Integral + Derivative. The classic control technique for keeping a process variable at a set-point.

Definition¶

u(t) = Kp · e(t) + Ki · ∫e(τ)dτ + Kd · de/dt

Where: - e(t) is the error (set-point - process variable) - Kp, Ki, Kd are tuning constants - u(t) is the control output (resize amount)

In Go, discrete form:

type PID struct {
    Kp, Ki, Kd float64
    setpoint   float64
    integral   float64
    lastError  float64
    lastTime   time.Time
    primed     bool
}

func (p *PID) Update(measured float64, now time.Time) float64 {
    err := p.setpoint - measured
    if !p.primed {
        p.primed = true
        p.lastError = err
        p.lastTime = now
        return p.Kp * err
    }
    dt := now.Sub(p.lastTime).Seconds()
    if dt <= 0 {
        return p.Kp * err
    }
    p.integral += err * dt
    derivative := (err - p.lastError) / dt
    p.lastError = err
    p.lastTime = now
    return p.Kp*err + p.Ki*p.integral + p.Kd*derivative
}

Application¶

Set-point: target utilization, e.g., 0.70.

Each tick:

util := computeUtilization()
controlSignal := pid.Update(util, time.Now())
// controlSignal is roughly: how many workers to add/remove
target := cur + int(math.Round(controlSignal))
target = clamp(target, floor, ceiling)
pool.Resize(target)

Tuning¶

The hard part. There is no universal Kp/Ki/Kd. Approaches:

1. Ziegler-Nichols. Tune Kp until oscillation, then back off. Set Ki and Kd by formula.
2. Manual. Start with small Kp (e.g., 5), Ki=0, Kd=0. Increase until responsive. Add small Ki for steady-state accuracy. Add small Kd if oscillation persists.
3. Auto-tuning. Various libraries do this online. Rarely used in worker pools.

Typical values for a worker pool:

Kp = 10
Ki = 0.5
Kd = 1.0

The exact values depend on your system. Tune empirically.

Why PID is overkill (usually)¶

PID is for systems where the process variable must be at a precise set-point. Worker pools don't need that — being at 65% utilization vs 75% is fine. A bang-bang controller with hysteresis serves most cases.

PID becomes useful when:

• You need very tight latency control
• You have a clear set-point (e.g., maintain p99 < 100 ms)
• The cost of overshoot is high
• You have time to tune carefully

For most pools, AIMD or simple threshold-based is enough. PID is a tool in the toolbox.

Anti-windup¶

A subtle PID issue: if the integral keeps accumulating during prolonged saturation (we're at ceiling, can't grow more, but error is still positive), the integral term grows without bound. When the situation resolves, the controller massively overshoots in the other direction.

Anti-windup: clamp the integral.

if p.integral > 1000 { p.integral = 1000 }
if p.integral < -1000 { p.integral = -1000 }

Or freeze integration when saturated:

if isSaturated() {
    // don't update integral this tick
} else {
    p.integral += err * dt
}

This is the kind of detail that turns a paper-correct PID into a production-correct PID.

Hysteresis as a Stability Mechanism¶

We have used hysteresis (different thresholds for up and down) throughout. Let us look at it more carefully.

The math¶

A system with a single threshold T exhibits chattering when the signal is near T. Crossing T triggers an action; the action reduces the signal back below T; the signal grows again; rinse, repeat.

Hysteresis introduces two thresholds, T_high > T_low. Crossing T_high triggers grow; crossing T_low triggers shrink. The deadband between them is where no action is taken.

The width of the deadband determines how much noise is filtered:

• Narrow (T_high - T_low = 0.1): filters noise within 10% range
• Wide (T_high - T_low = 0.5): filters noise within 50%

The wider the deadband, the more the system can drift before reacting. Trade-off: stability vs responsiveness.

Choosing widths¶

A good rule: deadband width = max(noise amplitude × 3, set-point × 0.3).

If your measured signal has a standard deviation of 0.05 around steady state, deadband of 0.15 (3 sigma) filters 99.7% of noise. If your set-point is 0.7 utilization, deadband of 0.2 (centered on set-point: 0.6-0.8) is reasonable.

Hysteresis vs cooldown¶

Hysteresis filters by value. Cooldown filters by time. They are complementary:

• Hysteresis without cooldown: no flap on the value axis, but you can resize every tick when signal slowly drifts across the threshold.
• Cooldown without hysteresis: no rapid-fire resize, but you flap as soon as cooldown elapses if signal is right at threshold.
• Both: solid stability.

Production systems always use both.

Adaptive hysteresis¶

If your workload pattern changes (e.g., daytime vs nighttime), fixed hysteresis may be too narrow at one time and too wide at another. Adaptive: track noise level, widen deadband when noise is high.

type AdaptiveBand struct {
    base        float64
    noise       *EWMA  // smoothed noise level
    current     float64
}

func (a *AdaptiveBand) Update(signal float64) {
    a.noise.Add(math.Abs(signal - a.lastSignal))
    a.current = a.base + a.noise.Value() * 3
    a.lastSignal = signal
}

The deadband widens when noise is high. Useful in production where workload character changes over the day.

Little's Law Driven Sizing¶

We have mentioned Little's Law. Let us use it.

Statement¶

For a stable queueing system:

L = λ × W

• L: average number of items in the system
• λ: throughput (items per second arriving and departing — they are equal in steady state)
• W: average time in system

Worker pool application¶

In a worker pool:

• L_workers: average number of busy workers
• λ: tasks per second
• W_process: average processing time

So:

busy_workers = throughput × processing_time

Example: 200 tasks/s, each taking 150 ms:

busy_workers = 200 × 0.15 = 30

You need at least 30 workers busy on average. With some idle capacity for variance and bursts, you provision 30 to 50 — say, 40.

Floor and ceiling from Little's Law¶

• Steady state: workers = λ × W_process
• Floor: enough workers for minimum throughput. If minimum throughput is 50 tasks/s, floor = 50 × 0.15 = 8.
• Ceiling: enough workers for maximum acceptable throughput. If max is 2000 tasks/s, ceiling = 2000 × 0.15 = 300.

Detecting saturation¶

If you observe:

λ × W_process > live_workers

You are saturated. Tasks are queueing. Grow.

If:

λ × W_process < live_workers × 0.5

You are over-provisioned. Shrink.

Worked example¶

A service processes 200 tasks/s. Each task takes 100 ms. Pool is at 40 workers.

By Little's Law: 200 × 0.1 = 20 workers busy on average. Pool of 40 means 50% utilization. Good headroom for bursts.

A burst arrives: throughput goes to 400 tasks/s briefly. Little's Law: needs 40 workers. Pool is exactly at limit. If burst is sustained: queue grows.

The autoscaler should grow the pool. Say to 60. New headroom: 33%.

If you predicted Little's Law and set up the autoscaler with foreknowledge of W_process, you can choose appropriate thresholds.

When Little's Law breaks down¶

• Non-stationary processes (workload character changes mid-test)
• Highly bursty workloads (steady-state assumptions fail)
• Workloads with feedback (slow downstream causes processing time to grow)

In those cases, Little's Law is a starting point, not a target.

Integration with Backpressure¶

Backpressure and autoscaling are partners. Let us see the full integration.

Layers of pressure¶

In a healthy production system:

1. Caller-side: clients implement retries with backoff. If we 503 them, they wait, retry.
2. Edge layer (HTTP): rate limiting at the front door. Reject excessive load.
3. Pool input: Submit returns error if pool is full.
4. Pool itself: autoscaler grows up to ceiling.
5. Downstream: circuit breaker if downstream is unhealthy.

Each layer protects the next. The autoscaler is layer 4; backpressure is layers 1-3 and 5.

Submit with backpressure¶

func (s *Service) Handle(req Request) Response {
    ctx, cancel := context.WithTimeout(req.Context(), 5 * time.Second)
    defer cancel()

    err := s.pool.Submit(ctx, func(ctx context.Context) {
        s.process(ctx, req)
    })
    switch err {
    case nil:
        return Response{Status: 200}
    case ErrPoolFull:
        return Response{Status: 503, Retry: true}
    case context.DeadlineExceeded:
        return Response{Status: 504}
    default:
        return Response{Status: 500}
    }
}

The caller sees 503 immediately when the pool is at capacity. The caller decides retry.

Token bucket fronting the pool¶

Sometimes you don't want to expose backpressure at the pool. Instead, fail fast at the edge:

type ServiceTokens struct {
    pool     *Pool
    limiter  *rate.Limiter
}

func (s *ServiceTokens) Handle(req Request) Response {
    if !s.limiter.Allow() {
        return Response{Status: 429}
    }
    err := s.pool.Submit(...)
    // ...
}

The limiter caps tokens/sec. Excess fails immediately with 429. The pool is protected from sudden bursts; the autoscaler adjusts the limiter rate based on pool capacity.

Autoscaler-aware limiter¶

Couple the limiter rate to pool capacity:

func (s *Service) updateLimiter() {
    cap := s.pool.Size()
    // assume ~100ms processing
    rate := float64(cap) / 0.1  // tokens per second
    s.limiter.SetLimit(rate.Limit(rate))
}

Called every tick. The limiter rate tracks the pool's actual capacity. As pool grows, limiter relaxes. As pool shrinks, limiter tightens.

This is integrated control: pool size and limiter rate move together.

Integration with Circuit Breakers¶

A circuit breaker monitors a downstream's health. When unhealthy, it short-circuits — calls fail immediately instead of waiting for timeout.

Why combine with autoscaling¶

We saw the failure mode at middle level: slow downstream → workers tied up → queue grows → autoscaler grows pool → more workers slamming slow downstream → downstream collapses.

The circuit breaker stops this:

func (s *Service) work(ctx context.Context, req Request) {
    err := s.breaker.Call(func() error {
        return s.callDownstream(ctx, req)
    })
    if err == ErrCircuitOpen {
        // fail fast; don't tie up the worker
        return
    }
    // ...
}

When breaker is open, the worker returns immediately. The pool's load is what matters, not downstream's. The autoscaler's signals (wait time, queue depth) reflect actual pool work, not downstream slowness.

Breaker state as a signal¶

You can also use breaker state directly in the autoscaler:

func (a *Autoscaler) shouldGrow() bool {
    if a.breaker.IsOpen() {
        return false  // downstream is sick; don't grow
    }
    return a.signal.WaitP99() > a.threshold
}

This is a veto: while downstream is sick, do not grow the pool. Doing so would worsen the situation.

Half-open state¶

A breaker in half-open lets a few requests through to test downstream. The autoscaler should also be conservative — grow slowly, if at all, during half-open.

func (a *Autoscaler) growStep() int {
    switch a.breaker.State() {
    case Open:
        return 0
    case HalfOpen:
        return 1  // tiny test step
    case Closed:
        return 4  // normal
    }
    return 0
}

Multi-Pool Coordination¶

Many services have multiple pools (one per downstream, per priority class, per tenant). Coordinating their autoscaling is its own challenge.

Independent autoscalers¶

Simplest: each pool has its own autoscaler. They don't talk to each other.

Pros: simple, isolated. Cons: they may collectively over-grow (each thinks it should grow; total exceeds host capacity).

Shared resource budget¶

Define a global budget: total workers across all pools < N.

type GlobalBudget struct {
    mu       sync.Mutex
    capacity int
    used     int
}

func (g *GlobalBudget) RequestGrowth(n int) (granted int) {
    g.mu.Lock()
    defer g.mu.Unlock()
    if g.used + n > g.capacity {
        granted = g.capacity - g.used
    } else {
        granted = n
    }
    g.used += granted
    return granted
}

func (g *GlobalBudget) Release(n int) {
    g.mu.Lock()
    defer g.mu.Unlock()
    g.used -= n
    if g.used < 0 { g.used = 0 }
}

Each pool's autoscaler asks the budget for growth permission. Across pools, total never exceeds capacity.

Priority-based allocation¶

Higher-priority pools get growth requests granted first. Lower-priority pools wait.

type PrioritizedBudget struct {
    capacity int
    used     int
    // queue of pending requests, sorted by priority
}

Production frameworks like Kubernetes Horizontal Pod Autoscaler use similar logic at cluster scale.

Per-tenant fairness¶

If pools serve different tenants, fairness matters. Several models:

• Equal shares. Each tenant guaranteed N/T workers, can use more if others don't.
• Weighted shares. High-paying tenants get more.
• Per-tenant quotas. Each has a max; no inter-tenant priority.

Implementation depends on policy. Track per-tenant utilization; allocate growth to whoever needs it most.

Predictive Scaling¶

Reactive autoscaling grows after the load arrives. Predictive grows before.

Time-of-day patterns¶

If your traffic is consistently high at 09:00, pre-warm at 08:55.

type SchedulePoint struct {
    Hour int
    Min  int
    Target int
}

func (p *PatternedScaler) tick(now time.Time) {
    for _, sp := range p.schedule {
        if now.Hour() == sp.Hour && now.Minute() == sp.Min {
            p.pool.Resize(sp.Target)
        }
    }
}

Simple but effective. Many services have stable daily patterns.

Predictive models¶

More sophisticated: use a forecast. Linear regression, ARIMA, neural network. The autoscaler computes the predicted load 5-10 minutes ahead and resizes preemptively.

For most services, a simple weighted-average prediction works:

func (p *Forecast) PredictNext(window time.Duration) float64 {
    // average over last N samples weighted by recency
    return weightedAverage(p.samples)
}

Hybrid: reactive + predictive¶

Run both. Predictive sets a baseline target; reactive corrects for unexpected variation.

func (a *Autoscaler) tick() {
    predicted := a.forecast.Predict()
    reactive := a.measure()
    target := max(predicted, reactive)
    a.pool.Resize(clamp(target, a.floor, a.ceiling))
}

When prediction is accurate, the pool is right-sized before load arrives. When prediction is wrong, reactive catches up.

Limits of prediction¶

• Sudden events (a marketing email blast, an external incident) cannot be predicted.
• Predictions are sometimes wrong; over-trusting them wastes capacity.
• Complex models cost CPU and engineering time.

For most services, time-of-day schedule + reactive autoscaler suffices. Heavy prediction is worth it only at scale.

Workload Modeling¶

To tune autoscaler, model the workload. Three dimensions matter.

Arrival rate¶

• Mean (λ̄): tasks per second over long window
• Variance: how bursty?
• Pattern: time-of-day, day-of-week
• Spikes: rare, large bursts

Service time¶

• Mean (W̄): average processing time
• Variance: are all tasks similar, or bimodal?
• Tail: how heavy is p99 / p999?

Coupling¶

• Are tasks independent, or do they share resources?
• Downstream effects: does one slow task slow others?
• Burst correlation: do bursts correlate with slow downstream?

A workload model can be as simple as:

type WorkloadModel struct {
    MeanArrivalRate float64
    PeakArrivalRate float64
    MeanServiceMs   float64
    P99ServiceMs    float64
}

From this, derive:

• Steady state pool size: MeanArrivalRate × MeanServiceMs / 1000 × 1.3 (30% headroom)
• Burst pool size: PeakArrivalRate × P99ServiceMs / 1000 (for tail-driven)
• Floor: MeanArrivalRate × 0.3 × MeanServiceMs / 1000
• Ceiling: 2 × PeakArrivalRate × P99ServiceMs / 1000

These are baselines for the autoscaler's floor, ceiling, and thresholds.

Continuous modeling¶

A production-grade autoscaler can update the model continuously:

type LiveModel struct {
    arrivalEWMA   *EWMA
    serviceEWMA   *EWMA
    p99Service    *HDR
}

Each task's start time updates arrival rate (EWMA). Each task's duration updates service time (EWMA, HDR for percentile). The autoscaler decisions use live model values.

This is a form of self-tuning: as workload character changes, the autoscaler adapts thresholds.

Cost-Aware Scaling¶

In cloud environments, every worker has a cost. Cost-aware autoscaling factors this in.

Cost dimensions¶

• Per-worker memory. Tied to stack size (small) + heap usage per task.
• Per-task downstream cost. Each call to a paid downstream API.
• Compute cost. Cloud bill per CPU-second.
• SLO violation cost. Customer impact from missed latency.

Cost-driven decision¶

func (a *Autoscaler) costAwareDecide(signals Signals, cur int) int {
    // Estimate cost of growing by 1 worker
    workerCost := a.workerHourCost
    // Estimate benefit: latency improvement * value-of-improvement
    latencyImprovement := a.estimateLatencyImprovement(cur)
    benefit := latencyImprovement * a.valuePerLatencyMs
    if benefit > workerCost {
        return cur + 1
    }
    return cur
}

The autoscaler grows only when expected benefit exceeds cost. Hard to estimate accurately; in practice, codify as thresholds.

Budget caps¶

Simple: don't grow beyond a budget.

func (a *Autoscaler) checkBudget(targetSize int) int {
    monthlyBudget := a.budget.Remaining()
    workerMonths := workerCostPerMonth * float64(targetSize)
    if workerMonths > monthlyBudget {
        // log alert, cap to budget
        return int(monthlyBudget / workerCostPerMonth)
    }
    return targetSize
}

Production systems often pair autoscaling with explicit budget caps. The autoscaler is allowed to grow as long as budget permits; beyond that, the system degrades gracefully (longer queues, backpressure).

Spot/preemptible workers¶

In some environments, you can mix on-demand and spot workers. Spot is cheaper but can be revoked.

Autoscaler runs two pools:

type DualPool struct {
    stable     *Pool  // on-demand
    spot       *Pool  // spot
}

func (d *DualPool) Submit(task func()) bool {
    // try spot first; fall back to stable
    if d.spot.Submit(task) {
        return true
    }
    return d.stable.Submit(task)
}

The autoscaler grows the spot pool aggressively when allowed; falls back to stable when spot capacity is constrained.

Pluggable Autoscaler Architectures¶

To build a production autoscaler that you can evolve, design it as composable parts.

Interfaces¶

type Pool interface {
    Size() int
    Resize(int) error
}

type Signal interface {
    Value() float64
    Name() string
}

type Decider interface {
    Decide(cur int, signals []Signal) (target int, reason string)
}

type Cooldown interface {
    AllowUp(now time.Time) bool
    AllowDown(now time.Time) bool
    RecordResize(now time.Time, dir Direction)
}

The autoscaler is a runner¶

type Autoscaler struct {
    Pool     Pool
    Signals  []Signal
    Decider  Decider
    Cooldown Cooldown
    Bounds   Bounds
    Interval time.Duration
    Logger   *slog.Logger
}

func (a *Autoscaler) Run(ctx context.Context) {
    ticker := time.NewTicker(a.Interval)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case now := <-ticker.C:
            cur := a.Pool.Size()
            target, reason := a.Decider.Decide(cur, a.Signals)
            target = a.Bounds.Clamp(target)
            if target == cur { continue }
            dir := DirectionUp
            if target < cur { dir = DirectionDown }
            switch dir {
            case DirectionUp:
                if !a.Cooldown.AllowUp(now) { continue }
            case DirectionDown:
                if !a.Cooldown.AllowDown(now) { continue }
            }
            if err := a.Pool.Resize(target); err != nil {
                a.Logger.Warn("resize failed", "err", err)
                continue
            }
            a.Cooldown.RecordResize(now, dir)
            a.Logger.Info("resized", "from", cur, "to", target, "reason", reason)
        }
    }
}

Each piece is swappable. Want AIMD instead of threshold? Swap Decider. Want time-of-day cooldown? Swap Cooldown. Want a different signal? Add a new Signal implementation.

Decider implementations¶

type ThresholdDecider struct { /* thresholds */ }
type AIMDDecider struct { /* additive/multiplicative params */ }
type PIDDecider struct { /* Kp, Ki, Kd, setpoint */ }
type ScheduleDecider struct { /* time-of-day schedule */ }
type CompositeDecider struct {
    Deciders []Decider
    Strategy MergeStrategy  // max, min, priority
}

A CompositeDecider combines several. The merge strategy decides how. "Max" grows aggressively (take the largest target); "min" grows conservatively; "priority" uses the first non-zero decision.

Testing each piece in isolation¶

Pure functions/structs. No real pool needed for tests:

func TestAIMDDeciderGrowsAdditive(t *testing.T) {
    d := &AIMDDecider{GrowStep: 1, ShrinkFactor: 0.25}
    target, _ := d.Decide(20, []Signal{
        &FakeSignal{Value_: 0.9, Name_: "util"},
    })
    if target != 21 { t.Errorf("expected 21, got %d", target) }
}

Mock everything. The runner's logic is fully covered by tests on its parts.

Coding Patterns¶

Pattern: Builder for autoscaler¶

type Builder struct {
    pool     Pool
    signals  []Signal
    decider  Decider
    cooldown Cooldown
    bounds   Bounds
    interval time.Duration
}

func NewBuilder(pool Pool) *Builder {
    return &Builder{
        pool:     pool,
        decider:  &ThresholdDecider{},
        cooldown: &SimpleCooldown{Up: 3 * time.Second, Down: 60 * time.Second},
        bounds:   Bounds{Min: 1, Max: 100},
        interval: 500 * time.Millisecond,
    }
}

func (b *Builder) WithDecider(d Decider) *Builder { b.decider = d; return b }
func (b *Builder) WithSignal(s Signal) *Builder    { b.signals = append(b.signals, s); return b }
// ... etc

func (b *Builder) Build() *Autoscaler {
    return &Autoscaler{
        Pool: b.pool, Signals: b.signals, Decider: b.decider,
        Cooldown: b.cooldown, Bounds: b.bounds, Interval: b.interval,
    }
}

Builder makes configuration readable and validated. Common pattern in Go for any multi-option struct.

Pattern: pluggable metrics¶

type MetricsSink interface {
    GaugeSet(name string, v float64)
    CounterInc(name string)
    HistogramObserve(name string, v float64)
}

type Autoscaler struct {
    Metrics MetricsSink
}

Implementations: Prometheus, OpenTelemetry, stdout, no-op (for tests).

Pattern: middleware around Submit¶

type SubmitMiddleware func(next SubmitFunc) SubmitFunc

func WithMetrics(m MetricsSink) SubmitMiddleware {
    return func(next SubmitFunc) SubmitFunc {
        return func(task func()) bool {
            m.CounterInc("submits")
            return next(task)
        }
    }
}

func WithLogging(l *slog.Logger) SubmitMiddleware { /* ... */ }
func WithTimeout(d time.Duration) SubmitMiddleware { /* ... */ }

Chain middlewares for cross-cutting concerns.

Pattern: observable resize events¶

type ResizeEvent struct {
    Old, New  int
    Reason    string
    Signals   map[string]float64
    Timestamp time.Time
}

type Observer interface {
    OnResize(e ResizeEvent)
}

Pluggable observers. One logs to stdout; one writes to a database for forensic analysis; one emits a metric.

Pattern: declarative policy¶

autoscaler:
  signals:
    - type: wait_p99
      window: 30s
    - type: utilization
      window: 10s
  decider:
    type: aimd
    grow_step: 2
    shrink_factor: 0.20
  cooldown:
    up: 3s
    down: 60s
  bounds:
    min: 4
    max: 128

Load policy from config; build autoscaler. Lets ops tune without redeploying.

Clean Code¶

• Names match domain. Signal, Decider, Cooldown, Bounds. Not Helper, Manager, Util.
• Each function does one thing. Decide doesn't enforce cooldown; Cooldown does.
• Constants for thresholds at the top. Easy to scan, easy to override.
• Document policy. Why this Kp? Why this cooldown? Future-you will thank you.
• Test interfaces, not implementations. Mock the pool; assert on Resize calls.
• Configuration as data. YAML/JSON > buried magic numbers.
• Composition over inheritance. CompositeDecider wraps Deciders; doesn't subclass them.

Error Handling and Recovery¶

Pool resize failure¶

func (a *Autoscaler) safeResize(target int) {
    if err := a.Pool.Resize(target); err != nil {
        a.Metrics.CounterInc("resize_errors")
        a.Logger.Warn("resize failed", "target", target, "err", err)
        // Don't update cooldown; retry next tick
        return
    }
}

Autoscaler panic¶

func (a *Autoscaler) Run(ctx context.Context) {
    defer func() {
        if r := recover(); r != nil {
            a.Logger.Error("autoscaler panicked", "panic", r, "stack", string(debug.Stack()))
            // Restart? Alert? Depend on policy.
        }
    }()
    a.runInner(ctx)
}

Signal collection failure¶

func (a *Autoscaler) safeSignal(s Signal) float64 {
    defer func() {
        if r := recover(); r != nil {
            a.Logger.Warn("signal panic", "signal", s.Name(), "panic", r)
        }
    }()
    return s.Value()
}

Closed pool¶

type Pool interface {
    Resize(int) error
}

var ErrPoolClosed = errors.New("pool closed")

// Autoscaler treats this as terminal:
case errors.Is(err, ErrPoolClosed):
    a.Logger.Info("pool closed; stopping autoscaler")
    return

Performance Tips¶

• Decision functions are pure; cache nothing.
• Avoid logging at info on every tick if tick rate is high; sample.
• Histograms over sort for p99 (O(buckets) vs O(n log n)).
• Use atomic operations; avoid mutex on hot paths.
• Sample signals; don't measure every task.
• Use time.Ticker, not time.After-in-loop (allocates each iteration).
• Avoid map iteration in decision functions; use struct fields.
• Profile under load. Look for unexpected GC pressure.

Best Practices¶

1. Design as composable interfaces (Pool, Signal, Decider, Cooldown).
2. Tune empirically. No "right" Kp for a PID. Measure.
3. Always have bounds. Floor, ceiling, max-step-per-tick.
4. Integrate with backpressure and circuit breakers from day 1.
5. Cost-aware: every growth has a cost; weigh against benefit.
6. Predictive when patterns are clear; reactive always.
7. Multi-pool coordination via global budget.
8. Workload model drives initial sizing; live model updates over time.
9. Test deciders as pure functions.
10. Document policy.

Edge Cases and Pitfalls¶

Pitfall: PID windup¶

Mentioned above. Integral term grows during saturation; overshoots after.

Pitfall: PID without anti-windup is unsafe¶

Combine Kp, Ki, Kd with limits on integral; pause integration during saturation.

Pitfall: AIMD with no minimum step¶

cur - cur*0.25 rounds to 0 for small cur. Always enforce minimum step of 1.

Pitfall: Multiple autoscalers fighting over global budget¶

If two pools' autoscalers both request budget at the same time, one wins, the other waits. If the winner releases later (shrink), the other could over-grab. Use a proper queue or distributed lock for the budget.

Pitfall: Predictive over-trust¶

Predictive autoscaler grows in anticipation; load doesn't arrive; pool is oversized. Always cap predictive growth at some multiple of reactive growth.

Pitfall: Cost-aware that misses SLO¶

Cost-aware decision says "don't grow"; SLO is breached. Add SLO veto: regardless of cost, grow to meet SLO.

Pitfall: Coupled feedback loops¶

Pool A grows → uses downstream X → X gets slow → Pool A's tasks slow → autoscaler grows Pool A more. Positive feedback. Defense: detect downstream sickness; veto growth.

Pitfall: Hysteresis with adaptive band that shrinks too quickly¶

Adaptive band updates each tick. If noise spikes briefly, band widens. If noise drops, band shrinks too fast, flapping resumes. Use smoothing on the band update itself.

Pitfall: Decider that returns target outside bounds¶

If clamp is only in the runner, a misbehaving decider can pass insane values. Validate inside the decider too (defensive).

Pitfall: Cooldown counted from completion, not from start¶

Tricky to get right. Track from when the resize finished (e.g., after Resize returns). Otherwise spawning many workers might make cooldown count from wrong time.

Common Mistakes¶

1. PID without anti-windup.
2. AIMD that rounds to zero step.
3. Multi-pool autoscalers without coordination.
4. Predictive scaler that ignores reactive corrections.
5. Hysteresis without cooldown (or vice versa).
6. Composite decider with no merge strategy specified.
7. Forgetting circuit breaker integration.
8. Cost-aware that lets SLO breach to save money.
9. Pluggable architecture so abstract that it is unreadable.
10. Tests that mock so much they don't test anything real.

Common Misconceptions¶

• "PID is the best controller." Often overkill. Bang-bang + hysteresis is usually enough.
• "Predictive is better than reactive." Only if your model is good. Mostly use both.
• "Cost-aware means cheap." No — it means informed. Sometimes the answer is "spend more."
• "AIMD converges to optimum." Converges to a stable state, not necessarily optimum.
• "More pluggability is better." No — readability matters more than maximum flexibility.

Tricky Points¶

• AIMD's stability depends on the measurement function, not just the algorithm. Garbage in, garbage out.
• PID Kd is sensitive to noise; high Kd amplifies measurement noise into oscillation.
• Multi-pool budget must be released on shrink; missing this leaks budget.
• Predictive models drift; retrain periodically.
• Backpressure-pool coupling: changing pool size mid-burst can affect backpressure thresholds; coordinate them.
• The autoscaler tick is not the only signal; events (deploy, manual override) also trigger resize.

Test¶

Test the decider as a pure function¶

func TestAIMDDecider(t *testing.T) {
    d := &AIMDDecider{GrowStep: 1, ShrinkFactor: 0.25, GrowPredicate: func(s Signals) bool {
        return s.Util > 0.85
    }, ShrinkPredicate: func(s Signals) bool {
        return s.Util < 0.30
    }}
    cases := []struct {
        cur  int
        sig  Signals
        want int
    }{
        {cur: 10, sig: Signals{Util: 0.9}, want: 11},
        {cur: 100, sig: Signals{Util: 0.2}, want: 75},
        {cur: 10, sig: Signals{Util: 0.5}, want: 10},
    }
    for _, c := range cases {
        got, _ := d.Decide(c.cur, c.sig)
        if got != c.want { t.Errorf("cur=%d sig=%v want=%d got=%d", c.cur, c.sig, c.want, got) }
    }
}

Test integration with a fake pool¶

type FakePool struct {
    size int
    resizes []int
}

func (p *FakePool) Size() int       { return p.size }
func (p *FakePool) Resize(n int) error {
    p.resizes = append(p.resizes, n)
    p.size = n
    return nil
}

func TestAutoscalerLoop(t *testing.T) {
    pool := &FakePool{size: 8}
    sig := &FakeSignal{Value_: 0.9}
    d := &AIMDDecider{...}
    a := &Autoscaler{Pool: pool, Signals: []Signal{sig}, Decider: d, Interval: 5 * time.Millisecond, ...}
    ctx, cancel := context.WithCancel(context.Background())
    go a.Run(ctx)
    time.Sleep(50 * time.Millisecond)
    cancel()
    if pool.size <= 8 { t.Errorf("expected grow, got %d", pool.size) }
}

Test stability under noise¶

func TestStabilityUnderNoise(t *testing.T) {
    pool := &FakePool{size: 16}
    sig := &NoisySignal{Mean: 0.5, Variance: 0.1}
    d := &AIMDDecider{...}
    a := &Autoscaler{...}
    ctx, cancel := context.WithCancel(context.Background())
    go a.Run(ctx)
    time.Sleep(2 * time.Second)
    cancel()
    // Pool should not have resized more than a few times
    if len(pool.resizes) > 10 {
        t.Errorf("excessive resize under noise: %d", len(pool.resizes))
    }
}

Tricky Questions¶

1. AIMD vs MIAD: which one for a worker pool? AIMD almost always. MIAD overshoots and stays high.

2. PID Kp too high: what happens? System oscillates around set-point, possibly diverging.

3. PID Ki too high: what happens? Slow oscillation; very slow recovery from disturbances.

4. What is the role of Kd? Damps oscillation by responding to rate of change. Caveat: amplifies noise.

5. How do you detect oscillation? Monitor resize/min counter. > 30/min usually means oscillation.

6. What if Little's Law says you need 50 workers but you have 5 CPU cores? Two cases: CPU-bound (cannot exceed cores) or I/O-bound (50 is fine). Diagnose with profiling.

7. Why integrate breaker with autoscaler? To prevent positive feedback on downstream failure.

8. Why cost-aware? Sometimes saving money matters more than latency. Make the trade explicit.

9. What is anti-windup? Limiting PID integral so it doesn't grow unboundedly during saturation.

10. Predictive scaling: when does it pay off? When patterns are strong (time-of-day) and prediction model is cheap. For random workloads: no.

Cheat Sheet¶

// AIMD
target = cur + 1         (grow)
target = cur - cur/4     (shrink)

// PID
err = setpoint - measured
integral += err * dt
deriv = (err - lastErr) / dt
output = Kp*err + Ki*integral + Kd*deriv
clamp(integral, -limit, +limit)   // anti-windup

// Little's Law
workers = throughput * service_time

// Integration:
//   Pool ↔ Backpressure (Submit returns ErrPoolFull)
//   Pool ↔ Breaker (veto growth when open)
//   Pool ↔ Limiter (limiter rate tracks pool capacity)
//   Multi-pool ↔ Budget (global budget enforces limit)

// Pluggable arch:
//   Pool, Signal, Decider, Cooldown, Bounds → Autoscaler

Self-Assessment Checklist¶

• I can implement AIMD with anti-windup-equivalent (minimum step)
• I can implement PID with anti-windup
• I can size a pool from Little's Law
• I can integrate breaker, limiter, backpressure with autoscaler
• I can architect a pluggable autoscaler with composable parts
• I can diagnose oscillation, retry storms, positive feedback
• I can build a predictive scaler with reactive fallback
• I can design cost-aware policies
• I can test the decider in isolation from real pool
• I can design multi-pool budget coordination

Summary¶

Junior taught mechanism. Middle taught policy. Senior teaches systems thinking.

Big themes: - AIMD: stable algorithm borrowed from TCP - PID: control-theory technique; sometimes overkill - Little's Law: analytical baseline for sizing - Integration: autoscaler doesn't live alone — combine with backpressure, breakers, limiters - Pluggable architecture: composable, testable, swappable parts - Cost-aware and predictive: layered on top of reactive - Multi-pool coordination: global budget, fairness

Professional level zooms in on production internals: ants/tunny/pond, distributed coordination, capacity planning math, and the deep operational knowledge you need at scale.

What You Can Build¶

• A production autoscaler with composable signals, deciders, and cooldowns
• A multi-pool service with global budget and fairness
• A predictive scaler with time-of-day patterns and reactive correction
• A cost-aware autoscaler that respects budgets
• A PID-driven utilization controller for tight latency control

Further Reading¶

• "TCP/IP Illustrated" — Stevens, on AIMD
• "Process Dynamics and Control" — for PID
• "Site Reliability Engineering" — Google, chapter on autoscaling
• "Designing Data-Intensive Applications" — Kleppmann, on backpressure
• Papers on AWS Auto Scaling Group internals
• Linux kernel's CPU governor (similar control loop ideas)

Related Topics¶

• Backpressure patterns
• Circuit breaker pattern
• Rate limiting
• Capacity planning
• Distributed coordination

Deep Dive: Building a PID Controller for a Worker Pool¶

We have introduced PID. Let us implement and tune one for a real worker pool.

Target¶

Keep worker utilization at 70%. When utilization drifts up, grow. When it drifts down, shrink. Smooth, no oscillation.

Code¶

package main

import (
    "context"
    "math"
    "sync/atomic"
    "time"
)

type PIDController struct {
    Kp, Ki, Kd float64
    Setpoint   float64

    integral   float64
    lastError  float64
    lastTime   time.Time
    primed     bool

    // anti-windup
    IntegralMin, IntegralMax float64
}

func (p *PIDController) Step(measured float64, now time.Time) float64 {
    err := p.Setpoint - measured

    if !p.primed {
        p.primed = true
        p.lastError = err
        p.lastTime = now
        return p.Kp * err
    }

    dt := now.Sub(p.lastTime).Seconds()
    if dt <= 0 {
        return p.Kp * err
    }

    p.integral += err * dt
    if p.integral > p.IntegralMax { p.integral = p.IntegralMax }
    if p.integral < p.IntegralMin { p.integral = p.IntegralMin }

    derivative := (err - p.lastError) / dt
    p.lastError = err
    p.lastTime = now

    return p.Kp*err + p.Ki*p.integral + p.Kd*derivative
}

func (p *PIDController) Reset() {
    p.integral = 0
    p.lastError = 0
    p.primed = false
}

Wiring into a pool¶

type PIDAutoscaler struct {
    Pool     *Pool
    Pid      *PIDController
    Interval time.Duration
    Floor    int
    Ceiling  int

    Util func() float64
}

func (a *PIDAutoscaler) Run(ctx context.Context) {
    ticker := time.NewTicker(a.Interval)
    defer ticker.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case now := <-ticker.C:
            u := a.Util()
            control := a.Pid.Step(u, now)
            cur := a.Pool.Size()
            // Interpret control as a fractional delta in pool size
            target := cur + int(math.Round(control * float64(cur) / 10))
            if target < a.Floor { target = a.Floor }
            if target > a.Ceiling { target = a.Ceiling }
            if target != cur {
                a.Pool.Resize(target)
            }
        }
    }
}

The control output is multiplied by cur/10 so the delta is proportional to current size — larger pools change in larger steps.

Tuning¶

Starting values: - Kp = 10 (proportional gain) - Ki = 0.5 (small integral) - Kd = 1.0 (small derivative) - Setpoint = 0.7

Run; observe behavior. If overshoots: reduce Kp. If steady-state error: increase Ki. If oscillates: reduce Kd or Kp.

Ziegler-Nichols method:

1. Set Ki = Kd = 0.
2. Increase Kp until sustained oscillation. Call this critical Kp = Ku, period T_u.
3. Set Kp = 0.6 Ku, Ki = 1.2 Ku / T_u, Kd = 0.075 Ku T_u.

This is a starting point; manual tuning usually follows.

Real-world challenge: integral windup¶

Imagine util is consistently 0.95 (above setpoint of 0.7). The error is -0.25 each tick. Integral grows: -0.25, -0.5, -0.75, ... The control output becomes very negative; the autoscaler aggressively shrinks. But wait — we want to grow, not shrink, when util is high.

Sign error! The setpoint is "we want util to be 70%, not higher." Higher util → grow. So error should be measured - setpoint = +0.25. Sign was wrong.

Let me fix:

err := measured - p.Setpoint  // not setpoint - measured

With this fix: - High util → positive error → positive control → grow ✓ - Low util → negative error → negative control → shrink ✓

Now integral windup: if we are at ceiling and can't grow, error stays positive, integral keeps growing. Anti-windup helps: pause integration when at ceiling.

if cur >= a.Ceiling && control > 0 {
    // pause integration during saturation in the grow direction
    a.Pid.integral -= err * dt  // undo this tick's increment
}

This is the operational reality of PID. The math is straightforward; the engineering is the corner cases.

When PID is genuinely useful¶

For a worker pool, PID becomes the right choice when:

• You care about precise utilization (e.g., maintaining 70% exactly for cost reasons)
• The signal has measurable trends that PID can follow
• You have time to tune carefully
• The system is slow enough that PID's smoothness helps

For most pools, AIMD or simple threshold suffices. PID is a senior-level capability you may or may not need.

Deep Dive: Implementing AIMD with Anti-Hysteresis¶

Standard AIMD oscillates slightly around the optimal pool size. Sometimes this is fine; sometimes it isn't. Anti-hysteresis is a small modification that reduces oscillation amplitude.

The idea¶

After a multiplicative shrink, the pool is smaller than optimal. Reactive grow brings it back. The cycle repeats; pool oscillates.

If we remember recent oscillations, we can grow back to the post-shrink size faster, or hold longer before shrinking again.

type AntiHystAIMD struct {
    GrowStep      int
    ShrinkFactor  float64
    History       []int  // recent target sizes
}

func (a *AntiHystAIMD) Decide(cur int, sig Signals) int {
    a.History = append(a.History, cur)
    if len(a.History) > 20 {
        a.History = a.History[1:]
    }
    switch {
    case shouldGrow(sig):
        // Grow toward median of recent history, faster
        med := median(a.History)
        if med > cur {
            return min(cur+a.GrowStep*2, med)
        }
        return cur + a.GrowStep
    case shouldShrink(sig):
        return cur - max(1, int(float64(cur)*a.ShrinkFactor))
    }
    return cur
}

When the pool oscillates around a steady size, the median of history captures that. Growing toward the median is faster than naive additive.

This is a heuristic; production behavior depends on workload. It is most useful for workloads where the optimal size is stable.

Trade-off¶

• Pro: less oscillation, faster recovery from shrink
• Con: slower to adapt when the actual right size changes

For workloads with stable optima, anti-hysteresis is a win. For workloads where the right size varies, it can be worse than pure AIMD.

Deep Dive: Time-Series Forecasting for Predictive Scaling¶

Predictive scaling needs a forecast. Let us look at three approaches.

Linear extrapolation¶

Simplest. Take recent samples; fit a line; extrapolate.

type LinearForecast struct {
    samples []point  // (time, value)
}

type point struct {
    t time.Time
    v float64
}

func (f *LinearForecast) Add(t time.Time, v float64) {
    f.samples = append(f.samples, point{t, v})
    if len(f.samples) > 100 { f.samples = f.samples[1:] }
}

func (f *LinearForecast) Predict(at time.Time) float64 {
    if len(f.samples) < 2 { return 0 }
    // least-squares regression on (t, v)
    var sumT, sumV, sumTV, sumTT float64
    base := f.samples[0].t.Unix()
    for _, p := range f.samples {
        t := float64(p.t.Unix() - base)
        sumT += t
        sumV += p.v
        sumTV += t * p.v
        sumTT += t * t
    }
    n := float64(len(f.samples))
    m := (n*sumTV - sumT*sumV) / (n*sumTT - sumT*sumT)
    b := (sumV - m*sumT) / n
    tt := float64(at.Unix() - base)
    return m*tt + b
}

Good for short horizons (next 1-2 minutes). Fails for non-linear trends.

Exponential smoothing (Holt-Winters)¶

Captures trend and seasonality. Built-in to many time-series libraries.

type HoltWinters struct {
    alpha, beta, gamma float64
    season int        // length of seasonal period
    level, trend float64
    seasonal []float64
}

Implementation is involved; libraries like github.com/montanaflynn/stats or a custom port of statsmodels can help.

Useful for daily patterns: predict tomorrow morning's load based on the last 7 mornings.

Neural networks¶

Overkill for most worker pools, but used in large-scale systems. LSTM or transformer trained on time-series of pool load.

The cost: training infrastructure, model serving, model staleness handling. The benefit: better predictions for complex non-linear workloads.

For 95% of services, linear or exponential smoothing is enough. Reach for neural networks only if the workload truly demands it.

Deep Dive: Multi-Pool Budget Implementation¶

We sketched multi-pool budgets earlier. Now a complete implementation.

type Budget struct {
    mu      sync.Mutex
    total   int
    used    int
    waiters []chan int
}

func NewBudget(total int) *Budget {
    return &Budget{total: total}
}

func (b *Budget) Request(want int) int {
    b.mu.Lock()
    defer b.mu.Unlock()
    avail := b.total - b.used
    if avail >= want {
        b.used += want
        return want
    }
    granted := avail
    b.used += granted
    return granted  // partial grant
}

func (b *Budget) Release(n int) {
    b.mu.Lock()
    defer b.mu.Unlock()
    b.used -= n
    if b.used < 0 { b.used = 0 }
    // notify waiters
    for len(b.waiters) > 0 && b.used < b.total {
        ch := b.waiters[0]
        b.waiters = b.waiters[1:]
        select { case ch <- 1: default: }
    }
}

func (b *Budget) Available() int {
    b.mu.Lock()
    defer b.mu.Unlock()
    return b.total - b.used
}