Unlimited Goroutines — Professional Level¶

Table of Contents¶

1. Introduction
2. The Architecture View
3. Concurrency Budgets in Microservices
4. Capacity Planning From First Principles
5. Kubernetes Pod Sizing vs Goroutine Counts
6. Vertical vs Horizontal Bounding
7. Graceful Degradation Strategies
8. Circuit Breakers as Goroutine Governors
9. Observability for Goroutine Counts at Fleet Scale
10. runtime.NumGoroutine as a Production Metric
11. pprof goroutine Profiles in Production
12. Continuous Profiling
13. Incident Postmortems
14. Service Level Objectives for Concurrency
15. Cross-Team Concurrency Contracts
16. Cost Modelling: Goroutines and the Cloud Bill
17. Multi-Tenant Architectures
18. Load Testing for Bounded Systems
19. Chaos Engineering for Concurrency
20. Static Analysis and Lint Rules
21. Migration Strategies for Large Codebases
22. Production Case Studies
23. The Long View: Engineering Discipline
24. Self-Assessment
25. Summary

Introduction¶

The senior-level material treated unlimited goroutines as an implementation anti-pattern: how to detect, bound, test, and refactor it within a service. The professional-level material treats it as an architectural concern: how to design a fleet of services where bounds are first-class properties, where capacity is planned, where degradation is graceful, and where the entire organisation has an answer to "what happens at 10x load."

At this level, the question is no longer "should I bound this fan-out?" — the answer is always yes. The question is:

• What is the right bound for this service, given its position in the call graph and the resources it shares with peers?
• How do I ensure the bound, set today, is still correct in six months when the load doubles?
• How do I detect, in production, that a bound is being violated, before users do?
• How does my service's bound interact with my dependencies' bounds?
• How do I migrate a 500 000-line codebase from unbounded to bounded, without rewriting it?
• What does my incident response look like when a bound fails?
• How do I budget concurrency across tens or hundreds of services?

These are questions for staff engineers, senior staff, principal engineers — the people whose job is the system, not the file. The material below is dense; it expects you have already mastered the senior level and want to operate the same patterns at organisation scale.

The Architecture View¶

Stand back from any single Go service and look at your system as a graph: nodes are services, edges are RPC calls, weights are concurrency budgets. From this view, several truths become evident:

1. Bounds are global properties. A bound on service A's outgoing calls to service B affects B's incoming load, B's outgoing calls to C, and C's behaviour to D. The bound stack runs through the entire graph.

2. The slowest path determines the system's bound. Following Amdahl's Law in reverse — the system's maximum useful concurrency is limited by its narrowest tier. If your database connection pool is 50, no amount of upstream concurrency helps beyond serving 50 concurrent DB-bound operations.

3. Bounds form a feedback loop with capacity planning. You set bounds based on capacity; capacity is planned based on bounds. Adjust one and the other needs revisiting.

4. Pod scaling is bound-aware. Adding pods multiplies each pod's bounds. If each pod allows 100 concurrent requests and you scale from 10 to 20 pods, total system concurrency doubles. But downstream services see twice the load and may be overwhelmed.

5. Bounds are inheritance, not composition. A service that calls 5 others does not compose 5 bounds into one; it inherits the minimum of them, divided by its fan-out factor.

The architecture view forces you to draw the diagram. Until you have drawn the bound stack across services, you do not understand your system's behaviour at peak load.

The bound diagram¶

For a 5-service system, the bound diagram looks like:

[Edge: 5000 RPS, 500 in-flight]
       |
       v
[API Gateway: 1000 RPS, 200 in-flight]
       |
       +---> [Auth Service: 400 RPS, 100 in-flight]
       |
       +---> [Catalog Service: 800 RPS, 150 in-flight]
       |       |
       |       +---> [Database: 50 connections]
       |
       +---> [Recommendations: 500 RPS, 100 in-flight]
              |
              +---> [Model Service: 300 RPS, 60 in-flight]
              +---> [Database: 50 connections]

Reading this diagram:

• The Edge admits 500 concurrent requests at most.
• Each Edge request fans out to ~3 internal calls (auth + catalog + recommendations).
• 500 × 3 = 1500 inbound to internal tier. But internal tier capacity is 200 + 100 + 150 + 100 = 550. Oversubscribed.

The diagram reveals the problem before any request flows. The fix: either reduce Edge admission, increase internal capacity, or reduce per-request fan-out (cache, deduplicate, batch).

The bound contract language¶

To formalise bounds across teams, adopt a small vocabulary in API documentation:

• Max RPS: requests per second a service guarantees to absorb.
• Max in-flight: simultaneous requests a service guarantees to absorb.
• Per-caller cap: maximum in-flight per identified caller.
• Burst tolerance: how long the service tolerates above-Max-RPS before degrading.
• Degradation policy: what happens when above capacity (reject / queue / partial response).

Every service publishes these. Every caller reads them. Mismatches are caught in design review, not in production.

Concurrency Budgets in Microservices¶

A concurrency budget is the total in-flight work a logical scope (service, team, organisation) is allowed to perform. It is to concurrency what a financial budget is to spending: a deliberately-set ceiling, allocated across uses, monitored for adherence.

Single-service budget¶

The simplest case: one service's total in-flight concurrency. The budget is implemented as a global semaphore or HTTP server option.

package main

import (
    "context"
    "net/http"
    "time"

    "golang.org/x/sync/semaphore"
)

const (
    GlobalBudget = 200
)

var globalSem = semaphore.NewWeighted(GlobalBudget)

func admit(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx, cancel := context.WithTimeout(r.Context(), 1*time.Second)
        defer cancel()
        if err := globalSem.Acquire(ctx, 1); err != nil {
            http.Error(w, "overloaded", http.StatusServiceUnavailable)
            return
        }
        defer globalSem.Release(1)
        next.ServeHTTP(w, r)
    })
}

Every request acquires from globalSem. If at capacity, the request waits up to 1 second; if still no slot, returns 503.

Multi-tier service budget¶

A service that does work on multiple tiers (read-heavy + write-heavy + background) splits its budget:

type Budget struct {
    Read       *semaphore.Weighted
    Write      *semaphore.Weighted
    Background *semaphore.Weighted
}

func NewBudget(read, write, background int64) *Budget {
    return &Budget{
        Read:       semaphore.NewWeighted(read),
        Write:      semaphore.NewWeighted(write),
        Background: semaphore.NewWeighted(background),
    }
}

The total is read + write + background. The split is policy: protect writes from being starved by reads, ensure background work always has minimum capacity.

Organisation-wide budget¶

Across services, the budget is allocated by team. Team A operates the catalog service; their concurrency budget is 200 in-flight (which they may split across multiple pods). Team B operates payments; their budget is 50 (smaller because payments are higher-stakes and slower).

The total organisational budget is bounded by infrastructure cost. A budget exceeds infrastructure if and only if the cost is justified by traffic. Quarterly capacity reviews adjust budgets up or down.

Budget allocation¶

A budget is allocated in three ways:

1. By tier: read vs write vs background.
2. By tenant: customer-A vs customer-B, with isolation.
3. By caller: incoming-from-service-X vs incoming-from-service-Y, to detect noisy callers.

Each layer adds a semaphore. The total in-flight at any moment is the minimum of the layers, applied in sequence.

Budget exhaustion¶

When the budget is exhausted, the policy options are:

• Reject: return 503. The caller decides whether to retry.
• Queue: hold the request in memory until a slot opens. Bounded queue depth.
• Degrade: return a partial response. Skip enrichment, serve from cache.
• Shed: drop a percentage of requests (often randomly, sometimes prioritised).
• Throttle: rate-limit the caller per their identity.

Different parts of the service use different policies. Edge services typically reject; internal services typically queue; background pools degrade or drop.

Capacity Planning From First Principles¶

Capacity planning is the practice of deciding, before a service is loaded, how many resources it needs. It is mostly arithmetic.

The Little's Law derivation¶

Given: - L = in-flight count (concurrency) - λ = arrival rate (requests per second) - W = mean response time (seconds per request)

Little's Law: L = λW.

To set the bound (L), you must measure W (or estimate it) and decide on a target λ.

Example: a service whose p95 response time is 100 ms (W = 0.1s). To handle 1000 RPS, in-flight = 1000 × 0.1 = 100. Set the bound to 100 (or 120 for headroom).

Tail latency vs mean latency¶

W in Little's Law is the mean latency. But the bound must handle the tail. If p95 = 100ms but p99 = 500ms, the tail occasionally backs up. The bound should be sized to accommodate the tail without permanent backup.

Practical: set L = RPS × p99 for safety, then test. If queues regularly fill, raise; if you never approach the bound, lower (saves memory).

Bound vs throughput trade-off¶

A higher bound = higher throughput up to some point. Beyond that point, additional concurrency adds overhead (scheduler, GC, contention) and decreases throughput. The optimum is found by load testing.

A canonical curve:

Throughput
   ^
   |       ___________
   |      /           \___
   |     /                \___
   |    /                     \____
   |   /
   |  /
   |/
   +------------------------------> Concurrency
   0   N_opt           N_thrash

N_opt is the optimum; beyond it, throughput declines. The decline is gradual at first, then steep at N_thrash where the scheduler is saturated.

In practice, set the bound at 0.7 × N_opt for headroom.

Capacity for cyclic load¶

If your service has daily peaks (10x baseline), capacity-plan for the peak. Provisioning for the average leaves you outage-prone at peak.

Capacity = Peak_RPS × Peak_p99_latency / 0.7 (utilisation)

If peak is 5000 RPS and peak p99 is 200ms:

Capacity = 5000 × 0.2 / 0.7 ≈ 1430 in-flight

This is the total across pods. If each pod handles 100 in-flight, you need 15 pods at peak (and 1-2 at baseline).

Capacity for bursty load¶

Bursty load is different: brief spikes of N× baseline lasting a few seconds. Two strategies:

1. Smoothed: queue the burst, drain at steady rate. Requires bounded queue depth and accepts some latency.
2. Spiky: maintain enough capacity to absorb the burst. Higher cost, lower latency.

The choice depends on user-visible latency tolerance.

Capacity planning checklist¶

Before launching a service, answer:

• What is the expected peak RPS?
• What is the expected p99 latency at peak?
• What is the bound (Little's Law)?
• How many pods, given per-pod bound?
• What is the auto-scale policy?
• What is the degradation behaviour above capacity?
• What is the alert threshold (e.g. 70% utilisation)?
• What is the recovery path after overload?

Each answer is a number or a sentence. The full set is your capacity plan.

Kubernetes Pod Sizing vs Goroutine Counts¶

In Kubernetes, pod sizing (CPU, memory) constrains goroutine count. The relationship is:

• CPU limit determines GOMAXPROCS (with automaxprocs library or manual configuration).
• Memory limit determines maximum live goroutines (each consuming stack + heap).
• Pod count determines aggregate fleet capacity.

Setting GOMAXPROCS correctly¶

By default, GOMAXPROCS is runtime.NumCPU() — the host's CPU count. Inside a container with a CPU limit of 1.0, this can return 32 (host CPUs) even though the container only has 1 CPU's worth of compute. The result: excessive scheduler thrash.

Use automaxprocs:

package main

import (
    _ "go.uber.org/automaxprocs"
)

This package reads the container's CPU limit and sets GOMAXPROCS accordingly. Without it, your Go process happily creates 32 Ps for 32 logical processors, while the kernel CFS-throttles you to 1 CPU's worth of execution. Every M sits idle most of the time, but each consumes scheduling bookkeeping.

Memory limit and goroutine count¶

Each goroutine consumes: - 2 KB initial stack (often grows to 8-64 KB in production). - A few hundred bytes of g struct overhead. - Heap allocations made during execution.

A pod with a 512 MB memory limit and 50% headroom (256 MB available) allows roughly 256 MB / 16 KB ≈ 16 000 goroutines before stack memory dominates — and that's before counting heap. In practice, you want goroutine count to be well under this, since heap allocations vastly exceed stack costs.

Rule of thumb: target peak goroutine count = 10% of memory_limit / 64KB. This leaves 90% of memory for actual work.

Pod count and fleet bounds¶

If each pod can serve 100 concurrent requests and you need 5000 RPS at 100ms latency, you need: - Per-pod: 100 in-flight = 1000 RPS. - Total: 5 pods. - Headroom (for rolling deploys, hot pods): 7-8 pods.

The fleet's aggregate bound is pod_count × per_pod_bound. Auto-scaling adjusts pod count based on observed load.

HPA and concurrency metrics¶

The default Kubernetes Horizontal Pod Autoscaler (HPA) scales on CPU. For concurrency-bound services, scale on a custom metric:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-service
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  minReplicas: 5
  maxReplicas: 50
  metrics:
  - type: Pods
    pods:
      metric:
        name: process_goroutines_avg
      target:
        type: AverageValue
        averageValue: "5000"

This scales up when average goroutine count exceeds 5000 per pod. The metric must be exposed via the metrics adapter (Prometheus adapter is common).

Pod sizing trade-offs¶

• Many small pods: better isolation (one pod's load does not affect another), more rebalance overhead, more idle resources.
• Few large pods: better resource utilisation, larger blast radius per failure, harder to schedule.

The trade-off is industry-dependent. SaaS multi-tenancy often favours many small pods. Batch processing often favours few large pods.

Vertical vs Horizontal Bounding¶

There are two axes for adding capacity: vertical (larger pods) and horizontal (more pods). The bound strategy differs.

Vertical bounding¶

Larger pods = higher per-pod concurrency. CPU and memory limits scale up, so the in-pod bound can be higher.

Pros: - Simpler architecture: fewer pods to manage. - Better cache locality: more work happens on the same machine. - Lower coordination overhead between pods.

Cons: - Larger blast radius: one pod failure affects more requests. - Slower rolling restarts: each pod takes longer to drain. - Limited by physical hardware: you cannot have a 1 TB pod on a 256 GB host.

The vertical limit on a pod is the machine. Beyond ~32 GB / 16 CPUs, vertical scaling becomes diminishing-returns: GC pauses extend, scheduler complexity grows, single-pod throughput plateaus.

Horizontal bounding¶

More pods, smaller each. Fleet capacity = pod_count × per_pod.

Pros: - Smaller blast radius. - Better deployment incrementality. - Often cheaper (smaller instance types are cheaper per CPU).

Cons: - More inter-pod coordination (load balancers, service discovery). - Higher latency for distributed operations. - Per-pod overhead (each pod runs the same boot sequence).

When to scale which way¶

Rule of thumb: - Start horizontal. Default to many small pods. - Scale vertical when you have a single-process bound (e.g. an in-memory cache that benefits from being larger). - Scale vertical when GC pauses are not yet a problem and adding CPU/memory improves throughput linearly.

In Go specifically: - Vertical scaling beyond ~8 GB heap often degrades because GC overhead climbs. - Horizontal scaling has no equivalent upper limit; you can have 1000 1-CPU pods.

Bounding policy¶

The per-pod bound interacts with the fleet count: - Fleet bound = pod count × per-pod bound. - If fleet bound exceeds downstream capacity, the downstream is the cap. - Adjust either pod count or per-pod bound to match.

In practice, the per-pod bound is set conservatively (room for retries, jitter), and the pod count auto-scales to match demand.

Graceful Degradation Strategies¶

Graceful degradation is the practice of giving the user something when the system cannot give them everything. Without it, every failure mode is a catastrophe.

Strategy 1: Cached fallback¶

type Service struct {
    primary *primaryClient
    cache   *cache.Cache
    sem     *semaphore.Weighted
}

func (s *Service) Get(ctx context.Context, key string) (Value, error) {
    if !s.sem.TryAcquire(1) {
        if v, ok := s.cache.Get(key); ok {
            return v, nil // degraded: cache hit
        }
        return Value{}, ErrOverloaded
    }
    defer s.sem.Release(1)
    v, err := s.primary.Get(ctx, key)
    if err != nil { return Value{}, err }
    s.cache.Set(key, v)
    return v, nil
}

When at capacity, return a cached value if available. The user gets a stale-but-correct answer.

Strategy 2: Partial response¶

type Result struct {
    User      User
    Recommendations []Item
    Recent          []Item
    Degraded        bool
}

func (s *Service) GetProfile(ctx context.Context, userID string) Result {
    user, _ := s.users.Get(ctx, userID)
    var (
        recs []Item
        recent []Item
        degraded bool
    )
    if s.sem.TryAcquire(1) {
        defer s.sem.Release(1)
        recs, _ = s.recs.Get(ctx, userID)
        recent, _ = s.recent.Get(ctx, userID)
    } else {
        degraded = true
    }
    return Result{User: user, Recommendations: recs, Recent: recent, Degraded: degraded}
}

When at capacity, return the essential user data; skip the optional enrichments. The frontend knows the response is degraded and can render accordingly.

Strategy 3: Static fallback¶

For very high-traffic public services, when everything is down, return a static page:

func handler(w http.ResponseWriter, r *http.Request) {
    if !s.health.IsHealthy() {
        http.ServeFile(w, r, "static/offline.html")
        return
    }
    // ... normal path
}

The user sees "we're experiencing high load, please try again." This is better than a 500.

Strategy 4: Prioritised admission¶

Reject low-priority traffic first, keep serving high-priority:

func admit(req *http.Request) bool {
    priority := classifyPriority(req)
    switch priority {
    case High:
        return s.highSem.TryAcquire(1)
    case Medium:
        if s.systemLoad() > 0.8 { return false }
        return s.medSem.TryAcquire(1)
    case Low:
        if s.systemLoad() > 0.6 { return false }
        return s.lowSem.TryAcquire(1)
    }
}

Background jobs are dropped first; logged-in user requests are dropped last.

Strategy 5: Asynchronous fallback¶

Accept the request, return an "ack" immediately, do the work later if and when capacity permits:

func (s *Service) AcceptForLater(w http.ResponseWriter, r *http.Request) {
    if !s.backgroundPool.TrySubmit(processJob{req: r}) {
        http.Error(w, "queue full", http.StatusServiceUnavailable)
        return
    }
    w.WriteHeader(http.StatusAccepted)
    // 202 — accepted, will process eventually
}

The user gets a 202 with a job ID. They poll or webhook for completion.

Picking the right strategy¶

Most production services combine 2-3 strategies depending on endpoint.

Circuit Breakers as Goroutine Governors¶

A circuit breaker is a state machine that disables calls to a known-failing dependency. The states:

• Closed: calls flow normally.
• Open: calls fail immediately, no actual call made.
• Half-open: a small number of trial calls flow; if they succeed, transition back to closed; if they fail, back to open.

Why it bounds concurrency¶

Without a circuit breaker, when a downstream fails, every caller goroutine times out (waits for the configured deadline before failing). At 30s timeouts and 1000 RPS, that's 30 000 goroutines waiting on a dead downstream. Each holds resources.

With a circuit breaker, after a threshold of failures, the breaker opens. New calls fail immediately. The waiting goroutines are not created. Resources are not held.

The breaker is, in effect, a zero-concurrency limit on calls to a failing dependency.

A circuit breaker library (Sony's gobreaker)¶

package downstream

import (
    "context"
    "errors"

    "github.com/sony/gobreaker"
)

type Client struct {
    breaker *gobreaker.CircuitBreaker
    raw     RawClient
}

func New(raw RawClient) *Client {
    settings := gobreaker.Settings{
        Name:        "downstream",
        MaxRequests: 1,
        Interval:    60 * time.Second,
        Timeout:     30 * time.Second,
        ReadyToTrip: func(counts gobreaker.Counts) bool {
            failureRatio := float64(counts.TotalFailures) / float64(counts.Requests)
            return counts.Requests >= 20 && failureRatio >= 0.5
        },
    }
    return &Client{
        breaker: gobreaker.NewCircuitBreaker(settings),
        raw:     raw,
    }
}

func (c *Client) Do(ctx context.Context, req Request) (Response, error) {
    result, err := c.breaker.Execute(func() (interface{}, error) {
        return c.raw.Do(ctx, req)
    })
    if err != nil {
        if errors.Is(err, gobreaker.ErrOpenState) {
            return Response{}, ErrUpstreamDown
        }
        return Response{}, err
    }
    return result.(Response), nil
}

The breaker trips at 50% failure rate over 20+ requests. Once open, all calls fail with ErrOpenState immediately.

Combining with bounded fan-out¶

The circuit breaker prevents new goroutines from being created during downstream failure, but existing ones must still terminate. The combination of breaker + bounded fan-out + context cancellation gives:

1. New requests with open breaker: fail instantly, no goroutine spawn.
2. In-flight requests: timeout via context, return error.
3. Bounded fan-out: prevents queue buildup during transition.

Without the breaker, step 1 spawns a goroutine that may wait 30s. With the breaker, step 1 is zero-cost.

Anti-pattern: breaker without bound¶

Some teams add a breaker but no goroutine bound:

for _, x := range items {
    go func(x Item) {
        _, err := breaker.Execute(func() (interface{}, error) {
            return downstream.Call(x)
        })
        // ...
    }(x)
}

The breaker prevents downstream calls but does not prevent the spawn. A 1 000 000-item loop still spawns 1 000 000 goroutines (now just doing breaker.Execute, which returns ErrOpenState in nanoseconds). Memory still climbs, GC still pauses. The breaker is necessary but not sufficient.

Always bound the fan-out before the breaker.

Breaker state observability¶

Export breaker state as a metric:

prometheus.NewGaugeFunc(prometheus.GaugeOpts{
    Name: "circuit_breaker_state",
    ConstLabels: prometheus.Labels{"name": "downstream"},
}, func() float64 {
    switch breaker.State() {
    case gobreaker.StateClosed: return 0
    case gobreaker.StateHalfOpen: return 1
    case gobreaker.StateOpen: return 2
    }
    return -1
})

Alert if the breaker is open for > 5 minutes; the downstream is consistently failing and needs intervention.

Observability for Goroutine Counts at Fleet Scale¶

A fleet of services emits many concurrency metrics. The observability layer must aggregate, alert, and visualise across pods, services, and time.

The metrics hierarchy¶

• Process level: runtime.NumGoroutine(), runtime.MemStats.
• Pool level: per-pool in-flight, queue depth, submitted, completed, failed.
• Request level: per-request fan-out factor, latency, errors.
• Service level: total RPS, total in-flight, p50/p95/p99 latency.
• Fleet level: sum across pods, percentile of pod-level metrics.

Each level is exported as a Prometheus metric. The aggregation happens in Prometheus queries.

A canonical metric set¶

package metrics

import (
    "runtime"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    Goroutines = promauto.NewGaugeFunc(prometheus.GaugeOpts{
        Name: "process_goroutines",
        Help: "Current number of goroutines",
    }, func() float64 { return float64(runtime.NumGoroutine()) })

    HeapInUse = promauto.NewGaugeFunc(prometheus.GaugeOpts{
        Name: "process_heap_in_use_bytes",
    }, func() float64 {
        var ms runtime.MemStats
        runtime.ReadMemStats(&ms)
        return float64(ms.HeapInuse)
    })

    PoolInflight = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "pool_inflight",
    }, []string{"pool"})

    PoolQueueDepth = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "pool_queue_depth",
    }, []string{"pool"})

    PoolSubmitted = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "pool_submitted_total",
    }, []string{"pool"})

    PoolCompleted = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "pool_completed_total",
    }, []string{"pool", "outcome"})

    SemWaiters = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "semaphore_waiters",
    }, []string{"semaphore"})

    SemHolders = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "semaphore_holders",
    }, []string{"semaphore"})
)

Apply to every pool and semaphore in your service. The label dimensions are what makes the metrics queryable later.

Recording rules and SLI definition¶

In Prometheus, define recording rules for fleet aggregates:

groups:
- name: concurrency
  interval: 30s
  rules:
  - record: service:goroutines:sum
    expr: sum by (service) (process_goroutines)
  - record: service:goroutines:p99
    expr: quantile by (service) (0.99, process_goroutines)
  - record: pool:inflight:saturation
    expr: pool_inflight / on(pool, service) group_left pool_capacity

Now your dashboard queries service:goroutines:sum (total goroutines across fleet) and pool:inflight:saturation (per-pool utilisation, 0-1).

Alerts¶

Sample alert rules:

groups:
- name: concurrency_alerts
  rules:
  - alert: GoroutineCountHigh
    expr: process_goroutines > 50000
    for: 5m
    labels: { severity: warning }
    annotations:
      summary: "{{ $labels.instance }} has high goroutine count"

  - alert: GoroutineCountGrowingRapidly
    expr: deriv(process_goroutines[5m]) > 100
    for: 10m
    labels: { severity: critical }
    annotations:
      summary: "{{ $labels.instance }} goroutine count is growing — likely leak"

  - alert: PoolSaturated
    expr: pool:inflight:saturation > 0.95
    for: 5m
    labels: { severity: warning }

Tune the thresholds to your baseline. The point is to alert on shape (growing, saturated), not absolute values.

Dashboards¶

A typical concurrency dashboard has:

1. Time-series of process_goroutines per service, stacked.
2. Heat map of per-pod goroutine count (rows = pods, color = count).
3. Saturation gauges for each named pool.
4. Distribution of fan-out factors (which endpoints fan out the most).
5. Inflight by tenant (top 10 tenants by concurrency consumption).
6. Goroutine count vs request rate scatter — points should cluster on a line; outliers are anomalous.

The dashboard is the operator's lens. New incidents are diagnosed from these views.

runtime.NumGoroutine as a Production Metric¶

runtime.NumGoroutine() is a single function that returns the count of live goroutines in the process. It is the single most important concurrency metric.

What it measures¶

• All goroutines that have been started and have not returned.
• Excludes the system goroutines that are not visible (a few runtime internals).
• Approximate: there is a small race window during a goroutine's start/end where it may not be counted.

How to expose it¶

import (
    "runtime"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var goroutineCount = promauto.NewGaugeFunc(prometheus.GaugeOpts{
    Name: "process_goroutines",
}, func() float64 { return float64(runtime.NumGoroutine()) })

The GaugeFunc is sampled on Prometheus scrape (every 15-30s typically). The function is cheap (it reads an atomic counter).

Interpreting the value¶

Absolute numbers are service-specific. The metric's power is in its shape over time: flat is good, monotonically rising is a leak, oscillating with load is healthy.

The "two-snapshot diff" technique¶

To find a leak:

1. Wait for service to be idle.
2. Take snapshot: start := runtime.NumGoroutine().
3. Run the suspected work (e.g. send 100 requests).
4. Wait for it to complete.
5. Take snapshot: end := runtime.NumGoroutine().
6. If end > start, you have leaked end - start goroutines.

Apply in a test:

func TestNoLeak(t *testing.T) {
    runtime.GC() // collect anything pending
    start := runtime.NumGoroutine()

    // run work
    for i := 0; i < 100; i++ { doStuff() }

    time.Sleep(time.Second) // let teardown happen
    runtime.GC()
    end := runtime.NumGoroutine()

    if end > start+5 { // allow small jitter
        t.Errorf("leaked %d goroutines", end-start)
    }
}

For more precision, use goleak.

Pitfalls¶

• Race with background GC. GC may finalise goroutines during the read; the count is approximate.
• Service warm-up. Many services start background goroutines on first request. The first request inflates the count; subsequent ones do not.
• Connection pooling. Idle pooled connections often have a goroutine reading from the socket (for HTTP/2 multiplexing). These goroutines are alive but doing nothing.

Interpret the value with these caveats. The metric is most useful in trend, not absolute.

pprof goroutine Profiles in Production¶

The goroutine profile is the diagnostic tool when NumGoroutine is high and you need to know why.

Enabling the endpoint¶

import (
    "net/http"
    _ "net/http/pprof"
)

func init() {
    go http.ListenAndServe("localhost:6060", nil)
}

This serves /debug/pprof/goroutine (and other profiles) on port 6060. In production, bind to localhost or a dedicated debug subnet — never expose pprof publicly.

Fetching a profile¶

curl -s http://pod:6060/debug/pprof/goroutine?debug=1 > goroutines.txt

debug=1 is human-readable. debug=2 is even more detailed (includes goroutine IDs, wait reasons).

A line in the output:

500 @ 0x40123a 0x402345 0x402567 ...
#       0x40123a        myapp/worker.(*Pool).run+0x4a
#       0x402345        runtime.gopark+0x12

Reads as: 500 goroutines are at this stack; their stack starts at worker.(*Pool).run. The gopark further down means they are blocked.

Aggregation by stack¶

Use pprof to aggregate:

go tool pprof -top -cum http://pod:6060/debug/pprof/goroutine

Output is a list of functions ranked by cumulative goroutine count. The top entry is the function holding the most goroutines.

The diff technique¶

Take two profiles, 30 seconds apart:

curl -s http://pod:6060/debug/pprof/goroutine > go1.pb.gz
sleep 30
curl -s http://pod:6060/debug/pprof/goroutine > go2.pb.gz
go tool pprof -base=go1.pb.gz go2.pb.gz

The diff shows goroutines that exist in both snapshots — the persistent ones, the leaks. Ephemeral goroutines (request handlers) appear in one but not the other.

Automation¶

Build a tool that:

1. Pulls goroutine profile every 5 minutes.
2. Diffs against the previous snapshot.
3. Alerts if any stack has > 100 goroutines and grew.

This catches slow leaks before they cause an outage. Many companies have a Prometheus exporter that scrapes the profile and exports goroutine_count_by_stack as a high-cardinality metric.

Reading complex stacks¶

Some stacks are nested deeply:

1042 @ 0x405abc 0x405def 0x40623f 0x406abc 0x407def ...
#       0x405abc        mongodb-driver.(*Client).executeOperation+0x12
#       0x405def        mongodb-driver.(*Client).Find+0x3a
#       0x40623f        myapp/repo.(*UserRepo).GetUsers+0x1a
#       0x406abc        myapp/handler.(*UserHandler).List+0x4d
#       0x407def        net/http.(*ServeMux).ServeHTTP+0x21

The stack reads top-down: the deepest frame is the current execution point. 1042 goroutines are blocked in executeOperation, which means they are waiting on MongoDB. Either MongoDB is slow, the pool is exhausted, or a query is hanging.

The diagnostic chain: 1042 stuck → MongoDB pool size 50? → 1042 / 50 = 20x oversubscribed → check MongoDB latency.

The "wait reason"¶

debug=2 includes the wait reason:

goroutine 12345 [chan receive, 5 minutes]:

"chan receive, 5 minutes" means the goroutine has been blocked on a channel receive for 5 minutes. Long waits are suspicious — they often indicate a leak.

Common wait reasons: - chan receive: blocked on <-ch. - chan send: blocked on ch <- v (buffer full, no receiver). - select: blocked on select { ... }. - sleep: in time.Sleep. - sync.Mutex.Lock: contending for a mutex. - IO wait: waiting on a socket/file.

A leaked goroutine almost always has one of these reasons. The reason tells you what it is waiting for.

Continuous Profiling¶

Continuous profiling is the practice of capturing profiles every few seconds, storing them indefinitely, and analysing them retroactively. Tools: Pyroscope, Polar Signals, Datadog Profiler, Google Cloud Profiler, Grafana Phlare.

Why continuous¶

Sampling on-demand only captures the current state. By the time you suspect a leak, the leak has been growing for hours; you cannot reproduce the early stages.

Continuous profiling captures every state. You can rewind:

• "Show me goroutine profile from 03:15 UTC yesterday."
• "Diff CPU profile before and after the 02:00 deploy."
• "Trend of memory allocation by stack over the last week."

This is the difference between sampling diagnostics and observability.

Integration¶

For Pyroscope:

import (
    "github.com/grafana/pyroscope-go"
)

func main() {
    pyroscope.Start(pyroscope.Config{
        ApplicationName: "myapp",
        ServerAddress:   "https://pyroscope.example.com",
        ProfileTypes: []pyroscope.ProfileType{
            pyroscope.ProfileCPU,
            pyroscope.ProfileAllocObjects,
            pyroscope.ProfileAllocSpace,
            pyroscope.ProfileInuseObjects,
            pyroscope.ProfileInuseSpace,
            pyroscope.ProfileGoroutines,
            pyroscope.ProfileMutexCount,
            pyroscope.ProfileMutexDuration,
            pyroscope.ProfileBlockCount,
            pyroscope.ProfileBlockDuration,
        },
    })
    // ... rest of app
}

This sends profiles every 10 seconds (configurable). Storage and querying happen on the Pyroscope server.

What to look for¶

Daily review: - Top goroutine-holding functions; are any unexpected? - Top heap-allocating functions; are any growing week-over-week? - Mutex contention hot spots; do they correlate with latency spikes?

Per-incident: - Profile diff before and after the incident. - Stack of the leaked goroutines (if NumGoroutine spiked). - CPU consumed by what function (if CPU spiked).

The profile is the source of truth. Logs say "request failed"; profiles say "request failed because 1042 goroutines were blocked in MongoDB pool".

Cost considerations¶

Continuous profiling has overhead: 1-2% CPU and some network. For most services, this is acceptable. For latency-sensitive services, sample less frequently (every 30 seconds instead of every 10).

Storage is the bigger cost. Profiles are large; weeks of retention can be many GB per service. Tier storage: recent (hot) and old (cold, compressed).

Incident Postmortems¶

A postmortem for a concurrency-related incident has a structure. Below is a template you can use.

Template¶

TITLE: <one-line summary, no blame>
DATE: <YYYY-MM-DD>
DURATION: <HH:MM start to HH:MM end>
SEVERITY: <SEV1/2/3>
AUTHOR: <name>

SUMMARY
-------
<2-3 paragraphs: what happened, impact, root cause>

TIMELINE
--------
<UTC time>  <event>
<UTC time>  <event>
...

ROOT CAUSE
----------
<technical detail: which code, which input, which mechanism>

CONTRIBUTING FACTORS
--------------------
- <factor that didn't directly cause but made it worse>
- <another>

WHAT WENT WELL
--------------
- <quick alerting>
- <effective rollback>

WHAT WENT POORLY
----------------
- <slow detection>
- <missing runbook>

ACTION ITEMS
------------
- [P0] <fix the root cause>
  Owner: <name>, Due: <date>
- [P1] <improve detection>
  Owner: <name>, Due: <date>
- [P2] <documentation>
  Owner: <name>, Due: <date>

Example postmortem (synthetic)¶

TITLE: Catalog Service OOM cascade
DATE: 2026-04-15
DURATION: 14:23 to 14:51 UTC
SEVERITY: SEV1
AUTHOR: jdoe

SUMMARY
-------
At 14:23 UTC, the catalog-service pods began OOMing in sequence. Within
8 minutes, 6 of 8 pods had restarted. Latency p99 climbed from 80ms to
8 seconds during the incident; error rate reached 12%. Recovery began
at 14:51 when an admission control config rollout completed.

The root cause was unbounded fan-out in the bulk-update endpoint.
A customer issued a single bulk-update request with 50,000 SKUs.
The endpoint spawned one goroutine per SKU, each making 3 downstream
calls, for 150,000 simultaneous goroutines. Memory consumption per pod
went from 800 MB to 4.1 GB within 90 seconds, exceeding the 4 GB pod
limit.

TIMELINE
--------
14:23  Customer X submits POST /v1/catalog/bulk-update with 50,000 SKUs
14:23  catalog-pod-3 NumGoroutine alert fires (>50,000)
14:24  catalog-pod-3 OOMKilled
14:24  catalog-pod-5 receives the rebalanced request
14:25  catalog-pod-5 OOMKilled
14:25  pager goes off
14:31  On-call engineer identifies the customer request in logs
14:35  On-call attempts to block the customer via WAF
14:38  Block in effect; new requests rejected
14:40  In-flight pods still OOMing
14:45  Hotfix PR: add SetLimit(64) to fan-out
14:48  Hotfix deployed to canary
14:51  Canary stable; rolling deploy completes

ROOT CAUSE
----------
The bulk-update endpoint in catalog/bulk.go had:

  for _, sku := range req.SKUs {
      go func(sku SKU) {
          inv, _ := inventory.Update(ctx, sku)
          price, _ := pricing.Update(ctx, sku, inv)
          search.Index(ctx, sku, inv, price)
      }(sku)
  }

This unbounded fan-out had been in the codebase since 2024-08. Until
this incident, no customer had sent more than ~500 SKUs in a single
request; the bound was implicit in usage.

CONTRIBUTING FACTORS
--------------------
- No per-request batch size limit.
- No alerting on per-request fan-out factor.
- The bulk-update endpoint was tested with 1000 SKUs; not with 50,000.
- The customer's intent (bulk seasonal update) was unknown to ops.

WHAT WENT WELL
--------------
- The goroutine-count alert fired immediately (5 minutes from incident
  start to page).
- The on-call had a runbook for "high goroutine count" that listed the
  bulk-update endpoint as a known suspect.
- The WAF block was effective in seconds.

WHAT WENT POORLY
----------------
- 8-minute detection-to-mitigation. Customers experienced 8 minutes of
  partial outage.
- The hotfix took 13 minutes to prepare and deploy.
- We had no per-customer rate limit; one customer took down the service.

ACTION ITEMS
------------
- [P0] Add SetLimit(64) to bulk-update endpoint. [DONE in hotfix]
- [P0] Add max batch size = 1000 to bulk-update API.
  Owner: jdoe, Due: 2026-04-17.
- [P1] Add per-customer concurrency cap.
  Owner: jdoe, Due: 2026-04-22.
- [P1] Audit all bulk endpoints for unbounded fan-out.
  Owner: catalog team, Due: 2026-04-29.
- [P2] Update runbook with the diagnostic steps used in this incident.
  Owner: oncall lead, Due: 2026-04-25.
- [P2] Add per-endpoint fan-out factor metric.
  Owner: platform team, Due: 2026-05-15.

This template makes the cause explicit, the fix verifiable, and the prevention measurable.

Common postmortem patterns for unbounded fan-out¶

After many such postmortems, you see patterns:

1. The customer didn't know. The behaviour was within API spec, but the customer's input was 10-100x typical.
2. The team didn't test the edge case. Tests covered "typical" not "large" inputs.
3. The bound existed implicitly. "Bulk endpoint" had no explicit cap.
4. The cascade was the real damage. One pod OOM is fine; the rebalance cascade is what makes it an outage.
5. The fix is small. Adding SetLimit(N) is 1 line. The fix is small precisely because the absence is the bug.

Service Level Objectives for Concurrency¶

SLOs (Service Level Objectives) are quantified commitments to availability and performance. For concurrency, they look like:

• "p99 latency < 200ms under load up to 1000 RPS."
• "No requests rejected for capacity reasons more than 1% of the time."
• "Background queue depth < 1000 99% of the time."

Each SLO is measured continuously and budgeted: a small percentage of violations is allowed (the error budget), and if you exceed the budget you must invest in reliability rather than features.

Concurrency SLI examples¶

For each SLI, you set an SLO. The SLO is the contract.

Budgeting¶

If your SLO is "99.9% success rate" over a 30-day window, your error budget is 0.1% = 43 minutes per month. If you spend all 43 minutes on real incidents, you cannot risk further failures; freeze deploys, increase test coverage, harden the system. If you spend nothing, you have headroom for new features and rapid iteration.

The budget mechanism aligns incentives: stability and velocity trade off explicitly.

Concurrency-specific SLOs¶

• Saturation SLO: pool utilisation < 80% 99% of the time. If exceeded, scale or add tenants.
• Leak SLO: goroutine count growth rate < 10/minute over 1-hour windows. If exceeded, investigate.
• Rejection SLO: < 0.1% of requests rejected for capacity. If exceeded, scale or shed lower-priority traffic.

Tracking SLOs across the fleet¶

SLO tracking is done in tools like Sloth, Pyrra, or custom dashboards. The output is a real-time view of "how close are we to the budget" for each service.

Cross-Team Concurrency Contracts¶

A concurrency contract is an explicit agreement between two teams: caller and callee. The contract specifies:

• Max RPS the callee accepts.
• Max in-flight the callee accepts.
• Per-caller quota.
• Burst tolerance.
• Degradation policy.
• Notification SLA for capacity changes.

Why explicit contracts¶

Without them, capacity is implicit and assumed. The caller assumes "the callee can handle anything." The callee assumes "the caller is well-behaved." When either assumption breaks, the system fails and there is no party at fault.

With them, capacity is owned. The callee owns the publishing; the caller owns adhering.

Contract template¶

SERVICE: catalog-api
VERSION: v3.2

CAPACITY
  Max RPS: 5000
  Max in-flight: 1000
  Burst tolerance: 30s

PER-CALLER QUOTAS
  default: 100 RPS, 20 in-flight
  payments: 500 RPS, 50 in-flight (premium tier)
  search: 1000 RPS, 100 in-flight (premium tier)

DEGRADATION
  At RPS > 5000: return 503 with Retry-After: 1
  At in-flight > 1000: return 503 with Retry-After: 5
  Per-caller exceed: return 429 with Retry-After (jittered)

ENDPOINTS
  /v1/catalog/get/{id}: GET, low cost
  /v1/catalog/list: GET, medium cost
  /v1/catalog/bulk: POST, high cost, max batch 1000
  /v1/catalog/import: POST, very high cost, max batch 100, async only

CHANGE POLICY
  Capacity changes announced 2 weeks in advance.
  Per-caller quota changes announced 1 week in advance.
  Breaking changes announced 6 months in advance.

This contract is published in the team's docs. Every caller team reads and acknowledges. Annual reviews update.

Enforcement¶

Both sides enforce: - Caller-side: HTTP client wrapped with rate limiter and semaphore matching the quota. - Callee-side: admission middleware that rejects above the published numbers.

The two sides should be aligned. If the caller side is missing or under-configured, the callee's load-shedding catches it; if the callee side is missing, the caller's rate limiter prevents misbehaviour.

Contract violations¶

When violated: 1. The caller's metrics show 429s/503s. 2. The callee's metrics show rejection rate climbing. 3. An incident is triggered (PagerDuty alert). 4. The teams discuss: is the contract wrong, or is the implementation wrong?

This is the productive form of inter-team friction. The contract creates a shared vocabulary; violations create concrete discussion points.

Cost Modelling: Goroutines and the Cloud Bill¶

Goroutines are not free, and at scale, their cost shows up in the cloud bill. Cost modelling is the practice of attributing infrastructure cost to the goroutines that consume it.

Goroutine cost in dollars¶

Approximate per-goroutine cost: - Stack memory: 4-16 KB amortised. At 16 KB and $0.005/GB-hour (AWS r5.large EBS-only): ~$0.07 per million goroutine-hours. - CPU: a goroutine doing 100 ms work per second of wall time consumes 10% of a CPU. On a $0.10/hour CPU, that's $0.01/hour. - I/O: each connection consumes a TCP stream (~1 KB kernel state) + downstream bandwidth.

These numbers are tiny per goroutine but multiply by 1 million goroutines × 24 hours: ~$1.70 for memory, $240 for CPU.

Cost of a single OOM¶

When a pod OOMs: - Pod restart: ~30 seconds of unavailable capacity. - Pending request drops: lost revenue (if the service is revenue-generating). - Cascading restarts: other pods absorb the load and OOM in turn.

A 5-minute partial outage of a moderate-traffic SaaS can be $10,000-$100,000 in lost revenue and reputation. The cost of not bounding is enormous.

Cost-aware sizing¶

When choosing the bound:

• Lower bound = less in-flight = smaller pods = lower bill.
• Higher bound = more in-flight = larger pods = higher bill.
• Optimum bound = highest throughput per dollar.

Run a load test at different bounds. Compute throughput / cost. Pick the bound that maximises it.

Example: at bound 50, pod size = 1 GB RAM, throughput = 100 RPS, pod cost = $20/month. Cost per RPS = $0.20.

At bound 200, pod size = 4 GB RAM, throughput = 300 RPS, pod cost = $80. Cost per RPS = $0.27.

The smaller bound is cheaper per request. Use it unless throughput needs override (e.g. you're under-provisioned and need to scale up fast).

Memory savings from bounded fan-out¶

Suppose a service unbounded-fan-outs to 10 000 goroutines at peak. Each goroutine consumes (let's say) 50 KB heap + 8 KB stack = 58 KB. Total: 580 MB at peak.

Bounded to 100 goroutines: 5.8 MB at peak.

The bounded version fits in a 64 MB pod; the unbounded version needs a 1 GB pod. Pod cost difference: ~16x. For a fleet of 100 pods, that is a real dollar amount.

TCO of unbounded code¶

The total cost of ownership of a single unbounded fan-out site, over its lifetime, includes:

• Storage (logs, profiles).
• Bandwidth (occasional spikes).
• On-call response time (engineer-hours).
• Incident postmortem time.
• Lost revenue during outages.
• Reputation damage.

The lifetime cost easily reaches $100,000-$1,000,000 for a high-traffic service. The fix is 1 line of code. The cost-benefit is laughably favourable.

Multi-Tenant Architectures¶

In a multi-tenant SaaS, one customer's load must not affect another's. Tenant isolation is the bound stack made tenant-aware.

Tenant isolation patterns¶

1. Shared-everything: all tenants share the same process, pool, semaphores. Cheapest. Tenant load can cross-affect.
2. Per-tenant pools: each tenant has its own pool. Better isolation. Memory cost proportional to tenant count.
3. Tenant-scoped semaphores: shared pool, per-tenant semaphore for fairness. Good middle ground.
4. Dedicated pods: one tenant per pod. Best isolation. Expensive at scale.
5. Tiered isolation: premium tenants on dedicated pods, free-tier shares. Cost-effective.

Most SaaS uses pattern 3 or 5.

Pattern 3 implementation¶

type TenantBudget struct {
    mu      sync.RWMutex
    sems    map[string]*semaphore.Weighted
    limit   int64
    sweep   *time.Ticker
}

func New(limit int64) *TenantBudget {
    tb := &TenantBudget{
        sems:  make(map[string]*semaphore.Weighted),
        limit: limit,
        sweep: time.NewTicker(time.Hour),
    }
    go tb.sweeper()
    return tb
}

func (tb *TenantBudget) sem(tenant string) *semaphore.Weighted {
    tb.mu.RLock()
    s, ok := tb.sems[tenant]
    tb.mu.RUnlock()
    if ok { return s }
    tb.mu.Lock()
    defer tb.mu.Unlock()
    if s, ok := tb.sems[tenant]; ok { return s }
    s = semaphore.NewWeighted(tb.limit)
    tb.sems[tenant] = s
    return s
}

func (tb *TenantBudget) Acquire(ctx context.Context, tenant string) error {
    return tb.sem(tenant).Acquire(ctx, 1)
}

func (tb *TenantBudget) Release(tenant string) {
    tb.sem(tenant).Release(1)
}

func (tb *TenantBudget) sweeper() {
    for range tb.sweep.C {
        tb.mu.Lock()
        for k, s := range tb.sems {
            if !s.TryAcquire(tb.limit) {
                continue // someone is holding
            }
            s.Release(tb.limit)
            delete(tb.sems, k)
        }
        tb.mu.Unlock()
    }
}

The sweeper periodically removes idle tenants from the map. The TryAcquire(limit) is a probe: if all tokens are free, the tenant is idle.

Per-tenant SLOs¶

Some tenants have premium SLOs; their requests get priority. Implement via priority queues:

type Job struct {
    Tenant string
    Pri    Priority
    Fn     func()
}

type PriorityPool struct {
    high *list.List
    norm *list.List
    low  *list.List
    mu   sync.Mutex
    cv   *sync.Cond
}

func (pp *PriorityPool) Submit(j Job) {
    pp.mu.Lock()
    defer pp.mu.Unlock()
    switch j.Pri {
    case High: pp.high.PushBack(j)
    case Normal: pp.norm.PushBack(j)
    case Low: pp.low.PushBack(j)
    }
    pp.cv.Signal()
}

func (pp *PriorityPool) take() Job {
    pp.mu.Lock()
    defer pp.mu.Unlock()
    for {
        if pp.high.Len() > 0 { return pp.pop(pp.high) }
        if pp.norm.Len() > 0 { return pp.pop(pp.norm) }
        if pp.low.Len() > 0 { return pp.pop(pp.low) }
        pp.cv.Wait()
    }
}

func (pp *PriorityPool) pop(l *list.List) Job {
    e := l.Front()
    l.Remove(e)
    return e.Value.(Job)
}

High-priority jobs are taken before normal, which are taken before low. Starvation is possible (low never runs if high is constant); add an anti-starvation mechanism (e.g. every 10th take is forced to be low).

Tenant fairness¶

Fair-share scheduling: each tenant gets an equal share of the budget regardless of how many requests they submit. Implement via a per-tenant queue and round-robin take:

type FairPool struct {
    queues  map[string]*list.List
    tenants []string
    cur     int
    mu      sync.Mutex
}

func (fp *FairPool) take() Job {
    fp.mu.Lock()
    defer fp.mu.Unlock()
    for i := 0; i < len(fp.tenants); i++ {
        fp.cur = (fp.cur + 1) % len(fp.tenants)
        tn := fp.tenants[fp.cur]
        q := fp.queues[tn]
        if q.Len() > 0 {
            e := q.Front()
            q.Remove(e)
            return e.Value.(Job)
        }
    }
    panic("no jobs but take called") // caller should wait
}

Round-robin takes one job from each tenant in turn. A tenant with 1000 queued jobs gets the same throughput as a tenant with 1.

Trade-offs¶

• Round-robin is fair but cache-unfriendly (jumps tenants on each take).
• Per-tenant pools are isolated but waste memory for idle tenants.
• Shared pool with semaphores is the cheapest fair approach.

Pick based on the tenant distribution (few large vs many small) and the cost of cross-tenant impact.

Load Testing for Bounded Systems¶

A bounded system must be load-tested for two things:

1. Throughput at bound: at what RPS does the bound start admitting at capacity?
2. Behaviour past bound: what happens when load exceeds capacity?

Test 1: Find the bound's capacity¶

Drive RPS upward gradually. Measure: - Success rate. - Latency p50/p99. - In-flight count.

Plot RPS vs latency. You will see latency stable, then a knee where it climbs sharply. The knee is your bound's capacity. Operate at 70% of it.

Test 2: Test load shedding¶

Drive RPS to 2x capacity. Measure: - Rejection rate (% of requests returned 503). - Latency for admitted requests. - Are admitted requests still successful?

A correctly-bounded system shows rejection rate climbing while admitted latency stays bounded. A misconfigured system shows latency climbing toward infinity with no rejections — meaning the bound is being violated (work is being admitted past capacity).

Test 3: Test recovery¶

Drive RPS to 5x capacity for 30 seconds, then drop back to 50%. Measure: - How quickly does latency return to normal? - Are there leftover queued items that take minutes to drain? - Are any goroutines still in-flight after the load drops?

A healthy system recovers in seconds. A broken system has minutes of residual latency or goroutine count.

Test tools¶

• wrk for HTTP-only load.
• vegeta for HTTP with detailed metrics.
• k6 for scripted scenarios.
• Locust for distributed Python-based.
• Custom Go programs for protocol-specific (gRPC, websockets).

Always run against a staging environment that matches production. Load testing against prod is dangerous (you can cause an outage).

Continuous load testing¶

Run a small load test every commit (a smoke test). Run a full load test weekly. The full load test catches regressions in throughput or latency.

# example .github/workflows/load.yml
on:
  schedule:
    - cron: "0 0 * * 0" # weekly
jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - run: docker-compose up -d
      - run: k6 run --vus 100 --duration 10m loadtest.js
      - run: docker-compose logs > logs.txt
      - uses: actions/upload-artifact@v3
        with: { name: load-results, path: logs.txt }

The artifact is the report. Compare week-over-week.

Chaos Engineering for Concurrency¶

Chaos engineering: deliberately inject failures to validate the system handles them. For concurrency, the failures are:

• Goroutine count spike (start 10 000 sleep-goroutines).
• Memory pressure (allocate 80% of pod memory).
• Downstream slowness (proxy with artificial delay).
• Downstream failure (proxy returning 500).
• Network partition (drop packets between services).

A chaos library¶

package chaos

import (
    "context"
    "math/rand"
    "time"
)

type Config struct {
    Enabled bool
    Probability float64
    DelayMin, DelayMax time.Duration
}

func Inject(ctx context.Context, cfg Config) {
    if !cfg.Enabled || rand.Float64() > cfg.Probability {
        return
    }
    delay := cfg.DelayMin + time.Duration(rand.Int63n(int64(cfg.DelayMax-cfg.DelayMin)))
    select {
    case <-ctx.Done():
    case <-time.After(delay):
    }
}

Wrap downstream calls with Inject. In a chaos run, every call has a probability of artificial slowness.

Validating bounds under chaos¶

Run the chaos for 10 minutes while monitoring: - Goroutine count: should stay near baseline. - Success rate: may dip but recover. - Rejection rate: may climb but not 100%.

If the goroutine count spikes during chaos, your bound isn't working — the artificial slowness is causing fan-out to accumulate.

Game days¶

Quarterly, run a "game day": a planned chaos event in a staging environment, with the on-call team responding as if it were real. Train both the system and the operators.

Static Analysis and Lint Rules¶

The "unbounded fan-out" pattern is statically detectable in many cases. A linter can catch new instances before they merge.

A starter analyser¶

package gofor

import (
    "go/ast"
    "go/types"

    "golang.org/x/tools/go/analysis"
    "golang.org/x/tools/go/analysis/passes/inspect"
    "golang.org/x/tools/go/ast/inspector"
)

var Analyzer = &analysis.Analyzer{
    Name:     "gofor",
    Doc:      "checks for `go` inside `for` over slices/channels without a bound",
    Requires: []*analysis.Analyzer{inspect.Analyzer},
    Run:      run,
}

func run(pass *analysis.Pass) (interface{}, error) {
    insp := pass.ResultOf[inspect.Analyzer].(*inspector.Inspector)
    insp.Preorder([]ast.Node{(*ast.RangeStmt)(nil)}, func(n ast.Node) {
        rs := n.(*ast.RangeStmt)
        if !hasGoInside(rs.Body) {
            return
        }
        if isBoundedRange(rs.X, pass.TypesInfo) {
            return
        }
        pass.Reportf(rs.Pos(), "unbounded go inside for range (likely fan-out anti-pattern)")
    })
    return nil, nil
}

func hasGoInside(b *ast.BlockStmt) bool {
    found := false
    ast.Inspect(b, func(n ast.Node) bool {
        if _, ok := n.(*ast.GoStmt); ok {
            found = true
            return false
        }
        return true
    })
    return found
}

func isBoundedRange(expr ast.Expr, info *types.Info) bool {
    // Bounded if expression is a literal with a known small upper bound,
    // or an array (not a slice) with fixed size.
    t := info.TypeOf(expr)
    _, isArr := t.(*types.Array)
    return isArr
}

The linter flags any go inside a for range over a slice or channel. Arrays (fixed-size) are excluded. The user can suppress with a comment if intentional:

//nolint:gofor // bounded by len(backends) which is a fixed 8
for _, b := range backends {
    go b.Init()
}

Integrating with golangci-lint¶

Add to .golangci.yml:

linters:
  enable:
    - gofor

Run on every PR. The PR fails if any new unbounded fan-out is introduced.

Beyond static analysis¶

Some patterns are not statically detectable. For instance:

func processAll(items []Item) {
    for _, it := range items {
        go process(it)
    }
}

Here, the linter can flag. But:

func processAll(items []Item) {
    go doProcessing(items)
}

func doProcessing(items []Item) {
    for _, it := range items {
        go process(it)
    }
}

The linter still flags doProcessing's loop. Good.

But:

func processAll(items []Item) {
    for _, it := range items {
        spawn(process, it)
    }
}

func spawn(fn func(Item), it Item) {
    go fn(it)
}

The go is in spawn, not in processAll's loop directly. The linter would need a cross-function analysis. This is a known gap.

For such cases, code review must catch them. The linter is a safety net, not a complete defence.

Migration Strategies for Large Codebases¶

You inherit a 500 000-line Go codebase. Hundreds of fan-out sites, many unbounded. You cannot rewrite it all at once. What is the migration?

Phase 1: Audit and Inventory¶

Catalog every go statement, classified: - Safe (input is statically bounded). - Risky (input is dynamically bounded). - Dangerous (input is unbounded, user-controlled, or stream-based).

Tools: grep + manual triage + automated lint output.

A spreadsheet with file, line, function, classification, estimated effort. Hundreds of rows is normal.

Phase 2: High-risk first¶

Sort by risk. Top 10% by risk → fix first. The fix: 1. Identify the bound. 2. Choose the cure (errgroup, semaphore, pool). 3. Add a test that fails if unbounded reappears. 4. Code review. 5. Deploy with monitoring.

A team can fix ~5 sites per week. 50 sites → 10 weeks for the top decile.

Phase 3: Lint to prevent regression¶

Once the worst sites are fixed, add the lint to CI. New unbounded fan-outs are blocked at PR time. Existing ones (still in lower-risk tiers) are suppressed with a comment and a TODO.

Phase 4: Burn down the TODOs¶

Each sprint, fix some of the TODOs. Track progress. Aim for 100% in 6-12 months.

Phase 5: Verify¶

After all known sites are fixed, run a stress test of the entire codebase. Inject pathological inputs (millions of items, gigabytes of data). Confirm no service OOMs.

Phase 6: Maintain¶

The lint remains. Code review remains. New engineers receive training on the anti-pattern. The codebase has built immunity.

Estimated timeline¶

For 500 000 lines of Go with ~500 unbounded sites, expect: - Phase 1: 1-2 months. - Phase 2-3: 3-6 months. - Phase 4: 6-12 months. - Phase 5-6: ongoing.

The investment is substantial but pays off in reduced incidents.

Production Case Studies¶

Case Study A: The Backplane Migration¶

Background. A SaaS company had a "backplane" service that fanned out customer events to 12 internal subscribers. The fan-out was per-event:

for _, sub := range subscribers {
    go sub.Deliver(event)
}

12 was hard-coded; the loop was safe in isolation. But the backplane handled 50 000 events/s, and subscribers was a snapshot of a registry; sometimes during deploys it briefly had 30 entries (old + new). Then 50 000 × 30 = 1 500 000 goroutines/s spawned.

Fix. Convert to a per-subscriber bounded pool. Each subscriber has its own goroutine and an input channel of capacity 1024. The backplane pushes events to channels; subscribers pull at their own pace.

Outcome. Memory at peak dropped from 8 GB to 800 MB. Latency p99 dropped because there was no more scheduler thrash.

Case Study B: The Polling Loop¶

Background. A monitoring service polled 5000 endpoints every 30 seconds. Naive implementation:

ticker := time.NewTicker(30 * time.Second)
for range ticker.C {
    for _, endpoint := range endpoints {
        go poll(endpoint)
    }
}

5000 goroutines every 30 seconds. If a poll took 28 seconds (some did), the next tick's goroutines stacked on top.

Fix. Bounded errgroup, polling time bounded by the tick interval:

for range ticker.C {
    g, gctx := errgroup.WithContext(ctx)
    g.SetLimit(200)
    pollCtx, cancel := context.WithTimeout(gctx, 25*time.Second)
    for _, e := range endpoints {
        e := e
        g.Go(func() error { return poll(pollCtx, e) })
    }
    g.Wait()
    cancel()
}

Now each tick's polling is bounded in time and concurrency. Slow endpoints time out at 25s, not stack indefinitely.

Outcome. Service became stable. Previously, hot endpoints would silently accumulate goroutines, leading to weekly OOM-driven restarts. After the fix, restart frequency dropped to once a month (for unrelated reasons).

Case Study C: The Cron Storm¶

Background. A scheduled job ran daily at midnight UTC, computing reports for all customers. Each customer's report was independent:

func runReports() {
    for _, customer := range customers {
        go generateReport(customer)
    }
}

10 000 customers. Each report took 5-30 seconds. At midnight, the service spawned 10 000 goroutines, each opening a database connection.

Fix. Bounded pool sized to database connection limit:

func runReports() {
    g, gctx := errgroup.WithContext(context.Background())
    g.SetLimit(50) // matches DB pool
    for _, customer := range customers {
        customer := customer
        g.Go(func() error {
            return generateReport(gctx, customer)
        })
    }
    _ = g.Wait()
}

The job takes longer (sequentially through batches) but doesn't crash the database.

Outcome. Database CPU at midnight dropped from 100% (with errors) to 60% (steady-state). Job duration extended from "crashes and partial" to "completes in 25 minutes."

Case Study D: The Goroutine-per-Connection Database¶

Background. A custom database server in Go used one goroutine per connection. Worked fine up to 10 000 concurrent connections, beyond which scheduler thrash dominated.

Fix. Hybrid model: a fixed pool of worker goroutines plus an event loop using net.Listener directly. Connections register interest; workers process events.

Outcome. Capacity climbed from 10 000 to 100 000 concurrent connections per node. Latency variance dropped because the scheduler was no longer overloaded.

Lesson. "Goroutine per connection" is fine up to tens of thousands. Beyond that, an event-loop architecture is more efficient.

Case Study E: The Hidden Library Fan-Out¶

Background. A service used a custom logging library. Each log call spawned a goroutine to ship the log line to a remote aggregator. Under heavy logging (debug mode), the goroutine count exploded.

Fix. Logging library refactored: a single shipper goroutine drains a channel. Log calls just enqueue; shipper batches.

Outcome. Debug-mode logging now had similar resource use to info-mode. No more goroutine explosions from chatty code.

Lesson. Library-level fan-out is hidden from callers but compounds when the library is used heavily. Audit libraries for hidden goroutine spawning.

The Long View: Engineering Discipline¶

The unlimited-goroutines anti-pattern is not merely a Go quirk; it is a microcosm of a broader engineering discipline. The discipline:

1. Make implicit assumptions explicit. Every go is an assumption that the goroutine count is bounded by something. Make the bound visible.

2. Choose numbers deliberately. "32 workers because the database has 50 connections" is an answer. "32 workers because it felt right" is not.

3. Measure before deciding. Bounds derived from load tests are correct. Bounds derived from intuition are guesses.

4. Test the failure mode. A test that passes when bounds are respected is necessary; a test that fails when bounds are violated is essential.

5. Monitor in production. Tests catch known failure modes. Monitoring catches unknown ones.

6. Plan for the long tail. Most days, your service runs at 1% capacity. Plan for the 99th-percentile day.

7. Iterate on the bound. Bounds set at launch are wrong; bounds tuned in production are correct.

8. Document the rationale. Every bound should have a comment, a load test, a runbook entry. Future engineers should not have to guess.

9. Build immunity. Linters, code review checklists, training, postmortems — all build organisational immunity to the anti-pattern.

10. Treat reliability as a feature. Bounding is not "ops work" or "perfectionism." It is what makes the service reliable. Reliability is a feature.

These are the habits that distinguish a professional from a senior. The professional sees the entire system; the senior sees the code in front of them. Both are necessary; the professional level is the next step.

Self-Assessment¶

Before claiming professional-level competence, answer:

1. Draw the bound stack for a service you operate. Identify the tightest tier. What metric would tell you that tier is saturated?

2. A new service joins your microservice graph. Write its concurrency contract.

3. Your service handles 200 RPS at p99 = 300ms. What is the in-flight bound? What is the pod count for 1000 RPS at the same latency? Show the math.

4. Design a per-tenant concurrency cap with a 1-hour idle timeout. Sketch the data structures.

5. Configure HPA to scale on goroutine count instead of CPU. Write the YAML.

6. Your fleet has 50 pods. NumGoroutine on pod-27 is steadily climbing at 200/min while other pods are flat. How do you diagnose the cause? List 5 specific commands or actions.

7. Design a chaos test for unbounded fan-out. What do you inject; what do you observe; what does success look like?

8. Write a postmortem for the case study A in this document (the Backplane Migration). Include timeline, root cause, action items.

9. You have 500 unbounded fan-out sites across 50 services. Sketch a 6-month migration plan with milestones.

10. Define a concurrency SLO for your service. Justify the chosen value.

These are the questions a staff engineer might ask in an interview or design review. If you can answer them with concrete code, math, and trade-offs, you are operating at the professional level.

Summary¶

• The professional view is architectural: bounds are a property of the system, not a single service.
• Concurrency budgets are allocated across services, tenants, and tiers. Each is owned and monitored.
• Capacity planning is arithmetic: Little's Law, peak load, headroom, pod sizing. No guesswork.
• Kubernetes pod limits constrain in-pod concurrency. GOMAXPROCS must match the CPU limit (use automaxprocs).
• Graceful degradation requires explicit strategies: cache, partial, static, prioritised, async. Choose per endpoint.
• Circuit breakers prevent goroutine accumulation during downstream failure. Combine with bounded fan-out.
• Observability is fleet-level: process_goroutines is the single most important metric.
• pprof goroutine profiles in production diagnose the unknown. Continuous profiling is the long-term tool.
• Postmortems for concurrency incidents follow a template. The action items are concrete and time-bound.
• Concurrency SLOs commit to specific bounds. Error budgets enforce them.
• Cross-team contracts make capacity explicit. Both sides enforce.
• Cost is real: bounded fan-out saves money. Unbounded fan-out costs incidents, hours, and revenue.
• Multi-tenant architectures isolate tenants via per-tenant semaphores, priority queues, or dedicated pods.
• Load tests must validate behaviour past the bound, not just at the bound.
• Chaos engineering validates bounds under adversarial conditions.
• Static analysis catches new violations; code review catches the rest.
• Migration of a large codebase is a multi-month project, ordered by risk.
• The discipline is broader than Go: making assumptions explicit, measuring, testing, monitoring.

If you embody these principles, you are no longer thinking about "the unlimited-goroutines anti-pattern." You are thinking about a system whose every part is bounded, monitored, and accountable.

Appendix A: A Complete Production-Grade Service¶

This is a worked example of a production service with all the patterns from this document.

A.1 — Service structure¶

package server

import (
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "log/slog"
    "net/http"
    "runtime"
    "sync"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "github.com/sony/gobreaker"
    "go.uber.org/automaxprocs/maxprocs"
    "golang.org/x/sync/errgroup"
    "golang.org/x/sync/semaphore"
)

type Config struct {
    Addr             string
    PerRequestFanout int
    GlobalInflight   int64
    PerTenantInflight int64
    BackendInflight   int64
}

type Server struct {
    cfg          Config
    global       *semaphore.Weighted
    tenantBudget *TenantBudget
    backendSem   *semaphore.Weighted
    breaker      *gobreaker.CircuitBreaker
    log          *slog.Logger
}

func New(cfg Config, log *slog.Logger) *Server {
    return &Server{
        cfg:          cfg,
        global:       semaphore.NewWeighted(cfg.GlobalInflight),
        tenantBudget: NewTenantBudget(cfg.PerTenantInflight),
        backendSem:   semaphore.NewWeighted(cfg.BackendInflight),
        breaker:      newBreaker(),
        log:          log,
    }
}

func newBreaker() *gobreaker.CircuitBreaker {
    return gobreaker.NewCircuitBreaker(gobreaker.Settings{
        Name:        "backend",
        MaxRequests: 1,
        Interval:    60 * time.Second,
        Timeout:     30 * time.Second,
    })
}

// ... continued

A.2 — Metrics¶

var (
    requestTotal = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "request_total",
    }, []string{"endpoint", "outcome"})
    requestDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name: "request_duration_seconds",
    }, []string{"endpoint"})
    inflightGauge = promauto.NewGaugeFunc(prometheus.GaugeOpts{
        Name: "process_inflight",
    }, func() float64 {
        // would need a counter; omitted for brevity
        return 0
    })
    goroutineGauge = promauto.NewGaugeFunc(prometheus.GaugeOpts{
        Name: "process_goroutines",
    }, func() float64 { return float64(runtime.NumGoroutine()) })
)

func init() {
    if _, err := maxprocs.Set(); err != nil {
        slog.Default().Warn("automaxprocs", "error", err)
    }
}

A.3 — Admission and routing¶

func (s *Server) admit(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
        defer cancel()
        if !s.global.TryAcquire(1) {
            w.Header().Set("Retry-After", "1")
            http.Error(w, "overloaded", http.StatusServiceUnavailable)
            requestTotal.WithLabelValues("admit", "rejected").Inc()
            return
        }
        defer s.global.Release(1)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

func (s *Server) Routes() http.Handler {
    mux := http.NewServeMux()
    mux.Handle("/metrics", promhttp.Handler())
    mux.Handle("/v1/aggregate", s.admit(http.HandlerFunc(s.Aggregate)))
    return mux
}

A.4 — The handler¶

func (s *Server) Aggregate(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    defer requestDuration.WithLabelValues("aggregate").Observe(time.Since(start).Seconds())
    tenant := r.Header.Get("X-Tenant-ID")
    if tenant == "" {
        http.Error(w, "missing tenant", http.StatusBadRequest)
        requestTotal.WithLabelValues("aggregate", "bad_request").Inc()
        return
    }

    var req struct {
        Queries []string `json:"queries"`
    }
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        requestTotal.WithLabelValues("aggregate", "bad_request").Inc()
        return
    }
    if len(req.Queries) > 100 {
        http.Error(w, "too many queries", http.StatusRequestEntityTooLarge)
        requestTotal.WithLabelValues("aggregate", "too_large").Inc()
        return
    }
    if err := s.tenantBudget.Acquire(r.Context(), tenant); err != nil {
        http.Error(w, "tenant overloaded", http.StatusTooManyRequests)
        requestTotal.WithLabelValues("aggregate", "tenant_rate_limit").Inc()
        return
    }
    defer s.tenantBudget.Release(tenant)

    g, gctx := errgroup.WithContext(r.Context())
    g.SetLimit(s.cfg.PerRequestFanout)

    results := make([]string, len(req.Queries))
    var mu sync.Mutex
    for i, q := range req.Queries {
        i, q := i, q
        g.Go(func() error {
            res, err := s.callBackend(gctx, q)
            if err != nil { return err }
            mu.Lock()
            results[i] = res
            mu.Unlock()
            return nil
        })
    }
    if err := g.Wait(); err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        requestTotal.WithLabelValues("aggregate", "error").Inc()
        return
    }
    _ = json.NewEncoder(w).Encode(map[string]any{"results": results})
    requestTotal.WithLabelValues("aggregate", "ok").Inc()
}

func (s *Server) callBackend(ctx context.Context, q string) (string, error) {
    if err := s.backendSem.Acquire(ctx, 1); err != nil { return "", err }
    defer s.backendSem.Release(1)
    res, err := s.breaker.Execute(func() (interface{}, error) {
        return backendCall(ctx, q)
    })
    if err != nil {
        if errors.Is(err, gobreaker.ErrOpenState) {
            return "", fmt.Errorf("backend unavailable")
        }
        return "", err
    }
    return res.(string), nil
}

func backendCall(ctx context.Context, q string) (string, error) {
    select {
    case <-ctx.Done():
        return "", ctx.Err()
    case <-time.After(50 * time.Millisecond):
        return "ok:" + q, nil
    }
}