GOMAXPROCS — Find the Bug¶

Each puzzle below shows a snippet with a performance bug related to GOMAXPROCS tuning. Diagnose, then check your answer. Difficulty rises gradually. These are performance bugs, not correctness bugs — the programs all compile and produce correct output; they just perform badly.

Bug 1 — The Hard-Coded Constant¶

package main

import (
    "log"
    "net/http"
    "runtime"
)

func init() {
    runtime.GOMAXPROCS(4)
}

func handler(w http.ResponseWriter, r *http.Request) {
    _, _ = w.Write([]byte("ok"))
}

func main() {
    log.Printf("GOMAXPROCS=%d", runtime.GOMAXPROCS(0))
    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Symptom. Service runs on 32-core nodes; CPU utilisation never exceeds 4 cores; throughput is capped.

Bug 2 — The "Headroom" Mistake¶

package main

import (
    "log"
    "runtime"
)

func main() {
    n := runtime.NumCPU() * 4
    runtime.GOMAXPROCS(n)
    log.Printf("GOMAXPROCS=%d for headroom", n)
    // ... service
}

Symptom. p99 latency is 20% worse than on a similar service that does not call GOMAXPROCS.

Bug 3 — The I/O Multiplier¶

package main

import (
    "log"
    "runtime"
)

func main() {
    // "Our service is I/O bound, so we need more parallelism for blocking calls."
    runtime.GOMAXPROCS(runtime.NumCPU() * 2)
    log.Printf("GOMAXPROCS=%d (I/O service)", runtime.GOMAXPROCS(0))
    // ... HTTP server with downstream calls
}

Symptom. No measurable improvement over GOMAXPROCS=NumCPU; in fact slight p99 regression.

Bug 4 — The Container Trap¶

// Go 1.15 service running in a pod with cpu: 2 on a 64-core host.

package main

import (
    "log"
    "runtime"
)

func main() {
    log.Printf("GOMAXPROCS=%d NumCPU=%d", runtime.GOMAXPROCS(0), runtime.NumCPU())
    // ... CPU-bound HTTP service
}

Symptom. Startup log shows GOMAXPROCS=64 NumCPU=64. cAdvisor shows constant CFS throttling. p99 is 50× the expected baseline.

Bug 5 — The Forgotten Debug Override¶

package main

import (
    "log"
    "net/http"
    "runtime"
)

func main() {
    runtime.GOMAXPROCS(1) // TODO: remove this, just for race debugging
    log.Printf("GOMAXPROCS=%d", runtime.GOMAXPROCS(0))
    http.HandleFunc("/", handleAll)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

func handleAll(w http.ResponseWriter, r *http.Request) { /* ... */ }

Symptom. Service deployed to production; throughput is 1/16th of expected.

Bug 6 — The Sweep on the Same Machine¶

A junior engineer runs:

for n in 1 2 4 8 16; do
    GOMAXPROCS=$n ./service &
    pid=$!
    sleep 1
    wrk -t8 -c64 -d10s http://localhost:8080/
    kill $pid
done

Symptom. Throughput rises until GOMAXPROCS=4, then drops dramatically. Engineer concludes the optimum is 4.

Bug 7 — The Single-Run Conclusion¶

Engineer reports: "I ran the sweep. GOMAXPROCS=8 gives 102 k RPS, GOMAXPROCS=12 gives 100 k RPS. So 8 is better. Closing ticket."

Bug 8 — The Sidecar Surprise¶

A pod has two containers: main (the Go service) and proxy (Envoy sidecar). Both at cpu: 4. Service runs at GOMAXPROCS=4. Envoy uses 1 core on average.

Symptom. Node-level CPU is at 90%; many similar pods on the node throttle; debugging shows total cgroup usage of main + proxy per pod is ~5 cores against ~4 cores of quota.

Bug 9 — The NUMA Catastrophe¶

A latency-critical Go service runs on a dual-socket 32-core-per-socket bare-metal server. GOMAXPROCS=64. p99 is 18 ms; target 8 ms. Engineer raises GOMAXPROCS=128 thinking "more parallelism".

Symptom. p99 spikes to 35 ms; throughput unchanged.

Bug 10 — The Dynamic-Tuning Oscillation¶

A team builds an in-process auto-tuner: when p99 exceeds the SLO, call runtime.GOMAXPROCS(current - 1); when p99 is well below, call runtime.GOMAXPROCS(current + 1). Adjustment runs every 5 seconds.

Symptom. Service flips between GOMAXPROCS=7 and GOMAXPROCS=9 every 5–10 seconds. p99 has new sawtooth pattern. Throughput is below baseline.

Bug 11 — The Friday Spike¶

A latency-critical service runs at p99 = 6 ms Mon–Thu. Every Friday afternoon at 14:00, p99 jumps to 80 ms for ~20 minutes.

Bug 12 — The Mismatched GOMAXPROCS and GOMEMLIMIT¶

A service has GOMAXPROCS=16 (cgroup quota correctly detected) and no GOMEMLIMIT. Container memory limit is 2Gi. Heap grows to 1.9 GiB, the kernel OOM-kills the process.

Bug 13 — The Misleading CPU Graph¶

Operator sees a service running at 30% CPU utilisation. "We have plenty of headroom — let's add more traffic." Adding traffic causes p99 to skyrocket.

Bug 14 — The "Less Is More" That Was Right¶

A team had GOMAXPROCS=NumCPU=32. They tried GOMAXPROCS=24 and p99 dropped from 12 ms to 7 ms; throughput dropped 10%. They reverted because "throughput matters more". Now p99 is back to 12 ms and they cannot understand why the latency win does not transfer to production.

Bug 15 — The Per-Tenant GOMAXPROCS Request¶

Product asks: "Tenant A is paying 10× more than tenant B. Can we give A more CPU?" Engineering writes a wrapper that calls runtime.GOMAXPROCS(16) before tenant A's requests and runtime.GOMAXPROCS(8) before tenant B's.

Symptom. Both tenants see catastrophic p99 spikes. Audit logs show procresize happening hundreds of times per second.

Bug 16 — The Failing CI Sweep¶

The team set up a nightly GOMAXPROCS sweep in CI. For two weeks it produced consistent numbers. Today it shows wildly different results — GOMAXPROCS=8 is 30% better than yesterday's GOMAXPROCS=8.

Self-Assessment¶

You should be able to diagnose Bugs 1–8 in under five minutes each at junior–middle level, Bugs 9–14 at senior level, and Bugs 15–16 at professional level. If you found yourself surprised, re-read the relevant level's file.

Cross-reference: the planned 10-scheduler-deep-dive/03-gomaxprocs-tuning/find-bug.md chapter (coming soon) for mechanics-side bugs that complement these performance-focused ones.