Premature Concurrency Optimization — Senior Level¶

Table of Contents¶

1. Introduction: the senior shift
2. When concurrency hurts
3. Sequential is faster than you think
4. Amdahl's law for goroutines
5. Gustafson, Karp-Flatt, and the practical companions to Amdahl
6. Context-switch costs
7. Scheduler overhead in detail
8. False sharing
9. Cache lines, NUMA, and memory traffic
10. Measurement-first methodology
11. Designing the right experiment
12. Reviewing a "make it concurrent" PR
13. Sequential vs parallel decision tree
14. Latency vs throughput at the design table
15. The "shape" of the workload
16. Cost models you can carry in your head
17. Coordination overhead
18. Why parallelizing tiny tasks loses
19. Channel and mutex hidden costs
20. Common premature optimisations and their refutations
21. Building a culture of measurement
22. Organisational norms
23. Worked example: a CSV pipeline that got slower
24. Worked example: image thumbnailer
25. Worked example: HTTP request fan-out
26. Worked example: a hot counter
27. Worked example: parallel map
28. Reading a CPU profile after a concurrency change
29. Reading a trace after a concurrency change
30. Statistical significance for benchmarks
31. GOMAXPROCS at the senior level
32. GC interactions with goroutine count
33. Self-assessment
34. Summary

Introduction: the senior shift¶

At the junior level the rule was "write the simple version, measure, then optimise." At middle level you applied the rule inside an existing codebase. At senior level the rule becomes a responsibility: you own a body of code that other engineers will modify, and your job is to make sure their first instinct is not "let me add a goroutine" but "let me measure first."

This document is about three things in roughly equal measure:

1. The physics. Why concurrency frequently adds cost — context switches, false sharing, cache-line bouncing, scheduler overhead, GC pressure from extra allocations. Senior engineers can quote rough numbers on these without rerunning the benchmark.
2. The methodology. How to design experiments that produce a defensible answer. How to read pprof, trace, and benchstat output. How to know when N=10 is enough and when you need N=200.
3. The leadership. How to gate "performance" PRs, how to write a benchmarking policy, how to teach the team to look at the numbers before the diff.

If you do these three things well, your service does not become a graveyard of dead optimisations, and your team learns to discipline its instincts. If you do them poorly, every quarter brings a new pile of sync.Pool and "actor" code that nobody dares delete and nobody is sure works.

When concurrency hurts¶

The phrase "concurrency hurts" is precise. We are not saying "goroutines are bad" or "channels are slow." We are saying that, for a given workload, the act of running it concurrently can make the program slower, less reliable, and harder to maintain than the obvious sequential version. There are eight specific mechanisms by which concurrency hurts performance. A senior engineer recognises them on sight.

1. Per-goroutine fixed overhead exceeds the per-item work. Spawning, scheduling, finishing, and synchronising a goroutine costs roughly 1–3 microseconds on modern hardware. If the work the goroutine does is, say, "decode 30 bytes of varint," the bookkeeping is 100× the work. You are paying the courier more than the package is worth.
2. Coordination cost dominates the critical section. A naive range over a slice is one memory access per element, predictable, pipelined by the CPU. A channelised version pays a mutex-lock-and-park on every send and receive. The CPU does the same arithmetic; the runtime does much more bookkeeping.
3. False sharing. Two goroutines writing to different fields in the same cache line invalidate that line on every write. The CPU now reads the same line back and forth between cores at the speed of L2 or L3 (10–60 ns) rather than L1 (1 ns). You pay 10–60× per write.
4. Mutex contention pile-up. A protected section short enough to be fine sequentially becomes a queue when 16 goroutines compete for it. Each waiter is parked and woken (a few microseconds round-trip); the protected work itself runs the same speed it always did.
5. Stale cache lines from migration. When the scheduler moves a goroutine between Ps, the new core's cache is cold for that goroutine's stack and recently-touched data. The first iteration on the new core suffers L1/L2 misses (10–30 ns each) until the line is warm again.
6. GC and allocator amplification. Concurrent workers each have their own mcache and per-P state. Allocating from N goroutines in tight loops can multiply the GC's mark work, and increase the rate at which mcaches are flushed back to the central allocator. The GC's CPU share grows; the application's shrinks.
7. System-call queueing. Going wide on a single bottleneck — a database connection pool, a single file handle, an HTTP/2 stream — adds queue depth without adding throughput. The bottleneck did not get wider; you just put more workers in line for it.
8. Critical-path lengthening from coordination synchronisation. Fork-join workloads pay for the slowest worker. A single straggler caused by a hot CPU, a GC pause, or a stuck network call can leave 31 goroutines waiting on 1. The serial version had no straggler — it had one path.

Each of these is testable. Each shows up in pprof, trace, or perf counters. A senior engineer makes them visible before the optimisation is merged.

Sequential is faster than you think¶

A common mental model says "more goroutines = more parallelism = more speed." Modern CPUs and well-written compilers undermine that intuition in three ways.

Pipelining and ILP¶

A single core executes more than one instruction per cycle. Decoders, schedulers, and reorder buffers inside the core hide memory latency by issuing independent instructions while a load is in flight. A tight loop that reads arr[i], adds, and stores out[i] retires at well over 1 element per cycle on a modern x86. SIMD widens this further: AVX2 processes 4 doubles per instruction; AVX-512 processes 8.

If you split that loop across N goroutines, you are no longer feeding one well-tuned pipeline; you are starting N pipelines on N cores, each of which needs warm caches and tight branch prediction. The breakeven point is not "1 core's worth of work" — it is "1 core's worth of work, plus all the warm-up cost N times."

Branch prediction warmth¶

The CPU's branch predictor learns the pattern of a loop. After a few iterations it predicts the back-edge with near-perfect accuracy. When you split work across N goroutines, each goroutine warms up its own branch predictor state. The first iterations of each are slower.

Allocator and TLB locality¶

Sequentially walking a slice keeps the TLB and L2 hot. Splitting it across cores spreads the touched pages — the original program had 1 TLB working set, the new one has N. If the data exceeds L2 per core, you trade intra-core misses for inter-core misses.

A concrete number¶

// Sequential
func sumSeq(xs []float64) float64 {
    s := 0.0
    for _, x := range xs {
        s += x
    }
    return s
}

With len(xs) == 10_000_000 this runs at roughly 5 ms on a typical laptop CPU. A parallel version that splits across 8 goroutines runs at... roughly 5 ms too, sometimes 6 ms because of the coordination, sometimes 4 ms if the data was not warm in L2. The gain is a fraction of the dispersion. Below 10 million elements there is no consistent gain at all. Below 1 million elements the parallel version loses every time.

A senior engineer can predict this. A team that benchmarks ahead of merging discovers it. A team that does not benchmarks discovers it three quarters later when someone notices the parallel sum showing up in a flame graph at 3% of CPU.

Amdahl's law for goroutines¶

Amdahl's law gives the upper bound on speedup from parallelism:

            1
speedup = -----------
          (1 - p) + p/N

where p is the parallelisable fraction and N the number of cores. If p = 0.95 and N = 16, the maximum speedup is 1 / (0.05 + 0.95/16) = 9.14×. Note that you never reach 16× even with 16 cores; serial fractions dominate at scale.

For Go specifically there are three Amdahl-style serial fractions that are easy to miss:

1. Pre-fan-out setup¶

items, err := loadAllItems(ctx)   // serial: reading from DB
if err != nil { return err }
// ... fan out

If loadAllItems is half your wall time, parallelising the post-fan-out step caps your speedup at 2× regardless of cores.

2. Joining the result¶

var sum int64
for _, n := range results {
    sum += n
}

Aggregating N results is O(N) and serial. With short per-result work, the aggregation can be the new dominator.

3. Channel send/receive¶

Every channel hop synchronises. In a pipeline stage1 -> stage2 -> stage3, even if stage 2 fans out to 8 workers, the channel between stages 1 and 2 is a serial bottleneck. The total wall time is bounded by stage1 plus the slowest stage's per-element latency.

Amdahl's law as a budget check¶

Senior engineers do an Amdahl check before writing the parallel code:

1. Roughly, what fraction of wall time is in the loop I want to parallelise?
2. Multiply that fraction by the rough core count you can use.
3. Subtract overhead at, say, 5% of total runtime per goroutine boundary.

If the answer is less than 1.3× speedup, do not parallelise. The simple version wins, because complexity has a cost and 30% is rarely worth it.

A worked Amdahl calculation¶

Suppose a function takes 100 ms wall time, of which:

• 10 ms is input parsing (serial)
• 80 ms is a CPU-bound loop you could parallelise
• 10 ms is output formatting (serial)

With 8 cores, ideal speedup of the loop is 80 ms / 8 = 10 ms, total 10 + 10 + 10 = 30 ms. Real-world: add 5 ms goroutine overhead and 5 ms join overhead — 35 ms. Net speedup: 100 / 35 = 2.86×. Useful, but a factor of 8 not a factor of 14. If the function is called at 100 req/s, you saved 6.5 seconds per second — clearly worth the engineering.

Now suppose the same function takes 5 ms wall time, of which:

• 1 ms input parsing
• 3 ms CPU-bound loop
• 1 ms output formatting

With 8 cores, loop in 0.375 ms, total 1 + 0.375 + 1 = 2.375 ms. Real-world: add 1 ms goroutine overhead, 3.375 ms. Net speedup: 5 / 3.375 = 1.48×. Worth maybe a day of engineering, not three.

Amdahl turns "should we parallelise" from a feeling into a number.

Gustafson, Karp-Flatt, and the practical companions to Amdahl¶

Amdahl assumes a fixed problem size and asks how much faster N cores make it. Gustafson's law inverts the question: given N cores, how much larger a problem can we solve in the same time? Most production workloads are Gustafson-shaped: the data set grows, the user base grows, the dashboard widget gets faster or shows more detail in the same time budget.

Gustafson's formula:

speedup = (1 - p) + p * N

If p = 0.95 and N = 16, Gustafson speedup is 0.05 + 0.95 * 16 = 15.25×. Why the optimism? Because we are not trying to do the same work faster; we are trying to do more work in the same time. Gustafson is the law that justifies, say, "with these 16 cores we can now compute a 16× larger window of analytics per dashboard refresh."

Karp-Flatt for diagnosing real systems¶

If you have measured serial time T1 and parallel time Tn on n cores, Karp-Flatt gives the experimentally determined serial fraction:

        (1/speedup) - (1/n)
e = ---------------------------
              1 - (1/n)

If e grows as you add cores, you have a synchronisation bottleneck (mutex contention, false sharing). If e stays roughly constant, your serial fraction is a real serial part you have not addressed yet.

A senior engineer reaches for Karp-Flatt before redesigning a slow parallel workload. "We saw e = 0.1 at 4 cores and e = 0.3 at 16 cores — that is a synchronisation problem, not a serial fraction; let's profile the mutex wait, not refactor the algorithm." Without Karp-Flatt the debate is anecdotal.

Context-switch costs¶

A goroutine context switch on Go's M:N scheduler is dramatically cheaper than an OS thread context switch — but it is not free. On x86_64 Linux a goroutine-to-goroutine switch is roughly 150–250 ns in 2024-era CPUs when both goroutines are runnable on the same P. Switching to a goroutine on a different P (because of work stealing) costs more, because the destination CPU's caches are cold for the goroutine's stack.

What happens during a goroutine switch¶

1. The current goroutine reaches a safe point — function preamble, channel op, stack growth check, or asynchronous preemption signal.
2. The scheduler saves the current g.sched (register file, PC, SP).
3. The scheduler picks the next runnable G from the local P's runq.
4. The scheduler swaps the M's pointer from old G to new G.
5. The CPU restores the new G's registers.

Steps 2 and 5 are roughly free (one cache-resident memory operation each). Step 3 is contention-free if the runq is non-empty. The actual cost is the cache miss profile: jumping to the new G's stack frame and code may evict lines the old G had warm.

What does not happen¶

There is no kernel transition. No TLB flush. No address-space change. No syscall. That is why goroutine switches are 100–1000× faster than OS thread switches.

When switches start to dominate¶

If your goroutine does 100 ns of useful work and then waits on a channel, you are paying:

• 100 ns of work
• ~150 ns to save state
• ~150 ns to restore state on the next goroutine
• N times the work-set cache miss to warm L1/L2 for the new goroutine

The ratio of useful to overhead is roughly 1:3. A serial loop running the same 100 ns of work is at 1:0. The serial loop is 3× faster regardless of how many cores you have.

Numbers to memorise¶

Senior engineers think in these numbers. If you can compute "how many cache misses does this design introduce per item" you can estimate the speedup before writing the code.

Scheduler overhead in detail¶

Go's scheduler is excellent, but every bit of work it does is work your CPU is not doing on your program. There are five scheduler overheads worth understanding.

1. Work stealing¶

When a P's local runqueue is empty, it tries to steal half of another P's runqueue. Stealing requires:

• Atomic CAS on the source P's head/tail pointers.
• Cache coherence traffic to move the stolen Gs' descriptors to the new P.
• A few hundred cycles of bookkeeping.

In a workload where Ps are mostly idle, stealing happens often and is cheap. In a workload where every P is busy, stealing is rare. The pathological case is a workload where one P generates work in bursts: other Ps repeatedly try to steal, fail, and go idle.

2. Global runqueue checks¶

Every 61 scheduler ticks (a runtime constant), a P checks the global runqueue before its local one. This prevents starvation of goroutines pushed to the global queue (e.g. on stack growth or runtime.Gosched). The check is cheap, but it adds occasional cross-P cache traffic.

3. Sysmon¶

A monitor goroutine sysmon runs at periodic intervals, checking for:

• Goroutines stuck in long syscalls (it hands off the P to another M).
• Goroutines running too long without a safe point (asynchronous preemption since Go 1.14).
• Idle Ms to retire.
• GC trigger conditions.

Sysmon itself uses a tiny fraction of CPU, but its work creates scheduler activity that affects your program — preemptions, hand-offs, and runqueue pokes.

4. Preemption¶

Asynchronous preemption fires a SIGURG to a goroutine running too long. Receiving the signal and rewinding to a safe point costs roughly 2–4 µs. If your goroutines all run shorter than 10 ms, you never see preemption. If you have a tight loop with no function calls, preemption fires every ~10 ms.

5. P-handoff on blocking syscall¶

A goroutine entering a blocking syscall releases its P. The handoff costs roughly 1 µs. The cost is per-syscall, not per-goroutine, so it adds up if your design issues a syscall for every item. Channel-based pipelines with explicit batching can amortise this.

Profile evidence¶

Look at top in pprof's CPU profile. If you see runtime.findrunnable, runtime.schedule, runtime.gopark, runtime.goroutineReady collectively above 5% of CPU, the scheduler is doing too much work for your workload. Common causes: too many short-lived goroutines, too many channel hops, too small a batch size.

False sharing¶

False sharing is the single most surprising performance bug in concurrent code, and the one most often introduced by "let me just add a goroutine."

What it is¶

CPU caches operate on cache lines, typically 64 bytes on x86_64 and 128 bytes on Apple Silicon. When core A writes to address X and core B writes to address Y, and X and Y fall on the same cache line, both cores must coordinate to invalidate each other's copies. The actual data they touch is different — there is no logical conflict — but the cache hardware does not see logical conflicts; it sees cache lines.

The result: each write causes a cache invalidation, the other core reads the line back from L2 or L3, and the cost per write balloons from ~1 ns (L1) to ~30 ns (L3) or worse (memory).

A concrete example¶

type Counters struct {
    A int64
    B int64
}

var c Counters

// Goroutine 1
for i := 0; i < 1e9; i++ {
    c.A++
}

// Goroutine 2
for i := 0; i < 1e9; i++ {
    c.B++
}

On x86_64, A and B are adjacent fields in a 64-byte cache line. Goroutines on different cores writing to A and B ping-pong the line between L1 caches. Throughput drops by a factor of 10–30 compared to either loop alone.

The fix: padding¶

type Counters struct {
    A int64
    _ [56]byte    // pad to a full cache line
    B int64
    _ [56]byte
}

Now A and B live in separate cache lines and the goroutines do not interfere. Throughput recovers.

Alternative: runtime.CacheLineSize (Go 1.22+ proposal pending) or convention-driven padding constants:

const cacheLine = 64

type Counters struct {
    A int64
    _ [cacheLine - 8]byte
    B int64
}

Where false sharing hides¶

• Stride-1 access on a shared slice from multiple workers (worker 0 writes indices 0,8,16,...; worker 1 writes 1,9,17,...). Many of those writes share lines.
• Per-goroutine fields in a struct array indexed by goroutine ID.
• []atomic.Int64 counters indexed by worker ID without padding.
• Slabs of small structs in a shared backing array.

Detection¶

perf c2c record / perf c2c report on Linux highlights cache-line contention by virtual address. The runtime/race detector does not find false sharing (it is not a race, just a performance bug). Benchmarks that scale sublinearly with cores are the most reliable signal.

Senior heuristic¶

If a parallel implementation gives 1.5× speedup on 8 cores rather than ~7×, false sharing is the first hypothesis. Pad and re-run; if you go from 1.5× to 6×, you found it.

Cache lines, NUMA, and memory traffic¶

False sharing is the headline cache effect, but the broader story is memory traffic. The CPU's memory hierarchy is roughly:

L1d (per core)   ~32 KB    ~1 ns
L2  (per core)   ~512 KB   ~3 ns
L3  (shared)     ~10 MB    ~12 ns
DRAM             GBs       ~70 ns
Remote NUMA      GBs       ~150 ns

A goroutine running on core 0 with its data in L1 retires loads at ~1 ns. The same goroutine after being stolen onto core 4 reads from L3 or DRAM for the first several thousand cycles. The cost depends on data set size.

NUMA in Go¶

On multi-socket servers, memory is partitioned per socket. A page of memory has a home NUMA node — the socket whose cores can reach it fastest. Cross-NUMA accesses cost roughly 2× a local-NUMA access.

Go is NUMA-agnostic. The runtime does not pin Ps to NUMA nodes. The scheduler may move a goroutine across sockets at any time. On NUMA hardware this can hurt: imagine a goroutine allocates a large slice on socket 0's memory, then is stolen to socket 1 — every read crosses the interconnect.

Mitigations¶

• On NUMA hardware, set CPU affinity at the process level (taskset, numactl) to constrain Go to one socket if your service fits.
• Pre-fault and allocate per-worker data on the worker's preferred socket. Linux provides numa_alloc_onnode; Go does not expose this, so for true NUMA-aware Go services you often need cgo wrappers.
• For most services it is sufficient to avoid NUMA hardware until you measure that you need it. A 16-core single-socket box has no NUMA penalty.

Going wide vs going deep¶

The senior insight: more cores rarely beat warmer caches. If your service is a 10-µs request that touches 4 KB of data, scaling to 64 cores buys nothing unless you have at least 64 requests in flight at all times, because a single request's data fits in L1 anyway. The cores sit half-idle waiting on slow memory traffic from other goroutines that thought they were "scaling."

Measurement-first methodology¶

Senior engineering means refusing to merge optimisations that are not measured. Here is the methodology:

Step 1: Establish a representative workload¶

A workload is a function plus a distribution of inputs and a load pattern. "Random ints" is not a workload — "the size distribution of our actual request batches over the last week, replayed at production rate" is a workload.

Sources for representative inputs:

• Anonymised production captures (PCAP, request logs).
• Replayed traces from observability systems.
• Synthetic distributions calibrated against production histograms.

The benchmark you ship should use input that resembles production. Senior engineers reject benchmarks that use make([]int, 1e6) filled with ascending integers as "synthetic and not generalisable."

Step 2: Define the metric and the SLO¶

Throughput? p50 latency? p99 latency? Memory? GC pause?

Pick one or two; do not chase all five. Have an SLO in mind: "the change should not regress p99 latency on the 1k-batch case by more than 5%."

Step 3: Write the benchmark¶

Use testing.B, b.ReportAllocs(), b.ResetTimer(). Make the benchmark deterministic: seed the RNG, control input size with -benchtime. Run with -cpu=1,2,4,8 to cover the GOMAXPROCS matrix.

Step 4: Run it enough times to be statistically meaningful¶

-count=10 minimum. -count=20 for noisy benchmarks. Use benchstat to compare the baseline and the optimised version. Report geomean and per-benchmark deltas with p<0.05.

Step 5: Inspect the profile¶

-cpuprofile, -memprofile, -blockprofile, -mutexprofile. Before and after. If you cannot explain the speedup with a profile delta, your benchmark is wrong or noisy.

Step 6: Decide¶

Speedup × use_frequency > engineering_cost + maintenance_cost?

If yes: merge. If no: archive the branch with a benchmark and a note for next time. If equivocal: pick the simpler implementation.

A checklist for the senior reviewer¶

When reviewing a "make it concurrent" PR, ask the author:

1. Where is the benchmark?
2. Where is the baseline?
3. Where is the benchstat output?
4. Where is the pprof profile showing the win?
5. Is the win larger than the noise floor (p < 0.05)?
6. Does the win hold at -cpu=1?
7. Does the win hold under GC pressure (run with GOGC=20)?
8. Does the win hold at production input distribution?
9. What does the trace look like under load?
10. Is the new code clearer, or have we taken on complexity for the speedup?

If any answer is "I have not measured that," the PR is not ready.

Designing the right experiment¶

A benchmark is an experiment. Experiments should isolate the variable under test. Here are the experiment design errors senior engineers reject most often.

Confounding the measurement¶

func BenchmarkParallel(b *testing.B) {
    items := loadItemsFromDisk()   // confound 1: disk
    for i := 0; i < b.N; i++ {
        var wg sync.WaitGroup
        wg.Add(len(items))
        for _, item := range items {
            go func(item Item) {
                defer wg.Done()
                process(item)
            }(item)
        }
        wg.Wait()
    }
}

Two confounds: the disk read is inside the loop, and the goroutine spawn cost is conflated with the parallel processing. The right design is to load once outside, reset the timer, then measure the parallel loop alone.

Warm-up¶

The first iteration of a Go benchmark suffers cold caches, JIT-like instruction caches, allocator warm-up. go test runs N iterations and divides; with N=1 the warm-up dominates. With N=1000 it amortises. Use -benchtime=2s or -benchtime=10000x so each measurement is many iterations.

Background interference¶

Benchmarks run on a busy laptop are noisy. Run on an isolated machine, disable turbo boost (so frequency is fixed), disable Spectre mitigations only if your production runs without them. Pin to specific cores.

Senior-level benchmark machines:

• Dedicated, not a developer laptop.
• Frequency scaling disabled or pinned (cpupower frequency-set -g performance).
• Background services (cron, snapd, indexers) quiesced.
• Same machine for baseline and candidate, in a controlled run order.

Input bias¶

Tests written by the author tend to favour their hypothesis. Senior reviewers ask for a benchmark suite that covers:

• Small input (where overhead might dominate).
• Medium input (the common case).
• Large input (where parallelism should win).
• Adversarial input (e.g. all values identical, all values distinct, very long elements).

A win on "medium random" that fails on "small skewed" is rarely a real win.

Variance¶

Re-run the same -count=20 twice. If the geomean differs by >2%, your variance is too high. Increase -benchtime, isolate the machine, or change the workload to be more deterministic.

Reviewing a "make it concurrent" PR¶

Here is a checklist for a thoughtful PR review when someone proposes adding concurrency:

1. Does the description state the latency or throughput goal? "Add parallel processing" is not a goal. "Reduce p99 of batch import from 800 ms to 400 ms" is a goal.
2. Is there a benchmark covering the goal metric? If not, decline the PR and ask for one.
3. Does the benchmark cover the right input distribution? Compare the benchmark inputs to recent production data.
4. Has the author run benchstat? Are the deltas statistically significant?
5. Is -cpu=1 regressed? A parallel design that loses on 1 CPU is buggy or expensive. Smaller-than-expected datasets running on a single-CPU pod will pay the cost without the benefit.
6. Does the trace show good parallelism? Open go tool trace; the goroutines should overlap. If goroutines run sequentially because of a shared mutex, the design is broken.
7. Is the worker count bounded? Unbounded fan-out is a memory bug waiting to happen. The PR should justify N workers, often as runtime.GOMAXPROCS(0) or a config constant.
8. Are errors propagated correctly? Fan-out designs lose the first error or accumulate all errors; the team should be explicit about which.
9. Are panics recovered? A panic in a worker should not crash the service if the service handles many independent batches.
10. Is cancellation honoured? ctx.Done() should short-circuit worker loops.
11. Are profiles attached? A pprof CPU + alloc profile of the new code under benchmark load.
12. Are the resource costs proportional? "We get 1.1× speedup at 8× CPU and 2× memory" is rarely worth merging.

A senior engineer is not the bad guy. They are an advocate for the team's future self. The PR that gets merged is the one that is defensible a year later when the speedup is no longer visible.

Sequential vs parallel decision tree¶

A practical decision tree for "should this be concurrent?":

Is the work CPU-bound?
├── No (I/O-bound): does it serialise on a single resource (one DB, one disk)?
│   ├── Yes -> concurrency adds queue depth, not throughput; profile the bottleneck instead.
│   └── No  -> concurrency may help (multiple independent I/Os in flight). Bound by the slowest.
└── Yes (CPU-bound): is the per-item work > ~1 µs?
    ├── No -> sequential is almost always better. Goroutine overhead dominates.
    └── Yes: is the total work > ~10 ms at production sizes?
        ├── No -> sequential is competitive; the simpler code wins.
        └── Yes: can you bound the worker pool to GOMAXPROCS?
            ├── No -> fix the design first.
            └── Yes: are the items independent (no shared mutable state)?
                ├── No -> coordination cost likely exceeds the win.
                └── Yes -> *measure* first, then parallelise.

This tree is not gospel. It is a default. Each box of the tree should have a benchmark backing the decision in your specific service.

Latency vs throughput at the design table¶

A frequent confusion: "we want to be faster." Faster how?

Concurrency primarily improves throughput and sometimes single-request latency for large requests. It rarely helps a small request's p50 latency, and it often hurts p99 because tail latency in concurrent systems is dominated by the worst worker.

A senior conversation in a design review:

• "We want to be faster" -> "Faster how?"
• "We want lower p99 on batch endpoint" -> "Is the batch endpoint already concurrent? Where does p99 come from in pprof?"
• "We want more req/s" -> "What is current saturation? Are workers stuck on a single DB connection?"

The right answer to "we want to be faster" is rarely "let's add goroutines." It is "let's profile and understand where the time goes."

The "shape" of the workload¶

Concurrency decisions hinge on workload shape. Senior engineers categorise workloads before optimising.

CPU-bound, embarrassingly parallel¶

Work on each item is independent, takes >>1 µs, and uses little memory. Examples: hashing files, decoding many small JPEGs, compressing a list of blobs. Concurrency wins big; bound the pool to GOMAXPROCS.

CPU-bound, dependent¶

The items have ordering or dependency constraints (e.g. update a tree, with each update depending on the previous). Concurrency rarely wins; serial usually is correct, and any parallelisation needs careful design.

I/O-bound, many independent endpoints¶

Many remote calls to many endpoints. Concurrency wins on wall time, bounded by the slowest endpoint.

I/O-bound, single endpoint¶

Many calls to one endpoint. Concurrency may not help; the endpoint or the connection pool is the bottleneck. You may add latency from connection queueing.

Memory-bound¶

Work spends most of its time waiting on cache or DRAM. Concurrency rarely helps; cache contention often hurts. Look at false sharing, prefetching, and locality first.

Mixed¶

Most real workloads. Profile, then decide. The dominant phase determines whether concurrency is the right tool.

Cost models you can carry in your head¶

Senior engineers carry rough cost models that let them estimate before benchmarking. Memorise:

• Goroutine spawn: ~1 µs end-to-end (allocation + scheduler enqueue).
• Goroutine context switch on same P: ~200 ns.
• Channel send/recv, buffered, no block: ~50 ns.
• Channel send/recv, unbuffered, with park: ~250 ns.
• Mutex lock, uncontended: ~10 ns.
• Mutex lock, contended (parked): ~1 µs.
• Atomic load/store: ~1 ns.
• Atomic CAS, uncontended: ~5 ns.
• Atomic CAS, contended: ~30–100 ns (cache-line bounce).
• Map access: ~20–50 ns.
• Slice append (no growth): ~3 ns.
• GC pause, modest heap: ~100 µs to a few ms.

If your per-item work is below 1 µs, any concurrency primitive that runs per item will dominate. If your per-item work is above 100 µs, even unbuffered channels are noise. The "interesting" zone is 1–100 µs of work per item; that is where most optimisation debates live, and it is where benchmarks are most informative.

Coordination overhead¶

Every concurrent design has coordination overhead. The cheaper the work, the more visible the coordination.

Counting the coordination¶

A typical worker pool incurs, per item:

• 1 channel send (producer enqueues)
• 1 channel recv (worker dequeues)
• 1 result channel send (worker emits)
• 1 result channel recv (aggregator receives)
• 4 × channel cost = ~200 ns at best, ~1 µs at worst.

If the work is 1 µs, you have doubled the wall time. If the work is 100 µs, the overhead is 1%.

Avoiding coordination¶

• Chunk the input. Send slices, not items. 100 items per send drops the per-item overhead by 100×.
• Avoid double channels. A result channel is often replaceable by a write into a slice indexed by worker ID, joined at the end.
• Use errgroup.SetLimit instead of a channel-based semaphore.
• Skip workers entirely when work is short; iterate serially and call it a day.

Why parallelizing tiny tasks loses¶

Spelled out: imagine a task that takes 100 ns. You parallelise it across 8 workers. The per-task overhead is 200 ns (goroutine context switch + channel send/recv). The parallel version spends 200 ns of coordination per 100 ns of work — it is 2× slower than the serial version regardless of cores.

Now imagine the task takes 10 µs. The per-task overhead is still 200 ns. Coordination is 2% of work. Eight workers give you roughly 7.8× speedup at the limit. The parallel version wins decisively.

The lesson: parallelism scales with work granularity. Increase the granularity (chunk the input) until the per-item overhead is small relative to the per-item work. Stop fan-out below that threshold.

Channel and mutex hidden costs¶

A sync.Mutex is roughly 8 bytes. Uncontended lock/unlock is ~10 ns. Contended lock can park the goroutine, which costs ~1 µs round-trip.

A chan is heavier:

• Buffered chan: ring buffer of element slots, head/tail indices, mutex, two wait queues.
• Allocation: tens of bytes, sometimes hundreds for the buffer.

Compared to a slice + mutex, a channel costs more per operation. The reason to use a channel is clarity of ownership and signalling, not raw speed. The fastest concurrent design is usually neither pure-channels nor pure-mutexes: it is a slice handed to N workers each owning a range, with one final mutex-protected join.

A common surprise¶

ch := make(chan int, 1024)
go producer(ch)
go consumer(ch)

If the consumer keeps up with the producer, the channel is hot in cache and operations cost ~50 ns each. If the consumer falls behind, the buffer fills, the producer parks, and every send costs ~250 ns. Steady-state throughput halves.

The fix is often not a larger buffer (which delays the parking but does not prevent it) but a slower producer or a faster consumer.

Common premature optimisations and their refutations¶

A non-exhaustive list of "optimisations" that more often than not lose:

1. sync.RWMutex for a hot read path. RWMutex has higher uncontended cost than Mutex and is only faster if the read sections are long and contention is severe. For sub-microsecond reads, Mutex wins.
2. sync.Pool for objects that fit in a register. Pool has overhead. Reusing a []byte of 16 bytes is slower than allocating it.
3. Lock-free queues for trivial workloads. A lock-free MPMC queue is harder to reason about and rarely faster than chan for the message rates real services see.
4. Atomic counters across cores. A single global counter incremented by many goroutines suffers cache-line bouncing. A per-goroutine local counter joined at the end is much faster.
5. Sharded map for small maps. A 32-shard map costs 32× the allocation of a single map. Unless contention is measured, single map wins.
6. Goroutine per request item. Spawning goroutines per leaf operation costs more than the leaf operation. Use a fixed worker pool.
7. Buffered channels "to avoid blocking." Buffered channels delay backpressure, hiding overload. A small buffer is fine; a large buffer hides a problem.
8. atomic.Value for hot config. Each load is a barrier. If the config rarely changes, a copy-on-write pointer with atomic.Pointer is often the same cost; if it never changes after startup, no synchronisation is needed at all.
9. Custom scheduler / goroutine pools for short tasks. The runtime scheduler is excellent. Reinventing it via a pool of goroutines that pull from a channel is usually slower than go f() directly.
10. CRDTs for single-writer state. Conflict-free replicated data types are for true multi-master replication. Internal in-process state rarely needs them.

A senior engineer challenges each of these on sight, asking the author to prove that the simpler version was inadequate.

Building a culture of measurement¶

You can document discipline, but unless the team practises it, the docs are decoration. Culture is what people do without being told.

Make benchmarks first-class¶

• A benchmarks/ directory or per-package *_bench_test.go files, run in CI on dedicated hardware.
• A baseline file checked in alongside the code, updated on intentional changes only.
• A weekly benchmark trend dashboard that flags regressions.

Make profiles first-class¶

• A Profile this button in your service's debug endpoints. A simple HTTP server that exposes pprof and trace endpoints behind auth.
• A "run a profile" runbook on every on-call rotation.
• A culture where the first investigation step for a slow endpoint is "did you profile it?"

Make benchstat mandatory¶

For any PR claiming a perf improvement, the description must include benchstat output. PRs without it are returned.

Make undo cheap¶

Every "optimisation" PR should include the benchmark and a one-line revert plan. If the optimisation hurts in production, the team can revert without ceremony.

Calibration days¶

Once a quarter, the team picks one "optimisation" already in the code and re-measures it. Either re-confirm or remove. Code that is not re-confirmed is presumed cargo-culted.

Organisational norms¶

Beyond a single team, the org needs norms.

The "no concurrency without measurement" gate¶

Code reviewers reject "make it concurrent" PRs that do not include a benchmark and a pprof.

The "burden of proof" rule¶

The author of the optimisation owns the proof. Reviewers do not need to prove the change is harmful; the author must show it helps.

Public benchmark archive¶

Each merged optimisation gets a one-page write-up: workload, baseline, candidate, delta, decision. This builds institutional memory and prevents repeating the same evaluation in two years.

Performance bug bounty¶

Senior engineers can offer "bring me a measured 10% improvement of X and I will mentor you through merging it." This rewards measurement, not bravado.

Worked example: a CSV pipeline that got slower¶

A team had a 1 GB CSV file to process: parse each row, validate, transform, emit. The original implementation was a tight serial loop using bufio.Scanner. A junior engineer proposed parallelising row processing across 8 workers.

The proposed design¶

func processCSV(r io.Reader) error {
    rows := make(chan []byte, 1024)
    var wg sync.WaitGroup
    for i := 0; i < 8; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for row := range rows {
                processRow(row)
            }
        }()
    }

    sc := bufio.NewScanner(r)
    for sc.Scan() {
        rows <- append([]byte(nil), sc.Bytes()...)
    }
    close(rows)
    wg.Wait()
    return sc.Err()
}

The numbers¶

Sequential: 9.4 s. Parallel: 11.7 s.

The parallel version was 24% slower. Why?

1. processRow was 5 µs per row. The channel send/recv was ~250 ns each — 10% overhead.
2. The append([]byte(nil), ...) copy was new work the serial version did not do. The serial version processed the scanner's internal buffer in place. The copy added 200 ns per row.
3. The producer was the same goroutine that read the file. It was already saturating one core; the workers were idle half the time waiting for input.
4. All 8 workers wrote to a shared output file via a mutex. Contention was modest but measurable.

The fix¶

Two options were tested.

Option A: keep parallel but batch. The scanner sends batches of 100 rows. Per-batch overhead is now 2.5 ns per row instead of 250 ns. Workers do 500 µs of work per dequeue. The output mutex is taken once per batch. Result: 4.1 s. Big win.

Option B: keep sequential, eliminate the copy. No goroutines, no channels. Use bufio.Reader.ReadSlice to avoid copies. Result: 6.2 s. A 34% improvement with no concurrency.

The team kept Option A because the batch was a real algorithmic improvement. The lesson was that the original "parallel" PR was actually two things bundled: a parallel design and a needless per-row copy. The senior reviewer asked the question that disentangled them, and the team learned to never propose "make it concurrent" without measuring against "make the sequential version better."

Worked example: image thumbnailer¶

A service generates JPEG thumbnails. Input: a directory of 10000 files. Two implementations:

Sequential¶

for _, name := range files {
    if err := thumbnail(name); err != nil {
        return err
    }
}

Wall time on 8 cores: 92 s.

Parallel with worker pool¶

sem := make(chan struct{}, 8)
var g errgroup.Group
for _, name := range files {
    name := name
    sem <- struct{}{}
    g.Go(func() error {
        defer func() { <-sem }()
        return thumbnail(name)
    })
}
return g.Wait()

Wall time on 8 cores: 14 s.

A 6.6× speedup on 8 cores. Why does this one win?

• thumbnail is CPU-bound (JPEG decode/resize/encode) and takes ~7 ms per image. Per-item work is 7000× a goroutine spawn.
• The output is independent per file; no shared mutable state.
• The pool is bounded; no goroutine explosion.

The Amdahl serial fraction here is "directory walk" which is < 1% of wall time. The speedup is close to the theoretical maximum of 8 minus overhead.

Conclusion: this is a good place for concurrency. The senior reviewer notes it, benchmarks confirm it, the PR merges.

Worked example: HTTP request fan-out¶

A handler aggregates data from 5 microservices. Sequential calls take ~50 ms each; total 250 ms.

Parallel design¶

g, ctx := errgroup.WithContext(ctx)
var users []User
var orders []Order
var inventory Inventory
var prices Prices
var recommendations []Rec

g.Go(func() error { var err error; users, err = svc.Users(ctx); return err })
g.Go(func() error { var err error; orders, err = svc.Orders(ctx); return err })
g.Go(func() error { var err error; inventory, err = svc.Inventory(ctx); return err })
g.Go(func() error { var err error; prices, err = svc.Prices(ctx); return err })
g.Go(func() error { var err error; recommendations, err = svc.Recommendations(ctx); return err })
if err := g.Wait(); err != nil {
    return err
}

Wall time: ~60 ms (bounded by the slowest call plus a small amount of overhead).

Why this wins:

• I/O-bound; goroutines do not consume CPU while blocked on network.
• Independent endpoints; no shared bottleneck.
• Network latency dominates; goroutine overhead is invisible.

The 5× speedup is essentially free. This is the textbook case for concurrency in Go services, and is so reliably useful that "fan out independent service calls" is rarely a controversial PR.

Worked example: a hot counter¶

A service tracks 5 counters: requests, errors, bytes_in, bytes_out, p99_seconds. Each handler increments all 5.

Naive: one global mutex¶

type Stats struct {
    mu sync.Mutex
    Requests, Errors, BytesIn, BytesOut int64
}

Under 16-concurrent-handler load, the mutex is hot. Profile shows 8% of CPU in sync.(*Mutex).Lock.

Premature optimisation: atomics¶

type Stats struct {
    Requests, Errors, BytesIn, BytesOut int64
}

func (s *Stats) AddRequest() {
    atomic.AddInt64(&s.Requests, 1)
}

Same fields, same cache line, 16 goroutines hammering atomics on it. Cache-line bouncing. Result: slower than the mutex version, because the atomics each generate cache-coherence traffic and the mutex version batches multiple field updates under one acquisition.

Better: per-CPU sharded counters¶

type Stats struct {
    shards []struct {
        Requests, Errors, BytesIn, BytesOut int64
        _ [32]byte // pad to cache line
    }
}

func (s *Stats) AddRequest() {
    p := runtime.GOMAXPROCS(0) // approximate "shard by P"
    shard := fastrand() % uint32(p)
    atomic.AddInt64(&s.shards[shard].Requests, 1)
}

This works but is over-engineered. The simpler win:

Simplest: per-goroutine local, flush on response¶

Each handler counts in local variables, then flushes once at end:

func handler(...) {
    var localReq, localBytes int64
    // ... do work, mutate localReq, localBytes
    atomic.AddInt64(&stats.Requests, localReq)
    atomic.AddInt64(&stats.BytesOut, localBytes)
}

One atomic op per handler instead of N. No cache contention. Profile drops counter overhead from 8% to 0.1%.

The lesson: the obvious "use atomics" is often worse than "use a mutex"; the right answer is usually "reduce the number of operations," not "use a cheaper operation."

Worked example: parallel map¶

A common ask: "let's make a thread-safe map that scales better than sync.Map."

Attempt 1: sharded map, 32 shards¶

type ShardedMap[K comparable, V any] struct {
    shards [32]struct {
        mu sync.RWMutex
        m  map[K]V
    }
}

For a workload with 10000 keys and 80/20 read/write under 16 concurrent goroutines:

• Single sync.Mutex-protected map: 6.8 µs/op.
• sync.Map: 4.2 µs/op.
• Sharded 32-way: 4.8 µs/op.
• Sharded 32-way with RWMutex: 5.1 µs/op.

The 32-way sharding lost on this workload. Why? The hash function (compute the shard index) cost 50 ns per access. Most contention was already absorbed by the runtime's locking strategy. RWMutex lost to Mutex because the read sections were too short to benefit from reader-parallelism.

Attempt 2: lock-free map¶

Implementations of lock-free hash maps in Go (xsync.Map, third-party libraries) can win for very read-heavy workloads with many cores. They lose when:

• The map has many writes.
• The map is small.
• The contended span is short.

Always measure with realistic data and load. Do not introduce a third-party "fast map" unless you can show a 2× win on your benchmark suite and the code is well-understood.

Reading a CPU profile after a concurrency change¶

A senior engineer reading a pprof CPU profile of a concurrent program looks for:

Runtime overhead¶

Functions that show up in CPU profiles include scheduler internals. A healthy program might show 1–3% of CPU in runtime.findrunnable, runtime.schedule, runtime.gopark, runtime.lock. If those are above 10%, the scheduler is doing too much work for the workload — typical causes are too many short goroutines, too many channel hops, or excessive contention.

Allocation overhead¶

runtime.mallocgc, runtime.makeslice, runtime.mapassign_* show up when the workload allocates a lot. If the parallel version allocates more than the serial version, the GC will eat the savings.

Mutex contention¶

sync.(*Mutex).Lock, sync.(*Mutex).Unlock, and sync.runtime_SemacquireMutex show up under contention. If runtime_SemacquireMutex is visible, you are actually parking — that is the expensive path.

False sharing signal¶

False sharing does not show up as a labelled function. It appears as elevated CPU time in user code, plus high IPC (instructions per cycle) drops in perf counters. The flame graph shows nothing wrong; the wall-clock disagrees with the user CPU.

Comparison method¶

Run the profile with -base to compare candidate to baseline:

go tool pprof -base before.pb.gz after.pb.gz

This shows deltas: functions that got more or less time. A successful optimisation has a clear "thing that went down." A bad optimisation has lots of small wins and losses with no net direction.

Reading a trace after a concurrency change¶

go test -trace trace.out then go tool trace trace.out reveals the timeline. What to look for:

Parallelism in the goroutine view¶

Open "Goroutine analysis" and look at the time each goroutine spent in each state. If most workers spent most of their time runnable but not running, the scheduler is starved (rare in Go; usually means you have hundreds of thousands of goroutines and they cycle).

If workers spent most time blocked on channels, the producer is the bottleneck. Increase batch size or speed up the producer.

If workers spent most time in syscalls, they were waiting on I/O. Concurrency is helping if there are >1 I/Os in flight, hurting if they queue on one resource.

GC pauses¶

go tool trace highlights GC pauses. Long pauses are a clue that the parallel version is allocating more than the serial.

Per-P utilisation¶

Each P's timeline shows how much of wall time it spent running goroutines. Idle stripes mean the workload was not saturating that core. If 4 of 8 Ps were idle most of the run, your nominal "8-way parallel" was actually "4-way."

Statistical significance for benchmarks¶

Benchmarks are noisy. To compare two variants:

Run with -count=N¶

go test -bench=. -count=20 -cpu=1,2,4,8 > before.txt
# ... change code ...
go test -bench=. -count=20 -cpu=1,2,4,8 > after.txt
benchstat before.txt after.txt

benchstat reports:

• Mean ± standard deviation per benchmark.
• Geomean across benchmarks.
• Welch's t-test p-value for each pair.

A delta with p > 0.05 is not statistically significant; do not claim a speedup.

How many runs are enough?¶

Rule of thumb: increase -count until the within-run variance stabilises. For a benchmark with 1% noise, -count=10 suffices. For a benchmark with 10% noise, -count=50 or more is needed; you also probably want a quieter machine.

Tail-aware metrics¶

benchstat reports central tendency. If you care about p99, write a benchmark that measures p99 directly (using b.Loop plus a slice of timings, or a histogram).

Beware of compiler tricks¶

A benchmark whose result is discarded can be optimised away. Use runtime.KeepAlive or b.ReportMetric to consume the result, or sink to a package-level variable.

var sink int

func BenchmarkX(b *testing.B) {
    for i := 0; i < b.N; i++ {
        sink = work()
    }
}

GOMAXPROCS at the senior level¶

GOMAXPROCS controls the number of Ps. The default is the number of logical CPUs available to the process (CPU set as detected by runtime.NumCPU; since Go 1.22 also limited by cgroup CPU quotas on Linux, with full automatic adjustment expected in 1.23+).

What changes with GOMAXPROCS¶

• More Ps means more parallelism, bounded by available cores.
• More Ps means more scheduler bookkeeping; each P has its own runq, mcache, etc.
• More Ps means more memory: each P's overhead is a few KB.

Pitfalls¶

• Containerised environments where runtime.NumCPU reports the host CPU count, not the cgroup limit. Older Go versions ran on 64 Ps inside a 2-CPU container, wasting scheduler effort and increasing GC contention.
• Setting GOMAXPROCS=1 is sometimes used to debug concurrency bugs; production should not run this way.
• Setting GOMAXPROCS higher than available cores does not increase parallelism; it adds overhead.

Modern Go and cgroups¶

Go 1.22+ on Linux respects the cgroup CPU quota when computing GOMAXPROCS. Go 1.23+ may adjust automatically when the quota changes. Senior engineers in container environments check runtime.GOMAXPROCS(0) at startup to confirm the value matches the container's actual quota.

Measuring per--cpu performance¶

Always benchmark with -cpu=1,2,4,8. A parallel design that wins at 8 cores but loses at 1 is dangerous: small-instance deployments or scheduling jitter can hit the lossy regime.

GC interactions with goroutine count¶

The Go garbage collector scales with heap size and allocation rate, not goroutine count directly. But goroutine count affects allocation:

• More goroutines, each allocating per item, multiplies the allocation rate.
• More goroutines, each with their own buffer, increases the live heap.
• More goroutines means more stacks. Stacks are not free; each is at least 2 KB initially, growing as needed.

Concrete: a parallel decoder with allocator-per-worker¶

type Worker struct {
    buf []byte // 4 KB per worker
}

workers := make([]*Worker, runtime.GOMAXPROCS(0))

8 workers × 4 KB = 32 KB. Negligible. But:

type Worker struct {
    cache map[uint32]Item // grows with input
}

Now each worker has its own cache; total memory is 8× a single shared cache. If items are global (same set of inputs), per-worker caching hurts both memory and locality.

GC pause amplification¶

A high-frequency allocator (a parallel parser allocating one struct per row) under 8 workers issues 8× the allocations. The GC's mark cost grows roughly with live heap; if the live set also grew, pauses grow.

A senior engineer instruments GC via runtime.ReadMemStats, runtime/metrics, and the trace tool. Goals:

• Total time in GC < 5% of wall clock.
• p99 pause < 10 ms (or whatever the SLO demands).
• Allocations per request close to constant as load grows.

If a concurrency change worsens any of these, the change is suspect even if microbenchmarks look better.

Self-assessment¶

Test your senior-level grasp:

1. A teammate proposes parallelising a []int sum over 8 workers. The slice has 10 elements. What do you say?
2. A trace shows 8 worker goroutines spending most of their wall time in chan recv. What is the likely problem?
3. A benchmark shows a 2× speedup on an 8-core developer laptop and a 0.7× regression in a 2-CPU production pod. What is the explanation?
4. runtime.GOMAXPROCS(0) is 64 in a 4-CPU container. Why? How do you fix it?
5. A sync.Pool was added for []byte allocations and shows a 10% speedup on BenchmarkX. Profile shows GC time is unchanged. What might be going on?
6. False sharing makes a "parallel" design slower than the serial. Name two ways to detect it.
7. A team claims "we use lock-free." How do you challenge the claim?
8. Amdahl's law says max speedup is 5× even with 32 cores. Explain in one sentence.
9. A goroutine handle leak grows the heap by 2 MB/min. Where do you look first?
10. A PR adds 4 goroutines to a 100-µs request and shows -2% latency in microbench. Should you merge?

If you can answer all 10 in a few sentences and cite the underlying mechanism (cache line, scheduler tick, GC pace), you are at senior level. If not, the rest of this section's writings will fill the gaps.

Summary¶

Senior-level practice on premature concurrency optimisation rests on three pillars:

1. Physical mechanism awareness. You know why concurrency might hurt: scheduler overhead, context switches, false sharing, GC pressure, cache effects. You can predict the rough magnitude.
2. Measurement discipline. You demand benchmarks, profiles, and benchstat output before merging. You refuse "I think it is faster" as a justification.
3. Cultural leadership. You set the bar for your team. You build the norms, the runbooks, the gates. You make discipline easy to follow and the alternative hard to slip through.

When all three are in place, the team writes concurrent code only when the workload calls for it, and the codebase contains only optimisations that have earned their keep. When any one is missing, the codebase accumulates layers of "performance" that nobody can justify, nobody can remove, and nobody can prove worked.

The next file, professional.md, extends this thinking to production: SLOs, profile-driven workflows, profile-and-then-decide pipelines, and the deeper architectural decisions about when to scale a service horizontally vs. add more concurrency within a process.

Appendix A: senior-level reading list¶

• Knuth's original paper and the misquote: Structured Programming with go to Statements, 1974.
• Brendan Gregg, Systems Performance, chapters on CPU and memory.
• The Go runtime source: src/runtime/proc.go, src/runtime/chan.go, src/runtime/mgc.go. These files are readable Go; the comments are extensive.
• Dmitry Vyukov's writings on lock-free algorithms (1024cores.net), particularly the cost analyses.
• Ulrich Drepper, What Every Programmer Should Know About Memory. Old but the mechanisms are unchanged.
• Cliff Click, Why Locks Aren't a Bottleneck on x86. A useful counter to "locks are slow" folklore.

These are not assigned reading; they are the senior engineer's reference shelf when a design conversation calls for a specific number or a specific mechanism. Cite them in PR reviews when you want to convince a teammate that their cache-line intuition is wrong.

Appendix B: a one-page rubric for "should this be concurrent?"¶

Print this and tape it above the desk:

1. What is the per-item work in nanoseconds?
2. < 100 ns: no concurrency.
3. 100 ns – 1 µs: serial; consider batching to make work-per-item > 1 µs if parallelism is required.
4. 1 µs – 100 µs: candidate for concurrency if there is enough total work.

5. 100 µs: concurrency usually wins; bound the pool.

6. What is the total work in seconds?
7. < 1 ms total: never concurrency; the goroutine spawn is half the work.
8. 1 ms – 100 ms total: concurrency only if per-item work is in the candidate band.

9. 100 ms total: concurrency is worth investigating.

10. Are the items independent?
11. No: do not bother; the coordination will dominate.
12. Yes: candidate.
13. Is the bottleneck CPU, I/O, or memory?
14. CPU: parallelism bounded by cores.
15. I/O on many endpoints: parallelism bounded by slowest endpoint.
16. I/O on one endpoint: parallelism is a queue, not a speedup.
17. Memory: parallelism often hurts.
18. Have you benchmarked the sequential version with the obvious algorithmic wins (batching, avoiding copies, reusing buffers)?
19. No: do that first. The sequential version is almost always better than you think.
20. Yes: now you can fairly compare.
21. Have you computed an Amdahl estimate?
22. No: do it on a napkin.
23. Yes: is the speedup > 1.3×? If not, skip the change.
24. Have you written the benchmark?
25. No: write it first.
26. Yes: run with -count=20 -cpu=1,2,4,8.
27. Have you run benchstat?
28. No: do it.
29. Yes: is the win significant at p < 0.05?
30. Have you profiled before and after?
31. No: do it. The profile is the proof.
32. Yes: can you point to the function that got faster?
33. Have you considered the maintenance cost?
    • The team will read this code for years. A 10% speedup that doubles complexity is rarely worth it.

If you can say yes to every item with evidence, the PR is ready for review. If not, fix the gap before opening the PR.

Appendix C: anti-pattern catalogue¶

A taxonomy of premature concurrency optimisations seen in real Go services, with the typical fix:

Each row is a topic to discuss in a postmortem when it appears. Each row is a paragraph to add to the team's design review checklist.

Appendix D: a sample benchmark suite¶

A reference structure for benchmarking a concurrency-related change:

package mywork_test

import (
    "context"
    "math/rand"
    "runtime"
    "sync"
    "testing"
    "time"

    "example.com/mywork"
)

func makeItems(n int, seed int64) []mywork.Item {
    r := rand.New(rand.NewSource(seed))
    items := make([]mywork.Item, n)
    for i := range items {
        items[i] = mywork.Item{
            ID:    int64(i),
            Value: r.Float64(),
            Name:  randomString(r, 8, 64),
        }
    }
    return items
}

func randomString(r *rand.Rand, minLen, maxLen int) string {
    n := minLen + r.Intn(maxLen-minLen)
    b := make([]byte, n)
    for i := range b {
        b[i] = byte('a' + r.Intn(26))
    }
    return string(b)
}

func BenchmarkSequential(b *testing.B) {
    sizes := []int{100, 1_000, 10_000, 100_000}
    for _, n := range sizes {
        items := makeItems(n, 42)
        b.Run(sizeLabel(n), func(b *testing.B) {
            b.ReportAllocs()
            b.ResetTimer()
            for i := 0; i < b.N; i++ {
                _ = mywork.ProcessSequential(items)
            }
        })
    }
}

func BenchmarkParallel(b *testing.B) {
    sizes := []int{100, 1_000, 10_000, 100_000}
    workers := []int{1, 2, 4, 8}
    for _, n := range sizes {
        items := makeItems(n, 42)
        for _, w := range workers {
            b.Run(sizeLabel(n)+"/W"+itoa(w), func(b *testing.B) {
                b.ReportAllocs()
                b.ResetTimer()
                for i := 0; i < b.N; i++ {
                    _ = mywork.ProcessParallel(context.Background(), items, w)
                }
            })
        }
    }
}

func sizeLabel(n int) string {
    if n >= 1000 {
        return "N" + itoa(n/1000) + "k"
    }
    return "N" + itoa(n)
}

func itoa(n int) string { return strconvItoa(n) }

func BenchmarkSequentialUnderLoad(b *testing.B) {
    items := makeItems(10_000, 42)
    var bg sync.WaitGroup
    stop := make(chan struct{})
    for i := 0; i < runtime.GOMAXPROCS(0); i++ {
        bg.Add(1)
        go func() {
            defer bg.Done()
            for {
                select {
                case <-stop:
                    return
                default:
                    busy()
                }
            }
        }()
    }
    defer func() { close(stop); bg.Wait() }()

    b.ReportAllocs()
    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        _ = mywork.ProcessSequential(items)
    }
}

func busy() {
    // a small amount of CPU work to simulate other load
    _ = time.Now().UnixNano()
}

This sketch (in real code you would also use b.SetParallelism, b.RunParallel where appropriate) covers:

• Size sweep
• Worker-count sweep
• Allocation reporting
• Background load (so you see how the benchmark behaves under shared CPU)

Run with -count=20 -cpu=1,2,4,8 -benchtime=2s and feed into benchstat.

Appendix E: case stories of premature optimisation reversals¶

Three short stories from real teams (sanitised).

Story 1: the parallel CSV exporter¶

A reporting service generated CSV exports. The original implementation used a single goroutine to write rows to an io.Writer. A developer parallelised row generation across 8 workers writing to a channel, with a single consumer writing to the file.

Performance: 1.4× faster on a developer laptop. Production: 0.9× slower (regression) on the 2-CPU app pod.

Diagnosis: the worker goroutines spent most of their time blocked on the channel send (the file writer was slow, 50 MB/s). On the laptop the disk was fast and the bottleneck was actually the row generation; on production the disk was slow (network-attached) and the bottleneck was the writer. The parallel design moved CPU off the critical path on a fast disk but added scheduling overhead on a slow disk.

Resolution: revert to sequential. The team added a per-deployment benchmark that runs against the target environment, not a developer laptop.

Story 2: the lock-free deduplicator¶

A team built a custom lock-free hash set to deduplicate event IDs. The set was used by an HTTP handler that ingested ~1000 events/sec.

Performance: a microbenchmark showed 1.8× over a sync.Map. Production: identical throughput, increased p99 latency by 30%.

Diagnosis: the lock-free set was correct but had pathological behaviour under bursts of similar keys (it used linear probing and burst hot-spots caused long probe chains). sync.Map did not have this issue. The microbenchmark used random keys; production keys were not random.

Resolution: revert to sync.Map. Add a benchmark that uses the actual key distribution captured from production logs.

Story 3: the prematurely sharded queue¶

A team replaced a single Go channel with a 16-way "sharded queue" (each shard a channel, producers round-robin by key). The goal was to scale throughput.

Performance: microbenchmark showed flat performance; the team merged anyway because "it scales better."

Two years later: no measurable contention on the original channel; the shard hash adds 100 ns per send; 16 channels mean 16 garbage-collection roots; iteration across shards is impossible without a custom collector. Total code: 280 lines.

Resolution: replaced with a single chan and a sync.Mutex around the rare administrative operations. Net: removed 280 lines, no performance regression.

What the stories have in common¶

• The original developer believed the parallel design was faster.
• The benchmark was misleading: wrong inputs, wrong environment, wrong dimension.
• The simpler version was eventually shown to be at least as good.
• The cost of the parallel design was paid in maintenance, not in CPU.

Senior engineers prevent these stories from being written, by requiring evidence at merge time and by maintaining a culture where "I removed the optimisation because it did not pay" is praised.

Appendix F: a half-page glossary¶

• Amdahl's law: ceiling on speedup from parallelism given a serial fraction.
• Backpressure: signalling upstream to slow down when a downstream stage is overwhelmed.
• Cache line: smallest unit the CPU cache moves at a time; 64 B on x86_64.
• Context switch: handoff from one runnable thread/goroutine to another; ~200 ns for goroutines, ~5 µs for OS threads.
• False sharing: two cores writing to different fields in the same cache line, causing coherence traffic.
• GOMAXPROCS: the number of Ps; effectively the parallel-execution slots.
• Karp-Flatt metric: experimentally determined serial fraction from measured speedups.
• Locality: how close (in address or time) related accesses are; high locality means warm caches.
• NUMA: non-uniform memory access; multi-socket boxes where memory has a "home" socket.
• pprof: Go's profiler; samples stacks for CPU, allocation, mutex, block.
• Preemption: forcibly suspending a goroutine at a safe point; asynchronous since Go 1.14.
• Premature optimisation: any optimisation made without measurement that justifies it.
• trace: Go's tracer; records every scheduler and GC event for visualisation.
• Work stealing: a P with an empty runq steals goroutines from a non-empty P.

Appendix G: end-of-chapter checklist¶

If you take three things from this senior-level chapter:

1. Concurrency has a physical cost. Memorise the numbers: 200 ns goroutine switch, ~1 µs mutex contention, 30 ns cache-line bounce.
2. Measurement-first is non-negotiable. No benchmark, no profile, no merge.
3. Culture matters. Your team's discipline is the durable thing. The code you write today will be edited by people who learned from you. Teach them by example: write the benchmark first, run benchstat, attach the pprof.

When in doubt, prefer the simpler design. The next engineer to maintain the code will thank you, and the production system will be no slower for the choice.

Appendix H: extended deep-dive — false sharing, demonstrated¶

A small reproducible Go program that demonstrates false sharing on hardware that has 64 B cache lines:

package main

import (
    "fmt"
    "runtime"
    "sync"
    "sync/atomic"
    "time"
)

type Adjacent struct {
    A int64
    B int64
}

type Padded struct {
    A int64
    _ [56]byte
    B int64
    _ [56]byte
}

func runAdjacent(iterations int64) time.Duration {
    var counters Adjacent
    var wg sync.WaitGroup
    wg.Add(2)
    start := time.Now()
    go func() {
        defer wg.Done()
        for i := int64(0); i < iterations; i++ {
            atomic.AddInt64(&counters.A, 1)
        }
    }()
    go func() {
        defer wg.Done()
        for i := int64(0); i < iterations; i++ {
            atomic.AddInt64(&counters.B, 1)
        }
    }()
    wg.Wait()
    return time.Since(start)
}

func runPadded(iterations int64) time.Duration {
    var counters Padded
    var wg sync.WaitGroup
    wg.Add(2)
    start := time.Now()
    go func() {
        defer wg.Done()
        for i := int64(0); i < iterations; i++ {
            atomic.AddInt64(&counters.A, 1)
        }
    }()
    go func() {
        defer wg.Done()
        for i := int64(0); i < iterations; i++ {
            atomic.AddInt64(&counters.B, 1)
        }
    }()
    wg.Wait()
    return time.Since(start)
}

func main() {
    runtime.GOMAXPROCS(2)
    const iters = int64(1e8)
    adj := runAdjacent(iters)
    pad := runPadded(iters)
    fmt.Printf("adjacent: %s\npadded:   %s\nratio:    %.2fx\n",
        adj, pad, float64(adj)/float64(pad))
}

On a typical x86_64 box, the adjacent version is 4–8× slower. The padded version is bounded by the speed of the atomic instructions themselves. Show this program to a developer once and the concept of false sharing becomes intuition.

Variant: write-then-read sharing¶

False sharing also affects reads, not just writes. A producer writing to a field and a consumer reading a different field on the same line still exchange the line.

type ProducerConsumer struct {
    seq  int64 // producer writes
    used int64 // consumer reads
}

Same fix: pad. If your design has two roles touching the same struct, separate their data into distinct cache lines.

A common debugging anecdote¶

A team observed that adding a new field to a hot struct slowed the program by 15%. Removing the field restored speed. The new field sat next to a heavily-mutated counter; before the addition, the counter was alone on its line. After, two writers were contending for the same line. The fix: move the new field into a separate struct or pad it explicitly.

The lesson is to be aware of struct layout in code-generated hot paths. go vet -copylocks and fieldalignment (golang.org/x/tools/go/analysis/passes/fieldalignment) can flag obvious issues; nothing flags false sharing automatically — only profiling and benchmarks do.

Appendix I: extended deep-dive — goroutine spawn vs worker pool¶

Two implementations of a parallel map function:

Spawn-per-item¶

func MapSpawn[T any, U any](xs []T, f func(T) U) []U {
    out := make([]U, len(xs))
    var wg sync.WaitGroup
    for i := range xs {
        i := i
        wg.Add(1)
        go func() {
            defer wg.Done()
            out[i] = f(xs[i])
        }()
    }
    wg.Wait()
    return out
}

Bounded worker pool¶

func MapPool[T any, U any](xs []T, f func(T) U, workers int) []U {
    out := make([]U, len(xs))
    var idx atomic.Int64
    var wg sync.WaitGroup
    for w := 0; w < workers; w++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                i := idx.Add(1) - 1
                if int(i) >= len(xs) {
                    return
                }
                out[i] = f(xs[i])
            }
        }()
    }
    wg.Wait()
    return out
}

Benchmark¶

func BenchmarkMap(b *testing.B) {
    xs := make([]int, 100_000)
    for i := range xs {
        xs[i] = i
    }
    f := func(x int) int { return x * 2 } // very cheap

    b.Run("Sequential", func(b *testing.B) {
        for i := 0; i < b.N; i++ {
            out := make([]int, len(xs))
            for j, x := range xs {
                out[j] = f(x)
            }
            _ = out
        }
    })

    b.Run("Spawn", func(b *testing.B) {
        for i := 0; i < b.N; i++ {
            _ = MapSpawn(xs, f)
        }
    })

    b.Run("Pool/8", func(b *testing.B) {
        for i := 0; i < b.N; i++ {
            _ = MapPool(xs, f, 8)
        }
    })
}

Typical results¶

Sequential:      ~75 µs/op
Spawn:          ~92 ms/op   (>1000x slower!)
Pool/8:          ~120 µs/op (1.6x slower than sequential)

Spawn-per-item loses catastrophically because spawning 100,000 goroutines costs orders of magnitude more than the work each does. Pool/8 loses to sequential because the work is too cheap; the atomic increment and the cache traffic outweigh the gain.

When does the parallel pool win?¶

Increase the per-item cost:

f := func(x int) int {
    sum := 0
    for i := 0; i < 1000; i++ {
        sum += i * x
    }
    return sum
}

Now f costs ~1 µs. Results:

Sequential:    ~100 ms/op
Spawn:         ~120 ms/op   (still loses to sequential)
Pool/8:        ~17 ms/op    (5.9x faster than sequential)

The pool wins; the spawn does not (because spawn cost is still per-item). At 10 µs of f, the pool wins by closer to 8×; at 100 ns of f, sequential wins by 2×. The crossover is between 500 ns and 1 µs of per-item work in a typical Go runtime, depending on hardware.

Senior takeaway¶

Always use a bounded worker pool for fan-out, never go per item except in I/O fan-out where each goroutine is long-lived. The bound should be runtime.GOMAXPROCS(0) for CPU work, or a measured higher number for I/O work.

Appendix J: extended deep-dive — channel cost in detail¶

chan is a wonderful primitive for communication. It is not a wonderful primitive for fast data transfer. The internal structure of a buffered chan (in runtime/chan.go):

type hchan struct {
    qcount   uint           // number of elements in queue
    dataqsiz uint           // size of circular queue
    buf      unsafe.Pointer // points to dataqsiz array
    elemsize uint16
    closed   uint32
    timer    *timer
    elemtype *_type
    sendx    uint           // send index
    recvx    uint           // receive index
    recvq    waitq          // receivers waiting
    sendq    waitq          // senders waiting
    lock     mutex          // protects ALL the above
}

Every channel operation acquires lock. On Linux this is a futex-backed mutex; on the uncontended path it is ~10 ns; on the contended path it parks the goroutine for ~1 µs.

A "send" copies the value into the buffer (or directly to a waiting receiver). A "receive" pops from the buffer (or from a waiting sender). Both touch the same hchan struct and acquire the same lock.

What this means¶

• A channel between two goroutines with no contention is ~50 ns per send/recv.
• A channel with many senders or receivers contends on the lock; effective cost rises.
• A channel that becomes "always full" or "always empty" parks one side; ~250 ns per op.
• A channel with select cases has additional bookkeeping for each case.

Faster alternatives¶

For very-high-throughput single-producer single-consumer scenarios (>1 M ops/sec), a custom ring buffer using atomics on indices can outperform chan by 2–4×. The complexity is significant; only do this with a benchmark forcing the issue.

For batching, send a slice of items per send. The per-item amortised cost drops from 50 ns to a few ns. This is almost always the right optimisation if you find yourself fighting channel cost.

// Slow: 1 item per send
ch := make(chan Item, 64)
for _, it := range items {
    ch <- it
}

// Fast: batches of 64 items per send
batches := make(chan []Item, 4)
batch := make([]Item, 0, 64)
for _, it := range items {
    batch = append(batch, it)
    if len(batch) == cap(batch) {
        batches <- batch
        batch = make([]Item, 0, 64)
    }
}
if len(batch) > 0 {
    batches <- batch
}

The batched version pays the channel cost 1/64 as often. Workers receive a slice and iterate locally, keeping data hot in L1.

Appendix K: extended deep-dive — mutex micro-benchmarks¶

What does a sync.Mutex actually cost?

Uncontended¶

func BenchmarkMutexUncontended(b *testing.B) {
    var mu sync.Mutex
    for i := 0; i < b.N; i++ {
        mu.Lock()
        mu.Unlock()
    }
}

Typical result: 8–12 ns/op. The mutex is in L1 the whole time; the lock is a simple CAS.

Contended¶

func BenchmarkMutexContended(b *testing.B) {
    var mu sync.Mutex
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            mu.Lock()
            // very short critical section
            mu.Unlock()
        }
    })
}

Typical result with 8 cores: 200–800 ns/op. The cache line bounces between cores; goroutines occasionally park.

Inside a critical section that does real work¶

func BenchmarkMutexWithWork(b *testing.B) {
    var mu sync.Mutex
    counter := 0
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            mu.Lock()
            for i := 0; i < 100; i++ {
                counter += i
            }
            mu.Unlock()
        }
    })
}

Now the work inside the critical section dominates the mutex cost. Lock acquisition is amortised over the protected work.

RWMutex comparisons¶

func BenchmarkRWMutexReadOnly(b *testing.B) {
    var mu sync.RWMutex
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            mu.RLock()
            mu.RUnlock()
        }
    })
}

Typical result: 50–100 ns/op uncontended (RWMutex is more expensive per-op than Mutex), 200+ ns/op contended.

Senior takeaway¶

sync.RWMutex is not free per read. Its higher per-op cost only pays back when the critical section is long and the read/write ratio is high. For short reads (<100 ns of work inside the lock), sync.Mutex is faster.

Appendix L: extended deep-dive — pprof reading by example¶

Here is a synthetic example of a profile we will read together. A program that processes work with three implementations under load.

package main

import (
    "log"
    "net/http"
    _ "net/http/pprof"
    "runtime"
    "sync"
    "sync/atomic"
    "time"
)

var counter int64

func mutexImpl(items []int) int {
    var mu sync.Mutex
    sum := 0
    var wg sync.WaitGroup
    chunks := splitWork(items, runtime.GOMAXPROCS(0))
    for _, chunk := range chunks {
        chunk := chunk
        wg.Add(1)
        go func() {
            defer wg.Done()
            local := 0
            for _, x := range chunk {
                local += x
            }
            mu.Lock()
            sum += local
            mu.Unlock()
        }()
    }
    wg.Wait()
    return sum
}

func atomicImpl(items []int) int {
    var sum int64
    var wg sync.WaitGroup
    chunks := splitWork(items, runtime.GOMAXPROCS(0))
    for _, chunk := range chunks {
        chunk := chunk
        wg.Add(1)
        go func() {
            defer wg.Done()
            for _, x := range chunk {
                atomic.AddInt64(&sum, int64(x))
            }
        }()
    }
    wg.Wait()
    return int(sum)
}

func sequentialImpl(items []int) int {
    sum := 0
    for _, x := range items {
        sum += x
    }
    return sum
}

func splitWork(items []int, n int) [][]int {
    chunkSize := (len(items) + n - 1) / n
    out := make([][]int, 0, n)
    for i := 0; i < len(items); i += chunkSize {
        end := i + chunkSize
        if end > len(items) {
            end = len(items)
        }
        out = append(out, items[i:end])
    }
    return out
}

func main() {
    go func() { log.Println(http.ListenAndServe("localhost:6060", nil)) }()
    items := make([]int, 1_000_000)
    for i := range items {
        items[i] = i
    }
    for i := 0; ; i++ {
        atomic.AddInt64(&counter, int64(mutexImpl(items)))
        atomic.AddInt64(&counter, int64(atomicImpl(items)))
        atomic.AddInt64(&counter, int64(sequentialImpl(items)))
        if i%100 == 0 {
            log.Println("iter", i, "counter", atomic.LoadInt64(&counter))
            time.Sleep(time.Millisecond)
        }
    }
}

Visit http://localhost:6060/debug/pprof/profile?seconds=30 to grab a CPU profile. Open in go tool pprof -http=:8080 and view "Top". What you will see:

• atomicImpl consumes the most CPU — atomic.AddInt64 in tight loop is a major bottleneck.
• mutexImpl consumes less than atomicImpl because each goroutine accumulates locally and locks once.
• sequentialImpl consumes the least.

Now the question for a senior reviewer: "Is the atomic version actually fastest?" Sometimes the answer is no. Replace BenchmarkAtomic with a benchmark and you may find that on heavily contended hot atomics, the mutex version beats the atomic version by 3×. The reason is exactly the cache-line bouncing we discussed.

The profile tells you what is consuming CPU; it does not tell you whether the strategy is right. Always pair the profile with a benchmark.

Appendix M: extended deep-dive — trace reading by example¶

go test -bench=BenchmarkMap -trace trace.out, then go tool trace trace.out. The trace UI offers:

1. View trace — the timeline view.
2. Goroutine analysis — per-goroutine summaries.
3. Network blocking profile — flame graph of time in network syscalls.
4. Synchronization blocking profile — flame graph of time blocked on chan/mutex.
5. Syscall blocking profile — flame graph of time in syscalls.
6. Scheduler latency profile — time spent in runqueue vs running.

What the timeline tells you¶

If you zoom in on a 10 ms window, you see horizontal lanes (one per P) and stacked colored boxes (one per running goroutine instance). Look for:

• Idle gaps: Ps that are not running anything. If you expected 8-way parallelism and see 4 lanes idle most of the time, your parallelism is half what it could be.
• Synchronisation marks: red lines connecting a sender and a receiver. Many in a row means heavy channel traffic.
• GC stripes: black or pink bands across all Ps during GC. Long ones are a problem.
• Preemption arrows: the runtime preempted a goroutine; rare in well-behaved code.

Goroutine analysis: "where did the time go?"¶

For each goroutine class (grouped by the goroutine's entry function), the trace tool reports:

• Total time alive.
• Time in Grunning (actually executing).
• Time in Grunnable (in the runqueue but not running).
• Time in Gwaiting (blocked on chan, mutex, syscall, network).
• Time in GC.

A worker goroutine that is Grunnable for 80% of its lifetime is starved — there are more runnable goroutines than Ps. A worker that is Gwaiting for 80% is bottlenecked on something else (usually channel input).

Worked: identifying a channel bottleneck¶

A team's parallel renderer had 8 workers but trace showed:

• Worker time: 30% running, 70% in Gwaiting on chan recv.
• Producer goroutine: 95% running, fully saturating one P.

Diagnosis: the producer was the bottleneck. Workers were idle waiting for input. The "parallel" design was actually one-CPU-bound.

Resolution: the producer was reading a file row-by-row and copying each row. Replaced with bufio.Reader.ReadSlice to avoid copies; saturated 4 cores instead of 1. Throughput rose 4×.

The trace told us what to fix. Without it, the team might have added more workers (which would not have helped).

Appendix N: extended deep-dive — benchstat in detail¶

benchstat is the canonical tool for comparing benchmarks. Install with go install golang.org/x/perf/cmd/benchstat@latest.

A typical invocation¶

go test -bench=. -count=20 -cpu=1,2,4,8 > old.txt
# ... change code ...
go test -bench=. -count=20 -cpu=1,2,4,8 > new.txt
benchstat old.txt new.txt

Reading the output¶

                            │   old.txt    │              new.txt               │
                            │    sec/op    │   sec/op     vs base               │
ParallelMap/N1k-8              25.1µ ± 2%   12.8µ ± 1%  -49.0% (p=0.000 n=20)
ParallelMap/N10k-8             254µ  ± 2%   135µ  ± 2%  -46.9% (p=0.000 n=20)
ParallelMap/N100k-8           2.55m  ± 1%   1.36m ± 2%  -46.7% (p=0.000 n=20)
geomean                        253µ          133µ        -47.6%

Key columns:

• sec/op: median time per op, with confidence interval.
• vs base: percent change.
• p: Welch's t-test p-value. p < 0.05 means the change is statistically significant.
• n: number of measurements (-count).

What to look for¶

• Consistent direction: if some sizes are faster and others slower, the change is workload-sensitive. Investigate.
• Significant p: insignificant changes are noise, not improvement.
• Reasonable variance (the ± X%): high variance (>10%) means the machine is noisy or the benchmark is bad.
• Geomean: a single number to summarise; useful for headlines but never the whole story.

Anti-patterns in benchstat reports¶

• Cherry-picking: showing the one benchmark that improved without showing the ones that regressed.
• Bench-shopping: writing a benchmark designed to show the change in a flattering light.
• Ignoring p: claiming 3% improvement when p = 0.4.

A senior reviewer checks all benchmarks, not just the headline; verifies p; and looks at variance.

Appendix O: extended deep-dive — GOMAXPROCS in container environments¶

In a Docker or Kubernetes pod, runtime.NumCPU() historically returns the host CPU count, not the cgroup quota. So a pod with cpu: 2 on an 8-CPU node ran with GOMAXPROCS=8, scheduling 8 Ps onto 2 CPUs. The result: each P-quantum was tiny, scheduler overhead spiked, throughput dropped.

Manifestations¶

• p99 latency higher than p99 on the same code with GOMAXPROCS=2.
• High runtime.findrunnable in CPU profile.
• Many short bursts of CPU activity interleaved with kernel scheduler wait.

Fixes¶

1. Set GOMAXPROCS explicitly from a config:

if v := os.Getenv("GOMAXPROCS"); v != "" {
    if n, err := strconv.Atoi(v); err == nil {
        runtime.GOMAXPROCS(n)
    }
}

1. Use automaxprocs (go.uber.org/automaxprocs): this library reads the cgroup quota and sets GOMAXPROCS accordingly. Until Go's built-in support matures, this is a common dependency.
2. Upgrade to Go 1.22+: starting Go 1.22 the runtime is cgroup-aware.

Side note: hyperthreading and GOMAXPROCS¶

On Intel CPUs with hyperthreading, 2 "logical CPUs" share one physical core. runtime.NumCPU() reports the logical count; GOMAXPROCS defaults to it. For CPU-bound workloads, this can hurt: 2 Ps fighting for one physical core perform worse than 1 P fully using it.

Some teams set GOMAXPROCS to half the logical count for pure CPU work. Most teams leave it alone. Senior engineers measure both and pick the winner per workload.

Appendix P: extended deep-dive — GC tuning under concurrency¶

The Go GC is a concurrent mark-and-sweep collector with:

• A trigger heap size (GOGC, default 100, meaning collect when the live heap doubles).
• A pacer that targets a fixed CPU share for GC.
• Concurrent marking that runs on application goroutines (a small fraction of mark work is done by application goroutines via "assist credit").

Concurrency interactions¶

• High allocation rate from many goroutines triggers GC more often.
• High live-heap from many goroutines (each keeping per-worker state) raises the trigger.
• GC assist can steal cycles from application goroutines under pressure; this shows up as runtime.gcDrain* in CPU profiles.

Tuning levers¶

• GOGC=200 (default *2): less frequent GC at the cost of higher memory.
• GOGC=50: more frequent GC at the cost of more CPU spent in GC.
• GOMEMLIMIT (Go 1.19+): a soft cap on total memory; the GC runs more aggressively as you approach the limit, preventing OOM.
• debug.SetGCPercent, debug.SetMemoryLimit: runtime equivalents.

How concurrency affects choice¶

A service that fan-outs and allocates per item is more sensitive to GC tuning than a sequential service. Senior engineers benchmark with GOGC=20 (aggressive) and GOGC=400 (relaxed) to see whether the concurrent version is paying excessive GC cost.

Appendix Q: deep-dive — channel select cost¶

A select with multiple cases has higher overhead than a single channel op. The runtime evaluates each case, ordering them randomly, and may park on multiple wait queues.

select {
case ch1 <- x:
case ch2 <- x:
case <-ctx.Done():
}

This costs ~150–300 ns even uncontended. A two-case select on hot channels is fine. A 10-case select in a tight loop is expensive.

Common pattern: ctx cancellation¶

select {
case ch <- item:
case <-ctx.Done():
    return ctx.Err()
}

This is the right idiom. The overhead pays for cancellability. Do not avoid it for perf.

Anti-pattern: select for state machine¶

for {
    select {
    case x := <-input:
        // ...
    case <-timer.C:
        // ...
    case <-ctx.Done():
        return
    }
}

Acceptable. But if input is hot and timer is rare, the select cost is paid on every input. A separate timer goroutine sending on a dedicated channel avoids the multi-case cost — at the price of an extra goroutine. Measure both.

Appendix R: deep-dive — sync.Map vs alternatives¶

sync.Map is not a general-purpose concurrent map. The documentation is explicit:

The Map type is optimized for two common use cases: 1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, 2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys.

For other workloads — many writers, broad key set — a mutex-protected map is usually faster.

When sync.Map wins¶

• A cache with stable keys after warmup.
• A registry where each key has a single writer.

When sync.Map loses¶

• Frequent writes across all keys.
• Many cores fighting over the same keys.
• Need for iteration or size queries.

Senior pattern¶

Start with a sync.Mutex + map. Benchmark. Switch to sync.Map only if the benchmark shows a meaningful win for your access pattern.

Appendix S: deep-dive — when concurrency actually wins¶

To balance the document, here are the cases where concurrency clearly wins. A senior engineer should recognise these too.

1. I/O fan-out to independent endpoints. Network-bound calls overlap freely.
2. Embarrassingly parallel CPU work with >>1 µs per item. Image processing, hash, compression.
3. Pipelining with naturally rate-matched stages. Each stage saturates a different resource (CPU, disk, network).
4. Bulk processing with batched coordination. Coordination cost amortised over batches.
5. Concurrent I/O for streaming. A goroutine per long-lived connection in a low-load server.
6. Concurrent map-reduce on big data. When the data is in RAM and the reduction is associative.

In all of these, the work is large compared to coordination, the items are independent, and the bottleneck is parallelisable. When all three hold, concurrency wins decisively.

Appendix T: deep-dive — the "context-switch tax"¶

A famous internal Google diagram (variously attributed) shows the cost of various operations. For CPU-bound concurrent code, the relevant number is roughly:

goroutine switch: ~200 ns
useful work per switch: needs to be > 200 ns to break even

If your "useful work" is iterating one item of a slice (10 ns), the tax is 20×. You lose.

If your "useful work" is a function that does 100 µs of computation, the tax is 0.2%. You win.

The lesson is to increase the granularity of work-per-goroutine until the tax is negligible. This is why batching is the most common single fix for "my parallel code is slow."

Appendix U: more on Amdahl, with diagrams¶

Amdahl's law:

                1
S(N) = -------------------
        (1 - p) + (p / N)

Plotted as N grows, S(N) asymptotes to 1 / (1 - p). For p = 0.9 the asymptote is 10×; for p = 0.99 it is 100×.

What this means in practice:

• A workload that is 50% serial can never go faster than 2× regardless of cores.
• A workload that is 90% serial can never go faster than 1.11×.
• A workload that is 1% serial caps at 100×.

The lesson: find the serial fraction and reduce it before adding cores. A 95% parallel workload at 16 cores gets 9.1×. Reducing serial to 99% gets 13.9×. Reducing serial fraction is often higher leverage than adding cores.

Diagrams¶

N      p=0.5    p=0.9    p=0.99   p=0.999
1      1.0      1.0      1.0      1.0
2      1.33     1.82     1.98     1.998
4      1.60     3.08     3.88     3.988
8      1.78     4.71     7.48     7.94
16     1.88     6.40     13.91    15.76
32     1.94     7.80     24.43    30.62
64     1.97     8.77     39.26    57.06
128    1.98     9.34     56.39    101.6
infty  2.0      10.0     100.0    1000.0

A senior engineer looks at this table mentally before adding cores. "Is my workload above 99% parallel? If not, going beyond 16 cores is wasted."

Appendix V: deep-dive — Go-specific Amdahl pitfalls¶

In Go, the serial fraction is often invisible. Three common hidden serial fractions:

1. The fan-out producer¶

g, ctx := errgroup.WithContext(ctx)
for _, item := range items { // serial loop
    item := item
    g.Go(func() error { ... })
}
return g.Wait()

The for loop iterates serially. If you have 1 million items and the goroutine-spawn cost is 1 µs, the producer takes 1 second before any work starts. Workers finish quickly; the producer is the wall time.

Fix: chunk the input, spawn a fixed number of goroutines each iterating its own chunk.

2. The fan-in aggregator¶

out := make(chan Result, len(items))
for _, item := range items {
    item := item
    go func() { out <- process(item) }()
}
results := make([]Result, 0, len(items))
for range items {
    results = append(results, <-out)
}

The aggregator is serial — one goroutine reading the channel. If each process is 100 µs and there are 1 M items, the aggregator runs 1 M times per 100 µs = 10000 ns per receive. If channel recv is ~50 ns, fine; but if any iteration of the aggregator does meaningful work, it becomes the bottleneck.

Fix: write to indexed slice; no aggregator needed.

3. The closing channel¶