Concurrency — Optimize & Reconcile¶

Correctness and performance are not opposites, but they have an order. A racy program that is fast is worthless; it produces wrong answers quickly. So the discipline is fixed: make it correct, measure it, then reduce coordination cost. Every scenario below starts from a correct-but-slow (or fast-but-wrong) concurrent design, shows the measurement that reveals the problem, and resolves it with the cheapest synchronization that preserves correctness. Numbers are order-of-magnitude figures from commodity x86-64 hardware (Skylake-class, 64-byte cache lines, ~3 GHz); treat them as ratios to reproduce on your own box, not constants.

Table of Contents¶

1. Lock contention turns a parallel map into a serial one
2. Read-write lock when reads dominate
3. False sharing destroys per-thread counters
4. Mutex vs atomic vs channel: choosing the cheapest primitive
5. Thread-pool sizing: CPU-bound vs I/O-bound
6. More parallelism makes it slower (Universal Scalability Law)
7. Batching to amortize synchronization
8. Context-switch and goroutine/thread overhead
9. The Python GIL: multiprocessing vs native extensions
10. Back-pressure prevents unbounded memory growth
11. Lock-free stack vs mutex stack: when CAS wins and when it loses
12. Per-CPU / thread-local sharding for hot counters
13. Lock held across I/O serializes everything
14. Channel fan-out with the wrong buffer size

Rules of Thumb · Related Topics

The decision flow¶

Scenario 1 — Lock contention turns a parallel map into a serial one¶

You parallelized an aggregation across 16 worker threads. CPU utilization sits at ~110% (barely more than one core) instead of the ~1600% you expected. Throughput is worse than the single-threaded version.

class WordCounter {
    private final Map<String, Integer> counts = new HashMap<>();
    private final Object lock = new Object();

    void count(String word) {
        synchronized (lock) {              // every thread serializes here
            counts.merge(word, 1, Integer::sum);
        }
    }
}

Sixteen threads each grab the same monitor for every word. The critical section is the entire work, so the threads run strictly one at a time — plus they pay lock acquisition, cache-line transfer of the monitor word, and OS park/unpark when contended. You have built a parallel program that runs slower than serial.

class WordCounter {
    private final ConcurrentHashMap<String, LongAdder> counts = new ConcurrentHashMap<>();

    void count(String word) {
        counts.computeIfAbsent(word, k -> new LongAdder()).increment();
    }
}

Scenario 2 — Read-write lock when reads dominate¶

A configuration cache is read on every request (~200k reads/sec) and reloaded every 30 seconds. It is guarded by a plain mutex. Under load, request latency p99 spikes even though writes are rare.

type ConfigCache struct {
    mu  sync.Mutex
    cfg *Config
}

func (c *ConfigCache) Get() *Config {
    c.mu.Lock()
    defer c.mu.Unlock()
    return c.cfg            // readers serialize against each other for no reason
}

A sync.Mutex is exclusive: two readers that only want to observe cfg still block each other. With 200k reads/sec across many goroutines, the mutex becomes a serialization point even though no reader mutates anything.

func (c *ConfigCache) Get() *Config {
    c.mu.RLock()
    defer c.mu.RUnlock()
    return c.cfg
}

type ConfigCache struct {
    cfg atomic.Pointer[Config]
}
func (c *ConfigCache) Get() *Config   { return c.cfg.Load() }   // wait-free read
func (c *ConfigCache) Set(n *Config)  { c.cfg.Store(n) }        // atomic publish

Scenario 3 — False sharing destroys per-thread counters¶

You gave each worker its own counter to avoid contention — yet 8 threads run barely faster than 1. There is no shared lock, no shared variable each thread writes. So why?

type Counters struct {
    n [8]int64    // one slot per worker; 8 * 8 bytes = 64 bytes
}
// worker i does, in a tight loop:  c.n[i]++

All 8 int64s sit in the same 64-byte cache line. When core 0 writes n[0], the MESI protocol invalidates that line in every other core's L1. Core 1 writing n[1] then takes a cache miss to re-fetch the line, writes, and invalidates everyone again. The line ping-pongs between cores on every increment though the threads touch logically disjoint data. This is false sharing.

type paddedCounter struct {
    n   int64
    _   [56]byte    // pad: 8 (int64) + 56 = 64 bytes, one full line
}
type Counters struct {
    c [8]paddedCounter
}

@jdk.internal.vm.annotation.Contended
static final class Cell { volatile long value; }

Scenario 4 — Mutex vs atomic vs channel: choosing the cheapest primitive¶

A request counter is incremented on every request. The team is debating: mutex, atomic, or a channel feeding a single owner goroutine? Each is correct. Which is cheapest, and when?

// Option A: mutex
mu.Lock(); count++; mu.Unlock()
// Option B: atomic
atomic.AddInt64(&count, 1)
// Option C: channel to an owner goroutine
incCh <- 1

Scenario 5 — Thread-pool sizing: CPU-bound vs I/O-bound¶

A service uses a fixed pool of 4 threads (machine has 4 cores) for everything. The CPU-bound image-resize endpoint is fine, but the I/O-bound "call 3 upstream APIs" endpoint achieves terrible throughput — the 4 threads spend most of their time blocked on sockets, and requests queue up behind them.

threads ≈ cores × targetUtilization × (1 + waitTime / computeTime)

ExecutorService cpuPool = Executors.newFixedThreadPool(Runtime.getRuntime().availableProcessors());
ExecutorService ioPool  = Executors.newFixedThreadPool(64);   // sized via Little's Law

Scenario 6 — More parallelism makes it slower (Universal Scalability Law)¶

You added threads expecting linear speedup. From 1→4 threads it scaled well (~3.6×). From 8→16 it regressed — 16 threads were slower than 8. Adding hardware made the system worse.

            N
C(N) = ---------------------------
       1 + α(N−1) + βN(N−1)

Scenario 7 — Batching to amortize synchronization¶

A pipeline pushes one item at a time onto a shared, locked queue: 1,000,000 items/sec, each enqueue taking a lock. The lock is the bottleneck even though the per-item work is trivial.

for (Item item : items) {
    queue.put(item);     // one lock acquisition per item
}

List<Item> batch = new ArrayList<>(1024);
for (Item item : items) {
    batch.add(item);
    if (batch.size() == 1024) {
        queue.putAll(batch);   // one lock acquisition per 1024 items
        batch.clear();
    }
}
if (!batch.isEmpty()) queue.putAll(batch);

var local int64
for _, x := range chunk { local += work(x) }
atomic.AddInt64(&total, local)   // one atomic per chunk, not per element

Scenario 8 — Context-switch and goroutine/thread overhead¶

A naive design spawns one OS thread per task and ends up creating 50,000 threads. The box thrashes: memory explodes, the scheduler spends most of its time switching, and throughput collapses.

sem := make(chan struct{}, runtime.GOMAXPROCS(0)*2) // bound concurrency for CPU work
for _, task := range tasks {
    sem <- struct{}{}
    go func(t Task) { defer func() { <-sem }(); process(t) }(task)
}

Scenario 9 — The Python GIL: multiprocessing vs native extensions¶

A CPU-bound Python function (numeric crunching) is parallelized with threading across 8 threads on an 8-core box. It runs no faster than single-threaded — sometimes slightly slower.

from threading import Thread
threads = [Thread(target=crunch, args=(chunk,)) for chunk in chunks]
for t in threads: t.start()
for t in threads: t.join()   # 8 threads, ~1 core of throughput

from concurrent.futures import ProcessPoolExecutor
with ProcessPoolExecutor(max_workers=8) as ex:
    results = list(ex.map(crunch, chunks))

# Cython:  with nogil:  for i in range(n): ...

Scenario 10 — Back-pressure prevents unbounded memory growth¶

A producer reads a file (or Kafka topic) far faster than the consumer can process. With an unbounded queue between them, memory climbs until OOM. With no queue, you lose data or block badly. The "fix" of making the queue bigger just delays the crash.

ch := make(chan Item)          // ... or worse, an unbounded slice the producer appends to
go func() {
    for item := range source { ch <- item }   // producer races ahead
}()
for item := range ch { slowProcess(item) }     // consumer can't keep up

ch := make(chan Item, 1024)    // bounded
go func() {
    for item := range source {
        ch <- item             // BLOCKS when full → producer is throttled to consumer's pace
    }
    close(ch)
}()
for item := range ch { slowProcess(item) }

Scenario 11 — Lock-free stack vs mutex stack: when CAS wins and when it loses¶

A team replaced a mutex-guarded stack with a lock-free CAS-based one "because lock-free is faster," and benchmarks under their real workload showed it was slower and burned CPU spinning.

// Lock-free push (Treiber stack)
for {
    old := head.Load()
    node.next = old
    if head.CompareAndSwap(old, node) { break }   // retries on contention
}

Scenario 12 — Per-CPU / thread-local sharding for hot counters¶

A metrics counter (requestsTotal) is incremented on every request across all cores. Even as a single atomic, at millions of increments/sec it's a bottleneck: every core fights for that one cache line.

var requestsTotal int64
// hot path on every core:
atomic.AddInt64(&requestsTotal, 1)

LongAdder requestsTotal = new LongAdder();
requestsTotal.increment();        // hits a striped, @Contended cell → no cross-core contention
long total = requestsTotal.sum(); // aggregates cells; reads are rarer than writes

type shardedCounter struct {
    shards [256]paddedCounter   // padded to avoid false sharing (Scenario 3)
}
func (c *shardedCounter) inc(shardIdx int) { c.shards[shardIdx&255].n++ }  // owner-only write
func (c *shardedCounter) sum() (t int64)   { for i := range c.shards { t += atomic.LoadInt64(&c.shards[i].n) }; return }

Scenario 13 — Lock held across I/O serializes everything¶

A cache-fill function holds the lock while it calls the database. Under load, every thread that wants the cache queues behind the one thread doing a slow network round-trip.

synchronized (lock) {
    Value v = cache.get(key);
    if (v == null) {
        v = db.load(key);        // 20 ms network call — INSIDE the lock!
        cache.put(key, v);
    }
    return v;
}

Value v = cache.get(key);          // fast read (use a concurrent map)
if (v != null) return v;
Value loaded = db.load(key);       // slow I/O — NO lock held
return cache.computeIfAbsent(key, k -> loaded);  // brief, in-memory only

v, _, _ := group.Do(key, func() (any, error) { return db.Load(key) }) // one in-flight load per key

Scenario 14 — Channel fan-out with the wrong buffer size¶

A fan-out worker pool uses an unbuffered channel to distribute work. CPU sits idle and throughput is low, because every producer send must rendezvous with a ready worker — producers block waiting for handoff and workers block waiting for the next item, so the pipeline is full of stalls.

jobs := make(chan Job)            // unbuffered: every send waits for a receiver
for i := 0; i < workers; i++ { go worker(jobs) }
for _, j := range allJobs { jobs <- j }   // producer blocks on each send

jobs := make(chan Job, workers*2)   // buffer absorbs timing jitter

Rules of Thumb¶

1. Correct first, fast second — always. A data race that produces wrong results occasionally is worse than a slow program. Never remove synchronization to gain speed; reduce contention instead.
2. Measure, don't guess. Use a lock profiler (async-profiler/JFR for Java, go test -bench + pprof + -race, Python cProfile/py-spy). Idle CPU + high lock-wait = contention; high cs in vmstat = too many threads; perf c2c HITM = false sharing.
3. Pick the least-powerful primitive that's correct: atomic (single word) < mutex (multi-field invariant) < channel/queue (ownership transfer + back-pressure). Uncontended they're all "fast enough"; contention is the 10–100× multiplier.
4. Reduce contention before changing primitives. Shard (LongAdder, striped maps, per-thread/per-CPU state) so threads stop touching the same cache line. A sharded mutex beats a contended atomic.
5. Pad concurrently-written hot fields to a cache line (@Contended, manual padding) to kill false sharing — but only the fields you measured, since padding wastes memory.
6. Hold locks only over in-memory state, never across I/O. The critical section duration is your global serialization floor.
7. Size pools by workload: CPU-bound ≈ #cores; I/O-bound by Little's Law (cores × util × (1 + wait/compute)), or stop blocking threads entirely (goroutines, virtual threads, asyncio).
8. Respect the Universal Scalability Law: more parallelism eventually goes retrograde due to O(N²) coherency cost. Find the measured peak and cap there.
9. Batch to amortize per-op synchronization cost — but cap batch size + add a flush timer so latency doesn't suffer.
10. Bound every buffer. Unbounded queues convert a rate mismatch into an OOM. A bounded channel/queue is your back-pressure.
11. Lock-free ≠ fast. CAS loops can lose to parking mutexes under high single-point contention. Prefer removing contention (sharding) to going lock-free, and use library structures over hand-rolled ones.
12. Python: the GIL means threading only helps I/O. For CPU parallelism use multiprocessing, native extensions that release the GIL (NumPy/Cython nogil), or a free-threaded build.

Related Topics¶

• README.md — the positive concurrency rules this file optimizes against.
• find-bug.md — race conditions and deadlocks to spot before you optimize (correctness gate).
• professional.md — production-grade concurrency review standards.
• ../14-immutability/README.md — immutable snapshots eliminate read-side locking (Scenario 2's copy-on-write).
• ../README.md — Clean Code chapter index.
• ../../refactoring/README.md — refactoring toward seams that make contention easy to isolate and shard.