Cache Coherence — Senior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Architectural View: Cores, Caches, Memory, Fabric
5. MOESI and Directory Coherence
6. Multi-Socket NUMA in Practice
7. Designing Cache-Friendly Data Structures
8. Padding Strategies in Depth
9. Cross-Architecture Design Considerations
10. Operational Tooling at Scale
11. Real Production Case Studies
12. Coding Patterns at the Senior Level
13. Anti-Patterns and How to Spot Them
14. Best Practices
15. Edge Cases & Pitfalls
16. Common Mistakes
17. Tricky Points
18. Tricky Questions
19. Cheat Sheet
20. Self-Assessment Checklist
21. Summary
22. What You Can Build
23. Further Reading
24. Related Topics
25. Diagrams & Visual Aids

Introduction¶

Focus: "I own the architecture of a multi-core Go service. I need to design data structures, choose primitives, and operate the service at scale, all with cache coherence in mind."

Junior-level knowledge is "what is false sharing." Middle-level knowledge is "the MESI state machine and what each Go atomic compiles to." Senior-level knowledge is system design with coherence as a first-class concern.

A senior Go engineer designing a high-throughput service must consider:

• Data structure choices. Whether to use sync.Map, hand-rolled sharded maps, immutable snapshots, or something else, and why.
• Memory layout. Padding strategies, NUMA-aware allocation, alignment guarantees.
• Operational realities. How to detect coherence problems in production, how to bound their impact, what metrics to expose.
• Cross-architecture portability. Code that runs on x86 servers, ARM servers, Apple silicon dev machines, all behaving sensibly.
• Trade-offs. When padding is right, when sharding is right, when copy-on-write is right, when "just use a mutex" is right.

This file teaches those judgements. By the end you will:

• Be able to design a per-CPU sharded structure with appropriate fall-back behaviour and a clean API.
• Be able to reason about cross-socket coherence costs and decide when they matter.
• Know how to handle NUMA in Go without controlling the OS thread scheduler directly.
• Recognise the coherence anti-patterns in code reviews and prescribe specific fixes.
• Have a deployment / monitoring strategy for catching coherence regressions before they hit production.

This is the level at which "Go is a high-level language" stops being protective. The hardware shows through. The patterns you learn here are the same patterns you would apply in C++ or Rust at the same scale.

Prerequisites¶

• Required: Solid command of middle-level cache coherence: MESI states, store buffers, what sync/atomic does on x86 and ARM.
• Required: Production Go experience. You have shipped Go services that handle real load.
• Required: Comfort with profiling tools (pprof, perf, perf c2c) and reading their output.
• Helpful: Some C or assembly reading ability for examining go tool objdump output.
• Helpful: Operational experience with at least one of: Linux kernel scheduler tuning, NUMA, CPU pinning, cgroups.

Glossary¶

Architectural View: Cores, Caches, Memory, Fabric¶

A senior engineer thinks about a multi-core system as a small distributed system.

A single socket, modern x86¶

A typical 16-core Skylake-X / Cascade Lake / Ice Lake socket:

Socket (one die)
├── 16 cores
│   each with:
│   ├── L1d (32 KB, ~1 ns)
│   ├── L1i (32 KB, ~1 ns)
│   └── L2  (1 MB,  ~4 ns)
├── Shared L3 (32 MB, ~12 ns) connected by a mesh
├── 6-8 memory channels
└── DRAM (~80 ns)

The cores are connected by a mesh (Intel) or ring (older Intel) or Infinity Fabric (AMD). Each mesh stop hosts cores plus an L3 slice plus a memory controller fragment. Coherence traffic travels across the mesh.

AMD EPYC (Zen 2/3/4)¶

AMD's design is different. Cores are grouped into CCXes (Core Complexes) of 4–8 cores; CCXes are placed on CCDs (Core Compute Dies); multiple CCDs share a separate IO die for memory and IO.

Socket
├── CCD 0
│   ├── CCX 0 (4-8 cores, shared L3)
│   └── CCX 1 (4-8 cores, shared L3)
├── CCD 1 ...
└── IO Die (memory controllers, PCIe)

Coherence within a CCX is fast. Across CCXes on the same CCD is slower. Across CCDs is slower still. Cross-socket is slowest.

This matters because a Go service on EPYC may see different scaling depending on where the OS places its threads. Pinning to one CCX is sometimes worth it for very latency-sensitive workloads.

Apple Silicon¶

M1/M2/M3 chips have:

• Performance cores in a cluster (4-8 cores), each with private L1d (128 KB) and L1i (192 KB), shared L2 (12 MB cluster).
• Efficiency cores in a separate cluster.
• Unified memory architecture: the GPU shares the same memory.

Cache lines are 128 bytes. The cluster topology means coherence within a P-cluster is fast; across clusters is more expensive. The Go scheduler does not know about clusters — it sees logical CPUs.

ARM server CPUs (Ampere Altra, AWS Graviton 3/4)¶

Often more uniform than AMD or Apple — many cores connected to a single coherent fabric. Cache lines are 64 bytes (Graviton 3) or 128 bytes (some configurations). Coherence is fast and reasonably uniform.

Multi-socket systems¶

Most production servers have one or two sockets. Some HPC machines have four or eight. Each socket connects to others via UPI (Intel) or Infinity Fabric (AMD). Cross-socket coherence is roughly 5-10× slower than same-socket.

For a senior, the key idea: cache lines are messages on a network. The topology of that network affects coherence costs. Locality on the network matters.

MOESI and Directory Coherence¶

Middle-level focused on MESI. Senior-level adds two more concepts.

MOESI: Owned state¶

MOESI adds the Owned (O) state: this core has the only modified copy AND other cores have shared copies. Compared to MESI:

• In MESI, when a core in M state is read by another, it must write back to memory and downgrade to S.
• In MOESI, the M-state core can instead transition to O and forward the value to the reader (who goes to S). The line stays dirty; no write-back needed yet.

Benefit: less memory bandwidth. The dirty line stays in the cache hierarchy until truly evicted.

AMD has historically used MOESI; Intel uses MESIF (where F = Forward state for read-friend caches).

Directory coherence¶

Snoop-based coherence broadcasts. It scales to a few cores per socket but not to thousands of cores or many sockets. Directory coherence keeps a table: for each cache line in the system, which cores hold it?

When a core wants RFO, it queries the directory, which sends invalidations only to the listed cores. No broadcast.

Modern multi-socket systems use directories (or hybrid schemes). The snoop filter discussed earlier is a directory-like structure local to one socket.

Implications for Go code:

• On a small system (few cores, one socket), broadcast is cheap; directory features are noise.
• On a large system (multiple sockets, many cores), directory lookups add latency. Targeted invalidations are cheaper than broadcasts.
• Code that touches many lines may overflow the directory's capacity, causing fallback to broadcast. This is rare in user-space Go but happens in kernels and databases.

Bandwidth vs latency¶

Coherence costs come in two flavours:

• Latency: time per individual coherence operation (~30 ns same-socket, ~200 ns cross-socket).
• Bandwidth: total coherence traffic the fabric can carry (often the dominant cost at scale).

A workload may be latency-bound (a few hot lines, ping-ponging) or bandwidth-bound (many lines, churning through the fabric). The fixes differ. Latency-bound: pad to keep lines local. Bandwidth-bound: reduce sharing or move work to colder data.

Multi-Socket NUMA in Practice¶

Most senior engineers do not need NUMA mastery — single-socket VMs hide it. Those who do, here are the practical considerations.

First-touch allocation on Linux¶

When a Go program calls make([]byte, 1<<30), Linux does not actually allocate physical pages immediately. It assigns virtual addresses; pages are faulted in on first touch. The OS allocates each page on the NUMA node of the thread that first touched it.

Implications:

• If a slice is initialised by a single goroutine, all pages end up on that goroutine's NUMA node.
• If the slice is later used by threads on other nodes, every access pays the remote-node penalty.
• For NUMA-aware code, ensure that the threads that touch memory most also allocate it.

In Go, the runtime allocator (mheap) does not currently expose NUMA awareness. Allocations go to arenas managed by the runtime; arenas may span NUMA nodes invisibly.

numactl for process pinning¶

The simplest NUMA tool: bind a Go process to one node.

numactl --cpunodebind=0 --membind=0 ./myservice

This restricts the process to socket 0's cores and memory. No cross-socket access possible. Throughput often improves 20–50% for shared-memory workloads.

For Go services running on multi-socket bare-metal, consider running one process per socket (with separate ports or shared load balancer) instead of one process spanning sockets.

Per-thread NUMA pinning (cgo)¶

Within a process, pinning individual threads to NUMA nodes requires cgo:

//#include <sched.h>
import "C"
import "runtime"

func pinToNode(node int) {
    runtime.LockOSThread()
    // Compute the CPU set for the given node...
    // Call sched_setaffinity via cgo.
}

This is rarely worth it for Go services. Better: design the service to be NUMA-oblivious by sharding state per CPU and accepting that local coherence is good enough.

NUMA-aware sharding¶

A common pattern in databases (and applicable to high-throughput Go services):

• One shard per CPU socket.
• All threads on socket 0 use shard 0; threads on socket 1 use shard 1.
• Merging across shards is rare.

In Go, this is approximated by sharding per GOMAXPROCS and trusting the runtime not to migrate frequently across sockets. For mission-critical services, pin threads.

Designing Cache-Friendly Data Structures¶

A senior's main job: design data structures and APIs that are cache-friendly by construction.

Principle 1: Separate hot from cold¶

Within a struct, group hot (frequently mutated) fields and cold (rarely accessed) fields. Pad between them.

type Server struct {
    // hot
    requests atomic.Int64
    _        cpu.CacheLinePad
    errors   atomic.Int64
    _        cpu.CacheLinePad

    // cold (set once at construction)
    name    string
    address string
    created time.Time
    config  *Config
}

The cold fields share lines freely. The hot fields are isolated.

Principle 2: Shard by access pattern¶

If you have N counters each potentially touched by M cores, you have N×M sharing problems. Restructure as M shards each holding N counters, indexed by CPU.

type Counters struct {
    perCPU []shardedCounters
}

type shardedCounters struct {
    requests int64
    errors   int64
    bytes    int64
    _        [40]byte // pad to 64
}

func (c *Counters) RecordRequest(cpu int) {
    atomic.AddInt64(&c.perCPU[cpu%len(c.perCPU)].requests, 1)
}

Within a shard, the three counters share a line — but only one CPU writes them. No contention. Merge at snapshot time.

Principle 3: Make reads cheaper than writes¶

If a value is read 1000× more often than written, design for read-cheapness:

• Use atomic.Pointer[Snapshot] for the canonical reference.
• Readers do a single atomic Load — cheap, line in S.
• Writers atomically swap the pointer — rare, brief invalidation.

type ConfigStore struct {
    snapshot atomic.Pointer[Config]
    _        cpu.CacheLinePad
}

Principle 4: Use the standard library¶

sync.Pool, sync.Map, runtime/internal/atomic types are already tuned. Reach for them before writing your own.

Principle 5: Document layout invariants¶

A struct with carefully arranged padding is fragile. A future maintainer can wreck it without realising. Document:

type Counter struct {
    // INVARIANT: value must be on its own cache line.
    // The padding below must not be removed or changed.
    value atomic.Int64
    _     cpu.CacheLinePad
}

Principle 6: Test layout at startup¶

func init() {
    c := &Counter{}
    addr := uintptr(unsafe.Pointer(&c.value))
    if addr%uintptr(cpu.CacheLinePadSize) != 0 {
        panic("Counter.value misaligned")
    }
}

Catch surprises early.

Padding Strategies in Depth¶

Beyond the basic _ [56]byte pattern, several padding strategies are worth knowing.

Strategy 1: Inline padding fields¶

type X struct {
    hot int64
    _   [56]byte
}

Simple. Anonymous. Cannot be inspected. Standard.

Strategy 2: Typed padding fields¶

type CacheLinePad [56]byte

type X struct {
    hot int64
    pad CacheLinePad
}

Named, can be referenced. Useful if you want to assert size in tests.

Strategy 3: cpu.CacheLinePad¶

import "golang.org/x/sys/cpu"

type X struct {
    hot int64
    _   cpu.CacheLinePad
}

Portable across architectures. The package picks the right size for the build target.

Strategy 4: Embedded padded struct¶

type Atomic64 struct {
    v atomic.Int64
    _ [56]byte
}

type X struct {
    counter1 Atomic64
    counter2 Atomic64
}

Each Atomic64 is its own cache line. Use when you need many independent atomics in one struct.

Strategy 5: Padding within an array slot¶

type Bucket struct {
    value int64
    _     [56]byte
}

type BucketArray [16]Bucket

Each slot is 64 bytes. Indexing into the array gets cache-line-isolated cells.

Strategy 6: Aligned allocation¶

For globals:

//go:align 64
var counter int64

Combined with following padding, gives a cache-line-aligned global.

For heap allocations, Go's allocator does not directly support align-N. Workarounds: allocate a larger buffer and slice out the aligned portion. Rare.

Strategy 7: Computed padding¶

When you have to pad something whose size depends on a type:

type Padded[T any] struct {
    v T
    _ [64 - unsafe.Sizeof(*new(T))%64]byte
}

The padding expression rounds the struct up to 64 bytes. Tricky and breaks with generics that have non-constant sizes.

Strategy 8: Manual alignment via slice tricks¶

buf := make([]byte, 128)
alignedAddr := (uintptr(unsafe.Pointer(&buf[0])) + 63) &^ 63
ptr := (*int64)(unsafe.Pointer(alignedAddr))

Hideous but sometimes necessary.

Choosing among them¶

For most Go code: use cpu.CacheLinePad after each hot field. It is portable, clear, and battle-tested.

For library code: hand-pad to specific sizes if you can document the platform assumptions.

For runtime / kernel-like code: align constructs manually with //go:align and explicit pads.

Cross-Architecture Design Considerations¶

Go binaries run on x86-64, ARM64 (Linux servers, Apple silicon), and 32-bit ARM (rare). Senior code must work across.

Cache line size¶

• x86-64: 64 bytes.
• ARM64 server (Ampere, Graviton): 64 bytes.
• Apple silicon: 128 bytes.

Solution: cpu.CacheLinePad, sized at build time per target.

Atomic alignment¶

• 32-bit ARM: 64-bit atomics require 8-byte alignment.
• 64-bit ARM and x86: 64-bit atomics are naturally aligned in well-formed structs.

Solution: use atomic.Int64 (typed) which guarantees alignment via noCopy and internal layout, or place 64-bit atomics first in struct definitions.

Memory model¶

• x86: TSO. Stores are not reordered after loads (much). Atomics on x86 are sequentially consistent.
• ARM: weak. Without fences, almost anything can be reordered.

Solution: always use sync/atomic and sync primitives. Never roll your own with bare loads/stores. Go's memory model abstracts the differences.

Atomic instruction availability¶

• x86: LOCK prefix always available.
• ARMv8 base: only LL/SC.
• ARMv8.1 LSE: direct LDADD, CAS, etc.

The Go toolchain picks at build time. For server-class ARM, you almost always get LSE.

Performance characteristics¶

The order of magnitude is the same. Worry about coherence, not architecture.

Endianness¶

All modern platforms in scope are little-endian. Cross-endian is essentially dead. Ignore.

Operational Tooling at Scale¶

At scale, you cannot debug coherence problems by trial and error. You need monitoring and tooling.

Production pprof¶

Enable continuous CPU profiling on production hosts. Sample every minute. Aggregate. Look for:

• Atomic operations at the top.
• sync.(*Mutex).Lock and related.
• Runtime atomics (runtime/internal/atomic.*).

Tools: pyroscope, parca, Datadog Continuous Profiler.

Hardware counters¶

If your environment allows perf, deploy a sidecar that runs perf stat periodically and exports counters to your metrics system. Key counters:

• cache-misses
• LLC-load-misses
• offcore_response.* series
• cycle_activity.stalls_l3_miss

These will tell you whether your service is coherence-bound.

perf c2c in canary¶

perf c2c is too heavy for steady-state production. Use it in canary or staging:

sudo perf c2c record -F 4000 -a -- ./run-load-test
sudo perf c2c report > c2c.report

Review the top hot lines. If any are from your code, investigate.

Custom telemetry¶

Some services expose runtime metrics like runtime/metrics plus custom counters for cache-friendliness:

• A counter incremented on every cross-CPU shard access.
• A latency histogram for atomic operations.
• A heuristic "coherence health" derived from atomic-op latencies.

These are advanced and rare. Usually, pprof + perf are sufficient.

Benchmarks in CI¶

Every PR runs a benchmark suite:

- name: benchmark
  run: |
    go test -bench=. -benchtime=10s -cpu=1,4,16 -count=5 > bench.out
    benchstat baseline.out bench.out
    # fail if any benchmark regresses by more than X%

This catches accidental coherence regressions before deploy.

Real Production Case Studies¶

Case Study 1: A search service that did not scale past 8 cores¶

Setup: A Go service serving search queries. Each query consults a per-route counter. 30k RPS at 8 cores; 32k RPS at 16 cores.

Diagnosis: pprof showed 22% in atomic.AddInt64. perf c2c flagged the counter slice as the hot line.

Fix: Padded the counter struct to 64 bytes. Each route's counter on its own line.

Result: 30k → 70k RPS at 16 cores. CPU usage dropped 35%.

Lesson: Padding a 24-byte struct to 64 bytes per counter cost ~10 MB of memory (a million routes × 40 extra bytes). Negligible compared to the gain.

Case Study 2: A logging library that became the bottleneck¶

Setup: A Go service logged each request via a popular logger. Throughput plateaued at 50k RPS regardless of cores.

Diagnosis: pprof showed massive time in logger.Output. Looking at the logger source: a global mutex around the writer.

Fix: Switched to a logger with per-goroutine buffers and periodic flushes.

Result: Throughput doubled. The fix was algorithmic — avoid the mutex on the hot path — not just padding.

Lesson: Sometimes the fix is not padding or sharding but redesigning the algorithm to avoid sharing entirely.

Case Study 3: A websocket fanout that bounced cache lines¶

Setup: A service maintained 100k websocket connections, each in its own goroutine. Outbound messages were fanned out from a publisher goroutine via a shared channel.

Diagnosis: Throughput was poor. The shared channel's hchan struct bounced between cores.

Fix: Sharded the channel. 64 shards. Each subscriber registered to one shard.

Result: 5× throughput improvement.

Lesson: Shared channels are often coherence hotspots. Sharding fixes it.

Case Study 4: A metrics library that destroyed scaling at 64 cores¶

Setup: A metrics library used by every handler exposed atomic counters. On a 64-core box, the service used 90% CPU at 100k RPS — only 1.5x the throughput of an 8-core box.

Diagnosis: All counters were packed in a single struct. Counters shared lines. Every handler invalidated lines for every other handler.

Fix: Migrated to per-CPU sharded counters with periodic flushes to the canonical counter.

Result: Linear scaling restored. Throughput jumped to 800k RPS.

Lesson: A widely-used library with bad cache behaviour can dominate a service's performance.

Case Study 5: A "lock-free" queue that was slower than a mutex queue¶

Setup: A team implemented an MPMC lock-free queue using atomics. Benchmarks showed it slower than a chan T with capacity.

Diagnosis: The queue's head and tail indices shared a cache line. Cells were 32 bytes each, sharing lines with neighbouring cells.

Fix: Padded head and tail. Padded each cell to 64 bytes. The queue became 3× faster than channels.

Lesson: "Lock-free" does not mean "fast." Layout still matters.

Coding Patterns at the Senior Level¶

Pattern: snapshot pointer + atomic swap¶

For read-mostly state:

type Service struct {
    config atomic.Pointer[Config]
    _      cpu.CacheLinePad
}

func (s *Service) Update(c *Config) { s.config.Store(c) }
func (s *Service) Get() *Config     { return s.config.Load() }

Readers do an atomic pointer load. Writers do an atomic pointer store. Snapshots are immutable; readers can dereference fields freely.

Pattern: per-CPU sharded with merge¶

type Counter struct {
    shards []paddedInt64
}

type paddedInt64 struct {
    v atomic.Int64
    _ cpu.CacheLinePad
}

func New(n int) *Counter {
    return &Counter{shards: make([]paddedInt64, n)}
}

func (c *Counter) Add(cpu int) {
    c.shards[cpu%len(c.shards)].v.Add(1)
}

func (c *Counter) Sum() int64 {
    var s int64
    for i := range c.shards {
        s += c.shards[i].v.Load()
    }
    return s
}

Pick a shard by CPU hint (request ID, goroutine ID hash, runtime P ID). Each shard is on its own cache line. No contention if the hint is well-distributed.

Pattern: read-mostly map with copy-on-write¶

type Map[K comparable, V any] struct {
    m atomic.Pointer[map[K]V]
    _ cpu.CacheLinePad
}

func (m *Map[K, V]) Get(k K) (V, bool) {
    p := m.m.Load()
    if p == nil {
        var zero V
        return zero, false
    }
    v, ok := (*p)[k]
    return v, ok
}

func (m *Map[K, V]) Set(k K, v V) {
    for {
        old := m.m.Load()
        next := make(map[K]V, lenOrZero(old)+1)
        if old != nil {
            for k2, v2 := range *old {
                next[k2] = v2
            }
        }
        next[k] = v
        if m.m.CompareAndSwap(old, &next) {
            return
        }
    }
}

func lenOrZero[K comparable, V any](p *map[K]V) int {
    if p == nil { return 0 }
    return len(*p)
}

Reads are essentially free after the initial load. Writes are expensive (full copy). Good for read-mostly, small maps with rare updates.

Pattern: hot/cold split with deferred write-back¶

type Worker struct {
    // hot
    localCount int64       // touched only by this worker
    _          cpu.CacheLinePad

    // cold (touched only on flush)
    globalCount *atomic.Int64
    flushEvery  int
}

func (w *Worker) Inc() {
    w.localCount++
    if w.localCount >= int64(w.flushEvery) {
        w.globalCount.Add(w.localCount)
        w.localCount = 0
    }
}

99% of increments stay local. The global counter is updated once per flushEvery operations.

Pattern: epoch-based reclamation¶

For high-throughput data structures with safe memory reclamation:

type Epoch struct {
    workers []paddedInt64
    global  atomic.Int64
}

func (e *Epoch) Enter(worker int) {
    e.workers[worker].v.Store(e.global.Load() | 1)
}

func (e *Epoch) Exit(worker int) {
    e.workers[worker].v.Store(0)
}

func (e *Epoch) SafeToReclaim(epoch int64) bool {
    for i := range e.workers {
        v := e.workers[i].v.Load()
        if v != 0 && v < epoch {
            return false
        }
    }
    return true
}

Each worker advertises its epoch. Reclamation waits until all workers have advanced past the target epoch.

Pattern: padded mutex slot¶

type MutexSlot struct {
    mu sync.Mutex
    _  cpu.CacheLinePad
}

type Registry struct {
    slots [256]MutexSlot
    data  [256]map[string]interface{}
}

func (r *Registry) Lock(key string) {
    h := hash(key) % 256
    r.slots[h].mu.Lock()
}

A sharded lock registry. Each shard's lock is on its own line.

Pattern: aligned ring buffer with stale indices¶

type Ring[T any] struct {
    head        atomic.Uint64
    cachedTail  uint64           // producer-local stale view of tail
    _           cpu.CacheLinePad

    tail        atomic.Uint64
    cachedHead  uint64           // consumer-local stale view of head
    _           cpu.CacheLinePad

    buf  []T
    mask uint64
}

The producer reads cachedTail (its own variable) most of the time. Only when the cached value says "full" does it refresh from tail. Symmetric for the consumer.

This pattern is the heart of LMAX Disruptor. Throughput approaches the speed of writing a single value to L1 cache per operation.

Anti-Patterns and How to Spot Them¶

Anti-pattern: "global atomic counter"¶

var counter atomic.Int64

func bump() { counter.Add(1) }

Bad if called from many cores. Shard it.

Anti-pattern: "stats struct with packed atomics"¶

type Stats struct {
    A, B, C, D atomic.Int64
}

All four atomics share one or two cache lines. Pad them apart or shard.

Anti-pattern: "mutex right next to protected data"¶

type Resource struct {
    mu sync.Mutex
    data int64
}

Every lock/unlock invalidates the data line. Pad between.

Anti-pattern: "RWMutex for short critical sections"¶

var mu sync.RWMutex
var x int

func get() int { mu.RLock(); defer mu.RUnlock(); return x }

The RLock/RUnlock costs more than the protected work. Use a Mutex or an atomic.Pointer pattern.

Anti-pattern: "shared channel with many senders/receivers"¶

ch := make(chan Job, 1000)
for i := 0; i < 64; i++ { go worker(ch) }

The channel's internal state bounces. Shard the channel.

Anti-pattern: "naive spinlock"¶

for !atomic.CompareAndSwapUint32(&lock, 0, 1) {}

Every spinner issues a CAS that requires line in M. Use test-then-CAS, or just use a sync.Mutex.

Anti-pattern: "per-goroutine atomic counter"¶

var counters = map[uint64]*atomic.Int64{}
var mu sync.Mutex

func get(id uint64) *atomic.Int64 {
    mu.Lock()
    c, ok := counters[id]
    if !ok { c = new(atomic.Int64); counters[id] = c }
    mu.Unlock()
    return c
}

The map's mutex is hot. Use per-P slots via sync.Pool-style design.

Best Practices¶

• Design data structures with coherence as a first-class concern. Hot/cold separation, sharding, snapshots — pick the right tool for each shared variable.
• Prefer sync.Map, sync.Pool, atomic.Pointer over hand-rolled equivalents. The standard library is tuned.
• Always benchmark with -cpu=1,2,4,8,16 (and 32 if your build farm supports). Scaling collapse is the canary.
• Document layout invariants in the type itself. A future engineer who deletes the _ [56]byte field must see why it is there.
• Use cpu.CacheLinePad for portable code. Hand-pad when you have measured and need every byte.
• Add CI benchmarks that fail on regression. Coherence regressions are the kind that creep in via "harmless" refactors.
• Provide a path to disable padding for memory-constrained environments. (Rare; usually you want padding always on.)
• Educate the team. A widely-used internal library tuned for cache is a force multiplier; a poorly-tuned one is a drag on every service.
• Operate with observability. Monitor pprof's top functions on production. If atomics or mutexes climb, investigate.
• Profile in canary, fix in code, verify in canary again. Production is not the right place to discover coherence problems.

Edge Cases & Pitfalls¶

• NUMA on cloud VMs. Most are single-node. Some are not. Check lscpu on each instance type you target.
• runtime.GOMAXPROCS. A surprise change (e.g., container CPU quota) can affect per-P sharding decisions.
• Cgroups CPU limits. A 16-core container limited to 4 CPUs by cgroup still has NumCPU()=16 in Go. Use automaxprocs or set GOMAXPROCS explicitly.
• Memory pressure under high contention. Coherence-bound code wastes CPU; if scaled down, CPU time pressure shows as memory pressure (more GC).
• Hyperthreading. Two threads on one physical core share L1 — no false sharing between them. Benchmark on physical cores for honest numbers.
• Apple silicon big.LITTLE. Performance cores and efficiency cores have different cache behaviour. Pinning to P-cores requires undocumented APIs; usually not worth it.
• Burst traffic. A service that handles 100x normal traffic in spikes may surface coherence problems that are invisible at baseline.
• Containerised deployments. Many containers per host may compete for shared L3 even if logically isolated. The "noisy neighbour" problem includes coherence.

Common Mistakes¶

1. Padding only inside a struct, not between adjacent allocations. Two structs in a slice may share a line even if each is internally padded — pad the struct to a full line.
2. Using int32 thinking smaller is better. Smaller fields fit more per line, increasing sharing probability.
3. Allocating from many goroutines without per-P pools. Causes allocator contention even when individual workloads look benign.
4. Setting GOMAXPROCS higher than physical cores. Over-subscribing causes scheduler thrashing and cache thrashing.
5. Trusting micro-benchmarks at single GOMAXPROCS. They lie about coherence behaviour.
6. Forgetting that time.Now() reads shared state. In hot loops it dominates.
7. Using runtime.LockOSThread() and assuming it pins to a core. It pins to a thread; the OS still moves the thread.
8. Mixing reads and writes on the same line. Either all-read or pad apart.

Tricky Points¶

• A "shared, read-only" line is not actually read-only at hardware level. Initial allocation is a write. If many cores read it after a writer touches it, it's still in S, but the first write costs an invalidation.
• MOESI vs MESI affects bandwidth more than latency. Owned state delays write-back. Useful for chips with limited memory bandwidth.
• Snoop filter capacity is finite. Very large working sets can overflow, falling back to broadcasts. Rare in user-space but possible in database workloads.
• runtime.NumCPU() may differ from runtime.GOMAXPROCS(). Use GOMAXPROCS for sizing shards because that is the number of Ps.
• The Go GC's write barrier touches per-P state on every pointer write. Heavy pointer mutation during GC marking touches a known small set of lines per P; usually fine, but visible in profiles.
• runtime.LockOSThread does prevent migration between goroutines, but the OS can still migrate the OS thread between cores. True pinning requires cgo.

Tricky Questions¶

1. Why is read-mostly with atomic.Pointer faster than read-mostly with RWMutex? Because atomic.Pointer.Load is a single MOV; readers do not write any state. RWMutex's RLock writes the reader counter.

2. When is LockOSThread actually useful for cache locality? Rarely. It pins goroutine to thread, not thread to core. Useful when combined with cgo CPU pinning, but that is outside normal Go.

3. How does Go's runtime ensure per-P data structures stay on the right CPU? It doesn't — the OS may migrate threads. Per-P pinning is fast because it usually matches the CPU; the runtime accepts occasional mismatch.

4. Why does sync.Pool use 128-byte padding instead of 64? To cover Apple silicon (128-byte cache lines). The waste on x86 (64 extra bytes) is negligible.

5. What is the difference between a snoop filter and a directory? Both track which cores hold each line. A snoop filter is local (per-socket), while a directory typically spans the system. Modern multi-socket coherence uses directories at the system level and snoop filters within sockets.

6. Why does a "lock-free" algorithm sometimes outperform a mutex and sometimes lose? Lock-free wins when the critical section is short and contention is low. Mutex wins when contention is high or critical section is long, because the runtime can park the loser instead of spinning. The crossover depends on the specific algorithm and contention level.

7. Why is time.Now() slow in tight loops on some platforms? It reads a kernel-shared page (vDSO). The page is cache-resident but globally synchronised. Many cores reading it pay coherence costs.

8. What does runtime.Gosched() cost in coherence terms? It puts the goroutine back on a runqueue; the runqueue is per-P with stealing. The goroutine may resume on a different P, losing cache locality from its previous run.

9. Why does sync.WaitGroup.Add(N) followed by N Done() calls work fine for small N but cost a lot for huge N? The WaitGroup state is a single counter. N Dones invalidate the counter line N times. Use sharded barriers for huge fan-outs.

10. How would you design a counter that scales to 1M ops/sec per core on a 64-core machine? Per-P sharded counter, padded slots, occasional batched flushes. Reads are O(N) over shards.

Cheat Sheet¶

Senior-level decision flow:

  Is this state shared?
    No -> Move on.
    Yes -> What is the read/write ratio?
              Read-heavy (>100:1) -> atomic.Pointer[Snapshot]
              Balanced -> sync.Mutex with padding, or sharded mutex
              Write-heavy by many cores -> per-CPU shard + merge
              Single writer / many readers -> snapshot pointer

  Is the working set fits in L3?
    Yes -> Coherence cost is per-line bouncing.
    No  -> Coherence cost is amortised; capacity matters more.

  Is the system multi-socket?
    Yes -> Cross-socket coherence is 5-10× same-socket. Consider numactl.
    No  -> Standard padding/sharding sufficient.

Common patterns:
  - snapshot pointer: atomic.Pointer[T], readers cheap, writers swap
  - per-CPU shard: []paddedSlot, indexed by hint, merged on snapshot
  - copy-on-write: atomic.Pointer[Map], cheap reads, expensive writes
  - hot/cold split: pad between hot and cold fields
  - epoch counters: per-worker padded slots advertising progress

Bounds to remember:
  - Cache line: 64B (x86, ARM server), 128B (Apple).
  - Same-socket invalidation: ~30 ns.
  - Cross-socket invalidation: ~150-300 ns.
  - Uncontended atomic: ~6 ns.
  - Bouncing atomic: ~30-100 ns.

Self-Assessment Checklist¶

• I can design a per-CPU sharded counter from scratch with appropriate padding.
• I can identify a NUMA system and decide whether per-socket sharding helps.
• I can read a perf c2c report and translate it to a code-level fix.
• I can debate the trade-offs between sync.Map, hand-rolled sharded maps, and copy-on-write maps for a given workload.
• I can defend a padding decision with benchmark numbers.
• I can recognise and refactor the common anti-patterns in code review.
• I can set up CI benchmarks that catch coherence regressions.
• I can explain why runtime.LockOSThread is not a cache-locality tool.
• I can design APIs for read-mostly state that are coherence-cheap by construction.

Summary¶

Senior-level cache coherence is a design discipline. Hot fields are isolated; cold fields share. Shared state is sharded or snapshotted. Hardware counters and perf c2c are your verification tools. Multi-socket NUMA matters on big iron; cloud VMs usually hide it. Apple silicon needs 128-byte awareness. The standard library has the patterns; copy them. CI benchmarks catch regressions before deploy. Coherence is no longer a surprise — it is a constraint you design with.

What You Can Build¶

• A high-throughput RPC server whose per-route counters scale linearly to 64 cores.
• A metrics library used company-wide whose contribution to CPU is negligible at any scale.
• A lock-free queue that outperforms channels.
• A read-mostly configuration store with millisecond update latency and microsecond read latency.
• A per-P sharded data structure for any custom workload that needs per-CPU semantics.

Further Reading¶

• What Every Programmer Should Know About Memory — Ulrich Drepper.
• Is Parallel Programming Hard, And, If So, What Can You Do About It? — Paul McKenney.
• Intel Optimization Reference Manual (free PDF), chapters on cache and atomics.
• AMD Software Optimization Guide.
• ARM Architecture Reference Manual ARMv8.
• Go source: src/sync/, src/runtime/, src/runtime/internal/atomic/.
• LMAX Disruptor paper.
• Apple's Apple Silicon CPU Architecture (chip details).

Related Topics¶

• 02-acquire-release — Memory ordering on top of coherence.
• 03-sequential-consistency — Strongest ordering; expensive globally.
• 05-false-sharing — Applied detection.
• sync.Map, sync.Pool — Standard-library coherence-aware structures.
• runtime.GOMAXPROCS — Per-P sizing.
• Distributed systems — Cache coherence is the in-machine version of CAP-style consistency.

Diagrams & Visual Aids¶

SENIOR-LEVEL DESIGN OVERVIEW

  +-------------------------------------------+
  |       Application Service                 |
  +-------------------------------------------+
        |                |               |
  [Read-mostly]   [Per-CPU shard]   [Read/write]
        |                |               |
  atomic.Pointer  []paddedSlot     sync.Mutex
   [Snapshot]                       (padded)
        |                |               |
  Cheap reads     Local writes      Serialised
  Rare writes     Rare merge        access

  Choose primitive by access pattern.

NUMA-AWARE STRUCTURE

  Socket 0                       Socket 1
  +-------------+                +-------------+
  | Cores 0-15  |                | Cores 16-31 |
  | Shard 0     |                | Shard 1     |
  | (allocated  |                | (allocated  |
  |  on node 0) |                |  on node 1) |
  +-------------+                +-------------+
       |                              |
       |   UPI link (slow)           |
       +------------------------------+
            (rare cross-socket)

  Locality preserved within each socket.
  Merge happens off the hot path.

COHERENCE COST HEATMAP

  Cores per box | Same-socket | Cross-socket
  --------------+-------------+--------------
       4        |   negligible|   negligible
      16        |   noticeable|        n/a (1 sock)
      32        |    big      |   huge if 2 socks
      64        |    big      |   huge
     128        |    big      |   prohibitive
     256+       | catastrophic|   prohibitive

  Padding moves you to the left column (no contention).
  Sharding moves you to the left column (less contention).
  NUMA pinning moves you to the left column (locality).

Extended Section: Designing a Production-Grade Per-CPU Counter Library¶

A senior engineer often designs reusable libraries. Here is a complete walkthrough of building a production-grade per-CPU counter library, with attention to coherence, ergonomics, observability, and edge cases.

Requirements¶

• Scale to 64+ cores with no contention.
• Handle dynamic CPU count changes (containers can have CPU limits changed at runtime).
• Provide read consistency: a Sum() call returns a stable snapshot.
• Be usable for both counters (monotonic) and gauges (settable).
• Expose metrics for monitoring health (per-shard distribution, contention levels).
• Be safe across architectures (x86, ARM, Apple).

API design¶

package coherentcounter

import (
    "sync/atomic"
    "runtime"
    "golang.org/x/sys/cpu"
)

type Counter struct {
    shards    []shard
    shardMask uint64
}

type shard struct {
    value atomic.Int64
    _     cpu.CacheLinePad
}

// New creates a Counter with the next power-of-two number of shards
// greater than or equal to GOMAXPROCS.
func New() *Counter {
    n := nextPow2(runtime.GOMAXPROCS(0))
    return &Counter{
        shards:    make([]shard, n),
        shardMask: uint64(n - 1),
    }
}

func nextPow2(n int) int {
    if n <= 1 { return 1 }
    p := 1
    for p < n { p *= 2 }
    return p
}

Power-of-two shard counts allow fast modulo via bitmask.

Add with shard hint¶

// Add increments by delta. The hint should be a fast-changing value
// like a goroutine ID or request ID for good distribution.
func (c *Counter) Add(hint uint64, delta int64) {
    c.shards[hint&c.shardMask].value.Add(delta)
}

Note: this avoids procPin for portability. The hint is the caller's responsibility.

Sum with consistency guarantee¶

func (c *Counter) Sum() int64 {
    var total int64
    for i := range c.shards {
        total += c.shards[i].value.Load()
    }
    return total
}

Sum is not an atomic snapshot — concurrent Adds may be reflected in some shards and not others. For most counter use cases, this is acceptable; the result is "approximately correct," monotonically increasing, eventually consistent.

For strict snapshot semantics, you would need to halt all writers, which defeats the point of sharding. Document the trade-off.

Per-CPU Add variant for advanced users¶

//go:linkname runtime_procPin runtime.procPin
func runtime_procPin() int

//go:linkname runtime_procUnpin runtime.procUnpin
func runtime_procUnpin()

// AddPinned increments using the current P as the shard hint.
// Slightly faster than Add(hint, delta) but uses unstable runtime APIs.
func (c *Counter) AddPinned(delta int64) {
    pid := runtime_procPin()
    c.shards[pid&int(c.shardMask)].value.Add(delta)
    runtime_procUnpin()
}

The procPin/procUnpin calls pin the goroutine to its P, ensuring the shard hint matches the current P. This is what sync.Pool does internally.

Observability hooks¶

// ShardStats returns the per-shard values.
// Useful for monitoring distribution skew.
func (c *Counter) ShardStats() []int64 {
    stats := make([]int64, len(c.shards))
    for i := range c.shards {
        stats[i] = c.shards[i].value.Load()
    }
    return stats
}

// Skew returns max/min of shard values, indicating distribution quality.
// A skew near 1 means perfect distribution; > 5 means hint is poor.
func (c *Counter) Skew() float64 {
    s := c.ShardStats()
    if len(s) == 0 { return 1 }
    min, max := s[0], s[0]
    for _, v := range s[1:] {
        if v < min { min = v }
        if v > max { max = v }
    }
    if min == 0 { return float64(max) }
    return float64(max) / float64(min)
}

Skew is a powerful debugging signal: if Skew is 100, your hint distribution is broken and one shard is doing all the work.

Dynamic resize support (advanced)¶

If GOMAXPROCS changes (e.g., due to container limits), the counter can be resized:

func (c *Counter) Resize(newN int) {
    if newN == len(c.shards) { return }
    newShards := make([]shard, nextPow2(newN))
    // Migrate values: sum old, distribute among new
    var total int64
    for i := range c.shards {
        total += c.shards[i].value.Load()
    }
    newShards[0].value.Store(total)
    c.shards = newShards
    c.shardMask = uint64(len(newShards) - 1)
}

This is racy — concurrent Add during Resize loses updates. Production-grade resize needs a versioned or atomic-swap design. For most services, just size for the maximum at startup.

Documentation¶

Every public type should explain:

• Memory cost (N × cache-line bytes).
• Best-distribution hints.
• Read consistency guarantee.
• Edge cases (zero shards, very high skew).

Without docs, a future user will reach for Counter.AddPinned without understanding the procPin caveat.

Testing¶

Tests should cover:

• Correctness: total matches expected.
• Memory layout: shards are on different cache lines (use unsafe).
• Scaling benchmark: ns/op stays flat from 1 to NumCPU cores.
• Skew under known good hint: should be near 1.

Benchmarks¶

func BenchmarkCounter(b *testing.B) {
    c := New()
    b.RunParallel(func(pb *testing.PB) {
        var hint uint64
        for pb.Next() {
            hint++
            c.Add(hint, 1)
        }
    })
}

Run with -cpu=1,4,16,64. The expected result: flat or slightly improving ns/op, total throughput scaling linearly.

Extended Section: A Tour of sync.Map Internals at Senior Depth¶

We touched on sync.Map at middle level. A senior should know it well enough to know when not to use it.

The sync.Map structure¶

type Map struct {
    mu Mutex

    read atomic.Pointer[readOnly]

    dirty map[any]*entry

    misses int
}

type readOnly struct {
    m       map[any]*entry
    amended bool // true if the dirty map contains some key not in m.
}

type entry struct {
    p atomic.Pointer[any]
}

The read is read-only from concurrent readers' perspective. It's a snapshot. Writers may mutate dirty under mu.

The hot read path¶

func (m *Map) Load(key any) (value any, ok bool) {
    read := m.read.Load()
    e, ok := read.m[key]
    if !ok && read.amended {
        m.mu.Lock()
        // Recheck under lock; lookup in dirty if needed.
        ...
        m.mu.Unlock()
    }
    if !ok { return nil, false }
    return e.load()
}

When the key is in read, the load is:

1. Atomic Load of the read pointer.
2. Plain map lookup.
3. Atomic Load of the entry's value pointer.

No lock. Coherence cost: pointer load (line in S, cheap).

The write path¶

Writes are slow but rare:

func (m *Map) Store(key, value any) {
    read := m.read.Load()
    if e, ok := read.m[key]; ok && e.tryStore(&value) { return }

    m.mu.Lock()
    read = m.read.Load()
    if e, ok := read.m[key]; ok {
        // Already exists, just update.
        e.unexpungeLocked()
        m.dirty[key] = e
        e.storeLocked(&value)
    } else if e, ok := m.dirty[key]; ok {
        e.storeLocked(&value)
    } else {
        if !read.amended {
            m.dirtyLocked()
            m.read.Store(&readOnly{m: read.m, amended: true})
        }
        m.dirty[key] = newEntry(value)
    }
    m.mu.Unlock()
}

Writes serialise on mu. Coherence: the mutex line bounces among writers; but writes are assumed rare.

Miss accounting and promotion¶

After many misses (key not in read), the dirty map is promoted to read:

func (m *Map) missLocked() {
    m.misses++
    if m.misses < len(m.dirty) {
        return
    }
    m.read.Store(&readOnly{m: m.dirty})
    m.dirty = nil
    m.misses = 0
}

This is the moment of catch-up. Future reads of those keys hit read.

When sync.Map is the right choice¶

• Read-mostly workload (>10:1 reads to writes).
• Key set is stable; growth is slow.
• Keys can be hashed (i.e., any works).
• Concurrent access from many cores.

When sync.Map is the wrong choice¶

• Write-heavy workload (a regular map + Mutex may be faster).
• Strict typing is needed (the any API is awkward).
• The key set churns rapidly (sync.Map's promotion overhead becomes significant).
• Predictable memory usage required (sync.Map's transient duplication of map state can spike memory).

Alternative: sharded map with Mutex¶

type ShardedMap struct {
    shards []mapShard
}

type mapShard struct {
    mu sync.Mutex
    m  map[string]any
    _  cpu.CacheLinePad
}

func NewShardedMap(n int) *ShardedMap {
    shards := make([]mapShard, n)
    for i := range shards { shards[i].m = make(map[string]any) }
    return &ShardedMap{shards: shards}
}

func (m *ShardedMap) Get(key string) (any, bool) {
    s := &m.shards[hash(key)%len(m.shards)]
    s.mu.Lock()
    v, ok := s.m[key]
    s.mu.Unlock()
    return v, ok
}

Trade-offs:

• Each shard has its own mutex. Contention is proportional to keys hashing to the same shard.
• Padding ensures shards do not false-share.
• Memory: N shards × (mutex + small map + pad).
• Predictable: no transient duplication.

For write-heavy workloads, sharded maps often beat sync.Map. For read-heavy, sync.Map is usually faster.

Extended Section: Building a Cache-Aware HTTP Server¶

A senior may design an HTTP server with coherence baked in. A blueprint.

Top-level structure¶

type Server struct {
    // hot stats — per CPU
    stats *coherentcounter.Counter
    _     cpu.CacheLinePad

    // hot config — atomic snapshot
    config atomic.Pointer[Config]
    _      cpu.CacheLinePad

    // cold setup
    listener net.Listener
    mu       sync.Mutex // for cold operations
    routes   map[string]Handler
}

The hot fields (stats, config) are isolated. Cold fields share lines but are accessed at startup and shutdown only.

Per-request work¶

func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
    s.stats.Add(uint64(time.Now().UnixNano()), 1) // bumped per request
    cfg := s.config.Load()                        // cheap atomic load
    h := s.routes[r.URL.Path]                     // map lookup (read-only)
    h(w, r, cfg)
}

Per-request, we touch:

• One per-CPU shard (one cache line, no contention).
• The config pointer (cache line in S).
• The routes map (read-only after init).

No mutex on the hot path. No false sharing.

Reloading config¶

func (s *Server) Reload(c *Config) {
    s.config.Store(c)
}

One atomic store. Briefly invalidates the config line for readers, who pick up the new pointer on their next request.

Adding routes (cold path)¶

func (s *Server) AddRoute(path string, h Handler) {
    s.mu.Lock()
    s.routes[path] = h
    s.mu.Unlock()
}

This is racy with concurrent reads. Production-grade requires either:

• All routes added before serving begins (typical pattern), or
• Copy-on-write of the routes map with atomic swap.

For most services, the former is sufficient. For services with dynamic routing, use the COW pattern.

Metrics export¶

func (s *Server) Metrics() map[string]int64 {
    return map[string]int64{
        "requests": s.stats.Sum(),
    }
}

Sum walks all shards — O(NumCPU). Called every few seconds, not in the hot path. Affordable.

Extended Section: Cache-Aware Database Driver¶

For a Go database driver maintaining a connection pool, cache awareness affects throughput.

Connection pool layout¶

type Pool struct {
    mu       sync.Mutex
    idle     []*Conn
    inUse    int
    maxOpen  int
    _        cpu.CacheLinePad

    waiters  chan struct{}
    closed   atomic.Bool
}

The mu, idle, inUse are touched per Acquire/Release. Pad before waiters and closed to keep them on a separate line.

Per-conn state¶

type Conn struct {
    id        uint64
    _         cpu.CacheLinePad

    inUse     atomic.Bool
    lastUsed  atomic.Int64
    _         cpu.CacheLinePad

    // bulk state (rarely contended)
    sock      net.Conn
    enc       *Encoder
    dec       *Decoder
}

inUse and lastUsed are mutated on every Acquire/Release. They share a line — fine, because each Conn is only handled by one goroutine at a time. But across many Conns, the array of Conns may share lines.

Better: place Conns far apart in memory by allocating each via a separate new, or pad the Conn struct to a multiple of a cache line.

Health checks¶

A background goroutine periodically scans the pool for stale connections:

func (p *Pool) gcLoop() {
    t := time.NewTicker(30 * time.Second)
    defer t.Stop()
    for range t.C {
        p.mu.Lock()
        // scan p.idle for expired conns
        p.mu.Unlock()
    }
}

The scan reads many Conn.lastUsed fields, putting them in S. The next Acquire on those conns must upgrade back to M. For most workloads, this is a tolerable cost — 30-second scans are rare. If hot-path Acquires are dominant, consider a separate per-Conn last-used timestamp that the GC ignores.

Extended Section: Cache Effects in the Go Runtime¶

The Go runtime itself is cache-aware. A senior should know the major touch points.

The GMP scheduler¶

Each P has its own runqueue. Goroutines stolen from other Ps cross cache boundaries. Per-P locality is the runtime's preferred shape.

Key fields in runtime.p:

• Local runqueue (runq, runqhead, runqtail)
• runqtail is written by the current P; read by stealers.
• The P struct is padded internally to prevent inter-P false sharing.

The memory allocator (mheap, mcache)¶

Each P has a per-P allocator cache (mcache). Allocations come from the cache when possible — no global lock. The mcache is per-P, padded.

When the mcache empties, the P fetches more from the central allocator (mcentral), which is sharded by size class.

Garbage collector¶

The GC has per-P workbuffers for tri-colour marking. Per-P means no contention during marking.

During the write barrier phase, writes to pointers go through a small per-P buffer. The buffer is padded.

Network poller¶

The network poller is conceptually shared, but registrations and notifications are routed per-P where possible.

Conclusion¶

Most runtime hot paths use per-P sharding plus padding. Read runtime/proc.go, runtime/mcache.go, runtime/mgc.go to see real industrial-grade examples.

Extended Section: Operational Stories¶

Story 1 — The case of the 99th percentile spike¶

A team noticed p99 latency spiking every few minutes. Their service was otherwise healthy. They traced it to GC pauses, which were short (1-2ms) but happened to coincide with bursts of cross-CPU writes that the runtime's write barrier amplified.

The fix: reduce pointer mutation in the hot path. Changed several *Foo fields to value types Foo. p99 settled.

Story 2 — The container that lied about CPU count¶

A team deployed Go services in containers with CPU quota. Inside the container, runtime.NumCPU() returned the host's CPU count (32), not the container's quota (4). The Go scheduler created 32 Ps, each with its own per-P state. The service spent significant CPU on inter-P coordination that should never have happened.

The fix: import go.uber.org/automaxprocs (or set GOMAXPROCS=4 explicitly). Per-P sharding aligned with actual CPU count. CPU usage dropped 25%.

Story 3 — The cold cache after deploy¶

A new version of a service was deployed. For the first few minutes, throughput was 30% lower than expected. The team initially blamed the new code. Then they noticed throughput recovered after about 5 minutes.

Explanation: every restart starts with cold caches. Hot lookup tables (config, route maps, prepared statements) had to be re-fetched from memory. Once they settled into L3, throughput returned to normal.

The lesson: cache state is itself a performance asset. Hot data accumulates over time. Don't conflate cold-cache slowness with code regressions.

Story 4 — The cross-socket coherence storm¶

A team ran a Go service on a 2-socket bare-metal box. They noticed inconsistent latencies: a 95th percentile of 8ms but 99.5th of 35ms. The OS scheduler was placing goroutines on both sockets, and the service's shared counters were bouncing across sockets when active goroutines happened to be split.

The fix: numactl --cpunodebind=0 --membind=0 to pin to one socket. Tail latency settled. The other socket was used for a separate service.

The lesson: NUMA bites on bare metal. Cloud VMs usually hide it.

Story 5 — The sync.Map that grew without bound¶

A team used sync.Map for a cache of computed values. Keys were per-request and never reused. Over time, the map's internal dirty and read maps grew, never being garbage-collected, because every key was still being inserted.

Memory usage climbed steadily. p99 latency rose with the map size (longer hash chains).

The fix: switched to a fixed-size LRU. sync.Map is for stable key sets; for ever-growing maps, use a cache with eviction.

The lesson: sync.Map has assumptions about access patterns. Check yours.

Story 6 — The library upgrade that hurt¶

A team upgraded a popular open-source library. The new version added internal metrics — atomic counters incremented on every operation. The atomic counters shared cache lines with frequently-read data structures.

Throughput dropped 15%. The team rolled back, filed a bug, contributed a padding fix.

The lesson: dependencies can introduce coherence problems. Profile after every major upgrade.

Story 7 — The "improvement" that regressed performance¶

A team refactored a hot struct from:

type Stats struct {
    Counter int64
    _       [56]byte
    Name    string
}

to:

type Stats struct {
    Name    string
    Counter int64
}

…because the new ordering was "more natural." Throughput collapsed.

The lesson: padding is load-bearing. Document it; resist "natural" reorderings that erase it. Code reviewers should challenge struct reorders.

Story 8 — The hot mutex that wasn't a mutex¶

pprof showed huge time in sync.(*Mutex).Lock. The team assumed lock contention. They added more granular locks. Performance got worse.

The mutex itself was not contended — the same goroutine acquired and released it rapidly with no waiters. But the mutex shared a cache line with a counter the goroutine also wrote. Every lock/unlock invalidated the counter line on another core that was reading it.

The fix: padded the mutex. Throughput returned to expected.

The lesson: high time in Lock does not always mean contention. Sometimes it is coherence on the mutex's line.

Extended Section: A Sample Architecture Document¶

When senior engineers design systems, they document architecture. A sample paragraph from a fictitious service spec:

Per-route counters use the coherentcounter library with 64 shards on 16-core hosts. Shards are indexed by request ID hash. Skew is monitored via Counter.Skew() and alerted on if > 5. Memory cost per counter is ~4 KB. Sum() is called by /metrics scrape every 15s; cost is O(64) loads, negligible.

Configuration is exposed via atomic.Pointer[Config]. Writers (admin API) update the pointer; readers do a single load per request. Memory cost per config snapshot is ~16 KB; we hold at most two snapshots (previous and current) for safety during rollout.

Route table is built at startup and exposed via atomic.Pointer[RouteTable]. Dynamic route changes are accepted via the admin API and trigger a full copy-on-write of the route map. Latency p99 of /admin/route is < 10ms.

No sync.Mutex is held on the per-request path. All shared state is read via atomic loads of snapshot pointers or via per-CPU shards. False sharing is prevented by cpu.CacheLinePad after every hot field.

The service is expected to scale linearly to 64 cores. CI benchmarks at -cpu=1,8,32,64 enforce this. A 10% regression in scaling triggers a build failure.

That kind of paragraph is what differentiates a senior architecture from a junior implementation.

Extended Section: Performance Budgets¶

A senior approach: assign performance budgets to each component.

For a typical Go HTTP service handling 100k RPS at 50% CPU on a 16-core box, per-request CPU budget is:

0.5 × 16 cores × 10^9 ns/sec / 100,000 RPS = 80,000 ns/request

Roughly 80 microseconds per request. Within that:

• Network I/O: 10-20 μs.
• Authentication / parsing: 5-10 μs.
• Business logic: 30-40 μs.
• Logging / metrics: 5-10 μs.
• Buffer.

If metrics alone consume 10 μs, you have a problem. If padding reduces it to 2 μs, you have an 8% budget recovery.

This is how senior engineers justify cache-aware design at the budget level. "10% of CPU on coherence" is concrete and unacceptable; "we should fix false sharing" is abstract and ignorable.

Extended Section: Quantitative Analysis of a Counter Library¶

Cost analysis of the per-CPU counter library:

With sharding plus padding, throughput is 50× the contended atomic on 16 cores. On 64 cores, the ratio is closer to 200×. Linear scaling.

Memory cost: 64 cores × 64 bytes = 4 KB per counter. For a service with 100 counters, 400 KB total. Trivial.

This kind of analysis is what convinces senior engineers (and their bosses) to invest in the right design.

Extended Section: Twenty Questions for a Senior Code Review¶

When reviewing a PR that touches shared state, ask:

1. Which goroutines mutate which fields?
2. Are mutated fields within 64 bytes of each other?
3. Are mutated fields within 64 bytes of read-mostly fields?
4. Is there a benchmark with -cpu=1,2,4,8 showing scaling?
5. Are atomic types padded with cpu.CacheLinePad?
6. Are arrays of structs padded per-element?
7. Are mutexes adjacent to other hot fields?
8. Is RWMutex used for short critical sections (likely wrong)?
9. Is there a sharing across NUMA nodes assumption?
10. Does the code use sync.Map/sync.Pool where appropriate?
11. Are there hand-rolled lock-free algorithms (red flag)?
12. Is there a spin loop on an atomic (red flag)?
13. Are stats counters in a struct that has other hot fields?
14. Is the config snapshot pattern used for read-mostly state?
15. Are CI benchmarks running with multiple -cpu values?
16. Are there hardware-counter assertions (perf c2c validation)?
17. Is layout documented in comments?
18. Are layout invariants checked at init time?
19. Does the API expose shard hints to callers?
20. Are there observability hooks for skew/contention?

A "no" to any of these is not a blocker — but ask why.

Extended Section: The Twenty Most Common Coherence Bugs¶

1. Adjacent atomic counters in one struct.
2. Slice of small structs sharing lines.
3. Mutex right next to protected data.
4. RWMutex for short reads.
5. Shared channel for many workers.
6. Global atomic in hot loop.
7. Spin loop on contended atomic.
8. Padding only some fields (others still share).
9. Padding inside struct but not between adjacent allocations.
10. Padding for x86 but not Apple silicon.
11. NUMA-naive design on bare metal.
12. Cgroup mismatch (GOMAXPROCS too high).
13. runtime.LockOSThread assumed to pin to core.
14. time.Now() in hot loops.
15. sync.Map for write-heavy workloads.
16. Hand-rolled spinlock without test-then-CAS.
17. Sharding without padding (neighbouring shards share).
18. Refcount field on the data line.
19. Long-lived shared structures allocated on wrong NUMA node.
20. Hot fields between read-mostly fields, fragmenting the read pattern.

Memorise these. They are the bulk of real-world incidents.

Extended Section: A Walk Through runtime.mheap for Educational Value¶

The Go runtime's heap allocator demonstrates senior-grade cache awareness.

runtime/mheap.go:

type mheap struct {
    lock      mutex
    pagesInUse uint64
    ...

    central [numSpanClasses]struct {
        mcentral mcentral
        pad      [cpu.CacheLinePadSize - unsafe.Sizeof(mcentral{})%cpu.CacheLinePadSize]byte
    }

    ...
}

The central array has one mcentral per size class. Each is padded to a cache line because allocations of different size classes can happen concurrently on different Ps, and the centrals are touched per allocation.

runtime/mcache.go:

type mcache struct {
    ...
    tiny       uintptr
    tinyoffset uintptr
    ...
    alloc [numSpanClasses]*mspan
    ...
}

mcache is per-P. No padding inside — single-threaded access. But the array of mcache (one per P) in runtime.allp is padded internally so that different Ps' mcaches don't share lines.

This is how real high-performance runtimes are built. Read more of runtime/proc.go, runtime/mgc.go, runtime/sema.go for additional examples.

Extended Section: A Quick Tour of Lock-Free Algorithms in Go¶

Lock-free programming is a separate discipline, but cache coherence is at its core. A senior should know the major shapes.

Treiber stack¶

type Stack[T any] struct {
    head atomic.Pointer[node[T]]
}

type node[T any] struct {
    value T
    next  *node[T]
}

func (s *Stack[T]) Push(v T) {
    n := &node[T]{value: v}
    for {
        head := s.head.Load()
        n.next = head
        if s.head.CompareAndSwap(head, n) {
            return
        }
    }
}

func (s *Stack[T]) Pop() (T, bool) {
    for {
        head := s.head.Load()
        if head == nil { var zero T; return zero, false }
        if s.head.CompareAndSwap(head, head.next) {
            return head.value, true
        }
    }
}

Coherence behaviour: the head pointer's line bounces under contention. Throughput plateaus.

Practical improvement: use sync.Pool for the nodes (per-P) and a "back-off" on CAS failure.

Michael-Scott queue¶

A classic MPMC lock-free queue:

type Queue[T any] struct {
    head atomic.Pointer[node[T]]
    tail atomic.Pointer[node[T]]
}

type node[T any] struct {
    value T
    next  atomic.Pointer[node[T]]
}

Head and tail share patterns. Pad them apart. Cells are heap-allocated; the GC handles reclamation.

LMAX Disruptor in Go¶

A bounded ring buffer with cached indices:

type Disruptor[T any] struct {
    write       atomic.Uint64
    cachedRead  uint64
    _           cpu.CacheLinePad

    read        atomic.Uint64
    cachedWrite uint64
    _           cpu.CacheLinePad

    cells       []cell[T]
    mask        uint64
}

type cell[T any] struct {
    seq  atomic.Uint64
    data T
    _    [40]byte
}

Producer and consumer mostly read their own cached counterparts. Only when "full" or "empty" do they refresh from the shared atomic. Cells are padded.

Throughput approaches one cache-line write per operation per core.

What to take away¶

Lock-free is powerful but unforgiving. Coherence layout is the foundation. Get it wrong and lock-free is slower than mutex-based.

Extended Section: When Cache Coherence Meets the Go GC¶

The Go GC interacts with coherence in subtle ways.

Write barrier¶

Every pointer write during the mark phase goes through a write barrier. The barrier records the write in a per-P buffer. The buffer is occasionally flushed to a global queue.

Cache effects:

• The per-P buffer is a hot line, but only the current P writes it.
• The global queue is touched on flush — a coherence event, but rare.

For heavy pointer-mutation workloads (linked-list-style algorithms), the write barrier adds visible overhead during GC. Profile to confirm.

Mark assist¶

When the heap grows faster than the GC can keep up, the allocator forces user goroutines to do "mark assist" work. Each assist is a small chunk of GC marking — touching shared GC state.

Cache effects: more cross-P traffic during heavy allocation under heavy heap growth.

Fix: pre-size structures; avoid surprise allocations in hot paths.

Sweep¶

After the mark phase, the sweep phase reclaims free objects. Per-P sweep buffers ensure each P sweeps its own pages. No coherence traffic for sweep itself.

Concurrent collection¶

The whole GC runs concurrently with user goroutines. There is no "stop the world" except for very short barrier synchronisations.

Cache effects during STW: every P touches a shared barrier — invalidations occur, but each STW is sub-millisecond.

Extended Section: Cache Coherence and Garbage Collection: Putting It Together¶

A senior must reason about both. An example workload:

• High-throughput RPC, 100k RPS, 8 cores.
• Per-request allocates ~10 small objects (allocator chooses size classes).
• Per-request mutates ~20 pointer fields (GC write barriers fire during marking).
• Per-request bumps 5 counters.

If the counters are unpadded: ~25 ns per request lost to false sharing. At 100k RPS × 8 cores = 800k counter ops/sec wasted on coherence. ~2-3% of CPU.

If the counters are padded but allocator is contended (no per-P pools): another 5-10% lost to allocator coherence.

If GC writes barriers hit shared buffers (rare): another 1-2% during marking, transient.

Total potential coherence cost: 10-15% of CPU before optimisation. After per-P sharding + padding + sync.Pool reuse: 1-2%.

The opportunity cost of not fixing coherence is roughly 10% of the machine. On a $1000/month box, that is $100/month. Across a fleet of 100 boxes, $10k/month. For one engineer-day of work, an excellent ROI.

This is the senior-level economic argument.

Extended Section: Cache Locality and Big Data Workloads¶

For services that process large datasets, cache locality (not just coherence) becomes paramount.

Working set size¶

If your dataset fits in L3 (~32 MB), random access is fast — every access hits cache. If it spills to DRAM, each access is 80 ns.

For a hashmap with 1 million entries × 24 bytes per entry = 24 MB, you fit in L3 but only just. Adding ten more bytes per entry pushes you out.

Implication: data structure compactness matters. Padding for false sharing costs compactness. There is a trade-off.

For very large maps, prefer:

• Concurrent maps with shards smaller than L3.
• Open addressing rather than chained (better cache friendliness).
• Compact representations (interned strings, packed structs).

Sequential access patterns¶

When you iterate a slice in order, the hardware prefetcher pulls in the next several lines. Random access defeats the prefetcher.

For workloads that scan large datasets:

• Store in contiguous arrays, not linked lists.
• Sort by access order before processing.
• Use struct-of-arrays instead of array-of-structs for partial access.

Example: instead of []Record where you only read Record.Score, store []int of scores separately. The CPU pulls fewer lines.

Cache thrashing¶

If many threads each have working sets larger than their private cache shares, lines evict each other constantly. This is cache thrashing, distinct from coherence.

Fix: reduce per-thread working set, or partition data so each thread owns its slice.

Extended Section: A Final Set of Twenty Senior-Level Drills¶

1. Design a per-CPU counter library API. List the trade-offs you considered.
2. Explain why sync.Map is faster than a sharded map for read-mostly workloads.
3. Walk through what happens at the hardware level when two cores both call atomic.AddInt64 on the same variable.
4. When does padding hurt performance?
5. How do you detect false sharing in production without perf c2c?
6. Design an HTTP server's request handler to be cache-friendly.
7. Estimate the coherence cost of a particular workload.
8. Explain the trade-offs of sync.Pool vs hand-rolled per-P caches.
9. Why is runtime.LockOSThread not sufficient for cache locality?
10. Design a config store that handles 1M reads/sec and 1 write/sec.
11. Explain numactl and when to use it.
12. Why do snoop filters help and when do they overflow?
13. Design a metrics library that scales to 1000 counters and 1M ops/sec.
14. When is RWMutex better than Mutex? When is it worse?
15. Explain the cache effect of garbage collection write barriers.
16. Design a lock-free ring buffer with cached indices.
17. Why does Apple silicon need different padding constants?
18. Explain MOESI and why AMD uses it.
19. How would you set up CI to catch coherence regressions?
20. Estimate the cost of cross-socket coherence on a 2-socket box.

Treat each as a 5-minute design exercise. If you can speak to each fluently, you are senior-level.

Closing¶

Senior cache coherence is design discipline plus operational maturity plus economic argument. You design data structures with coherence in mind, you operate them with the right tooling, and you justify the work to non-technical stakeholders with concrete numbers. The professional level adds runtime-internals depth.

Read professional.md when you are contributing to high-performance libraries or the Go runtime itself.

Long Appendix: A Reference Implementation of Common Patterns¶

Below is a single self-contained file demonstrating eight cache-aware patterns, each thoroughly commented. A senior reading this should be able to extract any pattern and adapt it.

// Package coherence demonstrates production-grade cache-aware patterns.
package coherence

import (
    "fmt"
    "runtime"
    "sync"
    "sync/atomic"
    "unsafe"

    "golang.org/x/sys/cpu"
)

// =============================================================
// Pattern 1: Padded atomic counter (basic building block)
// =============================================================

type PaddedCounter struct {
    v atomic.Int64
    _ cpu.CacheLinePad
}

func (c *PaddedCounter) Inc() { c.v.Add(1) }
func (c *PaddedCounter) Add(d int64) { c.v.Add(d) }
func (c *PaddedCounter) Load() int64 { return c.v.Load() }

// =============================================================
// Pattern 2: Per-CPU sharded counter
// =============================================================

type ShardedCounter struct {
    shards []PaddedCounter
    mask   uint64
}

func NewShardedCounter() *ShardedCounter {
    n := nextPow2(runtime.GOMAXPROCS(0))
    return &ShardedCounter{
        shards: make([]PaddedCounter, n),
        mask:   uint64(n - 1),
    }
}

func (c *ShardedCounter) Add(hint uint64, d int64) {
    c.shards[hint&c.mask].Add(d)
}

func (c *ShardedCounter) Sum() int64 {
    var s int64
    for i := range c.shards {
        s += c.shards[i].Load()
    }
    return s
}

func nextPow2(n int) int {
    if n <= 1 { return 1 }
    p := 1
    for p < n { p *= 2 }
    return p
}

// =============================================================
// Pattern 3: Read-mostly snapshot pointer
// =============================================================

type Snapshot[T any] struct {
    p atomic.Pointer[T]
    _ cpu.CacheLinePad
}

func (s *Snapshot[T]) Set(v *T) { s.p.Store(v) }
func (s *Snapshot[T]) Get() *T  { return s.p.Load() }

// =============================================================
// Pattern 4: Copy-on-write map
// =============================================================

type COWMap[K comparable, V any] struct {
    m atomic.Pointer[map[K]V]
}

func NewCOWMap[K comparable, V any]() *COWMap[K, V] {
    m := make(map[K]V)
    c := &COWMap[K, V]{}
    c.m.Store(&m)
    return c
}

func (c *COWMap[K, V]) Get(k K) (V, bool) {
    m := c.m.Load()
    v, ok := (*m)[k]
    return v, ok
}

func (c *COWMap[K, V]) Set(k K, v V) {
    for {
        old := c.m.Load()
        next := make(map[K]V, len(*old)+1)
        for k2, v2 := range *old {
            next[k2] = v2
        }
        next[k] = v
        if c.m.CompareAndSwap(old, &next) {
            return
        }
    }
}

// =============================================================
// Pattern 5: Hot/cold split struct
// =============================================================

type Service struct {
    // hot
    requests atomic.Int64
    _        cpu.CacheLinePad
    errors   atomic.Int64
    _        cpu.CacheLinePad

    // cold
    name     string
    address  string
    config   *Snapshot[Config]
    handlers map[string]Handler
}

type Config struct{}
type Handler func()

// =============================================================
// Pattern 6: Padded mutex registry (sharded locks)
// =============================================================

type MutexShard struct {
    mu sync.Mutex
    _  cpu.CacheLinePad
}

type LockRegistry struct {
    shards []MutexShard
    mask   uint64
}

func NewLockRegistry() *LockRegistry {
    n := 64 // tunable
    return &LockRegistry{
        shards: make([]MutexShard, n),
        mask:   uint64(n - 1),
    }
}

func (r *LockRegistry) Lock(key uint64) *sync.Mutex {
    m := &r.shards[key&r.mask].mu
    m.Lock()
    return m
}

// =============================================================
// Pattern 7: Aligned ring buffer with cached indices
// =============================================================

type RingBuffer[T any] struct {
    write       atomic.Uint64
    cachedRead  uint64
    _           cpu.CacheLinePad

    read        atomic.Uint64
    cachedWrite uint64
    _           cpu.CacheLinePad

    cells []ringCell[T]
    mask  uint64
}

type ringCell[T any] struct {
    seq  atomic.Uint64
    data T
    _    [40]byte
}

func NewRingBuffer[T any](size int) *RingBuffer[T] {
    size = nextPow2(size)
    cells := make([]ringCell[T], size)
    for i := range cells {
        cells[i].seq.Store(uint64(i))
    }
    return &RingBuffer[T]{
        cells: cells,
        mask:  uint64(size - 1),
    }
}

func (r *RingBuffer[T]) Enqueue(v T) bool {
    for {
        pos := r.write.Load()
        cell := &r.cells[pos&r.mask]
        seq := cell.seq.Load()
        if seq == pos {
            if r.write.CompareAndSwap(pos, pos+1) {
                cell.data = v
                cell.seq.Store(pos + 1)
                return true
            }
        } else if seq < pos {
            return false // full
        }
    }
}

func (r *RingBuffer[T]) Dequeue() (T, bool) {
    var zero T
    for {
        pos := r.read.Load()
        cell := &r.cells[pos&r.mask]
        seq := cell.seq.Load()
        if seq == pos+1 {
            if r.read.CompareAndSwap(pos, pos+1) {
                v := cell.data
                cell.seq.Store(pos + r.mask + 1)
                return v, true
            }
        } else if seq < pos+1 {
            return zero, false // empty
        }
    }
}

// =============================================================
// Pattern 8: Layout assertion
// =============================================================

func AssertOnDifferentLines(name string, a, b unsafe.Pointer) {
    cl := uintptr(cpu.CacheLinePadSize)
    if uintptr(a)/cl == uintptr(b)/cl {
        panic(fmt.Sprintf("%s: fields share a cache line (%p, %p)", name, a, b))
    }
}