Acquire / Release — Professional Level¶

Table of Contents¶

1. Introduction
2. Cross-Language Memory Models
3. C++ memory_order Reference
4. Rust Ordering Reference
5. Java volatile and VarHandle
6. Go's Sequential Consistency Choice
7. How Go Compiles Atomics
8. Per-Architecture Cost Models
9. The Go Runtime and Memory Barriers
10. Fence Elision
11. Linux Kernel RCU vs Go RCU
12. Hazard Pointers in Depth
13. Wait-Free Synchronization Theory
14. Universal Constructions
15. Read-Mostly Reclamation
16. NUMA Effects
17. Designing a New Concurrency Primitive
18. Formal Verification
19. Future of Go's Memory Model
20. Summary
21. Further Reading

Introduction¶

The professional level looks under the hood. You'll learn:

• Exactly what machine code Go emits for atomic.Store on x86, ARM64, and RISC-V.
• How Go's seq-cst choice compares to C++/Rust's flexible memory_order.
• The Go runtime's role in scheduling around memory barriers.
• Per-architecture cost models with cycle counts.
• Hazard pointers, epoch-based reclamation, and why Go's GC subsumes them.
• Formal verification techniques for concurrent algorithms.
• How to design a new concurrency primitive from scratch.

This level is for engineers who:

• Contribute to language runtimes (Go, Rust, C++).
• Write performance-critical libraries used by thousands of services.
• Design database engines, message queues, or storage systems.
• Investigate subtle production bugs caused by memory-model violations.

It is NOT required for application development. If you reached the senior level, you can ship correct concurrent Go code for the rest of your career without reading this file. This file is for the curious, the systems-builders, and the language designers.

Cross-Language Memory Models¶

Every modern language has a memory model. They differ in three dimensions:

1. Default behavior: what does a plain read/write mean?
2. Atomic vocabulary: what ordering options are exposed?
3. Synchronization primitives: what primitives provide which orderings?

A quick comparison:

Notes:

• "Race UB" means a data race is undefined behavior. The compiler is free to assume races don't happen.
• Java and C# guarantee that all reads return some value previously written (even if you raced) — no garbage, no torn reads of refs/booleans/etc.
• Go's choice of seq-cst-only is the most opinionated: simple to use, slightly slower on weakly ordered hardware.

C++ memory_order Reference¶

C++ exposes six memory orders:

enum memory_order {
    memory_order_relaxed,
    memory_order_consume,
    memory_order_acquire,
    memory_order_release,
    memory_order_acq_rel,
    memory_order_seq_cst
};

memory_order_relaxed¶

Atomic but unordered. No happens-before guarantees beyond the atomicity itself. Used for counters that don't need ordering (e.g., performance metrics).

std::atomic<int> counter{0};
counter.fetch_add(1, std::memory_order_relaxed);

Counter increments. No release of other writes; no acquire of others' writes.

memory_order_acquire and memory_order_release¶

The pair we've been discussing. Release on the producer side, acquire on the consumer side, on the same location.

std::atomic<bool> ready{false};
int data;

// Producer:
data = 42;
ready.store(true, std::memory_order_release);

// Consumer:
while (!ready.load(std::memory_order_acquire)) { }
assert(data == 42);

memory_order_acq_rel¶

For read-modify-write operations: the read is an acquire, the write is a release. Used for fetch_add, compare_exchange, etc.

std::atomic<int> v{0};
v.fetch_add(1, std::memory_order_acq_rel);

memory_order_seq_cst¶

Sequential consistency. The default. Provides a single global order of all seq_cst operations.

std::atomic<int> v{0};
v.store(1, std::memory_order_seq_cst);

memory_order_consume¶

Deprecated since C++17. The idea was a weaker form of acquire that only synchronizes for data dependencies. In practice, no compiler implements it correctly; everyone falls back to acquire.

Cost on real hardware (x86-64)¶

• relaxed: plain mov.
• acquire (load): plain mov (x86 loads are already acquire).
• release (store): plain mov (x86 stores are already release).
• acq_rel (RMW): LOCK CMPXCHG or similar locked instruction.
• seq_cst: locked instruction with implicit full fence; for plain stores, often XCHG.

So on x86, relaxed and acquire/release cost the same for loads/stores. The interesting cost difference is for seq_cst on writes (needs an extra mfence).

Cost on ARM64¶

• relaxed: plain LDR/STR.
• acquire load: LDAR.
• release store: STLR.
• acq_rel RMW: LDAXR/STXR with retry, or LDAR+STLR for newer ARMv8.1.
• seq_cst: LDAR/STLR plus DMB ISH for full fence.

On ARM, the difference between relaxed and acquire/release is one instruction. Between acq/rel and seq_cst, an additional DMB ISH is required.

Rust Ordering Reference¶

Rust's atomics mirror C++:

use std::sync::atomic::Ordering;

let v = AtomicU32::new(0);
v.store(1, Ordering::Release);
let x = v.load(Ordering::Acquire);

Ordering variants: Relaxed, Acquire, Release, AcqRel, SeqCst. (No Consume.)

Rust's compiler (rustc, via LLVM) generates the same instructions as Clang would for equivalent C++. Performance is essentially identical.

The main difference is the type system: Rust forces you to choose an Ordering on every call, which both empowers and forces you to think.

Java volatile and VarHandle¶

Java's volatile:

• A volatile read is an acquire.
• A volatile write is a release.
• All volatiles share a global ordering (almost like seq-cst but only among volatiles).

So Java's volatile is roughly Go's sync/atomic for individual fields. Stronger than C++ memory_order_acquire/release.

java.util.concurrent.atomic.AtomicInteger.compareAndSet is seq-cst.

java.lang.invoke.VarHandle (Java 9+) exposes explicit memory orderings like C++. Most code doesn't need this.

The Java Memory Model is actually well-defined for racy programs: a racy read returns some value previously written (or the initial 0), never garbage. This is unique among mainstream languages. The cost: compiler optimizations are more constrained.

Go's Sequential Consistency Choice¶

Why did Go pick seq-cst for all atomics?

1. Simplicity for programmers. No need to reason "is this acq_rel or seq_cst?" Programmers can't shoot themselves in the foot with a too-weak ordering.

2. DRF-SC is the goal. "Data-race-free programs are sequentially consistent." Most Go code is DRF. Making atomics seq-cst aligns with the DRF-SC theorem.

3. Modest cost on x86. x86 is essentially TSO (total store order), which is close to seq-cst. The extra cost is minimal for the dominant deployment platform.

4. Acceptable cost on ARM. ARMv8 has dedicated acquire/release load/store instructions; the extra fence for seq-cst is one DMB per store.

5. Avoids subtle bugs. Many real-world C++ bugs come from picking acq_rel when seq_cst was needed. Go eliminates this class.

The trade-off: code that could benefit from relaxed ordering on tight loops pays a small cost. For most code, this is invisible. For very hot atomic loops on weakly-ordered hardware, it's a few percent.

The Go team's bet: simplicity > peak performance for the 99% case. So far, it's been a good bet.

How Go Compiles Atomics¶

Let's trace x.Store(1) for var x atomic.Int32 through the compiler.

Step 1: package source¶

package main

import "sync/atomic"

var x atomic.Int32

func main() {
    x.Store(1)
}

Step 2: atomic.Int32.Store implementation¶

In src/sync/atomic/type.go:

type Int32 struct {
    _ noCopy
    v int32
}

func (x *Int32) Store(val int32) { StoreInt32(&x.v, val) }

StoreInt32 is declared in src/sync/atomic/doc.go:

func StoreInt32(addr *int32, val int32)

with the body in assembly per architecture.

Step 3: amd64 assembly¶

src/sync/atomic/asm.s redirects to runtime/internal/atomic. The actual implementation for amd64 is in runtime/internal/atomic/asm_amd64.s:

TEXT runtime/internal/atomic·Store(SB), NOSPLIT, $0-12
    MOVQ ptr+0(FP), BX
    MOVL val+8(FP), AX
    XCHGL AX, 0(BX)
    RET

The key instruction is XCHGL (exchange long). This is implicitly locked on x86 (any memory operand of XCHG is treated as having an implicit LOCK prefix), so it acts as a full memory barrier — providing seq-cst semantics.

A simpler MOVL would suffice for release semantics on x86, but to get seq-cst Go uses XCHG (or MOV + MFENCE; XCHG is shorter).

Step 4: amd64 atomic load¶

TEXT runtime/internal/atomic·Load(SB), NOSPLIT, $0-12
    MOVQ ptr+0(FP), AX
    MOVL 0(AX), AX
    MOVL AX, ret+8(FP)
    RET

Just a plain MOVL. On x86, loads are already acquire by default (TSO), so no fence is needed.

Step 5: arm64 store¶

TEXT runtime/internal/atomic·Store(SB), NOSPLIT, $0-12
    MOVD ptr+0(FP), R0
    MOVW val+8(FP), R1
    STLRW R1, (R0)
    RET

STLRW is "store release word." It guarantees that all prior writes by this CPU become visible before this store.

For seq-cst, Go (on older ARMv8) follows STLR with DMB ISH (data memory barrier, inner-shareable). On ARMv8.3+, STLR alone provides RC sc (release consistent with sequential consistency for cross-domain access).

Step 6: arm64 atomic load¶

TEXT runtime/internal/atomic·Load(SB), NOSPLIT, $0-12
    MOVD ptr+0(FP), R0
    LDARW (R0), R0
    MOVW R0, ret+8(FP)
    RET

LDARW is "load acquire word." It guarantees that subsequent reads/writes by this CPU happen after this load is visible globally.

Step 7: CAS on amd64¶

TEXT runtime/internal/atomic·Cas(SB),NOSPLIT,$0-17
    MOVQ ptr+0(FP), BX
    MOVL old+8(FP), AX
    MOVL new+12(FP), CX
    LOCK
    CMPXCHGL CX, 0(BX)
    SETEQ ret+16(FP)
    RET

The LOCK CMPXCHGL instruction is the heart of CAS on x86: atomically compare-and-swap with a LOCK prefix (which makes it a full barrier).

Step 8: CAS on arm64¶

TEXT runtime/internal/atomic·Cas(SB),NOSPLIT,$0-17
    MOVD ptr+0(FP), R0
    MOVW old+8(FP), R1
    MOVW new+12(FP), R2
loop:
    LDAXRW (R0), R3
    CMPW R1, R3
    BNE fail
    STLXRW R2, (R0), R4
    CBNZ R4, loop
    MOVB $1, ret+16(FP)
    RET
fail:
    MOVB $0, ret+16(FP)
    RET

On ARM, CAS is implemented with LL/SC: LDAXR (load exclusive with acquire) and STLXR (store exclusive with release). If the exclusive reservation is lost (between LDAXR and STLXR), STLXR fails and we retry.

This LL/SC pair is more complex than x86's CMPXCHG but allows more flexibility (you can compute on the loaded value before storing).

What this means for performance¶

• x86: stores cost an XCHG (~5-8 ns), loads are free (~1 ns), CAS costs LOCK CMPXCHG (~10-15 ns).
• ARM64: stores cost STLR + DMB (~5 ns), loads cost LDAR (~2 ns), CAS costs LL/SC with possible retry (~10-20 ns).
• RISC-V: similar to ARM64 but with explicit fence instructions.

In all cases, the operations are bounded — wait-free. Performance varies by architecture but is in the same ballpark.

Per-Architecture Cost Models¶

A more detailed look at concurrency primitives on common architectures:

x86-64 (Intel/AMD modern)¶

Operation                      Cycles    ns (3 GHz)
----                            ------    ---------
Plain MOV                       1          0.3
LOCK MOV (XCHG)                 20-40      6-13
LOCK CMPXCHG (success)          25-45      8-15
LOCK CMPXCHG (failure)          ~10        ~3
MFENCE                          5-10       1-3
Empty MOV+RET                   2          0.6
Function call                   3-5        1-2

Notes: numbers vary by microarchitecture. Cache-cold operations are much more expensive.

ARM64 (Apple, AWS Graviton, ARM Server)¶

Operation                      Cycles    ns (3 GHz)
----                            ------    ---------
Plain LDR/STR                   1-3        0.3-1
LDAR (acquire)                  3-5        1-2
STLR (release)                  3-5        1-2
LDAXR + STLXR (CAS)             10-20      3-7
DMB ISH                         3-5        1-2

ARM is generally on par with x86, with slightly cheaper atomics on average but more variance.

RISC-V¶

Operation                      Cycles    ns (clock-dependent)
----                            ------    ---------
Plain lw/sw                     1          ~ns
AMOSWAP                         varies     ~ns
LR/SC pair                      varies     ~ns
FENCE rw,rw                     varies     ~ns

RISC-V is too new for stable cost data; varies wildly by implementation.

PowerPC¶

PowerPC has very weak ordering. Atomics often require LWSYNC fences. Go's atomics on POWER use these fences appropriately.

The Go Runtime and Memory Barriers¶

The Go runtime itself uses memory barriers in several places:

Goroutine scheduling¶

When a goroutine is descheduled and resumed on a different P (processor), the runtime emits appropriate barriers to ensure the goroutine's view of memory is consistent. This is invisible to user code but essential.

Garbage collection¶

The concurrent GC interacts with the user program in nuanced ways. Write barriers (different from memory barriers, despite the name) track reference updates during GC. The runtime emits memory barriers at specific points to ensure GC sees a consistent view.

Stack growth¶

When a goroutine's stack grows, the runtime copies the stack to a new location. This requires careful synchronization to ensure the goroutine resumes with the right pointer adjustments.

Channels¶

Channel operations involve sending/receiving via a heap-allocated hchan. The runtime uses an internal mutex (not sync.Mutex, but runtime.mutex) plus atomic state to coordinate.

sync.Mutex slow path¶

When contended, sync.Mutex parks the calling goroutine via runtime_SemacquireMutex. This eventually calls into kernel-level futex (Linux) or equivalent. The wake-up path uses atomics plus runtime scheduling.

For each of these, the runtime carefully chooses primitives to maintain Go's memory model promises. User code doesn't see this complexity — but understanding it helps when debugging runtime-level issues.

Fence Elision¶

Compilers can sometimes elide memory fences when they prove the fence isn't needed.

Example: a release store followed by another release store to the same location. The first release fence is redundant (the second covers everything).

Go's compiler does some elision but not as aggressively as Clang/GCC for C++. The Go team's preference is correctness > peak performance.

Examples where fences are NOT elided:

• A release store followed by a non-atomic store followed by another release store. Both releases are necessary (the non-atomic store could be reordered otherwise).
• A pair of release stores on different locations. Both are needed.

Examples where fences COULD be elided (and Go may or may not):

• Two consecutive release stores on the same location.
• A release store immediately followed by a release fence.

For most code, this doesn't matter. For tight atomic loops, you might want to inspect the generated assembly with go tool objdump.

Linux Kernel RCU vs Go RCU¶

The Linux kernel's RCU implementation is famously complex. It handles:

1. Read-side critical sections: marked with rcu_read_lock() / rcu_read_unlock(). These are no-ops on the read side — just compiler barriers.

2. Grace periods: the writer waits until all CPUs have passed through a context switch (or other quiescent state). At that point, no reader holds an old pointer.

3. Deferred reclamation: call_rcu(callback) schedules a function to run after the next grace period.

4. Synchronize: synchronize_rcu() blocks until the next grace period.

Why is this complex in C? Because:

• No GC; memory must be explicitly freed.
• Readers can be preempted; you can't assume read-side critical sections are short.
• Multiple CPUs run independently; "all readers are done" must be detected explicitly.

In Go, the GC subsumes all of this. A reader holds a *T; as long as it does, the GC won't free it. When the reader returns, the local variable goes out of scope, and the GC can reclaim.

The cost: GC pauses (though Go's GC is concurrent and pause times are typically <1 ms).

For most Go services, this is fine. For real-time systems where >1 ms pauses are unacceptable, you'd implement explicit RCU-like reclamation — but rarely in pure Go.

Hazard Pointers in Depth¶

Hazard pointers are a memory reclamation scheme for lock-free structures. Each reader publishes (in a thread-local atomic slot) the pointers it's currently "holding." Writers, before freeing a pointer, check that no hazard pointer matches.

Pseudo-implementation in Go (simplified, ignoring per-thread storage):

var hazards [MaxThreads]atomic.Pointer[node]

func read(idx int, p atomic.Pointer[node]) *node {
    for {
        n := p.Load()
        hazards[idx].Store(n)
        if p.Load() == n {
            return n // hazard registered before any retire
        }
    }
}

func retire(n *node) {
    waitForHazards(n)
    free(n) // or queue for delayed free
}

func waitForHazards(n *node) {
    for {
        clean := true
        for i := 0; i < MaxThreads; i++ {
            if hazards[i].Load() == n {
                clean = false
                break
            }
        }
        if clean {
            return
        }
        runtime.Gosched()
    }
}

In a language without GC, this is the only way to safely free nodes in a lock-free data structure. In Go, the GC handles this for you, so hazard pointers are rarely needed.

Exception: when you bypass GC (e.g., using sync.Pool to recycle nodes, or unsafe.Pointer for foreign memory), you need explicit reclamation.

The full hazard pointer protocol has subtleties about ordering (the store of the hazard pointer must be visible before the re-check of p.Load()). In C++ this requires memory_order_seq_cst or careful use of fences. In Go, the seq-cst default handles it.

Wait-Free Synchronization Theory¶

Maurice Herlihy's 1991 paper "Wait-Free Synchronization" laid out the theoretical foundations:

• A consensus number of an object is the maximum number of threads that can solve consensus using only that object.
• Atomic registers (loads/stores) have consensus number 1.
• Atomic test-and-set has consensus number 2.
• Atomic compare-and-swap has consensus number infinity (∞).

Implication: only CAS (or equivalent) is "universal" — it can implement wait-free versions of any object. Loads/stores alone cannot solve consensus for 2+ threads.

This is why CAS is the workhorse of lock-free programming.

The theorem also implies that you cannot implement a wait-free queue using only atomic loads/stores. You need CAS (or LL/SC, or another universal primitive).

In Go, all the universal primitives are available: atomic.CompareAndSwap, atomic.Swap, atomic.LoadAndStore. Use them for lock-free designs.

Universal Constructions¶

Herlihy also showed that CAS lets you build a wait-free version of any object, mechanically. The construction:

1. Each operation is encoded as a small struct.
2. Threads compete to install their operation at the next slot in a log.
3. After installing, threads "help" by applying logged operations in order.

The construction is wait-free but slow: every operation involves a CAS plus log traversal. In practice, you use it as a proof of feasibility, not for production code.

For production wait-free code, design for the specific problem. Examples:

• Wait-free queue: Vyukov's MPMC bounded ring.
• Wait-free stack: difficult; usually settle for lock-free.
• Wait-free hashmap: Cliff Click's NonBlockingHashMap (Java).

Read-Mostly Reclamation¶

For read-mostly structures, reclamation is the bottleneck. Several approaches:

Epoch-based reclamation¶

Maintain a global epoch counter. Readers enter the current epoch; writers may free nodes from epochs N-2 or older once all readers have passed through.

Implementation requires per-thread epoch tracking. In Go, this can be done with per-P storage via runtime hooks, but it's complex.

Quiescent-state-based reclamation (QSBR)¶

Similar to epoch-based but tied to specific quiescent states (e.g., context switches). Used in Linux kernel RCU.

Stamp-based reclamation¶

Each pointer carries a version stamp. Readers track the highest stamp they've seen; writers free pointers below the minimum.

Reference counting¶

Each pointer carries an atomic refcount. Decrement on release; free when zero.

For Go, the GC handles all of these implicitly. You rarely need to choose.

NUMA Effects¶

On large servers with multiple sockets (NUMA — Non-Uniform Memory Access), memory access cost depends on which socket the data lives on.

• Local memory: ~80 ns.
• Remote memory: ~120-200 ns.
• Cache invalidations across sockets: ~200-500 ns.

For NUMA-aware concurrent code:

• Pin goroutines to a socket (Go doesn't expose this directly; use GOMAXPROCS and taskset).
• Allocate per-socket data.
• Use per-socket sharding rather than global state.

Most Go services don't care because they run on single-socket cloud VMs. For NUMA databases (e.g., Postgres on a big bare-metal server), the runtime/scheduling implications matter.

Designing a New Concurrency Primitive¶

When you design a new primitive, the process:

1. Specify the contract. What does it guarantee? Acquire? Release? Linearizable? Wait-free?
2. Sketch the algorithm. Use the smallest set of primitives (atomics, CAS).
3. Identify happens-before chains. Document every release-acquire pair.
4. Reason about ABA. If pointers are recycled, you need generation counters or hazard pointers.
5. Prove correctness. TLA+ or hand-written proof.
6. Implement. Use sync/atomic; document the contract.
7. Test. Stress test with -race -count=100.
8. Benchmark. Compare to alternatives.
9. Document. Include examples, contract, performance characteristics.

Example: designing a wait-free read snapshot of multiple atomics.

Contract: Snapshot() returns a consistent view of fields X, Y, Z at some instant.

Sketch: seqlock with three atomic fields + a generation counter.

Happens-before: writer increments gen (odd), updates fields, increments gen (even). Reader reads gen, fields, gen. Match → consistent.

ABA: only if gen wraps around. With uint64, this is 2^64 increments — effectively never.

Implementation: see the seqlock section in senior.md.

This is the workflow for any new primitive.

Formal Verification¶

For library code where correctness must be ironclad, formal verification helps.

TLA+¶

TLA+ models concurrent systems as state machines. You write a specification of the behavior; the TLC model checker explores reachable states.

Example: spec for a lock-free queue.

EXTENDS Naturals, Sequences

VARIABLE queue, locked

Init == queue = <<>> /\ locked = FALSE

Enqueue(v) == 
  /\ ~locked
  /\ locked' = TRUE
  /\ queue' = Append(queue, v)
  /\ UNCHANGED <<>>

Unlock ==
  /\ locked
  /\ locked' = FALSE
  /\ UNCHANGED queue

TLC explores all interleavings, checking invariants. If a violation exists, TLC produces a trace.

Real TLA+ specs for lock-free queues are longer (modeling each CAS, each retry), but the principle is the same.

Promela / SPIN¶

Similar tool, popular for kernel-level verification.

Hand-written proofs¶

A natural-language proof in the comments, citing the memory model axioms. Faster than formal verification; less rigorous.

For Go code, hand-written proofs are typically sufficient. For database engines or message brokers, TLA+ is worth the investment.

Future of Go's Memory Model¶

Go's memory model has been stable since 2009 with a major clarification in 2022. Future evolution:

• Atomic types with explicit ordering: There has been discussion of adding atomic.LoadAcquire / atomic.StoreRelease for performance. Not adopted as of Go 1.22.
• Wait-free primitives: more wait-free types in the standard library.
• Better runtime support for NUMA: opaque to user code, but improving.
• Generic atomics: already added (atomic.Pointer[T] in 1.19).

The Go team's philosophy: prefer simplicity, add complexity only with strong evidence. The current memory model is unlikely to change radically.

Summary¶

The professional level demands:

• Knowledge of Go's atomics down to the machine instruction.
• Cross-language perspective (C++, Rust, Java).
• Runtime internals and how the scheduler interacts with memory barriers.
• Theoretical foundations: consensus, wait-freedom, universal construction.
• Practical reclamation: hazard pointers, epochs, QSBR.
• NUMA awareness.
• The ability to design and verify new primitives.

If you've absorbed everything in this file, you can:

• Contribute to a language runtime.
• Design a database engine's concurrency layer.
• Diagnose production bugs caused by memory-model violations.
• Translate algorithms between languages with different memory models.
• Teach concurrency at the staff level.

You are equipped to push the state of the art forward, not just consume it.

Further Reading¶

• Maurice Herlihy, "Wait-Free Synchronization" (TOPLAS 1991).
• Hans Boehm, "Threads Cannot Be Implemented as a Library" (PLDI 2005).
• Russ Cox, "Hardware Memory Models" (2021).
• Russ Cox, "Programming Language Memory Models" (2021).
• Sarita Adve and Hans-J. Boehm, "Memory Models: A Case for Rethinking Parallel Languages and Hardware" (CACM 2010).
• Doug Lea, "The JSR-133 Cookbook" (2005).
• C++ Standard ISO/IEC 14882:2020, Section [intro.races].
• Rust Reference, "Memory Model" chapter.
• McKenney et al., "Is Parallel Programming Hard, And, If So, What Can You Do About It?" (latest edition).
• Vyukov's blog: https://www.1024cores.net.

End of professional level content — extended below.

Appendix A: Reading the Go Assembly Output¶

To understand what your code becomes, use go tool objdump or compile with -S:

go build -gcflags="-S" -o /dev/null . 2>&1 | less

This dumps the assembly for every function. Look for sync/atomic calls — they translate to inlined instructions or direct calls into runtime/internal/atomic.

For deeper inspection:

go build -o myprog .
go tool objdump -s '^main\.' myprog

Dumps the disassembly of main.* functions. You'll see the actual machine instructions: LOCK CMPXCHG, XCHG, LDAR, STLR, etc.

Example: tracing x.Store(1)¶

Source:

var x atomic.Int32

func setX() {
    x.Store(1)
}

Disassembly (amd64):

main.setX:
  MOVL    $1, AX
  MOVQ    main.x(SB), BX
  XCHGL   AX, 0(BX)
  RET

The XCHGL is the seq-cst store. Note: no explicit LOCK prefix needed because XCHG is implicitly locked on memory operands.

Example: tracing x.Load()¶

Source:

func getX() int32 {
    return x.Load()
}

Disassembly (amd64):

main.getX:
  MOVQ    main.x(SB), AX
  MOVL    0(AX), AX
  RET

Just a MOV. x86's strong ordering makes plain loads acquire by default.

Example: tracing x.CompareAndSwap(0, 1)¶

func casX() bool {
    return x.CompareAndSwap(0, 1)
}

Disassembly:

main.casX:
  XORL    AX, AX
  MOVL    $1, CX
  MOVQ    main.x(SB), DX
  LOCK
  CMPXCHGL CX, 0(DX)
  SETEQ   AX
  RET

LOCK CMPXCHGL is the heart. The SETEQ extracts whether the comparison succeeded.

Appendix B: Memory Ordering on Specific Hardware¶

Different generations of CPU have different memory model details.

Intel x86-64¶

• Total Store Order (TSO).
• All stores are visible in program order.
• Loads can be reordered before stores to different addresses (store buffer).
• Atomic operations (LOCK-prefixed) act as full memory barriers.
• MFENCE is a full barrier; LFENCE and SFENCE are specific.

AMD x86-64¶

• Identical to Intel for ordering purposes (TSO).
• Minor microarchitectural differences in atomic performance.

Apple M-series (ARM64)¶

• ARMv8.5+ with custom enhancements.
• Strong implementation of acquire/release semantics.
• LDAR and STLR are first-class instructions.
• DMB ISH for full barriers.

AWS Graviton (ARM64)¶

• Standard ARMv8.4.
• Similar performance to Apple Silicon for atomics.

ARM Cortex-A series (general)¶

• Weakly ordered.
• LDAR/STLR for acquire/release.
• DMB ISH for full fences.

RISC-V¶

• Even weaker than ARM.
• FENCE rw,rw for full barriers.
• Atomic instructions: LR/SC pair, AMOSWAP, AMOADD, etc.

POWER (ppc64)¶

• Very weakly ordered.
• lwsync for acquire/release.
• sync for full fence.

The Go runtime emits the appropriate barriers for each. Your code is portable.

Appendix C: C++ vs Go — Side-by-Side Patterns¶

Pattern: lazy init¶

C++:

#include <atomic>
#include <mutex>

std::atomic<Service*> instance{nullptr};
std::mutex mu;

Service* get() {
    Service* s = instance.load(std::memory_order_acquire);
    if (s) return s;
    std::lock_guard<std::mutex> lock(mu);
    s = instance.load(std::memory_order_relaxed);
    if (!s) {
        s = new Service();
        instance.store(s, std::memory_order_release);
    }
    return s;
}

Go:

var (
    once     sync.Once
    instance *Service
)

func Get() *Service {
    once.Do(func() { instance = newService() })
    return instance
}

Go's sync.Once hides the DCL pattern entirely.

Pattern: lock-free stack¶

C++ (Treiber stack with hazard pointers):

template<typename T>
class Stack {
    struct Node {
        T val;
        std::atomic<Node*> next;
    };
    std::atomic<Node*> head{nullptr};
public:
    void push(T v) {
        Node* n = new Node{std::move(v), nullptr};
        Node* old = head.load(std::memory_order_relaxed);
        do {
            n->next.store(old, std::memory_order_relaxed);
        } while (!head.compare_exchange_weak(old, n,
            std::memory_order_release, std::memory_order_relaxed));
    }

    std::optional<T> pop() {
        // hazard pointer management omitted
        Node* old = head.load(std::memory_order_acquire);
        while (old && !head.compare_exchange_weak(old, old->next.load(),
            std::memory_order_acquire, std::memory_order_acquire)) {
        }
        if (!old) return {};
        T v = std::move(old->val);
        retire(old);
        return v;
    }
};

Go (Treiber stack):

type Stack[T any] struct {
    head atomic.Pointer[node[T]]
}

type node[T any] struct {
    val  T
    next *node[T]
}

func (s *Stack[T]) Push(v T) {
    n := &node[T]{val: v}
    for {
        n.next = s.head.Load()
        if s.head.CompareAndSwap(n.next, n) {
            return
        }
    }
}

func (s *Stack[T]) Pop() (T, bool) {
    for {
        top := s.head.Load()
        if top == nil {
            var zero T
            return zero, false
        }
        if s.head.CompareAndSwap(top, top.next) {
            return top.val, true
        }
    }
}

Go is dramatically shorter. The GC handles reclamation; no hazard pointers needed.

Pattern: read-mostly state¶

C++ (read-copy-update via shared_ptr):

std::atomic<std::shared_ptr<Config>> current;

std::shared_ptr<Config> get() {
    return current.load();
}

void update(std::shared_ptr<Config> next) {
    current.store(std::move(next));
}

(C++20's atomic handles refcounting.)

Go:

var current atomic.Pointer[Config]

func Get() *Config { return current.Load() }
func Set(c *Config) { current.Store(c) }

Same idea, simpler syntax in Go.

Appendix D: Rust vs Go — Side-by-Side¶

Rust's Arc<T> is the equivalent of Go's GC-managed pointer.

Pattern: lazy init¶

Rust:

use std::sync::OnceLock;

static INSTANCE: OnceLock<Service> = OnceLock::new();

fn get() -> &'static Service {
    INSTANCE.get_or_init(|| Service::new())
}

Go:

var (
    once     sync.Once
    instance *Service
)

func Get() *Service {
    once.Do(func() { instance = newService() })
    return instance
}

Roughly equivalent.

Pattern: atomic counter¶

Rust:

use std::sync::atomic::{AtomicI64, Ordering};

static COUNTER: AtomicI64 = AtomicI64::new(0);

pub fn inc() { COUNTER.fetch_add(1, Ordering::SeqCst); }
pub fn get() -> i64 { COUNTER.load(Ordering::SeqCst) }

Go:

var counter atomic.Int64

func Inc() { counter.Add(1) }
func Get() int64 { return counter.Load() }

Notice Rust forces Ordering choice; Go always uses seq-cst.

Pattern: channel-based fan-out¶

Rust (using crossbeam):

use crossbeam_channel::{bounded, Sender, Receiver};

let (tx, rx) = bounded::<i32>(10);
for _ in 0..4 {
    let rx = rx.clone();
    std::thread::spawn(move || {
        while let Ok(v) = rx.recv() {
            process(v);
        }
    });
}

Go:

ch := make(chan int, 10)
for i := 0; i < 4; i++ {
    go func() {
        for v := range ch {
            process(v)
        }
    }()
}

Go's syntax is more compact; Rust requires explicit cloning of receivers.

Appendix E: Java's Memory Model — Detail¶

Java has a unique memory model:

1. Volatile: read = acquire, write = release. All volatiles are totally ordered.
2. Synchronized: provides full memory barriers around enter/exit. Lock acquire is acquire; release is release.
3. Final fields: a special guarantee that, after construction, final fields are visible to all readers without synchronization. This makes String and Integer thread-safe by construction.
4. Atomic classes (AtomicInteger, etc.): seq-cst semantics.
5. VarHandle (Java 9+): exposes C++-like explicit orderings.

Java's volatile is roughly equivalent to Go's sync/atomic for a single field — both provide release/acquire.

But Java's memory model also defines behavior under data races: a racy read returns some value previously written. This is stronger than Go (where racy reads are undefined behavior).

The cost: Java compilers cannot reorder as aggressively. Many optimizations that GCC/Clang perform on C++ are forbidden by the JMM.

Appendix F: Why Go Doesn't Expose memory_order¶

The Go team's reasoning, paraphrased:

1. Most Go code doesn't need it. 99% of concurrency in Go is through channels and mutexes; the memory ordering is implicit.

2. Programmers get it wrong. C++ memory_order is famously misused. Even experts have written buggy lock-free code.

3. Seq-cst is good enough. The performance overhead vs. acq/rel is small (a few ns) and only matters in atomic-heavy hot loops.

4. Simpler model is easier to specify. Go's memory model is shorter than C++'s precisely because it avoids the matrix of orderings.

5. If you need it, you can drop to assembly. For runtime contributors or extreme performance, runtime/internal/atomic (internal) and Go assembly give finer control.

The Go community has discussed adding atomic.LoadAcquire / atomic.StoreRelease periodically. It hasn't been adopted because:

• The cost savings are marginal for most code.
• It would add complexity to a deliberately simple model.
• Existing patterns (atomic.Pointer with CoW, sync.Once) cover the high-value cases.

This may change in the future, but as of Go 1.22, seq-cst is the only choice.

Appendix G: The Cost of Sequential Consistency, Measured¶

Let's measure the real cost of seq-cst vs. release-only.

Benchmark setup:

package main

import (
    "sync/atomic"
    "testing"
)

var v atomic.Int32

func BenchmarkAtomicStoreSeqCst(b *testing.B) {
    for i := 0; i < b.N; i++ {
        v.Store(int32(i))
    }
}

Equivalent C++ benchmark with memory_order_release:

#include <atomic>

std::atomic<int> v;

void bench(int n) {
    for (int i = 0; i < n; i++) {
        v.store(i, std::memory_order_release);
    }
}

Same with memory_order_seq_cst:

void bench_sc(int n) {
    for (int i = 0; i < n; i++) {
        v.store(i, std::memory_order_seq_cst);
    }
}

Results (approximate, x86-64 modern):

• C++ release: ~5 ns/op
• C++ seq-cst: ~6 ns/op
• Go seq-cst: ~6 ns/op

ARM64 results:

• C++ release: ~3 ns/op (STLR alone)
• C++ seq-cst: ~5 ns/op (STLR + DMB ISH)
• Go seq-cst: ~5 ns/op

The seq-cst penalty is 1-2 ns per store. For most code, negligible. For tight atomic loops at millions of ops per second, ~5-10% throughput cost.

Appendix H: Production Story — A Real Atomic Bug¶

A production incident:

A team had a counter incremented from many goroutines via atomic.AddInt64. The counter feeds metrics every 10 seconds.

The metric occasionally jumped backward by ~50% then recovered. Investigation:

• The counter type was correct (atomic.Int64).
• The increment was correct (Add(1)).
• The read was via Load().

After a week of debugging, they realized: the metrics handler also reset the counter every 10 seconds via Store(0). The reset was racing with concurrent increments.

go func() {
    for {
        time.Sleep(10 * time.Second)
        report(counter.Load())
        counter.Store(0) // RACE: increments lost
    }
}()

If an increment landed between the Load and the Store, it was lost (the Store overwrote it).

Fix: use Swap(0) to atomically read-and-reset:

v := counter.Swap(0)
report(v)

Now the read and reset are atomic. No increments lost.

Lesson: even with correct atomics, the protocol must be correct. Atomics provide atomicity per-operation; you must compose them correctly.

Appendix I: A NUMA Scaling Case Study¶

A team running PostgreSQL on a 4-socket server (128 cores) observed weird scaling: throughput peaked at 64 connections, decreased thereafter.

Investigation: every connection was spinning on a shared atomic counter (PostgreSQL's xlog.PgXact). At 64+ cores, cache-line bouncing between sockets dominated.

Fix in PG: shard the counter per-socket. Each socket increments its local counter; the global value is the sum on read.

This is a NUMA-specific scaling pattern. Few Go services hit this issue because they typically run on single-socket cloud VMs. But for those that do, sharding is the answer.

In Go, the equivalent for a heavily-contended counter:

type ShardedCounter struct {
    shards []paddedInt64
}

type paddedInt64 struct {
    n atomic.Int64
    _ [56]byte
}

func (c *ShardedCounter) Inc() {
    s := getCurrentShard()
    c.shards[s].n.Add(1)
}

func (c *ShardedCounter) Get() int64 {
    var sum int64
    for i := range c.shards {
        sum += c.shards[i].n.Load()
    }
    return sum
}

Shards equal to NumCPU (or 2x for some workloads) avoid cross-socket contention.

Appendix J: Designing a Lock-Free Hashmap¶

A real lock-free hashmap is among the hardest data structures. Key challenges:

1. Concurrent resize. The hashmap must grow when full, but readers and writers must keep working during resize.

2. Deletion. Marking entries as "deleted" without breaking concurrent traversal.

3. Memory reclamation. Freeing removed nodes safely.

Production implementations:

• Java's ConcurrentHashMap (Cliff Click's design, JDK 8+).
• C++'s Folly ConcurrentHashMap.
• Go's sync.Map.

sync.Map is the simplest. It uses a "read-only snapshot + dirty mutex-protected delta" pattern, not true lock-free.

For a research-quality lock-free hashmap in Go, consult Cliff Click's papers. The Go equivalent would be ~500-1000 lines. Most Go programs don't need this.

Simplified split-ordered list approach¶

A hashmap based on split-ordered linked lists (Shalev & Shavit, 2006):

1. Maintain a linked list ordered by reverse-bit hash.
2. Buckets are pointers into this list at specific points.
3. Inserting follows the linked list (lock-free) and updates the bucket pointer.
4. Resize doubles the number of buckets; new buckets are inserted into the existing list (lazy).

Implementation involves several atomic Pointer fields, careful CAS sequences, and a bit of reverse-bit math. It's a research-quality implementation in any language.

For Go, the recommendation is: use sync.Map or a sharded sync.RWMutex + map. Don't roll your own lock-free hashmap unless you're prepared for a 6-month project.

Appendix K: Database Engine Concurrency¶

Database engines (Postgres, MySQL, SQLite, RocksDB) use elaborate concurrency mechanisms:

• MVCC (Multi-Version Concurrency Control): each row has multiple versions; readers see the snapshot at transaction start.
• Write-Ahead Logging (WAL): writes are appended to a log; the log is fsynced periodically.
• Two-Phase Locking (2PL): each transaction acquires locks; locks held until commit.
• Optimistic Concurrency Control (OCC): transactions proceed without locks; conflict detected at commit time.

Each has implications for memory ordering:

• MVCC: snapshot publication needs acquire/release on the version pointer.
• WAL: fsync provides a durability barrier (different from memory ordering).
• 2PL: lock acquisition is acquire; release is release.
• OCC: CAS on the row's version stamp.

If you build a storage engine in Go, you'll touch all of these. Understanding acquire/release semantics is necessary.

Appendix L: Message Queue Concurrency¶

Message queues (Kafka, RabbitMQ, NATS) handle:

• Per-partition ordering.
• At-least-once vs exactly-once delivery.
• Concurrent producers and consumers.

The publication semantics within a single broker:

• A producer's send must be durable (fsync) before ack.
• A consumer's offset commit must be durable before reusing the consumer slot.

For an in-process Go queue (like a worker pool), the publication is simpler: channels (or atomic ring buffers) handle it.

Appendix M: A Walkthrough of sync/atomic Source¶

Read these files in your Go installation:

• src/sync/atomic/doc.go: declarations of all functions.
• src/sync/atomic/type.go: type wrappers (Int32, Pointer[T], etc.).
• src/sync/atomic/value.go: atomic.Value implementation.
• src/runtime/internal/atomic/: per-architecture assembly.

You'll see:

• Type wrappers are minimal — they delegate to runtime/internal/atomic.
• atomic.Value uses an internal mutex (fastpathlock) for type checking on first store.
• Assembly is short, mostly one or two instructions per operation.

Reading the runtime/internal/atomic per-architecture files is illuminating. You'll see the exact assembly emitted for amd64, arm64, ppc64, riscv64, mips, etc.

Appendix N: Implementing a Wait-Free Counter¶

A wait-free counter sounds trivial: atomic.Int64.Add(1). But under extreme contention, each Add takes longer because of cache invalidation across cores.

True wait-freedom under contention: each goroutine increments its own per-CPU counter; readers sum them.

type WFCounter struct {
    shards []paddedInt64
}

func NewWFCounter() *WFCounter {
    return &WFCounter{shards: make([]paddedInt64, runtime.NumCPU())}
}

func (c *WFCounter) Inc() {
    p := getProcID() // runtime-specific
    c.shards[p].n.Add(1)
}

func (c *WFCounter) Read() int64 {
    var s int64
    for i := range c.shards {
        s += c.shards[i].n.Load()
    }
    return s
}

Each Inc is wait-free per goroutine (one atomic operation on a private cache line). Read is O(NumCPU).

Caveats:

• getProcID() isn't directly exposed in Go. The closest equivalent is runtime_procPin (internal). Hashing the goroutine ID is a workaround.
• Read sums shards at some moment; the sum may not match any single instant. For monitoring, this is fine.

Appendix O: Implementing a Hierarchical Counter¶

For very high-throughput counters with many readers, hierarchical counters reduce read cost:

• Level 0: per-CPU shards (fast Inc).
• Level 1: aggregated periodically into a smaller set of shards.
• Level 2: global counter.

Reads at level 2 are cheap (one atomic load) but may be slightly stale. Writes at level 0 are wait-free.

This pattern is used in eBPF performance counters and similar telemetry.

In Go, implementing this is non-trivial because the aggregation thread must run periodically and the GC must handle the cross-level pointers. For most monitoring use cases, the simple sharded counter is good enough.

Appendix P: Beyond Acquire/Release — Release Consistency Variants¶

Modern hardware sometimes supports stronger or weaker forms than acquire/release:

• Release Consistency (RC): the canonical form.
• Lazy Release Consistency (LRC): writes are buffered until acquire.
• Eager Release Consistency: writes are published on release without waiting for acquire.
• Entry Consistency: each shared object has its own synchronization variable.
• Causal+ Consistency: causality plus per-object orderings.

These are research models for distributed systems and some hardware. Go uses RC for atomics (under seq-cst as a stronger form).

If you work on distributed systems (not single-process Go), you'll encounter these. The relationships are formalized in Adve & Hill, "Weak Ordering — A New Definition" (1990) and subsequent papers.

Appendix Q: Cache Coherence Protocols¶

CPU caches use coherence protocols to keep multiple cores in sync.

MESI (Modified, Exclusive, Shared, Invalid)¶

Four states per cache line:

• Modified: the line is dirty; this cache has the only copy.
• Exclusive: the line matches memory; this cache has the only copy.
• Shared: multiple caches have read-only copies.
• Invalid: the line is not in this cache.

Transitions:

• Read miss → Shared (if elsewhere) or Exclusive (if not).
• Write miss → invalidate other caches → Modified.
• Other write → Invalid here, Modified elsewhere.

MOESI (adds Owned)¶

• Owned: a hybrid of Shared (others have copies) and Modified (this cache must write back). Used in AMD CPUs.

MESIF (adds Forward)¶

• Forward: one cache is responsible for sourcing a Shared line on miss. Reduces cache-to-cache transfer overhead in NUMA. Used in Intel CPUs.

For programmers, the protocol details rarely matter — but they explain why cross-core writes are expensive (state transitions) and why false sharing hurts (innocent writes invalidate each other's cache lines).

Appendix R: When the Compiler Lies¶

Compilers may reorder code in ways you don't expect, as long as the reorder is invisible within a single goroutine. To force the compiler not to reorder, use atomic operations.

Example:

func f() {
    x = 1
    y = 2
}

The compiler may emit y = 2; x = 1 if it sees no dependency. For single-goroutine code, this is fine.

For multi-goroutine code, the atomic acts as a barrier:

func f() {
    x = 1
    atomic.StoreInt32(&y, 2) // also a compiler barrier
}

Now x = 1 must happen before y = 2 (both in source code order and in the emitted assembly).

For non-atomic shared memory, use a mutex:

func f() {
    mu.Lock()
    x = 1
    y = 2
    mu.Unlock()
}

The Lock and Unlock are compiler barriers; nothing inside is reordered out.

Appendix S: Atomicity Granularity¶

Go's atomic operations work on specific sizes:

• atomic.Int32/atomic.Uint32: 32-bit.
• atomic.Int64/atomic.Uint64: 64-bit (aligned).
• atomic.Uintptr: pointer-sized.
• atomic.Pointer[T]: pointer-sized.
• atomic.Bool: implemented as uint32.
• atomic.Value: variable-size, mutex-protected.

For other sizes:

• Pack into one of the above. E.g., two int16s in an int32.
• Use an atomic.Pointer[T] to a heap-allocated struct.
• Use a mutex.

128-bit atomics (CMPXCHG16B on x86-64, paired CAS on ARM) are not exposed in sync/atomic. They're available in runtime/internal/atomic but not part of the public API.

Appendix T: Atomicity and Alignment¶

64-bit atomic operations require 8-byte alignment on most architectures. Go's atomic.Int64 etc. struct types guarantee alignment.

For raw int64 fields:

type Bad struct {
    flag int32
    n    int64 // may be at offset 4 — misaligned on 32-bit ARM
}

On 64-bit platforms, this is fine. On 32-bit ARM, atomic.AddInt64(&b.n, 1) crashes.

Fix:

type Good struct {
    n    atomic.Int64 // wrapper guarantees alignment
    flag int32
}

Or use a _ [4]byte pad:

type Manual struct {
    flag int32
    _    [4]byte
    n    int64 // now aligned
}

Modern Go (1.19+) prefers the struct wrapper.

Appendix U: Cross-Goroutine State and Stack Movement¶

Go's runtime moves goroutine stacks when they grow. If a non-atomic pointer to a stack variable is shared across goroutines, the stack movement can invalidate the pointer.

This is rare because:

• Most shared state is heap-allocated.
• The compiler usually escapes captured variables to the heap.

But beware:

func bad() {
    var x int
    go func() {
        // captures &x; may escape to heap implicitly
        atomic.StoreInt32((*int32)(unsafe.Pointer(&x)), 1)
    }()
}

If x escapes to heap, fine. If somehow it doesn't (rare), the pointer could move. Don't rely on this.

The lesson: don't share pointers to local variables across goroutines via unsafe. Stick to heap-allocated state.

Appendix V: Profiling Memory Barriers¶

You can sometimes see memory barriers in CPU profiles:

$ go test -cpuprofile=cpu.out -bench=AtomicHeavy
$ go tool pprof -text cpu.out

Look for:

• runtime.atomicstore64: 64-bit atomic store.
• runtime.atomicload64: 64-bit atomic load.
• runtime.casgo: CAS for pointer.
• runtime.lock/runtime.unlock: internal mutex.
• sync.(*Mutex).lockSlow: contended mutex path.

If lockSlow is high, you have mutex contention. If casgo is high with many retries, CAS is contended.

The block profile (runtime.SetBlockProfileRate(1)) shows where goroutines block. The mutex profile shows which mutexes are contended.

For atomic-heavy code, look at the disassembly to see exactly what instructions are emitted.

Appendix W: Reasoning About Concurrent Code Mathematically¶

For very rigorous reasoning, encode concurrent code as a transition system:

• States: tuples of all shared variables and per-thread program counters.
• Transitions: enabled atomic actions.
• Invariants: properties that hold in all reachable states.

Tools like TLA+ explore this transition system exhaustively (for small enough state spaces).

For Go code, the manual reasoning might look like:

Invariant: at any time, either `ptr == nil` or `ptr->initialized == true`.

Proof:
1. The only place `ptr` is written is at line 42, which sets `ptr->initialized = true` before the store.
2. The store is atomic.Store, which is a release fence.
3. Any acquire on ptr observes the released value.
4. The release fence ensures `initialized = true` is visible before `ptr` is non-nil.
5. Therefore the invariant holds.

This level of rigor is rare in application code but standard for kernel/runtime code.

Appendix X: Production Concurrency Stories¶

X.1 — The runaway goroutine¶

A team noticed memory usage growing slowly over days. runtime.NumGoroutine() showed 100,000+ goroutines. Investigation: each request started a goroutine that waited on a channel, but the channel was never sent to.

Fix: use context for cancellation.

X.2 — The deadlock that wasn't¶

A team reported a deadlock under load. go tool trace showed a goroutine waiting on a mutex held by another goroutine. But the holder was also "waiting" — actually doing slow I/O.

Fix: don't hold mutexes during slow I/O. Use a pattern like:

mu.Lock()
key := pickKey()
mu.Unlock()
value := slowFetch(key)
mu.Lock()
cache[key] = value
mu.Unlock()

X.3 — The torn map¶

A team's map[string]int mysteriously had inconsistent values. Race detector confirmed: concurrent reads and writes.

Fix: sync.RWMutex around the map.

X.4 — The miscounted metric¶

A counter showed wrong totals because the reader did Load + Store(0) without atomic combination. Concurrent increments were lost.

Fix: Swap(0) for read-and-reset.

X.5 — The forgotten cancellation¶

A worker pool kept processing items even after shutdown. The shutdown signal closed a channel, but workers were busy with current items and didn't check.

Fix: check context at each iteration:

for {
    select {
    case <-ctx.Done():
        return
    case item := <-jobs:
        process(item)
    }
}

Appendix Y: The Future of Atomics in Go¶

Likely future additions:

• atomic.LoadAcquire / atomic.StoreRelease for explicit ordering.
• More wait-free types (e.g., atomic.Queue[T]).
• Better runtime support for NUMA.
• Generic atomic structs (currently you wrap with atomic.Pointer[T]).

Unlikely:

• Relaxed memory ordering (Go team prefers simplicity).
• 128-bit atomics (rarely needed).
• Hardware transactional memory exposed (HTM has been a research disappointment).

The Go team prioritizes pragmatic improvements over feature parity with C++ or Rust.

Appendix Z: A Closing Thought¶

Concurrency is fundamentally about time and visibility. Acquire/release semantics codify the rules of "what happens before what" across multiple cores.

Go gives you a simple, opinionated answer: seq-cst for atomics, well-defined synchronization for higher-level primitives, undefined behavior for races. This trades a small performance margin for huge gains in programmer productivity.

At the professional level, you understand all the layers: the language model, the runtime, the compiler, the hardware. You can reason about concurrent code with rigor that matches academic papers. You can design new primitives.

Whether you ever use this depth in your day job depends on your role. For a runtime contributor, every page; for a typical application developer, the senior level suffices.

But knowing the depth is what separates technical mastery from competence.

End of professional level — continued below.

Appendix AA: Implementing sync.RWMutex from Scratch¶

To understand sync.RWMutex, let's build one. The contract:

• RLock(): acquires a shared lock. Multiple readers may hold simultaneously.
• RUnlock(): releases a shared lock.
• Lock(): acquires an exclusive lock. Blocks until no readers and no other writer.
• Unlock(): releases an exclusive lock.

Invariants:

• At most one writer at a time.
• No readers while a writer is active.
• A waiting writer eventually acquires (writer preference).

type RWMutex struct {
    readers atomic.Int32
    writer  atomic.Bool
    writeMu sync.Mutex
    cond    *sync.Cond
    condMu  sync.Mutex
}

func (rw *RWMutex) RLock() {
    for {
        if rw.writer.Load() {
            rw.condMu.Lock()
            for rw.writer.Load() {
                rw.cond.Wait()
            }
            rw.condMu.Unlock()
            continue
        }
        rw.readers.Add(1)
        if rw.writer.Load() {
            rw.readers.Add(-1)
            continue
        }
        return
    }
}

func (rw *RWMutex) RUnlock() {
    if rw.readers.Add(-1) == 0 && rw.writer.Load() {
        rw.condMu.Lock()
        rw.cond.Signal()
        rw.condMu.Unlock()
    }
}

func (rw *RWMutex) Lock() {
    rw.writeMu.Lock()
    rw.writer.Store(true)
    rw.condMu.Lock()
    for rw.readers.Load() > 0 {
        rw.cond.Wait()
    }
    rw.condMu.Unlock()
}

func (rw *RWMutex) Unlock() {
    rw.writer.Store(false)
    rw.condMu.Lock()
    rw.cond.Broadcast()
    rw.condMu.Unlock()
    rw.writeMu.Unlock()
}

The standard library's sync.RWMutex is more efficient (uses semaphores), but this illustrates the protocol.

Publication: readers.Add and writer.Store are atomic. The condition variable adds wait/wake on top.

Appendix AB: Implementing a Channel from Atomics¶

A channel using mutex + condition variables (simplified):

type AtomicChan[T any] struct {
    buf    []T
    cap    uint64
    head   uint64
    tail   uint64
    mu     sync.Mutex
    notFull  sync.Cond
    notEmpty sync.Cond
    closed atomic.Bool
}

func (c *AtomicChan[T]) Send(v T) bool {
    c.mu.Lock()
    defer c.mu.Unlock()
    for c.head-c.tail >= c.cap && !c.closed.Load() {
        c.notFull.Wait()
    }
    if c.closed.Load() {
        return false
    }
    c.buf[c.head%c.cap] = v
    c.head++
    c.notEmpty.Signal()
    return true
}

func (c *AtomicChan[T]) Recv() (T, bool) {
    c.mu.Lock()
    defer c.mu.Unlock()
    for c.head == c.tail && !c.closed.Load() {
        c.notEmpty.Wait()
    }
    if c.head == c.tail {
        var zero T
        return zero, false
    }
    v := c.buf[c.tail%c.cap]
    c.tail++
    c.notFull.Signal()
    return v, true
}

func (c *AtomicChan[T]) Close() {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.closed.Store(true)
    c.notFull.Broadcast()
    c.notEmpty.Broadcast()
}

The standard library's chan is hand-tuned in the runtime — thousands of lines. This sketch is illustrative.

Appendix AC: Lock-Free vs Fine-Grained Locks¶

For most code, fine-grained locking is the right answer:

type StripedMutex struct {
    locks [256]sync.Mutex
}

func (s *StripedMutex) Lock(k string) {
    h := fnv32(k)
    s.locks[h%256].Lock()
}

func (s *StripedMutex) Unlock(k string) {
    h := fnv32(k)
    s.locks[h%256].Unlock()
}

Different keys map to different mutexes; contention is reduced by 256x in the best case.

Lock-free has its place (tight critical sections, real-time), but fine-grained covers the common case at a fraction of the complexity.

Appendix AD: Hardware Transactional Memory¶

Intel TSX, IBM POWER, others support HTM:

if (_xbegin() == _XBEGIN_STARTED) {
    x = 1; y = 2;
    _xend();
} else {
    // fallback
}

Transactions abort on conflict; you need a lock-based fallback. Aborts are common; debugging is hard.

Go doesn't expose HTM. Intel deprecated TSX in 2021 for security reasons. HTM may return; as of 2026, not mainstream.

Appendix AE: Persistent Memory and Memory Ordering¶

Persistent memory (Optane, NVDIMMs) survives power loss. New primitives:

• CLFLUSHOPT, CLWB: flush cache lines.
• SFENCE: store fence for persistence.

Programming model:

data = newValue;
_mm_clwb(&data);
_mm_sfence();
// data is now durable

Go doesn't directly support persistent memory. Use cgo + libpmem for niche needs. The Go memory model has no notion of durability.

Appendix AF: GPU Concurrency¶

GPUs have different memory models. CUDA's __threadfence() is a full barrier; __threadfence_block() is local to a block.

Go doesn't natively support GPU. Use cgo for CUDA/OpenCL. The CPU-GPU boundary has its own synchronization primitives.

Appendix AG: Distributed Consistency Models¶

Distributed systems have consistency models:

• Linearizability: atomic, totally ordered.
• Sequential consistency: all nodes agree on order.
• Causal consistency: causal pairs ordered.
• Eventual consistency: replicas eventually converge.

Acquire/release in single-process Go gives linearizability locally. Across processes, you need consensus algorithms (Paxos, Raft).

Appendix AH: Consensus Algorithms¶

In Go:

• hashicorp/raft
• etcd-io/raft

These run on top of Go's memory model and provide cross-node consistency.

Appendix AI: Traced Operations¶

AI.1 — m.Store(k, v) on sync.Map¶

1. Hash k.
2. Load "read" pointer (atomic).
3. If k in read, CAS its value entry.
4. Else acquire dirty mutex, add to dirty, release.

Publication: each atomic Load/Store and mutex provides acq/rel.

AI.2 — g.Wait() on errgroup.Group¶

1. Calls wg.Wait().
2. Each Done is a release.
3. Wait acquires when counter reaches 0.

AI.3 — ch <- v then <-ch¶

1. Sender locks channel mutex, copies into buffer, signals.
2. Receiver locks mutex, copies out, signals.

Publication: writes before send visible after receive.

Appendix AJ: When Does Go Reorder?¶

Common intra-goroutine optimizations:

• Loop-invariant code motion.
• Common subexpression elimination.
• Dead store elimination.
• Code straightening.

Atomic operations and sync primitives are compiler barriers. The compiler does not reorder across them.

Appendix AK: Production Notes¶

• Race detector overhead: ~5-10x. Use in tests, not prod.
• Mutex contention shows in pprof's lockSlow.
• Atomics scale well usually.
• Channels bounded ~500K ops/sec. Higher? Shard.
• GC doesn't interfere with atomics.
• NUMA rarely matters on cloud VMs.

Appendix AL: Decision Flowchart¶

Shared between goroutines? - NO → local, no sync.
                              YES → continue.

Immutable after publish? - YES → atomic.Pointer (read-heavy) or sync.Once (init).
                          NO → continue.

Multi-step transaction? - YES → sync.Mutex.
                         NO → continue.

Single-word? - YES → sync/atomic.
              NO → sync.Mutex.

Need to signal/pass value? - YES → channel.

Appendix AM: Acceptance Criteria¶

You're professional-level if you can:

1. Write DCL without docs.
2. Implement Treiber stack in <30 lines.
3. Explain memory model in 5 minutes.
4. Trace atomic.Pointer.Store to assembly.
5. Fix a publication bug from race report.
6. Choose primitive by workload analysis.
7. Profile and optimize atomic paths.
8. Design new concurrent primitive.
9. Understand fence elision.
10. Translate concurrency between languages.

Appendix AN: References¶

• Go source: src/sync/, src/sync/atomic/, src/runtime/.
• Linux kernel RCU docs.
• C++ standard memory model.
• Rust nomicon.
• Postgres source.
• CockroachDB (Go).
• TiKV (Rust).

Appendix AO: Wrap-Up¶

You've reached the end of the professional file. You should understand:

• The full stack: source → IR → assembly → machine code.
• Cross-language concurrency.
• Cost models per architecture.
• Runtime's role in barriers.
• Reclamation beyond GC.
• Cache-coherence and NUMA.

From here, deeper mastery comes from building systems that exercise these patterns.

Go forth.

End of the AO section.

Appendix AP: Deep Dive — sync.Mutex Internals¶

Go's sync.Mutex is sophisticated. Let's trace through its implementation.

type Mutex struct {
    state int32
    sema  uint32
}

const (
    mutexLocked = 1 << iota
    mutexWoken
    mutexStarving
    mutexWaiterShift = iota
    starvationThresholdNs = 1e6
)

The state field packs: - Bit 0: locked. - Bit 1: woken (a waiter was just woken). - Bit 2: starvation mode. - Bits 3+: waiter count.

Fast path: uncontended Lock¶

func (m *Mutex) Lock() {
    if atomic.CompareAndSwapInt32(&m.state, 0, mutexLocked) {
        return
    }
    m.lockSlow()
}

One CAS. Most lock acquisitions take this path.

Slow path¶

Spins briefly. If still locked, enqueues on semaphore. After 1 ms wait, enters starvation mode — next holder hands lock directly to longest-waiting goroutine, preventing starvation.

Unlock¶

func (m *Mutex) Unlock() {
    new := atomic.AddInt32(&m.state, -mutexLocked)
    if new != 0 {
        m.unlockSlow(new)
    }
}

Publication: every Lock's CAS is acquire; every Unlock's atomic subtract is release.

Appendix AQ: Deep Dive — sync.Once¶

type Once struct {
    done atomic.Uint32
    m    Mutex
}

func (o *Once) Do(f func()) {
    if o.done.Load() == 0 {
        o.doSlow(f)
    }
}

func (o *Once) doSlow(f func()) {
    o.m.Lock()
    defer o.m.Unlock()
    if o.done.Load() == 0 {
        defer o.done.Store(1)
        f()
    }
}

This is the canonical DCL. Fast path: one atomic load.

Subtleties: - The load is atomic to avoid race. - Inner check inside mutex avoids running f twice. - defer o.done.Store(1) ensures even panic marks done.

Publication: done.Store(1) is a release. Subsequent done.Load() == 1 is an acquire.

Appendix AR: Deep Dive — sync.WaitGroup¶

type WaitGroup struct {
    state1 atomic.Uint64 // high 32: counter, low 32: waiter count
    state2 atomic.Uint32 // semaphore
}

Packs counter and waiter count into one atomic to avoid race when counter hits 0 just as a Wait increments waiters.

Each Done is acq-rel. When counter hits 0, runtime signals all waiters. Writes before Done visible after Wait returns.

Appendix AS: Deep Dive — sync.Map¶

Two underlying maps: - read: atomic-Pointer to read-only snapshot. - dirty: mutex-protected.

Reads check read first (lock-free). Misses fall through to dirty under mutex. When misses exceed read size, dirty is promoted to read.

Read src/sync/map.go — ~400 lines of careful design.

Appendix AT: Deep Dive — chan¶

type hchan struct {
    qcount   uint
    dataqsiz uint
    buf      unsafe.Pointer
    elemsize uint16
    closed   uint32
    elemtype *_type
    sendx    uint
    recvx    uint
    recvq    waitq
    sendq    waitq
    lock     mutex
}

Send: 1. Lock chan. 2. If receiver waiting → direct hand-off, wake. 3. Else if buffer has space → store, advance. 4. Else → park sender, release lock.

Receive: symmetric.

Close: lock, set flag, wake all waiters.

Publication: the lock provides acq/rel for buffered transfers; direct hand-offs synchronize both sides under the lock.

Appendix AU: Cost Comparison¶

Microbenchmarks (rough, uncontended):

plain memory access              ~0.3 ns
atomic.Load(uint32)              ~1 ns
atomic.Store(uint32)             ~5 ns
atomic.CompareAndSwap            ~8 ns
sync.Mutex.Lock+Unlock           ~15 ns
sync.RWMutex.RLock+RUnlock       ~20 ns
chan send/recv buffered          ~80 ns
chan send/recv unbuffered        ~200 ns
sync.Once.Do cached              ~1 ns

Contention can amplify any of these 10-100x.

Appendix AV: When Performance Matters¶

• HFT.
• Real-time A/V.
• Hot path of network proxy.
• Storage engine.
• Game server tick.

For these, profile relentlessly. Choose the lightest primitive.

For typical web services, readability wins.

Appendix AW: Proper Spinlocks¶

type SpinLock struct {
    flag atomic.Bool
}

func (s *SpinLock) Lock() {
    for {
        if s.flag.CompareAndSwap(false, true) {
            return
        }
        for s.flag.Load() {
            runtime.Gosched()
        }
    }
}

func (s *SpinLock) Unlock() {
    s.flag.Store(false)
}

First CAS; if fails, wait via relaxed load (doesn't ping cache). Use only for <100 ns critical sections.

Appendix AX: NUMA Sharding¶

type NUMAAware struct {
    perSocket [4]*Cache
}

func (n *NUMAAware) Get(k string) (Value, bool) {
    socket := currentSocket()
    return n.perSocket[socket].Get(k)
}

Requires runtime.LockOSThread or taskset to pin. Rarely needed on cloud VMs.

Appendix AY: Lock-Free Refcounting¶

type Refcounted struct {
    refs atomic.Int32
    val  any
}

func (r *Refcounted) Inc() { r.refs.Add(1) }
func (r *Refcounted) Dec() {
    if r.refs.Add(-1) == 0 {
        // free
    }
}

Only Inc from a context already holding a reference. Rarely needed in Go.

Appendix AZ: Polling Channels¶

select {
case v := <-ch:
    // got
default:
    // would block
}

Useful for opportunistic check without blocking.

Appendix BA: Libraries to Study¶

• golang.org/x/sync/errgroup
• golang.org/x/sync/singleflight
• golang.org/x/sync/semaphore
• github.com/hashicorp/raft
• github.com/etcd-io/etcd
• github.com/cockroachdb/cockroach

Each is a master class. Read patterns; emulate.

Appendix BB: Concurrency Bug Database¶

Search for "data race," "deadlock," "memory ordering" in:

• github.com/golang/go issues.
• Kubernetes issues.
• etcd issues.
• CockroachDB issues.

Postmortems are gold.

Appendix BC: Diagrams¶

GO'S MEMORY MODEL HIERARCHY
===========================

      Application Code
            |
      sync, sync/atomic, channels
            |
      Compiler (gc / gccgo)
            |
      runtime/internal/atomic
            |
      Per-arch assembly (XCHG, LDAR, etc.)
            |
      Hardware memory model (TSO, ARMv8, etc.)

PUBLICATION STAGES
==================

Producer:                       Consumer:
[build *T]                          
    |                              
[release]  ──synchronizes-with──► [acquire]
                                      |
                                  [use *T]  ← writes visible

Appendix BD: Final Wrap¶

You've read the professional file. From here, mastery comes from:

• Implementing systems.
• Reading source.
• Reading papers.
• Discussing with peers.
• Teaching.

Concurrency is hard. Acquire/release is the foundation. Build on it.

End of professional file, continued below.

Appendix BE: Inside the Go Scheduler¶

The Go scheduler (GMP — Goroutines, Machine threads, Processors) is intertwined with memory ordering. When a goroutine migrates between Ms (OS threads), the runtime emits barriers.

You don't write code for this directly. But knowing it explains why your atomic operations always observe consistent state, even with goroutine migration.

Appendix BF: Channel Lock Elision¶

For single-sender/single-receiver channels with no contention, Go's runtime can sometimes skip the internal mutex. Pure optimization; semantics identical.

Appendix BG: GC and Atomics¶

Concurrent GC has brief stop-the-world pauses (<1 ms). During STW, all goroutines pause at safe points. Atomic operations naturally compose with GC; correctness is preserved.

Go's GC is non-moving (mostly), so atomic pointers don't change addresses during GC. This is friendlier than moving GCs for lock-free pointer structures.

Appendix BH: Goroutine-Local Storage¶

Go doesn't expose TLS. Alternative: sync.Map keyed by goroutine ID. The community generally avoids this pattern; prefer passing context explicitly.

Appendix BI: A Highly-Concurrent Counter¶

For millions of increments per second:

type ConcurrentCounter struct {
    cells [256]paddedInt64
}

type paddedInt64 struct {
    val atomic.Int64
    _   [56]byte
}

func (c *ConcurrentCounter) Add(delta int64) {
    idx := getProcID() % 256
    c.cells[idx].val.Add(delta)
}

func (c *ConcurrentCounter) Sum() int64 {
    var s int64
    for i := range c.cells {
        s += c.cells[i].val.Load()
    }
    return s
}

Each P writes to its own cell; Sum reads all. 16 KB memory; wait-free Add; O(256) Sum.

Appendix BJ: Sharded RWMutex Map¶

type ShardedMap[K comparable, V any] struct {
    shards [256]struct {
        mu sync.RWMutex
        m  map[K]V
    }
}

Each shard has its own RWMutex. Contention distributed across 256 buckets.

Appendix BK: Common Bottlenecks¶

1. Global mutex on hot map → shard.
2. Single semaphore → per-shard.
3. GC pressure → sync.Pool.
4. Single channel → multiple channels.
5. CAS retry loops under contention → mutex (often faster).
6. False sharing → padding.

Profile first.

Appendix BL: Reference MPSC Queue¶

Vyukov's MPSC (simplified):

type MPSC[T any] struct {
    head atomic.Pointer[mpscNode[T]]
    tail *mpscNode[T]
    stub mpscNode[T]
}

type mpscNode[T any] struct {
    next atomic.Pointer[mpscNode[T]]
    val  T
}

func (q *MPSC[T]) Push(v T) {
    n := &mpscNode[T]{val: v}
    prev := q.head.Swap(n)
    prev.next.Store(n)
}

func (q *MPSC[T]) Pop() (T, bool) {
    tail := q.tail
    next := tail.next.Load()
    if tail == &q.stub {
        if next == nil { var zero T; return zero, false }
        q.tail = next
        tail = next
        next = next.next.Load()
    }
    if next != nil {
        q.tail = next
        return next.val, true
    }
    head := q.head.Load()
    if tail != head { return q.Pop() }
    var zero T
    return zero, false
}

Wait-free producers, lock-free consumer.

Appendix BM: Approximate Counter¶

type ApproxCounter struct {
    val  atomic.Int64
    perP []atomic.Int64
}

func (c *ApproxCounter) Inc() {
    p := getProcID()
    if c.perP[p].Add(1) > 100 {
        delta := c.perP[p].Swap(0)
        c.val.Add(delta)
    }
}

func (c *ApproxCounter) Value() int64 {
    s := c.val.Load()
    for i := range c.perP {
        s += c.perP[i].Load()
    }
    return s
}

Reduces global pressure 100x; read is slightly stale.

Appendix BN: Lock-Free Lists¶

Harris's algorithm uses "logically deleted" marks on next pointers. 200+ lines for full implementation. Rarely needed in Go.

Appendix BO: Counter Pattern Matrix¶