Copy-on-Write — Middle Level¶

Table of Contents¶

1. Introduction
2. When COW Is the Right Tool
3. The Snapshot Discipline
4. Writer Coordination Strategies
5. COW Maps in Practice
6. COW Slices in Practice
7. Snapshot Consistency
8. Write Amplification and Batching
9. Comparing COW to RWMutex
10. Comparing COW to sync.Map
11. Sharded COW
12. Snapshot Versioning
13. Watchers and Notifications
14. Hot-Reload Patterns
15. Error Handling and Validation
16. Observability
17. Testing COW Code
18. Benchmark-Driven Decisions
19. Common Pitfalls at Scale
20. Best Practices
21. Self-Assessment
22. Summary

Introduction¶

Focus: "I understand the basics. Now: how do I design a system around COW? When is it wrong? How do I avoid the write-amplification trap? How does this interact with the rest of my architecture?"

At the middle level, COW becomes a system-design question, not a primitive-usage question. You already know how to load and store a snapshot, how to deep-copy a slice, and how to serialize writers. The next questions are:

• How do I choose between COW, sync.RWMutex, sync.Map, and a sharded-mutex map?
• How big can my snapshot get before write cost becomes a problem?
• How do I handle a flood of writes without overwhelming the GC?
• How do I expose snapshots to many goroutines without accidentally pinning old ones?
• How do I let other parts of the system react to snapshot changes (watchers)?
• How do I keep the snapshot consistent across a multi-step write?
• How do I measure all of this so I know my design is right?

The junior level taught the mechanics. The middle level teaches the engineering judgement: shape the design to the workload, measure relentlessly, and have a fallback plan when the workload changes.

When COW Is the Right Tool¶

The decision tree for "which concurrent data structure" should be applied before any code is written. COW sits at one corner of the design space.

The decision tree¶

                Reads >> Writes (>1000:1)?
                       |
              ┌────────┴────────┐
             yes                 no
              |                   |
   Snapshot small enough?    Map-shaped?
   (<10 MB; <1 GB upper)         |
              |             ┌────┴────┐
       ┌──────┴──────┐     yes        no
      yes            no    |           |
       |             |   sync.Map   sync.Mutex /
   Multi-field   sync.RWMutex      sync.RWMutex
   consistency             or sharded-mutex map
   needed?
       |
   ┌───┴───┐
  yes      no
   |       |
  COW   atomic primitives
        (Int64, Bool, etc.)

The four canonical workloads¶

The "magic ratio" rule of thumb: if writes are more than 1% of reads, reconsider COW. Below that threshold, COW typically wins on read latency, and the write cost is amortized over many reads.

Sizes matter as much as ratios¶

A 1 GB snapshot with one write per hour is theoretically COW-compatible — the per-write rebuild costs ~10 seconds and a 1 GB allocation, but it happens rarely. In practice, the allocation pressure spikes the GC, the rebuild blocks the writer for 10 seconds, and the operator who triggered it wonders why their dashboard hung.

Past 100 MB, switch to:

• Persistent data structures (HAMTs, finger trees, RRB trees) that share structure between versions.
• Incremental snapshots that combine a small "delta layer" with a large "base layer".
• Sharded COW where each shard is independently COW.

These appear in senior.md. At middle level, the rule of thumb is: if your snapshot is over a few MB and you write more than once a minute, plan to switch strategies.

Latency variance, not just throughput¶

COW's killer feature is predictable read latency. An RWMutex can give you a 100 ms tail latency if a writer holds the lock for 100 ms. COW gives you a 2 ns reader cost, always.

For SLO-sensitive services — p99.9 latency budgets in the low milliseconds — the absence of writer-induced tail latency is reason enough to choose COW even when throughput is similar.

The Snapshot Discipline¶

The junior level introduced "don't mutate published snapshots". The middle level treats that discipline as a type design problem, not a runtime hope.

Designing for deep immutability¶

A snapshot type should look immutable. Aim for these patterns:

Pattern 1: All fields are value types¶

type Snapshot struct {
    Version int64
    Origin  string
    Limit   int
    // no slices, no maps, no pointers
}

If the snapshot is small and field-only, deep immutability is automatic. The shallow copy is the deep copy.

Pattern 2: Slices and maps are immutable by convention¶

type Snapshot struct {
    Hosts []string                  // never mutated after publish
    Lookup map[string]*HostInfo     // map never mutated; *HostInfo also immutable
}

This is the common case. The type system cannot enforce immutability, so:

• Document the contract in a doc comment.
• Provide accessor methods that return copies or func accessors.
• Always rebuild from scratch in writers — never append in place, never assign to a map key.

Pattern 3: Wrap mutable inner types in accessors¶

type Snapshot struct {
    hosts []string // unexported
}

func (s *Snapshot) Hosts() []string {
    out := make([]string, len(s.hosts))
    copy(out, s.hosts)
    return out
}

func (s *Snapshot) HostAt(i int) string { return s.hosts[i] }

func (s *Snapshot) ForEachHost(fn func(string)) {
    for _, h := range s.hosts {
        fn(h)
    }
}

Accessors enforce immutability at the API boundary. The slice is never directly handed out.

Pattern 4: Use immutable sub-types¶

Use library types like string (immutable in Go) or roll your own:

type immutableSet struct {
    m map[string]struct{} // accessed only via Contains
}

func (s *immutableSet) Contains(k string) bool {
    _, ok := s.m[k]
    return ok
}

The internal m is never exposed.

Reasoning about deep copies¶

Every slice, map, or pointer inside the snapshot triggers a question: "if a writer modifies this field, does it need its own copy?"

The sync.Mutex row is a trap. go vet catches some cases (copylocks analyzer), but if you put a sync.Mutex inside a snapshot type, every snapshot you copy carries a fresh mutex — which is almost never what you want.

Type-system tricks¶

Some teams use a private "snapshot key" to make the snapshot type unforgeable:

type snapshotKey struct{}

type Snapshot struct {
    _      snapshotKey // can only be constructed by this package
    fields ...
}

func NewSnapshot(...) Snapshot { return Snapshot{...} }

Now external code cannot synthesize a Snapshot; they can only get one from your constructors. Combined with unexported fields, this approaches immutability-by-construction.

Returning a clone instead of a snapshot¶

For the highest safety, return a clone every time:

func (s *Store) Get() *Config {
    return s.cur.Load().Clone()
}

The cost is a full deep copy on every read. For a 10-field config with no slices, this is ~50 ns — still cheaper than an RWMutex. For larger snapshots, this destroys the whole point of COW.

Use this only when the caller is untrusted (e.g., third-party plugin) and the snapshot is small.

Writer Coordination Strategies¶

The junior level showed one writer pattern: a sync.Mutex around the load-build-store dance. At middle level, choose between three patterns based on writer characteristics.

Strategy 1: Plain writer mutex¶

mu.Lock()
defer mu.Unlock()
old := cur.Load()
next := build(old)
cur.Store(next)

Use when: writes are infrequent and short. Cleanest, easiest to reason about.

Avoid when: writes are bursty or any writer may hold the lock for a long time.

Strategy 2: CAS loop¶

for {
    old := cur.Load()
    next := build(old)
    if cur.CompareAndSwap(old, next) {
        return
    }
    // another writer interleaved; retry
}

Use when: writers must not block each other, and the build step is short (so retries are cheap).

Avoid when: build is expensive — every retry repeats it.

Caveat: under heavy contention, CAS can starve. Use a backoff:

for attempt := 0; ; attempt++ {
    old := cur.Load()
    next := build(old)
    if cur.CompareAndSwap(old, next) {
        return
    }
    if attempt > 100 {
        time.Sleep(time.Duration(attempt) * time.Microsecond)
    }
}

Strategy 3: Single-writer goroutine¶

Funnel all writes through a single goroutine fed by a channel:

type updateReq struct {
    apply func(*Config)
    done  chan struct{}
}

var updateCh = make(chan updateReq, 64)

func writerLoop() {
    for req := range updateCh {
        old := cur.Load()
        next := *old
        req.apply(&next)
        cur.Store(&next)
        close(req.done)
    }
}

func Update(fn func(*Config)) {
    done := make(chan struct{})
    updateCh <- updateReq{apply: fn, done: done}
    <-done
}

Use when: you want backpressure on writers (the channel buffer), centralized logging of writes, or batched writes.

Avoid when: simplicity matters more than control.

Strategy 4: Batched writer¶

Buffer pending writes and apply them in batches:

type batcher struct {
    pending []func(*Config)
    mu      sync.Mutex
    timer   *time.Timer
}

func (b *batcher) Enqueue(fn func(*Config)) {
    b.mu.Lock()
    defer b.mu.Unlock()
    b.pending = append(b.pending, fn)
    if b.timer == nil {
        b.timer = time.AfterFunc(10*time.Millisecond, b.flush)
    }
}

func (b *batcher) flush() {
    b.mu.Lock()
    work := b.pending
    b.pending = nil
    b.timer = nil
    b.mu.Unlock()

    // One rebuild for all queued updates.
    writeMu.Lock()
    defer writeMu.Unlock()
    old := cur.Load()
    next := *old
    for _, fn := range work {
        fn(&next)
    }
    cur.Store(&next)
}

Use when: many small writes arrive in bursts and a single combined rebuild is much cheaper than N separate rebuilds.

Cost: writers no longer see their changes reflected synchronously. Use callbacks or polling if they need to confirm.

Choosing among the four¶

The mutex is the right default. Move to CAS only if you have measured the mutex as a bottleneck. Move to single-writer when you want centralized control. Move to batched when GC pressure or rebuild cost is the bottleneck.

COW Maps in Practice¶

A COW map is a map exposed through an atomic pointer. The pattern is well-known but has subtleties at production scale.

The textbook COW map¶

type COWMap[K comparable, V any] struct {
    cur atomic.Pointer[map[K]V]
    mu  sync.Mutex
}

func NewCOWMap[K comparable, V any]() *COWMap[K, V] {
    m := make(map[K]V)
    out := &COWMap[K, V]{}
    out.cur.Store(&m)
    return out
}

func (m *COWMap[K, V]) Get(k K) (V, bool) {
    v, ok := (*m.cur.Load())[k]
    return v, ok
}

func (m *COWMap[K, V]) Set(k K, v V) {
    m.mu.Lock()
    defer m.mu.Unlock()
    old := *m.cur.Load()
    next := make(map[K]V, len(old)+1)
    for kk, vv := range old {
        next[kk] = vv
    }
    next[k] = v
    m.cur.Store(&next)
}

func (m *COWMap[K, V]) Delete(k K) {
    m.mu.Lock()
    defer m.mu.Unlock()
    old := *m.cur.Load()
    if _, ok := old[k]; !ok {
        return
    }
    next := make(map[K]V, len(old)-1)
    for kk, vv := range old {
        if kk != k {
            next[kk] = vv
        }
    }
    m.cur.Store(&next)
}

The cost model¶

For a map of size N:

• Get: O(1) — one atomic load, one map lookup.
• Set: O(N) — full rebuild of the map.
• Delete: O(N) — full rebuild.
• Memory: at any moment, holds one current map; old maps stick around until GC.

A 10 000-entry map with string keys and int64 values is ~600 KB. Rebuilding it takes ~500 µs and produces 600 KB of garbage. At 1 write per second, this is fine. At 100 writes per second, you are pushing 60 MB/sec to the GC.

When to walk away from COWMap¶

The break-even point depends on workload, but a rough rule:

• Map size < 1 000 entries: COW is fine up to ~1 000 writes/sec.
• Map size 1 000 – 100 000: COW is fine up to ~10 writes/sec.
• Map size > 100 000: COW writes are painful; consider sharded COW or sync.Map.

COWMap with pre-sized capacity¶

If you know your map's typical size, pre-size to avoid rehash:

next := make(map[K]V, len(old)*2) // double the size to leave headroom

The *2 hedges against future inserts that would otherwise trigger a rehash. Cost: more memory per snapshot.

COWMap with structural sharing? No.¶

A naive COWMap rebuilds the whole map per write. The fancy alternative is a HAMT (hash array-mapped trie) which shares structure between versions. Go does not have HAMTs in the standard library; community libraries (e.g., github.com/benbjohnson/immutable) implement them. See senior.md for when this is worth the dependency.

Range over a COWMap¶

func (m *COWMap[K, V]) Range(fn func(k K, v V) bool) {
    for k, v := range *m.cur.Load() {
        if !fn(k, v) {
            return
        }
    }
}

Critical: for range evaluates m.cur.Load() once. The iteration sees a consistent snapshot. A concurrent writer producing a new snapshot does not affect the in-progress iteration.

A common bug: forgetting that Load returns a pointer¶

func (m *COWMap[K, V]) Set(k K, v V) {
    m.mu.Lock()
    defer m.mu.Unlock()
    old := m.cur.Load() // *map[K]V, NOT map[K]V
    (*old)[k] = v       // BUG: mutates the published snapshot!
    // ...
}

The fix: dereference and copy explicitly. This bug is caught by the race detector but only after the first concurrent reader trips on it.

COW Slices in Practice¶

Slices are simpler than maps but bring their own surprises.

Append-only slice¶

type Log[T any] struct {
    cur atomic.Pointer[[]T]
    mu  sync.Mutex
}

func (l *Log[T]) Append(v T) {
    l.mu.Lock()
    defer l.mu.Unlock()
    old := *l.cur.Load()
    next := make([]T, len(old)+1)
    copy(next, old)
    next[len(old)] = v
    l.cur.Store(&next)
}

O(n) per append; you cannot escape this without a chunked representation.

Slice as snapshot vs slice as buffer¶

If your slice represents a rolling buffer ("last 100 events"), COW is wasteful — every append rebuilds the entire buffer. Use a ring buffer or a sync.Mutex-protected slice. COW is best for slices that are built incrementally and then frozen.

A frozen builder pattern¶

type Builder struct {
    pending []Event
}

func (b *Builder) Add(e Event) { b.pending = append(b.pending, e) }

func (b *Builder) Freeze() *Snapshot {
    return &Snapshot{events: b.pending}
}

The writer accumulates in a private builder, then publishes one snapshot. No intermediate publishes.

Slice of pointers¶

type Snapshot struct {
    events []*Event
}

Cheap to copy the slice (header + pointer-sized elements). But: if any reader mutates *event, the data is shared. Make Event immutable too.

Pre-allocation¶

For known-size snapshots:

next := make([]T, 0, len(old)+1)
next = append(next, old...)
next = append(next, v)

Two appends with pre-allocated capacity avoid intermediate allocations.

Slice slicing pitfall¶

old := *cur.Load()
next := old[:len(old)-1] // BUG: shares backing array
cur.Store(&next)

Slicing creates a new header but the same backing array. A reader iterating old sees the truncated portion or, worse, witnesses an in-flight write if you later append to next. Always copy:

next := make([]T, len(old)-1)
copy(next, old)

Snapshot Consistency¶

The whole point of COW is that one snapshot is consistent for the whole reader. This is more powerful than it sounds.

Multi-field reads are atomic by construction¶

c := cfg.Load()
if c.LogLevel == "debug" && c.MaxConnections > 100 { ... }

Both fields come from the same snapshot. No torn read possible.

Multi-snapshot reads are not atomic¶

cfg := cfg.Load()
flags := flags.Load() // ANOTHER store
if cfg.LogLevel == "debug" && flags.Enabled("foo") { ... }

A writer could update flags between the two loads. If you need consistency across both, merge them into one snapshot or accept that they may be momentarily inconsistent.

Pinning a snapshot for the duration of a request¶

The cleanest pattern: load once per request and pass the pointer down.

func handle(w http.ResponseWriter, r *http.Request) {
    cfg := cfg.Load()
    ctx := context.WithValue(r.Context(), cfgKey{}, cfg)
    process(r.WithContext(ctx))
}

Every downstream call uses the same snapshot. Even if a Store happens mid-request, the request finishes with its original snapshot.

Snapshot freshness vs consistency¶

There is a tension:

• Freshness means "give me the latest value." Requires Load at point of use.
• Consistency means "give me a stable, internally-consistent view." Requires Load once at top.

COW excels at consistency but does not give you sub-snapshot freshness. If you need both — "the latest LogLevel and the latest Hosts, even if they were published separately" — you need either:

• A single atomic pointer for the combined config.
• A locking mechanism around both fields.
• Acceptance that you have racy/inconsistent reads.

Cross-store transactions¶

You cannot atomically update two atomic.Pointer[T]s. The closest you can do:

mu.Lock()
defer mu.Unlock()
a.Store(newA)
b.Store(newB)

Two writers serialise via mu. But a reader that does a.Load() then b.Load() may catch the moment between the two stores. If this matters, group into one snapshot.

Consistency vs scale¶

A single huge snapshot is consistent but expensive to rebuild. A collection of small snapshots is cheap to rebuild but loses cross-snapshot consistency. The middle-level engineer picks the right granularity:

• A web server's config: one snapshot.
• A microservice's downstream-service list: one snapshot per downstream service (each can be reloaded independently).
• A user profile cache: shard by user ID.

Write Amplification and Batching¶

COW's main weakness is write amplification: a single field change requires rebuilding the entire snapshot.

Measuring amplification¶

For a 100 KB snapshot containing 1000 entries, each Set allocates 100 KB. If you Set 100 keys in a row, you allocate 10 MB.

for k, v := range pending {
    cowmap.Set(k, v) // one rebuild per Set
}

Batched updates¶

Group writes:

func (m *COWMap[K, V]) SetMany(pairs map[K]V) {
    m.mu.Lock()
    defer m.mu.Unlock()
    old := *m.cur.Load()
    next := make(map[K]V, len(old)+len(pairs))
    for k, v := range old {
        next[k] = v
    }
    for k, v := range pairs {
        next[k] = v
    }
    m.cur.Store(&next)
}

One rebuild for the whole batch. The amortized cost per Set drops dramatically.

Time-window batching¶

If writes arrive continuously, accumulate and flush:

var (
    pendingMu sync.Mutex
    pending   = make(map[string]string)
    flushTimer *time.Timer
)

func ScheduleSet(k, v string) {
    pendingMu.Lock()
    defer pendingMu.Unlock()
    pending[k] = v
    if flushTimer == nil {
        flushTimer = time.AfterFunc(10*time.Millisecond, flush)
    }
}

func flush() {
    pendingMu.Lock()
    batch := pending
    pending = make(map[string]string)
    flushTimer = nil
    pendingMu.Unlock()

    cowmap.SetMany(batch)
}

The cost: writes are delayed by up to 10 ms. The win: one snapshot rebuild per 10 ms instead of N rebuilds per N writes.

Adaptive batching¶

If load is high, batch larger windows:

windowSize := 10 * time.Millisecond
if writeRate > 1000 {
    windowSize = 100 * time.Millisecond
}

Adaptive heuristics keep amortized cost roughly constant across load levels.

Coalescing writes¶

If two writes target the same key, only the latest needs to land:

pending[k] = v // overwrites previous value for k

Coalescing is automatic with a map-based pending buffer. With a slice-based one (e.g., event log), use a dedup pass before flush.

Knowing when batching is too much¶

Batching breaks the "writes are visible synchronously" property. If a writer needs to know its update has landed (e.g., to acknowledge a client), it must wait for the flush. Provide callbacks or futures:

type pendingWrite struct {
    k, v string
    done chan struct{}
}

Writers await <-done after submitting. Latency is now bounded by the batch window.

Comparing COW to RWMutex¶

When RWMutex is right¶

• The structure is large and writes are merely "occasional" rather than "rare".
• You need to mutate fields in place — perhaps because copying is impractical (e.g., a structure that contains a sync.Mutex or open *os.File).
• The read path needs to block writers for some operation (e.g., taking a consistent snapshot of a live system).

Why RWMutex is sometimes wrong for read-mostly¶

• RLock always atomically increments a reader counter and decrements on RUnlock. Even uncontended, this is ~10–30 ns.
• Under contention with a writer, RLock may block until the writer is done (Go's RWMutex is writer-preferring as of 1.18+).
• The reader counter is a single cache line; under heavy read load, it ping-pongs across CPUs.

A COW read is one atomic load (~1.5 ns), and the cache line holding the pointer is read-shared across all CPUs (no ping-pong). For 1 M reads/sec across 16 cores, COW is roughly 10× faster than RWMutex.

Benchmark sketch¶

func BenchmarkRWMutexRead(b *testing.B) {
    var mu sync.RWMutex
    cfg := &Config{}
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            mu.RLock()
            _ = cfg.LogLevel
            mu.RUnlock()
        }
    })
}

func BenchmarkCOWRead(b *testing.B) {
    var p atomic.Pointer[Config]
    p.Store(&Config{})
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            _ = p.Load().LogLevel
        }
    })
}

Typical results on a recent Intel laptop:

BenchmarkRWMutexRead-8     30000000          45 ns/op
BenchmarkCOWRead-8        500000000           2.1 ns/op

20× difference. On NUMA hardware the gap is larger.

When RWMutex wins¶

A workload with 50% writes:

BenchmarkRWMutexMixed-8    10000000         140 ns/op
BenchmarkCOWMixed-8         2000000         800 ns/op  (snapshot rebuild dominates)

For balanced workloads, RWMutex (or a plain Mutex) is clearly faster.

Comparing COW to sync.Map¶

sync.Map is the standard library's lock-free-on-the-read-path map. It uses a different read-mostly strategy:

• A read-only atomic.Value "read map" — fast path for reads.
• A sync.Mutex-protected "dirty map" — slow path for writes.
• A promotion mechanism that moves dirty entries into the read map when the dirty grows large.

When sync.Map is right¶

• Map-shaped data with mixed read/write.
• Keys are accessed independently (no need for multi-key snapshot consistency).
• The set of keys is mostly stable; new keys are rare.

The Go documentation calls out two use cases as ideal:

1. When the entry for a given key is only ever written once but read many times, as in caches that only grow.
2. When multiple goroutines read, write, and overwrite entries for disjoint sets of keys.

When sync.Map is wrong¶

• You need a consistent multi-key snapshot. sync.Map.Range does not give one — its docs say "Range does not necessarily correspond to any consistent snapshot of the Map's contents."
• You need fast Len(). sync.Map has no Len; you must Range and count.
• Keys are frequently created and deleted across goroutines — the read map promotion is too slow to keep up.

Benchmark sketch¶

For a hot read on a map containing 1 000 entries:

BenchmarkSyncMapRead-8     50000000          22 ns/op
BenchmarkCOWMapRead-8     300000000           3.5 ns/op  (atomic load + map lookup)

COW wins for pure read. sync.Map wins for mixed read/write when writes touch disjoint keys.

The hybrid pattern¶

A small subset of cases benefit from putting a sync.Map inside a COW snapshot:

• The snapshot dictates which sync.Map is current.
• Operators can wholesale-swap to a fresh sync.Map (e.g., for cache clearing).
• Within a generation, reads and writes are sync.Map-fast.

type Cache struct {
    m *sync.Map
}

var c atomic.Pointer[Cache]

func init() { c.Store(&Cache{m: &sync.Map{}}) }

func Get(k string) (string, bool) {
    v, ok := c.Load().m.Load(k)
    if !ok { return "", false }
    return v.(string), true
}

func ClearAll() {
    c.Store(&Cache{m: &sync.Map{}})
}

This composes the two patterns cleanly.

Sharded COW¶

For a large map with frequent writes, shard:

const NShards = 32

type ShardedCOW[K comparable, V any] struct {
    shards [NShards]struct {
        cur atomic.Pointer[map[K]V]
        mu  sync.Mutex
    }
    hash func(K) uint64
}

func (s *ShardedCOW[K, V]) Get(k K) (V, bool) {
    sh := &s.shards[s.hash(k)%NShards]
    v, ok := (*sh.cur.Load())[k]
    return v, ok
}

func (s *ShardedCOW[K, V]) Set(k K, v V) {
    sh := &s.shards[s.hash(k)%NShards]
    sh.mu.Lock()
    defer sh.mu.Unlock()
    old := *sh.cur.Load()
    next := make(map[K]V, len(old)+1)
    for kk, vv := range old {
        next[kk] = vv
    }
    next[k] = v
    sh.cur.Store(&next)
}

Benefits¶

• Each shard's rebuild touches only N/Shards entries.
• Writer contention is reduced 32× (writers to different shards don't block each other).
• GC pressure spread across smaller allocations.

Costs¶

• No cross-shard atomicity. Reading all shards in Range is not a consistent snapshot.
• More complexity. Avoid for small maps.

Choosing the shard count¶

• Powers of 2 make hash(k) & (N-1) a fast bit-mask.
• 16, 32, 64 are common.
• More shards = more memory overhead (per-shard mutex + atomic pointer).
• Diminishing returns past 4× the number of CPU cores.

Snapshot Versioning¶

Tagging each snapshot with a monotonic version unlocks several patterns.

The version field¶

type Snapshot struct {
    Version int64
    Data    map[string]string
}

func (s *Store) Set(k, v string) {
    s.mu.Lock()
    defer s.mu.Unlock()
    old := s.cur.Load()
    next := &Snapshot{
        Version: old.Version + 1,
        Data:    make(map[string]string, len(old.Data)+1),
    }
    for kk, vv := range old.Data { next.Data[kk] = vv }
    next.Data[k] = v
    s.cur.Store(next)
}

Use case 1: Change detection¶

A consumer can poll and detect changes:

func (s *Store) Changed(since int64) (bool, int64) {
    cur := s.cur.Load()
    return cur.Version != since, cur.Version
}

Cheaper than diffing the whole snapshot.

Use case 2: Optimistic concurrency¶

A client reads a snapshot, modifies, and conditionally writes:

func (s *Store) CompareAndSet(expectedVer int64, fn func(*Snapshot) *Snapshot) bool {
    s.mu.Lock()
    defer s.mu.Unlock()
    old := s.cur.Load()
    if old.Version != expectedVer {
        return false // someone else wrote first
    }
    next := fn(old)
    next.Version = old.Version + 1
    s.cur.Store(next)
    return true
}

Use case 3: Replay / audit¶

If you persist every snapshot (rare!), the version is the natural primary key. More commonly, the version is logged on every write for audit.

Pitfall: monotonic but not gapless¶

If you use CAS, failed attempts produce version-number gaps. Don't rely on contiguous versions.

Watchers and Notifications¶

Real systems often need "do something when the snapshot changes." There are several patterns.

Pattern 1: Synchronous watcher list¶

type Watcher func(old, new *Config)

type Store struct {
    cur      atomic.Pointer[Config]
    mu       sync.Mutex
    watchers []Watcher
}

func (s *Store) Watch(w Watcher) { s.mu.Lock(); defer s.mu.Unlock(); s.watchers = append(s.watchers, w) }

func (s *Store) Update(fn func(*Config)) {
    s.mu.Lock()
    defer s.mu.Unlock()
    old := s.cur.Load()
    next := *old
    fn(&next)
    s.cur.Store(&next)
    for _, w := range s.watchers {
        w(old, &next) // synchronous; blocks under writer mutex
    }
}

Pro: simple, ordered, no goroutines.

Con: a slow watcher blocks all writers and other watchers.

Pattern 2: Asynchronous watcher¶

for _, w := range s.watchers {
    go w(old, &next)
}

Pro: writer is fast.

Con: watchers may run out of order; concurrent watchers may interfere.

Pattern 3: Channel-based notification¶

type Store struct {
    cur atomic.Pointer[Config]
    mu  sync.Mutex
    chs []chan *Config
}

func (s *Store) Subscribe() <-chan *Config {
    s.mu.Lock()
    defer s.mu.Unlock()
    ch := make(chan *Config, 1)
    s.chs = append(s.chs, ch)
    return ch
}

func (s *Store) Update(fn func(*Config)) {
    s.mu.Lock()
    defer s.mu.Unlock()
    old := s.cur.Load()
    next := *old
    fn(&next)
    s.cur.Store(&next)
    for _, ch := range s.chs {
        select {
        case ch <- &next:
        default: // drop if subscriber is slow
        }
    }
}

Pro: subscribers consume at their own pace, no goroutines spawned per update.

Con: drop policy must be considered. Buffered channels with non-blocking send and "skip if full" is a common compromise.

Pattern 4: Version-based polling¶

Subscribers don't get pushed updates; they poll:

last := int64(0)
for {
    cur := s.cur.Load()
    if cur.Version != last {
        handle(cur)
        last = cur.Version
    }
    time.Sleep(50 * time.Millisecond)
}

Simple, no infrastructure. Good when the polling latency is acceptable.

Pattern 5: Closed-channel "edge" notification¶

Every write closes a channel and replaces it with a fresh one. Subscribers wait on the channel and re-load when it closes.

type Store struct {
    cur atomic.Pointer[Config]
    mu  sync.Mutex
    ch  atomic.Pointer[chan struct{}]
}

func (s *Store) Notify() <-chan struct{} { return *s.ch.Load() }

func (s *Store) Update(...) {
    // ... store next
    old := s.ch.Load()
    next := make(chan struct{})
    s.ch.Store(&next)
    close(*old)
}

Subscribers:

for {
    notif := s.Notify()
    cur := s.Get()
    process(cur)
    <-notif // blocks until next update
}

Edge-triggered: one wakeup per change, regardless of subscriber count. Cheap.

Hot-Reload Patterns¶

A hot-reload is COW's canonical use case. Get this right and your service can survive most config-change incidents.

Idiom: Reload returns an error; on error, the old snapshot remains current¶

func (s *Store) Reload() error {
    next, err := s.load() // I/O, parsing, validation
    if err != nil {
        return err
    }
    s.cur.Store(next)
    return nil
}

A failed reload should never crash the service.

Idiom: Validate before publish¶

func validate(c *Config) error {
    if c.ListenAddr == "" {
        return errors.New("listen_addr required")
    }
    if c.MaxRetries < 0 {
        return errors.New("max_retries must be >= 0")
    }
    for _, h := range c.AllowedHosts {
        if _, err := url.Parse(h); err != nil {
            return fmt.Errorf("bad host %q: %w", h, err)
        }
    }
    return nil
}

Validation lives between parse and publish. A valid *Config reaches Store; an invalid one is returned as an error.

Idiom: Source pluggability¶

Decouple "where the config comes from" from "how it's published":

type Source interface {
    Load(context.Context) (*Config, error)
}

type FileSource struct{ Path string }
type EtcdSource struct{ Client *etcd.Client; Key string }
type EnvSource struct{}

func (s *Store) Reload(ctx context.Context) error {
    next, err := s.src.Load(ctx)
    if err != nil { return err }
    if err := validate(next); err != nil { return err }
    s.cur.Store(next)
    return nil
}

Testing becomes trivial: inject a Source that returns a fixture.

Idiom: Reload-triggered side effects¶

Some config changes require side effects beyond updating the snapshot — e.g., a new TLS cert requires reloading the server's TLS config.

func (s *Store) Reload() error {
    next, err := s.load()
    if err != nil { return err }
    if err := s.applySideEffects(next); err != nil {
        return err // don't publish if side effects fail
    }
    s.cur.Store(next)
    return nil
}

If side effects can fail partway, you have a partial-failure problem — common in config reload. Mitigations:

• Make side effects idempotent so retrying is safe.
• Use a "two-phase" reload: prepare, then commit. Prepare can fail; commit cannot.

Idiom: Concurrent reloads¶

Reload() is called from a signal handler, a poll loop, an admin API, and a test, all concurrently. The writer mutex serialises them. The second concurrent call simply waits for the first.

Idiom: Periodic reload with exponential backoff on failure¶

func PollLoop(ctx context.Context, s *Store, base time.Duration) {
    delay := base
    for {
        err := s.Reload()
        if err != nil {
            delay = min(delay*2, time.Minute)
        } else {
            delay = base
        }
        select {
        case <-time.After(delay):
        case <-ctx.Done():
            return
        }
    }
}

Error Handling and Validation¶

COW makes "validate before publish" easy and obvious. Use this strength.

Validation surface area¶

• Schema: required fields present, types correct.
• Semantic: ranges, mutual exclusion (e.g., "if X then not Y").
• Cross-field: consistency (e.g., MaxRetries × RetryDelay < TotalTimeout).
• External: references to existing resources (e.g., DB connectivity).

Validation order¶

Cheap checks first; expensive last:

func validate(c *Config) error {
    if c.MaxRetries < 0 { return ... }   // O(1)
    for _, h := range c.AllowedHosts {    // O(N)
        if _, err := url.Parse(h); err != nil { return ... }
    }
    if err := pingDB(c.DBURL); err != nil { return ... } // O(network)
    return nil
}

Fail fast on cheap errors; reserve expensive checks for after the cheap ones pass.

Error wrapping¶

return fmt.Errorf("invalid host %q at index %d: %w", h, i, err)

Operators read these errors. Make them actionable.

Publishing partial validation¶

Sometimes a "partial" snapshot is acceptable — e.g., new hosts are added even if some failed validation. Decide explicitly:

var validHosts []string
for _, h := range raw.Hosts {
    if _, err := url.Parse(h); err != nil {
        log.Printf("skipping invalid host %q: %v", h, err)
        continue
    }
    validHosts = append(validHosts, h)
}
next := *old
next.Hosts = validHosts
cur.Store(&next)

Log all skipped items; emit a metric for "invalid items skipped".

Atomic validation across updates¶

Update already gives you atomicity: you can validate cross-field consistency after applying all changes.

func (s *Store) Update(fn func(*Config) error) error {
    s.mu.Lock()
    defer s.mu.Unlock()
    old := s.cur.Load()
    next := *old
    if err := fn(&next); err != nil { return err }
    if err := validate(&next); err != nil { return err }
    s.cur.Store(&next)
    return nil
}

Observability¶

Make COW visible. Three pieces of telemetry pay for themselves.

Metric 1: Snapshot version (or generation)¶

metrics.ConfigGeneration.Set(float64(cfg.Load().Version))

If the gauge is the same on two pods, they have the same config. Operators love this.

Metric 2: Snapshot age¶

type Snapshot struct {
    PublishedAt time.Time
    // ...
}

metrics.ConfigAgeSeconds.Set(time.Since(cfg.Load().PublishedAt).Seconds())

A stuck reload is visible as ever-rising age.

Metric 3: Reload success/failure counters¶

func (s *Store) Reload() error {
    defer func() {
        metrics.ReloadAttempts.Inc()
    }()
    // ...
    if err != nil {
        metrics.ReloadFailures.Inc()
        return err
    }
    metrics.ReloadSuccesses.Inc()
    return nil
}

A reload-failure spike correlates with operator changes; show it on the deploy dashboard.

Logs at write boundaries¶

log.Printf("config reloaded: version=%d hosts=%d log_level=%s",
    next.Version, len(next.Hosts), next.LogLevel)

One log line per snapshot publish. Easy to grep, easy to alert on.

expvar for debugging¶

expvar.Publish("config", expvar.Func(func() any { return cfg.Load() }))

Hit /debug/vars to see the live snapshot. Invaluable in incidents.

pprof.Labels to identify long readers¶

pprof.SetGoroutineLabels(pprof.WithLabels(ctx, pprof.Labels("snapshot", "v42")))

A goroutine profile then shows which version each goroutine is holding — useful for finding "this goroutine pinned an old 100 MB snapshot for 10 minutes."

Testing COW Code¶

The race detector is your best friend. Make every COW test run with -race.

Test 1: Basic load/store¶

func TestLoadStore(t *testing.T) {
    s := NewStore(&Config{N: 1})
    if s.Get().N != 1 { t.Fatal("initial") }
    s.Update(func(c *Config) { c.N = 2 })
    if s.Get().N != 2 { t.Fatal("after update") }
}

Test 2: Old snapshot remains valid¶

func TestOldSnapshot(t *testing.T) {
    s := NewStore(&Config{N: 1})
    old := s.Get()
    s.Update(func(c *Config) { c.N = 2 })
    if old.N != 1 {
        t.Fatal("old snapshot mutated!")
    }
}

This is the immutability assertion.

Test 3: Concurrent reads do not race¶

func TestConcurrentReads(t *testing.T) {
    s := NewStore(&Config{Hosts: []string{"a", "b"}})
    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for j := 0; j < 1000; j++ {
                _ = s.Get().Hosts
            }
        }()
    }
    wg.Wait()
}

Run with -race. Any read-side bug surfaces immediately.

Test 4: Concurrent writes serialize correctly¶

func TestConcurrentWrites(t *testing.T) {
    s := NewStore(&Counter{N: 0})
    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            s.Update(func(c *Counter) { c.N++ })
        }()
    }
    wg.Wait()
    if s.Get().N != 100 {
        t.Fatalf("expected 100, got %d", s.Get().N)
    }
}

If you used CAS without a loop, this test fails. If you used a mutex, it passes.

Test 5: Mixed read/write under race¶

The "torture test". Run reads and writes concurrently for several seconds with the race detector enabled.

func TestRaceTorture(t *testing.T) {
    s := NewStore(&Config{Hosts: []string{}})
    stop := make(chan struct{})
    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
        wg.Add(1)
        go func(i int) {
            defer wg.Done()
            for {
                select {
                case <-stop: return
                default:
                    s.Update(func(c *Config) {
                        c.Hosts = append([]string(nil), c.Hosts...)
                        c.Hosts = append(c.Hosts, "h")
                    })
                }
            }
        }(i)
    }
    for i := 0; i < 50; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for {
                select {
                case <-stop: return
                default:
                    _ = len(s.Get().Hosts)
                }
            }
        }()
    }
    time.Sleep(2 * time.Second)
    close(stop)
    wg.Wait()
}

If the race detector reports anything, you have an immutability bug.

Test 6: Reload error path¶

func TestReloadFailure(t *testing.T) {
    s := NewStore(&Config{N: 1})
    src := &mockSource{err: errors.New("nope")}
    s.src = src
    if err := s.Reload(); err == nil {
        t.Fatal("expected error")
    }
    if s.Get().N != 1 {
        t.Fatal("snapshot changed despite error")
    }
}