Concurrent Counters — Middle Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Core Concepts
5. Real-World Analogies
6. Mental Models
7. Pros & Cons
8. Use Cases
9. Code Examples
10. Coding Patterns
11. Clean Code
12. Product Use / Feature
13. Error Handling
14. Security Considerations
15. Performance Tips
16. Best Practices
17. Edge Cases & Pitfalls
18. Common Mistakes
19. Common Misconceptions
20. Tricky Points
21. Test
22. Tricky Questions
23. Cheat Sheet
24. Self-Assessment Checklist
25. Summary
26. What You Can Build
27. Further Reading
28. Related Topics
29. Diagrams & Visual Aids

Introduction¶

Focus: "When atomic.Int64.Add(1) is not enough — multi-counter coordination, CompareAndSwap loops, expvar, and the basic shard pattern."

At the junior level we treated a counter as a single atomic value. That works beautifully for the textbook cases. As soon as you need any of the following, the junior toolbox runs out:

• "Increment counter A and set counter B, atomically." (Two atomics are not atomic together.)
• "Increment only if the result would not exceed a limit." (A read-then-add race.)
• "Track the maximum value ever observed." (A CAS loop.)
• "Expose dozens of counters to operators." (You need a registry.)
• "Take a consistent snapshot of multiple related counters." (A read race.)
• "Make this counter scale past one cache line of contention." (Sharding.)

This file covers the next layer. You will learn:

• How CompareAndSwap works and the loop pattern around it
• How to use atomic.Pointer[T] to swap an entire snapshot struct atomically
• The expvar package end-to-end: expvar.Int, expvar.Float, expvar.Map, expvar.Func, and the registry behind /debug/vars
• The basic shard pattern: N independent counters per "slot", summed at read time
• How to integrate counters with Prometheus / OpenTelemetry while still using atomic.Int64 underneath
• When a counter is fundamentally the wrong primitive (and you really need a histogram)

You do not yet need to know about cache-line padding, sloppy/per-CPU counters, LongAdder-style designs, or HDR histograms. Those come at the senior and professional levels.

Prerequisites¶

• Required: Junior-level fluency with atomic.Int64, sync.Mutex, sync.WaitGroup, defer, and the race detector.
• Required: Familiarity with Go generics (the typed atomic.Pointer[T] API from Go 1.19+).
• Required: Comfort writing test benchmarks (func BenchmarkXxx(b *testing.B)).
• Helpful: Some operational experience — what a /metrics endpoint is, what an alert is, what an SRE looks at when paging on a counter.
• Helpful: Awareness of the broader observability landscape: Prometheus, OpenTelemetry, StatsD, Datadog. You do not need to be an expert; the names should ring bells.

If you can write the broken count++ program, fix it three different ways, and explain the race detector's output to a colleague, you are ready.

Glossary¶

Core Concepts¶

CompareAndSwap and the lock-free loop¶

The fundamental building block for any algorithm that wants to read a value, compute a derived value, and write the derived value without using a mutex is CompareAndSwap (CAS).

type AtomicMax struct{ v atomic.Int64 }

func (m *AtomicMax) Observe(x int64) {
    for {
        cur := m.v.Load()
        if x <= cur {
            return // already the max or larger; nothing to do
        }
        if m.v.CompareAndSwap(cur, x) {
            return // we won the race; cur was still the value, now it is x
        }
        // we lost; someone else changed the value; reread and retry
    }
}

The pattern is:

1. Read the current value.
2. Decide what the new value should be (here, x if x > cur, otherwise do nothing).
3. Atomically set the value to new only if it is still cur.
4. If the CAS fails, someone else changed the value; go back to step 1.

This is lock-free: no goroutine blocks, no goroutine sleeps, no mutex is held. Under contention, some retries are wasted, but no goroutine ever stops making progress. Compared to a mutex, the latency for an uncontended operation is roughly the same (both are one atomic instruction); under contention, the CAS loop tends to do more wasted work but avoids OS-level park/unpark.

The same pattern builds:

• AtomicMin (mirror image)
• AtomicAverage (CAS-loop over sum and count)
• AtomicSet (CAS-loop adding an element)
• Linked lists, queues, stacks (with much more care)

atomic.Pointer[T] and snapshots¶

Sometimes you do not want to track one number — you want to atomically replace a whole struct. Example: a snapshot of metrics that operators read.

type Snapshot struct {
    Requests  int64
    Errors    int64
    InFlight  int64
    Timestamp time.Time
}

type Metrics struct {
    snap atomic.Pointer[Snapshot]
}

func (m *Metrics) Publish(s *Snapshot) {
    m.snap.Store(s)
}

func (m *Metrics) Read() *Snapshot {
    return m.snap.Load()
}

Two threads cannot accidentally see "Requests from time T1 but InFlight from time T2" — the pointer swap is atomic, and the struct is immutable after Publish. The reader either sees the old snapshot or the new one, never a mix.

For writes you typically have one goroutine producing snapshots (a "metrics publisher"); for reads, any number of goroutines may call Read. This decouples the cost of recording from the cost of reading.

expvar end-to-end¶

The standard library's expvar package gives you a registry of named metrics, a JSON endpoint at /debug/vars, and types for the common metric shapes — for free.

import (
    "expvar"
    "net/http"
)

var (
    requests = expvar.NewInt("requests_total")
    errors   = expvar.NewInt("errors_total")
    inflight = expvar.NewInt("inflight")
)

func handler(w http.ResponseWriter, r *http.Request) {
    requests.Add(1)
    inflight.Add(1)
    defer inflight.Add(-1)
    if err := doWork(r); err != nil {
        errors.Add(1)
        http.Error(w, err.Error(), 500)
        return
    }
    w.Write([]byte("ok"))
}

func main() {
    http.HandleFunc("/", handler)
    // /debug/vars is automatically registered when you import expvar
    http.ListenAndServe(":8080", nil)
}

curl localhost:8080/debug/vars returns:

{
  "cmdline": ["./yourbinary"],
  "memstats": { ... runtime stats ... },
  "requests_total": 14,
  "errors_total": 1,
  "inflight": 0
}

The package's types include:

• Int — atomic int64
• Float — atomic-ish float64 (uses math.Float64bits + atomic.Uint64)
• String — atomic-ish string (mutex-protected)
• Map — keyed collection of named vars
• Func — a function that returns any JSON-serialisable value (for live-computed metrics)

var memStats = expvar.NewMap("mem")
memStats.Set("heap_alloc", expvar.Func(func() any {
    var s runtime.MemStats
    runtime.ReadMemStats(&s)
    return s.HeapAlloc
}))

expvar.Func is a clever escape hatch: if you do not want to maintain a metric — you just want to compute and emit it at scrape time — wrap the producer in expvar.Func.

The basic sharded counter¶

When one atomic.Int64 is hit by 32 cores at very high rate, the cache line holding it ping-pongs between cores. Throughput per core falls dramatically. The basic fix:

type Sharded struct {
    shards [16]atomic.Int64
}

func (s *Sharded) Inc() {
    // Pick a shard. The choice of which shard a goroutine uses
    // is the interesting design question; here we use the goroutine
    // address as a cheap proxy.
    n := uintptr(unsafe.Pointer(&s)) // or pid, goroutine id, etc.
    s.shards[n%16].Add(1)
}

func (s *Sharded) Get() int64 {
    var total int64
    for i := range s.shards {
        total += s.shards[i].Load()
    }
    return total
}

Each shard lives at a different address; in the ideal case, each shard sits on its own cache line and the cores rarely fight over the same line. The cost is Get() becomes O(shards) — but reads of metrics are typically rare (once per scrape, every 15 seconds) while writes are common (once per request, every microsecond).

This is the basic idea. The good version of this is the senior file's topic: cache-line padding, the right shard count, the right hashing strategy, and (for the truly hot case) per-CPU shards via runtime_procPin. For now, the basic "N shards picked by a cheap hash" is enough to dramatically reduce contention.

Counter vs histogram¶

A counter answers "how many?" A histogram answers "what shape?".

If your operations team asks "what is the p99 latency of this handler?", a counter cannot tell you. You need a histogram (a bucketed distribution of observations). The professional file covers HDR histograms. For now: know the distinction. Many engineers reach for a counter when they need a histogram and end up with total_latency_ns / total_requests as their only signal — which discards all the tail behaviour and misleads operators.

Counters are great for:

• Counts of events (requests, errors, bytes)
• Sums of values (total bytes transferred)
• Gauges (current value of something)

Counters are wrong for:

• Distributions (latency, response size)
• Percentiles (p50, p95, p99)
• Anomaly detection on tail behaviour

Real-World Analogies¶

The CAS loop as "trying to grab the last seat"¶

Imagine a single chair in the middle of a room, and ten people who all want to sit in it. The CAS-loop way to grab the chair:

1. Look at the chair: is it empty?
2. If empty, simultaneously sit down (the "compare and swap" — sit only if still empty).
3. If two people try at the exact same moment, only one of them actually sits; the others bounce off.
4. The losers go back to step 1 (look again).

No queueing, no announcer, no one in charge. Sometimes you bounce off and have to try again. But under low contention, you always sit on the first try.

expvar as a public bulletin board¶

expvar is a bulletin board outside the building. Anyone who needs to publish a number tacks up a card with NewInt("requests_total"). Anyone who wants to know the current state walks by and reads all the cards. The publisher does not have to know who is reading; the readers do not need to coordinate.

Sharded counters as 16 separate tip jars¶

A bar with one tip jar and 16 bartenders: every tip causes a small fight as the bartenders reach in at once. Replace with one tip jar per bartender: no fights, and at the end of the night the manager sums up all the jars.

atomic.Pointer[Snapshot] as a presidential briefing¶

The president needs a daily briefing. Staff write a fresh briefing each morning and put it on the president's desk; the old one is discarded. The president reads the briefing as a whole; she never gets a half-old, half-new document. The "atomic pointer swap" is the assistant placing the new briefing on the desk and removing the old one.

Mental Models¶

CAS is "I bet the value is still X"¶

Every CAS is a bet. You read X, you compute a new value, you bet that the value is still X by the time you try to write. If the bet wins, your write succeeds. If the bet loses, you retry. Lock-free programming is the art of structuring algorithms around CAS bets.

Atomic pointer = swap the whole world¶

When the state you need is more than one word, you cannot atomic-update each word individually — you would get torn snapshots. Instead, build an immutable struct, allocate a new one when you want to change state, and atomic-swap the pointer. Every reader sees a consistent view of some version, even if not the latest.

A registry is just a map[string]Stringer¶

expvar looks magical until you read the source. It is:

var vars sync.Map // map[string]Var

with a Publish(name, v) that does vars.Store(name, v), a Do(f) that iterates the map, and an HTTP handler that walks the map and writes JSON. That is the whole package.

Sharding is "make the hot spot N times less hot"¶

Instead of one atomic that everyone contends on, have N atomics. If your sharding is good (uniform distribution, separate cache lines), per-shard contention falls roughly to 1/N. For N=16 and 32 cores, contention is moderate; for N=128 and 32 cores, contention is essentially gone. The cost is N×8 bytes of memory and an O(N) read.

Pros & Cons¶

CompareAndSwap loops¶

Pros - Lock-free; no goroutine ever sleeps on a lock - Excellent latency under low contention - Works for arbitrary read-modify-write operations - Composable with other atomics

Cons - Live-lock under extreme contention: many goroutines repeatedly fail their CAS - Harder to reason about than a mutex - ABA problem: if a value goes A → B → A between your read and your CAS, the CAS succeeds but the value "really" changed twice. For monotonic counters this is rarely an issue; for pointer-based structures it is a major concern.

atomic.Pointer[T] snapshots¶

Pros - Atomic update of arbitrarily complex state - Wait-free reads (one atomic load) - Natural fit for "read-mostly, occasionally publish" workloads

Cons - Allocates a new snapshot on every update (GC pressure) - Old snapshots can outlive the swap if readers are slow (memory waste) - Cannot update part of the snapshot; always whole-replacement

expvar¶

Pros - Standard library, zero dependencies - Automatic JSON endpoint - Free runtime metrics (memstats, cmdline) - Atomic underneath; no special care needed

Cons - JSON only — no Prometheus, no OpenTelemetry, no histograms - Registers globally; tests that import expvar get the endpoint as a side effect - The default endpoint is on http.DefaultServeMux, which is a security hazard if exposed publicly - String type uses a mutex internally (not atomic for the string itself)

Basic sharded counter¶

Pros - Dramatically reduces contention - Simple to implement - Reads are still cheap if shards are few (16 fits in one cache line group)

Cons - Picking the right shard count and hashing strategy is workload-dependent - Reads scale O(shards), so you should not read in a hot loop - The naive version still suffers from false sharing (shards adjacent in memory); the senior file's padded version fixes it

Use Cases¶

Use case: atomic "increment with cap"¶

type Capped struct {
    v   atomic.Int64
    max int64
}

func (c *Capped) Inc() bool {
    for {
        cur := c.v.Load()
        if cur >= c.max {
            return false
        }
        if c.v.CompareAndSwap(cur, cur+1) {
            return true
        }
    }
}

Returns true if the increment was allowed, false if at cap. Used in admission controllers, rate limiters, and worker-pool size limits.

Use case: leader election by atomic stamp¶

type Leader struct {
    holder atomic.Int64 // pid of current leader, or 0
}

func (l *Leader) TryClaim(myPID int64) bool {
    return l.holder.CompareAndSwap(0, myPID)
}

func (l *Leader) Release(myPID int64) bool {
    return l.holder.CompareAndSwap(myPID, 0)
}

The first goroutine to CAS from 0 to its PID is the leader. Others fall through and become followers.

Use case: epoch-based reclamation stamp¶

The "atomic counter as logical clock" pattern: bump on every write, readers compare before-and-after to detect concurrent modification. Building block for seqlocks, RCU, and STM.

Use case: read-mostly stats snapshot¶

A long-running daemon that updates its statistics every few seconds and serves them on an HTTP endpoint. Updates are infrequent; reads are frequent. atomic.Pointer[Stats] is the perfect tool.

Use case: per-route HTTP counter¶

Tens of routes, each wanting its own counter. sync.Map[string]*atomic.Int64 plus a LoadOrStore is the textbook pattern.

Use case: bytes-per-second meter¶

A counter that resets every second; an external scraper reads Swap(0) and reports the per-second rate. The CAS-loop version supports millisecond windows too.

Code Examples¶

CAS-loop maximum¶

package counters

import "sync/atomic"

type Max struct{ v atomic.Int64 }

// Observe records x; the value stored becomes max(prev, x).
func (m *Max) Observe(x int64) {
    for {
        cur := m.v.Load()
        if x <= cur {
            return
        }
        if m.v.CompareAndSwap(cur, x) {
            return
        }
        // Lost the CAS; reread and retry.
    }
}

func (m *Max) Get() int64 { return m.v.Load() }

CAS-loop running sum¶

For a "running average" you would track sum and count separately, but you cannot atomically update two int64s. Pack them:

type Avg struct {
    state atomic.Uint64 // high 32 bits: count; low 32 bits: sum
}

func (a *Avg) Observe(x int32) {
    if x < 0 {
        x = 0
    }
    for {
        cur := a.state.Load()
        count := uint32(cur >> 32)
        sum := uint32(cur)
        next := uint64(count+1)<<32 | uint64(sum+uint32(x))
        if a.state.CompareAndSwap(cur, next) {
            return
        }
    }
}

func (a *Avg) Mean() float64 {
    cur := a.state.Load()
    count := uint32(cur >> 32)
    sum := uint32(cur)
    if count == 0 {
        return 0
    }
    return float64(sum) / float64(count)
}

This is correct but inelegant. For real averages you would use HDR histograms (professional level) or accept the slight cost of a mutex.

atomic.Pointer[T] snapshot¶

package metrics

import (
    "sync/atomic"
    "time"
)

type Snapshot struct {
    At        time.Time
    Requests  int64
    Errors    int64
    InFlight  int64
}

type Publisher struct {
    requests atomic.Int64
    errors   atomic.Int64
    inflight atomic.Int64
    current  atomic.Pointer[Snapshot]
}

func (p *Publisher) RecordRequest()      { p.requests.Add(1) }
func (p *Publisher) RecordError()        { p.errors.Add(1) }
func (p *Publisher) Inflight() func()    { p.inflight.Add(1); return func() { p.inflight.Add(-1) } }

func (p *Publisher) RefreshSnapshot() {
    s := &Snapshot{
        At:       time.Now(),
        Requests: p.requests.Load(),
        Errors:   p.errors.Load(),
        InFlight: p.inflight.Load(),
    }
    p.current.Store(s)
}

func (p *Publisher) Current() *Snapshot {
    return p.current.Load()
}

RefreshSnapshot should be called periodically (every second is typical). Current is wait-free — one atomic load.

Note: the three Load() calls inside RefreshSnapshot are not all at the same instant. The snapshot is an approximation. For exact consistency you would need a more elaborate scheme (versioned counters, RCU). For metrics, "approximate at second resolution" is the standard.

expvar integration¶

package server

import (
    "expvar"
    "net/http"
)

var (
    requests = expvar.NewInt("server_requests_total")
    inflight = expvar.NewInt("server_inflight")
    errors   = expvar.NewMap("server_errors")
)

func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
    requests.Add(1)
    inflight.Add(1)
    defer inflight.Add(-1)
    if err := s.process(r); err != nil {
        errors.Add(classify(err), 1)
        http.Error(w, err.Error(), 500)
        return
    }
    w.Write([]byte("ok"))
}

func classify(err error) string {
    // map any error to a stable bucket name
    switch {
    case errors.Is(err, context.DeadlineExceeded):
        return "timeout"
    case errors.Is(err, io.EOF):
        return "eof"
    default:
        return "unknown"
    }
}

server_errors becomes a Map in JSON:

"server_errors": { "timeout": 12, "unknown": 3 }

Map.Add is goroutine-safe; it uses an internal mutex around an atomic.Int64 per key. The middle-level lesson: when you need per-key counts and the key set is unbounded, expvar.Map is the easy answer.

A basic sharded counter¶

package counters

import "sync/atomic"

const shards = 64

type Sharded struct {
    cells [shards]atomic.Int64
}

func (s *Sharded) Inc(shardKey uint64) {
    s.cells[shardKey%shards].Add(1)
}

func (s *Sharded) Get() int64 {
    var total int64
    for i := range s.cells {
        total += s.cells[i].Load()
    }
    return total
}

The shard key is the caller's responsibility. Typical choices:

• uint64(runtime_procPin()) — the current P's index (runtime-private; senior topic)
• A goroutine-local random seed
• uint64(uintptr(unsafe.Pointer(localObject))) — address of some thread-local
• A "good enough" fast hash of the goroutine ID

A naive but workable choice is to use rand.Uint64() per increment — random shard selection. This loses the locality benefit of always hitting the same shard but is trivially uniform.

The senior file shows the right way to pick a shard, with padding for false sharing and per-P locality.

A "sloppy" counter (preview)¶

A sloppy counter trades exact freshness for throughput. Each goroutine maintains a local counter that it can increment without any synchronisation; periodically, the local value is added to a global atomic.

type Sloppy struct {
    global atomic.Int64
}

type LocalCounter struct {
    local  int64
    flush  int64
    parent *Sloppy
}

func (s *Sloppy) Local(flushAt int64) *LocalCounter {
    return &LocalCounter{flush: flushAt, parent: s}
}

func (l *LocalCounter) Inc() {
    l.local++
    if l.local >= l.flush {
        l.parent.global.Add(l.local)
        l.local = 0
    }
}

func (l *LocalCounter) Flush() {
    if l.local > 0 {
        l.parent.global.Add(l.local)
        l.local = 0
    }
}

func (s *Sloppy) Get() int64 {
    return s.global.Load()
}

Caveats:

• The LocalCounter is not safe to share across goroutines — each goroutine must hold its own.
• The global count lags reality by up to flushAt - 1 per goroutine.
• You must call Flush() when the goroutine exits or the local count is lost.

Sloppy counters are the senior-level next step from sharded counters. Mentioned here so you know the term.

Coding Patterns¶

Pattern: increment-with-undo¶

n := c.Add(1)
if n > limit {
    c.Add(-1)
    return ErrTooMany
}

Slight over-shoot: between the Add and the Add(-1), the counter is briefly past the limit. Other concurrent increments might see this state. For an exact cap, use CAS:

for {
    cur := c.Load()
    if cur >= limit {
        return ErrTooMany
    }
    if c.CompareAndSwap(cur, cur+1) {
        return nil
    }
}

Pattern: snapshot-and-reset¶

prev := c.Swap(0)
report(prev)

Swap(0) is atomic — you cannot lose increments between the read and the reset. Anything done after Swap is on the captured value prev; the counter is freshly zero.

Pattern: publish-snapshot¶

For multi-field state, replace the whole struct atomically:

type State struct { ... }
var state atomic.Pointer[State]

// publisher (rare):
state.Store(&newState)

// reader (frequent):
s := state.Load()
use(s)

Pattern: per-key counter via sync.Map¶

type Counters struct{ m sync.Map }

func (c *Counters) Inc(key string) {
    v, ok := c.m.Load(key)
    if !ok {
        v, _ = c.m.LoadOrStore(key, &atomic.Int64{})
    }
    v.(*atomic.Int64).Add(1)
}

For a fixed key set known at startup, use a map[string]*atomic.Int64 built once — no sync.Map needed.

Pattern: counter exposed via expvar.Func for derived metrics¶

expvar.Publish("avg_request_size", expvar.Func(func() any {
    n := requestsTotal.Value()
    if n == 0 {
        return 0.0
    }
    return float64(bytesInTotal.Value()) / float64(n)
}))

expvar.Func evaluates at scrape time. Use it for computed metrics that you do not want to maintain incrementally.

Pattern: sharded write, summed read¶

type Sharded struct{ cells [N]atomic.Int64 }

func (s *Sharded) Inc(key uint64)     { s.cells[key%N].Add(1) }
func (s *Sharded) Get() (n int64)     {
    for i := range s.cells { n += s.cells[i].Load() }
    return
}

Pattern: thread-local accumulation + periodic flush¶

type local struct{ n int64 }

var localStats sync.Pool // pool of *local, one per goroutine

func record() {
    l := localStats.Get().(*local)
    l.n++
    if l.n >= 1024 {
        global.Add(l.n)
        l.n = 0
    }
    localStats.Put(l)
}

sync.Pool gives near-per-goroutine semantics. Not exact, but the GC will reclaim unused locals, which is desirable.

Pattern: CAS-loop accumulator¶

func atomicMax(p *atomic.Int64, x int64) {
    for {
        cur := p.Load()
        if x <= cur { return }
        if p.CompareAndSwap(cur, x) { return }
    }
}

func atomicMin(p *atomic.Int64, x int64) {
    for {
        cur := p.Load()
        if x >= cur { return }
        if p.CompareAndSwap(cur, x) { return }
    }
}

func atomicAddIfLT(p *atomic.Int64, delta, cap int64) bool {
    for {
        cur := p.Load()
        if cur+delta > cap { return false }
        if p.CompareAndSwap(cur, cur+delta) { return true }
    }
}

These three are the workhorses of middle-level lock-free counter code.

Clean Code¶

Wrap atomics in domain types¶

// BAD: callers see raw atomics; easy to misuse.
var requestCount atomic.Int64

// GOOD: callers see a domain type with named operations.
type RequestCounter struct{ v atomic.Int64 }
func (r *RequestCounter) Increment()  { r.v.Add(1) }
func (r *RequestCounter) Value() int64 { return r.v.Load() }

Document monotonicity in the type system, not just comments¶

// MonotonicCounter never decreases.
type MonotonicCounter struct{ v atomic.Int64 }
func (m *MonotonicCounter) Inc() { m.v.Add(1) }
func (m *MonotonicCounter) Add(n uint64) { m.v.Add(int64(n)) } // never negative
func (m *MonotonicCounter) Value() int64 { return m.v.Load() }
// No Dec, no Set, no Reset.

Name registry keys with care¶

expvar registrations live forever. Two packages registering the same name will panic at init. Establish a project-wide naming convention:

<service>_<subsystem>_<metric>_<unit>

e.g., myapi_db_queries_total, myapi_cache_hits_total, myapi_inflight_requests.

Snapshot types are immutable¶

If you use atomic.Pointer[Snapshot], declare Snapshot as a value with no setters. Each snapshot is "frozen" the moment it is published.

type Snapshot struct {
    At        time.Time
    Requests  int64
    Errors    int64
    InFlight  int64
}

No methods that modify. If you need to derive a new snapshot, allocate a new one.

Avoid global counters in libraries¶

A library that internally uses expvar.NewInt("foo") registers a globally-visible metric on import. That is rarely what library users want. Prefer to expose a *Counters struct that callers can register where they like:

type Counters struct {
    Requests atomic.Int64
    Errors   atomic.Int64
}

func (c *Counters) RegisterTo(m *expvar.Map) {
    m.Set("requests", expvar.Func(func() any { return c.Requests.Load() }))
    m.Set("errors", expvar.Func(func() any { return c.Errors.Load() }))
}

Product Use / Feature¶

Admission control¶

atomic.Int64 plus a CAS-loop cap gives you a perfect "max in-flight" gate.

Quota enforcement¶

Per-user atomic counters with periodic reset. For low cardinality (a few users), sync.Map[uid]*atomic.Int64; for high cardinality, you need a distributed scheme (Redis INCR, database token bucket).

Rate-derived metrics¶

Combine a counter with a periodic snapshotter to produce a per-second rate. Operations teams almost always want rates, not totals.

Feature flag rollouts¶

A counter per (flag, bucket) tracks how often each variant was selected. expvar.Map is the natural fit.

Per-tenant accounting¶

Counters per tenant for billing. Use sharded counters within a tenant if the tenant is hot; otherwise a single atomic.Int64 per tenant is fine.

Internal observability¶

Goroutine pools, channel depths, retry counts, breaker trips — all naturally counters. The middle-level shape is: one counter per state transition, summed at read time, exposed via expvar.

Error Handling¶

Atomic operations cannot fail. CAS can "fail" — meaning the swap did not happen — but that is part of normal operation, not an error. Wrap CAS loops carefully:

func (c *Capped) Inc() error {
    for {
        cur := c.v.Load()
        if cur >= c.max {
            return ErrAtCap
        }
        if c.v.CompareAndSwap(cur, cur+1) {
            return nil
        }
    }
}

The only "error" is hitting the cap. CAS loop retries are not errors.

For expvar, the failure modes are:

• NewInt(name) panics if name is already registered. Catch this at init.
• The HTTP handler can panic if a Func-backed var panics during serialisation. Always recover in custom Func implementations.
• The default /debug/vars is wired to http.DefaultServeMux. If you also serve user traffic on the default mux, you have leaked your metrics. Always use a dedicated http.ServeMux for /debug/*.

Security Considerations¶

Counter-based authorisation is fragile¶

"Allow the third request from this user" using a per-user atomic counter has race windows. Two requests arriving simultaneously may both see the count at 2, both CAS to 3, and the third request is not the "third" one — it is the first to succeed. For real authorisation, use explicit per-session state with proper synchronisation, not raw counters.

Exposing expvar to the public internet¶

/debug/vars reveals all your metrics — request volumes, error rates, internal state. Attackers can profile your service from outside. Always authenticate /debug/* or bind it to a private interface.

Counter overflow as a DoS vector¶

If an attacker can drive a counter at line rate, can they overflow it? int64 is impractical to overflow (~292 years at 1 billion ops/sec). But int32 (~4 billion ops, achievable in tens of minutes at peak) is risky. Use int64 unless you have a specific reason not to.

Information leakage through timing¶

A CAS loop that retries under contention has measurable timing variance. In theory an attacker can use this as a side channel to infer concurrent load. In practice, this matters for cryptographic counters but rarely for application metrics.

Don't log raw counter values per request¶

Including a global counter value in every request log line can reveal request ordering to anyone who reads the logs. Not always a problem, but worth thinking about for sensitive deployments.

Performance Tips¶

Bench, do not guess¶

Every claim in this section should be re-verified on your hardware:

func BenchmarkAtomicAdd(b *testing.B) {
    var c atomic.Int64
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            c.Add(1)
        }
    })
}

Run with -cpu=1,2,4,8,16,32. Watch ops/sec per CPU drop as parallelism grows. That drop is your contention cost — and the motivation for sharding.

Atomic ops are ~5 ns each, but not free¶

A single uncontended atomic add is 5–15 ns on modern x86-64. Under contention it can be hundreds of nanoseconds. Mutex Lock/Unlock is 25–50 ns uncontended; under contention it can blow up to microseconds or more (if it ends up parking goroutines).

Load is cheaper than Add¶

A bare Load is typically a single MOV with an acquire fence — half the cost of an Add (which involves a LOCK XADD). If you can replace many Add(1)s with a single accumulated Add(n), do it.

Swap is cheaper than Load + Store combined¶

Plus it is atomic with respect to other operations. Always prefer Swap.

CAS loops are cheap in low contention, expensive in high contention¶

Under low contention, a CAS loop almost always succeeds first try and is the same cost as an Add. Under high contention, retries waste CPU. If your CAS is failing more than ~50% of the time, you have a contention problem that sharding (or a mutex with backoff) might solve better.

Shard count: 2–4× the number of cores¶

Too few shards: still contention. Too many shards: cache-thrashing on the read path. 64 is a defensible default for most workloads. 16 for low core counts (4–8 cores). 256 only for very-large boxes (64+ cores).

Read latency matters less than write latency¶

If reads happen once per scrape (every 15 seconds), spending 1 microsecond summing 1024 shards is fine. If reads happen on every request, sum fewer shards or use a periodic-snapshot pattern.

Best Practices¶

• Default to atomic.Int64. Reach for fancier patterns only when measurement shows a problem.
• Use CompareAndSwap for read-modify-write where the new value depends on the old.
• Use atomic.Pointer[T] for multi-field consistent snapshots.
• Use expvar for "expose me to the operators with no ceremony".
• Use a dedicated ServeMux for /debug/*; never expose the default mux publicly.
• Document monotonicity in the type system if possible.
• Benchmark with -cpu to confirm scaling behaviour.
• Avoid registering metrics in init() from a library — let the caller register.
• Always use pointer receivers for atomic-bearing structs.
• go vet warnings about copying atomics are never spurious.

Edge Cases & Pitfalls¶

CAS loop on a hot location¶

If 32 goroutines are all CAS-looping the same counter, most retries will fail. Throughput collapses. Solutions: switch to plain Add if you only need increment; shard; or use a mutex with backoff.

ABA in CAS loops on pointers¶

If a CAS loops on a pointer that goes from A to B and back to A between read and CAS, the CAS succeeds — but the meaning of A may have changed. For counters this is rarely an issue (counts going down and back up is suspicious anyway); for pointer-based data structures it requires versioned pointers or epoch-based reclamation. Senior topic.

atomic.Pointer[T] swap leaves old snapshot allocated¶

Until the last reader drops the old snapshot, it stays in memory. Long-lived readers can pin many generations of snapshots. Mitigation: short-lived readers, snapshot size budget, periodic forced refresh.

expvar.Int.Set is non-atomic-relative-to-Add¶

Set(x) is atomic.StoreInt64; Add(y) is atomic.AddInt64. If two goroutines do Set(0) and Add(1) simultaneously, the result is one of {0, 1} depending on order. This is "atomic" in the safety sense but not "atomic" in the transaction sense. For "reset and resume", use Swap(0).

Sharded counter shard skew¶

If your shard-key choice is correlated with which goroutine is hot (e.g., always shard 0 because the hash of the master goroutine ID is 0), you have not solved contention. Verify shard balance empirically.

expvar.Map.Init() vs expvar.NewMap¶

NewMap(name) registers globally; Init() initialises an existing map without registering. Mixing them in tests is a source of "duplicate name" panics. Use NewMap once at process startup, Init never (for most cases).

atomic.Int64{}.Add on a copy¶

Copying a struct that holds an atomic.Int64 is a bug go vet will flag. Confirmed by reading: every method must be on a pointer receiver.

atomic.Pointer[T] to nil¶

Load() on a never-Stored atomic.Pointer[T] returns nil. Always handle that case:

s := state.Load()
if s == nil {
    s = &Snapshot{} // safe default
}

Snapshot publisher that itself races¶

The publisher reads all the underlying atomics and packages them into a snapshot. If the underlying atomics are still being updated, the snapshot reflects a slightly inconsistent moment. For metrics, this is acceptable; for transactional state, it is not.

Common Mistakes¶

1. Using Load + Store instead of Swap for reset. Loses increments in between.
2. CAS-looping on a field that is also Add-updated by others. The CAS keeps failing.
3. Registering the same expvar name twice. Panic at init.
4. Holding a mutex while calling expvar.Int.Add. The mutex is unnecessary; remove it.
5. Forgetting to Flush() a sloppy counter on goroutine exit. Up to flushAt increments are lost.
6. Using sharded counters when an unsharded counter is fine. Premature optimisation.
7. Snapshotting a multi-counter struct with multiple Loads instead of atomic.Pointer[T]. Inconsistent snapshots.
8. expvar exposed on default mux that is also the public mux. Information leak.
9. int32 counter that can be driven to overflow. DoS vector.
10. Treating CAS failure as an error. It is not; just retry.

Common Misconceptions¶

• "CAS is wait-free." It is lock-free, not wait-free. Some goroutines may retry many times; only the system as a whole is guaranteed to make progress.
• "atomic.Pointer[T] is free." It allocates on every Store (you allocated the struct yourself). GC pressure is real.
• "expvar is for production monitoring." It is fine for low-traffic JSON-scraping monitoring but not as scalable or expressive as Prometheus client libraries. Use as a "starter" tool.
• "Sharded counters always help." Not at low contention. They add read cost and code complexity for no benefit unless you have measured contention.
• "Sloppy counters lose data." They lose freshness, not correctness — the total is right once everyone flushes.

Tricky Points¶

CompareAndSwap returns whether the swap happened¶

if !c.CompareAndSwap(0, 1) {
    // someone else has already set it
}

The boolean is the only signal. There is no "previous value" returned by Go's CAS API; if you need both, use Swap (which atomically replaces and returns the prior value).

atomic.Pointer[T] is a value, not a pointer¶

var p atomic.Pointer[State] // a value containing a pointer
p.Store(&State{})           // OK

The type itself contains the pointer. Do not pass an atomic.Pointer[State] by value — that copies the embedded atomic, which is a bug.

expvar is initialised lazily¶

expvar.NewInt("foo") may not be visible at /debug/vars until you actually call http.ListenAndServe (which triggers the side-effect registration via init). In tests, force-import _ "expvar" to register.

expvar.Map.Add does an atomic op per call¶

You might be tempted to grab the inner *atomic.Int64 and call Add directly:

v := stats.Get("foo")
if v == nil {
    stats.Set("foo", new(expvar.Int))
    v = stats.Get("foo")
}
v.(*expvar.Int).Add(1)

Beware races between Get and Set. The Add(key, delta) method on expvar.Map is atomic; prefer it.

Sharded counter alignment¶

If your shards live in adjacent memory (e.g., [64]atomic.Int64), several may share the same 64-byte cache line. False sharing! Senior file shows the padded version.

CAS-loop livelock¶

In the worst case, all goroutines may keep retrying without anyone making progress. Probability is low but non-zero. The standard mitigation is exponential backoff inside the loop:

backoff := time.Nanosecond
for {
    cur := p.Load()
    next := compute(cur)
    if p.CompareAndSwap(cur, next) {
        return
    }
    time.Sleep(backoff)
    backoff *= 2
}

Rarely needed in practice for simple counters, but worth knowing.

Snapshot freshness vs cost¶

How often should you RefreshSnapshot? Too often and you waste CPU on snapshot work; too rare and operators see stale data. Match scrape interval (typically 5–15s for Prometheus).

Test¶

package counters

import (
    "sync"
    "sync/atomic"
    "testing"
)

func TestMax_Observe(t *testing.T) {
    var m Max
    var wg sync.WaitGroup
    for i := int64(0); i < 100; i++ {
        i := i
        wg.Add(1)
        go func() { defer wg.Done(); m.Observe(i) }()
    }
    wg.Wait()
    if got := m.Get(); got != 99 {
        t.Errorf("expected 99, got %d", got)
    }
}

func TestCAS_Capped(t *testing.T) {
    c := &Capped{max: 100}
    var ok atomic.Int64
    var fail atomic.Int64
    var wg sync.WaitGroup
    for i := 0; i < 1000; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            if c.Inc() { ok.Add(1) } else { fail.Add(1) }
        }()
    }
    wg.Wait()
    if got := ok.Load(); got != 100 {
        t.Errorf("expected 100 successes, got %d", got)
    }
    if got := fail.Load(); got != 900 {
        t.Errorf("expected 900 failures, got %d", got)
    }
}

func TestSharded(t *testing.T) {
    var s Sharded
    var wg sync.WaitGroup
    for i := uint64(0); i < 10000; i++ {
        i := i
        wg.Add(1)
        go func() { defer wg.Done(); s.Inc(i) }()
    }
    wg.Wait()
    if got := s.Get(); got != 10000 {
        t.Errorf("expected 10000, got %d", got)
    }
}

func BenchmarkSingleAtomic(b *testing.B) {
    var c atomic.Int64
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            c.Add(1)
        }
    })
}

func BenchmarkSharded(b *testing.B) {
    var s Sharded
    var key uint64
    b.RunParallel(func(pb *testing.PB) {
        myKey := atomic.AddUint64(&key, 1)
        for pb.Next() {
            s.Inc(myKey)
        }
    })
}

Run both benchmarks at -cpu=1,8,32 and watch the ratio. At -cpu=1, single-atomic wins (no contention; sharded pays extra for shard lookup). At -cpu=32, sharded should be 5–20× faster.

Tricky Questions¶

Q: What happens if a CAS loop runs forever? A: In theory, livelock — possible but extremely rare for simple counters. In practice, exponential backoff or switching to a mutex when contention is detected solves it. Lock-free algorithms are still guaranteed to make progress overall — at least one operation succeeds per cycle — but individual operations can starve.

Q: Is atomic.Pointer[T].Load() wait-free? A: Yes — single atomic load, never retries, never blocks. The most predictable read primitive Go offers.

Q: Why doesn't Go provide a typed atomic.Map? A: sync.Map exists; it is mutex-and-atomic-based, not fully atomic. A truly atomic map (CAS on the whole table) would have poor write performance. The standard library's choice is the right trade-off.

Q: Does expvar.Int.Add work concurrently with expvar.Int.Set? A: Yes — both go through atomic.AddInt64 / atomic.StoreInt64. But the result is the last writer wins in the temporal sense; concurrent Set and Add can produce any value in {x, x+delta} depending on ordering. Sequence them carefully.

Q: How do I monitor a sharded counter without expensive reads? A: Either (a) accept the O(shards) read cost (cheap for shards ≤ 1024); (b) maintain a published snapshot updated by a background goroutine.

Q: Can I use atomic.Pointer[T] to implement copy-on-write? A: Yes — it is the textbook implementation. Read the old snapshot, copy with modifications, CAS the pointer. Retry on CAS failure.

Q: What's the difference between expvar.Map and a sync.Map of atomic.Int64? A: expvar.Map exposes to /debug/vars automatically and uses a mutex (not lock-free). sync.Map[string, *atomic.Int64] is faster for hot keys but you have to expose it yourself.

Q: When should I use expvar over Prometheus client lib? A: expvar for tiny services, internal tools, and "just give me a JSON endpoint". Prometheus for production-scale, multi-label, multi-histogram metrics. They can coexist.

Q: Can I use atomics to implement a queue? A: Yes — Michael-Scott queue, LMAX Disruptor, ring buffers — all are atomic-based. They are not simple; senior/professional topic. For most use cases, chan is the right answer.

Q: Is atomic.Pointer[T].CompareAndSwap ABA-safe? A: No — if a pointer goes A → B → A, the CAS sees it as unchanged. For counter-like uses this is rarely an issue (you usually use atomic.Int64 for counters, not pointers). For pointer-based data structures, you need versioned pointers or hazard pointers — senior/professional topic.

Q: Why does expvar.Float exist when atomic doesn't have a Float64? A: It uses atomic.Uint64 plus math.Float64bits / math.Float64frombits to store and load. The same trick works for any non-int atomic.

Cheat Sheet¶

// CAS loop
for {
    cur := v.Load()
    if !canApply(cur, x) { return }
    if v.CompareAndSwap(cur, apply(cur, x)) { return }
}

// Snapshot publisher
var snap atomic.Pointer[State]
snap.Store(&State{...}) // publish
s := snap.Load()        // read

// expvar
var c = expvar.NewInt("name")
c.Add(1)
c.Value()

// Sharded
var s [64]atomic.Int64
s[key%64].Add(1)
total := int64(0)
for i := range s { total += s[i].Load() }

Self-Assessment Checklist¶

• I can write a CAS loop for max/min/cap-enforced increment without looking it up
• I can explain when Swap(0) is correct and when Load()+Store(0) is broken
• I can use atomic.Pointer[T] for multi-field snapshots
• I have written a small expvar-backed metrics endpoint
• I understand the difference between counter and histogram
• I know the basic shape of a sharded counter and when it helps
• I can write a benchmark with RunParallel and read its scaling
• I know when not to shard

Summary¶

The middle level of concurrent counters is about composition and exposure: composing multiple atomics into consistent snapshots (atomic.Pointer[T]), composing read-modify-write into lock-free operations (CAS loops), composing per-key counters (sync.Map + atomic), and exposing all of it to operators (expvar).

The basic sharded counter sits at the bridge between middle and senior. It is the answer to "my one atomic is the bottleneck"; the senior file refines it with cache-line padding, sloppy / per-CPU semantics, and LongAdder-style dynamic sharding.

The professional file completes the story with HDR histograms (for distributions, not just counts), NUMA-aware shard placement, and the design of an observability subsystem that exposes everything to Prometheus, OpenTelemetry, and custom backends — all built on the primitives you learned here.

What You Can Build¶

• A cap-enforced admission controller
• A leader election by atomic stamp
• A multi-field snapshot publisher
• An HTTP server with expvar-exposed metrics
• A sharded counter (basic; the senior version is harder)
• A per-route HTTP counter with sync.Map
• A MaxLatencyTracker with CAS-loop max
• A sloppy counter (the senior file polishes this)
• A read-mostly state publisher

Further Reading¶

• sync/atomic package docs: https://pkg.go.dev/sync/atomic
• expvar package docs: https://pkg.go.dev/expvar
• The Go memory model: https://go.dev/ref/mem
• Russ Cox, "Updating the Go Memory Model" (2021): https://research.swtch.com/gomm
• Maurice Herlihy & Nir Shavit, The Art of Multiprocessor Programming — chapters on CAS and lock-free algorithms
• Doug Lea's java.util.concurrent.LongAdder source (for inspiration on dynamic sharding)
• LMAX Disruptor papers on cache-aware concurrent data structures

Related Topics¶

• Junior counters (this file's prerequisite)
• Senior counters (cache-line padding, sharded counters in depth, sloppy counters)
• Professional counters (HDR histograms, expvar+Prometheus integration, NUMA)
• atomic.Pointer[T] (broader use beyond snapshots)
• sync.Map
• The Go race detector
• Memory ordering in the Go memory model
• Lock-free data structures (queues, stacks)
• RCU and epoch-based reclamation (advanced)

Diagrams & Visual Aids¶

CAS loop flow¶

read current value
  |
  v
compute desired new value
  |
  v
attempt CAS(current, new)
  |
  +----- success: return
  |
  +----- failure: someone changed it; restart

Snapshot publisher¶

+-------------+       +-------------+       +-------------+
| publisher   | ----> | atomic.     | <---- | reader      |
| (1/sec)     |       | Pointer[S]  |       | (frequent)  |
+-------------+       +-------------+       +-------------+
       |                     ^                     |
       v                     |                     v
   allocates              swaps in                Load()
   new Snapshot           atomically              returns
                                                  current S

Sharded counter¶

goroutine A --hash--> shard 3 ----+
goroutine B --hash--> shard 7 ----|
goroutine C --hash--> shard 3 ----|----- sum at read time ----> total
goroutine D --hash--> shard 1 ----|
goroutine E --hash--> shard 9 ----+

expvar registry¶

init():                              GET /debug/vars:
  expvar.NewInt("requests")
  expvar.NewInt("errors")            for each name in registry:
  expvar.NewMap("by_route")            write name, JSON value
                                     (memstats, cmdline auto-included)

That is the middle-level toolkit. The senior file picks up at the question "but why does my sharded counter scale only 4×, not 16×?" The answer is cache-line padding, and the rabbit hole goes deep.

Deep Dive: CAS in Detail¶

The CAS instruction is the single most important primitive in modern concurrent programming. Almost every lock-free algorithm boils down to one or more CAS loops. Understanding it deeply pays off forever.

What the hardware does¶

CMPXCHG on x86-64 (with the LOCK prefix for atomicity across cores) compares the value at a memory address with a value in the AX register. If equal, the register's content is stored to memory; if not, the memory value is loaded into AX. Either way, the result is reported via the zero flag (ZF).

LOCK CMPXCHG [counter], ebx   ; if [counter] == eax, [counter] <- ebx; else eax <- [counter]

The LOCK prefix means: for the duration of this instruction, the cache line containing [counter] is exclusively owned. No other CPU may read or write it. This is what makes the operation atomic.

On ARMv8.1+ there is LDADD-style atomic that does load-add-store as one operation. On older ARM cores, CAS is built from LDREX/STREX (load-exclusive / store-exclusive) — the CPU monitors whether anyone else touched the cache line between LDREX and STREX, and STREX fails if so:

loop:
  LDREX r0, [counter]    ; load with exclusive monitor
  CMP   r0, r1            ; compare with expected
  BNE   fail              ; not equal, return false
  STREX r2, r3, [counter] ; store new value if monitor still set
  CBNZ  r2, loop          ; STREX failed (someone interfered); retry
  MOV   r0, #1            ; success
fail:
  ...

Notice how the ARM "single CAS" is actually a retry loop in the instruction set itself. The sync/atomic package wraps this so you never see it.

Why CAS loops are needed¶

A single CAS is one attempt. The CAS loop turns that into a logical "atomic update with a computed new value". The compute is not protected by the CAS — it sees only the cur value at the moment of the read. If anyone else changes the value between your read and your CAS, your CAS fails and you start over.

for {
    cur := v.Load()
    next := f(cur)      // computation outside any lock
    if v.CompareAndSwap(cur, next) {
        return next
    }
    // Lost; cur is stale; reread.
}

This is the structure. It is universal. Memorise it.

When CAS loops are the wrong tool¶

• Long compute between read and CAS. The longer f(cur) takes, the more likely someone else will win the race. Throughput drops.
• Pointer-based ABA. If cur and next are pointers and cur could be reallocated between your read and your CAS, the CAS may succeed despite the meaning having changed.
• Multi-word atomicity. CAS works on one word. Multi-word CAS exists in hardware (DCAS, LL/SC variants) but is not exposed by Go.
• High contention. A live-locking CAS storm at 32 cores is worse than a mutex. The mutex parks losers; CAS keeps them spinning.

CAS loop with limited retries¶

In rare cases you want to give up if CAS keeps failing — for instance, an admission controller that should not spin forever:

const maxRetries = 10

func (c *Capped) IncWithLimit() bool {
    for i := 0; i < maxRetries; i++ {
        cur := c.v.Load()
        if cur >= c.max {
            return false
        }
        if c.v.CompareAndSwap(cur, cur+1) {
            return true
        }
    }
    return false // assume overload
}

Use sparingly. Returning false after maxRetries is a heuristic, not a correctness guarantee.

CAS loop with exponential backoff¶

Under extreme contention, spinning hurts everyone. Adding a tiny backoff helps:

func (m *Max) Observe(x int64) {
    backoff := 1
    for {
        cur := m.v.Load()
        if x <= cur {
            return
        }
        if m.v.CompareAndSwap(cur, x) {
            return
        }
        for i := 0; i < backoff; i++ {
            runtime.Gosched()
        }
        if backoff < 1024 {
            backoff *= 2
        }
    }
}

runtime.Gosched() yields to other goroutines, giving the contender a chance to publish. Doubling the wait time reduces overall contention. Rarely necessary for counters; very useful for high-contention data structures.

Spurious-failure CAS variants¶

Some platforms have a "weak CAS" that can fail even when the value matches, in exchange for slightly cheaper execution. Go's CompareAndSwap is the strong form on all platforms — it fails only if the value differs. You do not need to worry about spurious failure.

Deep Dive: atomic.Pointer[T] and Snapshot Patterns¶

The shape¶

var snap atomic.Pointer[Snapshot]

// publisher
new := buildSnapshot()
snap.Store(new) // atomic

// reader
s := snap.Load() // atomic
use(s)

The snapshot itself must be immutable after Store. If you ever mutate a published snapshot, a reader may see a half-updated state.

Copy-on-write update¶

func (m *Metrics) update(fn func(s Snapshot) Snapshot) {
    for {
        cur := m.snap.Load()
        var curVal Snapshot
        if cur != nil {
            curVal = *cur
        }
        next := fn(curVal)
        if m.snap.CompareAndSwap(cur, &next) {
            return
        }
    }
}

// usage
m.update(func(s Snapshot) Snapshot {
    s.Requests++
    s.LastAt = time.Now()
    return s
})

Every update allocates a new Snapshot. Concurrent updates retry. The result: a fully consistent multi-field state, updated atomically.

Cost: allocation per update. For metrics updated thousands of times per second, this is too expensive — prefer separate atomic.Int64s and snapshot-by-copy at scrape time. For state updated rarely (configuration changes, leader elections), copy-on-write via atomic.Pointer[T] is excellent.

The seqlock alternative¶

For data that is written rarely and read frequently, a seqlock can outperform atomic.Pointer[T] by avoiding allocations:

type SeqLock struct {
    seq  atomic.Uint64
    data Snapshot // protected by seq's parity
}

func (s *SeqLock) Write(fn func(*Snapshot)) {
    s.seq.Add(1)  // make odd
    fn(&s.data)
    s.seq.Add(1)  // make even
}

func (s *SeqLock) Read() Snapshot {
    for {
        v1 := s.seq.Load()
        if v1%2 == 1 {
            continue // write in progress
        }
        d := s.data // copy
        v2 := s.seq.Load()
        if v1 == v2 {
            return d
        }
        // changed during read; retry
    }
}

Caveats: the reader is not lock-free in the strict sense (it may retry forever if writes are constant), and the data copy can be expensive for large structs. For small, frequently-read snapshots, seqlocks are excellent. For Go specifically, atomic.Pointer[T] is simpler and usually fast enough. Seqlocks are more common in the Linux kernel and embedded systems.

atomic.Value: the predecessor¶

Before atomic.Pointer[T] (Go 1.19+) there was atomic.Value — an untyped, interface-based atomic-stored value. It still works:

var snap atomic.Value
snap.Store(&Snapshot{...})
s := snap.Load().(*Snapshot)

The downsides vs atomic.Pointer[T]:

• Type-asserted on every Load — runtime cost and no compile-time safety.
• Once you call Store(x), all subsequent Stores must be the same concrete type — accidental mismatch panics.
• No CompareAndSwap method (until Go 1.17 added it; the typed Pointer[T] is still cleaner).

In new code, prefer atomic.Pointer[T]. In old code, leave atomic.Value alone unless refactoring nearby.

Deep Dive: Reading expvar Source¶

The entire expvar package is ~250 lines. Read it. Some excerpts (paraphrased):

// expvar.Int
type Int struct {
    i atomic.Int64
}

func (v *Int) Value() int64       { return v.i.Load() }
func (v *Int) String() string {
    return strconv.FormatInt(v.i.Load(), 10)
}
func (v *Int) Add(delta int64) { v.i.Add(delta) }
func (v *Int) Set(value int64) { v.i.Store(value) }

// expvar.Float
type Float struct {
    f atomic.Uint64 // store float64 bits
}

func (v *Float) Value() float64 {
    return math.Float64frombits(v.f.Load())
}
func (v *Float) Add(delta float64) {
    for {
        cur := v.f.Load()
        nxtVal := math.Float64frombits(cur) + delta
        nxt := math.Float64bits(nxtVal)
        if v.f.CompareAndSwap(cur, nxt) {
            return
        }
    }
}

Notice: Float.Add is a CAS loop because there is no atomic float-add instruction. Under contention this can be much slower than Int.Add. Knowing this helps: if you find yourself adding to an expvar.Float from many goroutines, consider keeping it as Int of nanoseconds (for time) or basis-points (for ratios) and converting at read time.

// expvar.Map
type Map struct {
    m      sync.Map // map[string]Var
    keysMu sync.RWMutex
    keys   []string // sorted; for stable iteration
}

func (v *Map) Add(key string, delta int64) {
    i, ok := v.m.Load(key)
    if !ok {
        var dup bool
        i, dup = v.m.LoadOrStore(key, new(Int))
        if !dup {
            v.addKey(key)
        }
    }
    if iv, ok := i.(*Int); ok {
        iv.Add(delta)
    }
}

Map.Add is the double-checked load-or-store pattern, with the inner increment delegated to Int.Add (atomic). Reading this teaches you the idiomatic concurrent map+counter combo for free.

// expvar.Func
type Func func() any
func (f Func) Value() any   { return f() }
func (f Func) String() string {
    v, _ := json.Marshal(f())
    return string(v)
}

Func is the bridge from "I have a derived value" to "I need to expose it as JSON". The function runs at scrape time.

// expvar's HTTP handler
func expvarHandler(w http.ResponseWriter, r *http.Request) {
    w.Header().Set("Content-Type", "application/json; charset=utf-8")
    fmt.Fprintf(w, "{\n")
    first := true
    Do(func(kv KeyValue) {
        if !first { fmt.Fprintf(w, ",\n") }
        first = false
        fmt.Fprintf(w, "%q: %s", kv.Key, kv.Value)
    })
    fmt.Fprintf(w, "\n}\n")
}