Concurrent Counters — Senior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Core Concepts
5. Real-World Analogies
6. Mental Models
7. Pros & Cons
8. Use Cases
9. Code Examples
10. Coding Patterns
11. Clean Code
12. Architecture & Design
13. Error Handling
14. Security Considerations
15. Performance Engineering
16. Best Practices
17. Edge Cases & Pitfalls
18. Common Mistakes
19. Common Misconceptions
20. Tricky Points
21. Test
22. Tricky Questions
23. Cheat Sheet
24. Self-Assessment Checklist
25. Summary
26. What You Can Build
27. Further Reading
28. Related Topics
29. Diagrams & Visual Aids

Introduction¶

Focus: "Why does my sharded counter scale 4×, not 16×? False sharing, cache-line padding, per-CPU shards, sloppy counters, and a Go analog of Java's LongAdder."

By the middle level you have learned that a single atomic.Int64.Add(1) becomes a bottleneck under heavy contention and that "shard it across N atomics" is the natural fix. You measured your sharded counter on a 16-core machine, expected a 16× speedup, and got 3.8×. This file is about that gap.

The 3.8× ceiling has two causes:

1. False sharing. Your N atomic counters are packed into a single array [N]atomic.Int64, which means several of them share each 64-byte CPU cache line. When two cores write to "different" shards, the cache line still bounces between them — they are not really independent.
2. Suboptimal shard selection. If the shard key is random per call, you lose locality and hop between shards on every write. If the shard key is per-goroutine but uniformly random, you can still get hot spots when traffic shape is skewed.

The fix to (1) is cache-line padding. The fix to (2) is per-P sharding (one shard per Go runtime processor, accessed via runtime_procPin).

Beyond those two, the senior toolkit includes:

• Sloppy counters — per-goroutine local accumulators that flush periodically to a global; trade freshness for throughput.
• LongAdder-style auto-growing sharding — Java's class that grows its cell array under contention, so you do not have to pick the right N upfront.
• Counter Reset semantics — how to atomically read-and-clear N shards.
• Multi-counter snapshots — coordinating reads of related sharded counters.

You will leave this file able to write a counter that scales linearly to as many cores as you have, with an understanding deep enough to diagnose mysteries like "throughput drops at exactly N cores" and "this shard is hotter than others by 10×".

Prerequisites¶

• Required: Middle-level fluency: CAS loops, atomic.Pointer[T], expvar, basic sharded counters.
• Required: Familiarity with CPU caches at a conceptual level. You should know what L1/L2/L3 are, what a cache line is (64 bytes on x86-64), and what cache coherence means at the "many cores touch the same line → it ping-pongs" level.
• Required: Ability to read Go assembly and run go tool pprof, go tool trace, and perf.
• Helpful: Some exposure to the Go runtime — what a P is in GMP, where the scheduler lives, what runtime.lockOSThread does. The senior-level per-CPU shard pattern uses runtime-private API; we will look at how it is exposed in user code.
• Helpful: Awareness of Java's LongAdder and Striped64. The design idea — dynamic, contention-driven sharding — translates directly.

Glossary¶

Core Concepts¶

False sharing in detail¶

When you write [N]atomic.Int64, the Go runtime allocates N contiguous 8-byte values. On a system with 64-byte cache lines, eight int64s share one cache line. If goroutines on different cores write to indices 0 and 1 (or any pair within the same 8-element block), the cache line bounces between cores on every write — even though logically these are "different" shards.

Concretely, the MESI protocol works like this:

1. Core A reads shard 0. The cache line containing shards 0..7 is loaded into A's L1 cache in Shared state.
2. Core A writes to shard 0. The line transitions to Modified; B's copy (if any) is Invalidated.
3. Core B wants to write to shard 1. The line is Invalid in B's cache. B sends a coherence request; A flushes its line to L2/L3; B loads the line in Exclusive state.
4. Core B writes to shard 1. The line is now Modified in B's cache; A's copy is Invalidated.
5. Core A wants to write to shard 0 again — repeat from step 3.

Each transition costs ~50–200 ns (cache-line transfer between cores). If two cores hammer shards 0 and 1, every write costs the price of a coherence round-trip, completely defeating the purpose of sharding.

The fix is cache-line padding: ensure each atomic sits on its own cache line.

Cache-line padding patterns¶

Three idiomatic ways to pad in Go:

Pattern 1: explicit byte padding

type PaddedCounter struct {
    v   atomic.Int64
    _   [56]byte // pad to 64-byte cache line
}

type Sharded struct {
    cells [N]PaddedCounter
}

Each PaddedCounter is exactly 64 bytes. Adjacent cells in the array no longer share a line.

Pattern 2: golang.org/x/sys/cpu.CacheLinePad

import "golang.org/x/sys/cpu"

type PaddedCounter struct {
    _ cpu.CacheLinePad
    v atomic.Int64
    _ cpu.CacheLinePad
}

cpu.CacheLinePad is [CacheLinePadSize]byte, where CacheLinePadSize is set per-architecture (typically 64; 128 on POWER). Future-proof against architectures with larger cache lines. The extra padding before and after isolates the counter from neighbours in either direction.

Pattern 3: alignment-based padding using struct layout

type PaddedCounter struct {
    pad0 [8]uint64
    v    atomic.Int64
    pad1 [7]uint64
}

Less common; explicit-byte version is preferred for readability.

The padding adds memory: 56 bytes per shard. For 256 shards on a 16-core box, that is 16 KB. Trivial.

Per-P sharding via runtime_procPin¶

The runtime's procPin/procUnpin functions pin the calling goroutine to a specific P and return its index. This lets you write a sharded counter where each P always uses the same shard:

//go:linkname runtime_procPin runtime.procPin
func runtime_procPin() int

//go:linkname runtime_procUnpin runtime.procUnpin
func runtime_procUnpin()

type PerPCounter struct {
    cells []PaddedCounter
}

func New() *PerPCounter {
    return &PerPCounter{cells: make([]PaddedCounter, runtime.GOMAXPROCS(0))}
}

func (c *PerPCounter) Inc() {
    p := runtime_procPin()
    c.cells[p].v.Add(1)
    runtime_procUnpin()
}

func (c *PerPCounter) Get() int64 {
    var total int64
    for i := range c.cells {
        total += c.cells[i].v.Load()
    }
    return total
}

Properties:

• Each P has its own shard. While a goroutine runs on P3, all its writes go to shard 3.
• Because Ps run on at most one OS thread at a time, two writes to shard P3 from "different goroutines" still happen on the same thread — no contention on that cache line.
• Add itself remains atomic (you could in principle drop it to a non-atomic write because there is no cross-thread contention, but the reader still needs an atomic Load; cleaner to keep it symmetric).
• The number of shards is GOMAXPROCS, which is runtime.NumCPU() by default. Perfect match for the workload.

Caveats:

• runtime_procPin is private runtime API. The //go:linkname directive accesses it. The Go team has historically maintained this function and the linkname trick works, but you are off the supported path.
• Calling code may not block / yield while pinned. You can only do "small, fast" operations between procPin and procUnpin.
• GOMAXPROCS may change at runtime; if it grows, you have not allocated cells for the new Ps. Either lock GOMAXPROCS, or oversize the cell array, or detect and resize.
• Each Inc costs an extra atomic-ish operation (the pin itself is cheap — a couple of cycles).

This is the Go analog of Java's LongAdder for the "always pin to a known small N" case. For dynamic N, see the next section.

LongAdder-style auto-growing sharding¶

Java's LongAdder solves "pick the right shard count" by growing the array dynamically. The state machine:

1. Start with a single atomic base.
2. On add(delta), try to CAS base + delta into base.
3. If the CAS fails (contention detected), allocate or grow a Cell[] and have the contending threads write to cells instead.
4. Each thread is assigned a "probe" — a pseudo-random thread-local hash. The probe picks a cell.
5. If the probe-targeted cell is contended, re-hash the probe and try again. If still contended, grow the cell array.
6. Sum() reads base + sum of all cells.

The Go translation (sketched):

type LongAdder struct {
    base  atomic.Int64
    cells atomic.Pointer[[]Cell]
    busy  atomic.Int32 // CAS lock for growing
}

type Cell struct {
    v atomic.Int64
    _ [56]byte // padding
}

func (a *LongAdder) Add(delta int64) {
    cells := a.cells.Load()
    if cells == nil {
        // Try the simple base CAS first.
        if a.base.CompareAndSwap(a.base.Load(), a.base.Load()+delta) {
            return
        }
        // Contention; install cells (with locking).
        a.allocateCells()
        cells = a.cells.Load()
    }
    probe := getThreadProbe()
    idx := probe % uint32(len(*cells))
    if !(*cells)[idx].v.CompareAndSwap((*cells)[idx].v.Load(), (*cells)[idx].v.Load()+delta) {
        // Contention on this cell; either rehash probe or grow cells.
        a.handleContention(probe)
        a.Add(delta) // retry
    }
}

func (a *LongAdder) Sum() int64 {
    total := a.base.Load()
    if cells := a.cells.Load(); cells != nil {
        for i := range *cells {
            total += (*cells)[i].v.Load()
        }
    }
    return total
}

In practice this is complex enough that you should either use a community library (search "go longadder") or write it carefully with tests. The senior takeaway is: dynamic sharding exists, it solves the "what N to pick" problem, and the implementation cost is moderate.

For most Go workloads, a fixed per-P sharded counter (N = GOMAXPROCS) is simpler and nearly as good.

Sloppy counters¶

The "sloppy counter" comes from the Tornado operating system and is used in the Linux kernel for many statistics. The idea:

• Each thread (or goroutine in Go) maintains a private counter, incremented without any synchronisation.
• When the local counter exceeds a threshold, it is flushed atomically to a global counter.
• Reads of the global counter return a value that lags reality by up to threshold * numThreads.

type Sloppy struct {
    global atomic.Int64
}

type Local struct {
    n     int64
    flush int64
    parent *Sloppy
}

func (s *Sloppy) Local(threshold int64) *Local {
    return &Local{flush: threshold, parent: s}
}

func (l *Local) Inc() {
    l.n++
    if l.n >= l.flush {
        l.parent.global.Add(l.n)
        l.n = 0
    }
}

func (l *Local) Flush() {
    if l.n > 0 {
        l.parent.global.Add(l.n)
        l.n = 0
    }
}

func (s *Sloppy) Get() int64 {
    return s.global.Load()
}

Per-goroutine Local is not concurrent-safe; each goroutine must have its own. Pattern:

func worker(s *Sloppy) {
    local := s.Local(1024)
    defer local.Flush()
    for job := range jobs {
        local.Inc()
        process(job)
    }
}

Tradeoffs:

• Throughput: vastly higher than even sharded atomic — each Inc is just l.n++, no atomic.
• Freshness: lags by up to threshold * numGoroutines. For a kernel that flushes every 1024 events, with 16 cores, the lag is ~16K events. For metrics, almost always fine.
• Crash safety: un-flushed increments are lost on panic. Use defer Flush().
• Memory: one Local struct per goroutine. Cheap.

Sloppy counters are the right answer when:

• You have many goroutines making many small increments
• Exact value is not needed at any given moment
• Some loss on crash is acceptable

They are wrong when:

• You need exact counts for billing or auditing
• Goroutines are short-lived (the Local cost dominates)
• You need millisecond-fresh values

Counter reset semantics¶

For a sharded counter, "reset to zero" is not a single atomic operation. You must Swap(0) each shard in turn:

func (s *Sharded) Reset() int64 {
    var total int64
    for i := range s.cells {
        total += s.cells[i].v.Swap(0)
    }
    return total
}

Properties:

• Writes that happen during Reset may land in either the old (about-to-be-zeroed) shard or the just-zeroed shard.
• If you reset shard 0 first, an increment to shard 0 immediately after is preserved; an increment to shard 1 happening "before" the reset of shard 1 is counted in the return value.
• Total preservation: every increment is counted exactly once across consecutive Resets. The boundary is fuzzy in time but not in count.

For metrics, this fuzziness is acceptable. For billing, you would need a generation number scheme.

Multi-counter snapshot¶

If you have several related sharded counters (requests, errors, inflight) and want a coherent snapshot — all three values "at the same instant" — sharded atomics give you no guarantees. The standard fixes:

1. Acceptance. Decide that metrics snapshots do not need to be coherent. They almost never do.
2. Generation-stamp. Bump a generation counter before and after each batch of increments; readers retry if the generation changes mid-read. Seqlock-style.
3. atomic.Pointer[Snapshot]. A publisher thread periodically reads all counters into an immutable snapshot and atomically swaps the pointer. Readers see a coherent view at some bounded staleness.

Option 3 is the practical choice. Bound the staleness to your scrape interval (15s for Prometheus) and the publisher cost amortises to near-zero.

Real-World Analogies¶

False sharing as a shared whiteboard¶

Imagine a whiteboard divided into 8 boxes; 8 employees each "own" one box and write tallies in them. The whiteboard hangs in a room with one door; only one person at a time may enter, and entering takes 100 ms. Two employees who want to write in different boxes still queue at the door — they fight for the room, not the boxes.

Padding gives each employee their own room. Now they can all write simultaneously.

Per-P shards as a chef per station¶

A restaurant with 8 stations, each chef tied to a station, each station with its own ingredients. Chef 3 always uses ingredients on station 3; chef 7 always uses station 7. No fighting over a shared pantry. Reads (the manager wanting to tally end-of-shift inventory) walk all 8 stations.

Sloppy counter as a tip jar that empties into a vault¶

Bartenders drop tips into individual jars. Once a jar fills to $50, it is poured into the central vault. The total tips ever earned (vault content) lags reality by up to (jar capacity × jar count). Bartenders never fight over the vault; the vault accountant reads the vault only at end of shift.

LongAdder as a hotel adding desks during a rush¶

A hotel front desk has one receptionist. When the queue grows, a second desk opens. When it grows more, a third. When demand quiets, desks close. Threads "self-assign" to whichever desk is least busy. The total guest count is sum of all desks plus a "checked in earlier" baseline.

Mental Models¶

"Cache line is the unit of contention"¶

When you think about scaling, do not think "two cores writing to two atomics" — think "two cores writing to the same cache line". Different atomics, same line, same contention.

"Pad to 64, accept the memory cost"¶

For high-contention atomics, always pad. Memory is cheap; cache-line ping-pongs are expensive.

"Per-P is the right answer when the runtime cooperates"¶

Where runtime_procPin is acceptable (most server code), per-P shards give you essentially-linear scaling. The only constraint: small, non-blocking operations between pin and unpin.

"Sloppy is the right answer when you can spare freshness"¶

The CPU price of an atomic add is dominated by the cache traffic. A purely local increment with periodic flush costs ~1 ns. The trade is staleness, which for monitoring is almost always acceptable.

"Dynamic sharding is the right answer when you cannot predict load"¶

LongAdder is the move when the contention shape changes over time and you cannot pick a fixed N upfront. For most Go services, fixed per-P is enough.

Pros & Cons¶

Cache-line padding¶

Pros - Eliminates false sharing - One-time, mechanical fix - Cheap (a few KB extra memory)

Cons - Boilerplate (_ [56]byte) - Requires knowing your architecture's line size - Easy to forget when reshaping structs

Per-P sharded counters¶

Pros - Effectively linear scaling up to GOMAXPROCS - Each shard is uncontended in steady state - Small read cost (O(P))

Cons - Relies on runtime_procPin (private API) - Cannot block while pinned - GOMAXPROCS changes require care - Each P has its own cache line for the shard, consuming memory

Sloppy counters¶

Pros - Highest throughput of any pattern - Each increment is a single non-atomic memory write - Scales perfectly because there is no contention

Cons - Lossy on crash (un-flushed local data) - Stale by up to threshold × goroutines - Per-goroutine local storage required - Wrong for exact counting

LongAdder (dynamic sharded)¶

Pros - No need to pick N upfront - Adapts to actual contention - Excellent under bursty load

Cons - Complex implementation - Higher read cost (sum over a dynamically-sized array) - Memory grows under contention, may not shrink - Not in Go standard library

Use Cases¶

• Service-wide request counter at 1M+ RPS: per-P shards with padding
• In-flight gauge: padded atomic.Int64 (only one variable, no sharding needed; padding still helps if it shares a line with other hot atomics)
• Bytes processed per pipeline stage: sloppy counters with per-worker flush
• Per-route HTTP counter at high cardinality: dynamic (LongAdder-style) or per-P with hash of route
• Billing-grade counter: not in-memory; use a database. In-memory sharded with persistence checkpoint is acceptable for some applications.
• GC pause counter (within Go runtime itself): atomic; called rarely enough that no sharding needed.
• Cache hit/miss counter: padded sharded; reads are infrequent.

Code Examples¶

Padded sharded counter (production-quality)¶

package counters

import (
    "math/rand/v2"
    "sync/atomic"

    "golang.org/x/sys/cpu"
)

type cell struct {
    _ cpu.CacheLinePad
    v atomic.Int64
    _ cpu.CacheLinePad
}

type Sharded struct {
    cells []cell
    mask  uint64
}

// New creates a sharded counter with shards rounded up to the next
// power of 2 at least equal to numShards.
func New(numShards int) *Sharded {
    n := 1
    for n < numShards {
        n <<= 1
    }
    return &Sharded{cells: make([]cell, n), mask: uint64(n - 1)}
}

func (s *Sharded) Inc() {
    s.cells[rand.Uint64()&s.mask].v.Add(1)
}

func (s *Sharded) Add(delta int64) {
    s.cells[rand.Uint64()&s.mask].v.Add(delta)
}

func (s *Sharded) Get() int64 {
    var total int64
    for i := range s.cells {
        total += s.cells[i].v.Load()
    }
    return total
}

func (s *Sharded) Reset() int64 {
    var total int64
    for i := range s.cells {
        total += s.cells[i].v.Swap(0)
    }
    return total
}

Notes:

• Power-of-2 sizes let us replace % with bitwise & — slightly faster.
• rand.Uint64() from math/rand/v2 is per-goroutine internally; no contention.
• Each cell is padded on both sides; no two cells share a line.
• Memory: each cell is at least 128 bytes (pad + atomic + pad); 64 shards = 8 KB. Negligible.

Benchmark scaling on a 16-core machine, ops/sec:

The naive sharded plateaus due to false sharing. The padded sharded scales near-linearly.

Per-P sharded counter¶

package counters

import (
    "runtime"
    "sync/atomic"
    _ "unsafe" // for go:linkname

    "golang.org/x/sys/cpu"
)

//go:linkname runtime_procPin runtime.procPin
func runtime_procPin() int

//go:linkname runtime_procUnpin runtime.procUnpin
func runtime_procUnpin()

type cellP struct {
    _ cpu.CacheLinePad
    v atomic.Int64
    _ cpu.CacheLinePad
}

type PerP struct {
    cells []cellP
}

// NewPerP creates a per-P counter sized for the current GOMAXPROCS.
// Changing GOMAXPROCS after this call may use too few cells; in that
// case extra writes wrap to existing cells and contention may slightly
// increase.
func NewPerP() *PerP {
    n := runtime.GOMAXPROCS(0)
    if n < 1 {
        n = 1
    }
    return &PerP{cells: make([]cellP, n)}
}

func (c *PerP) Inc() {
    p := runtime_procPin()
    c.cells[p%len(c.cells)].v.Add(1)
    runtime_procUnpin()
}

func (c *PerP) Add(delta int64) {
    p := runtime_procPin()
    c.cells[p%len(c.cells)].v.Add(delta)
    runtime_procUnpin()
}

func (c *PerP) Get() int64 {
    var total int64
    for i := range c.cells {
        total += c.cells[i].v.Load()
    }
    return total
}

Notes:

• We use //go:linkname to access the unexported runtime.procPin. This is documented but not officially blessed; the Go team has not removed it because too many libraries (including sync.Pool) rely on it.
• The compiler needs import _ "unsafe" for the linkname directive to work.
• Between procPin and procUnpin you must not block, allocate heavy, or call user code that might. The atomic add is safe.
• If GOMAXPROCS grows after NewPerP, the modulo wrap will reintroduce some contention. For most services this is acceptable.

Sloppy counter¶

package counters

import "sync/atomic"

type Sloppy struct {
    global atomic.Int64
}

type Local struct {
    parent  *Sloppy
    n       int64
    flushAt int64
}

func (s *Sloppy) Local(flushAt int64) *Local {
    return &Local{parent: s, flushAt: flushAt}
}

func (l *Local) Inc() {
    l.n++
    if l.n >= l.flushAt {
        l.parent.global.Add(l.n)
        l.n = 0
    }
}

func (l *Local) Add(delta int64) {
    l.n += delta
    if l.n >= l.flushAt {
        l.parent.global.Add(l.n)
        l.n = 0
    }
}

func (l *Local) Flush() {
    if l.n > 0 {
        l.parent.global.Add(l.n)
        l.n = 0
    }
}

func (s *Sloppy) Get() int64 {
    return s.global.Load()
}

Usage:

func worker(s *Sloppy) {
    local := s.Local(1024)
    defer local.Flush()
    for j := range jobs {
        local.Inc()
        process(j)
    }
}

Read at end-of-process or periodic snapshot. Lag is bounded by flushAt * activeWorkers.

Tiny LongAdder analog¶

package counters

import (
    "sync"
    "sync/atomic"
    "unsafe"

    "golang.org/x/sys/cpu"
)

const initialCells = 4

type adderCell struct {
    _ cpu.CacheLinePad
    v atomic.Int64
    _ cpu.CacheLinePad
}

type LongAdder struct {
    base   atomic.Int64
    cellsP atomic.Pointer[[]adderCell]
    mu     sync.Mutex
}

func (a *LongAdder) Add(delta int64) {
    cellsPtr := a.cellsP.Load()
    if cellsPtr == nil {
        // Try the base first.
        cur := a.base.Load()
        if a.base.CompareAndSwap(cur, cur+delta) {
            return
        }
        // Contention. Install cells.
        a.installCells()
        cellsPtr = a.cellsP.Load()
    }
    cells := *cellsPtr
    probe := threadProbe()
    idx := probe % uint32(len(cells))
    cell := &cells[idx]
    cur := cell.v.Load()
    if cell.v.CompareAndSwap(cur, cur+delta) {
        return
    }
    // Cell contended; grow or rehash.
    a.handleContention(delta, probe, cellsPtr)
}

func (a *LongAdder) Sum() int64 {
    total := a.base.Load()
    if cellsPtr := a.cellsP.Load(); cellsPtr != nil {
        for i := range *cellsPtr {
            total += (*cellsPtr)[i].v.Load()
        }
    }
    return total
}

func (a *LongAdder) installCells() {
    a.mu.Lock()
    defer a.mu.Unlock()
    if a.cellsP.Load() != nil {
        return
    }
    cells := make([]adderCell, initialCells)
    a.cellsP.Store(&cells)
}

func (a *LongAdder) handleContention(delta int64, probe uint32, oldCells *[]adderCell) {
    // Simplified: just CAS-loop on the cell with rehashed probe.
    // Real LongAdder would grow the cell array if contention persists.
    a.mu.Lock()
    cells := *a.cellsP.Load()
    if len(cells) == len(*oldCells) && len(cells) < 1024 {
        newCells := make([]adderCell, len(cells)*2)
        // Migrate existing values into new cells (not strictly required;
        // could simply enlarge with zero-init and let new traffic land there).
        for i := range cells {
            newCells[i].v.Store(cells[i].v.Load())
        }
        a.cellsP.Store(&newCells)
    }
    a.mu.Unlock()
    // Retry on a (possibly larger) cell array.
    a.Add(delta)
}

// threadProbe returns a per-goroutine pseudorandom uint32.
// Real LongAdder uses a per-thread hash that mutates on contention.
func threadProbe() uint32 {
    var x int
    return uint32(uintptr(unsafe.Pointer(&x)))
}

Real LongAdder is several hundred lines. This is a starting sketch. The salient ideas:

• Try base first; cells only on contention.
• Cells start small, grow under sustained contention.
• Per-thread probe selects a cell; probe rehashes on contention.
• Read is base + sum of all cells.

For production, use a community port. The point of writing your own is understanding.

Coding Patterns¶

Pattern: pad-then-pack¶

When you have several small atomics that together form one logical unit and are accessed together, pack them in one struct on one line:

type Stats struct {
    Requests atomic.Int64
    Errors   atomic.Int64
    Inflight atomic.Int64
    _        [40]byte // pad whole struct to next cache line
}

Multiple Stats instances in an array do not contend with each other. Inside one Stats, all three counters share a line — which is fine because they are typically incremented together.

Pattern: per-P shard with fallback¶

func (c *Counter) Inc() {
    if c.cells != nil {
        p := runtime_procPin()
        c.cells[p%len(c.cells)].v.Add(1)
        runtime_procUnpin()
        return
    }
    c.base.Add(1)
}

Cells are only installed under contention (LongAdder-style). Cheap fast path.

Pattern: bulk flush¶

If your goroutine is about to exit and has accumulated local counts:

defer local.Flush()

Always pair the Local constructor with the Flush defer.

Pattern: snapshot-publisher loop¶

go func() {
    t := time.NewTicker(time.Second)
    defer t.Stop()
    for {
        select {
        case <-stop:
            return
        case <-t.C:
            snap := newSnapshot(
                counter1.Get(),
                counter2.Get(),
                counter3.Get(),
            )
            publishedSnap.Store(snap)
        }
    }
}()

One publisher, many readers. Readers do one Load.

Pattern: read-mostly + occasional reset¶

func periodic() {
    for range ticker.C {
        n := c.Reset()
        sink(n)
    }
}

Reset is O(shards). Acceptable at second granularity.

Clean Code¶

Document the contention model¶

Every counter struct should have a one-line comment on its concurrency story.

// Sharded is a counter sharded across NumCPU cells, each padded to a
// cache line. Inc is safe for concurrent use from any number of
// goroutines; Get is O(shards) and intended for infrequent reads.
type Sharded struct { ... }

Encapsulate the padding¶

Do not let _ cpu.CacheLinePad proliferate across your codebase. Wrap padded atomics in named types:

type PaddedInt64 struct {
    _ cpu.CacheLinePad
    V atomic.Int64
    _ cpu.CacheLinePad
}

Now consumers reach for PaddedInt64 as a single concept.

Hide runtime_procPin behind an interface¶

procPin is dangerous in user code (you must not block while pinned). Wrap it:

// withP runs fn while pinned to a single P. fn must be brief.
func withP(fn func(p int)) {
    p := runtime_procPin()
    fn(p)
    runtime_procUnpin()
}

Now consumers cannot accidentally block while pinned, because the body is a clear lambda.

Pick one paradigm per project¶

Mixing per-P, sloppy, and dynamic-sharded counters in the same codebase is confusing. Pick one based on your dominant workload and use it everywhere.

Architecture & Design¶

When to introduce sharding¶

Default to a single atomic.Int64. Introduce sharding only when:

1. Profiling shows hot time in atomic add for this counter
2. Throughput is plateauing as you add cores
3. The counter is hit at high rate (>1M ops/sec across all cores)

For most counters, even in high-RPS services, a single atomic.Int64 is fine. Sharding adds memory and complexity for no benefit when the contention is low.

When to introduce padding¶

Always when the atomic is high-rate enough to matter. The cost is bytes; the benefit is real.

When to introduce per-P¶

When the runtime is your friend (you control GOMAXPROCS, you don't run on weird platforms) and the atomic add is in your hot path. The Go runtime itself uses per-P for many of its counters.

When to introduce sloppy¶

When exact freshness is unimportant and throughput is critical. Internal "we processed N bytes" meters for log shippers are the canonical use case.

When to introduce LongAdder¶

When the contention shape changes over time and you cannot predict the right shard count. For most Go services, a fixed-N per-P shard is simpler and adequate.

Coordinating multiple counters¶

If you have related counters (requests, errors, inflight), do not coordinate them on the increment path. Coordinate them at the snapshot boundary using atomic.Pointer[Snapshot]. The snapshot publisher reads each counter, packages them, and publishes.

Multi-shard reset¶

If you need "reset all sharded counters at the same moment", you cannot. The best approximation:

1. Snapshot all counters into a single struct (via Pointer[T]).
2. Reset each shard.
3. Accept that increments between (1) and (2) are counted in the snapshot but also in the next interval. Or vice versa. The total is preserved across consecutive intervals.

For monitoring, this fuzziness is acceptable. For accounting, use a different design (e.g., generation numbers).

Error Handling¶

The same as middle level: atomic operations do not fail; CAS "failure" is part of normal operation. The senior-level additions:

• Per-P shard mismatch on GOMAXPROCS change. Add will modulo into a smaller-than-expected array; no panic, but contention reappears. Detect at startup and warn.
• Sloppy counter overflow. int64 overflow is impractical, but int32 local accumulators can overflow if flushAt is set too high.
• LongAdder cell array exhaustion. Cap the cell array growth to prevent runaway memory under pathological contention.

Security Considerations¶

• runtime_procPin is private API. A future Go version could rename or remove it. Pin Go versions in CI, run integration tests on new Go versions before adopting.
• Sloppy counters are crash-lossy. Do not use them for billing or auditing.
• Per-P shards reveal CPU count to attackers reading metrics output. Not always a leak, but worth noting.
• Counter padding wastes cache. In a memory-constrained environment, the extra cache lines hurt other workloads' caching. Measure.

Performance Engineering¶

Methodology¶

1. Benchmark first. A counter that is not in your profile is not worth optimising.
2. Measure scaling. go test -bench=. -cpu=1,2,4,8,16,32. The shape of the curve tells you the contention story.
3. Look at perf top. If LOCK XADD or atomic.AddInt64 is in the top entries during your workload, it is a hot atomic.
4. Look at perf c2c (Linux). Detects false sharing directly. Hugely valuable.
5. Inspect cache-line layout. Use unsafe.Sizeof and unsafe.Offsetof to verify your padding.

Common diagnoses¶

• "Throughput plateaus at N cores." Cache-line contention. Pad or shard.
• "Adding shards gives sublinear speedup." False sharing — pad.
• "Per-P shards still slow." Check GOMAXPROCS; check that the workload actually uses all Ps.
• "Sloppy counter shows stale values." Increase flush frequency or add a periodic flush from a coordinator goroutine.
• "Memory grew under load." LongAdder-style growth without shrinkage; bound the cell array.

Tools¶

• go test -bench=. -benchmem -cpu=1,2,4,8,16,32 — scaling curves
• go tool pprof -http=:8080 cpu.prof — CPU profile
• go tool trace trace.out — scheduler view
• perf top, perf c2c — Linux-specific, deep cache analysis
• cachegrind, valgrind --tool=cachegrind — cache simulation

Anti-pattern: padding everything¶

Padding a low-traffic atomic wastes memory for no benefit. Pad atomics that profile as hot; leave others alone.

Anti-pattern: too many shards¶

256 shards on a 4-core machine is silly — most shards are never touched, and the read path is unnecessarily expensive. Size shards to ~2-4× cores.

Anti-pattern: sloppy counter where exact is needed¶

Sloppy counters are seductive (much faster!) but lossy. Never use them for billing, auditing, or critical business state.

Best Practices¶

• Default to a single padded atomic.Int64. Add sharding only when measurement shows contention.
• Always pad high-contention atomics. Use cpu.CacheLinePad for portability.
• Prefer per-P shards over random shards when procPin is available and acceptable.
• Use sloppy counters for high-throughput, loss-tolerant counts only.
• Snapshot multi-counter state via atomic.Pointer[Snapshot], not by reading each counter separately.
• Document the contention model in the type's comment.
• Benchmark scaling with -cpu for every new high-traffic counter.
• Cap dynamic-sharded growth to prevent runaway memory.

Edge Cases & Pitfalls¶

GOMAXPROCS changes at runtime¶

If your per-P counter was sized at startup for 16 Ps and runtime later sets GOMAXPROCS to 32, half the Ps will share shards. Either lock GOMAXPROCS or oversize.

procPin while blocking¶

You must not call time.Sleep, allocate large objects, or call user code (which might block) while pinned. The atomic add is safe; everything else is forbidden.

Sloppy counter where flush goroutine dies¶

If your goroutine panics, defer Flush() runs and the local data is preserved. If your goroutine hangs forever, the local data is stranded. Add periodic flushers as a safety net.

LongAdder cells leaked on growth¶

Old cell arrays remain referenced until GC; under heavy growth, you can briefly retain N cell arrays. Bound the growth and trigger GC if it matters.

Sharded counter with non-power-of-2 size¶

shards[key % 100] is slower than shards[key & 63]. Round shard count up to the next power of 2.

False sharing across struct boundaries¶

type S struct {
    a atomic.Int64
    b atomic.Int64
}
var instances [10]S

instances[0].b and instances[1].a may share a cache line. Pad S to a cache line if both are hot.

cpu.CacheLinePad is per-architecture¶

On Apple Silicon, cpu.CacheLinePadSize is 128 (some M-series chips have 128-byte lines). Padding to 64 is not enough. Always use cpu.CacheLinePad, not a hand-rolled _ [56]byte.

runtime_procPin on Wasm/JS¶

Wasm and JS targets do not implement procPin in the same way; behaviour may degrade. Verify on your target platforms.

Reading sharded counters from inside a hot loop¶

Per-write Get() is O(shards). For 64 shards, that is 200+ ns — many times the cost of the increment. Cache the result locally if you use it many times.

Counter Reset racing with snapshot¶

If you Reset() while a publisher is computing a snapshot, the snapshot may include partial-reset values. Coordinate reset with snapshot windows.

Common Mistakes¶

1. Padding the array, not the element. [N]atomic.Int64 with a _ [56]byte after the array still has 8 cells per line.
2. Forgetting cpu.CacheLinePad is per-architecture. Hardcoding 56 bytes assumes 64-byte lines.
3. Calling user code while procPinned. Panic if the runtime preempts; subtle if not.
4. Sloppy counter without defer Flush(). Lost data on panic.
5. LongAdder without growth cap. Memory explodes under pathological contention.
6. Resetting a sharded counter under the assumption that all shards are zero at the same instant. They never are.
7. Snapshotting by reading each counter separately and assuming consistency. Use atomic.Pointer[T].
8. Premature sharding. Single atomic is fine for the vast majority of counters.

Common Misconceptions¶

• "Padding is just for fun." No — false sharing is a measurable, dominant cost at high contention.
• "More shards is always better." Past 2-4× cores, additional shards waste memory and slow reads.
• "Per-P is the same as per-core." Almost, but not exactly: a P is bound to an OS thread while it runs Go code, but the OS thread itself can migrate between cores. NUMA effects are still possible.
• "LongAdder is always faster than a sharded counter." Under low contention, the dynamic machinery is overhead. Fixed sharded is faster for sustained moderate load.
• "Sloppy counters lose data." They lose staleness; in steady state the count is correct. They lose unfflushed deltas on crash.

Tricky Points¶

Per-P shard sees writes from other goroutines on the same P¶

When the scheduler moves another goroutine onto P3, that goroutine's Inc also goes to cell 3. So "per-P" does not mean "per-goroutine" — but because only one goroutine runs on P3 at a time, there is no cross-thread contention on cell 3's cache line.

procPin is not just LockOSThread¶

procPin is faster and lighter than runtime.LockOSThread. It prevents preemption during the pinned section but does not bind to a specific OS thread the way LockOSThread does. For atomic-add workloads, procPin is the right tool.

Sum of shards is not atomic¶

Get() walks shards and reads each one. By the time you reach shard 63, shards 0..62 may have been further incremented. The returned value is "monotonically increasing in time" but is not "the value at any single instant".

LongAdder.Sum() returns a sloppy answer too¶

Java's LongAdder.sum() is documented as "an estimate". Same caveat as Go's sharded Get().

Padding cost compounds with shard count¶

64 shards × 128 bytes (with both-side padding) = 8 KB per counter. For one counter, trivial. For 100 counters, 800 KB. For a large metrics namespace, this adds up. Consider sharing padding across logically-related counters.

Test¶

package counters

import (
    "runtime"
    "sync"
    "sync/atomic"
    "testing"
    "unsafe"

    "golang.org/x/sys/cpu"
)

func TestCellAlignment(t *testing.T) {
    var s Sharded = *New(4)
    a0 := uintptr(unsafe.Pointer(&s.cells[0].v))
    a1 := uintptr(unsafe.Pointer(&s.cells[1].v))
    diff := a1 - a0
    if diff < uintptr(cpu.CacheLinePadSize) {
        t.Errorf("cells too close: %d bytes apart", diff)
    }
}

func TestSharded_Correct(t *testing.T) {
    s := New(64)
    const N = 100000
    var wg sync.WaitGroup
    for i := 0; i < N; i++ {
        wg.Add(1)
        go func() { defer wg.Done(); s.Inc() }()
    }
    wg.Wait()
    if got := s.Get(); got != N {
        t.Errorf("expected %d, got %d", N, got)
    }
}

func TestPerP_Correct(t *testing.T) {
    p := NewPerP()
    const N = 100000
    var wg sync.WaitGroup
    for i := 0; i < N; i++ {
        wg.Add(1)
        go func() { defer wg.Done(); p.Inc() }()
    }
    wg.Wait()
    if got := p.Get(); got != N {
        t.Errorf("expected %d, got %d", N, got)
    }
}

func TestSloppy_Correct(t *testing.T) {
    var s Sloppy
    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            local := s.Local(100)
            defer local.Flush()
            for j := 0; j < 1000; j++ {
                local.Inc()
            }
        }()
    }
    wg.Wait()
    if got := s.Get(); got != 100*1000 {
        t.Errorf("expected %d, got %d", 100*1000, got)
    }
}

func BenchmarkSingle(b *testing.B) {
    var c atomic.Int64
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            c.Add(1)
        }
    })
}

func BenchmarkSharded(b *testing.B) {
    s := New(runtime.GOMAXPROCS(0) * 4)
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            s.Inc()
        }
    })
}

func BenchmarkPerP(b *testing.B) {
    p := NewPerP()
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            p.Inc()
        }
    })
}

func BenchmarkSloppy(b *testing.B) {
    var s Sloppy
    b.RunParallel(func(pb *testing.PB) {
        local := s.Local(1024)
        defer local.Flush()
        for pb.Next() {
            local.Inc()
        }
    })
}

Run all four at -cpu=1,4,16,32:

go test -bench=. -cpu=1,4,16,32 -benchtime=2s

Expected shape: Single's ops/sec falls as cores rise; Sharded scales until ~16 cores then plateaus; PerP scales linearly to GOMAXPROCS; Sloppy is fastest at every concurrency level.

Tricky Questions¶

Q: How can I tell if I have false sharing without perf c2c? A: Run a microbenchmark with N goroutines hammering N "different" atomics. If the throughput is much worse than N goroutines each hammering its own isolated atomic, you have false sharing. Adding padding and re-measuring confirms.

Q: What is the right shard count? A: 2-4× GOMAXPROCS. For a 16-core machine, 32-64 shards. Round to power of 2 to use bitwise mask.

Q: Why not always use per-P? A: It relies on private runtime API. For libraries that ship widely, you do not want that dependency. For application code at known Go versions, per-P is excellent.

Q: Does cpu.CacheLinePad work on ARM64? A: Yes. The package picks the right size per architecture.

Q: Is there overhead in procPin itself? A: A few cycles. Roughly the cost of an atomic add. Worth it when the alternative is cross-core cache traffic.

Q: Can I use sloppy counters with channels for the flush? A: Yes — a local.IncSendOnFlush(ch) pattern works. But the simple "add to global atomic" flush is usually fine.

Q: How do I Reset a sloppy counter? A: You need every Local to flush first, then s.global.Swap(0). Coordinating "every local flush" requires a barrier (a generation number on the global, locals check generation on Inc and flush themselves if it has bumped). Non-trivial.

Q: Does LongAdder shrink? A: Java's does not. Once grown, the cell array stays large. This is intentional — shrinking is expensive and the assumption is that contention recurs.

Q: What about CPU pinning at the OS level? A: Useful for NUMA-aware workloads. Combine with per-P shards: pin the OS thread, then use procPin for the shard index. Professional level.

Q: How does sync.Pool use this technique? A: sync.Pool has a per-P shard internally. It is the canonical example of procPin usage in the standard library.

Cheat Sheet¶

// Padded shard
type cell struct {
    _ cpu.CacheLinePad
    v atomic.Int64
    _ cpu.CacheLinePad
}

// Per-P pin
p := runtime_procPin()
cells[p].v.Add(1)
runtime_procUnpin()

// Sloppy
local := s.Local(1024)
defer local.Flush()
local.Inc()

// LongAdder-style
adder.Add(1)
n := adder.Sum()

// Shard count heuristic
shards := nextPowerOf2(4 * runtime.GOMAXPROCS(0))

Self-Assessment Checklist¶

• I can explain false sharing to a colleague
• I have measured the difference between padded and unpadded sharded counters
• I can write per-P sharded counters using runtime_procPin (or know why I shouldn't)
• I have implemented and tested a sloppy counter
• I can sketch a LongAdder and explain the trade-offs vs fixed sharded
• I can interpret a go test -bench -cpu=... scaling curve
• I know when not to shard

Summary¶

Senior-level concurrent counters are about making them scale. The progression is:

1. Single atomic — fine for most.
2. Padded single atomic — for moderate contention.
3. Padded sharded with random shard — for high contention without runtime access.
4. Per-P sharded — when runtime cooperation is acceptable.
5. Sloppy — when freshness is dispensable.
6. LongAdder-style dynamic — when contention shape is unpredictable.

Each level adds memory and code complexity; each pays back at higher contention. Profile before you climb the ladder; do not climb past your need.

The professional file completes the story with HDR histograms (because counters are not enough for latency), NUMA-aware shard placement, deep expvar/Prometheus integration, and the design of a full observability subsystem.

What You Can Build¶

• A padded sharded counter that scales near-linearly to 32 cores
• A per-P counter using runtime_procPin
• A sloppy counter with periodic flushing
• A LongAdder analog (basic version)
• A multi-counter coherent snapshot publisher
• Benchmarks proving scaling claims
• Diagnosis flows for "counter is hot in profile"

Further Reading¶

• Java's LongAdder / Striped64 source code (OpenJDK)
• Cliff Click, "A Lock-Free Hash Table" — for inspiration on dynamic sizing
• Linux kernel's percpu_counter.c — the textbook sloppy counter
• Doug Lea, "JSR-166 Concurrency Updates" — historical background
• Russ Cox, "Hardware Memory Models" series
• Ulrich Drepper, "What Every Programmer Should Know About Memory"
• golang.org/x/sys/cpu package source
• sync/atomic, sync.Pool, runtime source in the Go standard library

Related Topics¶

• Junior and middle counters (prerequisites)
• Professional counters (HDR histograms, expvar+Prometheus, NUMA)
• sync.Pool (uses per-P shards under the hood)
• The Go scheduler (GMP model)
• Lock-free data structures
• Cache coherence protocols (MESI, MOESI)
• NUMA architectures

Diagrams & Visual Aids¶

Cache line layout — naive vs padded¶

naive [4]atomic.Int64 (32 bytes in one cache line):
| c0 | c1 | c2 | c3 | c4 | c5 | c6 | c7 |    cells 0..7 share one line

padded cell (each in its own line):
| pad | c0 | pad | ... | pad | c1 | pad | ...
  ^---- 64 bytes ----^     ^---- 64 bytes ----^

Per-P assignment¶

P0 --+
P1 -|--> each writes only to its own cell
P2 -|    (no inter-P contention)
P3 --+

Sloppy counter flow¶

goroutine A: l.n++ ... l.n++ ... (l.n >= flush) -> global.Add(l.n); l.n=0
goroutine B: l.n++ ... l.n++ ... (l.n >= flush) -> global.Add(l.n); l.n=0
goroutine C: l.n++ ...                             (still local)

reader: global.Load() -> sees A's and B's flushed totals, missing C's local

LongAdder decision tree¶

Add(delta):
  cells is nil:
    try CAS base += delta
    succeeded: done
    failed: install cells, retry
  cells not nil:
    probe = thread-local hash
    cell = cells[probe % len(cells)]
    try CAS cell += delta
    succeeded: done
    failed: rehash probe; if persistent contention, grow cells

That is the senior-level concurrent counter toolkit. The professional file adds HDR histograms, NUMA, and full observability subsystem design.

Deep Dive: Cache Coherence in Detail¶

To design counters that scale, you must understand how the CPU keeps cache lines consistent across cores. The MESI protocol (and its variants MOESI, MESIF) is the foundation. The full state machine has four states per cache line per core:

• M (Modified) — this core has the line exclusively and has written to it; other cores' copies are stale.
• E (Exclusive) — this core has the only clean copy; nobody else has it cached.
• S (Shared) — multiple cores have read-only copies that match memory.
• I (Invalid) — this core does not have a valid copy.

Transitions:

• A read miss on a line currently in M state in another core forces that core to flush (writeback) and downgrade to S, while this core upgrades from I to S.
• A write to a line in S state forces all other holders to I (an invalidation), and this core upgrades to M.
• A read of a line that nobody else has becomes E.
• Writing to a line in E silently upgrades to M (no bus traffic).

Each transition involves an inter-core message. The latency is:

• L1 hit (uncontended): ~1 ns
• L2 hit: ~3-10 ns
• L3 hit (in this socket): ~20-50 ns
• Cross-socket cache transfer: 100-300 ns
• Memory: 50-100 ns local, 200+ ns remote (NUMA)

When two cores hammer the same cache line with writes, every write transitions the line through I → M and back. The line bounces between cores at L3-or-worse latency. This is why uncontended atomics cost ~10 ns and contended atomics cost ~200 ns each.

False sharing manifests as cache-line bouncing even though the logical values written by different cores are different. From the cache controller's perspective, "we wrote into this line" is the only granularity that matters. The fix — putting each hot value on its own line — is the only fix.

perf c2c for diagnosis¶

On Linux, perf c2c reports cache-line contention directly. Sample workflow:

$ perf c2c record -F 99 -- ./your_app
$ perf c2c report

The report shows which cache lines are bounced between cores and which functions touched them. False sharing jumps out as "two cache lines accessed by N cores, M HITMs each second".

Without perf c2c, you can still diagnose by:

1. Add padding.
2. Re-benchmark.
3. If throughput jumps, false sharing was your problem.

This is crude but works.

Deep Dive: Reading sync.Pool for procPin Patterns¶

sync.Pool is the canonical example of procPin in the standard library. Its internal structure:

type Pool struct {
    noCopy noCopy
    local     unsafe.Pointer // local fixed-size per-P pool, actual type is [P]poolLocal
    localSize uintptr        // size of the local array
    ...
}

type poolLocal struct {
    poolLocalInternal
    pad [128 - unsafe.Sizeof(poolLocalInternal{})%128]byte
}

func (p *Pool) pin() (*poolLocalInternal, int) {
    pid := runtime_procPin()
    s := runtime_LoadAcquintptr(&p.localSize)
    l := p.local
    if uintptr(pid) < s {
        return indexLocal(l, pid), pid
    }
    return p.pinSlow()
}

Things to notice:

• poolLocal is padded to 128 bytes — Apple Silicon has 128-byte lines, so 128 is the safe choice.
• The local array is sized to GOMAXPROCS.
• pin() returns both the local pointer and the P index. The pin must be held while you operate on the local.
• pinSlow handles GOMAXPROCS changes by reallocating.

Studying this teaches you a lot about how to write your own per-P infrastructure. The linkname-to-runtime trick, the padding to the largest known line size, the slow-path for resize — these are the patterns to copy.

Deep Dive: Writing a Production Per-P Counter¶

Let us write a per-P counter that handles all the edge cases:

package counters

import (
    "runtime"
    "sync"
    "sync/atomic"
    "unsafe"
    _ "unsafe" // for go:linkname

    "golang.org/x/sys/cpu"
)

//go:linkname runtime_procPin runtime.procPin
func runtime_procPin() int

//go:linkname runtime_procUnpin runtime.procUnpin
func runtime_procUnpin()

type pcell struct {
    _ cpu.CacheLinePad
    v atomic.Int64
    _ cpu.CacheLinePad
}

type PerP struct {
    mu       sync.Mutex
    cellsPtr atomic.Pointer[[]pcell]
}

// NewPerP creates a per-P counter sized for current GOMAXPROCS.
// If GOMAXPROCS later grows, the counter resizes itself transparently
// (at the cost of a one-time slow path for the next Inc).
func NewPerP() *PerP {
    p := &PerP{}
    p.resize(runtime.GOMAXPROCS(0))
    return p
}

func (p *PerP) resize(n int) {
    p.mu.Lock()
    defer p.mu.Unlock()
    old := p.cellsPtr.Load()
    if old != nil && len(*old) >= n {
        return
    }
    newCells := make([]pcell, n)
    if old != nil {
        for i := range *old {
            newCells[i].v.Store((*old)[i].v.Load())
        }
    }
    p.cellsPtr.Store(&newCells)
}

func (p *PerP) Inc() {
    pid := runtime_procPin()
    cells := *p.cellsPtr.Load()
    if pid < len(cells) {
        cells[pid].v.Add(1)
        runtime_procUnpin()
        return
    }
    runtime_procUnpin()
    // Slow path: GOMAXPROCS grew.
    p.resize(runtime.GOMAXPROCS(0))
    p.Inc()
}

func (p *PerP) Add(delta int64) {
    pid := runtime_procPin()
    cells := *p.cellsPtr.Load()
    if pid < len(cells) {
        cells[pid].v.Add(delta)
        runtime_procUnpin()
        return
    }
    runtime_procUnpin()
    p.resize(runtime.GOMAXPROCS(0))
    p.Add(delta)
}

func (p *PerP) Get() int64 {
    cells := *p.cellsPtr.Load()
    var total int64
    for i := range cells {
        total += cells[i].v.Load()
    }
    return total
}

func (p *PerP) Reset() int64 {
    cells := *p.cellsPtr.Load()
    var total int64
    for i := range cells {
        total += cells[i].v.Swap(0)
    }
    return total
}

// assertCachelineAligned is a runtime check that cells are padded.
func assertCachelineAligned(cells []pcell) bool {
    if len(cells) < 2 {
        return true
    }
    diff := uintptr(unsafe.Pointer(&cells[1])) - uintptr(unsafe.Pointer(&cells[0]))
    return diff >= uintptr(cpu.CacheLinePadSize)
}

Key design points:

• atomic.Pointer[[]pcell] for the cell slice; resizing swaps the pointer.
• cells[pid] is read after procPin, so the slice pointer is stable for the duration of our access.
• If pid >= len(cells), GOMAXPROCS grew; we unpin, resize, and retry.
• Resize uses a mutex; concurrent resizes are deduplicated.
• Reset is O(P) but called rarely.
• All counter math goes through the padded atomic cells.

This is production-grade. The main runtime cost is the procPin/procUnpin pair (a few cycles) plus the atomic add — comparable to a single unsharded atomic but contention-free.

Deep Dive: Benchmarking Sharded Counters Rigorously¶

Bad benchmarks lie. Here is a rigorous one for sharded counter design:

package counters

import (
    "runtime"
    "sync"
    "sync/atomic"
    "testing"
)

type Bench struct {
    name string
    fn   func()
}

func BenchmarkAll(b *testing.B) {
    var single atomic.Int64
    naive := [64]atomic.Int64{}
    padded := New(64)
    perP := NewPerP()

    benches := []Bench{
        {"single", func() { single.Add(1) }},
        {"naive[hash]", func() {
            i := runtime_FastRand()
            naive[i%64].Add(1)
        }},
        {"padded[hash]", func() {
            padded.Inc()
        }},
        {"perP", func() {
            perP.Inc()
        }},
    }

    for _, bb := range benches {
        b.Run(bb.name, func(b *testing.B) {
            b.ResetTimer()
            b.RunParallel(func(pb *testing.PB) {
                for pb.Next() {
                    bb.fn()
                }
            })
        })
    }
}

//go:linkname runtime_FastRand runtime.fastrand
func runtime_FastRand() uint32

Run at multiple core counts:

for c in 1 2 4 8 16 32; do
  go test -bench=BenchmarkAll -cpu=$c -benchtime=3s
done

Expected results (illustrative, on a 16-core x86-64):

(Numbers vary widely by hardware; run on yours.)

Interpretation:

• single scales worst: contention dominates.
• naive (no padding) suffers from false sharing past ~4 cores.
• padded scales until cache traffic between distant cores becomes visible.
• perP is essentially flat — perfect scaling.

This is the chart that wins design arguments. Generate it for your service.

Deep Dive: Sharded Counters with Coordinated Reset¶

A nuance of Reset on a sharded counter: between resetting shard 0 and shard 63, increments to shards 1-63 are still arriving. Those are "lost" to the return value of the current Reset but will appear in the next Reset.

For monitoring, this is fine. For exact billing, you need a versioning scheme:

type VersionedShard struct {
    cells [N]struct {
        v   atomic.Int64
        ver atomic.Uint64 // bumped each reset
    }
}

func (s *VersionedShard) Inc(epoch uint64) {
    idx := pickShard()
    for {
        cur := s.cells[idx].v.Load()
        if s.cells[idx].ver.Load() > epoch {
            // This shard was reset after epoch; our increment belongs in the new epoch.
            // Decide: drop, count in new epoch, or retry.
            return
        }
        if s.cells[idx].v.CompareAndSwap(cur, cur+1) {
            return
        }
    }
}

This is brittle. For real exact-counting, use a different design entirely (e.g., write to an append-only log, count in a batch job). Senior takeaway: do not try to make sharded counters atomically resettable.

Deep Dive: Sloppy Counter Variations¶

The basic sloppy counter has many useful variations.

Time-based flush¶

Instead of flushing every N increments, flush every T milliseconds:

type TimedLocal struct {
    parent *Sloppy
    n      int64
    nextAt time.Time
    period time.Duration
}

func (l *TimedLocal) Inc() {
    l.n++
    if time.Now().After(l.nextAt) {
        l.parent.global.Add(l.n)
        l.n = 0
        l.nextAt = time.Now().Add(l.period)
    }
}

Trades: bounded freshness in time (always within period seconds) regardless of rate.

Adaptive threshold¶

If the global value is read often, flush more aggressively; if rarely, flush less:

type AdaptiveLocal struct {
    parent  *Sloppy
    n       int64
    flushAt int64 // adjusted based on observed read frequency
}

Requires the global to track reader-frequency. Rarely worth the complexity.

Centrally-coordinated flush¶

A separate goroutine periodically requests all locals to flush. Requires a registry of locals and a flush channel per local:

type Coordinator struct {
    locals []chan struct{}
    mu     sync.Mutex
}

func (c *Coordinator) Register(local *Local) chan struct{} {
    ch := make(chan struct{}, 1)
    c.mu.Lock()
    c.locals = append(c.locals, ch)
    c.mu.Unlock()
    return ch
}

func (c *Coordinator) FlushAll() {
    c.mu.Lock()
    defer c.mu.Unlock()
    for _, ch := range c.locals {
        select { case ch <- struct{}{}: default: }
    }
}