Concurrent Counters — Professional Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Beyond Counters: Why Histograms
5. HDR Histograms in Depth
6. Implementing an HDR-Backed Counter
7. Percentile-Preserving Merging
8. Time-Bucketed Histograms
9. NUMA-Aware Counter Architecture
10. Observability Subsystem Architecture
11. expvar + Prometheus + OpenTelemetry
12. Multi-Process Counter Aggregation
13. Cardinality Management
14. Adaptive Counters
15. Counters in Extreme Environments
16. Performance Engineering at Scale
17. Operational Concerns
18. Security & Compliance
19. Testing Strategy
20. Common Mistakes at Scale
21. Tricky Questions
22. Cheat Sheet
23. Self-Assessment Checklist
24. Summary
25. What You Can Build
26. Further Reading
27. Related Topics
28. Diagrams & Visual Aids

Introduction¶

Focus: "Histograms (HDR), per-CPU counters at extreme scale, full observability subsystem design, NUMA, multi-process, cardinality, and the operational craft of metrics at production-grade volume."

At the senior level you built counters that scale to many cores. You can pad, shard, per-P, sloppy, and dynamic-shard. You are no longer afraid of false sharing. You can interpret a perf c2c report and a -bench -cpu curve.

This file is about the design of systems of counters: full observability subsystems, integration with Prometheus and OpenTelemetry, NUMA-aware shard placement, HDR histograms for distributions, percentile-preserving merging, and the operational concerns that show up only at scale (cardinality, multi-process aggregation, exposure security).

You will learn:

• Why and when to use HDR histograms instead of counters
• How to implement a high-throughput, low-error HDR-histogram-backed metric
• Time-bucketed histograms for sliding windows
• NUMA-aware shard placement for multi-socket systems
• The full architecture of a production observability subsystem: primitives → registry → exposition → transport → backend
• How expvar, Prometheus, and OpenTelemetry coexist and when to choose which
• Multi-process counter aggregation patterns
• Cardinality bombing — how to avoid blowing up your metrics backend
• Adaptive counters that adjust behaviour based on observed load
• Operational concerns: dashboards, alerts, runbooks built on counter outputs

By the end, you can design the metrics subsystem for a service handling millions of RPS across hundreds of instances and dozens of teams.

Prerequisites¶

• Required: Senior-level mastery of concurrent counters: padding, sharding, per-P, sloppy, LongAdder.
• Required: Experience operating production services. You should have been paged on a counter-driven alert at least once.
• Required: Familiarity with Prometheus and at least one of OpenTelemetry, Datadog, or Honeycomb.
• Required: Comfort with Go runtime internals — what runtime.NumGoroutine measures, how the GC interacts with allocations, where the stack lives.
• Helpful: Experience with NUMA-aware service deployment (multi-socket bare metal or large cloud instances).
• Helpful: Background in observability theory: SLOs, error budgets, signal types (RED, USE, golden signals).

Glossary¶

Beyond Counters: Why Histograms¶

A counter answers "how many?". It does not answer "what shape?".

Suppose your service averages 50 ms response time. Operators wake up at 3 AM because p99 latency spiked. Your counter cannot tell them what p99 is — it only has the total time and total count. You compute the average from those. The average has been constant. The p99 has spiked. You see nothing.

The cause: a few slow requests dominate the tail without moving the average. You need a histogram.

A histogram records the distribution of observations. It can answer:

• What is p50 / p95 / p99 / p99.9?
• What fraction of requests took longer than 100 ms?
• Is the distribution bimodal?
• Has the shape changed over time?

For latency, response size, queue depth, and many other measures, a histogram is the right primitive.

Naive histogram¶

A simple histogram is an array of buckets, each counting observations in a range:

type Histogram struct {
    buckets []atomic.Int64
    bounds  []float64
}

func (h *Histogram) Observe(v float64) {
    for i, b := range h.bounds {
        if v <= b {
            h.buckets[i].Add(1)
            return
        }
    }
    h.buckets[len(h.bounds)].Add(1) // +Inf bucket
}

Each Observe walks the bounds array to find the right bucket. For 20 buckets, this is ~20 comparisons — fast. For 100 buckets, slower. For HDR's typical 1700+ buckets, this is too slow.

HDR histograms use a clever bucket layout that lets you find the bucket in O(1) via bit operations.

HDR Histograms in Depth¶

The HDR histogram (designed by Gil Tene at Azul Systems) provides:

• Constant-time Record(value) regardless of bucket count
• Bounded relative error (configurable; typically 0.001 = 0.1%)
• Fixed memory footprint
• Coordinated-omission compensation
• Mergeable across processes

Bucket layout¶

The HDR histogram divides the value range into "buckets" of equal relative width. For a target precision of 3 significant digits (relative error 0.001), each "decade" of values is divided into 2048 sub-buckets, with the bucket index growing by 1 for each next-decade region.

Concretely: values 1.000-1.001 fall in one bucket; 1.001-1.002 in the next; 10.00-10.01 in another; etc. The bucket boundaries are spaced logarithmically with linear sub-spacing.

The math: for a value v, the bucket index is

exp = log2(v) - log2(2 * subBucketCount)
if exp < 0:
    sub = v
    index = sub
else:
    sub = v >> exp
    index = (exp + 1) * subBucketCount + sub - subBucketCount

In code, this becomes a small handful of bit operations — constant time.

Memory layout¶

For a tracking range of 1 to 60_000_000 (e.g., latency in microseconds, up to 60s) with 3-digit precision:

• subBucketCount = 2048 (2^11)
• buckets = 17 (one per power of 2 of the range)
• total buckets ≈ 17 * 2048 = 34,816
• each bucket is an int64 → ~272 KB

Larger than a simple counter; smaller than millions of raw observations.

Recording¶

func (h *HDR) RecordValue(v int64) {
    if v < 0 || v > h.highest {
        return // out of range
    }
    idx := h.indexOf(v)
    atomic.AddInt64(&h.counts[idx], 1)
}

indexOf is a few bit operations. Add is one atomic. Total: ~10-20 ns per record. Fast enough to record every event.

Querying¶

func (h *HDR) ValueAtPercentile(p float64) int64 {
    total := h.TotalCount()
    threshold := int64(float64(total) * p / 100)
    var sum int64
    for i, c := range h.counts {
        sum += atomic.LoadInt64(&c)
        if sum >= threshold {
            return h.valueFromIndex(i)
        }
    }
    return h.highest
}

TotalCount is O(buckets). ValueAtPercentile is also O(buckets) in the worst case. Both run at scrape time, infrequently.

Coordinated omission¶

If you measure latency by timing a function and the function takes 100 ms when it should take 1 ms, the calling loop falls behind. The next call's measured time still looks like 1 ms — but really it should be measured as "100 ms (waiting) + 1 ms (running)". HDR has a RecordValueWithExpectedInterval method that compensates:

func (h *HDR) RecordValueWithExpectedInterval(v, expected int64) {
    h.RecordValue(v)
    if v > expected {
        for missing := expected; missing < v; missing += expected {
            h.RecordValue(v - missing)
        }
    }
}

This synthesises missing observations to fill the gap. Crucial for honest latency measurement under load.

Go implementation¶

Use github.com/HdrHistogram/hdrhistogram-go (or its newer v2):

import "github.com/HdrHistogram/hdrhistogram-go"

h := hdrhistogram.New(1, 60_000_000, 3) // min, max, significant digits
h.RecordValue(123)
p99 := h.ValueAtQuantile(99)

Caveats:

• The library is not thread-safe for RecordValue from multiple goroutines. Wrap with a mutex, shard, or use atomic.AddInt64 on the underlying count slice directly (advanced).
• The Snapshot returns a deep copy; cheap to merge with another histogram via Add(other).

Concurrency wrapper¶

type ConcurrentHDR struct {
    shards []shard
}

type shard struct {
    _ cpu.CacheLinePad
    mu sync.Mutex
    h  *hdrhistogram.Histogram
    _ cpu.CacheLinePad
}

func New(minV, maxV int64, sig int) *ConcurrentHDR {
    n := runtime.GOMAXPROCS(0)
    s := make([]shard, n)
    for i := range s {
        s[i].h = hdrhistogram.New(minV, maxV, sig)
    }
    return &ConcurrentHDR{shards: s}
}

func (c *ConcurrentHDR) RecordValue(v int64) {
    s := &c.shards[runtime_fastrand()%uint32(len(c.shards))]
    s.mu.Lock()
    s.h.RecordValue(v)
    s.mu.Unlock()
}

func (c *ConcurrentHDR) Merged() *hdrhistogram.Histogram {
    out := hdrhistogram.New(c.shards[0].h.LowestTrackableValue(), c.shards[0].h.HighestTrackableValue(), int(c.shards[0].h.SignificantFigures()))
    for i := range c.shards {
        c.shards[i].mu.Lock()
        out.Merge(c.shards[i].h)
        c.shards[i].mu.Unlock()
    }
    return out
}

Each shard has its own histogram, protected by its own mutex. Merging is O(shards × buckets). Done at scrape time.

For ultimate performance, manipulate the underlying count slice with atomics directly:

type LockFreeShard struct {
    _      cpu.CacheLinePad
    counts []atomic.Int64 // same layout as hdrhistogram.Histogram.counts
    _      cpu.CacheLinePad
}

But you must replicate HDR's indexOf logic exactly, and the library does not expose it. Most teams accept the mutex; the wins of going fully lock-free are modest.

Implementing an HDR-Backed Counter¶

Let us build a full HDR-backed latency counter for an HTTP handler:

package metrics

import (
    "net/http"
    "runtime"
    "sync"
    "sync/atomic"
    "time"

    "github.com/HdrHistogram/hdrhistogram-go"
    "golang.org/x/sys/cpu"
)

type LatencyMetric struct {
    name    string
    shards  []latencyShard
    count   atomic.Int64 // total observations (for fast count without merge)
    sum     atomic.Int64 // total nanos (for average)
}

type latencyShard struct {
    _  cpu.CacheLinePad
    mu sync.Mutex
    h  *hdrhistogram.Histogram
    _  cpu.CacheLinePad
}

func NewLatencyMetric(name string) *LatencyMetric {
    n := runtime.GOMAXPROCS(0)
    shards := make([]latencyShard, n)
    for i := range shards {
        shards[i].h = hdrhistogram.New(1, 60_000_000_000, 3) // 1ns to 60s, 3 sig digits
    }
    return &LatencyMetric{name: name, shards: shards}
}

//go:linkname runtime_fastrand runtime.fastrand
func runtime_fastrand() uint32

func (m *LatencyMetric) Observe(d time.Duration) {
    nanos := d.Nanoseconds()
    m.count.Add(1)
    m.sum.Add(nanos)
    idx := uint32(runtime_fastrand()) % uint32(len(m.shards))
    s := &m.shards[idx]
    s.mu.Lock()
    s.h.RecordValue(nanos)
    s.mu.Unlock()
}

func (m *LatencyMetric) Count() int64 { return m.count.Load() }
func (m *LatencyMetric) Sum() int64   { return m.sum.Load() }
func (m *LatencyMetric) Mean() float64 {
    c := m.count.Load()
    if c == 0 {
        return 0
    }
    return float64(m.sum.Load()) / float64(c)
}

func (m *LatencyMetric) Quantile(p float64) int64 {
    merged := m.merged()
    return merged.ValueAtQuantile(p * 100)
}

func (m *LatencyMetric) merged() *hdrhistogram.Histogram {
    out := hdrhistogram.New(1, 60_000_000_000, 3)
    for i := range m.shards {
        m.shards[i].mu.Lock()
        out.Merge(m.shards[i].h)
        m.shards[i].mu.Unlock()
    }
    return out
}

// Middleware that wraps an http.Handler and records latency.
func (m *LatencyMetric) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        start := time.Now()
        next.ServeHTTP(w, r)
        m.Observe(time.Since(start))
    })
}

Usage:

metrics := NewLatencyMetric("http_request_duration")
http.Handle("/api/users", metrics.Middleware(usersHandler))

At scrape time:

fmt.Fprintf(w, "count=%d\n", metrics.Count())
fmt.Fprintf(w, "mean=%.2fns\n", metrics.Mean())
fmt.Fprintf(w, "p50=%dns\n", metrics.Quantile(0.5))
fmt.Fprintf(w, "p95=%dns\n", metrics.Quantile(0.95))
fmt.Fprintf(w, "p99=%dns\n", metrics.Quantile(0.99))
fmt.Fprintf(w, "p999=%dns\n", metrics.Quantile(0.999))

Properties:

• Per-shard records with their own mutex; no central lock.
• count and sum are atomic, fast-path for rate-derived metrics.
• HDR shards merge at scrape time only.
• Cache-line padding prevents shard contention.
• Each shard is one HDR histogram (~272 KB); N shards × 16 cores = 4.4 MB total. Acceptable.

This is a production-grade latency metric.

Percentile-Preserving Merging¶

When you have shards (or multiple processes), you need to merge their histograms to get a global percentile. The naive approach:

merged := hdrhistogram.New(1, max, 3)
for _, shard := range shards {
    merged.Merge(shard.h)
}
p99 := merged.ValueAtQuantile(99)

This works because HDR histograms are additive: bucket counts sum. The merged histogram has the bucket counts of the sum of all shards. Percentiles computed from the merged histogram are correct for the union of observations.

Crucially, you cannot average percentiles across shards. p99 of shard A averaged with p99 of shard B is not the global p99. The merge is necessary.

Cross-process merging¶

When you have multiple Go processes each emitting a histogram, the metric backend must merge them. Prometheus does this via histogram_quantile(0.99, sum(rate(http_duration_seconds_bucket[5m])) by (le)) — it sums the bucket counts across processes, then computes the quantile from the sum.

This is correct because Prometheus histograms are bucket-based (like HDR) and bucket counts are additive.

OpenMetrics histograms work the same way.

Don't average percentiles¶

A common operational mistake: 10 processes each emit p99_latency (a scalar). The dashboard averages them and displays the result. That value is meaningless.

The right way: emit histogram bucket counts; let the backend compute the percentile on the merged distribution.

HDR-specific merging tricks¶

HDR has a Merge method that takes the bucket counts of one histogram and adds them to the receiver. It is O(buckets), typically ~1 ms for a full 34K-bucket histogram. At scrape time (every 15 s), this is irrelevant.

For very large numbers of shards or processes, you can do hierarchical merging:

shards: [s0, s1, s2, ..., s31]
level1: [merge(s0..s7), merge(s8..s15), merge(s16..s23), merge(s24..s31)]
final: merge(level1)

Tree-style merging is O(N log N) work but parallelisable. Rarely needed for the sizes typical in Go services.

Time-Bucketed Histograms¶

A single HDR histogram accumulates forever (or until you reset it). For "last 5 minutes", you need a ring of histograms, one per time bucket:

type TimedHistogram struct {
    buckets []*hdrhistogram.Histogram
    period  time.Duration
    cur     atomic.Int64 // current bucket index
    mu      sync.Mutex
}

func NewTimed(numBuckets int, bucketPeriod time.Duration) *TimedHistogram {
    buckets := make([]*hdrhistogram.Histogram, numBuckets)
    for i := range buckets {
        buckets[i] = hdrhistogram.New(1, 60_000_000_000, 3)
    }
    return &TimedHistogram{buckets: buckets, period: bucketPeriod}
}

func (t *TimedHistogram) Observe(v int64) {
    idx := t.cur.Load() % int64(len(t.buckets))
    t.mu.Lock()
    t.buckets[idx].RecordValue(v)
    t.mu.Unlock()
}

func (t *TimedHistogram) Tick() {
    next := (t.cur.Load() + 1) % int64(len(t.buckets))
    t.mu.Lock()
    t.buckets[next].Reset()
    t.cur.Store(next)
    t.mu.Unlock()
}

func (t *TimedHistogram) Snapshot() *hdrhistogram.Histogram {
    out := hdrhistogram.New(1, 60_000_000_000, 3)
    t.mu.Lock()
    for _, b := range t.buckets {
        out.Merge(b)
    }
    t.mu.Unlock()
    return out
}

func (t *TimedHistogram) Run(stop <-chan struct{}) {
    ticker := time.NewTicker(t.period)
    defer ticker.Stop()
    for {
        select {
        case <-stop:
            return
        case <-ticker.C:
            t.Tick()
        }
    }
}

For "last 5 minutes with 1-minute resolution", use 5 buckets of 1 minute. Snapshot() returns a histogram covering the last 5 minutes (approximately — the current bucket may be partial).

For higher concurrency, shard each time bucket the same way the simple LatencyMetric shards.

Sliding windows are surprisingly hard¶

You may have noticed: a goroutine Observes while another goroutine Ticks. Which bucket does the observe land in? Depends on timing. Edge effects.

Mitigations:

• Accept fuzziness at bucket boundaries (typical).
• Use sub-second buckets for finer resolution (more memory).
• Use exponentially-decaying weighted averages instead of ring buffers (different design entirely).

For most monitoring, ring buffers of 1-minute or 1-second buckets are fine.

NUMA-Aware Counter Architecture¶

On a multi-socket server, memory near a socket is faster than memory near another socket. Cross-socket cache transfers can be 200-500 ns vs 50-100 ns same-socket.

For counters, this matters when:

• Your goroutines run on cores spread across sockets.
• Your counter cells happen to live in memory near one socket.
• Cross-socket writes dominate.

Detection¶

On Linux, numactl --hardware shows your NUMA topology. lscpu shows NUMA nodes and core mappings.

In Go, the runtime does not expose NUMA topology. You can read it manually:

data, _ := os.ReadFile("/sys/devices/system/node/online")
// parses to "0-1" for two-socket system

Mitigation¶

The brutal but effective approach: pin OS threads to sockets. Each socket gets its own counter cells.

// Pseudo-code; requires CGO and platform-specific syscalls.
func PinToSocket(threadID, socket int) error {
    return setaffinity(threadID, cpusForSocket(socket))
}

type NumaCounter struct {
    perSocket []*Sharded
}

func (n *NumaCounter) Inc() {
    sock := currentSocket() // expensive
    n.perSocket[sock].Inc()
}

The currentSocket() call is expensive on Linux (a syscall or vdso). Caching it per-OS-thread mitigates the cost — combine with runtime.LockOSThread to make a goroutine stick to a thread, which sticks to a core, which sticks to a socket.

When it matters¶

For a single-socket cloud VM (most deployments), NUMA is a non-issue. For:

• Multi-socket bare metal
• Cloud instances with > 32 vCPUs (often 2 sockets)
• HPC workloads
• Database servers

NUMA can be a 2-5× factor in counter contention.

Practical advice¶

If your service runs on multi-socket hardware and counter contention is a measurable problem:

1. Confirm NUMA topology of your deployment.
2. Pin OS threads to sockets (one Go binary per socket, or taskset at launch).
3. Use per-socket sharded counters.
4. Verify with perf c2c that cross-socket traffic dropped.

For most services, none of this is necessary. NUMA awareness is the last 10× when you've already optimised everything else.

Observability Subsystem Architecture¶

A complete observability subsystem for a Go service includes:

+--------------------------------------------------+
|              Application Code                    |
|  counter.Inc()  latency.Observe(d)  ...          |
+------------------------+-------------------------+
                         |
+------------------------v-------------------------+
|        Counter / Histogram Primitives            |
|  atomic.Int64  Sharded  PerP  Sloppy  HDR        |
+------------------------+-------------------------+
                         |
+------------------------v-------------------------+
|             Metric Registry                      |
|  named lookup, type registration, labels         |
+--------+--------+--------+----------+-----------+
         |        |        |          |
         v        v        v          v
   +-----+--+ +---+--+ +---+----+ +---+----+
   |expvar  | |Prom  | |OTLP    | |Custom  |
   |JSON    | |Text  | |gRPC/HTTP||sink    |
   +--------+ +------+ +--------+ +--------+
        |        |         |          |
        v        v         v          v
   /debug/vars  /metrics  collector   logs/files

+--------------------------------------------------+
|              External Backend                    |
|  Prometheus, Grafana, OpenTelemetry, Datadog     |
+--------------------------------------------------+

Each layer is independently designed:

• Primitives: senior-level work — the right shape for each contention profile.
• Registry: holds named metrics, manages labels, prevents cardinality bombing.
• Exposition: serialises metrics in various formats.
• Transport: HTTP pull, gRPC push, file rotation, etc.
• Backend: Prometheus, Datadog, etc. Out of scope for this file.

Why multiple exposition formats?¶

• expvar JSON: free, built-in, useful for local development and ad-hoc curl.
• Prometheus text: industry-standard pull format; works with most monitoring stacks.
• OTLP: modern push format; supports labels, traces, exemplars.
• Custom: e.g., to a corporate logging system.

A service often exposes 2-3 of these simultaneously. The registry abstracts the formats.

Registry design¶

type Registry struct {
    mu   sync.RWMutex
    m    map[string]MetricFamily
    help map[string]string
    labels map[string]LabelSpec
}

type MetricFamily interface {
    Encode(format Format, w io.Writer, name string) error
}

type Format int
const (
    FormatJSON Format = iota
    FormatProm
    FormatOTLP
)

Each metric implements Encode for each format it cares about. The registry dispatches based on the requested format.

For OTLP specifically, the encoding involves Protocol Buffers and gRPC — heavier than text formats. Many services use the OpenTelemetry SDK directly for OTLP and expvar/Prometheus for the other formats.

Labels and cardinality¶

A labeled counter like http_requests_total{status, route, user} can have unbounded cardinality if any label has unbounded values (especially user). Cardinality bombing is a classic observability incident:

• Application emits a unique label combo per request.
• Metric backend's memory explodes.
• Backend dies; alerting is gone; outage cascades.

Mitigations:

• Enforce a cardinality limit per metric (e.g., 10K distinct combos).
• Use a _total{status} for status code (low cardinality) and don't include user.
• For per-user counts, use traces/spans, not metrics.

The registry should track cardinality and refuse new combos past the limit:

func (r *Registry) AddObservation(metric string, labels Labels, value float64) error {
    family := r.m[metric]
    if family.Cardinality() >= r.labels[metric].MaxCardinality {
        family.IncrementDropped(1)
        return ErrCardinalityLimit
    }
    family.Add(labels, value)
    return nil
}

This protects the backend at the cost of dropped observations. Dropped observations are themselves a metric.

expvar + Prometheus + OpenTelemetry¶

Real services often expose all three. How do they coexist?

Common pattern: one source of truth, multiple exposers¶

type Counter struct {
    v atomic.Int64
    // ... possibly sharded
}

func (c *Counter) Inc()       { c.v.Add(1) }
func (c *Counter) Value() int64 { return c.v.Load() }

// Expose via expvar
func (c *Counter) Expvar() expvar.Var {
    return expvar.Func(func() any { return c.Value() })
}

// Expose via Prometheus
func (c *Counter) Prometheus(name, help string) prometheus.Collector {
    return prometheus.NewCounterFunc(
        prometheus.CounterOpts{Name: name, Help: help},
        func() float64 { return float64(c.Value()) },
    )
}

// Expose via OTLP — using OpenTelemetry SDK
func (c *Counter) OpenTelemetry(meter metric.Meter, name string) {
    counter, _ := meter.Int64ObservableCounter(name)
    meter.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
        o.ObserveInt64(counter, c.Value())
        return nil
    }, counter)
}

The underlying atomic.Int64 is the source of truth. Three wrapper functions expose it to three different systems. Each scrape/push reads the current value.

When to choose each¶

• expvar only: tiny services, internal tools, "is the process alive?" debugging.
• Prometheus: standard for modern container/Kubernetes deployments.
• OpenTelemetry: when traces and metrics are correlated (e.g., exemplars linking metrics to traces). Future-proof.
• All three: large services with diverse consumers; the overhead is small if you wrap one underlying primitive.

Performance overlap¶

Each exposure costs at scrape time, not at increment time. The increment is still one atomic. The Prometheus scrape iterates registered collectors; expvar serialises the map; OTLP packages the observation set. None of these touch the hot path.

Cardinality across systems¶

Be careful: a labeled counter has cardinality on each emitter. If http_requests_total{status, route} has 100 distinct combos, each backend sees those 100 time series. Cardinality multiplies if you double-expose. Coordinate label sets across exposers.

Multi-Process Counter Aggregation¶

A single Go process is one thing; a fleet of Go processes (containers, instances, replicas) is another. The metric backend aggregates across processes — but only for compatible metric shapes.

Counters: easy¶

Each process's counter is monotonically increasing. The backend sums them across processes (or uses rate() per process and sums). Prometheus does this natively.

Caveat: counter resets (process restarts) cause apparent decreases. Prometheus handles this with the rate() function, which detects and skips resets.

Gauges: tricky¶

Each process has its own gauge value. The backend's job is unclear:

• Sum (for "total requests in flight across fleet")
• Max (for "max queue depth seen across fleet")
• Average (for "average CPU per process")
• Histogram (for "distribution of memory usage")

Pick the right aggregation per metric. Prometheus does this with sum(), max(), avg() in PromQL.

Histograms: subtle¶

Each process emits bucket counts. The backend sums bucket counts across processes, then computes the percentile from the sum. Correct.

You cannot compute percentiles per process and then average them. That gives a meaningless number.

Counter resets are confusing¶

On process restart, the counter goes from N back to 0. The backend sees "N decreased to 0", which it must interpret as "process restarted, counter resumed from 0". Prometheus's rate() handles this correctly; bespoke logic may not.

A more durable pattern: each process emits both a counter and an instance_id label. The backend treats counter{instance_id=X} as a per-instance metric and sums across instances using sum without(instance_id) (rate(counter[5m])).

Cardinality Management¶

Cardinality is the number of distinct label combinations. High cardinality is the #1 metric-backend outage cause.

Sources of high cardinality¶

• User ID labels
• Trace/request ID labels
• Anything user-controlled

Bounding cardinality¶

1. Label whitelist: only allow labels from a known set.
2. Value bucketing: group user IDs into 100 buckets by hash; emit user_bucket=42 instead of user_id=....
3. Cardinality limits at metric layer: refuse new label combos past N.
4. Pre-aggregation: before sending to backend, group by stable labels and drop high-cardinality ones.

Cardinality in the registry¶

type MetricFamily struct {
    series sync.Map // map[labelHash]*Counter
    count  atomic.Int64
    limit  int64
}

func (m *MetricFamily) Inc(labels Labels) {
    h := labels.Hash()
    if _, ok := m.series.Load(h); !ok {
        if m.count.Load() >= m.limit {
            // over budget; drop
            atomic.AddInt64(&m.dropped, 1)
            return
        }
        m.series.LoadOrStore(h, &Counter{})
        m.count.Add(1)
    }
    v, _ := m.series.Load(h)
    v.(*Counter).Inc()
}

The dropped counter is itself a metric — emit it so operators can see when cardinality limits are biting.

Cardinality across the fleet¶

A label pod_name with 100 values per instance × 1000 instances = 100K distinct series. The backend has to store them all. Audit label cardinality regularly.

Adaptive Counters¶

Counters that adjust behaviour based on observed load are an advanced pattern. Examples:

Adaptive shard count¶

Start unsharded. If contention is detected (CAS failures, high atomic latency), grow to N shards. If contention disappears, optionally shrink (rare).

Adaptive flush threshold¶

A sloppy counter that flushes every K increments. Under low load, K can be small (fast freshness). Under high load, K grows (less contention). Detect load via increment rate.

Adaptive precision¶

A histogram that uses high precision (3 sig digits) by default but degrades to lower precision (2 sig digits) under memory pressure.

Adaptive sampling¶

A counter that increments every Nth event under high load (saving cost) but every event under low load. Combined with sample tracking, the metric is corrected at scrape time.

All four are advanced patterns. They are rarely needed but extremely powerful when applicable. Java's LongAdder is one example of adaptive sharding. Most other adaptations are bespoke.

Counters in Extreme Environments¶

High-frequency trading¶

Latency-sensitive systems require deterministic counters. Sloppy is best (no contention). Pre-allocate everything; avoid GC during trading hours.

Embedded / IoT¶

Memory is constrained. Padding is expensive. Use small (int32) counters where possible; avoid HDR.

GPU compute (CUDA via cgo)¶

Per-thread counters are tricky because GPU "threads" are not Go goroutines. Aggregate at kernel boundaries.

WebAssembly¶

Single-threaded (in browsers; Wasi has threads). No concurrency, no atomics needed. int64 is fine.

Network functions / DPDK-style¶

Pin everything to cores. Per-CPU counters with no atomic — just direct writes, since no other thread will touch the cell.

These environments require deep knowledge of the platform. The patterns from this file transfer with adjustments.

Performance Engineering at Scale¶

Profiling discipline¶

• Profile every release.
• Compare profiles week-over-week.
• Set budgets for counter overhead (e.g., < 1% of CPU).
• Track increment latency, not just count.

Continuous benchmarking¶

• Run benchmarks in CI on dedicated hardware.
• Detect regressions automatically.
• Alert on > 10% slowdown.

Production instrumentation of counters¶

Counters of counters: track how many increments per second your counters do. If a counter is hit at 1 GHz, you have a problem. If at 1 Hz, do not bother optimising.

Tail latency in counter ops¶

Atomic ops have predictable mean but tail latency under contention can be 100× the mean. Measure p99 of increment latency, not just throughput.

NUMA in production¶

Test in staging on hardware that matches production. If staging is single-socket and prod is multi-socket, NUMA bites you the day you go live.

Operational Concerns¶

Dashboards¶

A counter feeds:

• Rate gauges: rate(http_requests_total[5m])
• SLO panels: 1 - (rate(http_errors_total[5m]) / rate(http_requests_total[5m]))
• Heatmaps: histogram_quantile(0.99, rate(http_duration_seconds_bucket[5m]))
• Top-N tables: by labels (status, route, etc.)

Design dashboards before designing metrics — clarify what operators need.

Alerts¶

A counter feeds:

• Error rate alerts: rate(errors[5m]) > 10
• SLO budget alerts: slo_remaining < 0.5
• Anomaly alerts: rate dropped 50% vs last week

Counter-driven alerting is the workhorse of SRE. The counter is the alert's truth.

Runbooks¶

When an alert fires, the runbook says:

• Which counter triggered (http_5xx_total).
• What its baseline is.
• What graph to check (the dashboard with that counter).
• What to do (rollback, scale, page).

Counter names appearing in runbooks should be stable. Renaming a counter breaks runbooks.

Security & Compliance¶

Counters can leak data¶

A counter failed_logins_total{username} leaks usernames. A counter requests_by_ip{ip} leaks IPs. PII in labels is a compliance issue.

Audit labels for PII before shipping.

Counters can be DoS targets¶

If your counter increments on attacker input, an attacker can drive a counter to high cardinality, blowing up your backend. Validate inputs before they become labels.

Counters can leak internal state¶

/metrics endpoints expose request volumes, error rates, capacity. Attackers profile your system from outside if /metrics is public.

Always authenticate /metrics. Bind to private interfaces.

GDPR and counters¶

Counters with PII labels may be subject to GDPR retention rules. Audit your metrics for compliance, the same as logs.

Testing Strategy¶

Correctness tests¶

Per-counter unit tests: N goroutines × M increments → total == N*M. Always under -race.

Concurrency tests¶

Stress tests: thousands of goroutines, millions of increments. Check final value.

Scaling tests¶

-bench -cpu=1,2,4,8,16 on dedicated hardware. Verify scaling claims.

Integration tests¶

Test the full metrics pipeline: increment → exposition → scrape → backend → query. Use a local Prometheus instance.

Snapshot tests¶

For each metric format (JSON, Prometheus, OTLP), assert the serialised output matches a golden file.

Cardinality tests¶

For each labeled counter, test that excess labels are dropped (not silently growing the registry).

Fuzz tests¶

Throw random label values, random delta values, random concurrency levels. Look for panics, deadlocks, or wrong outputs.

Common Mistakes at Scale¶

1. No cardinality limit on labels. Backend OOMs.
2. Averaging percentiles across processes. Numbers look fine, are meaningless.
3. Counter reset on every scrape. Loses data between scrapes.
4. No tests for counter correctness under load. Race conditions hide.
5. Unauthenticated /metrics. Information leak.
6. Padding only some counters. Inconsistent performance.
7. Per-process counters as "the answer". Without aggregation across the fleet, you cannot see the whole picture.
8. High-precision HDR for low-rate counters. Wastes memory.
9. Coordinated omission not addressed. Latency measurements are wrong.
10. Histograms in tight inner loops. Even cheap HDR records cost; can dominate at extreme rates.

Tricky Questions¶

Q: How do you handle counter reset across process restarts? A: Don't try to. Use rate() in the query layer; Prometheus and OTLP handle resets gracefully. For exact cumulative counts across restarts, persist to a database — but rarely needed.

Q: What's the right HDR precision? A: 2-3 significant digits. More wastes memory; less loses fidelity at tails.

Q: How do you debug a counter that is "wrong"? A: First, confirm it really is wrong (not just lagged). Then look for: missing atomics, sharded reset races, cardinality drops, label mismatches. Last resort: enable counter trace logging.

Q: Should I use a single labeled counter or multiple counters by label? A: Labeled if cardinality is bounded; multiple if not. Labels are easier in queries; multiple are easier in code.

Q: How do you migrate from expvar to Prometheus? A: Keep expvar exposed; add Prometheus alongside. Cut over Grafana dashboards. Remove expvar later if desired. Coexistence is cheap.

Q: What about exemplars? A: An exemplar is a sample trace ID attached to a counter or histogram observation. OpenTelemetry supports this; correlation between metrics and traces is the killer feature.

Q: How do you handle histograms with values much larger than expected? A: HDR caps at highest value; out-of-range values are dropped. Configure highest generously (e.g., 60s for HTTP latency).

Q: Do I need NUMA awareness in cloud deployments? A: Usually not. Most cloud VMs are single-socket. For very large instances (96+ vCPUs) or dedicated bare metal, yes.

Q: How do you size HDR memory? A: Each bucket is 8 bytes (int64). For (1ns - 60s, 3 sig digits): ~34K buckets ~= 272 KB. Multiply by shards. Multiply by metric count. Budget accordingly.

Q: Can you build the whole observability stack in Go? A: Yes. Prometheus is written in Go. OpenTelemetry has full Go SDKs. The standard library expvar is built in. Go is the default language of observability tooling.

Cheat Sheet¶

// HDR histogram concurrent wrapper
type LatencyMetric struct {
    shards []shard
}

func (m *LatencyMetric) Observe(d time.Duration) {
    s := &m.shards[runtime_fastrand()%uint32(len(m.shards))]
    s.mu.Lock()
    s.h.RecordValue(d.Nanoseconds())
    s.mu.Unlock()
}

// Multi-process aggregation: sum bucket counts, then quantile
// (Prometheus does this with histogram_quantile)

// Cardinality bound
if m.count.Load() >= m.limit {
    m.dropped.Add(1)
    return
}

// Expose to expvar, Prometheus, OTLP via wrappers around one atomic

Self-Assessment Checklist¶

• I can implement a sharded HDR histogram and explain the trade-offs.
• I understand coordinated omission and how RecordValueWithExpectedInterval addresses it.
• I can sketch a sliding-window histogram using a ring of HDR.
• I can design a multi-format exposition layer (expvar + Prometheus + OTLP).
• I know not to average percentiles across processes.
• I can bound cardinality in a labeled counter registry.
• I can detect and mitigate NUMA effects on counter contention.
• I have built and operated a production metrics pipeline end-to-end.
• I can write counter-driven SLO alerts that page on real incidents.
• I can audit a counter codebase for PII leaks, cardinality risks, and contention hotspots.

Summary¶

Professional-level concurrent counters are about systems, not primitives. You compose counters and histograms into observability subsystems, integrate with multiple backends, handle multi-process aggregation, bound cardinality, and manage operational concerns.

HDR histograms extend the counter primitive to distributions; sharded HDR scales to many cores; sliding windows give responsive rate metrics; NUMA awareness handles multi-socket deployments. Above the primitives, the registry layer manages exposition, labels, and cardinality. Above the registry, transport pushes data to Prometheus, OpenTelemetry, or custom backends. Above the backend, dashboards, alerts, and runbooks drive operations.

A senior built counters that scaled. A professional builds the system that uses them — and ensures every counter, every label, every dashboard, every alert is intentional.

What You Can Build¶

• A sharded HDR-histogram-backed latency metric
• A sliding-window rate metric with sub-second resolution
• A multi-format exposition layer (expvar + Prometheus + OTLP simultaneously)
• A cardinality-bounded labeled counter registry
• A NUMA-aware counter for multi-socket bare metal
• A full observability subsystem for a 100K-RPS Go service
• SLO alerts and runbooks driven by counter outputs
• A cross-language counter aggregation system (Go process emitting to Prometheus, Python aggregator reading)
• A metric system that compensates for coordinated omission
• An adaptive counter that adjusts shard count under load

Further Reading¶

• Gil Tene, "How Not to Measure Latency" — the canonical talk on coordinated omission.
• github.com/HdrHistogram/HdrHistogram — the reference HDR implementation in Java.
• github.com/HdrHistogram/hdrhistogram-go — the Go port.
• Prometheus's client_golang source — production-grade Go metrics library.
• OpenTelemetry's Go SDK — modern observability protocol.
• Google's SRE Book chapters on monitoring and alerting.
• Brendan Gregg's "Systems Performance" — for USE method, profiling techniques.
• Tom Wilkie, "Monitoring Microservices the Red Way" — for RED method.
• Heinrich Hartmann, "Statistics for Engineers" — on percentiles and distributions.

Related Topics¶

• All previous levels of concurrent counters (junior, middle, senior)
• HDR histograms in depth
• Prometheus / OpenTelemetry / Datadog metric models
• Distributed tracing (counters with exemplars)
• SLO and SLI design
• Cardinality theory in observability
• NUMA architectures
• Embedded / real-time concurrent programming

Diagrams & Visual Aids¶

HDR histogram bucket layout¶

exponent:   0   1   2     ...      16
sub-buckets:|--2048--||--2048--|...|--2048--|
values:     1-2     2-4   4-8       ~16M-32M

Each "sub-bucket" is a separate counter; recording is bit-math index calc.

Sharded HDR architecture¶

goroutine -> hash -> shard 17 -> mutex -> hdrhistogram.RecordValue
goroutine -> hash -> shard 42 -> mutex -> hdrhistogram.RecordValue

scrape -> merge all shards -> hdrhistogram with summed bucket counts
       -> ValueAtQuantile(99)

Sliding window¶

buckets: [B0][B1][B2][B3][B4]  (5 minutes, 1 minute each)
         ^
         current

after 1 minute tick:
buckets: [B0][B1][B2][B3][B4]
              ^
              current

(B5's data is overwritten, oldest minute drops off)

Multi-format exposition¶

       +--------------+
       | atomic.Int64 |   <-- one source of truth
       +------+-------+
              |
        +-----+----+----+----+
        |          |    |    |
        v          v    v    v
     expvar    Prom OTLP custom
     /vars   /metrics gRPC files

Cardinality bound¶

register("user_action", labels=[action, user_bucket])
  - action: 10 values
  - user_bucket: 100 values (hashed from user_id)
  cardinality = 10 * 100 = 1000  (acceptable)

NOT: labels=[action, user_id]
  - user_id: millions of values
  cardinality = 10 * millions  (UNACCEPTABLE)

Closing Thought¶

You started with count++ being wrong. You end with the design of full observability subsystems.

That is the journey of concurrent programming in miniature. Counters are the lens through which we see the whole field.

Build systems that count well. Operators will thank you. SREs will sleep better. And you will know, on the inside, exactly why each line of code is the way it is.

Deep Dive: The HDR Bucket Math¶

Understanding HDR's bucket layout from first principles helps you debug edge cases.

Goal¶

Record values in a wide range (e.g., 1 ns to 60 s = factor of 10^10) with bounded relative error (e.g., 0.1%) and constant-time recording.

Naive logarithmic¶

You could use log2(value) and quantise to N buckets per power of 2:

bucket = log2(value) * N

Computing log2 is slow (floating point, transcendental). HDR avoids it.

HDR's trick: clz (count leading zeros)¶

For an integer v, bitLen(v) = 64 - clz(v) is roughly log2(v) + 1. clz is a single CPU instruction on most architectures. So bitLen is constant time.

HDR uses bitLen to determine the "exponent" portion of the bucket. The "sub-bucket" portion is the next-few-bits of the value (after removing the leading 1 implied by bitLen).

Concretely, with subBucketHalfCountMagnitude = 11 (subBucketHalfCount = 2048):

bucketIndex(v):
    if v < subBucketHalfCount:
        return v
    pow2 = bitLen(v) - 1
    if pow2 < subBucketHalfCountMagnitude:
        sub = v - subBucketHalfCount
        return sub
    bucket = pow2 - subBucketHalfCountMagnitude
    sub = (v >> bucket) - subBucketHalfCount
    return bucket * subBucketHalfCount * 2 + sub

(Simplified — real HDR has more nuance.)

The cost: ~10 CPU cycles. Constant time, independent of value or bucket count.

Bucket value range¶

Given a bucket index i, what range of values does it cover? The inverse calculation:

valueFromIndex(i):
    bucket = i / subBucketHalfCount - 1
    sub = i % subBucketHalfCount + subBucketHalfCount
    return sub << max(0, bucket)

The width of bucket i grows with the exponent. At the bottom of the range, widths are 1; at the top, widths are large but the relative error is still bounded by 2^-11 ≈ 0.05%.

Resolution vs memory¶

Significant digits 3 (default) ≈ relative error 0.1% ≈ subBucketCount 2048. Each "decade" of values has 2048 buckets. For 17 decades, that's ~34K buckets × 8 bytes = 272 KB.

For 4 significant digits, double the sub-bucket count, double the memory.

Most workloads use 3 sig digits. Some critical-tail-latency workloads use 4.

Implications¶

Knowing the bucket layout helps you:

• Debug "why is my p99 in bucket X?" — compute valueFromIndex(X).
• Estimate memory cost for new histograms.
• Understand why some bucket widths are 1 ns and others are seconds.
• Know that recording is truly constant time.

Deep Dive: Coordinated Omission in Practice¶

Coordinated omission (CO) is the silent killer of latency measurements. Walk through a concrete example.

The setup¶

A load generator fires 1000 requests/sec, expects each to take 1 ms.

ticker := time.NewTicker(time.Millisecond)
for range ticker.C {
    start := time.Now()
    doRequest()
    latency := time.Since(start)
    histogram.RecordValue(latency.Nanoseconds())
}

The problem¶

Suppose request #500 takes 100 ms instead of 1 ms. While that one request is running, the ticker fires 99 more times — and the loop is blocked. So instead of recording 1 slow request and 99 fast ones, you record 1 100ms-request followed by 99 "fast" requests (1 ms each) that were supposed to fire during the 100 ms blockage but didn't.

Result: histogram shows p99 = ~100 ms (one slow value out of ~1000). Reality: every request during the 100 ms blockage was also effectively slow (waiting + processing). True p99 should be far higher.

HDR's fix¶

RecordValueWithExpectedInterval(value, expected):

• Records the actual value once.
• For each expected interval that elapsed beyond the actual measurement, records value - expected, value - 2*expected, ... to fill in the missing observations.

histogram.RecordValueWithExpectedInterval(latency.Nanoseconds(), time.Millisecond.Nanoseconds())

Now the histogram includes synthesised observations for the 99 missing requests, each at their "actual" elapsed-time-since-they-should-have-fired.

When to use¶

Whenever your measurement loop is itself rate-limited (firing on a ticker, processing from a channel with a known rate). Most production HTTP handlers do not coordinate-omit — each request is fired by an independent client, so blocking one handler does not delay others.

But anywhere you simulate load with a fixed-rate generator, CO is a risk. HDR's API gives you the fix.

Without CO compensation¶

Tools like wrk2 (a wrk fork by Gil Tene) build CO compensation into the load generator itself. Use them for accurate load testing.

Deep Dive: Building a Multi-Metric Snapshot System¶

A production observability system often needs coherent snapshots of many counters and histograms at "the same moment". For most monitoring, this is overkill — Prometheus scrapes a process every 15 s and computes rates and percentiles independently. But for special cases (correlating across metrics in real time), you need a coherent snapshot.

type SystemSnapshot struct {
    At              time.Time
    Requests        int64
    Errors          int64
    InFlight        int64
    BytesIn         int64
    BytesOut        int64
    LatencyP50      int64
    LatencyP99      int64
    GCPauseNs       uint64
    HeapAllocBytes  uint64
    Goroutines      int
}

type System struct {
    requests *Sharded
    errors   *Sharded
    inflight atomic.Int64
    bytesIn  *Sharded
    bytesOut *Sharded
    latency  *LatencyMetric
    snap     atomic.Pointer[SystemSnapshot]
}

func (s *System) Refresh() {
    var ms runtime.MemStats
    runtime.ReadMemStats(&ms)
    merged := s.latency.merged()
    s.snap.Store(&SystemSnapshot{
        At:             time.Now(),
        Requests:       s.requests.Get(),
        Errors:         s.errors.Get(),
        InFlight:       s.inflight.Load(),
        BytesIn:        s.bytesIn.Get(),
        BytesOut:       s.bytesOut.Get(),
        LatencyP50:     merged.ValueAtQuantile(50),
        LatencyP99:     merged.ValueAtQuantile(99),
        GCPauseNs:      ms.PauseTotalNs,
        HeapAllocBytes: ms.HeapAlloc,
        Goroutines:     runtime.NumGoroutine(),
    })
}

func (s *System) Current() *SystemSnapshot {
    return s.snap.Load()
}

func (s *System) Run(ctx context.Context, interval time.Duration) {
    t := time.NewTicker(interval)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-t.C:
            s.Refresh()
        }
    }
}

The publisher runs in a single goroutine; it does the merge work once per refresh. Readers (HTTP /status handler, in-process consumers) do one Load. Memory cost: ~100 bytes per snapshot, allocated per refresh.

For higher-precision needs, you would version the snapshot and ensure all underlying counters are read between consistent versions. Rarely worth the complexity.

Deep Dive: Histogram Aggregation Across the Fleet¶

A 1000-instance Go fleet, each instance emits a latency histogram. The dashboard wants a fleet-wide p99.

Prometheus way¶

Each instance exposes its histogram bucket counts:

http_duration_seconds_bucket{le="0.001"} 1234
http_duration_seconds_bucket{le="0.005"} 5678
http_duration_seconds_bucket{le="0.01"}  9012
http_duration_seconds_bucket{le="+Inf"}  10000

Prometheus scrapes all instances. PromQL:

histogram_quantile(0.99, sum(rate(http_duration_seconds_bucket[5m])) by (le))

Reads:

• rate(http_duration_seconds_bucket[5m]) — per-instance, per-bucket rate over 5 minutes.
• sum(...) by (le) — sum across instances, grouped by bucket boundary.
• histogram_quantile(0.99, ...) — compute p99 from the summed bucket counts.

The result is the fleet-wide p99 over the last 5 minutes. Correct because bucket counts are additive.

OpenTelemetry way¶

Each instance pushes histogram observations via OTLP. The collector aggregates across instances. Same math, different transport.

Custom way¶

You roll your own metric backend (rare; usually a bad idea). You collect histograms from each instance, merge bucket counts, compute quantile. Same math.

Don't average¶

Whatever you do, do not average per-instance p99s. The fleet p99 is not the average of per-instance p99s. The merge-then-quantile approach is the only correct one.

Deep Dive: Custom Exposition Format¶

Sometimes you need a custom exposition format — e.g., to push to a corporate logging system that doesn't speak Prometheus.

type SinkLine struct {
    Name      string
    Value     float64
    Timestamp time.Time
    Labels    map[string]string
}

type Sink interface {
    Emit(line SinkLine) error
}

type Exporter struct {
    sink Sink
    reg  *Registry
}

func (e *Exporter) Run(ctx context.Context, interval time.Duration) {
    t := time.NewTicker(interval)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case now := <-t.C:
            e.reg.ForEach(func(name string, m Metric) {
                switch v := m.(type) {
                case *Counter:
                    e.sink.Emit(SinkLine{Name: name, Value: float64(v.Value()), Timestamp: now})
                case *Gauge:
                    e.sink.Emit(SinkLine{Name: name, Value: float64(v.Value()), Timestamp: now})
                case *LatencyMetric:
                    h := v.merged()
                    e.sink.Emit(SinkLine{Name: name + "_p50", Value: float64(h.ValueAtQuantile(50)), Timestamp: now})
                    e.sink.Emit(SinkLine{Name: name + "_p99", Value: float64(h.ValueAtQuantile(99)), Timestamp: now})
                    e.sink.Emit(SinkLine{Name: name + "_count", Value: float64(h.TotalCount()), Timestamp: now})
                }
            })
        }
    }
}

The exporter walks the registry, encodes each metric to a sink-specific format, emits. Run it as a goroutine; the sink can be a UDP socket, an HTTP POST to a corporate endpoint, or anything else.

This is exactly the StatsD pattern (UDP push). For modern systems, prefer OTLP, but custom exporters remain useful for legacy integrations.

Deep Dive: Persistent Counters¶

For business-critical counts (orders placed, credits used), you cannot rely on in-memory atomics across process restarts. Persistence patterns:

Pattern 1: Atomic + periodic database flush¶

type PersistentCounter struct {
    inMem   atomic.Int64
    db      *sql.DB
    key     string
    flushAt time.Duration
}

func (p *PersistentCounter) Inc() { p.inMem.Add(1) }

func (p *PersistentCounter) Get() int64 {
    var dbVal int64
    p.db.QueryRow("SELECT v FROM counters WHERE k=$1", p.key).Scan(&dbVal)
    return dbVal + p.inMem.Load()
}

func (p *PersistentCounter) Flush() error {
    delta := p.inMem.Swap(0)
    _, err := p.db.Exec("UPDATE counters SET v=v+$1 WHERE k=$2", delta, p.key)
    if err != nil {
        // Roll back in-memory delta if we failed to persist.
        p.inMem.Add(delta)
        return err
    }
    return nil
}

func (p *PersistentCounter) Run(ctx context.Context) {
    t := time.NewTicker(p.flushAt)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            p.Flush()
            return
        case <-t.C:
            p.Flush()
        }
    }
}

Trade-offs:

• Throughput high (in-memory increment).
• Loses up to one flush interval of data on crash.
• Database is the source of truth.

Pattern 2: Write-ahead log¶

Every increment writes a log entry; periodically the log is compacted. Survives crashes; higher write cost per increment.

Pattern 3: Distributed counter service¶

A dedicated service (e.g., Redis with INCR) owns the counter. Other processes RPC to it. Atomic across processes; expensive per-increment.

For most metrics, persistence is unnecessary — counters reset on restart, and rate-derived metrics are robust to resets. For billing-grade counters, choose pattern 1 or 3 depending on requirements.

Deep Dive: Counter Telemetry for the Counter System Itself¶

The metrics subsystem itself should be observable. Counters about counters:

• metrics_counter_inc_total — total increments across all counters
• metrics_counter_inc_dropped_total — increments dropped due to cardinality limits
• metrics_registry_size — number of registered metrics
• metrics_registry_unique_series — unique label combinations
• metrics_scrape_duration_seconds — how long scrapes take
• metrics_histogram_record_duration_ns — histogram record latency

These are all themselves counters/gauges. Use them to detect:

• Cardinality bombs (rapid growth in unique series)
• Scrape timeouts (high scrape duration)
• Hot counters (high inc rate on one metric)
• Lost data (high dropped count)

Pre-register these at process startup. Operators rely on them when diagnosing observability outages.

Deep Dive: Building a SLO-Driven Alert from Counters¶

A Service Level Objective (SLO) of "99.9% of requests succeed" maps to two counters:

• http_requests_total — all requests
• http_errors_total — failed requests

The SLO says: errors / requests < 0.001 over the SLO window (often 30 days).

In Prometheus, the SLO query:

1 - (
  sum(rate(http_errors_total[30d]))
  /
  sum(rate(http_requests_total[30d]))
)

Alert on SLO budget exhaustion: if 99% of the 30-day budget is consumed in the first 24 hours, page. The math is "budget burn rate":

sum(rate(http_errors_total[1h])) / sum(rate(http_requests_total[1h])) > 0.01 * 24

Two counters drive an entire SLO. Counter design quality directly affects SLO accuracy.

Deep Dive: Counters and Distributed Tracing¶

OpenTelemetry's exemplars attach a trace ID to a histogram observation:

histogram.Observe(latency.Seconds(),
    attribute.String("status", "ok"),
    metric.WithExemplarTrace(traceID))

Now when you see an outlier in a histogram, you can jump to the specific trace that produced it. Killer feature.

Implementation: each histogram bucket stores not only a count but also a few "exemplar" observations (with their trace IDs). Adds memory but enables the jump.

Prometheus supports exemplars in OpenMetrics format. The Go OpenTelemetry SDK has full exemplar support.

Deep Dive: Counter Considerations for Multi-Tenant Services¶

A service that serves multiple tenants needs per-tenant counters. Choices:

Per-tenant labels¶

counter.With(labels.Tenant(tenantID)).Inc()

Cardinality risk if tenants are numerous. Bound or sample.

Per-tenant counter objects¶

counters := tenants.GetOrCreate(tenantID)
counters.Requests.Inc()

No label cardinality issue in the metric backend (each tenant has its own counter namespace). But memory grows with tenant count.

Hybrid¶

Top-tenant labels (most-active tenants) get their own counters; long-tail tenants are aggregated. Bounded cardinality with detail on the important tenants.

Tenant isolation¶

A noisy tenant should not affect another tenant's counter measurements. Per-tenant counters with separate cell arrays achieve this.

Deep Dive: Hot Patching Counter Behaviour¶

In a long-running service, you may want to change counter behaviour without restart:

• Change cardinality limit
• Add/remove a label
• Change shard count
• Switch between counter and histogram

The runtime hot-patch pattern:

type RegistryConfig struct {
    MaxCardinality int
    DefaultShards  int
}

var config atomic.Pointer[RegistryConfig]

func init() {
    config.Store(&RegistryConfig{MaxCardinality: 10000, DefaultShards: 64})
}

func (r *Registry) reload(c *RegistryConfig) {
    config.Store(c)
}

A config endpoint accepts new config, calls reload. New increments respect the new config; existing data stays. For more invasive changes (new label, new metric), restart is usually simpler.

Deep Dive: Counter Performance Across Go Versions¶

Go's sync/atomic performance has evolved:

• Go 1.0: basic atomic functions, alignment-sensitive on 32-bit.
• Go 1.19: typed atomic.Int64/atomic.Uint64/atomic.Pointer[T]. Compiler enforces alignment.
• Go 1.20: improved inlining of atomic operations.
• Go 1.22: further inlining; faster atomic.Add on ARM64.

Re-benchmark when upgrading. Sometimes a Go upgrade improves counter throughput materially with no code change.

Deep Dive: Counter Compression in Storage¶

If you persist counters or transmit them in bulk, compression matters. A counter time series is monotonic-mostly; gorilla compression (Facebook's time-series compression) achieves ~1.5 bytes per sample for typical metrics. Prometheus uses it; so does any modern TSDB.

You do not implement it yourself. But understanding it helps you size storage budgets:

• 1 metric × 1 sample / 15 s × 30 days = 172,800 samples × 1.5 bytes ≈ 260 KB.
• 1000 metrics × 1000 instances = 1M time series × 260 KB = 260 GB per month.

For dozens of instances and hundreds of metrics, storage is cheap. For thousands of each, plan carefully.

Deep Dive: Counter Privacy¶

Counters often contain hints about individual user behaviour. Examples:

• user_logins_total{user_id="alice"} — explicit PII.
• requests_by_country{country="GB"} — coarser PII (combinable).
• error_rate{path="/users/alice/settings"} — path-based PII leak.

GDPR-compliant counter design:

1. Audit every label for PII.
2. Hash or bucket high-cardinality labels.
3. Set retention policies on metric storage.
4. Provide data-subject-access tooling for counter data.

The same care you take with logs applies to metrics.

Deep Dive: Counter Validation¶

A counter "validator" detects malformed values:

• Negative deltas on monotonic counters
• Out-of-range histogram observations
• Excessive label values
• Sudden 10× rate changes (anomaly)

type ValidatedCounter struct {
    name      string
    v         atomic.Int64
    minDelta  int64
    maxDelta  int64
    anomaly   anomalyDetector
}

func (c *ValidatedCounter) Add(delta int64) {
    if delta < c.minDelta || delta > c.maxDelta {
        log.Printf("counter %s: delta %d out of range", c.name, delta)
        return
    }
    c.v.Add(delta)
    c.anomaly.observe(delta)
}

Pay the validation cost on increment if your input is untrusted; pay it at scrape time otherwise.

Deep Dive: Counter Migrations¶

Renaming a counter or changing its semantics is operationally painful:

• Dashboards reference the old name.
• Alerts depend on the old name.
• Runbooks document the old name.

Migration pattern:

1. Add the new counter alongside the old. Both increment in parallel.
2. Dual-display in dashboards. Verify the new counter matches the old.
3. Migrate alerts to the new counter.
4. Update runbooks.
5. Remove the old counter.

Steps 2-4 may take weeks. Do not rush.

Deep Dive: Counter A/B Testing¶

Splitting traffic between two code paths and comparing counter outputs:

if hash(userID) < threshold {
    counterA.Inc()
    pathA(req)
} else {
    counterB.Inc()
    pathB(req)
}

Both counters feed the same metric backend; the experiment dashboard plots them side by side. Statistically-significant differences validate the experiment.

For more sophisticated experiments, use a dedicated experimentation framework (LaunchDarkly, Optimizely, internal). Counters are still the foundation.

Deep Dive: Counters and Replay¶

A counter built on top of an event log can be replayed:

type ReplayableCounter struct {
    v atomic.Int64
}

func (c *ReplayableCounter) Apply(event Event) {
    if event.Type == "increment" {
        c.v.Add(1)
    }
}

func Replay(c *ReplayableCounter, log []Event) {
    for _, e := range log {
        c.Apply(e)
    }
}

This is event sourcing. The counter is derived from the log; can be reconstructed at any point in time. Powerful for audit, debugging, and recovery.

Deep Dive: HDR Histogram Edge Cases¶

Recording values below lowest¶

HDR's lowest parameter defines the minimum trackable value. Recording below it depends on the implementation:

• Some HDR ports clamp to lowest.
• Others drop the observation.
• Others extend the lowest bucket.

In hdrhistogram-go, recording 0 (when lowest=1) is dropped. Be aware.

Recording values above highest¶

Always dropped or clamped to highest. Configure highest generously (e.g., 60s for HTTP latency, even if you expect 1s).

Significant digits¶

Higher precision → more memory → slower record? No, record time is constant. Only memory and merge time grow.

Merging histograms with different bounds¶

You cannot directly merge histograms with different lowest/highest values. Reshape one to match the other first.

Resetting¶

Most HDR implementations have a Reset() method that zeros all bucket counts. For sliding-window use, reset the oldest bucket each tick.

Snapshotting¶

hdrhistogram-go provides Snapshot() which returns a deep copy. Use it when you want a point-in-time copy that does not change as new observations arrive.

Deep Dive: Production Postmortem Patterns Involving Counters¶

Common postmortem causes:

"We didn't see the error rate climbing"¶

Counter exists but no alert is wired to it. Fix: every important counter has an alert.

"The counter said zero but errors were happening"¶

Counter only increments on a specific code path; errors flowed through a different path. Fix: counter at the entry point of all paths.

"We averaged p99s"¶

Misuse of percentiles. Fix: use histogram_quantile in PromQL.

"Cardinality bomb"¶

A label was user-controlled. Fix: bound cardinality or remove the label.

"Counter wrapped past int32 max"¶

Used the wrong type. Fix: int64 by default.

"Counter reset on deploy hid a regression"¶

rate() handles resets correctly; ad-hoc dashboards may not. Fix: use PromQL rate().

"Sloppy counter lost data on crash"¶

A sloppy counter for billing. Fix: choose persistence; sloppy counters are not durable.

Postmortems on counter-related incidents drive future design decisions.

Deep Dive: A Reference Implementation Sketch of a Full Subsystem¶

package observability

import (
    "context"
    "io"
    "net/http"
    "runtime"
    "sync"
    "sync/atomic"
    "time"

    "github.com/HdrHistogram/hdrhistogram-go"
    "golang.org/x/sys/cpu"
)

// Subsystem is the root observability container for a Go service.
// One per process; passed via context.
type Subsystem struct {
    mu       sync.RWMutex
    counters map[string]*shardedCounter
    gauges   map[string]*atomic.Int64
    hists    map[string]*hdrShard
    info     map[string]string
}

func New() *Subsystem {
    return &Subsystem{
        counters: map[string]*shardedCounter{},
        gauges:   map[string]*atomic.Int64{},
        hists:    map[string]*hdrShard{},
        info:     map[string]string{},
    }
}

// --- Counter ---

func (s *Subsystem) Counter(name, help string) *Counter {
    s.mu.Lock()
    defer s.mu.Unlock()
    if c, ok := s.counters[name]; ok {
        return (*Counter)(c)
    }
    c := newShardedCounter()
    s.counters[name] = c
    s.info[name] = help
    return (*Counter)(c)
}

type Counter shardedCounter

func (c *Counter) Inc() {
    (*shardedCounter)(c).inc()
}

func (c *Counter) Add(n int64) {
    (*shardedCounter)(c).add(n)
}

func (c *Counter) Value() int64 {
    return (*shardedCounter)(c).get()
}

type shardedCell struct {
    _ cpu.CacheLinePad
    v atomic.Int64
    _ cpu.CacheLinePad
}

type shardedCounter struct {
    cells []shardedCell
}

func newShardedCounter() *shardedCounter {
    return &shardedCounter{cells: make([]shardedCell, 64)}
}

func (s *shardedCounter) inc() {
    s.cells[runtime_fastrand()%64].v.Add(1)
}

func (s *shardedCounter) add(n int64) {
    s.cells[runtime_fastrand()%64].v.Add(n)
}

func (s *shardedCounter) get() int64 {
    var t int64
    for i := range s.cells {
        t += s.cells[i].v.Load()
    }
    return t
}

// --- Gauge ---

func (s *Subsystem) Gauge(name, help string) *Gauge {
    s.mu.Lock()
    defer s.mu.Unlock()
    if g, ok := s.gauges[name]; ok {
        return (*Gauge)(g)
    }
    var g atomic.Int64
    s.gauges[name] = &g
    s.info[name] = help
    return (*Gauge)(&g)
}

type Gauge atomic.Int64

func (g *Gauge) Inc()             { (*atomic.Int64)(g).Add(1) }
func (g *Gauge) Dec()             { (*atomic.Int64)(g).Add(-1) }
func (g *Gauge) Set(n int64)      { (*atomic.Int64)(g).Store(n) }
func (g *Gauge) Value() int64     { return (*atomic.Int64)(g).Load() }

// --- Histogram ---

func (s *Subsystem) Histogram(name, help string) *Histogram {
    s.mu.Lock()
    defer s.mu.Unlock()
    if h, ok := s.hists[name]; ok {
        return (*Histogram)(h)
    }
    h := newHDRShard()
    s.hists[name] = h
    s.info[name] = help
    return (*Histogram)(h)
}

type Histogram hdrShard

type hdrShard struct {
    shards []histShardCell
}

type histShardCell struct {
    _ cpu.CacheLinePad
    mu sync.Mutex
    h  *hdrhistogram.Histogram
    _ cpu.CacheLinePad
}

func newHDRShard() *hdrShard {
    n := runtime.GOMAXPROCS(0)
    h := &hdrShard{shards: make([]histShardCell, n)}
    for i := range h.shards {
        h.shards[i].h = hdrhistogram.New(1, 60_000_000_000, 3)
    }
    return h
}

func (h *Histogram) Observe(v int64) {
    s := &(*hdrShard)(h).shards[runtime_fastrand()%uint32(len((*hdrShard)(h).shards))]
    s.mu.Lock()
    s.h.RecordValue(v)
    s.mu.Unlock()
}

func (h *Histogram) Quantile(q float64) int64 {
    merged := h.merged()
    return merged.ValueAtQuantile(q * 100)
}

func (h *Histogram) merged() *hdrhistogram.Histogram {
    out := hdrhistogram.New(1, 60_000_000_000, 3)
    for i := range (*hdrShard)(h).shards {
        (*hdrShard)(h).shards[i].mu.Lock()
        out.Merge((*hdrShard)(h).shards[i].h)
        (*hdrShard)(h).shards[i].mu.Unlock()
    }
    return out
}

// --- Exposition: Prometheus text ---

func (s *Subsystem) WritePrometheus(w io.Writer) {
    s.mu.RLock()
    defer s.mu.RUnlock()
    for name, c := range s.counters {
        fmt.Fprintf(w, "# HELP %s %s\n# TYPE %s counter\n%s %d\n",
            name, s.info[name], name, name, c.get())
    }
    for name, g := range s.gauges {
        fmt.Fprintf(w, "# HELP %s %s\n# TYPE %s gauge\n%s %d\n",
            name, s.info[name], name, name, g.Load())
    }
    for name, h := range s.hists {
        merged := (&hdrShard{shards: h.shards})
        snap := func() *hdrhistogram.Histogram {
            out := hdrhistogram.New(1, 60_000_000_000, 3)
            for i := range merged.shards {
                merged.shards[i].mu.Lock()
                out.Merge(merged.shards[i].h)
                merged.shards[i].mu.Unlock()
            }
            return out
        }()
        fmt.Fprintf(w, "# HELP %s %s\n# TYPE %s histogram\n", name, s.info[name], name)
        for _, p := range []int{50, 90, 95, 99, 999} {
            fmt.Fprintf(w, "%s{quantile=\"%.3f\"} %d\n", name, float64(p)/1000, snap.ValueAtQuantile(float64(p)/10))
        }
        fmt.Fprintf(w, "%s_count %d\n", name, snap.TotalCount())
    }
}

// --- HTTP exposition ---

func (s *Subsystem) Handler() http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "text/plain")
        s.WritePrometheus(w)
    })
}

//go:linkname runtime_fastrand runtime.fastrand
func runtime_fastrand() uint32