time.Ticker — Professional Level (Production Patterns)¶

Table of Contents¶

1. Introduction
2. The Production Mental Model
3. Heartbeats: The Most Common Use Case
4. Telemetry Flush Loops
5. Jittered Tick Across Replicas
6. Backpressure on Slow Consumers
7. Observability: Metrics for Skipped Ticks
8. Drift Correction and Wall-Clock Alignment
9. Coalescing Many Tickers Into One
10. Per-Connection Tickers at Scale
11. Graceful Shutdown of Ticker-Driven Loops
12. Testing Ticker-Driven Code
13. Production Incident #1: The Stampede at 00:00:00 UTC
14. Production Incident #2: The Quiet Heartbeat
15. Production Incident #3: The Cache Stampede
16. Production Incident #4: The Drifting Sampler
17. Production Incident #5: The Leaking Ticker
18. Production Incident #6: The Backpressure Cliff
19. Production Incident #7: The Reset Race
20. Patterns for Distributed Schedulers
21. Ticker vs Alternatives in Production
22. Operational Runbook
23. Self-Assessment
24. Summary

Introduction¶

A time.Ticker looks innocent in a code review. Three lines: open it, drain its channel, defer Stop. In production those three lines turn into incidents. A heartbeat ticker on ten thousand pods aligned to the second creates a thundering herd against the control plane. A telemetry flush ticker on a node with a slow uplink quietly drops ticks until the buffer at the receiver overflows and the entire metrics pipeline stalls. A reset call on an active ticker leaves a stale value in t.C and the next iteration of the loop fires instantly, double-billing whatever the loop was throttling. A forgotten Stop on a long-lived background worker leaks a few hundred timers per request and the timer heap grows until runtime.adjustTimers shows up at the top of a CPU profile.

This document is a field manual. It is not about how time.Ticker works internally — the senior page covers the four-heap scheduler, the channel buffer change in Go 1.23, the assembly path of runtime.tickerproc. This file is about how to operate a ticker-driven system: how to shape the load it generates, how to monitor it, how to recover from the kinds of failures that show up at three in the morning when a region is melting and the on-call has been paged for the fifth time in an hour. Every pattern here was paid for by a real outage. Every incident here happened somewhere, sometimes more than once.

We assume you already know the API. time.NewTicker(d) *Ticker returns a ticker that delivers the current time on t.C every d. t.Stop() halts further delivery but does not close t.C and does not drain pending values. t.Reset(d) changes the interval. As of Go 1.23 the ticker is GC-friendly: a ticker whose goroutine is no longer referencing it can be collected without an explicit Stop call, although you should still call Stop because (a) it makes intent explicit, (b) it works on older Go versions, and (c) it lets the runtime reclaim the timer immediately rather than waiting for the next GC cycle.

The patterns are organised from the most common (heartbeats, telemetry) to the most subtle (drift correction, coalescing). The incidents at the end are presented as postmortems: timeline, root cause, fix, and lessons. Read them. They are how the patterns earned the right to be here.

The Production Mental Model¶

Before any pattern, internalise four invariants that distinguish a toy ticker from a production ticker.

Invariant 1: The channel is a buffered slot of size one. From Go 1.23 onwards t.C has capacity 1 (make(chan Time, 1)). The runtime delivers a tick by trying a non-blocking send; if the slot is full the tick is dropped. There is no queue of pending ticks. If your handler takes longer than the interval, every overrun tick after the first is silently lost. You will not see them as a backlog. You will see them as "the metric is flat".

Invariant 2: Delivery time is best-effort, not deadline. The runtime guarantees that the next delivery is at least d after the previous one. It does not guarantee that it is exactly d, or even close to d under scheduler pressure. On a busy machine a 100ms ticker can deliver at 105ms, then 98ms, then 250ms when GC stops the world. Your design must tolerate jitter from below and lateness from above.

Invariant 3: Wall-clock and monotonic are different. time.Now() returns both a wall reading and a monotonic reading. The ticker fires on monotonic time, which is immune to NTP slew and DST. If you compare the value received on t.C to a wall-clock deadline elsewhere, you may compare unlike quantities. Use time.Since and time.Until rather than subtracting time.Time values; they prefer monotonic readings if both sides have one.

Invariant 4: A ticker holds a goroutine reference until Stop. Pre-1.23, an unreferenced but un-stopped ticker leaks the goroutine that owns its timer along with one heap entry per ticker. Post-1.23, an unreachable ticker can be GC'd, but the GC may not run for a while. In a 10k-RPS service that creates one ticker per request, "a while" is enough heap pressure to matter. Always defer t.Stop().

We will return to these invariants throughout. They are the bedrock under every pattern.

Heartbeats: The Most Common Use Case¶

A heartbeat is a periodic message from a worker to a coordinator saying "I am alive and making progress." Examples: a pod posting to a service registry, a consumer renewing a lease against a distributed lock, a long-running task reporting progress to a watcher goroutine.

Baseline implementation¶

package heartbeat

import (
    "context"
    "log/slog"
    "time"
)

type Reporter interface {
    Report(ctx context.Context, payload Payload) error
}

type Payload struct {
    NodeID    string
    Sequence  uint64
    Timestamp time.Time
    Healthy   bool
}

// Run sends a heartbeat every interval until ctx is cancelled.
// It blocks; callers should run it in its own goroutine.
func Run(ctx context.Context, r Reporter, nodeID string, interval time.Duration, log *slog.Logger) error {
    t := time.NewTicker(interval)
    defer t.Stop()

    var seq uint64
    // Send one immediately so consumers see liveness without waiting a full interval.
    if err := r.Report(ctx, Payload{NodeID: nodeID, Sequence: seq, Timestamp: time.Now(), Healthy: true}); err != nil {
        log.Warn("initial heartbeat failed", "err", err)
    }
    seq++

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            if err := r.Report(ctx, Payload{NodeID: nodeID, Sequence: seq, Timestamp: now, Healthy: true}); err != nil {
                log.Warn("heartbeat failed", "err", err, "seq", seq)
                // Do not return; transient failures are normal.
            }
            seq++
        }
    }
}

Three things in the baseline that matter:

1. defer t.Stop() is on the line after NewTicker. Never separate them by more than one line of code. If a panic happens between NewTicker and defer, you leak the ticker.
2. The first heartbeat fires before the loop. Consumers should not have to wait interval seconds to observe liveness for the first time. Sending immediately also lets you detect a misconfigured reporter on startup rather than on the first tick.
3. A failed report is logged, not returned. A heartbeat that crashes on a transient HTTP 503 is worse than no heartbeat — the worker would be marked dead while it is still working. The loop continues; the consumer's job is to debounce.

Why a heartbeat is harder than it looks¶

A naïve heartbeat loop has at least seven failure modes:

Each pattern in this section addresses at least one of these. The baseline above only solves the "leak on shutdown" line, because of the defer t.Stop() and the ctx.Done() branch.

Heartbeats with deadline-based delivery¶

If the consumer cares about "heartbeat received within X seconds of expected time", give the reporter a deadline so it does not block forever on a slow coordinator. The deadline must be shorter than the interval, otherwise reports overlap.

func RunWithDeadline(ctx context.Context, r Reporter, nodeID string, interval time.Duration, log *slog.Logger) error {
    if interval <= 0 {
        return errors.New("heartbeat: interval must be positive")
    }
    // The deadline is half the interval. A late report is no use; abort it.
    reportTimeout := interval / 2
    if reportTimeout < 100*time.Millisecond {
        reportTimeout = 100 * time.Millisecond
    }

    t := time.NewTicker(interval)
    defer t.Stop()

    var seq uint64
    send := func(now time.Time) {
        rctx, cancel := context.WithTimeout(ctx, reportTimeout)
        defer cancel()
        if err := r.Report(rctx, Payload{NodeID: nodeID, Sequence: seq, Timestamp: now, Healthy: true}); err != nil {
            log.Warn("heartbeat failed", "err", err, "seq", seq)
        }
        seq++
    }
    send(time.Now())

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            send(now)
        }
    }
}

Two notes:

• reportTimeout := interval / 2 is a heuristic. The exact ratio depends on the coordinator's SLO. For a 30s interval with a coordinator that occasionally takes 8s, set the timeout to 10s. The rule of thumb: timeout < interval, and (timeout × max-concurrent-retries) < interval.
• We use the tick value now (the runtime's view of when the tick fired) rather than calling time.Now() inside the loop. This is not just stylistic; on a busy system time.Now() may be a few milliseconds after the tick, and propagating that bias every iteration accumulates drift if a downstream system uses the timestamp to compute deltas.

Heartbeats with an outgoing buffer¶

If the report path is asynchronous — say, a message queue or an in-memory channel sent to another goroutine that batches — the heartbeat loop should not block on a full buffer. Drop the heartbeat instead.

func RunBuffered(ctx context.Context, out chan<- Payload, nodeID string, interval time.Duration, log *slog.Logger) error {
    t := time.NewTicker(interval)
    defer t.Stop()
    var seq, dropped uint64

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            p := Payload{NodeID: nodeID, Sequence: seq, Timestamp: now, Healthy: true}
            select {
            case out <- p:
                // sent
            default:
                dropped++
                log.Warn("heartbeat buffer full, dropping",
                    "dropped_total", dropped, "seq", seq)
            }
            seq++
        }
    }
}

The pattern select { case out <- p: default: } is the canonical Go idiom for a non-blocking send. It maps cleanly to the production constraint: a heartbeat is throwaway data; missing one is far less bad than blocking the heartbeat goroutine, which is exactly the situation we are trying to avoid by sending heartbeats in the first place.

The dropped counter belongs in a metric. We will return to that in Observability: Metrics for Skipped Ticks.

Heartbeats with grace on transient failures¶

A heartbeat that posts to a coordinator over HTTP is going to fail occasionally. The naïve "log and continue" is fine if the coordinator's debouncing handles a few drops. If the coordinator marks a worker dead after one missed heartbeat, you need retries.

type retryReporter struct {
    inner Reporter
    log   *slog.Logger
}

func (r retryReporter) Report(ctx context.Context, p Payload) error {
    var lastErr error
    for attempt := 0; attempt < 3; attempt++ {
        if err := r.inner.Report(ctx, p); err == nil {
            return nil
        } else {
            lastErr = err
        }
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(time.Duration(100*(1<<attempt)) * time.Millisecond):
        }
    }
    return lastErr
}

Two notes on time.After here:

1. time.After allocates a fresh timer each call. Inside a select that loops thousands of times per second this is a problem; inside a heartbeat retry that runs at most three times per interval it is fine.
2. The backoff is 100ms, 200ms, 400ms. Total worst case is 700ms plus three round-trips. Make sure that is less than interval/2 if you also have a per-call timeout.

Better is to share a time.Timer:

func (r retryReporter) Report(ctx context.Context, p Payload) error {
    timer := time.NewTimer(0)
    defer timer.Stop()
    if !timer.Stop() {
        <-timer.C
    }

    var lastErr error
    for attempt := 0; attempt < 3; attempt++ {
        if err := r.inner.Report(ctx, p); err == nil {
            return nil
        } else {
            lastErr = err
        }
        backoff := time.Duration(100*(1<<attempt)) * time.Millisecond
        timer.Reset(backoff)
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
        }
    }
    return lastErr
}

This is overkill for three retries per heartbeat. It would matter if you had a thousand retries per second; see Coalescing Many Tickers Into One.

Detecting your own heartbeat is broken¶

A common bug: the heartbeat loop runs forever, but the worker thread it is meant to monitor has died, so heartbeats report a worker that is no longer doing work. Solve by routing the heartbeat through the worker itself:

type Worker struct {
    work    chan unit
    healthy atomic.Bool
}

func (w *Worker) Process(ctx context.Context) {
    for {
        select {
        case <-ctx.Done():
            return
        case <-w.work:
            // do something
            w.healthy.Store(true)
        }
    }
}

func (w *Worker) Beat(ctx context.Context, r Reporter, interval time.Duration) error {
    t := time.NewTicker(interval)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            healthy := w.healthy.Swap(false) // reset; worker re-sets when it makes progress
            p := Payload{Timestamp: now, Healthy: healthy}
            r.Report(ctx, p)
        }
    }
}

Now a heartbeat with Healthy: false means the worker has not made progress since the last heartbeat. A consumer that sees three such heartbeats in a row should mark the worker stuck even though it is still posting.

This trick — heartbeat-as-liveness-probe-of-internal-state — is what separates a watchdog from a "this goroutine is still scheduled" signal. The latter is almost never what you want.

Telemetry Flush Loops¶

The second-most-common ticker pattern is "buffer metrics in memory and flush periodically." Examples: an in-process StatsD client, a Prometheus push-gateway client, an OpenTelemetry batch span processor, a log forwarder that uploads gzipped lines every five seconds.

Why flush periodically¶

Sending every event synchronously is expensive: a syscall per event, a round-trip per event, and a serialisation step per event. Batching amortises all three. The tradeoff is staleness: an event recorded at T appears at the receiver at T + flush-interval.

Reasonable flush intervals:

Baseline batch flusher¶

package metrics

import (
    "context"
    "log/slog"
    "sync"
    "time"
)

type Event struct {
    Name   string
    Value  float64
    Labels map[string]string
    Time   time.Time
}

type Sink interface {
    Send(ctx context.Context, batch []Event) error
}

type Flusher struct {
    sink     Sink
    interval time.Duration
    maxBatch int
    log      *slog.Logger

    mu     sync.Mutex
    buffer []Event

    in chan Event
}

func NewFlusher(sink Sink, interval time.Duration, maxBatch int, log *slog.Logger) *Flusher {
    return &Flusher{
        sink: sink, interval: interval, maxBatch: maxBatch, log: log,
        in: make(chan Event, 1024),
    }
}

// Record is the non-blocking producer side.
func (f *Flusher) Record(e Event) {
    select {
    case f.in <- e:
    default:
        f.log.Warn("telemetry: in-queue full, dropping event", "name", e.Name)
    }
}

// Run is the consumer side; one goroutine.
func (f *Flusher) Run(ctx context.Context) error {
    t := time.NewTicker(f.interval)
    defer t.Stop()

    flush := func() {
        f.mu.Lock()
        batch := f.buffer
        f.buffer = nil
        f.mu.Unlock()
        if len(batch) == 0 {
            return
        }
        sctx, cancel := context.WithTimeout(ctx, f.interval)
        defer cancel()
        if err := f.sink.Send(sctx, batch); err != nil {
            f.log.Warn("flush failed", "err", err, "events", len(batch))
        }
    }

    for {
        select {
        case <-ctx.Done():
            flush() // final flush on shutdown
            return ctx.Err()
        case <-t.C:
            flush()
        case e := <-f.in:
            f.mu.Lock()
            f.buffer = append(f.buffer, e)
            full := len(f.buffer) >= f.maxBatch
            f.mu.Unlock()
            if full {
                flush() // size-based flush even before the timer fires
            }
        }
    }
}

Things to notice:

• Two flush triggers: time and size. Either can fire first. Without size, a burst of events would balloon the buffer before the next tick.
• Final flush on ctx.Done(). Shutdown should not discard buffered data. If the shutdown is forced by a SIGKILL there is nothing you can do; for SIGTERM you have a few seconds to finish the upload.
• Mutex around buffer. The flush path takes the buffer atomically, leaving the producer free to append a new buffer. Holding the mutex during Send would block producers for the duration of the upload.
• Send timeout = interval. A send that takes longer than one interval means data accumulates faster than it ships. The timeout caps the damage; missing data is preferable to indefinite buffering.

Decoupling the ticker from the work¶

A subtle problem in the baseline: while flush() is running (which can take up to interval), the next tick can fire but is consumed by the case <-t.C: immediately when the previous handler returns. From the outside the second tick appears to be on time, but the actual flush rhythm is determined by how long each flush takes.

For most telemetry this is fine. For a strict-rhythm flush (e.g., billing rollups that must align to wall-clock minute boundaries), do the work in a separate goroutine and let the ticker stay punctual:

func (f *Flusher) RunRhythmic(ctx context.Context) error {
    t := time.NewTicker(f.interval)
    defer t.Stop()

    sem := make(chan struct{}, 1) // at most one flush in flight
    flush := func() {
        select {
        case sem <- struct{}{}:
        default:
            f.log.Warn("flush skipped, previous still running")
            return
        }
        go func() {
            defer func() { <-sem }()
            f.mu.Lock()
            batch := f.buffer
            f.buffer = nil
            f.mu.Unlock()
            if len(batch) == 0 {
                return
            }
            sctx, cancel := context.WithTimeout(ctx, f.interval)
            defer cancel()
            if err := f.sink.Send(sctx, batch); err != nil {
                f.log.Warn("flush failed", "err", err, "events", len(batch))
            }
        }()
    }

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-t.C:
            flush()
        }
    }
}

The sem channel of capacity one is a non-blocking lock: at most one flush is in flight. If a flush is already running when a tick fires, the tick is dropped (and logged). This is deliberate backpressure: we would rather skip a flush than queue them.

Streaming the flush¶

A third pattern, when batches are big and the sink supports it: stream rather than send. The ticker fires a "begin flush" signal; the producer keeps appending into a streaming write that ends when the next tick fires.

type StreamingSink interface {
    Begin(ctx context.Context) (StreamWriter, error)
}

type StreamWriter interface {
    Write(e Event) error
    Close() error
}

func (f *Flusher) RunStreaming(ctx context.Context) error {
    t := time.NewTicker(f.interval)
    defer t.Stop()

    var writer StreamWriter
    closeWriter := func() {
        if writer != nil {
            if err := writer.Close(); err != nil {
                f.log.Warn("close stream", "err", err)
            }
            writer = nil
        }
    }
    defer closeWriter()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-t.C:
            closeWriter()
            w, err := f.sink.Begin(ctx)
            if err != nil {
                f.log.Warn("begin stream", "err", err)
                continue
            }
            writer = w
        case e := <-f.in:
            if writer == nil {
                // No stream open; drop or buffer.
                continue
            }
            if err := writer.Write(e); err != nil {
                f.log.Warn("stream write", "err", err)
                closeWriter()
            }
        }
    }
}

This is the model used by AWS Kinesis Firehose, by Datadog's trace agent during high-cardinality periods, and by some Loki shippers. The advantage: no in-memory buffer growth; backpressure shows up immediately as Write errors. The disadvantage: the sink must support framed streaming, which excludes most HTTP-POST receivers.

What goes wrong with naive flushers¶

The single most common bug in flusher code is conflating "flush on tick" with "process input." If you write:

for range t.C {
    flush()
}

…then the only way new events enter the buffer is for Record to be a producer to a shared buffer protected by a lock. That works, but it means Record can block if the lock is held by flush. If flush takes 200ms, every Record during that window is stalled. On a hot path that records ten thousand events per second, you have just created a 200ms latency spike for every flush. The pattern above — separate input channel, ticker just signals a flush — avoids this entirely.

The second-most-common bug is forgetting the final flush. A process that exits cleanly via ctx.Done() should still upload its last batch. The defer t.Stop() does not do this; you need explicit code in the ctx.Done() branch.

The third-most-common bug is unbounded buffer growth. If Send is failing and you keep appending, you eat memory until OOM. Cap the buffer.

const maxBufferedEvents = 100_000

func (f *Flusher) recordSafe(e Event) {
    f.mu.Lock()
    defer f.mu.Unlock()
    if len(f.buffer) >= maxBufferedEvents {
        f.log.Warn("telemetry: buffer at cap, dropping", "name", e.Name)
        return
    }
    f.buffer = append(f.buffer, e)
}

Jittered Tick Across Replicas¶

A fleet of N replicas of the same service, each running the same ticker at the same interval, will (absent jitter) all fire at almost the same time. The variance comes only from startup time differences. After a few hours, NTP slew and the runtime's monotonic clock keep them within a few hundred microseconds of each other. The result is a coordinated burst against whatever downstream system they all hit.

If the downstream is a control plane, a cache, a database, a queue, or an external API, the burst translates to a periodic latency spike at the tick boundary. The latency spike causes timeouts on the slow replicas, which retry, which makes the spike worse. This pattern is called a stampede. It is the second most common cause of cascading failure I have personally seen, after retry-without-backoff.

Per-replica jitter¶

The fix is to add a per-replica random offset to the tick. Each replica picks a number uniformly between zero and the interval, sleeps that long once at startup, and then ticks at the interval.

func RunJittered(ctx context.Context, interval time.Duration, do func()) error {
    if interval <= 0 {
        return errors.New("interval must be positive")
    }
    // Uniform jitter in [0, interval).
    offset := time.Duration(rand.Int63n(int64(interval)))
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(offset):
    }
    t := time.NewTicker(interval)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-t.C:
            do()
        }
    }
}

After the initial jitter, every replica still ticks at exactly interval. They are now uniformly distributed within a single interval window: the load on the downstream is spread evenly.

Use a per-process *rand.Rand seeded from crypto/rand if you care that two processes started at the same wall second do not pick the same jitter. With Go 1.20+ and math/rand/v2, the default source is already seeded randomly per process, so a plain rand.Int63n is fine.

Per-tick jitter¶

A subtler variant: jitter every tick, not just the first one. This is used when you suspect the downstream has some other coordinated load — say, a database that runs a checkpoint every minute — and you want to avoid aligning with it consistently.

func RunPerTickJittered(ctx context.Context, base, jitter time.Duration, do func()) error {
    if base <= 0 {
        return errors.New("base must be positive")
    }
    timer := time.NewTimer(base)
    defer timer.Stop()
    for {
        // Reset the timer with base + random jitter.
        d := base + time.Duration(rand.Int63n(int64(jitter)))
        timer.Reset(d)
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
        }
        do()
    }
}

Notice this uses time.Timer (one-shot, reset every iteration) not time.Ticker. A ticker has a fixed interval; if you want jitter per tick you cannot use a ticker. The pattern is "timer-as-ticker."

Bounded jitter¶

Pure jitter [0, interval) is too aggressive for some use cases. You may want "tick at 60s but smear over a 5s window centred on minute boundaries." The pattern is:

func nextAt(now time.Time, base time.Duration, jitterWindow time.Duration) time.Duration {
    // Find the next multiple of base, then add jitter in [-jitterWindow/2, +jitterWindow/2].
    bucket := now.Truncate(base).Add(base)
    j := time.Duration(rand.Int63n(int64(jitterWindow))) - jitterWindow/2
    next := bucket.Add(j)
    if next.Before(now) {
        next = next.Add(base)
    }
    return next.Sub(now)
}

func RunAligned(ctx context.Context, base, jitter time.Duration, do func()) error {
    timer := time.NewTimer(nextAt(time.Now(), base, jitter))
    defer timer.Stop()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
        }
        do()
        timer.Reset(nextAt(time.Now(), base, jitter))
    }
}

This is what you want for "billing run every hour on the hour, but spread across replicas over a 60-second window." The bucket boundary is the wall-clock hour; the jitter offsets within ±30 seconds of that boundary.

Deterministic jitter¶

If you need reproducibility — for example, you want every replica's tick offset to be stable across restarts so a debugging session can predict when it will fire — derive the jitter from the replica's identity rather than a random source.

import "hash/fnv"

func deterministicJitter(replicaID string, interval time.Duration) time.Duration {
    h := fnv.New64a()
    h.Write([]byte(replicaID))
    sum := h.Sum64()
    return time.Duration(sum % uint64(interval))
}

If you rename a replica, its jitter changes. If two replicas have the same ID (rolling restart, replica reuses pod name) they get the same jitter, which defeats the purpose. Hash on hostname + start-time for safety.

Anti-pattern: jitter at the consumer¶

A natural mistake: instead of jittering the producer, jitter the consumer. "We'll have the receiver sleep a random amount before processing." This is worse, because the receiver still gets the burst, just delays it. The peak load on the receiver's input queue is the same, just shifted. Jitter belongs at the source.

Measuring jitter effectiveness¶

Add a metric that records, for each tick, the wall-clock time at which the work started. After a few hours, plot a histogram. If jitter is working, you see a uniform distribution across the interval. If it is not, you see a tight peak near a boundary.

var tickStartHistogram = prometheus.NewHistogram(prometheus.HistogramOpts{
    Name:    "ticker_start_offset_seconds",
    Help:    "Offset within the tick interval when work started",
    Buckets: prometheus.LinearBuckets(0, 1, 60),
})

func RunWithMetric(ctx context.Context, interval time.Duration, do func()) error {
    offset := time.Duration(rand.Int63n(int64(interval)))
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-time.After(offset):
    }
    t := time.NewTicker(interval)
    defer t.Stop()
    start := time.Now()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-t.C:
            elapsed := time.Since(start).Seconds()
            bucketSec := math.Mod(elapsed, interval.Seconds())
            tickStartHistogram.Observe(bucketSec)
            do()
        }
    }
}

In production this metric is more useful than you would expect. Once we found a "jittered" cron pool that was synchronizing because every replica seeded math/rand with time.Now().Unix() and they all started in the same second.

Backpressure on Slow Consumers¶

A ticker fires; the handler is still finishing the previous tick. What happens? The Go runtime tries a non-blocking send on t.C. The slot has capacity one. If it is full, the tick is silently dropped. There is no error, no log, no signal. From the application's perspective the gap looks identical to the ticker being slower.

For some workloads this is exactly the right behaviour. A heartbeat that fires every second but is sometimes processed every two seconds is fine: the consumer skips a beat, you log it, life goes on. For other workloads — billing, GDPR-required state transitions, exactly-once cleanup — dropping a tick is unacceptable.

Detecting drops¶

The simplest detection is to check the elapsed time inside the handler:

func RunWithDropDetection(ctx context.Context, interval time.Duration, do func()) error {
    t := time.NewTicker(interval)
    defer t.Stop()
    last := time.Now()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            gap := now.Sub(last)
            if gap > 2*interval {
                slog.Warn("ticker drop detected",
                    "expected", interval, "actual", gap,
                    "dropped", int(gap/interval)-1)
            }
            last = now
            do()
        }
    }
}

This counts dropped ticks. Note now is the tick time as delivered by the runtime, not time.Now(). The tick time is the time the runtime scheduled the tick, so the gap directly reflects how many tick periods elapsed between the last delivered tick and this one. If three ticks were dropped, the gap is 4× the interval.

Strategy 1: shed load¶

The simplest backpressure response is "do less work per tick." If your handler decides per-tick whether to act, you can shed cheaply:

type LoadShedder struct {
    rate atomic.Uint64 // multiplier per 100; 100 = full load, 50 = half load
}

func (l *LoadShedder) shouldRun() bool {
    r := l.rate.Load()
    if r >= 100 {
        return true
    }
    return rand.Uint64()%100 < r
}

func (l *LoadShedder) Run(ctx context.Context, interval time.Duration, do func()) error {
    t := time.NewTicker(interval)
    defer t.Stop()
    last := time.Now()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            gap := now.Sub(last)
            if gap > 2*interval && l.rate.Load() > 25 {
                old := l.rate.Add(^uint64(9)) // -10
                slog.Warn("shedding load", "rate", old-10)
            }
            last = now
            if l.shouldRun() {
                do()
            }
        }
    }
}

rate starts at 100. Each detected drop drops it by 10. A separate goroutine raises it back to 100 slowly. This is a primitive form of additive-increase, multiplicative-decrease.

Strategy 2: drain pending work concurrently¶

If the work is parallelisable, run multiple workers behind one ticker. The ticker enqueues; workers dequeue.

type Task struct {
    Seq  uint64
    Time time.Time
}

func RunParallel(ctx context.Context, interval time.Duration, workers int, do func(Task)) error {
    t := time.NewTicker(interval)
    defer t.Stop()
    in := make(chan Task, workers*2)
    var wg sync.WaitGroup
    for i := 0; i < workers; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for task := range in {
                do(task)
            }
        }()
    }
    defer func() {
        close(in)
        wg.Wait()
    }()

    var seq uint64
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            task := Task{Seq: seq, Time: now}
            seq++
            select {
            case in <- task:
            default:
                slog.Warn("all workers busy, dropping task", "seq", task.Seq)
            }
        }
    }
}

Capacity of in is workers * 2, so a brief slowdown does not drop immediately. Beyond that, the non-blocking send sheds load. The number of workers is the maximum concurrency you tolerate; for a ticker fed work that takes longer than interval/workers to complete, the queue fills and drops resume.

Strategy 3: dynamic interval¶

If the work's duration depends on input that varies, adapt the interval. Slow down when handlers are slow.

type AdaptiveTicker struct {
    base    time.Duration
    current time.Duration
    min     time.Duration
    max     time.Duration
}

func (a *AdaptiveTicker) Run(ctx context.Context, do func() time.Duration) error {
    a.current = a.base
    timer := time.NewTimer(a.current)
    defer timer.Stop()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
        }
        dur := do()
        // If work took > half the interval, slow down. If < quarter, speed up.
        switch {
        case dur > a.current/2:
            a.current = min(a.current*2, a.max)
        case dur < a.current/4:
            a.current = max(a.current/2, a.min)
        }
        timer.Reset(a.current)
    }
}

This uses time.Timer, not time.Ticker, because the interval changes per iteration. You can also call t.Reset(d) on a ticker, but you have to be careful about the Reset race; see Production Incident #7.

Strategy 4: bounded delay¶

Sometimes you cannot drop a tick. The work must run eventually. Then queue it; if the queue overflows, block (deliberate backpressure on the producer, which is the ticker).

func RunNoDrop(ctx context.Context, interval time.Duration, queueCap int, do func(time.Time)) error {
    in := make(chan time.Time, queueCap)
    go func() {
        for now := range in {
            do(now)
        }
    }()
    t := time.NewTicker(interval)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            close(in)
            return ctx.Err()
        case now := <-t.C:
            // Blocking send; if queue is full, the next tick from the runtime is dropped
            // while we wait. But once we unblock, all future ticks are honored.
            select {
            case in <- now:
            case <-ctx.Done():
                close(in)
                return ctx.Err()
            }
        }
    }
}

A subtle property: while the loop is blocked in in <- now, the runtime continues to deliver ticks to t.C, but only one can fit in t.C. Subsequent ticks are dropped at the runtime level. When we unblock, we receive the single most recent tick, send it to the queue, and continue. So this isn't strictly no-drop; it is "no drop within the queue, but the in-flight tick can still be dropped." For true no-drop you need to advance an internal clock and emit missed ticks to the queue:

func RunCatchUp(ctx context.Context, interval time.Duration, queueCap int, do func(time.Time)) error {
    in := make(chan time.Time, queueCap)
    go func() {
        for now := range in {
            do(now)
        }
    }()
    t := time.NewTicker(interval)
    defer t.Stop()
    last := time.Now()
    for {
        select {
        case <-ctx.Done():
            close(in)
            return ctx.Err()
        case now := <-t.C:
            // Catch up on any ticks the runtime dropped.
            for last.Add(interval).Before(now) {
                last = last.Add(interval)
                select {
                case in <- last:
                case <-ctx.Done():
                    close(in)
                    return ctx.Err()
                }
            }
            last = now
            select {
            case in <- now:
            case <-ctx.Done():
                close(in)
                return ctx.Err()
            }
        }
    }
}

Now the queue receives one entry per interval that should have elapsed, even if the runtime dropped some. This is the only way to get exactly-once tick semantics from time.Ticker. For most workloads it is overkill; for billing it is what you need.

Observability: Metrics for Skipped Ticks¶

You cannot fix what you cannot see. Every production ticker should emit metrics. The minimal set:

Reference instrumentation¶

package tickerobs

import (
    "context"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    iterations = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "ticker_iterations_total",
        Help: "Number of ticker handler invocations.",
    }, []string{"name"})

    drops = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "ticker_drops_total",
        Help: "Number of ticks dropped by the runtime due to slow consumer.",
    }, []string{"name"})

    duration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "ticker_duration_seconds",
        Help:    "Wall-clock duration of ticker handler.",
        Buckets: prometheus.DefBuckets,
    }, []string{"name"})

    interval = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "ticker_interval_seconds",
        Help: "Current configured ticker interval.",
    }, []string{"name"})

    lastSuccess = promauto.NewGaugeVec(prometheus.GaugeOpts{
        Name: "ticker_last_success_timestamp_seconds",
        Help: "Unix timestamp of last successful tick.",
    }, []string{"name"})

    errors = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "ticker_errors_total",
        Help: "Number of ticker handler errors.",
    }, []string{"name"})
)

type Loop struct {
    Name     string
    Interval time.Duration
    Do       func(context.Context, time.Time) error
}

func (l Loop) Run(ctx context.Context) error {
    interval.WithLabelValues(l.Name).Set(l.Interval.Seconds())
    t := time.NewTicker(l.Interval)
    defer t.Stop()

    last := time.Now()
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case now := <-t.C:
            gap := now.Sub(last)
            if gap > 2*l.Interval {
                missed := int(gap/l.Interval) - 1
                drops.WithLabelValues(l.Name).Add(float64(missed))
            }
            last = now

            start := time.Now()
            err := l.Do(ctx, now)
            duration.WithLabelValues(l.Name).Observe(time.Since(start).Seconds())
            iterations.WithLabelValues(l.Name).Inc()
            if err != nil {
                errors.WithLabelValues(l.Name).Inc()
            } else {
                lastSuccess.WithLabelValues(l.Name).Set(float64(time.Now().Unix()))
            }
        }
    }
}

This is the canonical "instrumented loop" pattern. We use it for every ticker in production code.

Useful queries¶

Given the above, Prometheus queries that find broken tickers:

# Tickers that have not made progress in 5 minutes (despite being expected to)
time() - ticker_last_success_timestamp_seconds > 300

# Tickers dropping more than 10% of their ticks
rate(ticker_drops_total[5m]) / rate(ticker_iterations_total[5m]) > 0.1

# Tickers where p99 handler duration exceeds interval (will start dropping)
histogram_quantile(0.99, rate(ticker_duration_seconds_bucket[5m])) > ticker_interval_seconds

# Tickers where handler error rate > 5%
rate(ticker_errors_total[5m]) / rate(ticker_iterations_total[5m]) > 0.05

The last query is the most actionable. If p99 handler duration > interval for a sustained period, your ticker is dropping ticks even if it does not look like it.

Alerting¶

groups:
- name: tickers
  rules:
  - alert: TickerStalled
    expr: time() - ticker_last_success_timestamp_seconds > 300
    for: 1m
    labels:
      severity: page
    annotations:
      summary: "Ticker {{ $labels.name }} has not succeeded in 5 minutes"

  - alert: TickerDroppingTicks
    expr: rate(ticker_drops_total[5m]) / rate(ticker_iterations_total[5m]) > 0.1
    for: 10m
    labels:
      severity: warn
    annotations:
      summary: "Ticker {{ $labels.name }} is dropping >10% of ticks"

  - alert: TickerHandlerSlower
    expr: histogram_quantile(0.99, rate(ticker_duration_seconds_bucket[5m])) > ticker_interval_seconds
    for: 5m
    labels:
      severity: warn
    annotations:
      summary: "Ticker {{ $labels.name }} handler p99 exceeds interval"

These three alerts cover 90% of ticker failure modes. The first catches "loop dead." The second catches "loop is slower than it should be." The third predicts the second by a few minutes — by the time you see drops, you have been over-budget on duration for a while.

Logging¶

Logs and metrics serve different purposes. Metrics tell you the rate; logs tell you the what. For a ticker, log on drop, on error, and on interval change. Do not log on every successful tick — at a high rate this drowns the rest of the log stream.

if missed > 0 {
    slog.WarnContext(ctx, "ticker drop",
        "name", l.Name,
        "missed", missed,
        "gap", gap,
        "interval", l.Interval)
}
if err != nil {
    slog.ErrorContext(ctx, "ticker handler",
        "name", l.Name,
        "err", err,
        "duration", time.Since(start))
}

Use slog.WarnContext so the log carries the trace ID if one is in context. We have caught more issues by correlating a WARN drop log with a trace ID than by reading metrics alone.

Drift Correction and Wall-Clock Alignment¶

A time.Ticker measures intervals on the monotonic clock. The interval d between consecutive ticks is guaranteed to be at least d. If a tick fires late (handler runs long, scheduler is busy), the next tick is not delivered early to compensate; the runtime simply schedules the next tick d after the previous one. Over time the ticker drifts away from any wall-clock anchor.

For most workloads this is fine. For billing, snapshots, and time-windowed analytics it is wrong.

What drift looks like¶

func main() {
    t := time.NewTicker(1 * time.Second)
    defer t.Stop()
    start := time.Now()
    for i := 0; i < 10; i++ {
        <-t.C
        expected := start.Add(time.Duration(i+1) * time.Second)
        actual := time.Now()
        fmt.Printf("tick %d: drift = %v\n", i, actual.Sub(expected))
        time.Sleep(50 * time.Millisecond) // simulate slow handler
    }
}

Expected output: drift grows by ~50ms per tick. After 100 ticks the drift is ~5 s. The ticker fires at relative times 1s, 2s, 3s from the previous tick, but each tick's handling takes 50ms, so by the time we reach the next <-t.C we are 50ms late. The runtime delivers the next tick 1s after the previous delivery, not 1s after the previous expected time. The drift compounds.

Strategy: anchor to wall clock with a timer¶

If you want "fire at top of each second, regardless of handler duration", drop time.Ticker and use time.Timer with explicit re-arming relative to wall time.

func RunAligned(ctx context.Context, interval time.Duration, do func(time.Time)) error {
    if interval <= 0 {
        return errors.New("interval must be positive")
    }
    // Align first tick to the next interval boundary.
    nextDeadline := time.Now().Truncate(interval).Add(interval)
    timer := time.NewTimer(time.Until(nextDeadline))
    defer timer.Stop()

    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
        }
        do(nextDeadline)
        nextDeadline = nextDeadline.Add(interval)
        // If we are behind, skip to the next future deadline.
        if time.Now().After(nextDeadline) {
            now := time.Now()
            elapsed := now.Sub(nextDeadline)
            skip := elapsed / interval
            nextDeadline = nextDeadline.Add((skip + 1) * interval)
        }
        timer.Reset(time.Until(nextDeadline))
    }
}