Timer Leaks — Professional Level (Detection, Telemetry, Retrofits)¶

Table of Contents¶

1. Introduction
2. Why Timer Leaks Are Different
3. What Production Looks Like When Timers Leak
4. The Runtime Timer Heap, Briefly
5. Detection Layer One: runtime.MemStats
6. Detection Layer Two: pprof Heap Profiles
7. Detection Layer Three: pprof Goroutine Profiles
8. Detection Layer Four: Allocation Profiles
9. Detection Layer Five: Execution Traces
10. Detection Layer Six: Live Timer Count Metric
11. Goleak in CI
12. Building a Leak Reproducer
13. Real Incident: The time.After RPC Loop
14. Real Incident: The Forgotten Reset
15. Real Incident: The Cancelled context.Context That Wasn't
16. Real Incident: The Ticker Inside a Hot Path
17. Real Incident: The Reconnect Loop
18. Real Incident: The Per-Request Health Check
19. Postmortem Template for Timer Leaks
20. Retrofitting Old Codebases
21. Static Analysis: vet, staticcheck, Custom Analyzers
22. Wrappers That Make Misuse Hard
23. Capacity Planning and Cost of Timers
24. Continuous Profiling in Production
25. Self-Assessment
26. Summary

Introduction¶

A leak that needs three days to manifest is the most expensive bug a Go service can carry. It survives unit tests, survives integration tests, survives canary windows, survives smoke tests, survives the entire pre-production gauntlet, and is then released into the fleet where it sits quietly accumulating cost for the lifetime of every long-running process. Timer leaks are this kind of bug. They do not break the binary; they erode it. Heap grows by a few hundred kilobytes per hour, goroutine count drifts upward by a handful per minute, GC cycles take a little longer each pass, the timer heap fills with entries scheduled hours into the future, and eventually the process either hits a memory limit, slows below its SLO, or is restarted as a routine maintenance action that masks the underlying cause for another release cycle.

This document is about how to find timer leaks before they cost you a postmortem, and how to retrofit a fleet that already has them. We are going to look at the runtime's timer machinery in just enough detail to understand what a "leaked" timer actually is, what the runtime keeps alive on its behalf, and what each diagnostic surface in the Go toolchain shows about it. We will then build a stack of detection layers from the simplest (a metric scraped every fifteen seconds) to the most invasive (a tracing build that records every timer event), and we will walk through six real incidents drawn from production postmortems. Finally, we will design retrofits — code review patterns, linters, wrappers, telemetry, and CI guards — that turn the implicit risk of time.After and friends into something a team can carry safely.

The audience for this file is engineers who already know how to use time.After, time.NewTimer, time.AfterFunc, and time.Ticker. The junior file in this same folder covers the basic mechanics: how the API behaves, when timers fire, what Stop returns, what the standard idioms are. Here we assume that foundation and focus on operating a Go service in which timers may leak — observing them, attributing them, fixing them, and preventing regressions.

A note on Go versions. The timer machinery underwent a major rewrite in Go 1.23, which dramatically reduced the cost of time.After and made many old idioms obsolete. We will call out where 1.21, 1.22, and 1.23 differ — but most of this document is structured around the worst case, which is the pre-1.23 behaviour, because most production fleets run a mix of versions and any retrofitting plan must support the oldest binaries in the field.

Why Timer Leaks Are Different¶

Timer leaks share a family resemblance with goroutine leaks and channel-blocking leaks, but they have three properties that set them apart and that drive the rest of this document.

Property one: timers are runtime-managed, not goroutine-backed¶

Every other concurrency primitive in Go that can leak — a channel, a mutex, a goroutine — has a corresponding object in user code that you can reach and inspect. A leaked goroutine appears in runtime.Stack. A leaked channel send is a goroutine blocked in chansend, visible in any goroutine profile. A leaked mutex hold is, in the worst case, a goroutine blocked in sync.(*Mutex).Lock, also visible. But a leaked timer is, by default, not a goroutine. It is an entry in an internal min-heap managed by the runtime; until it fires (or you call Stop), it consumes memory and a slot in the heap, but there is no goroutine attached to it. Goroutine profiling will not find it. Stack traces will not find it. The only places where a leaked timer surfaces are heap profiles (under time.NewTimer, time.startTimer, or similar symbols), the timer-count statistics exposed by the runtime, and execution traces.

This means the standard "is anything leaking?" debugging recipe — dump goroutines, look for unbounded growth, find the function that spawns them — does not work for timer leaks. You need to learn a different set of tools.

Property two: a leaked timer is GC-reachable¶

When you write time.After(5*time.Second), the runtime allocates a *runtime.timer (or a *time.Timer wrapping one, depending on Go version) and registers it with the per-P timer heap. Even after you drop your reference to the returned channel, the runtime still holds a reference to the timer through the heap. The GC therefore cannot collect the timer until it fires or is stopped — even if no user-code reference exists. This is critical: it explains why timer leaks accumulate over hours instead of being cleaned up on the next GC. The runtime is, in a precise sense, keeping the leak alive on your behalf.

Go 1.23 weakened this: now stopped timers and timers without active channel receivers can be collected more aggressively, but the model is still "the runtime owns the timer until it expires." In any version, the correct mental model is: a timer occupies runtime memory for at least duration after it is started, no matter what user code does with the returned channel.

Property three: the cost is silent and gradual¶

A leaked goroutine costs at least 2 KiB of stack plus its g struct (~376 bytes on amd64). A leaked timer costs roughly 80–120 bytes for the timer struct, plus the channel it owns (~96 bytes), plus a slot in the per-P timer heap. The total per-timer cost is on the order of 200 bytes. A service that allocates one leaked timer per RPC at 10,000 RPC/s leaks roughly 2 MiB/s — small enough to be invisible on a memory chart for the first hour, but ruinous over a day. Compare this to a leaked goroutine, which leaks 10× as much memory per occurrence and is therefore noticed sooner.

Timer leaks have another silent dimension: they degrade GC pause times. Every timer in the heap is a root, in effect — the runtime must walk the timer heap on every GC cycle. A million timers in the heap can add tens of milliseconds to GC pause time, which is the kind of degradation that shows up in a tail-latency dashboard long before it shows up in a memory dashboard.

What Production Looks Like When Timers Leak¶

Before we instrument anything, let's establish what timer leaks look like to the operator. Knowing the symptoms is half the battle, because most timer leaks are first observed indirectly — through their downstream effects on the GC, the heap, the scheduler, or the API SLOs.

Symptom one: slowly rising heap¶

Run kubectl top pod or your equivalent every few hours. A service with no leak has a sawtooth heap pattern: GC drops it, allocation raises it, GC drops it again. A service with a timer leak has the same sawtooth — but the baseline drifts upward by a small percentage per hour. The minimum value of the sawtooth after each GC is the live heap, and that minimum grows linearly with uptime. After three days, the live heap might be 1.8 GiB on a process that started at 200 MiB.

This is the most reliable symptom, but it is unspecific: any leak (timers, goroutines, caches, log buffers) produces it. The next symptoms narrow the search.

Symptom two: rising goroutine count without obvious cause¶

Even though leaked timers are not goroutines, they are usually coupled to goroutine leaks. For example: a goroutine waits in a select that has a time.After branch. If the goroutine never exits because the channel it is waiting on never closes, the time.After timer is leaked alongside the goroutine. So in practice, timer leaks often co-occur with goroutine leaks, and a rising goroutine count is a useful early-warning signal.

Symptom three: increasing GC CPU share¶

GOGC=100 (the default) targets a steady-state heap that is roughly 2× the live heap. As the live heap grows, GC has more work per cycle, and the fraction of CPU spent in GC grows. The Go runtime exposes this fraction; if your dashboard shows GC CPU rising from 3% to 8% to 12% over the course of a week of uninterrupted uptime, something is leaking. Timers contribute to this because every timer in the heap is a memory allocation the GC must trace.

Symptom four: degraded tail latency¶

The 99th percentile latency of an API drifts upward over days. The mean is stable; the median is stable; only the tail moves. This is the signature of a GC problem: longer pauses make a handful of requests slow without affecting the average. Timer leaks contribute to this because (a) they grow the heap, lengthening GC pauses, and (b) they grow the per-P timer heap, lengthening the time the scheduler spends checking for ready timers.

Symptom five: rising runtime.NumGoroutine() plotted against a flat runtime.MemStats.Mallocs - Frees¶

This one is more subtle: if your fleet exports both runtime.NumGoroutine() and the live allocation count Mallocs - Frees, and both grow in parallel, you have a goroutine leak. If goroutines are flat and Mallocs - Frees grows, you have something else — possibly a timer leak, possibly a regular memory leak. We will refine this signal below.

Symptom six: an alarm that fires once and goes away¶

A timer leak can produce a very specific alarm: a single brief spike of heap growth at the moment of process startup, followed by smooth growth. The startup spike is the worker pool initializing; the smooth growth is timers slowly accumulating. Operators often dismiss this pattern as "warmup" and move on. Don't.

The Runtime Timer Heap, Briefly¶

To debug timer leaks effectively, you need to know what the runtime is doing on your behalf. This section is a compressed tour; for the full picture see the specification file.

In Go 1.14 and later, every P (logical processor) owns a min-heap of pending timers, sorted by when (the absolute time at which the timer should fire, in monotonic nanoseconds). When you call time.NewTimer(d), the runtime:

1. Allocates a time.Timer struct, which contains a C chan Time and an embedded runtimeTimer.
2. Computes when = nanotime() + int64(d).
3. Calls runtime.addtimer, which inserts the timer into the local P's timer heap. (Before 1.14, all timers lived on a single global heap behind a global mutex; this was the bottleneck the per-P design fixed.)
4. Returns the *time.Timer.

When the timer's when arrives, the runtime, via sysmon and the scheduler, sends the current time on t.C. If nothing is reading from t.C, the send blocks — but only briefly, because t.C has a buffer of size 1. The send completes, the timer's state is timerDeleted (Go 1.14+ terminology), and the timer is removed from the heap on the next opportunity.

When you call t.Stop(), the runtime atomically transitions the timer's state and arranges for it to be removed from the heap. If the timer had already fired and the value sits in t.C, Stop returns false; otherwise it returns true. The classic drain pattern,

if !t.Stop() {
    <-t.C
}

is needed because if the timer has fired but no one has read t.C, the next select over t.C will spuriously trigger.

The cost of a timer is therefore: - The time.Timer struct (runtimeTimer ~80 bytes + chan Time ~96 bytes ≈ 176 bytes). - A slot in the per-P timer heap (8–16 bytes). - An entry in the GC's root set until the timer is removed from the heap.

Multiply by the number of leaked timers and you have your accounting.

time.After is just sugar¶

The implementation of time.After in time/sleep.go is approximately:

func After(d Duration) <-chan Time {
    return NewTimer(d).C
}

Note what is not there: any way to call Stop on the timer. The *Timer is dropped, only its C channel survives, and the runtime keeps the timer in its heap until d elapses. This is the entire reason for the slogan "don't use time.After in loops."

Go 1.23: a major rewrite¶

Go 1.23 changed two things relevant to leak debugging:

1. When a *Timer becomes unreachable (i.e., no goroutine holds a reference to it and no one is reading from t.C), it can now be garbage collected even if it has not fired. The runtime arranges this through a special weak-reference mechanism on the timer's internal state.
2. The timer's channel t.C is created with a "synchronous" delivery model rather than the old async send-to-buffer pattern.

In practice, this means many time.After leaks that were catastrophic in Go 1.21 become merely "wasteful" in Go 1.23: the timer is still allocated and still consumes CPU until the next GC, but it does not accumulate indefinitely. Crucially, this only helps if the surrounding code drops references to the channel. If a goroutine sits in select { case <-ch: ...; case <-time.After(d): ... } and ch never delivers, the goroutine still holds the channel reference and the timer is still leaked. The Go 1.23 change cannot rescue you from a goroutine leak.

We will return to version differences throughout this document; for now, the key point is that no version of Go saves you from a leaked goroutine that owns a timer's channel.

Detection Layer One: runtime.MemStats¶

The cheapest, lowest-overhead leak detector you can deploy is runtime.ReadMemStats. It returns a snapshot of memory statistics, including counters that grow over the process lifetime. By scraping these counters every 15–30 seconds and exporting them as metrics, you can detect leaks of any kind — including timer leaks — with very little cost.

The relevant fields are:

The two derived quantities that matter most for leak detection are:

• Live heap: HeapAlloc immediately after a GC. This is the floor of the sawtooth.
• Live objects: Mallocs - Frees. This is the number of objects currently alive, regardless of size.

A timer leak shows up as a slowly growing Mallocs - Frees (each leaked timer is one or two objects) and a slowly growing HeapAlloc floor.

Exporting the metrics¶

Here is a minimal exporter that you can drop into any service:

package memstats

import (
    "runtime"
    "time"

    "github.com/prometheus/client_golang/prometheus"
)

type Exporter struct {
    HeapAlloc      prometheus.Gauge
    HeapObjects    prometheus.Gauge
    LiveObjects    prometheus.Gauge
    NumGC          prometheus.Counter
    PauseTotalNs   prometheus.Counter
    Goroutines     prometheus.Gauge
    lastPauseTotal uint64
    lastNumGC      uint32
}

func New(reg prometheus.Registerer) *Exporter {
    e := &Exporter{
        HeapAlloc: prometheus.NewGauge(prometheus.GaugeOpts{
            Name: "go_heap_alloc_bytes",
            Help: "Current heap allocation in bytes",
        }),
        HeapObjects: prometheus.NewGauge(prometheus.GaugeOpts{
            Name: "go_heap_objects",
            Help: "Number of allocated heap objects",
        }),
        LiveObjects: prometheus.NewGauge(prometheus.GaugeOpts{
            Name: "go_live_objects",
            Help: "Mallocs minus frees",
        }),
        NumGC: prometheus.NewCounter(prometheus.CounterOpts{
            Name: "go_gc_cycles_total",
            Help: "Total GC cycles",
        }),
        PauseTotalNs: prometheus.NewCounter(prometheus.CounterOpts{
            Name: "go_gc_pause_total_nanoseconds",
            Help: "Cumulative GC pause time in nanoseconds",
        }),
        Goroutines: prometheus.NewGauge(prometheus.GaugeOpts{
            Name: "go_goroutines",
            Help: "Number of goroutines",
        }),
    }
    reg.MustRegister(e.HeapAlloc, e.HeapObjects, e.LiveObjects,
        e.NumGC, e.PauseTotalNs, e.Goroutines)
    return e
}

func (e *Exporter) Run(interval time.Duration, stop <-chan struct{}) {
    t := time.NewTicker(interval)
    defer t.Stop()
    var ms runtime.MemStats
    for {
        select {
        case <-stop:
            return
        case <-t.C:
            runtime.ReadMemStats(&ms)
            e.HeapAlloc.Set(float64(ms.HeapAlloc))
            e.HeapObjects.Set(float64(ms.HeapObjects))
            e.LiveObjects.Set(float64(ms.Mallocs - ms.Frees))
            if ms.NumGC > e.lastNumGC {
                e.NumGC.Add(float64(ms.NumGC - e.lastNumGC))
                e.lastNumGC = ms.NumGC
            }
            if ms.PauseTotalNs > e.lastPauseTotal {
                e.PauseTotalNs.Add(float64(ms.PauseTotalNs - e.lastPauseTotal))
                e.lastPauseTotal = ms.PauseTotalNs
            }
            e.Goroutines.Set(float64(runtime.NumGoroutine()))
        }
    }
}

This costs almost nothing in steady state — ReadMemStats is fast (microseconds) but it does stop-the-world briefly. At 15-second intervals the overhead is negligible.

What to alert on¶

The slope of go_live_objects over a 24-hour window is the single most useful leak signal. A healthy service has a flat or sawtooth slope; a leaky service has a positive linear slope. Configure an alert that fires when deriv(go_live_objects[6h]) > 100/s (or some service-appropriate threshold) — this catches leaks early, before they affect availability.

A more specific timer-leak signal: divide go_live_objects by go_goroutines. If the ratio grows, you are leaking objects that are not goroutines — a strong hint at timers, channels, or caches.

Limitations of MemStats¶

MemStats does not distinguish between timer leaks and other heap leaks. It tells you something is leaking but not what. To attribute, you need pprof.

runtime/metrics (Go 1.16+)¶

A more modern alternative to MemStats is the runtime/metrics package. It exposes a stable set of metric names with explicit units. Notable for our purposes:

import "runtime/metrics"

samples := []metrics.Sample{
    {Name: "/sched/goroutines:goroutines"},
    {Name: "/gc/heap/objects:objects"},
    {Name: "/gc/heap/allocs:bytes"},
    {Name: "/gc/cycles/total:gc-cycles"},
    {Name: "/sched/latencies:seconds"},
}
metrics.Read(samples)

runtime/metrics is preferred over MemStats because it doesn't stop-the-world. It also covers some metrics MemStats doesn't, like scheduler latency histograms — which are useful for noticing that GC pauses are eating into runnable goroutines.

There is, regrettably, no /timers/total metric in the standard runtime/metrics namespace in any Go version up to 1.23. The closest available is /gc/heap/objects:objects, which counts all heap objects. We will build our own timer counter later in this document.

Detection Layer Two: pprof Heap Profiles¶

pprof heap profiles are the most powerful tool for attributing memory growth to specific allocation sites. A heap profile is a sampled list of allocation call stacks, weighted by the number of bytes (or objects) currently alive. For timer leak debugging it is indispensable, because time.NewTimer allocations show up at very specific symbols.

Enabling pprof in production¶

The cheapest and most reliable approach is net/http/pprof. Import it for side effects and serve it on a dedicated debug port:

import (
    "net/http"
    _ "net/http/pprof"
)

func init() {
    go func() {
        // Bind only to localhost or to an internal NIC.
        http.ListenAndServe("127.0.0.1:6060", nil)
    }()
}

Profile endpoints (assuming default mux):

• /debug/pprof/heap — current heap profile
• /debug/pprof/goroutine — goroutine profile
• /debug/pprof/profile — CPU profile (default 30 s)
• /debug/pprof/trace — execution trace
• /debug/pprof/allocs — allocation profile (all allocations since process start, not just live)

Capturing a heap profile¶

From a developer's laptop, with port-forwarding into the pod:

kubectl port-forward pod/api-7d5 6060:6060 &
go tool pprof -http=:0 http://localhost:6060/debug/pprof/heap

This launches a browser with the interactive pprof UI. The inuse_space view (the default) shows currently-alive memory grouped by call stack. For timer leaks, look for stacks that include:

• time.NewTimer
• time.startTimer
• time.After
• time.AfterFunc
• runtime.addtimer

In inuse_objects view, the same stacks show how many timer objects are currently alive. The ratio of inuse_space to inuse_objects for each stack is roughly the per-object size: for time.NewTimer, expect 80–200 bytes per object depending on the version.

Reading a heap profile¶

A typical leaky service's inuse_objects view, sorted by self count:

      flat  flat%   sum%        cum   cum%
   1.2M     35%    35%       1.2M    35%   time.NewTimer
    600K    17%    52%        600K    17%   time.After (inline)
    250K     7%    59%        250K     7%   bytes.makeSlice
    ...

If time.NewTimer or time.After is the top entry, you have a timer leak. The diff against a profile taken an hour earlier confirms it:

go tool pprof -base=heap-baseline.pb.gz -http=:0 heap-now.pb.gz

The base view shows only growth since the baseline, which makes leaks unambiguous: only leaked allocation sites have non-zero growth.

runtime.SetBlockProfileRate and friends¶

In addition to the heap profile, the runtime exposes:

• SetBlockProfileRate(rate int) — captures profiles of goroutines blocked on synchronization
• SetMutexProfileFraction(rate int) — captures contended mutex stacks

Neither of these directly catches timer leaks, but they are useful when investigating related problems (e.g., a goroutine leaked because it was blocked on a mutex, dragging a timer with it).

Sampling rate and accuracy¶

The default heap profile sampling rate is one sample per 512 KiB allocated. For a typical timer leak you need thousands of leaked timers before any individual stack shows up. For more sensitive detection, set:

runtime.MemProfileRate = 1024 // sample every 1 KiB

at process startup. This costs CPU (memory profile overhead grows roughly linearly with sample rate), but for staging or development debugging it's worth it.

Programmatic capture¶

For automated capture (e.g., a "snapshot every hour and ship to S3" job):

package profile

import (
    "context"
    "fmt"
    "os"
    "runtime/pprof"
    "time"
)

func SnapshotHeap(ctx context.Context, dir string) error {
    ts := time.Now().UTC().Format("20060102T150405Z")
    path := fmt.Sprintf("%s/heap-%s.pb.gz", dir, ts)
    f, err := os.Create(path)
    if err != nil {
        return err
    }
    defer f.Close()
    if err := pprof.Lookup("heap").WriteTo(f, 0); err != nil {
        return fmt.Errorf("write heap profile: %w", err)
    }
    return nil
}

func RunHourly(ctx context.Context, dir string) {
    t := time.NewTicker(1 * time.Hour)
    defer t.Stop()
    for {
        select {
        case <-ctx.Done():
            return
        case <-t.C:
            if err := SnapshotHeap(ctx, dir); err != nil {
                // log and continue
            }
        }
    }
}

Adapt to your storage backend; the key is that you have one profile per hour available to diff against once you suspect a leak.

Heap profile gotcha: inuse_space lies about timers (slightly)¶

The timer object itself is small, but it holds a reference to a channel, which is allocated separately. In the heap profile, the channel allocation shows up under a different stack (runtime.makechan), so the per-timer cost is split across two stacks. To get the true per-timer cost, you need to look at both time.NewTimer and the channel allocation it triggers. In practice, looking at time.NewTimer alone is sufficient to attribute the leak; you just have to remember that the total memory cost is roughly double the count multiplied by the size shown for the NewTimer line.

Detection Layer Three: pprof Goroutine Profiles¶

A goroutine profile is a snapshot of every goroutine in the process with its call stack. It is captured from:

curl -o goroutines.pb.gz http://localhost:6060/debug/pprof/goroutine
go tool pprof -http=:0 goroutines.pb.gz

Or, for human-readable output:

curl http://localhost:6060/debug/pprof/goroutine?debug=2

While leaked timers are not goroutines, leaked goroutines often drag timers with them. Specifically, a goroutine in this state is a strong indicator of a paired timer leak:

goroutine 12345 [select, 4 minutes]:
main.handler(...)
    /app/handler.go:42 +0x108
created by main.serve in goroutine 1
    /app/server.go:88 +0x60

The [select, 4 minutes] tag tells you a goroutine has been parked in a select for four minutes. If the select includes a time.After branch, that timer is leaked alongside the goroutine.

Identifying suspicious goroutines¶

Look for stacks with high count and [chan receive] or [select] parking reasons. A leak pattern is:

goroutine profile: total 142000
   141500 @ 0x4f8c00 0x4f8d60 0x4f9000 0x509b00 ...
#   0x509aff   main.waitForReply+0x5f   /app/rpc.go:88
#   0x509200   main.handleRequest+0x40  /app/rpc.go:42

When a single stack accounts for >90% of goroutines and is parked in chan receive or select, you have a leak. Open rpc.go:88 and look for either an unbounded <-ch or a select with time.After — both are candidates.

When timer leaks are not visible in goroutine profiles¶

If your leak is purely "spawn a timer, drop the reference, never start a goroutine to wait for it," goroutine profiles will not show anything. Pattern:

for {
    if shouldNotify() {
        time.AfterFunc(5*time.Second, sendAlert) // leaks if shouldNotify is hot
    }
}

Here time.AfterFunc schedules a function to run later but does not spawn a goroutine until it fires. Each call leaks the *Timer struct, but no goroutine is created until the timer fires (and then the runtime borrows one of its own goroutines, briefly). Goroutine profiles are blind to this kind of leak.

Differential goroutine profiles¶

If you have two snapshots, diff them by count:

go tool pprof -base=goroutines-baseline.pb.gz -http=:0 goroutines-now.pb.gz

The diff shows stacks whose goroutine count grew. This is the most useful single command for finding goroutine leaks in a running service.

runtime.Stack for emergency inspection¶

When pprof isn't available, you can dump goroutine stacks programmatically:

func dumpStacks() string {
    buf := make([]byte, 1<<20)
    n := runtime.Stack(buf, true)
    return string(buf[:n])
}

Pass true to include all goroutines. For a healthy service this is 5–50 KiB; for a leaking one it can be hundreds of MiB, which is precisely why you should rarely call this in production. Reserve it for crash-handlers and last-ditch debugging.

Counting goroutines per stack¶

A useful diagnostic is grouping goroutines by their stack trace and reporting counts:

package gprof

import (
    "fmt"
    "io"
    "runtime/pprof"
    "strings"
)

func WriteSummary(w io.Writer) error {
    p := pprof.Lookup("goroutine")
    var sb strings.Builder
    if err := p.WriteTo(&sb, 1); err != nil {
        return err
    }
    // Format with debug=1 is "count @ stack hash\nstack...\n\n"
    counts := map[string]int{}
    blocks := strings.Split(sb.String(), "\n\n")
    for _, b := range blocks {
        lines := strings.Split(b, "\n")
        if len(lines) < 2 {
            continue
        }
        header := lines[0]
        var n int
        fmt.Sscanf(header, "%d", &n)
        // Use first frame as a coarse key
        if len(lines) >= 3 {
            counts[lines[2]] += n
        }
    }
    for stack, n := range counts {
        if n > 100 { // threshold
            fmt.Fprintf(w, "%5d %s\n", n, stack)
        }
    }
    return nil
}

A handler that exposes this on /debug/goroutines/summary is a cheap way to spot leaks without leaving the SSH session.

Detection Layer Four: Allocation Profiles¶

The inuse_space heap profile shows currently-alive allocations, which is what you want for finding current leaks. But sometimes you want to know which call sites allocate most often, even if those allocations are short-lived. The allocation profile is alloc_space and alloc_objects.

go tool pprof -alloc_space http://localhost:6060/debug/pprof/heap

For timer leak debugging, allocation profiles are useful in one specific case: a "transient" leak where a code path allocates many timers per request but they all get garbage-collected within seconds. The heap profile won't show this (the timers are gone by the time you look) but the allocation profile will. A high alloc_objects count for time.NewTimer tells you something is hammering on time.After even if you don't see live leaks.

This matters because, even pre-1.23 with non-leaking patterns, an extreme volume of timer allocations stresses the GC and the timer heap. A service that allocates 100 K time.After calls per second will spend nontrivial CPU in runtime.addtimer and runtime.deltimer, and will have a hot timer heap on every P. The allocation profile makes this visible.

alloc_objects vs alloc_space¶

For timers, alloc_objects is the more useful view, because timer objects are small and uniform. A profile showing time.NewTimer at the top of alloc_objects but not alloc_space tells you you have a high-rate problem, not necessarily a high-volume problem.

Detection Layer Five: Execution Traces¶

Execution traces (runtime/trace) capture per-goroutine, per-event timelines of what the runtime is doing. They are heavier than profiles — typically 10–100 MiB per minute of capture — but they reveal scheduler behaviour that profiles cannot, including timer-related events.

Capturing a trace¶

curl -o trace.out 'http://localhost:6060/debug/pprof/trace?seconds=30'
go tool trace trace.out

This opens a web UI with several views:

• View trace: a per-CPU timeline
• Goroutine analysis: per-goroutine state distribution
• Network blocking profile: time spent waiting on network IO
• Synchronization blocking profile: time spent waiting on channels/mutexes

For timer debugging, the most useful trace events are:

• runtime.timerproc: the runtime function that fires due timers
• runtime.goparkunlock followed by runtime.goready with a wait reason of timer
• Goroutines that wake up after a long park

What to look for¶

A trace from a service with a timer leak will show, in the "Synchronization blocking profile," large amounts of time attributed to "timer" wait reasons across many goroutines. Each entry tells you which goroutine, which timer duration, and which call site. Cross-referenced against your code, this often pinpoints the exact time.After causing the leak.

Tracing in production¶

Execution tracing has nontrivial overhead — typically 1–5% CPU during capture — and produces files large enough that you should not run continuously. Reserve traces for one-off investigations:

1. Set up an alert that fires when goroutines or live objects cross a threshold.
2. On alert, capture a 30-second trace.
3. Ship the trace to S3 for offline analysis.

For most operators, a heap profile diff is sufficient to find the leak. Traces are the next-step tool when the profile alone doesn't tell you why timers are leaking.

go tool trace and timer events¶

The trace tool's event types include EvTimerGoroutine (deprecated as of 1.22) and EvTimer events for individual fires. The Go 1.23 trace format records timer creation, modification, and deletion as discrete events. Programmatic analysis is possible with golang.org/x/exp/trace:

import "golang.org/x/exp/trace"

func analyseTrace(path string) error {
    f, err := os.Open(path)
    if err != nil {
        return err
    }
    defer f.Close()
    r, err := trace.NewReader(f)
    if err != nil {
        return err
    }
    timerCounts := map[string]int{}
    for {
        ev, err := r.ReadEvent()
        if err == io.EOF {
            break
        }
        if err != nil {
            return err
        }
        if ev.Kind() == trace.EventStateTransition {
            st := ev.StateTransition()
            if st.Reason == "timer" {
                stack := ev.Stack().String()
                timerCounts[stack]++
            }
        }
    }
    for stack, n := range timerCounts {
        fmt.Printf("%5d %s\n", n, stack)
    }
    return nil
}

For production fleets running tracing in cron-job style, this kind of analysis produces a daily "timer hotspot" report that complements your standing dashboards.

Detection Layer Six: Live Timer Count Metric¶

So far we have used the standard Go tools. Now we build a custom signal that no standard tool provides: a live count of timers known to the runtime. With this counter you can graph timer-heap size directly, alert on drift, and correlate with deploys.

The Go runtime does not expose a public timer-count API. We need to count timers ourselves by intercepting their creation and destruction. The straightforward approach is to provide a thin wrapper around time.NewTimer, time.NewTicker, and time.AfterFunc, increment a counter on creation, and decrement on stop.

The wrapper package¶

// Package timed wraps time package primitives with a live counter.
package timed

import (
    "sync/atomic"
    "time"
)

var liveCount int64

// LiveCount returns the number of timers currently outstanding.
func LiveCount() int64 {
    return atomic.LoadInt64(&liveCount)
}

// Timer is a counted wrapper around time.Timer.
type Timer struct {
    *time.Timer
    stopped int32
}

// NewTimer creates a counted timer.
func NewTimer(d time.Duration) *Timer {
    atomic.AddInt64(&liveCount, 1)
    return &Timer{Timer: time.NewTimer(d)}
}

// Stop stops the timer and decrements the counter.
func (t *Timer) Stop() bool {
    if atomic.CompareAndSwapInt32(&t.stopped, 0, 1) {
        atomic.AddInt64(&liveCount, -1)
    }
    return t.Timer.Stop()
}

// Reset resets the timer. If the timer was already stopped, this re-counts it.
func (t *Timer) Reset(d time.Duration) bool {
    if atomic.CompareAndSwapInt32(&t.stopped, 1, 0) {
        atomic.AddInt64(&liveCount, 1)
    }
    return t.Timer.Reset(d)
}

// Ticker is a counted wrapper around time.Ticker.
type Ticker struct {
    *time.Ticker
    stopped int32
}

func NewTicker(d time.Duration) *Ticker {
    atomic.AddInt64(&liveCount, 1)
    return &Ticker{Ticker: time.NewTicker(d)}
}

func (t *Ticker) Stop() {
    if atomic.CompareAndSwapInt32(&t.stopped, 0, 1) {
        atomic.AddInt64(&liveCount, -1)
    }
    t.Ticker.Stop()
}

// AfterFunc schedules f to run after d. The returned Timer can be cancelled.
func AfterFunc(d time.Duration, f func()) *Timer {
    atomic.AddInt64(&liveCount, 1)
    t := &Timer{}
    t.Timer = time.AfterFunc(d, func() {
        if atomic.CompareAndSwapInt32(&t.stopped, 0, 1) {
            atomic.AddInt64(&liveCount, -1)
        }
        f()
    })
    return t
}

This wrapper has limitations:

1. It counts only timers created through the wrapper. Anything still using time.After or raw time.NewTimer bypasses the counter.
2. The counter does not decrement automatically when a timer fires; you must call Stop (or schedule the firing through AfterFunc, which we patched) or the count remains.

The second limitation is a feature: a timer that fires without being stopped is still occupying the timer heap until the runtime cleans it up, and counting it as "live" until then is honest.

Forcing fleet adoption¶

To make the wrapper effective fleet-wide, ban the time package's leak-prone APIs and require the wrapper. The simplest enforcement is a custom analyzer.

// Package banafter is a vet-style analyzer that forbids time.After in selects.
package banafter

import (
    "go/ast"
    "golang.org/x/tools/go/analysis"
)

var Analyzer = &analysis.Analyzer{
    Name: "banafter",
    Doc:  "Forbid time.After (use timed.NewTimer with explicit Stop instead)",
    Run:  run,
}

func run(pass *analysis.Pass) (interface{}, error) {
    for _, file := range pass.Files {
        ast.Inspect(file, func(n ast.Node) bool {
            call, ok := n.(*ast.CallExpr)
            if !ok {
                return true
            }
            sel, ok := call.Fun.(*ast.SelectorExpr)
            if !ok {
                return true
            }
            ident, ok := sel.X.(*ast.Ident)
            if !ok {
                return true
            }
            if ident.Name == "time" && sel.Sel.Name == "After" {
                pass.Reportf(call.Pos(),
                    "time.After is forbidden; use timed.NewTimer with Stop")
            }
            return true
        })
    }
    return nil, nil
}

Wire this into your CI, fail the build on violations, and you have an enforced policy. Combined with the wrapper's metric, you now have observability and a guarantee that everything counted is being counted.

Exporting the counter¶

import (
    "github.com/prometheus/client_golang/prometheus"
    "yourorg/timed"
)

var liveTimersGauge = prometheus.NewGaugeFunc(
    prometheus.GaugeOpts{
        Name: "go_live_timers",
        Help: "Number of timers outstanding (created through timed package)",
    },
    func() float64 {
        return float64(timed.LiveCount())
    },
)

func init() {
    prometheus.MustRegister(liveTimersGauge)
}

Graph this against goroutines and live objects, and you have a three-dimensional view of leakiness.

Per-call-site counters¶

A more advanced version of the wrapper attributes counts to specific call sites:

package timed

import (
    "runtime"
    "sync"
    "sync/atomic"
    "time"
)

type siteCounter struct {
    count int64
    name  string
}

var (
    siteMu sync.RWMutex
    sites  = map[uintptr]*siteCounter{}
)

func caller(skip int) (uintptr, string) {
    pcs := make([]uintptr, 1)
    n := runtime.Callers(skip+2, pcs)
    if n == 0 {
        return 0, "unknown"
    }
    f, _ := runtime.CallersFrames(pcs).Next()
    return pcs[0], f.Function
}

func incrSite() *siteCounter {
    pc, name := caller(1)
    siteMu.RLock()
    sc, ok := sites[pc]
    siteMu.RUnlock()
    if !ok {
        siteMu.Lock()
        sc, ok = sites[pc]
        if !ok {
            sc = &siteCounter{name: name}
            sites[pc] = sc
        }
        siteMu.Unlock()
    }
    atomic.AddInt64(&sc.count, 1)
    return sc
}

// ... NewTimer, etc., call incrSite() and store sc in the Timer

This is more expensive (runtime.Callers is not free) but yields a per-call-site live count that maps directly to source lines. In production, this kind of attribution lets you spot the leaking call site in seconds.

Goleak in CI¶

go.uber.org/goleak is the standard tool for detecting goroutine leaks in tests. It works by snapshotting the set of goroutines at the start of a test and asserting that no extras remain at the end. While goleak doesn't directly detect timer leaks, it catches the goroutine component of paired leaks and is therefore a strong line of defence.

Basic usage¶

package mypkg_test

import (
    "testing"

    "go.uber.org/goleak"
)

func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}

This adds a check after every test that no goroutines were left behind. If a test creates a goroutine that loops on time.After and forgets to shut it down, goleak fails the test with a stack dump.

Per-test usage¶

For finer control:

func TestSomething(t *testing.T) {
    defer goleak.VerifyNone(t)
    // ... test body ...
}

VerifyNone runs at defer time and reports any goroutines that survived the test body.

Ignoring known goroutines¶

Some packages (e.g., HTTP clients, OpenTelemetry exporters) intentionally keep background goroutines alive. Use IgnoreTopFunction to whitelist them:

goleak.VerifyTestMain(m,
    goleak.IgnoreTopFunction("internal/poll.runtime_pollWait"),
    goleak.IgnoreTopFunction("go.opencensus.io/stats/view.(*worker).start"),
)

Be conservative with ignores; every entry is a place a future leak can hide.

Why goleak doesn't catch pure timer leaks¶

If your leak is time.AfterFunc(5*time.Minute, f) with no goroutine attached, goleak misses it. The leaked timer holds a closure reference but produces no live goroutine until it fires. To catch this in tests you need a timer counter:

func TestNoTimerLeaks(t *testing.T) {
    before := timed.LiveCount()
    // ... test body ...
    if after := timed.LiveCount(); after > before {
        t.Fatalf("timer leak: %d timers outstanding", after-before)
    }
}

This works if your test exclusively uses the timed wrapper. For a mixed codebase, you can sample pprof.Lookup("heap") and assert no growth in time.NewTimer allocations — but this is fragile because pprof samples.

CI integration patterns¶

A typical CI pipeline for a Go service:

# .github/workflows/test.yml
- run: go test -race ./...           # races + goleak
- run: go test -tags=timerleak ./... # extra timer checks
- run: staticcheck ./...
- run: vet ./...
- run: ./bin/banafter ./...          # custom analyzer

The timerleak build tag opts into stricter checks (like the per-test timer counter above), which can be too noisy for every test but invaluable as a separate guard.

Goleak and test parallelism¶

Goleak's VerifyNone interacts poorly with t.Parallel() because parallel tests share goroutines. The standard pattern is:

1. Use VerifyTestMain for whole-package checks.
2. Avoid t.Parallel() in tests that spawn goroutines you want to verify.

For services where parallel testing is essential, you can spawn goroutines under a context that's cancelled at end of test and Wait for them via a WaitGroup. This is a more disciplined pattern anyway and removes the need for goleak verification per test.

Building a Leak Reproducer¶

Before deploying a fix, build a reproducer. A reproducer is a tiny self-contained program that exhibits the same leak signature as production, runs in seconds, and is small enough to step through with a debugger. It is the foundation for everything downstream: regression tests, benchmarks, and confidence in the fix.

Anatomy of a reproducer¶

A typical timer-leak reproducer has three parts:

1. A hot loop that exercises the suspected leak pattern.
2. A periodic check of timer count or heap size.
3. Termination on success (leak detected) or timeout (no leak in N seconds).

Here is a reproducer for the canonical time.After in a select:

package main

import (
    "context"
    "fmt"
    "runtime"
    "time"
)

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    workCh := make(chan int, 1)
    done := make(chan struct{})

    // Worker that "races" timer against channel; channel always wins.
    go func() {
        defer close(done)
        for i := 0; i < 1_000_000; i++ {
            workCh <- i
            select {
            case <-workCh:
                // got it, fast path
            case <-time.After(10 * time.Second):
                fmt.Println("slow path")
            }
            if ctx.Err() != nil {
                return
            }
        }
    }()

    // Sampler
    var ms runtime.MemStats
    var lastHeap uint64
    t := time.NewTicker(500 * time.Millisecond)
    defer t.Stop()
    for i := 0; i < 20; i++ {
        <-t.C
        runtime.ReadMemStats(&ms)
        if i > 0 {
            growth := int64(ms.HeapAlloc) - int64(lastHeap)
            fmt.Printf("heap=%d Δ=%+d goroutines=%d\n",
                ms.HeapAlloc, growth, runtime.NumGoroutine())
        }
        lastHeap = ms.HeapAlloc
    }
    cancel()
    <-done
}

Run this on Go 1.21:

heap=1.2MB Δ=+50KB  goroutines=3
heap=2.5MB Δ=+1.3MB goroutines=3
heap=3.8MB Δ=+1.3MB goroutines=3
...

The heap grows about 1.3 MB every 500 ms, which is roughly 130 K timers/s × 200 bytes each. On Go 1.23 the growth is much smaller because the runtime collects unreferenced timers more aggressively — but rerunning the reproducer with GOGC=off makes the leak visible on any version.

Reproducer as a regression test¶

Once the leak is fixed, the reproducer becomes a test:

// +build !short

package mypkg_test

import (
    "context"
    "runtime"
    "testing"
    "time"
)

func TestNoTimerLeakInHotPath(t *testing.T) {
    if testing.Short() {
        t.Skip("long-running leak test")
    }
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    var before, after runtime.MemStats
    runtime.GC()
    runtime.ReadMemStats(&before)

    hotPath(ctx, 100_000)

    runtime.GC()
    runtime.ReadMemStats(&after)

    growth := int64(after.HeapAlloc) - int64(before.HeapAlloc)
    if growth > 1_000_000 { // 1 MB tolerance
        t.Fatalf("hot path leaked %d bytes", growth)
    }
}

This kind of test is slow (it usually needs >1 second to run) and is best placed behind a build tag or run only in CI.

Reproducing in a sandbox¶

Sometimes the leak is sensitive to scheduling or to specific runtime tunables. A useful debugging mode:

GODEBUG=gctrace=1,schedtrace=1000 GOGC=10 ./repro

gctrace=1 prints every GC cycle; schedtrace=1000 prints scheduler statistics every second; GOGC=10 runs GC aggressively so leaks show up faster. The schedtrace output includes counts of P-local timer queues, which is the closest you get to a built-in timer count metric.

Programmatic timer count¶

A useful diagnostic in reproducers is to extract the runtime's view of the timer heap. This is not exposed publicly, but it can be approximated via metrics:

import "runtime/metrics"

samples := []metrics.Sample{
    {Name: "/sched/goroutines:goroutines"},
    {Name: "/gc/heap/objects:objects"},
}
metrics.Read(samples)
fmt.Printf("goroutines=%v objects=%v\n",
    samples[0].Value.Uint64(), samples[1].Value.Uint64())

Object growth without goroutine growth is a strong signal of timer-style leaks.

Stress test reproducer¶

For load-style reproducers:

package main

import (
    "context"
    "runtime"
    "sync"
    "time"
)

func main() {
    ctx, cancel := context.WithCancel(context.Background())

    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            for ctx.Err() == nil {
                select {
                case <-time.After(time.Hour):
                case <-ctx.Done():
                    return
                }
            }
        }()
    }

    // Let leaks accumulate
    time.Sleep(30 * time.Second)

    var ms runtime.MemStats
    runtime.ReadMemStats(&ms)
    println("heap=", ms.HeapAlloc)

    cancel()
    wg.Wait()
}

100 goroutines, each parked on time.After(time.Hour). After 30 seconds of running, the heap is ~50 MB. Each goroutine has one outstanding timer; the leak is paired with a goroutine leak. This is a simple but realistic reproducer.

Real Incident: The time.After RPC Loop¶

This section is the first of six incident reports drawn from real-world postmortems. Names, services, and exact numbers have been generalized; the patterns are the durable lesson.

Context¶

A photo-sharing service operated a Go-based RPC gateway that proxied requests from mobile clients to dozens of internal services. The gateway maintained a pool of long-lived RPC connections, each with a worker goroutine that read responses off the socket. Code, simplified:

func (c *Conn) readLoop(ctx context.Context) {
    for {
        select {
        case msg := <-c.incoming:
            c.dispatch(msg)
        case <-time.After(30 * time.Second):
            c.keepalive()
        case <-ctx.Done():
            return
        }
    }
}

This ran on Go 1.20.

Symptoms¶

Three days after a feature deploy that doubled the gateway's RPC fanout, oncall began receiving alerts: heap usage on gateway pods was climbing steadily, latency p99 was creeping up, and several pods OOM-killed in a 24-hour window. Restarting the pods restored healthy heap, then the climb began again.

Investigation¶

Heap dumps taken pre-restart showed time.NewTimer as the top allocator (87% of live objects). Goroutine profiles were unremarkable — the gateway maintained roughly 5,000 connection workers, which matched expectations. So the leak was not in goroutines: it was in timers.

The trace of c.dispatch revealed the root cause: under the new fanout, c.incoming rarely went idle for more than a few milliseconds. Each iteration of the readLoop was creating a fresh time.After(30 * time.Second) timer, getting a message off the channel, and looping. The timer never fired and was never stopped — but the runtime retained it for 30 seconds. With ~5,000 connections each iterating thousands of times per second, the per-connection timer count grew to 150,000+ timers each. Total live timers: ~750 M. At ~200 bytes each, that's ~150 GiB — the gateway could not even allocate that much; it was OOM-killed long before reaching it.

Fix¶

Replace time.After with a long-lived time.Timer that is stopped and restarted each iteration:

func (c *Conn) readLoop(ctx context.Context) {
    t := time.NewTimer(30 * time.Second)
    defer t.Stop()
    for {
        if !t.Stop() {
            select {
            case <-t.C:
            default:
            }
        }
        t.Reset(30 * time.Second)
        select {
        case msg := <-c.incoming:
            c.dispatch(msg)
        case <-t.C:
            c.keepalive()
        case <-ctx.Done():
            return
        }
    }
}

After the fix, the live timer count per goroutine returned to 1 (the long-lived timer) and the heap stabilized.

Lessons¶

1. time.After in a hot loop is a leak unless the loop iteration is rare. "Rare" here means "less often than the timer's duration."
2. Heap profiles are the right first step. Goroutine profiles were misleading because the actual symptom was timers, not goroutines.
3. The post-1.23 runtime would have mitigated this leak considerably, but not eliminated it: the timers were still allocated and still occupied the timer heap for up to 30 seconds. The leak would have been slower, not absent.
4. A linter that flagged time.After in a function loop would have caught this in code review.

Postmortem note: detection lag¶

The leak deployed on day 0; the alert fired on day 3; the fix shipped on day 5. The deployment that introduced the leak doubled the gateway's request rate, which doubled the number of iterations through the leaky loop, which doubled the rate of leaked timers. Without that capacity change, the leak would have remained subclinical for weeks. Many timer leaks are like this: they exist in code for months, dormant, and surface only when traffic patterns shift.

Real Incident: The Forgotten Reset¶

Context¶

A payment processor used a time.Timer as a watchdog for long-running operations. The watchdog reset itself on each progress signal; if it fired, it aborted the operation.

func process(ctx context.Context, op *Op) error {
    watchdog := time.NewTimer(5 * time.Second)
    defer watchdog.Stop()

    progress := op.Progress()
    for {
        select {
        case <-progress:
            watchdog.Reset(5 * time.Second) // bug?
        case <-watchdog.C:
            return ErrTimeout
        case <-ctx.Done():
            return ctx.Err()
        }
    }
}

The intended behaviour: each progress signal resets the watchdog. If 5 seconds pass without progress, abort.

Symptoms¶

A small but growing fraction of operations were aborting with ErrTimeout, even when they were clearly making progress (other instrumentation confirmed it). The error rate grew over time, suggesting drift rather than a workload change.

Investigation¶

Reset on a time.Timer has subtle semantics. From the Go documentation (Go 1.22 and earlier):

Reset should be invoked only on stopped or expired timers with drained channels.

The bug: when progress arrived after the watchdog had already fired (but before its channel was drained), calling Reset did not actually re-arm the timer. Instead, the old fired-but-unread tick was now ready on the channel, and the next iteration's select picked it up and aborted.

This race was rare — it required the watchdog to fire in the very narrow window between the progress signal arriving and the Reset call. But it happened often enough that, over the lifetime of a long-running batch, a few percent of operations hit it.

Fix¶

Drain the channel before Reset:

case <-progress:
    if !watchdog.Stop() {
        select {
        case <-watchdog.C:
        default:
        }
    }
    watchdog.Reset(5 * time.Second)

Or, for cleaner code, use context.WithTimeout and refactor to use cancellation rather than timer reset.

Lessons¶

1. Reset is not idempotent and not race-free. The standard pattern is Stop, drain, Reset.
2. Subtle timer bugs often hide behind correctness bugs (e.g., spurious timeouts) rather than memory leaks. They are part of the same family of mistakes.
3. As of Go 1.23, Reset on a fired-but-unread timer behaves more sensibly: the prior tick is discarded. But your code must still work correctly on pre-1.23 runtimes if you support them.

Detection plan¶

The bug above is a correctness bug, not a leak; we include it because the runtime mechanics are identical. The same wrapper that counts live timers can be enhanced to log timer state transitions, which makes Reset/Stop race conditions easier to diagnose:

func (t *Timer) Reset(d time.Duration) bool {
    prev := atomic.LoadInt32(&t.state)
    log.Printf("reset prev=%d", prev)
    return t.Timer.Reset(d)
}

Counting how often Reset is called on a not-fully-drained timer gives you a metric for code paths that may have this race.

Real Incident: The Cancelled context.Context That Wasn't¶

Context¶

A microservice used context.WithTimeout to bound outbound HTTP calls:

func (c *Client) Fetch(parent context.Context, url string) ([]byte, error) {
    ctx, cancel := context.WithTimeout(parent, 30*time.Second)
    defer cancel()
    req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
    resp, err := c.do.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    return io.ReadAll(resp.Body)
}

This looks correct. defer cancel() ensures the timer behind the context is stopped when the function returns. So far, so safe.

But the production code looked slightly different:

func (c *Client) Fetch(parent context.Context, url string) ([]byte, error) {
    ctx, _ := context.WithTimeout(parent, 30*time.Second) // bug: discarded cancel
    req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)
    resp, err := c.do.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    return io.ReadAll(resp.Body)
}

The cancel function was thrown away.

Symptoms¶

The service's heap grew slowly — about 10 MB/day. Pods were restarted weekly and the issue had never been investigated; it was filed as "expected slow growth."

When the team finally took a heap profile, context.WithTimeout's internal timer creation showed up at 35% of live objects. Each call to Fetch leaked one timer for up to 30 seconds (the timeout duration) — but because requests took on average <500 ms, the timers far outlived their actual usefulness.

Investigation¶

context.WithTimeout returns a (ctx, cancel func()) pair. The cancel function not only marks the context as cancelled but also calls Stop on the underlying timer. If you don't call cancel, the timer is not stopped, and it sits in the runtime timer heap until its duration elapses.

Go's go vet has a check called lostcancel that catches exactly this mistake — but in this codebase, vet had been disabled in CI years ago because of a misleading false positive that the team never got around to fixing. The leak was invisible to all the regular guards.

Fix¶

Restore the cancel call:

ctx, cancel := context.WithTimeout(parent, 30*time.Second)
defer cancel()

And re-enable go vet in CI, including the -lostcancel check.

Lessons¶

1. context.WithTimeout (and WithDeadline) creates a timer. Discarding the cancel function leaks it.
2. Go vet's lostcancel analyzer catches this; it is enabled by default and should never be disabled.
3. Per-request leaks scale with QPS. A service doing 1,000 RPS that leaks one timer per request for 30 seconds keeps 30,000 leaked timers live continuously. At 200 bytes each, that's ~6 MB — small enough to hide.
4. The standard idiom in code review is "WithTimeout should always be followed by defer cancel()." Train your reviewers, and document the rule.

Detection plan¶

Beyond go vet, two additional defences:

1. A custom analyzer that flags any call to context.WithTimeout, context.WithDeadline, or context.WithCancel whose second return value is _. This is stricter than lostcancel because it triggers even when the analyzer cannot prove a leak.

// Pseudocode for the analyzer logic.
if assign, ok := node.(*ast.AssignStmt); ok {
    if len(assign.Lhs) == 2 {
        if id, ok := assign.Lhs[1].(*ast.Ident); ok && id.Name == "_" {
            // Check Rhs is context.WithTimeout/WithDeadline/WithCancel
        }
    }
}

1. A wrapper around context.WithTimeout in your code base that mandatorily registers a finalizer or logs the cancel function's stack:

package contextx

import (
    "context"
    "runtime"
    "time"
)

func WithTimeout(parent context.Context, d time.Duration) (context.Context, context.CancelFunc) {
    ctx, cancel := context.WithTimeout(parent, d)
    pc, file, line, _ := runtime.Caller(1)
    timersLeakCounter.WithLabelValues(callerName(pc), file, line).Inc()
    return ctx, func() {
        timersLeakCounter.WithLabelValues(callerName(pc), file, line).Dec()
        cancel()
    }
}

This gives per-call-site leakage metrics. Any growing counter is a code path that doesn't call cancel.

Real Incident: The Ticker Inside a Hot Path¶

Context¶

A service exported per-request metrics. The metrics path created a time.Ticker to periodically flush buffered metric samples to a remote collector.

func (m *Metrics) Record(name string, v float64) {
    if m.lastFlush.IsZero() {
        m.startFlusher()
    }
    m.buf = append(m.buf, sample{name, v, time.Now()})
}

func (m *Metrics) startFlusher() {
    t := time.NewTicker(1 * time.Second)
    go func() {
        for range t.C {
            m.flush()
        }
    }()
    m.lastFlush = time.Now()
}

The author's intent: start the flusher exactly once. The bug: Record was called concurrently from many goroutines, and the lastFlush.IsZero() check was not synchronized. Multiple goroutines passed the check simultaneously and started multiple flushers.

Symptoms¶

After about 15 minutes of uptime, the service's metrics path started failing because the remote collector rejected duplicate samples. The flusher count grew unboundedly: roughly 30 new flushers per second once traffic ramped up.

Each flusher was a goroutine + a ticker + a 1-second-period buffer flush. The leak was therefore all three: goroutines, timers (one per ticker), and CPU (the flushers competed for the lock on m.buf).

Investigation¶

The goroutine profile was unambiguous:

goroutine profile: total 50032
   49998 @ ...
#   0x73a200   main.(*Metrics).startFlusher.func1+0x40 /app/metrics.go:21

50,000 goroutines, all of them flushers. Heap profile showed time.NewTicker at the top of live objects, confirming the paired timer leak.

Fix¶

Make the start-once check atomic:

type Metrics struct {
    once sync.Once
    // ...
}

func (m *Metrics) Record(name string, v float64) {
    m.once.Do(m.startFlusher)
    // ...
}

After the fix, exactly one flusher ran per Metrics instance.

Lessons¶

1. sync.Once is the standard tool for "exactly once" initialization. Don't roll your own with a flag.
2. A double-init bug is doubly bad when the init creates a timer: you now leak both a goroutine and a timer, compounding the cost.
3. A Ticker is a long-lived timer. Leaking one is more expensive than leaking a one-shot Timer because the ticker persists until Stop (which the leaked goroutine never calls).
4. The fix to this leak required a heap profile and a goroutine profile. Neither alone would have been enough: heap profile shows the timer leak but not the goroutine count; goroutine profile shows the goroutines but not the timers. Always look at both.

Detection plan¶

The single best detection for this kind of bug is goleak in CI. A unit test that records metrics, finishes, and runs goleak.VerifyNone(t) would have failed instantly on the first concurrent run. Goleak's failure message would have included the flusher's stack, pointing the author at metrics.go:21 immediately.

Real Incident: The Reconnect Loop¶

Context¶

A streaming ingest service used a long-lived TCP connection to a backend message broker. On disconnect, it reconnected with exponential backoff. The relevant code:

func (c *Client) run(ctx context.Context) {
    backoff := 1 * time.Second
    for ctx.Err() == nil {
        if err := c.connect(ctx); err == nil {
            backoff = 1 * time.Second // reset on success
            continue
        }
        select {
        case <-time.After(backoff):
        case <-ctx.Done():
            return
        }
        backoff *= 2
        if backoff > time.Minute {
            backoff = time.Minute
        }
    }
}

The leaking call: time.After(backoff) inside a for ... select. On every reconnect attempt, the timer was allocated; on success path (no select), no timer was needed.

Symptoms¶

In a healthy environment with rare disconnects, the leak was invisible — the loop slept for a second, the timer fired, all was well. But during an upstream outage where reconnects happened thousands of times per second (with backoff quickly hitting the 1-minute ceiling), the leak accelerated: every reconnect attempt scheduled a 1-minute timer that was rarely consumed before being replaced by the next.

After two hours of upstream brownout, the ingest service consumed 8 GB of memory and OOM-killed. The brownout had been mild (broker was returning 503s, not refusing connections); the side effect was catastrophic.

Investigation¶

Heap profile during the brownout:

3.2GB time.NewTimer
2.1GB chan Time (timer channels)
0.4GB *time.Timer
0.1GB other

99% of heap was timer-related. Live timer count, extracted from the runtime's schedtrace output: 8.3 million. With 60-second timeouts and an attempt-rate of ~140 K/s, that math checks: 140,000 × 60 = 8.4 M.

Fix¶

func (c *Client) run(ctx context.Context) {
    backoff := 1 * time.Second
    t := time.NewTimer(time.Hour) // initial dummy; will be reset
    if !t.Stop() {
        <-t.C
    }
    defer t.Stop()
    for ctx.Err() == nil {
        if err := c.connect(ctx); err == nil {
            backoff = 1 * time.Second
            continue
        }
        t.Reset(backoff)
        select {
        case <-t.C:
        case <-ctx.Done():
            return
        }
        backoff *= 2
        if backoff > time.Minute {
            backoff = time.Minute
        }
    }
}

Reuse a single timer. The leak is gone.

Lessons¶

1. A reconnect loop is a hot loop during incidents. Even if you assume reconnects are rare in steady state, allocate as if they could be common, because during incidents they will be.
2. Leak symptoms can be triggered by upstream brownouts. The leaky code is in your service, but the workload that exposes it comes from elsewhere. Test with chaos-engineering scenarios that simulate flapping upstreams.
3. Connection management code is one of the most leak-prone patterns in Go. Always allocate timers outside loops, always defer-stop, always reuse via Reset.

Detection plan¶

This service did not have per-timer-call-site metrics. After the incident, the team added:

1. A live_timers_per_site gauge using the per-call-site wrapper described earlier.
2. A chaos test that flapped the upstream broker every 30 seconds for an hour and asserted that the ingest service's heap remained bounded.

Both would have caught the bug pre-deploy.

Real Incident: The Per-Request Health Check¶

Context¶

An API gateway performed an upstream health check before forwarding each request. The check used time.AfterFunc to schedule a fallback action if the upstream took too long:

func (g *Gateway) Forward(ctx context.Context, req *Request) (*Response, error) {
    fallback := time.AfterFunc(2*time.Second, func() {
        g.recordSlowUpstream(req.upstream)
    })
    resp, err := g.upstream.Call(ctx, req)
    // bug: fallback is never stopped
    return resp, err
}

If the upstream responded in <2 seconds (the common case), the fallback function ran needlessly 2 seconds after the response. The bug was both a correctness issue (spurious "slow upstream" metrics) and a leak: the *Timer was retained for the full 2 seconds.

Symptoms¶

The "slow upstream" metric was massively inflated. The team initially thought their upstream was misbehaving; only after weeks of escalating with the upstream team did someone notice that the slow-count exactly equalled the request-count.

The leak component was less prominent but real: at 5,000 RPS, the service held ~10,000 unnecessary *Timer allocations continuously (2 s × 5,000/s).

Fix¶

func (g *Gateway) Forward(ctx context.Context, req *Request) (*Response, error) {
    fallback := time.AfterFunc(2*time.Second, func() {
        g.recordSlowUpstream(req.upstream)
    })
    defer fallback.Stop()
    resp, err := g.upstream.Call(ctx, req)
    return resp, err
}

A single defer fallback.Stop() fixed both the correctness bug and the leak.

Lessons¶

1. time.AfterFunc returns a *Timer. The return value exists precisely so you can stop it. If you discard it, you have signed up for leaking the timer for the full duration.
2. A pattern: any time.AfterFunc whose return value is _ is suspicious. A linter that flags this would have caught the bug.
3. The leak component was small relative to the correctness component, but both came from the same root cause. Often a "small leak" is symptom of an undetected correctness bug. Investigate, don't dismiss.

Detection plan¶

// Pseudocode for a custom analyzer.
if call, ok := node.(*ast.CallExpr); ok {
    if sel, ok := call.Fun.(*ast.SelectorExpr); ok {
        if pkg, ok := sel.X.(*ast.Ident); ok && pkg.Name == "time" &&
            sel.Sel.Name == "AfterFunc" {
            // Check that the enclosing statement is not an assignment
            // (i.e., the return value is discarded)
        }
    }
}

Couple this with the timer-count metric and you have both prevention and detection.

Postmortem Template for Timer Leaks¶

Every leak fixed in production should produce a postmortem. The postmortem's job is not blame; it is to ensure the same shape of leak does not recur, and to populate your team's institutional memory. A timer-leak-specific template:

## Postmortem: <service-name> timer leak

### Summary
- One paragraph: what was leaked, when, for how long, what user impact.

### Timeline
- T+0: Deploy of feature X
- T+72h: Alert fired (heap > threshold)
- T+74h: Triage identified time.After in handler.go:42
- T+78h: Patch deployed, leak stopped

### Root cause
- Code snippet of the leaking pattern
- Why it leaked (which API behaviour was misunderstood)
- Which Go version: 1.X.Y

### Why we didn't catch it earlier
- Was vet on? Was staticcheck on? Was goleak on?
- Was there a metric that should have shown it?
- Why didn't the metric alert?

### Impact
- Memory leaked per hour: X MB
- Time to OOM: Y hours
- User-visible impact: Z

### Mitigations
- Code fix (link to PR)
- Detection improvement (new metric / new alert / new linter)
- Documentation update

### Open follow-ups
- [ ] Audit other handlers for same pattern
- [ ] Add lint rule for X
- [ ] Add chaos test for Y