The Go Execution Tracer — Specification¶

Focus: Precise reference for runtime/trace, the events the runtime emits, and the go tool trace viewer that consumes them.

Sources: - runtime/trace package: https://pkg.go.dev/runtime/trace - net/http/pprof package: https://pkg.go.dev/net/http/pprof - cmd/trace source: https://github.com/golang/go/tree/master/src/cmd/trace - Diagnostics guide: https://go.dev/doc/diagnostics - Flight recorder proposal: https://go.dev/issue/63185

1. What the tracer is¶

The Go execution tracer is a runtime-integrated event recorder. Unlike CPU profiling, which samples PCs on a timer, the tracer logs every scheduler decision, GC phase, syscall transition, and goroutine state change on every P (processor). The result is a chronological log from which the viewer reconstructs precise wall-clock timelines for every goroutine, every P, and every OS thread.

2. Public API — runtime/trace¶

trace.Start and trace.Stop are global: the runtime tracks one active writer per process.

3. Capture methods¶

import _ "net/http/pprof"
// curl 'http://localhost:6060/debug/pprof/trace?seconds=5' > trace.out

4. Event taxonomy¶

The runtime emits ~40 event kinds. The viewer groups them into a small number of categories.

Each event carries: timestamp, goroutine ID, P ID, M (OS thread) ID, and event-specific arguments (e.g., the reason for a block, the new size goal).

5. The viewer (go tool trace)¶

go tool trace trace.out parses the file and starts a local HTTP server (default 127.0.0.1: ephemeral port) that renders the trace in a browser. The main views:

The flame view is the most powerful and the most overwhelming. Start in the analysis views to find a candidate goroutine, then jump into the timeline at that point.

6. The flame timeline layout¶

Reading top to bottom, the timeline shows:

PROCS  : P0   ████████░░████  ░░████░░░░██  (goroutine ID per slice)
         P1   ░░░░██████░░░░  ████████░░██
GC     :      ░░░░██MARK░░░░  ░░░░░░░░░░░░
STW    :                 ▌▌
HEAP   : ─live──goal─────────────────────────
THREADS: M0  M1  M2  M3

• The PROCS lanes show which goroutine ran on each P at each moment. Idle = gap.
• The GC lane shows mark and sweep phases.
• Vertical bars in the STW lane mark stop-the-world points.
• The HEAP lane plots live and next_gc goals over time.
• The THREADS lane shows OS threads (Ms) and which P (if any) they ran.

Clicking a slice opens a sidebar with the goroutine ID, the call stack at the start of the slice, and links to the events that bracketed it.

7. Goroutine states the tracer distinguishes¶

A trace slice tells you, for every nanosecond of every goroutine, which of these it was in.

8. User-level instrumentation¶

The runtime accepts three user events to embed semantic meaning in the trace.

Tasks accept arbitrary nesting via context.Context. Regions cannot span goroutines.

9. Tracer overhead¶

Overhead is measured per-event, not per-call. Each event writes ~10-30 bytes to a per-P buffer.

Output size: roughly 1-5 MB per second of wall clock at moderate concurrency. A 30-second trace from a busy service can produce 50-200 MB.

10. Trace vs CPU profile vs heap profile¶

The CPU profile is the right first stop when "the program is slow." The trace is the right first stop when "the program is slow at that moment," when latency tails are wide, or when concurrency is misbehaving.

11. Flight recorder (Go 1.25+)¶

The flight recorder is a circular trace buffer that runs continuously but only writes on demand.

fr, _ := trace.NewFlightRecorder(trace.FlightRecorderConfig{
    MinAge:   5 * time.Second,
    MaxBytes: 16 << 20,
})
fr.Start()
defer fr.Stop()

// At incident time:
f, _ := os.Create("incident.out")
fr.WriteTo(f)
f.Close()

Steady-state overhead is comparable to a full trace, but no output is produced unless WriteTo is called. The captured window is the most recent MaxBytes of events that are at least MinAge old.

12. File format¶

The binary trace format is documented in internal/trace/format.go. Practical notes:

• Each event is variable-length, LEB128-encoded.
• Events are batched per-P; the reader merges streams by timestamp.
• Headers carry the format version, frequency of the monotonic clock, and procedure tables.
• The format is not stable across major Go versions in detail — read with the matching go toolchain.

The golang.org/x/exp/trace library provides a stable programmatic reader for custom analysis.

13. Limitations¶

• A trace records what the Go runtime sees. Time spent in cgo, in the kernel beyond a syscall return, or in another process is opaque.
• The viewer becomes sluggish above ~500 MB. Slice large traces with --starttime/--endtime or by recording shorter intervals.
• The "Synchronization blocking profile" attributes a block to the goroutine that blocked, not to the goroutine that held the resource. Identifying the holder requires reading the timeline.
• Goroutine IDs are reused after a goroutine exits; if you need to correlate across reuse, attach a trace.NewTask.

14. Related references¶

• runtime/trace package: https://pkg.go.dev/runtime/trace
• Diagnostics guide: https://go.dev/doc/diagnostics
• Dmitry Vyukov, original tracer design: https://docs.google.com/document/d/1FP5apqzBgr7ahCCgFO-yoVhk4YZrNIDNf9RybngBc14
• Felix Geisendorfer, "Reading a Go trace": https://blog.felixge.de/reading-go-execution-traces/
• Michael Knyszek, flight recorder proposal: https://go.dev/issue/63185