Optimization Workflow — Professional¶

1. The production framing¶

In a production service, the optimization workflow is not "an engineer runs pprof and tunes a function." It is a continuous, team-level practice with budgets, gates, runbooks, and an incident pipeline. The professional job, roughly:

1. Define SLOs that express what "fast enough" means to users.
2. Translate SLOs into per-service performance budgets that the team can engineer against.
3. Continuously measure real user latency, tail behavior, allocation rate, GC CPU, and resource ceilings.
4. Gate releases on regression of the four numbers that matter.
5. Triage and fix the performance bugs that escape the gates.
6. Run periodic deep optimization sprints with executive support, not in spare time.

The rest of this file is what each of those looks like in practice.

2. SLOs and the error budget¶

A performance SLO is two numbers and a window:

"p99 of POST /checkout < 300 ms over a rolling 28-day window, with a 99.9% achievement target."

The "achievement target" leaves a budget: 0.1% of the 28 days (about 40 minutes) can be over the threshold without violating the contract. That budget is what gives the team room to take engineering risks.

Three principles:

The professional alternative to "optimize when someone complains" is "optimize when the error budget says to."

3. Translating SLO into budget¶

The user-visible SLO of 300 ms p99 must be allocated across all the systems that contribute. A typical allocation:

The application's 100 ms is the actual engineering target. Performance work in the service is judged against whether p99 of the service's own processing stays under 100 ms. The senior engineer who set the 100 ms is now the professional who has to defend it across releases.

4. The four dashboards every service needs¶

Each chart should have an alert pointed at the threshold that maps to an action. "p99 climbing toward SLO" is not an alert; "p99 over 250 ms for 5 minutes" is.

5. Performance budgets per route¶

The 100 ms application budget is itself further decomposed into per-route budgets:

The budget is enforced via dashboards. A PR that pushes /checkout p99 from 80 ms to 95 ms in canary is not "still in budget"; it is "burning the budget" and the team should know about it before it merges.

6. The CI gate¶

Microbenchmarks run in CI on every PR. The output is fed through benchstat against a known-good baseline branch.

git fetch origin main
git checkout origin/main
go test -bench=. -benchmem -run=^$ -count=10 ./... > /tmp/base.txt
git checkout -
go test -bench=. -benchmem -run=^$ -count=10 ./... > /tmp/new.txt
benchstat /tmp/base.txt /tmp/new.txt

The gating rule is configured per benchmark:

The CI script enforces this. PRs that violate must add justification or include an off-set elsewhere.

7. Continuous profiling¶

Capturing a profile only when on-call gets paged is too late. The professional answer is continuous profiling: a background agent captures short profiles (10–30 seconds, every few minutes) and ships them to a central store.

Tools:

The value is twofold:

1. When an incident happens, you have profile data from before the incident.
2. Slow drift across releases is visible as a change in profile shape, not just numbers.

A service running a continuous profiler exposes drift weeks before it shows up as an SLO breach.

8. The PGO pipeline¶

Profile-Guided Optimization (Go 1.21+) is integrated into the build pipeline.

production → continuous profile collection → 24h aggregate → default.pgo
                                                     ↓
                                                go build -pgo=auto
                                                     ↓
                                                next release binary

Steps in detail:

1. The continuous profiler aggregates 24 hours of CPU profiles into a single representative profile.
2. A scheduled job pulls the aggregate, validates it (sample count > 100k, average load > N), and writes it as default.pgo in the source tree.
3. The next CI build picks up default.pgo automatically with -pgo=auto.
4. The team reviews the PR that bumps the profile, looking for unexpected shape changes.

Typical wins: 2–10% on CPU. The discipline is keeping the profile fresh — a stale profile produces a binary optimized for the wrong workload. Re-collect monthly or on major workload changes.

9. The canary and the rollback¶

Every release is rolled out gradually. The professional shape of the rollout:

The auto-rollback hook reverts the deployment if any threshold is crossed in any stage. The release engineer doesn't have to be online; the automation does it.

This is the production version of "re-measure after the change." Microbenchmarks said the change was a win; the canary will tell you whether it's a win at real load.

10. Regression incident response¶

When a regression slips through the gate and hits production:

1. Confirm. Latency dashboard shows a step change at the deploy time? Compare canary vs. previous.
2. Rollback first, diagnose second. The service was fine yesterday; restore that state.
3. Bisect. Which commit in the release caused the regression?
4. Capture. A profile from the regressed version (if a pod still exists) vs. a profile from the fixed version.
5. Fix forward. Either revert that commit alone or fix the root cause in a follow-up.
6. Post-mortem. Why did the gate miss this? Add a benchmark or alert that would have caught it.

The post-mortem step is what makes the system better. Each escape is an opportunity to harden the gate.

11. Team practices¶

Performance is a property of the system, but it is maintained by the team. Practices that work:

The professional team treats performance as a first-class engineering concern, not as "we'll get to it after features."

12. The runtime/metrics panel¶

Build one Grafana panel per service exporting these:

These map directly to the prometheus collector exposed by prometheus/client_golang with GoRuntimeMetricsCollection. One panel per service, identical across services, so on-call engineers don't have to relearn each service.

13. Cost-aware optimization¶

At scale, performance is also cost.

Engineering hours are also money. A 10% improvement that costs $500/month in compute but two weeks of engineer time at $5000/week is a $9500 investment for $6000/year of return. The professional makes that math explicit before committing.

14. The optimization sprint¶

Periodic focused optimization works better than continuous trickle-optimization. Structure of a one-week sprint:

The output of the week is (a) measurable improvements landed and (b) a written record of what was tried and why. Both compound across sprints; the second sprint is faster than the first because the team is better calibrated.

15. Cross-team performance¶

When the bottleneck is in a service owned by another team, the workflow includes communication:

The professional doesn't say "their service is slow" and move on. Cross-team performance work is some of the highest-leverage performance work because the bottlenecks usually involve more than one team.

16. Long-lived performance documentation¶

Each service maintains a PERFORMANCE.md (or wiki page) with:

This is the document a new team member reads in their first week. It is the memory of a team. Without it, every new engineer relearns the same lessons.

17. The gates against entropy¶

Performance regresses without active resistance. The five mechanisms that hold the line:

Each one catches what the others miss. A team running all five is far more performant than a team running just one or two.

18. Summary¶

Professional optimization is institutional: SLOs translate into per-route budgets; budgets are enforced by CI gates and canary thresholds; continuous profiling makes drift visible; PGO closes the loop from production back to compile; and team practices keep the work funded against feature pressure. The individual engineer's loop — measure, identify, change, re-measure — is unchanged, but it runs against a backdrop of institutional support and accountability. That's how performance survives organizational entropy.

Further reading¶

• Google SRE book on SLOs: https://sre.google/sre-book/service-level-objectives/
• Go PGO documentation: https://go.dev/doc/pgo
• Grafana Pyroscope: https://grafana.com/oss/pyroscope/
• Polar Signals Parca: https://www.parca.dev/
• Datadog Continuous Profiler: https://docs.datadoghq.com/profiler/
• Susan Fowler, "Production-Ready Microservices" (book), chapter on performance