Integration Tests — Professional¶

← Back

The professional view treats integration tests as a service offered to the rest of the engineering organization. The contract is: every push gets a fast, reliable signal that the system's contracts with its dependencies still hold. Achieving that at scale requires budgets, ownership and continuous tuning — not just clever Go code.

1. Test pyramid at the org level¶

Publish quotas for each tier and review them quarterly. Example numbers from a mid-size Go monorepo of about 600k lines:

Anything taller than this pyramid drives developers toward t.Skip() and erodes signal. Anything flatter rarely catches contract bugs. Track counts and durations in a quarterly review with squad leads.

2. Ownership matters¶

Without an owner, the integration suite drifts. Assign one squad — often "platform" or "developer experience" — the mandate to:

• Pin image digests and rotate them.
• Maintain the testenv harness.
• Track P95 wall time across CI runs.
• Review and prune flakes.

Squads keep ownership of the test bodies they write; the platform owns the substrate.

3. Container orchestration¶

Pin image digests, not tags. postgres:16-alpine is reproducible only until the publisher pushes a new build with the same tag. Store digests in a single Go file checked into the repo:

package images

const (
    Postgres = "postgres@sha256:1f...c4"
    Redis    = "redis@sha256:99...01"
    Kafka    = "confluentinc/cp-kafka@sha256:7a...ff"
)

Refresh digests on a schedule, with a bot opening a PR that re-runs the full suite. The PR description includes a diff of the upstream release notes for the version bump.

4. CI service containers¶

GitHub Actions and GitLab CI expose service containers natively. When images are massive and the test fleet is large, declarative services beat testcontainers because they share an image cache across jobs.

jobs:
  integration:
    runs-on: ubuntu-24.04
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_PASSWORD: test
        ports: ["5432:5432"]
        options: >-
          --health-cmd "pg_isready -U postgres"
          --health-interval 5s
          --health-timeout 5s
          --health-retries 5
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
        with: { go-version: '1.24' }
      - run: go test -tags=integration ./...
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/postgres?sslmode=disable

The same workflow exists for GitLab via the services: keyword.

A hybrid strategy works well: services declared in CI for the hot path (Postgres, Redis), testcontainers-go inside the test process for less common dependencies (LocalStack, Wiremock, custom images).

5. Flake budget¶

Treat flakes as bugs. Establish a budget such as "no more than 0.5% of test runs may be auto-retried" and burn it down. A flaky test scheduled for quarantine should:

1. Be tagged //go:build integration && flaky.
2. Have a ticket open with an owner.
3. Be excluded from the merge-gate.
4. Be deleted if the ticket ages out beyond two sprints.

This breaks the "retry until green" cycle that hides genuine race conditions. Auto-retries are tempting because they hide pain in the short term while inflating it in the long term.

6. Observability inside the suite¶

Wrap container start, schema apply and seed steps with structured logs and metrics so that slow suites can be diagnosed:

defer trace.StartRegion(ctx, "postgres-start").End()
metrics.Histogram("test_setup_seconds", time.Since(t0).Seconds(), "phase", "pg")

gotestsum --junitfile out.xml exports timings to your CI dashboard. The top ten longest tests are the next optimization target. Run trends weekly; sudden growth in any one test is your early-warning system.

7. Sharding¶

Once the suite passes 5 minutes wall-clock, shard across N runners. go test does not shard natively, so use a wrapper that distributes packages by hash:

PACKAGES=$(go list -tags=integration ./...)
SHARD_PACKAGES=$(echo "$PACKAGES" | awk -v s=$SHARD -v n=$TOTAL '{ if (NR % n == s) print }')
go test -tags=integration -count=1 $SHARD_PACKAGES

Eight shards run eight times faster (minus startup overhead). Plot the critical-path shard duration on the dashboard.

8. Cost discipline¶

Each container costs CPU minutes. On AWS m6i.large running 100 PRs/day times 4 minutes equals roughly 6.7 instance-hours daily. Container reuse and image caching cut that in half. Track test_minutes_per_pr as a KPI; revisit when the number grows by more than 20% quarter-over-quarter.

Cloud runners are billed per-minute. If you can shave one minute from average suite wall-clock, multiply by daily PR count and engineering hourly cost to get the financial return.

9. Documentation and onboarding¶

A TESTING.md at the repo root names:

• The build tag (integration).
• The harness package (internal/testenv).
• Local prerequisites (Docker, Go version).
• The standard run commands and where to look in CI when something fails.

Engineers should be writing their first integration test within ninety minutes of onboarding. If the friction is higher, the suite is too bespoke.

10. Avoid common organizational antipatterns¶

• A single "QA team" owns all integration tests. Squads stop investing in test quality.
• The suite is allowed to break overnight, gated only by a daily build. Bugs land in main for hours.
• Test execution is decoupled from PR review. People merge red builds.
• Flake quarantines become permanent.

Each of these tends to escalate; correct them before they harden.

11. Quarterly health review¶

Run a recurring meeting (60 minutes) where engineering leads inspect:

• The integration suite's wall-clock P95.
• The flake rate.
• The slowest 10 tests and their owners.
• Container images pinned to digests; pending refreshes.
• Coverage of new features merged in the previous quarter.

Outcomes are a short list of action items assigned to squads. A quarterly cadence catches drift before it becomes a fire.

12. Image lifecycle policy¶

Container images go stale. Establish a written policy:

• Pin all production-shape images by digest.
• Refresh digests monthly via an automated PR.
• Block merges that introduce images without a digest.
• Allow :latest only in throwaway exploration, never in CI.

A short Go file makes the policy enforceable in code:

package images

import "regexp"

var digestRE = regexp.MustCompile(`^[a-z0-9./-]+@sha256:[a-f0-9]{64}$`)

const (
    Postgres = "postgres@sha256:..."
    Redis    = "redis@sha256:..."
)

func init() {
    for _, img := range []string{Postgres, Redis} {
        if !digestRE.MatchString(img) {
            panic("image not pinned by digest: " + img)
        }
    }
}

A go vet rule or linter such as golangci-lint custom analyzer catches drift at PR time.

13. Capacity planning¶

Container-based testing has real capacity needs:

• Disk: each Postgres container needs ~50 MB. A worker running 10 parallel tests needs at least 500 MB free.
• Memory: Postgres defaults to 128 MB shared buffers. Multiply by parallelism.
• File descriptors: each container opens ~20 sockets. macOS defaults to 256 fds per process; raise via ulimit -n 4096 in the test wrapper.
• Network: Docker creates a bridge per project. Heavy parallelism may hit kernel networking limits; sysctl net.core.somaxconn is a common culprit.

Document these in TESTING.md so new engineers do not silently hit limits.

14. Runbook for "integration tests are slow"¶

Common diagnostic tree:

1. Check the long pole: gotestsum --jsonfile out.json | jq -s 'sort_by(-.elapsed)[:10]'. Investigate the top three.
2. Check container reuse: are tests spinning their own containers instead of sharing via TestMain?
3. Check parallelism: is -parallel set lower than necessary?
4. Check image pulls: is the Docker cache warm? docker images should list all required images locally.
5. Check the host: high disk IO wait or CPU saturation explains suspicious test latency.

Capturing this in a runbook saves the next engineer half a day.

15. Runbook for "integration tests are flaky"¶

1. Identify the test from the CI logs. Note the failure message.
2. Re-run locally with -count=10 -shuffle=on. Reproduce.
3. If not reproducible locally, examine timing: missing wait strategy, time.Sleep, fragile network call.
4. Check shared state: package-level variables touched by parallel tests.
5. Check determinism: random seed, map iteration order, time-of-day dependence.
6. Fix the root cause. Resist auto-retry as a long-term answer.

A flake retrospective every quarter spreads this knowledge across the team.

16. Onboarding¶

Every new engineer should write and ship an integration test on day one. Provide a guided walkthrough:

• Clone the repo.
• Read TESTING.md (10 minutes).
• Open the harness package (internal/testenv) and read the public surface (15 minutes).
• Pick an existing test, copy it, modify the assertions, run locally.
• Open a PR against a starter exercise. Reviewer signs off in a day.

Within 24 hours, the engineer has a green build in CI and basic familiarity with the suite. The first week is the easiest time to absorb conventions; capture it.

17. Multi-repo coordination¶

Larger organizations have many repositories, each with its own integration suite. To keep them consistent:

• Publish the testenv harness as a private Go module that other repos depend on.
• Tag image digests in a shared file referenced from each repo.
• Run periodic surveys: count flake rate, runtime, coverage across repos. Repos drifting from the mean get a friendly nudge.
• Hold a cross-repo "test infra" guild meeting monthly.

Without coordination, every team reinvents subtly different harnesses; upgrading one common dependency (e.g. Postgres major version) becomes weeks of toil.

18. Vendor-lock considerations¶

testcontainers-go and ory/dockertest both work without Docker, but the practical reality is they need a Docker-compatible daemon: Docker Desktop, Podman, colima, or a Docker-in-Docker setup on CI.

Senior teams document the supported runtimes and gate CI to the same runtime that developers use locally. Heterogeneity here amplifies flakes — a test that works on Docker Desktop fails on Podman because of subtle network namespace differences.

19. Long-term cost reduction strategies¶

Over many years, these strategies compound:

• Caching at every layer. Image registry cache, build cache, module cache, test result cache.
• Right-sizing CI runners. Larger machines test faster, but bill per minute. Find the sweet spot empirically.
• Selective tests. When a PR touches only one package, only run that package's integration tests plus a small smoke set. Tools like Bazel or custom scripts compute the dependency graph.
• Containers as a service. Some teams run a hosted pool of warmed-up Postgres containers; tests check one out, use it, return it. Saves the 3-second start time entirely.

None of these are easy. All of them pay back in a sufficiently large suite.

20. The maturity model¶

A simple model for tracking where your suite is:

Most organizations operate at level 2-3. Senior engineers push toward 4-5. Each step has clear ROI; none is glamorous.

21. Final professional thought¶

Integration tests at scale are an operational practice, not a coding exercise. The code is straightforward — the discipline, ownership and budgets are what determine whether the suite is a help or a hindrance five years from now.

A team that takes its integration suite seriously ships faster than one that treats it as overhead. The compounding effect of preventing production incidents through realistic pre-merge testing dwarfs the cost of the suite itself.

Invest in the suite; the suite invests in your team.

22. Case study — a tier-3 incident traced to test gaps¶

Real shape of incidents you can prevent:

A payment service rolled out a schema change that added a NOT NULL constraint without a default. Unit tests with a mocked database passed because the mock did not enforce constraints. The deploy hit production and 0.4% of transactions failed for ninety seconds before rollback.

Post-mortem action: an integration test that exercises the migration against a real Postgres, asserting INSERT statements still succeed without explicit NOT NULL columns. The test would have caught the issue in CI.

This is the canonical case for integration tests. Each gap-driven incident, properly investigated, generates one or more new integration tests that ratchet down the production failure surface.

23. Designing for replaceability¶

Service dependencies change. Postgres becomes Aurora. Kafka becomes Pulsar. Redis becomes Dragonfly. The integration harness should make these migrations cheap.

Design principles:

• Test against an interface, not a concrete database. The harness returns DBTX, not *pgx.Conn.
• Pin image digests in one file; swap to alternative images in that one place.
• Avoid driver-specific assertions; use errors.Is against sentinel errors.
• Keep migrations vendor-neutral where possible; isolate vendor- specific syntax behind a flag.

When the day comes to migrate, you change the harness's images.go and the substrate underneath, not every test.

24. Reporting standards¶

A professional integration suite reports:

• Pass/fail count per package, per run.
• Median, P95, P99 wall time per test.
• Setup time vs assertion time per test.
• Number of flake-retries triggered.
• Counts of skipped tests with reasons.
• Coverage percentage (go test -cover) when meaningful.

Dashboards aggregate these by week, month, quarter. Trends matter as much as absolute values; sudden growth is a leading indicator of suite rot.

25. Compliance and audit considerations¶

Some industries require evidence that tests ran before release. Tools like Sigstore can sign test artifacts; CI systems can attest to test execution.

For regulated environments:

• Store JUnit XML reports as build artifacts for seven years.
• Tie commit SHAs to test runs in your CI dashboard.
• Reproduce a historical test run by checking out the commit and pinning the image digests recorded at the time.

This level of rigor is overkill for many teams; it is essential for finance and healthcare.

26. People aspects¶

Behind every healthy suite is a small group of engineers who care about test quality more than features. Recognize them; protect their time; promote them. The most underrated career path in a backend team is becoming the person who keeps the suite green.

Tradition: rotate "test sheriff" duty weekly. The sheriff watches for new flakes, triages them within 24 hours, and pulls in domain owners to fix. Two months of sheriff duty is the fastest way to internalize the suite's quirks.

27. The professional mindset¶

Junior engineers ask "how do I write this test?" Senior engineers ask "will this test be reliable?". Professional engineers ask "will this test, in aggregate with the suite, give the organization the confidence to ship?"

The mindset shift is from individual tests to suite-level properties: flake rate, wall time, coverage, cost, ownership. Practising that shift is what makes you professional.

28. Closing the professional page¶

You now understand how to scale Go integration testing from a single test to an organizational practice. The patterns transfer to any language, but the Go-specific tooling — testcontainers-go, ory/dockertest, build tags, TestMain — gives you a complete toolkit.

Apply the patterns where they fit. Resist applying them where they do not. The judgement of when each pattern earns its complexity is what distinguishes professional engineers.

29. Migration playbooks¶

When the org adopts a new testing pattern (say, switching from dockertest to testcontainers-go), publish a playbook:

1. Pilot in one squad for two weeks. Measure pain points.
2. Document migration steps in MIGRATIONS.md.
3. Provide a Slack channel for questions.
4. Tag remaining repos by ETA. Track in a dashboard.
5. After all repos migrate, remove the old library from go.mod.

A formal playbook prevents the half-migration trap where two patterns coexist for years.

30. Test data lifecycle¶

In production, you carefully archive and delete old data. In tests, you generate fresh data each run. The harness should:

• Mark every test database with a prefix (t_).
• Run a cleanup job nightly that drops orphans.
• Refuse to run if the host has more than N test databases (a sign of past leaks).

func init() {
    var n int
    _ = admin.QueryRow(`SELECT count(*) FROM pg_database WHERE datname LIKE 't_%'`).Scan(&n)
    if n > 100 {
        log.Fatalf("too many leaked test dbs (%d); clean up before running", n)
    }
}

Guard rails save you from "the suite is slow because we are running 3000 left-over databases" debugging sessions.

31. Tooling investments worth making¶

In order of return on investment:

1. A testenv harness package. Single most leveraged investment.
2. gotestsum with JUnit XML. Required for any CI dashboard.
3. Image digest pinning policy. Prevents drift.
4. Flake tracker. A spreadsheet is enough; a dashboard is better.
5. CI shard runner. Worthwhile beyond 5-minute suites.

Each of these takes one engineer-week to set up and saves multiples of that over a quarter.

32. Tracking value delivered¶

Quantify what integration tests catch:

• Every production incident has a "could integration tests have caught this?" field in the post-mortem template.
• If yes, the action item is "write the missing test" — and it ships the same week.
• Track the count over time. Trend should slope down (fewer escapes) as coverage grows.

This metric, more than any other, defends the integration suite's budget when leadership asks why CI takes so long.

33. Healthy disagreements¶

Some debates have no universal answer; pick a side per team:

• testcontainers-go vs dockertest? Either works; do not mix.
• Database-per-test vs transaction-per-test? Mix as needed.
• Run E2E in CI or only nightly? Depends on your release cadence.
• Test coverage as a merge gate? Useful threshold (60-70%); rigid 100% wastes time.

Healthy teams disagree and decide. Unhealthy teams either avoid the conversation or revisit it endlessly. Document the decision; revisit yearly.

34. A note on AI-assisted testing¶

By 2026, AI assistants help generate test scaffolds. They are useful for:

• Filling in factories from struct definitions.
• Generating happy-path tests for new handlers.
• Suggesting wait strategies for unfamiliar containers.

They are weaker at:

• Choosing which boundaries to integration-test.
• Designing harness abstractions.
• Diagnosing flakes.

Use AI to accelerate well-understood patterns; reserve human judgement for the design choices that determine suite quality.

35. The closing professional thought¶

Integration tests are an operational practice. They are how a team keeps a complex system trustworthy as it evolves. They are not a substitute for code review, design discipline, or operational expertise — they are one tool among several.

When the team trusts the integration suite, every other engineering practice gets easier. Refactors land confidently. Migrations proceed incrementally. Releases happen on schedule.

That trust is what you are building. Every well-written integration test is a contribution. Every flake you fix is a contribution. Every new engineer you onboard into the harness is a contribution.

The suite is the team's reflex memory.

36. Hand-off checklist¶

When a senior engineer leaves and someone else inherits the harness, the hand-off should cover:

• Where the harness lives (internal/testenv).
• Which images are pinned where (internal/testenv/images.go).
• Which CI workflows run integration tests.
• What the current flake rate looks like.
• Outstanding flake quarantines and their tickets.
• Vendor relationships (e.g., LocalStack, Wiremock).

A short document at internal/testenv/OWNERS.md covers most of this. Update it whenever ownership changes.

37. Last word on culture¶

The most effective integration suites belong to teams where every engineer feels permission to fix a flake on sight. The least effective suites belong to teams that defer to "the testing team" or "the platform team" for every issue.

Empowerment is cheap to grant and immediate in returns. Document the suite well, hold a short workshop, and stand back. Engineers do the right thing when they have the tools and the trust.

38. The single most important graph¶

If you only track one chart for the integration suite, track this:

P95 wall time per integration test run, weekly, for the past year.

A flat line means the suite is healthy. A creeping line means investment is overdue. A jagged line means flakes need attention.

Project that graph at every quarterly review. Engineering leadership will start asking the right questions; the suite gets the budget it needs.

39. The end of the professional page¶

You have read every page in this section. The integration testing craft, from junior first-test through professional governance, lives on a single set of patterns. Each tier adds another layer of sophistication, but none discards the previous tier's lessons.

The next time you write or review an integration test in Go, the patterns should feel natural. If they do not yet, that is fine — re-read the page that covers the patterns you find awkward. Practice is what closes the gap.

The integration suite is one of the highest-leverage investments a backend team makes. Treat it with the seriousness it deserves and it will repay your team for years.

40. Where to refer back¶

When organizational questions arise:

• Quotas: Section 1.
• Ownership: Section 2.
• CI: Sections 4, 16.
• Flakes: Sections 5, 15.
• Sharding: Section 7.
• Cost: Sections 8, 19.
• Onboarding: Section 16.
• Maturity model: Section 20.
• Migrations: Section 29.
• Tooling investments: Section 31.

The page is a reference, not a story. Read it through once; come back to specific sections as situations arise.

41. Acknowledgement¶

The patterns documented in this section reflect the collective experience of the Go community over a decade of building production services. Credit goes to the authors of testcontainers-go, ory/dockertest, golang-migrate and the many smaller libraries that comprise the modern Go testing ecosystem.

Without their work, none of this would be possible. The community contribution is the substrate; this section is a way to navigate it.

42. Section complete¶

You have read every page of the integration testing section. From spec to professional, from first test to organizational governance, the same patterns recur in increasing depth.

Apply them where they fit. Skip them where they do not. Trust your judgment — and refine it by reading test code from teams you respect.

43. One more time¶

The compounding gains from a healthy integration suite are easy to underestimate and impossible to retrofit cheaply. Build the substrate early; tend it consistently; the work pays back over the entire life of the codebase.

That is the professional view. Go put it into practice.

When you look back in five years at the suite you helped build, you will recognize the bugs it caught, the engineers it onboarded, and the production incidents it prevented. The compounding effect of disciplined investment in integration testing is hard to see quarter-by-quarter and impossible to miss in retrospect.

Build the suite. Tend the suite. Trust the suite.

The end.