Modules & Packages — Optimize & Reconcile¶

Package structure is not only a readability concern — it is the single largest lever on incremental build time, startup latency, and binary size. A compiler recompiles, retests, and re-links along package boundaries; a god package or an import cycle forces the whole graph through the slow path on every edit. This file reconciles clean module design with build/runtime cost and shows, with concrete numbers, why good boundaries are also a performance win.

Table of Contents¶

1. God package forces a full recompile on every edit
2. Import cycle defeats incremental compilation (Go)
3. Fan-in hotspot: the types package everyone imports
4. init() ordering inflates startup time (Go)
5. Java static initializer doing eager I/O at class-load
6. Python import-time work blocks CLI startup
7. Barrel file defeats tree-shaking (TS/JS)
8. Unused import keeps dead code in the Go binary
9. Test selection: rebuild/retest only affected packages
10. Deep import chains serialize the build (no parallelism)
11. Re-exporting a third-party type couples your build to theirs
12. internal/ vs leaking types: the API surface as a recompile blast radius
13. Java JPMS / split packages and module-graph resolution

Rules of Thumb · Related Topics

Scenario 1 — God package forces a full recompile on every edit¶

Scenario. A Go service has a single model package holding 140 types — every domain entity, every DTO, every enum. 22 of the 30 packages in the repo import model. A junior adds one field to Invoice (one of those 140 types).

Measurement / reasoning. Go's unit of compilation and caching is the package, not the file. Change any file in model and the compiler invalidates the cached object for the entire model package, then transitively recompiles every package that imports it.

$ touch model/invoice.go && go build ./...
# recompiled: model + 22 importers + their importers = 26 packages
# cold-ish incremental build: 11.4s

Compare the same one-field edit after splitting model into billing, catalog, identity, shipping:

$ touch billing/invoice.go && go build ./...
# recompiled: billing + 4 importers = 5 packages
# incremental build: 2.1s

The edit touched identical bytes of source. The 5.4× difference is entirely the blast radius of the package boundary.

Scenario 2 — Import cycle defeats incremental compilation (Go)¶

Scenario. order imports payment for payment.Charge; payment imports order for order.Status. Go refuses to compile import cycles outright, so a developer "fixes" it by merging both into one package orderpayment. The cycle is gone — but so is the boundary.

Measurement / reasoning. Go's compiler builds a DAG of packages and compiles independent nodes in parallel, caching each. A cycle cannot be a DAG node, so the only way to satisfy Go is to collapse the cycle into one package. That one package is now a recompile unit spanning two concerns: any edit to ordering logic recompiles payment logic and vice-versa.

# before merge (cycle, won't build):  import cycle not allowed
# after naive merge: orderpayment is one 3,400-line package
$ touch orderpayment/status.go && go build ./...   # 4.8s, recompiles all of payment too

Languages that do permit cycles (Java, Python) pay differently: the cycle becomes a single strongly-connected component that incremental tools (Gradle, Bazel, Pants) must treat as one recompile/retest unit. You lose granularity exactly where you most wanted it.

// package order
type Charger interface { Charge(amount Money) error }   // order owns the abstraction

func (o *Order) Settle(c Charger) error { return c.Charge(o.Total()) }

// package payment — depends on order's interface implicitly (structural), no import back
func (p *Processor) Charge(amount Money) error { /* ... */ }

$ touch order/status.go && go build ./...   # 1.3s, payment untouched and cache-hit

Scenario 3 — Fan-in hotspot: the types package everyone imports¶

Scenario. A types (or pb generated-protobuf) package is imported by 90% of the codebase. It is well-factored and feature-grouped — but it changes weekly because the API schema evolves.

Measurement / reasoning. Fan-in is the number of packages that depend on a node. Recompile cost on edit ≈ (fan-in) × (avg compile time of a dependent). A package with fan-in 90 and a 0.2s average dependent cost means every schema tweak costs ~18s of recompilation across the graph.

$ go list -f '{{range .Imports}}{{println .}}{{end}}' ./... | sort | uniq -c | sort -rn | head
     90 myapp/types
     61 myapp/internal/errs
     44 myapp/internal/log

The dangerous combination is high fan-in × high churn. High fan-in is fine if the package is stable (think errors, time — imported everywhere, edited never). High churn is fine if fan-in is low.

Scenario 4 — init() ordering inflates startup time (Go)¶

Scenario. A CLI tool takes 900ms to print --help. Profiling shows the time is spent before main even runs.

Measurement / reasoning. Go runs every imported package's init() functions at startup, in dependency order, before main. A transitively-imported config package has:

func init() {
    cfg = loadFromConsul()        // network round-trip, 600ms
    metrics = registerPrometheus() // 80ms
}

Every binary that imports config — even --help — pays 680ms before doing anything. You can see the init cost:

$ GODEBUG=inittrace=1 ./tool --help 2>&1 | sort -k4 -rn | head
init myapp/config @0.4 ms, 612 ms clock, ...
init myapp/metrics @613 ms, 81 ms clock, ...

// no package-level init that does I/O
func New(ctx context.Context) (*Config, error) {   // called from main, after flag parsing
    return loadFromConsul(ctx)
}

var (
    once sync.Once
    cfg  *Config
)
func Get() *Config { once.Do(func() { cfg = mustLoad() }); return cfg }

Scenario 5 — Java static initializer doing eager I/O at class-load¶

Scenario. A Spring service's cold start is 6.5s. A CountryData utility class loads a 40MB CSV in a static block the first time any code references the class — and class-load is triggered transitively at context-refresh.

Measurement / reasoning. JVM class initialization runs the static initializer on first active use of the class. If a frequently-touched package references CountryData (even via a constant), the 40MB parse happens during bean wiring, on the critical startup path.

class CountryData {
    static final Map<String, Country> ALL;
    static {
        ALL = parseCsv(load("/countries.csv"));   // 40MB, ~1.8s, on class-load
    }
}

Measure with -verbose:class and -Xlog:startuptime, or use the JFR "Class Loading" + "Java Application/Initialization" events. The 1.8s shows up as a stall with no thread doing useful concurrent work.

class CountryData {
    private static final class Holder {                 // initialization-on-demand holder
        static final Map<String, Country> ALL = parseCsv(load("/countries.csv"));
    }
    static Map<String, Country> all() { return Holder.ALL; }
}

Scenario 6 — Python import-time work blocks CLI startup¶

Scenario. mytool --version takes 1.4s. The top-level __init__.py does from .commands import *, which imports pandas, torch, and a cloud SDK — none needed for --version.

Measurement / reasoning. Python executes module bodies at import time, and import is transitive and eager. A wildcard barrel import in the package root drags in the entire dependency tree before argparse even sees the flags.

$ python -X importtime -c "import mytool" 2>&1 | sort -k2 -rn | head -3
import time:  720146 |  720146 | torch
import time:  410233 |  430880 |   pandas
import time:  ...

torch alone is 720ms of import. The user asked for a version string.

def predict(args):
    import torch          # paid only when `predict` actually runs
    import pandas as pd
    ...

def version(args):
    print(__version__)    # zero heavy imports

Scenario 7 — Barrel file defeats tree-shaking (TS/JS)¶

Scenario. A frontend imports one helper: import { formatDate } from "@acme/utils". The production bundle grows by 180KB because @acme/utils is a barrel (index.ts re-exporting 60 modules including a date library, a charting helper, and a markdown parser).

Measurement / reasoning. Tree-shaking (dead-code elimination by the bundler) relies on the static import graph and on modules being side-effect-free. A barrel index.ts that re-exports everything makes every consumer appear to depend on every member. If any re-exported module has a side effect — or the bundler can't prove it doesn't ("sideEffects" not set in package.json) — the whole barrel is retained.

# import one function, ship the world:
$ npx source-map-explorer dist/main.js
@acme/utils  ............ 180 KB  (formatDate is 0.4 KB of it)

Scenario 8 — Unused import keeps dead code in the Go binary¶

Scenario. A 6MB Go service grew to 48MB. A debug-only package imports net/http/pprof and a heavyweight visualization library at package scope, "just in case."

Measurement / reasoning. Go's linker performs dead-code elimination, but it works at the granularity of reachable symbols. An imported package's init() and any symbol reachable from main are retained. net/http/pprof's init() registers HTTP handlers — that init is reachable the moment the package is imported, which roots a large dependency subtree the linker then cannot drop.

$ go tool nm -size server | sort -k2 -rn | head     # biggest symbols
$ go build -ldflags="-s -w" ... && ls -lh server    # strip symbol table: 48M -> 39M
# but the real bloat is the retained pprof/viz subtree, not symbols

(Go won't even compile an unused import — imported and not used — so the cost here is from imports that are referenced but only by dead-by-intent code paths.)

//go:build debug

package main
import _ "net/http/pprof"   // only compiled into the `-tags debug` binary

Scenario 9 — Test selection: rebuild/retest only affected packages¶

Scenario. CI runs the full suite (14 min) on every PR, even one-line changes. The team blames test count; the real lever is package boundaries.

Measurement / reasoning. Build tools that hash inputs (go test cache, Bazel, Nx, Pants, Turborepo) skip any test target whose transitive inputs are unchanged. The unit of skipping is the package/target. With good boundaries, a one-line change in billing reruns only billing and its dependents.

# Go: per-package result caching
$ go test ./...                 # first run: 14m12s, compiles+runs everything
$ touch billing/price.go
$ go test ./...                 # 41s: only billing + 4 dependents rerun; rest "(cached)"

# Nx / Bazel: affected-only graph
$ nx affected --target=test --base=main   # tests only projects touched by the diff
$ bazel test //...                         # cache-hits everything unchanged; runs the SCC of the edit

The catch: a god package or an import cycle (Scenarios 1–2) collapses these affected-sets. If everything depends on model, then "affected by a model edit" = "everything," and caching buys nothing.

Scenario 10 — Deep import chains serialize the build (no parallelism)¶

Scenario. A Go module builds in 95s on a 16-core machine, but CPU sits near 15% the whole time. The dependency graph is a near-linear chain: a → b → c → d → … → p (16 packages, each importing only the next).

Measurement / reasoning. Compilers parallelize the build by walking the package DAG and compiling independent nodes concurrently. A chain has no independent nodes — b can't start until a is done, c waits on b, and so on. Build time ≈ sum of the chain (serial), not max-over-parallel-layers.

$ go build -p 16 ./...         # -p sets parallelism; chain ignores it
# critical path = 16 sequential compiles ~= 95s, cores idle

Contrast a wide graph where one core package fans out to 16 independent leaves: those 16 compile in parallel, and wall-time ≈ core + one layer.

Scenario 11 — Re-exporting a third-party type couples your build to theirs¶

Scenario. Your api package re-exports github.com/stripe/stripe-go's Charge type directly in its public signatures. Every package that touches your API transitively compiles against the entire Stripe SDK.

Measurement / reasoning. When a public function signature contains a foreign type, every importer must resolve and compile against that foreign package. A Stripe SDK bump (frequent) now invalidates the compile cache of every package in your fan-in, and the SDK's own large transitive tree is dragged into your build graph.

$ go list -deps ./api | grep -c stripe     # 38 stripe sub-packages pulled in transitively
# stripe minor version bump -> recompiles all 38 + everyone importing ./api

This is the "re-export third-party types" anti-pattern from the README, measured as build cost: hidden coupling becomes hidden recompilation.

// your package — owns this type, stable, no stripe in the signature
type Payment struct { ID string; AmountMinor int64; Currency string }

// translation lives in ONE adapter package that imports stripe
func fromStripe(c *stripe.Charge) Payment { /* ... */ }

Scenario 12 — internal/ vs leaking types: the API surface as a recompile blast radius¶

Scenario. A library package exposes a struct with all-public fields. Consumers reach into those fields directly. The maintainer wants to change the internal layout but every field change is a breaking, recompile-everything event.

Measurement / reasoning. The exported surface of a package is the set of things its importers can be coupled to — and therefore the set of changes that force recompilation (and, in compiled-distribution languages, re-release) of every consumer. A wide public surface = a wide recompile blast radius. A narrow one lets you change internals freely with zero downstream rebuilds (cache hits).

// wide surface: every field is a coupling point + a recompile trigger
type Client struct {
    HTTP    *http.Client
    BaseURL string
    Retries int
    cache   map[string]string  // can't even change this layout without risk
}

type Client struct{ /* all unexported */ }
func New(opts ...Option) *Client { /* ... */ }
func (c *Client) Get(ctx context.Context, path string) (*Response, error) { /* ... */ }

Scenario 13 — Java JPMS / split packages and module-graph resolution¶

Scenario. Migrating a Java app to the module system, the build fails with "module reads package X from both A and B" (a split package), and module resolution adds noticeable time to large multi-module Gradle builds.

Measurement / reasoning. JPMS resolves a module graph at compile and launch. Split packages (the same package name living in two modules) are forbidden because the runtime must map each package to exactly one module — an ambiguity that also wrecks incremental compilation, since the compiler can't attribute a recompile to a single owning module.

# module-info.java
module com.acme.billing {
    requires com.acme.core;
    exports com.acme.billing.api;        // only this package is visible downstream
    // com.acme.billing.internal is NOT exported -> not a recompile trigger for consumers
}

The exports clause is the Java analogue of Go's internal/: it defines the public recompile surface. Anything not exported can change without forcing dependents to recompile.

Rules of Thumb¶

• Recompile blast radius = boundary quality. If editing one file recompiles half the repo, the boundary is wrong — measure it (go list -deps, reverse-import counts, Gradle/Bazel affected sets), don't eyeball it.
• High fan-in is only safe with low churn. A package everyone imports must be stable. Split it by churn: stable core (keep high fan-in) vs. volatile surface (reduce fan-in).
• Cycles are a build bug, not just a smell. They collapse the parallelizable DAG into one serial recompile/retest unit. The clean fix (invert the dependency) is the fast fix.
• init() / static blocks / module bodies must not do I/O. Register, don't connect. Push expensive work behind lazy entry points (sync.Once, holder idiom, lazy imports). Startup latency is paid by every invocation, including --help.
• A package's public surface is its minimum shippable + recompilable unit. Narrow it: internal/, JPMS exports, curated index.ts. Encapsulation is a build-incrementality strategy.
• Barrels and wildcard re-exports trade author convenience for every consumer's bundle/startup. Import from specific modules; mark packages side-effect-free.
• An import is a binary-size and build-graph dependency, not just a name. Build-tag out optional heavyweight subsystems; confine volatile third-party SDKs to a single adapter package.
• Optimize graph depth, not just node count. A long import chain serializes the build and idles cores; a wide, shallow DAG parallelizes.
• Don't re-introduce a god package for speed. Merging packages to "break a cycle" or "reduce hops" usually worsens incremental build by widening recompile units. Invert dependencies instead.
• Keep tests hermetic so package-level caching works. Non-determinism (clock, network, global state) silently disables go test/Bazel/Nx caching and forces full reruns.

Related Topics¶

• find-bug.md — detecting cycles, fan-in hotspots, and god packages statically.
• professional.md — test hygiene and hermeticity that keep package-level caching honest.
• Chapter README — the positive module/package design rules these optimizations reconcile with.
• Boundaries — confining third-party and volatile surfaces behind stable adapters.
• Refactoring — seams, dependency inversion, and Move Method for restructuring a dependency graph without changing behavior.