Expand-Contract Refactors — Find the Bug¶

Category: Anti-Patterns at Scale → Expand-Contract Refactors Covers (collectively): Parallel Change (expand-contract) · Backward & forward compatibility · Deprecation windows · Schema / API / event / DB evolution · Dual-write / dual-read & Tolerant Reader

This file is critical-reading practice. Each snippet below is a plausible migration step that someone shipped believing it was safe. Read it the way a reviewer who has caused an outage reads it, and answer:

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

These aren't syntax bugs — every snippet compiles and passes its own unit tests. The failure lives in the ordering of separate deploys/migrations and only appears when old and new versions run together, when a backfill is mid-flight, or when an in-flight message arrives. That's exactly what makes them dangerous: each PR looks correct in isolation.

How to use this file: write your own answer before expanding the collapsible. The skill is seeing the missing step in the sequence, not recalling the name of the pattern.

Table of Contents¶

1. Drop-the-column rename in one migration
2. The dual-write that swallows the new write
3. Reads switched before the backfill finished
4. Consumer ships before producer (reversed deploy order)
5. Old path removed while a monthly job still calls it
6. The strict reader that breaks on the expand
7. The backfill that takes the database down

Snippet 1 — Drop-the-column rename in one migration¶

-- migration V37__rename_email_column.sql
-- "Renaming users.email to users.email_address. Ship it with the app PR that uses the new name."
ALTER TABLE users RENAME COLUMN email TO email_address;

The matching application PR changes every query from email to email_address and is deployed in the same release.

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

Snippet 2 — The dual-write that swallows the new write¶

// Dual-writing to the new column during a migration.
public void save(User user) {
    repository.updateEmail(user.getId(), user.getEmail());        // old column — the source of truth

    try {
        repository.updateEmailAddress(user.getId(), user.getEmail()); // new column — best effort
    } catch (Exception e) {
        log.warn("could not write email_address, will fix later", e); // swallow & continue
    }
}

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

@Transactional
public void save(User user) {
    repository.updateEmail(user.getId(), user.getEmail());
    repository.updateEmailAddress(user.getId(), user.getEmail()); // same tx — both or neither
}

Snippet 3 — Reads switched before the backfill finished¶

# Migration runbook, executed top to bottom in one sitting:
#   1. ALTER TABLE users ADD COLUMN email_address VARCHAR(320);   # done
#   2. deploy dual-write code                                     # done
#   3. start backfill job (40M rows, ~6 hours)                    # RUNNING...
#   4. deploy read-switch code                                    # <-- shipped now

def get_email(user_id: int) -> str:
    row = db.query_one("SELECT email_address FROM users WHERE id = %s", user_id)
    return row["email_address"]   # reads the new column

The engineer deployed step 4 right after kicking off step 3, "to get the migration over with before lunch."

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

-- Step 4 is forbidden until this returns 0.
SELECT count(*) FROM users WHERE email_address IS NULL;

Snippet 4 — Consumer ships before producer (reversed deploy order)¶

# OrderService (producer) — PR #812, in code review, not yet deployed.
def publish_order(order):
    emit("OrderPlaced", {"order_id": order.id, "total": order.total, "region": order.region})

# ShippingConsumer — PR #815, deployed FIRST (it was approved first).
def handle_order_placed(event):
    region = event["region"]                 # NEW required field
    rate = SHIPPING_RATES[region]            # KeyError if region missing
    ship(event["order_id"], rate)

region is a new field. The consumer PR merged and deployed Tuesday; the producer PR is still in review.

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

region = event.get("region", "UNKNOWN")   # default; never KeyError on old events
rate = SHIPPING_RATES.get(region, DEFAULT_RATE)

Snippet 5 — Old path removed while a monthly job still calls it¶

// Contraction PR: "legacyExport is deprecated. grep shows no callers. Deleting it."
// Reviewer approved: "Confirmed, `rg legacyExport` returns nothing in the service repo. LGTM."

- func legacyExport(r Report) ([]byte, error) {
-     // ... 80 lines of the old CSV format ...
- }

rg legacyExport in the service repo genuinely returns zero hits. The PR merges and deploys.

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

Snippet 6 — The strict reader that breaks on the expand¶

// Client SDK that calls a partner's REST API.
private static final ObjectMapper MAPPER = new ObjectMapper()
    .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, true); // strict

public Account parse(String json) throws IOException {
    return MAPPER.readValue(json, Account.class); // Account has id, name, balance
}

The partner runs a textbook expand: they add one new, optional field tier to the Account response — a fully backward-compatible, additive change. Your client starts throwing in production.

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

private static final ObjectMapper MAPPER = new ObjectMapper()
    .configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false); // ignore unknowns

Snippet 7 — The backfill that takes the database down¶

-- Backfill step of an otherwise-correct expand-contract migration.
-- "Copy the old column into the new one for all existing rows."
UPDATE users SET email_address = email;   -- 40M rows, one statement

Dual-write was deployed first, the new column exists, and the read-switch is correctly gated on completion. The sequence is right. The migration is run during business hours.

What's broken about the sequencing? What's the concrete production symptom? How do you fix the order?

-- Loop in the app/job, advancing :lo by batch size, sleeping between batches.
UPDATE users
SET    email_address = email
WHERE  id BETWEEN :lo AND :lo + 5000
  AND  email_address IS NULL;     -- idempotent: skip already-filled rows
-- commit; sleep(200ms); check replication lag; repeat.

Summary¶

• Every bug here compiles and passes its own tests — the defect is in the ordering of separate deploys/migrations or in a step's execution, visible only when old and new run together, a backfill is mid-flight, or an old message arrives.
• Contract too early (Snippets 1, 5) is the recurring killer: dropping a column in the deploy, or deleting a path on grep instead of a runtime cross-repo zero-callers metric over a full business cycle.
• Dual-write must be atomic (Snippet 2): a best-effort second write that drops on error silently diverges the two stores and gives false confidence.
• Reads after backfill, never during (Snippet 3): switching early serves NULLs for un-backfilled rows — a silent, self-healing correctness bug.
• Deploy order is load-bearing (Snippet 4): never require a field no deployed producer emits; consumer-first for additive, producer-first for required, tolerant readers to forgive the gap.
• Tolerant readers (Snippet 6) make the producer's expand invisible; strict readers turn safe additive changes into outages.
• The backfill is part of the migration's safety (Snippet 7): batch, throttle, make idempotent and resumable, or the "zero-downtime" migration causes the downtime itself.

Related Topics¶

• tasks.md — do these migrations correctly, with the full sequence and the remaining-callers gate.
• interview.md — the deploy-ordering rules and "why atomic fails" in Q&A form.
• senior.md — the codebase-scale walkthrough behind these snippets.
• optimize.md — the migration that's correct but was never finished (dual-path left on forever).
• Strangler Fig & Seams — broken incremental replacements, the macro sibling of these bugs.
• Architecture Fitness Functions — making "no caller of the old path" a build-failing rule.
• Anti-Pattern Budgets & Ratcheting — keeping old-path callers monotonically decreasing.
• Architecture → Anti-Patterns — system-level contract-evolution siblings.