Fragile Tests — Find the Bug¶

Category: Testing Anti-Patterns → Fragile Tests — a test that breaks when you change code without changing its behavior.

This file is critical-reading practice. Each snippet below is a plausible test in Go, Java, or Python. Your job is to read it the way a good reviewer does and answer three questions:

What is this test coupled to? What behavior-preserving change would break it? How would you rewrite it to assert the contract instead?

The "bug" here is rarely a crash — it's a latent flaw in what the test asserts. A fragile test passes today and breaks tomorrow for the wrong reason: someone refactored the implementation without changing its behavior, and the test went red anyway. Some snippets are deliberately innocent-looking — tests that would sail through a casual review because they're green and look thorough. A few are traps: tests that look fragile but are actually correct, because the thing they pin really is the contract. Read slowly, then open the answer.

How to use this file: read each snippet and write your own answer before expanding the collapsible. The skill you're training is asking "would this survive a behavior-preserving refactor?" — not recalling the name.

Table of Contents¶

1. The thorough-looking user test
2. Verifying the save
3. The exact JSON
4. The sorted-looking result
5. Reading the cache
6. The log says so
7. Charge exactly once

Snippet 1 — The thorough-looking user test¶

# Python — looks diligent: asserts "everything" about the created user.
def test_create_user():
    u = create_user("Sam", "sam@example.com")
    assert u == User(
        id=1,
        name="Sam",
        email="sam@example.com",
        status="ACTIVE",
        created_at=datetime(2026, 6, 10, 12, 0, 0),
        version=1,
    )

What is this test coupled to? What behavior-preserving change would break it? How would you rewrite it?

def test_create_user_makes_active_user():
    u = create_user("Sam", "sam@example.com")
    assert u.name == "Sam"
    assert u.email == "sam@example.com"
    assert u.status == "ACTIVE"
    assert u.id is not None          # an id exists; value not pinned
    # created_at and version are incidental to THIS behavior → not asserted.

Snippet 2 — Verifying the save¶

// Java — a deletion test that "checks the database was updated."
@Test
void deleteUser_removesUser() {
    UserRepo repo = mock(UserRepo.class);
    UserService service = new UserService(repo);

    service.delete(42L);

    verify(repo).deleteById(42L);   // the only assertion
}

What is this test coupled to? What behavior-preserving change would break it? How would you rewrite it?

@Test
void deleteUser_makesUserUnretrievable() {
    FakeUserRepo repo = new FakeUserRepo();
    repo.save(new User(42L, "sam@x.io"));
    UserService service = new UserService(repo);

    service.delete(42L);

    assertThat(repo.findById(42L)).isEmpty();   // the actual contract: it's gone
}

Snippet 3 — The exact JSON¶

// Go — an API handler test that checks the response body.
func TestUserEndpoint(t *testing.T) {
    rec := httptest.NewRecorder()
    handler.ServeHTTP(rec, newRequest("GET", "/users/1"))

    expected := `{"id":1,"name":"Sam","email":"sam@x.io","roles":["admin"]}`
    assert.Equal(t, expected, rec.Body.String())
}

What is this test coupled to? What behavior-preserving change would break it? How would you rewrite it?

func TestUserEndpoint_returnsUser(t *testing.T) {
    rec := httptest.NewRecorder()
    handler.ServeHTTP(rec, newRequest("GET", "/users/1"))

    require.Equal(t, http.StatusOK, rec.Code)
    var got struct {
        ID    int      `json:"id"`
        Name  string   `json:"name"`
        Roles []string `json:"roles"`
    }
    require.NoError(t, json.Unmarshal(rec.Body.Bytes(), &got))

    assert.Equal(t, 1, got.ID)
    assert.Equal(t, "Sam", got.Name)
    assert.Equal(t, []string{"admin"}, got.Roles)
}

Snippet 4 — The sorted-looking result¶

# Python — two tests on a function that returns active user ids.
def get_active_ids(users):
    return [u.id for u in users if u.active]   # preserves input order... today

def test_active_ids():
    users = [User(1, True), User(2, False), User(3, True)]
    assert get_active_ids(users) == [1, 3]     # Test A

def test_active_ids_sorted():
    result = sort_ids_descending(get_active_ids(users))
    assert result == [3, 1]                     # Test B

One of these is fragile and one is correct. Which is which, and why?

def test_active_ids_returns_active_set():
    users = [User(1, True), User(2, False), User(3, True)]
    assert set(get_active_ids(users)) == {1, 3}   # membership, not order

Snippet 5 — Reading the cache¶

// Java — testing a memoizing service.
@Test
void compute_cachesResult() {
    PriceService svc = new PriceService();

    svc.price("ABC");
    svc.price("ABC");   // second call should hit the cache

    // Reach into the private cache via reflection to "prove" memoization.
    Map<String, Money> cache =
        (Map<String, Money>) ReflectionTestUtils.getField(svc, "cache");
    assertThat(cache).containsKey("ABC");
}

What is this test coupled to? What behavior-preserving change would break it? How would you rewrite it?

@Test
void compute_doesExpensiveWorkOncePerKey() {
    AtomicInteger calls = new AtomicInteger();
    PriceService svc = new PriceService(sku -> {     // inject the expensive op
        calls.incrementAndGet();
        return Money.of("10.00");
    });

    Money first  = svc.price("ABC");
    Money second = svc.price("ABC");

    assertThat(first).isEqualTo(second);             // same result
    assertThat(calls.get()).isEqualTo(1);            // expensive work ran ONCE — that's caching
}

Snippet 6 — The log says so¶

# Python — an order processor that logs its work.
def test_process_order(caplog):
    process_order(order_42)
    assert "Order 42 processed successfully for customer Sam (total: $99.00)" in caplog.text

What is this test coupled to? What behavior-preserving change would break it? How would you rewrite it?

def test_process_order_confirms_and_charges():
    process_order(order_42)
    assert order_42.status == "CONFIRMED"            # the real outcome
    assert payments.charged(order_42.id) == Money("99.00")
    # No assertion on log text — the behavior is verified directly.

def test_process_order_emits_audit_event(audit_sink):
    process_order(order_42)
    e = audit_sink.last()
    assert e.type == "order.processed"   # stable contract field
    assert e.order_id == 42
    # The human-readable message is free to change.

Snippet 7 — Charge exactly once¶

// Go — testing a payment service that retries on transient failure.
func TestCharge_exactlyOnce(t *testing.T) {
    gw := NewFakeGateway()                 // a fake at the real boundary
    gw.FailNextN(2)                        // first two attempts flake, third succeeds
    svc := NewPayments(gw, RetryPolicy{Max: 3})

    err := svc.Charge(order42)

    require.NoError(t, err)
    assert.Equal(t, 1, gw.SuccessfulCharges(order42.ID))  // charged exactly once
}

This asserts on an interaction (the charge count). Is it fragile? Why or why not?

// Fragile: pins the internal retry choreography, not the contract.
gw.AssertCalled(t, "Attempt")            // how many internal attempts? implementation detail
mock.InOrder(gw.Attempt, gw.Attempt, gw.Commit)  // freezes the retry algorithm

Summary — patterns of spotting¶

You don't spot a fragile test by a single bad keyword — you spot it by asking one question of every assertion: "would this survive a behavior-preserving refactor, and would it still fail if the behavior actually broke?" The repeatable moves from these seven snippets:

• A "thorough" full-object assertion is usually over-specification (Snippet 1). It pins generated ids, timestamps, and unrelated fields; assert only what the behavior promises, and existence not value for generated data.
• verify(mock).x() as the main assertion tests the mechanism, not the outcome (Snippet 2). It passes even when the result is wrong and breaks on any re-plumbing. Use a fake, assert the observable result.
• Byte-for-byte serialized output pins key order and forbids additive fields (Snippet 3). Parse, then assert on the values a consumer reads.
• Exact order is fragile only when order isn't the contract (Snippet 4 trap). Ask whether the function promises a set or a sequence before crying fragile.
• Reflection / private-field access bypasses the contract entirely (Snippet 5). Assert the observable effect (e.g. "expensive work ran once") through a seam, never the internal storage.
• Asserting on log prose couples to the most volatile text you have (Snippet 6). Assert the behavior the log describes; use structured events if emission is contractual.
• Interaction testing is correct when the interaction is the contract (Snippet 7 trap) — exactly-once, ordering, protocol — at a real boundary. It's fragile only when it pins internal choreography.

The meta-lesson: fragility is about what you assert, not how green the test is today. Two of these seven snippets pin exactly the thing they should and are correct; the other five pin details the contract never promised and will go red on the next harmless refactor. Read every assertion and ask whether it names a promise the code actually makes.

Related Topics¶

• junior.md — what a fragile test looks like and the three first habits.
• middle.md — the four creep patterns and the contract-vs-implementation rule.
• tasks.md — fix-it exercises that build the same muscle from the writing side.
• optimize.md — refactor a whole over-specified test file end-to-end.
• professional.md — when an interaction or exact-match assertion is correct, not fragile.
• Over-Mocking — the find-bug file for mock-induced fragility.
• The mocking-strategies and unit-testing-patterns skills — state vs interaction verification, fakes vs mocks.