Property-Based Testing — Find the Bug¶
Spot the defects below. Each snippet either has a buggy implementation that PBT would surface, or has a buggy property/generator that hides bugs.
Bug 1 — broken Reverse¶
func Reverse(s []int) []int {
out := make([]int, len(s))
for i, v := range s {
out[len(s)-i] = v
}
return out
}
The index len(s)-i is off by one — element 0 writes past the end and the last index is never assigned. PBT property Reverse(Reverse(x)) == x would report the panic immediately and shrink to [0].
Fix: out[len(s)-1-i] = v.
Bug 2 — property that always passes¶
There is no assertion. The "property" computes a value and throws it away, so it cannot fail. A green test that never asserts is worse than no test — it gives false confidence.
Fix: assert something, e.g. if n*2/2 != n && n != math.MinInt { t.Fatalf(...) }.
Bug 3 — too-narrow generator¶
The generator never hits negative numbers or values large enough to overflow. factorial(21) overflows int64 and becomes negative; PBT will not catch this because the generator caps at 10.
Fix: widen the range, or document the precondition explicitly.
Bug 4 — round-trip that compares the wrong thing¶
rapid.Check(t, func(t *rapid.T) {
s := rapid.String().Draw(t, "s")
b, _ := json.Marshal(s)
var out string
_ = json.Unmarshal(b, &out)
if len(out) != len(s) {
t.Fatal("mismatch")
}
}
Comparing only len will miss mojibake or escape bugs: "a b" and "ab" would compare equal in length but differ in content. The property must compare full equality.
Fix: if out != s { t.Fatalf("got %q want %q", out, s) }.
Bug 5 — sort property that ignores duplicates¶
rapid.Check(t, func(t *rapid.T) {
in := rapid.SliceOf(rapid.Int()).Draw(t, "in")
out := mySort(in)
set := map[int]bool{}
for _, v := range in { set[v] = true }
for _, v := range out {
if !set[v] { t.Fatal("foreign value") }
}
})
Putting inputs in a set loses multiplicity. A buggy sort that returns [1, 1, 1] for input [1, 2, 3] would pass. The property must check that out is a permutation, not merely a subset.
Fix: compare sorted copies, or compare multiset counts.
Bug 6 — checking sortedness only¶
return []int{} would satisfy this property for any input. Sortedness alone is not enough; you must also check the output is a permutation of the input.
Bug 7 — relying on map iteration order in a generator¶
gen := func(t *rapid.T) []string {
m := rapid.MapOf(rapid.String(), rapid.Int()).Draw(t, "m")
out := []string{}
for k := range m { out = append(out, k) }
return out
}
Map iteration order is randomised by Go itself, on top of the rapid seed. Two runs with the same rapid seed can produce different inputs, breaking reproducibility.
Fix: collect keys then sort.Strings(out) before returning.
Bug 8 — using time.Now() inside a property¶
rapid.Check(t, func(t *rapid.T) {
n := rapid.Int().Draw(t, "n")
if time.Now().Unix()%2 == 0 && process(n) < 0 {
t.Fatal("negative")
}
})
The property depends on wall clock — failures are not reproducible. PBT properties must be deterministic given the rapid seed.
Bug 9 — silent error swallow¶
rapid.Check(t, func(t *rapid.T) {
s := rapid.String().Draw(t, "s")
b, err := encode(s)
if err != nil { return } // skip
out, _ := decode(b)
require.Equal(t, s, out)
})
If encode is buggy and returns an error for valid inputs, the property silently skips those cases. PBT thinks everything passes.
Fix: require.NoError(t, err) — turn errors into failures unless the generator can legitimately produce invalid input.
Bug 10 — comparing maps by stringifying¶
rapid.Check(t, func(t *rapid.T) {
m := rapid.MapOf(rapid.String(), rapid.Int()).Draw(t, "m")
b, _ := json.Marshal(m)
var got map[string]int
json.Unmarshal(b, &got)
if fmt.Sprintf("%v", got) != fmt.Sprintf("%v", m) {
t.Fatal("round-trip mismatch")
}
})
fmt.Sprintf("%v", map) prints map keys in iteration order, which is randomised in Go. The property will fail spuriously on maps with the same content but different print order.
Fix: compare with reflect.DeepEqual or compare sorted key/value pairs.
Bug 11 — generator that never produces edge cases¶
gen := rapid.IntRange(1, 100)
rapid.Check(t, func(t *rapid.T) {
n := gen.Draw(t, "n")
require.NotZero(t, Factorial(n))
})
Bug in Factorial(0): returns 0 instead of 1. The generator caps at 1, so the bug never appears. PBT cannot find what the generator does not produce.
Fix: rapid.IntRange(0, 100) — include the boundary.
Bug 12 — property that mutates the input¶
rapid.Check(t, func(t *rapid.T) {
xs := rapid.SliceOf(rapid.Int()).Draw(t, "xs")
sort.Ints(xs) // mutates rapid's slice!
require.True(t, isSorted(xs))
})
rapid may reuse underlying memory between runs. Mutating the drawn slice can corrupt subsequent iterations or shrinking.
Fix: copy before mutating.
Bug 13 — flaky property due to goroutine¶
rapid.Check(t, func(t *rapid.T) {
n := rapid.IntRange(1, 100).Draw(t, "n")
var sum atomic.Int64
var wg sync.WaitGroup
for i := 0; i < n; i++ {
wg.Add(1)
go func() { defer wg.Done(); sum.Add(1) }()
}
wg.Wait()
if sum.Load() != int64(n) {
t.Fatal()
}
})
The property is well-formed and should pass. But if you forget wg.Add or use a non-atomic counter, the property becomes flaky — not deterministic from the seed. The seed determines n; goroutine scheduling determines whether the bug fires.
Use -race to catch the actual data race; do not rely on PBT alone for concurrency bugs.
Bug 14 — wrong oracle¶
rapid.Check(t, func(t *rapid.T) {
xs := rapid.SliceOf(rapid.Int()).Draw(t, "xs")
if MyMax(xs) != naiveMin(xs) { // typo!
t.Fatal()
}
})
The oracle is naiveMin instead of naiveMax. The property fails on almost every input. But that looks like a bug in MyMax until you read carefully.
Lesson: name oracles with care. Code review the property as carefully as the implementation.
Bug 15 — Float comparison without tolerance¶
rapid.Check(t, func(t *rapid.T) {
a := rapid.Float64().Draw(t, "a")
b := rapid.Float64().Draw(t, "b")
if (a+b)+1 != a+(b+1) {
t.Fatal("addition not associative")
}
})
Float addition is not associative. Even when arithmetic is correct, the property fails for many inputs because == compares bit-identical floats. The property is bogus — it fails for correct float arithmetic.
Fix: use a tolerance, or restrict to integer-like floats.
Bug 16 — silent type assertion panic¶
rapid.Check(t, func(t *rapid.T) {
val := rapid.Custom(func(t *rapid.T) interface{} {
return rapid.Int().Draw(t, "v")
}).Draw(t, "val")
n := val.(int) + 1 // OK in this case, but...
_ = n
})
If the Custom generator were widened to OneOf(genInt, genString), the type assertion would panic on string draws. PBT would catch the panic, but the report becomes confusing.
Fix: assert the type properly or use a typed generator.
Bug 17 — over-aggressive filter¶
Less than one in a million ints satisfies the predicate. rapid will discard millions of draws, eventually erroring with "too many discards".
Fix: generate values that satisfy the predicate directly, e.g. rapid.Map(rapid.Int(), func(n int) int { return n * 1000000 }).
Bug 18 — missing copy before mutating¶
rapid.Check(t, func(t *rapid.T) {
xs := rapid.SliceOf(rapid.Int()).Draw(t, "xs")
reverseInPlace(xs) // mutates the drawn slice
reverseInPlace(xs) // reverses back
// Looks fine, but rapid may reuse memory between runs.
})
The property happens to be self-correcting (reverse twice = identity), so it may not visibly fail. But the next iteration may see a reversed slice, leading to non-determinism in shrinking.
Fix: always copy first.