Coordination Anti-Patterns — Find the Bug¶

Category: Concurrency Anti-Patterns → Coordination — two or more lock holders that fail to make progress together. Covers (collectively): Lock Ordering Inconsistency → Deadlock · Holding a Lock During I/O · Wrong Lock Granularity

This file is critical-reading practice for the hardest class of concurrency bug: the one where every individual line is correct and every function passes its unit test, yet the combination deadlocks, stalls, or melts your throughput under the right interleaving or load. Coordination bugs are not crashes — they are failures to make progress together. They show up at 3 a.m. under production traffic and vanish the moment you attach a debugger.

For each snippet, read it the way an on-call engineer reviewing a "service hangs intermittently" incident would, and answer three questions:

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

A note on vocabulary you'll need throughout. A deadlock is a cycle in the wait-for graph: a directed graph whose nodes are threads and whose edge A → B means "thread A is blocked waiting for a lock currently held by thread B." If that graph ever contains a cycle, every thread on the cycle is blocked forever — none can release the lock the next one needs. The four Coffman conditions for deadlock are mutual exclusion, hold-and-wait, no preemption, and circular wait; breaking any one prevents the deadlock, and "establish a global lock order" is simply the standard way to break circular wait.

How to use this file: read each snippet and write your own answer before expanding the collapsible. The skill you are training is constructing the bad interleaving in your head, not recalling the pattern's name. Several snippets combine two anti-patterns, and at least one is a deliberate trap — code that looks deadlock-prone but is provably safe.

A note on Python: the GIL serializes bytecode execution, but it does not save you here. Two threads can still each hold one threading.Lock and block forever waiting for the other — the GIL is released while a thread is blocked on a lock. Holding a lock across a requests.get() is just as ruinous in Python as in Go or Java. The GIL prevents data races on a single object, not coordination deadlocks.

Table of Contents¶

1. Transfer between two accounts
2. The cache that calls the database
3. The registry with a notify callback
4. One lock to rule the service
5. The graph with a lock per node
6. Two managers, two mutexes
7. The rate limiter that phones home
8. The ordered transfer that looks safe
9. The metrics map under one mutex
10. The connection pool and the logger
11. Striped locks with a cross-stripe move
12. The read lock that upgrades
13. The single mutex that only looks scary
14. The event bus that fans out under lock

Snippet 1 — Transfer between two accounts¶

// Go — moving money between two locked accounts
type Account struct {
    mu      sync.Mutex
    balance int64
}

func Transfer(from, to *Account, amount int64) error {
    from.mu.Lock()
    defer from.mu.Unlock()
    to.mu.Lock()
    defer to.mu.Unlock()

    if from.balance < amount {
        return errors.New("insufficient funds")
    }
    from.balance -= amount
    to.balance += amount
    return nil
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

func Transfer(from, to *Account, amount int64) error {
    first, second := from, to
    if from.id > to.id {            // always lock the lower id first
        first, second = to, from
    }
    first.mu.Lock();  defer first.mu.Unlock()
    second.mu.Lock(); defer second.mu.Unlock()

    if from.balance < amount {
        return errors.New("insufficient funds")
    }
    from.balance -= amount
    to.balance += amount
    return nil
}

Snippet 2 — The cache that calls the database¶

// Java — a write-through cache shared by all request threads
public class ProductCache {
    private final Map<Long, Product> map = new HashMap<>();
    private final ReentrantLock lock = new ReentrantLock();

    public Product get(long id) {
        lock.lock();
        try {
            Product p = map.get(id);
            if (p == null) {
                p = db.loadProduct(id);     // network + DB round-trip, 20–500 ms
                map.put(id, p);
            }
            return p;
        } finally {
            lock.unlock();
        }
    }
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

private final ConcurrentMap<Long, CompletableFuture<Product>> map = new ConcurrentHashMap<>();

public Product get(long id) {
    return map.computeIfAbsent(id, k ->
        CompletableFuture.supplyAsync(() -> db.loadProduct(k))   // load off the lock
    ).join();
}

Snippet 3 — The registry with a notify callback¶

// Go — an observable registry; listeners are invoked on every change
type Registry struct {
    mu        sync.Mutex
    items     map[string]int
    listeners []func(key string)
}

func (r *Registry) Set(key string, val int) {
    r.mu.Lock()
    defer r.mu.Unlock()
    r.items[key] = val
    for _, fn := range r.listeners {
        fn(key)                       // call listener while holding r.mu
    }
}

func (r *Registry) Get(key string) int {
    r.mu.Lock()
    defer r.mu.Unlock()
    return r.items[key]
}

// A listener registered elsewhere:
reg.listeners = append(reg.listeners, func(key string) {
    log.Printf("changed %s -> %d", key, reg.Get(key))   // re-enters reg.mu
})

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

func (r *Registry) Set(key string, val int) {
    r.mu.Lock()
    r.items[key] = val
    listeners := make([]func(string), len(r.listeners))
    copy(listeners, r.listeners)        // snapshot under lock
    r.mu.Unlock()                        // release BEFORE calling out

    for _, fn := range listeners {       // listeners run lock-free; may re-enter safely
        fn(key)
    }
}

Snippet 4 — One lock to rule the service¶

// Go — the only synchronization in a high-QPS pricing service
type PricingService struct {
    mu       sync.Mutex
    rates    map[string]float64   // currency -> rate, updated ~once/minute
    requests uint64               // total served
}

func (s *PricingService) Price(item Item, currency string) float64 {
    s.mu.Lock()
    defer s.mu.Unlock()

    s.requests++
    rate := s.rates[currency]
    base := item.cents
    // ... 40 lines of pure CPU pricing math using base and rate ...
    return float64(base) * rate * marginFor(item) // marginFor is pure, ~5µs
}

func (s *PricingService) UpdateRates(r map[string]float64) {
    s.mu.Lock()
    defer s.mu.Unlock()
    s.rates = r
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

type PricingService struct {
    rates    atomic.Pointer[map[string]float64]  // swapped wholesale on update
    requests atomic.Uint64
}

func (s *PricingService) Price(item Item, currency string) float64 {
    s.requests.Add(1)                  // lock-free counter
    rates := *s.rates.Load()           // lock-free read of an immutable snapshot
    rate := rates[currency]
    base := item.cents
    return float64(base) * rate * marginFor(item)   // math runs fully parallel
}

func (s *PricingService) UpdateRates(r map[string]float64) {
    s.rates.Store(&r)                  // publish a new immutable map
}

Snippet 5 — The graph with a lock per node¶

// Go — a concurrent graph; each node guards its own edges
type Node struct {
    mu    sync.Mutex
    id    int
    edges map[*Node]int
}

// Adds a bidirectional edge, locking both endpoints.
func Connect(a, b *Node, weight int) {
    a.mu.Lock()
    defer a.mu.Unlock()
    b.mu.Lock()
    defer b.mu.Unlock()
    a.edges[b] = weight
    b.edges[a] = weight
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

func Connect(a, b *Node, weight int) {
    first, second := a, b
    if a.id > b.id {
        first, second = b, a
    }
    first.mu.Lock();  defer first.mu.Unlock()
    second.mu.Lock(); defer second.mu.Unlock()
    a.edges[b] = weight
    b.edges[a] = weight
}

Snippet 6 — Two managers, two mutexes¶

// Java — two subsystems that occasionally need to coordinate
class UserManager {
    private final Object lock = new Object();
    private final SessionManager sessions;

    void logout(long userId) {
        synchronized (lock) {                 // (1) UserManager.lock
            clearUserState(userId);
            sessions.endAll(userId);          // calls into SessionManager
        }
    }
    void onSessionExpired(long userId) {      // called BY SessionManager
        synchronized (lock) {                 // (4) UserManager.lock
            clearUserState(userId);
        }
    }
}

class SessionManager {
    private final Object lock = new Object();
    private UserManager users;

    void endAll(long userId) {
        synchronized (lock) {                 // (2) SessionManager.lock
            killSessions(userId);
        }
    }
    void expire(long userId) {                // background reaper thread
        synchronized (lock) {                 // (3) SessionManager.lock
            killSessions(userId);
            users.onSessionExpired(userId);   // calls back into UserManager
        }
    }
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

void logout(long userId) {
    synchronized (lock) { clearUserState(userId); }   // release before calling out
    sessions.endAll(userId);                           // no UserManager.lock held here
}
void expire(long userId) {
    synchronized (lock) { killSessions(userId); }      // release before calling back
    users.onSessionExpired(userId);                    // no SessionManager.lock held here
}

Snippet 7 — The rate limiter that phones home¶

# Python — a distributed rate limiter consulted on every request
import threading, requests

class RateLimiter:
    def __init__(self):
        self._lock = threading.Lock()
        self._counts = {}   # key -> int, in-process fallback

    def allow(self, key, limit):
        with self._lock:
            # Ask the central quota service whether this key is over budget.
            resp = requests.post("http://quota-svc/check",
                                 json={"key": key}, timeout=2.0)   # network call
            remaining = resp.json()["remaining"]
            self._counts[key] = remaining
            return remaining > 0

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

class RateLimiter:
    def __init__(self):
        self._lock = threading.Lock()
        self._counts = {}

    def allow(self, key, limit):
        resp = requests.post("http://quota-svc/check",
                             json={"key": key}, timeout=2.0)   # no lock held
        remaining = resp.json()["remaining"]
        with self._lock:                                       # lock only the dict write
            self._counts[key] = remaining
        return remaining > 0

Snippet 8 — The ordered transfer that looks safe¶

// Go — like Snippet 1, but the author "fixed" the ordering. Did they?
type Account struct {
    mu  sync.Mutex
    id  int
    bal int64
}

func lockBoth(a, b *Account) (first, second *Account) {
    if a.id <= b.id {
        a.mu.Lock()
        b.mu.Lock()
        return a, b
    }
    b.mu.Lock()
    a.mu.Lock()
    return b, a
}

func Transfer(from, to *Account, amt int64) error {
    f, s := lockBoth(from, to)
    defer f.mu.Unlock()
    defer s.mu.Unlock()
    if from.bal < amt {
        return errors.New("insufficient")
    }
    from.bal -= amt
    to.bal += amt
    return nil
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

func Transfer(from, to *Account, amt int64) error {
    if from == to {
        return nil                       // no-op; avoids double-locking one mutex
    }
    f, s := lockBoth(from, to)
    defer f.mu.Unlock()
    defer s.mu.Unlock()
    ...
}

Snippet 9 — The metrics map under one mutex¶

// Go — per-endpoint latency metrics, written on every request
type Metrics struct {
    mu      sync.Mutex
    byRoute map[string]*Histogram
}

func (m *Metrics) Observe(route string, latency time.Duration) {
    m.mu.Lock()
    defer m.mu.Unlock()
    h, ok := m.byRoute[route]
    if !ok {
        h = NewHistogram()
        m.byRoute[route] = h
    }
    h.Record(latency)        // updates the histogram's internal buckets
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

type Metrics struct {
    mu      sync.RWMutex
    byRoute map[string]*Histogram
}

func (m *Metrics) Observe(route string, latency time.Duration) {
    m.mu.RLock()
    h, ok := m.byRoute[route]
    m.mu.RUnlock()
    if !ok {
        m.mu.Lock()
        if h, ok = m.byRoute[route]; !ok {   // re-check under write lock
            h = NewHistogram()
            m.byRoute[route] = h
        }
        m.mu.Unlock()
    }
    h.Record(latency)        // Histogram has its own internal sync / atomics
}

Snippet 10 — The connection pool and the logger¶

// Java — a JDBC-style pool whose checkout logs via a shared audit logger
class ConnectionPool {
    private final Object poolLock = new Object();
    private final AuditLogger audit;          // shared, also used elsewhere

    Connection acquire() {
        synchronized (poolLock) {
            Connection c = waitForFree();      // blocks until a conn frees up
            audit.record("acquire", c.id());   // (A) poolLock -> auditLock
            return c;
        }
    }
}

class AuditLogger {
    private final Object auditLock = new Object();
    private ConnectionPool pool;               // logger writes audit rows to the DB!

    void record(String ev, long id) {
        synchronized (auditLock) {
            buffer.add(ev, id);
            if (buffer.full()) {
                Connection c = pool.acquire();  // (B) auditLock -> poolLock
                flushToDb(c);
            }
        }
    }
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

Connection acquire() {
    Connection c;
    synchronized (poolLock) { c = takeFreeIfAvailable(); }   // don't block under lock
    if (c == null) c = waitForFreeOutsideLock();
    audit.record("acquire", c.id());        // logged after releasing poolLock
    return c;
}
// AuditLogger flushes asynchronously on its own thread + its own dedicated connection,
// so record() never calls back into the request-path pool while holding auditLock.

Snippet 11 — Striped locks with a cross-stripe move¶

// Java — a sharded map using lock striping for throughput
class StripedStore {
    private final Object[] stripes = new Object[16];
    private final Map<String, Long>[] buckets = new HashMap[16];
    // (constructor fills stripes[i] = new Object() and buckets[i] = new HashMap())

    private int idx(String k) { return (k.hashCode() & 0x7fffffff) % 16; }

    // Move a value from one key to another (possibly different stripe).
    void move(String src, String dst, long delta) {
        synchronized (stripes[idx(src)]) {       // lock src's stripe
            synchronized (stripes[idx(dst)]) {   // lock dst's stripe
                buckets[idx(src)].merge(src, -delta, Long::sum);
                buckets[idx(dst)].merge(dst, delta, Long::sum);
            }
        }
    }
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

void move(String src, String dst, long delta) {
    int i = idx(src), j = idx(dst);
    int lo = Math.min(i, j), hi = Math.max(i, j);
    synchronized (stripes[lo]) {
        synchronized (stripes[hi]) {          // lo == hi is fine: synchronized is reentrant
            buckets[i].merge(src, -delta, Long::sum);
            buckets[j].merge(dst, delta, Long::sum);
        }
    }
}

Snippet 12 — The read lock that upgrades¶

// Go — a config store using RWMutex, with a "refresh if stale" read path
type Config struct {
    mu      sync.RWMutex
    data    map[string]string
    fetched time.Time
}

func (c *Config) Get(key string) string {
    c.mu.RLock()
    defer c.mu.RUnlock()

    if time.Since(c.fetched) > time.Minute {   // stale?
        c.mu.Lock()                            // try to upgrade to write lock
        c.data = reload()
        c.fetched = time.Now()
        c.mu.Unlock()
    }
    return c.data[key]
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

func (c *Config) Get(key string) string {
    c.mu.RLock()
    stale := time.Since(c.fetched) > time.Minute
    if !stale {
        defer c.mu.RUnlock()
        return c.data[key]
    }
    c.mu.RUnlock()                 // drop read lock BEFORE taking write lock

    c.mu.Lock()
    if time.Since(c.fetched) > time.Minute {   // double-check; someone may have reloaded
        c.data = reload()
        c.fetched = time.Now()
    }
    val := c.data[key]
    c.mu.Unlock()
    return val
}

Snippet 13 — The single mutex that only looks scary¶

// Go — a bounded job queue; producers and consumers share one mutex + cond
type Queue struct {
    mu    sync.Mutex
    cond  *sync.Cond
    items []Job
    max   int
}

func (q *Queue) Push(j Job) {
    q.mu.Lock()
    defer q.mu.Unlock()
    for len(q.items) == q.max {
        q.cond.Wait()              // releases mu while waiting, re-acquires on wake
    }
    q.items = append(q.items, j)
    q.cond.Broadcast()
}

func (q *Queue) Pop() Job {
    q.mu.Lock()
    defer q.mu.Unlock()
    for len(q.items) == 0 {
        q.cond.Wait()
    }
    j := q.items[0]
    q.items = q.items[1:]
    q.cond.Broadcast()
    return j
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

Snippet 14 — The event bus that fans out under lock¶

// Go — an in-process event bus delivering to all subscribers synchronously
type Bus struct {
    mu   sync.Mutex
    subs map[string][]chan Event
}

func (b *Bus) Subscribe(topic string) <-chan Event {
    b.mu.Lock()
    defer b.mu.Unlock()
    ch := make(chan Event)            // UNBUFFERED channel
    b.subs[topic] = append(b.subs[topic], ch)
    return ch
}

func (b *Bus) Publish(topic string, e Event) {
    b.mu.Lock()
    defer b.mu.Unlock()
    for _, ch := range b.subs[topic] {
        ch <- e                       // blocks until a receiver is ready
    }
}

What's the coordination bug? Under what interleaving or load does it manifest? How would you fix it?

func (b *Bus) Publish(topic string, e Event) {
    b.mu.Lock()
    subs := append([]chan Event(nil), b.subs[topic]...)  // snapshot under lock
    b.mu.Unlock()                                        // release before delivering

    for _, ch := range subs {
        select {
        case ch <- e:                  // deliver without holding b.mu
        default:                       // drop or queue if subscriber is slow; never block the bus
        }
    }
}

Summary — how to spot coordination bugs¶

You don't catch coordination bugs by reading a line — you catch them by constructing the bad interleaving and by tracing locks across function and subsystem boundaries. The repeatable moves from these fourteen snippets:

• List the locks each path acquires, in acquisition order. If two paths take the same two locks in opposite orders, you have a potential circular wait — draw the wait-for graph and look for a cycle (Snippets 1, 5, 6, 10, 11). The fix is almost always a global lock order keyed on something stable (id, index): always lock the lower one first.
• Follow the calls made while a lock is held. Any I/O, network call, channel send, or call into foreign/unknown code under a lock is a hazard: it either collapses throughput (Snippets 2, 7, 14) or, if that foreign code re-enters your locks, deadlocks (Snippets 3, 6, 14). The rule: snapshot what you need under the lock, release, then do the slow/foreign work.
• Watch for re-entrancy and lock upgrades. A non-re-entrant mutex re-locked on the same thread is an instant self-deadlock (Snippets 3, 8-self-case); an RWMutex read lock that tries to become a write lock deadlocks against its own readers (Snippet 12). Drop the lock and re-acquire with a double-check instead.
• Right-size the lock. One global lock around a hot CPU path or an independent-per-route map throws away all parallelism (Snippets 4, 9); the cure is to shrink the lock to the smallest consistent unit and pick the right primitive (atomic counter, RWMutex, immutable snapshot, per-item lock). But beware: finer locks multiply the ordering obligation (Snippets 5, 11) — granularity and ordering are not independent decisions.
• Resist false positives. A single mutex can never deadlock on lock ordering; a cond.Wait() releases its mutex while sleeping; a consistently-ordered two-lock acquire is safe regardless of defer order (Snippets 8, 13). Verify the actual acquisition order and whether the lock is truly held during a wait — don't flag a structure by its silhouette.

The meta-lesson: every coordination bug is invisible in any single function. Transfer looks fine; logout looks fine; Publish looks fine. The defect lives in the interleaving of two threads, or in the transitive call that reaches a second lock, or in the load profile that turns a held lock into a queue. To find it you must simulate two threads at once and trace locks through the calls they make — exactly the muscle these snippets train.

Related Topics¶

• tasks.md — small concurrent programs to fix from the writing side.
• junior.md — what a deadlock and a lock-held-during-I/O look like for the first time.
• middle.md — detecting these in review and the safer default patterns.
• senior.md — debugging a deadlock in production and refactoring a contended path.
• professional.md — lock-free alternatives and memory-ordering depth.
• optimize.md — implementations to make both safe and fast.
• Synchronization Misuse → Find the Bug — the sibling category (locks and memory primitives applied wrongly).
• Shared State → Find the Bug — the sibling category (mutable state crossing threads).
• Concurrency Roadmap — the positive patterns and primitives behind these cures.
• Async Anti-Patterns — the event-loop sibling chapter where deadlocks take an async form.
• Distributed Systems — coordination at the network scale (distributed locks, leader election).