Error Propagation in Pipelines — Junior Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Glossary
4. Core Concepts
5. Real-World Analogies
6. Mental Models
7. Why Pipeline Errors Are Special
8. The First Error Pattern
9. Introducing errgroup
10. Wrapping Errors with %w
11. Pros and Cons
12. Use Cases
13. Code Examples
14. Coding Patterns
15. Clean Code
16. Product Use
17. Error Handling
18. Security Considerations
19. Performance Tips
20. Best Practices
21. Edge Cases and Pitfalls
22. Common Mistakes
23. Common Misconceptions
24. Tricky Points
25. Test
26. Tricky Questions
27. Cheat Sheet
28. Self-Assessment Checklist
29. Summary
30. What You Can Build
31. Further Reading
32. Related Topics
33. Diagrams and Visual Aids

Introduction¶

Focus: "A stage in my pipeline failed. How do I make sure the caller learns about it, and how do I stop the other stages from working on something nobody will read?"

A pipeline in Go is a chain of stages, each running in its own goroutine, connected by channels. Data flows from the first stage to the last. Each stage reads from its input channel, does some work, and writes to its output channel.

The simplest pipeline is three stages:

generator  -->  worker  -->  collector
   ch1            ch2

The happy path is easy. The hard part is the unhappy path. Suppose worker fails on the third item. What should happen?

• The worker should stop pulling new items from ch1.
• The generator should stop producing new items (otherwise it blocks forever on a channel nobody reads).
• The collector should stop waiting on ch2.
• The error should travel back to whoever called the pipeline.
• All the goroutines should exit cleanly, with no leaks.

A single returned error from a single function is straightforward in Go. But in a pipeline, the error is born inside a goroutine that the caller has no direct reference to. The caller does not own its stack. There is no try/catch reaching across goroutines. You must propagate the error through channels, sync primitives, or a coordination type like errgroup.Group that wraps both.

This file teaches the foundation. By the end you will:

• Know why a goroutine returning an error is not the same as a function returning an error
• Understand the "first error wins" pattern and how errgroup implements it
• Use context.Context to cancel sibling stages when one fails
• Wrap errors with fmt.Errorf("...: %w", err) so the caller can identify them
• Avoid the four most common mistakes: ignored errors, leaked goroutines on error, double-close panics on channels, and lost cancellation

You do not need to know about errors.Join, multi-error aggregation, panic recovery, or compensating rollback yet. Those come at the middle and senior levels.

Prerequisites¶

• Required: Go 1.20 or newer (1.21+ recommended). errors.Join exists since 1.20 but is covered at the middle level. go version.
• Required: Comfort with goroutines and channels at the level of 01-goroutines and 02-channels. You should be able to write a producer-consumer with a single goroutine and a buffered channel without thinking hard.
• Required: Familiarity with context.Context — at minimum context.WithCancel, <-ctx.Done(), and ctx.Err().
• Required: You can read and write a function returning error.
• Helpful: Knowing what defer close(ch) does and why it is the standard pattern for telling consumers a producer is finished.

If you can write the following two functions without help, you are ready:

func gen(ctx context.Context, n int) <-chan int {
    out := make(chan int)
    go func() {
        defer close(out)
        for i := 0; i < n; i++ {
            select {
            case <-ctx.Done():
                return
            case out <- i:
            }
        }
    }()
    return out
}

func consume(ch <-chan int) {
    for v := range ch {
        fmt.Println(v)
    }
}

Glossary¶

Core Concepts¶

1. A goroutine has no return value¶

In sequential code, a function returns its error:

func step1() error { ... }
func step2() error { ... }

if err := step1(); err != nil { return err }
if err := step2(); err != nil { return err }

In concurrent code, you start a goroutine like go step1(). The go statement does not give you back a value. Whatever step1 returns is discarded by the runtime. If step1 fails, you must arrange to publish the error somewhere the caller can see it.

There are three common publication channels:

1. A dedicated error channel: errCh := make(chan error, 1).
2. A shared error variable protected by a mutex.
3. An errgroup.Group, which combines both above behind a clean API.

2. Errors must travel across a synchronisation boundary¶

If goroutine A writes to a variable and goroutine B reads it without synchronisation, the program has a data race. The race detector will flag it. So when you put an error somewhere, you must either:

• Use an atomic operation or mutex
• Use a channel send/receive (which synchronises by the Go memory model)
• Use a primitive like sync.Once or errgroup.Group that handles it for you

This is why the naive "just write to a global error variable" pattern is wrong even though it "works most of the time" in casual testing.

3. The first stage to fail should stop the others¶

In most pipelines, after the first failure further work is wasted. The classic pattern is:

• The failing stage records the error
• A cancellation signal is broadcast to siblings
• Siblings notice the signal and exit
• The caller receives the first error

This is what errgroup.WithContext automates.

4. Channel close is a separate concept from error¶

In Go, closing a channel means "no more values will be sent." It does not mean "an error occurred." A consumer ranging over a channel exits the loop on close. So the question "did the producer finish normally or with an error?" is answered by also checking an error variable — never by inspecting the channel alone.

A naive design that tries to encode error-versus-success into channel state — for instance, leaving a channel open on error and closing it on success — is fragile and confusing. Keep these two concerns separate.

5. The caller must Wait() before reading the error¶

You cannot read an error variable before the writing goroutine has set it. With errgroup you call g.Wait(), which blocks until every goroutine has returned and then gives you the first non-nil error. Reading g before Wait is meaningless.

Real-World Analogies¶

A factory assembly line. Each station does one operation on the product moving along the belt. If station 3 detects a defect, it cannot just stop its own arm — the belt is still feeding it. Someone has to press the emergency stop, which signals stations 1, 2, 4, and 5 to halt too. The defect report (the error) is then carried back to the foreman.

A relay race where one runner trips. The next runner is already moving forward expecting the baton. The teammate behind sees the fall and shouts "stop!" so everyone clears the track. The coach (the caller) gets the report.

A restaurant kitchen. Appetiser, main, dessert stations work in parallel for one table. If the main is burned, there is no point plating the dessert. The expediter (the coordinator) calls "fire the table again from the top," cancelling outstanding work. The waiter (the caller) is told what went wrong.

Three plumbers fixing a leaky house. One discovers a cracked pipe upstream. The others should stop tightening downstream fittings — their work is useless until the cracked pipe is replaced. Without a foreman, they will keep working blindly. The foreman is the context.Context.

Mental Models¶

Model 1: Stages are independent processes connected by pipes¶

Think of a Go pipeline as a Unix pipeline: cat file | grep foo | wc -l. Each program is independent. If grep exits with an error, the kernel sends SIGPIPE to cat and closes the pipe to wc. In Go, the analogue of SIGPIPE is context.Cancel, and the analogue of pipe-close is close(channel). Your job as the pipeline author is to wire up both signals correctly.

Model 2: The pipeline is a state machine with two terminal states¶

Running -> Done and Running -> Failed(err). Every goroutine in the pipeline must be designed to land in one of these states without leaking. Designing each stage to exit when its input closes and when ctx.Done() fires is the trick. Either condition is sufficient.

Model 3: Errors flow up the call tree, data flows down the pipe¶

Data moves from generator to collector through channels. Errors move from any stage back up to the caller through a shared error-capture mechanism. These are two different flows and they should not share a channel.

Model 4: errgroup is a WaitGroup with a brain¶

A WaitGroup knows "are we done?" An errgroup.Group knows "are we done, and if anyone failed, give me the first error." Internally it is a WaitGroup plus a sync.Once-protected error field plus (optionally) a derived Context.

Why Pipeline Errors Are Special¶

In a single-threaded function, an error is local. The stack frame still exists, the variables are still alive, and you simply return. In a pipeline:

1. The error is born in a sibling goroutine. The parent has no stack to unwind. The error has to be moved.
2. Other goroutines are still running. Without explicit cancellation they will keep producing or consuming, wasting CPU, memory, network, and (in the worst case) issuing irreversible side effects like writing to a database.
3. Some goroutines are blocked. A stage waiting on ch <- value cannot just "be told" about the error — it has to be unblocked, which means either the receiver must drain the channel or the sender must select on ctx.Done().
4. Resources are held. Open files, database connections, network sockets, allocated buffers — every running stage may be holding something that needs releasing.
5. The caller may have already moved on. If the caller's context is cancelled (timeout, parent cancellation), goroutines that don't check ctx.Done() will leak.

Each of these is solvable. The composite solution is what production-grade Go calls "structured concurrency": every spawned goroutine has a defined termination condition, every error has a route to the surface, and every blocking operation is select-ed against cancellation.

The First Error Pattern¶

The most common policy for pipelines is first error wins. The first stage to fail:

1. Records its error.
2. Triggers cancellation.
3. Subsequent errors from other stages are discarded (or logged separately).

The caller receives exactly one error: the one that "started the failure cascade." This matches user expectations — when a CLI tool prints an error and exits, you want the root cause, not a list of secondary effects.

A bare-bones implementation:

func run() error {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    errCh := make(chan error, 3) // buffered = number of stages, prevents blocking
    var wg sync.WaitGroup

    wg.Add(3)
    go func() { defer wg.Done(); errCh <- stage1(ctx) }()
    go func() { defer wg.Done(); errCh <- stage2(ctx) }()
    go func() { defer wg.Done(); errCh <- stage3(ctx) }()

    // Wait for all to finish, but cancel as soon as one fails.
    done := make(chan struct{})
    go func() { wg.Wait(); close(done) }()

    var firstErr error
    for {
        select {
        case err := <-errCh:
            if err != nil && firstErr == nil {
                firstErr = err
                cancel()
            }
        case <-done:
            return firstErr
        }
    }
}

This works, but it has corner cases (buffer size, draining, the for-select racing with done). Nobody writes it from scratch. The community settled on golang.org/x/sync/errgroup, which gives you the same semantics in two lines.

Introducing errgroup¶

errgroup.Group is the workhorse for first-error-wins pipelines.

import "golang.org/x/sync/errgroup"

func run(ctx context.Context) error {
    g, ctx := errgroup.WithContext(ctx)

    g.Go(func() error { return stage1(ctx) })
    g.Go(func() error { return stage2(ctx) })
    g.Go(func() error { return stage3(ctx) })

    return g.Wait()
}

What this does:

• errgroup.WithContext(ctx) returns a *Group and a derived context. The derived context is cancelled automatically the moment any g.Go function returns a non-nil error.
• g.Go(fn) spawns a goroutine that runs fn. Internally, the group does wg.Add(1) for you. The function's return value is captured atomically.
• g.Wait() blocks until every spawned goroutine has returned. It then returns the first non-nil error captured. If every goroutine returned nil, Wait returns nil.

The "first non-nil error" is captured with a sync.Once, so concurrent failures are safe. The cancellation happens the moment the first error is captured, before Wait returns — every other stage that respects the context can stop early.

A complete tiny pipeline:

package main

import (
    "context"
    "fmt"
    "log"

    "golang.org/x/sync/errgroup"
)

func main() {
    if err := run(context.Background()); err != nil {
        log.Fatal(err)
    }
}

func run(parent context.Context) error {
    g, ctx := errgroup.WithContext(parent)

    nums := make(chan int)
    doubled := make(chan int)

    // Stage 1: generate
    g.Go(func() error {
        defer close(nums)
        for i := 0; i < 5; i++ {
            select {
            case <-ctx.Done():
                return ctx.Err()
            case nums <- i:
            }
        }
        return nil
    })

    // Stage 2: transform
    g.Go(func() error {
        defer close(doubled)
        for n := range nums {
            if n == 3 {
                return fmt.Errorf("worker: refuse to process %d", n)
            }
            select {
            case <-ctx.Done():
                return ctx.Err()
            case doubled <- n * 2:
            }
        }
        return nil
    })

    // Stage 3: collect
    g.Go(func() error {
        for v := range doubled {
            fmt.Println(v)
        }
        return nil
    })

    return g.Wait()
}

Three things to notice:

1. Every send is wrapped in select with <-ctx.Done(). Without this, when the worker returns with an error, the generator would block forever trying to send into nums.
2. defer close(out) is the responsibility of each stage's sender. Even when failing, you must close so downstream range loops exit. (The defer runs whether the function returns normally or via error.)
3. The collector returns nil — its only job is to drain. If it returned an error, that error might overwrite the meaningful one from the worker, depending on ordering. (At the middle level we look at how to avoid this.)

If you run the program, you see 0 2 4 printed (some interleaving possible), then the program exits with the error worker: refuse to process 3. The generator stopped because the worker returned, doubled was closed by the worker, the collector finished naturally, and Wait returned the worker's error.

Wrapping Errors with %w¶

When the worker returns fmt.Errorf("worker: refuse to process %d", n), the caller sees one string and that is fine for a top-level message. But in a real system you often want to keep the original error around — perhaps a sql.ErrNoRows or a network timeout — while adding context that says where it came from.

This is what %w does in fmt.Errorf:

err := db.QueryRow(...).Scan(&x)
if err != nil {
    return fmt.Errorf("worker: load user %d: %w", id, err)
}

The returned error formats as "worker: load user 42: sql: no rows in result set" when you print it. But it also satisfies errors.Is(err, sql.ErrNoRows), because the wrapped error is preserved inside.

The caller code:

if err := run(ctx); err != nil {
    if errors.Is(err, sql.ErrNoRows) {
        // expected case, handle gracefully
    } else {
        log.Println("unexpected error:", err)
    }
}

This is the foundation of typed error handling in Go. Without %w, every layer would erase the original error and the caller would have to do string matching, which is fragile.

Rules of thumb at the junior level:

• Always wrap errors crossing a stage boundary, so the chain says "stage X: stage Y: underlying detail."
• Use one verb per wrap. fmt.Errorf("worker: %w", err) adds one layer. Do not concatenate two %w verbs.
• Lowercase, no trailing punctuation. fmt.Errorf("connect to db: %w", err), not fmt.Errorf("Connect to DB.: %w", err). The standard library style is lowercase, colon-separated, no period.
• Do not wrap if you also do not add information. fmt.Errorf("%w", err) is pointless. Just return err.

A complete example combining errgroup with wrapping:

func fetchAll(ctx context.Context, urls []string) ([]Result, error) {
    g, ctx := errgroup.WithContext(ctx)
    results := make([]Result, len(urls))

    for i, u := range urls {
        i, u := i, u
        g.Go(func() error {
            r, err := fetchOne(ctx, u)
            if err != nil {
                return fmt.Errorf("fetch %s: %w", u, err)
            }
            results[i] = r
            return nil
        })
    }
    if err := g.Wait(); err != nil {
        return nil, err
    }
    return results, nil
}

If fetchOne for URL https://example.com/users/42 returned context.DeadlineExceeded, the caller sees: "fetch https://example.com/users/42: context deadline exceeded" and errors.Is(err, context.DeadlineExceeded) returns true.

Pros and Cons¶

Pros of the first-error-wins / errgroup pattern¶

• Simple mental model. "Someone failed; everyone stops." Matches how callers think about CLIs and RPC handlers.
• Cancellation is automatic. No manual cancel() plumbing — the group does it.
• Composable. Groups can be nested. One stage of a parent group can run its own subgroup.
• Production-tested. errgroup has been in golang.org/x/sync since 2016 and is used in nearly every major Go service.
• Fast. Internally cheap: a WaitGroup, a sync.Once, and one cancel function pointer.
• Race-free. First-error capture uses Once. No mutex acquisition on every error.

Cons¶

• You lose secondary errors. If two stages fail at the same time, only one bubbles up. The other is silently dropped (unless you capture it separately for logging).
• Cancellation is cooperative. A stage that does not check <-ctx.Done() will keep running after a sibling fails. The group still waits for it.
• Resource cleanup is your job. errgroup does not call Close() on anything. Each stage is responsible for its own defer.
• No backpressure built in. If your stages run at different speeds, channel buffering is still your concern.
• No panic safety by default. A panic in a g.Go function crashes the program. (At senior level we cover wrapping it.)

Use Cases¶

• ETL pipelines: read rows, transform, write — first error fails the batch.
• Parallel HTTP fetches: fan out to N URLs, return on first failure or all succeed.
• Build systems: compile multiple files concurrently, surface the first compile error.
• Image processing: decode, resize, encode in a pipeline; bad image fails the request.
• Streaming aggregation: read messages, parse, batch, flush — any parse failure surfaces.
• CLI tools: parallelise file walks, return first error so the user sees a clear root cause.

When not to use first-error-wins:

• Batch jobs where you want to report every failed item (use aggregation, covered at senior level).
• Background workers that should keep processing on individual failures (handle errors per-item, not per-pipeline).
• User-facing forms where you want to show all validation errors at once.

Code Examples¶

Example 1: Two-stage pipeline with cancellation¶

func sumSquares(ctx context.Context, nums []int) (int64, error) {
    g, ctx := errgroup.WithContext(ctx)
    in := make(chan int)
    squared := make(chan int64)

    g.Go(func() error {
        defer close(in)
        for _, n := range nums {
            select {
            case <-ctx.Done():
                return ctx.Err()
            case in <- n:
            }
        }
        return nil
    })

    g.Go(func() error {
        defer close(squared)
        for n := range in {
            if n < 0 {
                return fmt.Errorf("negative input %d", n)
            }
            select {
            case <-ctx.Done():
                return ctx.Err()
            case squared <- int64(n) * int64(n):
            }
        }
        return nil
    })

    var total int64
    g.Go(func() error {
        for v := range squared {
            total += v
        }
        return nil
    })

    if err := g.Wait(); err != nil {
        return 0, err
    }
    return total, nil
}

The collector mutates total without a mutex — that is safe because every g.Go goroutine runs concurrently with the others, but g.Wait() provides a happens-before relationship: the writes to total are visible to the caller after Wait returns. (We will look closer at this in the memory-model section at the senior level.)

Example 2: Fan-out fetch¶

func fetchAll(ctx context.Context, urls []string) ([]string, error) {
    g, ctx := errgroup.WithContext(ctx)
    bodies := make([]string, len(urls))

    for i, u := range urls {
        i, u := i, u
        g.Go(func() error {
            req, err := http.NewRequestWithContext(ctx, "GET", u, nil)
            if err != nil {
                return fmt.Errorf("build request %s: %w", u, err)
            }
            resp, err := http.DefaultClient.Do(req)
            if err != nil {
                return fmt.Errorf("fetch %s: %w", u, err)
            }
            defer resp.Body.Close()
            b, err := io.ReadAll(resp.Body)
            if err != nil {
                return fmt.Errorf("read %s: %w", u, err)
            }
            bodies[i] = string(b)
            return nil
        })
    }
    if err := g.Wait(); err != nil {
        return nil, err
    }
    return bodies, nil
}

When one fetch fails, the context is cancelled. All in-flight http.Client.Do calls observe the cancellation (via http.NewRequestWithContext) and abort, freeing TCP connections immediately. Without context propagation, slow URLs would keep loading after the first failure for as long as their server allows.

Example 3: Wrapping in a multi-stage chain¶

func processFile(ctx context.Context, path string) error {
    f, err := os.Open(path)
    if err != nil {
        return fmt.Errorf("open %s: %w", path, err)
    }
    defer f.Close()

    rows, err := parse(ctx, f)
    if err != nil {
        return fmt.Errorf("parse %s: %w", path, err)
    }

    if err := store(ctx, rows); err != nil {
        return fmt.Errorf("store %s: %w", path, err)
    }
    return nil
}

The final error chain for a corrupted file might read: "parse /var/data/x.csv: line 42: unexpected EOF". The caller can errors.Is(err, io.ErrUnexpectedEOF) to handle it specifically while still showing the human-friendly message.

Example 4: Sentinel error as a control signal¶

var errNotFound = errors.New("not found")

func lookup(ctx context.Context, id int) (User, error) {
    u, err := db.Get(ctx, id)
    if errors.Is(err, sql.ErrNoRows) {
        return User{}, fmt.Errorf("lookup %d: %w", id, errNotFound)
    }
    if err != nil {
        return User{}, fmt.Errorf("lookup %d: %w", id, err)
    }
    return u, nil
}

func handler(w http.ResponseWriter, r *http.Request) {
    u, err := lookup(r.Context(), 42)
    switch {
    case errors.Is(err, errNotFound):
        http.NotFound(w, r)
    case err != nil:
        http.Error(w, "internal error", http.StatusInternalServerError)
    default:
        json.NewEncoder(w).Encode(u)
    }
}

The point: errNotFound is a package-level sentinel. Callers compare with errors.Is, not ==, so wrapping does not break the comparison. We cover sentinels in depth at the middle level.

Example 5: A pipeline returning partial results plus an error¶

For some tasks, partial results are valuable even on failure (e.g. "we fetched 4 of 5 URLs"). Done naively, this is brittle. The idiomatic shape is:

func fetchAll(ctx context.Context, urls []string) ([]Body, error) {
    g, ctx := errgroup.WithContext(ctx)
    bodies := make([]Body, len(urls))

    for i, u := range urls {
        i, u := i, u
        g.Go(func() error {
            b, err := fetch(ctx, u)
            if err != nil {
                return fmt.Errorf("fetch %s: %w", u, err)
            }
            bodies[i] = b
            return nil
        })
    }
    err := g.Wait()
    // bodies contains successful results in their positions; failed indices are zero values.
    return bodies, err
}

The caller checks err, but can still inspect bodies for what arrived before the cancellation. This is the simplest "best-effort" pattern; it works because g.Wait() ensures all writes to bodies happen-before the return.

Coding Patterns¶

Pattern: g, ctx := errgroup.WithContext(parent) is line one¶

Every error-aware pipeline starts with this. Forgetting WithContext means your group will not cancel siblings on error; you only get error capture. That is rarely what you want.

Pattern: defer close(out) inside the sender goroutine¶

Every stage that owns an output channel must close it on its way out. The cleanest place is the very first defer of the goroutine body:

g.Go(func() error {
    defer close(out)
    // ... do work, send to out, return error or nil ...
})

Pattern: every send is select { case <-ctx.Done(): ...; case out <- v: }¶

This prevents blocking when downstream has already failed.

Pattern: every receive uses for v := range in¶

The consumer terminates naturally when the sender closes. If the consumer also needs to check ctx.Done(), use select:

for {
    select {
    case <-ctx.Done():
        return ctx.Err()
    case v, ok := <-in:
        if !ok { return nil }
        // handle v
    }
}

The for-range is simpler and works when the only reason to stop is a closed channel. The explicit select is needed when the consumer wants to stop even before the channel closes (e.g. it has work of its own that can fail).

Pattern: capture loop variables explicitly when launching goroutines¶

Even on Go 1.22+ where per-iteration scoping is the default, an explicit i, u := i, u line makes the intent obvious and protects you on older Go versions.

Pattern: wrap once per layer, not per error site¶

If you wrap inside step1 and again inside its caller, the user sees a clean chain. If every site inside step1 adds its own wrap, the chain is noisy. One wrap per logical boundary.

Clean Code¶

A clean pipeline at the junior level has these properties:

1. One function constructs and runs the pipeline end-to-end. Stages are small functions or closures inside it.
2. All channels are created in one place. No channel is created by one stage and consumed by a completely separate one — that makes ownership unclear.
3. defer close(out) is the first statement of every sender goroutine. No exceptions.
4. No nil error handling. Either return nil cleanly or return an error. Never panic for control flow.
5. No goroutine reads a shared variable without g.Wait() first. Writes from g.Go functions are visible after Wait; before that, they are races.
6. No time.Sleep for synchronisation. Always use <-ctx.Done() or a channel.
7. Errors are wrapped at every cross-stage boundary. The final error tells the user which stage, what input, and what underlying cause.

A clean three-stage pipeline reads top-to-bottom like:

func run(ctx context.Context, input <-chan Job) error {
    g, ctx := errgroup.WithContext(ctx)
    parsed := make(chan Parsed)
    enriched := make(chan Enriched)

    g.Go(func() error { defer close(parsed);   return parseStage(ctx, input, parsed) })
    g.Go(func() error { defer close(enriched); return enrichStage(ctx, parsed, enriched) })
    g.Go(func() error {                        return writeStage(ctx, enriched) })

    return g.Wait()
}

This is the shape to aim for. Three lines of g.Go, three named stage functions, one Wait. Anything more complicated should be questioned.

Product Use¶

Where errgroup pipelines appear in real products¶

• API request handlers. When one request triggers parallel sub-requests (load user, load orders, load preferences), errgroup.WithContext(r.Context()) is the standard wrapper.
• Batch import jobs. Read CSV, validate, dedupe, insert — each row is a unit, the pipeline is one batch.
• Search indexers. Crawl, parse, tokenise, write to index.
• Image and video transcoders. Decode, transform, encode in parallel for each output format.
• Webhook fan-out. When delivering an event to N subscribers, an errgroup keeps things parallel while surfacing the first delivery failure.
• Internal tooling. Database migrations, log analyzers, deployment scripts.

Product trade-off: fail fast vs degrade gracefully¶

A user-facing handler usually wants fail fast: surface the first error and let the user retry. A scheduled batch job often wants the opposite: log the failure, continue processing the rest, and produce a report. The product decision drives whether you use errgroup (fail fast) or per-item error handling with aggregation (degrade gracefully). The middle and senior files cover the second pattern.

Observability hook¶

In production, before returning the error from g.Wait(), log it (or add a structured log via your logger). errgroup does not log anything itself.

err := g.Wait()
if err != nil {
    logger.Error("pipeline failed", "err", err)
}
return err

This double-surface — log inside, return outside — is the standard production shape.

Error Handling¶

The junior-level checklist:

1. Wrap with %w when crossing a stage boundary. Never bare-return err if more context would help.
2. Never fmt.Errorf("error: %s", err.Error()). That loses the chain.
3. Compare errors with errors.Is and errors.As, never with == (unless you really know it is a known sentinel and not wrapped).
4. If a stage returns context.Canceled because the parent cancelled, that is the error. Do not invent a "wrapper" error like errors.New("cancelled").
5. Treat nil returns as success. Treat any non-nil as failure. There is no third state.
6. Return one error per stage. If a stage discovers two errors at once, wrap one and log the other, or move to middle-level aggregation.

A "context.Canceled vs real error" check is the most common subtlety:

err := g.Wait()
switch {
case err == nil:
    // success
case errors.Is(err, context.Canceled):
    // parent cancelled us — usually not a failure of *our* pipeline
    return nil
case errors.Is(err, context.DeadlineExceeded):
    return fmt.Errorf("pipeline timed out: %w", err)
default:
    return fmt.Errorf("pipeline failed: %w", err)
}

Security Considerations¶

Pipeline error propagation has a few subtle security angles:

• Do not leak secrets in error messages. If your stage handles a credential, fmt.Errorf("auth with key %s: %w", apiKey, err) will print the key. Use fmt.Errorf("auth: %w", err) instead.
• Do not return raw error strings to untrusted users. Internal errors may reveal database structure, file paths, internal hostnames. Map errors to a user-facing form at the HTTP boundary.
• Resource exhaustion under failure. If your error path leaks goroutines or open files, an attacker who triggers errors repeatedly can DoS your service. Audit your stages' cleanup paths.
• Don't log full requests on every error. Logs are often shipped to less-trusted systems. Strip PII before serialising.

These are not pipeline-specific, but they are easy to forget when your wrap looks innocent.

Performance Tips¶

At the junior level, performance is about correctness first and avoiding obviously wasteful work second.

• Buffer channels when appropriate. A buffered channel with capacity 1 between two stages doubles potential parallelism if stages are uneven. Don't go overboard — over-buffering hides backpressure problems.
• Cancel early. Every stage that respects ctx.Done() saves the work it would have done. Don't write a stage that completes a long operation before checking cancellation.
• Don't allocate errors on the hot path. If you might call fmt.Errorf millions of times per second, prefer a pre-defined sentinel for the common case.
• Close output channels promptly. A delayed close keeps the downstream consumer blocked, delaying overall pipeline shutdown on failure.
• Use errgroup.SetLimit if you have a huge fan-out. Capping parallelism at, say, runtime.NumCPU()*4 avoids resource thrash. (Covered in middle.)

At the architecture level, the biggest performance win on the error path is fast cancellation: when one stage fails, every sibling notices and exits within milliseconds. A pipeline that takes 30 seconds to "wind down" after a failure has cancellation bugs.

Best Practices¶

1. Always use errgroup.WithContext, never bare errgroup.Group unless you have a specific reason. The context-bound version is what people expect.
2. One sender per channel. If you have multiple goroutines that want to send to the same channel, route them through a fan-in stage. This makes close unambiguous.
3. Cancellation is everyone's job. Every blocking operation (send, recv, db.Query, http.Do) should be cancellable via ctx.
4. Stages are functions, not anonymous code. A stage you can name and unit-test is a stage you can reason about. Inline goroutine bodies become unmaintainable past three stages.
5. Log inside, return outside. Log the error in the stage where you first know about it (with context), then return it. This gives you a trail in logs even if the caller silently handles the error.
6. Never recover() and ignore. If you must recover from a panic in a stage, convert it to an error and return it. Hiding panics is hiding bugs.
7. Wait exactly once. Calling g.Wait() a second time on a re-used group is a bug (the group is single-use).

Edge Cases and Pitfalls¶

Pitfall 1: Forgotten close, downstream blocks forever¶

g.Go(func() error {
    for _, v := range items {
        out <- v
    }
    return nil // <-- forgot defer close(out)
})

The downstream for v := range out never returns. g.Wait() blocks forever.

Pitfall 2: Send without select, blocks on cancelled context¶

g.Go(func() error {
    defer close(out)
    for _, v := range items {
        out <- v   // <-- no select on ctx.Done()
    }
    return nil
})

If the downstream stage fails and stops reading, this goroutine blocks forever on out <- v. The group never closes.

Pitfall 3: Buffered error channel with the wrong size¶

errCh := make(chan error, 1)  // assumes only one stage produces error
g.Go(func() error { errCh <- stage1(); return nil })
g.Go(func() error { errCh <- stage2(); return nil })  // <-- blocks if first wrote already

This is one reason people use errgroup instead of rolling their own.

Pitfall 4: Reading the error variable before Wait¶

var firstErr error
g.Go(func() error { /* writes to firstErr */ return nil })
fmt.Println(firstErr)  // <-- race; goroutine may not have run yet
g.Wait()

Reading shared state before Wait is a race. Read after.

Pitfall 5: Recovering a panic but not draining¶

g.Go(func() error {
    defer func() { recover() }()
    panic("boom")
})

The goroutine returns nil and the panic is swallowed. The pipeline appears successful. This is worse than the panic itself. If you recover, convert to error and return it.

Pitfall 6: Mixing errgroup with manual cancel¶

g, ctx := errgroup.WithContext(parent)
ctx2, cancel := context.WithCancel(ctx)
defer cancel()
g.Go(func() error { return doWork(ctx2) })

Here cancel is called when run returns, which is fine, but it is easy to write code that calls cancel accidentally early. Trust errgroup to do the cancellation; don't layer your own.

Pitfall 7: Returning ctx.Err() when you don't want to¶

When a sibling fails, every other stage receives ctx.Err() == context.Canceled and many implementations dutifully return it. But errgroup.Wait returns the first error, not necessarily the root cause. If your worker returned context.Canceled first (because it noticed cancellation before the real failure reached Wait), you may see context.Canceled instead of the meaningful error. The fix: structure stages so that cancellation-from-sibling returns ctx.Err() only if nothing else has been seen. In practice, since errgroup captures via sync.Once (first wins), this is usually self-correcting — but be aware that during shutdown your stage's "error" might be cancellation noise.

Common Mistakes¶

1. Naked go func() inside a function that returns an error. The function returns before the goroutine, and the error is lost. Use errgroup.
2. Returning before draining. If a stage returns early without reading the rest of its input channel, the producer blocks forever.
3. Returning err.Error() as errors.New — errors.New(err.Error()) is a destructive copy that loses the wrap chain.
4. Using == to compare wrapped errors. err == io.EOF fails if someone wrapped EOF. Use errors.Is(err, io.EOF).
5. Sharing an errgroup between requests. Each request should have its own. Groups are single-use.
6. Forgetting to handle ctx.Done() in select. Every blocking primitive in a stage needs a cancellation branch.
7. Wrapping nil. fmt.Errorf("foo: %w", nil) creates an error whose Unwrap returns nil. The chain looks corrupt. Always check if err != nil first.

Common Misconceptions¶

"errgroup catches panics."

No. A panic in a g.Go function crashes the program. You must recover manually (see senior level).

"g.Wait returns when context is cancelled."

No. g.Wait returns when every spawned goroutine has returned. If a goroutine ignores cancellation, Wait blocks. The context cancellation is just a signal; cooperation is required.

"errgroup.WithContext cancels on any error, including context.Canceled."

Almost. It cancels on any non-nil error, but if the cancellation reason was a parent context, the group's context was already cancelled — the error captured is still the first one observed.

"Wrapping makes errors bigger and slower."

fmt.Errorf("...: %w", err) allocates a small wrapper struct. The cost is microseconds at most. Worry about other things first.

"Sentinel errors are bad style."

Far from it. They are the simplest way to express "this specific failure mode" and they compose with errors.Is. The advice "prefer error types over sentinels" applies to broad categories like *os.PathError, not to focused markers like io.EOF.

Tricky Points¶

• errgroup.Group zero value is usable, but you do not get context cancellation. Always prefer errgroup.WithContext.
• g.Go does not block on Add if there is no Wait — but the spawned goroutine is still running. Lifetime is bound by Wait, not by Go.
• Order of defer close(out) matters relative to the rest of the body. Put it first so it runs last (defers stack LIFO).
• errgroup v1 vs errgroup.Group{}. Both work; WithContext is the production form.
• The captured error is the first goroutine's return value that was non-nil. Even if a later goroutine has a "better" error, you get the first.
• g.Wait returns the same value if called repeatedly. It is safe to read after Wait, but you usually only call it once.

Test¶

func TestRunFailsOnNegative(t *testing.T) {
    _, err := sumSquares(context.Background(), []int{1, 2, -1})
    if err == nil {
        t.Fatal("expected error, got nil")
    }
    if !strings.Contains(err.Error(), "negative") {
        t.Fatalf("wrong error: %v", err)
    }
}

func TestRunCancelsSiblings(t *testing.T) {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    g, gctx := errgroup.WithContext(ctx)
    var fastDone, slowDone bool

    g.Go(func() error {
        time.Sleep(10 * time.Millisecond)
        return errors.New("fast fail")
    })
    g.Go(func() error {
        defer func() { slowDone = true }()
        select {
        case <-gctx.Done():
            return gctx.Err()
        case <-time.After(time.Second):
            return nil
        }
    })
    err := g.Wait()
    if err == nil || !strings.Contains(err.Error(), "fast fail") {
        t.Fatalf("expected fast fail, got %v", err)
    }
    if !slowDone {
        t.Fatal("slow goroutine did not exit")
    }
    _ = fastDone
}

The second test verifies the core promise: when one stage fails, sibling stages exit promptly via ctx.Done().

Tricky Questions¶

Q. If I call g.Go 100 times and then g.Wait, does it spawn 100 goroutines simultaneously?

Yes, by default. Each g.Go immediately spawns a goroutine. The function does not return until Wait is called. If you want to cap parallelism, use g.SetLimit(n) (added in errgroup in 2022; covered at middle level).

Q. Does errgroup panic if I call g.Go after g.Wait?

No — Go still runs the function in a goroutine and the result is added, but the function call to Wait has already returned with its current state. Calling Go after Wait is a programming error, even if it does not panic. Use a fresh group.

Q. What if my stage function is func(ctx context.Context) error and I want to pass it to g.Go (which expects func() error)?

Wrap with a closure:

g.Go(func() error { return myStage(ctx) })

This is the universal adapter.

Q. If two stages fail with different errors at the exact same instant, which one wins?

Whichever wrote to the internal sync.Once first. The order is unspecified; it depends on scheduler timing. If you care about both, use aggregation (senior level).

Q. Can I nest errgroups?

Yes. The outer group's context becomes the parent of the inner group's context. The inner group's cancellation does not affect the outer. The outer's cancellation cascades to the inner.

Cheat Sheet¶

import "golang.org/x/sync/errgroup"

g, ctx := errgroup.WithContext(parent)

g.Go(func() error {
    defer close(out)
    for v := range in {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case out <- transform(v):
        }
    }
    return nil
})

if err := g.Wait(); err != nil {
    return fmt.Errorf("pipeline: %w", err)
}

Rules:

• errgroup.WithContext, not errgroup.Group{}.
• defer close(out) in the sender.
• select on every send.
• for range on every receive (unless you also need ctx.Done()).
• %w to wrap on the way out.
• errors.Is to identify on the way in.

Self-Assessment Checklist¶

• I can explain why goroutine errors are different from sequential errors.
• I can write an errgroup-based pipeline of three stages without referring to docs.
• I always defer close(out) in sender goroutines.
• I always select on ctx.Done() for sends and recv loops.
• I wrap errors with %w when crossing stages.
• I use errors.Is/errors.As to identify errors, never bare ==.
• I know what happens if I forget WithContext.
• I can name three real-world systems where this pattern fits.

Building Up From Scratch: Five Iterations¶

The fastest way to internalise pipeline error propagation is to build the same pipeline five times, each time fixing one problem the previous iteration had. We'll process a slice of integers, double each one, and return the sum. If we encounter a negative input, the pipeline must fail.

Iteration 1: Sequential baseline¶

func sumDoubled(nums []int) (int, error) {
    total := 0
    for _, n := range nums {
        if n < 0 {
            return 0, fmt.Errorf("negative: %d", n)
        }
        total += n * 2
    }
    return total, nil
}

Trivially correct. No concurrency. The error returns up the stack like any other Go error. This is the reference behaviour we want from every concurrent version.

Iteration 2: Naive goroutines (broken)¶

func sumDoubled(nums []int) (int, error) {
    total := 0
    var mu sync.Mutex

    for _, n := range nums {
        n := n
        go func() {
            if n < 0 {
                fmt.Println("error: negative", n)
                return
            }
            mu.Lock()
            total += n * 2
            mu.Unlock()
        }()
    }
    // The main goroutine returns immediately; total is unfinished.
    return total, nil
}

Problems:

• Returns before goroutines complete. total is whatever it was when main got there.
• The error is printed, not returned. The caller has no way to know it happened.
• No cancellation — even if we knew about the error, the other goroutines keep running.
• No coordination at all.

Iteration 3: WaitGroup + error variable (still broken)¶

func sumDoubled(nums []int) (int, error) {
    total := 0
    var mu sync.Mutex
    var firstErr error
    var wg sync.WaitGroup

    for _, n := range nums {
        n := n
        wg.Add(1)
        go func() {
            defer wg.Done()
            if n < 0 {
                mu.Lock()
                if firstErr == nil {
                    firstErr = fmt.Errorf("negative: %d", n)
                }
                mu.Unlock()
                return
            }
            mu.Lock()
            total += n * 2
            mu.Unlock()
        }()
    }
    wg.Wait()
    return total, firstErr
}

Better. We wait for all goroutines, capture the first error, return both. But:

• All goroutines run to completion even after a failure. With heavy work, this is wasteful.
• No context propagation — if the caller cancels, we don't know.
• The "first error" check is locked behind a mutex; using sync.Once is cleaner.

Iteration 4: errgroup, fail fast¶

func sumDoubled(ctx context.Context, nums []int) (int, error) {
    g, _ := errgroup.WithContext(ctx)
    total := 0
    var mu sync.Mutex

    for _, n := range nums {
        n := n
        g.Go(func() error {
            if n < 0 {
                return fmt.Errorf("negative: %d", n)
            }
            mu.Lock()
            total += n * 2
            mu.Unlock()
            return nil
        })
    }
    if err := g.Wait(); err != nil {
        return 0, err
    }
    return total, nil
}

Much better. First-error wins, captured via sync.Once inside errgroup. Context cancellation triggers on first failure (though our goroutines don't check it — improvement coming). Returning 0 on error matches the sequential baseline.

But notice we discarded the derived ctx — the goroutines don't honor cancellation. In this trivial example each goroutine is so short it doesn't matter, but the habit is dangerous.

Iteration 5: Production-grade¶

func sumDoubled(ctx context.Context, nums []int) (int, error) {
    g, ctx := errgroup.WithContext(ctx)
    total := 0
    var mu sync.Mutex

    for _, n := range nums {
        n := n
        g.Go(func() error {
            select {
            case <-ctx.Done():
                return ctx.Err()
            default:
            }
            if n < 0 {
                return fmt.Errorf("negative: %d", n)
            }
            // Simulate work the cancellation could shortcut:
            select {
            case <-ctx.Done():
                return ctx.Err()
            case <-time.After(10 * time.Millisecond):
            }
            mu.Lock()
            total += n * 2
            mu.Unlock()
            return nil
        })
    }
    if err := g.Wait(); err != nil {
        return 0, err
    }
    return total, nil
}

Now:

• g.Wait() returns the first error.
• errgroup.WithContext cancels the derived ctx on first failure.
• Each goroutine checks <-ctx.Done() at meaningful points and bails out early.
• The mutex protects total; the writes happen-before g.Wait returns.
• Context cancellation from the caller propagates naturally.

This is the iterated form you want in production. Notice how short each version actually is — the patterns themselves are not long. The discipline is what's hard.

Walking Backwards: Reading Existing Pipeline Code¶

Sometimes you join a team and inherit a pipeline. Here's a checklist to read it quickly:

1. Find g.Wait(). This is the join point. Everything between the group's creation and Wait() is the body.
2. Count g.Go calls. This is the stage count. If there's a g.Go inside a loop, that's a fan-out.
3. For each channel, find who closes it. There should be exactly one sender that runs defer close(out).
4. For each send, check for select on ctx.Done(). If absent, the stage can block on cancellation.
5. For each db.Query, http.Do, etc., check that ctx is passed. Without it, the operation cannot be cancelled.
6. Check the error path. Are errors wrapped with %w? Are sentinel errors defined at package level?
7. Identify the resources. Open files, connections, locks. Are they defer-released?
8. Look at the caller. Does it errors.Is against specific failures? If yes, those must be wrapped, not swallowed.

This eight-point audit, done in two minutes, tells you whether the pipeline is "production safe" or "works on a sunny day."

Worked Example 1: A Complete CSV Importer¶

Let us build a realistic pipeline end-to-end so every concept above is grounded in code you can compile.

The task: read a CSV of users, parse each row into a User struct, validate it, then insert into a database. Failure of any row fails the whole import (junior-level policy — at senior we revisit). Cancellation from the caller (timeout, Ctrl-C) stops the import cleanly.

package importer

import (
    "context"
    "encoding/csv"
    "errors"
    "fmt"
    "io"
    "strconv"

    "golang.org/x/sync/errgroup"
)

type User struct {
    ID    int64
    Email string
    Age   int
}

var (
    ErrBadRow       = errors.New("bad row")
    ErrUnderage     = errors.New("underage user")
    ErrDuplicateID  = errors.New("duplicate id")
)

type DB interface {
    Insert(ctx context.Context, u User) error
}

func Import(ctx context.Context, r io.Reader, db DB) (int, error) {
    g, ctx := errgroup.WithContext(ctx)

    rawRows := make(chan []string, 16)
    users   := make(chan User,    16)

    // Stage 1: read CSV rows
    g.Go(func() error {
        defer close(rawRows)
        cr := csv.NewReader(r)
        for {
            row, err := cr.Read()
            if errors.Is(err, io.EOF) {
                return nil
            }
            if err != nil {
                return fmt.Errorf("csv read: %w", err)
            }
            select {
            case <-ctx.Done():
                return ctx.Err()
            case rawRows <- row:
            }
        }
    })

    // Stage 2: parse + validate
    g.Go(func() error {
        defer close(users)
        for row := range rawRows {
            u, err := parseRow(row)
            if err != nil {
                return fmt.Errorf("parse row %v: %w", row, err)
            }
            if err := validate(u); err != nil {
                return fmt.Errorf("validate %d: %w", u.ID, err)
            }
            select {
            case <-ctx.Done():
                return ctx.Err()
            case users <- u:
            }
        }
        return nil
    })

    // Stage 3: insert
    var inserted int
    g.Go(func() error {
        for u := range users {
            if err := db.Insert(ctx, u); err != nil {
                return fmt.Errorf("insert %d: %w", u.ID, err)
            }
            inserted++
        }
        return nil
    })

    if err := g.Wait(); err != nil {
        return inserted, err
    }
    return inserted, nil
}

func parseRow(row []string) (User, error) {
    if len(row) != 3 {
        return User{}, fmt.Errorf("%w: expected 3 columns, got %d", ErrBadRow, len(row))
    }
    id, err := strconv.ParseInt(row[0], 10, 64)
    if err != nil {
        return User{}, fmt.Errorf("%w: id: %v", ErrBadRow, err)
    }
    age, err := strconv.Atoi(row[2])
    if err != nil {
        return User{}, fmt.Errorf("%w: age: %v", ErrBadRow, err)
    }
    return User{ID: id, Email: row[1], Age: age}, nil
}

func validate(u User) error {
    if u.Age < 13 {
        return ErrUnderage
    }
    return nil
}

Trace through what happens for a few scenarios:

1. Healthy import of 100 rows. Each row flows through the three stages. When csv.Read returns io.EOF, the reader returns nil. close(rawRows) is deferred, so it runs. The parser sees the close, exits its for range, returns nil, close(users) runs. The inserter sees that and exits. g.Wait() returns nil. Import returns (100, nil).

2. Bad CSV row at index 42. The parser returns fmt.Errorf("parse row %v: %w", row, ErrBadRow). The errgroup captures the error and cancels the context. The reader, on its next iteration, hits ctx.Done() and returns. The inserter notices its input channel closing (when the parser exits) and finishes whatever rows it had queued, then exits. The whole pipeline shuts down within milliseconds. Import returns (inserted_so_far, err) and the caller can use errors.Is(err, ErrBadRow) to identify the cause.

3. Caller cancels with ctx.Cancel(). Every <-ctx.Done() fires. Every stage returns ctx.Err() (which is context.Canceled). g.Wait returns context.Canceled. The caller sees errors.Is(err, context.Canceled) and treats it as user-initiated, not a bug.

4. DB insert fails. The inserter returns the error. The group cancels. The parser, mid-select, exits. The reader exits. Import returns with the insert error. The number of rows actually inserted depends on timing, which is why returning inserted alongside the error is useful — the caller knows the partial state.

This example demonstrates everything covered: errgroup, wrap with %w, sentinel errors (ErrBadRow, ErrUnderage), defer close, select on send, partial result reporting, and context propagation to the DB.

Worked Example 2: Parallel HTTP Health Checks¶

Different shape — fan-out where each worker is independent.

package healthcheck

import (
    "context"
    "errors"
    "fmt"
    "io"
    "net/http"
    "time"

    "golang.org/x/sync/errgroup"
)

type Result struct {
    URL    string
    Status int
    OK     bool
    Err    error
}

func CheckAll(ctx context.Context, urls []string) ([]Result, error) {
    g, ctx := errgroup.WithContext(ctx)
    results := make([]Result, len(urls))

    for i, u := range urls {
        i, u := i, u
        g.Go(func() error {
            r, err := checkOne(ctx, u)
            results[i] = Result{URL: u, Status: r, OK: err == nil, Err: err}
            // Note: we do NOT return err here.
            // We want every URL checked, not first-error-wins.
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return results, err
    }
    return results, nil
}

func checkOne(ctx context.Context, url string) (int, error) {
    cctx, cancel := context.WithTimeout(ctx, 5*time.Second)
    defer cancel()

    req, err := http.NewRequestWithContext(cctx, "GET", url, nil)
    if err != nil {
        return 0, fmt.Errorf("build request: %w", err)
    }
    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return 0, fmt.Errorf("do: %w", err)
    }
    defer resp.Body.Close()
    _, _ = io.Copy(io.Discard, resp.Body)
    if resp.StatusCode >= 500 {
        return resp.StatusCode, fmt.Errorf("server error: %d", resp.StatusCode)
    }
    return resp.StatusCode, nil
}

var ErrAtLeastOneFailed = errors.New("at least one check failed")

func CheckAllStrict(ctx context.Context, urls []string) ([]Result, error) {
    results, _ := CheckAll(ctx, urls)
    for _, r := range results {
        if !r.OK {
            return results, fmt.Errorf("%w: %s", ErrAtLeastOneFailed, r.URL)
        }
    }
    return results, nil
}