tunny — Professional Level¶

Table of Contents¶

1. Introduction
2. Prerequisites
3. Operating Model
4. CPU-Bound Workloads in Production
5. Image Processing Pipelines
6. Integration With HTTP Handlers
7. Integration With gRPC
8. Integration With Message Queues
9. Observability Stack
10. Graceful Shutdown
11. Capacity Planning
12. Load Testing
13. Resource Limits and cgroups
14. Multi-Tenancy
15. Deployment Topologies
16. Case Study — Image Service at 100M images/day
17. Case Study — PDF Renderer With Heavy Memory
18. Case Study — JSON Schema Validator With CPU Cap
19. Case Study — Crypto Signing Service
20. Case Study — ML Inference Worker Pool
21. Incident Response
22. Postmortems
23. Configuration Management
24. Runtime Tuning
25. Production Best Practices
26. Production Anti-Patterns
27. Self-Assessment Checklist
28. Summary
29. Further Reading
30. Related Topics

Introduction¶

Focus: "How do I keep a tunny-based service healthy in production for years?"

You have used tunny in scripts, side projects, and small services. You understand the API and the internals. Now you must run it in production. The concerns shift:

• It is not enough that the pool works. It must work under load, for years, across deployments, with operators on-call.
• Pool sizing becomes an operational decision, revisited as traffic shifts.
• Observability becomes essential — you cannot debug what you cannot see.
• Shutdown must be coordinated with HTTP servers, signal handlers, and downstream systems.
• Capacity planning becomes a cycle: measure, predict, adjust.

This file covers all of those. It is less about code, more about decisions. Most of the code is shape rather than completeness.

After this file you should be able to:

• Deploy a tunny-based service to production with confidence.
• Set up observability so the pool's behaviour is visible.
• Plan capacity for predicted growth.
• Handle production incidents that touch the pool.
• Write a postmortem about a tunny-related outage.
• Tune GOMAXPROCS, GOGC, and other runtime knobs alongside pool size.

Prerequisites¶

• Comfortable with everything in junior.md, middle.md, and senior.md.
• Familiar with HTTP servers in Go (net/http, or a framework).
• Familiar with Prometheus / OpenTelemetry / structured logging.
• Familiar with deployment systems (Kubernetes, ECS, Nomad, etc).
• Have run a production service for at least three months.
• Familiar with pprof, go tool trace, basic system tools (top, lsof, tcpdump at need).

This file talks at the level of a service owner, not a beginner.

Operating Model¶

A tunny-based service has the following operating-model shape:

   Inbound traffic
        │
        ▼
   HTTP / gRPC / queue consumer
        │ (one goroutine per request/message)
        ▼
   Authentication, validation, etc.
        │
        ▼
   pool.ProcessCtx(ctx, payload)   ← the bottleneck
        │
        ▼
   Worker code
        │
        ▼
   Result back to handler
        │
        ▼
   Response / ack

Every concern in this file falls along this pipeline:

• Inbound shape — HTTP, gRPC, queue. Each has its own context propagation.
• Pool sizing — matched to the workload's CPU/memory profile and downstream limits.
• Observability — metrics at every stage.
• Shutdown — orderly tear-down from front to back.

Keep this picture in mind. Every section is a refinement of one piece.

CPU-Bound Workloads¶

Tunny is at its best for CPU-bound workloads. Examples:

• Image manipulation (resize, encode, watermark).
• Text processing (tokenization, normalization, stemming).
• Cryptographic operations (signing, verification, encryption, KDF).
• Compression (gzip, zstd, brotli).
• Parsing (large JSON, XML, custom formats).
• ML inference (small models running in-process).

For all of these, the rule is:

• Pool size = runtime.NumCPU() is a safe default.
• Each worker holds reusable buffers/codecs as needed.
• HTTP / gRPC handlers call ProcessCtx with the request context.
• Memory footprint = pool_size * per_worker_state.

A common pitfall: sizing the pool to the request rate rather than the CPU budget. If you serve 1000 RPS, that does not mean you need 1000 workers. If each call is 10 ms and you have 8 cores, you need ~8 workers (perhaps with a buffer of 16 for variance).

A specific calculation:

• RPS * call_time = work_capacity_needed
• 1000 RPS * 10 ms = 10 worker-seconds of work per second
• On 8 cores, you can do up to 8 worker-seconds of work per second
• Therefore: you cannot keep up at this rate. You need 10+ cores, or to reduce call_time, or to drop traffic.

This basic queueing-theory exercise is the foundation of capacity planning. Run it for your workload before deploying.

Image Processing Pipelines¶

The canonical tunny use case. A few patterns from real deployments:

Pattern: Single pool, multi-format¶

If most images are similar in cost, one pool suffices. The worker decodes whatever format arrives.

type ImageWorker struct {
    buf bytes.Buffer
}

func (w *ImageWorker) Process(p any) any {
    job := p.(ImageJob)
    w.buf.Reset()
    f, err := os.Open(job.InPath)
    if err != nil {
        return ImageResult{Err: err}
    }
    defer f.Close()
    if _, err := w.buf.ReadFrom(f); err != nil {
        return ImageResult{Err: err}
    }
    img, format, err := image.Decode(&w.buf)
    if err != nil {
        return ImageResult{Err: err}
    }
    return processImage(img, format, job)
}

Pool sized to NumCPU. Each worker has its own buffer.

Pattern: Format-segregated pools¶

If formats have very different costs (PNG is 5x slower than JPEG in your benchmarks), separate them. Each pool can be sized for its workload.

type Service struct {
    jpeg *tunny.Pool
    png  *tunny.Pool
    webp *tunny.Pool
}

Per-pool sizing means no head-of-line blocking across formats. Heavy PNG traffic does not starve JPEG callers.

Pattern: Pre-resize before re-encode¶

For thumbnail services, often the input is huge (e.g. 20 MP) and the output is small (e.g. 200x200). Decoding the whole thing wastes CPU. Use a decoder that supports fractional decode:

func (w *ImageWorker) Process(p any) any {
    job := p.(ImageJob)
    // libjpeg-turbo or similar supports DCT-domain shrinking
    img, err := decodeShrunk(job.InData, job.OutSize)
    if err != nil {
        return ImageResult{Err: err}
    }
    // resize the smaller image
    return resize(img, job.OutSize)
}

This is a worker-level optimisation that can give 4x speedup. The pool size stays the same; throughput goes up.

Pattern: Streaming uploads with pool offload¶

Inbound HTTP requests contain raw image data. The handler reads the body, then passes it to the pool. The pool does decode+process+encode. The handler writes the result back.

http.HandleFunc("/resize", func(w http.ResponseWriter, r *http.Request) {
    data, err := io.ReadAll(r.Body)
    if err != nil { ... }
    result, err := svc.pool.ProcessCtx(r.Context(), data)
    if err != nil { ... }
    w.Write(result.([]byte))
})

A few decisions:

• Should we read the body in the handler or in the worker? In the worker, the read happens in pool-capacity time. In the handler, the read happens regardless of pool capacity. For small payloads, either is fine.
• Should we limit body size? Always, with http.MaxBytesReader. Otherwise a 1 GB upload can OOM your service.

Integration With HTTP Handlers¶

The canonical pattern:

type Service struct {
    pool *tunny.Pool
}

func (s *Service) handler(w http.ResponseWriter, r *http.Request) {
    payload, err := parseRequest(r)
    if err != nil {
        http.Error(w, err.Error(), 400)
        return
    }

    ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
    defer cancel()

    result, err := s.pool.ProcessCtx(ctx, payload)
    if err != nil {
        switch {
        case errors.Is(err, context.Canceled):
            // client gave up
            return
        case errors.Is(err, context.DeadlineExceeded):
            http.Error(w, "timeout", 504)
            return
        case errors.Is(err, tunny.ErrPoolNotRunning):
            http.Error(w, "shutting down", 503)
            return
        }
        http.Error(w, err.Error(), 500)
        return
    }

    writeResponse(w, result)
}

Key points:

• Use r.Context() — propagates client disconnect.
• Wrap in WithTimeout — clamps maximum work duration.
• Map error categories to HTTP status codes.
• Body size limits up front.

For high-throughput services, also consider:

• Pre-validation in the handler so bad inputs do not reach the pool.
• Admission control via QueueLength — return 503 if too many are waiting.
• Per-tenant pools if SLA isolation is needed.

Admission control¶

A simple admission rule:

if s.pool.QueueLength() > s.maxQueue {
    w.Header().Set("Retry-After", "5")
    http.Error(w, "busy", 503)
    return
}

The service stays responsive even under overload. Callers retry later, when the queue has drained.

Choosing maxQueue is a trade-off:

• Too low: you reject traffic you could have served.
• Too high: queue time exceeds caller's timeout, you do work nobody wants.

Rule of thumb: maxQueue = pool_size * avg_call_time / target_p99_latency. For 8 workers, 100 ms calls, 1 s target: maxQueue = 80. Round to 100.

Integration With gRPC¶

gRPC's interceptors give you a clean place to insert the pool:

func PoolInterceptor(pool *tunny.Pool) grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
        result, err := pool.ProcessCtx(ctx, request{ctx: ctx, req: req, handler: handler})
        if err != nil {
            return nil, mapError(err)
        }
        r := result.(response)
        return r.resp, r.err
    }
}

Or you can put the pool inside specific RPC methods, depending on whether all RPCs are CPU-bound.

For streaming RPCs, the pool is less natural — each message in the stream is a separate call, but the stream itself is long-lived. You typically iterate messages and submit each to the pool. Backpressure on the stream comes from the pool naturally.

Integration With Message Queues¶

A common batch-processing shape:

func consume(ctx context.Context, pool *tunny.Pool, q Queue) {
    for {
        msg, err := q.Read(ctx)
        if err != nil {
            return
        }
        result, err := pool.ProcessCtx(ctx, msg)
        if err != nil {
            q.Nack(msg)
            continue
        }
        q.Ack(msg, result)
    }
}

Multiple consumer goroutines, all sharing the same pool. The pool caps in-flight work; the queue provides durable buffering.

Variations:

• Parallel consumers — N goroutines reading from the queue, all submitting to the pool. The pool's size is independent of N.
• At-most-once vs at-least-once — handled at the queue layer (when do you ack?), not in tunny.
• Batch ack — accumulate results, ack in batches for throughput.

This is the shape used for image processing pipelines that ingest from S3 events, Kafka, NATS, etc.

Observability Stack¶

You cannot operate tunny in production without observability. Three things you must export:

1. Pool size and queue length¶

poolSizeGauge.Set(float64(pool.GetSize()))
queueLengthGauge.Set(float64(pool.QueueLength()))

Update these on a timer (every 5-15 seconds) or expose as Prometheus gauges that call the methods directly.

2. Call durations¶

Wrap Process calls:

func (s *Service) Do(ctx context.Context, in In) (Out, error) {
    timer := callDurationHistogram.WithLabelValues(s.name).NewTimer()
    defer timer.ObserveDuration()
    // ... ProcessCtx
}

Histogram with buckets reflecting your expected distribution. For 10-100 ms calls, buckets like [1, 5, 10, 25, 50, 100, 250, 500, 1000] ms.

3. Outcomes (success/error/timeout)¶

outcomeCounter.WithLabelValues(s.name, outcome).Inc()

Where outcome is one of ok, error, timeout, cancelled, pool_closed.

These three families of metrics let you build dashboards:

• Throughput — outcome counters integrated over time.
• Latency distribution — histogram percentiles.
• Saturation — queue length vs pool size.

Set alerts on:

• queue_length > pool_size * 5 for more than 1 minute (saturation).
• p99_latency > target for more than 5 minutes (degradation).
• outcome:error rate > 1% for more than 1 minute (failure).
• outcome:timeout rate > 5% for more than 1 minute (cap problem).

Graceful Shutdown¶

Production services receive SIGTERM. The pool must be closed gracefully — finish in-flight work, then exit. The standard shape:

func main() {
    pool := tunny.New(runtime.NumCPU(), workerFactory)
    server := &http.Server{Addr: ":8080", Handler: makeMux(pool)}

    go func() {
        if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatal(err)
        }
    }()

    sig := make(chan os.Signal, 1)
    signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
    <-sig

    log.Println("shutdown initiated")

    // Phase 1: stop accepting new HTTP requests.
    shutdownCtx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()
    if err := server.Shutdown(shutdownCtx); err != nil {
        log.Printf("server shutdown: %v", err)
    }

    // Phase 2: drain in-flight pool work.
    for pool.QueueLength() > 0 {
        select {
        case <-shutdownCtx.Done():
            log.Println("forced close, queue still nonzero")
            break
        case <-time.After(100 * time.Millisecond):
        }
    }

    // Phase 3: close the pool.
    pool.Close()

    log.Println("shutdown complete")
}

Three phases:

1. Stop accepting new traffic (HTTP server shutdown).
2. Drain in-flight work (wait for queue to empty).
3. Close the pool (releases workers).

The shutdown context bounds the whole thing. If draining takes too long, you force-close.

In Kubernetes:

• Configure terminationGracePeriodSeconds longer than the worst-case Process time.
• Use a preStop hook for additional delay if needed.
• The shutdown context should be shorter than the grace period.

Capacity Planning¶

Capacity planning answers: "How big should my service be?" For a tunny-based service:

1. Measure call duration — p50, p95, p99.
2. Measure call rate — requests/sec, peak and sustained.
3. Compute work-seconds needed — rate * duration.
4. Choose pool size — at least work_seconds_per_second * safety_margin.
5. Choose number of replicas — to provide redundancy and headroom.

Worked example:

• Service receives 500 RPS peak.
• Each call takes 50 ms (p95).
• Work-seconds/sec = 500 * 0.05 = 25.
• On 8-core pods, each pod provides 8 work-seconds/sec.
• Need at least 4 pods (32 work-seconds). For redundancy, 6.
• Pool size on each pod: 8 (one per core).

For headroom against bursts, you might want 8 pods. Cost vs latency trade-off.

This is queueing theory in disguise. The Erlang-C formula and M/M/c models apply. Most engineers do not need the formulas; an intuitive sizing followed by load tests is enough.

Load Testing¶

Test under realistic load before deploying. Tools: hey, wrk, k6, vegeta, custom Go programs.

A simple hey invocation:

hey -c 50 -z 30s -m POST -D image.jpg http://service/resize

50 concurrent clients, 30 seconds. Measure p95, p99, error rate. Compare to single-pod capacity.

Sanity test: monotonic load. Start at 10 RPS, double every minute until errors appear. Find the breaking point. Set production traffic well below.

Burst test: send a sudden 10x burst. Observe how queue length grows and whether latency degrades gracefully or catastrophically.

Long-running soak test: sustain expected production traffic for 24 hours. Look for memory growth, file descriptor leaks, slow degradation.

These three (linear, burst, soak) are the minimum before declaring a tunny-based service production-ready.

Resource Limits and cgroups¶

In containers, runtime.NumCPU() returns the host's CPU count, not the container's CPU quota. This is a footgun:

• Container has 1 CPU quota.
• runtime.NumCPU() returns 64.
• Pool is sized to 64.
• Service is wildly oversubscribed; latency suffers.

Use runtime.GOMAXPROCS(0) after setting it correctly, or use a library like automaxprocs:

import _ "go.uber.org/automaxprocs"

This sets GOMAXPROCS from the container's CPU quota. Then size your pool:

pool := tunny.New(runtime.GOMAXPROCS(0), workerFactory)

This way, your pool size matches reality.

Similarly for memory: set Go's GOMEMLIMIT to a fraction of the container's memory limit. Otherwise the GC may run too aggressively or too laxly.

Multi-Tenancy¶

If your service serves multiple tenants and they have SLA isolation needs, consider:

Option A — shared pool¶

All tenants use the same pool. Simplicity. Risk: a noisy tenant degrades others.

Option B — per-tenant pool¶

Each tenant gets a small pool. Isolation. Cost: many pools, more goroutines, more operational complexity.

Option C — weighted shared pool¶

Use BlockUntilReady with a weighted limiter per tenant. Complex but flexible.

The right answer depends on:

• Number of tenants (handful → per-tenant; thousands → weighted shared).
• SLA strictness (strict → isolated).
• Cost sensitivity (high → shared).

In practice, option B works for B2B services with <100 tenants. Option A works for everything else.

Deployment Topologies¶

A few production topologies:

Topology 1 — Single binary, single pool¶

The most common. One service, one pool. Scales horizontally by deploying more pods.

Topology 2 — Single binary, multiple pools¶

The service has several workloads (image resize, hash, validate). Each has its own pool. Independent tuning.

Topology 3 — Sidecar pool¶

A small "compute" sidecar runs tunny; other services call it. Useful when the compute workload is shared across services.

Topology 4 — Job consumer¶

The service is a queue consumer, not an HTTP server. Tunny is the in-process compute layer. Scales by adding consumers, not request handlers.

Topology 5 — Lambda / serverless¶

Tunny inside a serverless function. Cold-start cost. Limited by function runtime memory. Niche use case.

Pick the topology that matches your traffic pattern. Default to Topology 1.

Case Study — Image Service at 100M images/day¶

A real production deployment processes 100 million images per day. Topology:

• 50 Kubernetes pods, each with 8 CPU cores and 16 GB memory.
• Each pod runs a tunny pool of size 8.
• Each worker holds a bytes.Buffer and a image.Image slot for reuse.
• Inbound: HTTPS from CDN. Outbound: to object storage.
• Average call duration: 75 ms (decode + resize + encode).

Throughput per pod: 8 cores * (1000 / 75) ms/sec = ~106 RPS. Total throughput: 50 * 106 = ~5300 RPS. Daily: 5300 * 86400 = ~458M. Plenty of headroom over 100M.

The pool size of 8 is exactly NumCPU. Larger sizes were tested; they hurt p99 latency due to context switching.

Observability: Prometheus exports queue length, call duration histogram, outcome counters. Grafana dashboards show real-time saturation. Alerts on queue length > 50 sustained.

Shutdown: 30 s grace period, 5 s drain timeout, 5 s pool close timeout. Most pods complete in flight in under 5 s.

Lessons:

• Pool size matters more than replica count for latency.
• Per-worker buffers eliminate most GC pressure.
• Queue length is the single most important metric.
• Soak testing for 24 hours caught a slow goroutine leak we would not have seen otherwise.

Case Study — PDF Renderer With Heavy Memory¶

PDF generation is memory-hungry: each render allocates 100-500 MB temporary. With unbounded concurrency you OOM.

Setup:

• Pods with 64 GB memory.
• Pool size 8 — limits in-flight renders to 8 * 500 MB = 4 GB peak.
• Pool size came from min(NumCPU, MemoryBudget / PerRenderMem).

Workers do not reuse buffers (PDF renderer is third-party C code; we cannot control its allocations). Memory pressure is the dominant constraint, not CPU.

Lesson: pool size is not always CPU-driven. Memory can be the constraint. Profile peak memory per call and divide your budget.

Case Study — JSON Schema Validator With CPU Cap¶

A validator service receives JSON documents and validates against schemas. Each validation: 1-10 ms CPU. The schemas are large (megabytes) and pre-loaded.

Setup:

• Pods with 16 CPU cores.
• Pool size: 32 (2x NumCPU).
• Each worker holds a compiled schema validator. The compilation is expensive (~5 s) so worker construction is slow at startup.

Why 2x NumCPU? Validation has some IO (occasional schema lookup) and the 2x buffer keeps CPUs busy during the brief waits.

Startup time: 5 s schema compilation per worker, parallelised → ~5 s total cold-start. Health probe gated on pool readiness.

Lesson: factory time matters. Slow factories mean slow cold starts. If you can pre-compile and ship a binary blob, do so. If not, accept the cold-start latency.

Case Study — Crypto Signing Service¶

A service signs payloads with a private key. Signing is CPU-bound (a few ms per call) and the key is held in memory (or in an HSM).

Setup:

• Pool size: NumCPU. Each worker holds a reference to the shared key.
• If using HSM, each worker holds an HSM session. Pool size matched to HSM session limit.
• Signing latency: 2-5 ms.

BlockUntilReady enforces HSM rate limit if it has one.

Terminate returns the HSM session.

Lesson: when external resources gate your pool size, the pool size matches the external limit, not local CPU.

Case Study — ML Inference Worker Pool¶

A small image classification model (50 MB) is loaded per worker. Inference takes 10-50 ms.

Setup:

• Pool size: NumCPU / 2 (because the model fits in L3 cache shared by pairs of cores; oversubscribing causes cache contention).
• Model loaded once per worker, in the factory. ~200 ms cold start.
• Inputs pre-resized; outputs are JSON.

Memory: 50 MB * pool_size. On an 8-core pod, 200 MB for models, well within budget.

Lesson: cache-aware sizing can matter for ML workloads. NumCPU / 2 is not arbitrary — it matches the cache topology.

Incident Response¶

When a tunny-related incident fires, the playbook:

1. Check queue length. If high and growing, the pool is saturated.
2. Check call duration. If high, the work itself is slow — investigate downstream.
3. Check error rate. If high, work is failing — look at logs.
4. Check pod count. If low (e.g. due to OOM kills), the service is underprovisioned for traffic.
5. Check GC pause times. If high, allocation pressure is the problem.

Mitigation toolbox:

• Scale up pods. Linear capacity. Slow (minutes to spin up).
• Increase pool size. Free CPU is used. Risk: over-subscription.
• Drop traffic. Admission control returns 503. Saves the service at the cost of users.
• Roll back. If the issue started with a deploy, revert.

After mitigation, postmortem. Always.

Postmortems¶

A tunny incident postmortem template:

1. Timeline. When did metrics deviate? When did alerts fire? When did mitigation start? When did the incident end?
2. Impact. How many requests, how many users, how much money?
3. Root cause. What changed in the system? What was the proximate cause?
4. Contributing factors. What weakened the system before the change?
5. Detection. How did we notice? Could we have noticed sooner?
6. Mitigation. What did we do? What worked?
7. Action items. Specific, owned, dated.

Tunny-related action items often include:

• "Add a queue-length alert at 50."
• "Document the pool size for this workload and require sign-off to change."
• "Add a load test that simulates the failure pattern."
• "Add a circuit breaker upstream of the pool."

Repetition matters. Each postmortem makes the next outage less bad.

Configuration Management¶

Pool size should be configurable, not hardcoded. Suggested config shape:

pool:
  size: 8       # or "auto" for NumCPU
  max_queue: 100
  process_timeout: 5s

Read at startup:

type Config struct {
    Pool struct {
        Size           string        `yaml:"size"`
        MaxQueue       int           `yaml:"max_queue"`
        ProcessTimeout time.Duration `yaml:"process_timeout"`
    } `yaml:"pool"`
}

func PoolSize(cfg Config) int {
    if cfg.Pool.Size == "auto" {
        return runtime.NumCPU()
    }
    n, err := strconv.Atoi(cfg.Pool.Size)
    if err != nil || n < 1 {
        log.Fatalf("bad pool size: %s", cfg.Pool.Size)
    }
    return n
}

Reading from environment variables similarly. Avoid hardcoded magic numbers.

For dynamic environments (Kubernetes), expose these as ConfigMap entries. Restart pods to pick up changes; tunny does not support hot-reloading config (and even if it did, SetSize is the wrong tool for "I changed my mind about sizing").

Runtime Tuning¶

GOMAXPROCS interacts with pool size:

• GOMAXPROCS=1 and pool.size=8 → 8 workers but only 1 can run at a time. Pool size is wasted.
• GOMAXPROCS=8 and pool.size=1 → 1 worker uses 1 core; 7 cores idle.
• GOMAXPROCS=8 and pool.size=8 → ideal for CPU-bound.

Set GOMAXPROCS from the container's CPU quota (automaxprocs). Set pool size to match. They should agree.

GOGC (default 100) controls GC frequency. For low-latency services, set it higher (200-400) to reduce GC pause frequency at the cost of more memory. For memory-constrained services, leave it at 100.

GOMEMLIMIT (Go 1.19+) caps total memory. Set to ~90% of container limit. Above this, GC runs more aggressively.

These three knobs (GOMAXPROCS, GOGC, GOMEMLIMIT) plus pool size are the main tunables. Tune them together.

Production Best Practices¶

1. One pool per workload. Multiple pools per service is fine; one pool serving many workloads is a code smell.
2. Size pools to constraints, not traffic. CPU, memory, downstream limits — pick the binding constraint.
3. Always use ProcessCtx in HTTP handlers. Pass r.Context().
4. Always recover panics in Process. Production services should not crash on bad input.
5. Always close the pool in shutdown. Graceful shutdown means no work in flight when Close is called.
6. Export queue length as a metric. It is your primary saturation signal.
7. Export call duration as a histogram. With buckets matching your SLO.
8. Alert on saturation, not just errors. Errors are the lagging indicator; saturation is the leading.
9. Test under load before deploying. A pool that works at 10 RPS may not at 1000.
10. Profile in production. pprof over HTTPS, rate-limited, behind auth.

Production Anti-Patterns¶

1. Pool created inside HTTP handler. Goroutine leak. Performance disaster.
2. Pool size from os.Getenv without validation. WORKERS=abc should not parse to runtime.NumCPU() silently.
3. No shutdown handling. SIGTERM kills work in flight, leaves transactions abandoned.
4. No observability. You cannot debug what you cannot see.
5. One pool for everything. Heavy work blocks light work. Mixed metrics.
6. Pool size = host CPU count in containers. Use container quota, not host.
7. No load tests. Real traffic is the first stress test. Don't.
8. No admission control. Queue grows unboundedly until OOM.
9. Worker that calls back into the pool. Deadlock under saturation.
10. No alerting on queue length. Outage rolls in silently.

If any of these are in your service, fix them before tomorrow.

Self-Assessment Checklist¶

• My service's pool size is read from config, not hardcoded.
• My service exports QueueLength as a Prometheus gauge.
• My service has an alert on QueueLength > pool_size * 5.
• My service uses ProcessCtx in all HTTP handlers.
• My service recovers panics in Process.
• My service has tested graceful shutdown end-to-end.
• My service has a soak test that runs for at least 1 hour.
• My service has documented its pool sizing rationale.
• My service has a postmortem template ready.
• My service's runtime uses automaxprocs or equivalent.

If you can check all ten boxes, your tunny deployment is genuinely production-ready.

Summary¶

• Tunny in production is about operations, not just code.
• Pool size is the most important decision; tune to constraints.
• Observability (queue length, call duration, outcome) is essential.
• Graceful shutdown is a three-phase dance: stop accepting, drain, close.
• Capacity planning is queueing theory in light disguise.
• Common workloads (images, PDFs, validation, crypto, ML) have specific patterns.
• Common pitfalls (pool per handler, no shutdown, no alerts) are well-known.

If you internalise just one thing from this file: observability and capacity planning are not optional. They are the difference between a tunny service that runs for years and one that fires the on-call every month.

Further Reading¶

• "Production-Ready Microservices" by Susan Fowler — operational principles.
• "Site Reliability Engineering" (Google) — the SRE book, free online.
• The Go runtime documentation — GOMAXPROCS, GOGC, GOMEMLIMIT.
• automaxprocs and automemlimit by Uber and others.
• Prometheus, Grafana, OpenTelemetry documentation.

Related Topics¶

• Junior, middle, senior tunny — the prerequisites.
• Specification (specification.md) — quick API lookup.
• Optimize (optimize.md) — tactical performance tuning.
• Observability stack elsewhere in this roadmap.
• Container orchestration elsewhere in this roadmap.

You are now operational on tunny. Build, deploy, observe, iterate.

Extended Walkthrough — Building a Production Image Service End-to-End¶

Time to put everything together. Below is a complete production-shape image service. We will build it in stages and discuss every operational decision along the way.

Stage 1 — directory structure¶

imgservice/
├── cmd/
│   └── server/
│       └── main.go
├── internal/
│   ├── config/
│   │   └── config.go
│   ├── pool/
│   │   └── pool.go
│   ├── server/
│   │   └── server.go
│   ├── metrics/
│   │   └── metrics.go
│   └── shutdown/
│       └── shutdown.go
├── go.mod
└── Dockerfile

Five packages: config, pool, server, metrics, shutdown. Each with one responsibility.

Stage 2 — config.go¶

package config

import (
    "fmt"
    "os"
    "runtime"
    "strconv"
    "time"

    "gopkg.in/yaml.v3"
)

type Config struct {
    HTTP struct {
        Addr            string        `yaml:"addr"`
        ReadTimeout     time.Duration `yaml:"read_timeout"`
        WriteTimeout    time.Duration `yaml:"write_timeout"`
        ShutdownTimeout time.Duration `yaml:"shutdown_timeout"`
    } `yaml:"http"`
    Pool struct {
        Size           string        `yaml:"size"`
        MaxQueue       int           `yaml:"max_queue"`
        ProcessTimeout time.Duration `yaml:"process_timeout"`
    } `yaml:"pool"`
    Metrics struct {
        Addr string `yaml:"addr"`
    } `yaml:"metrics"`
}

func Load(path string) (*Config, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, err
    }
    defer f.Close()
    var cfg Config
    if err := yaml.NewDecoder(f).Decode(&cfg); err != nil {
        return nil, err
    }
    if cfg.HTTP.Addr == "" {
        cfg.HTTP.Addr = ":8080"
    }
    if cfg.Metrics.Addr == "" {
        cfg.Metrics.Addr = ":9090"
    }
    if cfg.HTTP.ReadTimeout == 0 {
        cfg.HTTP.ReadTimeout = 30 * time.Second
    }
    if cfg.HTTP.WriteTimeout == 0 {
        cfg.HTTP.WriteTimeout = 30 * time.Second
    }
    if cfg.HTTP.ShutdownTimeout == 0 {
        cfg.HTTP.ShutdownTimeout = 30 * time.Second
    }
    if cfg.Pool.ProcessTimeout == 0 {
        cfg.Pool.ProcessTimeout = 10 * time.Second
    }
    if cfg.Pool.MaxQueue == 0 {
        cfg.Pool.MaxQueue = 100
    }
    return &cfg, nil
}

func (c *Config) PoolSize() int {
    if c.Pool.Size == "" || c.Pool.Size == "auto" {
        return runtime.NumCPU()
    }
    n, err := strconv.Atoi(c.Pool.Size)
    if err != nil {
        panic(fmt.Errorf("bad pool size: %w", err))
    }
    if n < 1 {
        panic("pool size must be >= 1")
    }
    return n
}

Default values for everything. auto resolves to NumCPU. Validation happens at load time so the service fails fast on bad config.

Stage 3 — pool.go¶

package pool

import (
    "bytes"
    "context"
    "fmt"
    "image"
    _ "image/jpeg"
    "image/png"
    "io"
    "sync"
    "time"

    "github.com/Jeffail/tunny"
    "golang.org/x/image/draw"
)

type Job struct {
    Data   []byte
    Width  int
    Height int
}

type Result struct {
    Out []byte
    Err error
}

type worker struct {
    inBuf  bytes.Buffer
    outBuf bytes.Buffer

    mu     sync.Mutex
    cancel context.CancelFunc
}

func (w *worker) Process(p any) any {
    job := p.(Job)
    ctx, cancel := context.WithCancel(context.Background())
    w.mu.Lock()
    w.cancel = cancel
    w.mu.Unlock()
    defer func() {
        w.mu.Lock()
        w.cancel = nil
        w.mu.Unlock()
        cancel()
    }()
    defer func() {
        if r := recover(); r != nil {
            // (caught here, return below — but we already returned)
            _ = r // already logged by metrics layer
        }
    }()
    return w.process(ctx, job)
}

func (w *worker) process(ctx context.Context, job Job) Result {
    w.inBuf.Reset()
    w.inBuf.Write(job.Data)

    type imgRes struct {
        img image.Image
        err error
    }
    ch := make(chan imgRes, 1)
    go func() {
        img, _, err := image.Decode(&w.inBuf)
        ch <- imgRes{img, err}
    }()

    var src image.Image
    select {
    case <-ctx.Done():
        return Result{Err: ctx.Err()}
    case r := <-ch:
        if r.err != nil {
            return Result{Err: fmt.Errorf("decode: %w", r.err)}
        }
        src = r.img
    }

    dst := image.NewRGBA(image.Rect(0, 0, job.Width, job.Height))
    draw.CatmullRom.Scale(dst, dst.Bounds(), src, src.Bounds(), draw.Over, nil)

    w.outBuf.Reset()
    if err := png.Encode(&w.outBuf, dst); err != nil {
        return Result{Err: fmt.Errorf("encode: %w", err)}
    }

    // Copy to avoid sharing internal buffer with caller.
    out := make([]byte, w.outBuf.Len())
    copy(out, w.outBuf.Bytes())
    return Result{Out: out}
}

func (w *worker) BlockUntilReady() {}

func (w *worker) Interrupt() {
    w.mu.Lock()
    if w.cancel != nil {
        w.cancel()
    }
    w.mu.Unlock()
}

func (w *worker) Terminate() {}

type Pool struct {
    inner    *tunny.Pool
    maxQueue int
    timeout  time.Duration
}

func New(size, maxQueue int, timeout time.Duration) *Pool {
    return &Pool{
        inner: tunny.New(size, func() tunny.Worker {
            return &worker{}
        }),
        maxQueue: maxQueue,
        timeout:  timeout,
    }
}

var ErrBusy = fmt.Errorf("pool busy")

func (p *Pool) Resize(ctx context.Context, data []byte, w, h int) ([]byte, error) {
    if int(p.inner.QueueLength()) > p.maxQueue {
        return nil, ErrBusy
    }
    ctx, cancel := context.WithTimeout(ctx, p.timeout)
    defer cancel()
    r, err := p.inner.ProcessCtx(ctx, Job{Data: data, Width: w, Height: h})
    if err != nil {
        return nil, err
    }
    res := r.(Result)
    return res.Out, res.Err
}

func (p *Pool) QueueLength() int64 { return p.inner.QueueLength() }
func (p *Pool) Size() int          { return p.inner.GetSize() }

func (p *Pool) Close() { p.inner.Close() }

// ReadAllLimited reads up to maxBytes from r.
func ReadAllLimited(r io.Reader, maxBytes int64) ([]byte, error) {
    return io.ReadAll(io.LimitReader(r, maxBytes))
}

Lots happening:

• Per-worker inBuf and outBuf for reuse.
• Interrupt cancels a per-call context.
• The decode runs in a sub-goroutine because Go's image decoders are not context-aware. The select lets us bail out promptly.
• ErrBusy is returned when the queue is too long.
• Per-call timeout via context.WithTimeout.
• Output is copied to avoid sharing the internal buffer.

Stage 4 — metrics.go¶

package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    PoolSize = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "imgsvc_pool_size",
        Help: "Configured pool size",
    })
    PoolQueue = promauto.NewGauge(prometheus.GaugeOpts{
        Name: "imgsvc_pool_queue_length",
        Help: "Current queue length",
    })
    ProcessDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
        Name:    "imgsvc_process_duration_seconds",
        Help:    "Process call duration",
        Buckets: []float64{0.001, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10},
    }, []string{"outcome"})
    RequestCounter = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "imgsvc_requests_total",
        Help: "Total requests",
    }, []string{"outcome"})
)

Three metric families: pool size, queue length, request distribution. Each is exposed by promauto (which auto-registers).

Stage 5 — server.go¶

package server

import (
    "errors"
    "io"
    "log"
    "net/http"
    "strconv"
    "time"

    "imgservice/internal/metrics"
    "imgservice/internal/pool"

    "github.com/Jeffail/tunny"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

const maxUpload = 10 * 1024 * 1024 // 10 MB

type Server struct {
    Pool *pool.Pool
}

func (s *Server) Routes() http.Handler {
    mux := http.NewServeMux()
    mux.HandleFunc("/resize", s.handleResize)
    mux.HandleFunc("/healthz", s.handleHealthz)
    return mux
}

func (s *Server) handleResize(w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    outcome := "ok"
    defer func() {
        metrics.ProcessDuration.WithLabelValues(outcome).Observe(time.Since(start).Seconds())
        metrics.RequestCounter.WithLabelValues(outcome).Inc()
    }()

    width, _ := strconv.Atoi(r.URL.Query().Get("w"))
    height, _ := strconv.Atoi(r.URL.Query().Get("h"))
    if width <= 0 || height <= 0 || width > 4096 || height > 4096 {
        outcome = "bad_request"
        http.Error(w, "invalid size", 400)
        return
    }

    r.Body = http.MaxBytesReader(w, r.Body, maxUpload)
    data, err := io.ReadAll(r.Body)
    if err != nil {
        outcome = "bad_body"
        http.Error(w, err.Error(), 400)
        return
    }

    out, err := s.Pool.Resize(r.Context(), data, width, height)
    if err != nil {
        switch {
        case errors.Is(err, pool.ErrBusy):
            outcome = "busy"
            w.Header().Set("Retry-After", "5")
            http.Error(w, "busy", 503)
        case errors.Is(err, context.DeadlineExceeded):
            outcome = "timeout"
            http.Error(w, "timeout", 504)
        case errors.Is(err, context.Canceled):
            outcome = "cancelled"
            // client gave up; no response needed
        case errors.Is(err, tunny.ErrPoolNotRunning):
            outcome = "shutting_down"
            http.Error(w, "shutting down", 503)
        default:
            outcome = "error"
            log.Printf("resize error: %v", err)
            http.Error(w, "server error", 500)
        }
        return
    }

    w.Header().Set("Content-Type", "image/png")
    w.Header().Set("Content-Length", strconv.Itoa(len(out)))
    if _, err := w.Write(out); err != nil {
        outcome = "write_error"
        return
    }
}

func (s *Server) handleHealthz(w http.ResponseWriter, r *http.Request) {
    if int(s.Pool.QueueLength()) > s.Pool.Size()*10 {
        http.Error(w, "overloaded", 503)
        return
    }
    w.Write([]byte("ok"))
}

func MetricsHandler() http.Handler { return promhttp.Handler() }

The handler:

• Times the request from start to end.
• Labels by outcome for metrics.
• Limits body size with http.MaxBytesReader.
• Maps internal errors to HTTP status codes.
• Health probe checks queue saturation.

Stage 6 — shutdown.go¶

package shutdown

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "imgservice/internal/pool"
)

func Wait(srv *http.Server, metricsSrv *http.Server, p *pool.Pool, grace time.Duration) {
    sig := make(chan os.Signal, 1)
    signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
    <-sig
    log.Println("shutdown initiated")

    ctx, cancel := context.WithTimeout(context.Background(), grace)
    defer cancel()

    // 1. Stop accepting traffic.
    if err := srv.Shutdown(ctx); err != nil {
        log.Printf("http shutdown: %v", err)
    }
    if err := metricsSrv.Shutdown(ctx); err != nil {
        log.Printf("metrics shutdown: %v", err)
    }

    // 2. Drain in-flight pool work.
    for p.QueueLength() > 0 {
        select {
        case <-ctx.Done():
            log.Println("drain timeout, forcing close")
            p.Close()
            return
        case <-time.After(100 * time.Millisecond):
        }
    }

    // 3. Close the pool.
    p.Close()
    log.Println("shutdown complete")
}

Three phases as described earlier, with a hard deadline.

Stage 7 — main.go¶

package main

import (
    "context"
    "flag"
    "log"
    "net/http"
    "time"

    "imgservice/internal/config"
    "imgservice/internal/metrics"
    "imgservice/internal/pool"
    "imgservice/internal/server"
    "imgservice/internal/shutdown"

    _ "go.uber.org/automaxprocs"
)

func main() {
    cfgPath := flag.String("config", "/etc/imgservice/config.yaml", "config file")
    flag.Parse()

    cfg, err := config.Load(*cfgPath)
    if err != nil {
        log.Fatalf("config: %v", err)
    }

    p := pool.New(cfg.PoolSize(), cfg.Pool.MaxQueue, cfg.Pool.ProcessTimeout)
    metrics.PoolSize.Set(float64(p.Size()))

    // Background ticker for queue length.
    go func() {
        t := time.NewTicker(5 * time.Second)
        defer t.Stop()
        for range t.C {
            metrics.PoolQueue.Set(float64(p.QueueLength()))
        }
    }()

    srv := &http.Server{
        Addr:         cfg.HTTP.Addr,
        Handler:      (&server.Server{Pool: p}).Routes(),
        ReadTimeout:  cfg.HTTP.ReadTimeout,
        WriteTimeout: cfg.HTTP.WriteTimeout,
    }
    metricsSrv := &http.Server{
        Addr:    cfg.Metrics.Addr,
        Handler: server.MetricsHandler(),
    }

    go func() {
        log.Printf("http listening on %s", cfg.HTTP.Addr)
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("http: %v", err)
        }
    }()
    go func() {
        log.Printf("metrics listening on %s", cfg.Metrics.Addr)
        if err := metricsSrv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("metrics: %v", err)
        }
    }()

    shutdown.Wait(srv, metricsSrv, p, cfg.HTTP.ShutdownTimeout)
    _ = context.Background()
}

Wiring: load config, create pool, start servers, wait for shutdown. The whole main.go is 60 lines.

Stage 8 — Dockerfile¶

FROM golang:1.22-alpine AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /out/server ./cmd/server

FROM gcr.io/distroless/static
COPY --from=build /out/server /server
EXPOSE 8080 9090
ENTRYPOINT ["/server", "--config=/etc/imgservice/config.yaml"]

Multi-stage build. Distroless final image (~20 MB).

Stage 9 — Kubernetes manifest¶

apiVersion: apps/v1
kind: Deployment
metadata:
  name: imgservice
spec:
  replicas: 4
  selector:
    matchLabels: {app: imgservice}
  template:
    metadata:
      labels: {app: imgservice}
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: server
        image: imgservice:latest
        resources:
          requests: {cpu: "4", memory: "2Gi"}
          limits: {cpu: "4", memory: "4Gi"}
        ports:
        - containerPort: 8080
        - containerPort: 9090
        livenessProbe:
          httpGet: {path: /healthz, port: 8080}
          periodSeconds: 10
        readinessProbe:
          httpGet: {path: /healthz, port: 8080}
          periodSeconds: 5
        volumeMounts:
        - name: config
          mountPath: /etc/imgservice
      volumes:
      - name: config
        configMap:
          name: imgservice-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: imgservice-config
data:
  config.yaml: |
    http:
      addr: ":8080"
    pool:
      size: "auto"
      max_queue: 100
      process_timeout: 10s
    metrics:
      addr: ":9090"

Notes:

• terminationGracePeriodSeconds: 60 allows the service to complete draining.
• CPU limit is the integer 4 — automaxprocs will set GOMAXPROCS=4.
• The pool sizes itself based on GOMAXPROCS.
• Health probes hit /healthz, which reports overload via 503.

Stage 10 — Prometheus alerts¶

groups:
- name: imgservice
  rules:
  - alert: ImgServicePoolSaturated
    expr: imgsvc_pool_queue_length > imgsvc_pool_size * 5
    for: 1m
    labels: {severity: warning}
    annotations:
      summary: "Pool queue too long on {{ $labels.instance }}"

  - alert: ImgServiceHighErrorRate
    expr: |
      sum(rate(imgsvc_requests_total{outcome="error"}[5m])) /
      sum(rate(imgsvc_requests_total[5m])) > 0.01
    for: 5m
    labels: {severity: critical}
    annotations:
      summary: "Error rate above 1% on {{ $labels.instance }}"

  - alert: ImgServiceP99HighLatency
    expr: |
      histogram_quantile(0.99,
        sum(rate(imgsvc_process_duration_seconds_bucket[5m])) by (le)
      ) > 1.0
    for: 5m
    labels: {severity: warning}
    annotations:
      summary: "p99 latency above 1s"

Three alerts covering saturation, errors, and latency. The cornerstone of production observability.

Stage 11 — Grafana dashboard sketch¶

Panels:

1. Request rate (per outcome): sum(rate(imgsvc_requests_total[1m])) by (outcome).
2. p50/p95/p99 latency: histogram quantiles.
3. Queue length: imgsvc_pool_queue_length.
4. Pool size: imgsvc_pool_size.
5. Queue saturation %: imgsvc_pool_queue_length / imgsvc_pool_size.
6. Error rate %: derived from outcome counter.

Hang the dashboard in the team's war room. Glance at it daily.

Stage 12 — Load test¶

# Linear ramp.
hey -c 10 -z 1m -m POST -D image.jpg "http://localhost:8080/resize?w=256&h=256"
hey -c 50 -z 1m -m POST -D image.jpg "http://localhost:8080/resize?w=256&h=256"
hey -c 100 -z 1m -m POST -D image.jpg "http://localhost:8080/resize?w=256&h=256"

# Burst.
hey -c 200 -n 5000 -m POST -D image.jpg "http://localhost:8080/resize?w=256&h=256"

# Sustained.
hey -c 50 -z 10m -m POST -D image.jpg "http://localhost:8080/resize?w=256&h=256"

Watch metrics during each. Tune pool size and max queue based on the data.

This is what a production tunny service looks like end-to-end. About 400 lines of Go, plus configuration. The pool is one piece of the puzzle, but with the right scaffolding, it works reliably.

Operating This Service Day to Day¶

A week in the life of imgservice:

Monday — deploy¶

Push the new version. Canary on one pod, observe metrics for 30 minutes, then roll the rest. Watch for:

• p99 latency change.
• Error rate change.
• Memory or goroutine count drift.

Tuesday — alerts fire¶

ImgServicePoolSaturated fires. Check Grafana. Queue is at 80 for 5 minutes. CPU is at 100%. Decision: scale to 6 pods.

Make the change. Queue drops. p99 drops. Alert clears.

Postmortem: traffic up 50% from previous week. Update capacity plan.

Wednesday — quiet day¶

Routine. Glance at dashboards over coffee. Nothing burning.

Thursday — a worker panics¶

A new file format arrives that crashes the decoder. The pool's recover catches it. Outcome counter shows a spike in error. Logs show the panic stack.

File a bug. Add a unit test for the input. Roll out a fix.

Friday — shutdown test¶

Manually trigger a pod restart during peak load. Verify:

• In-flight requests complete.
• Queue drains.
• No requests are abandoned mid-response.

Pass. Document the test.

Saturday — soak test catches an issue¶

A scheduled soak test on staging shows memory creeping up over 12 hours. Investigation finds a small leak in the encoder — bytes.Buffer growing without bound.

Fix: cap the buffer size, re-allocate if exceeded.

Sunday — quiet¶

The system works.

This rhythm is what good operations looks like. Most days are boring. The exceptions are caught early, investigated thoroughly, and prevented from recurring.

Advanced Operational Patterns¶

A few patterns that come up in mature production deployments.

Pattern: Two-level pools¶

The outer pool handles HTTP routing. The inner pool handles compute. The outer is sized for connection limits; the inner for CPU.

type Service struct {
    routing *tunny.Pool
    compute *tunny.Pool
}

Useful when routing has light pre-processing (auth, validation) that you do not want to mix with heavy compute.

Pattern: Warm-up¶

If your service has slow worker startup (model loading), do not accept traffic until warm:

func (s *Service) Ready() bool {
    return s.pool.Size() == s.targetSize
}

Readiness probe returns 503 until ready. Kubernetes does not route traffic to unready pods.

Pattern: Slow consumer detection¶

Track per-call latency. If a consumer is consistently slow:

• Log it.
• Consider tighter timeouts for that caller.
• Consider rate limiting that caller.

Tunny does not give you per-caller stats; you must instrument them in your handler.

Pattern: Adaptive timeout¶

If most calls finish in 50 ms, set the timeout to 500 ms (10x). If load grows, callers wait longer, and the timeout still catches genuine stalls.

For automated adaptation, track the p99 over the last hour and set the timeout to 2x or 3x p99.

Pattern: Bulkheading¶

Separate pools for different priorities or tenants. A noisy tenant cannot affect others. We covered this earlier; restating because it is critical for SaaS.

Pattern: Hedged requests¶

For tail-latency sensitive workloads, send a backup request after the median latency:

result := make(chan Result, 2)
go func() { result <- pool.Process(x) }()
time.AfterFunc(median, func() {
    go func() { result <- pool.Process(x) }()
})
r := <-result

The first result wins. The second is wasted work. Useful when occasional slowness is the enemy.

Caveat: doubles the work. Use selectively.

Pattern: Circuit breaker around the pool¶

If the pool starts failing consistently, open a circuit breaker to shed traffic upstream. Tunny does not have this built in; use gobreaker or similar.

Pattern: Distributed pool via NATS / Kafka¶

Multiple service instances cooperate via a message queue. Each instance has its own local tunny pool. The queue provides cross-instance load balancing.

This is the "consume from queue" pattern at scale.

Debugging in Production¶

When things go wrong, your tools:

pprof¶

curl http://service/debug/pprof/profile?seconds=30 > cpu.pprof
go tool pprof cpu.pprof

Look at top --cum and web to see where CPU goes.

curl http://service/debug/pprof/heap > heap.pprof
go tool pprof heap.pprof

Look for big allocators.

curl http://service/debug/pprof/goroutine?debug=2 > goroutines.txt

Inspect stacks. Look for many goroutines in unusual places.

Logs¶

Structured logs with the request ID, the worker ID (if you assigned one), the outcome.

A typical log line:

{"ts": "...", "level": "info", "msg": "request done", "id": "req-123", "worker": 4, "duration_ms": 87, "outcome": "ok"}