Graceful Shutdown — Professional Level¶

Table of Contents¶

1. Introduction
2. UNIX Signals at the Kernel Level
3. Go Runtime Signal Handling
4. The os/signal Package Internals
5. Inside signal.NotifyContext
6. Inside http.Server.Shutdown
7. Inside http.Server.Close
8. The net.Listener Lifecycle
9. Connection State Tracking in http.Server
10. SIGKILL Mechanics at the Kernel Level
11. Container Runtime Signal Delivery
12. Kubernetes kubelet Internals
13. PID 1 and Signal Forwarding
14. Go Runtime Internals During Shutdown
15. Performance Characteristics of Shutdown
16. Version-Specific Behaviour
17. Reading the Standard Library Source
18. Performance Profiling of Shutdown
19. Edge Cases at the Lowest Level
20. Conclusion

Introduction¶

At professional level we leave architecture behind and read the source. What does the Linux kernel do when SIGTERM is delivered? How does the Go runtime intercept it without disrupting the running program? What is the actual algorithm inside http.Server.Shutdown? How does Kubernetes' kubelet decide to send SIGKILL?

The questions are no longer "how do I design my service" but "what exactly happens, byte by byte, when these things occur."

This file is for engineers who:

• Debug kernel-level signal handling issues.
• Maintain Go itself or libraries that need internal-level correctness.
• Work in environments where the standard patterns don't fit.
• Want to understand why the patterns we use exist.

It is not required reading for most production work. But it is the deepest layer, and a small number of incidents per year reach into it. When they do, this knowledge is the difference between "we don't know" and "we found it."

UNIX Signals at the Kernel Level¶

A signal is one of the oldest IPC mechanisms in UNIX, dating to Bell Labs. The kernel-side mechanics:

Signal numbers and types¶

The kernel reserves a small numeric range (1–64 on Linux) for signals. The signal number maps to a name like SIGTERM. Real-time signals (SIGRTMIN–SIGRTMAX) extend the range and offer FIFO delivery guarantees; standard signals can be coalesced.

For graceful shutdown, we care about standard signals 1–15 mostly:

• SIGHUP (1): terminal hangup, often reused for reload.
• SIGINT (2): terminal interrupt (Ctrl+C).
• SIGQUIT (3): terminal quit (Ctrl+), default action is core dump.
• SIGTERM (15): polite terminate.
• SIGKILL (9): forcible terminate.
• SIGSTOP (19): forcible stop (cannot be caught).

Signal delivery flow¶

When the kernel decides to deliver a signal to process P:

1. Pending bit set. In P's task_struct, the bit for the signal is set in the pending field.
2. Wake up. If P is sleeping (in a syscall), the kernel wakes it.
3. Signal handling on return to userspace. When P next returns from kernel to userspace, before resuming user code, the kernel checks pending signals.
4. Action selection. The kernel looks up the signal's action: default, ignore, or user-defined handler.
5. User handler invocation. If user-defined, the kernel arranges for the handler to be called: pushes a special frame onto P's stack, sets PC to the handler.
6. Handler returns. When the handler returns, the kernel restores the original stack frame and resumes user code.

Signals and threads¶

In multi-threaded processes (Go's case), signal delivery is more complex:

• Each thread has its own signal mask (sigprocmask). A masked signal is held pending for that thread.
• The kernel chooses an arbitrary thread for delivery — specifically, a thread that doesn't have the signal masked.
• If all threads have the signal masked, the signal stays pending at the process level until some thread unmasks.

For Go, the runtime configures a dedicated signal-handling thread (or thread group) and the rest of the threads usually have most signals masked.

Coalescing¶

For standard (non-realtime) signals, multiple deliveries of the same signal before the process handles the first are coalesced. The process sees only one delivery. This is fine for SIGTERM (you only care that it arrived) but problematic for SIGCHLD (you want to reap each child exit).

Realtime signals¶

SIGRTMIN+0 through SIGRTMAX (typically 34–64) are queued, not coalesced. Each delivery is preserved. Used for high-frequency notifications. Go uses one of these (SIGURG) internally for async preemption.

signal(2) vs sigaction(2)¶

The kernel-level API. signal(2) is old and has inconsistent semantics across systems. sigaction(2) is the modern, portable API. Go uses sigaction.

kill(2) and friends¶

kill(pid, sig) sends a signal to a process. tgkill(tgid, tid, sig) sends to a specific thread within a process. pkill is a command-line wrapper. Kubernetes' kubelet ultimately invokes a kill syscall.

What kill -9 actually does¶

kill -9 <pid> sends signal 9 (SIGKILL). The kernel sees this and:

1. Sets the SIGKILL pending bit.
2. Wakes the process.
3. Cleans up the process: closes file descriptors, releases memory, removes from process table.
4. Does not invoke any user-space handler — SIGKILL is not catchable.

The process is gone within microseconds.

Go Runtime Signal Handling¶

Go's runtime handles signals in a way that is largely transparent to user code but quite sophisticated underneath.

The signal-handling thread¶

The runtime starts a dedicated thread early on (in runtime.signal_handle). This thread:

• Has all signals unmasked (so it receives them all).
• Loops on a signal receiver.
• Dispatches signals to runtime internals and to user-space subscribers via the os/signal package.

sigaction installation¶

The runtime calls sigaction(2) for every signal it cares about, installing its own handler in runtime.sighandler. This handler:

1. Saves the original signal context.
2. Checks if the signal should be forwarded to a user handler (registered via signal.Notify).
3. If a user handler is registered, the signal is queued internally.
4. Returns to the kernel; kernel resumes the interrupted code.

A goroutine inside the runtime (signal_recv in runtime/sigqueue.go) wakes up periodically and consumes the queued signals, then sends them on the appropriate channels.

signal_recv goroutine¶

Reads from the internal queue. For each signal:

• Iterates over the registered channels for that signal.
• Sends the signal on each channel (non-blocking).

If a channel is full, the send is silently dropped. This is why signal.Notify channels must be buffered.

Signal masking on M (thread) startup¶

When the runtime starts a new OS thread (M), it sets the thread's signal mask to block most signals, except a few essential ones (like SIGURG for preemption). This ensures signals go to the dedicated handling thread, not to arbitrary application threads.

Async preemption via SIGURG¶

Since Go 1.14, the runtime sends SIGURG to a goroutine's hosting thread to preempt long-running goroutines (without cooperation). The handler in runtime.sighandler intercepts SIGURG, checks if preemption is intended, and arranges for the goroutine to yield.

This is one reason you should never subscribe to SIGURG via signal.Notify — you would interfere with preemption.

Synchronous vs asynchronous signals¶

The runtime distinguishes:

• Synchronous signals (SIGSEGV, SIGBUS, SIGFPE, etc.): caused by the program itself (invalid memory access). Default action: panic the goroutine.
• Asynchronous signals (SIGTERM, SIGINT, etc.): sent externally. Default action: terminate the process.

The user-handler queue is for asynchronous signals; synchronous ones go straight to panic.

Signal context save/restore¶

On signal delivery, the runtime saves the CPU registers (in mcontext_t) before invoking the handler. After the handler returns, registers are restored. This is what makes signal handling transparent to user code.

The os/signal Package Internals¶

The os/signal package is the user-facing API for signal handling. Let's read its key functions.

signal.Notify(c chan<- os.Signal, sig ...os.Signal)¶

// from src/os/signal/signal.go
func Notify(c chan<- os.Signal, sig ...os.Signal) {
    if c == nil {
        panic("os/signal: Notify using nil channel")
    }
    handlers.Lock()
    defer handlers.Unlock()

    h := handlers.m[c]
    if h == nil {
        if handlers.ref == nil {
            handlers.ref = make(map[uint32]int64)
        }
        h = new(handler)
        handlers.m[c] = h
    }

    add := func(n int) {
        if n < 0 {
            return
        }
        if !h.want(n) {
            h.set(n)
            if handlers.ref[uint32(n)] == 0 {
                enableSignal(n)
            }
            handlers.ref[uint32(n)]++
        }
    }

    if len(sig) == 0 {
        for n := 0; n < numSig; n++ {
            add(n)
        }
    } else {
        for _, s := range sig {
            add(signum(s))
        }
    }
}

What it does:

1. Acquires the global handlers lock.
2. Gets or creates a handler struct for the channel.
3. For each signal:
4. If not already set, mark it as "wanted" on this handler.
5. If reference count for this signal is 0, call enableSignal(n) which calls into the runtime to start delivering this signal to user-space.
6. Increment reference count.

The reference count is what allows multiple Notify calls for the same signal: enabling happens once; disabling happens when ref count drops to zero (via Stop).

signal.Stop(c chan<- os.Signal)¶

func Stop(c chan<- os.Signal) {
    handlers.Lock()
    defer handlers.Unlock()
    h := handlers.m[c]
    if h == nil {
        return
    }
    delete(handlers.m, c)

    for n := 0; n < numSig; n++ {
        if h.want(n) {
            handlers.ref[uint32(n)]--
            if handlers.ref[uint32(n)] == 0 {
                disableSignal(n)
            }
        }
    }
}

Symmetric to Notify:

1. Removes the handler from the map.
2. For each signal previously wanted by this handler:
3. Decrement reference count.
4. If reference count is 0, call disableSignal(n) to stop delivery.

This is what makes defer signal.Stop(ch) non-optional: without it, the reference count stays elevated and signals continue being captured by the runtime even though no one is listening.

The process function¶

The internal heart of os/signal:

func process(sig os.Signal) {
    n := signum(sig)
    if n < 0 {
        return
    }

    handlers.Lock()
    defer handlers.Unlock()

    for c, h := range handlers.m {
        if h.want(n) {
            // Non-blocking send.
            select {
            case c <- sig:
            default:
            }
        }
    }
}

When the runtime calls process (via the signal_recv goroutine), this function iterates over all handlers, and for each handler interested in this signal, does a non-blocking send.

Three critical observations:

1. The send is non-blocking. A full channel silently drops the signal.
2. Multiple handlers receive. If you have two signal.Notify calls for SIGTERM, both channels get the signal.
3. The lock is global. During process, no Notify/Stop can run. Brief pause, but worth knowing.

The runtime side: runtime/sigqueue.go¶

The runtime has its own queue. signal_recv reads from it:

// from src/runtime/sigqueue.go
func signal_recv() uint32 {
    for {
        // Serve any signals from the local copy.
        for i := uint32(0); i < _NSIG; i++ {
            if sig.recv[i/32]&(1<<(i&31)) != 0 {
                sig.recv[i/32] &^= 1 << (i & 31)
                return i
            }
        }
        // ...wait for new signals...
    }
}

A bit-set tracks which signals have arrived. signal_recv clears the bit and returns the signal number.

User-space code (os/signal.loop) reads from signal_recv in a loop and calls process for each.

func loop() {
    for {
        process(syscall.Signal(signal_recv()))
    }
}

Why the buffered channel matters¶

Recall the channel-must-be-buffered rule. Here's why exactly:

• process does a non-blocking send (select { case c <- sig: default: }).
• If the channel is unbuffered AND no goroutine is reading, the send goes to default — signal dropped.
• If the channel is buffered (cap 1) AND no goroutine is reading, the send fills the buffer — signal preserved.

Buffer of 1 is enough because subsequent sends still go to default (drop), but the first one is preserved.

Inside signal.NotifyContext¶

The Go 1.16+ helper. Source:

// from src/os/signal/signal.go
func NotifyContext(parent context.Context, signals ...os.Signal) (ctx context.Context, stop context.CancelFunc) {
    ctx, cancel := context.WithCancel(parent)
    c := &signalCtx{
        Context: ctx,
        cancel:  cancel,
        signals: signals,
    }
    c.ch = make(chan os.Signal, 1)
    Notify(c.ch, signals...)
    if ctx.Err() == nil {
        go func() {
            select {
            case <-c.ch:
                c.cancel()
            case <-c.Done():
            }
        }()
    }
    return c, c.stop
}

The implementation is small:

1. Derive a context with WithCancel.
2. Create a buffered (cap 1) channel.
3. Register the channel with signal.Notify.
4. Spawn a goroutine that:
5. Cancels the derived context on first signal arrival.
6. Or returns when the parent is cancelled.

The stop function is c.stop:

func (c *signalCtx) stop() {
    c.cancel()
    Stop(c.ch)
}

stop cancels the context (so <-ctx.Done() returns) and deregisters the channel from signal.Notify.

Why defer stop() is mandatory¶

Without defer stop():

• The signal handler is still registered. The reference count is elevated.
• The goroutine in step 4 is leaked (waiting on c.ch).
• enableSignal(n) was called but disableSignal(n) is never called.

In a long-running program, this is a small but real leak that accumulates if NotifyContext is called repeatedly (e.g., in tests).

Why the goroutine inside¶

NotifyContext spawns its own goroutine to bridge channel-receive to context-cancellation. This goroutine is part of the API. Stopping it requires calling stop (or for the parent to be cancelled, which the <-c.Done() branch handles).

WithCancelCause enhancement (Go 1.20+)¶

A small but useful pattern:

func notifyWithCause(parent context.Context, sigs ...os.Signal) (context.Context, func()) {
    ctx, cancel := context.WithCancelCause(parent)
    ch := make(chan os.Signal, 1)
    signal.Notify(ch, sigs...)
    go func() {
        select {
        case s := <-ch:
            cancel(fmt.Errorf("signal: %v", s))
        case <-ctx.Done():
        }
    }()
    return ctx, func() {
        signal.Stop(ch)
        cancel(nil)
    }
}

Now context.Cause(ctx) returns the actual signal that caused the cancellation. Useful for logging.

Inside http.Server.Shutdown¶

The full Shutdown implementation, abbreviated. Source: src/net/http/server.go.

func (srv *Server) Shutdown(ctx context.Context) error {
    srv.inShutdown.Store(true)

    srv.mu.Lock()
    lnerr := srv.closeListenersLocked()
    for _, f := range srv.onShutdown {
        go f()
    }
    srv.mu.Unlock()
    srv.listenerGroup.Wait()

    pollIntervalBase := time.Millisecond
    nextPollInterval := func() time.Duration {
        // ... exponential backoff capped at 500ms ...
    }

    timer := time.NewTimer(nextPollInterval())
    defer timer.Stop()
    for {
        if srv.closeIdleConns() {
            return lnerr
        }

        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
            timer.Reset(nextPollInterval())
        }
    }
}

Step by step:

Step 1: Set the shutdown flag¶

srv.inShutdown.Store(true)

An atomic flag. Other parts of http.Server check this to reject new operations.

Step 2: Close listeners¶

lnerr := srv.closeListenersLocked()

Iterates over all registered listeners and closes them. The internal accept loop in Serve sees the close and returns. This is what stops new connections.

closeListenersLocked:

func (srv *Server) closeListenersLocked() error {
    var err error
    for ln := range srv.listeners {
        if cerr := (*ln).Close(); cerr != nil && err == nil {
            err = cerr
        }
    }
    return err
}

Each listener's Close() is called. Errors are collected; first one is returned.

Step 3: Fire OnShutdown callbacks¶

for _, f := range srv.onShutdown {
    go f()
}

Each callback runs in its own goroutine. Shutdown does not wait for them.

Step 4: Wait for listener-accept goroutines¶

srv.listenerGroup.Wait()

listenerGroup is a sync.WaitGroup tracking the accept loop goroutines. Each accept loop decrements when it returns (which happens because the listener is closed).

Step 5: Polling loop on idle / active connections¶

for {
    if srv.closeIdleConns() {
        return lnerr
    }
    select {
    case <-ctx.Done():
        return ctx.Err()
    case <-timer.C:
        timer.Reset(nextPollInterval())
    }
}

closeIdleConns returns true when all connections are closed (both idle and active). The loop wakes periodically (initially 1 ms, exponentially up to 500 ms) and checks.

closeIdleConns¶

func (srv *Server) closeIdleConns() bool {
    srv.mu.Lock()
    defer srv.mu.Unlock()
    quiescent := true
    for c := range srv.activeConn {
        st, unixSec := c.getState()
        if st == StateNew && unixSec < time.Now().Unix()-5 {
            st = StateIdle
        }
        if st != StateIdle || unixSec >= time.Now().Unix() {
            quiescent = false
            continue
        }
        c.rwc.Close()
        delete(srv.activeConn, c)
    }
    return quiescent
}

For each tracked connection:

• Get its state and last-state-change timestamp.
• If state is StateIdle (keep-alive, no active request): close it and remove from the map.
• If state is StateNew and >5 seconds old: treat as idle.
• Otherwise (e.g., StateActive, StateHandlingReadingRequest): leave it; mark quiescent = false.

Return value: true if all connections were closed.

Implications¶

• Idle connections close fast. The first iteration closes all of them.
• Active connections must finish naturally. No interruption.
• The polling interval is bounded by 500 ms. Even on the happy path (all connections idle), the check takes <1 ms but subsequent checks (if any) wait up to 500 ms. So an idle server can shut down in <5 ms.
• An active connection holds the entire shutdown until it transitions to idle.

Why polling?¶

Shutdown could have used per-connection channels for notification. Polling was chosen for simplicity. The cost is small (one check every 500 ms, against a small map) compared to typical handler durations.

Inside http.Server.Close¶

Close is much simpler than Shutdown:

func (srv *Server) Close() error {
    srv.inShutdown.Store(true)
    srv.mu.Lock()
    defer srv.mu.Unlock()
    err := srv.closeListenersLocked()

    // Unlock srv.mu while waiting for listenerGroup.
    // The group Add and Done calls are made with srv.mu held,
    // to avoid adding a new listener in the window between
    // us setting inShutdown above and waiting here.
    srv.mu.Unlock()
    srv.listenerGroup.Wait()
    srv.mu.Lock()

    for c := range srv.activeConn {
        c.rwc.Close()
        delete(srv.activeConn, c)
    }
    return err
}

Steps:

1. Set the shutdown flag.
2. Close listeners.
3. Wait for accept goroutines to return.
4. Close every active connection.

The last step is the difference: Close does not wait for handlers to finish. It calls rwc.Close() on every connection, which immediately interrupts read/write operations in the handler. The handler typically sees an io.EOF or *net.OpError on its next read/write.

What the handler sees¶

A handler in the middle of writing the response:

w.Write([]byte("..."))
// If Close was called, this returns *net.OpError.

Most handlers ignore write errors. The truncated response is sent (or not). Connection is closed at the TCP layer. Client sees a connection reset.

Why use Close after Shutdown failed?¶

After Shutdown returned context.DeadlineExceeded, some active connections are still hanging on. Close interrupts them. Without Close, the process cannot exit cleanly because the connection-handling goroutines are blocked.

RegisterOnShutdown vs Close¶

OnShutdown hooks run at the start of Shutdown. They are not called by Close. If you have hooks that need to run even on force-close, call them separately.

The net.Listener Lifecycle¶

The underlying mechanism for accepting connections.

net.Listener interface¶

type Listener interface {
    Accept() (Conn, error)
    Close() error
    Addr() Addr
}

Accept blocks until a new connection arrives. Close interrupts Accept (it returns an error indicating the listener is closed).

*net.TCPListener internals¶

On Linux, Accept calls the accept4(2) syscall. The kernel's TCP stack maintains a queue of completed handshakes; accept4 pops the next one.

Close calls close(2) on the listener's file descriptor. Subsequent accept4 calls fail with EINVAL (or sometimes EBADF). The Go runtime maps this to a sentinel error indicating "use of closed network connection."

File descriptor cleanup¶

The kernel reclaims the listener's fd. The TCP socket's accept queue is drained — incoming SYNs after the close are RST'd by the kernel.

Reuse-port and graceful restart¶

Linux's SO_REUSEPORT lets multiple processes bind to the same port. Used for hot-restart: the new process binds; the old drains its connections; the kernel load-balances between them.

Setting it in Go:

lc := net.ListenConfig{
    Control: func(network, address string, c syscall.RawConn) error {
        return c.Control(func(fd uintptr) {
            syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET,
                syscall.SO_REUSEPORT, 1)
        })
    },
}
ln, _ := lc.Listen(context.Background(), "tcp", ":8080")

Once bound with SO_REUSEPORT, another process can also bind; both receive new connections.

TLS listener¶

tls.NewListener(ln, tlsConfig) wraps a raw listener. Accept returns *tls.Conns. Close closes the underlying raw listener. The TLS handshake happens after Accept, in a per-connection goroutine.

Connection State Tracking in http.Server¶

http.Server tracks each connection's state in a small state machine.

States¶

type ConnState int

const (
    StateNew ConnState = iota
    StateActive
    StateIdle
    StateHijacked
    StateClosed
)

• StateNew: just accepted, no bytes read yet.
• StateActive: in the middle of reading or writing a request/response.
• StateIdle: in keep-alive, waiting for next request.
• StateHijacked: handler took over the connection (WebSocket).
• StateClosed: closed.

State transitions¶

[Accept] -> StateNew

[bytes received, request parsed] -> StateActive

[response sent, connection kept alive] -> StateIdle

[next request begins] -> StateActive

[connection closed] -> StateClosed

[handler hijacked] -> StateHijacked  (not tracked further)

Why Shutdown cares¶

Shutdown.closeIdleConns() iterates over activeConn:

• For each in StateIdle: close it.
• For each in StateActive: leave it.
• For each in StateHijacked: not in the map (removed on transition).

So hijacked connections are not tracked. WebSockets are not waited for.

ConnState hook¶

You can register a callback on every state transition:

srv.ConnState = func(c net.Conn, s http.ConnState) {
    log.Printf("conn %v: %v", c.RemoteAddr(), s)
}

Useful for diagnostics. Performance cost is small but real (the callback runs per request per connection).

SIGKILL Mechanics at the Kernel Level¶

What happens when kill -9 lands.

Inside the kernel¶

When the kernel receives SIGKILL for process P:

1. Set the SIGKILL flag. Even if signal masks block it, SIGKILL is unmaskable; the kernel proceeds.
2. Mark P for termination. The process exit state is set.
3. Schedule P to wake up. If P is sleeping in a syscall, the syscall is interrupted (EINTR).
4. Execute do_exit. The kernel's process-cleanup routine runs:
5. Close all file descriptors.
6. Release memory mappings.
7. Send SIGCHLD to the parent.
8. Remove from the process table.
9. Reap. When the parent calls wait, P is fully removed.

The whole thing takes microseconds.

What survives a SIGKILL¶

• File contents already written to disk. Synchronous writes (with fsync) are persisted.
• TCP packets already in the kernel send queue. Sent to the network.
• IPC objects. Pipes, shared memory — until the last user of each closes.

What does NOT survive¶

• Userspace buffers. Anything in bufio.Writer, in os.File write buffers (before fsync), in Go's log buffer — gone.
• In-flight HTTP responses. TCP layer sends RST.
• Database transactions. Not committed unless transaction was already committed.
• Goroutines. All gone.
• defers. None run.

Why this matters for graceful shutdown¶

If your shutdown plan relies on defer for cleanup, SIGKILL skips it. Graceful shutdown is the only way to ensure deferred cleanup runs.

OOM killer and SIGKILL¶

On OOM (out of memory), the kernel's OOM killer picks a process and sends SIGKILL. Your process has no warning; it's just gone.

Mitigation: configure oom_score_adj (some scripts give the application a higher score than infrastructure components, so the app is killed first instead of, say, kubelet). Set memory limits in K8s so OOM affects you, not your noisy neighbour.

Inside the container runtime¶

Container runtimes (containerd, CRI-O, Docker) wrap the application process in a cgroup. SIGKILL from the kubelet is delivered to the container's PID 1, which the runtime kills. If PID 1 is your Go binary, it's gone. If PID 1 is a shell wrapper, the shell is killed but the Go binary may continue briefly (depending on the wrapper's signal handling) until the kernel cleans up the cgroup.

Container Runtime Signal Delivery¶

How signals propagate from the kubelet to your Go process.

Layers¶

kubelet
  v
CRI (gRPC interface)
  v
container runtime (containerd, CRI-O)
  v
runc (OCI runtime)
  v
container init (PID 1 in the namespace)
  v
... container's process tree ...
  v
Go binary (somewhere in the tree)

The signal must traverse all of these layers.

kubelet's StopContainer¶

When kubelet decides to stop a container:

1. Calls CRI's StopContainer with a timeout.
2. The runtime translates this to "send SIGTERM, wait for exit, then SIGKILL if not exited."
3. The signal is delivered to the container's init (PID 1 in the container's PID namespace).

PID namespace¶

In a container, processes see a different PID space. The container's "PID 1" is your application (or a wrapper). From the host's perspective, this same process has a different PID.

Signal propagation to children¶

Signals to PID 1 in a namespace are NOT automatically forwarded to child processes. If your Go binary is PID 1, SIGTERM goes to it directly. If a shell wrapper is PID 1 and your Go binary is a child, the shell must forward.

Most shells do NOT forward SIGTERM to children. The classic problem:

CMD ["sh", "-c", "/app/server"]  # sh is PID 1; doesn't forward

The Go binary never sees SIGTERM. After the grace period, SIGKILL lands on the shell, which dies, which terminates the Go binary.

Fixes¶

ENTRYPOINT ["/app/server"]  # Go binary is PID 1

Or use a minimal init like tini:

ENTRYPOINT ["/sbin/tini", "--", "/app/server"]

Tini forwards signals to children.

dumb-init and tini¶

Both are small init systems for containers. They forward signals and reap zombies. K8s 1.17+ has shareProcessNamespace: true which uses a built-in pause container for similar functionality.

Kubernetes kubelet Internals¶

The pod-termination flow from kubelet's perspective.

kubelet's PodTerminator¶

When a pod is marked Terminating:

1. Get pod spec.
2. Run preStop hooks (if any).
3. SendStopSignal(SIGTERM) to all containers.
4. Start grace timer (terminationGracePeriodSeconds).
5. Watch for container exit.
6. On timer expiration, SendStopSignal(SIGKILL).
7. Wait for runtime to confirm container removed.
8. Cleanup pod resources.

preStop hooks¶

Implemented in kubelet's lifecycle package. For exec:

cmd := exec.Command("sh", "-c", spec.Exec.Command)
// ... run in the container namespace ...
err := cmd.Run()

For httpGet:

req, _ := http.NewRequest("GET", spec.HTTPGet.Path, nil)
// ... add headers, scheme, host:port ...
resp, err := client.Do(req)

Both have a default timeout of 30 seconds. If preStop exceeds the timeout, kubelet logs a warning and proceeds to SIGTERM.

Grace period budgeting¶

The grace timer starts after preStop completes. So total wall-clock from "begin terminating" to "SIGKILL" is approximately:

preStop duration + terminationGracePeriodSeconds

This is why you often see longer-than-expected pod terminations: preStop adds to the budget.

Container exit detection¶

kubelet polls the container runtime for state changes. When the runtime reports the container exited, kubelet records the exit code and cleans up.

The polling interval is short (sub-second). Process exit is typically detected within 100 ms.

Force-stop on grace expiration¶

If the timer expires, kubelet sends SIGKILL via the runtime. The kernel cleans up the process. kubelet then runs through the normal cleanup path.

kubelet exit codes¶

The pod's exit reason is exposed:

state:
  terminated:
    reason: Completed       # exit 0
    exitCode: 0

Or:

state:
  terminated:
    reason: Error           # non-zero exit
    exitCode: 137           # SIGKILL (128 + 9)

Exit code 137 is the universal "SIGKILLed" indicator. If you see it, your pod hit the grace timer.

metrics¶

kubelet emits metrics:

• kubelet_container_log_filesystem_used_bytes
• kubelet_running_pods
• kubelet_node_name

Plus container-level metrics that reveal slow shutdowns. The K8s ecosystem has dashboards for these.

PID 1 and Signal Forwarding¶

A common production gotcha.

Why PID 1 matters¶

In a container, PID 1 has special semantics in the kernel:

• Default signal handlers for SIGTERM, SIGINT, etc. are no-op (PID 1 is supposed to handle them explicitly).
• PID 1 is the parent of orphaned processes — when their actual parent dies, they reparent to PID 1.
• PID 1 must reap zombies, or the process table fills up.

Go binaries as PID 1¶

A Go binary as PID 1 must handle these responsibilities. Fortunately, the Go runtime does it:

• The runtime's signal handler intercepts SIGTERM/SIGINT etc. normally.
• Subprocess management (exec.Cmd) reaps subprocess exits via Wait.

Most Go services as PID 1 work correctly. The only common failure is if you spawn subprocesses and don't Wait for them — zombies accumulate.

Shell wrappers and PID 1¶

CMD ["sh", "-c", "/app/server"]

The shell is PID 1. The Go binary is PID 2 (or higher). SIGTERM goes to the shell. The shell's default action for SIGTERM as PID 1 is... nothing. The signal is ignored.

After grace period, SIGKILL lands on the shell. The kernel kills it. The Go binary, having lost its parent, reparents to... no one (since PID 1 is gone too). The kernel kills the whole container's process tree as part of cleanup.

The Go binary never had a chance to handle SIGTERM.

Tini and dumb-init¶

Both are minimal PID-1 inits that:

• Forward SIGTERM/SIGINT to all children.
• Reap zombies.

ENTRYPOINT ["/sbin/tini", "--", "/app/server"]

Now tini is PID 1. Tini receives SIGTERM, forwards to the Go binary. The Go binary handles it normally.

shareProcessNamespace¶

K8s 1.17+ has a pod-spec field shareProcessNamespace: true. All containers in the pod share a PID namespace. A pause container (k8s.gcr.io/pause) becomes PID 1. The pause container is a minimal init that reaps zombies.

With this set, your Go binary is NOT PID 1; the pause container is. Signal forwarding is automatic.

Detecting PID 1 in code¶

if os.Getpid() == 1 {
    log.Println("running as PID 1; signal handling should be tested")
}

Useful for diagnostic logging.

Go Runtime Internals During Shutdown¶

What does the runtime do during srv.Shutdown and after main returns?

main return and exit¶

When main returns:

1. The runtime calls runtime.exit(0).
2. The runtime invokes deferred functions (none at this point, all main's defers already ran).
3. The runtime calls os.Exit(0).

os.Exit(0):

1. Calls user-registered runtime/debug.SetPanicOnFault etc. handlers (rarely used).
2. Calls runtime.Goexit for the main goroutine? No — that would only exit the main goroutine.
3. Calls the C exit(0) syscall.

The kernel cleans up the process.

os.Exit vs runtime.Goexit¶

• os.Exit(n): immediate process termination. No defers run.
• runtime.Goexit(): terminates the current goroutine. Does run deferred functions in that goroutine.

If os.Exit is called from main, no defers anywhere run. If runtime.Goexit is called from main, main's defers run, but the process continues with other goroutines until they exit.

Use os.Exit only via log.Fatalf at the top of main, after all defers.

Goroutine cleanup at exit¶

When the process exits via os.Exit, all goroutines are summarily terminated. The runtime does not "unwind" their stacks. Any deferred functions in those goroutines do not run.

This is why critical cleanup (Sentry flush, log flush) should happen in main before os.Exit, not in deferred functions of other goroutines.

runtime.GC() at shutdown?¶

The runtime does NOT trigger GC at process exit. There is no reason to: the kernel reclaims all memory.

If you want resource cleanup that runs at exit, register it explicitly.

Finalizers¶

runtime.SetFinalizer(obj, fn) registers fn to run when obj is garbage-collected. Finalizers do NOT run at process exit. They run only when GC reclaims memory. At process exit, all objects are still alive (from the runtime's perspective); no GC happens.

Do not rely on finalizers for cleanup that needs to run at shutdown. Use explicit Close calls or defers.

Signal-handling thread at exit¶

The runtime's signal-handling thread terminates when the process exits. No special handling needed.

Performance Characteristics of Shutdown¶

Numbers help. Here are typical measurements.

signal.NotifyContext overhead¶

Constant: about 200 ns to set up, 50 ns per signal type registered. Goroutine spawn: ~1 µs.

Total: typically <2 µs at startup. Negligible.

Signal delivery latency¶

From kernel-marked-pending to user-channel-receive: typically 10–100 µs. Under load: up to several ms.

For graceful shutdown, this latency is irrelevant — the shutdown takes seconds, not microseconds.

srv.Shutdown on idle server¶

Steps:

• inShutdown.Store: ~50 ns.
• closeListenersLocked: ~10 µs (close one fd).
• listenerGroup.Wait: ~1 µs (no pending Accepts).
• closeIdleConns: O(n) for n idle connections. Closing a TCP fd is ~5 µs each.
• Polling loop: returns immediately after first iteration.

Total for 100 idle connections: ~500 µs. Sub-millisecond.

srv.Shutdown with active connections¶

Dominated by the polling loop. Each iteration: ~10 µs of overhead plus the sleep (1ms → 500ms exponentially).

For an 8-second drain (real-world handler latency), expect about 15 iterations: 1 + 2 + 4 + 8 + 16 + 32 + 64 + 128 + 256 + 500 + 500 + ... ms.

Total: 8 seconds + ~50 ms of polling overhead. The polling is not the bottleneck.

srv.Close overhead¶

Same as Shutdown minus the polling loop, plus the active-connection closure loop. For 100 active connections: ~500 µs.

Context cancellation propagation¶

cancel() is O(n) for n direct children. Each child's cancel is called recursively. For a tree of depth d and width w, total: O(w^d).

In typical services, the tree is shallow (depth 2–3) and narrow (width 10–100). Total: microseconds.

Sentry / OTLP flush¶

These are external network calls. Typical latency: 50–500 ms. The 2*time.Second flush timeout is comfortable.

Total shutdown latency budget¶

A clean shutdown of a service with no active connections: ~100 ms.

A shutdown with 10 active connections finishing in 2 seconds: ~2.5 seconds.

A shutdown with 100 active connections, p99 handler 8s: ~8.5 seconds.

A shutdown with 1000 active connections, p99 handler 8s: ~10 seconds.

Sublinear in active connection count because they drain in parallel.

Version-Specific Behaviour¶

Go's shutdown story has evolved.

Go 1.0–1.7¶

No http.Server.Shutdown. Third-party packages (manners, graceful) provided it.

Go 1.8 (Feb 2017)¶

http.Server.Shutdown added. Includes RegisterOnShutdown.

The first Shutdown did not properly handle HTTP/2. Patched in 1.8.1.

Go 1.9¶

Improvements to keep-alive handling. Idle connections close more reliably.

Go 1.11–1.13¶

Minor improvements. Shutdown honours BaseContext and ConnContext (added 1.13).

Go 1.14¶

Async preemption via SIGURG. The runtime now interrupts long-running goroutines without cooperation. Side effect: subscribing to SIGURG via signal.Notify interferes with preemption.

Go 1.16¶

signal.NotifyContext added. Major ergonomic improvement.

Go 1.20¶

context.WithCancelCause added. Useful for richer cancellation reasons.

Go 1.21¶

context.AfterFunc added. context.WithDeadlineCause added.

Go 1.22¶

Loop-variable scoping changed. Some old for i := range ...; go f(i) patterns work differently.

Go 1.23¶

Iterators (range func) added. Not directly shutdown-related but enable cleaner iteration patterns.

Go 1.24+¶

Future improvements to context, signals, and net/http likely.

Compatibility implications¶

A service that targets Go 1.14+ can use async preemption confidently. A service that targets 1.16+ can use signal.NotifyContext cleanly. A service that targets 1.20+ can use cancel-with-cause.

Most production services run 1.21+ today.

Reading the Standard Library Source¶

For the professional, reading the source is essential. A few files to know.

src/os/signal/signal.go¶

The user-facing signal.Notify, signal.Stop, signal.NotifyContext. About 350 lines. Clear, well-commented.

src/os/signal/signal_unix.go¶

UNIX-specific signal handling. The loop function is the goroutine that reads from signal_recv.

src/runtime/sigqueue.go¶

The internal signal queue. signal_recv, signal_enable, signal_disable. About 250 lines. Detailed comments on the bit-set implementation.

src/runtime/signal_unix.go¶

The runtime's signal handler. About 1000 lines. Dense, but readable. Shows how the runtime intercepts signals and dispatches them.

src/net/http/server.go¶

The HTTP server implementation. Shutdown, Close, Serve, ListenAndServe. About 4000 lines. The Shutdown method is around line 2400. Read it; it is well-commented.

src/net/net.go, src/net/fd_unix.go¶

Listener and connection internals. Close semantics for fd's.

src/context/context.go¶

Context implementation. WithCancel, WithTimeout, propagation. About 800 lines.

Recommendation¶

Pick one file. Spend an hour reading it. Take notes. The Go standard library is some of the most polished open-source code; you will learn idioms by reading.

Performance Profiling of Shutdown¶

For the professional engineer concerned with shutdown performance:

Profiling tools¶

• pprof (CPU, memory, goroutines, blocking).
• runtime.Trace (Go's execution tracer; visualises scheduling).
• dlv (Go debugger; can inspect goroutine states during shutdown).
• perf (Linux; for kernel-side profiling).
• bpftrace (eBPF; for kernel events including signal delivery).

A pprof workflow for slow shutdown¶

If shutdown is slow:

1. Trigger shutdown.
2. Before it completes, capture: curl http://host:port/debug/pprof/goroutine?debug=2.
3. Examine: which goroutines are blocked? On what?

The goroutine dump reveals stuck goroutines. Most are usually:

• HTTP handlers waiting on downstream.
• Workers in a blocking syscall.
• The Shutdown's polling loop itself.

The remaining are the bug.

A trace for shutdown timeline¶

import _ "net/http/pprof"
import "runtime/trace"

// at shutdown start:
f, _ := os.Create("/tmp/shutdown.trace")
trace.Start(f)
defer trace.Stop()
defer f.Close()

The trace shows every scheduling decision. Open with go tool trace. Useful for understanding contention at shutdown.

Counting handler durations during shutdown¶

Instrument the middleware:

defer func(t time.Time, p string) {
    metric.HandlerDurationDuringShutdown.WithLabelValues(p).Observe(time.Since(t).Seconds())
}(time.Now(), r.URL.Path)

After production for a week, the histograms reveal the slow paths.

Kernel-side profiling¶

For extremely deep diagnosis:

bpftrace -e 'kprobe:do_signal { @[comm, pid] = count(); }'

Counts signal-handling kprobe hits per process. Useful when investigating "signal not delivered" mysteries.

Edge Cases at the Lowest Level¶

A grab bag of edge cases the professional engineer encounters.

Edge case: signal during select with no ctx.Done case¶

select {
case msg := <-ch:
case <-time.After(time.Second):
}

A signal arriving during this select does NOT interrupt it. The runtime queues the signal for delivery via the signal channel; user-space select is unaffected. To make select responsive to shutdown, include <-ctx.Done().

Edge case: blocking syscall during shutdown¶

A goroutine in a blocking syscall (e.g., a sync read from a slow file) cannot be interrupted by context cancellation. The syscall blocks indefinitely until it completes or the file descriptor is closed.

For network reads, closing the connection's fd makes the syscall return EINTR or io.EOF. For disk reads, you usually cannot interrupt.

Workaround: do not perform blocking disk I/O without async wrappers.

Edge case: time.Sleep and signals¶

time.Sleep(d) blocks in a way that does NOT respond to signals. A signal arrives, the runtime handles it, but time.Sleep keeps sleeping.

For shutdown-aware sleep:

select {
case <-time.After(d):
case <-ctx.Done():
}

Edge case: zero-byte Read returning before signal¶

A Read returning 0 bytes (without error) means EOF or nothing yet. Some implementations may return immediately after a signal interrupts. This is implementation-defined and can confuse handlers.

Edge case: socket linger¶

A TCP socket has a SO_LINGER option. With linger enabled, close blocks until queued data is sent. During shutdown, this can prolong Close calls.

tcpConn, _ := conn.(*net.TCPConn)
tcpConn.SetLinger(0) // immediate close, drop unsent data

Most servers don't need to mess with linger.

Edge case: TIME_WAIT after shutdown¶

After your server closes a connection, the TCP state on the server side moves to TIME_WAIT for 60 seconds (Linux default). The port may not be immediately reusable. Set SO_REUSEADDR:

lc := net.ListenConfig{
    Control: func(_, _ string, c syscall.RawConn) error {
        return c.Control(func(fd uintptr) {
            syscall.SetsockoptInt(int(fd), syscall.SOL_SOCKET,
                syscall.SO_REUSEADDR, 1)
        })
    },
}

The Go runtime sets this by default for HTTP servers, but for raw net.Listen you may need to do it manually.

Edge case: file descriptor exhaustion¶

A shutdown that accumulates file descriptors (a leak) can hit EMFILE ("too many open files"). The shutdown itself becomes slow because each new socket fails.

Diagnose: lsof -p <pid>. Solve: find the leak.

Edge case: ulimit and shutdown¶

The process's ulimit -n (max open files) caps how many connections it can serve. During shutdown, connections being closed don't count toward the limit; idle ones do.

Edge case: connection close in TLS¶

A TLS connection has a graceful close (close-notify alert). Without it, the peer sees an unexpected close. tls.Conn.Close() sends close-notify; raw *net.TCPConn.Close() does not.

When http.Server.Close is called, it uses conn.Close() which for TLS connections invokes the proper close-notify. So this is usually correct.

Edge case: forks during shutdown¶

If your service spawns subprocesses during normal operation, what about during shutdown? Don't. Shutdown is a "no new work" phase; no new processes should be created.

If you accidentally spawn during shutdown, the child may inherit closed file descriptors and behave unexpectedly.

Edge case: shutdown during init¶

If signal.NotifyContext is called before dependent initialisation (DB, Redis), and a signal arrives during init, the init may fail with context.Canceled. This is desirable behaviour — init bails out cleanly.

But if init's error handling is wrong, the program may keep running with a partially-initialised state. Always check ctx.Err() in init and propagate cancellation up.

Edge case: deferred mutex unlocks during panic¶

mu.Lock()
defer mu.Unlock()
panic("oops")

Deferred Unlock runs before the panic propagates. The mutex is correctly released. Subsequent goroutines can acquire it.

But if the panic is during shutdown, the program may exit before subsequent goroutines run. The mutex's state doesn't matter at that point.

Edge case: runtime.LockOSThread and signals¶

A goroutine pinned to an OS thread via runtime.LockOSThread has its own signal mask considerations. Most user code doesn't need this; CGo callers may.

Conclusion¶

The professional level treats graceful shutdown as a mechanism to understand, not just an architecture to apply. The kernel's signal delivery, the runtime's interception, the standard library's implementation, the container runtime's choreography, the kubelet's algorithm — each is a layer with its own internals.

Knowing them is the difference between:

• "The service won't shut down cleanly; let me file a ticket."
• "The service has a SIGTERM-not-delivered issue because PID 1 is the shell. Add tini or change CMD to ENTRYPOINT."

The first is a junior diagnosis. The second is a professional one.

You have now read four levels of graceful shutdown content. The junior file gave you the recipe. The middle file gave you the system. The senior file gave you the architecture. This professional file gave you the mechanism.

A reader who has internalised all four pages can:

• Write graceful shutdown in any Go service from scratch.
• Design shutdown for any system architecture.
• Debug shutdown issues at any layer.
• Mentor others through the patterns.

That is the bar. Onwards to the specifications, interview questions, hands-on tasks, find-the-bug exercises, and optimization challenges in the remaining files. Mastery is built through repetition and practice.

Appendix: A Reading Trail Through the Source¶

For the deeply curious, a recommended order to read the relevant Go source files:

1. src/os/signal/signal.go (350 lines) — Understand Notify, Stop, NotifyContext.
2. src/os/signal/signal_unix.go (~100 lines) — The signal-receive loop.
3. src/runtime/sigqueue.go (250 lines) — The internal signal queue.
4. src/runtime/signal_unix.go (~1000 lines, skim) — Runtime's signal handler.
5. src/context/context.go (800 lines) — WithCancel, WithTimeout.
6. src/net/net.go and src/net/fd_unix.go (skim) — Listener and conn internals.
7. src/net/http/server.go (4000 lines, focus on Shutdown and Close) — HTTP server.

A weekend spent on this list makes you a de facto expert. The Go source is some of the cleanest production code in any language.

Appendix: A Glossary of Kernel-Level Terms¶

Terms that come up in this file:

A working knowledge of these terms is the prerequisite for kernel-level debugging.

Appendix: A One-Page Mental Model of the Whole Stack¶

+---------------------------------------+
|        Customer's browser             |
|        (sees 503 or success)          |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     Cloud LB (ALB, GCP LB, etc.)      |
|     Health check, endpoint routing    |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     K8s kube-proxy (iptables/IPVS)    |
|     Service-to-pod routing            |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     Pod's container network (CNI)      |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     Container runtime (containerd)     |
|     PID namespace, cgroup              |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     Container's init (PID 1)           |
|     Signal forwarding (or not)         |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     Go binary                          |
|     Runtime signal handler             |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     os/signal package                  |
|     Channel-based delivery             |
+----------------|----------------------+
                 |
+----------------v----------------------+
|     Application code                   |
|     signal.NotifyContext, srv.Shutdown |
+---------------------------------------+

Every layer above your application code is doing work for you. Understanding what each layer does — and what it does NOT do — is the professional's superpower.

When something goes wrong at shutdown, the problem is in one of these layers. The professional engineer diagnoses methodically, layer by layer, until they find it.

Appendix: How to Become a Professional-Level Engineer on This Topic¶

A condensed plan:

1. Read this file twice. Slowly. Take notes.
2. Read the source files in the trail above. One per evening.
3. Run experiments. Modify standard library; observe behaviour. Use go install for local builds.
4. Use strace to watch signals in real-time:

   strace -p $(pidof yourbinary) -e signal
   

5. Use bpftrace to watch kernel events.
6. Read incident postmortems from cloud providers; many involve shutdown.
7. Contribute to open source. Go itself, popular libraries.
8. Write a small replacement. Implement a tiny Shutdown-like in your own package.
9. Mentor others. Teaching reveals gaps.
10. Stay current. Read Go release notes; track issues in golang/go.

A year of this turns "I read Mike Lin's article" into "I know the runtime well."

Final Words for the Professional Reader¶

The professional layer is rarefied. Most teams don't need this depth. Some do.

The skills here apply far beyond graceful shutdown:

• Kernel signal handling is also relevant for process management, debugging, profiling.
• Go runtime knowledge transfers to performance tuning generally.
• Container runtime understanding helps in capacity planning and SRE work.
• The "read the source" discipline applies to every problem.

A professional engineer who masters this file has the toolkit to solve any low-level issue in a Go service. Shutdown is just where this knowledge first becomes load-bearing.

The graceful shutdown topic is complete. The remaining files (specification, interview, tasks, find-bug, optimize) are reference and practice material.

Carry the knowledge forward.

Extended Topic: Detailed Walkthrough of Signal Path¶

For the truly curious, an annotated tour of what happens when SIGTERM arrives at a Go process.

Time t=0: The kernel decides to deliver¶

The kubelet has invoked kill(pid, 15) via its runtime layer. The kernel's kill syscall implementation:

// simplified
SYSCALL_DEFINE2(kill, pid_t, pid, int, sig) {
    return do_send_signal(pid, sig);
}

do_send_signal looks up the target process, checks permissions (sender must have same UID or be root), then dispatches.

Time t=0+ε: Setting pending bit¶

In the target's task_struct, the SIGTERM bit (15) is set in pending.signal. If the process has a signal mask blocking SIGTERM (rare), it stays pending until unblocked.

Time t=0+δ: Wake the target¶

If the target is sleeping in a syscall (e.g., read from a TCP socket), the kernel marks the task as runnable and the wait queue's wake mechanism kicks in. The target thread becomes runnable.

The runtime is typically scheduled within microseconds.

Time t=tiny: Return to userspace¶

The kernel returns from the syscall (or from a timer-interrupt-induced context switch). Before resuming userspace, the kernel checks for pending signals.

For each pending signal, the kernel looks up the action:

• Default: take default action (terminate, ignore, etc.).
• Ignore: do nothing.
• User handler: invoke it.

For Go, every catchable signal has a user handler installed by the runtime.

Time t=tiny+: Runtime signal handler¶

The kernel arranges to call runtime.sigtramp (assembly) which sets up the Go call stack and calls runtime.sighandler.

runtime.sighandler does:

1. Save the interrupted goroutine's state.
2. Identify the signal.
3. Determine if it's synchronous (caused by current goroutine) or asynchronous.
4. For async signals: forward to user-space via signal_recv queue.
5. Restore the interrupted goroutine's state.
6. Return to kernel; kernel resumes the interrupted code.

The actual handler executes in ~microseconds, on the same thread that was interrupted.

Time t=tiny+more: signal_recv wakes¶

The runtime's signal_recv goroutine is parked on a futex. The signal-enqueue path wakes it.

signal_recv reads the signal number from the bit-set, returns it. The caller (os/signal.loop) calls process(sig).

Time t=tiny++: Channel send¶

process(sig) iterates handlers, does non-blocking sends. For our signal.NotifyContext channel, the channel has cap 1 and is empty. The send succeeds.

Time t=tiny+++: User goroutine awakes¶

The signal.NotifyContext's internal goroutine is parked on <-c.ch. The send wakes it. It calls c.cancel().

Time t=tiny++++: Context propagation¶

c.cancel() walks the context tree. Each derived context's Done() channel is closed. All goroutines waiting on those channels wake.

Time t=tiny+++++: User code reacts¶

main's <-rootCtx.Done() returns. main proceeds to call srv.Shutdown(ctx).

Total latency¶

From kernel signal-pending bit set to main's <-rootCtx.Done() returning: typically 50–500 µs. Mostly scheduler latency.

This is fast enough that the latency is invisible in shutdown logs. The shutdown delay is dominated by what comes next (draining handlers), not signal-handling latency.

Extended Topic: How signal.Notify Tracks Subscribers¶

The handlers global:

var handlers struct {
    sync.Mutex
    m   map[chan<- os.Signal]*handler
    ref map[uint32]int64
}

• m maps channels to handler structs.
• ref counts how many handlers are subscribed to each signal.

The handler struct:

type handler struct {
    mask [(numSig + 31) / 32]uint32
}

func (h *handler) want(sig int) bool {
    return (h.mask[sig/32]>>(uint(sig)&31))&1 != 0
}

func (h *handler) set(sig int) {
    h.mask[sig/32] |= 1 << (uint(sig) & 31)
}

Each handler has a bit-set of "wanted" signals. Compact and fast.

Multiple Notify calls¶

If you call Notify twice for the same channel and different signals:

ch := make(chan os.Signal, 1)
signal.Notify(ch, syscall.SIGTERM)
signal.Notify(ch, syscall.SIGINT)

Both calls find the same handler struct in m[ch]. Each sets the corresponding bit in the mask. The ref for each signal is incremented.

After this, ch receives both SIGTERM and SIGINT. Equivalent to:

signal.Notify(ch, syscall.SIGTERM, syscall.SIGINT)

Stop with multiple subscriptions¶

If two channels subscribe to SIGTERM and one calls Stop:

ch1 := make(chan os.Signal, 1)
ch2 := make(chan os.Signal, 1)
signal.Notify(ch1, syscall.SIGTERM)
signal.Notify(ch2, syscall.SIGTERM)
// ref[SIGTERM] = 2

signal.Stop(ch1)
// ref[SIGTERM] = 1 (still > 0; signal still delivered to ch2)

disableSignal(SIGTERM) is NOT called until ref drops to 0. Good design — multiple subscribers don't interfere.

What happens if you Notify then Stop then Notify again¶

ch := make(chan os.Signal, 1)
signal.Notify(ch, syscall.SIGTERM)  // ref=1, enabled
signal.Stop(ch)                      // ref=0, disabled
signal.Notify(ch, syscall.SIGTERM)  // ref=1, enabled again

Standard library handles this cleanly. The handler struct may or may not be recreated; either way, subscription is correct.

Race conditions¶

signal.Notify and signal.Stop use handlers.Mutex. They are race-free.

The internal process function also uses this mutex. During process, Notify/Stop are blocked briefly. Brief contention; no practical issue.

Extended Topic: The HTTP Server's Connection Tracking Loop¶

The full lifecycle of a request from http.Server's perspective.

Accept¶

func (srv *Server) Serve(l net.Listener) error {
    // ... boilerplate ...
    for {
        rw, err := l.Accept()
        if err != nil {
            select {
            case <-srv.getDoneChan():
                return ErrServerClosed
            default:
            }
            // ... handle error ...
        }
        connCtx := ctx
        if cc := srv.ConnContext; cc != nil {
            connCtx = cc(connCtx, rw)
        }
        c := srv.newConn(rw)
        c.setState(c.rwc, StateNew, runHooks)
        go c.serve(connCtx)
    }
}

Each accepted connection becomes a *conn struct. setState(StateNew) adds it to srv.activeConn. A goroutine handles it.

c.serve¶

func (c *conn) serve(ctx context.Context) {
    // ... handshake, panic recover, etc. ...
    for {
        w, err := c.readRequest(ctx)
        if c.r.remain != c.server.initialReadLimitSize() {
            c.setState(c.rwc, StateActive, runHooks)
        }
        if err != nil {
            // ... handle errors, close connection ...
            return
        }
        c.curReq.Store(w)
        // serve the request
        serverHandler{c.server}.ServeHTTP(w, w.req)
        // ... cleanup ...
        if !w.shouldReuseConnection() {
            // ... close ...
            return
        }
        c.setState(c.rwc, StateIdle, runHooks)
        // ... wait for next request ...
    }
}

The loop: read request, mark active, serve, mark idle, wait for next request. State transitions are tracked.

State on Shutdown¶

When Shutdown is called, it scans activeConn. Connections in StateIdle are closed immediately. Connections in StateActive continue serving; their state will transition to StateIdle after the handler returns. The polling loop catches them then.

Hijack¶

If the handler calls w.(http.Hijacker).Hijack(), the connection is removed from activeConn and the handler takes over the raw conn. Shutdown is no longer aware of it.

BaseContext and ConnContext¶

srv.BaseContext = func(_ net.Listener) context.Context {
    return rootCtx
}
srv.ConnContext = func(ctx context.Context, c net.Conn) context.Context {
    return ctx
}

BaseContext provides the context for the listener (every connection inherits). ConnContext derives a per-connection context.

These were added in Go 1.13 to make http.Server shutdown-aware. By passing rootCtx as the base, every connection's context is cancelled when rootCtx is cancelled — which means every handler's r.Context() is also cancelled.

This is a powerful pattern often overlooked. Setting BaseContext to rootCtx makes ALL handlers shutdown-aware automatically:

srv := &http.Server{
    BaseContext: func(_ net.Listener) context.Context {
        return rootCtx
    },
}

After this, no need for serviceCtx-linking middleware. The handlers' contexts are already linked.

Effect on Shutdown¶

Shutdown does NOT cancel BaseContext for active connections. The handlers' contexts remain alive until the connections close (or Close is called).

But because BaseContext derives from rootCtx, and rootCtx is already cancelled when shutdown begins, the handlers' contexts are already cancelled. Handlers that respect r.Context() exit promptly.

This is a major optimization for shutdown latency. Without it, handlers would only know to exit when their stream-context fires, which may take a while.

Extended Topic: The defer Mechanism and Shutdown¶

A subtle aspect of defer that matters during shutdown.

How defer works¶

When defer fn() is executed, the function and its arguments are evaluated, then a record is pushed onto the goroutine's defer chain. When the function returns (or panics), records pop and execute in LIFO order.

defer cost¶

Pre-Go 1.13, each defer was a heap allocation. Now it's stack-allocated in most cases (small fast path). Cost: ~50 ns per defer for the fast path; ~1 µs for heap-allocated.

defer and os.Exit¶

os.Exit calls the C exit(0) syscall directly. It does NOT unwind the Go call stack. No defers run.

This is the most common shutdown-related defer gotcha.

defer and panic¶

A panic does unwind the stack. Defers run in reverse order until a recover catches the panic (or the goroutine itself terminates).

A panic in shutdown code propagates through main's defers. If main's deferred function is something like db.Close(), it might not run (depending on what panic'd).

Defensive pattern:

func main() {
    defer func() {
        if r := recover(); r != nil {
            log.Printf("panic at shutdown: %v", r)
        }
    }()
    // ... rest of main ...
}

The recover at the top of main lets shutdown defers complete even if something panics.

defer performance under shutdown¶

If your shutdown registers many defers, they all run at main's return. For 100 defers, total cost ~5 µs. Negligible.

The slow part of shutdown is not defers; it's I/O.

runtime.Goexit and defers¶

runtime.Goexit() terminates the current goroutine, running its defers. It does NOT affect other goroutines.

Useful for "exit this goroutine cleanly without panicking." Rarely used in shutdown.

Extended Topic: Memory Behaviour During Shutdown¶

How does memory usage change during shutdown?

Allocations¶

Shutdown itself allocates minimally — a few timer structs, a few small slices. Negligible.

Handlers in flight may continue allocating until they return. If GC is needed, it happens normally.

Goroutine stack reclamation¶

When a goroutine exits, its stack is reclaimed by the runtime. This happens at exit, not at GC.

If your shutdown waits for many goroutines to exit, you'll see live-stack memory dropping as they finish.

GC during shutdown¶

The Go GC runs concurrently. It doesn't stop because of shutdown. If shutdown is slow because of contention, GC may run more often.

For shutdown latency analysis, GC is usually not the bottleneck.

Final GC at exit?¶

The runtime does NOT run a final GC at process exit. There's no point — the kernel reclaims all memory.

If you have finalizers that need to run, they don't run at exit. Use explicit cleanup.

Heap snapshots during shutdown¶

For diagnosis, capture a heap snapshot during a slow shutdown:

curl http://host:port/debug/pprof/heap > heap.prof
go tool pprof heap.prof

The snapshot shows what's allocated. Slow shutdowns sometimes reveal leaked goroutines or large pending buffers.

Extended Topic: Container Runtime Deep Dive¶

The path from kubelet to your process.

containerd's signal-sending¶

containerd is the dominant container runtime in modern K8s. When kubelet asks it to stop a container:

1. containerd's gRPC API receives StopContainer(timeout).
2. containerd queries the container's state.
3. containerd invokes runc kill <container-id> TERM (or via its task layer).
4. runc reads the cgroup info and uses kill(pid, SIGTERM) on the container init process.
5. containerd waits up to timeout for the container to exit.
6. If timeout elapses, containerd invokes runc kill <container-id> KILL.

Why runc kill is special¶

runc kill knows the container's PID 1 (in the host's PID space) from its container state. It sends the signal directly. There's no shell or wrapper involved.

Differences with Docker¶

Docker (with its old daemon) had its own signal-sending path. Behaviour was largely the same. Modern Docker (with containerd backend) just uses containerd directly.

Differences with podman¶

Podman is a runtime alternative. Signal sending is similar; uses runc underneath.

Differences with gVisor and Kata¶

Sandboxed runtimes like gVisor (Google) and Kata (lightweight VMs) intercept syscalls. Signal handling can have small additional latency. Behaviour from the application's perspective is the same.

Extended Topic: K8s Pod Lifecycle Edge Cases¶

A few non-obvious K8s behaviours.

"Phantom" Terminating pods¶

A pod stuck in Terminating state for a long time often means the application is not exiting. kubelet has sent SIGTERM and is waiting. If the timer fires, it sends SIGKILL.

If even after SIGKILL the pod is stuck, the issue is at the container runtime level. Sometimes restarting the kubelet or rebooting the node helps.

ForceDelete (--force --grace-period=0)¶

kubectl delete pod --force --grace-period=0 skips the grace period and immediately removes the pod object from the API server. The kubelet may or may not have terminated the container yet.

This is dangerous: the actual process may still be running, holding resources. Use sparingly.

Pre-emptible / spot nodes¶

Cloud-provider spot nodes can be reclaimed with ~30 seconds notice. K8s gets a signal; the kubelet drains pods. Your terminationGracePeriodSeconds must fit in the spot-eviction window.

For services on spot nodes, terminationGracePeriodSeconds: 25 is a safe default.

Node draining (kubectl drain)¶

kubectl drain <node> evicts all pods from a node (for maintenance). Each pod goes through the normal termination lifecycle. PodDisruptionBudgets are respected.

kubectl rollout restart¶

Restarts all pods in a Deployment without changing the spec. Each pod is recreated; the old goes through termination, the new starts. Useful for picking up new configs or recovering from transient state.

StatefulSet termination¶

StatefulSets terminate in reverse-ordinal order (largest index first). Each pod's termination follows the normal flow.

Job and CronJob¶

Job pods terminate when the work completes. They typically don't need graceful shutdown for in-flight work (they have no in-flight work after completion). But they may need to flush metrics, traces, logs.

CronJob is just a Job scheduler. Same applies.

Extended Topic: HTTP/2 Specifics During Shutdown¶

HTTP/2 has its own quirks for shutdown.

GOAWAY frame¶

When http.Server.Shutdown is called, the HTTP/2 layer sends a GOAWAY frame to each HTTP/2 client. The frame says: "do not open new streams on this connection."

The client may finish existing streams but should not start new ones. After the client's last stream finishes, the connection is closed.

Client behaviour¶

A well-behaved client (browsers, Go's http.Client) respects GOAWAY and opens a new connection (perhaps to a different backend) for new requests.

A misbehaving client might ignore GOAWAY and keep trying. The server eventually times out and closes the connection.

Trailers and GOAWAY¶

Some HTTP/2 servers send trailers (final response headers after the body). During shutdown, in-flight streams must complete normally including trailers.

http.Server.Shutdown handles this.

Push streams¶

HTTP/2 server push is rarely used. If used, push streams are in-flight and waited for like any other stream.

HTTP/3 (QUIC) considerations¶

HTTP/3 over QUIC has its own connection lifecycle. The pattern is similar — send a "no new streams" signal, drain existing — but the implementation differs.

Go's standard library does not yet have HTTP/3 support (as of late 2025). Third-party libraries (quic-go) exist. The shutdown pattern transfers.

Extended Topic: TLS Handshake During Shutdown¶

A subtle interaction.

Shutdown and in-progress handshake¶

A TCP connection accepted but mid-handshake is in StateNew. Its state will transition to StateActive (or to closed if handshake fails).

If Shutdown runs while the handshake is in progress, the connection counts as StateNew. The 5-second rule in closeIdleConns will eventually close it. But until then, it counts as active.

For fast shutdown, ReadHeaderTimeout is your friend — it bounds the time a connection can sit in StateNew.

Client certificate verification¶

If you require client certificates, the handshake includes a verification step that may involve fetching CRLs or OCSP responses. If those network calls are slow, the handshake can take seconds.

For fast shutdown, OCSP stapling (server pre-fetches OCSP response) avoids the per-handshake fetch.

Session resumption¶

TLS 1.3 session tickets allow resumption. Resumed handshakes are faster. Shutdown of a server invalidates outstanding tickets (eventually), so clients fall back to full handshake on reconnect.

This is rarely a problem unless you're serving extremely high RPS.

Extended Topic: NetworkPolicy and Shutdown¶

K8s NetworkPolicies can affect shutdown:

• A policy that blocks egress prevents Shutdown from flushing traces / Sentry events / Kafka producer.
• A policy applied mid-deploy can cause the pod's drain to fail (outbound calls blocked).

When designing NetworkPolicies, allow egress to observability backends.

Common policy mistake¶

egress:
- to:
  - podSelector:
      matchLabels:
        app: api-service
  ports:
  - protocol: TCP
    port: 8080

This allows egress to a specific service. But traces / metrics are typically sent to different destinations. Add allowances:

- to:
  - namespaceSelector:
      matchLabels:
        name: observability
  ports:
  - protocol: TCP
    port: 443

Otherwise, your pod's traces are silently dropped at shutdown.

Extended Topic: Service Mesh Sidecars and Shutdown¶

A common architecture: each pod has the application container and an Envoy sidecar.

The drain coordination problem¶

Both containers receive SIGTERM. If the Envoy sidecar exits first, the application loses its egress (Envoy proxies outbound calls in many mesh configurations). If the application exits first, in-flight requests may fail.

Istio's holdApplicationUntilProxyStarts¶

This pod-spec annotation makes the application container wait for the sidecar to be ready before it starts. It also reverses the order at shutdown: Envoy exits last.

annotations:
  proxy.istio.io/config: |
    holdApplicationUntilProxyStarts: true

This is the recommended setting for most Istio installations.

The Envoy drain_time_s¶

Envoy's drain time configures how long it waits for in-flight requests after receiving its drain signal. Tune to match your application's shutdown budget.

annotations:
  proxy.istio.io/config: |
    terminationDrainDuration: 25s

Sidecar lifecycle hooks¶

- name: istio-proxy
  lifecycle:
    preStop:
      exec:
        command: ["pilot-agent", "request", "POST", "quitquitquit"]

This explicit preStop for Envoy tells it to start draining. The application's preStop can sleep to let Envoy notice.

Extended Topic: Drain Coordination with Service Discovery¶

In dynamic environments, services discover each other via a registry (Consul, etcd, K8s Service).

Deregistration on shutdown¶

A service registers itself when starting; should deregister when stopping. Most service registries support this:

// at start
if err := registry.Register(myService); err != nil { return err }

// at shutdown
if err := registry.Deregister(myService); err != nil {
    log.Printf("deregister failed: %v", err)
}

Deregistration is a phase in the shutdown machine — after readiness flip, before drain.

Stale registrations¶

If a service crashes without deregistering, its entry stays in the registry until the next health check fails. For Consul: ~30 seconds. For K8s Service: depends on probe configuration.

Stale registrations cause clients to try-and-fail to reach a dead pod. Mitigate via:

• Always deregister on shutdown (handles graceful cases).
• Short health-check intervals (handles crashes).
• TTL-based registration that auto-expires.

Coordinated multi-service drain¶

If service A depends on service B, and both are draining, the order matters. A should drain before B (so A doesn't need B anymore). This requires explicit coordination — often via a deploy orchestrator.

Extended Topic: BPF Tracing of Shutdown¶

For ultra-low-level debugging, eBPF tools are powerful.

Tracing signal delivery¶

sudo bpftrace -e '
    tracepoint:signal:signal_deliver {
        printf("PID %d got signal %d at %lld\n",
            pid, args->sig, nsecs);
    }
'

Real-time output of every signal delivered to every process. Filter by pid:

sudo bpftrace -e '
    tracepoint:signal:signal_deliver /pid == 12345/ {
        printf("signal %d\n", args->sig);
    }
'

Tracing syscall durations¶

If shutdown is slow, eBPF can show which syscall is slow:

sudo bpftrace -e '
    tracepoint:syscalls:sys_enter_close {
        @start[tid] = nsecs;
    }
    tracepoint:syscalls:sys_exit_close /@start[tid]/ {
        @duration = hist((nsecs - @start[tid]) / 1000);
        delete(@start[tid]);
    }
'

Shows histogram of close syscall durations in microseconds.

Tracing Go goroutines¶

eBPF can hook into Go runtime functions. The golang.org/x/exp/trace package gives a higher-level view.

When to reach for eBPF¶

When standard tools (pprof, traces, logs) are insufficient. Typically:

• Investigating signals from outside the application.
• Diagnosing kernel-level latency.
• Understanding interactions with the container runtime.

A senior engineer might use eBPF once a year; a kernel-adjacent professional uses it monthly.

Extended Topic: Goroutine Leak Detection at Shutdown¶

A common bug: goroutines that don't exit at shutdown.

Detection in production¶

After shutdown, examine the goroutine count:

goroutinesBefore := runtime.NumGoroutine()
_ = mgr.Run(ctx, 30*time.Second)
goroutinesAfter := runtime.NumGoroutine()
if goroutinesAfter > expectedRemaining {
    log.Printf("leaked goroutines: %d", goroutinesAfter)
    buf := make([]byte, 1<<20)
    n := runtime.Stack(buf, true)
    log.Printf("stacks:\n%s", buf[:n])
}

The stack dump shows which goroutines are still running.

Detection in tests¶

go.uber.org/goleak is a popular library:

import "go.uber.org/goleak"

func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)
}

After every test, goleak checks for unexpected goroutines. The test fails if any leaked.

Common leak patterns¶

• Goroutine that watches a channel but never observes ctx.Done().
• Goroutine started by a library that doesn't accept cancellation.
• Goroutine in a for { select { case x := <-ch: ... } } with no Done case.

Each is fixed by adding a <-ctx.Done() case.

Library-imposed leaks¶

Some third-party libraries spawn goroutines that don't honour cancellation. The library may have a Close method that releases them:

defer thirdPartyClient.Close()

Or you may need to fork the library and add cancellation. File an issue upstream.

Extended Topic: A Tour of Process Termination on Linux¶

For the professional, a complete trip through do_exit.

When a process exits (whether via syscall exit() or via signal):

1. do_exit(code) is called.
2. Profile / accounting. Final stats recorded.
3. exit_mm. Memory mappings released.
4. exit_files. File descriptors closed (including listeners, sockets).
5. exit_fs. Working directory etc. released.
6. exit_signals. Signal-related cleanup.
7. exit_notify. Parent notified via SIGCHLD.
8. task_struct becomes a zombie. Waits for parent's wait.
9. wait reaps. Final cleanup of task_struct.

Steps 4 and 5 are where TCP sockets close, listeners close, etc. The kernel handles this regardless of whether the process exits cleanly or via SIGKILL.

The application's job (in graceful shutdown) is to handle anything the kernel doesn't: flushing user-space buffers, committing transactions, releasing distributed locks.

Extended Topic: The Future of Graceful Shutdown in Go¶

Speculative but informed.

Likely improvements¶

• Better tooling for shutdown observability (e.g., a runtime/shutdown package).
• Standardisation of phase-machine patterns into a golang.org/x/... package.
• More integration with K8s lifecycle hooks (e.g., a Go-native preStop library).
• Improved HTTP/3 support and corresponding drain mechanics.

Unlikely improvements¶

• Built-in graceful shutdown for every server type. Go's design philosophy is to provide primitives.
• Automatic signal-to-context bridging without signal.NotifyContext. The current API is already minimal.

What stays stable¶

• The signal handling API.
• The HTTP Shutdown / Close contract.
• The context cancellation model.

A skill investment in current shutdown patterns has a long shelf life.

Extended Topic: Shutdown in WebAssembly Runtimes¶

For completeness: Go-on-WASM doesn't have signals. The WASM runtime (browser, wasmtime, etc.) controls the lifecycle.

If your Go code targets WASM, graceful shutdown is the runtime's concern, not yours. The patterns in this Roadmap don't apply.

WASM is increasingly used for plugins, edge computing, and serverless. Most "graceful shutdown" engineering is for traditional Go servers.

Extended Topic: Shutdown in Embedded / IoT Go¶

Go compiles to many platforms including some embedded targets. Embedded environments often lack signals or have constrained signal handling.

If your Go binary runs on a bare-metal SBC or in an RTOS, consult the target documentation. The signal-based pattern may not apply.

Extended Topic: A Final Read of http.Server.Shutdown¶

The complete implementation, annotated. Source is in src/net/http/server.go. Here we reproduce a stripped-down version:

func (srv *Server) Shutdown(ctx context.Context) error {
    // (1) Set the shutdown flag.
    srv.inShutdown.Store(true)

    // (2) Acquire the server mutex.
    srv.mu.Lock()

    // (3) Close all listeners.
    lnerr := srv.closeListenersLocked()

    // (4) Fire OnShutdown callbacks (each in its own goroutine).
    for _, f := range srv.onShutdown {
        go f()
    }

    srv.mu.Unlock()

    // (5) Wait for accept loops to return.
    srv.listenerGroup.Wait()

    // (6) Polling loop.
    pollIntervalBase := time.Millisecond
    nextPollInterval := func() time.Duration {
        // exponential backoff up to 500ms
        // ... (omitted for brevity) ...
    }

    timer := time.NewTimer(nextPollInterval())
    defer timer.Stop()

    for {
        // (7) Check if all connections are closed.
        if srv.closeIdleConns() {
            return lnerr
        }

        // (8) Wait for next poll or context cancellation.
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-timer.C:
            timer.Reset(nextPollInterval())
        }
    }
}

Each numbered step:

1. Set the flag. Atomic write. Other operations check this flag to refuse work.
2. Lock. Protects internal maps.
3. Close listeners. Each listener's Close is called. Accept loops will return.
4. Fire hooks. Each OnShutdown callback in its own goroutine. The server does NOT wait for them.
5. Wait for accept loops. listenerGroup is a WaitGroup tracking the accept goroutines.
6. Setup polling. Exponential backoff from 1ms to 500ms.
7. Try to close idle connections. Returns true if all connections are closed (idle and active).
8. Wait or fail. Sleep until next poll, or return on context cancellation.

The whole function is about 30 lines. Read it. Understand it. It is the single most important shutdown function in Go.

closeIdleConns in detail¶

func (srv *Server) closeIdleConns() bool {
    srv.mu.Lock()
    defer srv.mu.Unlock()
    quiescent := true
    for c := range srv.activeConn {
        st, unixSec := c.getState()
        // StateNew connections are idle if > 5 seconds old
        if st == StateNew && unixSec < time.Now().Unix()-5 {
            st = StateIdle
        }
        if st != StateIdle || unixSec >= time.Now().Unix() {
            quiescent = false
            continue
        }
        c.rwc.Close()
        delete(srv.activeConn, c)
    }
    return quiescent
}

For each tracked connection:

• Check its current state.
• If StateNew for >5 seconds: treat as StateIdle (force-close stalled handshakes).
• If StateIdle (and not exactly at the current second): close it.
• Otherwise: leave it, mark as not quiescent.

Return true if all connections closed.

The 5-second rule for StateNew is interesting: it bounds the time a connection can sit in pre-request state. Helps if a client opens a connection and never sends data.

Extended Topic: A Comparison with Other Languages¶

For perspective, how graceful shutdown looks in other ecosystems.

Node.js¶

process.on('SIGTERM', () => {
  server.close(() => {
    process.exit(0);
  });
});

Single-threaded event loop. server.close stops accepting new connections; the event loop processes in-flight ones. Similar in spirit to Go but lacks the explicit deadline.

Python (FastAPI / Uvicorn)¶

Uvicorn has built-in graceful shutdown. Hookable via lifespan events:

@app.on_event("shutdown")
async def shutdown():
    await close_db()

Less low-level than Go; the framework handles signal trapping.

Java (Spring Boot)¶

server.shutdown.graceful config + spring.lifecycle.timeout-per-shutdown-phase. The framework provides graceful shutdown out of the box.

Rust (tokio)¶

let signal = tokio::signal::ctrl_c();
let server = ServerBuilder::new().build()
    .with_graceful_shutdown(async {
        signal.await.unwrap();
    });

Similar to Go in pattern. with_graceful_shutdown accepts a future that resolves on shutdown signal.

Common theme¶

Every modern ecosystem has a graceful shutdown mechanism. The patterns transfer. Go's is one of the more explicit and low-level, which has trade-offs: more flexibility, more boilerplate.

Extended Topic: Closing the Loop¶

The professional file has gone deep into mechanics. Let's loop back to why it matters.

Most Go production engineers don't need this depth. The junior file covers 95% of practical needs. The middle file covers 99%. The senior file makes you world-class for the typical service.

The professional level is for:

• Library authors. Building shared infrastructure needs deep correctness.
• SREs investigating rare incidents. When standard tools fail, the kernel-level view is the last resort.
• Performance engineers. Squeezing the last millisecond of shutdown latency.
• Educators. You can't teach what you don't understand at the mechanism level.

If you don't fall into these categories, you can read this file once and move on. The senior file is the practical ceiling for most.

For those who do fall into these categories, this file is a map. The territory is the Go source code. Go explore.

Final, Definitively Final Words¶

Graceful shutdown is a 15-line pattern. The reason it took five thousand lines of Roadmap content to teach is that the implications of those 15 lines reach into every layer of the stack — kernel signal delivery, runtime scheduler, network library, HTTP framework, container runtime, orchestrator, load balancer, monitoring stack.

A reader who has finished all five files (index, junior, middle, senior, professional) has the deepest understanding of Go shutdown patterns achievable. They will encounter shutdown issues with calm; they will design systems with care; they will mentor teams toward operational excellence.

Onwards to the practice material in the remaining files. Concepts without practice fade. Practice them.

Thank you for reading.

Appendix A: The Complete signal_unix.go Walkthrough¶

Go's src/runtime/signal_unix.go is the heart of signal handling. A guided tour of the key functions, paraphrased.

initsig¶

Called once at runtime startup. For each signal in the runtime's table:

for i := uint32(0); i < _NSIG; i++ {
    t := &sigtable[i]
    if t.flags == 0 || t.flags&_SigDefault != 0 {
        continue
    }
    // ... install handler ...
    setsig(i, abi.FuncPCABIInternal(sighandler))
}

Sets up the runtime as the signal handler for every "interesting" signal. setsig calls sigaction(2) under the hood.

sighandler¶

The actual signal handler invoked by the kernel. Pseudocode:

func sighandler(sig uint32, info *siginfo, ctxt unsafe.Pointer, gp *g) {
    _g_ := getg()
    c := &sigctxt{info, ctxt}

    // Check if it's a known synchronous fault (SIGSEGV, etc.)
    if sig == _SIGURG && doSigPreempt(...) {
        // Async preemption — handle and return
        return
    }

    if sig < uint32(len(sigtable)) {
        flags := sigtable[sig].flags
        if flags&_SigPanic != 0 && gp.throwsplit == 0 {
            // Synchronous signal — turn into panic
            // ...
        }
    }

    if !sigsend(sig) {
        // No user-space subscriber — take default action
        if flags&_SigKill != 0 {
            dieFromSignal(sig)
        }
    }
}

The key call: sigsend(sig). This enqueues the signal for the signal_recv goroutine.

sigsend¶

func sigsend(s uint32) bool {
    bit := uint32(1) << uint(s%32)
    // Check if anyone is interested
    if !sig.wanted[s/32]&bit != 0 {
        return false
    }
    // Atomically set the bit
    for {
        mask := sig.mask[s/32]
        if mask&bit != 0 {
            return true // already pending
        }
        if atomic.Cas(&sig.mask[s/32], mask, mask|bit) {
            break
        }
    }
    // Wake the receiver
    notewakeup(&sig.note)
    return true
}

The signal is recorded in a bit-set. If a receiver was sleeping (via notewakeup), it wakes up.

signal_recv¶

func signal_recv() uint32 {
    for {
        // Try to find a pending signal
        for i := uint32(0); i < _NSIG; i++ {
            j := i / 32
            b := uint32(1) << (i % 32)
            if sig.mask[j]&b != 0 {
                atomic.Cas(&sig.mask[j], sig.mask[j], sig.mask[j]&^b)
                return i
            }
        }
        // Wait for new signals
        notetsleepg(&sig.note, -1)
        noteclear(&sig.note)
    }
}