Skip to content

gvisor: LinkEndpointFilter.Attach never forwards nil, so GVisor.Close leaks all inbound dispatcher goroutines and fds #85

Description

@k1ng

Summary

GVisor.Close() stops the inbound machinery via t.endpoint.Attach(nil). The fdbased
endpoint's teardown only runs on a literal nil dispatcher:

// internal/fdbased_darwin/endpoint.go (Attach)
if dispatcher == nil && e.dispatcher != nil {
    for _, dispatcher := range e.inboundDispatchers {
        dispatcher.Stop()
    }
    ...
    e.Wait()
    return
}

But t.endpoint is a LinkEndpointFilter, whose Attach wraps the dispatcher
unconditionally:

// stack_gvisor_filter.go
func (w *LinkEndpointFilter) Attach(dispatcher stack.NetworkDispatcher) {
	w.LinkEndpoint.Attach(&networkDispatcherFilter{dispatcher, w.BroadcastAddress, w.Writer})
}

So on Close the underlying endpoint receives a non-nil networkDispatcherFilter{nil, …},
neither branch of Attach matches, and Stop()/Wait() never run. Every GVisor.Close
permanently leaks:

  • the recvMMsg dispatcher goroutine, parked in kevent (pinning an OS thread). The tun fd is
    closed elsewhere, and closing an fd silently removes its kqueue registrations — so the
    goroutine waits forever on a stop-fd that will never be written;
  • the processor.start goroutines (ProcessorsPerChannel);
  • the qdisc fifo dispatchLoop goroutine;
  • the stopfd pipe fds (plus the kqueue fd).

Affects any process that recreates the gvisor stack in-place (config reload / reconnect in a
library embedding). Long-run numbers from such a process on macOS: ~15 goroutines + several fds
leaked per stack recreation; after 18 days of periodic reconnects, ~730 leaked goroutines, ~70
kevent-pinned OS threads, hundreds of fds. Verified on v0.8.9 and current dev (the
ForwardDispatcher refactor keeps the unconditional wrap). The filter is cross-platform, so
the Linux fdbased endpoint should be affected the same way; verified on darwin.

Reproduction (in-process, darwin, -tags with_gvisor)

before := runtime.NumGoroutine()
for i := 0; i < 10; i++ {
    tunIf, _ := tun.New(tunOpts)                  // any working tun options
    stack, _ := tun.NewStack("gvisor", tun.StackOptions{Tun: tunIf, TunOptions: tunOpts, ...})
    stack.Start()
    stack.Close()
    tunIf.Close()
}
time.Sleep(time.Second)
after := runtime.NumGoroutine()   // grows by ~(2 + ProcessorsPerChannel) per iteration

A goroutine profile after the loop shows the piles at
fdbased_darwin.(*processor).start, qdisc/fifo.(*queueDispatcher).dispatchLoop, and
rawfile_darwin.BlockingRecvMMsgUntilStopped (in kevent).

Fix

Forward nil as nil:

func (w *LinkEndpointFilter) Attach(dispatcher stack.NetworkDispatcher) {
	if dispatcher == nil {
		w.LinkEndpoint.Attach(nil)
		return
	}
	w.LinkEndpoint.Attach(&networkDispatcherFilter{dispatcher, w.BroadcastAddress, w.Writer})
}

Running this one-liner in production: goroutine and fd counts are flat across stack
recreations since.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions