The issue I fixed is about Kubernetes’ in-memory watch cache. This part of the code is responsible for maintaining an in-memory cache (I believe it’s implemented as a ring buffer) and keeping it in sync as etcd changes. The problem originates from this line in the watch() function in staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:

go watcher.processInterval(ctx, cacheInterval, requiredResourceVersion)

This line sits near the end of the function, and it’s arguably the most important part: it starts a goroutine that keeps the cache watcher in sync. Everything before it — fetching the RV, configuring options, checking feature gates — is just preparation.

Back to the issue: it shows that a specific kind of client can cause a goroutine leak in the API server’s watch implementation. Here’s the reproduction case:

sendInitialEvents := true
timeoutSeconds := int64(time.Hour.Seconds())
watcher, _ := client.CoreV1().Pods(namespace).Watch(ctxOnce, metav1.ListOptions{
	ResourceVersion:      "0",
	ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan,
	SendInitialEvents:    &sendInitialEvents,
	Watch:                true,
	AllowWatchBookmarks:  true,
	TimeoutSeconds:       &timeoutSeconds,
})
defer watcher.Stop()
for {
	select {
	case <-watcher.ResultChan():
                // do not deal with event
                time.Sleep(time.Hour)
	case <-ctx.Done():
		return
	}
}

This runs a watch client that only becomes active once an hour. One detail worth noting is the “0” resource version — it asks for the full history.

The next step is digging into the underlying goroutine function, which leads to processInterval in staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go. The function receives a cacheInterval, which holds a source field, its iteration logic, a resourceVersion, and an initialEventsEndBookmark. This type is used to drain the interval between the cache and etcd.

So what causes it to get stuck? Digging deeper, we find this:

c.sendWatchCacheEvent(event)

Inside it:

	select {
	case c.result <- *watchEvent:
		c.markBookmarkAfterRvSent(event)
		sentAt = c.clock.Now()
	case <-c.done:
	}

This is the source of the issue: the select statement blocks on case c.result <- *watchEvent once the client stops consuming from the channel — exactly what happens in the repro, where the watch only reads once an hour.

With that, we can take a broader view of the control path. Watch prepares the cache watch, then calls processInterval, which holds the main watch-and-sync logic. There are two phases inside it. First, for requests still in the initial phase, it replays the full history from etcd into c.result via sendWatchCacheEvent. If the client stops consuming those events, this causes the same stall described in the issue.

With the full picture in hand, the next question is how to fix it. My first idea, based on a comment on the issue, was to evict references that are no longer in the watch cache. But I quickly realized this was tricky and not worth it — it would likely require adding reverse references from each event/snapshot so they could be traced back on deletion, and it’s a hot-path operation that could hurt performance.

So I went with something simpler: use a fixed timeout — 30 seconds — and terminate the watcher if it makes no progress sending events during the initial phase. The timer resets on every successful send, so it only bounds how long a watch can stall completely, not how long it takes to deliver everything. This protects slow-but-progressing watchers, including ones with a large initial snapshot, from being evicted unfairly — a risk a single total-time budget would carry. This fixes the repro and resolves the stuck-connection issue.

Note that processInterval calls processe at the end, meaning the connection has finished the initial phase and moved into the streaming phase. Since process also calls the same sendWatchCacheEvent, I initially assumed my fix would affect it too — so I dug further. It turns out it doesn’t need the fix: the streaming phase already has its own stall-handling path. When it gets stuck, the c.input channel blocks, and on the producer side, add() soon detects this and closes the stalled watcher. That’s why it doesn’t need the timeout fix.

PR: https://github.com/kubernetes/kubernetes/pull/141899

Issue: https://github.com/kubernetes/kubernetes/issues/131721