[{"content":"This issue is relate to the controller and handler, which is the part of code I am not familiar. Therefore it consumes me days to understand how does it work inside the rcd server.\nCRD server(maybe it is not precise?) is contained inside the api-server and is called as apiextension-server actually in code. It handles CRD request and events when the event arrives, which is sent by a shared Informer. Here I want to add some about Informer: it\u0026rsquo;s a specialized component for syncing, draining and dispatching event to its destination.\nThen comes to our controller - from the CRD-server will get those events in real-time, triggering its pre-programmed logic and probably cause status change on its resource. And the second part is handler. The handler is the same type we know from go std http lib. It\u0026rsquo;s a function and it receives responseWriter and request ptr, too. The request to CRD will be dispatch to the func. And each of controller has its own handler usually for.\nNow we have the all we need to know to understand the issue, let\u0026rsquo;s see what happen to this issue first: When someone update the CRD spec and request to the endpoint immediately, the server\u0026rsquo;s return a stale response - it doesn\u0026rsquo;t follow the latest change. That means the problem is come from a refreshness problem. What cause that? The spec change should generate an event that consumed by controller and informer, so that they can update their status. If we go deeper in the code, we will find that informer send events to them in a not fixed sequence - that\u0026rsquo;s correctand deliberate, because of that it won\u0026rsquo;t stall on waiting a slow consumer. Go further by checking the code of handler and controller, I found that the discovery controller takes responsible on publishing the discovery document. It should be an important part because the original issue tells the url mismatch symptom. But its logic is fine.\nSo I focus on the crdHandler which is the true handler managing crd REST storage. Then I found the root: the handler use a crd.UID to search from a possibily outdated cache. It didn\u0026rsquo;t checkif the cache is up-to-date or not at all. So if the Informer sends the event to both discovery controller and crdHandler, that\u0026rsquo;s where the race occure. If the discovery controller is faster than the crdHandler, when the request arrive, the crdHandler still keeps a stale cache without noticing the spec is changed. In contrast is the normal and expected path.\nNow we find the problem, which is the hardest part. The fix is very clear. Since we can\u0026rsquo;t change the informer behavior that cause a huge blast range, what we can do is let the crdHandler update is store whenever the spec is changed. So I remove the crd.UID cache map finding functiong. Let it use generation to judge if the spec is ever changed.\nThere is a gotcha here. The generation is good to use but is not enough. If the expectedName(in crd status) is conflicted with the other one , which will be update by server with zero generation bump, the crdHandler won\u0026rsquo;t know that and may build a wrong storage. So we need add a function to check equality between them to judge if it need to update the storage.\n","permalink":"https://blog.datayurei.com/en/posts/k8s-issue-130235/","summary":"\u003cp\u003eThis issue is relate to the controller and handler, which is the part of code I am not familiar.\nTherefore it consumes me days to understand how does it work inside the rcd server.\u003c/p\u003e\n\u003cp\u003eCRD server(maybe it is not precise?) is contained inside the api-server and is called as apiextension-server\nactually in code. It handles CRD request and events when the event arrives, which is sent by a shared Informer. Here I\nwant to add some about Informer: it\u0026rsquo;s a specialized component for syncing, draining and dispatching event to its destination.\u003c/p\u003e","title":"Understand how does CRD server work when going though issue#130235"},{"content":"The issue I fixed is about Kubernetes\u0026rsquo; in-memory watch cache. This part of the code is responsible for maintaining an in-memory cache (I believe it\u0026rsquo;s implemented as a ring buffer) and keeping it in sync as etcd changes. The problem originates from this line in the watch() function in staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:\ngo watcher.processInterval(ctx, cacheInterval, requiredResourceVersion) This line sits near the end of the function, and it\u0026rsquo;s arguably the most important part: it starts a goroutine that keeps the cache watcher in sync. Everything before it — fetching the RV, configuring options, checking feature gates — is just preparation.\nBack to the issue: it shows that a specific kind of client can cause a goroutine leak in the API server\u0026rsquo;s watch implementation. Here\u0026rsquo;s the reproduction case:\nsendInitialEvents := true timeoutSeconds := int64(time.Hour.Seconds()) watcher, _ := client.CoreV1().Pods(namespace).Watch(ctxOnce, metav1.ListOptions{ ResourceVersion: \u0026#34;0\u0026#34;, ResourceVersionMatch: metav1.ResourceVersionMatchNotOlderThan, SendInitialEvents: \u0026amp;sendInitialEvents, Watch: true, AllowWatchBookmarks: true, TimeoutSeconds: \u0026amp;timeoutSeconds, }) defer watcher.Stop() for { select { case \u0026lt;-watcher.ResultChan(): // do not deal with event time.Sleep(time.Hour) case \u0026lt;-ctx.Done(): return } } This runs a watch client that only becomes active once an hour. One detail worth noting is the \u0026ldquo;0\u0026rdquo; resource version — it asks for the full history.\nThe next step is digging into the underlying goroutine function, which leads to processInterval in staging/src/k8s.io/apiserver/pkg/storage/cacher/cache_watcher.go. The function receives a cacheInterval, which holds a source field, its iteration logic, a resourceVersion, and an initialEventsEndBookmark. This type is used to drain the interval between the cache and etcd.\nSo what causes it to get stuck? Digging deeper, we find this:\nc.sendWatchCacheEvent(event) Inside it:\nselect { case c.result \u0026lt;- *watchEvent: c.markBookmarkAfterRvSent(event) sentAt = c.clock.Now() case \u0026lt;-c.done: } This is the source of the issue: the select statement blocks on case c.result \u0026lt;- *watchEvent once the client stops consuming from the channel — exactly what happens in the repro, where the watch only reads once an hour.\nWith that, we can take a broader view of the control path. Watch prepares the cache watch, then calls processInterval, which holds the main watch-and-sync logic. There are two phases inside it. First, for requests still in the initial phase, it replays the full history from etcd into c.result via sendWatchCacheEvent. If the client stops consuming those events, this causes the same stall described in the issue.\nWith the full picture in hand, the next question is how to fix it. My first idea, based on a comment on the issue, was to evict references that are no longer in the watch cache. But I quickly realized this was tricky and not worth it — it would likely require adding reverse references from each event/snapshot so they could be traced back on deletion, and it\u0026rsquo;s a hot-path operation that could hurt performance.\nSo I went with something simpler: use a fixed timeout — 30 seconds — and terminate the watcher if it makes no progress sending events during the initial phase. The timer resets on every successful send, so it only bounds how long a watch can stall completely, not how long it takes to deliver everything. This protects slow-but-progressing watchers, including ones with a large initial snapshot, from being evicted unfairly — a risk a single total-time budget would carry. This fixes the repro and resolves the stuck-connection issue.\nNote that processInterval calls processe at the end, meaning the connection has finished the initial phase and moved into the streaming phase. Since process also calls the same sendWatchCacheEvent, I initially assumed my fix would affect it too — so I dug further. It turns out it doesn\u0026rsquo;t need the fix: the streaming phase already has its own stall-handling path. When it gets stuck, the c.input channel blocks, and on the producer side, add() soon detects this and closes the stalled watcher. That\u0026rsquo;s why it doesn\u0026rsquo;t need the timeout fix.\nPR: https://github.com/kubernetes/kubernetes/pull/141899\nIssue: https://github.com/kubernetes/kubernetes/issues/131721\n","permalink":"https://blog.datayurei.com/en/posts/k8s-issue-131721/","summary":"Something about the Cache machanism I learn when I solving issue 131721 in k8s","title":"Something I learned about kubernetes api-server watch mechanism when try solving issue#131721"},{"content":" This post was originally written in Chinese.\nNightreign runs with an unusually low frame rate on Linux, while CPU and GPU usage both remain very low. After searching around on Google, I found the answer in a GitHub Proton issue:\nhttps://github.com/ValveSoftware/Proton/issues/8751#issuecomment-2957426376\nhttps://www.phoronix.com/news/Linux-Splitlock-Hurts-Gaming\nSome games, anti-cheat systems, drivers, or runtimes can trigger split locks. When Linux detects this, it throttles the related process to prevent it from slowing down the whole system.\nThis mechanism is good for system fairness, but for games it can show up as lower frame rates, stuttering, slower loading, or increased input latency.\nAfter disabling it with kernel.split_lock_mitigate=0, the kernel no longer penalizes the process that triggers split locks, so game performance can recover. The tradeoff is that if a program continues triggering many split locks, overall system responsiveness may be affected.\nFor a daily-use Linux PC, I think it is completely reasonable to disable this and it should not have much impact.\nThe specific fix is as follows. To apply it once and take effect immediately:\nsudo sysctl kernel.split_lock_mitigate=0 To restore the original behavior, set it back to 1:\nsudo sysctl kernel.split_lock_mitigate=1 To make it persistent, add the following to /etc/sysctl.d/99-splitlock.conf:\nkernel.split_lock_mitigate=0 ","permalink":"https://blog.datayurei.com/en/posts/game-optimization-on-linux/","summary":"A Linux gaming performance issue caused by split lock mitigation, and how to work around it.","title":"Frame Rate Issues in Some Games Such as Elden Ring Nightreign on Linux"},{"content":" This post was originally written in Chinese.\nAfter running into a number of small issues on Windows over the past few days, I decided to switch my main system to Ubuntu. That led me into figuring out how to play games on Linux.\nWhile testing HELLDIVERS 2, I found that fullscreen mode did not work the way it did in other games. After switching to fullscreen, Waybar was still visible. Even after closing Waybar, a small strip of the desktop could still be seen at the top or bottom of the screen. I also tried using Gamescope, but it either crashed or failed to start.\nThen, by chance, I accidentally reset Niri\u0026rsquo;s scale back to 1 and realized the problem was caused by fractional scaling. As long as the scale is set back to an integer value, everything works as expected.\nAlthough Wayland is already quite usable today, support for features like fractional scaling is still not perfect. If possible, it is better to avoid enabling it.\n","permalink":"https://blog.datayurei.com/en/posts/wayland-niri-gaming-fullscreen-bug/","summary":"A fullscreen issue that can appear when running games on Linux with Wayland and Niri.","title":"Fullscreen Issues When Gaming with Niri"},{"content":" This post was originally written in Chinese.\nThis is the initial post used to validate the Hugo and GitHub Actions deployment flow.\n","permalink":"https://blog.datayurei.com/en/posts/hello-hugo/","summary":"Initial post used to validate the Hugo and GitHub Actions deployment flow.","title":"Hello Hugo"}]