This issue is relate to the controller and handler, which is the part of code I am not familiar. Therefore it consumes me days to understand how does it work inside the rcd server.
CRD server(maybe it is not precise?) is contained inside the api-server and is called as apiextension-server actually in code. It handles CRD request and events when the event arrives, which is sent by a shared Informer. Here I want to add some about Informer: it’s a specialized component for syncing, draining and dispatching event to its destination.
Then comes to our controller - from the CRD-server will get those events in real-time, triggering its pre-programmed logic and probably cause status change on its resource. And the second part is handler. The handler is the same type we know from go std http lib. It’s a function and it receives responseWriter and request ptr, too. The request to CRD will be dispatch to the func. And each of controller has its own handler usually for.
Now we have the all we need to know to understand the issue, let’s see what happen to this issue first: When someone update the CRD spec and request to the endpoint immediately, the server’s return a stale response - it doesn’t follow the latest change. That means the problem is come from a refreshness problem. What cause that? The spec change should generate an event that consumed by controller and informer, so that they can update their status. If we go deeper in the code, we will find that informer send events to them in a not fixed sequence - that’s correctand deliberate, because of that it won’t stall on waiting a slow consumer. Go further by checking the code of handler and controller, I found that the discovery controller takes responsible on publishing the discovery document. It should be an important part because the original issue tells the url mismatch symptom. But its logic is fine.
So I focus on the crdHandler which is the true handler managing crd REST storage. Then I found the root: the handler use a crd.UID to search from a possibily outdated cache. It didn’t checkif the cache is up-to-date or not at all. So if the Informer sends the event to both discovery controller and crdHandler, that’s where the race occure. If the discovery controller is faster than the crdHandler, when the request arrive, the crdHandler still keeps a stale cache without noticing the spec is changed. In contrast is the normal and expected path.
Now we find the problem, which is the hardest part. The fix is very clear. Since we can’t change the informer behavior that cause a huge blast range, what we can do is let the crdHandler update is store whenever the spec is changed. So I remove the crd.UID cache map finding functiong. Let it use generation to judge if the spec is ever changed.
There is a gotcha here. The generation is good to use but is not enough. If the expectedName(in crd status) is conflicted with the other one , which will be update by server with zero generation bump, the crdHandler won’t know that and may build a wrong storage. So we need add a function to check equality between them to judge if it need to update the storage.