Skip to content

Share one client budget across REST clients, or state it per client ​

cmd/main.go sets QPS and Burst on the rest config, and each REST client builds its own token bucket from them, so QPS 50 does not bound the controller's total writes. The design's rate-limit section said it did until Q50.

controller-runtime keeps one REST client per kind and per typed or unstructured form. So the two Pod writes in a resize pass draw from two buckets: writePodStatus patches a typed *corev1.Pod and applyResize patches an unstructured one. The cache key is the GVK plus forceDisableProtoBuf, which is true for an unstructured object (getResource in vendor/sigs.k8s.io/controller-runtime/pkg/client/client_rest_resources.go), and vendor/k8s.io/client-go/rest/config.go builds a new limiter per REST client when rest.Config.RateLimiter is unset.

A throwaway probe confirmed the split: both functions on a client.New client against a stub server, at QPS 1, Burst 1. A second call to the same function waited 1.0s, a call to the other function waited 0s, and with one shared RateLimiter a call to the other function waited 1.0s. That probe did not go through a manager or a real apiserver.

The reviewer of Q50 measured the cost through a real manager on envtest 1.37, at QPS 50, Burst 100, with 60 nodes × 8 pods and one dryRun flip:

ClientResizesDrainAll PATCH requestsAveragePeak in 1s
As shipped4808.6s960111/s298/s
With one shared rest.Config.RateLimiter48027.5s144052/s148/s

The shared run's extra 480 are inferred to be annotation rewrites, and that row measures the limiter alone rather than a full suite with it.

Decide whether a shared limiter is worth the slower drain, then either share it or document the budget as per REST client in docs/runbook.md. Sharing it would also put each resize behind status annotation writes in one queue, so measure that before choosing. If the budget stays split, also correct the two places that still call it global: the comment above restConfig in cmd/main.go, and the ClientQPS and ClientBurst field comments in api/v1alpha1/headroomconfig_types.go, which generate the CRD's descriptions.