Manage pods with pod-level resources where the resize is allowed

Demand: someone runs pods with pod-level resources on a version whose InPlacePodLevelResourcesVerticalScaling gate is on and wants them shrunk on a full node — or Event: Headroom's oldest supported Kubernetes version has that gate on.

Pods with non-empty spec.resources are unmanaged on every version, because 1.35 forbids the resize and 1.36.1 and 1.37 refuse any container limit above a pod-level CPU limit. The cost is wherever the gate is on: measured on kind v1.37.0 before that gate, a pod with 100m of requests shrank to 100m on a full node, and with it the pod stays at 1 CPU.

Measured on 2026-11-08 on kind: on 1.36.1 and 1.37, the resize subresource changes spec.resources.limits.cpu up and down, and the kubelet applies it to the pod cgroup. So Headroom can manage these pods by writing its pod total into the pod-level limit and leaving container limits alone. The constraints are below.

Upstream

The gate belongs to KEP-5419, "Pod Level Resources Support With In Place Pod Vertical Scaling". It is beta and on by default from 1.36, and the KEP targets stable for 1.38. This was read from the tracking issue and the KEP metadata on 2026-11-06.

kubernetes#143257 (open as of 2026-11-06) reports that with this gate on, the scheduler counts a pod's RuntimeClass overhead twice when the pod-level status fields are set. Headroom's booking counts overhead once since Q56, so on such a node Headroom would see more slack than the scheduler books. This is untested, and it matters only for pods that combine pod-level resources with a RuntimeClass overhead.

Measurement

Single-node kind clusters (kh-plr-*), probed one at a time:

• kindest/node:v1.36.1@sha256:3489c7674813ba5d8b1a9977baea8a6e553784dab7b84759d1014dbd78f7ebd5
• kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
• the Makefile's KIND_NODE_IMAGE, kindest/node:v1.35.0@sha256:452d707d…, as the control

Each API server's /metrics reported:

v1.35.0  kubernetes_feature_enabled{name="InPlacePodLevelResourcesVerticalScaling",stage="ALPHA"} 0
v1.36.1  kubernetes_feature_enabled{name="InPlacePodLevelResourcesVerticalScaling",stage="BETA"} 1
v1.37.0  kubernetes_feature_enabled{name="InPlacePodLevelResourcesVerticalScaling",stage="BETA"} 1

All pods run one pause container. Every patch is a strategic-merge patch on the subresource, of this form:

kubectl -n plr patch pod a-nolimit --subresource resize -p '{"spec":{"resources":{"limits":{"cpu":"800m"}}}}'

For each patch, the probe recorded the API server's answer, spec.resources and status.resources about 6 s later, and cpu.max for the pod cgroup (kubelet-kubepods-burstable-pod<uid>.slice) and the app container's cri-containerd-<id>.scope, read with docker exec on the node. Do not use --type merge for container patches. A JSON merge patch replaces the whole containers list, so the container's requests vanish and default to its new limit. The resulting refusal (must be greater than or equal to aggregate container requests of 900m) comes from the probe, not from Kubernetes.

The pods:

Pod	spec.resources	Container c
a-nolimit	req cpu 100m / mem 64Mi, lim cpu 500m / mem 128Mi	req cpu 100m
b-ctrlimit	same as a	req cpu 100m, lim cpu 400m
c-memonly	req mem 64Mi, lim mem 128Mi	req cpu 110m
d-memonly-ctrlimit	same as c	req cpu 110m, lim cpu 110m
e-mempinned (1.37 only)	req cpu 100m / mem 128Mi, lim cpu 500m / mem 128Mi	req cpu 100m
f-guaranteed (1.37 only)	req cpu 500m / mem 128Mi, lim cpu 500m / mem 128Mi	none

Results

On 1.35.0, the API server refused every patch below, including the container-only ones. The refusal was the same each time, and no cgroup changed:

The Pod "a-nolimit" is invalid: []: Forbidden: resize of pods with pod-level resources is forbidden when InPlacePodLevelResourcesVerticalScalingEnabled feature gate is disabled

1.36.1 and 1.37.0 gave the same admission result and the same cpu.max on every patch:

Patch	Admitted	Pod cpu.max	Container cpu.max
a: pod lim 500m → 800m	yes	50000 → 80000 100000	follows the pod, 80000
a: pod lim 800m → 300m	yes	30000 100000	30000
b: pod lim 500m → 800m	yes	80000 100000	stays 40000
b: pod lim 800m → 450m	yes	45000 100000	40000
b: pod lim 450m → 300m, under the container's 400m	no	unchanged	unchanged
b: pod lim → 1000m, then container lim 400m → 900m	yes, both	100000 100000	90000
b: pod 1500m and container 1400m in one patch	yes	150000 100000	140000
c: add pod lim 800m to a pod born without one	yes	max → 80000 100000	80000 on 1.36.1; still max 6 s later on 1.37.0 (the next patch then wrote 20000)
c: pod lim 800m → 200m	yes	20000 100000	20000
c: pod lim → 100m, under the pod's 110m request	no	unchanged	unchanged
d: pod lim → 800m	yes	11000 → 80000 100000	stays 11000
d: then container lim 110m → 700m	yes	80000	70000
d: then pod lim → 300m, under the container's 700m	no	unchanged	unchanged

The two refusals read:

The Pod "b-ctrlimit" is invalid: spec.resources.containers[0][cpu].limits: Invalid value: "400m": must be less than or equal to pod limits of 300m
The Pod "c-memonly" is invalid: spec.resources.requests: Invalid value: "110m": must be less than or equal to cpu limit of 100m

No container restarted. Six seconds after a patch, status.resources.limits.cpu sometimes still showed the old value alongside PodResizeInProgress=True, while cpu.max already showed the new one. The status caught up by the next read.

Pod-level QoS, tested on 1.37.0 only:

$ kubectl -n plr patch pod e-mempinned --subresource resize -p '{"spec":{"resources":{"limits":{"cpu":"100m"}}}}'
The Pod "e-mempinned" is invalid: spec: Invalid value: "Burstable": Pod QOS Class may not change as a result of resizing
$ kubectl -n plr patch pod e-mempinned --subresource resize -p '{"spec":{"resources":{"limits":{"cpu":"101m"}}}}'
pod/e-mempinned patched
$ kubectl -n plr patch pod f-guaranteed --subresource resize -p '{"spec":{"resources":{"limits":{"cpu":"800m"}}}}'
The Pod "f-guaranteed" is invalid: spec: Invalid value: "Guaranteed": Pod QOS Class may not change as a result of resizing

Four things differed from what this row assumed before:

• Only d got a pod-level CPU limit filled in at create on 1.37.0. Its spec read limits: {cpu: 110m, memory: 128Mi}, the sum of the container limits. On 1.36.1 the spec has no pod-level CPU limit, though status.resources reports 110m. c, whose container sets only a request, got a filled-in pod-level CPU request (110m) on both versions and no limit. The fill-in therefore needs every container to set a CPU limit.
• The earlier claim "a pod born at 110m could not go above 110m" holds for container patches only. A raise of the pod-level limit itself was admitted and applied (d: 110m → 800m).
• On 1.36.1 the kubelet's status.resources.requests.cpu read 79m for b (spec 100m) and 104m for c and d (spec 110m) after their first pod-level resize. On 1.37.0 it matched spec. PodBookedCPUMilli reads status resources, so on 1.36.1 it would under-count these pods' booked CPU by up to 21%. The cause was not investigated.
• On 1.37.0, a pod-level limit of 101m came out as cpu.max=11000 100000 and status 110m, with no resize in progress. The cause was not investigated, and other values were not tried.

What managing them would take

Controller. Headroom computes the pod total exactly as it does now, then patches spec.resources.limits.cpu instead of the per-container split. Containers without their own CPU limit take the pod cgroup's value, as a and c show. Containers with their own limit keep it as a sub-cap. The pod-level limit is refused below two values, so the target's floor is the larger of the pod-level CPU request and the largest container CPU limit. Headroom should leave container limits untouched. Lowering the pod limit below a tenant-set container limit would mean rewriting that container's limit too. The paired patch b showed this is possible in one call, but it moves a value the tenant chose.

Version gate. The 1.35 refusal is is invalid wrapping Forbidden, so it takes the generic error path. The eligibility check needs to know the gate state. The server version is one option. A --dry-run=server resize is another, but it was not tried here.

QoS floor. Pod-level resources set the QoS class on their own: e's container has no limits, yet moving the pod-level CPU limit to the pod-level request was refused as a QoS change. So the counterpart of guaranteedAtRequest for these pods is a pod-level check. When pod-level memory request equals pod-level memory limit, the target must stay strictly above the pod-level CPU request; 101m was admitted against a 100m request. A pod that is Guaranteed through pod-level resources (f) cannot be raised at all, so it stays unmanaged. Eligible's Burstable check already gives that result.

Webhook seed. Today the webhook skips these pods (HasPodLevelResources). Its counterpart would seed spec.resources.limits.cpu = round(pod-level CPU request × multiplier), quantized, when the pod sets a pod-level CPU request and no pod-level CPU limit. The seed must be at least the largest container CPU limit, and strictly above the request where pod-level memory is pinned. It should never seed container limits on such a pod. On 1.37, a pod whose containers all have CPU limits gets a pod-level limit summed from them, and that sum would become the pod's ceiling. Nobody has checked whether that fill-in runs before or after mutating admission.