Kubernetes Gets Serious About GPU Scheduling and Agent Control

Kubernetes’ next AI push targets GPU efficiency, massive clusters, and safer agents.

Plenty of people still underestimate how much AI runs on Kubernetes. In the CNCF January 2026 Annual Cloud Native Survey, 66% of organizations hosting generative AI models used Kubernetes for inference. Add everyone running other AI workloads on the platform, and the majority of AI users are leaning on it. At KubeCon North America 2026, the CNCF and its vendors laid out what comes next: smarter scheduling, larger control planes, faster container startup, and tighter limits on autonomous agents.

The next milestone is Kubernetes 1.38, due December 16. It brings gang scheduling and more accelerator-allocation options closer to stable status. But the message reached past a single release. Karpenter, the open-source node lifecycle manager that provisions and terminates compute in response to unscheduled pods, is picking up its own improvements.

Making the Control Plane Keep Up

AWS’s Head of Containers, Mike Stefaniak, put the core problem bluntly in his keynote: “AI workloads can spin up 10s of 1000s of GPU instances per cluster, and things break at that scale.” His answer is to push fixes discovered in huge production deployments upstream instead of keeping them in a provider’s private fork. That's the right instinct. Fork-only patches help one vendor and leave everyone else to rediscover the same failures.

AWS hit bottlenecks while pushing past Kubernetes’ conventional 5,000-node design envelope toward 100,000 nodes. In a million-pod benchmark, a lock on the API server’s watch cache left writers waiting more than half a second. A lazy-snapshot implementation cut that wait to two milliseconds.

Other changes removed serialization in Horizontal Pod Autoscaler processing and replaced repeated scheduler volume-reference calculations with incremental updates. Stefaniak reported scheduling rates above 300 pods per second under sustained backlog in AWS’s largest clusters. He said those fixes already shipped in Kubernetes 1.37.

For 1.38, AWS co-authored a proposal to compress API objects before they're written to etcd, the backing data store. “It’s accepted for 1.38 alpha, and the first code was actually merged just last week,” Stefaniak said. The immediate goal is relieving storage pressure in very large clusters.

Scheduling Whole AI Jobs

The bigger shift is treating distributed AI workloads as coordinated groups instead of unrelated pods. Kubernetes’ gang-scheduling enhancement does that, and the Workload and PodGroup APIs supply native primitives for building those groups.

The payoff is simple. A training job where only some workers fit can sit on expensive GPUs making no progress. Gang scheduling holds the group until its required minimum membership can be placed. It coordinates admission. It doesn’t guarantee every container starts at the same instant, but it stops GPUs from burning time on half-built jobs.

The roadmap also targets beta maturity for topology-aware placement and CompositePodGroups. Those cover communication-sensitive placement and workloads made of several coordinated groups, which describes distributed training and disaggregated inference well.

Dynamic Resource Allocation (DRA) fills in the hardware side. It describes accelerator capabilities and capacity more precisely, and 1.38 pushes several features toward general availability: partitionable devices, consumable capacity, fractional capacity ranges, and device binding conditions. Allocation stops being a matter of counting whole GPUs.

Getting GPUs Online Faster

GPUs aren’t being left out. Stefaniak said Karpenter now models devices on the instance types it can provision. That closes the gap between DRA’s device inventories and just-in-time node creation. “GPU capacity can scale from zero and be optimally placed,” he said. Note that this is Karpenter work, not a Kubernetes 1.38 feature.

Container startup is the next bottleneck. AI images can reach tens of gigabytes, and parallel downloads just move the delay into unpacking layers. An upstreamed parallel-unpack implementation in containerd 2.2 cut a 10GB image’s registry-to-node time from two minutes to consistently under 45 seconds in AWS testing. Shaving that delay matters when you're paying for idle accelerators while the image loads.

Agents Need Limits Too

Efficiency now covers agents, not just models. At the press briefing, Jonathan Bryce, the CNCF Executive Director, framed the challenge plainly: “This is how do we take the same footprint, the same power consumption, the same set of hardware, and get more out of it.”

Bryce pointed to the default-deny approach to agent permissions in OpenShell and to how Agent Substrate handles idle agents. He cited a demonstration that went from one agent per node to 65 by putting sleeping and working agents in rotation around their tasks. It's still work in progress, but the density numbers are hard to ignore.

Portability remains unfinished. Dilpreet Bindra, VMware Senior Director of Engineering, said, “Our goal is to define and verify how workloads should behave across any environment, reducing the fragmentation and ensuring portability.” Vendor-neutral behavior specs would matter a lot to teams that don't want to be locked into one cloud's GPU stack.

The direction is clear: AI is pushing Kubernetes to coordinate workloads, hardware, and security as one problem. What ships in December still depends on the release process, and running these pieces reliably in production will take longer. Still, if the scheduling and allocation work lands as planned, AI teams should get more output from the same hardware, and that's the number that will drive the next round of platform decisions.