Buoyant Kinference Brings GPU-Aware AI Inference Management to Kubernetes

GPU scarcity and low utilization can coexist in the same AI infrastructure. Buoyant’s open-source Kinference project targets that mismatch with a control plane for coordinating vLLM inference workloads across Kubernetes clusters, giving platform teams more say over resource use, service quality and where requests run. The premise is straightforward: when accelerators are scarce, keeping them busy matters, but simply sending more work to them isn’t enough if that degrades response times or violates operating constraints. Kinference is meant to give teams a way to balance those competing demands across clusters.

Coordination for distributed vLLM

Kinference sits above vLLM, the inference engine and server developed by researchers at the University of California, Berkeley’s Sky Computing Lab. vLLM uses PagedAttention to improve memory throughput and includes an inference server for handling network traffic. Kinference adds a management layer for workloads spread across Kubernetes environments, where operators may need to coordinate capacity that’s divided among separate clusters or teams. The distinction matters: an inference engine handles model serving, while a control plane can apply policies to where workloads run and how available resources are assigned.

Buoyant CEO William Morgan says the control plane is intended to help IT teams scale those workloads with more control over cost and performance. Teams can set operating boundaries for performance, cost and compliance, then manage inference within them. That’s a useful framing because capacity decisions aren’t purely technical: an operator may have to weigh response time against the expense of using external infrastructure, or keep certain workloads within defined boundaries. The software’s practical value will depend on whether those policies can be expressed clearly and applied reliably as demand changes.

That matters as organizations consider running models on infrastructure they operate themselves, where they can assign capacity to workloads instead of paying cloud providers per token. Self-hosting can make resource planning more visible, but it also transfers responsibility for keeping systems supplied and serving requests. Teams have to understand what their hardware can handle and how competing jobs affect one another. If you're working in this space, the appeal is less about owning hardware for its own sake and more about controlling the tradeoffs that come with serving models at scale.

Self-hosting also gives teams greater control over how sensitive data reaches a model. Security concerns and compliance requirements can make organizations wary of sending that data to services managed by third parties. Morgan also points to latency: AI applications often involve repeated interactions, so delays can affect the user experience. A request that takes longer may influence the next step in a conversation or workflow, making response time part of product quality rather than merely an infrastructure metric. That said, keeping data and workloads in-house doesn’t remove the need to manage access, capacity and performance carefully.

Using GPU capacity more deliberately

Kinference’s central resource-management idea is to account for signals such as GPU cache data and queue status when directing work across AI workloads. Those signals can provide a more useful view of current conditions than a simple tally of allocated devices: a GPU may be assigned to a workload while its memory or queue state affects how much additional work it can serve. Morgan says GPU utilization is often in the single digits, even as organizations face a broad shortage of accelerators. Better coordination could help teams keep more of their installed GPU capacity productive. This is more significant than it looks, since the gap between having GPUs and getting useful work from them can shape the economics of an entire deployment.

The control plane can also route inference requests to cloud providers when an organization’s conditions call for it. Those rules might respond to a saturated GPU pool, a particular tenant’s usage or the characteristics of an incoming request. That gives teams a way to keep routine or sensitive work on infrastructure they control while sending selected requests elsewhere when local capacity is constrained. The usefulness of that approach depends on whether the routing rules match the organization’s actual priorities, and whether operators can understand why a request took a particular path. A fallback to cloud capacity may relieve local pressure, but it also changes where the work runs and how its cost is accounted for.

For cost visibility, Kinference tracks GPU use by tenant, group or team. That attribution can help operators see which workloads consume capacity and make more informed choices about infrastructure allocation. It may also make internal discussions about shared resources less abstract: teams can compare demand with observed usage rather than relying only on requests for more hardware. The tracking won’t settle every allocation dispute, especially where workloads have different performance needs, but it can give operators a clearer starting point. (And this is the part most people overlook.) Better measurement only helps if teams use it to revise how they allocate capacity.

A new scheduling problem for Kubernetes teams

AI inference is joining the mix of applications platform teams already run on Kubernetes, adding workloads with demanding and sometimes variable requirements. The number of AI workloads on clusters remains unclear, but their growth raises a practical question: how should teams allocate limited compute across inference and other services? Kubernetes already gives platform teams tools for managing applications, yet GPU-backed inference introduces resource behavior that may require more specialized signals. A scheduler that treats every workload as an interchangeable request for compute can miss the difference between a busy queue, available GPU memory and an application’s need to respond quickly.

Kinference brings workload policy, GPU scheduling signals and optional cloud routing into one control plane. Its value will depend on how well those controls translate into higher utilization and predictable service for real deployments. A policy interface can make decisions easier to define, but it can’t guarantee that the underlying capacity is sufficient or that every workload behaves as expected. The operational test is whether teams can keep services responsive while avoiding idle resources and unwanted spillover costs.

What the control plane could change

For teams building self-hosted AI, the concrete takeaway is that managing inference capacity may require a layer that understands both Kubernetes workloads and the state of the GPUs serving them. Kinference’s approach reflects a broader operational challenge: as model serving becomes part of ordinary application infrastructure, platform teams need to connect workload policies with the hardware conditions beneath them. The details matter. A system that reports utilization without helping teams act on it offers limited relief, while a routing policy that’s difficult to audit can create new uncertainty.

What this means for you depends on where your bottleneck sits. If GPU queues, tenant demands and service targets are pulling in different directions, a dedicated control plane may make those tradeoffs easier to manage. If capacity is already well matched to demand, another layer of policy may add operational work without changing the outcome. Kinference makes a clear case for coordinating distributed inference; its impact will be measured in day-to-day results. More useful capacity, with predictable service.