Kubernetes News

• Kubernetes v1.37: Native Histograms Graduates to Beta

  I'm excited to announce that native histogram support for Kubernetes metrics is graduating to Beta and is enabled by default in Kubernetes v1.37!

  Native histograms (previously introduced as Alpha in Kubernetes v1.36 under KEP-5808) bring high-resolution, low-cardinality observability to Kubernetes metrics. By adopting Prometheus Native Histograms, Kubernetes components now expose latency and duration metrics with far greater accuracy while significantly reducing telemetry storage and scraping overhead.

  Why move beyond classic histograms?

  Since the early days of Kubernetes observability, duration and latency metrics (such as API server request latencies or scheduling durations) have relied on classic Prometheus histograms.

  Classic histograms require metric authors to define a static list of cumulative bucket boundaries (le labels), such as 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. While familiar, this approach introduces three major challenges:

  1. The Bucket Guessing Game: If a workload's latency profile changes, for example, shifting into microsecond ranges or experiencing long-tail tail latencies beyond the highest bucket, the histogram loses visibility. Specifying bucket boundaries upfront requires knowing the distribution before observing it
  2. High Cardinality & Storage Cost: With classic histograms, each bucket boundary is exported as a separate time series (_bucket{le="..."}). A histogram with 10 buckets across multiple labels multiplies the number of time series by 10, increasing memory consumption in Prometheus and inflating time series database (TSDB) storage costs
  3. Interpolation Error in Quantiles: Calculating percentiles using histogram_quantile() relies on linear interpolation between static bucket boundaries. When bucket spans are coarse, quantile calculations can suffer from significant estimation error

  What are Prometheus native histograms?

  Prometheus Native Histograms replace static user-defined buckets with dynamic, exponential buckets.

  Instead of emitting a separate time series for every single bucket boundary, a native histogram is stored as a single time series containing a rich schema of positive and negative spans, zero thresholds, and exponential scaling factors.

  • High Resolution Automatically: Exponential buckets dynamically adjust to any value range — from nanoseconds to hours — without requiring pre-configured bucket boundaries
  • Up to 90% Fewer Time Series: By consolidating buckets into structured spans within a single time series, scraping and storage overhead are dramatically reduced
  • Accurate Quantile Calculation: Quantiles can be calculated with mathematical bounds on error (≃5% worst-case relative error under default settings) across the entire spectrum of observations

  How native histograms work in Kubernetes

  In Kubernetes, native histogram support is implemented directly inside the shared metrics subsystem (k8s.io/component-base/metrics).

  Figure 1 illustrates how native histogram metrics are processed and exposed across Kubernetes components.

  

  1. Dual exposition for zero breaking changes

  A primary design requirement for KEP-5808 was zero disruption for existing observability stacks. When the NativeHistograms feature gate is enabled, Kubernetes components use dual exposition:

  • Classic buckets (h.Bucket) are still emitted alongside native spans. Existing Prometheus servers, dashboards, and alerting rules that rely on traditional text scraping or classic bucket labels continue to work unmodified
  • Native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand native histograms

  2. Tuned default exponential configuration

  When NativeHistograms is enabled, the k8s.io/component-base/metrics package automatically applies standardized exponential options to all histogram metrics:

  • BucketFactor: 1.1: Configures exponential buckets where each bucket is at most 10% wider than the preceding one. This guarantees a mathematically bounded worst-case relative error of at most ~5% for quantile calculations regardless of whether an operation takes 1 millisecond or 10 seconds.
  • MaxBucketNumber: 160: Caps the maximum number of buckets per histogram to 160. Following OpenTelemetry SDK recommendations for base-2 exponential histogram aggregation, this limit protects component memory usage even under extreme outlier distributions.

  3. Broad component support

  Because native histograms are integrated into component-base/metrics, all major Kubernetes control plane and node components inherit support automatically, including:

  • kube-apiserver (e.g., apiserver_request_duration_seconds, authentication/authorization metrics, validation latencies)
  • kube-scheduler (e.g., scheduler_plugin_execution_duration_seconds, scheduler_scheduling_algorithm_duration_seconds)
  • kubelet (node-level container runtime and pod lifecycle metrics)
  • kube-controller-manager and kube-proxy

  How to scrape native histograms

  The simple answer: upgrade to Kubernetes v1.37, and it works.

  Because Kubernetes v1.37 enables NativeHistograms by default, your cluster is already emitting dual-exposition metrics. How you configure Prometheus to scrape native histograms depends on your Prometheus version:

  1. Prometheus scrape configuration by version

  • Prometheus 3.0+ (Recommended): Use explicit per-job configuration in your scrape_configs rather than global flags (the global --enable-feature=native-histograms flag is deprecated in Prometheus 3.9+):

    scrape_configs:
    - job_name:'kubernetes-apiservers'
    scrape_native_histograms:true
    always_scrape_classic_histograms:true# Recommended during transition
    

    You must read the caution in Migrating dashboards and alerts in the Native Histograms documentation. In summary: always set always_scrape_classic_histograms: true during your transition period. Without this setting, Prometheus will only ingest the native format and stop ingesting classic _bucket, _count, and _sum series. Setting always_scrape_classic_histograms: true ensures existing dashboards (histogram_quantile(..._bucket...)) and alerts continue to work while you migrate them to native histograms.

  • Prometheus 2.40 – 2.x: Enable Native Histograms globally by starting Prometheus with the feature flag:

    prometheus --enable-feature=native-histograms
    

    Note that in Prometheus 2.x, this is an all-or-nothing setting for all scrape targets.

  2. Verify Protobuf dual exposition

  Standard Prometheus text scraping (application/openmetrics-text or plain text format) only transfers classic buckets. When scrape_native_histograms is enabled, Prometheus automatically negotiates Protobuf format with Kubernetes endpoints.

  You can verify that a Kubernetes component is exporting native histograms using curl with an Accept header specifying Protobuf. For example:

  ## THIS IS NOT SECURE. ONLY DO THIS IN A TEST CONTEXT.
  curl --insecure \
   -H "Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited" \
   --header "Authorization: Bearer$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   https://localhost:6443/metrics
  

  When decoded, the returned MetricFamily for histogram metrics (like apiserver_request_duration_seconds) will contain both traditional bucket entries and populated schema / positive_span fields.

  Querying native histograms in PromQL

  Once native histograms are ingested into Prometheus, you can query them using standard PromQL histogram functions without needing static le bucket labels or _bucket suffixes:

  # 1. Calculating P99 latency for a single target:
  # Classic histogram (requires _bucket suffix):
  histogram_quantile(0.99,rate(apiserver_request_duration_seconds_bucket[5m]))
  
  # Native histogram (operates directly on the metric name):
  histogram_quantile(0.99,rate(apiserver_request_duration_seconds[5m]))
  
  # 2. Aggregating across multiple instances (e.g., all API servers):
  # Classic histogram (requires sum by (le) to preserve bucket boundaries):
  histogram_quantile(0.99,sumby(le)(rate(apiserver_request_duration_seconds_bucket[5m])))
  
  # Native histogram (no grouping by le required!):
  histogram_quantile(0.99,sum(rate(apiserver_request_duration_seconds[5m])))
  

  With native histograms, functions like histogram_quantile() operate directly on the dynamic exponential spans inside the time series, producing highly accurate quantiles without static bucket interpolation error.

  For official documentation on querying Native Histograms in PromQL, see:

  • PromQL histogram_quantile documentation
  • PromQL histogram_fraction documentation
  • PromQL histogram_sum documentation
  • PromQL histogram_count documentation
  • PromQL histogram_count documentation
  • PromQL histogram_stddev and histogram_stdvar documentation

  Dashboard migration & rollback strategy

  Recommended migration workflow

  To safely transition your monitoring infrastructure to Native Histograms without breaking existing alerts or dashboards, I recommend a four-step migration workflow:

  1. Enable Both Formats: In your Prometheus 3.x scrape config, set scrape_native_histograms: true AND always_scrape_classic_histograms: true so both formats are collected safely during transition
  2. Migrate Queries: Update your Grafana dashboards and Prometheus alerting rules from classic quantile queries (histogram_quantile(..._bucket...)) to native histogram queries (histogram_quantile(...)), and replace references to classic _count and _sum series with histogram_count(...) and histogram_sum(...)
  3. Verify in Staging/Production: Validate that all dashboards and SLO alerts fire and graph correctly using the new native histogram queries
  4. Unlock ~10x Storage Savings: Once migration is complete, set always_scrape_classic_histograms: false. Prometheus will stop ingesting the static _bucket, _count, and _sum time series, reducing your histogram time series count by up to 90%!

  Opt-out and rollback flexibility

  Because native histograms are dual-exposed, using them is entirely opt-in from a collector perspective:

  • Instant Collector Rollback: If you need to stop ingesting native histograms, simply set scrape_native_histograms: false in your Prometheus job configuration. No Kubernetes restart is required, and Prometheus will immediately resume scraping only the classic format without data loss
  • Component Feature Gate Rollback: Administrators can also disable the feature gate on Kubernetes components using --feature-gates=NativeHistograms=false (requires component restart)

  What's next & how to get involved

  As native histograms progress toward General Availability (GA) in future Kubernetes releases, SIG Instrumentation will continue evaluating ecosystem readiness, performance characteristics, and long-term plans for eventually deprecating static classic buckets once native histogram adoption becomes ubiquitous across the monitoring community.

  • Read the KEP-5808 page or the KEP GitHub issue to learn more.
  • Read the Prometheus Native Histograms specification and PromQL querying functions documentation
  • Get involved with SIG Instrumentation on Slack in #sig-instrumentation or join the weekly SIG meetings

  Acknowledgements

  A huge thank you to contributors across SIG Instrumentation and component owners who collaborated on the design, implementation, testing, and review of native histograms in Kubernetes!

• Kubernetes v1.37: Scheduler Preemption for In-Place Pod Resize (Alpha)

  In Kubernetes, resource allocation has historically been a static decision made during a Pod's initial scheduling and placement. With the graduation of the core in-Place Pod resize feature to General Availability in v1.35, application developers and cluster operators gained the powerful ability to dynamically adjust CPU and memory allocations of running containers without incurring disruptive restarts or application downtime.

  However, in-place resizing introduced a unique resource scheduling gap: if a running Pod requested a resource scale-up that exceeded the host node's allocatable headroom, the Kubelet was forced to mark the request as Deferred. The Pod would remain parked in this state indefinitely, waiting for resources on the node to naturally free up.

  To bridge this scheduling gap, Kubernetes v1.37 introduces scheduler preemption for in-place Pod resize (Alpha), behind the InPlacePodVerticalScalingSchedulerPreemption feature gate. This feature allows the Kubernetes scheduler to actively free up capacity on a fully-utilized node by preempting lower-priority workloads, enabling the pending in-place resizes of critical, higher-priority applications to succeed.

  The "deferred" resize challenge

  To understand why this preemption mechanism is needed, it is helpful to look at how Kubernetes handles running Pod resizing. When a user or controller (such as the Vertical Pod Autoscaler) updates the resource requests of an active container, the Kubelet evaluates whether the underlying node has enough spare allocatable capacity to fulfill the increase.

  If the node's resources are fully utilized and cannot satisfy the new limits, the Kubelet sets the container's resizeStatus (reported in the Pod's status.containerStatuses[]) to Deferred. Unlike an Infeasible resize request (which is immediately rejected because it exceeds physical machine boundaries, namespace limit ranges, or admission quotas) a Deferred status indicates that the request is valid but is temporarily unable to be actuated, waiting until node capacity becomes available.

  Before the introduction of this preemption mechanism, a Pod's in-place resize scale-up request could become permanently blocked if the node was heavily utilized. Even when a critical application (such as an in-memory database or a real-time web server) required more memory to prevent an imminent out-of-memory (OOM) crash, and the node lacked free capacity, the resize remained Deferred.

  In this scenario, cluster administrators had limited choices:

  1. Manually evict lower-priority Pods from the node to clear resource headroom.
  2. Rely on the cluster autoscaler to eventually spin up a larger node and reschedule the Pod. However, this is an operation that is highly disruptive and violates the core "no restart" value proposition of in-place scaling.
  3. Rely on a custom autoscaling solution, for example a cluster autoscaler that can trigger dynamic node resizing operations itself.

  Because the kube-scheduler was unaware of deferred resizes on running Pods, it could not leverage standard priority-based preemption to evict lower-priority workloads and make room for the higher-priority running Pod's resource growth.

  Why this matters

  In production Kubernetes environments, cluster administrators strive to maximize resource utilization and efficiency. A common strategy is to bin-pack unused capacity on not-yet-full nodes with lower-priority workloads, such as batch jobs, background data processing, or best-effort tasks.

  Without scheduler preemption for in-place resizing, this created a major operational dilemma. If lower-priority workloads consumed the remaining headroom on a node, higher-priority applications running on that same node would become blocked (Deferred) when they needed to scale up to handle sudden traffic surges or memory spikes. Operators were forced to choose between running low-utilization clusters with idle buffer capacity or risking that critical workloads could not resize when needed.

  With scheduler preemption for in-place Pod resize, you can confidently bin-pack unused space across your clusters with lower-priority workloads without worrying about them degrading higher-priority Pods or blocking their scale-up requests. If a high-priority workload requires an in-place resize that exceeds available node capacity, the scheduler automatically preempts the lower-priority Pods to clear headroom. You achieve high cluster utilization and cost efficiency while preserving the responsiveness and reliability of critical services.

  Architectural mechanics: How it works

  Scheduler preemption for in-place Pod resize integrates directly into the core scheduling cycle to coordinate resources dynamically and safely.

  Centralized scheduler tracking

  The kube-scheduler monitors the cluster for running Pods with a Deferred resize status condition. Normally, Pods with spec.nodeName populated are considered successfully placed and bypass the active scheduling queue. Under this feature gate, the scheduler intercepts Pods carrying the Deferred condition, permitting them to remain in active scheduling evaluations specifically to trigger preemption. The scheduler maintains continuous tracking of these Pods until the Kubelet successfully completes the resize actuation.

  Single-node preemption boundary

  Unlike placement preemption, which evaluates all nodes in a cluster to find the best scheduling fit, preemption for in-place resizing is strictly localized to the Pod's currently assigned node. The scheduler identifies eligible lower-priority "victim" Pods on the same host and initiates their graceful eviction, freeing up local capacity. Preemption is strictly scoped to the same node where the deferred Pod is running; if a node cannot accommodate the resize even after evicting all eligible lower-priority workloads, the resize remains in the Deferred state.

  Resource reservation safety

  To prevent scheduling races and double-allocation, the scheduler treats resources requested for a resize as already consumed. This enables the Kubelet to actuate the resize once the preemption takes effect.

  Separation of concerns & critical admission

  When a node is under resource pressure, the Kubelet includes a local mechanism known as the critical Pod admission handler. During initial Pod admission, if a critical system Pod arrives on a node that lacks spare capacity, this local handler can directly evict lower-priority Pods on that node to guarantee admission for the critical workload.

  A significant architectural benefit of this new feature is the strict separation of concerns between the Kubelet and the scheduler. Under the InPlacePodVerticalScalingSchedulerPreemption feature gate, the Kubelet's critical Pod admission handler does not perform local preemption checks or trigger local evictions for in-place resizing operations. Instead, the Kubelet defers the request and delegates the preemption decision entirely to the scheduler. This guarantees that a single, centralized orchestrator manages all resize-related preemption logic, respecting global priorities, Pod disruption budgets (PDBs), and graceful termination policies.

  Managing competing updates & races

  If a competing, higher-priority resize request is submitted for another running Pod on the same node during an active preemption cycle, the Kubelet prioritizes the higher-priority request. The scheduler is designed to observe these updates and will dynamically trigger a new round of preemption if more capacity is required to fulfill the new state.

  Node-level preemption configuration

  Administrators and automated controllers (such as a cluster autoscaler) can disable preemption specifically for in-place resizes on particular nodes. This is configured using the new spec.podPreemptionPolicy field in the Node Spec:

  apiVersion:v1
  kind:Node
  metadata:
  name:batch-workload-node
  spec:
  podPreemptionPolicy:
  disableResizePreemption:
  - "cluster-autoscaler.kubernetes.io/disable-preemption"
  - "operator.example.com/policy-override"
  

  An example use case for this policy is when a controller would prefer to size down other pods or dynamically adjust the node capacity itself when possible, only enabling scheduler preemption as a last resort.

  Try it out!

  To utilize scheduler preemption for in-place Pod resize:

  • Your cluster must be running Kubernetes v1.37 or later across both the control plane and all worker nodes.
  • The InPlacePodVerticalScalingSchedulerPreemption feature gate must be enabled across all control plane components (kube-apiserver, kube-scheduler) and the kubelet.

  Mini-tutorial: Observe resize preemption in action

  To see this feature in action locally, you can test scheduler preemption on a single-node kind cluster with constrained CPU headroom.

  1. Create a kind cluster with scheduler resize preemption enabled

  Create a kind cluster configuration file named kind-config.yaml with the InPlacePodVerticalScalingSchedulerPreemption feature gate enabled:

  # kind-config.yaml
  kind:Cluster
  apiVersion:kind.x-k8s.io/v1alpha4
  featureGates:
  InPlacePodVerticalScalingSchedulerPreemption:true
  

  Create the cluster using this configuration, passing the --image flag to ensure the cluster is running Kubernetes v1.37 (or later):

  kind create cluster --config kind-config.yaml --image kindest/node:v1.37.0
  

  Note:

  Make sure that the node image you specify corresponds to a Kubernetes v1.37 cluster or later (such as kindest/node:v1.37.0). Older Kubernetes releases do not support the InPlacePodVerticalScalingSchedulerPreemption feature gate.

  Once your cluster is ready, inspect the node to check how many allocatable CPU cores it has:

  kubectl get nodes -o custom-columns=NAME:.metadata.name,ALLOCATABLE_CPU:.status.allocatable.cpu
  

  In a standard local kind environment, the output shows 8 allocatable CPU cores:

  NAME ALLOCATABLE_CPU
  kind-control-plane 8
  

  2. Create PriorityClasses and deploy Pods

  Create two PriorityClasses and deploy a low-priority Pod (requesting 3 CPU) alongside a high-priority Pod (requesting 4 CPU). Together, these workloads consume 7 of the 8 available CPU cores, leaving 1 CPU of free allocatable headroom on the node.

  # preemption-demo.yaml
  apiVersion:scheduling.k8s.io/v1
  kind:PriorityClass
  metadata:
  name:high-priority
  value:1000000
  globalDefault:false
  description:"High priority workload"
  ---
  apiVersion:scheduling.k8s.io/v1
  kind:PriorityClass
  metadata:
  name:low-priority
  value:1000
  globalDefault:false
  description:"Low priority workload"
  ---
  apiVersion:v1
  kind:Pod
  metadata:
  name:low-priority-pod
  spec:
  priorityClassName:low-priority
  containers:
  - name:worker
  image:nginx
  resources:
  requests:
  cpu:"3"
  memory:"500Mi"
  limits:
  cpu:"3"
  memory:"500Mi"
  ---
  apiVersion:v1
  kind:Pod
  metadata:
  name:high-priority-pod
  spec:
  priorityClassName:high-priority
  containers:
  - name:app
  image:nginx
  resources:
  requests:
  cpu:"4"
  memory:"1Gi"
  limits:
  cpu:"4"
  memory:"1Gi"
  

  Save this manifest to preemption-demo.yaml and apply it:

  kubectl apply -f preemption-demo.yaml
  

  Wait until both Pods are running on the node:

  kubectl get pods
  

  Output:

  NAME READY STATUS RESTARTS AGE
  high-priority-pod 1/1 Running 0 9s
  low-priority-pod 1/1 Running 0 9s
  

  3. Request an in-place scale-up

  Patch the high-priority Pod to increase its CPU request from 4 to 6 (+2 CPU delta). Because only 1 CPU of headroom is free on the node, this resize request exceeds remaining allocatable capacity:

  kubectl patch pod high-priority-pod --subresource resize --patch \
   '{"spec":{"containers":[{"name":"app", "resources":{"requests":{"cpu":"6"}, "limits":{"cpu":"6"}}}]}}'
  

  4. Inspect the preemption event on the low-priority Pod

  With InPlacePodVerticalScalingSchedulerPreemption enabled, the scheduler intercepts the Deferred resize condition on high-priority-pod and targets low-priority-pod for preemption.

  To verify that the scheduler actively preempted the low-priority Pod, inspect its events:

  kubectl get events --field-selector involvedObject.name=low-priority-pod
  

  In the event stream (or via kubectl describe pod low-priority-pod), you will see a Preempted event emitted by the scheduler:

  LAST SEEN TYPE REASON OBJECT MESSAGE
  5s Normal Preempted pod/low-priority-pod Preempted by pod 97dba925-6b5f-4e2f-99f9-d51c30016586 on node kind-control-plane
  5s Normal Killing pod/low-priority-pod Stopping container worker
  

  5. Trace the resize event lifecycle on the high-priority Pod

  Next, inspect the event history on high-priority-pod to observe how the resize progressed from being deferred to successfully completed:

  kubectl get events --field-selector involvedObject.name=high-priority-pod
  

  You will observe a sequence of events as the Kubelet coordinates with the scheduler:

  LAST SEEN TYPE REASON OBJECT MESSAGE
  33s Warning ResizeDeferred pod/high-priority-pod Pod resize OutOfcpu: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2,"error":"Node didn't have enough resource: cpu, requested: 6000, used: 3950, capacity: 8000"}
  32s Normal ResizeStarted pod/high-priority-pod Pod resize started: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}
  32s Normal ResizeCompleted pod/high-priority-pod Pod resize completed: {"containers":[{"name":"app","resources":{"limits":{"cpu":"6","memory":"1Gi"},"requests":{"cpu":"6","memory":"1Gi"}}}],"generation":2}
  

  1. ResizeDeferred: The Kubelet initially marks the resize request as deferred (Warning) due to insufficient CPU headroom on the node (OutOfcpu).
  2. ResizeStarted: Once the scheduler preempts low-priority-pod and capacity is released, the Kubelet accepts the new allocation and begins actuating the resize.
  3. ResizeCompleted: The Kubelet successfully updates container cgroup limits via the container runtime without restarting the Pod.

  Finally, verify that the allocated CPU on the container reflects the new request (appending {"\n"} to the JSONPath query ensures a trailing newline in your terminal):

  kubectl get pod high-priority-pod -o jsonpath='{.status.containerStatuses[0].allocatedResources.cpu}{"\n"}'
  

  Output:

  6
  

  This confirms that the in-place resize succeeded.

  Getting involved

  This feature represents a major step forward for resource scheduling, bringing enterprise-grade density control and workload prioritization to dynamic resource scaling. We invite cluster operators, platform architects, and developers to enable the InPlacePodVerticalScalingSchedulerPreemption feature gate in their testing environments and share feedback.

  If you want to share your experience with this feature, please get in touch with the community via SIG Scheduling or SIG Node channels!

• Kubernetes v1.37: Introducing Node Lifecycle Conditions

  Kubernetes has many ways to describe what is happening on a Node. Readiness, taints, Pod state, labels, annotations, and provider-specific APIs each expose part of the picture. What has been missing is a shared, Kubernetes-owned way to say that a Node is draining, undergoing maintenance, or undergoing Graceful Node Shutdown.

  Kubernetes v1.37 introduces five well-known Node conditions that provide that description:

  • DrainInProgress
  • Drained
  • MaintenancePlanned
  • MaintenanceInProgress
  • GracefulNodeShutdownInProgress

  The new Node lifecycle conditions

  Condition	What it reports
  DrainInProgress	The Node is actively being drained according to the administrator's chosen drain criteria.
  Drained	The Node has reached the drain criteria selected by the administrator.
  MaintenancePlanned	The Node is expected to undergo a change in the future.
  MaintenanceInProgress	The Node is actively undergoing maintenance.
  GracefulNodeShutdownInProgress	Graceful Node Shutdown is determined to be in progress on the Node.

  Maintenance can include hardware or software rollout, remediation, decommissioning, or debugging. Whether maintenance requires a drain depends on its impact. A Kubernetes upgrade usually should follow a drain, while a kernel live patch might not need one.

  Like other Node conditions, each lifecycle condition uses status to report whether the observation is active:

  • True: the lifecycle state is currently observed.
  • False: the lifecycle state is not currently observed.
  • Unknown: Kubernetes cannot determine whether the lifecycle state is active.

  The reason provides a stable, machine-readable cause for the current status, and message can provide additional human-readable detail.

  For example, an authorized maintenance controller could publish:

  # Node .status excerpt
  status:
  conditions:
  - type:MaintenancePlanned
  status:"True"
  reason:MaintenanceWindow
  lastTransitionTime:"2026-12-09T12:00:00Z"
  message:"Hardware maintenance is scheduled for this Node"
  

  What changes in Kubernetes v1.37

  The v1.37 release reserves these names as well-known NodeConditionType constants and introduces the Alpha NodeLifecycleConditions feature gate, which is disabled by default. In v1.37 the gate is effectively a no-op: it does not restrict who can set these conditions, and no core component reads them. It exists so that the built-in behavior planned for future releases — controllers that consume these conditions — can be opted into when it arrives. You do not need to enable it to start publishing the conditions today.

  For this release, an administrator or an administrator-authorized controller is responsible for setting and clearing the lifecycle conditions.

  In this first release, no core workload controller changes its behavior based on these conditions, but an administrator can publish them to communicate maintenance and drains to cluster users.

  How to use lifecycle conditions today

  The immediate value is operational clarity. Administrators and lifecycle automation can use these conditions as a common status channel for Node lifecycle work that already happens today.

  For example, maintenance automation can set MaintenancePlanned when a future maintenance window is scheduled, then set MaintenanceInProgress when work starts. Drain automation can set DrainInProgress when it begins evicting Pods and Drained when the administrator's selected drain criteria have been met. The GracefulNodeShutdownInProgress condition can report that Graceful Node Shutdown is in progress on the Node.

  The recommended pattern is to use lifecycle conditions to report status, while lifecycle operations are managed through other mechanisms. Continue to use existing Kubernetes mechanisms such as kubectl cordon, kubectl drain, taints, and workload-specific controls to change scheduling or eviction behavior. Use lifecycle conditions to make the state of that work visible to people, dashboards, alerts, and automation that choose to consume the signal.

  When setting a condition, use True while the lifecycle state is active. Set the condition to False, or remove it, when the state is no longer active. Use a stable reason value and a clear message so that both people and automation can understand why the condition changed. Cluster administrators should also decide which component owns each lifecycle condition to avoid conflicting writes.

  Why a shared signal matters

  Node lifecycle affects components across the cluster. The kubelet, node lifecycle controller, workload controllers, scheduler, autoscalers, storage operators, and external maintenance systems all need some understanding of what is happening to a Node.

  Today, each component has to reconstruct that understanding from indirect signals. One controller might look at Node readiness, another at taints, and another at Pods that are terminating or missing. Infrastructure providers and operators often add their own labels or annotations.

  Those signals remain useful for their intended purposes, but they do not answer the same question. A taint can influence scheduling or eviction, for example, but it does not attest that a drain is in progress or that an administrator's drain criteria have been met. A NotReady Node does not explain whether the cause is an unexpected failure, a graceful shutdown, or planned maintenance.

  Without shared lifecycle context, independently correct components can make conflicting decisions. A DaemonSet controller can replace a Pod that the kubelet intentionally terminated during graceful shutdown. A Job controller can wait indefinitely for a terminal Pod phase on a Node that an administrator is removing. A storage operator might learn about maintenance only after drain has already started.

  The new conditions provide a stable place on the Node for that missing context, as part of the larger effort to enhance Node Lifecycle management.

  The foundation for lifecycle-aware Kubernetes

  The value of a shared signal comes from what can consume it — core controllers, administrators, or the ecosystem of lifecycle projects. Follow-up enhancements can build on the conditions without every component inventing a different way to infer Node lifecycle state.

  Consider a long-standing DaemonSet rollout edge case. A Node that is broken or undergoing maintenance can remain unavailable for reasons unrelated to the new DaemonSet revision. That Node still consumes the rollout's availability budget, which can slow or block the controller from progressing the rollout on healthy Nodes.

  The DaemonSet controller knows that a Pod is unavailable, but it cannot tell whether the new revision failed or an administrator intentionally took the Node out of service. Readiness, taints, and Pod state expose pieces of the situation, but none provides authoritative maintenance context.

  The MaintenanceInProgress condition creates a Kubernetes-owned place to publish that context. Future work can define how the DaemonSet controller uses it for rollout ordering, availability accounting, and status reporting. Those behaviors still require careful design, but the goal is for administrators to no longer have to manually adjust the rollout.

  Future expansions and getting involved

  Node lifecycle is a cross-cutting problem. Solving it starts with components sharing enough context to make compatible decisions. The next stage is to build on Node Lifecycle Conditions to improve scenarios such as Graceful Node Shutdown, drain, and maintenance. Longer-term lifecycle coordination may require explicit ownership, locking, and potentially a dedicated API.

  The Kubernetes ecosystem already includes many solutions for Node maintenance, remediation, drain, autoscaling, and fleet management. The experience behind those projects is essential to building a foundation that works across different environments and operational models. The Node Lifecycle Working Group, SIG Node, and SIG Apps invite maintainers and users to share their use cases and ideas to shape the future work.

  Follow the work through KEP-5683: Node Lifecycle Conditions. To participate in our discussions, join one of our groups:

  • Node Lifecycle Working Group
  • SIG Node
  • SIG Apps

• Kubernetes v1.37: Advancing Workload-Aware Scheduling

  AI/ML and complex batch workloads continue to push the boundaries of Kubernetes scheduling. Following the foundational workload-centric enhancements introduced in previous releases, Kubernetes v1.37 delivers the next major milestone in the Workload-Aware Scheduling (WAS) journey. In this release, the core Workload and PodGroup APIs—enabling gang scheduling—along with Workload-Aware Preemption (WAP) and shared DRA ResourceClaims for PodGroups, all graduate to Beta, solidifying their role in the Kubernetes ecosystem.

  To address the hierarchical scheduling requirements of modern high-performance distributed workloads, v1.37 introduces the new CompositePodGroup API. This new API allows expressing multi-level topology constraints, gang scheduling, and preemption policies for complex, heterogeneous groups of Pods. Crucially, this architectural expansion unlocks native scheduling support for advanced workload structures commonly managed by higher-order extension APIs such as JobSet and LeaderWorkerSet (LWS).

  Alongside these API additions, v1.37 focuses on streamlining adoption by introducing a new set of controller integration APIs and the workloadbuilder Go library. These provide standardized building blocks that significantly simplify how out-of-tree controllers can integrate with WAS capabilities. Utilizing these new tools, the native Job controller integration has been upgraded to fully consume the expanded WAS APIs—enabling advanced scheduling policies, flexible disruption modes, and topology-aware scheduling for standard batch workloads.

  Gang scheduling and Workload / PodGroup APIs

  Kubernetes v1.37 delivers a major milestone: Workload / PodGroup APIs and gang scheduling are officially graduating to Beta. This graduation signals that native, "all-or-nothing" scheduling for workloads is solidifying for wider adoption.

  Key updates to the API and gang scheduling algorithm in this release include:

  Beta graduation and API versioning changes

  The core Workload and PodGroup APIs have been promoted to v1beta1, meaning they are now one step away from General Availability (GA). For early adopters who have been testing these features, take note of the alpha versioning transition: v1alpha2 has been entirely replaced by v1alpha3. This transition introduces breaking changes designed to clean up the API structure around disruptionMode.

  Native PodGroup queueing

  A significant under-the-hood improvement in v1.37 makes the PodGroup a first-class citizen in the scheduling queue. Previously, even if belonging to a PodGroup, all member Pods were queued individually. Now, only the top-level PodGroup object is queued. This ensures all Pods share the same queueing behavior and lays the groundwork for more advanced PodGroup queueing strategies in the future.

  Dynamic elasticity with minCount mutability

  In earlier iterations, the minCount field, which dictates the minimum number of Pods required to successfully schedule a PodGroup, was strictly immutable. In v1.37, minCount is now mutable. This API change unlocks flexibility for elastic workloads. Controllers can now dynamically adjust the minimum required size of a gang on the fly, allowing workloads to gracefully degrade or expand without interrupting already-scheduled Pods.

  Workload-aware preemption

  In Kubernetes v1.37 the separate WorkloadAwarePreemption feature gate for workload-aware preemption was merged into the GenericWorkload feature gate, becoming a core part of the gang scheduling effort.

  While the core concepts of workload-aware preemption stay the same, there are some differences between the v1.36 and v1.37 releases:

  Performance and optimality

  To check whether a preemptor can fit in the cluster thanks to preemption, the scheduler simulates the removal of all potential victims and re-runs the scheduling algorithm. After that it tries to reprieve as many victims as possible. In the v1.36 release, the scheduling algorithm was run for each victim reprieval, verifying whether with the victim reprieved, the algorithm can still find a valid placement for the preemptor. In v1.37, the scheduling algorithm is run only once and the preemptor Pods are assumed based on its output. Later, the reprieval checks whether a victim can still run in its place with the preemptor assumed.

  PodGroup as a victim

  One of the limitations of v1.36 was the fact that the default preemption for single Pods was not aware of PodGroups and was not respecting their disruptionMode fields, allowing for disruption of single Pods even when the PodGroup had disruptionMode: {all: {}} set. Kubernetes v1.37 removes this limitation; the default preemption now respects the PodGroup disruptionMode field.

  Rename of the disruptionMode fields

  During the promotion of the API to Beta, the disruptionMode field was changed to decouple its naming from the PodGroup object, allowing consistent naming across PodGroups and CompositePodGroups. The modes changed as follows: PodGroup became all, and Pod became single.

  Support for preemptionPolicy

  In v1.36, the PodGroup does not have a preemptionPolicy field. The PodGroup can perform preemption as long as none of the Pods forming it has preemptionPolicy: Never set. In v1.37, when the PodGroupPreemptionPolicy feature gate is enabled, a PodGroup also has a preemptionPolicy field. It serves as an authoritative field for whether a PodGroup can perform preemption.

  CompositePodGroup API

  In Kubernetes v1.36, workload-aware scheduling established a clean separation between static workload templates (Workload) and runtime group state (PodGroup), but the supported scheduling policies were limited to a single, flat group. The CompositePodGroup API, introduced in Kubernetes v1.37, extends this model to support hierarchical scheduling requirements.

  This API allows its consumers to express multi-level scheduling requirements by organizing a workload in a tree-shaped hierarchy consisting of CompositePodGroup and PodGroup objects. Each CompositePodGroup carries policies and constraints that apply to other groups (CompositePodGroups and/or PodGroups), similar to how PodGroups govern scheduling behavior for a flat group of Pods. The scheduler treats such a hierarchy as a single scheduling unit and aims to satisfy the requirements specified by every group within that hierarchy.

  Defining a workload hierarchy

  To express multi-level scheduling requirements, you define a hierarchy of templates in a Workload object. Controllers then create the corresponding CompositePodGroup and PodGroup objects from that hierarchy.

  To support this, the Workload API is extended with the spec.compositePodGroupTemplates field. Each CompositePodGroupTemplate defines a template for a parent CompositePodGroup and directly nests the templates (podGroupTemplates and/or compositePodGroupTemplates) from which its child groups derive.

  Below is a sample Workload object that defines a two-level template hierarchy:

  apiVersion:scheduling.k8s.io/v1beta1
  kind:Workload
  metadata:
  name:example-workload
  annotations:
  kubernetes.io/description:"Two-level workload hierarchy requiring 4 worker Pods and 1 driver Pod to schedule together."
  spec:
  compositePodGroupTemplates:
  - name:workload-root
  schedulingPolicy:
  gang:
  minGroupCount:2
  podGroupTemplates:
  - name:workers
  schedulingPolicy:
  gang:
  minCount:4
  - name:driver
  schedulingPolicy:
  gang:
  minCount:1
  

  After creating example-workload, a controller can stamp out the corresponding runtime group objects from these templates:

  1. A root CompositePodGroup that references the workload-root template in example-workload and carries its group-level scheduling policy (gang scheduling with minGroupCount: 2):

     apiVersion:scheduling.k8s.io/v1alpha3
     kind:CompositePodGroup
     metadata:
     name:example-root-group
     annotations:
     kubernetes.io/description:"Root group coordinating gang scheduling across child worker and driver PodGroups."
     spec:
     workloadRef:
     workloadName:example-workload
     templateName:workload-root
     schedulingPolicy:
     gang:
     minGroupCount:2
     

  2. Two child PodGroup objects (example-workload-workers and example-workload-driver) that reference their respective leaf templates in example-workload and link to the root group via parentCompositePodGroupName:

     apiVersion:scheduling.k8s.io/v1beta1
     kind:PodGroup
     metadata:
     name:example-workload-workers
     annotations:
     kubernetes.io/description:"Worker group requiring at least 4 Pods to be scheduled together."
     spec:
     parentCompositePodGroupName:example-root-group
     workloadRef:
     workloadName:example-workload
     templateName:workers
     schedulingPolicy:
     gang:
     minCount:4
     ---
     apiVersion:scheduling.k8s.io/v1beta1
     kind:PodGroup
     metadata:
     name:example-workload-driver
     annotations:
     kubernetes.io/description:"Driver group requiring 1 Pod to schedule alongside the workers."
     spec:
     parentCompositePodGroupName:example-root-group
     workloadRef:
     workloadName:example-workload
     templateName:driver
     schedulingPolicy:
     gang:
     minCount:1
     

  How multi-level gang scheduling works

  To schedule a hierarchical workload, kube-scheduler evaluates the entire group tree as a unified scheduling unit:

  • Recursive evaluation: The scheduler traverses the hierarchy from the root CompositePodGroup down to the leaf PodGroup objects. At each level, a parent CompositePodGroup is considered schedulable only when its child groups satisfy its scheduling policy (for example, placing at least minGroupCount of child groups when using the gang policy), while each leaf PodGroup must satisfy its own Pod-level policy (for example, placing at least minCount of member Pods when using the gang policy).
  • All-or-nothing scheduling: Once a valid combination of child groups is found that satisfies the requirements of the root CompositePodGroup, the Pods across the entire hierarchy are scheduled and bound atomically. If the root group cannot satisfy its policy constraints, the entire hierarchy remains unschedulable and no Pods are bound, preventing partial deployments and deadlocks.

  Workload-aware preemption for the CompositePodGroup API

  Kubernetes v1.37 extends workload-aware preemption to support CompositePodGroup hierarchies as well. Specifically, if a CompositePodGroup cannot be scheduled due to insufficient capacity in the cluster, the scheduler can invoke preemption to evict lower-priority workloads in order to fit the Pods belonging to that CompositePodGroup.

  A CompositePodGroup can be selected for preemption as well. To specify the desired behavior during preemption, workload owners can specify an appropriate disruptionMode in the CompositePodGroup spec:

  • single: Allows individual child groups within the CompositePodGroup to be preempted and disrupted independently. This is the behavior when disruptionMode is not set.
  • all: Enforces "all-or-nothing" disruption semantics across the entire CompositePodGroup hierarchy. If any Pod within the descendant subtree must be preempted, the scheduler evicts all Pods across the entire hierarchy together.

  Topology-aware scheduling

  In Kubernetes v1.37, topology-aware scheduling expands to support complex, multi-level workload hierarchies and delivers performance improvements for existing single-level deployments.

  Multi-level topology-aware scheduling

  In Kubernetes v1.36, we introduced foundational topology-aware scheduling, allowing you to define co-location constraints directly on a PodGroup. While effective for single-level groupings, complex distributed workloads—such as large-scale AI/ML training, JobSet deployments, or disaggregated inference via LeaderWorkerSet (LWS)—often require co-location across multiple levels of cluster infrastructure simultaneously.

  For example, an entire workload may need to run within a single availability zone, while different parts of that workload (such as specific worker groups or driver processes) require strict co-location within specific server racks.

  In Kubernetes v1.37, alongside the new CompositePodGroup API (scheduling.k8s.io/v1alpha3), topology-aware scheduling expands to support multi-level topology-aware scheduling. You can now express complex co-location requirements by specifying topology constraints at different levels of a group hierarchy.

  Top-down topology constraint resolution

  During hierarchical scheduling, the kube-scheduler resolves multi-level topology constraints in a top-down manner. Specifically, topology domains that are considered during the scheduling of a child group are confined within a topology domain that corresponds to the placement assumed by the parent group.

  Configuration and runtime execution

  Using the updated Workload API (scheduling.k8s.io/v1beta1), you can configure multi-level topology constraints directly within compositePodGroupTemplates. In the example below, the parent template constrains the overall workload to a single availability zone (topology.kubernetes.io/zone), while child templates for workers and driver constrain their respective Pods to server racks (topology.example.com/rack) within that selected zone:

  apiVersion:scheduling.k8s.io/v1beta1
  kind:Workload
  metadata:
  name:multi-level-tas-workload
  namespace:job-ns
  annotations:
  kubernetes.io/description:"Workload defining zone-level co-location for the root group and rack-level co-location for child groups."
  spec:
  compositePodGroupTemplates:
  - name:root
  schedulingPolicy:
  gang:
  minGroupCount:2
  schedulingConstraints:
  topology:
  - key:topology.kubernetes.io/zone
  podGroupTemplates:
  - name:workers
  schedulingPolicy:
  gang:
  minCount:8
  schedulingConstraints:
  topology:
  - key:topology.example.com/rack
  - name:driver
  schedulingPolicy:
  gang:
  minCount:1
  schedulingConstraints:
  topology:
  - key:topology.example.com/rack
  

  When a controller creates an instance of this workload at runtime, it spawns the corresponding runtime objects from these templates:

  1. The root CompositePodGroup referencing the root template, carrying the availability zone topology constraint and the hierarchical gang scheduling policy.
  2. The two child PodGroup objects (tas-workload-workers and tas-workload-driver), each referencing the root CompositePodGroup as their parent group via the parentCompositePodGroupName spec field:

  apiVersion:scheduling.k8s.io/v1alpha3
  kind:CompositePodGroup
  metadata:
  name:tas-workload-root
  namespace:job-ns
  annotations:
  kubernetes.io/description:"Root group constraining the entire workload to a single availability zone."
  spec:
  workloadRef:
  workloadName:multi-level-tas-workload
  templateName:root
  schedulingPolicy:
  gang:
  minGroupCount:2
  schedulingConstraints:
  topology:
  - key:topology.kubernetes.io/zone
  ---
  apiVersion:scheduling.k8s.io/v1beta1
  kind:PodGroup
  metadata:
  name:tas-workload-workers
  namespace:job-ns
  annotations:
  kubernetes.io/description:"Worker group requiring 8 Pods co-located within a single rack in the selected zone."
  spec:
  parentCompositePodGroupName:tas-workload-root
  workloadRef:
  workloadName:multi-level-tas-workload
  templateName:workers
  schedulingPolicy:
  gang:
  minCount:8
  schedulingConstraints:
  topology:
  - key:topology.example.com/rack
  ---
  apiVersion:scheduling.k8s.io/v1beta1
  kind:PodGroup
  metadata:
  name:tas-workload-driver
  namespace:job-ns
  annotations:
  kubernetes.io/description:"Driver group requiring 1 Pod placed in a rack within the selected zone."
  spec:
  parentCompositePodGroupName:tas-workload-root
  workloadRef:
  workloadName:multi-level-tas-workload
  templateName:driver
  schedulingPolicy:
  gang:
  minCount:1
  schedulingConstraints:
  topology:
  - key:topology.example.com/rack
  

  During scheduling, the scheduler evaluates multiple candidate availability zones across the cluster for tas-workload-root. For each candidate zone, it subdivides the nodes by rack topology to explore feasible rack placements for tas-workload-workers and tas-workload-driver strictly within that zone, systematically evaluating multiple combinations across available zones and racks before making a scheduling decision.

  By allowing topology constraints to be modeled hierarchically, Kubernetes v1.37 provides a structured way to express multi-level co-location requirements across complex cluster infrastructures.

  Performance improvements for single-level TAS

  Alongside the Alpha introduction of multi-level hierarchies, Kubernetes v1.37 reduces the cost of placement evaluation for existing single-level topology-aware scheduling. We are continuously working to optimize the efficiency of placement evaluation algorithms in kube-scheduler and plan to deliver further performance improvements in future releases.

  Controller Integration APIs

  Kubernetes v1.37 introduces new standard building blocks so that every controller can expose the same scheduling primitives in their own APIs, and share the same logic for translating them into scheduling objects. These primitives express specific scheduling behaviors — such as policies or disruption logic — while leaving the field naming flexible for each controller. A prime example of this is the native Job controller, which we detail in the next section.

  Types prefixed with WorkloadPodGroup describe a leaf group of Pods; types prefixed with WorkloadCompositePodGroup describe a group of groups. A controller embeds them verbatim into its own API, under whatever field name fits its domain:

  • WorkloadPodGroupSchedulingPolicy — either basic, meaning standard Pod-by-Pod scheduling, or gang with a minCount. The composite variant takes a minGroupCount instead.
  • WorkloadPodGroupSchedulingConstraints — the topology constraints (topology[].key) the group's Pods must be co-located within.
  • WorkloadPodGroupDisruptionMode — single or all, with the preemption semantics described earlier in this post.
  • WorkloadPodGroupResourceClaim — the ResourceClaims shared across the group.

  Only the shapes are shared, so controllers retain full autonomy over how they name and nest these fields in their own APIs.

  The workloadbuilder library turns that intent into the scheduling objects. A controller describes its workload as a tree of WorkloadItem nodes — a node with children compiles to a CompositePodGroupTemplate, a node without children to a PodGroupTemplate — and attaches its own defaults plus the user-supplied building blocks to each node. From there, Validate() reports problems back at the exact field path within the controller's own API, BuildWorkload() compiles the tree into a Workload, and NewPodGroup() and NewCompositePodGroup() stamp out the runtime group objects.

  Validation is deny-by-default: a controller declares the policies and disruption modes it actually supports through AllowedPolicies and AllowedDisruptionModes, and anything outside those lists is rejected. Building blocks added in future releases therefore stay unavailable until a controller explicitly opts into them.

  For hierarchical workloads where a parent controller owns the Workload and delegates group creation to its children, NewBuilderFromExistingWorkload lets a child materialize only its own PodGroup from the parent's Workload.

  Neither the building blocks nor the library have a feature gate of their own; they become user-visible through whichever controller adopts them. The native Job controller is the first to do so, and we detail it in the next section.

  Integration with the Job controller

  Building upon the new controller integration APIs, the Job API now features an explicit .spec.scheduling field, so you can declare how a Job should be scheduled instead of relying on the Job controller to infer it from the Job's shape. This expands support well beyond static, indexed, and fully-parallel Jobs.

  .spec.scheduling is composed of the building blocks described above:

  • schedulingPolicy — basic for standard Pod-by-Pod scheduling, or gang for all-or-nothing scheduling.
  • schedulingConstraints — the topology domain the Job's Pods must be co-located within.
  • disruptionMode — whether the Job's Pods can be preempted individually (single) or only as a whole (all).
  • resourceClaims — the ResourceClaims shared by all of the Job's Pods.

  For example:

  apiVersion:batch/v1
  kind:Job
  metadata:
  name:distributed-training-job
  annotations:
  kubernetes.io/description:"Distributed Job using explicit WAS scheduling with gang policy and zone topology constraints."
  spec:
  parallelism:8
  completions:8
  scheduling:
  schedulingPolicy:
  gang:{}# minCount omitted → defaults to parallelism (8)
  schedulingConstraints:
  topology:
  - key:topology.kubernetes.io/zone
  disruptionMode:
  all:{}
  template:
  spec:
  containers:
  ...
  

  Omitting .spec.scheduling, or omitting schedulingPolicy within it, selects the basic policy, which behaves exactly like standard Job scheduling today.

  For every Job it manages, the controller compiles this configuration into a Workload and a PodGroup owned by the Job, and sets .spec.schedulingGroup.podGroupName on each Pod it creates so the scheduler treats them as one group. Once created, .spec.scheduling is immutable, with one exception: schedulingPolicy.gang.minCount can be updated, which lets you resize a running gang.

  DRA ResourceClaim support for workloads

  As the core WAS APIs mature, so do their integrations with Dynamic Resource Allocation (DRA). Kubernetes v1.36 introduced the DRAWorkloadResourceClaims feature gate. The associated feature allows ResourceClaims to be replicated and reserved for entire PodGroups and shared by all their member Pods:

  apiVersion:scheduling.k8s.io/v1beta1
  kind:PodGroup
  metadata:
  name:training-job-workers-pg
  spec:
  ...
  resourceClaims:
  - name:pg-claim
  resourceClaimTemplateName:my-claim-template
  ---
  apiVersion:v1
  kind:Pod
  metadata:
  name:topology-aware-workers-pg-pod
  spec:
  ...
  schedulingGroup:
  podGroupName:training-job-workers-pg
  resourceClaims:
  - name:pg-claim
  resourceClaimTemplateName:my-claim-template
  

  In Kubernetes v1.37, the DRAWorkloadResourceClaims feature gate graduated to Beta.

  While the API and core functionality of the feature remain unchanged, one change eliminates some potentially surprising behavior when disabling the feature. Previously when one of a Pod's spec.resourceClaims referenced a ResourceClaimTemplate and matched one of its PodGroup's spec.resourceClaims and the DRAWorkloadResourceClaims feature gate was disabled, a ResourceClaim was created for the Pod instead of the PodGroup. In that scenario in v1.37, no ResourceClaim is created at all. This change prevents Kubernetes from creating a flood of ResourceClaims from a ResourceClaimTemplate and potentially exhausting DRA resources when a claim intended to be shared by a whole PodGroup is replicated for each and every Pod in the group.

  For more information, see the feature documentation.

  What's next?

  The Workload-Aware Scheduling Working Group (WG WAS) is currently finalizing its plans for the Kubernetes v1.38 release cycle. While the roadmap is still taking shape (stay tuned!), the following key initiatives are already planned:

  • Graduation of Workload and PodGroup APIs to GA: Solidifying the core foundation of workload-aware scheduling as a stable Kubernetes API.
  • Graduation of Topology-Aware Scheduling (TAS) and CompositePodGroup (CPG) to Beta: Bringing these advanced placement and hierarchical scheduling features to Beta stability.
  • Graduation of controller integration building blocks to Beta: Further refining the integration APIs to ensure a robust developer experience.
  • Increased adoption and integration: Expanding the ecosystem by integrating workload-aware scheduling with other controllers, with a particular focus on hierarchical orchestrators such as JobSet.
  • Kueue Integration: Fostering closer alignment between WAS and Kueue. In the near term, we aim to ensure Kueue is fully aware of WAS features for seamless interoperability. In the long term, we envision Kueue leveraging WAS as its underlying engine for capabilities like gang-scheduling and topology-aware placement.

  Getting started

  Many of the workload-aware scheduling improvements are now available as Beta features in v1.37, while new advanced capabilities are introduced in Alpha. Both Beta and Alpha features here are disabled by default and require manual enablement.

  Beta features:

  • Workload API, gang scheduling, and preemption: The GenericWorkload feature gate (which now integrates gang scheduling and workload-aware preemption) is Beta and disabled by default on the kube-apiserver, kube-controller-manager and kube-scheduler. Ensure your manifests are updated to use the scheduling.k8s.io/v1beta1 API group.
  • DRA ResourceClaim support for workloads: Enable the DRAWorkloadResourceClaims feature gate on the kube-apiserver, kube-controller-manager, kube-scheduler and kubelet.

  Alpha features:

  • Topology-aware scheduling: Enable the TopologyAwareWorkloadScheduling feature gate on the kube-apiserver and kube-scheduler.

  • CompositePodGroup API: Enable the CompositePodGroup feature gate on the kube-apiserver, kube-controller-manager and kube-scheduler, and ensure the scheduling.k8s.io/v1alpha3 API version is enabled. Note that enabling CompositePodGroup on the kube-controller-manager also requires the TopologyAwareWorkloadScheduling feature gate to be enabled.

  • Workload API integration with the Job controller: Enable the WorkloadWithJob feature gate on the kube-apiserver and kube-controller-manager.

  • PodGroup preemptionPolicy: Enable the PodGroupPreemptionPolicy feature gate on the kube-apiserver and kube-scheduler.

  Controller integration APIs:

  The new workloadbuilder library is available to developers building both out-of-tree and in-tree controllers who want to integrate with WAS. It does not require a feature gate. You can explore the library and find usage examples directly in the kubernetes/component-helpers repository.

  We encourage you to try out workload-aware scheduling in your test clusters and share your experiences to help shape the future of Kubernetes scheduling. You can send your feedback by:

  • Reaching out via Slack (#wg-workload-aware-scheduling).
  • Joining the WG Workload-Aware Scheduling or SIG Scheduling meetings.
  • Filing a new issue in the Kubernetes repository.

  Learn more

  To dive deeper into the architecture and design of these features, read the KEPs:

  • KEP-4671: Gang Scheduling Support in Kubernetes
  • KEP-5710: Workload-aware preemption
  • KEP-5732: Topology-aware workload scheduling
  • KEP-6012: CompositePodGroup API
  • KEP-6089: WAS: Controller Integration APIs
  • KEP-5547: WAS: Integrate Workload APIs with Job controller
  • KEP-5729: DRA: ResourceClaim Support for Workloads

• Kubernetes v1.37: KubeletInUserNamespace (aka Rootless mode) Graduates to Beta

  Kubernetes v1.37 promotes the KubeletInUserNamespace feature gate to beta. With this feature enabled, all of the node components (kubelet, CRI and OCI runtimes, CNI plugins, and kube-proxy) can run as a non-root user on the host, using a Linux user namespace. This technique is also known as rootless mode. The work started as an experiment in 2018, and was merged into Kubernetes v1.22 (2021) as an alpha feature (Kubernetes Enhancement Proposal KEP-2033).

  This feature should not be confused with user namespaces for pods (hostUsers: false with the UserNamespacesSupport feature gate, GA since v1.36), which puts pods in user namespaces but still runs the node components as root. These two features do not conflict. Moreover, they can be combined to nest Kubernetes inside Kubernetes without resorting to the full privileged: true.

  Why run the node components in a user namespace?

  Because the node components have historically had container-breakout vulnerabilities that could compromise full root privileges on the host.

  Examples of such vulnerabilities include:

  • CVE-2022-0811 ("cr8escape"): CRI-O could be tricked into setting arbitrary sysctls, such as kernel.core_pattern, resulting in arbitrary code execution as root on the host
  • CVE-2023-27561: runc could be tricked into bypassing the masked paths of a container via a volume mount race, exposing the host's procfs files (a regression of CVE-2019-19921)
  • CVE-2024-10220: the kubelet could be made to execute arbitrary commands as root via gitRepo volumes (gitRepo volumes had a similar vulnerability, CVE-2018-11235, back in 2018 too)
  • CVE-2025-31133: runc could be tricked into bind-mounting attacker-controlled paths and writing to the host's procfs files, such as /proc/sysrq-trigger and /proc/sys/kernel/core_pattern
  • CVE-2026-53488: containerd could be tricked into executing arbitrary commands on the host, via crafted labels in a container image

  By running the node components in a user namespace, the potential damage is confined to the non-root user's account. Notably, an attacker cannot conceal their intrusion by modifying the kernel, the boot loader, or the firmware.

  It should still be noted that user namespaces are not effective for mitigating vulnerabilities in the kernel itself. User namespaces should be used in conjunction with traditional hardening measures such as seccomp to prevent containers from invoking unnecessary system calls.

  Use cases

  • Production clusters: mitigate potential container-breakout vulnerabilities.
  • Shared machines (e.g., HPC): users can deploy Kubernetes without asking the machine administrator for root privileges, and without the risk of accidentally breaking other users' environments.
  • Laptops: prevent a local cluster from accidentally breaking the host system configuration, e.g., the host iptables rules used for VPNs.
  • AI sandbox: a Kubernetes application developer may create a dedicated local user account for running an AI coding agent and a test Kubernetes cluster. This setup is useful for preventing the AI agent from breaking the host when it is deceived by malicious information on the Internet.
  • Kubernetes-in-Kubernetes: a nested cluster can run inside a parent cluster as a user-namespaced pod (hostUsers: false), isolating workloads more strictly than Kubernetes API namespaces do.
  • Bootstrapping: a temporary unprivileged cluster can be used to bootstrap an actual cluster, e.g., with Cluster API.

  How does it work?

  A Linux kernel user namespace maps a host level non-root user (e.g., UID 1000) to a fake root user inside the namespace. The UID 0 privileges are limited to the inside of the namespace. The fake root is enough for most of the node components' tasks: mounting volumes, creating cgroups, and configuring the network namespaces of pods. It still comes with some caveats that may break compatibility with specific CNI and CSI drivers, though.

  The user namespace has to be created outside of Kubernetes. For example, Rootless Docker can be used to prepare the user namespace in which Kubernetes runs.

  The KubeletInUserNamespace feature gate itself is quite "boring": basically it just lets the kubelet ignore permission errors that occur when setting some sysctl values (e.g., vm.overcommit_memory and kernel.panic) and when watching kernel messages via /dev/kmsg.

  See Running Kubernetes Node Components as a Non-root User for further information.

  What changed from Alpha to Beta?

  • The KubeletInUserNamespace feature gate is now enabled by default. Enabling the gate does not put the kubelet into a user namespace automatically, so nothing changes for existing "rootful" clusters.
  • kubectl get nodes -o yaml now reports whether nodes are running in a user namespace via the runningInUserNamespace property. A cluster administrator can use this property to set node labels or taints, to avoid scheduling workloads that need real root privileges (e.g., some CNI plugin installers) onto rootless nodes.
  • For Kubernetes' own CI/CD testing, the node conformance end to end tests now run on a rootless cluster (ci-kubernetes-e2e-kind-rootless).

  Several related improvements have also happened outside the promotion of the feature gate itself:

  • Linux kernel v6.3 (2023): added support for idmapped tmpfs.
  • Kubernetes v1.33 (2025): enabled the UserNamespacesSupport feature gate by default, allowing user-namespaced pods (hostUsers: false) to be created without extra configuration.
  • containerd v2.1 (2025): added support for writable cgroups.

  With these improvements, a Kubernetes cluster with KubeletInUserNamespace can now also be nested inside Kubernetes pods with hostUsers: false (UserNamespacesSupport).

  How to use it

  kind

  The easiest way is to use kind (a Kubernetes SIG Testing project) to run a Kubernetes cluster in rootless Docker, rootless nerdctl, or rootless Podman:

  # Example using Docker
  dockerd-rootless-setuptool.sh install
  kind create cluster
  

  Depending on the host configuration, you may need additional configuration for systemd, kernel modules, sysctl, etc.

  See the Docker documentation and the kind documentation for further information.

  minikube

  minikube (a Kubernetes SIG Cluster Lifecycle project) also supports running a Kubernetes cluster in rootless Docker or rootless Podman:

  dockerd-rootless-setuptool.sh install
  minikube start --driver=docker
  

  See the minikube documentation for further information.

  Usernetes

  Usernetes (a third-party project) is a distribution of rootless Kubernetes, maintained by the author of this article. The project began in 2018, and it is where the KubeletInUserNamespace feature gate originally came from.

  Unlike kind and minikube, Usernetes supports creating a cluster with multiple rootless Docker / Podman / nerdctl nodes, connected using VXLAN via the Flannel CNI plugin.

  Usernetes also experimentally supports a Kubernetes-in-Kubernetes mode.

  k3s

  k3s (a CNCF Sandbox project) also supports rootless mode. Unlike kind, minikube, and the current generation of Usernetes, rootless k3s does not rely on an external runtime such as rootless Docker.

  What's next?

  Depending on feedback and adoption, the Kubernetes project plans to graduate this feature to General Availability (GA) in a future release. If you have feedback on this feature, please open an issue in the kubernetes/kubernetes repository.

  The project is also discussing several Kubernetes Enhancement Proposals that may contribute to simplifying Kubernetes-in-Kubernetes with this feature:

  • KEP-5474: Enable Writable cgroups for unprivileged containers
  • KEP-5714: Allow specifying whether to unshare cgroup namespaces

  Getting involved

  We always welcome new contributors. If you would like to get involved, you can join the Node Special Interest Group (SIG Node).

  If you would like to share feedback, you can do so on our public Slack channel (visit https://slack.k8s.io/ for an invitation if you need one).

  Special thanks to everyone who helped design and implement this feature, including but not limited to (in alphabetical order):

  • Bing Hongtao (HirazawaUi)
  • Jordan Liggitt (liggitt)
  • Sergey Kanzhelev (SergeyKanzhelev)
  • Tim Hockin (thockin)