Using NVIDIA Run:ai GPU Fractioning on Amazon EKS
1 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · chưa có số tương tác · Nguồn crawl
Chi tiết nguồn
<p>GPU sharing is not new: Multi-Instance GPU (MIG), time-slicing, and Multi-Process Service (MPS) address different isolation and utilization needs. NVIDIA Run:ai GPU Fractioning builds on MPS with Kubernetes-integrated scheduling, per-Pod memory contracts, enforcement, and metrics.</p> <p><a href="https://github.com/dsx-ai-factory/run-ai" target="_blank" rel="noopener">NVIDIA Run:ai</a> is AI workload orchestration software for Kubernetes. Its GPU Fractioning Operator can help several teams share one or two GPUs on <a href="https://aws.amazon.com/eks/" target="_blank" rel="noopener">Amazon Elastic Kubernetes Service (Amazon EKS)</a>. Notebooks, model evaluations, and small inference services can leave teams waiting even when GPU memory remains available. In this walkthrough’s baseline, the NVIDIA device plugin exposes the single GPU as one <code>nvidia.com/gpu</code> resource, with GPU sharing disabled. Two Pods each requesting <code>nvidia.com/gpu: 1</code> cannot run concurrently on that node, even when their combined GPU-memory demand fits.</p> <p><a href="https://github.com/kai-scheduler/gpu-fractioning/" target="_blank" rel="noopener">NVIDIA Run:ai GPU Fractioning</a> adds Kubernetes scheduling integration, enforced GPU-memory limits, and per-Pod metrics to software-based sharing. The approach applies to NVIDIA GPUs that meet the GPU Fractioning Operator’s hardware and software requirements, including GPUs without MIG. We use one <a href="https://aws.amazon.com/ec2/" target="_blank" rel="noopener">Amazon Elastic Compute Cloud (Amazon EC2)</a> g5.xlarge instance with an NVIDIA A10G Tensor Core GPU as the validated example. Other configurations require compatibility checks and validation.</p> <p>NVIDIA <a href="https://developer.nvidia.com/blog/nvidia-runai-is-now-open-source-as-part-of-nvidia-dsx-ai-factory-platform/" target="_blank" rel="noopener">announced</a> the <a href="https://github.com/dsx-ai-factory/run-ai" target="_blank" rel="noopener">open source release of NVIDIA Run:ai</a> on October 13, 2026. This example uses two of its four modular components: <a href="https://github.com/kai-scheduler/KAI-Scheduler" target="_blank" rel="noopener">KAI Scheduler</a> and <a href="https://github.com/kai-scheduler/gpu-fractioning/" target="_blank" rel="noopener">GPU Fractioning Operator</a>. KAI Resource Management and Workload Map address organizational governance and additional workload types, outside this walkthrough’s scope.</p> <p>In this post, we show how to build a custom <a href="https://aws.amazon.com/linux/amazon-linux-2023/" target="_blank" rel="noopener">Amazon Linux 2023 (AL2023)</a> image and configure GPU-memory sharing on Amazon EKS. In the validated G5/A10G example, you establish a whole-GPU baseline, give each Pod an 8 GiB memory contract, and verify 6 GiB allocations. You then inspect per-Pod metrics, test rejection of a 9 GiB allocation, and clean up the disposable environment.</p> <h2 id="choose-a-gpu-sharing-approach">Choose a GPU sharing approach</h2> <p>MIG provides hardware partitions on <a href="https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-gpus.html" target="_blank" rel="noopener">supported GPUs</a>. <a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html" target="_blank" rel="noopener">Device-plugin time-slicing</a> shares a GPU without replica memory or fault isolation. MPS permits concurrent CUDA processes. Fault containment depends on configuration. NVIDIA Run:ai GPU Fractioning adds enforced per-container memory limits and supports time-slicing (default) or sm-sharing compute modes. <a href="https://developer.nvidia.com/blog/memory-enforced-gpu-sharing-in-kubernetes-with-nvidia-cuda-13-4/" target="_blank" rel="noopener">NVIDIA documents</a> separate workload error domains for its default mode and a shared error domain for sm-sharing. These are not security guarantees. This walkthrough tests memory limits, not fault containment or compute performance. A memory limit does not reserve proportional compute.</p> <p>Amazon EKS documents several GPU sharing options: <a href="https://docs.aws.amazon.com/eks/latest/userguide/device-management-nvidia-mig.html" target="_blank" rel="noopener">MIG</a>, time-slicing with the <a href="https://docs.aws.amazon.com/eks/latest/userguide/device-management-nvidia-time-slicing.html#eks-time-slicing-device-plugin-al2023" target="_blank" rel="noopener">NVIDIA device plugin</a>, and time-slicing with the <a href="https://docs.aws.amazon.com/eks/latest/userguide/device-management-nvidia-time-slicing.html#eks-time-slicing-karpenter-consumable-capacity" target="_blank" rel="noopener">NVIDIA DRA driver and Karpenter</a>. For guidance on choosing among them, see the <a href="https://docs.aws.amazon.com/eks/latest/best-practices/aiml-compute.html#_optimize_gpu_resource_allocation_with_time_slicing_mig_and_fractional_gpu_allocation" target="_blank" rel="noopener">GPU sharing guidance</a> in the Amazon EKS Best Practices Guide. Choose GPU fractioning when you need enforced per-Pod GPU-memory limits on GPUs without MIG and can run fixed-size GPU node groups.</p> <p><a href="https://docs.aws.amazon.com/eks/latest/userguide/device-management.html" target="_blank" rel="noopener">Dynamic Resource Allocation (DRA)</a> provides Kubernetes APIs for requesting and allocating devices. GPU sharing and isolation depend on the device driver’s implementation and enabled features. KAI Scheduler and GPU Operator support DRA. This walkthrough uses NvFractions annotations, not DRA.</p> <p>GPU fractioning can help interactive notebooks with idle periods, short evaluation jobs, GPU compatibility tests, and small inference services. Choose each GPU-memory request from measured peak usage plus overhead, then test latency under concurrent load. The demonstrated density improvement does not establish a throughput gain or a universal cost reduction.</p> <h2 id="solution-overview">Solution overview</h2> <p
- Using NVIDIA Run:ai GPU Fractioning on Amazon EKS AWS Machine Learning Blog · 19 thg 11, 2026