Deploying a large language model on Kubernetes unlocks elastic scaling, high availability, and operational consistency for production-grade AI applications. Whether you are serving a fine-tuned Llama 3 model or a massive Mixtral 8x7B, Kubernetes provides the orchestration layer to manage GPU resources, handle rolling updates, and route inference traffic intelligently.
However, the process involves more than just running a Docker container. You need to prepare a GPU-enabled cluster, containerize the model server, define resource limits, and set up autoscaling that reacts to real-time demand. This guide walks you through the entire workflow with concrete steps and a real-world scenario, showing how to keep costs under control while delivering low-latency predictions.
Preparing Your Kubernetes Cluster for LLM Workloads
Before launching any inference pod, your cluster must be equipped with NVIDIA GPU nodes and the necessary device plugins. Most managed Kubernetes services—such as GKE, EKS, or AKS—offer GPU-accelerated node pools out of the box. When setting up, ensure the nvidia-device-plugin DaemonSet is deployed so pods can request resources like nvidia.com/gpu: 1.
Network-attached storage is critical for model weights that can exceed 100 GB. Solutions like ReadWriteMany PVCs backed by Filestore or EFS allow multiple pods to share the same model cache, reducing download times. Consider also the node taints and tolerations: GPU nodes should be tainted with nvidia.com/gpu to prevent non-GPU workloads from being scheduled there, and your inference pods need the matching toleration.
Finally, enable cluster autoscaling to add GPU nodes during spikes. For cost awareness, define node pool labels to distinguish on-demand from spot/preemptible GPUs; spot nodes can slash compute costs by up to 60% for batch or non-critical inference.
Deploying an LLM Inference Service Step by Step
Assume you need to serve a 13-billion-parameter model for real-time customer support summarization. A typical approach uses vLLM inside a Kubernetes Deployment with an HPA that scales on inference requests per second.
Step 1 – Containerize the model server. Write a Dockerfile that downloads the model from Hugging Face or a cloud storage bucket into a folder, installs vLLM, and sets the entry point. Use a multi-stage build to separate model weights from the inference code, so the final image is not bloated. The model weights can be mounted via a PersistentVolume to avoid rebuilding the image when you switch models.
Step 2 – Define a Deployment. Create a YAML manifest for a Deployment with one or more replicas. Set resource requests and limits: for a 13B model, 1×A100-40GB is usually sufficient for a small number of concurrent requests. Specify the GPU resource request (nvidia.com/gpu: 1) and a toleration for GPU taints. Mount the PVC containing model weights at /models.
Step 3 – Create a Service and configure scaling. Expose the inference endpoint as a ClusterIP Service, then create an HPA that targets average CPU or custom metrics like llm_requests_per_second. During high traffic, the HPA spins up additional pods; when demand drops, it scales down. Combined with cluster autoscaling, the overall infrastructure flexes with load.
Step 4 – Add monitoring. Integrate Prometheus to track GPU memory usage, inference latency, and request throughput. A Grafana dashboard can alert when GPU utilization exceeds 90% for sustained periods, prompting manual or automatic scaling adjustments.
Concrete Scenario Example
A fintech startup needed to deploy a 13B LLM on GKE for on-the-fly fraud detection message summaries. They provisioned a mix of A100 nodes: a fixed on-demand node pool for the baseline serving replicas and a spot-instance pool for burst capacity. Using vLLM’s PagedAttention and the HPA on a custom latency metric, they maintained P95 latency under 800 ms while handling 200 requests per second. During off-peak hours, the HPA scaled down to a single replica, and the unused spot nodes were terminated automatically. By working with CnCloud’s cost optimization service—which right-sized their persistent volumes and applied reseller discounts—the team achieved up to 30% savings on their monthly cloud bill compared to their initial over-provisioned configuration. The entire deployment was set up with a single kustomize overlay, making it easy to replicate across development and production environments.
Cost-Saving Strategies for Production LLM Deployments
Running LLMs on Kubernetes can be expensive, but deliberate planning keeps costs in check. First, leverage mixed node pools: always keep one node pool with guaranteed on-demand GPUs for minimum availability, and another with spot/preemptible GPUs for excess capacity. The HPA should scale up only to the spot pool when demand exceeds the on-demand baseline.
Second, tune the resource requests. Overly generous memory requests waste GPU hours; profile your model’s actual peak memory and set requests close to that value with a small buffer. Enable GPU sharing if your inference server supports it (e.g., vLLM’s tensor parallelism across processes on the same GPU), which can increase throughput per GPU.
Third, incorporate a startup probe with a long initial delay to avoid premature killing of pods that are downloading or loading models. This prevents wasteful pod restarts that would consume extra GPU time. Finally, use Kubernetes cost allocation tools (like kube-cost or GCP cost allocation tags) to track spending per team or model, turning cloud costs into an engineering metric rather than a surprise.
Conclusion
Running an LLM on Kubernetes gives you the elasticity and reliability modern AI applications demand. By following a structured approach—preparing a GPU cluster, containerizing the inference server, setting up autoscaling, and applying cost controls—you can move from a prototype to a resilient production service. The fintech scenario illustrates how the right architectural choices, combined with expert cost governance, can save significant amounts while keeping performance high. Whether you are on GCP, AWS, or any other cloud, the principles remain the same; refined resource management and smart scaling make LLM deployments both scalable and affordable.