KubeCon + CloudNativeCon Europe 2025

Deploying large language models in Kubernetes environments faces a critical challenge: the cold start problem.When auto-scaling workloads with tools like Knative, the latency from loading large model weights into GPU memory slows response times, degrades performance, and increases costs.Traditional methods rely on loading weights sequentially into CPU memory then to the GPU,which is slow and inefficient.This talk introduces Run:ai Model Streamer, an open-source tool that mitigates cold starts by streaming model weights to GPU memory while reading them from storage in parallel.It integrates seamlessly into inference engine containers and Kubernetes workflows, enabling parallelized weight streaming without modifying weight formats, making it an easy-to-adopt solution for Kubernetes-based AI deployments.We’ll share benchmarking results comparing storage backends like GP3 SSDs, IO2 SSDs, and S3, highlighting performance improvements, cost savings, and best practices from these experiments.