KubeCon + CloudNativeCon Europe 2025

Large Language Models are increasing in popularity and the training performance in Kubernetes at scale has become the biggest challenges for enterprises. How to achieve the optimal performance and linearity for a huge training job, such as 100k GPUs? What are the three most critical factors that affect performance? How to optimize performance step by step?

In this talk we will present an end to end analysis of the bottleneck of LLM training in Kubernetes at scale. And then show how the insufficient resource management and network topology awareness in Kubernetes affect the performance. Finally we will introduce the new resource management model, LLM dedicated training workload and scheduling solution which are initiated in the Volcano open source community and demonstrate how to use it to get optimal performance and linearity.