KubeCon + CloudNativeCon Europe 2025

With the development of large model technology, industry-leading large models now have the capability to train at a scale of up to 100,000 GPUs. This scale often exceeds the capacity limits of a single K8s cluster. A feasible solution is to adopt a multi-K8s cluster joint training approach.
To achieve multi-K8s cluster joint training, two key challenges need to be addressed: adapting single K8s cluster training tasks to run in a multi-K8s cluster environment, and ensuring the synchronization and efficient transmission of training parameters and checkpoint data across clusters.
In this presentation, we will share China Mobile’s practical experience in achieving parallel training on cross-region multi-K8s clusters, utilizing over 10,000 GPUs with Kubeflow’s Training Operator and VolcanoJob, with no modifications required. Additionally, we will introduce optimized methods to accelerate cross-region data synchronization during training.