KubeCon + CloudNativeCon Europe 2025

While there are performance and security reasons for operating Spark from bare metal Apache Hadoop clusters, cloud-based installations using Kubernetes as the cluster manager are becoming more and more mainstream due to superior scalability, flexibility, and cost-effectiveness for variable workloads. However, the migration of Spark from bare metal clusters to a cloud-based cluster environment poses a number of non-trivial challenges from a technical as well as from a human/organizational perspective. Specifically, these challenges include but are not limited to dealing with difficulties in achieving query performance parity, differences in scheduling and resource management, security in a multi-tenancy context, and the provisioning of sufficient introspection for aiding diagnostics and configuration adjustments. This case study recounts challenges encountered and solutions implemented while migrating Spark from bare metal to Kubernetes managed cloud in a large corporate environment.