KubeCon + CloudNativeCon Europe 2025

Existing solutions for serving AI models in Kubernetes are often difficult to deploy and manage with complex workflows and a lack of user-friendly design. This talk introduces a custom container runtime that leverages Ollama as the serving backend, simplifying the deployment and operation of AI models in Kubernetes environments.

A custom container runtime extends the standard container execution workflow by integrating additional capabilities directly into the container lifecycle. Solutions like gVisor and Kata Containers are prominent examples, leveraging this technology to enhance container security by isolating workloads or providing lightweight virtualized environments. In our case, we apply the same principle to AI model serving, enabling native deployment of open-source AI models within Kubernetes.