KubeCon + CloudNativeCon Europe 2025

The Kubernetes API is awesome and so tempting to use, especially when building Observability Solutions. Nobody wants to just get raw IP addresses and ports in their network or request telemetry, it’s much better to see your pod and service metadata. But what’s even better is that getting information about all the nodes in your cluster can help you produce amazing service graphs.

This talk is a story of how we took down the Kubernetes API in our biggest production cluster at Grafana, by deploying observability tools which make heavy use of the Kubernetes API. We’ll show you the techniques we used to avoid repeating our mistakes, by applying configuration changes and building services which helped us shield the Kubernetes API from the information thirsty observability tools, while keeping the functionality intact.