Experience with Hard Multi-Tenancy in Kubernetes Using Kata Containers

Databricks is building a serverless platform for performance-sensitive workloads such as Data Lakehouse on Kubernetes clusters. Because each cluster runs code on behalf of multiple customers, they need "hard multi-tenant" container isolation. This talk discusses their experience using Kata Containers, an open-source container runtime that provides strong isolation by running containers in micro-VMs. The presentation covers how they built a hard compute and network isolation layer among untrusted workloads in Kubernetes clusters leveraging Kata Containers, network policy, and network security groups. They share first-hand experience integrating Kata Containers with Kubernetes in production, highlighting challenges, trade-offs between security, performance, and cost, and workarounds for heterogeneity across public cloud providers.