Learn Kata Containers - Multi-tenancy & Isolation Best Practices

#Introduction

Sixteen episodes of learning come down to one practical question: how do you design a multi-tenant cluster that's genuinely secure? Kata Containers gives you the tool — a microVM per pod — but a tool without usage patterns is a weapon misused. Episode 16 covers those patterns: how to organize tenancy, isolate resources, and separate workload trust levels in a single cluster.

This is the episode that brings it all together: everything you learned about architecture, resources, networking, and security is now assembled into a defensible cluster design.

#One Pod, One MicroVM: The Basic Multi-tenancy Pattern

#Tenancy at the Pod Level

The core multi-tenancy pattern with Kata is simple: one tenant = one pod = one microVM. No microVM sharing between tenants. Each tenant runs its own pod, and each pod is isolated by hardware.

Why is this pattern correct? Because mixing several tenants in one microVM means moving the inter-tenant isolation problem back to the container level inside the guest — destroying the main purpose of using Kata. A shared guest kernel in one VM means tenants can attack each other through guest kernel flaws.

#Design Consequences

• Every tenant has its own microVM — the number of microVMs can be large; make sure nodes have enough capacity.
• A Kubernetes namespace per tenant — one namespace per tenant for clear RBAC and policy.
• The same RuntimeClass stays in place — kata for all tenants; what differs is namespace, quota, and policy.

Namespace-level separation makes policy (episodes 14-15) easy to target per tenant.

#Resource Isolation via the VM

#Why the VM Gives Stronger Resource Isolation

With regular containers, resources are limited by cgroups — a tenant using excessive CPU affects its neighbors despite cgroups, especially for memory and I/O. With Kata, each microVM has its own resource allocation (episode 5): memory is allocated to the VM, vCPUs are allocated to the VM. A tenant can't "inflate" another tenant's resources.

The practical impact: noisy neighbors almost disappear at the primary resource level. Tenant A's microVM wasting CPU can't touch the memory already allocated to tenant B's microVM.

#Quota and Limits per Tenant

Apply a quota per namespace so each tenant is bounded from the top:

kubectl create namespace tenant-a
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: ResourceQuota
metadata:
  name: tenant-a-quota
  namespace: tenant-a
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
EOF

ResourceQuota limits the total resources of the tenant-a namespace. Because every pod is a microVM that holds its resources, a namespace quota means the total capacity limit of the tenant's microVMs — a firm, easily calculable control.

#Sandbox Patterns for Untrusted Workloads

#Sandbox as a Workload Category

For genuinely untrusted workloads — code executed from users, third-party plugins, AI agents running tools — use an explicit sandbox pattern: a separate namespace, Kata as the mandatory RuntimeClass, restricted egress, and short lifetimes.

A healthy sandbox design:

• A dedicated namespace with its own ResourceQuota.
• Admission policy that requires runtimeClassName: kata (episode 15).
• NetworkPolicy with deny-by-default egress (episode 14).
• Image policy that rejects unsigned images.
• A TTL or job that cleans up sandboxes automatically.

This is the pattern production deployments use to execute unknown code — and we'll expand it to AI agent sandboxes in episode 19.

#Sandbox Is Not a Dedicated Node

An important distinction: the sandbox pattern here is a workload category, not a dedicated node. The microVM provides isolation at the hardware level, so sandboxes can live on the same node as other workloads without threatening them — as long as the node itself is secure.

#Don't Mix Trusted and Untrusted Workloads on the Same Node

#Two Trust Levels, Two Tactics

Even though microVMs are highly isolating, there's one consideration that makes production teams distinguish nodes: the trust level of the node itself. Example scenarios:

• Nodes with GPU passthrough (episode 10) give the guest access to physical devices — the host is more exposed to guest behavior.
• Nodes running high-trust workloads (payment, control plane) are better shielded from every possibility.

Best practice: don't mix untrusted workloads with very sensitive workloads on the same node. Use node taints and tolerations to separate them:

kubectl taint nodes node-sandbox tier=sandbox:NoSchedule
kubectl label nodes node-sandbox workload=untrusted

kubectl taint nodes node-sandbox tier=sandbox:NoSchedule prevents regular pods from landing on the sandbox node. Untrusted pods add a toleration to be scheduled there — while trusted workloads never touch that node.

#Per-VM Monitoring

#Why Per-VM, Not Per-Node

In a Kata cluster, a single node can host dozens of microVMs. Monitoring that only sees "the node is healthy" misses per-tenant problems: one microVM wasting CPU or running out of memory isn't visible from a node average. Monitoring must be per-microVM.

Metrics to monitor per microVM:

• Resource consumption: CPU, memory, and I/O of each VM.
• Boot latency: time from pod creation to ready — changes signal KVM or kernel problems.
• Error rate: shim, VMM, or agent failures.
• Churn: too many microVM restarts signals a problem.

#Monitoring Tools

kubelet exposes pod metrics (including cgroup CPU/memory), and kubectl top is a starting point:

kubectl top pod -A
kubectl top node

kubectl top pod -A shows resource consumption per pod — per microVM. For deeper metrics (per-VMM process, virtio I/O), export node metrics with a standard exporter. Integrate with Prometheus + Grafana to see trends per namespace and per tenant.

To see the hypervisor memory footprint per pod from the host side:

VMM processes per pod

ps aux | grep -E "(qemu|cloud-hypervisor)" | awk '{print $11, $6/1024 "MB"}'

ps aux | grep -E "(qemu|cloud-hypervisor)" shows each VMM process with its memory consumption — a quick way to see how much memory each microVM holds on the host.

#Conclusion

What you should take away:

• One tenant = one pod = one microVM; don't mix tenants in a single VM.
• Resource isolation via the VM removes noisy neighbors at the primary level.
• ResourceQuota per namespace bounds the microVM capacity per tenant.
• Sandbox pattern: separate namespace, mandatory RuntimeClass, deny-by-default egress.
• Taints + tolerations separate untrusted workloads from sensitive ones.
• Per-VM monitoring is mandatory: resources, boot latency, errors, and churn.

In the next episode, episode 17, we'll cover Kata 4.0 & the Rust runtime (runtime-rs) — what it means for the Rust runtime to become the default, Dragonball optimizations, infrastructure security focus, and the Go-to-Rust transition in the 3.x releases with a monthly cadence. This is the future direction of the project you're learning.