Learn Cloud Native & CNCF - AI & Machine Learning in the CNCF Ecosystem

#Introduction

In episode 10 we saw one number that changed the industry's perspective: 66% of organizations hosting generative AI models use Kubernetes for inference. That number led many to call Kubernetes "the de facto OS for AI" — the platform where artificial intelligence runs in production.

The meeting of AI and cloud native is no coincidence. These two worlds have complementary problems: AI needs infrastructure that can manage expensive compute resources like GPUs efficiently, and cloud native has Kubernetes, which happens to excel at exactly that.

By the end of this episode you will understand Kubernetes' role in the AI world, know mechanisms like DRA and KEDA, and see how CNCF responds to the AI era through TAG AI and its new projects.

#Kubernetes as an AI Platform

#Why AI Needs Kubernetes

Modern AI models — especially generative AI — rely heavily on GPUs, which are expensive and scarce. GPUs can't be left idle; every idle minute means wasted money. Kubernetes offers what the AI world needs: efficient, automatic resource scheduling.

With Kubernetes, AI workloads can be scheduled to nodes with GPUs, scaled on demand, and recovered automatically on failure. Everything that once made AI operations complicated is exactly where Kubernetes excels.

#GPU Inference on Kubernetes

Inference — running a model to produce answers — is the most common AI workload on Kubernetes. Because inference requests come from real users, it needs fast scaling and high availability, just like a regular web service.

The main difference is the GPU requirement. To manage GPUs efficiently, Kubernetes needs a special mechanism — and that's where DRA comes in.

#DRA and GPU Scheduling

#Dynamic Resource Allocation

DRA, or Dynamic Resource Allocation, is a Kubernetes mechanism that lets specialized resources like GPUs be allocated dynamically according to workload needs. Before DRA, GPU allocation tended to be rigid and inflexible.

With DRA, a Pod can request GPUs in a more explicit and managed way. This is an important foundation for making Kubernetes genuinely comfortable to use as a large-scale AI platform.

#Seeing GPU Capacity in a Cluster

To see how GPUs look from Kubernetes' side, you can check node capacity with the command below — a real illustration of the concept we just discussed:

kubectl get nodes -o custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia.com/gpu
kubectl describe node | grep -i "nvidia.com/gpu"

The kubectl get nodes command above shows the GPU column on each node, while kubectl describe node shows the allocation details. On a cluster with GPUs, this column will contain numbers; on a regular cluster, it's empty.

#KEDA and Autoscaling for AI

#Event-Based Autoscaling

AI workloads aren't always constant — they can spike suddenly when many users send requests at once. To handle this, KEDA — the graduated project we met in episode 5 — provides event-based autoscaling, including for GPU-related metrics.

KEDA monitors metrics from various sources and automatically adds or removes Pods. In the AI context, this means infrastructure can follow real demand without paying for idle capacity.

#Efficiency That Becomes FinOps

Good autoscaling is the bridge between this episode and episode 13 on FinOps. By adjusting capacity to demand, organizations don't pay for idle GPUs — and that's a huge saving given GPU prices.

#Kubeflow and MLOps Tooling

#A Tool Ecosystem for Machine Learning

Beyond core Kubernetes, there's Kubeflow — a machine learning platform running on top of Kubernetes that unifies model training, tuning, and serving in one workflow. Kubeflow is an example of how AI needs are translated into cloud native tools.

The MLOps ecosystem around Kubernetes keeps growing: tools for training models, tracking experiments, and serving inference. For those interested in the intersection of AI and DevOps, this area is a broad and still-developing field.

#Monitoring ML Workloads

When an ML platform runs on Kubernetes, it's managed like any other workload: with namespaces, Pods, and monitorable resources. The commands below show how ML workloads are seen by operators:

kubectl get pods -n kubeflow
kubectl get deployments -A | grep -i "serve"

The kubectl get pods -n kubeflow command shows the Pods belonging to the Kubeflow platform, and grep -i "serve" searches for workloads related to model serving. This shows that AI workloads ultimately get managed as ordinary Kubernetes workloads.

#TAG AI and New Projects

#The Official Home for AI in CNCF

CNCF responds to the AI era structurally through TAG AI — one of the youngest TAGs we met in episode 7. This TAG is the forum for discussing how AI and cloud native influence each other, and the birthplace of guidance and best practices.

Alongside this, more and more AI and LLM projects are entering the sandbox. This pattern resembles the early days of cloud native: new technology arrives, then the ecosystem absorbs it through community-maintained projects.

#Watching It Develop

For those following this series to the end, this is one of the most interesting areas to monitor: every quarter brings new AI projects into the CNCF catalog. Reading the catalog regularly — the habit we practiced in episode 6 — will let you see cloud native AI trends early.

#Shadow AI and Security Challenges

#A New Threat in CI/CD

As AI proliferates, a new threat called Shadow AI emerges: employees using AI tools without the knowledge or oversight of IT and security teams. In the cloud native context, Shadow AI can appear in CI/CD pipelines — for example, AI-generated code that goes unreviewed.

Shadow AI poses real risks: code whose quality and security aren't guaranteed, and internal data that could leak to third-party services. This is one reason TAG Security and TAG AI work together on new threat models.

#Managing Risk Without Stifling

The challenge is finding balance: harnessing AI's power without sacrificing security. A healthy approach is clear policy, disciplined review, and team awareness — not total prohibition, which actually drives covert usage.

#Closing

Episode 17 dissected the meeting of AI and cloud native: Kubernetes as an AI platform for GPU inference, DRA for dynamic resource allocation, KEDA for autoscaling, Kubeflow as MLOps tooling, and the Shadow AI challenge amid a fast-growing trend.

Key takeaways:

• 66% of AI organizations use Kubernetes for generative AI inference.
• DRA enables dynamic GPU allocation in Kubernetes.
• KEDA provides event-based autoscaling for AI workloads.
• Kubeflow is an ML platform running on top of Kubernetes.
• TAG AI is the official forum for AI discussions in the CNCF ecosystem.
• Shadow AI is a new security threat that needs balanced management.

In episode 18 we'll discuss the trend changing how organizations work: platform engineering and internal developer platforms — the IDP concept, golden paths, and the role of TAG Platform Engineering and related projects.