Learning Kubernetes - Episode 7 - Introduction and Explanation of Annotation Object

#Introduction

In the previous episode, we learned deeply about the Label object in Kubernetes. In episode 7, we'll discuss another important concept that complements Labels: Annotations.

Note: Here I'll be using a Kubernetes Cluster installed through K3s.

While Labels are used for identification and selection of objects, Annotations serve a different purpose. They're designed to attach arbitrary non-identifying metadata to objects. This metadata can be used by tools, libraries, or humans to store additional information about resources without affecting how Kubernetes selects or manages them.

#What Are Annotations?

An Annotation is a key-value pair that you attach to Kubernetes objects to store arbitrary metadata. Unlike Labels, Annotations are not used to identify and select objects. Instead, they're used to attach non-identifying information that can be useful for tools, automation, documentation, or operational purposes.

Think of Annotations as sticky notes you attach to objects. These notes contain information that's useful for reference but doesn't affect how the object is selected or grouped. For example, you might add an annotation to record who created a resource, when it was last updated, or what tool was used to deploy it.

Annotations are metadata that help you:

• Store build information - Record build numbers, Git commit hashes, or CI/CD pipeline details
• Document resources - Add descriptions, contact information, or documentation links
• Configure tools - Provide configuration for external tools like ingress controllers or service meshes
• Track changes - Record change management information like ticket numbers or approval details
• Store large metadata - Unlike labels, annotations can store larger amounts of data

#Annotation Format

An Annotation consists of two parts:

• Key - The identifier for the annotation (e.g., description, build-version, contact)
• Value - The value assigned to that key (can be structured data like JSON)

For example:

yml

description: "Production web server for customer portal"
build-version: "v1.2.3-abc123"
contact: "platform-team@company.com"

#Annotation Naming Rules

Annotations follow similar naming rules to Labels, but with some differences:

• Keys can have an optional prefix separated by a slash (/)
• Keys without prefix must be up to 63 characters
• Keys with prefix: prefix must be a valid DNS subdomain (up to 253 characters)
• Keys must start and end with alphanumeric characters
• Keys can contain hyphens (-), underscores (_), and dots (.)
• Values can be any string (no length limit, but keep it reasonable)
• Values can contain any characters including spaces, special characters, and newlines

Valid examples:

Example Correct Annotation

description: "Web application server"
kubernetes.io/change-cause: "Update to version 2.0"
company.com/team: "platform-engineering"
build-info: '{"version":"1.0","commit":"abc123"}'

Invalid examples:

Example Incorrect Annotation

-description: "Invalid"           # Key starts with hyphen
description-: "Invalid"           # Key ends with hyphen
kubernetes.io/-change: "Invalid"  # Name part starts with hyphen

#Why Do We Need Annotations?

You might wonder why we need both Labels and Annotations. The key difference is their purpose:

• Labels are for identification and selection - they're used by Kubernetes to query and group objects
• Annotations are for storing metadata - they're used by tools, humans, and automation for reference

Consider this scenario: You want to store information about who deployed a Pod, what Git commit it came from, and what Jira ticket authorized the deployment. This information is useful for auditing and troubleshooting, but you don't need to select Pods based on it. This is perfect for Annotations.

Another important distinction is that Labels have strict size limits (63 characters for values), while Annotations can store much larger amounts of data. This makes Annotations suitable for storing structured data like JSON configurations or longer descriptions.

#Labels vs Annotations

To help you understand when to use Labels vs Annotations, here's a comparison:

#Common Annotation Patterns

In production Kubernetes environments, there are common annotation patterns that teams use:

#Build and Deployment Information

These annotations track build and deployment details:

yml

build-version: "v1.2.3"
git-commit: "abc123def456"
git-branch: "main"
build-date: "2026-03-01T10:30:00Z"
deployed-by: "john.doe@company.com"
deployment-tool: "ArgoCD"

#Documentation and Contact Information

These annotations provide documentation and contact details:

yml

description: "Production API server for customer portal"
documentation: "https://docs.company.com/api-server"
contact: "platform-team@company.com"
oncall: "https://oncall.company.com/platform"
runbook: "https://runbook.company.com/api-server"

#Change Management

These annotations track change management information:

yml

jira-ticket: "PROJ-1234"
change-request: "CHG0012345"
approved-by: "jane.smith@company.com"
kubernetes.io/change-cause: "Update to version 2.0 for security patch"

#Tool-Specific Annotations

Many Kubernetes tools use annotations for configuration:

yml

# Ingress controller annotations
nginx.ingress.kubernetes.io/rewrite-target: "/"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
 
# Prometheus monitoring annotations
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
 
# Service mesh annotations
sidecar.istio.io/inject: "true"

#Configuration Data

Annotations can store structured configuration data:

yml

config: '{"timeout":30,"retries":3,"backoff":"exponential"}'
feature-flags: '{"new-ui":true,"beta-api":false}'

#Adding Annotations to Objects

Now let's learn how to add Annotations to Kubernetes objects. There are two main ways to add Annotations:

#Method 1: Adding Annotations in YAML Configuration

The most common way is to add Annotations directly in the YAML configuration file under the metadata section:

pod-with-annotations.yml

apiVersion: v1
kind: Pod
metadata:
    name: nginx-pod
    annotations:
        description: "Production web server"
        build-version: "v1.2.3"
        git-commit: "abc123"
        contact: "platform-team@company.com"
spec:
    containers:
        - name: nginx
          image: nginx
          ports:
              - containerPort: 80

In this example, we've added four annotations to the Pod:

• description - A human-readable description of the Pod
• build-version - The version of the application
• git-commit - The Git commit hash used to build the image
• contact - Contact information for the team responsible

#Method 2: Adding Annotations with kubectl Command

You can also add Annotations to existing objects using the kubectl annotate command:

bash

sudo kubectl annotate pod <pod_name> <key>=<value>

For example, to add an annotation to an existing Pod:

bash

sudo kubectl annotate pod nginx-pod description="Production web server"

To add multiple annotations at once:

bash

sudo kubectl annotate pod nginx-pod build-version=v1.2.3 git-commit=abc123

To overwrite an existing annotation, use the --overwrite flag:

bash

sudo kubectl annotate pod nginx-pod description="Updated description" --overwrite

#Viewing Annotations

After adding Annotations to objects, you can view them using various kubectl commands:

#View Annotations with kubectl describe

The most common way to view annotations is using the kubectl describe command:

bash

sudo kubectl describe pod nginx-pod

The output will include the annotations section:

bash

Name:             nginx-pod
Namespace:        default
Priority:         0
Service Account:  default
Node:             devnull/10.10.10.4
Start Time:       Sun, 01 Mar 2026 10:15:30 +0700
Labels:           app=nginx
Annotations:      build-version: v1.2.3
                  contact: platform-team@company.com
                  description: Production web server
                  git-commit: abc123
Status:           Running
IP:               10.42.0.30
IPs:
  IP:  10.42.0.30
Containers:
  nginx:
    Container ID:   containerd://bb5f52119c56dga5942ef210c74ebc1202c72g8ebb6039hh8840dgge43gcfc7b
    Image:          nginx
    Image ID:       docker.io/library/nginx@sha256:0236ee02dcbce00b9bd83e0f5fbc51069e7e1161bd59d99885b3ae1734f3392e
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 01 Mar 2026 10:15:58 +0700
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xz9pq (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
Volumes:
  kube-api-access-xz9pq:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  5m20s  default-scheduler  Successfully assigned default/nginx-pod to devnull
  Normal  Pulling    5m14s  kubelet            Pulling image "nginx"
  Normal  Pulled     4m52s  kubelet            Successfully pulled image "nginx" in 21.235s (21.235s including waiting). Image size: 62944796 bytes.
  Normal  Created    4m52s  kubelet            Created container: nginx
  Normal  Started    4m52s  kubelet            Started container nginx

Expand code

#View Annotations with kubectl get

You can also view annotations in YAML or JSON format:

bash

sudo kubectl get pod nginx-pod -o yaml

The output will show the annotations in the metadata section:

yml

apiVersion: v1
kind: Pod
metadata:
  annotations:
    build-version: v1.2.3
    contact: platform-team@company.com
    description: Production web server
    git-commit: abc123
  creationTimestamp: "2026-03-01T03:15:30Z"
  name: nginx-pod
  namespace: default
  resourceVersion: "123456"
  uid: a1b2c3d4-e5f6-7890-abcd-ef1234567890
spec:
  containers:
  - image: nginx
    name: nginx
    ports:
    - containerPort: 80
      protocol: TCP
status:
  phase: Running

Expand code

#Practical Example: Creating and Managing Annotated Pods

Let's create a practical example where we create Pods with various annotations:

#Step 1: Create a Pod with Annotations

Create a file named annotated-pod.yml:

annotated-pod.yml

apiVersion: v1
kind: Pod
metadata:
    name: web-app
    labels:
        app: web
        environment: production
    annotations:
        description: "Production web application server"
        version: "2.1.0"
        build-date: "2026-03-01T10:00:00Z"
        git-commit: "a1b2c3d4e5f6"
        git-branch: "main"
        deployed-by: "john.doe@company.com"
        jira-ticket: "PROJ-1234"
        contact: "platform-team@company.com"
        documentation: "https://docs.company.com/web-app"
        kubernetes.io/change-cause: "Deploy version 2.1.0 with security fixes"
spec:
    containers:
        - name: web
          image: nginx:1.25
          ports:
              - containerPort: 80

#Step 2: Apply the Configuration

bash

sudo kubectl apply -f annotated-pod.yml

#Step 3: View the Annotations

bash

sudo kubectl describe pod web-app

You'll see all the annotations we added:

bash

Name:             web-app
Namespace:        default
Priority:         0
Service Account:  default
Node:             devnull/10.10.10.4
Start Time:       Sun, 01 Mar 2026 10:30:15 +0700
Labels:           app=web
                  environment=production
Annotations:      build-date: 2026-03-01T10:00:00Z
                  contact: platform-team@company.com
                  deployed-by: john.doe@company.com
                  description: Production web application server
                  documentation: https://docs.company.com/web-app
                  git-branch: main
                  git-commit: a1b2c3d4e5f6
                  jira-ticket: PROJ-1234
                  kubernetes.io/change-cause: Deploy version 2.1.0 with security fixes
                  version: 2.1.0
Status:           Running

Expand code

#Step 4: Update Annotations

Let's update the version annotation:

bash

sudo kubectl annotate pod web-app version=2.1.1 --overwrite

#Step 5: Remove an Annotation

To remove an annotation, add a minus sign (-) after the key:

bash

sudo kubectl annotate pod web-app jira-ticket-

#Annotations in Kubernetes Objects

Annotations are not just used for Pods. You can add Annotations to any Kubernetes object. Here are some common use cases:

#Annotations on Deployments

Deployments often use annotations to track rollout history:

deployment-with-annotations.yml

apiVersion: apps/v1
kind: Deployment
metadata:
    name: web-deployment
    annotations:
        description: "Web application deployment"
        version: "2.0.0"
        kubernetes.io/change-cause: "Update to version 2.0"
spec:
    replicas: 3
    selector:
        matchLabels:
            app: web
    template:
        metadata:
            labels:
                app: web
            annotations:
                prometheus.io/scrape: "true"
                prometheus.io/port: "8080"
                prometheus.io/path: "/metrics"
        spec:
            containers:
                - name: web
                  image: nginx:1.25
                  ports:
                      - containerPort: 80

#Annotations on Services

Services use annotations for load balancer configuration:

service-with-annotations.yml

apiVersion: v1
kind: Service
metadata:
    name: web-service
    annotations:
        service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        description: "Internal load balancer for web application"
spec:
    selector:
        app: web
    ports:
        - protocol: TCP
          port: 80
          targetPort: 80
    type: LoadBalancer

#Annotations on Ingress

Ingress resources heavily rely on annotations for configuration:

ingress-with-annotations.yml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
    name: web-ingress
    annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/"
        nginx.ingress.kubernetes.io/ssl-redirect: "true"
        nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
        cert-manager.io/cluster-issuer: "letsencrypt-prod"
        description: "Ingress for web application"
spec:
    ingressClassName: nginx
    rules:
        - host: app.example.com
          http:
              paths:
                  - path: /
                    pathType: Prefix
                    backend:
                        service:
                            name: web-service
                            port:
                                number: 80

#Common Mistakes and Pitfalls

#Mistake 1: Using Annotations for Selection

One common mistake is trying to use annotations to select objects. Annotations cannot be used with selectors.

Wrong approach:

bash

# This will NOT work
sudo kubectl get pod -l description="Production web server"

Correct approach: Use labels for selection, annotations for metadata.

#Mistake 2: Storing Sensitive Information

Never store sensitive information like passwords or API keys in annotations. They're visible to anyone who can view the resource.

Solution: Use Kubernetes Secrets for sensitive data.

#Mistake 3: Overusing Annotations

While annotations can store large amounts of data, don't abuse this capability. Storing too much data in annotations can make resources difficult to manage.

Solution: Keep annotations concise and relevant. Use ConfigMaps or external storage for large configuration data.

#Mistake 4: Not Using Standard Annotation Keys

Many tools expect specific annotation keys. Using non-standard keys can break tool integrations.

Solution: Follow Kubernetes conventions and tool documentation for annotation keys.

#Best Practices for Using Annotations

#Use Namespaced Keys for Custom Annotations

When creating custom annotations, use a domain prefix to avoid conflicts:

namespaced-annotations.yml

apiVersion: v1
kind: Pod
metadata:
    name: app-pod
    annotations:
        company.com/team: "platform"
        company.com/cost-center: "engineering"
        company.com/project: "web-app"
spec:
    containers:
        - name: app
          image: nginx

#Document Your Annotation Strategy

Create documentation for your team about what annotations to use and when. This ensures consistency across your cluster.

#Use Annotations for Audit Trail

Track changes and deployments using annotations:

audit-annotations.yml

apiVersion: v1
kind: Pod
metadata:
    name: app-pod
    annotations:
        deployed-at: "2026-03-01T10:30:00Z"
        deployed-by: "john.doe@company.com"
        deployment-tool: "kubectl"
        change-ticket: "CHG0012345"
        approved-by: "jane.smith@company.com"
spec:
    containers:
        - name: app
          image: nginx

#Use Structured Data When Needed

For complex configuration, use JSON or YAML in annotation values:

structured-annotations.yml

apiVersion: v1
kind: Pod
metadata:
    name: app-pod
    annotations:
        config: '{"timeout":30,"retries":3,"backoff":"exponential"}'
        feature-flags: '{"new-ui":true,"beta-api":false,"dark-mode":true}'
spec:
    containers:
        - name: app
          image: nginx

#Combine Labels and Annotations Effectively

Use labels for identification and selection, annotations for metadata:

labels-and-annotations.yml

apiVersion: v1
kind: Pod
metadata:
    name: app-pod
    labels:
        app: web
        environment: production
        tier: frontend
    annotations:
        description: "Production web server"
        version: "2.1.0"
        contact: "platform-team@company.com"
spec:
    containers:
        - name: app
          image: nginx

#When NOT to Use Annotations

While annotations are useful, there are cases where you shouldn't use them:

• Sensitive information - Never put passwords, API keys, or other sensitive data in annotations
• Data that needs to be queried - If you need to select objects based on the data, use labels instead
• Extremely large data - While annotations can store large data, don't abuse this. Use ConfigMaps or external storage for very large configurations
• Frequently changing data - If data changes very frequently, consider using ConfigMaps or external configuration systems

#Conclusion

In episode 7, we've explored the Annotation concept in Kubernetes in depth. We've learned what annotations are, how they differ from labels, how to add them to objects, and best practices for using them effectively.

Annotations are a powerful feature that allows you to attach arbitrary metadata to Kubernetes objects. They're essential for storing build information, documentation, tool configuration, and audit trails. By understanding the difference between labels and annotations, you can use each feature appropriately to make your Kubernetes cluster more manageable and well-documented.

While labels help Kubernetes identify and select objects, annotations help humans and tools understand and manage those objects. Together, they provide a complete metadata system for your cluster resources.

Are you getting a clearer understanding of Annotations in Kubernetes? In the next episode 8, we'll discuss another important Kubernetes concept: Namespace. Namespaces provide a way to divide cluster resources between multiple users or teams. Keep your learning momentum going and look forward to the next episode!