Learn Observability with the LGTM Stack - Multi-Tenancy & Access Control

#Introduction

As organizations grow, one observability stack is used by many teams: the platform team, the checkout team, the data team. Without isolation, one team could read or even corrupt another team's data. This is where multi-tenancy and access control come in.

This episode covers tenancy models, how Mimir, Loki, and Tempo separate data between tenants, RBAC in Grafana, and the authentication methods that guard the front door.

#Tenancy Models

#Three Approaches

Organization-based tenancy: each organization or business unit becomes a tenant.
Team-based tenancy: each engineering team becomes a tenant.
Environment-based tenancy: dev, staging, and prod are separated as tenants.

Tenant models

organization | team | environment

The organization | team | environment pattern — choose whichever best fits your organizational structure.

#Multi-Tenancy in Mimir

#Tenant ID and Isolation

Mimir identifies tenants via the X-Scope-OrgID header:

Remote write with a tenant ID

curl -s -H "X-Scope-OrgID: checkout" -X POST \
  http://localhost:9009/api/v1/push \
  --data-binary 'http_requests_total 1'

The header X-Scope-OrgID: checkout tells Mimir which tenant the data belongs to. Without this header, the request is rejected when multi-tenancy is active.

Per-tenant limits: ingestion and query limits are set per tenant.
Per-tenant storage: data blocks are separated per tenant in object storage.
Query isolation: queries only see data from the same tenant.

#Multi-Tenancy in Loki

#Configuration and Access

Loki uses a mechanism similar to X-Scope-OrgID on log pushes. Additionally:

Label-based tenancy: using labels to separate data — not a recommended approach for strict security.
Access control: per-tenant queries are restricted according to permissions.
Cost allocation: log volume per tenant is calculated for cost allocation.

Per-tenant limits concept in Loki

limits_config:
  per_tenant_override_config: /etc/loki/tenants.yaml

The file tenants.yaml holds per-tenant limits and special settings — for example different ingestion quotas for each team.

#Multi-Tenancy in Tempo

Tempo isolates traces per tenant:

Tenant isolation: traces are only visible to the tenant owner.
Trace filtering: searches are restricted per tenant.
Resource quotas: trace storage limits per tenant.

#RBAC in Grafana

#Organizations, Teams, and Roles

Grafana manages access through a hierarchy:

Organizations: fully separate spaces for different units.
Teams: groups of users within an organization.
Roles: Admin, Editor, and Viewer with different permissions.
Folder permissions: per-folder dashboard access.
Data source permissions: restrict queries to certain data sources.
Dashboard permissions: per-dashboard access.

Viewer permission concept

role: Viewer
permissions:
  - dashboards:read
  - datasources:query

The Viewer role may only read dashboards and run queries, without the ability to change configuration.

#Authentication Methods

Grafana and the LGTM backends support many methods:

Basic auth: username and password — enough for simple internal use.
OAuth2/OIDC: Google, GitHub, GitLab, and modern IdPs.
LDAP: integration with corporate directories.
SAML: enterprise and compliance needs.
API keys and service accounts: machine access for automation.

Info

The principle of least privilege applies in observability: give access according to need. The checkout team doesn't need to see the data team's data, and automation should use service accounts with a limited scope.

#Closing

In episode 25 you understood the organization, team, and environment tenancy models, the multi-tenancy mechanisms of Mimir, Loki, and Tempo with tenant IDs, Grafana RBAC with organizations, teams, roles, and permissions, and authentication methods from basic auth to SAML.

The key takeaways:

Mimir, Loki, and Tempo use tenant IDs for isolation.
Per-tenant limits protect the cluster from a single tenant.
Grafana RBAC restricts access through roles and permissions.
Choose an authentication method according to needs and compliance.
Apply least privilege across the whole stack.

In the next episode 26 we'll discuss cost optimization — reducing metric costs with cardinality control, log costs with sampling and retention, trace costs with intelligent sampling, and optimizing infrastructure costs. Good observability must still be economical — this is the art of balancing both.