Learn Backstage - Catalog Ingestion & Processing

#Introduction

In episode 4, you got to know entities and how to describe them via catalog-info.yaml. The next question: how does an entity get into the catalog? Episode 5 covers catalog ingestion and processing — the mechanisms for channeling entities in (locations and entity providers) along with the pipeline that turns raw files into ready-to-use entities.

A catalog is only useful if its contents stay current. So besides knowing how to write entities, you need to understand the two mechanisms that keep the catalog alive: locations pointing to sources, and the pipeline that processes them continuously.

#Ingestion

Ingestion is the process of bringing entities into the catalog. There are several approaches with different levels of ease and scale. The three approaches below aren't mutually exclusive — organizations often combine them: locations for specific entities, providers for whole repositories.

#Static Locations

The simplest way is to declare locations statically in app-config.yaml. Each location points directly to a single catalog-info.yaml source:

catalog:
  locations:
    - type: url
      target: https://github.com/acme/order-service/blob/main/catalog-info.yaml

Static locations are easy to understand and fit entities that are few in number. But for large organizations, registering every repository manually won't scale.

Locations like this can also be registered directly from the catalog UI — Backstage provides a form to register a new location without touching the config file.

#URL and File Locations

A location itself can be one of two source types: url for remote sources (for example GitHub, GitLab, or any website) and file for local files inside the workspace. The file type is usually used when you want to register an entity from a file on your machine or from a mounted file system. The only difference between them is the source — the processing afterward is identical. A location, whatever its type, is simply a pointer telling the catalog where to read entity files.

#Entity Providers

For large scale, Backstage provides entity providers — modules that automatically discover and ingest many entities at once from a single source. Providers cover a variety of integrations:

Instead of pointing one by one, a provider scans its source, finds all matching catalog-info.yaml files, and hands the results to the catalog. You just configure the pattern you want to search for — for example, all repositories in one organization. Which provider you choose depends on where you keep your code: for an organization that's uniform on one platform, a single provider is enough; for a mixed organization, several providers can be enabled together.

#The Processing Pipeline

After a location or provider produces entity sources, those candidates go through a series of processing stages called the processing pipeline:

This pipeline runs continuously: every time the source changes, the entity is refreshed through the same process. This flow from provider to stitching is what keeps the catalog in sync with your repositories without manual intervention.

The order of these stages is logical, not a single timeline — several entities can be at different stages at the same time because the pipeline runs in parallel and asynchronously.

A failure in one stage doesn't always stop the whole pipeline. The catalog records entities that failed to process and retries periodically when the source changes. This means an entity you just pushed may take a few seconds to appear in the portal — that's normal for an active catalog, not a sign that something is broken.

#GitHub Discovery Integration

The most common case is discovering entities from GitHub. The pattern is the same for GitLab, Azure DevOps, and other providers — only the provider name and source differ. First, configure the GitHub integration and the discovery provider in app-config.yaml:

integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN}
 
catalog:
  providers:
    github:
      providerId:
        organization: acme

Second, install the backend module that provides this provider:

yarn --cwd packages/backend add @backstage/plugin-catalog-backend-module-github

It's the catalog-backend-module-github module that connects the GitHub integration to the catalog pipeline. Once the module is installed and registered in the backend, every repository in the acme organization containing a catalog-info.yaml automatically appears as an entity in the catalog. Registration in the backend follows the New Backend System pattern you learned in episode 2 — calling backend.add() for the module in question.

#Conclusion

In this episode 5, you understood how the catalog fills itself: static locations for single entities, URL and file locations, and entity providers for automatic discovery from GitHub, GitLab, Azure DevOps, Bitbucket, S3, and the file system. You also learned about the processing pipeline from entity provider, reading, parsing, validation, relations, to stitching.

The key takeaways:

• Locations point to sources — static for simple cases, providers for large scale.
• Entity providers automate ingestion — one configuration can discover many repositories.
• The pipeline is a continuous process — the catalog stays in sync as long as sources change.
• GitHub discovery needs a token — store it via an environment variable, not in the config file.

In the next episode, episode 6, you'll turn the catalog into a productivity engine: Software Templates (Scaffolder) — YAML templates with a golden path that let engineers create new services self-service, complete with repositories, pipelines, and automatic catalog registration.