Learn Debezium - Connector Lifecycle & GitOps

#Introduction

Up to episode 10, connectors were registered with a direct curl to the REST API. In production, that practice is dangerous: no change history, no review process, and rollback becomes guesswork. Episode 11 introduces GitOps to manage connectors as code.

With GitOps, connector definitions live in a repository, changes go through pull requests, and CI/CD applies them to Kafka Connect. The result: a complete history, easy auditing, and the ability to return to a previous version in seconds.

#Managing Connector Configuration as Code

Store each connector as a JSON file in a clear repository structure:

connectors/
  mysql-inventory.json
  postgres-orders.json
  sqlserver-customers.json
scripts/
  apply.sh
  validate.sh
.env.example

The connector file is exactly the same body you send to the REST API:

{
  "name": "inventory-connector",
  "config": {
    "connector.class": "io.debezium.connector.mysql.MySqlConnector",
    "database.hostname": "mysql",
    "database.user": "${env:DEBEZIUM_DB_USER}",
    "database.password": "${env:DEBEZIUM_DB_PASSWORD}",
    "database.server.id": "223344",
    "topic.prefix": "dbserver1",
    "database.include.list": "inventory"
  }
}

Because this file can be reviewed like ordinary code, every property change becomes transparent to the whole team.

#Deploying via GitOps and Automation

A simple deployment script reads all files in the connectors folder and registers them. Note that the -d @file flag tells curl to read the body from a file, not from a literal string:

for f in connectors/*.json; do
  curl -s -X POST -H "Content-Type: application/json" \
    http://localhost:8083/connectors/ -d @"$f"
done

In full GitOps, this script isn't run manually — it's executed by the CI/CD pipeline whenever changes land on the main branch. Recommended flow:

• Commit the connector definition on a feature branch.
• Open a pull request for review.
• The staging pipeline applies the configuration and runs smoke tests.
• After approval, the production pipeline applies it.

With this pattern, what's in the repository is always the single source of truth for all connectors.

#Versioning and Rollback

Because connector definitions live in git, versioning is automatic. Every change is a commit with a message explaining why. To return to a previous version:

git log --oneline -- connectors/mysql-inventory.json
git checkout <commit-id> -- connectors/mysql-inventory.json

To update an existing connector, use the PUT endpoint:

curl -s -X PUT -H "Content-Type: application/json" \
  http://localhost:8083/connectors/inventory-connector/config -d @"connectors/mysql-inventory.json"

Remember that a configuration update triggers a connector task restart. Don't do mass rollbacks during peak hours without coordination.

#Validating Changes Before Rollout

Before actually applying, Kafka Connect provides a validate endpoint that checks the configuration without running the connector:

curl -s -X PUT -H "Content-Type: application/json" \
  http://localhost:8083/connector-plugins/io.debezium.connector.mysql.MySqlConnector/config/validate \
  -d @"connectors/mysql-inventory.json"

The response contains the list of properties along with valid/error status and default values. Get into the habit of running validation in the pipeline before deploying:

for f in connectors/*.json; do
  CLASS=$(jq -r '.config["connector.class"]' "$f")
  curl -sf -X PUT \
    "http://localhost:8083/connector-plugins/$CLASS/config/validate" \
    -H "Content-Type: application/json" -d @"$f" \
    | jq -e 'all(.configs[]; .value.errors == null or (.value.errors | length == 0))' \
    || exit 1
done

If validation fails, the pipeline stops and the change never reaches production.

#Conclusion

Episode 11 brings connector management to engineering level: definitions as code, automated deployment via GitOps, versioning with git, fast rollback, and validation before every rollout.

The key takeaways:

• Store each connector as a reviewable JSON file.
• A deploy script reads the connectors folder and registers them via the REST API.
• Git provides history, audit, and rollback for connector definitions.
• PUT to /connectors/{name}/config triggers a task restart, so coordinate the timing.
• Run validation via the /connector-plugins/.../config/validate endpoint in the pipeline.

In the next episode, episode 12, we'll discuss distributed deployment and high availability — running a Kafka Connect cluster in distributed mode, connector high availability, balanced workloads, and failover.