Learn gRPC - State, Data Management & Schema Evolution

#Introduction

Applications always evolve: new fields get added, types change, and even the meaning of an entity shifts. In the REST world, schema changes often mean a new API version. In gRPC, protobuf is designed so schemas can evolve without breaking old clients — if you follow the rules.

Episode 8 covers the art of schema evolution: changing .proto without breaking changes, keeping field numbers stable, migrating services gradually, and connecting gRPC contracts to real databases. This is the skill that most determines an API's longevity.

#Schema Evolution Fundamentals

#Add, Don't Change

The golden rule of protobuf: you may add fields, but never change existing ones. When a new server sends a new field, old clients simply ignore it; when an old client sends a field the server doesn't know, it's stored as an unknown field. Both directions stay compatible.

message Product {
  string id = 1;
  string name = 2;
  double price = 3;
  // addition: without changing existing numbers
  string currency = 8;
}

Adding currency = 8 doesn't disturb clients that only know numbers 1 through 3. Number 8 is chosen because it sits in the 1-byte block (1 to 15) while avoiding collisions.

#Never Change a Field's Type

Changing a field's type — for example from int32 to string — will produce garbage data or errors during decoding. This rule is non-negotiable. If a type absolutely must change, use a new field with a new number and deprecate the old one.

#Safe Field Numbering

#Using reserved to Protect Numbers

Protobuf provides the reserved keyword to lock down numbers and names of retired fields, so they can't be accidentally reused:

message Product {
  reserved 4, 5;
  reserved "old_price", "legacy_flag";
 
  string id = 1;
  string name = 2;
  double price = 3;
  string currency = 8;
}

The reserved 4, 5 line makes the compiler reject anyone trying to reuse those numbers. Reusing old numbers is the most dangerous mistake because old data will be read with new meaning.

#Practical Numbering Rules

• Numbers 1 through 15 use only one byte — reserve them for the most frequently sent fields.
• Numbers 16 through 2047 use two bytes — use them for rare fields.
• Leave room to grow: don't fill up the entire 1-to-15 block at once.

#Service Versioning Techniques

#Package Version as the Dividing Line

For major changes, gRPC uses package versioning instead of modifying old contracts. Change package catalog.v1 to catalog.v2 and create a new file:

package catalog.v2;
 
message Product {
  string id = 1;
  string name = 2;
  Money price = 3;
}
 
message Money {
  int64 amount = 1;
  string currency = 2;
}

New clients use catalog.v2, old clients keep using catalog.v1. Both can run side by side on the same server because the service names differ. Migration happens gradually: both versions live together until the old clients retire.

#Gradual Migration with the Strangler Pattern

Avoid big-bang cuts. Shift traffic little by little: move 1 percent of clients to the new version, watch the metrics, then increase the share. catalog.v1 is kept as a compatibility layer during the transition and removed only when no callers remain.

#Database Integration

#The Repository Pattern

A gRPC server should separate transport from data. Isolate database access in a repository, while gRPC handlers only map protobuf types to domain types:

type ProductRepo struct {
    db *sql.DB
}
 
func (r *ProductRepo) FindByID(id string) (*pb.Product, error) {
    row := r.db.QueryRow(
        `SELECT id, name, price, currency FROM products WHERE id = $1`, id,
    )
    var p pb.Product
    err := row.Scan(&p.Id, &p.Name, &p.Price, &p.Currency)
    if errors.Is(err, sql.ErrNoRows) {
        return nil, status.Error(codes.NotFound, "produk tidak ditemukan")
    }
    return &p, nil
}

Notice the mapping: sql.ErrNoRows is translated into codes.NotFound — internal errors don't leak to the client. The SQL query above uses the $1 placeholder so it's safe from SQL injection.

#Database Schema Migration

The database schema evolves through versioned migrations, for example with a tool like golang-migrate:

migrate -path db/migrations -database "$DB_URL" up

The migrate ... up command applies all migrations not yet run. Migrations are idempotent and recorded in a history table, so the database state can always be reproduced from scratch.

#Closing

Key takeaways:

• You may add fields; never change the type of an existing field.
• Field numbers must never change and never be reused; use reserved.
• Package versioning (catalog.v1, catalog.v2) lets two versions live side by side.
• Migrate gradually with the strangler pattern, not in one big cut.
• A repository separates data logic from gRPC handlers and maps database errors to status codes.
• Versioned database migrations keep state reproducible.

In episode 9 next, we cover health checking, reflection, and service discovery — implementing the standard grpc.health.v1.Health, enabling gRPC server reflection for debugging with grpcurl, and understanding modern service discovery via DNS, Consul, Kubernetes, and xDS. Your now-stable contract is ready to be monitored and discovered by other systems.