Learn C Language - Observability & Production Support

#Introduction

An application running in production needs more than correct code — it needs visibility. Episode 22 discusses observability: how to see what's happening inside a program while it runs, and how to investigate failures after they happen.

We'll build logging and tracing for C applications, monitor runtime metrics, then analyze crashes using core files with post-mortem debugging. Just as important, you'll learn release management and rollback, plus the documentation and code review habits that keep a project healthy.

This is the episode that unites all your technical skills into a complete production capability.

#Logging, Tracing, and Monitoring

#Designing Useful Logging

Good logs record important events with enough context: time, level, and location:

#include <stdio.h>
#include <time.h>
 
void log_info(const char *pesan) {
    time_t now = time(NULL);
    char waktu[32];
    struct tm *t = localtime(&now);
    strftime(waktu, sizeof(waktu), "%Y-%m-%d %H:%M:%S", t);
    fprintf(stderr, "[%s] INFO %s\n", waktu, pesan);
}

The log_info function above writes a timestamped message to stderr. Using stderr separates logs from normal data output so they can be redirected to a separate file or systemd.

#Log Levels and Tracing

Use levels: DEBUG, INFO, WARN, and ERROR, and let the minimum level be set at runtime. Tracing follows a flow through many components; in distributed applications, a trace ID ties together scattered logs. For monitoring, export metrics like request counts and latency to systems such as Prometheus through an HTTP endpoint or a file.

#Crash Dump Analysis and Core Files

#Enabling Core Files

When a program crashes, the kernel can save a memory snapshot to a core file. Enable and inspect it:

ulimit -c unlimited
./app

The command ulimit -c unlimited allows core files to be created. When the program crashes, a file named core appears in the working directory. Make sure the build uses -g so the core file contains readable symbol information. On production systems, set up core dumps to be collected into a dedicated directory and rotated automatically so they don't fill the disk.

#Post-mortem Debugging

Analyzing a crash without rerunning it — that's what distinguishes post-mortem debugging:

gdb -q ./app core

Inside gdb with a core file, bt prints the backtrace at the crash, info registers shows the CPU state, and print variabel reads variable values at the moment of failure. This backtrace answers the most important question: in which function and from which path the program fell.

#Getting Useful Stack Traces

For a meaningful backtrace, compile with -g -fno-omit-frame-pointer. In production, many teams also use breakpad to collect crash reports from users who can't access the server. Core files can also be downloaded and analyzed on a development machine.

Keep a core file from each release version as an archive. When a crash report arrives, compare it against that version's core file using matching symbol addresses. This habit turns confusing production incidents into structured investigations.

#Rollback Strategy and Release Management

#Versioning and Reproducibility

Every release must be identifiable and rebuildable: a version tag in the repository, dependency lockfiles, and binary hashes. Embed the version into the binary:

gcc -DVERSI=\"1.4.2\" main.c -o app
./app --version

-DVERSI=\"1.4.2\" compiles the version as a macro so the program can print it through --version. A clear version makes bug reporting easier and tracks which one is running on servers.

#Fast Rollback

Even with the best testing, a release can go wrong. Prepare a way back: deploying the previous version should be as easy as deploying the new one. A common technique is blue-green deployment — two sets of instances, the new version is tested then traffic is moved, and traffic is moved back to the old version when there's a problem. Staged releases and feature flags let you pull a problematic feature without redeploying.

#Documentation, Code Review, and Maintenance

#Documenting Interfaces

Good documentation explains a function's contract: parameters, return values, and error behavior. Header files are where interface documentation lives in C — module users read the header without opening the implementation. Don't document the obvious; document assumptions and edge cases.

#Code Review and Maintenance

Code review finds bugs before merging and spreads knowledge among team members. An effective review asks: is this code clear, does it handle errors, are there fragile parts. For maintenance, reduce technical debt gradually, keep dependencies updated, and monitor CVEs for the libraries you use.

#Closing

Key takeaways:

• Good logs record time, level, and context to stderr.
• Core files save a crash snapshot for post-mortem analysis.
• gdb reads core files with backtraces and variable contents.
• Embed the version into the binary so bug reporting becomes easy.
• Prepare rollback with blue-green or staged releases.
• Review code and document interfaces to keep a project healthy.

In the next episode 23, the final episode, we will discuss modern C11, C17, C23 features and future trends — stable features of the modern standard, concurrency and atomic operations, C's role in modern systems, embedded, and performance-critical work, up to strategies for keeping your C skills relevant in the technology ecosystem.