Learn C++ - Observability & Production Support

#Introduction

Applications running in production must be "visible" from the inside. When a problem occurs, you can't plug a debugger into a production server — all that remains are logs, metrics, and crash dumps. The ability to observe an application is what's called observability.

Episode 22 covers production support for C++ applications: structured logging with proper levels, tracing and monitoring with metrics, crash dump analysis and post-mortem debugging, release engineering with versioning and packaging, as well as ongoing maintenance and documentation.

#Structured Logging

#Log Levels and Formats

Good logs have levels (trace, debug, info, warning, error) and a structured format. Don't use std::cout for production logs — a logging library like spdlog gives you levels, formatting, and sinks to files or the network. Structured logs in JSON are easy for aggregation tools to process:

#include <spdlog/spdlog.h>
 
int main() {
    spdlog::info("server mulai di port {}", 8080);
    spdlog::warn("koneksi lambat: {} ms", 2500);
    spdlog::error("koneksi gagal: {}", "timeout");
}

spdlog::info("server mulai di port {}", 8080) writes a formatted log with placeholders. Use consistent levels: info for normal events, warning for suspicious conditions, and error for failures. You should be able to filter logs without changing code.

#Don't Log Sensitive Data

The security principle from episode 14 also applies to logs: never record passwords, tokens, or raw personal data. Limit log size and apply rotation so the disk doesn't fill up. Good logs store context, not just a message — include the request id, user id, and correlation with traces.

#Tracing and Monitoring

#Metrics and Distributed Tracing

Monitoring collects metrics: CPU, memory, latency, throughput, and error rates. Tracing follows a single request across many components. For C++ applications, OpenTelemetry C++ provides both:

#include <opentelemetry/trace/context.h>
// begin_span creates a span for one operation
auto span = tracer->StartSpan("proses-request");
// ... operation ...
span->End();

StartSpan("proses-request") creates a span recorded to the tracing backend, and span->End() closes it with a duration. The combination of logs, metrics, and traces forms complete observability — logs answer "what happened", traces answer "where", and metrics answer "how much".

#A Simple Usage Example

For applications without a large infrastructure, metrics can be published through your own endpoint read by Prometheus. Libraries like prometheus-cpp expose counters and histograms. Start with three essential metrics: request count, latency, and error rate — these three are already enough to detect service degradation.

#Crash Dump Analysis

#Reading Core Dumps

When a program crashes, the operating system can save a core dump — a snapshot of the process memory. This dump is analyzed with GDB to find the crash location:

ulimit -c unlimited
./app
gdb ./app core -batch -ex bt

ulimit -c unlimited enables core dumps, and after a crash, gdb ./app core -batch -ex bt shows the backtrace — the chain of function calls at the moment of the crash. The backtrace is the first clue toward the root cause.

#Post-mortem Debugging

An effective post-mortem pattern:

• Backtrace: where the program crashed.
• Variable values: inspect local variables in the crash frame with frame and info locals.
• Identical binary: make sure the production binary is the same as the one being debugged.
• Symbols: release builds should keep symbols in a separate file so the binary doesn't leak details.

Crash dump analysis should be triggered automatically. Systems like systemd or Apport capture core dumps, and crash addresses can be translated with addr2line. Combine this with sanitizers in CI (episode 11) so many bugs never reach production.

#Release Engineering and Packaging

#Versioning

Release engineering manages the software lifecycle: versioning, packaging, and releases. Use semantic versioning: MAJOR.MINOR.PATCH. MAJOR increases on incompatible changes, MINOR for compatible new features, PATCH for bug fixes. This version is injected into the binary at compile time:

project(app VERSION 2.3.1 LANGUAGES CXX)
configure_file(version.h.in version.h)
target_compile_definitions(app PRIVATE
    APP_VERSION="${PROJECT_VERSION}")

project(app VERSION 2.3.1 ...) sets the project version, and target_compile_definitions(... APP_VERSION="...") embeds it into the binary. An application that knows its own version makes production debugging much easier.

#Packaging

Packaging turns a binary into an installable artifact. On Linux, the common formats are .deb and .rpm; cpack from CMake produces both from a single description. Make sure the artifact includes the version, dependencies, and checksums. The CI from episode 19 automates the whole thing: build, test, package, and upload artifacts.

#Maintenance and Documentation

#Maintainable Code

Maintenance is the biggest part of software cost. Code that's good to maintain:

• Small functions with a single responsibility.
• Descriptive names.
• Comments that explain why, not what.
• Tests that catch regressions before release.
• Small, incremental changes instead of big refactors.

#API Documentation

API documentation with Doxygen generates a reference from code comments:

doxygen Doxyfile

doxygen Doxyfile generates HTML documentation from the /// and /** ... */ comments in the code. Documentation that lives alongside the code is always more up to date than separate documents. Add usage examples to the documentation — running examples are the best documentation.

#Conclusion

Here's what to take away:

• Structured logs with levels and JSON format make aggregation easy.
• OpenTelemetry provides tracing and metrics for C++ applications.
• Core dumps and GDB backtraces are the main post-mortem debugging tools.
• Semantic versioning and a version embedded in the binary ease support.
• cpack produces .deb and .rpm artifacts from CMake.
• Doxygen produces documentation that lives alongside the code.

In the final episode, episode 23, we'll discuss stable modern features and future trends — the latest stable standard features, C++'s role in game development, finance, embedded, and systems, the evolution of the ecosystem and modern libraries, as well as strategies for keeping your C++ skills relevant in the industry.