Learn C# - Observability & Monitoring

#Introduction

An application in production will fail — the only question is when. When it happens, you must quickly answer three questions: what went wrong, where it happened, and how severe the impact is. This is the definition of observability.

Observability isn't just about writing logs. It's the discipline of emitting three signals — logs, metrics, and traces — consistently from all services.

Episode 20 builds a complete observability system: logging with Serilog, telemetry with OpenTelemetry, health checks, and how to handle incidents in production.

#Logging with Serilog

#Structured Logs to Console and File

In episode 6 you were introduced to ILogger. Serilog strengthens it with structured logging and diverse sinks — file, console, and observability platforms:

dotnet add src/Toko.Api package Serilog.AspNetCore

Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .WriteTo.File("logs/toko-.log", rollingInterval: RollingInterval.Day)
    .Enrich.FromLogContext()
    .CreateLogger();
 
var builder = WebApplication.CreateBuilder(args);
builder.Host.UseSerilog();

WriteTo.File with rollingInterval: RollingInterval.Day creates one log file per day, preventing the file from ballooning. Serilog sends structured JSON logs, not just text, so they're easy to query on an observability platform.

#Structured Placeholders

Always use named placeholders instead of concatenation:

logger.LogInformation(
    "Order {OrderId} diproses oleh {UserId} sebesar {Total:C}",
    orderId, userId, total);

The placeholders {OrderId}, {UserId}, and {Total} become separate fields in the structured log — enabling searches like "all orders with ID X".

#Metrics, Tracing, and OpenTelemetry

#Instrumentation with OpenTelemetry

OpenTelemetry is the industry standard for telemetry. This library captures metrics and traces automatically from ASP.NET Core and HttpClient:

dotnet add src/Toko.Api package OpenTelemetry.Extensions.Hosting
dotnet add src/Toko.Api package OpenTelemetry.Instrumentation.AspNetCore

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing =>
    {
        tracing.AddAspNetCoreInstrumentation();
        tracing.AddHttpClientInstrumentation();
    })
    .WithMetrics(metrics =>
    {
        metrics.AddAspNetCoreInstrumentation();
    });

AddAspNetCoreInstrumentation automatically records each request's duration as a trace, and metrics like the number of requests per endpoint. Traces from different services can be combined to see one request's journey across microservices.

#Activity for Manual Traces

For manual instrumentation, use ActivitySource:

ActivitySource Sumber = new("Toko.Orders");
 
using var activity = Sumber.StartActivity("ProsesOrder");
activity?.SetTag("order.id", orderId);
activity?.SetTag("order.total", total);

StartActivity begins a trace segment with tags containing context. This is what creates the correlation IDs you saw in episode 17.

#Health Checks and Monitoring Endpoints

#Health Endpoints

Orchestration platforms need to know whether an application is alive and ready to accept traffic. ASP.NET Core provides built-in health checks:

builder.Services.AddHealthChecks()
    .AddDbContextCheck<TokoContext>();
 
var app = builder.Build();
app.MapHealthChecks("/health", new HealthCheckOptions
{
    ResponseWriter = async (ctx, report) =>
    {
        ctx.Response.ContentType = "application/json";
        await ctx.Response.WriteAsJsonAsync(report);
    }
});

MapHealthChecks("/health") provides the /health endpoint, which reports the status of the application and database. Kubernetes uses this endpoint as liveness and readiness probes to decide when a pod is ready to accept traffic.

#Monitoring with curl

curl -s http://localhost:8080/health

The curl -s http://localhost:8080/health command returns a Healthy or Unhealthy status along with check details. Polling this endpoint routinely is the foundation of alerting.

#Incident Response and Troubleshooting

#The Incident Handling Flow

When an incident occurs, follow a structured flow:

• Detection: alerts from metrics or failing health checks.
• Triage: check metric dashboards, the latest traces, and error logs to determine which service is affected.
• Mitigation: recover quickly — roll back the deployment or scale horizontally, rather than hunting for the root cause immediately.
• Post-mortem: once stable, analyze the root cause and write follow-up actions.

#Troubleshooting in Containers

Logs are scattered across many pods; gather them with kubectl:

kubectl logs deployment/toko-api --tail=200

The kubectl logs command shows the latest pod logs. For centralized logs, point the Serilog sink at a platform like Loki or CloudWatch so all logs are available in one searchable place.

#Closing

Key takeaways:

• Serilog provides structured logging with flexible sinks.
• Named placeholders make logs queryable, not just readable.
• OpenTelemetry captures metrics and traces automatically.
• Health checks are the source of truth for probes and alerting.
• Incidents are handled with detection, triage, mitigation, then post-mortem.

In the next episode 21 we move to the client side: desktop and cross-platform apps — .NET MAUI for cross-platform applications, the basics of WPF and WinForms, interoperability with native libraries, and packaging and distribution of desktop applications.