Learn Seccomp - Modes & Return Actions

#Introduction

In episode 2 you understood seccomp's architecture: how filters are evaluated in the kernel, the role of libseccomp, and how both STRICT and FILTER modes work. Episode 3 dives into the heart of decision-making: what does the kernel do when a syscall comes in, and what should you choose when writing a filter?

We'll cover two things: an in-depth comparison of the two operating modes from security and performance standpoints, then all the return actions (SECCOMP_RET_*) — their effects, when to use them, and why choosing the right one matters. This is the episode most often referenced back to throughout the series, so understand it well.

#SECCOMP_MODE_STRICT vs SECCOMP_MODE_FILTER

These two modes answer different eras (episode 1) and are installed differently (episode 2). Let's compare them systematically:

On the performance side, both are equally cheap. Seccomp BPF evaluation runs on the hottest syscall path, so it's deliberately designed to be as simple as possible: BPF programs limited in length (max 4096 instructions), loop-free, and guaranteed to terminate. The practical overhead is on the order of tens to hundreds of nanoseconds per syscall — far smaller than the cost of the syscall itself.

The security consideration is more interesting. Strict mode provides absolute guarantees in a brutal way: only four syscalls, and anything else kills the process. Seccomp-bpf is subtler: you decide how strict to be yourself, and since deny-lists tend to forget dangerous syscalls, the industry best practice is actually an allow-list (default deny) — only the syscalls that are genuinely needed are allowed. We'll explore this philosophy further through the return actions.

#Return Actions: The Kernel's Verdict on Every Syscall

When a filter is evaluated, the BPF program returns a return action that determines the syscall's fate. The kernel compares the return action values from all installed filters (if layered) and takes the most restrictive one.

#Which One to Use When?

SECCOMP_RET_KILL_PROCESS is the answer to "which syscalls must never happen under any circumstances?". If a process that never calls execve suddenly does, it's very likely being exploited — there's no reason to continue execution. Killing the process is the safest response and eliminates follow-up risk.

SECCOMP_RET_KILL_THREAD is useful when only one thread violated the rule and you want other threads to keep running. It's rarely used in practice — killing the whole process is usually cleaner from a security standpoint — but it's important to understand the difference when reading kernel documentation.

SECCOMP_RET_ERRNO is the most common default action choice for allow-list profiles (like Docker's and runc's default profiles). With SCMP_ACT_ERRNO(EPERM), every syscall not on the list fails with "Operation not permitted" — the application gets a clear error, and you can observe which syscalls the application actually needs without killing the process. It's the ideal "phased mode" for debugging an application's syscall requirements.

SECCOMP_RET_TRAP sends a SIGSYS that a user-space handler can catch. It's useful when you want to observe blocked syscalls while still letting the process continue, or when a syscall needs to be processed manually. Its overhead is larger because it involves a signal, so it's not for hot syscall paths.

SECCOMP_RET_TRACE hands the decision to a ptrace tracer. It's used by managed sandboxes: a supervisor inspects each syscall and decides to allow or deny it. A consequence: the process becomes dependent on the tracer — if the tracer dies, the default behavior must be defined carefully.

SECCOMP_RET_ALLOW is the foundation of both deny-lists and allow-lists. In an allow-list profile, it's the listed exception; in a deny-list, it's the default.

SECCOMP_RET_LOG lets you practice: the syscall is logged to the audit log but still allowed. It's the best way to understand an application's syscall behavior before the filter is fully enforced. Kernels older than 4.14 don't support it.

SECCOMP_RET_NOTIFY (kernel 5.0+) makes a syscall wait while notifying a supervisor in user-space via a notification fd. It's the most modern feature: policy can be dynamic based on application context, not just static. We'll cover it fully in episode 6.

#Practicing Strict Mode

As a conceptual closer, let's return to the practical strict mode example from episode 2 — a simple program that enables SECCOMP_MODE_STRICT:

#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <sys/prctl.h>
#include <linux/seccomp.h>
 
int main(void) {
    if (prctl(PR_SET_SECCOMP, SECCOMP_MODE_STRICT) == -1) {
        perror("prctl");
        syscall(SYS_exit, 1);
    }
    write(STDOUT_FILENO, "strict mode active\n", 19);
    syscall(SYS_exit, 0);
}

gcc -o strict-mode strict_mode.c
./strict-mode
strict mode active

#Conclusion

In episode 3 you mastered seccomp's core decisions:

• Strict mode: rigid, cheap, but impractical; filter mode: flexible and still lightweight.
• Seccomp overhead is very small — far below the cost of the syscall itself.
• Seven main return actions: KILL_PROCESS, KILL_THREAD, TRAP, ERRNO, TRACE, ALLOW, LOG, and NOTIFY.
• The golden priority rule: KILL beats ERRNO, ERRNO beats ALLOW.
• Pick the right default action first (SCMP_ACT_ALLOW for a deny-list, SCMP_ACT_ERRNO for an allow-list), then the rules.

Now you have the language and concepts to understand any filter. In the next episode 4, we'll practice for real: the basic libseccomp API — seccomp_init, seccomp_rule_add, seccomp_load, seccomp_reset, and seccomp_export_bpf, complete with your first C filter that blocks execve. See you then!