DevOps Peach

Security (DevSecOps) · Kubernetes security · Lesson 5 of 5

Runtime detection with Falco

Everything so far is preventive; Falco watches container system calls through eBPF and alerts on unusual behaviour such as a shell inside a container, reading sensitive files or odd outbound connections.

Practice interactively →

Why runtime security

Image scanning, policies and securityContext reduce the chance of compromise, but cannot block everything: a zero-day or a leaked credential can still put an attacker inside a container. Runtime detection observes real behaviour while running so you can detect and respond quickly.

Falco

Falco (a CNCF graduated project started by Sysdig) runs as a DaemonSet on each node. It reads system calls from the kernel through eBPF (the modern probe), enriches them with container and Kubernetes metadata (Pod, namespace, image), and compares them with rules.

flowchart LR
  C["Container<br/>execve, open, connect"] --> K["Kernel"]
  K -- "eBPF" --> F["Falco on the node"]
  F -- "rule matches" --> A["Alert"]
  A --> S["Falcosidekick<br/>Slack, SIEM, webhook"]

What the default rules catch

Rule Behaviour
Terminal shell in container Someone runs kubectl exec to open bash or sh in a container
Read sensitive file untrusted A process reads /etc/shadow
Write below binary dir Writing into /usr/bin, a sign of tool installation
Contact K8S API Server From Container An unexpected Pod calls the API server
Drop and execute new binary in container Download a file and run it, dropper-style

A simple rule:

- rule: Shell in app container
  desc: Detect an interactive shell in the app namespace
  condition: >
    spawned_process and container and k8s.ns.name = "app"
    and proc.name in (bash, sh, zsh)
  output: "Shell in container (user=%user.name pod=%k8s.pod.name cmd=%proc.cmdline)"
  priority: WARNING
  tags: [container, shell]

Installing and viewing alerts

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco -n falco --create-namespace \
  --set falcosidekick.enabled=true
kubectl logs -n falco -l app.kubernetes.io/name=falco -f

Warning Notice A shell was spawned in a container with an attached terminal
(user=root k8s.ns=app k8s.pod=web-7d9c-abcde container=web shell=bash)

Responding

Falco only detects. Responses are wired through Falcosidekick or Falco Talon: send to Slack or a SIEM, open a ticket, or automatically isolate the Pod (label it so a NetworkPolicy blocks it, or delete it). Keep a runbook for each alert type.

Similar tools: Tetragon (from Cilium, can block directly in the kernel), KubeArmor, Microsoft Defender for Containers, Amazon GuardDuty EKS Runtime Monitoring.

Traps: enabling every rule and drowning in noisy alerts, so tune exceptions for legitimate workloads; alerts with nobody on call; a legitimate debug container opening shells every day, so the team learns to ignore real alerts.

Practice interactively →