Container Security: Scanning, Runtime Protection, and Best Practices

Last year, I audited a startup's Kubernetes cluster and found 847 known CVEs across their 23 container images. Twelve of those CVEs were rated critical — including one that allowed remote code execution without authentication. The team had been running this setup in production for nine months. Their response? "We thought Docker containers were isolated and secure by default."

They're not alone in that assumption. A 2024 Sysdig Cloud-Native Security Report found that 87% of container images have high or critical vulnerabilities, and 73% of organizations have containers running as root in production. Containers provide process isolation, not security isolation. If your security strategy starts and ends with docker build, you have a problem.

This guide covers container security from image creation to runtime protection — the full lifecycle. Whether you're running Docker Compose on a single server or orchestrating thousands of pods on Kubernetes, these practices apply.

The Container Threat Model: What Are You Defending Against?

Before diving into tools and techniques, let's be precise about the threats. Container security spans four attack surfaces:

1. Image-Level Threats

• Vulnerable base images: Your FROM node:18 includes the entire Debian OS with hundreds of packages, many with known vulnerabilities
• Embedded secrets: API keys, database passwords, and certificates baked into image layers during build
• Malicious packages: Compromised npm/pip/gem packages that execute arbitrary code during installation
• Outdated dependencies: Application libraries with known CVEs that haven't been updated

2. Build Pipeline Threats

• Tampered base images: A compromised Docker Hub account pushing malicious versions of popular images
• Supply chain attacks: A compromised CI/CD pipeline injecting malicious code during build
• Unverified image provenance: No guarantee that the image in production is the one that was built and tested

3. Runtime Threats

• Container escape: Exploiting kernel vulnerabilities to break out of the container and access the host
• Privilege escalation: Containers running as root gaining access to host resources
• Lateral movement: Compromised container attacking other containers or services on the network
• Resource abuse: Cryptomining, DDoS participation, or data exfiltration from a compromised container

4. Orchestration Threats

• Kubernetes API exposure: Unauthenticated access to the Kubernetes API server
• RBAC misconfigurations: Overly permissive service accounts
• Secret management: Kubernetes Secrets stored as base64-encoded plaintext in etcd

According to Red Hat's 2024 State of Kubernetes Security Report, 67% of organizations have delayed or slowed deployment due to security concerns, and 37% have experienced a security incident related to containers or Kubernetes in the past 12 months.

Image Scanning: Your First Line of Defense

Image scanning analyzes container images for known vulnerabilities (CVEs), misconfigurations, embedded secrets, and compliance violations. It should run at three points: during development, in the CI/CD pipeline, and continuously in the registry.

Tool Comparison: The Big Five

Trivy: The Open-Source Standard

Trivy (by Aqua Security) has become the de facto standard for container scanning. It's fast, comprehensive, and integrates with every major CI/CD platform.

# Scan a container image
trivy image myapp:latest

# Scan with severity filter (only HIGH and CRITICAL)
trivy image --severity HIGH,CRITICAL myapp:latest

# Scan and fail CI if critical vulnerabilities found
trivy image --exit-code 1 --severity CRITICAL myapp:latest

# Scan a Dockerfile for misconfigurations
trivy config Dockerfile

# Scan for embedded secrets
trivy image --scanners secret myapp:latest

# Generate SBOM (Software Bill of Materials)
trivy image --format spdx-json --output sbom.json myapp:latest

CI/CD Integration: Scanning in GitHub Actions

# .github/workflows/container-security.yml
name: Container Security Scan
on: [push, pull_request]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'  # Fail the build

      - name: Upload scan results to GitHub Security
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: 'trivy-results.sarif'

Hardening Container Images: The Practical Guide

1. Use Minimal Base Images

The #1 mistake in container security is using full OS images as base. Every package in your base image is an attack surface. Here's the impact:

Google's Distroless images contain only your application and its runtime dependencies — no shell, no package manager, no utilities. This dramatically reduces the attack surface. If an attacker gains code execution inside a distroless container, they can't even run ls.

2. Multi-Stage Builds: Separate Build from Runtime

# Stage 1: Build (has all build tools, dev dependencies)
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
RUN npm run build

# Stage 2: Runtime (minimal, no build tools)
FROM gcr.io/distroless/nodejs20-debian12
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./

# Run as non-root user (distroless provides user 65534)
USER 65534

EXPOSE 3000
CMD ["dist/server.js"]

3. Never Run as Root

# Bad: running as root (default)
FROM node:20-alpine
WORKDIR /app
COPY . .
CMD ["node", "server.js"]

# Good: create and switch to non-root user
FROM node:20-alpine
RUN addgroup -g 1001 -S appgroup && \
    adduser -S appuser -u 1001 -G appgroup
WORKDIR /app
COPY --chown=appuser:appgroup . .
USER appuser
CMD ["node", "server.js"]

According to the Sysdig 2024 report, 73% of containers in production run as root. This means a container escape vulnerability grants the attacker root access on the host. Setting USER in your Dockerfile is the single highest-impact security improvement you can make.

Runtime Security: Detecting Threats in Running Containers

Image scanning catches known vulnerabilities before deployment. But what about zero-days? What about misuse of legitimate tools? What about an attacker who gets in through your application code? This is where runtime security comes in.

Falco: The Open-Source Runtime Security Standard

Falco (CNCF graduated project) monitors system calls made by containers and alerts on suspicious behavior. It's like an intrusion detection system specifically designed for containers.

# Install Falco via Helm
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
  --set falcosidekick.enabled=true \
  --set falcosidekick.config.slack.webhookurl="https://hooks.slack.com/xxx"

# Example Falco rules
- rule: Shell Spawned in Container
  desc: Detect shell execution in a container (potential intrusion)
  condition: >
    spawned_process and container and
    proc.name in (bash, sh, zsh, dash, ash) and
    not proc.pname in (cron, supervisord)
  output: >
    Shell spawned in container
    (user=%user.name container=%container.name shell=%proc.name
     parent=%proc.pname cmdline=%proc.cmdline image=%container.image.repository)
  priority: WARNING
  tags: [container, shell, mitre_execution]

- rule: Sensitive File Access
  desc: Detect access to sensitive files (passwords, keys)
  condition: >
    open_read and container and
    (fd.name startswith /etc/shadow or
     fd.name startswith /etc/passwd or
     fd.name contains id_rsa or
     fd.name contains .env)
  output: >
    Sensitive file read in container
    (user=%user.name file=%fd.name container=%container.name)
  priority: CRITICAL

Network Policies: Zero-Trust Container Networking

By default, every pod in a Kubernetes cluster can talk to every other pod. This is the container equivalent of putting every server in the same flat network — a compromised container can reach everything.

# Default deny all ingress and egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

---
# Allow specific communication: API -> Database
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-api-to-db
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: postgres
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api-server
      ports:
        - protocol: TCP
          port: 5432

---
# Allow API to reach external services (DNS + HTTPS only)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-egress
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Egress
  egress:
    - to: []
      ports:
        - protocol: UDP
          port: 53  # DNS
        - protocol: TCP
          port: 443  # HTTPS

Kubernetes Security: Hardening the Orchestrator

Pod Security Standards (PSS)

Kubernetes 1.25+ enforces Pod Security Standards natively (replacing the deprecated PodSecurityPolicy). There are three levels:

# Enforce restricted security on a namespace
apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Seccomp Profiles: Limiting System Calls

Seccomp (Secure Computing Mode) restricts which Linux system calls a container can make. The default Docker seccomp profile blocks ~60 of ~300+ syscalls, but you can create custom profiles that are even more restrictive:

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": ["SCMP_ARCH_X86_64"],
  "syscalls": [
    {
      "names": [
        "read", "write", "close", "fstat", "mmap", "mprotect",
        "munmap", "brk", "accept", "bind", "listen", "connect",
        "socket", "sendto", "recvfrom", "epoll_create1", "epoll_ctl",
        "epoll_wait", "clone", "execve", "exit_group", "futex",
        "getpid", "openat", "newfstatat"
      ],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

My Opinionated Container Security Stack

After securing container workloads across three organizations, here's the stack I recommend:

1. Don't start with tools — start with Dockerfiles. 80% of container security issues come from bad Dockerfiles: running as root, using bloated base images, and baking secrets into layers. Fix these before buying any security product.

2. Trivy is enough for most teams. Unless you're in a regulated industry requiring specific compliance reports, Trivy's free image scanning covers 95% of what Snyk and Sysdig charge thousands per month for. Invest the savings in engineering time to fix the vulnerabilities Trivy finds.

3. Network policies are non-negotiable. Every namespace should have a default-deny policy. If your CNI doesn't support network policies (I'm looking at you, default kubenet), switch to Calico or Cilium.

4. Runtime security (Falco) is a luxury for most startups. Unpopular opinion, but Falco has operational overhead. If you're a team of 5 engineers, focus on image scanning, non-root containers, and network policies first. Add Falco when you have a security engineer or reach SOC 2 requirements.

5. Sign your images. Cosign (from the Sigstore project) makes image signing trivial. Combined with an admission controller that verifies signatures, this prevents deploying unsigned or tampered images.

Action Plan: Container Security in 4 Weeks

Week 1: Image Hardening

• Audit all Dockerfiles: switch to minimal base images (alpine or distroless)
• Add USER directive to every Dockerfile — no more root containers
• Implement multi-stage builds to separate build and runtime dependencies
• Run Trivy locally against all your images and catalog the findings

Week 2: Pipeline Security

• Add Trivy scanning to CI/CD with exit-code 1 for CRITICAL severity
• Set up SBOM generation for every image build
• Implement image signing with Cosign
• Configure a private registry with automatic vulnerability scanning

Week 3: Runtime Hardening

• Implement default-deny network policies in all namespaces
• Enable Pod Security Standards (restricted level) on production namespaces
• Drop all Linux capabilities and add back only what's needed
• Enable read-only root filesystem where possible

Week 4: Monitoring and Response

• Set up continuous scanning of images in your registry
• Create alerts for new critical CVEs in deployed images
• Document your vulnerability response process: triage, patch, deploy timelines
• Run a tabletop exercise: "What do we do if image X is compromised?"

Sources and Further Reading

• Sysdig — 2024 Cloud-Native Security and Usage Report
• Red Hat — 2024 State of Kubernetes Security Report
• Aqua Security — Trivy (GitHub)
• Falco — Cloud-Native Runtime Security
• Google — Distroless Container Images
• Sigstore — Cosign Image Signing
• Kubernetes — Pod Security Standards
• OWASP — Docker Security Cheat Sheet
• NIST — National Vulnerability Database

I'm Ismat, and I build BirJob — Azerbaijan's job aggregator scraping 80+ sources daily.