Pangolin + CrowdSec + Authentik

1. What This Stack Does & Why

If you run services on your homelab — Jellyfin, Grafana, Vaultwarden, whatever — you eventually want to access them from outside your home by typing grafana.yourdomain.com instead of 192.168.1.50:3000. That requires a reverse proxy: something that receives HTTPS traffic from the internet and routes it to the right container on your LAN.

This guide deploys a reverse proxy stack with three layers of security that most homelab guides skip:

Component	What it does	Why you want it
Pangolin + Gerbil	Route management dashboard + WireGuard tunnel	Manage all your reverse proxy routes from a web UI instead of editing YAML files. The WireGuard tunnel means you don’t need to forward dozens of ports on your router — only one UDP port.
Traefik v3	The actual reverse proxy / load balancer	Handles TLS termination, certificate management, and request routing. It’s the engine under the hood — Pangolin is the dashboard that configures it.
CrowdSec	Web Application Firewall (WAF) + crowd-sourced threat intelligence	Blocks malicious requests (SQL injection, XSS, path traversal, brute force) before they reach your services. Shares threat data with 200k+ other CrowdSec users.
Authentik	Single Sign-On (SSO) via OpenID Connect	One login for all your services. Role-based access control lets you give friends access to Jellyfin but not Grafana.

How is this different from a standard Traefik setup?

In a traditional Traefik homelab setup, you add Docker labels to each container to define routes. This works but has drawbacks:

• Every route change requires editing compose files and redeploying containers
• Labels are scattered across dozens of compose files — hard to see the full picture
• Adding SSO requires a separate forward-auth container and complex label chains
• No built-in WireGuard tunnel — you need Cloudflare Tunnel or manual port forwarding

With Pangolin, you add routes from a web dashboard. Pangolin pushes the config to Traefik via an HTTP API — no labels, no container restarts. SSO is built in. The WireGuard tunnel (Gerbil + Newt) handles connectivity.

What is Newt? What is Gerbil?

Gerbil is the WireGuard server. It runs alongside Traefik and accepts incoming tunnel connections. It binds ports 80, 443, and 51820/udp on your host.

Newt is the WireGuard client. It runs on the same Docker network as your services and creates a tunnel back to Gerbil. When Traefik needs to reach a service like http://grafana:3000, the request travels through this tunnel. This is how services on your LAN become reachable from the internet without port-forwarding each one individually.

If you’re running everything on a single server (not a VPS + homelab split), Newt still serves a purpose: it registers your local services with Pangolin so they appear in the dashboard and can be managed centrally.

2. Architecture Overview

Internet (your public IP or VPS)
  │
  │  :80 (HTTP redirect)
  │  :443 (HTTPS)
  │  :51820/udp (WireGuard)
  ▼
┌─────────────────────────────────────────────────────────┐
│  Gerbil  (WireGuard server, binds all external ports)     │
│                                                         │
│  ┌───────────────────────────────────────────────────┐  │
│  │  Traefik v3.4  (network_mode: service:gerbil)       │  │
│  │                                                   │  │
│  │  Entrypoints:                                     │  │
│  │    :443 (websecure) ─── TLS termination           │  │
│  │    :8080 (traefik)  ─── Dashboard (LAN only)      │  │
│  │                                                   │  │
│  │  Providers:                                       │  │
│  │    HTTP ← polls Pangolin API every 5s for routes  │  │
│  │    File ← dynamic_config.yml (middleware + manual) │  │
│  │                                                   │  │
│  │  Plugins:                                         │  │
│  │    badger v1.3.1   ─── Pangolin auth middleware   │  │
│  │    bouncer v1.5.0  ─── CrowdSec WAF integration  │  │
│  └───────────────────────────────────────────────────┘  │
└────────────────────┬────────────────────────────────────┘
                     │
          ┌──────────┴──────────┐
          ▼                     ▼
┌──────────────────┐   ┌────────────────────┐
│  Pangolin         │   │  CrowdSec            │
│  API    :3001    │   │  LAPI    :8080      │
│  UI     :3002    │   │  AppSec  :7422      │
│  (route manager) │   │  Metrics :6060      │
│                  │   │  (WAF + ban engine)  │
│  Stores routes   │   │                     │
│  in SQLite DB    │   │  Reads Traefik logs  │
│  Pushes to       │   │  + inspects requests │
│  Traefik via API │   │  inline via plugin   │
└──────────────────┘   └────────────────────┘
          │
          ▼
┌──────────────────┐    ┌──────────────────────────────┐
│  Newt             │    │  Your services (proxy network) │
│  WireGuard client│───▶│                              │
│  Registers local │    │  grafana:3000                │
│  services with   │    │  audiobookshelf:80           │
│  Pangolin        │    │  vaultwarden:80              │
└──────────────────┘    │  jellyseerr:5055             │
                        │  ...                         │
          │             └──────────────────────────────┘
          ▼
┌──────────────────────────┐
│  Authentik (SSO)          │
│  Server  :9000 (HTTP)    │
│  Server  :9443 (HTTPS)   │
│  + PostgreSQL (local vol)│
│  + Redis                 │
│                          │
│  Handles OIDC login flow │
│  Returns identity claims │
│  to Pangolin             │
└──────────────────────────┘

How a request flows through the stack

Here’s what happens when someone visits https://grafana.yourdomain.com:

1. DNS resolves grafana.yourdomain.com to your public IP (or VPS IP). Cloudflare handles this via a wildcard CNAME or individual A records.
2. Gerbil receives the connection on port 443 and passes it to Traefik (they share a network namespace).
3. Traefik matches the hostname against its router rules. The route for grafana.yourdomain.com was pushed by Pangolin’s HTTP provider.
4. CrowdSec bouncer plugin runs first (it’s a middleware). It checks the client IP against the ban list and sends the request body to the AppSec engine for WAF inspection. If malicious → 403 Forbidden.
5. Pangolin’s SSO middleware (badger) checks if the route requires authentication. If yes, it checks for a valid session cookie. If no cookie → redirect to Authentik login.
6. Authentik handles login: username/password, MFA, etc. On success, it redirects back to Pangolin with an OIDC token containing the user’s email and role.
7. Pangolin validates the token, creates a session cookie, checks if the user’s role has access to this resource, and forwards the request to the backend service.
8. Traefik proxies the request to http://grafana:3000 via the Docker network (or WireGuard tunnel if on a different host).

All of this happens in milliseconds. The user just sees a login page (once), then Grafana.

3. Prerequisites & DNS Setup

3.1 What you need

• A Linux host with Docker Engine and Docker Compose v2 installed. Any Debian/Ubuntu-based distro works. Minimum 2 GB RAM.
• A domain name — e.g., yourdomain.com. This guide assumes DNS is managed by Cloudflare (free tier is fine).
• A public IP that can receive traffic on ports 80, 443, and 51820/udp. If your ISP uses CGNAT, you’ll need a cheap VPS (Hetzner, Oracle free tier) as a tunnel endpoint instead.
• A free CrowdSec account for console enrollment and crowd-sourced blocklists.

3.2 Firewall rules

Open these ports on your router/firewall and forward them to your Docker host:

Port	Protocol	Purpose
80	TCP	HTTP → HTTPS redirect (Gerbil)
443	TCP	HTTPS traffic (Gerbil → Traefik)
51820	UDP	WireGuard tunnel (Gerbil)

If you use ufw on the Docker host:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 51820/udp

Only three ports

This is the beauty of the WireGuard tunnel approach. You forward three ports total, regardless of how many services you run. Without a tunnel, you’d need to forward a separate port for each service, or use Cloudflare Tunnel (which has its own limitations).

3.3 Create a Cloudflare API token

Traefik needs this to automatically create TLS certificates via Let’s Encrypt DNS challenge.

1. Log into dash.cloudflare.com
2. Go to My Profile → API Tokens → Create Token
3. Use the “Edit zone DNS” template
4. Under Zone Resources, select Include → Specific zone → yourdomain.com
5. Click Continue to summary → Create Token
6. Copy the token — you’ll need it for the .env file later. This token can only edit DNS records for your domain, nothing else.

Why DNS challenge instead of HTTP challenge?

The HTTP challenge requires Let’s Encrypt to reach your server on port 80. The DNS challenge instead creates a temporary TXT record on your domain to prove ownership. This works even if port 80 isn’t open yet, and it supports wildcard certificates (*.yourdomain.com), so you don’t need a separate cert for each subdomain.

3.4 DNS records

In Cloudflare, create DNS records for your domain. You have two options:

Option A: Wildcard (recommended)

Type	Name	Content	Proxy
A	@	Your public IP	DNS only (grey cloud)
CNAME	*	yourdomain.com	DNS only (grey cloud)

This routes all subdomains (grafana.yourdomain.com, auth.yourdomain.com, etc.) to your server. New services work immediately without adding DNS records.

Option B: Individual records

Create an A record for each subdomain you want to expose. More control, but you need to add a record every time you add a service.

Grey cloud (DNS only), not orange cloud

Set proxy status to “DNS only” (grey cloud icon), not “Proxied” (orange). We want traffic to go directly to your server where Traefik handles TLS. Cloudflare’s proxy would interfere with the WireGuard tunnel and WebSocket connections.

3.5 Create the Docker network and directories

# Create the shared network all containers will use
docker network create proxy

# Compose files
mkdir -p /opt/stacks/pangolin
mkdir -p /opt/stacks/authentik

# Persistent config directories
mkdir -p /srv/docker/pangolin/config/traefik
mkdir -p /srv/docker/crowdsec/{config,data}
mkdir -p /srv/docker/traefik/{letsencrypt,logs}
mkdir -p /srv/docker/authentik/{media,templates}

3.6 Create a CrowdSec account

1. Go to app.crowdsec.net and create a free account
2. Go to Security Engines → Engines → Add Security Engine
3. Copy the enrollment key — it looks like cmd4uxk7n...
4. You’ll paste this into the .env file. When CrowdSec starts, it uses this key to register with the console, which gives you a web dashboard to see alerts, decisions, and crowd-sourced blocklists.

4. Understanding Traefik v3 (Crash Course)

Traefik is the engine that actually handles incoming requests. Pangolin configures it, but understanding these core concepts will save you hours of debugging.

4.1 Entrypoints

An entrypoint is a port that Traefik listens on. Think of it as a door into your reverse proxy.

entryPoints:
  websecure:           # Name (referenced by routers)
    address: ":443"    # Port to listen on
  traefik:
    address: ":8080"   # Dashboard port

In our setup, we only define two entrypoints: websecure (443) for all HTTPS traffic, and traefik (8080) for the dashboard. We do not define a web (port 80) entrypoint because Gerbil handles port 80 and redirects HTTP to HTTPS before Traefik even sees it.

4.2 Routers

A router matches incoming requests to backend services based on rules. The most common rule is hostname matching:

routers:
  my-grafana:
    rule: "Host(`grafana.yourdomain.com`)"   # Match this hostname
    entryPoints: [websecure]                  # Only listen on :443
    middlewares: [security-chain]             # Apply these middlewares first
    service: grafana-svc                     # Then forward to this service
    tls:
      certResolver: letsencrypt              # Use this cert resolver for TLS

You can also match on paths (PathPrefix(/api/)), headers, or combinations. The priority field controls which router wins when multiple rules match — higher number wins.

4.3 Services

A service is the backend your request gets forwarded to. In Docker setups, this is typically http://container-name:port:

services:
  grafana-svc:
    loadBalancer:
      servers:
        - url: "http://grafana:3000"

Traefik resolves grafana via Docker’s internal DNS because all containers are on the same proxy network.

4.4 Middlewares

Middlewares modify requests before they reach the backend. They’re applied in order. Our stack uses three, chained together:

Middleware	What it does
crowdsec-bouncer	Checks the client IP against CrowdSec’s ban list. Sends the request body to the AppSec WAF engine for SQL injection, XSS, and other attack detection. Blocked requests get a 403.
secure-headers	Adds security headers to responses: HSTS (force HTTPS), X-Frame-Options (prevent clickjacking), X-Content-Type-Options (prevent MIME sniffing), CSP, etc.
rate-limit	Limits each client to 50 requests/second with a burst of 100. Prevents abuse without affecting normal browsing.

These three are combined into a chain middleware called security-chain so you can apply all three with a single reference:

security-chain:
  chain:
    middlewares:
      - crowdsec-bouncer
      - secure-headers
      - rate-limit

4.5 Providers

Providers are how Traefik learns about routes. In a traditional setup, the Docker provider watches for containers with labels. Our setup uses two different providers:

Provider	Source	What it provides
HTTP provider	Polls http://pangolin:3001/api/v1/traefik-config every 5 seconds	All routes managed by Pangolin (the majority of your services). When you add a route in the Pangolin dashboard, Traefik picks it up within 5 seconds.
File provider	Watches dynamic_config.yml on disk	Middleware definitions (CrowdSec, headers, rate-limit) and manual routes that Pangolin doesn’t manage (like the Pangolin dashboard itself).

We do not use the Docker provider at all. No labels needed on any container.

4.6 Certificate resolvers

A certificate resolver handles TLS certificate issuance from Let’s Encrypt (or another ACME provider). Our config uses DNS challenge via Cloudflare:

certificatesResolvers:
  letsencrypt:
    acme:
      email: you@example.com           # Let's Encrypt registration email
      storage: /letsencrypt/acme.json   # Where certs are stored
      dnsChallenge:
        provider: cloudflare            # Use Cloudflare DNS API
        delayBeforeCheck: 30            # Wait 30s for DNS propagation

When Traefik sees a request for a hostname it doesn’t have a cert for, it automatically requests one from Let’s Encrypt by creating a temporary DNS TXT record via the Cloudflare API. Certs auto-renew before expiry. All certs are stored in acme.json.

4.7 Plugins

Traefik v3 supports plugins loaded at startup. Our stack uses two:

Plugin	Purpose
badger (v1.3.1)	Pangolin’s authentication middleware. When a route has SSO enabled, badger intercepts the request, checks for a valid Pangolin session, and redirects to the OIDC provider if needed. This is how Pangolin enforces SSO without a separate forward-auth container.
bouncer (v1.5.0)	CrowdSec’s Traefik plugin. Checks every request against CrowdSec’s local API (ban list) and optionally forwards request bodies to the AppSec WAF engine for deep inspection.

Plugins are declared in the experimental.plugins section of the static config. Traefik downloads them on first start.

4.8 network_mode: service:gerbil

This is a Docker networking feature that makes Traefik share Gerbil’s network namespace. In plain English: Traefik doesn’t get its own IP address or port bindings. Instead, it uses Gerbil’s. When Traefik listens on :443, that’s actually Gerbil’s port 443.

This is necessary because Gerbil is the WireGuard server that receives all incoming traffic. Traefik needs to be “inside” Gerbil’s network to process that traffic. The alternative would be complex iptables rules to forward traffic between containers.

Consequence: Traefik has no ports: section in the compose file. All port bindings are on the gerbil service. If Traefik tried to bind port 80 while Gerbil is already binding port 80, one of them would fail silently.

5. Part 1 — Pangolin + Gerbil + Traefik

5.1 Environment file

Create a .env file with your secrets. These values are referenced by ${VARIABLE} in the compose file.

# System
GID=1000                            # Your user's group ID (run: id -g)
TZ=Europe/Copenhagen                # Your timezone

# Cloudflare API token (created in step 3.3)
CF_API_KEY=your-cloudflare-api-token

# CrowdSec console enrollment (created in step 3.6)
CROWDSEC_ENROLL_KEY=your-crowdsec-enroll-key
ENROLL_INSTANCE_NAME=homelab        # Friendly name shown in CrowdSec console

# Gerbil — this tells Gerbil how to reach itself internally
GERBIL_REACHABLE_AT=http://gerbil:80

# Newt credentials — leave blank for now, fill in after first boot (step 5.7)
NEWT_ID=
NEWT_SECRET=

5.2 Pangolin config

This is the main Pangolin configuration. It defines your domain, admin credentials, and how Gerbil’s WireGuard tunnel works.

app:
  dashboard_url: "https://pangolin.yourdomain.com"
  log_level: "info"

server:
  external_port: 443     # The port clients connect to (HTTPS)
  internal_port: 3001    # Pangolin API (internal, not exposed publicly)
  next_port: 3002        # Pangolin Next.js UI (internal)
  secret: "generate-me"  # Run: openssl rand -hex 32

domains:
  domain1:
    base_domain: "yourdomain.com"
    cert_resolver: "letsencrypt"      # Must match the resolver name in Traefik config

traefik:
  cert_resolver: "letsencrypt"
  http_entrypoint: "web"              # Name of the HTTP entrypoint
  https_entrypoint: "websecure"       # Name of the HTTPS entrypoint

gerbil:
  base_endpoint: "gerbil.yourdomain.com"  # PUBLIC hostname for WireGuard
  start_port: 51820                        # First WireGuard port
  block_size: 24                           # Subnet size for tunnel IPs
  subnet_group: "100.90.128.0/24"          # Internal tunnel IP range

flags:
  require_email_verification: false        # Skip email verification (homelab)
  disable_signup_without_invite: true      # No public registration
  allow_raw_resources: true                # Allow IP:port targets (not just hostnames)
  allow_local_sites: true                  # Allow sites pointing to local IPs

users:
  server_admin:
    email: "you@example.com"
    password: "change-me"              # Initial admin password

gerbil.base_endpoint must NOT include a port

Pangolin automatically appends start_port (51820) to this hostname. If you write gerbil.yourdomain.com:51820, the result is gerbil.yourdomain.com:51820:51820 and WireGuard connections fail silently with no useful error message.

5.3 Traefik static config

This file is loaded once when Traefik starts. It defines entrypoints, providers, cert resolvers, and plugins. Changes require a Traefik restart.

global:
  checkNewVersion: false
  sendAnonymousUsage: false

# Enable the Traefik dashboard on :8080 (only accessible from LAN)
api:
  insecure: true
  dashboard: true

# Application log — useful for debugging startup issues and cert problems
log:
  level: INFO
  filePath: "/app/logs/traefik.log"

# Access log — records every HTTP request. CrowdSec parses this.
# MUST be JSON format for CrowdSec's Traefik parser to work.
accessLog:
  filePath: "/app/logs/access.log"
  format: json

# Prometheus metrics — useful if you have a Grafana/Prometheus monitoring stack
metrics:
  prometheus:
    addEntryPointsLabels: true
    addServicesLabels: true
    addRoutersLabels: true

# Entrypoints — the ports Traefik listens on
entryPoints:
  websecure:
    address: ":443"
    transport:
      respondingTimeouts:
        readTimeout: "30m"        # Long timeout for WebSocket connections
    http:
      tls:
        certResolver: letsencrypt # Auto-TLS for all routes on this entrypoint
  traefik:
    address: ":8080"              # Dashboard — do NOT expose publicly

# Certificate resolver — auto-issues Let's Encrypt certs via Cloudflare DNS
certificatesResolvers:
  letsencrypt:
    acme:
      email: you@example.com
      storage: /letsencrypt/acme.json
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 30      # Seconds to wait for DNS propagation

# Trust self-signed certs from backend services (e.g., Pangolin's internal API)
serversTransport:
  insecureSkipVerify: true

# Plugins — loaded on startup
experimental:
  plugins:
    badger:
      moduleName: "github.com/fosrl/badger"
      version: "v1.3.1"          # Pangolin's auth/SSO middleware
    bouncer:
      moduleName: "github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin"
      version: "v1.5.0"          # CrowdSec WAF integration

# Providers — how Traefik discovers routes
providers:
  http:
    endpoint: "http://pangolin:3001/api/v1/traefik-config"
    pollInterval: "5s"            # Check Pangolin for route changes every 5s
  file:
    filename: "/etc/traefik/dynamic_config.yml"
    watch: true                   # Hot-reload when file changes

5.4 Traefik dynamic config

This file is hot-reloaded whenever it changes (no restart needed). It defines middlewares and manual routes that aren’t managed by Pangolin.

http:
  middlewares:
    # CrowdSec bouncer — checks every request against the ban list
    # and sends request bodies to the AppSec WAF engine
    crowdsec-bouncer:
      plugin:
        bouncer:
          enabled: true
          crowdsecMode: "live"                          # Check bans in real-time
          crowdsecLapiUrl: "http://crowdsec:8080"       # CrowdSec's Local API
          crowdsecLapiKey: "REPLACE_AFTER_STEP_6.3"     # Bouncer API key
          crowdsecAppsecEnabled: true                   # Enable WAF inspection
          crowdsecAppsecUrl: "http://crowdsec:7422"     # AppSec engine endpoint
          crowdsecAppsecFailureBlock: true              # Block if AppSec is unreachable
          updateIntervalSeconds: 60                     # Cache ban list for 60s
          forwardedHeadersCustomName: "X-Forwarded-For" # Read real client IP from header

    # Security headers — added to every response
    secure-headers:
      headers:
        referrerPolicy: "same-origin"
        customResponseHeaders:
          X-Robots-Tag: "noindex, nofollow"   # Tell search engines not to index
          X-Content-Type-Options: "nosniff"   # Prevent MIME type sniffing
        forceSTSHeader: true
        stsSeconds: 31536000                  # HSTS: force HTTPS for 1 year
        stsIncludeSubdomains: true
        frameDeny: true                       # Prevent clickjacking (X-Frame-Options)
        contentTypeNosniff: true
        browserXssFilter: true                # Legacy XSS protection header

    # Rate limiter — prevent request flooding
    rate-limit:
      rateLimit:
        average: 50    # Requests per second (sustained)
        burst: 100     # Allowed burst above average

    # Chain all three into a single reusable middleware
    security-chain:
      chain:
        middlewares:
          - crowdsec-bouncer
          - secure-headers
          - rate-limit

  # ----- Manual routes (not managed by Pangolin) -----
  # Pangolin can't manage its own routes (chicken-and-egg), so we define them here.

  routers:
    # API + WebSocket traffic → Pangolin's API port (443 internally)
    pangolin-api:
      rule: "Host(`pangolin.yourdomain.com`) && PathPrefix(`/api/`)"
      entryPoints: [websecure]
      service: pangolin-api
      priority: 100       # Higher priority = matched first
      tls: { certResolver: letsencrypt }

    pangolin-websocket:
      rule: "Host(`pangolin.yourdomain.com`) && PathPrefix(`/ws`)"
      entryPoints: [websecure]
      service: pangolin-api
      priority: 100
      tls: { certResolver: letsencrypt }

    # Everything else → Pangolin's Next.js UI (port 3002)
    pangolin-ui:
      rule: "Host(`pangolin.yourdomain.com`)"
      entryPoints: [websecure]
      service: pangolin-ui
      priority: 1         # Lowest priority = catch-all
      tls: { certResolver: letsencrypt }

  services:
    pangolin-api:
      loadBalancer:
        servers:
          - url: "http://pangolin:443"    # Pangolin's internal API
    pangolin-ui:
      loadBalancer:
        servers:
          - url: "http://pangolin:3002"   # Pangolin's Next.js UI

Why does Pangolin need split routes?

Pangolin runs two internal servers: an API on port 443 and a Next.js UI on port 3002. The API handles /api/* and /ws (WebSocket). Everything else (the dashboard pages) is served by the UI. If you point all traffic to one port, either the API or the dashboard will break. The priority field ensures /api/ routes match before the catch-all.

5.5 Docker Compose

services:
  pangolin:
    image: fosrl/pangolin:latest
    container_name: pangolin
    restart: unless-stopped
    volumes:
      - /srv/docker/pangolin/config:/app/config
    ports:
      - "3003:3001"   # API — optional, for direct LAN access during setup
      - "3004:3002"   # UI  — optional, for direct LAN access during setup
    networks:
      - proxy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3001/api/v1/"]
      interval: 3s
      timeout: 5s
      retries: 15

  gerbil:
    image: fosrl/gerbil:latest
    container_name: gerbil
    restart: unless-stopped
    depends_on:
      pangolin:
        condition: service_healthy   # Wait for Pangolin to be ready
    command:
      - --reachableAt=${GERBIL_REACHABLE_AT}
      - --generateAndSaveKeyTo=/var/config/wg.key
      - --remoteConfig=http://pangolin:3001/api/v1/
    volumes:
      - /srv/docker/pangolin/config:/var/config
    cap_add:
      - NET_ADMIN      # Required for WireGuard (creates network interfaces)
      - SYS_MODULE      # Required for loading kernel modules
    ports:
      - "51820:51820/udp"    # WireGuard tunnel
      - "443:443"            # HTTPS (Traefik listens here via network_mode)
      - "80:80"              # HTTP redirect
      - "8080:8080"          # Traefik dashboard (LAN only)
    networks:
      - proxy

  traefik:
    image: traefik:v3.4
    container_name: traefik
    restart: unless-stopped
    network_mode: service:gerbil     # Share Gerbil's network — no ports needed here
    depends_on:
      pangolin:
        condition: service_healthy
    command:
      - --configFile=/etc/traefik/traefik_config.yml
    environment:
      - CLOUDFLARE_DNS_API_TOKEN=${CF_API_KEY}   # Used by ACME DNS challenge
    volumes:
      - /srv/docker/pangolin/config/traefik/traefik_config.yml:/etc/traefik/traefik_config.yml:ro
      - /srv/docker/pangolin/config/traefik/dynamic_config.yml:/etc/traefik/dynamic_config.yml:ro
      - /srv/docker/traefik/letsencrypt:/letsencrypt:rw
      - /srv/docker/traefik/logs:/app/logs:rw

  newt:
    image: fosrl/newt:latest
    container_name: newt
    restart: unless-stopped
    environment:
      - PANGOLIN_ENDPOINT=https://pangolin.yourdomain.com
      - NEWT_ID=${NEWT_ID}
      - NEWT_SECRET=${NEWT_SECRET}
    networks:
      - proxy
    depends_on:
      pangolin:
        condition: service_healthy

  crowdsec:
    image: crowdsecurity/crowdsec:latest
    container_name: crowdsec
    restart: unless-stopped
    networks:
      - proxy
    ports:
      - "6060:6060"           # Prometheus metrics endpoint
    environment:
      - GID=${GID}
      # Collections to install on first boot (space-separated)
      - COLLECTIONS=crowdsecurity/traefik crowdsecurity/http-cve crowdsecurity/whitelist-good-actors crowdsecurity/appsec-virtual-patching crowdsecurity/appsec-generic-rules crowdsecurity/base-http-scenarios crowdsecurity/sshd crowdsecurity/linux
      - ENROLL_KEY=${CROWDSEC_ENROLL_KEY}
      - ENROLL_INSTANCE_NAME=${ENROLL_INSTANCE_NAME}
      - TZ=${TZ}
      - USE_WAL=1             # Write-ahead logging for better SQLite performance
    volumes:
      - /srv/docker/crowdsec/config:/etc/crowdsec:rw
      - /srv/docker/crowdsec/data:/var/lib/crowdsec/data:rw
      - /srv/docker/traefik/logs:/traefik-logs:ro     # Read Traefik access logs
      - /var/log:/var/log:ro                          # Read system logs (sshd, etc.)
    security_opt:
      - no-new-privileges:true

networks:
  proxy:
    external: true

5.6 First boot

cd /opt/stacks/pangolin
docker compose up -d

Wait about 30 seconds for Pangolin to initialize its database and generate configs. You can watch the logs:

docker compose logs -f pangolin

Once you see Server started, open the Pangolin dashboard:

• Via HTTPS (if DNS is set up): https://pangolin.yourdomain.com
• Via LAN (always works): http://your-server-ip:3004

Log in with the email and password from config.yml.

acme.json permissions

After first boot, check the permissions on acme.json. Traefik requires it to be mode 600. If your storage is NFS (which defaults files to 777), fix it: chmod 600 /srv/docker/traefik/letsencrypt/acme.json Traefik will refuse to start (or silently fail to issue certs) if this file has wrong permissions.

5.7 Create a Newt site and configure credentials

1. In the Pangolin dashboard, go to Sites → Add Site
2. Give it a name (e.g., “homelab”) and choose Newt as the connection method
3. Pangolin generates a Site ID and Secret. Copy both.
4. Edit your .env file and fill in NEWT_ID and NEWT_SECRET
5. Restart Newt to pick up the credentials:

docker compose restart newt

Check that Newt connects successfully:

docker compose logs newt

You should see Connected to Pangolin. In the dashboard, the site status should show green/online.

6. Part 2 — CrowdSec WAF Integration

CrowdSec is already running from the compose file above. It auto-installed the collections from the COLLECTIONS environment variable on first boot. Now we need to configure what it monitors, generate a bouncer key, and verify everything works.

6.1 What CrowdSec’s collections do

Collection	What it detects
crowdsecurity/traefik	Parses Traefik JSON access logs — detects brute force, path scanning, bad user agents
crowdsecurity/http-cve	Detects probing for known CVEs (Log4Shell, Spring4Shell, etc.)
crowdsecurity/whitelist-good-actors	Whitelists known-good IPs (search engine bots, monitoring services) to avoid false positives
crowdsecurity/appsec-virtual-patching	WAF rules that block exploitation of known vulnerabilities in real-time
crowdsecurity/appsec-generic-rules	Generic WAF rules: SQL injection, XSS, path traversal, command injection, etc.
crowdsecurity/base-http-scenarios	Baseline HTTP attack detection (directory enumeration, credential stuffing patterns)
crowdsecurity/sshd	Parses /var/log/auth.log for SSH brute force attempts
crowdsecurity/linux	General Linux system log analysis

6.2 Acquisition config

This tells CrowdSec what log files to read and which parsers to use.

# Parse Traefik access logs (JSON format)
filenames:
  - /traefik-logs/access.log
labels:
  type: traefik
---
# AppSec WAF engine — inspects request bodies inline
# Traefik's bouncer plugin sends requests here before forwarding them
listen_addr: 0.0.0.0:7422
appsec_config: crowdsecurity/appsec-default
name: appsec
source: appsec
labels:
  type: appsec

Mount Traefik logs to /traefik-logs, not /var/log/traefik

CrowdSec also mounts /var/log:/var/log:ro to read system logs (sshd, etc.). If you mount Traefik logs to /var/log/traefik inside the container, it shadows or conflicts with the system /var/log mount. Use a separate path like /traefik-logs.

6.3 Generate a bouncer API key

The Traefik bouncer plugin needs an API key to talk to CrowdSec’s Local API (LAPI). Generate one:

docker exec crowdsec cscli bouncers add traefik-bouncer

This outputs a key like 86d1vOdL1ZlPmKBL/KgMi8JvfeGT70GY7Cv4W3OnTyM. Copy it and paste it into your dynamic_config.yml:

# In dynamic_config.yml, replace:
crowdsecLapiKey: "REPLACE_AFTER_STEP_6.3"
# With your actual key:
crowdsecLapiKey: "86d1vOdL1ZlPmKBL/KgMi8JvfeGT70GY7Cv4W3OnTyM"

Since the file provider has watch: true, Traefik picks up the change automatically (no restart needed). But if Traefik was failing to start because of the placeholder key, restart it:

docker compose restart traefik

6.4 Configure Prometheus metrics (optional but recommended)

After first boot, CrowdSec generates its main config at /srv/docker/crowdsec/config/config.yaml. Find the prometheus section and ensure level is set to full:

prometheus:
  enabled: true
  level: full          # "full" exposes cs_info, cs_lapi_*, cs_active_decisions
  listen_addr: 0.0.0.0
  listen_port: 6060

Why “level: full”?

The default level only exposes basic parser/bucket metrics. If you use Grafana to monitor CrowdSec (highly recommended), you need full to get cs_active_decisions (current bans), cs_lapi_decisions (API activity), and cs_info (version info). The CrowdSec Grafana dashboard (ID 21419) expects these metrics.

6.5 Verify CrowdSec is working

# Check that CrowdSec is parsing Traefik logs
docker exec crowdsec cscli metrics
# You should see lines parsed under "traefik" and "appsec"

# Check the bouncer is registered and connected
docker exec crowdsec cscli bouncers list
# Status should show "validated" (not "pending")

# Check installed collections
docker exec crowdsec cscli collections list

# Check for any active decisions (bans)
docker exec crowdsec cscli decisions list

6.6 How the two detection layers work together

Layer	Detection method	Speed	What it catches
AppSec engine(port 7422)	Inline request inspection — the bouncer plugin sends each request to the AppSec engine before forwarding it to the backend	Real-time (per-request)	SQL injection, XSS, path traversal, command injection, known CVE exploitation patterns. Blocks the individual request with 403.
Log parser(reads access.log)	Behavioral analysis — CrowdSec reads Traefik access logs and looks for patterns over time	Near real-time (log processing delay)	Brute force, credential stuffing, directory enumeration, aggressive scanning. Bans the IP for a configured duration (default: 4 hours).

Additionally, CrowdSec’s crowd-sourced blocklists (via console enrollment) pre-ban known-bad IPs before they even send a request. This is the “crowd” in CrowdSec — 200k+ users sharing threat intelligence.

6.7 Useful CrowdSec commands

# View real-time alerts
docker exec crowdsec cscli alerts list

# Manually ban an IP for 24 hours
docker exec crowdsec cscli decisions add --ip 1.2.3.4 --duration 24h --reason "manual ban"

# Unban an IP
docker exec crowdsec cscli decisions delete --ip 1.2.3.4

# View all active bans
docker exec crowdsec cscli decisions list

# Check CrowdSec console enrollment status
docker exec crowdsec cscli console status

7. Part 3 — Authentik SSO

Authentik is a self-hosted identity provider. In this stack, it handles login for all your SSO-protected services. Users authenticate once via Authentik and get access to everything their role allows.

Why Authentik when Pangolin already has SSO?

Pangolin has built-in user management — you can create accounts in the dashboard and protect routes with login. For a solo homelab, that works. I added Authentik because I needed more:

• MFA — Pangolin’s built-in auth is username/password only. Authentik supports TOTP, WebAuthn/FIDO2 hardware keys, and more. If your homelab is internet-facing, a password alone isn’t enough.
• Shared access with friends and family — I share some services (Audiobookshelf, Seerr) with people who shouldn’t see my admin tools (Grafana, Dozzle). Authentik’s group system feeds into Pangolin’s role-based access, so I can separate “Admin” from “Member” based on Authentik group membership.
• One login for everything — services like Grafana, Portainer, and Gitea support OIDC natively. With Authentik, they can all use the same identity provider directly — not just the ones routed through Pangolin.
• Self-service — users can reset their own passwords and manage their MFA devices without messaging me.
• Audit trail — Authentik logs every login, failed attempt, and session. Useful when you’re exposing services to the internet.

If you’re the only user and just want password-protected routes, skip Authentik entirely — Pangolin’s built-in SSO is enough. Add Authentik when you need MFA, multiple users with different access levels, or a centralized identity provider.

7.1 Environment file

# Generate strong secrets — do NOT use these example values
PG_PASS=generate-me            # openssl rand -base64 18
AUTHENTIK_SECRET_KEY=generate-me   # openssl rand -base64 36

7.2 Docker Compose

services:
  authentik-postgres:
    image: postgres:16-alpine
    container_name: authentik-postgres
    restart: unless-stopped
    volumes:
      - authentik_postgres:/var/lib/postgresql/data    # LOCAL volume, not NFS!
    environment:
      POSTGRES_DB: authentik
      POSTGRES_USER: authentik
      POSTGRES_PASSWORD: ${PG_PASS}
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d authentik -U authentik"]
      interval: 5s
      timeout: 5s
      retries: 10
    networks:
      - proxy

  authentik-redis:
    image: redis:7-alpine
    container_name: authentik-redis
    restart: unless-stopped
    command: --save 60 1 --loglevel warning
    volumes:
      - authentik_redis:/data
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      interval: 5s
      timeout: 5s
      retries: 10
    networks:
      - proxy

  authentik-server:
    image: ghcr.io/goauthentik/server:2025.2
    container_name: authentik-server
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_POSTGRESQL__HOST: authentik-postgres
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_REDIS__HOST: authentik-redis
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - /srv/docker/authentik/media:/media
      - /srv/docker/authentik/templates:/templates
    ports:
      - "9000:9000"      # HTTP
      - "9443:9443"      # HTTPS
    depends_on:
      authentik-postgres:
        condition: service_healthy
      authentik-redis:
        condition: service_healthy
    networks:
      - proxy

  authentik-worker:
    image: ghcr.io/goauthentik/server:2025.2
    container_name: authentik-worker
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
      AUTHENTIK_POSTGRESQL__HOST: authentik-postgres
      AUTHENTIK_POSTGRESQL__USER: authentik
      AUTHENTIK_POSTGRESQL__NAME: authentik
      AUTHENTIK_POSTGRESQL__PASSWORD: ${PG_PASS}
      AUTHENTIK_REDIS__HOST: authentik-redis
      AUTHENTIK_ERROR_REPORTING__ENABLED: "false"
    volumes:
      - /srv/docker/authentik/media:/media
      - /srv/docker/authentik/templates:/templates
    depends_on:
      authentik-postgres:
        condition: service_healthy
      authentik-redis:
        condition: service_healthy
    networks:
      - proxy

volumes:
  authentik_postgres:
    driver: local       # MUST be local — NFS causes data corruption
  authentik_redis:
    driver: local

networks:
  proxy:
    external: true

PostgreSQL MUST use a local Docker volume

NFS does not support fsync properly. PostgreSQL relies on fsync to ensure data is written to disk. On NFS, PostgreSQL will appear to work but silently corrupt data over time, eventually causing crashes. Always use driver: local for database volumes. This also applies to any other database (MySQL, MariaDB, SQLite under heavy write load).

Authentik 2025.2 still requires Redis

The Authentik team announced plans to remove the Redis dependency, but as of version 2025.2 it has not shipped yet. You must include Redis in your stack. Check the release notes for your version.

7.3 First boot

cd /opt/stacks/authentik

# Authentik runs as a non-root user internally.
# The media directory needs wide permissions, especially on NFS.
chmod 777 /srv/docker/authentik/media

docker compose up -d

Wait for all containers to be healthy:

docker compose ps

7.4 Initial admin setup

1. Open http://your-server-ip:9000/if/flow/initial-setup/ in your browser
2. Create your admin account. Choose a strong password — this is the superuser for your entire identity system.
3. After setup, you’ll land on the Authentik admin dashboard. Bookmark it.

7.5 Add Authentik route in Pangolin

So that Authentik is accessible at https://auth.yourdomain.com, create a route in the Pangolin dashboard:

1. In Pangolin, go to Resources → Add Resource
2. Set the domain to auth.yourdomain.com
3. Set the target to http://authentik-server:9000
4. Disable SSO on this route (set it to public/no authentication)

The Authentik route must NOT have SSO enabled

This is a chicken-and-egg problem: if SSO is enabled on the Authentik route, users need to log in via Authentik to reach Authentik. They can never get to the login page. Always keep auth.yourdomain.com public. Authentik has its own authentication — it doesn’t need Pangolin to protect it.

8. Part 4 — Connecting Authentik to Pangolin (OIDC)

OIDC (OpenID Connect) is the protocol that lets Pangolin delegate authentication to Authentik. When a user hits an SSO-protected route, Pangolin says “go prove your identity to Authentik” and Authentik sends back a signed token with the user’s email and role.

8.1 Create an OIDC Provider in Authentik

Field	Value
Name	Pangolin
Authorization flow	Select the default implicit-consent flow (or explicit-consent if you want users to approve scopes)
Redirect URIs	https://pangolin.yourdomain.com/api/v1/auth/idp/oidc/callback
Signing Key	Leave empty

1. In Authentik, go to Applications → Providers in the left sidebar
2. Click Create at the top
3. Select OAuth2/OpenID Connect Provider
4. Fill in:
5. Click Finish
6. On the provider detail page, copy the Client ID and Client Secret. You’ll need them in step 8.3.

Leave Signing Key empty

Pangolin validates tokens using the OIDC discovery endpoint (.well-known/openid-configuration), not a static signing key. If you select a signing key, Pangolin may fail to validate tokens and users will get stuck in a redirect loop.

8.2 Create an Application in Authentik

Field	Value
Name	Pangolin
Slug	pangolin (auto-generated from name)
Provider	Select “Pangolin” (the provider you just created)
Launch URL	https://pangolin.yourdomain.com

1. Go to Applications → Applications in the left sidebar
2. Click Create
3. Fill in:
4. Click Create

The slug matters

The slug determines the OIDC issuer URL: https://auth.yourdomain.com/application/o/**pangolin**/. If you change the slug, update the Issuer URL in Pangolin accordingly.

8.3 Configure OIDC in Pangolin

Field	Value	Notes
Name	Authentik	Display name on the login button
Issuer URL	https://auth.yourdomain.com/application/o/pangolin/	Must include trailing slash
Client ID	(from step 8.1)
Client Secret	(from step 8.1)
Identifier Path	email	Which token claim identifies the user
Scopes	openid email profile	Space-separated

1. In the Pangolin dashboard, go to Settings → Identity Providers
2. Click Add OIDC Provider
3. Fill in:
4. Click Save

Identifier Path must be “email”, not “sub”

The sub claim from Authentik is a UUID (like a1b2c3d4-...) that has no meaning in Pangolin. Using sub means Pangolin can’t match users to their accounts, resulting in “access denied” errors. Use email — it maps to the user’s actual email address, which Pangolin uses for user identification and role mapping.

8.4 Test the SSO flow

1. In Pangolin, create a test resource (or use an existing one) with SSO enabled
2. Open the resource’s URL (e.g., https://grafana.yourdomain.com) in a private/incognito browser window
3. You should be redirected to auth.yourdomain.com with an Authentik login page
4. Log in with your Authentik credentials
5. You should be redirected back to the service and see its content

If you get stuck in a redirect loop, check the troubleshooting section.

8.5 Role-based access control (RBAC)

By default, SSO is binary: either you’re logged in or you’re not. RBAC lets you create roles (e.g., Admin and Member) and restrict certain services to certain roles. This requires three pieces:

1. Groups in Authentik (to categorize users)
2. A custom scope mapping (to include the role in the OIDC token)
3. Role mapping config in Pangolin (to read the role from the token)

Step A: Create groups in Authentik

1. Go to Directory → Groups in the left sidebar
2. Click Create
3. Create a group called homelab-admins
4. Click into the group → Users tab → Add existing user → add yourself
5. Create another group called homelab-members (for friends/family you want to share services with)

Step B: Create a custom scope mapping

This tells Authentik to include a role field in the OIDC token based on group membership.

Field	Value
Name	Pangolin Role
Scope name	pangolin_role
Expression	(see below)

1. Go to Customization → Property Mappings
2. Click Create → Scope Mapping
3. Fill in:

Paste this Python expression:

# Returns "Admin" for homelab-admins members, "Member" for everyone else
if ak_is_group_member(request.user, name="homelab-admins"):
    return {"role": "Admin"}
return {"role": "Member"}

Step C: Add the scope to the OIDC provider

1. Go to Applications → Providers → Pangolin (the provider you created earlier)
2. Click Edit
3. In the Scopes section, add Pangolin Role to the selected scopes
4. Click Update

Step D: Update Pangolin’s OIDC config

1. In the Pangolin dashboard, go to Settings → Identity Providers → Authentik
2. Update the Scopes field to: openid email profile pangolin_role
3. Set the Organization role mapping for your org to: role
4. Click Save

Now when a user logs in, the OIDC token includes "role": "Admin" or "role": "Member", and Pangolin uses this to determine which resources they can access.

JMESPath syntax pitfalls in role mapping

Pangolin’s role mapping uses JMESPath expressions to extract values from the OIDC token. This is a common source of frustration:

• String literals need single quotes: 'Member'. Bare Member (no quotes) is interpreted as “access the field named Member on the token object” and returns null.
• Booleans need backticks: true. Bare true is also a field access attempt.
• If everyone gets the default role regardless of their Authentik group, the JMESPath expression is probably evaluating to null (field access on a non-existent field) instead of the expected string.
• The simplest mapping — just reading a top-level field — is the word role (which means “read the role field from the token”). This works because our Authentik scope mapping returns {"role": "Admin"} as a top-level field.

Step E: Assign roles to resources

In Pangolin, go to each resource and set which roles can access it:

Access level	Services (examples)	Configuration
Admin only	Grafana, Tautulli, Dozzle, Wazuh	SSO enabled, only “Admin” role assigned
Admin + Member	Audiobookshelf, Homepage, Seerr	SSO enabled, both “Admin” and “Member” roles assigned
Public (no SSO)	Authentik, Vaultwarden	SSO disabled — these services handle their own auth

9. Part 5 — Adding Services & Routes

9.1 Via Pangolin dashboard (recommended)

This is the normal way to add services. Example: exposing Grafana.

networks:
  - proxy

networks:
  proxy:
    external: true

1. Make sure Grafana’s container is on the proxy Docker network. In Grafana’s compose file, add:
2. In the Pangolin dashboard, go to Resources → Add Resource
3. Set domain: grafana.yourdomain.com
4. Set target: http://grafana:3000 (container name and port)
5. Enable SSO if you want Authentik login required
6. Click Create

Within 5 seconds, Pangolin pushes the route to Traefik. Open https://grafana.yourdomain.com and it should work.

9.2 Via Traefik file provider (manual routes)

Use this for services that need custom middleware, special routing rules, or services that Pangolin can’t manage (like Pangolin itself). Add to dynamic_config.yml:

  routers:
    my-service:
      rule: "Host(`myapp.yourdomain.com`)"
      entryPoints: [websecure]
      middlewares: [security-chain]      # Apply CrowdSec + headers + rate-limit
      service: my-service
      tls: { certResolver: letsencrypt }

  services:
    my-service:
      loadBalancer:
        servers:
          - url: "http://container-name:port"

The file is hot-reloaded — no restart needed.

9.3 Applying the CrowdSec security chain

• Pangolin-managed routes: In the Pangolin UI, you can attach middlewares under the resource’s advanced settings. Reference security-chain@file (the @file suffix tells Traefik the middleware is defined in the file provider, not the HTTP provider).
• File provider routes: Add middlewares: [security-chain] directly to the router definition as shown above.

9.4 Common services quick reference

Service	Target	SSO?	Notes
Grafana	http://grafana:3000	Yes	Admin-only recommended
Audiobookshelf	http://audiobookshelf:80	Yes	Has its own auth too
Vaultwarden	http://vaultwarden:80	No	Has its own auth; SSO would break browser extensions
Jellyseerr/Seerr	http://seerr:5055	No	Has its own user system
Tautulli	http://tautulli:8181	Yes	Admin-only
Dozzle	http://dozzle:8080	Yes	Shows Docker logs — admin-only

10. Part 6 — Backup & Restore

This stack has several stateful components. Losing any of them means reconfiguring from scratch. Here’s what to back up and how.

10.1 What to back up

Component	Data location	What it contains	Priority
Pangolin	/srv/docker/pangolin/config/	config.yml, SQLite database (all routes, sites, users), WireGuard keys	Critical
Traefik	/srv/docker/traefik/letsencrypt/	acme.json (all TLS certificates)	High (certs can be re-issued, but it takes time and hits rate limits)
Traefik config	/srv/docker/pangolin/config/traefik/	Static + dynamic config, middleware definitions	High
CrowdSec	/srv/docker/crowdsec/config/ + data/	Config, acquis.yaml, decision database, bouncer keys	Medium (can be rebuilt, but you lose ban history)
Authentik DB	Docker volume authentik_postgres	All users, groups, OIDC configs, flows, policies	Critical
Authentik media	/srv/docker/authentik/media/	Uploaded logos, custom CSS	Low
Compose files	/opt/stacks/pangolin/ + /opt/stacks/authentik/	compose.yaml + .env files	Critical

10.2 Backing up the Authentik database

Since Authentik’s PostgreSQL uses a local Docker volume (not a bind mount), you need to dump it:

# Create a SQL dump
docker exec authentik-postgres pg_dump -U authentik authentik > authentik_backup.sql

# To restore (after recreating the container with an empty volume):
cat authentik_backup.sql | docker exec -i authentik-postgres psql -U authentik authentik

10.3 Backing up Pangolin’s SQLite database

# Copy the database (it's a single file)
cp /srv/docker/pangolin/config/db/db.sqlite /path/to/backup/

# To restore: stop Pangolin, copy the file back, start Pangolin
docker compose -f /opt/stacks/pangolin/compose.yaml stop pangolin
cp /path/to/backup/db.sqlite /srv/docker/pangolin/config/db/
docker compose -f /opt/stacks/pangolin/compose.yaml start pangolin

10.4 Simple backup script

#!/bin/bash
# backup-proxy-stack.sh — run daily via cron
BACKUP_DIR="/path/to/backups/proxy-stack/$(date +%Y-%m-%d)"
mkdir -p "$BACKUP_DIR"

# Pangolin config + SQLite DB
cp -r /srv/docker/pangolin/config/ "$BACKUP_DIR/pangolin-config/"

# Traefik config + certs
cp -r /srv/docker/pangolin/config/traefik/ "$BACKUP_DIR/traefik-config/"
cp -r /srv/docker/traefik/letsencrypt/ "$BACKUP_DIR/traefik-certs/"

# CrowdSec config
cp -r /srv/docker/crowdsec/config/ "$BACKUP_DIR/crowdsec-config/"

# Authentik DB dump
docker exec authentik-postgres pg_dump -U authentik authentik > "$BACKUP_DIR/authentik.sql"

# Compose files + env
cp /opt/stacks/pangolin/compose.yaml "$BACKUP_DIR/"
cp /opt/stacks/pangolin/.env "$BACKUP_DIR/pangolin.env"
cp /opt/stacks/authentik/compose.yaml "$BACKUP_DIR/authentik-compose.yaml"
cp /opt/stacks/authentik/.env "$BACKUP_DIR/authentik.env"

# Retain last 30 days
find "$(dirname "$BACKUP_DIR")" -maxdepth 1 -type d -mtime +30 -exec rm -rf {} +

echo "Backup complete: $BACKUP_DIR"

11. Gotchas & Troubleshooting

11.1 NFS storage pitfalls

If you use NFS for config storage, read this carefully

• acme.json must be chmod 600 — NFS defaults files to 777. Traefik rejects acme.json with wrong permissions and silently fails to issue certificates.
• PostgreSQL must NOT be on NFS — NFS doesn’t support fsync, causing silent data corruption. Use local Docker volumes for all databases.
• SQLite under heavy writes — services like Radarr and Sonarr get “database is locked” errors on NFS. Move their config to local Docker volumes.
• Authentik media dir needs chmod 777 — the Authentik container runs as a non-root user that can’t write to NFS directories with default permissions.
• sed -i doesn’t work — it creates a temporary file and renames it, which NFS doesn’t allow. Copy the file off NFS, edit locally, copy back.
• Brace expansion in sudo doesn’t work — sudo mkdir -p /nfs/{a,b} fails. Use separate mkdir commands.

11.2 Port conflicts

Gerbil and Traefik port conflict

Gerbil binds ports 80, 443, and 8080. Traefik runs inside Gerbil’s network namespace, so when Traefik listens on :443, it’s using Gerbil’s port 443 binding. This works because they share a network.

But if you also define a web entrypoint on :80 in Traefik, it will conflict with Gerbil which already uses port 80 for HTTP-to-HTTPS redirect. Traefik should only bind :443 and :8080.

11.3 CrowdSec issues

acquis.yaml location

The acquisition config must be at /srv/docker/crowdsec/config/acquis.yaml (inside the config directory, mapped to /etc/crowdsec/ in the container). A common mistake is putting it at /srv/docker/crowdsec/acquis.yaml (outside the config directory), where CrowdSec won’t find it.

11.4 Debugging checklist

Common problems and solutions

Symptom	Likely cause	Fix
502 Bad Gateway	Target service not on proxy network, or wrong container name/port	Check docker network inspect proxy to verify the service is connected. Check container name matches exactly.
SSL certificate errors	acme.json has wrong permissions (NFS), or Cloudflare API token is invalid	chmod 600 acme.json. Check docker logs traefik for ACME errors.
CrowdSec not blocking anything	Bouncer not registered, or not connected to LAPI	docker exec crowdsec cscli bouncers list — status should be “validated”
OIDC redirect loop	SSO enabled on the auth.yourdomain.com route	Disable SSO on the Authentik route in Pangolin
OIDC “access denied” after login	Identifier path is sub instead of email	Change to email in Pangolin OIDC settings
Everyone gets “Member” role despite being in homelab-admins	JMESPath syntax: bare Member or true instead of 'Member' or true	Use single quotes for strings, backticks for booleans in JMESPath
WireGuard tunnel not connecting	base_endpoint in config.yml includes a port	Remove the port — Pangolin appends it automatically
Traefik dashboard shows no routers	Pangolin not healthy, or HTTP provider URL wrong	Check docker compose ps — pangolin should be healthy. Check the HTTP provider endpoint in traefik_config.yml.
Pangolin dashboard partly broken (API works, UI doesn’t or vice versa)	Missing split routes in dynamic_config.yml	Ensure you have separate routers for /api/ (→ port 443) and catch-all (→ port 3002)

11.5 Useful commands

Quick reference

# === CrowdSec ===
docker exec crowdsec cscli metrics              # Overall stats
docker exec crowdsec cscli alerts list           # Recent alerts
docker exec crowdsec cscli decisions list         # Active bans
docker exec crowdsec cscli bouncers list          # Bouncer status

# Ban/unban manually
docker exec crowdsec cscli decisions add --ip 1.2.3.4 --duration 24h --reason "manual ban"
docker exec crowdsec cscli decisions delete --ip 1.2.3.4

# === Traefik ===
# View all routers (requires curl + jq on the host)
curl -sk https://localhost/api/rawdata 2>/dev/null | jq '.routers | keys'

# Tail access logs (JSON) — useful for debugging routing
docker logs traefik -f --tail 50 2>&1 | grep -v "level=debug"

# === Pangolin ===
docker compose -f /opt/stacks/pangolin/compose.yaml logs -f pangolin

# === Authentik ===
docker logs authentik-server -f --tail 100

# === Newt ===
docker logs newt -f --tail 50    # Look for "Connected to Pangolin"

# === General ===
docker network inspect proxy     # See which containers are on the network
docker compose ps                # Check container health status