Harden geoblock, remove secrets from config, support dev host

Refs #1 ## Security * Remove the CrowdSec LAPI key from config.yml. Traefik does not interpolate env vars in its YAML config, so the key had to be literal there - which is why the file kept drifting out of version control. It is now defined as a docker-compose label, where compose substitutes ${CROWDSEC_LAPI_KEY} from .env. config.yml is therefore secret-free and stays in git. NOTE: this renames the middleware crowdsec-bouncer@file -> @docker. Site stacks referencing @file must be updated. The old key remains in this repo's history and must still be rotated and scrubbed - removing it here does not un-expose it. * Move the dashboard basic-auth hash out of docker-compose.yml into .env. In .env every '$' must be doubled to '$$', or compose silently collapses the hash to an empty string and the dashboard rejects every password. * Stop publishing port 8080. The API is not insecure-mode, so nothing served there; publishing it only widened exposure. ## Geoblock (the availability fix) * allowUnknownCountries: false -> true. Previously a geojs.io outage made every lookup "unknown" and therefore blocked, taking sites down for US visitors too - a third-party service in the critical path with no degraded mode. Now an outage degrades to "geoblocking ineffective". CrowdSec still blocks actual attackers. * cacheSize: 25 -> 5000. At 25, nearly every visitor triggered a fresh API call, adding up to 750ms and risking rate limits. ## Dev host support * Add traefik.dev.yml, selected by TRAEFIK_STATIC_CONFIG in .env (defaults to traefik.yml, so prod behaviour is unchanged). It differs from traefik.yml ONLY in the ACME challenge type: public port 80 forwards to prod for its renewals, so an HTTP-01 challenge for a dev hostname hits prod's Traefik, matches no router, and 404s. Dev uses DNS-01 via Route 53 instead, which needs no inbound connectivity. * Add .env.example documenting every variable. ## Captured from prod's working tree Prod had uncommitted drift; these were running but never committed: plugin bumps (bouncer v1.3.3->v1.3.5, geoblock v0.2.7->v0.2.8), the entryPoints HTTP->HTTPS redirect with priority 1, and httpClientTimeout: 0. The priority: 1 on that redirect is load-bearing - it keeps the redirect below normal routers so basil's Host(`localhost`) HTTP route still serves. Do not raise it. ## Verified on dev (prkl10), Traefik 3.7.10 7 routers / 8 middlewares, 0 warnings, 0 errors. Certificates issued via DNS-01. pihole and technitium routers preserved. Sites serving 200.
2026-08-05 02:09:45 -06:00
parent ef8c76211e
commit 0f76abd1fa
6 changed files with 263 additions and 41 deletions
--- a/.env.example
+++ b/.env.example
@@ -0,0 +1,30 @@
+# Copy to .env and fill in.  .env is gitignored.
+
+# Hostname for the Traefik dashboard. Must resolve publicly for the
+# Let's Encrypt HTTP-01 challenge to succeed, even though the router itself
+# is additionally restricted to the LAN by internal-whitelist@file.
+#   dev:  traefik-dev.pkartchner.com
+#   prod: traefik.pkartchner.com
+TRAEFIK_DASHBOARD_HOST=traefik-dev.pkartchner.com
+
+# Basic-auth users for the dashboard, htpasswd format.
+#
+# EVERY '$' MUST BE DOUBLED to '$$'. Compose interpolates the substituted value,
+# so a single '$' makes it read "$apr1" as an unset variable and silently
+# collapse the hash to an empty string — which produces a dashboard that
+# rejects every password rather than an obvious error.
+#
+# Generate and escape in one step:
+#   htpasswd -nb admin 'yourpassword' | sed -e 's/\$/\$\$/g'
+TRAEFIK_DASHBOARD_AUTH=admin:$$apr1$$CHANGEME$$REPLACETHISVALUE
+
+# CrowdSec bouncer API key. Generate after the crowdsec stack is up:
+#   docker exec crowdsec cscli bouncers add traefik-bouncer -o raw
+# Consumed by the crowdsec-bouncer middleware defined as a Docker label in
+# docker-compose.yml, so it never appears in any committed file.
+CROWDSEC_LAPI_KEY=
+
+# Static config file for THIS host.
+#   prod: leave unset (defaults to ./traefik.yml, HTTP-01)
+#   dev : ./traefik.dev.yml  (DNS-01, because public port 80 forwards to prod)
+# TRAEFIK_STATIC_CONFIG=./traefik.dev.yml
--- a/.gitignore
+++ b/.gitignore
@@ -1,18 +1,25 @@
-# Traefik sensitive files
+# Secrets — never commit
+.env
 acme.json
+acme.json.bak
 acme.json.backup

+# config.yml is COMMITTED. It is secret-free: the CrowdSec bouncer (the only
+# middleware needing a key) is defined as a Docker label in docker-compose.yml
+# so its key stays in .env.
+
 # Logs
+logs/
 *.log

-# OS files
+# CrowdSec runtime state (separate stack)
+crowdsec/
+
+# OS / IDE
 .DS_Store
 Thumbs.db
-
-# IDE
 .vscode/
 .idea/
 *.swp
 *.swo
 *~
-crowdsec/
--- a/config.yml
+++ b/config.yml
@@ -1,3 +1,8 @@
+# Traefik dynamic configuration.
+#
+# This file is SECRET-FREE and is committed to git. The CrowdSec bouncer, which
+# is the only middleware needing a credential, is defined as a Docker label in
+# docker-compose.yml so its key stays in .env (gitignored).
 http:
  routers:
    # Router for Gogs (if it's running outside Docker or on different network)
@@ -14,6 +19,30 @@ http:
    #   tls:
    #     certResolver: letsencrypt

+    # Pi-hole admin UI - internal only, hosted on 10.20.10.3
+    pihole:
+      rule: "Host(`dns.pkartchner.com`)"
+      entryPoints:
+        - https
+      service: pihole
+      middlewares:
+        - internal-whitelist
+        - secure-headers
+      tls:
+        certResolver: letsencrypt
+
+    # Technitium DNS UI - internal only, hosted on 10.20.10.3
+    technitium:
+      rule: "Host(`technitium.pkartchner.com`)"
+      entryPoints:
+        - https
+      service: technitium
+      middlewares:
+        - internal-whitelist
+        - secure-headers
+      tls:
+        certResolver: letsencrypt
+
  services:
    # Service for Gogs
    # Disabled - now using Gitea with Docker labels
@@ -22,8 +51,20 @@ http:
    #     servers:
    #       - url: "http://gogs.pkartchner.com:3000"

+    # Pi-hole on 10.20.10.3
+    pihole:
+      loadBalancer:
+        servers:
+          - url: "http://10.20.10.3:80"
+
+    # Technitium on 10.20.10.3
+    technitium:
+      loadBalancer:
+        servers:
+          - url: "http://10.20.10.3:5380"
+
  middlewares:
-    # Security headers
+    # Security headers — applied to every public site router.
    secure-headers:
      headers:
        forceSTSHeader: true
@@ -35,26 +76,28 @@ http:
        browserXssFilter: true
        referrerPolicy: "same-origin"

-    # IP whitelist for internal network access only
+    # Restrict a router to the internal LAN only.
    internal-whitelist:
-      ipWhiteList:
+      ipAllowList:
        sourceRange:
          - "10.20.10.0/24"
          - "10.20.140.0/24"
          - "127.0.0.1/32"

-    # Crowdsec bouncer middleware
-    crowdsec-bouncer:
-      plugin:
-        bouncer:
-          enabled: true
-          crowdsecMode: live
-          crowdsecLapiKey: ***REMOVED***
-          crowdsecLapiHost: crowdsec:8080
-          crowdsecLapiScheme: http
-          forwardedHeadersCustomName: X-Custom-Header
+    # NOTE: crowdsec-bouncer is deliberately NOT defined here.
+    #
+    # It carries an API key, and Traefik does not interpolate environment
+    # variables in this file — so defining it here would force a secret into
+    # version control, or force this whole file to be gitignored (which is what
+    # previously happened, costing the repo its "config lives in git" property).
+    #
+    # It is defined as a Docker label on the traefik container instead, where
+    # compose substitutes ${CROWDSEC_LAPI_KEY} from .env.
+    #
+    # Reference it as `crowdsec-bouncer@docker`, NOT `@file`.
+    # This file is therefore secret-free and safe to commit.

-    # GeoIP blocking - Allow only US traffic
+    # GeoIP blocking — allow US traffic only.
    geoblock:
      plugin:
        geoblock:
@@ -65,14 +108,25 @@ http:
          logApiRequests: true
          api: https://get.geojs.io/v1/ip/country/{ip}
          apiTimeoutMs: 750
-          cacheSize: 25
+          # 25 was the upstream default and is far too small: only 25 IPs are
+          # remembered, so nearly every visitor triggers a fresh lookup against
+          # a free third-party API, adding up to 750ms before WordPress starts
+          # and risking rate limits. A few thousand entries is a trivial amount
+          # of memory and turns the API call into a rare event.
+          cacheSize: 5000
          forceMonthlyUpdate: true
-          allowUnknownCountries: false
+          # FAIL OPEN. With this false, a geojs.io outage or slow response makes
+          # every lookup "unknown" and therefore blocked — taking the site down
+          # for US visitors too, because of a third-party service we do not
+          # control sitting in the path of every request. Allowing unknowns
+          # means an outage degrades to "geoblocking temporarily ineffective"
+          # instead of "site is down". CrowdSec still blocks actual attackers.
+          allowUnknownCountries: true
          unknownCountryApiResponse: nil
          countries:
            - US

-    # Rate limiting for Harbor - Prevent brute force attacks
+    # Rate limiting, used by Harbor on prod.
    harbor-ratelimit:
      rateLimit:
        average: 100
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,6 +1,6 @@
 services:
  traefik:
-    image: traefik:v2.10
+    image: traefik:latest
    container_name: traefik
    restart: always
    security_opt:
@@ -9,39 +9,64 @@ services:
      - traefik
      - crowdsec
    ports:
-      - "80:80"      # HTTP
-      - "443:443"    # HTTPS
-      - "8080:8080"  # Traefik Dashboard (optional, can be disabled)
+      - "80:80"
+      - "443:443"
+      # Port 8080 is deliberately NOT published. The dashboard is reached
+      # through the authenticated router below. Publishing 8080 only exposes
+      # the API when `api.insecure: true`, which we do not set.
    environment:
      - TZ=America/Denver
+      # Route 53 DNS-01 challenge. Values live in .env (gitignored).
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+      - AWS_REGION=${AWS_REGION}
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
-      - ./traefik.yml:/traefik.yml:ro
+      # Static config differs between hosts ONLY in the ACME challenge type:
+      # prod uses HTTP-01 (public port 80 lands there), dev must use DNS-01.
+      # Selected via .env so both environments share one branch instead of
+      # diverging permanently.
+      - ${TRAEFIK_STATIC_CONFIG:-./traefik.yml}:/traefik.yml:ro
      - ./acme.json:/acme.json
      - ./config.yml:/config.yml:ro
      - ./logs:/var/log/traefik
    labels:
      - "traefik.enable=true"
-      # Dashboard
-      - "traefik.http.routers.traefik.rule=Host(`traefik.pkartchner.com`)"
+      # Dashboard — LAN-only AND password protected.
+      - "traefik.http.routers.traefik.rule=Host(`${TRAEFIK_DASHBOARD_HOST}`)"
      - "traefik.http.routers.traefik.entrypoints=https"
      - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.middlewares=traefik-auth,internal-whitelist@file"
-      # Dashboard auth (username: admin, password: ***REMOVED***)
-      # Generate new password: echo $(htpasswd -nb admin yourpassword) | sed -e s/\\$/\\$\\$/g
-      - "traefik.http.middlewares.traefik-auth.basicauth.users=***REMOVED***"
-      # Global redirect to HTTPS
-      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
-      - "traefik.http.routers.http-catchall.entrypoints=http"
-      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
+      # Credentials come from .env (gitignored), not from this file.
+      # Generate with: htpasswd -nb admin 'yourpassword'
+      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_AUTH}"
+      # Named redirect middleware, referenced by site stacks on their HTTP router.
      - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
+      # CrowdSec bouncer.
+      #
+      # Defined HERE rather than in config.yml on purpose: Traefik does not
+      # interpolate environment variables in its YAML config files, so the API
+      # key would have to be written literally into config.yml — which is why
+      # that file was previously gitignored and the repo lost its "config lives
+      # in git" property. Compose DOES substitute ${...} in labels, so defining
+      # it here keeps the key in .env alone and lets config.yml be committed.
+      #
+      # Consequence: this middleware is `crowdsec-bouncer@docker`, NOT `@file`.
+      # Site stacks must reference it with the @docker suffix.
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.enabled=true"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecMode=live"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecLapiKey=${CROWDSEC_LAPI_KEY}"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecLapiHost=crowdsec:8080"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecLapiScheme=http"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.forwardedHeadersCustomName=X-Custom-Header"

 networks:
+  # Pre-create once per host:  docker network create traefik
  traefik:
    name: traefik
-    driver: bridge
+    external: true
  crowdsec:
    name: crowdsec
    external: true
--- a/traefik.dev.yml
+++ b/traefik.dev.yml
@@ -0,0 +1,81 @@
+# Traefik static configuration — DEV host (prkl10) ONLY.
+#
+# Identical to traefik.yml except for the ACME challenge type. Select it with
+# TRAEFIK_STATIC_CONFIG=./traefik.dev.yml in .env.
+#
+# Keep any other change in BOTH files, or dev stops mirroring prod.
+
+api:
+  dashboard: true
+  debug: false
+
+experimental:
+  plugins:
+    bouncer:
+      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
+      version: v1.3.5
+    geoblock:
+      moduleName: github.com/PascalMinder/geoblock
+      version: v0.2.8
+
+entryPoints:
+  http:
+    address: ":80"
+    http:
+      redirections:
+        entryPoint:
+          to: https
+          scheme: https
+          permanent: true
+          # priority 1 keeps this redirect BELOW normal routers, which default
+          # to a priority derived from rule length. That is what allows basil's
+          # Host(`localhost`) HTTP router to keep serving without redirecting.
+          # Do not raise this value.
+          priority: 1
+  https:
+    address: ":443"
+
+# Backends here use self-signed certs (harbor, etc). This disables verification
+# of upstream TLS for ALL services — matches prod, but it is a real weakening.
+# Prefer per-service serversTransport if you ever narrow this.
+serversTransport:
+  insecureSkipVerify: true
+
+providers:
+  docker:
+    endpoint: "unix:///var/run/docker.sock"
+    exposedByDefault: false
+    network: traefik
+    httpClientTimeout: 0
+  file:
+    filename: /config.yml
+    watch: true
+
+certificatesResolvers:
+  letsencrypt:
+    acme:
+      email: pkartch@gmail.com
+      storage: acme.json
+      # DNS-01, not HTTP-01. Public port 80 forwards to prod (10.20.10.18) so
+      # that it can renew production certs, which means an HTTP-01 challenge for
+      # a dev hostname lands on prod's Traefik, finds no matching router, and
+      # returns 404. DNS-01 proves control via a Route 53 TXT record instead and
+      # needs no inbound connectivity at all.
+      #
+      # Credentials come from AWS_* in .env (IAM user traefik-dev-dns01, scoped
+      # to the backyardhoneycomb.com and pkartchner.com zones).
+      dnsChallenge:
+        provider: route53
+        # Check propagation against public resolvers. The LAN resolvers
+        # (Pi-hole / Technitium) answer authoritatively for internal names and
+        # will not see the _acme-challenge TXT records, so leaving this unset
+        # makes the pre-check hang until it times out.
+        resolvers:
+          - "1.1.1.1:53"
+          - "8.8.8.8:53"
+
+log:
+  level: INFO
+
+accessLog:
+  filePath: "/var/log/traefik/access.log"
--- a/traefik.yml
+++ b/traefik.yml
@@ -1,3 +1,13 @@
+# Traefik static configuration — DEFAULT / PRODUCTION.
+#
+# Selected by TRAEFIK_STATIC_CONFIG in .env (defaults to this file).
+# The dev host uses traefik.dev.yml, which differs ONLY in the ACME challenge
+# type, because public port 80 forwards to prod so HTTP-01 cannot work there.
+#
+# NOTE: Traefik does NOT interpolate environment variables in this file. Any
+# secret must be a docker-compose label instead, where compose substitutes it
+# from .env. That is why the CrowdSec bouncer is defined in docker-compose.yml.
+
 api:
  dashboard: true
  debug: false
@@ -6,17 +16,31 @@ experimental:
  plugins:
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
-      version: v1.3.3
+      version: v1.3.5
    geoblock:
      moduleName: github.com/PascalMinder/geoblock
-      version: v0.2.7
+      version: v0.2.8

 entryPoints:
  http:
    address: ":80"
+    http:
+      redirections:
+        entryPoint:
+          to: https
+          scheme: https
+          permanent: true
+          # priority 1 keeps this redirect BELOW normal routers, which default
+          # to a priority derived from rule length. That is what allows basil's
+          # Host(`localhost`) HTTP router to keep serving without redirecting.
+          # Do not raise this value.
+          priority: 1
  https:
    address: ":443"

+# Backends here use self-signed certs (harbor, etc). This disables verification
+# of upstream TLS for ALL services — matches prod, but it is a real weakening.
+# Prefer per-service serversTransport if you ever narrow this.
 serversTransport:
  insecureSkipVerify: true

@@ -25,6 +49,7 @@ providers:
    endpoint: "unix:///var/run/docker.sock"
    exposedByDefault: false
    network: traefik
+    httpClientTimeout: 0
  file:
    filename: /config.yml
    watch: true
@@ -34,8 +59,8 @@ certificatesResolvers:
    acme:
      email: pkartch@gmail.com
      storage: acme.json
-      # Uncomment for production (remove caServer line for production)
-      # caServer: https://acme-staging-v02.api.letsencrypt.org/directory
+      # HTTP-01: public port 80 forwards to this host, so the challenge
+      # reaches us directly. The dev host cannot use this — see traefik.dev.yml.
      httpChallenge:
        entryPoint: http