Harden geoblock, remove secrets from config, support dev host

Refs #1

## Security

* Remove the CrowdSec LAPI key from config.yml. Traefik does not interpolate
  env vars in its YAML config, so the key had to be literal there - which is
  why the file kept drifting out of version control. It is now defined as a
  docker-compose label, where compose substitutes ${CROWDSEC_LAPI_KEY} from
  .env. config.yml is therefore secret-free and stays in git.

  NOTE: this renames the middleware crowdsec-bouncer@file -> @docker.
  Site stacks referencing @file must be updated.

  The old key remains in this repo's history and must still be rotated and
  scrubbed - removing it here does not un-expose it.

* Move the dashboard basic-auth hash out of docker-compose.yml into .env.
  In .env every '$' must be doubled to '$$', or compose silently collapses
  the hash to an empty string and the dashboard rejects every password.

* Stop publishing port 8080. The API is not insecure-mode, so nothing served
  there; publishing it only widened exposure.

## Geoblock (the availability fix)

* allowUnknownCountries: false -> true. Previously a geojs.io outage made
  every lookup "unknown" and therefore blocked, taking sites down for US
  visitors too - a third-party service in the critical path with no degraded
  mode. Now an outage degrades to "geoblocking ineffective". CrowdSec still
  blocks actual attackers.

* cacheSize: 25 -> 5000. At 25, nearly every visitor triggered a fresh API
  call, adding up to 750ms and risking rate limits.

## Dev host support

* Add traefik.dev.yml, selected by TRAEFIK_STATIC_CONFIG in .env (defaults to
  traefik.yml, so prod behaviour is unchanged). It differs from traefik.yml
  ONLY in the ACME challenge type: public port 80 forwards to prod for its
  renewals, so an HTTP-01 challenge for a dev hostname hits prod's Traefik,
  matches no router, and 404s. Dev uses DNS-01 via Route 53 instead, which
  needs no inbound connectivity.

* Add .env.example documenting every variable.

## Captured from prod's working tree

Prod had uncommitted drift; these were running but never committed:
plugin bumps (bouncer v1.3.3->v1.3.5, geoblock v0.2.7->v0.2.8), the
entryPoints HTTP->HTTPS redirect with priority 1, and httpClientTimeout: 0.

The priority: 1 on that redirect is load-bearing - it keeps the redirect
below normal routers so basil's Host(`localhost`) HTTP route still serves.
Do not raise it.

## Verified on dev (prkl10), Traefik 3.7.10

7 routers / 8 middlewares, 0 warnings, 0 errors. Certificates issued via
DNS-01. pihole and technitium routers preserved. Sites serving 200.

.env.example

@@ -0,0 +1,30 @@
+# Copy to .env and fill in.  .env is gitignored.
+
+# Hostname for the Traefik dashboard. Must resolve publicly for the
+# Let's Encrypt HTTP-01 challenge to succeed, even though the router itself
+# is additionally restricted to the LAN by internal-whitelist@file.
+#   dev:  traefik-dev.pkartchner.com
+#   prod: traefik.pkartchner.com
+TRAEFIK_DASHBOARD_HOST=traefik-dev.pkartchner.com
+
+# Basic-auth users for the dashboard, htpasswd format.
+#
+# EVERY '$' MUST BE DOUBLED to '$$'. Compose interpolates the substituted value,
+# so a single '$' makes it read "$apr1" as an unset variable and silently
+# collapse the hash to an empty string — which produces a dashboard that
+# rejects every password rather than an obvious error.
+#
+# Generate and escape in one step:
+#   htpasswd -nb admin 'yourpassword' | sed -e 's/\$/\$\$/g'
+TRAEFIK_DASHBOARD_AUTH=admin:$$apr1$$CHANGEME$$REPLACETHISVALUE
+
+# CrowdSec bouncer API key. Generate after the crowdsec stack is up:
+#   docker exec crowdsec cscli bouncers add traefik-bouncer -o raw
+# Consumed by the crowdsec-bouncer middleware defined as a Docker label in
+# docker-compose.yml, so it never appears in any committed file.
+CROWDSEC_LAPI_KEY=
+
+# Static config file for THIS host.
+#   prod: leave unset (defaults to ./traefik.yml, HTTP-01)
+#   dev : ./traefik.dev.yml  (DNS-01, because public port 80 forwards to prod)
+# TRAEFIK_STATIC_CONFIG=./traefik.dev.yml

.gitignore

@@ -1,18 +1,25 @@
-# Traefik sensitive files
+# Secrets — never commit
+.env
 acme.json
+acme.json.bak
 acme.json.backup
 
+# config.yml is COMMITTED. It is secret-free: the CrowdSec bouncer (the only
+# middleware needing a key) is defined as a Docker label in docker-compose.yml
+# so its key stays in .env.
+
 # Logs
+logs/
 *.log
 
-# OS files
+# CrowdSec runtime state (separate stack)
+crowdsec/
+
+# OS / IDE
 .DS_Store
 Thumbs.db
-
-# IDE
 .vscode/
 .idea/
 *.swp
 *.swo
 *~
-crowdsec/

config.yml

@@ -1,3 +1,8 @@
+# Traefik dynamic configuration.
+#
+# This file is SECRET-FREE and is committed to git. The CrowdSec bouncer, which
+# is the only middleware needing a credential, is defined as a Docker label in
+# docker-compose.yml so its key stays in .env (gitignored).
 http:
   routers:
     # Router for Gogs (if it's running outside Docker or on different network)
@@ -14,6 +19,30 @@ http:
     #   tls:
     #     certResolver: letsencrypt
 
+    # Pi-hole admin UI - internal only, hosted on 10.20.10.3
+    pihole:
+      rule: "Host(`dns.pkartchner.com`)"
+      entryPoints:
+        - https
+      service: pihole
+      middlewares:
+        - internal-whitelist
+        - secure-headers
+      tls:
+        certResolver: letsencrypt
+
+    # Technitium DNS UI - internal only, hosted on 10.20.10.3
+    technitium:
+      rule: "Host(`technitium.pkartchner.com`)"
+      entryPoints:
+        - https
+      service: technitium
+      middlewares:
+        - internal-whitelist
+        - secure-headers
+      tls:
+        certResolver: letsencrypt
+
   services:
     # Service for Gogs
     # Disabled - now using Gitea with Docker labels
@@ -22,8 +51,20 @@ http:
     #     servers:
     #       - url: "http://gogs.pkartchner.com:3000"
 
+    # Pi-hole on 10.20.10.3
+    pihole:
+      loadBalancer:
+        servers:
+          - url: "http://10.20.10.3:80"
+
+    # Technitium on 10.20.10.3
+    technitium:
+      loadBalancer:
+        servers:
+          - url: "http://10.20.10.3:5380"
+
   middlewares:
-    # Security headers
+    # Security headers — applied to every public site router.
     secure-headers:
       headers:
         forceSTSHeader: true
@@ -35,26 +76,28 @@ http:
         browserXssFilter: true
         referrerPolicy: "same-origin"
 
-    # IP whitelist for internal network access only
+    # Restrict a router to the internal LAN only.
     internal-whitelist:
-      ipWhiteList:
+      ipAllowList:
         sourceRange:
           - "10.20.10.0/24"
           - "10.20.140.0/24"
           - "127.0.0.1/32"
 
-    # Crowdsec bouncer middleware
-    crowdsec-bouncer:
-      plugin:
-        bouncer:
-          enabled: true
-          crowdsecMode: live
-          crowdsecLapiKey: ***REMOVED***
-          crowdsecLapiHost: crowdsec:8080
-          crowdsecLapiScheme: http
-          forwardedHeadersCustomName: X-Custom-Header
+    # NOTE: crowdsec-bouncer is deliberately NOT defined here.
+    #
+    # It carries an API key, and Traefik does not interpolate environment
+    # variables in this file — so defining it here would force a secret into
+    # version control, or force this whole file to be gitignored (which is what
+    # previously happened, costing the repo its "config lives in git" property).
+    #
+    # It is defined as a Docker label on the traefik container instead, where
+    # compose substitutes ${CROWDSEC_LAPI_KEY} from .env.
+    #
+    # Reference it as `crowdsec-bouncer@docker`, NOT `@file`.
+    # This file is therefore secret-free and safe to commit.
 
-    # GeoIP blocking - Allow only US traffic
+    # GeoIP blocking — allow US traffic only.
     geoblock:
       plugin:
         geoblock:
@@ -65,14 +108,25 @@ http:
           logApiRequests: true
           api: https://get.geojs.io/v1/ip/country/{ip}
           apiTimeoutMs: 750
-          cacheSize: 25
+          # 25 was the upstream default and is far too small: only 25 IPs are
+          # remembered, so nearly every visitor triggers a fresh lookup against
+          # a free third-party API, adding up to 750ms before WordPress starts
+          # and risking rate limits. A few thousand entries is a trivial amount
+          # of memory and turns the API call into a rare event.
+          cacheSize: 5000
           forceMonthlyUpdate: true
-          allowUnknownCountries: false
+          # FAIL OPEN. With this false, a geojs.io outage or slow response makes
+          # every lookup "unknown" and therefore blocked — taking the site down
+          # for US visitors too, because of a third-party service we do not
+          # control sitting in the path of every request. Allowing unknowns
+          # means an outage degrades to "geoblocking temporarily ineffective"
+          # instead of "site is down". CrowdSec still blocks actual attackers.
+          allowUnknownCountries: true
           unknownCountryApiResponse: nil
           countries:
             - US
 
-    # Rate limiting for Harbor - Prevent brute force attacks
+    # Rate limiting, used by Harbor on prod.
     harbor-ratelimit:
       rateLimit:
         average: 100

docker-compose.yml

@@ -1,6 +1,6 @@
 services:
   traefik:
-    image: traefik:v2.10
+    image: traefik:latest
     container_name: traefik
     restart: always
     security_opt:
@@ -9,39 +9,64 @@ services:
       - traefik
       - crowdsec
     ports:
-      - "80:80"      # HTTP
-      - "443:443"    # HTTPS
-      - "8080:8080"  # Traefik Dashboard (optional, can be disabled)
+      - "80:80"
+      - "443:443"
+      # Port 8080 is deliberately NOT published. The dashboard is reached
+      # through the authenticated router below. Publishing 8080 only exposes
+      # the API when `api.insecure: true`, which we do not set.
     environment:
       - TZ=America/Denver
+      # Route 53 DNS-01 challenge. Values live in .env (gitignored).
+      - AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
+      - AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
+      - AWS_REGION=${AWS_REGION}
     volumes:
       - /etc/localtime:/etc/localtime:ro
       - /var/run/docker.sock:/var/run/docker.sock:ro
-      - ./traefik.yml:/traefik.yml:ro
+      # Static config differs between hosts ONLY in the ACME challenge type:
+      # prod uses HTTP-01 (public port 80 lands there), dev must use DNS-01.
+      # Selected via .env so both environments share one branch instead of
+      # diverging permanently.
+      - ${TRAEFIK_STATIC_CONFIG:-./traefik.yml}:/traefik.yml:ro
       - ./acme.json:/acme.json
       - ./config.yml:/config.yml:ro
       - ./logs:/var/log/traefik
     labels:
       - "traefik.enable=true"
-      # Dashboard
-      - "traefik.http.routers.traefik.rule=Host(`traefik.pkartchner.com`)"
+      # Dashboard — LAN-only AND password protected.
+      - "traefik.http.routers.traefik.rule=Host(`${TRAEFIK_DASHBOARD_HOST}`)"
       - "traefik.http.routers.traefik.entrypoints=https"
       - "traefik.http.routers.traefik.tls.certresolver=letsencrypt"
       - "traefik.http.routers.traefik.service=api@internal"
       - "traefik.http.routers.traefik.middlewares=traefik-auth,internal-whitelist@file"
-      # Dashboard auth (username: admin, password: ***REMOVED***)
-      # Generate new password: echo $(htpasswd -nb admin yourpassword) | sed -e s/\\$/\\$\\$/g
-      - "traefik.http.middlewares.traefik-auth.basicauth.users=***REMOVED***"
-      # Global redirect to HTTPS
-      - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
-      - "traefik.http.routers.http-catchall.entrypoints=http"
-      - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"
+      # Credentials come from .env (gitignored), not from this file.
+      # Generate with: htpasswd -nb admin 'yourpassword'
+      - "traefik.http.middlewares.traefik-auth.basicauth.users=${TRAEFIK_DASHBOARD_AUTH}"
+      # Named redirect middleware, referenced by site stacks on their HTTP router.
       - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"
+      # CrowdSec bouncer.
+      #
+      # Defined HERE rather than in config.yml on purpose: Traefik does not
+      # interpolate environment variables in its YAML config files, so the API
+      # key would have to be written literally into config.yml — which is why
+      # that file was previously gitignored and the repo lost its "config lives
+      # in git" property. Compose DOES substitute ${...} in labels, so defining
+      # it here keeps the key in .env alone and lets config.yml be committed.
+      #
+      # Consequence: this middleware is `crowdsec-bouncer@docker`, NOT `@file`.
+      # Site stacks must reference it with the @docker suffix.
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.enabled=true"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecMode=live"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecLapiKey=${CROWDSEC_LAPI_KEY}"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecLapiHost=crowdsec:8080"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.crowdsecLapiScheme=http"
+      - "traefik.http.middlewares.crowdsec-bouncer.plugin.bouncer.forwardedHeadersCustomName=X-Custom-Header"
 
 networks:
+  # Pre-create once per host:  docker network create traefik
   traefik:
     name: traefik
-    driver: bridge
+    external: true
   crowdsec:
     name: crowdsec
     external: true

traefik.dev.yml

@@ -0,0 +1,81 @@
+# Traefik static configuration — DEV host (prkl10) ONLY.
+#
+# Identical to traefik.yml except for the ACME challenge type. Select it with
+# TRAEFIK_STATIC_CONFIG=./traefik.dev.yml in .env.
+#
+# Keep any other change in BOTH files, or dev stops mirroring prod.
+
+api:
+  dashboard: true
+  debug: false
+
+experimental:
+  plugins:
+    bouncer:
+      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
+      version: v1.3.5
+    geoblock:
+      moduleName: github.com/PascalMinder/geoblock
+      version: v0.2.8
+
+entryPoints:
+  http:
+    address: ":80"
+    http:
+      redirections:
+        entryPoint:
+          to: https
+          scheme: https
+          permanent: true
+          # priority 1 keeps this redirect BELOW normal routers, which default
+          # to a priority derived from rule length. That is what allows basil's
+          # Host(`localhost`) HTTP router to keep serving without redirecting.
+          # Do not raise this value.
+          priority: 1
+  https:
+    address: ":443"
+
+# Backends here use self-signed certs (harbor, etc). This disables verification
+# of upstream TLS for ALL services — matches prod, but it is a real weakening.
+# Prefer per-service serversTransport if you ever narrow this.
+serversTransport:
+  insecureSkipVerify: true
+
+providers:
+  docker:
+    endpoint: "unix:///var/run/docker.sock"
+    exposedByDefault: false
+    network: traefik
+    httpClientTimeout: 0
+  file:
+    filename: /config.yml
+    watch: true
+
+certificatesResolvers:
+  letsencrypt:
+    acme:
+      email: pkartch@gmail.com
+      storage: acme.json
+      # DNS-01, not HTTP-01. Public port 80 forwards to prod (10.20.10.18) so
+      # that it can renew production certs, which means an HTTP-01 challenge for
+      # a dev hostname lands on prod's Traefik, finds no matching router, and
+      # returns 404. DNS-01 proves control via a Route 53 TXT record instead and
+      # needs no inbound connectivity at all.
+      #
+      # Credentials come from AWS_* in .env (IAM user traefik-dev-dns01, scoped
+      # to the backyardhoneycomb.com and pkartchner.com zones).
+      dnsChallenge:
+        provider: route53
+        # Check propagation against public resolvers. The LAN resolvers
+        # (Pi-hole / Technitium) answer authoritatively for internal names and
+        # will not see the _acme-challenge TXT records, so leaving this unset
+        # makes the pre-check hang until it times out.
+        resolvers:
+          - "1.1.1.1:53"
+          - "8.8.8.8:53"
+
+log:
+  level: INFO
+
+accessLog:
+  filePath: "/var/log/traefik/access.log"

traefik.yml

@@ -1,3 +1,13 @@
+# Traefik static configuration — DEFAULT / PRODUCTION.
+#
+# Selected by TRAEFIK_STATIC_CONFIG in .env (defaults to this file).
+# The dev host uses traefik.dev.yml, which differs ONLY in the ACME challenge
+# type, because public port 80 forwards to prod so HTTP-01 cannot work there.
+#
+# NOTE: Traefik does NOT interpolate environment variables in this file. Any
+# secret must be a docker-compose label instead, where compose substitutes it
+# from .env. That is why the CrowdSec bouncer is defined in docker-compose.yml.
+
 api:
   dashboard: true
   debug: false
@@ -6,17 +16,31 @@ experimental:
   plugins:
     bouncer:
       moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
-      version: v1.3.3
+      version: v1.3.5
     geoblock:
       moduleName: github.com/PascalMinder/geoblock
-      version: v0.2.7
+      version: v0.2.8
 
 entryPoints:
   http:
     address: ":80"
+    http:
+      redirections:
+        entryPoint:
+          to: https
+          scheme: https
+          permanent: true
+          # priority 1 keeps this redirect BELOW normal routers, which default
+          # to a priority derived from rule length. That is what allows basil's
+          # Host(`localhost`) HTTP router to keep serving without redirecting.
+          # Do not raise this value.
+          priority: 1
   https:
     address: ":443"
 
+# Backends here use self-signed certs (harbor, etc). This disables verification
+# of upstream TLS for ALL services — matches prod, but it is a real weakening.
+# Prefer per-service serversTransport if you ever narrow this.
 serversTransport:
   insecureSkipVerify: true
 
@@ -25,6 +49,7 @@ providers:
     endpoint: "unix:///var/run/docker.sock"
     exposedByDefault: false
     network: traefik
+    httpClientTimeout: 0
   file:
     filename: /config.yml
     watch: true
@@ -34,8 +59,8 @@ certificatesResolvers:
     acme:
       email: pkartch@gmail.com
       storage: acme.json
-      # Uncomment for production (remove caServer line for production)
-      # caServer: https://acme-staging-v02.api.letsencrypt.org/directory
+      # HTTP-01: public port 80 forwards to this host, so the challenge
+      # reaches us directly. The dev host cannot use this — see traefik.dev.yml.
       httpChallenge:
         entryPoint: http