Prod geoblock: raise cacheSize and fail open when geojs.io is unreachable #1

New Issue

pkartch · 2026-08-05T07:52:49Z

pkartch commented

2026-08-05 07:52:49 +00:00

Two problems in the geoblock middleware in config.yml. Both were found and fixed on the dev host (prkl10) while building the WordPress migration stack; prod still carries them.

1. Site goes down if geojs.io does (availability risk)

allowUnknownCountries: false

Every non-cached request makes a live call to https://get.geojs.io/v1/ip/country/{ip} with a 750ms timeout. When that call fails or times out, the country is "unknown", and with this set to false an unknown country is blocked.

So if geojs.io is down, slow, or rate-limits us, every visitor is blocked, including US visitors. A free third-party service we do not control sits in the critical path of every request to every geoblocked site on this host, with no fallback and no degraded mode.

2. `cacheSize: 25` is far too small (latency + rate-limit risk)

cacheSize: 25

Only 25 IPs are remembered, so on any site with real traffic nearly every request becomes a fresh API call. That adds up to 750ms before WordPress even starts, and generates large call volume against a free API that may throttle us — which then triggers problem 1.

Fix

-          cacheSize: 25
+          cacheSize: 5000
-          allowUnknownCountries: false
+          allowUnknownCountries: true

With these, a geojs.io outage degrades to "geoblocking temporarily ineffective" instead of "site down". CrowdSec still blocks actual attackers, so the security posture does not depend on geoblock being up. 5000 cached entries is negligible memory and turns the API call into a rare event.

Gotcha: a restart is required

config.yml is bind-mounted as a single file. Editing it with anything that writes atomically (temp file + rename, which most editors do) creates a new inode, so the container keeps seeing the old file and Traefik's watch: true never fires. This is silent — the config on disk looks correct while the running container uses the old values.

After editing, verify from inside the container:

docker exec traefik sh -c 'grep -E "cacheSize:|allowUnknownCountries:" /config.yml'
docker compose up -d --force-recreate traefik

Worth considering mounting the directory instead of individual files to avoid this class of problem.

Verification

docker exec traefik sh -c 'grep -E "cacheSize:|allowUnknownCountries:" /config.yml'
curl -s -u admin:<pw> https://traefik.pkartchner.com/api/http/middlewares | grep -o 'geoblock[^,]*'

Expect status=enabled, error=none, and the new values.

apiTimeoutMs: 750 — less critical once failures fail open, but still 750ms of worst-case added latency per uncached visitor.
traefik.pkartchner.com currently resolves to the dev host (38.78.244.119 via prkhome), not prod, so prod's dashboard is not reachable at its own hostname.
The CrowdSec LAPI key and the dashboard basic-auth hash are committed in this repo's history and should be rotated with a history scrub. config.yml is now gitignored on dev, with config.yml.example committed in its place.

Two problems in the `geoblock` middleware in `config.yml`. Both were found and fixed on the dev host (`prkl10`) while building the WordPress migration stack; prod still carries them. ## 1. Site goes down if geojs.io does (availability risk) ```yaml allowUnknownCountries: false ``` Every non-cached request makes a live call to `https://get.geojs.io/v1/ip/country/{ip}` with a 750ms timeout. When that call fails or times out, the country is "unknown", and with this set to `false` an unknown country is **blocked**. So if geojs.io is down, slow, or rate-limits us, **every visitor is blocked, including US visitors**. A free third-party service we do not control sits in the critical path of every request to every geoblocked site on this host, with no fallback and no degraded mode. ## 2. `cacheSize: 25` is far too small (latency + rate-limit risk) ```yaml cacheSize: 25 ``` Only 25 IPs are remembered, so on any site with real traffic nearly every request becomes a fresh API call. That adds up to 750ms before WordPress even starts, and generates large call volume against a free API that may throttle us — which then triggers problem 1. ## Fix ```diff - cacheSize: 25 + cacheSize: 5000 - allowUnknownCountries: false + allowUnknownCountries: true ``` With these, a geojs.io outage degrades to "geoblocking temporarily ineffective" instead of "site down". CrowdSec still blocks actual attackers, so the security posture does not depend on geoblock being up. 5000 cached entries is negligible memory and turns the API call into a rare event. ## Gotcha: a restart is required `config.yml` is bind-mounted as a **single file**. Editing it with anything that writes atomically (temp file + rename, which most editors do) creates a new inode, so the container keeps seeing the old file and Traefik's `watch: true` never fires. This is silent — the config on disk looks correct while the running container uses the old values. After editing, verify from **inside** the container: ```bash docker exec traefik sh -c 'grep -E "cacheSize:|allowUnknownCountries:" /config.yml' docker compose up -d --force-recreate traefik ``` Worth considering mounting the directory instead of individual files to avoid this class of problem. ## Verification ```bash docker exec traefik sh -c 'grep -E "cacheSize:|allowUnknownCountries:" /config.yml' curl -s -u admin:<pw> https://traefik.pkartchner.com/api/http/middlewares | grep -o 'geoblock[^,]*' ``` Expect `status=enabled`, `error=none`, and the new values. ## Related, not fixed here - `apiTimeoutMs: 750` — less critical once failures fail open, but still 750ms of worst-case added latency per uncached visitor. - `traefik.pkartchner.com` currently resolves to the **dev** host (38.78.244.119 via `prkhome`), not prod, so prod's dashboard is not reachable at its own hostname. - The CrowdSec LAPI key and the dashboard basic-auth hash are committed in this repo's history and should be rotated with a history scrub. `config.yml` is now gitignored on dev, with `config.yml.example` committed in its place.

pkartch referenced this issue from a commit

2026-08-05 08:09:48 +00:00

Harden geoblock, remove secrets from config, support dev host

pkartch referenced a pull request that will close this issue

2026-08-05 08:10:50 +00:00

Harden geoblock, remove secrets from config, support dev host #2

pkartch closed this issue

2026-08-05 08:12:07 +00:00

pkartch referenced this issue from a commit

2026-08-05 08:15:50 +00:00

Harden geoblock, remove secrets from config, support dev host

Sign in to join this conversation.