Propagate this build's lessons into the scripts and every runbook

addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild
sets it up like every other host instead of leaving it to be discovered missing.

HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads
hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends
up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply,
and the plan's risks section carries the same warning plus the path-prefix cascade.

LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an
ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than
half-configured; the old Afterwards section is relettered N.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -344,11 +344,17 @@ Six containers, all loopback, published only through the gated `diag.` host:
       `observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy
 - [ ] CNAME `diag` → the main host
 - [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env`
-- [ ] `./configure.py --apply` (pulls ~500 MB of images on the first run)
+- [ ] `./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG` — **not `--apply`**: apply reads
+      the hostnames back out of the Caddyfile, so it cannot introduce a host that is not in there yet and will
+      leave the new name without a site block or a certificate without reporting anything
+      (HOSTNAME-CHANGE.md §D). Pulls ~500 MB of images on the first run.
 - [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing,
       `journalctl -u vanguard-deadman` is quiet
-- [ ] expect one round of corrections: Vector's journald field names and the LogsQL rule syntax could not be
+- [x] first-run corrections (2026-11-03): `current_boot_only: false` is rejected on systemd 257; the
-      checked without the services running
+      ingestion path is under `-http.pathPrefix` too (`/logs/insert/elasticsearch/`); every proxied UI needs
+      its own prefix or it renders broken; a Caddy `respond` body eats `{...}` so the index is a file
+- [ ] still to confirm with real traffic: Vector's journald field names (our timers showing the right
+      `service` label) and the LogsQL rule syntax (`vmalert-logs` logging no errors)
 - [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong
       way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every
       invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web
@@ -430,6 +436,12 @@ Checklist if adopted:
 - **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1.
 - **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
 - **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public.
+- **A new host needs `--full` or `--hosts`, never `--apply`.** `configure.py` is stateless and reads hostnames
+  back out of the Caddyfile, so apply silently omits a role that is not already there and the name ends up with
+  no certificate (HOSTNAME-CHANGE.md §D). Cost an hour when `diag.` was added.
+- **Observability path prefixes cascade.** Setting `-http.pathPrefix` (or Alertmanager's `--web.route-prefix`)
+  moves that service's ingestion, health and metrics paths too, so the Vector sink, the scrape config and the
+  deadman's probes all have to follow. LOGGING.md lists the four traps this produced.
 
 ## Monthly cost
 Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free.

FRESH-INSTALL.md

@@ -5,7 +5,7 @@ Everything in order. Scripted steps are commands; manual steps are click lists.
 README.md (section numbers given).
 
 ## 0. Before touching the server
-- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `auth`, `watch`, `git`, `bot` → `HOST`.
+- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `diag`, `auth`, `watch`, `git`, `bot` → `HOST`.
   Optional: the bare domain `A` → server IP (it redirects to `https://HOST/`).
 - The Discord bot lives in its **own** repo (ours `VanguardOrg/VanguardBot`), not in this one. Nothing here
   needs it to exist; section 7a wires it up afterwards.
@@ -41,7 +41,7 @@ build, **firewall** (keep this session open; test a new ssh in a second terminal
 
     cd ~/vanguard-stream && ./configure.py --full
 
-Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it
+Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`diag`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it
 redirects; Discord bot token + server ID (optional now, needed for the plugin feed); "is live" webhook (optional).
 Generates `.env` with every secret, the configs, starts all containers, waits, then runs the post-start fixups
 (Syncthing GUI auth off, Gitea admin user, Jellyfin plugin). `./configure.py --show` prints every credential
@@ -117,6 +117,16 @@ Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` fro
 6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the
    container. By hand: `./deployBotBundle.sh <bundle.zip>`; back out with `--rollback`.
 
+## 7b. Diagnostics — logs, metrics, alerts (README §5h)
+1. The `diag` CNAME must exist first. Six containers come up from `configure.py`: VictoriaLogs + Vector,
+   VictoriaMetrics, two vmalert instances and Alertmanager, all loopback, published only through `diag.BASE`
+   behind the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
+2. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it the stack is still correct: alerts evaluate, group and
+   show at `/alerts`, they are simply not delivered, and the deadman logs instead of posting.
+3. The deadman (`vanguard-deadman.timer`) is installed by `install.sh` and by every apply. It is the only thing
+   that can report the observability stack being dead, so check it runs: `systemctl list-timers vanguard-deadman.timer`.
+4. If this host is being ADDED to a running server, use `--full` or `--hosts` — not `--apply` (HOSTNAME-CHANGE.md §D).
+
 ## 8. Pipeline (README "Deployment")
 1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo
    → `./switchToClone.sh clone` (keeps `.env`, generated configs, plugin zips) → `./configure.py --apply`.
@@ -341,8 +351,21 @@ L7. dev$ ssh debian@HOST 'docker logs discord-bot --tail 20'
 L8. Private window → https://bot.BASE/ → Discord login → the bot's pages. Needs the `bot-users` group (admins
     need it too, or set BOT_REQUIRE_GROUP= and let the bot's WEB_ALLOWED_GROUPS decide).
 
-## M. Afterwards
+## M. Diagnostics (browser at https://diag.BASE/)
 
-M1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"'  → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply.
+M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "victoria|vector|vmalert|alertmanager"'
-M2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update).
+    → five Up. Vector restarting means a config error: docker logs vector --tail 20.
-M3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7.
+M2. https://diag.BASE/ → Discord login → the index. /logs is the one to open first; pick a service from the
+    stream fields and you should see several distinct ones.
+M3. https://diag.BASE/alerts → `Watchdog` listed as firing. That alert fires on purpose: it is the heartbeat
+    the deadman watches.
+M4. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
+    → no output and exit 0. Exit 1 lists what is wrong and, with no webhook set, says it could not send it.
+M5. Set `ALERT_WEBHOOK_URL` in `.env` when you have a channel for it, then `./configure.py --apply` — that
+    regenerates `observability/alertmanager.yml` WITH the Discord receiver, which is omitted while it is empty.
+
+## N. Afterwards
+
+N1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"'  → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply.
+N2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update).
+N3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7.

HOSTNAME-CHANGE.md

@@ -15,7 +15,7 @@ Roles: `main` (gateway page at `/`), `files` (copyparty), `clips`, `sync`, `live
 | Place | Holds |
 |---|---|
 | `Caddyfile` | every site block — the source of truth `configure.py` reads back (`# vanguard:<role>` markers) |
-| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT` |
+| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `DIAG_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT` |
 | `mediamtx.yml` | `webrtcAdditionalHosts: [MAIN_HOST]` (needs a MediaMTX restart to take effect) |
 | `docker-compose.yml` → container env | Jellyfin `PublishedServerUrl` (WATCH), Gitea `DOMAIN/ROOT_URL/SSH_DOMAIN` (GIT), bot `WEB_PUBLIC_URL` (BOT), MediaMTX `LIVE_HOST` (text of the "is live" post), act_runner `GITEA_INSTANCE_URL` (GIT). `compose up -d` in the apply recreates the affected containers (seconds each) |
 | Caddy TLS | certificates for new names are fetched automatically; old ones linger harmlessly |
@@ -122,8 +122,30 @@ name under BASE, the outpost host (`auth.`) is unchanged.
 5. Discord: re-pin `https://media.BASE/` (members), `https://clips.BASE/` (gamers), `https://files.BASE/` (editors).
 6. Old `ovh1` bookmarks stop working (no Caddy block, TLS fails) — point people at the gateway.
 
+## D. Adding a NEW host
+
+**`./configure.py --apply` cannot do this, and will not complain.** It is stateless by design: it reads the
+hostnames back out of the `Caddyfile` (`# vanguard:<role>` markers), so a role that is not in there yet has no
+value to write and the site is simply left out. The containers come up, Caddy reloads, and the new name has no
+site block and therefore no certificate — which presents as an SSL handshake error, not as an error from the
+apply. This cost an hour on 2026-11-03 when `diag.` was added.
+
+Introduce a new host with **`--full`** (asks for every hostname, defaults from the current config) or with
+**`--hosts`**, which is the non-interactive form and takes them positionally:
+
+    ./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG     # "" leaves one unset
+
+After that, `--apply` maintains it like any other host, because the marker is now in the file.
+
+1. DNS: `CNAME` (or `A`) for the new name → the box; check it resolves before continuing.
+2. Add the role to `configure.py`: a `write_caddy` block, a `<ROLE>_REQUIRE_GROUP` default in `write_env`,
+   the prompt in `full()`, the positional in `--hosts`, and the line in `show()`.
+3. `./configure.py --hosts …` with every hostname including the new one.
+4. Verify the marker landed: `grep -c "vanguard:<role>" Caddyfile` → 1. Then `curl -sI https://NEW/` → 302.
+5. Add it to `addCommunityMediaServer.sh` so a rebuild asks for it too.
+
 ## Checklist after any change
-    dev$ for h in HOST files.BASE clips.BASE sync.BASE live.BASE auth.BASE watch.BASE git.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done
+    dev$ for h in HOST files.BASE clips.BASE sync.BASE live.BASE diag.BASE auth.BASE watch.BASE git.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done
     https://HOST/whoami            → your identity (gate works on the new name)
     https://watch.BASE/sso/OID/start/authentik   → Jellyfin SSO round trip
     https://git.BASE/user/login    → "Sign in with authentik" round trip

LOGGING.md

@@ -108,6 +108,20 @@ a signal. Nothing inside the stack can report its own death; this can.
 a working configuration, not an error: `configure.py` generates `observability/alertmanager.yml` with the
 Discord notifier omitted, so nothing is half-configured and nothing fails to load.
 
+## Gotchas this build produced (all fixed; here so they are not rediscovered)
+
+- **`-http.pathPrefix` applies to EVERY endpoint, not just the UI.** With VictoriaLogs behind `/logs`, the
+  ingestion path is `/logs/insert/elasticsearch/`, the health probe is `/logs/health` and the scrape is
+  `/logs/metrics`. It rejects a write to the unprefixed path with a message naming the missing flag.
+- **Every proxied UI needs its own prefix set**, or it loads at the sub-path and then fetches its assets from
+  the domain root and renders broken. VictoriaMetrics and vmalert use `-http.pathPrefix`; Alertmanager uses
+  `--web.route-prefix` plus `--web.external-url`. Setting those also moves their metrics and health paths.
+- **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped;
+  earlier entries are still on the host for `journalctl --boot=-1`.
+- **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a
+  file (`diag/index.html`, mounted like the gateway and dock pages) instead of inlining it.
+- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
+
 ## Wiring sketches (the options that were weighed)
 
 - **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container

README.md

@@ -7,6 +7,10 @@
 
 **Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
 
+> Adding a host later needs `./configure.py --full` or `--hosts`, **not** `--apply`: apply reads the hostnames
+> back out of the Caddyfile, so a role that is not in there yet is silently left out and the new name ends up
+> with no certificate. HOSTNAME-CHANGE.md §D has the procedure.
+
 | Host | What | Who |
 |---|---|---|
 | `media.BASE` (main) | gateway page at `/` (role-aware links + how-tos), `/whoami`, `/logout` | everyone |
@@ -14,6 +18,7 @@
 | `clips.BASE` | self-service clip intake (pair your Syncthing) | any member |
 | `sync.BASE` | Syncthing admin UI | media-admins |
 | `live.BASE` | live portal, Go Live, OBS dock `/dock/`, clock source `/dock/clock/`, `/dump`, `/me`, `/time` | everyone (`LIVE_REQUIRE_GROUP`) |
+| `diag.BASE` | logs, metrics, alerts (§5h) | media-admins |
 | `watch.BASE` | Jellyfin | Discord via OIDC |
 | `git.BASE`, `bot.BASE`, `auth.BASE` | Gitea, bot web UI (`bot-users`), Authentik | — |
 

STATUS.md

@@ -59,7 +59,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | normalize sweep | editing standard, one copy per clip | live |
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
 | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
-| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **committed 2026-11-03, deploy pending**: needs the `diag` CNAME and `ALERT_WEBHOOK_URL`; expect one round of corrections to Vector's journald fields and the LogsQL rules |
+| Observability (Phase 13) | VictoriaLogs + Vector (logs, 30 d), VictoriaMetrics (metrics, 90 d, scrapes node_exporter / cAdvisor / MediaMTX / Caddy), vmalert ×2 + Alertmanager → Discord, all behind `diag.` (`media-admins`) | **deploying 2026-11-03**: containers up; four first-run corrections applied (systemd 257 rejects `current_boot_only: false`; `-http.pathPrefix` moves the ingestion path too; each proxied UI needs its own prefix; a Caddy `respond` body eats `{...}`). `diag.` needed `--hosts`, not `--apply` — apply cannot introduce a host (HOSTNAME-CHANGE.md §D). No `ALERT_WEBHOOK_URL` yet, which is a supported state: alerts evaluate and show at `/alerts` but are not delivered |
 | Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: all six components answering, the `Watchdog` heartbeat through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **committed 2026-11-03, deploy pending** |
 | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |

addCommunityMediaServer.sh

@@ -39,11 +39,13 @@ SYNC_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("sync",""
 SYNC=$(ask "Syncthing UI hostname" "${SYNC_CUR:-sync.$BASE}")
 CLIPS_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("clips",""))')
 CLIPS=$(ask "Clips self-service hostname" "${CLIPS_CUR:-clips.$BASE}")
+DIAG_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("diag",""))')
+DIAG=$(ask "Diagnostics hostname: logs, metrics, alerts" "${DIAG_CUR:-diag.$BASE}")
 
 # ---- DNS sanity ----------------------------------------------------------------------------------
 say "DNS check"
 MAIN_IP=$(ip_of "$MAIN"); ok=1
-for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS"; do
+for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG"; do
   ip=$(ip_of "$h")
   if [[ -z $ip ]]; then echo "  $h: does not resolve yet"; ok=0
   elif [[ $ip != "$MAIN_IP" ]]; then echo "  $h -> $ip (main is $MAIN_IP) — points somewhere else"; ok=0
@@ -56,7 +58,7 @@ fi
 
 # ---- write configs, .env, start containers -----------------------------------------------------
 say "Applying (configure.py --hosts)"
-VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" | sed -n '/== writing/,$p' | grep -vE '^  (techy|diceofire|[a-z0-9_-]+)  password' || true
+VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG" | sed -n '/== writing/,$p' | grep -vE '^  (techy|diceofire|[a-z0-9_-]+)  password' || true
 [[ $DRY == 1 ]] && { echo "(dry run: skipping container wait)"; exit 0; }
 
 # ---- wait ---------------------------------------------------------------------------------------
@@ -156,14 +158,15 @@ PHASE 3 — Jellyfin (direct play only on this CPU)
           Custom CSS:       paste jellyfin/branding.css
         -> Save; reload https://$WATCH/
 
-AUTHENTIK FORWARD-AUTH (gates every Vanguard host: $MAIN, $FILES, $CLIPS, $SYNC, $LIVE, $BOT) — once:
+AUTHENTIK FORWARD-AUTH (gates every Vanguard host: $MAIN, $FILES, $CLIPS, $SYNC, $LIVE, $DIAG, $BOT) — once:
   1. Applications -> Outposts -> edit "authentik Embedded Outpost" -> config: authentik_host: https://$AUTH
   2. Applications -> Create with provider -> Application "Vanguard sites"; Proxy Provider, authorization
      flow implicit-consent, mode "Forward auth (domain level)", authentication URL https://$AUTH,
      cookie domain ${BASE}, Invalidation flow default-provider-invalidation-flow (logout = full re-login)
      -> then edit the Embedded Outpost -> Applications: add it -> Update
   3. Per-host access is enforced by Caddy from X-Authentik-Groups: .env BOT_REQUIRE_GROUP=bot-users
-     (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member). ./configure.py apply regenerates.
+     (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member), DIAG_REQUIRE_GROUP=media-admins.
+   ./configure.py apply regenerates.
   4. Discord roles -> Authentik groups: Directory -> Groups: create bot-users (keep streamers/editors/admins);
      Discord source -> Scopes: "guilds guilds.members.read"; Customization -> Policies -> Expression Policy
      "discord-roles-to-groups" from authentik/discord-roles-to-groups.py (server id + role-id->group table);