Propagate this build's lessons into the scripts and every runbook

addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild
sets it up like every other host instead of leaving it to be discovered missing.

HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads
hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends
up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply,
and the plan's risks section carries the same warning plus the path-prefix cascade.

LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an
ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than
half-configured; the old Afterwards section is relettered N.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 16:32:09 -06:00
parent 18d443579d
commit 6f77764fec
7 changed files with 95 additions and 16 deletions

View File

@@ -108,6 +108,20 @@ a signal. Nothing inside the stack can report its own death; this can.
a working configuration, not an error: `configure.py` generates `observability/alertmanager.yml` with the
Discord notifier omitted, so nothing is half-configured and nothing fails to load.
## Gotchas this build produced (all fixed; here so they are not rediscovered)
- **`-http.pathPrefix` applies to EVERY endpoint, not just the UI.** With VictoriaLogs behind `/logs`, the
ingestion path is `/logs/insert/elasticsearch/`, the health probe is `/logs/health` and the scrape is
`/logs/metrics`. It rejects a write to the unprefixed path with a message naming the missing flag.
- **Every proxied UI needs its own prefix set**, or it loads at the sub-path and then fetches its assets from
the domain root and renders broken. VictoriaMetrics and vmalert use `-http.pathPrefix`; Alertmanager uses
`--web.route-prefix` plus `--web.external-url`. Setting those also moves their metrics and health paths.
- **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped;
earlier entries are still on the host for `journalctl --boot=-1`.
- **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a
file (`diag/index.html`, mounted like the gateway and dock pages) instead of inlining it.
- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
## Wiring sketches (the options that were weighed)
- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container