Propagate this build's lessons into the scripts and every runbook

addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild
sets it up like every other host instead of leaving it to be discovered missing.

HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads
hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends
up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply,
and the plan's risks section carries the same warning plus the path-prefix cascade.

LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an
ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than
half-configured; the old Afterwards section is relettered N.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 16:32:09 -06:00
parent 18d443579d
commit 6f77764fec
7 changed files with 95 additions and 16 deletions

View File

@@ -344,11 +344,17 @@ Six containers, all loopback, published only through the gated `diag.` host:
`observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy
- [ ] CNAME `diag` → the main host
- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env`
- [ ] `./configure.py --apply` (pulls ~500 MB of images on the first run)
- [ ] `./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG` — **not `--apply`**: apply reads
the hostnames back out of the Caddyfile, so it cannot introduce a host that is not in there yet and will
leave the new name without a site block or a certificate without reporting anything
(HOSTNAME-CHANGE.md §D). Pulls ~500 MB of images on the first run.
- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing,
`journalctl -u vanguard-deadman` is quiet
- [ ] expect one round of corrections: Vector's journald field names and the LogsQL rule syntax could not be
checked without the services running
- [x] first-run corrections (2026-11-03): `current_boot_only: false` is rejected on systemd 257; the
ingestion path is under `-http.pathPrefix` too (`/logs/insert/elasticsearch/`); every proxied UI needs
its own prefix or it renders broken; a Caddy `respond` body eats `{...}` so the index is a file
- [ ] still to confirm with real traffic: Vector's journald field names (our timers showing the right
`service` label) and the LogsQL rule syntax (`vmalert-logs` logging no errors)
- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong
way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every
invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web
@@ -430,6 +436,12 @@ Checklist if adopted:
- **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1.
- **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
- **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public.
- **A new host needs `--full` or `--hosts`, never `--apply`.** `configure.py` is stateless and reads hostnames
back out of the Caddyfile, so apply silently omits a role that is not already there and the name ends up with
no certificate (HOSTNAME-CHANGE.md §D). Cost an hour when `diag.` was added.
- **Observability path prefixes cascade.** Setting `-http.pathPrefix` (or Alertmanager's `--web.route-prefix`)
moves that service's ingestion, health and metrics paths too, so the Vector sink, the scrape config and the
deadman's probes all have to follow. LOGGING.md lists the four traps this produced.
## Monthly cost
Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free.