Propagate this build's lessons into the scripts and every runbook
addCommunityMediaServer.sh asks for the diag hostname, DNS-checks it and passes it to --hosts, so a rebuild sets it up like every other host instead of leaving it to be discovered missing. HOSTNAME-CHANGE.md gains §D, adding a new host, which is where the hour went: --apply is stateless and reads hostnames back out of the Caddyfile, so it silently omits a role that is not already there and the name ends up with no certificate. README and the Phase 13 checklist now point at --full or --hosts instead of --apply, and the plan's risks section carries the same warning plus the path-prefix cascade. LOGGING.md collects the four traps this produced so none of them is rediscovered. FRESH-INSTALL gains an ordered §7b and appendix M for the diag host, including that running without a webhook is correct rather than half-configured; the old Afterwards section is relettered N. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -344,11 +344,17 @@ Six containers, all loopback, published only through the gated `diag.` host:
|
||||
`observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy
|
||||
- [ ] CNAME `diag` → the main host
|
||||
- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env`
|
||||
- [ ] `./configure.py --apply` (pulls ~500 MB of images on the first run)
|
||||
- [ ] `./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG` — **not `--apply`**: apply reads
|
||||
the hostnames back out of the Caddyfile, so it cannot introduce a host that is not in there yet and will
|
||||
leave the new name without a site block or a certificate without reporting anything
|
||||
(HOSTNAME-CHANGE.md §D). Pulls ~500 MB of images on the first run.
|
||||
- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing,
|
||||
`journalctl -u vanguard-deadman` is quiet
|
||||
- [ ] expect one round of corrections: Vector's journald field names and the LogsQL rule syntax could not be
|
||||
checked without the services running
|
||||
- [x] first-run corrections (2026-11-03): `current_boot_only: false` is rejected on systemd 257; the
|
||||
ingestion path is under `-http.pathPrefix` too (`/logs/insert/elasticsearch/`); every proxied UI needs
|
||||
its own prefix or it renders broken; a Caddy `respond` body eats `{...}` so the index is a file
|
||||
- [ ] still to confirm with real traffic: Vector's journald field names (our timers showing the right
|
||||
`service` label) and the LogsQL rule syntax (`vmalert-logs` logging no errors)
|
||||
- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong
|
||||
way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every
|
||||
invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web
|
||||
@@ -430,6 +436,12 @@ Checklist if adopted:
|
||||
- **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1.
|
||||
- **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
|
||||
- **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public.
|
||||
- **A new host needs `--full` or `--hosts`, never `--apply`.** `configure.py` is stateless and reads hostnames
|
||||
back out of the Caddyfile, so apply silently omits a role that is not already there and the name ends up with
|
||||
no certificate (HOSTNAME-CHANGE.md §D). Cost an hour when `diag.` was added.
|
||||
- **Observability path prefixes cascade.** Setting `-http.pathPrefix` (or Alertmanager's `--web.route-prefix`)
|
||||
moves that service's ingestion, health and metrics paths too, so the Vector sink, the scrape config and the
|
||||
deadman's probes all have to follow. LOGGING.md lists the four traps this produced.
|
||||
|
||||
## Monthly cost
|
||||
Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free.
|
||||
|
||||
Reference in New Issue
Block a user