Phase 13 integration: install.sh, configure.py, and a deadman that degrades cleanly

install.sh: the three data dirs with correct owners (uid 1000 for the VictoriaMetrics pair, 65534 for
alertmanager), /var/lib/vanguard, the deadman script and timer, and /etc/vanguard/alert-webhook as a FILE —
compose bind-mounts it, so without it Docker creates a directory and alertmanager will not start.

configure.py: --full now asks for the alert webhook and stores it; --show says plainly when none is set; the
webhook lands at 0640 root:65534 rather than world-readable, since it is a credential for that channel.

The deadman no longer assumes a webhook. It trims and validates the file (the previous -s test passed on the
newline configure.py writes), skips delivery when there is none, still logs every finding, and exits non-zero
only when it found something it could not report — so the unit state is the signal. Verified for a missing
file, a newline-only file, a junk value, and the healthy case.

check.sh validates all five observability configs; confirmed it catches a deliberate break.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -320,51 +320,39 @@ permanent reference is needed for long sessions, not just a start burst.
       export at 2560×1440)
 - rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
 
-### Phase 13 — `diag.` host: logs, metrics, alerting, deadman  (NEXT)
+### Phase 13 — `diag.` host: logs, metrics, alerting, deadman  (built; needs DNS + a webhook)
 
-**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`.
-Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one
-host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer).
-VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already
-expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector.
+**Decided 2026-11-03: VictoriaLogs**, on `diag.BASE`, log UI at `/logs`. Rejected: ELK (a JVM heap and a
+growing index on a box that spends its nights running x264, to search one host's logs) and Grafana + Loki (two
+services and a dashboard tool we would only use as a log viewer). The framework everything feeds is
+`LOGGING.md`; as-built detail lives there and in README §5h.
 
+Six containers, all loopback, published only through the gated `diag.` host:
+`victorialogs` (30 d) · `vector` · `victoriametrics` (90 d) · `vmalert-metrics` · `vmalert-logs` · `alertmanager`.
+
+- [x] `configure.py`: `diag` host role, `DIAG_REQUIRE_GROUP` (default `media-admins`), `ALERT_WEBHOOK_URL`
+      asked by `--full`, written to `/etc/vanguard/alert-webhook` (0640 root:65534)
+- [x] `install.sh`: the three data dirs with the right owners, `/var/lib/vanguard`, the webhook file (compose
+      bind-mounts it, so without it Docker would create a *directory* and alertmanager would not start),
+      the deadman script and timer
+- [x] alerting: vmalert per datasource → Alertmanager → Discord. 11 metric rules, 5 LogsQL rules, inhibits so
+      a critical disk does not also report as filling
+- [x] deadman (`bin/vanguard-deadman.sh`, host timer, 5 min): component health, the `Watchdog` heartbeat
+      through vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. Reports on change, announces recovery
+      once, degrades to journal-only with no webhook and exits non-zero so the unit state is the signal
+- [x] collection = **option A**: vector holds the Docker socket. Root-equivalent on this host, recorded in
+      `observability/vector.yaml`. Upgrades: journald log driver everywhere, or a read-only socket proxy
 - [ ] CNAME `diag` → the main host
-- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`)
-- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with
-      `-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not**
-      strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans.
-- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep,
-      trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The
-      `vanguard.service` / `vanguard.category` labels already on every container become stream fields, so
-      filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them
-      (JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx,
-      multiline + regex for jellyfin).
-- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API,
-      and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The
-      alternative is a read-only socket proxy in front of it, one more container but no root. Reading the
-      json files directly avoids both and loses the container labels, which is most of the value.
-- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy`
-      longer than the rest would mean separate tenants — not worth it unless an audit needs it.
-- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
-      (node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
-      would consume them. Not required for logs.
-- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
-      (+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
-      those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
-      covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
-      Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
-      scraping the four taps that are already live. Full picture: four containers.
-- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
-      over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
-      in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
-      failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
-      version for thresholds, grouping and silencing, not a prerequisite for being warned.
-
-**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
-admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
-Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now,
-purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a
-logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command".
+- [ ] a Discord webhook for alerts (its own channel, not the live notices) → `ALERT_WEBHOOK_URL` in `.env`
+- [ ] `./configure.py --apply` (pulls ~500 MB of images on the first run)
+- [ ] verify: all six answer, `/logs` shows events from several services, `/alerts` lists `Watchdog` as firing,
+      `journalctl -u vanguard-deadman` is quiet
+- [ ] expect one round of corrections: Vector's journald field names and the LogsQL rule syntax could not be
+      checked without the services running
+- [ ] later: ops controls on the same host (restart buttons). Design note so it is not built the easy wrong
+      way — allowlisted dumpd endpoints, authorised from the Authentik headers like everything else, every
+      invocation logged as a logfmt event so the action lands beside the logs it is diagnosing. Never a web
+      page with Docker access, never a generic command runner.
 
 ### Phase 14 — push-to-deploy  (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
 - server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved