Phase 14: diag. host with VictoriaLogs at /logs, ops controls later

Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and
path, the collector choice, and the one real decision left for build time: how Vector reads container logs
without handing it root through the Docker socket. Also a design note for the later ops controls, so restart
buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CommunityDiscordMedia.md

@@ -384,6 +384,44 @@ Checklist if adopted:
       Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
 - [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron
 
+### Phase 14 — `diag.` host: logs now, ops controls later
+
+**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`.
+Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one
+host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer).
+VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already
+expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector.
+
+- [ ] CNAME `diag` → the main host
+- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`)
+- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with
+      `-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not**
+      strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans.
+- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep,
+      trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The
+      `vanguard.service` / `vanguard.category` labels already on every container become stream fields, so
+      filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them
+      (JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx,
+      multiline + regex for jellyfin).
+- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API,
+      and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The
+      alternative is a read-only socket proxy in front of it, one more container but no root. Reading the
+      json files directly avoids both and loses the container labels, which is most of the value.
+- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy`
+      longer than the rest would mean separate tenants — not worth it unless an audit needs it.
+- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
+      (node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
+      would consume them. Not required for logs.
+- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep
+      `event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook.
+
+**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
+admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
+Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now,
+purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a
+logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command".
+
+
 ## Risks / gotchas
 
 - **Phase 5 lockout.** Keep one SSH session open and a copyparty local admin account until the

LOGGING.md

@@ -1,7 +1,8 @@
-# Logging framework — store-agnostic
+# Logging framework
 
-Decided 2026-11-01. The log *store* (Grafana + Loki, VictoriaLogs, Netdata, …) is not chosen yet. Everything
+Decided 2026-11-01, and the store chosen on 2026-11-03: **VictoriaLogs**, on the `diag.BASE` host with its UI
-upstream of it is fixed now, so the choice later is a collector config, not a code change.
+at `/logs` (Phase 14 in `CommunityDiscordMedia.md`). Everything upstream of the store was fixed first, so
+adopting it is a collector config rather than a code change.
 
 ## Where the lines come from (the only two taps a collector needs)
 
@@ -76,17 +77,19 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie
 `X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the
 "who downloaded what" audit trail for the files host.
 
-## Retention (when a store exists)
+## Retention
 
-14 days for everything; 90 for `auth` and `deploy`. Both Loki and VictoriaLogs do per-stream retention.
+One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping
+`auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does.
 
 ## Wiring sketches (not deployed; pick one later)
 
 - **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
   labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
   Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
-- **VictoriaLogs**: single container, journald + docker via its agent or Vector; built-in UI behind the Caddy
+- **VictoriaLogs** (chosen): single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
-  forward-auth gate (`media-admins`). VictoriaMetrics beside it for metrics.
+  ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE`
+  (`media-admins`). VictoriaMetrics beside it would consume the four metric taps.
 - **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;
   behind the Caddy gate. Weakest at log search.
 - **ELK**: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a