Phase 14: diag. host with VictoriaLogs at /logs, ops controls later
Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and path, the collector choice, and the one real decision left for build time: how Vector reads container logs without handing it root through the Docker socket. Also a design note for the later ops controls, so restart buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -384,6 +384,44 @@ Checklist if adopted:
|
||||
Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
|
||||
- [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron
|
||||
|
||||
### Phase 14 — `diag.` host: logs now, ops controls later
|
||||
|
||||
**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`.
|
||||
Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one
|
||||
host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer).
|
||||
VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already
|
||||
expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector.
|
||||
|
||||
- [ ] CNAME `diag` → the main host
|
||||
- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`)
|
||||
- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with
|
||||
`-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not**
|
||||
strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans.
|
||||
- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep,
|
||||
trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The
|
||||
`vanguard.service` / `vanguard.category` labels already on every container become stream fields, so
|
||||
filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them
|
||||
(JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx,
|
||||
multiline + regex for jellyfin).
|
||||
- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API,
|
||||
and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The
|
||||
alternative is a read-only socket proxy in front of it, one more container but no root. Reading the
|
||||
json files directly avoids both and loses the container labels, which is most of the value.
|
||||
- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy`
|
||||
longer than the rest would mean separate tenants — not worth it unless an audit needs it.
|
||||
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
|
||||
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
|
||||
would consume them. Not required for logs.
|
||||
- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep
|
||||
`event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook.
|
||||
|
||||
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
|
||||
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
|
||||
Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now,
|
||||
purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a
|
||||
logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command".
|
||||
|
||||
|
||||
## Risks / gotchas
|
||||
|
||||
- **Phase 5 lockout.** Keep one SSH session open and a copyparty local admin account until the
|
||||
|
||||
Reference in New Issue
Block a user