Phase 14: diag. host with VictoriaLogs at /logs, ops controls later

Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and
path, the collector choice, and the one real decision left for build time: how Vector reads container logs
without handing it root through the Docker socket. Also a design note for the later ops controls, so restart
buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 15:33:47 -06:00
parent 12760a94c3
commit 3662eb3465
2 changed files with 48 additions and 7 deletions

View File

@@ -1,7 +1,8 @@
# Logging framework — store-agnostic
# Logging framework
Decided 2026-11-01. The log *store* (Grafana + Loki, VictoriaLogs, Netdata, …) is not chosen yet. Everything
upstream of it is fixed now, so the choice later is a collector config, not a code change.
Decided 2026-11-01, and the store chosen on 2026-11-03: **VictoriaLogs**, on the `diag.BASE` host with its UI
at `/logs` (Phase 14 in `CommunityDiscordMedia.md`). Everything upstream of the store was fixed first, so
adopting it is a collector config rather than a code change.
## Where the lines come from (the only two taps a collector needs)
@@ -76,17 +77,19 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie
`X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the
"who downloaded what" audit trail for the files host.
## Retention (when a store exists)
## Retention
14 days for everything; 90 for `auth` and `deploy`. Both Loki and VictoriaLogs do per-stream retention.
One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping
`auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does.
## Wiring sketches (not deployed; pick one later)
- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
- **VictoriaLogs**: single container, journald + docker via its agent or Vector; built-in UI behind the Caddy
forward-auth gate (`media-admins`). VictoriaMetrics beside it for metrics.
- **VictoriaLogs** (chosen): single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE`
(`media-admins`). VictoriaMetrics beside it would consume the four metric taps.
- **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;
behind the Caddy gate. Weakest at log search.
- **ELK**: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a