From 3662eb3465b651425c656fd85e30de31cd54ac41 Mon Sep 17 00:00:00 2001 From: Brandyn / Techy Date: Tue, 3 Nov 2026 15:33:47 -0600 Subject: [PATCH] Phase 14: diag. host with VictoriaLogs at /logs, ops controls later Records the store decision (VictoriaLogs over ELK and over Grafana+Loki, with the reasons), the hostname and path, the collector choice, and the one real decision left for build time: how Vector reads container logs without handing it root through the Docker socket. Also a design note for the later ops controls, so restart buttons end up as allowlisted dumpd endpoints rather than a web page with Docker access. Co-Authored-By: Claude Opus 5 (1M context) --- CommunityDiscordMedia.md | 38 ++++++++++++++++++++++++++++++++++++++ LOGGING.md | 17 ++++++++++------- 2 files changed, 48 insertions(+), 7 deletions(-) diff --git a/CommunityDiscordMedia.md b/CommunityDiscordMedia.md index b57b55a..73fd78f 100644 --- a/CommunityDiscordMedia.md +++ b/CommunityDiscordMedia.md @@ -384,6 +384,44 @@ Checklist if adopted: Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`. - [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron +### Phase 14 — `diag.` host: logs now, ops controls later + +**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`. +Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one +host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer). +VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already +expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector. + +- [ ] CNAME `diag` → the main host +- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`) +- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with + `-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not** + strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans. +- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep, + trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The + `vanguard.service` / `vanguard.category` labels already on every container become stream fields, so + filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them + (JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx, + multiline + regex for jellyfin). +- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API, + and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The + alternative is a read-only socket proxy in front of it, one more container but no root. Reading the + json files directly avoids both and loses the container labels, which is most of the value. +- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy` + longer than the rest would mean separate tenants — not worth it unless an audit needs it. +- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live + (node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs + would consume them. Not required for logs. +- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep + `event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook. + +**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an +admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access. +Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now, +purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a +logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command". + + ## Risks / gotchas - **Phase 5 lockout.** Keep one SSH session open and a copyparty local admin account until the diff --git a/LOGGING.md b/LOGGING.md index f2de9a1..454ce82 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -1,7 +1,8 @@ -# Logging framework — store-agnostic +# Logging framework -Decided 2026-11-01. The log *store* (Grafana + Loki, VictoriaLogs, Netdata, …) is not chosen yet. Everything -upstream of it is fixed now, so the choice later is a collector config, not a code change. +Decided 2026-11-01, and the store chosen on 2026-11-03: **VictoriaLogs**, on the `diag.BASE` host with its UI +at `/logs` (Phase 14 in `CommunityDiscordMedia.md`). Everything upstream of the store was fixed first, so +adopting it is a collector config rather than a code change. ## Where the lines come from (the only two taps a collector needs) @@ -76,17 +77,19 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie `X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the "who downloaded what" audit trail for the files host. -## Retention (when a store exists) +## Retention -14 days for everything; 90 for `auth` and `deploy`. Both Loki and VictoriaLogs do per-stream retention. +One global retention period in VictoriaLogs (`-retentionPeriod`); start at 30 days and watch the disk. Keeping +`auth` and `deploy` longer than the rest would need separate tenants, which is not worth it unless an audit does. ## Wiring sketches (not deployed; pick one later) - **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea). Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana. -- **VictoriaLogs**: single container, journald + docker via its agent or Vector; built-in UI behind the Caddy - forward-auth gate (`media-admins`). VictoriaMetrics beside it for metrics. +- **VictoriaLogs** (chosen): single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector + ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE` + (`media-admins`). VictoriaMetrics beside it would consume the four metric taps. - **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer; behind the Caddy gate. Weakest at log search. - **ELK**: rejected — JVM heap and index disk on a box doing x264; our data is structured records (→ MariaDB) and a