Phase 13 build: logs, metrics, alerting and a deadman on a diag. host
Six containers, all loopback-published, reachable only through the gated diag. host: victorialogs 30d retention, -http.pathPrefix=/logs so Caddy passes the prefix through unchanged vector docker_logs + journald -> VictoriaLogs (elasticsearch bulk, the documented path) victoriametrics 90d, scrapes the four taps that were already live plus the new services vmalert-metrics 11 metric rules: disk, load, memory, RAID, restart loops, a core container gone vmalert-logs 5 LogsQL rules: sweep failures, publish-deny bursts, failed bot deploys, cert trouble alertmanager groups and inhibits, posts to Discord, webhook read from a 0600 file not the config The deadman (bin/vanguard-deadman.sh, host timer, 5 min) is the part that does not trust any of the above: curl only, it checks each component answers, that the always-firing Watchdog alert is still arriving through vmalert -> Alertmanager -> dumpd /heartbeat, and the disks. It reports on change and announces recovery once. Verified here: failures listed, silence when nothing changes, disk threshold, recovery fired once. Collection uses the Docker socket (Phase 13 option A, chosen explicitly). That is root-equivalent on this host and is recorded as such in vector.yaml with the two upgrade paths. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -320,7 +320,53 @@ permanent reference is needed for long sessions, not just a start burst.
|
||||
export at 2560×1440)
|
||||
- rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
|
||||
|
||||
### Phase 13 — push-to-deploy (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
|
||||
### Phase 13 — `diag.` host: logs, metrics, alerting, deadman (NEXT)
|
||||
|
||||
**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`.
|
||||
Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one
|
||||
host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer).
|
||||
VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already
|
||||
expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector.
|
||||
|
||||
- [ ] CNAME `diag` → the main host
|
||||
- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`)
|
||||
- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with
|
||||
`-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not**
|
||||
strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans.
|
||||
- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep,
|
||||
trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The
|
||||
`vanguard.service` / `vanguard.category` labels already on every container become stream fields, so
|
||||
filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them
|
||||
(JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx,
|
||||
multiline + regex for jellyfin).
|
||||
- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API,
|
||||
and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The
|
||||
alternative is a read-only socket proxy in front of it, one more container but no root. Reading the
|
||||
json files directly avoids both and loses the container labels, which is most of the value.
|
||||
- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy`
|
||||
longer than the rest would mean separate tenants — not worth it unless an audit needs it.
|
||||
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
|
||||
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
|
||||
would consume them. Not required for logs.
|
||||
- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
|
||||
(+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
|
||||
those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
|
||||
covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
|
||||
Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
|
||||
scraping the four taps that are already live. Full picture: four containers.
|
||||
- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
|
||||
over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
|
||||
in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
|
||||
failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
|
||||
version for thresholds, grouping and silencing, not a prerequisite for being warned.
|
||||
|
||||
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
|
||||
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
|
||||
Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now,
|
||||
purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a
|
||||
logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command".
|
||||
|
||||
### Phase 14 — push-to-deploy (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
|
||||
- server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved
|
||||
- `vanguard-deploy pull|apply|rollback|pending|status` applies only the changed parts; `check.sh` gates it;
|
||||
MediaMTX restarts are deferred while anything is live (`deploy-pending.timer`)
|
||||
@@ -384,52 +430,6 @@ Checklist if adopted:
|
||||
Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
|
||||
- [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron
|
||||
|
||||
### Phase 14 — `diag.` host: logs now, ops controls later
|
||||
|
||||
**Decided 2026-11-03: VictoriaLogs**, on its own hostname `diag.BASE`, with the log UI at `diag.BASE/logs`.
|
||||
Rejected: ELK (a JVM heap and a growing index on a box that spends its nights running x264, to search one
|
||||
host's logs), and Grafana + Loki (two services and a dashboard tool we would only use as a log viewer).
|
||||
VictoriaLogs is one container with its own UI and the same `/metrics` endpoint style as the taps we already
|
||||
expose. The framework everything feeds from is `LOGGING.md`; this phase only adds the store and the collector.
|
||||
|
||||
- [ ] CNAME `diag` → the main host
|
||||
- [ ] `configure.py`: new host role `diag`, Caddy block gated by `DIAG_REQUIRE_GROUP` (default `media-admins`)
|
||||
- [ ] `victoria-logs` container, `127.0.0.1:9428`, data in `/srv/appdata/victorialogs`, run with
|
||||
`-http.pathPrefix=/logs` so its own UI emits `/logs/...` URLs; Caddy uses `handle /logs*` and does **not**
|
||||
strip the prefix (stripping breaks vmui's asset paths). `redir /logs /logs/select/vmui/ 302` for humans.
|
||||
- [ ] collector: **Vector**, one container, two sources — `journald` for our units (dumpd, normalize-sweep,
|
||||
trash-purge, deploy-pending, vgdeband) and Docker logs for the 15 containers. The
|
||||
`vanguard.service` / `vanguard.category` labels already on every container become stream fields, so
|
||||
filtering by category works without parsing. Parse stages per source as `LOGGING.md` lists them
|
||||
(JSON for caddy/authentik, logfmt for syncthing and our own scripts, one regex for mediamtx,
|
||||
multiline + regex for jellyfin).
|
||||
- [ ] **Decide at build time how Vector reads container logs.** Its `docker_logs` source needs the Docker API,
|
||||
and mounting the socket is root-equivalent on this host — the same exposure act_runner already has. The
|
||||
alternative is a read-only socket proxy in front of it, one more container but no root. Reading the
|
||||
json files directly avoids both and loses the container labels, which is most of the value.
|
||||
- [ ] retention: one global `-retentionPeriod` (start at 30d and watch the disk). Keeping `auth` and `deploy`
|
||||
longer than the rest would mean separate tenants — not worth it unless an audit needs it.
|
||||
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
|
||||
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
|
||||
would consume them. Not required for logs.
|
||||
- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
|
||||
(+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
|
||||
those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
|
||||
covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
|
||||
Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
|
||||
scraping the four taps that are already live. Full picture: four containers.
|
||||
- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
|
||||
over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
|
||||
in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
|
||||
failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
|
||||
version for thresholds, grouping and silencing, not a prerequisite for being warned.
|
||||
|
||||
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
|
||||
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
|
||||
Put each action behind a dumpd endpoint with a fixed allowlist (restart *this* named service, run the sweep now,
|
||||
purge trash now), authorise from the Authentik headers as everything else does, and log every invocation as a
|
||||
logfmt event so the action lands in the same place as the logs it is diagnosing. No generic "run a command".
|
||||
|
||||
|
||||
## Risks / gotchas
|
||||
|
||||
|
||||
Reference in New Issue
Block a user