Phase 14: say plainly that VictoriaLogs does not alert, and put a timer script ahead of the vmalert stack

Alerting is vmalert against its stats endpoints plus Alertmanager for delivery, and that covers log conditions
only: disk, load and restart loops are metrics and additionally need VictoriaMetrics over the four taps. Four
containers for the full picture, which is why a forty-line timer posting to the Discord webhook comes first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 15:35:39 -06:00
parent 3662eb3465
commit 09882833a0
2 changed files with 15 additions and 4 deletions

View File

@@ -412,8 +412,17 @@ expose. The framework everything feeds from is `LOGGING.md`; this phase only add
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
would consume them. Not required for logs.
- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep
`event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook.
- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
(+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
scraping the four taps that are already live. Full picture: four containers.
- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
version for thresholds, grouping and silencing, not a prerequisite for being warned.
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.