Phase 14: say plainly that VictoriaLogs does not alert, and put a timer script ahead of the vmalert stack

Alerting is vmalert against its stats endpoints plus Alertmanager for delivery, and that covers log conditions
only: disk, load and restart loops are metrics and additionally need VictoriaMetrics over the four taps. Four
containers for the full picture, which is why a forty-line timer posting to the Discord webhook comes first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-11-03 15:35:39 -06:00
parent 3662eb3465
commit 09882833a0
2 changed files with 15 additions and 4 deletions

View File

@@ -412,8 +412,17 @@ expose. The framework everything feeds from is `LOGGING.md`; this phase only add
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
would consume them. Not required for logs.
- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep
`event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook.
- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
(+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
scraping the four taps that are already live. Full picture: four containers.
- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
version for thresholds, grouping and silencing, not a prerequisite for being warned.
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.

View File

@@ -22,8 +22,10 @@ Every collector on the shortlist reads both. Metrics are the same story: see the
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` |
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
None of these are reachable from outside (loopback binds; nothing in the firewall). Alert thresholds to start with:
disk on `/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
None of these are reachable from outside (loopback binds; nothing in the firewall). Nothing evaluates them yet:
metric alerting needs VictoriaMetrics scraping them plus vmalert, and log alerting needs vmalert against
VictoriaLogs' stats endpoints with Alertmanager for delivery (Phase 14). Thresholds to start with: disk on
`/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)