Phase 14: say plainly that VictoriaLogs does not alert, and put a timer script ahead of the vmalert stack
Alerting is vmalert against its stats endpoints plus Alertmanager for delivery, and that covers log conditions only: disk, load and restart loops are metrics and additionally need VictoriaMetrics over the four taps. Four containers for the full picture, which is why a forty-line timer posting to the Discord webhook comes first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -412,8 +412,17 @@ expose. The framework everything feeds from is `LOGGING.md`; this phase only add
|
||||
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
|
||||
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
|
||||
would consume them. Not required for logs.
|
||||
- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep
|
||||
`event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook.
|
||||
- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
|
||||
(+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
|
||||
those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
|
||||
covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
|
||||
Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
|
||||
scraping the four taps that are already live. Full picture: four containers.
|
||||
- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
|
||||
over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
|
||||
in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
|
||||
failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
|
||||
version for thresholds, grouping and silencing, not a prerequisite for being warned.
|
||||
|
||||
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
|
||||
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
|
||||
|
||||
@@ -22,8 +22,10 @@ Every collector on the shortlist reads both. Metrics are the same story: see the
|
||||
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` |
|
||||
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
|
||||
|
||||
None of these are reachable from outside (loopback binds; nothing in the firewall). Alert thresholds to start with:
|
||||
disk on `/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
|
||||
None of these are reachable from outside (loopback binds; nothing in the firewall). Nothing evaluates them yet:
|
||||
metric alerting needs VictoriaMetrics scraping them plus vmalert, and log alerting needs vmalert against
|
||||
VictoriaLogs' stats endpoints with Alertmanager for delivery (Phase 14). Thresholds to start with: disk on
|
||||
`/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
|
||||
|
||||
## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user