Phase 14: say plainly that VictoriaLogs does not alert, and put a timer script ahead of the vmalert stack
Alerting is vmalert against its stats endpoints plus Alertmanager for delivery, and that covers log conditions only: disk, load and restart loops are metrics and additionally need VictoriaMetrics over the four taps. Four containers for the full picture, which is why a forty-line timer posting to the Discord webhook comes first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -412,8 +412,17 @@ expose. The framework everything feeds from is `LOGGING.md`; this phase only add
|
|||||||
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
|
- [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live
|
||||||
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
|
(node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs
|
||||||
would consume them. Not required for logs.
|
would consume them. Not required for logs.
|
||||||
- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep
|
- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query`
|
||||||
`event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook.
|
(+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against
|
||||||
|
those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain
|
||||||
|
covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy.
|
||||||
|
Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics
|
||||||
|
scraping the four taps that are already live. Full picture: four containers.
|
||||||
|
- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv`
|
||||||
|
over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error`
|
||||||
|
in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the
|
||||||
|
failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up
|
||||||
|
version for thresholds, grouping and silencing, not a prerequisite for being warned.
|
||||||
|
|
||||||
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
|
**Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an
|
||||||
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
|
admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access.
|
||||||
|
|||||||
@@ -22,8 +22,10 @@ Every collector on the shortlist reads both. Metrics are the same story: see the
|
|||||||
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` |
|
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` |
|
||||||
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
|
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
|
||||||
|
|
||||||
None of these are reachable from outside (loopback binds; nothing in the firewall). Alert thresholds to start with:
|
None of these are reachable from outside (loopback binds; nothing in the firewall). Nothing evaluates them yet:
|
||||||
disk on `/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
|
metric alerting needs VictoriaMetrics scraping them plus vmalert, and log alerting needs vmalert against
|
||||||
|
VictoriaLogs' stats endpoints with Alertmanager for delivery (Phase 14). Thresholds to start with: disk on
|
||||||
|
`/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
|
||||||
|
|
||||||
## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
|
## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user