diff --git a/CommunityDiscordMedia.md b/CommunityDiscordMedia.md index 73fd78f..77462f0 100644 --- a/CommunityDiscordMedia.md +++ b/CommunityDiscordMedia.md @@ -412,8 +412,17 @@ expose. The framework everything feeds from is `LOGGING.md`; this phase only add - [ ] switch on the metrics scrape at the same time if wanted: the four taps are already live (node_exporter 9100, cAdvisor 9101, MediaMTX 9998, Caddy 2019) and VictoriaMetrics beside VictoriaLogs would consume them. Not required for logs. -- [ ] alerts (the first thing that actually protects the stack): disk on `/srv` over 85 %, a sweep - `event=failed`, a container restart loop, MediaMTX dropped packets rising on a live path → Discord webhook. +- [ ] **alerts — VictoriaLogs does NOT alert.** It stores and queries, and exposes `/select/logsql/stats_query` + (+ `_range`) returning Prometheus-compatible results. Alerting is **vmalert** evaluating rules against + those endpoints, and **Alertmanager** receiving what fires (it has a native Discord receiver). That chain + covers *log* conditions only — a sweep `event=failed`, a burst of `publish_deny`, a failed bot deploy. + Disk, load and restart loops are *metrics*, so alerting on them additionally needs VictoriaMetrics + scraping the four taps that are already live. Full picture: four containers. +- [ ] **Do this first, instead, and keep it either way**: `bin/vanguard-watch.sh` on a timer — disk on `/srv` + over 85 %, any `*.normalize-failed.log`, a container missing from `docker compose ps`, dumpd `level=error` + in the last interval → one Discord webhook post. About forty lines, no new services, and it covers the + failure modes that actually take the platform down. The vmalert/Alertmanager stack is the grown-up + version for thresholds, grouping and silencing, not a prerequisite for being warned. **Later on the same host — ops controls.** Restart buttons and similar, so `diag.` becomes the one place an admin looks and acts. Design note so it is not built the easy wrong way: the page must not get Docker access. diff --git a/LOGGING.md b/LOGGING.md index 454ce82..3f7eaa9 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -22,8 +22,10 @@ Every collector on the shortlist reads both. Metrics are the same story: see the | `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` | | `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) | -None of these are reachable from outside (loopback binds; nothing in the firewall). Alert thresholds to start with: -disk on `/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path. +None of these are reachable from outside (loopback binds; nothing in the firewall). Nothing evaluates them yet: +metric alerting needs VictoriaMetrics scraping them plus vmalert, and log alerting needs vmalert against +VictoriaLogs' stats endpoints with Alertmanager for delivery (Phase 14). Thresholds to start with: disk on +`/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path. ## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)