techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Metrics taps: node_exporter + cAdvisor containers, MediaMTX and Caddy metrics on loopback (Prometheus format, store-agnostic)

Browse Source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Brandyn / Techy
2026-11-01 19:55:42 -06:00
parent 704bc8c4d3
commit 855651c5a4
5 changed files with 163 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
LOGGING.md
View File

@@ -10,9 +10,19 @@ upstream of it is fixed now, so the choice later is a collector config, not a co
| **journald** | our systemd units: `dumpd`, `normalize-sweep`, `trash-purge`, `deploy-pending`, `vgdeband` (transient), `deploy.sh` output | `journalctl -u <unit>`; collectors: Alloy `loki.source.journal`, Vector `journald`, VictoriaLogs journald ingestion, Netdata's journal plugin |
| **Docker json-file logs** | every container (`docker-compose.yml`), rotated 5 × 50 MB each | `docker logs <name>`; collectors read `/var/lib/docker/containers/*/*-json.log` and get the container **labels** below for free |
Every collector on the shortlist reads both. Metrics are the same story: node_exporter / cAdvisor expose the
Prometheus text format that Alloy, vmagent and Netdata all scrape; MediaMTX (`metrics: yes`) and Caddy have
endpoints too, currently off.
Every collector on the shortlist reads both. Metrics are the same story: see the taps table below — all four endpoints are on now, on loopback.
## Metrics taps (all Prometheus text format, loopback only — scrape them with Alloy, vmagent or Netdata)
| endpoint | what | from |
|---|---|---|
| `http://127.0.0.1:9100/metrics` | host: CPU, RAM, disk space + I/O, network, load, md RAID | node_exporter (host network, pid host) |
| `http://127.0.0.1:9101/metrics` | per container: CPU, memory, network, block I/O — labelled with `vanguard.service` / `vanguard.category` | cAdvisor |
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP sessions, dropped packets | MediaMTX `metrics: yes` |
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
None of these are reachable from outside (loopback binds; nothing in the firewall). Alert thresholds to start with:
disk on `/srv` > 85 %, load > cores for 15 min, a container restarting, MediaMTX dropped packets rising on a live path.
## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
@@ -74,8 +84,7 @@ On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie
- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
Metrics: Alloy scrapes node_exporter, cAdvisor, MediaMTX `:9998/metrics`, Caddy `:2019/metrics` → Prometheus or
VictoriaMetrics, same Grafana.
Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
- **VictoriaLogs**: single container, journald + docker via its agent or Vector; built-in UI behind the Caddy
forward-auth gate (`media-admins`). VictoriaMetrics beside it for metrics.
- **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;