bunch of shit

This commit is contained in:
2026-11-06 19:05:47 -06:00
parent ca01516d42
commit 630b253278
29 changed files with 2793 additions and 303 deletions

View File

@@ -20,7 +20,7 @@ journald input for our units. Metrics are the same story: see the taps table bel
| endpoint | what | from |
|---|---|---|
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets | MediaMTX `metrics: yes` |
| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets. `remoteAddr` labels are real client addresses now that MediaMTX runs on the host network; before, every publisher showed as the bridge gateway `172.18.0.1` | MediaMTX `metrics: yes` |
| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
| `http://127.0.0.1:8080/metrics` | ours: the normalise queue, encode time, live stream count | `bin/dumpd.py` |
@@ -36,6 +36,15 @@ Three layers on purpose, because each answers a different question and none of t
- **How bad is it, and is it getting worse** — the gauges: `vanguard_media_pending`,
`vanguard_media_pending_oldest_seconds`, `vanguard_media_failed_files`. A Lens dashboard over
`metricbeat-*` shows the trend across a night, which is the thing a list never can.
- **Stitching** — `event=stitched stream= parts= joins= missing_ms= seconds= out= trash=` when a stream's reconnect
fragments were joined; `event=stitch_wait` (stream still going), `event=stitch_skipped reason=too_many`
(500+ files: use vgstuck.sh), `event=stitch_break reason=gap|params_changed` (chain split there),
`event=stitch_failed` (fragments left alone, published one by one), `event=defer` (a fragment left alone
because a sibling is still being written).
- **Stop stream** — dumpd `stream_stop stream= kind= by= minutes= result=kicked|kick failed…`, then `publish_block user= minutes= by=`
/ `publish_unblock`, and every refused reconnect as `publish_deny reason=blocked by=`.
- **Wipe** — `component=vgstuck event=wipe_done recordings= stale_files= trash=` when an admin pressed Bin all
unprocessed on the Dump tab (dumpd logs `stuck_wipe by=`), with one `event=trashed file=` line per file before it.
- **Which recordings** — the sweep logs `event=parked file= stream= log=` once per run for every file a
failure log is holding back, plus `event=failed` with ffmpeg's own stderr when it first breaks. Both land
in `filebeat-*`, so a Discover search or a dashboard table names them and stays current. Names deliberately