diff --git a/LOGGING.md b/LOGGING.md index 23bc5e8..c06ccbe 100644 --- a/LOGGING.md +++ b/LOGGING.md @@ -29,6 +29,26 @@ and `docker` modules, not from an exporter — node_exporter and cAdvisor were r went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher). +### Stuck recordings: counts in Kibana, names in the logs, decisions on the host + +Three layers on purpose, because each answers a different question and none of them needs a new web page. + +- **How bad is it, and is it getting worse** — the gauges: `vanguard_media_pending`, + `vanguard_media_pending_oldest_seconds`, `vanguard_media_failed_files`. A Lens dashboard over + `metricbeat-*` shows the trend across a night, which is the thing a list never can. +- **Which recordings** — the sweep logs `event=parked file= stream= log=` once per run for every file a + failure log is holding back, plus `event=failed` with ffmpeg's own stderr when it first breaks. Both land + in `filebeat-*`, so a Discover search or a dashboard table names them and stays current. Names deliberately + do NOT come from the metrics: a label per filename would wreck index cardinality for no gain. +- **What to do about it** — `bin/vgstuck.sh` on the host. It classifies every unpublished recording as + `parked` (a failure log is blocking it, with the reason), `published` (an output already exists), + `waiting` (nothing blocking it yet) or `writing` (still recording), then `retry ` clears the + blocker so the sweep tries again and `drop ` moves it to `trash//` for the same three-day + undo as a dumped night. `drop --all` is refused. + +A leftover `.mp4.part` does **not** block anything — ffmpeg overwrites it with `-y`. It only wastes +disk, and `retry` clears it. + ### Our own metrics, and how to add one MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising @@ -135,8 +155,15 @@ to Discord, and the pipeline is split: Kibana rule --.index connector--> vanguard-alerts --vanguard-alerts.timer--> Discord webhook -- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index") and the heartbeat - rule. Run it once, *after* Elasticsearch security is on. +- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index"), the heartbeat rule, + the data view, and the "Vanguard ops" landing dashboard plus the `defaultRoute` that makes `diag.BASE` + open on it. Run it once, *after* Elasticsearch security is on. Name a step to run only that one: + `kibana-setup.py landing`. + Kibana deliberately stays at the **root** of that host. Moving it under `/kibana` would mean + `server.basePath` plus `rewriteBasePath`, a Caddy handle that does not strip the prefix, and appending the + prefix in both Beats' `setup.kibana`, this script, the alarm script and the deadman's status check — and a + miss in the Beats one crash-loops Metricbeat, because it exits when Kibana is unreachable. `defaultRoute` + buys the landing page without any of that. - **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind` as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused them, and replayable after the fact. diff --git a/README.md b/README.md index 1d2f6d0..d77c908 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,8 @@ works from ~/vanguard-stream without sudo. ./configure.py --full # first time: asks everything, then applies ./configure.py # later: add/remove/reset streamers and editors, then apply ./configure.py --show # print current users and the OBS settings to hand out - ./configure.py --set K=V # change one setting (Discord tokens/webhooks, retention) without a full pass + ./configure.py --set K=V # change one setting (Discord tokens/webhooks, retention, alarm mentions) + # run it with no arguments to list what is settable Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates `Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates @@ -437,8 +438,27 @@ Jellyfin and Gitea use OIDC providers instead, not this. ## 5h. Diagnostics: Kibana at `diag.HOST` -Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover, -dashboards, and alerting, with no sub-paths and no index page of links. +Four containers, one store, one UI. Kibana is served at the **root** of `diag.HOST`, with no sub-path to get +wrong. Arriving there opens a dashboard of ours, "Vanguard ops", via Kibana's `defaultRoute` setting: links to +every host and to the triage page, plus shortcuts into Discover, Dashboards, Stack Rules and Index Management. +It is an ordinary dashboard, so add metric panels beside the links (`vanguard_media_pending` first). + +Two paths on that host are ours rather than Kibana's: + +| path | what | +|---|---| +| `/stuck` | **stuck recordings**: everything that finished but never published, why each one is stuck, and Retry / Bin buttons. The list is readable by anyone the gate allows; the actions are enforced in dumpd for the admin groups, not hidden in the page. Binning moves the file to `trash//`, the same 3-day undo as a dumped night | +| `/stuck/api` | dumpd serves it. Both the page and `vgstuck.sh` on the host run the same script, so they cannot disagree about why something is stuck | + +Kibana deliberately does **not** live under `/kibana`. That would need `server.basePath` plus +`rewriteBasePath`, a Caddy handle that does not strip the prefix, and the prefix appended in both Beats' +`setup.kibana`, `kibana-setup.py`, `kibana-alarms.py` and the deadman's status check — and missing the Beats +one crash-loops Metricbeat, which exits when Kibana is unreachable. `defaultRoute` buys the landing page +without any of it. + +`observability/kibana-setup.py` creates the first-run objects and takes a step name, so you can redo one: +`connector`, `rule`, `dataview`, `landing`, or all of them when omitted. `observability/kibana-alarms.py` +creates the six alarm rules. | container | job | |---|---| @@ -573,6 +593,13 @@ Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later. ## Deployment (push to deploy — built, OFF by default) +What lands in `/usr/local/bin` is listed once, in `bin/install.list`, and read by `install.sh`, +`configure.py --apply` and `deploy.sh`. It used to be three hand-kept lists and they drifted: a push was +installing five of ten scripts, so `vgstuck.sh`, both sourced helpers and both timer scripts only reached the +host if somebody ran `--apply` by hand. `check.sh` now fails if the list names a missing file, or if +something in `bin/` is neither listed nor explicitly excluded (`mtx-on-ready.sh` is mounted into the MediaMTX +container; `vgbot-merge-sqlite.py` is a one-off migration). + The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea → `./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout. @@ -715,4 +742,6 @@ evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern. |------|------|------| | normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard | | trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days | -| dumpd.service | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time and on `clips.` at /me/clips | +| vanguard-alerts.timer | every 1 min | delivers Kibana's alert documents to Discord, with a 15-min per-rule repeat guard | +| vanguard-deadman.timer | every 5 min | watches the alerting stack from outside it, and the disks | +| dumpd.service | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time, on `clips.` at /me/clips and on `diag.` at /stuck/api | diff --git a/STATUS.md b/STATUS.md index aa7f8cf..764ff0c 100644 --- a/STATUS.md +++ b/STATUS.md @@ -60,9 +60,11 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live | | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) | | Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) | +| diag landing | `diag.…` opens on a "Vanguard ops" dashboard: a markdown panel of links to every host plus the stuck page, and shortcuts to Discover, Dashboards, Stack Rules and Index Management. Kibana still owns the root; `defaultRoute` just redirects there. Created by `kibana-setup.py landing`, so a rebuild recreates it | **added 2026-11-04** | | Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** | | Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything | | Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy | +| Stuck recordings | A page at `https://diag.…/stuck` for anyone past the gate, on the triage host (already `media-admins` only), linked from the gateway's Media admin card and from the diag landing dashboard: every unpublished recording, why it is stuck, and Retry / Bin buttons. dumpd serves `/stuck/api` and allows the actions only for the admin groups; both the page and the CLI run `bin/vgstuck.sh`, so they can never disagree. `bin/vgstuck.sh` triages anything that never published: it says why each one is stuck (`parked` by a failure log with ffmpeg's reason, `published` collision, `waiting`, or still `writing`), then `retry ` clears the blocker and `drop ` moves it to trash with the usual 3-day undo. `drop --all` is refused. The sweep also logs `event=parked` once per run so Kibana can name them, not just count them | **added 2026-11-04** | | Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** | | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending | | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live | @@ -117,6 +119,9 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested). - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply. - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only. +- **A high `x-google-min-bitrate` disables WebRTC congestion control.** Go Live sent `min-bitrate=6000`, so Chrome never encoded below 6 Mbps; on an uplink that could not carry it, the excess was dropped, MediaMTX logged `invalid FU-A packet (non-starting)` and every viewer saw blocky corruption. The loss was on the PUBLISH session (4761 packets for one streamer) while the viewer sessions lost nothing, which is the fingerprint: the server relays damage it received. Floor lowered to 1200 and `scaleResolutionDownBy` unpinned so the encoder can shed resolution. Found 2026-11-04. +- `tracks=['Opus','H264']` on a path means a browser (Go Live); `MPEG-4 Audio` means OBS. Quick way to tell who is publishing how. +- **`handle_path` strips the prefix, so an upstream's own redirects lose it.** MediaMTX redirects every HLS playlist request once (to `?cookieCheck=1`) with a root-relative `Location`, so the browser followed `///index.m3u8` into the portal's SPA catch-all, got `index.html`, and hls.js said `manifestParsingError` — a web page where a manifest should be. Fixed with `header_down Location ^/ /hls/`. `curl` against MediaMTX directly hides this completely, because there is no prefix to lose. Found 2026-11-04. - **Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked** in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a `tmcd` data stream is the fingerprint. First working Windows audio: 2026-11-03. - The sweep's band-removal branch **overwrote `$reason`**, which carried the probe's `audio=<codec>` marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got `-c:a copy` and shipped Opus inside an MP4. Fixed by keeping `probe_reason` separately; it only became reachable once Go Live audio worked. - **Authentik has two invalidation flows and the names are a trap.** `default-provider-invalidation-flow` is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so `/logout` drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. `default-invalidation-flow` ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03. diff --git a/about/index.html b/about/index.html index cdab28c..2c76876 100644 --- a/about/index.html +++ b/about/index.html @@ -120,6 +120,7 @@ <h2>Media admin</h2> <p class="lead" data-group="media-admins" data-yes="You are in <code>media-admins</code>: full rights on every folder, trash, Syncthing, dump anyone's night." data-no="Admin view — you are <b>not</b> in <code>media-admins</code>: copyparty will show you nothing and <code>/sync/</code> will 403 until you get the role."></p> <div class="links"> + <a class="card" id="l-stuck" href="#"><b>Stuck recordings</b><span>never published — see why, retry or bin</span></a> <a class="card" id="l-trash" href="#"><b>Trash</b><span>dumped nights, purged after 3 days</span></a> <a class="card" id="l-sync" href="#"><b>Syncthing</b><span>device pairing, folder states</span></a> <a class="card" href="/whoami"><b>Who am I</b><span>identity headers as the apps see them</span></a> @@ -129,6 +130,7 @@ <li>Group changes in Authentik apply on the next login: <a href="/logout">sign out</a> and back in.</li> <li>Dump anyone's night: the dock's Dump works on your own; for others use the API with <code>{"user":"name"}</code> or ask the owner.</li> <li>New gamer: they self-serve on the <a class="l-clips" href="#">clips page</a>; you only need to watch Syncthing accept the device.</li> + <li><a class="l-stuck-note" href="#">Stuck recordings</a> (on the diagnostics host, with the dashboards) lists anything that finished but never published, with the reason. Retry puts it back in the queue; binning keeps the same 3-day undo as a dumped night.</li> </ul> </details> </section> @@ -179,6 +181,8 @@ $('l-git').href = sub('git'); $('l-bot').href = sub('bot'); $('l-auth').href = s $('l-jfadmin').href = sub('watch') + '/web/#/dashboard'; $('l-gitadmin').href = sub('git') + '/-/admin'; $('l-dock').href = sub('live') + '/dock/'; $('dockurl').textContent = sub('live') + '/dock/'; $('l-files').href = sub('files') + '/'; $('l-finals').href = sub('files') + '/finals/'; $('l-trash').href = sub('files') + '/trash/'; $('l-sync').href = sub('sync') + '/'; +$('l-stuck').href = sub('diag') + '/stuck'; +document.querySelectorAll('a.l-stuck-note').forEach(a => a.href = sub('diag') + '/stuck'); $('l-clips').href = sub('clips') + '/'; document.querySelectorAll('a.l-clips').forEach(a => a.href = sub('clips') + '/'); $('gitssh').textContent = `ssh://git@${'git.' + base}:2222/org/repo.git`; (async () => { try { diff --git a/bin/dumpd.py b/bin/dumpd.py index 179bc1a..be1549b 100755 --- a/bin/dumpd.py +++ b/bin/dumpd.py @@ -21,6 +21,10 @@ POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a D HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot. May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body. +GET /stuck/api -> every unpublished recording and why it is stuck (any gated user) +POST /stuck/api -> {"action":"retry"|"drop","src":...}: clear the blocker, or move it to trash with the + usual 3-day undo. Admin groups only; both actions run bin/vgstuck.sh so the web page and the + CLI share one implementation. GET /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here. POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid @@ -425,6 +429,43 @@ def verify_token(token): return None return user.lower() +# ---- stuck recordings ----------------------------------------------------------------------------- +# GET /stuck/api every unpublished recording and why, for anyone the gate let through +# POST /stuck/api {"action":"retry"|"drop","src":"<path>"} — admins only +# The classification and the file moves both live in bin/vgstuck.sh, so the web page and the CLI can never +# disagree about why something is stuck, and "delete" means the same three-day trash in both. +STUCK_TOOL = os.environ.get("VGSTUCK_TOOL", "/usr/local/bin/vgstuck.sh") + +def stuck_list(): + done = subprocess.run([STUCK_TOOL, "json"], capture_output=True, text=True, timeout=180) + try: + return json.loads(done.stdout or "[]") + except ValueError: + log("stuck_list_failed", level="error", rc=done.returncode, err=(done.stderr or "")[:200]) + return [] + +def stuck_target(src): + """Resolve what the browser asked us to act on, or raise. This is the only thing between a request body + and a file move, so it is deliberately strict: inside the streams tree, an unpublished .live recording, + and really there.""" + path = Path(src).resolve() + root = (ROOT / "streams").resolve() + if root != path and root not in path.parents: + raise ValueError("that path is not inside the streams tree") + if f"{os.sep}.live{os.sep}" not in str(path) or path.suffix.lower() != ".mp4": + raise ValueError("that is not an unpublished recording") + if not path.is_file(): + raise ValueError("no such recording (already published or already binned?)") + return path + +def stuck_action(action, src, who): + path = stuck_target(src) + done = subprocess.run([STUCK_TOOL, action, str(path)], capture_output=True, text=True, timeout=600) + lines = [l for l in (done.stdout + done.stderr).splitlines() if l.strip()] + log(f"stuck_{action}", level="info" if done.returncode == 0 else "error", + file=str(path), by=who, rc=done.returncode) + return {"ok": done.returncode == 0, "rc": done.returncode, "output": lines[-8:]} + # ---- GET /metrics ----------------------------------------------------------------------------------- # Prometheus text for Metricbeat's prometheus module. Two sources are merged: # 1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/<component>.counters) @@ -619,6 +660,10 @@ class H(BaseHTTPRequestHandler): return self._json(200, {"user": who, "groups": groups, "admin": is_admin, "name": self.headers.get("X-Authentik-Name") or who, "email": self.headers.get("X-Authentik-Email") or "", "discord_id": proxy_claim(self.headers, "discord_id")}) + if self.path == "/stuck/api": + who, is_admin = identity(self.headers) + if not who: return self._json(401, {"error": "not authenticated"}) + return self._json(200, {"admin": is_admin, "user": who, "items": stuck_list()}) if self.path.startswith("/me/clips"): who, _ = identity(self.headers) if not who: return self._json(401, {"error": "not authenticated"}) @@ -747,6 +792,24 @@ class H(BaseHTTPRequestHandler): return self._json(401, {"error": "path must be <your-username>/<Title>"}) log("publish_allow", user=user, stream=path, ip=req.get("ip")) return self._json(200, {"ok": True}) + if self.path == "/stuck/api": + who, is_admin = identity(self.headers) + if not who: return self._deny() + if not is_admin: + return self._json(403, {"error": f"retrying and binning recordings is for {', '.join(sorted(ADMIN_GROUPS))}"}) + try: + body = json.loads(self.rfile.read(int(self.headers.get("Content-Length") or 0)) or b"{}") + except ValueError as error: + return self._json(400, {"error": str(error)}) + action = body.get("action") + if action not in ("retry", "drop"): + return self._json(400, {"error": "action must be retry or drop"}) + try: + return self._json(200, stuck_action(action, str(body.get("src", "")), who)) + except ValueError as error: + return self._json(400, {"error": str(error)}) + except Exception as error: + return self._json(500, {"error": str(error)}) if self.path.startswith("/me/clips"): who, _ = identity(self.headers) if not who: return self._json(401, {"error": "not authenticated"}) diff --git a/bin/install.list b/bin/install.list new file mode 100644 index 0000000..377f56b --- /dev/null +++ b/bin/install.list @@ -0,0 +1,19 @@ +# What gets installed into /usr/local/bin, and with which mode. ONE list, read by install.sh, +# configure.py --apply and deploy.sh, because three hand-kept copies drifted: a push-to-deploy was +# silently not installing vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script. +# check.sh fails if a file here is missing, or if something executable in bin/ is neither listed nor +# excluded below. +0755 bin/normalize-sweep.sh +0755 bin/trash-purge.sh +0755 bin/dumpd.py +0755 bin/vgstrip.py +0755 bin/vgdeband.sh +0755 bin/vgstuck.sh +0755 bin/vanguard-deadman.sh +0755 bin/vanguard-alerts.sh +# sourced, never executed +0644 bin/vgmetrics.sh +0644 bin/vgnotify.sh +# NOT installed on the host, on purpose: +# exclude bin/mtx-on-ready.sh mounted into the mediamtx container (docker-compose.yml) +# exclude bin/vgbot-merge-sqlite.py one-off migration, run from the checkout diff --git a/bin/normalize-sweep.sh b/bin/normalize-sweep.sh index a3709d8..2084e48 100755 --- a/bin/normalize-sweep.sh +++ b/bin/normalize-sweep.sh @@ -115,7 +115,12 @@ while IFS= read -r -d '' src; do if [[ -d "$d/.stfolder" ]]; then synced=1; stroot="$d"; break; fi d="$(dirname "$d")" done - [[ -e "$base.normalize-failed.log" ]] && continue + if [[ -e "$base.normalize-failed.log" ]]; then + # Say so every run. Skipping silently meant a parked file produced no log line after its original + # failure, so Kibana could only ever show a count and never which recording. bin/vgstuck.sh triages it. + vglog info "event=parked file=\"$src\" stream=\"$stream\" log=\"$base.normalize-failed.log\"" + continue + fi fuser -s "$src" 2>/dev/null && continue # still being written [[ -e "$src" ]] || continue # dumped meanwhile diff --git a/bin/vanguard-alerts.sh b/bin/vanguard-alerts.sh index 3f665a7..e45d631 100755 --- a/bin/vanguard-alerts.sh +++ b/bin/vanguard-alerts.sh @@ -65,6 +65,10 @@ webhook=$(vg_webhook 2>/dev/null || true) now=$(date +%s) cursor=$(state_get cursor); cursor="${cursor:-0}" +# Record every run, not just runs that did something. The deadman watches this file's age to tell whether +# delivery is still happening at all, and almost every run is quiet: without this the file went stale within +# minutes and the deadman reported the timer as dead, then recovered the moment an alert happened to flow. +state_set last_run "$now" # Everything newer than the cursor that is not a heartbeat, oldest first so the cursor advances safely. query=$(printf '{"size":%d,"sort":[{"@timestamp":"asc"}],"query":{"bool":{"filter":[{"range":{"@timestamp":{"gt":%s,"format":"epoch_millis"}}}],"must_not":[{"term":{"kind":"heartbeat"}}]}}}' "$BATCH" "$cursor") diff --git a/bin/vanguard-deadman.sh b/bin/vanguard-deadman.sh index b9951be..d209ce9 100755 --- a/bin/vanguard-deadman.sh +++ b/bin/vanguard-deadman.sh @@ -127,6 +127,11 @@ if [[ -f $ALERT_STATE ]]; then age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") )) (( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered" fi +# Freshness above proves the timer fires; this proves the run finished. A unit that starts every minute and +# dies immediately would otherwise look alive, because the file is written at the start of the run. +if systemctl is-failed --quiet vanguard-alerts.service 2>/dev/null; then + note "vanguard-alerts.service is in a failed state — $(systemctl show -p Result --value vanguard-alerts.service 2>/dev/null); alerts are written but not delivered" +fi # the disks, independently of any agent while read -r pct mount; do diff --git a/bin/vgstuck.sh b/bin/vgstuck.sh new file mode 100755 index 0000000..53ceaeb --- /dev/null +++ b/bin/vgstuck.sh @@ -0,0 +1,143 @@ +#!/bin/bash +# Triage recordings that never published: see why each one is stuck, then retry it or bin it. +# +# vgstuck.sh list everything stuck, with the reason (read-only, the default) +# vgstuck.sh retry <match> clear what is blocking it so the sweep tries again +# vgstuck.sh retry --all ...for everything that is parked +# vgstuck.sh drop <match> move the recording to trash: 3-day undo, same as "dump tonight" +# vgstuck.sh sweep kick the sweep once, without waiting on it +# vgstuck.sh json the same survey as JSON (what dumpd serves to the web page) +# +# <match> is any substring of the path (a streamer name, a title, a timestamp). It must match at least one +# recording, and `drop` refuses --all: binning every stuck recording in one keystroke is never a thing +# somebody meant to do. +# +# WHY A RECORDING STAYS IN .live/ +# parked a <Title>__<ts>.normalize-failed.log sits next to the intended output. The sweep skips the +# file entirely until that log is deleted, so it will never retry on its own. +# published an output <Title>__<ts>.mp4 already exists, so the sweep leaves the source alone. +# waiting nothing is blocking it; it is simply newer than MINS_IDLE or the sweep has not run yet. +# partial a leftover <output>.mp4.part. This does NOT block anything (ffmpeg overwrites it with -y); +# it only wastes disk, and retry removes it. +# +# Nothing here deletes media. `drop` MOVES to /srv/media/trash/<today>/, which trash-purge clears after +# three days, so a wrong call is recoverable for the same window as a dumped night. +set -uo pipefail + +ROOT="${MEDIA_ROOT:-/srv/media}" +TRASH="$ROOT/trash" +MINS_IDLE="${MINS_IDLE:-3}" + +vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgstuck %s\n' "$(date -Is)" "$lvl" "$*"; } + +# Everything about one .live source, as: src|out|state|age_s|bytes|dur_s|reason +survey() { + local src titledir user title stamp base out state reason dur + while IFS= read -r -d '' src; do + titledir="$(dirname "$(dirname "$src")")" + user="$(basename "$(dirname "$titledir")")" + title="$(basename "$titledir")" + stamp="$(basename "${src%.*}")" + base="$(dirname "$titledir")/${title}__${stamp}" + out="$base.mp4" + reason="" + if [[ -e "$base.normalize-failed.log" ]]; then + state=parked + reason="$(head -1 "$base.normalize-failed.log" 2>/dev/null)" + local why; why=$(grep -A2 '^--- ffmpeg stderr ---$' "$base.normalize-failed.log" 2>/dev/null | sed -n '2p') + [[ -n ${why:-} ]] && reason="$reason :: $why" + elif [[ -e "$out" ]]; then + state=published + reason="output already exists: $(basename "$out")" + elif [[ -n $(find "$src" -mmin "-$MINS_IDLE" -print -quit 2>/dev/null) ]]; then + state=writing + reason="touched in the last ${MINS_IDLE}m — still recording" + else + state=waiting + reason="nothing blocking it; the sweep has not got to it" + fi + [[ -e "$out.part" ]] && reason="$reason [leftover .part]" + dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$src" 2>/dev/null | cut -d. -f1) + printf '%s|%s|%s|%s|%s|%s|%s\n' "$src" "$out" "$state" \ + "$(( $(date +%s) - $(stat -c %Y "$src") ))" "$(stat -c %s "$src")" "${dur:-0}" "$reason" + done < <(find "$ROOT/streams" -path '*/.live/*' -name '*.mp4' -print0 2>/dev/null) +} + +show() { + local any=0 src out state age bytes dur reason + printf '\n %-9s %-9s %9s %8s %s\n' state waiting length size recording + while IFS='|' read -r src out state age bytes dur reason; do + any=1 + printf ' %-9s %-9s %9s %7.0fM %s\n' "$state" \ + "$(( age / 3600 ))h$(printf '%02d' $(( (age % 3600) / 60 )))m" \ + "$(printf '%d:%02d:%02d' $(( dur / 3600 )) $(( (dur % 3600) / 60 )) $(( dur % 60 )))" \ + "$(awk -v b="$bytes" 'BEGIN{print b/1000000}')" "${src#"$ROOT"/streams/}" + printf ' %s\n' "$reason" + done < <(survey | sort -t'|' -k3,3 -k4,4nr) + (( any )) || printf ' nothing is stuck.\n' + printf '\n retry one: vgstuck.sh retry <substring> bin one: vgstuck.sh drop <substring>\n\n' +} + +# rows matching the pattern, or everything when --all +selected() { + local pattern="$1" + if [[ $pattern == --all ]]; then survey; else survey | grep -F -- "$pattern" || true; fi +} + +do_retry() { + local pattern="$1" n=0 src out state age bytes dur reason + while IFS='|' read -r src out state age bytes dur reason; do + [[ $state == writing ]] && { vglog info "event=skip reason=still_recording file=\"$src\""; continue; } + rm -f "$out.part" # leftover partial output: harmless, but no reason to keep + rm -f "${out%.mp4}.normalize-failed.log" # THIS is what makes the sweep skip the file forever + n=$((n + 1)) + vglog info "event=retry file=\"$src\" was=$state" + done < <(selected "$pattern") + (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; } + vglog info "event=retry_done count=$n msg=\"run vgstuck.sh sweep, or wait for the timer\"" +} + +do_drop() { + local pattern="$1" night n=0 src out state age bytes dur reason rel dst + [[ $pattern == --all ]] && { vglog error 'event=refused msg="drop --all is not allowed; name them"' >&2; return 2; } + night="$(date +%F)" + while IFS='|' read -r src out state age bytes dur reason; do + [[ $state == writing ]] && { vglog info "event=skip reason=still_recording file=\"$src\""; continue; } + for victim in "$src" "${src%.*}.json" "${out%.mp4}.normalize-failed.log" "$out.part"; do + [[ -e $victim ]] || continue + rel="${victim#"$ROOT"/}" + dst="$TRASH/$night/$rel" + mkdir -p "$(dirname "$dst")" + mv -f "$victim" "$dst" && vglog info "event=trashed file=\"$rel\" undo_days=3" + done + rmdir "$(dirname "$src")" "$(dirname "$(dirname "$src")")" 2>/dev/null || true + n=$((n + 1)) + done < <(selected "$pattern") + (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; } + vglog info "event=drop_done count=$n trash=\"$TRASH/$night\"" +} + +# Same survey, as JSON. bin/dumpd.py serves this to the web page, so the classification lives in exactly +# one place and the browser and the CLI can never disagree about why something is stuck. +emit_json() { + survey | python3 -c ' +import json, sys +rows = [] +for line in sys.stdin: + parts = line.rstrip("\n").split("|", 6) + if len(parts) != 7: + continue + src, out, state, age, size, dur, reason = parts + rows.append({"src": src, "out": out, "state": state, "age_s": int(age or 0), + "bytes": int(size or 0), "duration_s": int(dur or 0), "reason": reason}) +print(json.dumps(rows))' +} + +case "${1:-list}" in + list|"") show ;; + json) emit_json ;; + retry) [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh retry <substring>|--all" >&2; exit 1; }; do_retry "$2" ;; + drop) [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh drop <substring>" >&2; exit 1; }; do_drop "$2" ;; + sweep) systemctl start --no-block normalize-sweep.service && vglog info "event=sweep_started" ;; + *) sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 1 ;; +esac diff --git a/check.sh b/check.sh index 4f67d04..b68c834 100755 --- a/check.sh +++ b/check.sh @@ -13,12 +13,43 @@ for f in configure.py bin/dumpd.py bin/vgstrip.py observability/kibana-setup.py python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; } done rm -rf __pycache__ bin/__pycache__ -python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; } -node -e " +# bin/install.list is the single source for what lands in /usr/local/bin. Fail if it names something that +# does not exist, or if a file in bin/ is neither listed nor explicitly excluded — that drift is exactly how +# deploy.sh ended up installing five of ten scripts. +if python3 - <<'PYMAN' >/dev/null 2>&1 +import re, sys +from pathlib import Path +text = Path("bin/install.list").read_text() +listed = dict((m.group(2), m.group(1)) for m in re.finditer(r"^(0\d{3}) (\S+)$", text, re.M)) +excluded = set(re.findall(r"^#\s+exclude (\S+)", text, re.M)) +present = set("bin/" + p.name for p in Path("bin").iterdir() if p.is_file() and p.name != "install.list") +missing = set(listed) - present +unclaimed = present - set(listed) - excluded +assert not missing, f"install.list names files that do not exist: {sorted(missing)}" +assert not unclaimed, f"bin/ files neither listed nor excluded in install.list: {sorted(unclaimed)}" +PYMAN +then say "bin/install.list" ok; else say "bin/install.list" "DRIFTED — see the assertion"; fail=1; fi +# Only files that are actually here. The server checkout is partial (scp, not a full clone), and a tool that +# reports "broken" when it means "absent" gets ignored — the same reason the deadman reports on change only. +yamls=(); for f in docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml; do [[ -f $f ]] && yamls+=("$f"); done +python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" "${yamls[@]}" \ + && say "yaml (${#yamls[@]} files)" ok || { say "yaml" "INVALID"; fail=1; } +if ! command -v node >/dev/null; then + say "page JS" "(skipped: node not installed here)" +else + node -e " const fs=require('fs'); -for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); } -new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" && say "dock + clock JS" ok || { say "dock + clock JS" "SYNTAX ERROR"; fail=1; } -diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null && say "vgclock.js in sync (dock/portal)" ok || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line" +for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html','stuck/index.html']) { if (!fs.existsSync(f)) continue; const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); } +if (fs.existsSync('dock/clock/vgclock.js')) new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" \ + && say "page JS" ok || { say "page JS" "SYNTAX ERROR"; fail=1; } +fi +if [[ -f live/src/vgclock.js && -f dock/clock/vgclock.js ]]; then + diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null \ + && say "vgclock.js in sync (dock/portal)" ok \ + || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line" +else + say "vgclock.js in sync" "(skipped: only one copy in this checkout)" +fi # generated Caddyfile / mediamtx / copyparty from a dry configure run must at least be produced + parse T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c " import configure as c, yaml diff --git a/configure.py b/configure.py index 67353f6..bd025a6 100755 --- a/configure.py +++ b/configure.py @@ -345,10 +345,23 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l # vanguard:diag {diag_host} {{ encode zstd gzip -{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))} # Kibana IS this host: logs (Discover), dashboards and alerting in one app, served at the root so there - # is no path prefix to get wrong and no index page of links. Elasticsearch itself is never published. - reverse_proxy 127.0.0.1:5601 {{ - flush_interval -1 +{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))} # Triage lives on this host, alongside the dashboards it belongs with. Both handles come BEFORE the + # catch-all below, and the Kibana proxy is wrapped in its own handle so the three are evaluated in the + # order written rather than by Caddy's directive ordering. + handle /stuck/api {{ + reverse_proxy 127.0.0.1:8080 # dumpd: the list, and retry/bin for the admin groups + }} + handle_path /stuck* {{ + root * /srv/stuck + try_files {{path}} /index.html + file_server # stuck-recording triage; this host is already media-admins only + }} + # Kibana IS the rest of this host: logs (Discover), dashboards and alerting in one app, served at the + # root so there is no path prefix to get wrong. Elasticsearch itself is never published. + handle {{ + reverse_proxy 127.0.0.1:5601 {{ + flush_interval -1 + }} }} }} """ @@ -390,7 +403,14 @@ groups: {{http.request.header.X-Authentik-Groups}}" reverse_proxy 127.0.0.1:8889 }} handle_path /hls/* {{ - reverse_proxy 127.0.0.1:8888 + # handle_path strips /hls before proxying, so MediaMTX's own redirects come back root-relative and + # lose the prefix. It redirects every playlist request once, to ?cookieCheck=1, so without this the + # browser follows Location: /<user>/<Title>/index.m3u8 into the SPA catch-all below, receives + # index.html, and hls.js reports "manifestParsingError" — a web page where a manifest should be. + reverse_proxy 127.0.0.1:8888 {{ + header_down Location ^/ /hls/ + flush_interval -1 # low-latency HLS sends partial segments; do not buffer them + }} }} handle {{ root * /srv/live @@ -709,23 +729,15 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"]) run(["sudo", "install", "-d", "/var/lib/vanguard"]) # deadman state run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"]) # script counters -> dumpd /metrics - # The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in - # /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md). - media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py", - "bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()] - if media: - run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"]) - if Path("bin/vgmetrics.sh").exists(): - run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"]) # sourced by the sweep and vgdeband - if Path("bin/vgnotify.sh").exists(): - run(["sudo", "install", "-m", "0644", "bin/vgnotify.sh", "/usr/local/bin/"]) # sourced by the alerts timer and the deadman + # Everything bin/install.list names. Without this an --apply updated the containers but left STALE + # binaries in /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time. + for mode, path in bin_manifest(): + run(["sudo", "install", "-m", mode, path, "/usr/local/bin/"]) if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")): - run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"]) run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"]) run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"]) # The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord. if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")): - run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"]) run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"]) run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"]) run(["sudo", "systemctl", "restart", "dumpd"]) @@ -741,6 +753,26 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis print(" run docker compose restart mediamtx between streams (cuts a live recording). Adding/removing") print(" streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.") +def bin_manifest(): + """(mode, path) for everything that belongs in /usr/local/bin, from bin/install.list. + + One list, shared with install.sh and deploy.sh. Three hand-kept copies had already drifted: a + push-to-deploy was installing five of the ten, so vgstuck.sh, both sourced helpers and both timer + scripts never reached the host unless somebody ran --apply by hand. + """ + out = [] + try: + for line in Path("bin/install.list").read_text().splitlines(): + line = line.strip() + if not line or line.startswith("#"): + continue + mode, path = line.split(None, 1) + if Path(path).exists(): + out.append((mode, path)) + except FileNotFoundError: + pass + return out + def compose_cmd(): """docker compose, with sudo only if this user cannot reach the daemon.""" if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0: diff --git a/deploy.sh b/deploy.sh index ccd2eb5..206dda1 100755 --- a/deploy.sh +++ b/deploy.sh @@ -46,20 +46,31 @@ apply_changes() { if [[ -x check.sh ]]; then asowner bash check.sh >/tmp/vg-check.log 2>&1 || { log "check.sh FAILED — aborting"; tail -20 /tmp/vg-check.log; notify "❌ deploy $new: check.sh failed"; return 1; }; fi # 1. host scripts + units - if has '^bin/'; then run install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/; fi + # bin/install.list, not a second hand-kept list: this one had drifted to five of ten files, so a push + # never installed vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script. + if has '^bin/'; then + while read -r mode path; do + [[ -f $path ]] && run install -m "$mode" "$path" /usr/local/bin/ + done < <(grep -vE '^[[:space:]]*(#|$)' bin/install.list) + fi if has '^systemd/'; then run install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/; run systemctl daemon-reload; fi if has '^deploy\.sh$'; then run install -m 0755 deploy.sh /usr/local/bin/vanguard-deploy; fi if has '^deployBotBundle\.sh$'; then run install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot; fi if has '^firewall/nftables\.conf$'; then run install -m 0644 firewall/nftables.conf /etc/nftables.conf; run nft -f /etc/nftables.conf; fi # 2. regenerate configs (Caddy reload, copyparty restart) — configure.py decides; MediaMTX is handled below - if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/'; then + if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/|^stuck/|^observability/kibana-setup\.py$'; then local before_mtx; before_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1) if [[ $DRY == 1 ]]; then log "\$ (as $OWNER) VG_DRY=1 python3 configure.py --apply"; else sudo -u "$OWNER" -H python3 configure.py --apply >/tmp/vg-configure.log 2>&1 || { log "configure --apply FAILED"; tail -20 /tmp/vg-configure.log; return 1; }; fi local after_mtx; after_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1) [[ $before_mtx == "$after_mtx" ]] || { log "mediamtx.yml changed"; restart_mediamtx_or_defer; } fi if has '^bin/dumpd\.py$|^systemd/dumpd\.service$'; then run systemctl restart dumpd; fi + # Beat and Kibana configs are bind-mounted, so a content change needs the container recreated. Nothing + # handled this before: a push that changed filebeat.yml silently did nothing. + if has '^observability/filebeat\.yml$'; then run docker compose up -d --force-recreate filebeat; fi + if has '^observability/metricbeat\.yml$'; then run docker compose up -d --force-recreate metricbeat; fi + if has '^observability/kibana\.yml$'; then run docker compose up -d --force-recreate kibana; fi # 3. compose: recreate changed services, but never MediaMTX while live if has '^docker-compose\.yml$|^bot/Dockerfile$|^bot/entrypoint\.sh$'; then diff --git a/docker-compose.yml b/docker-compose.yml index a5bebf0..3b3e97d 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -72,6 +72,7 @@ services: - ./dock:/srv/dock:ro - ./clips:/srv/clips:ro # clip-intake self-service page - ./about:/srv/about:ro # gateway page (/about) + - ./stuck:/srv/stuck:ro # stuck-recording triage page (main host /stuck) - /srv/live:/srv/live:ro # live portal static build (deployLive.sh) - caddy_data:/data - caddy_config:/config diff --git a/install.sh b/install.sh index 91d6892..a5bff06 100755 --- a/install.sh +++ b/install.sh @@ -56,9 +56,11 @@ for d in /srv/media/clips/*/; do done echo "== host scripts + units" -install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/ -install -m 0644 bin/vgmetrics.sh /usr/local/bin/ # sourced by the sweep/vgdeband; counters for dumpd's /metrics -install -m 0644 bin/vgnotify.sh /usr/local/bin/ # sourced by the alerts timer + deadman; Discord posts and mentions +# bin/install.list is the single list, shared with configure.py --apply and deploy.sh. Three hand-kept +# copies had drifted, so a push-to-deploy was installing half of them. +grep -vE '^\s*(#|$)' bin/install.list | while read -r mode path; do + install -m "$mode" "$path" /usr/local/bin/ +done install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot # dumpd runs this for the bot CI's POST /deploy-bot install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/ # watches the observability stack from outside it install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/ # takes Kibana's alert documents to Discord diff --git a/live/src/broadcast.js b/live/src/broadcast.js index 48cc33a..b73718e 100644 --- a/live/src/broadcast.js +++ b/live/src/broadcast.js @@ -34,7 +34,11 @@ export async function captureWindow() { } } -export const MIN_KBPS = 6000 +// A FLOOR, not a target. It was 6000, which told Chrome never to encode below 6 Mbps: on an uplink that +// could not sustain that, Chrome kept sending anyway, the network discarded the excess, and MediaMTX logged +// "invalid FU-A packet (non-starting)" while every viewer saw blocky corruption. A floor this low still +// skips the painfully slow cold ramp without disabling congestion control. +export const MIN_KBPS = 1200 export const START_KBPS = 9000 function preferCodecs(pc, kind = 'video') { @@ -54,7 +58,8 @@ function preferCodecs(pc, kind = 'video') { tx.setCodecPreferences([...high, ...base, ...vp9, ...rest]) } -// Chrome honours x-google-* fmtp hints: skip the slow bandwidth ramp and never sit below MIN_KBPS. +// Chrome honours x-google-* fmtp hints. start-bitrate skips the slow cold ramp, max-bitrate is the ceiling, +// and min-bitrate is a floor that must stay LOW: a high floor defeats congestion control entirely. function mungeBitrates(sdp) { return sdp.replace(/^a=fmtp:(\d+) (.*)$/gm, (line, pt, params) => /packetization-mode|profile-id|VP9/i.test(params) || /H264|VP9/i.test(sdp.match(new RegExp(`a=rtpmap:${pt} .*`))?.[0] || '') @@ -95,9 +100,10 @@ export async function publish({ stream, path, user, pass, onState }) { const vs = pc.getSenders().find(s => s.track?.kind === 'video') if (vs) { const p = vs.getParameters(); p.encodings = p.encodings?.length ? p.encodings : [{}] - p.encodings[0].maxBitrate = MAX_KBPS * 1000 + p.encodings[0].maxBitrate = MAX_KBPS * 1000 // a ceiling; congestion control may go far below it p.encodings[0].maxFramerate = 60 - p.encodings[0].scaleResolutionDownBy = 1 + // No scaleResolutionDownBy: pinning it to 1 forbids the encoder from shedding resolution under + // pressure, which leaves it nothing to give but packets it cannot deliver. p.degradationPreference = 'balanced' await vs.setParameters(p).catch(() => {}) } diff --git a/observability/kibana-setup.py b/observability/kibana-setup.py index 3e0af4e..07dee40 100755 --- a/observability/kibana-setup.py +++ b/observability/kibana-setup.py @@ -4,6 +4,9 @@ connector "Vanguard alert index" — the .index connector, writing into the vanguard-alerts index rule "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc data view "Vanguard logs + metrics" — filebeat-*,metricbeat-* so Discover and Lens have one source + landing "Vanguard ops" dashboard — our own links, and defaultRoute so diag.BASE opens on it + +Run one step on its own by naming it: kibana-setup.py landing (also: connector, rule, dataview, all) WHY AN INDEX AND NOT A WEBHOOK On a basic license the only connector types Kibana allows are .index and .server-log; every connector that @@ -38,11 +41,25 @@ delivery script recognises the heartbeat and keeps it out of Discord. Use "On st for real alarms; only the heartbeat wants to fire every interval. """ import json, os, time, urllib.error, urllib.request +from pathlib import Path # env_hosts() reads the stack's .env + +import sys + +STEPS = ("connector", "rule", "dataview", "landing") +WANTED = sys.argv[1] if len(sys.argv) > 1 else "all" +if WANTED not in STEPS + ("all",): + raise SystemExit(f"unknown step {WANTED!r}; pick one of: {', '.join(STEPS)}, all") +def wanted(step): + return WANTED in ("all", step) KIBANA = "http://127.0.0.1:5601" ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts") HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1", - "Authorization": "ApiKey " + os.environ["KEY"]} + "Authorization": "ApiKey " + os.environ["KEY"], + # Kibana 9 ships with server.restrictInternalApis on. /api/kibana/settings (defaultRoute) is an + # internal API and refuses without this header: "exists but is not available with the current + # configuration". Sending it on every call is harmless; loosening the restriction is not. + "x-elastic-internal-origin": "Kibana"} def wait_for_kibana(seconds=300): """Kibana has to be *available*, not merely listening. @@ -87,26 +104,32 @@ wait_for_kibana() # 1. the alert bus connector CONNECTOR_NAME = "Vanguard alert index" -existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")} -if CONNECTOR_NAME in existing: +connector_id = None +if wanted("connector") or wanted("rule"): + existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")} + if CONNECTOR_NAME in existing: connector_id = existing[CONNECTOR_NAME] print("connector already present:", connector_id) -else: - connector_id = req("POST", "/api/actions/connector", { + elif not wanted("connector"): + raise SystemExit(f"connector {CONNECTOR_NAME!r} does not exist yet — run the connector step first") + else: + connector_id = req("POST", "/api/actions/connector", { "name": CONNECTOR_NAME, "connector_type_id": ".index", # executionTimeField stamps every document, so a rule body never has to get the timestamp right. "config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"}, "secrets": {}})["id"] - print("connector created:", connector_id, "->", ALERT_INDEX) + print("connector created:", connector_id, "->", ALERT_INDEX) # 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked # harmless and meant the data view was never created on any system where the rule already existed, which # is every re-run. RULE_NAME = "Vanguard alerting heartbeat" -existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", []) +existing_rule = None if not wanted("rule") else next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", []) if r["name"] == RULE_NAME), None) -if existing_rule: +if not wanted("rule"): + pass +elif existing_rule: print("rule already present:", existing_rule["id"], "enabled" if existing_rule.get("enabled") else "DISABLED") else: @@ -152,10 +175,111 @@ else: # detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and # metrics, which is the whole reason they share a cluster. VIEW_TITLE = "filebeat-*,metricbeat-*" -views = req("GET", "/api/data_views").get("data_view", []) -if any(v.get("title") == VIEW_TITLE for v in views): +views = req("GET", "/api/data_views").get("data_view", []) if wanted("dataview") else [] +if not wanted("dataview"): + pass +elif any(v.get("title") == VIEW_TITLE for v in views): print("data view already present:", VIEW_TITLE) else: view = req("POST", "/api/data_views/data_view", { "data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}}) print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE) + + +# 4. the landing page. Kibana stays at the root of the diag host; this just makes arriving there open OUR +# dashboard instead of Kibana's home. A markdown panel holds the links, and `defaultRoute` is the +# advanced setting that does the redirect. That is the whole trick, and it avoids moving Kibana under a +# /kibana prefix — which would have meant touching both Beats' setup.kibana, this script, the alarm +# script and the deadman's status check, with a silent crash-loop waiting for whichever one was missed. +DASHBOARD_ID = "vanguard-ops" + +def env_hosts(): + """Hostnames out of the stack's .env, so the links are real rather than examples.""" + values = {} + for candidate in (Path(".env"), Path("/home/debian/vanguard-stream/.env")): + if candidate.is_file(): + for line in candidate.read_text().splitlines(): + if "=" in line and not line.startswith("#"): + key, value = line.split("=", 1) + values[key.strip()] = value.strip() + break + return values + +def landing_markdown(hosts): + def url(key, path="/"): + host = hosts.get(key) + return f"https://{host}{path}" if host else None + rows = [ + ("Stuck recordings", "/stuck", "finished but never published — why, and retry or bin"), + ("Live portal", url("LIVE_HOST"), "watch every active stream"), + ("OBS dock", url("LIVE_HOST", "/dock/"), "go live, titles, dump tonight"), + ("Files", url("FILES_HOST"), "browse, download, upload finals"), + ("Jellyfin", url("WATCH_HOST"), "recordings by streamer and season"), + ("Gateway", url("MAIN_HOST"), "role-aware links and how-tos"), + ("Gitea", url("GIT_HOST"), "code and CI"), + ("Bot", url("BOT_HOST"), "the Discord bot's own pages"), + ("Syncthing", url("SYNC_HOST"), "clip intake, device pairing"), + ("Authentik", url("AUTH_HOST", "/if/admin/"), "users, groups, policies"), + ] + lines = ["### Vanguard", "", "| | |", "|---|---|"] + for name, link, why in rows: + if link: + lines.append(f"| **[{name}]({link})** | {why} |") + lines += [ + "", "### In here", "", + "| | |", "|---|---|", + "| **[Discover](/app/discover)** | search every log line; filter on `service.name` or `vanguard.category` |", + "| **[Dashboards](/app/dashboards)** | the Metricbeat system and Docker dashboards |", + "| **[Stack Rules](/app/management/insightsAndAlerting/triggersActions/rules)** | the six alarms, and whether each is firing |", + "| **[Index Management](/app/management/data/index_management/indices)** | index sizes and ILM |", + "", + "_Alarms post to Discord through the `vanguard-alerts` index, not from Kibana directly: a basic " + "licence has no webhook connector. The host deadman watches that whole path from outside._", + ] + return "\n".join(lines) + +if wanted("landing"): + hosts = env_hosts() + panel = { + "type": "visualization", + "panelIndex": "1", + "gridData": {"x": 0, "y": 0, "w": 24, "h": 18, "i": "1"}, + "embeddableConfig": { + "savedVis": { + "id": "", "title": "", "description": "", "type": "markdown", + "params": {"fontSize": 12, "openLinksInNewTab": False, + "markdown": landing_markdown(hosts)}, + "uiState": {}, + "data": {"aggs": [], "searchSource": {"query": {"query": "", "language": "kuery"}, + "filter": []}}, + }, + "enhancements": {}, + }, + } + saved = req("POST", f"/api/saved_objects/dashboard/{DASHBOARD_ID}?overwrite=true", { + "attributes": { + "title": "Vanguard ops", + "description": "Landing page for the diagnostics host. Edit freely: add metric panels beside the links.", + "panelsJSON": json.dumps([panel]), + "optionsJSON": json.dumps({"hidePanelTitles": True, "useMargins": True, + "syncColors": False, "syncCursor": True, "syncTooltips": False}), + "timeRestore": False, + "kibanaSavedObjectMeta": { + "searchSourceJSON": json.dumps({"query": {"query": "", "language": "kuery"}, "filter": []})}, + }}) + # Prove it rather than assume it: read the dashboard back and count the panels Kibana actually stored. + stored = req("GET", f"/api/saved_objects/dashboard/{DASHBOARD_ID}") + panels = json.loads(stored.get("attributes", {}).get("panelsJSON", "[]")) + links = sum(1 for line in landing_markdown(hosts).splitlines() if line.startswith("| **[")) + print(f"dashboard: {stored.get('id')} | title {stored.get('attributes', {}).get('title')!r} | " + f"{len(panels)} panel(s) stored | {links} links in the markdown") + if not panels: + raise SystemExit(" the dashboard saved but with no panels — Kibana rejected the panel shape") + print("open:", f"/app/dashboards#/view/{DASHBOARD_ID}") + + route = f"/app/dashboards#/view/{DASHBOARD_ID}" + req("POST", "/api/kibana/settings", {"changes": {"defaultRoute": route}}) + now = req("GET", "/api/kibana/settings").get("settings", {}).get("defaultRoute", {}).get("userValue") + print("defaultRoute:", now or "(not set!)") + if now != route: + raise SystemExit(" Kibana did not keep that defaultRoute — set it by hand in Advanced Settings") diff --git a/stuck/index.html b/stuck/index.html new file mode 100644 index 0000000..28f761b --- /dev/null +++ b/stuck/index.html @@ -0,0 +1,113 @@ +<!doctype html> +<html lang="en"> +<head> +<meta charset="utf-8"> +<title>Stuck recordings — Vanguard + + + + +
+

Stuck recordings

+

Recordings that finished but never reached Jellyfin or copyparty. Retry puts one back in the + queue; bin moves it to the trash, recoverable for three days just like a dumped night.

+
Loading…
+
Refreshes every 30 seconds. Trends and history live on the diagnostics dashboard.
+
+ + +