Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms

Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

LOGGING.md

@@ -29,6 +29,26 @@ and `docker` modules, not from an exporter — node_exporter and cAdvisor were r
 went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of
 them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher).
 
+### Stuck recordings: counts in Kibana, names in the logs, decisions on the host
+
+Three layers on purpose, because each answers a different question and none of them needs a new web page.
+
+- **How bad is it, and is it getting worse** — the gauges: `vanguard_media_pending`,
+  `vanguard_media_pending_oldest_seconds`, `vanguard_media_failed_files`. A Lens dashboard over
+  `metricbeat-*` shows the trend across a night, which is the thing a list never can.
+- **Which recordings** — the sweep logs `event=parked file= stream= log=` once per run for every file a
+  failure log is holding back, plus `event=failed` with ffmpeg's own stderr when it first breaks. Both land
+  in `filebeat-*`, so a Discover search or a dashboard table names them and stays current. Names deliberately
+  do NOT come from the metrics: a label per filename would wreck index cardinality for no gain.
+- **What to do about it** — `bin/vgstuck.sh` on the host. It classifies every unpublished recording as
+  `parked` (a failure log is blocking it, with the reason), `published` (an output already exists),
+  `waiting` (nothing blocking it yet) or `writing` (still recording), then `retry <match>` clears the
+  blocker so the sweep tries again and `drop <match>` moves it to `trash/<today>/` for the same three-day
+  undo as a dumped night. `drop --all` is refused.
+
+A leftover `<output>.mp4.part` does **not** block anything — ffmpeg overwrites it with `-y`. It only wastes
+disk, and `retry` clears it.
+
 ### Our own metrics, and how to add one
 
 MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising
@@ -135,8 +155,15 @@ to Discord, and the pipeline is split:
 
     Kibana rule  --.index connector-->  vanguard-alerts  --vanguard-alerts.timer-->  Discord webhook
 
-- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index") and the heartbeat
+- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index"), the heartbeat rule,
-  rule. Run it once, *after* Elasticsearch security is on.
+  the data view, and the "Vanguard ops" landing dashboard plus the `defaultRoute` that makes `diag.BASE`
+  open on it. Run it once, *after* Elasticsearch security is on. Name a step to run only that one:
+  `kibana-setup.py landing`.
+  Kibana deliberately stays at the **root** of that host. Moving it under `/kibana` would mean
+  `server.basePath` plus `rewriteBasePath`, a Caddy handle that does not strip the prefix, and appending the
+  prefix in both Beats' `setup.kibana`, this script, the alarm script and the deadman's status check — and a
+  miss in the Beats one crash-loops Metricbeat, because it exits when Kibana is unreachable. `defaultRoute`
+  buys the landing page without any of that.
 - **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind`
   as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused
   them, and replayable after the fact.

README.md

@@ -69,7 +69,8 @@ works from ~/vanguard-stream without sudo.
     ./configure.py --full      # first time: asks everything, then applies
     ./configure.py             # later: add/remove/reset streamers and editors, then apply
     ./configure.py --show      # print current users and the OBS settings to hand out
-    ./configure.py --set K=V   # change one setting (Discord tokens/webhooks, retention) without a full pass
+    ./configure.py --set K=V   # change one setting (Discord tokens/webhooks, retention, alarm mentions)
+                               # run it with no arguments to list what is settable
 
 Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates
 `Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates
@@ -437,8 +438,27 @@ Jellyfin and Gitea use OIDC providers instead, not this.
 
 ## 5h. Diagnostics: Kibana at `diag.HOST`
 
-Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover,
+Four containers, one store, one UI. Kibana is served at the **root** of `diag.HOST`, with no sub-path to get
-dashboards, and alerting, with no sub-paths and no index page of links.
+wrong. Arriving there opens a dashboard of ours, "Vanguard ops", via Kibana's `defaultRoute` setting: links to
+every host and to the triage page, plus shortcuts into Discover, Dashboards, Stack Rules and Index Management.
+It is an ordinary dashboard, so add metric panels beside the links (`vanguard_media_pending` first).
+
+Two paths on that host are ours rather than Kibana's:
+
+| path | what |
+|---|---|
+| `/stuck` | **stuck recordings**: everything that finished but never published, why each one is stuck, and Retry / Bin buttons. The list is readable by anyone the gate allows; the actions are enforced in dumpd for the admin groups, not hidden in the page. Binning moves the file to `trash/<today>/`, the same 3-day undo as a dumped night |
+| `/stuck/api` | dumpd serves it. Both the page and `vgstuck.sh` on the host run the same script, so they cannot disagree about why something is stuck |
+
+Kibana deliberately does **not** live under `/kibana`. That would need `server.basePath` plus
+`rewriteBasePath`, a Caddy handle that does not strip the prefix, and the prefix appended in both Beats'
+`setup.kibana`, `kibana-setup.py`, `kibana-alarms.py` and the deadman's status check — and missing the Beats
+one crash-loops Metricbeat, which exits when Kibana is unreachable. `defaultRoute` buys the landing page
+without any of it.
+
+`observability/kibana-setup.py` creates the first-run objects and takes a step name, so you can redo one:
+`connector`, `rule`, `dataview`, `landing`, or all of them when omitted. `observability/kibana-alarms.py`
+creates the six alarm rules.
 
 | container | job |
 |---|---|
@@ -573,6 +593,13 @@ Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
 
 ## Deployment (push to deploy — built, OFF by default)
 
+What lands in `/usr/local/bin` is listed once, in `bin/install.list`, and read by `install.sh`,
+`configure.py --apply` and `deploy.sh`. It used to be three hand-kept lists and they drifted: a push was
+installing five of ten scripts, so `vgstuck.sh`, both sourced helpers and both timer scripts only reached the
+host if somebody ran `--apply` by hand. `check.sh` now fails if the list names a missing file, or if
+something in `bin/` is neither listed nor explicitly excluded (`mtx-on-ready.sh` is mounted into the MediaMTX
+container; `vgbot-merge-sqlite.py` is a one-off migration).
+
 The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea →
 `./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token
 secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout.
@@ -715,4 +742,6 @@ evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
 |------|------|------|
 | normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
 | trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
-| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time and on `clips.` at /me/clips |
+| vanguard-alerts.timer | every 1 min | delivers Kibana's alert documents to Discord, with a 15-min per-rule repeat guard |
+| vanguard-deadman.timer | every 5 min | watches the alerting stack from outside it, and the disks |
+| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time, on `clips.` at /me/clips and on `diag.` at /stuck/api |

STATUS.md

@@ -60,9 +60,11 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
 | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
 | Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
+| diag landing | `diag.…` opens on a "Vanguard ops" dashboard: a markdown panel of links to every host plus the stuck page, and shortcuts to Discover, Dashboards, Stack Rules and Index Management. Kibana still owns the root; `defaultRoute` just redirects there. Created by `kibana-setup.py landing`, so a rebuild recreates it | **added 2026-11-04** |
 | Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
 | Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
 | Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
+| Stuck recordings | A page at `https://diag.…/stuck` for anyone past the gate, on the triage host (already `media-admins` only), linked from the gateway's Media admin card and from the diag landing dashboard: every unpublished recording, why it is stuck, and Retry / Bin buttons. dumpd serves `/stuck/api` and allows the actions only for the admin groups; both the page and the CLI run `bin/vgstuck.sh`, so they can never disagree. `bin/vgstuck.sh` triages anything that never published: it says why each one is stuck (`parked` by a failure log with ffmpeg's reason, `published` collision, `waiting`, or still `writing`), then `retry <match>` clears the blocker and `drop <match>` moves it to trash with the usual 3-day undo. `drop --all` is refused. The sweep also logs `event=parked` once per run so Kibana can name them, not just count them | **added 2026-11-04** |
 | Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
 | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
@@ -117,6 +119,9 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
 - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
 - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
+- **A high `x-google-min-bitrate` disables WebRTC congestion control.** Go Live sent `min-bitrate=6000`, so Chrome never encoded below 6 Mbps; on an uplink that could not carry it, the excess was dropped, MediaMTX logged `invalid FU-A packet (non-starting)` and every viewer saw blocky corruption. The loss was on the PUBLISH session (4761 packets for one streamer) while the viewer sessions lost nothing, which is the fingerprint: the server relays damage it received. Floor lowered to 1200 and `scaleResolutionDownBy` unpinned so the encoder can shed resolution. Found 2026-11-04.
+- `tracks=['Opus','H264']` on a path means a browser (Go Live); `MPEG-4 Audio` means OBS. Quick way to tell who is publishing how.
+- **`handle_path` strips the prefix, so an upstream's own redirects lose it.** MediaMTX redirects every HLS playlist request once (to `?cookieCheck=1`) with a root-relative `Location`, so the browser followed `/<user>/<Title>/index.m3u8` into the portal's SPA catch-all, got `index.html`, and hls.js said `manifestParsingError` — a web page where a manifest should be. Fixed with `header_down Location ^/ /hls/`. `curl` against MediaMTX directly hides this completely, because there is no prefix to lose. Found 2026-11-04.
 - **Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked** in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a `tmcd` data stream is the fingerprint. First working Windows audio: 2026-11-03.
 - The sweep's band-removal branch **overwrote `$reason`**, which carried the probe's `audio=<codec>` marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got `-c:a copy` and shipped Opus inside an MP4. Fixed by keeping `probe_reason` separately; it only became reachable once Go Live audio worked.
 - **Authentik has two invalidation flows and the names are a trap.** `default-provider-invalidation-flow` is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so `/logout` drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. `default-invalidation-flow` ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03.

about/index.html

@@ -120,6 +120,7 @@
     <h2>Media admin</h2>
     <p class="lead" data-group="media-admins" data-yes="You are in <code>media-admins</code>: full rights on every folder, trash, Syncthing, dump anyone's night." data-no="Admin view — you are <b>not</b> in <code>media-admins</code>: copyparty will show you nothing and <code>/sync/</code> will 403 until you get the role."></p>
     <div class="links">
+      <a class="card" id="l-stuck" href="#"><b>Stuck recordings</b><span>never published — see why, retry or bin</span></a>
       <a class="card" id="l-trash" href="#"><b>Trash</b><span>dumped nights, purged after 3 days</span></a>
       <a class="card" id="l-sync" href="#"><b>Syncthing</b><span>device pairing, folder states</span></a>
       <a class="card" href="/whoami"><b>Who am I</b><span>identity headers as the apps see them</span></a>
@@ -129,6 +130,7 @@
         <li>Group changes in Authentik apply on the next login: <a href="/logout">sign out</a> and back in.</li>
         <li>Dump anyone's night: the dock's Dump works on your own; for others use the API with <code>{"user":"name"}</code> or ask the owner.</li>
         <li>New gamer: they self-serve on the <a class="l-clips" href="#">clips page</a>; you only need to watch Syncthing accept the device.</li>
+        <li><a class="l-stuck-note" href="#">Stuck recordings</a> (on the diagnostics host, with the dashboards) lists anything that finished but never published, with the reason. Retry puts it back in the queue; binning keeps the same 3-day undo as a dumped night.</li>
       </ul>
     </details>
   </section>
@@ -179,6 +181,8 @@ $('l-git').href = sub('git'); $('l-bot').href = sub('bot'); $('l-auth').href = s
 $('l-jfadmin').href = sub('watch') + '/web/#/dashboard'; $('l-gitadmin').href = sub('git') + '/-/admin';
 $('l-dock').href = sub('live') + '/dock/'; $('dockurl').textContent = sub('live') + '/dock/';
 $('l-files').href = sub('files') + '/'; $('l-finals').href = sub('files') + '/finals/'; $('l-trash').href = sub('files') + '/trash/'; $('l-sync').href = sub('sync') + '/';
+$('l-stuck').href = sub('diag') + '/stuck';
+document.querySelectorAll('a.l-stuck-note').forEach(a => a.href = sub('diag') + '/stuck');
 $('l-clips').href = sub('clips') + '/'; document.querySelectorAll('a.l-clips').forEach(a => a.href = sub('clips') + '/'); $('gitssh').textContent = `ssh://git@${'git.' + base}:2222/org/repo.git`;
 (async () => {
   try {

bin/dumpd.py

@@ -21,6 +21,10 @@ POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a D
                    HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
                    May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
                    into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body.
+GET  /stuck/api  -> every unpublished recording and why it is stuck (any gated user)
+POST /stuck/api  -> {"action":"retry"|"drop","src":...}: clear the blocker, or move it to trash with the
+             usual 3-day undo. Admin groups only; both actions run bin/vgstuck.sh so the web page and the
+             CLI share one implementation.
 GET  /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live
              gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here.
 POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid
@@ -425,6 +429,43 @@ def verify_token(token):
         return None
     return user.lower()
 
+# ---- stuck recordings -----------------------------------------------------------------------------
+# GET  /stuck/api   every unpublished recording and why, for anyone the gate let through
+# POST /stuck/api   {"action":"retry"|"drop","src":"<path>"} — admins only
+# The classification and the file moves both live in bin/vgstuck.sh, so the web page and the CLI can never
+# disagree about why something is stuck, and "delete" means the same three-day trash in both.
+STUCK_TOOL = os.environ.get("VGSTUCK_TOOL", "/usr/local/bin/vgstuck.sh")
+
+def stuck_list():
+    done = subprocess.run([STUCK_TOOL, "json"], capture_output=True, text=True, timeout=180)
+    try:
+        return json.loads(done.stdout or "[]")
+    except ValueError:
+        log("stuck_list_failed", level="error", rc=done.returncode, err=(done.stderr or "")[:200])
+        return []
+
+def stuck_target(src):
+    """Resolve what the browser asked us to act on, or raise. This is the only thing between a request body
+    and a file move, so it is deliberately strict: inside the streams tree, an unpublished .live recording,
+    and really there."""
+    path = Path(src).resolve()
+    root = (ROOT / "streams").resolve()
+    if root != path and root not in path.parents:
+        raise ValueError("that path is not inside the streams tree")
+    if f"{os.sep}.live{os.sep}" not in str(path) or path.suffix.lower() != ".mp4":
+        raise ValueError("that is not an unpublished recording")
+    if not path.is_file():
+        raise ValueError("no such recording (already published or already binned?)")
+    return path
+
+def stuck_action(action, src, who):
+    path = stuck_target(src)
+    done = subprocess.run([STUCK_TOOL, action, str(path)], capture_output=True, text=True, timeout=600)
+    lines = [l for l in (done.stdout + done.stderr).splitlines() if l.strip()]
+    log(f"stuck_{action}", level="info" if done.returncode == 0 else "error",
+        file=str(path), by=who, rc=done.returncode)
+    return {"ok": done.returncode == 0, "rc": done.returncode, "output": lines[-8:]}
+
 # ---- GET /metrics -----------------------------------------------------------------------------------
 # Prometheus text for Metricbeat's prometheus module. Two sources are merged:
 #   1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/<component>.counters)
@@ -619,6 +660,10 @@ class H(BaseHTTPRequestHandler):
             return self._json(200, {"user": who, "groups": groups, "admin": is_admin,
                                     "name": self.headers.get("X-Authentik-Name") or who, "email": self.headers.get("X-Authentik-Email") or "",
                                     "discord_id": proxy_claim(self.headers, "discord_id")})
+        if self.path == "/stuck/api":
+            who, is_admin = identity(self.headers)
+            if not who: return self._json(401, {"error": "not authenticated"})
+            return self._json(200, {"admin": is_admin, "user": who, "items": stuck_list()})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
             if not who: return self._json(401, {"error": "not authenticated"})
@@ -747,6 +792,24 @@ class H(BaseHTTPRequestHandler):
                 return self._json(401, {"error": "path must be <your-username>/<Title>"})
             log("publish_allow", user=user, stream=path, ip=req.get("ip"))
             return self._json(200, {"ok": True})
+        if self.path == "/stuck/api":
+            who, is_admin = identity(self.headers)
+            if not who: return self._deny()
+            if not is_admin:
+                return self._json(403, {"error": f"retrying and binning recordings is for {', '.join(sorted(ADMIN_GROUPS))}"})
+            try:
+                body = json.loads(self.rfile.read(int(self.headers.get("Content-Length") or 0)) or b"{}")
+            except ValueError as error:
+                return self._json(400, {"error": str(error)})
+            action = body.get("action")
+            if action not in ("retry", "drop"):
+                return self._json(400, {"error": "action must be retry or drop"})
+            try:
+                return self._json(200, stuck_action(action, str(body.get("src", "")), who))
+            except ValueError as error:
+                return self._json(400, {"error": str(error)})
+            except Exception as error:
+                return self._json(500, {"error": str(error)})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
             if not who: return self._json(401, {"error": "not authenticated"})

bin/install.list

@@ -0,0 +1,19 @@
+# What gets installed into /usr/local/bin, and with which mode. ONE list, read by install.sh,
+# configure.py --apply and deploy.sh, because three hand-kept copies drifted: a push-to-deploy was
+# silently not installing vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script.
+# check.sh fails if a file here is missing, or if something executable in bin/ is neither listed nor
+# excluded below.
+0755 bin/normalize-sweep.sh
+0755 bin/trash-purge.sh
+0755 bin/dumpd.py
+0755 bin/vgstrip.py
+0755 bin/vgdeband.sh
+0755 bin/vgstuck.sh
+0755 bin/vanguard-deadman.sh
+0755 bin/vanguard-alerts.sh
+# sourced, never executed
+0644 bin/vgmetrics.sh
+0644 bin/vgnotify.sh
+# NOT installed on the host, on purpose:
+#   exclude bin/mtx-on-ready.sh        mounted into the mediamtx container (docker-compose.yml)
+#   exclude bin/vgbot-merge-sqlite.py  one-off migration, run from the checkout

bin/normalize-sweep.sh

@@ -115,7 +115,12 @@ while IFS= read -r -d '' src; do
     if [[ -d "$d/.stfolder" ]]; then synced=1; stroot="$d"; break; fi
     d="$(dirname "$d")"
   done
-  [[ -e "$base.normalize-failed.log" ]] && continue
+  if [[ -e "$base.normalize-failed.log" ]]; then
+    # Say so every run. Skipping silently meant a parked file produced no log line after its original
+    # failure, so Kibana could only ever show a count and never which recording. bin/vgstuck.sh triages it.
+    vglog info "event=parked file=\"$src\" stream=\"$stream\" log=\"$base.normalize-failed.log\""
+    continue
+  fi
   fuser -s "$src" 2>/dev/null && continue            # still being written
   [[ -e "$src" ]] || continue                          # dumped meanwhile
 

bin/vanguard-alerts.sh

@@ -65,6 +65,10 @@ webhook=$(vg_webhook 2>/dev/null || true)
 
 now=$(date +%s)
 cursor=$(state_get cursor); cursor="${cursor:-0}"
+# Record every run, not just runs that did something. The deadman watches this file's age to tell whether
+# delivery is still happening at all, and almost every run is quiet: without this the file went stale within
+# minutes and the deadman reported the timer as dead, then recovered the moment an alert happened to flow.
+state_set last_run "$now"
 
 # Everything newer than the cursor that is not a heartbeat, oldest first so the cursor advances safely.
 query=$(printf '{"size":%d,"sort":[{"@timestamp":"asc"}],"query":{"bool":{"filter":[{"range":{"@timestamp":{"gt":%s,"format":"epoch_millis"}}}],"must_not":[{"term":{"kind":"heartbeat"}}]}}}' "$BATCH" "$cursor")

bin/vanguard-deadman.sh

@@ -127,6 +127,11 @@ if [[ -f $ALERT_STATE ]]; then
   age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") ))
   (( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered"
 fi
+# Freshness above proves the timer fires; this proves the run finished. A unit that starts every minute and
+# dies immediately would otherwise look alive, because the file is written at the start of the run.
+if systemctl is-failed --quiet vanguard-alerts.service 2>/dev/null; then
+  note "vanguard-alerts.service is in a failed state — $(systemctl show -p Result --value vanguard-alerts.service 2>/dev/null); alerts are written but not delivered"
+fi
 
 # the disks, independently of any agent
 while read -r pct mount; do

bin/vgstuck.sh

@@ -0,0 +1,143 @@
+#!/bin/bash
+# Triage recordings that never published: see why each one is stuck, then retry it or bin it.
+#
+#   vgstuck.sh                     list everything stuck, with the reason        (read-only, the default)
+#   vgstuck.sh retry <match>       clear what is blocking it so the sweep tries again
+#   vgstuck.sh retry --all         ...for everything that is parked
+#   vgstuck.sh drop  <match>       move the recording to trash: 3-day undo, same as "dump tonight"
+#   vgstuck.sh sweep               kick the sweep once, without waiting on it
+#   vgstuck.sh json                the same survey as JSON (what dumpd serves to the web page)
+#
+# <match> is any substring of the path (a streamer name, a title, a timestamp). It must match at least one
+# recording, and `drop` refuses --all: binning every stuck recording in one keystroke is never a thing
+# somebody meant to do.
+#
+# WHY A RECORDING STAYS IN .live/
+#   parked     a <Title>__<ts>.normalize-failed.log sits next to the intended output. The sweep skips the
+#              file entirely until that log is deleted, so it will never retry on its own.
+#   published  an output <Title>__<ts>.mp4 already exists, so the sweep leaves the source alone.
+#   waiting    nothing is blocking it; it is simply newer than MINS_IDLE or the sweep has not run yet.
+#   partial    a leftover <output>.mp4.part. This does NOT block anything (ffmpeg overwrites it with -y);
+#              it only wastes disk, and retry removes it.
+#
+# Nothing here deletes media. `drop` MOVES to /srv/media/trash/<today>/, which trash-purge clears after
+# three days, so a wrong call is recoverable for the same window as a dumped night.
+set -uo pipefail
+
+ROOT="${MEDIA_ROOT:-/srv/media}"
+TRASH="$ROOT/trash"
+MINS_IDLE="${MINS_IDLE:-3}"
+
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgstuck %s\n' "$(date -Is)" "$lvl" "$*"; }
+
+# Everything about one .live source, as: src|out|state|age_s|bytes|dur_s|reason
+survey() {
+  local src titledir user title stamp base out state reason dur
+  while IFS= read -r -d '' src; do
+    titledir="$(dirname "$(dirname "$src")")"
+    user="$(basename "$(dirname "$titledir")")"
+    title="$(basename "$titledir")"
+    stamp="$(basename "${src%.*}")"
+    base="$(dirname "$titledir")/${title}__${stamp}"
+    out="$base.mp4"
+    reason=""
+    if [[ -e "$base.normalize-failed.log" ]]; then
+      state=parked
+      reason="$(head -1 "$base.normalize-failed.log" 2>/dev/null)"
+      local why; why=$(grep -A2 '^--- ffmpeg stderr ---$' "$base.normalize-failed.log" 2>/dev/null | sed -n '2p')
+      [[ -n ${why:-} ]] && reason="$reason :: $why"
+    elif [[ -e "$out" ]]; then
+      state=published
+      reason="output already exists: $(basename "$out")"
+    elif [[ -n $(find "$src" -mmin "-$MINS_IDLE" -print -quit 2>/dev/null) ]]; then
+      state=writing
+      reason="touched in the last ${MINS_IDLE}m — still recording"
+    else
+      state=waiting
+      reason="nothing blocking it; the sweep has not got to it"
+    fi
+    [[ -e "$out.part" ]] && reason="$reason [leftover .part]"
+    dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$src" 2>/dev/null | cut -d. -f1)
+    printf '%s|%s|%s|%s|%s|%s|%s\n' "$src" "$out" "$state" \
+      "$(( $(date +%s) - $(stat -c %Y "$src") ))" "$(stat -c %s "$src")" "${dur:-0}" "$reason"
+  done < <(find "$ROOT/streams" -path '*/.live/*' -name '*.mp4' -print0 2>/dev/null)
+}
+
+show() {
+  local any=0 src out state age bytes dur reason
+  printf '\n  %-9s %-9s %9s %8s  %s\n' state waiting length size recording
+  while IFS='|' read -r src out state age bytes dur reason; do
+    any=1
+    printf '  %-9s %-9s %9s %7.0fM  %s\n' "$state" \
+      "$(( age / 3600 ))h$(printf '%02d' $(( (age % 3600) / 60 )))m" \
+      "$(printf '%d:%02d:%02d' $(( dur / 3600 )) $(( (dur % 3600) / 60 )) $(( dur % 60 )))" \
+      "$(awk -v b="$bytes" 'BEGIN{print b/1000000}')" "${src#"$ROOT"/streams/}"
+    printf '            %s\n' "$reason"
+  done < <(survey | sort -t'|' -k3,3 -k4,4nr)
+  (( any )) || printf '  nothing is stuck.\n'
+  printf '\n  retry one:  vgstuck.sh retry <substring>      bin one:  vgstuck.sh drop <substring>\n\n'
+}
+
+# rows matching the pattern, or everything when --all
+selected() {
+  local pattern="$1"
+  if [[ $pattern == --all ]]; then survey; else survey | grep -F -- "$pattern" || true; fi
+}
+
+do_retry() {
+  local pattern="$1" n=0 src out state age bytes dur reason
+  while IFS='|' read -r src out state age bytes dur reason; do
+    [[ $state == writing ]] && { vglog info "event=skip reason=still_recording file=\"$src\""; continue; }
+    rm -f "$out.part"                                  # leftover partial output: harmless, but no reason to keep
+    rm -f "${out%.mp4}.normalize-failed.log"           # THIS is what makes the sweep skip the file forever
+    n=$((n + 1))
+    vglog info "event=retry file=\"$src\" was=$state"
+  done < <(selected "$pattern")
+  (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; }
+  vglog info "event=retry_done count=$n msg=\"run vgstuck.sh sweep, or wait for the timer\""
+}
+
+do_drop() {
+  local pattern="$1" night n=0 src out state age bytes dur reason rel dst
+  [[ $pattern == --all ]] && { vglog error 'event=refused msg="drop --all is not allowed; name them"' >&2; return 2; }
+  night="$(date +%F)"
+  while IFS='|' read -r src out state age bytes dur reason; do
+    [[ $state == writing ]] && { vglog info "event=skip reason=still_recording file=\"$src\""; continue; }
+    for victim in "$src" "${src%.*}.json" "${out%.mp4}.normalize-failed.log" "$out.part"; do
+      [[ -e $victim ]] || continue
+      rel="${victim#"$ROOT"/}"
+      dst="$TRASH/$night/$rel"
+      mkdir -p "$(dirname "$dst")"
+      mv -f "$victim" "$dst" && vglog info "event=trashed file=\"$rel\" undo_days=3"
+    done
+    rmdir "$(dirname "$src")" "$(dirname "$(dirname "$src")")" 2>/dev/null || true
+    n=$((n + 1))
+  done < <(selected "$pattern")
+  (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; }
+  vglog info "event=drop_done count=$n trash=\"$TRASH/$night\""
+}
+
+# Same survey, as JSON. bin/dumpd.py serves this to the web page, so the classification lives in exactly
+# one place and the browser and the CLI can never disagree about why something is stuck.
+emit_json() {
+  survey | python3 -c '
+import json, sys
+rows = []
+for line in sys.stdin:
+    parts = line.rstrip("\n").split("|", 6)
+    if len(parts) != 7:
+        continue
+    src, out, state, age, size, dur, reason = parts
+    rows.append({"src": src, "out": out, "state": state, "age_s": int(age or 0),
+                 "bytes": int(size or 0), "duration_s": int(dur or 0), "reason": reason})
+print(json.dumps(rows))'
+}
+
+case "${1:-list}" in
+  list|"")  show ;;
+  json)     emit_json ;;
+  retry)    [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh retry <substring>|--all" >&2; exit 1; }; do_retry "$2" ;;
+  drop)     [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh drop <substring>" >&2; exit 1; }; do_drop "$2" ;;
+  sweep)    systemctl start --no-block normalize-sweep.service && vglog info "event=sweep_started" ;;
+  *)        sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 1 ;;
+esac

check.sh

@@ -13,12 +13,43 @@ for f in configure.py bin/dumpd.py bin/vgstrip.py observability/kibana-setup.py
   python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
 done
 rm -rf __pycache__ bin/__pycache__
-python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
+# bin/install.list is the single source for what lands in /usr/local/bin. Fail if it names something that
-node -e "
+# does not exist, or if a file in bin/ is neither listed nor explicitly excluded — that drift is exactly how
+# deploy.sh ended up installing five of ten scripts.
+if python3 - <<'PYMAN' >/dev/null 2>&1
+import re, sys
+from pathlib import Path
+text = Path("bin/install.list").read_text()
+listed = dict((m.group(2), m.group(1)) for m in re.finditer(r"^(0\d{3}) (\S+)$", text, re.M))
+excluded = set(re.findall(r"^#\s+exclude (\S+)", text, re.M))
+present = set("bin/" + p.name for p in Path("bin").iterdir() if p.is_file() and p.name != "install.list")
+missing = set(listed) - present
+unclaimed = present - set(listed) - excluded
+assert not missing, f"install.list names files that do not exist: {sorted(missing)}"
+assert not unclaimed, f"bin/ files neither listed nor excluded in install.list: {sorted(unclaimed)}"
+PYMAN
+then say "bin/install.list" ok; else say "bin/install.list" "DRIFTED — see the assertion"; fail=1; fi
+# Only files that are actually here. The server checkout is partial (scp, not a full clone), and a tool that
+# reports "broken" when it means "absent" gets ignored — the same reason the deadman reports on change only.
+yamls=(); for f in docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml; do [[ -f $f ]] && yamls+=("$f"); done
+python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" "${yamls[@]}" \
+  && say "yaml (${#yamls[@]} files)" ok || { say "yaml" "INVALID"; fail=1; }
+if ! command -v node >/dev/null; then
+  say "page JS" "(skipped: node not installed here)"
+else
+  node -e "
 const fs=require('fs');
-for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
+for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html','stuck/index.html']) { if (!fs.existsSync(f)) continue; const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
-new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" && say "dock + clock JS" ok || { say "dock + clock JS" "SYNTAX ERROR"; fail=1; }
+if (fs.existsSync('dock/clock/vgclock.js')) new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" \
-diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null && say "vgclock.js in sync (dock/portal)" ok || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line"
+    && say "page JS" ok || { say "page JS" "SYNTAX ERROR"; fail=1; }
+fi
+if [[ -f live/src/vgclock.js && -f dock/clock/vgclock.js ]]; then
+  diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null \
+    && say "vgclock.js in sync (dock/portal)" ok \
+    || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line"
+else
+  say "vgclock.js in sync" "(skipped: only one copy in this checkout)"
+fi
 # generated Caddyfile / mediamtx / copyparty from a dry configure run must at least be produced + parse
 T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "
 import configure as c, yaml

configure.py

@@ -345,10 +345,23 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
 # vanguard:diag
 {diag_host} {{
 	encode zstd gzip
-{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))}	# Kibana IS this host: logs (Discover), dashboards and alerting in one app, served at the root so there
+{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))}	# Triage lives on this host, alongside the dashboards it belongs with. Both handles come BEFORE the
-	# is no path prefix to get wrong and no index page of links. Elasticsearch itself is never published.
+	# catch-all below, and the Kibana proxy is wrapped in its own handle so the three are evaluated in the
-	reverse_proxy 127.0.0.1:5601 {{
+	# order written rather than by Caddy's directive ordering.
-		flush_interval -1
+	handle /stuck/api {{
+		reverse_proxy 127.0.0.1:8080          # dumpd: the list, and retry/bin for the admin groups
+	}}
+	handle_path /stuck* {{
+		root * /srv/stuck
+		try_files {{path}} /index.html
+		file_server                           # stuck-recording triage; this host is already media-admins only
+	}}
+	# Kibana IS the rest of this host: logs (Discover), dashboards and alerting in one app, served at the
+	# root so there is no path prefix to get wrong. Elasticsearch itself is never published.
+	handle {{
+		reverse_proxy 127.0.0.1:5601 {{
+			flush_interval -1
+		}}
 	}}
 }}
 """
@@ -390,7 +403,14 @@ groups: {{http.request.header.X-Authentik-Groups}}"
 		reverse_proxy 127.0.0.1:8889
 	}}
 	handle_path /hls/* {{
-		reverse_proxy 127.0.0.1:8888
+		# handle_path strips /hls before proxying, so MediaMTX's own redirects come back root-relative and
+		# lose the prefix. It redirects every playlist request once, to ?cookieCheck=1, so without this the
+		# browser follows Location: /<user>/<Title>/index.m3u8 into the SPA catch-all below, receives
+		# index.html, and hls.js reports "manifestParsingError" — a web page where a manifest should be.
+		reverse_proxy 127.0.0.1:8888 {{
+			header_down Location ^/ /hls/
+			flush_interval -1          # low-latency HLS sends partial segments; do not buffer them
+		}}
 	}}
 	handle {{
 		root * /srv/live
@@ -709,23 +729,15 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"])
     run(["sudo", "install", "-d", "/var/lib/vanguard"])                                                  # deadman state
     run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"])            # script counters -> dumpd /metrics
-    # The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in
+    # Everything bin/install.list names. Without this an --apply updated the containers but left STALE
-    # /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md).
+    # binaries in /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time.
-    media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py",
+    for mode, path in bin_manifest():
-                         "bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()]
+        run(["sudo", "install", "-m", mode, path, "/usr/local/bin/"])
-    if media:
-        run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"])
-    if Path("bin/vgmetrics.sh").exists():
-        run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"])                    # sourced by the sweep and vgdeband
-    if Path("bin/vgnotify.sh").exists():
-        run(["sudo", "install", "-m", "0644", "bin/vgnotify.sh", "/usr/local/bin/"])                     # sourced by the alerts timer and the deadman
     if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
-        run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
         run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"])
         run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"])
     # The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord.
     if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")):
-        run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"])
         run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"])
         run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"])
     run(["sudo", "systemctl", "restart", "dumpd"])
@@ -741,6 +753,26 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     print("            run  docker compose restart mediamtx  between streams (cuts a live recording). Adding/removing")
     print("            streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.")
 
+def bin_manifest():
+    """(mode, path) for everything that belongs in /usr/local/bin, from bin/install.list.
+
+    One list, shared with install.sh and deploy.sh. Three hand-kept copies had already drifted: a
+    push-to-deploy was installing five of the ten, so vgstuck.sh, both sourced helpers and both timer
+    scripts never reached the host unless somebody ran --apply by hand.
+    """
+    out = []
+    try:
+        for line in Path("bin/install.list").read_text().splitlines():
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            mode, path = line.split(None, 1)
+            if Path(path).exists():
+                out.append((mode, path))
+    except FileNotFoundError:
+        pass
+    return out
+
 def compose_cmd():
     """docker compose, with sudo only if this user cannot reach the daemon."""
     if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0:

deploy.sh

@@ -46,20 +46,31 @@ apply_changes() {
   if [[ -x check.sh ]]; then asowner bash check.sh >/tmp/vg-check.log 2>&1 || { log "check.sh FAILED — aborting"; tail -20 /tmp/vg-check.log; notify "❌ deploy $new: check.sh failed"; return 1; }; fi
 
   # 1. host scripts + units
-  if has '^bin/'; then run install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/; fi
+  # bin/install.list, not a second hand-kept list: this one had drifted to five of ten files, so a push
+  # never installed vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script.
+  if has '^bin/'; then
+    while read -r mode path; do
+      [[ -f $path ]] && run install -m "$mode" "$path" /usr/local/bin/
+    done < <(grep -vE '^[[:space:]]*(#|$)' bin/install.list)
+  fi
   if has '^systemd/'; then run install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/; run systemctl daemon-reload; fi
   if has '^deploy\.sh$'; then run install -m 0755 deploy.sh /usr/local/bin/vanguard-deploy; fi
   if has '^deployBotBundle\.sh$'; then run install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot; fi
   if has '^firewall/nftables\.conf$'; then run install -m 0644 firewall/nftables.conf /etc/nftables.conf; run nft -f /etc/nftables.conf; fi
 
   # 2. regenerate configs (Caddy reload, copyparty restart) — configure.py decides; MediaMTX is handled below
-  if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/'; then
+  if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/|^stuck/|^observability/kibana-setup\.py$'; then
     local before_mtx; before_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
     if [[ $DRY == 1 ]]; then log "\$ (as $OWNER) VG_DRY=1 python3 configure.py --apply"; else sudo -u "$OWNER" -H python3 configure.py --apply >/tmp/vg-configure.log 2>&1 || { log "configure --apply FAILED"; tail -20 /tmp/vg-configure.log; return 1; }; fi
     local after_mtx; after_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
     [[ $before_mtx == "$after_mtx" ]] || { log "mediamtx.yml changed"; restart_mediamtx_or_defer; }
   fi
   if has '^bin/dumpd\.py$|^systemd/dumpd\.service$'; then run systemctl restart dumpd; fi
+  # Beat and Kibana configs are bind-mounted, so a content change needs the container recreated. Nothing
+  # handled this before: a push that changed filebeat.yml silently did nothing.
+  if has '^observability/filebeat\.yml$'; then run docker compose up -d --force-recreate filebeat; fi
+  if has '^observability/metricbeat\.yml$'; then run docker compose up -d --force-recreate metricbeat; fi
+  if has '^observability/kibana\.yml$'; then run docker compose up -d --force-recreate kibana; fi
 
   # 3. compose: recreate changed services, but never MediaMTX while live
   if has '^docker-compose\.yml$|^bot/Dockerfile$|^bot/entrypoint\.sh$'; then

docker-compose.yml

@@ -72,6 +72,7 @@ services:
       - ./dock:/srv/dock:ro
       - ./clips:/srv/clips:ro               # clip-intake self-service page
       - ./about:/srv/about:ro               # gateway page (/about)
+      - ./stuck:/srv/stuck:ro               # stuck-recording triage page (main host /stuck)
       - /srv/live:/srv/live:ro              # live portal static build (deployLive.sh)
       - caddy_data:/data
       - caddy_config:/config

install.sh

@@ -56,9 +56,11 @@ for d in /srv/media/clips/*/; do
 done
 
 echo "== host scripts + units"
-install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
+# bin/install.list is the single list, shared with configure.py --apply and deploy.sh. Three hand-kept
-install -m 0644 bin/vgmetrics.sh /usr/local/bin/      # sourced by the sweep/vgdeband; counters for dumpd's /metrics
+# copies had drifted, so a push-to-deploy was installing half of them.
-install -m 0644 bin/vgnotify.sh /usr/local/bin/       # sourced by the alerts timer + deadman; Discord posts and mentions
+grep -vE '^\s*(#|$)' bin/install.list | while read -r mode path; do
+  install -m "$mode" "$path" /usr/local/bin/
+done
 install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot   # dumpd runs this for the bot CI's POST /deploy-bot
 install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/      # watches the observability stack from outside it
 install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/        # takes Kibana's alert documents to Discord

live/src/broadcast.js

@@ -34,7 +34,11 @@ export async function captureWindow() {
   }
 }
 
-export const MIN_KBPS = 6000
+// A FLOOR, not a target. It was 6000, which told Chrome never to encode below 6 Mbps: on an uplink that
+// could not sustain that, Chrome kept sending anyway, the network discarded the excess, and MediaMTX logged
+// "invalid FU-A packet (non-starting)" while every viewer saw blocky corruption. A floor this low still
+// skips the painfully slow cold ramp without disabling congestion control.
+export const MIN_KBPS = 1200
 export const START_KBPS = 9000
 
 function preferCodecs(pc, kind = 'video') {
@@ -54,7 +58,8 @@ function preferCodecs(pc, kind = 'video') {
   tx.setCodecPreferences([...high, ...base, ...vp9, ...rest])
 }
 
-// Chrome honours x-google-* fmtp hints: skip the slow bandwidth ramp and never sit below MIN_KBPS.
+// Chrome honours x-google-* fmtp hints. start-bitrate skips the slow cold ramp, max-bitrate is the ceiling,
+// and min-bitrate is a floor that must stay LOW: a high floor defeats congestion control entirely.
 function mungeBitrates(sdp) {
   return sdp.replace(/^a=fmtp:(\d+) (.*)$/gm, (line, pt, params) =>
     /packetization-mode|profile-id|VP9/i.test(params) || /H264|VP9/i.test(sdp.match(new RegExp(`a=rtpmap:${pt} .*`))?.[0] || '')
@@ -95,9 +100,10 @@ export async function publish({ stream, path, user, pass, onState }) {
   const vs = pc.getSenders().find(s => s.track?.kind === 'video')
   if (vs) {
     const p = vs.getParameters(); p.encodings = p.encodings?.length ? p.encodings : [{}]
-    p.encodings[0].maxBitrate = MAX_KBPS * 1000
+    p.encodings[0].maxBitrate = MAX_KBPS * 1000       // a ceiling; congestion control may go far below it
     p.encodings[0].maxFramerate = 60
-    p.encodings[0].scaleResolutionDownBy = 1
+    // No scaleResolutionDownBy: pinning it to 1 forbids the encoder from shedding resolution under
+    // pressure, which leaves it nothing to give but packets it cannot deliver.
     p.degradationPreference = 'balanced'
     await vs.setParameters(p).catch(() => {})
   }

observability/kibana-setup.py

@@ -4,6 +4,9 @@
   connector  "Vanguard alert index"        — the .index connector, writing into the vanguard-alerts index
   rule       "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc
   data view  "Vanguard logs + metrics"     — filebeat-*,metricbeat-* so Discover and Lens have one source
+  landing    "Vanguard ops" dashboard      — our own links, and defaultRoute so diag.BASE opens on it
+
+Run one step on its own by naming it:  kibana-setup.py landing   (also: connector, rule, dataview, all)
 
 WHY AN INDEX AND NOT A WEBHOOK
 On a basic license the only connector types Kibana allows are .index and .server-log; every connector that
@@ -38,11 +41,25 @@ delivery script recognises the heartbeat and keeps it out of Discord. Use "On st
 for real alarms; only the heartbeat wants to fire every interval.
 """
 import json, os, time, urllib.error, urllib.request
+from pathlib import Path          # env_hosts() reads the stack's .env
+
+import sys
+
+STEPS = ("connector", "rule", "dataview", "landing")
+WANTED = sys.argv[1] if len(sys.argv) > 1 else "all"
+if WANTED not in STEPS + ("all",):
+    raise SystemExit(f"unknown step {WANTED!r}; pick one of: {', '.join(STEPS)}, all")
+def wanted(step):
+    return WANTED in ("all", step)
 
 KIBANA = "http://127.0.0.1:5601"
 ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts")
 HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
-           "Authorization": "ApiKey " + os.environ["KEY"]}
+           "Authorization": "ApiKey " + os.environ["KEY"],
+           # Kibana 9 ships with server.restrictInternalApis on. /api/kibana/settings (defaultRoute) is an
+           # internal API and refuses without this header: "exists but is not available with the current
+           # configuration". Sending it on every call is harmless; loosening the restriction is not.
+           "x-elastic-internal-origin": "Kibana"}
 
 def wait_for_kibana(seconds=300):
     """Kibana has to be *available*, not merely listening.
@@ -87,26 +104,32 @@ wait_for_kibana()
 
 # 1. the alert bus connector
 CONNECTOR_NAME = "Vanguard alert index"
-existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+connector_id = None
-if CONNECTOR_NAME in existing:
+if wanted("connector") or wanted("rule"):
+  existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+  if CONNECTOR_NAME in existing:
     connector_id = existing[CONNECTOR_NAME]
     print("connector already present:", connector_id)
-else:
+  elif not wanted("connector"):
-    connector_id = req("POST", "/api/actions/connector", {
+    raise SystemExit(f"connector {CONNECTOR_NAME!r} does not exist yet — run the connector step first")
+  else:
+      connector_id = req("POST", "/api/actions/connector", {
         "name": CONNECTOR_NAME,
         "connector_type_id": ".index",
         # executionTimeField stamps every document, so a rule body never has to get the timestamp right.
         "config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"},
         "secrets": {}})["id"]
-    print("connector created:", connector_id, "->", ALERT_INDEX)
+      print("connector created:", connector_id, "->", ALERT_INDEX)
 
 # 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked
 #    harmless and meant the data view was never created on any system where the rule already existed, which
 #    is every re-run.
 RULE_NAME = "Vanguard alerting heartbeat"
-existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
+existing_rule = None if not wanted("rule") else next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
                       if r["name"] == RULE_NAME), None)
-if existing_rule:
+if not wanted("rule"):
+    pass
+elif existing_rule:
     print("rule already present:", existing_rule["id"],
           "enabled" if existing_rule.get("enabled") else "DISABLED")
 else:
@@ -152,10 +175,111 @@ else:
 #    detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and
 #    metrics, which is the whole reason they share a cluster.
 VIEW_TITLE = "filebeat-*,metricbeat-*"
-views = req("GET", "/api/data_views").get("data_view", [])
+views = req("GET", "/api/data_views").get("data_view", []) if wanted("dataview") else []
-if any(v.get("title") == VIEW_TITLE for v in views):
+if not wanted("dataview"):
+    pass
+elif any(v.get("title") == VIEW_TITLE for v in views):
     print("data view already present:", VIEW_TITLE)
 else:
     view = req("POST", "/api/data_views/data_view", {
         "data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}})
     print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE)
+
+
+# 4. the landing page. Kibana stays at the root of the diag host; this just makes arriving there open OUR
+#    dashboard instead of Kibana's home. A markdown panel holds the links, and `defaultRoute` is the
+#    advanced setting that does the redirect. That is the whole trick, and it avoids moving Kibana under a
+#    /kibana prefix — which would have meant touching both Beats' setup.kibana, this script, the alarm
+#    script and the deadman's status check, with a silent crash-loop waiting for whichever one was missed.
+DASHBOARD_ID = "vanguard-ops"
+
+def env_hosts():
+    """Hostnames out of the stack's .env, so the links are real rather than examples."""
+    values = {}
+    for candidate in (Path(".env"), Path("/home/debian/vanguard-stream/.env")):
+        if candidate.is_file():
+            for line in candidate.read_text().splitlines():
+                if "=" in line and not line.startswith("#"):
+                    key, value = line.split("=", 1)
+                    values[key.strip()] = value.strip()
+            break
+    return values
+
+def landing_markdown(hosts):
+    def url(key, path="/"):
+        host = hosts.get(key)
+        return f"https://{host}{path}" if host else None
+    rows = [
+        ("Stuck recordings", "/stuck", "finished but never published — why, and retry or bin"),
+        ("Live portal", url("LIVE_HOST"), "watch every active stream"),
+        ("OBS dock", url("LIVE_HOST", "/dock/"), "go live, titles, dump tonight"),
+        ("Files", url("FILES_HOST"), "browse, download, upload finals"),
+        ("Jellyfin", url("WATCH_HOST"), "recordings by streamer and season"),
+        ("Gateway", url("MAIN_HOST"), "role-aware links and how-tos"),
+        ("Gitea", url("GIT_HOST"), "code and CI"),
+        ("Bot", url("BOT_HOST"), "the Discord bot's own pages"),
+        ("Syncthing", url("SYNC_HOST"), "clip intake, device pairing"),
+        ("Authentik", url("AUTH_HOST", "/if/admin/"), "users, groups, policies"),
+    ]
+    lines = ["### Vanguard", "", "| | |", "|---|---|"]
+    for name, link, why in rows:
+        if link:
+            lines.append(f"| **[{name}]({link})** | {why} |")
+    lines += [
+        "", "### In here", "",
+        "| | |", "|---|---|",
+        "| **[Discover](/app/discover)** | search every log line; filter on `service.name` or `vanguard.category` |",
+        "| **[Dashboards](/app/dashboards)** | the Metricbeat system and Docker dashboards |",
+        "| **[Stack Rules](/app/management/insightsAndAlerting/triggersActions/rules)** | the six alarms, and whether each is firing |",
+        "| **[Index Management](/app/management/data/index_management/indices)** | index sizes and ILM |",
+        "",
+        "_Alarms post to Discord through the `vanguard-alerts` index, not from Kibana directly: a basic "
+        "licence has no webhook connector. The host deadman watches that whole path from outside._",
+    ]
+    return "\n".join(lines)
+
+if wanted("landing"):
+    hosts = env_hosts()
+    panel = {
+        "type": "visualization",
+        "panelIndex": "1",
+        "gridData": {"x": 0, "y": 0, "w": 24, "h": 18, "i": "1"},
+        "embeddableConfig": {
+            "savedVis": {
+                "id": "", "title": "", "description": "", "type": "markdown",
+                "params": {"fontSize": 12, "openLinksInNewTab": False,
+                           "markdown": landing_markdown(hosts)},
+                "uiState": {},
+                "data": {"aggs": [], "searchSource": {"query": {"query": "", "language": "kuery"},
+                                                      "filter": []}},
+            },
+            "enhancements": {},
+        },
+    }
+    saved = req("POST", f"/api/saved_objects/dashboard/{DASHBOARD_ID}?overwrite=true", {
+        "attributes": {
+            "title": "Vanguard ops",
+            "description": "Landing page for the diagnostics host. Edit freely: add metric panels beside the links.",
+            "panelsJSON": json.dumps([panel]),
+            "optionsJSON": json.dumps({"hidePanelTitles": True, "useMargins": True,
+                                       "syncColors": False, "syncCursor": True, "syncTooltips": False}),
+            "timeRestore": False,
+            "kibanaSavedObjectMeta": {
+                "searchSourceJSON": json.dumps({"query": {"query": "", "language": "kuery"}, "filter": []})},
+        }})
+    # Prove it rather than assume it: read the dashboard back and count the panels Kibana actually stored.
+    stored = req("GET", f"/api/saved_objects/dashboard/{DASHBOARD_ID}")
+    panels = json.loads(stored.get("attributes", {}).get("panelsJSON", "[]"))
+    links = sum(1 for line in landing_markdown(hosts).splitlines() if line.startswith("| **["))
+    print(f"dashboard: {stored.get('id')} | title {stored.get('attributes', {}).get('title')!r} | "
+          f"{len(panels)} panel(s) stored | {links} links in the markdown")
+    if not panels:
+        raise SystemExit("  the dashboard saved but with no panels — Kibana rejected the panel shape")
+    print("open:", f"/app/dashboards#/view/{DASHBOARD_ID}")
+
+    route = f"/app/dashboards#/view/{DASHBOARD_ID}"
+    req("POST", "/api/kibana/settings", {"changes": {"defaultRoute": route}})
+    now = req("GET", "/api/kibana/settings").get("settings", {}).get("defaultRoute", {}).get("userValue")
+    print("defaultRoute:", now or "(not set!)")
+    if now != route:
+        raise SystemExit("  Kibana did not keep that defaultRoute — set it by hand in Advanced Settings")

stuck/index.html

@@ -0,0 +1,113 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Stuck recordings — Vanguard</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#3d8bfd;--ok:#2ecc71;--warn:#e3a008;--bad:#e05260;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
+  main{max-width:1040px;margin:0 auto;padding:28px 16px 48px}
+  header{display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;margin-bottom:6px}
+  h1{font-size:26px;margin:0}.who{color:var(--mut)}.who b{color:var(--fg)}
+  p.lead{color:var(--mut);margin:4px 0 20px}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:16px 18px;margin-bottom:14px}
+  .row{display:grid;grid-template-columns:96px 1fr auto;gap:12px;align-items:start}
+  .state{font-size:12px;border-radius:999px;padding:2px 10px;text-align:center;align-self:start;white-space:nowrap}
+  .parked{background:#43242a;color:var(--bad)}.published{background:#3a3220;color:var(--warn)}
+  .waiting{background:#1e3147;color:var(--acc)}.writing{background:#1f3b2a;color:var(--ok)}
+  .name{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .meta{color:var(--mut);font-size:13px;margin-top:2px}
+  .why{color:var(--mut);font-size:13px;margin-top:6px;border-left:2px solid var(--line);padding-left:10px;white-space:pre-wrap}
+  .acts{display:flex;gap:8px}
+  button{font:inherit;font-size:13px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:6px 12px;cursor:pointer}
+  button:hover:not(:disabled){border-color:var(--acc)}button:disabled{opacity:.45;cursor:default}
+  button.danger:hover:not(:disabled){border-color:var(--bad);color:var(--bad)}
+  .note{color:var(--mut);font-size:13px}.err{color:var(--bad)}
+  pre{background:#121212;border-radius:8px;padding:10px;overflow:auto;font-size:12px;margin:8px 0 0}
+  footer{color:var(--mut);font-size:13px;margin-top:20px}
+</style>
+</head>
+<body>
+<main>
+  <header><h1>Stuck recordings</h1><div class="who" id="who"></div></header>
+  <p class="lead">Recordings that finished but never reached Jellyfin or copyparty. Retry puts one back in the
+  queue; bin moves it to the trash, recoverable for three days just like a dumped night.</p>
+  <div id="list"><section class="note">Loading…</section></div>
+  <footer>Refreshes every 30&nbsp;seconds. Trends and history live on the diagnostics dashboard.</footer>
+</main>
+<script>
+const API = '/stuck/api';
+let admin = false;
+
+const hms = s => { s = Math.max(0, s|0); const h = s/3600|0, m = (s%3600)/60|0; return h ? `${h}h ${m}m` : `${m}m`; };
+const dur = s => { s = Math.max(0, s|0); return `${s/3600|0}:${String((s%3600)/60|0).padStart(2,'0')}:${String(s%60).padStart(2,'0')}`; };
+const short = p => p.replace(/^.*\/streams\//, '').replace('/.live/', ' · ');
+
+async function load() {
+  const list = document.getElementById('list');
+  let data;
+  try {
+    const r = await fetch(API, { credentials: 'same-origin' });
+    if (r.status === 401) { list.innerHTML = '<section class="note err">Not signed in. Reload the page.</section>'; return; }
+    data = await r.json();
+  } catch (e) {
+    list.innerHTML = `<section class="note err">Could not reach the server: ${e.message}</section>`; return;
+  }
+  admin = !!data.admin;
+  document.getElementById('who').innerHTML = `signed in as <b>${data.user || '?'}</b>${admin ? '' : ' · read only'}`;
+
+  const items = (data.items || []).sort((a, b) =>
+    (a.state === b.state ? b.age_s - a.age_s : a.state.localeCompare(b.state)));
+  if (!items.length) { list.innerHTML = '<section class="note">Nothing is stuck. Everything recorded has published.</section>'; return; }
+
+  list.innerHTML = '';
+  for (const it of items) {
+    const el = document.createElement('section');
+    // "writing" means it is still being recorded: acting on it would cut a live recording, so no buttons.
+    const actionable = admin && it.state !== 'writing';
+    el.innerHTML = `
+      <div class="row">
+        <div class="state ${it.state}">${it.state}</div>
+        <div>
+          <div class="name">${short(it.src)}</div>
+          <div class="meta">waiting ${hms(it.age_s)} · ${dur(it.duration_s)} long · ${(it.bytes/1e6).toFixed(0)} MB</div>
+          <div class="why">${it.reason || ''}</div>
+        </div>
+        <div class="acts">
+          <button data-act="retry" ${actionable ? '' : 'disabled'}>Retry</button>
+          <button data-act="drop" class="danger" ${actionable ? '' : 'disabled'}>Bin</button>
+        </div>
+      </div>`;
+    for (const b of el.querySelectorAll('button')) {
+      b.onclick = () => act(it, b.dataset.act, el);
+    }
+    list.appendChild(el);
+  }
+}
+
+async function act(item, action, el) {
+  if (action === 'drop' && !confirm(`Move this recording to the trash?\n\n${short(item.src)}\n\nRecoverable for three days.`)) return;
+  for (const b of el.querySelectorAll('button')) b.disabled = true;
+  const out = document.createElement('pre');
+  out.textContent = action === 'retry' ? 'Clearing the blocker…' : 'Moving to trash…';
+  el.appendChild(out);
+  try {
+    const r = await fetch(API, {
+      method: 'POST', credentials: 'same-origin',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ action, src: item.src }),
+    });
+    const body = await r.json();
+    out.textContent = body.error ? `Failed: ${body.error}` : (body.output || []).join('\n') || 'Done.';
+    if (!body.error) setTimeout(load, 1200);
+  } catch (e) {
+    out.textContent = `Failed: ${e.message}`;
+  }
+}
+
+load();
+setInterval(load, 30000);
+</script>
+</body>
+</html>