Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms

Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

STATUS.md

@@ -60,9 +60,11 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
 | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
 | Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
+| diag landing | `diag.…` opens on a "Vanguard ops" dashboard: a markdown panel of links to every host plus the stuck page, and shortcuts to Discover, Dashboards, Stack Rules and Index Management. Kibana still owns the root; `defaultRoute` just redirects there. Created by `kibana-setup.py landing`, so a rebuild recreates it | **added 2026-11-04** |
 | Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
 | Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
 | Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
+| Stuck recordings | A page at `https://diag.…/stuck` for anyone past the gate, on the triage host (already `media-admins` only), linked from the gateway's Media admin card and from the diag landing dashboard: every unpublished recording, why it is stuck, and Retry / Bin buttons. dumpd serves `/stuck/api` and allows the actions only for the admin groups; both the page and the CLI run `bin/vgstuck.sh`, so they can never disagree. `bin/vgstuck.sh` triages anything that never published: it says why each one is stuck (`parked` by a failure log with ffmpeg's reason, `published` collision, `waiting`, or still `writing`), then `retry <match>` clears the blocker and `drop <match>` moves it to trash with the usual 3-day undo. `drop --all` is refused. The sweep also logs `event=parked` once per run so Kibana can name them, not just count them | **added 2026-11-04** |
 | Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
 | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
@@ -117,6 +119,9 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
 - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
 - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
+- **A high `x-google-min-bitrate` disables WebRTC congestion control.** Go Live sent `min-bitrate=6000`, so Chrome never encoded below 6 Mbps; on an uplink that could not carry it, the excess was dropped, MediaMTX logged `invalid FU-A packet (non-starting)` and every viewer saw blocky corruption. The loss was on the PUBLISH session (4761 packets for one streamer) while the viewer sessions lost nothing, which is the fingerprint: the server relays damage it received. Floor lowered to 1200 and `scaleResolutionDownBy` unpinned so the encoder can shed resolution. Found 2026-11-04.
+- `tracks=['Opus','H264']` on a path means a browser (Go Live); `MPEG-4 Audio` means OBS. Quick way to tell who is publishing how.
+- **`handle_path` strips the prefix, so an upstream's own redirects lose it.** MediaMTX redirects every HLS playlist request once (to `?cookieCheck=1`) with a root-relative `Location`, so the browser followed `/<user>/<Title>/index.m3u8` into the portal's SPA catch-all, got `index.html`, and hls.js said `manifestParsingError` — a web page where a manifest should be. Fixed with `header_down Location ^/ /hls/`. `curl` against MediaMTX directly hides this completely, because there is no prefix to lose. Found 2026-11-04.
 - **Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked** in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a `tmcd` data stream is the fingerprint. First working Windows audio: 2026-11-03.
 - The sweep's band-removal branch **overwrote `$reason`**, which carried the probe's `audio=<codec>` marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got `-c:a copy` and shipped Opus inside an MP4. Fixed by keeping `probe_reason` separately; it only became reachable once Go Live audio worked.
 - **Authentik has two invalidation flows and the names are a trap.** `default-provider-invalidation-flow` is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so `/logout` drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. `default-invalidation-flow` ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03.