Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms

Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

README.md

@@ -69,7 +69,8 @@ works from ~/vanguard-stream without sudo.
     ./configure.py --full      # first time: asks everything, then applies
     ./configure.py             # later: add/remove/reset streamers and editors, then apply
     ./configure.py --show      # print current users and the OBS settings to hand out
-    ./configure.py --set K=V   # change one setting (Discord tokens/webhooks, retention) without a full pass
+    ./configure.py --set K=V   # change one setting (Discord tokens/webhooks, retention, alarm mentions)
+                               # run it with no arguments to list what is settable
 
 Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates
 `Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates
@@ -437,8 +438,27 @@ Jellyfin and Gitea use OIDC providers instead, not this.
 
 ## 5h. Diagnostics: Kibana at `diag.HOST`
 
-Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover,
-dashboards, and alerting, with no sub-paths and no index page of links.
+Four containers, one store, one UI. Kibana is served at the **root** of `diag.HOST`, with no sub-path to get
+wrong. Arriving there opens a dashboard of ours, "Vanguard ops", via Kibana's `defaultRoute` setting: links to
+every host and to the triage page, plus shortcuts into Discover, Dashboards, Stack Rules and Index Management.
+It is an ordinary dashboard, so add metric panels beside the links (`vanguard_media_pending` first).
+
+Two paths on that host are ours rather than Kibana's:
+
+| path | what |
+|---|---|
+| `/stuck` | **stuck recordings**: everything that finished but never published, why each one is stuck, and Retry / Bin buttons. The list is readable by anyone the gate allows; the actions are enforced in dumpd for the admin groups, not hidden in the page. Binning moves the file to `trash/<today>/`, the same 3-day undo as a dumped night |
+| `/stuck/api` | dumpd serves it. Both the page and `vgstuck.sh` on the host run the same script, so they cannot disagree about why something is stuck |
+
+Kibana deliberately does **not** live under `/kibana`. That would need `server.basePath` plus
+`rewriteBasePath`, a Caddy handle that does not strip the prefix, and the prefix appended in both Beats'
+`setup.kibana`, `kibana-setup.py`, `kibana-alarms.py` and the deadman's status check — and missing the Beats
+one crash-loops Metricbeat, which exits when Kibana is unreachable. `defaultRoute` buys the landing page
+without any of it.
+
+`observability/kibana-setup.py` creates the first-run objects and takes a step name, so you can redo one:
+`connector`, `rule`, `dataview`, `landing`, or all of them when omitted. `observability/kibana-alarms.py`
+creates the six alarm rules.
 
 | container | job |
 |---|---|
@@ -573,6 +593,13 @@ Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
 
 ## Deployment (push to deploy — built, OFF by default)
 
+What lands in `/usr/local/bin` is listed once, in `bin/install.list`, and read by `install.sh`,
+`configure.py --apply` and `deploy.sh`. It used to be three hand-kept lists and they drifted: a push was
+installing five of ten scripts, so `vgstuck.sh`, both sourced helpers and both timer scripts only reached the
+host if somebody ran `--apply` by hand. `check.sh` now fails if the list names a missing file, or if
+something in `bin/` is neither listed nor explicitly excluded (`mtx-on-ready.sh` is mounted into the MediaMTX
+container; `vgbot-merge-sqlite.py` is a one-off migration).
+
 The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea →
 `./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token
 secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout.
@@ -715,4 +742,6 @@ evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
 |------|------|------|
 | normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
 | trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
-| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time and on `clips.` at /me/clips |
+| vanguard-alerts.timer | every 1 min | delivers Kibana's alert documents to Discord, with a 15-min per-rule repeat guard |
+| vanguard-deadman.timer | every 5 min | watches the alerting stack from outside it, and the disks |
+| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time, on `clips.` at /me/clips and on `diag.` at /stuck/api |