Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms

Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

bin/dumpd.py

@@ -21,6 +21,10 @@ POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a D
                    HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
                    May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
                    into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body.
+GET  /stuck/api  -> every unpublished recording and why it is stuck (any gated user)
+POST /stuck/api  -> {"action":"retry"|"drop","src":...}: clear the blocker, or move it to trash with the
+             usual 3-day undo. Admin groups only; both actions run bin/vgstuck.sh so the web page and the
+             CLI share one implementation.
 GET  /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live
              gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here.
 POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid
@@ -425,6 +429,43 @@ def verify_token(token):
         return None
     return user.lower()
 
+# ---- stuck recordings -----------------------------------------------------------------------------
+# GET  /stuck/api   every unpublished recording and why, for anyone the gate let through
+# POST /stuck/api   {"action":"retry"|"drop","src":"<path>"} — admins only
+# The classification and the file moves both live in bin/vgstuck.sh, so the web page and the CLI can never
+# disagree about why something is stuck, and "delete" means the same three-day trash in both.
+STUCK_TOOL = os.environ.get("VGSTUCK_TOOL", "/usr/local/bin/vgstuck.sh")
+
+def stuck_list():
+    done = subprocess.run([STUCK_TOOL, "json"], capture_output=True, text=True, timeout=180)
+    try:
+        return json.loads(done.stdout or "[]")
+    except ValueError:
+        log("stuck_list_failed", level="error", rc=done.returncode, err=(done.stderr or "")[:200])
+        return []
+
+def stuck_target(src):
+    """Resolve what the browser asked us to act on, or raise. This is the only thing between a request body
+    and a file move, so it is deliberately strict: inside the streams tree, an unpublished .live recording,
+    and really there."""
+    path = Path(src).resolve()
+    root = (ROOT / "streams").resolve()
+    if root != path and root not in path.parents:
+        raise ValueError("that path is not inside the streams tree")
+    if f"{os.sep}.live{os.sep}" not in str(path) or path.suffix.lower() != ".mp4":
+        raise ValueError("that is not an unpublished recording")
+    if not path.is_file():
+        raise ValueError("no such recording (already published or already binned?)")
+    return path
+
+def stuck_action(action, src, who):
+    path = stuck_target(src)
+    done = subprocess.run([STUCK_TOOL, action, str(path)], capture_output=True, text=True, timeout=600)
+    lines = [l for l in (done.stdout + done.stderr).splitlines() if l.strip()]
+    log(f"stuck_{action}", level="info" if done.returncode == 0 else "error",
+        file=str(path), by=who, rc=done.returncode)
+    return {"ok": done.returncode == 0, "rc": done.returncode, "output": lines[-8:]}
+
 # ---- GET /metrics -----------------------------------------------------------------------------------
 # Prometheus text for Metricbeat's prometheus module. Two sources are merged:
 #   1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/<component>.counters)
@@ -619,6 +660,10 @@ class H(BaseHTTPRequestHandler):
             return self._json(200, {"user": who, "groups": groups, "admin": is_admin,
                                     "name": self.headers.get("X-Authentik-Name") or who, "email": self.headers.get("X-Authentik-Email") or "",
                                     "discord_id": proxy_claim(self.headers, "discord_id")})
+        if self.path == "/stuck/api":
+            who, is_admin = identity(self.headers)
+            if not who: return self._json(401, {"error": "not authenticated"})
+            return self._json(200, {"admin": is_admin, "user": who, "items": stuck_list()})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
             if not who: return self._json(401, {"error": "not authenticated"})
@@ -747,6 +792,24 @@ class H(BaseHTTPRequestHandler):
                 return self._json(401, {"error": "path must be <your-username>/<Title>"})
             log("publish_allow", user=user, stream=path, ip=req.get("ip"))
             return self._json(200, {"ok": True})
+        if self.path == "/stuck/api":
+            who, is_admin = identity(self.headers)
+            if not who: return self._deny()
+            if not is_admin:
+                return self._json(403, {"error": f"retrying and binning recordings is for {', '.join(sorted(ADMIN_GROUPS))}"})
+            try:
+                body = json.loads(self.rfile.read(int(self.headers.get("Content-Length") or 0)) or b"{}")
+            except ValueError as error:
+                return self._json(400, {"error": str(error)})
+            action = body.get("action")
+            if action not in ("retry", "drop"):
+                return self._json(400, {"error": "action must be retry or drop"})
+            try:
+                return self._json(200, stuck_action(action, str(body.get("src", "")), who))
+            except ValueError as error:
+                return self._json(400, {"error": str(error)})
+            except Exception as error:
+                return self._json(500, {"error": str(error)})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
             if not who: return self._json(401, {"error": "not authenticated"})