bunch of shit

Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

HOSTNAME-CHANGE.md

@@ -5,9 +5,27 @@ domain. Both are mostly `configure.py --full` plus a known list of places outsid
 Do it between streams: the main host is MediaMTX's WebRTC advertised host, so a main-host change needs a
 MediaMTX restart.
 
-Roles: `main` (gateway page at `/`), `files` (copyparty), `clips`, `sync`, `live` (portal + `/dock/`, `/dock/clock/`,
+Roles: `main` (gateway page at `/`, `/onboarding/`), `files` (copyparty), `clips`, `sync`, `live` (portal + Go Live,
-`/dump`, `/me`, `/time`), `auth`, `watch`, `git`, `bot`. Ours: `media.`, `files.`, `clips.`, `sync.`, `live.`, … under
+`/dock/`, `/dock/clock/`, `/dump`, `/me`, `/time`, `/whoami`), `auth`, `watch`, `git`, `bot`, `diag` (Kibana at `/`,
-`dumpstervigilantes.rip`.
+`/streaming`, `/disk`, dumpd's `/streaming/api/*`), and optionally `apex` (the bare domain, a redirect to
+`APEX_REDIRECT`; it needs its own A record at the box or Caddy cannot get its certificate — as of 2026-11-06 the bare
+`dumpstervigilantes.rip` resolves elsewhere, so leave `APEX_HOST` empty or fix the record first). Ours: `media.`,
+`files.`, `clips.`, `sync.`, `live.`, `diag.`, … under `dumpstervigilantes.rip`. `ovh1.BASE` is ssh only.
+
+## The one rule: apps before Authentik
+
+Change the app, then Authentik, never the other way round. An app that already lives on its new name but whose
+Authentik provider still says the old one is simply unreachable until Authentik is updated; a provider pointed at
+a name the app does not answer on yet locks you out of the app, and if that app is the gate in front of the admin
+UI, out of everything. So in every procedure below Authentik comes after the servers, the containers, Jellyfin and
+Gitea — and the Discord developer portal after Authentik. The safety net at every step is akadmin's password
+card, `https://AUTH/if/flow/default-authentication-flow/?admin=1`, which bypasses the Discord forward.
+
+Note that `configure.py --apply` does touch Authentik on its own: the login redirect stage's target (the Discord
+source's login URL on the auth host) and the brand's default application. Both are Authentik-internal, neither
+decides whether an app is reachable, and the admin card ignores the redirect stage — so they cannot lock you out.
+The things that can — outpost host, proxy provider external host and cookie domain, Jellyfin/Gitea redirect
+URIs — are never changed by any script here.
 
 ## Where hostnames live
 
@@ -15,16 +33,19 @@ Roles: `main` (gateway page at `/`), `files` (copyparty), `clips`, `sync`, `live
 | Place | Holds |
 |---|---|
 | `Caddyfile` | every site block — the source of truth `configure.py` reads back (`# vanguard:<role>` markers) |
-| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `DIAG_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT` |
+| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `DIAG_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT`; during a move `ALT_BASE` (every site also answers under it) and `REDIRECT_BASE` (old names 308 to the new) |
 | `mediamtx.yml` | `webrtcAdditionalHosts: [MAIN_HOST]` (needs a MediaMTX restart to take effect) |
-| `docker-compose.yml` → container env | Jellyfin `PublishedServerUrl` (WATCH), Gitea `DOMAIN/ROOT_URL/SSH_DOMAIN` (GIT), bot `WEB_PUBLIC_URL` (BOT), MediaMTX `LIVE_HOST` (text of the "is live" post), act_runner `GITEA_INSTANCE_URL` (GIT). `compose up -d` in the apply recreates the affected containers (seconds each) |
+| `docker-compose.yml` → container env | Jellyfin `PublishedServerUrl` (WATCH), Gitea `DOMAIN/ROOT_URL/SSH_DOMAIN` (GIT), bot `WEB_PUBLIC_URL` (BOT), MediaMTX `LIVE_HOST` (text of the "is live" post), act_runner `GITEA_INSTANCE_URL` (GIT), Kibana `server.publicBaseUrl` via `DIAG_HOST` (DIAG). `compose up -d` in the apply recreates the affected containers (seconds each). MediaMTX is on the host network and is deliberately **not** recreated by the apply |
+| Authentik, by `configure.py --apply` (`--ak-branding` alone) | the login **redirect stage** target `https://AUTH/source/oauth/login/<discord-slug>/` (members never see an Authentik page), the brand's default application (the proxy provider whose external host is MAIN), logo/favicon/background paths (relative, no host). Re-run after an **auth** or **main** change |
+| Kibana landing dashboard, by `configure.py --apply` (`KEY=… kibana-setup.py landing` alone) | the Vanguard tile's links: LIVE, FILES, WATCH, MAIN, GIT, BOT, SYNC from `.env`, AUTH derived as `auth.<domain of MAIN>`; `/streaming`, `/disk` relative. Re-run after any change |
 | Caddy TLS | certificates for new names are fetched automatically; old ones linger harmlessly |
 
 ### Pages in the repo that derive sibling hosts from the browser's location (label names hard-coded)
 | File | Assumes |
 |---|---|
-| `about/index.html` | `files.`, `clips.`, `sync.`, `live.`, `auth.`, `watch.`, `git.`, `bot.` + the parent of the current host |
+| `about/index.html` | `files.`, `clips.`, `sync.`, `live.`, `auth.`, `watch.`, `git.`, `bot.`, `diag.` + the parent of the current host (`sub('<label>')`); `/onboarding/` relative |
-| `dock/index.html` | `auth.` (sign-out), ingest host prefilled from the current host (`live.`; any name of the box works for RTMP/SRT) |
+| `dock/index.html` | `auth.` (sign-out), ingest host prefilled from the current host (`live.`; any name of the box works for RTMP/SRT); `/dump`, `/me/*`, `/time` relative |
+| `live/` portal, `streaming/index.html`, `streaming/disk.html`, `about/onboarding/` | nothing: every call is same-origin (`/webrtc/`, `/hls/`, `/me/*`, `/streaming/api/*`) |
 | `switchToClone.sh`, `deployLive.sh` | defaults for `git.dumpstervigilantes.rip` / `media.dumpstervigilantes.rip` (override by env/arg, or edit) |
 | `README.md`, `FRESH-INSTALL.md` | examples only |
 Renaming a *label* (e.g. `watch` → `video`) means editing the first two; changing only the domain does not.
@@ -49,7 +70,9 @@ Renaming a *label* (e.g. `watch` → `video`) means editing the first two; chang
 | Streamers' OBS | dock URL (Custom Browser Docks), dock Settings → ingest host (re-save), clock browser source URL (dock → Timecode band → "Add band + clock source" again), any manual RTMP/SRT configs (copy from the dock) | live |
 | Jellyfin apps on phones/TVs | server address | watch |
 | Bookmarks, Discord pinned links | main (gateway), clips, files, watch, live | all |
-Not affected: Syncthing pairings (device IDs), MariaDB, stream tokens, the Gitea webhook (gateway IP), Authentik users/groups/policies.
+Not affected: Syncthing pairings (device IDs), MariaDB, stream tokens, the Gitea webhook (gateway IP), Authentik
+users/groups/policies, the alert pipeline (Discord webhook + loopback Elasticsearch), the deadman, the publish block
+list and the WebRTC session history (`.markers/`), MediaMTX's host-network ports.
 
 ## A. Renaming one subdomain (example: `watch.BASE` → `video.BASE`)
 
@@ -58,50 +81,80 @@ Not affected: Syncthing pairings (device IDs), MariaDB, stream tokens, the Gitea
    "Jellyfin hostname" prompt; `y`. Caddy reloads (new block, old gone), Jellyfin is recreated with the new
    published URL. `curl -sI https://video.BASE/ | head -1` → 200/302 once the certificate is issued.
    (Only the *main* host touches `mediamtx.yml`; other hosts need no MediaMTX restart.)
-3. Authentik: Applications → Providers → `Jellyfin` → Edit → redirect URI → the new host → Update.
+3. Jellyfin first: Dashboard → General → Branding → disclaimer form action → new host → Save.
-4. Jellyfin: Dashboard → General → Branding → disclaimer form action → new host → Save.
+   Dashboard → Plugins → Vanguard Metadata → Public URL → new host → Save. (`https://video.BASE/` now loads and
-   Dashboard → Plugins → Vanguard Metadata → Public URL → new host → Save.
+   fails only at the SSO step: expected until step 4.)
+4. Authentik last: Applications → Providers → `Jellyfin` → Edit → redirect URI → the new host → Update.
 5. If the *label* changed (`watch`→`video`), edit `about/index.html` (`sub('watch')` → `sub('video')`) — commit, push,
    `sudo vanguard-deploy pull`. (`.env APEX_REDIRECT` too if the bare domain pointed at it: `./configure.py --full`.)
 6. Private window: `https://video.BASE/` → Discord → in. The gateway `https://HOST/` links to the new name.
 7. Remove the old DNS record. Tell members with Jellyfin apps to update the server address.
 
 Per-host extras when it is a different host:
-- **auth**: Discord portal redirect; Authentik outpost `authentik_host` and the proxy provider's authentication URL;
+- **auth**, in this order: Jellyfin SSO plugin OID endpoint and Vanguard plugin Authentik URL; Gitea auth source
-  Jellyfin SSO plugin OID endpoint; Vanguard plugin Authentik URL; Gitea auth source discovery URL;
+  discovery URL; `dock/index.html` + `about/index.html` if the label changed; THEN Authentik's outpost
-  `dock/index.html` + `about/index.html` if the label changed. Everyone re-logs in.
+  `authentik_host` and the proxy provider's authentication URL (log in there with the admin card); THEN the
+  Discord portal redirect. Everyone re-logs in.
 - **git**: Authentik Gitea provider redirect; `./registerGiteaRunner.sh`; server clone remote + `~/.ssh/config.vanguard`;
   dev remote; `switchToClone.sh` default.
-- **live**, **bot**: only the compose env (automatic) and `about/index.html` if the label changed.
+- **live**, **bot**: only the compose env (automatic) and `about/index.html` if the label changed. Streamers who use
+  Go Live need nothing (same-origin); OBS users re-add the dock URL (the dock lives on `live`).
+- **diag**: compose env → Kibana's `publicBaseUrl` (automatic, Kibana recreated), the Kibana landing tile
+  (automatic in the apply), `DIAG_REQUIRE_GROUP` in `.env` if the gate group should differ, `about/index.html`
+  if the label changed. Alarm rules and the Discord alert bus carry no hostname.
+- **auth**, in addition to the list above: `configure.py --ak-branding` (the apply does it) so the Discord-forward
+  redirect stage points at the new auth host — until then every login lands on the old name; the admin card
+  moves to `https://NEW-AUTH/if/flow/default-authentication-flow/?admin=1` (the Kibana tile link follows).
 - **main (HOST)**: `mediamtx.yml` changes → `docker compose restart mediamtx` between streams; every streamer
   re-adds the dock URL, re-saves Settings (ingest host), and re-runs "Add band + clock source"; manual RTMP/SRT
   configs are re-copied from the dock; `deployLive.sh` default host.
 
 ## B. Changing the base domain (everything at once)
 
-Plan a maintenance window: Caddy, Jellyfin, Gitea, the bot and MediaMTX all restart, and every member logs in again
+No maintenance window: every site answers on **both** bases first (`--alias`), so the old names keep working for
-(the Authentik session cookie is bound to the base domain).
+members while the apps and Authentik are moved one by one; the flip (`--base`) is the last server step and turns the
+old names into redirects. Order follows the rule above: servers, then the apps, then Authentik, then Discord, then
+the flip. `OLD` = the current base, `NEW` = the new one. Members' sessions survive until step 6; from step 6 on
+they log in once on the new names.
 
-1. DNS on the new domain: `A HOST2 → IP`, CNAMEs `files`, `clips`, `sync`, `live`, `auth`, `watch`, `git`, `bot` → `HOST2` (optional bare domain A).
+1. **DNS on NEW** (nothing else until this resolves): `A media.NEW` or CNAMEs for `media`, `files`, `clips`, `sync`,
-   Verify they resolve before continuing. Leave the old domain's records in place for now.
+   `live`, `diag`, `auth`, `watch`, `git`, `bot` → the box. Leave every OLD record in place. Verify:
-2. Discord developer portal → OAuth2 → add redirect `https://auth.NEW/source/oauth/callback/discord/` (keep the old one until step 9).
+   `for h in media files clips sync live diag auth watch git bot; do printf '%-6s ' $h; getent hosts $h.NEW | awk '{print $1}' | tail -1; done`
-3. Server, between streams: `./configure.py --full` → new main hostname; the eight sub-hostnames default to the new
+   — ten identical addresses. No bare-domain record unless it really points at the box.
-   base (Enter); bare domain; `y`. Then `docker compose restart mediamtx` (webrtcAdditionalHosts changed).
+2. **Both bases live:** `./configure.py --alias NEW`. Only the Caddyfile changes (`media.OLD, media.NEW {` …) and
-   All sites answer on the new names within a minute (certificates).
+   Caddy reloads; certificates for the new names follow within a minute. Nothing else moves. Check:
-4. Authentik (log in at the NEW auth host with akadmin — Discord login is broken until step 5 is done):
+   `curl -sI https://auth.NEW/ | head -1` → 302, and the old names unchanged. `sudo vghosts-check.py` now also
-   - Outposts → Embedded Outpost → `authentik_host: https://auth.NEW`
+   probes every alias.
-   - Providers → `Vanguard sites` → authentication URL `https://auth.NEW`, **cookie domain `NEW`**
+3. **Discord developer portal** → OAuth2 → Redirects → **add** `https://auth.NEW/source/oauth/callback/discord/`
-   - Providers → `Jellyfin` → redirect `https://watch.NEW/sso/OID/redirect/authentik`
+   beside the old one.
-   - Providers → `Gitea` → redirect `https://git.NEW/user/oauth2/authentik/callback`
+4. **Jellyfin** (any name, SSO still works): SSO-Auth `authentik` provider OID endpoint
-5. Private window: `https://auth.NEW/` → Discord login works (the redirect from step 2 is used).
+   `https://auth.NEW/application/o/jellyfin/` — works at once, Authentik answers on both names; Branding login
-6. Jellyfin: SSO plugin OID endpoint `https://auth.NEW/application/o/jellyfin/`; Branding disclaimer form action
+   disclaimer `https://watch.NEW/sso/OID/start/authentik`; Vanguard Metadata Public URL `https://watch.NEW`,
-   `https://watch.NEW/sso/OID/start/authentik`; Vanguard plugin Public URL + Authentik URL. Test the SSO start URL.
+   Authentik URL `https://auth.NEW`.
-7. Gitea: auth source discovery URL `https://auth.NEW/application/o/gitea/.well-known/openid-configuration`;
+5. **Gitea**: Site administration → Authentication sources → `authentik` → discovery URL
-   `./registerGiteaRunner.sh`; server: `cd ~/vanguard-stream && git remote set-url origin ssh://git@git.NEW:2222/techy/Media-Server-Setup.git`
+   `https://auth.NEW/application/o/gitea/.well-known/openid-configuration` (works at once, same reason).
-   and edit `~/.ssh/config.vanguard` (Host line); dev box: `git remote set-url origin …`. Test `sudo vanguard-deploy pull`.
+   Server clone remote and `~/.ssh/config.vanguard` → `git.NEW` (the host answers already). Dev box remote likewise.
-8. Repo defaults: `switchToClone.sh` (REPO_SSH), `deployLive.sh` (HOST) — commit, push, pull.
+6. **Authentik, then the flip, back to back, between streams.** This is the only moment anything is unreachable,
-9. Streamers: re-add the dock at `https://live.NEW/dock/`, re-save Settings, "Add band + clock source" again.
+   and it is seconds:
-   Members: `https://HOST2/`; Jellyfin apps → new server address. Remove the old Discord redirect and the old DNS.
+   `./configure.py --ak-hosts OLD NEW --dry-run` (read it) → `./configure.py --ak-hosts OLD NEW` → immediately
+   `./configure.py --base NEW` → `docker compose restart mediamtx` → `./configure.py --ak-branding` →
+   `./registerGiteaRunner.sh`.
+   `--ak-hosts` moves the outpost host, the proxy provider (external host + cookie domain `NEW`) and the
+   Jellyfin/Gitea redirect URIs: from this instant a login only sticks on the NEW names. `--base` makes NEW the
+   primary everywhere (`.env`, `mediamtx.yml`, Jellyfin/Gitea/runner/bot/Kibana recreated, Kibana tile), drops
+   the alias, and gives every OLD name a block that 308s to its NEW name, so bookmarks and pinned links keep
+   working. Members on an old name get redirected and log in once.
+7. **Test, private window:** `https://auth.NEW/` → Discord → in. `https://media.OLD/` → 308 → `https://media.NEW/`
+   → gate → in. `https://watch.NEW/` SSO round trip. `https://git.NEW/user/login` authentik sign-in.
+   `sudo vanguard-deploy pull`. `sudo vghosts-check.py` → 0 failed (it also checks every OLD name 308s).
+8. **Repo defaults:** `deployLive.sh` (HOST), `switchToClone.sh` (REPO_SSH) → NEW; commit, push, pull.
+9. **People:** OBS streamers re-add the dock at `https://live.NEW/dock/` (the old URL redirects, but OBS docks
+   want the real one), re-save Settings, "Add band + clock source" again; Go Live users just reload. Jellyfin apps
+   → new server address. Pins can stay, they redirect; re-pin at leisure. Remove the OLD Discord redirect entry
+   once everyone has logged in. The OLD DNS records stay as long as the NEW names are CNAMEs onto them.
+
+Rollback before step 6: `./configure.py --alias ""` and revert the two Jellyfin/Gitea screens. Rollback after step 6:
+`./configure.py --ak-hosts NEW OLD` then `./configure.py --base OLD` (which makes the NEW names the redirects).
 
 ## C. Splitting the single main host into media / files / clips / sync (done 2026-11-01)
 
@@ -144,8 +197,40 @@ After that, `--apply` maintains it like any other host, because the marker is no
 4. Verify the marker landed: `grep -c "vanguard:<role>" Caddyfile` → 1. Then `curl -sI https://NEW/` → 302.
 5. Add it to `addCommunityMediaServer.sh` so a rebuild asks for it too.
 
-## Checklist after any change
+## E. Validating a change: `sudo vghosts-check.py`
-    dev$ for h in HOST files.BASE clips.BASE sync.BASE live.BASE diag.BASE auth.BASE watch.BASE git.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done
+
+One command on the server checks every place in the tables above against the Caddyfile, the same source of
+truth `configure.py` reads back, and prints `ok` / `FAIL` / `warn` per item plus a `todo` list for the things no
+script can read (the Discord developer portal, streamers' OBS). Exit code 1 if anything failed.
+
+    server$ cd ~/vanguard-stream && sudo vghosts-check.py          # everything
+    dev$    ./bin/vghosts-check.py --repo                           # repo defaults, about-page labels, docs only
+    server$ sudo vghosts-check.py --json                            # for a script
+
+**The order is the order of the change, and it matters.** The sections run top-down because each one depends
+on the one above it, so a failure early on is usually the *cause* of the failures below it; fix the first
+failure and re-run rather than chasing the later ones:
+
+| # | Section | Depends on | Why this position |
+|---|---|---|---|
+| 1 | DNS | — | nothing gets a certificate or a visitor until the name resolves to the box |
+| 2 | Caddyfile | 1 | the roles and names `--full`/`--hosts` wrote; Caddy fetches certificates from here |
+| 3 | `.env` | 2 | the same names as compose variables; a mismatch is a half-finished `--full` |
+| 4 | `mediamtx.yml` | 3 | `webrtcAdditionalHosts` = main host; only a MediaMTX restart (between streams) applies it |
+| 5 | containers | 3 | the env each container was *created* with: Jellyfin, Gitea, runner, bot, MediaMTX, Kibana |
+| 6 | HTTPS | 1, 2 | every name answers with a valid certificate; gated names bounce to the auth host |
+| 7 | Jellyfin | 5, 6 | SSO plugin OID endpoint, branding disclaimer, Vanguard plugin URLs; live SSO start |
+| 8 | Gitea | 5, 6 | `app.ini`, the auth source (live redirect), the runner's registration, the server clone's remote |
+| 9 | Authentik | 7, 8 | **last of the services**: outpost host, proxy provider external host + cookie domain, Jellyfin/Gitea redirect URIs — changing these before the apps is the one way to lock yourself out |
+| 10 | Discord portal | 9 | the OAuth2 redirect is printed exactly; Discord login is broken until it matches; akadmin's card still works |
+| 11 | repo | 2 | `deployLive.sh`/`switchToClone.sh` defaults, about-page labels = Caddy roles, no browser links to `ovh1` in the docs |
+| 12 | streamers | 2 | OBS dock URL, clock source, Jellyfin apps, Discord pins: listed, never checked |
+
+What it knows that a human forgets: a container keeps the environment it was created with, so step 5 catches a
+`.env` that changed without `docker compose up -d <service>`; Authentik's cookie domain decides whether one
+login spans every host; and `ovh1.BASE` must resolve (ssh) but must *not* answer https.
+
+By hand, for the round trips the script only probes:
     https://HOST/whoami            → your identity (gate works on the new name)
     https://watch.BASE/sso/OID/start/authentik   → Jellyfin SSO round trip
     https://git.BASE/user/login    → "Sign in with authentik" round trip

LOGGING.md

@@ -20,7 +20,7 @@ journald input for our units. Metrics are the same story: see the taps table bel
 
 | endpoint | what | from |
 |---|---|---|
-| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets | MediaMTX `metrics: yes` |
+| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets. `remoteAddr` labels are real client addresses now that MediaMTX runs on the host network; before, every publisher showed as the bridge gateway `172.18.0.1` | MediaMTX `metrics: yes` |
 | `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
 | `http://127.0.0.1:8080/metrics` | ours: the normalise queue, encode time, live stream count | `bin/dumpd.py` |
 
@@ -29,6 +29,35 @@ and `docker` modules, not from an exporter — node_exporter and cAdvisor were r
 went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of
 them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher).
 
+### Stuck recordings: counts in Kibana, names in the logs, decisions on the host
+
+Three layers on purpose, because each answers a different question and none of them needs a new web page.
+
+- **How bad is it, and is it getting worse** — the gauges: `vanguard_media_pending`,
+  `vanguard_media_pending_oldest_seconds`, `vanguard_media_failed_files`. A Lens dashboard over
+  `metricbeat-*` shows the trend across a night, which is the thing a list never can.
+- **Stitching** — `event=stitched stream= parts= joins= missing_ms= seconds= out= trash=` when a stream's reconnect
+  fragments were joined; `event=stitch_wait` (stream still going), `event=stitch_skipped reason=too_many`
+  (500+ files: use vgstuck.sh), `event=stitch_break reason=gap|params_changed` (chain split there),
+  `event=stitch_failed` (fragments left alone, published one by one), `event=defer` (a fragment left alone
+  because a sibling is still being written).
+- **Stop stream** — dumpd `stream_stop stream= kind= by= minutes= result=kicked|kick failed…`, then `publish_block user= minutes= by=`
+  / `publish_unblock`, and every refused reconnect as `publish_deny reason=blocked by=`.
+- **Wipe** — `component=vgstuck event=wipe_done recordings= stale_files= trash=` when an admin pressed Bin all
+  unprocessed on the Dump tab (dumpd logs `stuck_wipe by=`), with one `event=trashed file=` line per file before it.
+- **Which recordings** — the sweep logs `event=parked file= stream= log=` once per run for every file a
+  failure log is holding back, plus `event=failed` with ffmpeg's own stderr when it first breaks. Both land
+  in `filebeat-*`, so a Discover search or a dashboard table names them and stays current. Names deliberately
+  do NOT come from the metrics: a label per filename would wreck index cardinality for no gain.
+- **What to do about it** — `bin/vgstuck.sh` on the host. It classifies every unpublished recording as
+  `parked` (a failure log is blocking it, with the reason), `published` (an output already exists),
+  `waiting` (nothing blocking it yet) or `writing` (still recording), then `retry <match>` clears the
+  blocker so the sweep tries again and `drop <match>` moves it to `trash/<today>/` for the same three-day
+  undo as a dumped night. `drop --all` is refused.
+
+A leftover `<output>.mp4.part` does **not** block anything — ffmpeg overwrites it with `-y`. It only wastes
+disk, and `retry` clears it.
+
 ### Our own metrics, and how to add one
 
 MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising
@@ -135,8 +164,15 @@ to Discord, and the pipeline is split:
 
     Kibana rule  --.index connector-->  vanguard-alerts  --vanguard-alerts.timer-->  Discord webhook
 
-- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index") and the heartbeat
+- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index"), the heartbeat rule,
-  rule. Run it once, *after* Elasticsearch security is on.
+  the data view, and the "Vanguard ops" landing dashboard plus the `defaultRoute` that makes `diag.BASE`
+  open on it. Run it once, *after* Elasticsearch security is on. Name a step to run only that one:
+  `kibana-setup.py landing`.
+  Kibana deliberately stays at the **root** of that host. Moving it under `/kibana` would mean
+  `server.basePath` plus `rewriteBasePath`, a Caddy handle that does not strip the prefix, and appending the
+  prefix in both Beats' `setup.kibana`, this script, the alarm script and the deadman's status check — and a
+  miss in the Beats one crash-loops Metricbeat, because it exits when Kibana is unreachable. `defaultRoute`
+  buys the landing page without any of that.
 - **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind`
   as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused
   them, and replayable after the fact.
@@ -294,6 +330,14 @@ Discord notifier omitted, so nothing is half-configured and nothing fails to loa
   value is at `prometheus.metrics.<name>` with labels at `prometheus.labels.*`. The dumpd scrape sets
   `use_types` and `rate_counters`, so a gauge is at `prometheus.<name>.value` and a counter has both
   `.counter` and `.rate`. Look for the right one before concluding a metric is missing.
+- **`-r` and a non-CFR `-fps_mode` are mutually exclusive in ffmpeg.** It refuses the pair with
+  "One of -r/-fpsmax was specified together a non-CFR -vsync/-fps_mode" and then fails with
+  `Error opening output files: Invalid argument` — which looks like a path or permission problem and is not.
+  Capping the frame rate means re-timing, so the cap forces CFR and passthrough has to lose.
+- **`avg_frame_rate` alone is not a frame rate.** A looped or fragmented recording reported 95 fps for a
+  30 fps source (`r_frame_rate=30/1`, `avg_frame_rate=3625/38`), which sent it down the fps-cap path and broke
+  the encode. `r_frame_rate` alone is no better: it is a huge nominal value on genuine VFR. The sweep takes
+  the **lower of the two**.
 - **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
 
 ## Wiring sketches (the options that were weighed, kept so the decision is not re-argued)

README.md

@@ -69,6 +69,8 @@ works from ~/vanguard-stream without sudo.
     ./configure.py --full      # first time: asks everything, then applies
     ./configure.py             # later: add/remove/reset streamers and editors, then apply
     ./configure.py --show      # print current users and the OBS settings to hand out
+    ./configure.py --set K=V   # change one setting (Discord tokens/webhooks, retention, alarm mentions)
+                               # run it with no arguments to list what is settable
 
 Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates
 `Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates
@@ -126,22 +128,34 @@ logged-in Discord user each time Go Live is pressed and sets it as the OBS strea
 
 OBS → Settings → Output → Encoder: **B-frames = 0** (NVENC "Max B-frames: 0"; x264 add `bframes=0`). MediaMTX
 refuses WebRTC playback of H.264 with B-frames ("WebRTC doesn't support H264 streams with B-frames"), so the live
+page falls back to HLS. The dock checks the actual recording a few seconds after Go Live and shows a red banner
+if B-frames are present (and in Simple output mode with x264 it sets `bf=0` for you); the Live tab on `diag.`
+shows the same per publisher. Nothing can set it for you in Advanced output mode, so the live
 portal would fall back to HLS with seconds of delay. Keyframe interval 2 s. Recording quality is unaffected.
 OBS → Tools → WebSocket Server Settings → enable, set a password.
 OBS → Docks → Custom Browser Docks → `https://live.BASE/dock/` → log in with Discord once.
 In the dock's Settings: ingest host (prefilled), OBS WebSocket password/port, optional title prefix/suffix
 (`{date}` → YYYY-MM-DD, e.g. prefix `HelmetCam-` or suffix `-{date}`).
 Per stream: type a title, click **Go Live**. The dock shows the final stream name (`<you>/<Title>`) as you type
-and enforces the naming rules the metadata relies on: letters, digits, `_ . -` only, no double underscore
+and enforces the naming rules the metadata relies on: letters, digits, `_ -` only (no dots: `<title>.webrtc` is reserved for the audio mirror), no double underscore
 (it separates title and timestamp in filenames), no leading dot, max 64 chars.
-After a bad night: **Dump tonight** (dumps the night of the account you are logged in as; `admins`/`media-admins`
+After a bad night: **Dump tonight** in the dock dumps the night of the account you are logged in as, and only
-may dump anyone's).
+that, admin or not. Dumping someone else's night is triage: **Dump a night**, a tab of the streaming page on
+`diag.BASE/streaming#dump` (media-admins): the night's recordings and clips per streamer, a date picker, Dump per
+streamer or Dump everyone, two confirmations. Same endpoint, same 3-day undo; dumpd honours a `user` only on that
+route. The night is 17:00–02:00 Eastern.
 
 Manual OBS/other software: dock → **Manual OBS setup** shows the RTMP server, the stream key and the SRT URL
 with the token hidden (Show / Copy / Fresh token). Tokens expire after 12 h; a running stream is not cut when
 its token expires, but a reconnect needs a fresh one.
 
 Browser streaming without OBS: the live portal's **Go Live** (Windows Chrome 141+ for window audio + GPU encode).
+On a lossy uplink Go Live switches itself to TCP: when the server reports more than 0.3% packet loss over 20 s it
+re-publishes the same capture with only MediaMTX's TCP candidate, which costs latency and nothing else. The stats
+line shows loss, retransmit requests (nack) and `via TCP` once it has happened; the recording picks up one extra
+file at the switch, which the sweep stitches. The switch is remembered in that browser, so the next stream starts
+on TCP; the "Lossy uplink" checkbox in Go Live's setup shows and controls it. Starlink users: tick it before the
+first stream, the dish drops a burst on every satellite handoff (~15 s cycle) and that is what was splitting files.
 
 ## 5a. Onboarding a gamer's clips (Syncthing) — self-service
 
@@ -196,6 +210,15 @@ In the UI (https://sync.BASE/):
 
 ## 5c. Authentik (Discord login) and Jellyfin
 
+**Login page branding and the Discord forward are not clicked in the UI.** `./configure.py --apply` (or
+`./configure.py --ak-branding` on its own) sets them from the ORM every time: brand title, logo, favicon and
+background from `authentik/brand/*.svg`, the login-card CSS, the media gateway as the brand's default application,
+flow titles, and a redirect stage at the top of `default-authentication-flow` that sends members straight to
+Discord. Nobody sees an Authentik page unless they open the admin card at
+`https://auth.BASE/if/flow/default-authentication-flow/?admin=1` (akadmin + password; the link is on the diag
+landing dashboard). Change the look by editing the SVGs or the `CSS` list in `configure.py`, then re-run.
+A UI edit to those objects is overwritten by the next apply, on purpose.
+
 DNS: `auth`, `watch` (and `files`, `clips`, `sync`, `live`, `git`, `bot`) as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
 for every hostname, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
 adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
@@ -368,7 +391,8 @@ React + dockview page (`live/`) behind Authentik forward-auth: every active stre
 audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
 MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
 MediaMTX runs the `-ffmpeg` image for the on-ready hook `bin/mtx-on-ready.sh` (bind-mounted into the container):
-the optional "is live" Discord post (`LIVE_WEBHOOK_URL` in `.env`, asked by `configure.py --full`) and the
+the optional "is live" Discord post (`LIVE_WEBHOOK_URL`, one of the Discord settings grouped at the top of
+`.env`; set it with `./configure.py --set LIVE_WEBHOOK_URL=…`, which tells you MediaMTX needs a recreate) and the
 **WebRTC audio mirror**. Browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only) and
 OBS sends AAC, so for every stream whose audio is not already Opus the hook runs one ffmpeg that copies the video
 and transcodes the audio to Opus into the sibling path `<user>/<Title>.webrtc` (via MediaMTX's RTSP on the
@@ -416,9 +440,13 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
    Logout: copyparty's logout link, the live portal's "Sign out" and the dock's "sign out" go through
    `/outpost.goauthentik.io/sign_out`; `https://<any gated host>/logout` does the same from anywhere
    (e.g. from the Syncthing UI, which has no logout of its own). Set the provider's **Invalidation flow** to
-   `default-provider-invalidation-flow` so that also ends the Authentik login: every logout then forces a full
+   `default-invalidation-flow` — the one named **"Logout"**, the same flow Authentik's own logout button uses.
-   Discord re-login on the next visit (and picks up group changes). Discord itself may approve silently if
+   Do NOT use `default-provider-invalidation-flow`: despite the name it is called "Logged out of application"
-   you are still logged into Discord in that browser.
+   and ends the session for that one application only, leaving the SSO session alive. With it, `/logout` drops
+   the proxy cookie, forward-auth asks Authentik again, Authentik still knows you, and you are signed straight
+   back in — which looks exactly like the page merely refreshing. With `default-invalidation-flow` every
+   logout ends the whole session and forces a full Discord re-login on the next visit (and so picks up group
+   changes). Discord itself may still approve silently if you remain logged into Discord in that browser.
 5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them, and
    `https://HOST/me` adds the `discord_id` claim decoded from `X-Authentik-Jwt` — the quickest check that the
    `Vanguard profile` scope mapping is live. The raw token is deliberately never echoed: the browser does not
@@ -431,8 +459,30 @@ Jellyfin and Gitea use OIDC providers instead, not this.
 
 ## 5h. Diagnostics: Kibana at `diag.HOST`
 
-Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover,
+Four containers, one store, one UI. Kibana is served at the **root** of `diag.HOST`, with no sub-path to get
-dashboards, and alerting, with no sub-paths and no index page of links.
+wrong. Arriving there opens a dashboard of ours, "Vanguard ops", via Kibana's `defaultRoute` setting: links to
+every host and to the triage page, plus shortcuts into Discover, Dashboards, Stack Rules and Index Management.
+It is an ordinary dashboard, so add metric panels beside the links (`vanguard_media_pending` first).
+
+Two paths on that host are ours rather than Kibana's:
+
+| path | what |
+|---|---|
+| `/streaming` (Stuck tab) | **stuck recordings**: everything that finished but never published, why each one is stuck, and Retry / Bin buttons. The list is readable by anyone the gate allows; the actions are enforced in dumpd for the admin groups, not hidden in the page. Binning moves the file to `trash/<today>/`, the same 3-day undo as a dumped night |
+| `/streaming/api/stuck` | dumpd serves it. Both the page and `vgstuck.sh` on the host run the same script, so they cannot disagree about why something is stuck |
+
+Kibana deliberately does **not** live under `/kibana`. That would need `server.basePath` plus
+`rewriteBasePath`, a Caddy handle that does not strip the prefix, and the prefix appended in both Beats'
+`setup.kibana`, `kibana-setup.py`, `kibana-alarms.py` and the deadman's status check — and missing the Beats
+one crash-loops Metricbeat, which exits when Kibana is unreachable. `defaultRoute` buys the landing page
+without any of it.
+
+After any hostname change, `sudo vghosts-check.py` validates every place a name lives, in the order the change
+must be made (HOSTNAME-CHANGE.md §E).
+`observability/kibana-setup.py` creates the first-run objects and takes a step name, so you can redo one:
+`connector`, `rule`, `dataview`, `landing`, or all of them when omitted. Disk use by streamer, clipper, month, logs
+and system is its own page, `https://diag.BASE/disk`. `observability/kibana-alarms.py`
+creates the six alarm rules.
 
 | container | job |
 |---|---|
@@ -567,6 +617,13 @@ Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
 
 ## Deployment (push to deploy — built, OFF by default)
 
+What lands in `/usr/local/bin` is listed once, in `bin/install.list`, and read by `install.sh`,
+`configure.py --apply` and `deploy.sh`. It used to be three hand-kept lists and they drifted: a push was
+installing five of ten scripts, so `vgstuck.sh`, both sourced helpers and both timer scripts only reached the
+host if somebody ran `--apply` by hand. `check.sh` now fails if the list names a missing file, or if
+something in `bin/` is neither listed nor explicitly excluded (`mtx-on-ready.sh` is mounted into the MediaMTX
+container; `vgbot-merge-sqlite.py` is a one-off migration).
+
 The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea →
 `./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token
 secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout.
@@ -652,6 +709,11 @@ A `.env`-driven Authentik blueprint would remove the first line (planned).
 
 ## Layout on the server
 
+MediaMTX runs with `network_mode: host`, as its documentation recommends for WebRTC: the bridge put Docker's
+userland UDP proxy between browsers and port 8189 and it lost packets, which corrupted recordings and split
+them into fragments. Its ingest ports (1935, 8890/udp, 8189 tcp+udp) are accepted by the host firewall; its
+API, HLS, WHIP/WHEP signalling and metrics bind 127.0.0.1 and are reached only through Caddy and Metricbeat.
+
     /srv/media/streams/<user>/<Title>/.live/<ts>.mp4             recording in progress (hidden from Jellyfin/copyparty)
     /srv/media/streams/<user>/<Title>__<ts>.json                 capture time (UTC ms), timecode, source (strip|arrival), drift
     /srv/media/streams/<user>/<Title>__<ts>.mp4                  finished stream, one flat MP4 per recording, directly in
@@ -709,4 +771,6 @@ evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
 |------|------|------|
 | normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
 | trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
-| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time and on `clips.` at /me/clips |
+| vanguard-alerts.timer | every 1 min | delivers Kibana's alert documents to Discord, with a 15-min per-rule repeat guard |
+| vanguard-deadman.timer | every 5 min | watches the alerting stack from outside it, and the disks |
+| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time, on `clips.` at /me/clips and on `diag.` at /streaming/api/stuck |

STATUS.md

@@ -50,7 +50,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 
 | Service | Role | State |
 |---|---|---|
-| MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`); on-ready hook posts "is live" and runs the WebRTC audio mirror (`<path>.webrtc`, audio → Opus, never recorded) | live, token auth verified |
+| MediaMTX | **Host network since 2026-11-05** (MediaMTX's recommended deployment for WebRTC). On the bridge, every WebRTC media packet on 8189/udp was relayed by Docker's userland proxy, which lost ~0.4% of them: each loss corrupted an access unit, the recorder reopened its file, and one evening produced 471 fragments; every publisher also appeared as 172.18.0.1. Ingest (1935, 8890/udp, 8189) now arrives through the nftables INPUT chain; API, HLS, WHIP/WHEP signalling and metrics bind 127.0.0.1 and are reached only through Caddy and Metricbeat. RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`); on-ready hook posts "is live" and runs the WebRTC audio mirror (`<path>.webrtc`, audio → Opus, never recorded) | live, token auth verified |
 | copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (`editors`, `media-admins`), Discord-only, no WebDAV | live |
 | dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity (on `live.`), `/me/clips` Syncthing self-service (on `clips.`); OBS dock with timecode-band setup | identity from Authentik headers | live |
 | `media.` gateway, `clips.` self-service | static pages behind the gate | — | live; **first real pairing 2026-11-01 (thrahg)** — exposed a missing `.stlocal` on fresh folders, fixed |
@@ -60,9 +60,26 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
 | Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
 | Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
+| diag landing | `diag.…` opens on a "Vanguard ops" dashboard: a markdown panel of links to every host plus the stuck page, and shortcuts to Discover, Dashboards, Stack Rules and Index Management. Kibana still owns the root; `defaultRoute` just redirects there. Created by `kibana-setup.py landing`, so a rebuild recreates it | **added 2026-11-04** |
 | Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
 | Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
 | Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
+| Fragment stitching | A stream that dropped and came back is one recording in several `.live/` files. `bin/vgstitch.py` (run first by every sweep) joins them losslessly with the concat demuxer when there are fewer than `STITCH_MAX_PARTS` (500; was 20 until Kyle's lossy night showed a recorder restart every ~10 s is normal for 0.5% loss) and the gaps are under `STITCH_MAX_GAP_S` (10 min); the arrival-time file names place every piece. Joins are collapsed and recorded in the sidecar (`stitched.parts/joins`, `missing_ms`). Nothing in a stream's `.live/` is touched until no file there is open or fresh (`event=defer` / `event=stitch_wait`). A resolution/codec change breaks the chain (copy-concat is impossible across it), so Go Live now holds resolution and sheds frame rate instead. The fragments go to `trash/<night>/` for the 3-day undo; a failed stitch leaves them alone and the old per-file path publishes them | **added 2026-11-05** |
+| Reconnect fragments | Every publisher drop-and-reconnect opened a new MediaMTX recording; one bad night left **471** one- and two-second files in `.live/`, mostly from the 6 Mbps Go Live floor (fixed). Cleanup: `vgstuck.sh drop --under 5` or the page's bulk button, admin-only, confirm shows the count. Prevention: fragments are stitched first (row above); a **lone** stream recording under `MIN_LIVE_SECONDS` (10) goes to trash instead of being published. Clips are never subject to either: they are a few seconds long on purpose and never pass through `.live/` | **added 2026-11-04** |
+| Login branding + Discord forward | Done from the ORM by `configure.py` on every apply (`--ak-branding` runs just this): brand title/logo/favicon/background from `authentik/brand/*.svg` (installed into Authentik's own `/media/public/brand/`, the only unauthenticated place that can serve them), login-card CSS in the site palette with Discord blurple, brand default application = the media gateway (so `auth.` root never shows Authentik's app library), flow titles renamed, Discord attached to the identification stage. Then the forward: a **redirect stage first in `default-authentication-flow`** whose static target is the Discord source's login URL, bound to an expression policy that skips it when the flow URL carries `?admin=1` (the executor keeps the flow query in the session key `authentik/flows/get`, not `request.GET`). Members never see an Authentik page: protected link → Discord consent → back where they were (the outpost's `next` rides the same session key). akadmin's password card is `https://auth.BASE/if/flow/default-authentication-flow/?admin=1`, linked only from the diag landing dashboard | **added 2026-11-06** |
+| Frozen Go Live picture | Breezy: one frame, then one more per alt-tab. Window capture on Windows comes from the compositor and an **exclusive full-screen** game bypasses it, so Chrome only gets a frame when Windows composites one (alt-tab). Go Live now has a capture choice, window (default) or **whole screen** (`displaySurface: monitor`, desktop duplication: sees exclusive full-screen, carries system audio), remembered per browser, plus a hint naming the symptom and the better fix: the game in borderless/windowed full-screen keeps window audio and needs no change | **added 2026-11-06** |
+| Base-domain move, no outage | `configure.py --alias NEW` makes every Caddy site answer on both bases (`media.old, media.new {`), certificates issued, nothing else touched; apps and Authentik are then moved one by one while the old names keep working; `--ak-hosts OLD NEW` (dry-run first) moves Authentik last; `--base NEW` flips the primary, recreates the host-bearing containers, and turns the old names into 308 redirects (`REDIRECT_BASE`). `read_hosts` understands both Caddyfile forms; the sweep probes aliases and old-name redirects. Rule documented at the top of HOSTNAME-CHANGE.md: apps before Authentik, never the reverse | **added 2026-11-06** |
+| Hostname sweep | `bin/vghosts-check.py` (installed; `sudo vghosts-check.py` on the box, `--repo` on a dev box, `--json`): validates every place HOSTNAME-CHANGE.md lists against the Caddyfile — DNS, `.env`, `mediamtx.yml`, the env each running container was created with, HTTPS + gate redirects per host, Authentik (outpost host, proxy external host + cookie domain, Jellyfin/Gitea redirect URIs, the exact Discord callback to check by eye), Jellyfin config XML + live SSO start, Gitea `app.ini` + live auth-source redirect + runner + clone remote, repo defaults, about-page labels, docs with browser links to `ovh1`. Sections run in the order a change must be made (documented in HOSTNAME-CHANGE.md §E) because each depends on the previous; exit 1 on any failure | **added 2026-11-06** |
+| Onboarding checklist | `media.BASE/onboarding/` (`about/onboarding/index.html`, served by the gateway's file server, so behind the same Discord gate). For the person **doing** the onboarding: eight topics to tick as they are covered — who we are, about/verify, community, who Vanguard is, who Spartan are, comms, training, push to mute/talk — and a Clear button for the next recruit. Ticks are per browser. The gateway's `try_files` gained `{path}/index.html` so folder pages under `about/` resolve without a trailing slash | **added 2026-11-06** |
+| Disk page | `diag.BASE/disk`, its own page (`streaming/disk.html`, served from the same mount): filesystem bars, then recordings per **streamer by month** (month from the recording's own `__YYYY-MM-DD` stamp; raw `.src` and unpublished `.live` called out), **clips per clipper by month**, trash by night, **logs** (Elasticsearch indices by the date in their names, the ES data dir as the on-disk truth, journald, Docker json logs) and **system** (each `appdata/<app>`, live portal, bot, Docker's own image/volume figures). dumpd walks the tree with allocated-block sizes, caches ten minutes, `?refresh=1` / "Rewalk now" forces | **added 2026-11-06** |
+| Stop a stream (Live tab) | Each publisher row has **Stop stream** with a block length (kick only / 30 min / 2 h / 8 h, default 30 min). dumpd kicks the session through MediaMTX's API (`/v3/<webrtcsessions|rtmpconns|srtconns>/kick/<id>`) **and** writes the account into `.markers/publish-blocks.json`; `/mtx-auth` denies a blocked account on every publish, so OBS's auto-reconnect and a Go Live restart both get "publishing is paused for this account by an admin" until it expires or an admin presses Allow again in the new "Blocked from publishing" section. The block is written before the kick, so there is no reconnect window. Admin groups only; the recording so far is kept and the sweep publishes it as usual | **added 2026-11-06** |
+| Transport on the Live tab | MediaMTX's session API labels the selected ICE pair `host/udp/…` or `host/tcp/…`, so the publishers table now shows `webrtc udp` / `webrtc tcp` and the session start time, and a new "Publish sessions — last 24 h" table lists every WebRTC publish session with transport, start, end, duration, received/lost, and a "switched from udp" marker on the session that began the TCP leg: that is the when and the before/after loss of a lossy-uplink switch. dumpd keeps the history in `.markers/webrtc-history.json`, updated on every Live-tab poll and every metrics scrape, so it fills in while nobody is watching | **added 2026-11-06** |
+| Lossy uplinks (Go Live) | Kyle's night: one publish session, 16 recorder restarts, 165 processing errors, 0.5% loss. A loss burst that eats a keyframe's slices leaves MediaMTX's recorder an access unit with only SPS/PPS, it refuses it (`access unit doesn't contain an IDR or non-IDR NALU`) and the path restarts the recorder 2 s later = a new file every ~10 s. Three levers pulled: (1) retransmits — MediaMTX's WebRTC code calls pion `ConfigureNack` for every peer connection, so NACK is on; the Go Live stats line now shows `loss x% (n nack, m resent)` so a stream where NACKs stay at 0 under loss is visible at a glance; (2) the stitcher cap went 20 → 500 so such a night is one recording; (3) **automatic TCP**: Go Live samples the server's receiver reports every second and when loss over the last 20 s exceeds 0.3% (after at least 1500 packets) it ends the WHIP session and re-publishes the same tracks with only MediaMTX's TCP candidates (`webrtcLocalTCPAddress :8189`) left in the answer. One extra recording file (stitched), ~2 s of picture lost in the swap, then no loss can reach the recorder; latency is what TCP costs instead. It never flips back during a stream; the stats line says `via TCP` and why. Kyle is on **Starlink**, whose scheduler reshuffles every ~15 s and drops a burst at each handoff — that is the every-10-to-15-s restart cadence in his log, and no NACK can cover a burst that long. So the switch is also **remembered per browser** (`vg.golive.tcp`): once it has fired, the next stream starts on TCP, no 20 s of corrupted recording and no extra file; a "Lossy uplink" checkbox in Go Live's setup shows and controls it. First TCP night: stable, one file, zero recorder restarts — but soft: with no loss to react to, Chrome's estimator only sees the dish's delay swings and parks near the 1.2 Mbps floor. TCP publishes now use a **4 Mbps floor** (`TCP_MIN_KBPS`): safe there because the UDP failure mode of a floor (sending into loss) cannot happen on TCP, the worst case is added latency | **added 2026-11-06** |
+| B-frame enforcement | obs-websocket cannot set encoder options in Advanced output mode, but it can switch **profiles**, and encoder settings live in a profile. The dock has an "OBS profile to stream from" setting (default `Vanguard`) and switches OBS to it before every Go Live, plus a checkbox to do so when the dock loads, i.e. on OBS startup. A streamer sets the profile up once with B-frames 0; the post-Go-Live probe verifies the result. Missing profile = a red status line naming what to create, never a silent fallback | **added 2026-11-05** |
+| B-frame detection | MediaMTX refuses H.264 with B-frames over WebRTC, and nothing on the publish path can see it: obs-websocket cannot set encoder options in OBS's Advanced output mode, and MediaMTX only notices when a viewer connects. dumpd now reads the B-frame depth off the first fragments of the live recording (`ffprobe`, small probesize, ~50 ms, works on a growing file, cached 30 s). The **dock** polls `/me/stream-health` after Go Live and shows a red banner with per-encoder instructions; in Simple output mode with x264 it writes `bf=0` into the options itself first. The **Live tab** shows an encode column per publisher: "no B-frames" or "N B-frames · WebRTC refused" | **added 2026-11-05** |
+| Streaming page | `https://diag.…/streaming`, two tabs. **Live**: every publisher with its real address, packets received and lost on the way in, the last ten minutes of publish sessions / recorder restarts / processing errors, and a one-line verdict (loss, reconnects, or parameter changes) with the recorder log tail. **Stuck**: the triage below. Served by dumpd `/streaming/api/live` and `/streaming/api/stuck`; `/stuck` redirects | **added 2026-11-05** |
+| Dump a night (web) | Third tab of the streaming page on the triage host: the night's recordings and clips grouped by streamer with sizes, a date picker (defaults to dumpd's "tonight, else last night"), a Dump button per streamer and a Dump everyone button for admins. Two confirmations, then a POST to `/streaming/api/dump` on this host, so one cookie covers it. The dock's own Dump tonight button stays for a streamer's own night and is personal for everyone, admins included: dumpd honours a `user`/`all` only when the request came in on `/streaming/api/dump`, so the only place another streamer's night can be dumped is this tab. GET is a dry run; only the POST moves files, to `trash/<night>/` with the 3-day undo. Below the night table: **Bin all unprocessed**, which runs `vgstuck.sh wipe` — stops the sweep (killing any conversion in flight; the timer restarts it to an empty queue), then every `.live/` file from any night, parked or fresh, plus the failure logs and `.part` leftovers, then the emptied folders. The only thing it leaves is a file still held open after the sweep is down, i.e. a stream MediaMTX is recording this minute. `drop --all` stays refused so a pattern can never do this by accident | **added 2026-11-05** |
+| Stuck recordings | The Stuck tab of the page above for anyone past the gate, on the triage host (already `media-admins` only), linked from the gateway's Media admin card and from the diag landing dashboard: every unpublished recording, why it is stuck, and Retry / Bin buttons. dumpd serves `/streaming/api/stuck` and allows the actions only for the admin groups; both the page and the CLI run `bin/vgstuck.sh`, so they can never disagree. `bin/vgstuck.sh` triages anything that never published: it says why each one is stuck (`parked` by a failure log with ffmpeg's reason, `published` collision, `waiting`, or still `writing`), then `retry <match>` clears the blocker and `drop <match>` moves it to trash with the usual 3-day undo. `drop --all` is refused. The sweep also logs `event=parked` once per run so Kibana can name them, not just count them | **added 2026-11-04** |
 | Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
 | Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
@@ -106,6 +123,7 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 - Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as `<Title>__<ts>.mp4` in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
 - Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. **Gap filling is available** (`FILL_GAPS_MS` in `normalize-sweep.service`, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
 - The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
+- **An encode that fails with `Error opening output files: Invalid argument` is usually contradictory flags, not the path.** ffmpeg rejects `-r` together with a non-CFR `-fps_mode`; the real message is one line earlier in its stderr, which the sweep now captures into the failure log. Found 2026-11-03 on the 1 November test recordings, which also exposed that `avg_frame_rate` can be pure fiction (95 fps reported for a 30 fps source) — the sweep now takes the lower of r_frame_rate and avg_frame_rate.
 - A deploy block that is only half-run leaves old binaries in `/usr/local/bin`; `grep -c` the installed script before blaming logic.
 - MediaMTX does not reload `mediamtx.yml` live here (bind-mounted file): any change to it needs `docker compose restart mediamtx` between streams. Nothing day-to-day touches that file.
 - **Timecode verified live (2026-10-31)**: OBS + NVENC with the band → `source: strip`, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path.
@@ -116,6 +134,17 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
 - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
 - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
+- **"Enforce B-frames 0 through obs-websocket" is only possible in Simple output mode with x264.** Advanced-mode encoder settings live in `streamEncoder.json`, which the protocol does not expose. The enforceable thing is detection on the server (the recording reveals the reorder depth within seconds) with a loud dock banner, which covers every encoder. Found 2026-11-05 when an OBS stream was unwatchable with `has_b_frames=1`.
+- **A published UDP port on the Docker bridge goes through `docker-proxy`, a userland relay, and it drops packets under load.** The sign is every client appearing as the bridge gateway (`172.18.0.1`) in MediaMTX's metrics. For WebRTC media at ~1000 packets/s this produced steady ~0.4% loss that no browser-side setting could fix, because the loss was on our box. Anything doing real-time UDP belongs on `network_mode: host`, which is also what MediaMTX documents. Found 2026-11-05 after chasing it as a client bitrate problem first.
+- **Two MediaMTX path regexes that can both match a name is a coin toss, not "first rule wins".** The mirror rule (`…\.webrtc$`) and the generic rule both matched `Techy_multi_3.webrtc` because titles were allowed dots, and the generic one won: a **36 GB** audio mirror was recorded as a full duplicate of its source. Both rules now have a dot-free title segment, so a name satisfies exactly one, and titles no longer accept dots anywhere they are typed. Needs a MediaMTX recreate between streams. Found 2026-11-04.
+- **Never `ffprobe` fragmented MP4 in a request path.** `.live/` recordings have no duration in their header, so ffprobe reads the whole file to answer. The stuck-recordings survey did that per file, took over three minutes for 471 of them, dumpd's 180 s request ceiling killed the response, Caddy returned an empty 502 and the page said "unexpected end of JSON". Size and age come from `stat`; duration is opt-in (`vgstuck.sh list --durations`). Found 2026-11-04.
+- **A high `x-google-min-bitrate` disables WebRTC congestion control.** Go Live sent `min-bitrate=6000`, so Chrome never encoded below 6 Mbps; on an uplink that could not carry it, the excess was dropped, MediaMTX logged `invalid FU-A packet (non-starting)` and every viewer saw blocky corruption. The loss was on the PUBLISH session (4761 packets for one streamer) while the viewer sessions lost nothing, which is the fingerprint: the server relays damage it received. Floor lowered to 1200 and `scaleResolutionDownBy` unpinned so the encoder can shed resolution. Found 2026-11-04.
+- `tracks=['Opus','H264']` on a path means a browser (Go Live); `MPEG-4 Audio` means OBS. Quick way to tell who is publishing how.
+- **`handle_path` strips the prefix, so an upstream's own redirects lose it.** MediaMTX redirects every HLS playlist request once (to `?cookieCheck=1`) with a root-relative `Location`, so the browser followed `/<user>/<Title>/index.m3u8` into the portal's SPA catch-all, got `index.html`, and hls.js said `manifestParsingError` — a web page where a manifest should be. Fixed with `header_down Location ^/ /hls/`. `curl` against MediaMTX directly hides this completely, because there is no prefix to lose. Found 2026-11-04.
+- **Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked** in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a `tmcd` data stream is the fingerprint. First working Windows audio: 2026-11-03.
+- The sweep's band-removal branch **overwrote `$reason`**, which carried the probe's `audio=<codec>` marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got `-c:a copy` and shipped Opus inside an MP4. Fixed by keeping `probe_reason` separately; it only became reachable once Go Live audio worked.
+- **Authentik has two invalidation flows and the names are a trap.** `default-provider-invalidation-flow` is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so `/logout` drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. `default-invalidation-flow` ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03.
+- Prefer changing a proxy provider through the ORM (`ak shell`) over the UI: saving it in the UI re-adds the managed property mappings every time.
 - **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
 - A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
 - **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.

about/index.html

@@ -32,6 +32,7 @@
     <div class="links">
       <a class="card" id="l-watch" href="#"><b>Recordings &amp; clips</b><span>Jellyfin — by streamer, by day/week/month</span></a>
       <a class="card" id="l-live" href="#"><b>Live now</b><span>every active stream in one page, audio follows the window you click</span></a>
+      <a class="card" href="/onboarding/"><b>Onboarding checklist</b><span>for whoever is onboarding someone: tick the topics as you cover them</span></a>
       <a class="card" href="/logout"><b>Sign out</b><span>ends the Discord session for all Vanguard sites</span></a>
     </div>
     <details><summary>How watching works</summary>
@@ -65,7 +66,7 @@
       <ol>
         <li>OBS → Tools → WebSocket Server Settings → enable, set a password.</li>
         <li>OBS → Docks → Custom Browser Docks → add <code id="dockurl">…/dock/</code>; log in with Discord inside the dock.</li>
-        <li>OBS → Settings → Output → encoder <b>B-frames: 0</b> (NVENC "Max B-frames"). With B-frames the live page cannot play you over WebRTC and falls back to a delayed HLS view.</li>
+        <li>Make an OBS <b>profile named Vanguard</b> once (Profile → New) and in it set Output → Advanced → your encoder with <b>B-frames 0</b> (NVENC "Max B-frames", x264 <code>bf=0</code>) and keyframe interval 2 s. The dock switches OBS to that profile on every Go Live, and on startup if you tick the box in its settings; after Go Live it checks what the server actually received and turns red if B-frames got through. With B-frames the live page cannot play you over WebRTC and falls back to a delayed HLS view.</li>
         <li>Dock → Settings: OBS WebSocket password (the rest is prefilled). Optional title prefix/suffix (<code>{date}</code> = today).</li>
         <li>Dock → Timecode band → <b>Add band + clock source</b> while not streaming (gives editors a frame-accurate timecode).</li>
         <li>Per stream: type a title, <b>Go Live</b>. The dock mints a 12-hour token and sets your stream key itself.</li>
@@ -74,7 +75,7 @@
     <details><summary>Rules worth knowing</summary>
       <ul>
         <li>Everything streamed is recorded and visible to editors and members. <b>Dump tonight</b> throws a night away (3-day undo by an admin).</li>
-        <li>Titles: letters, digits, <code>_ . -</code>; no double underscore; the dock shows the final name as you type.</li>
+        <li>Titles: letters, digits, <code>_ -</code>; no dots (the server reserves <code>.webrtc</code>); no double underscore; the dock shows the final name as you type.</li>
         <li>Browser Go Live needs Windows + Chrome/Edge 141+ for window audio and the GPU encoder; on Linux use OBS.</li>
       </ul>
     </details>
@@ -120,6 +121,7 @@
     <h2>Media admin</h2>
     <p class="lead" data-group="media-admins" data-yes="You are in <code>media-admins</code>: full rights on every folder, trash, Syncthing, dump anyone's night." data-no="Admin view — you are <b>not</b> in <code>media-admins</code>: copyparty will show you nothing and <code>/sync/</code> will 403 until you get the role."></p>
     <div class="links">
+      <a class="card" id="l-stuck" href="#"><b>Streaming</b><span>who is live and losing packets; recordings that never published</span></a>
       <a class="card" id="l-trash" href="#"><b>Trash</b><span>dumped nights, purged after 3 days</span></a>
       <a class="card" id="l-sync" href="#"><b>Syncthing</b><span>device pairing, folder states</span></a>
       <a class="card" href="/whoami"><b>Who am I</b><span>identity headers as the apps see them</span></a>
@@ -129,6 +131,7 @@
         <li>Group changes in Authentik apply on the next login: <a href="/logout">sign out</a> and back in.</li>
         <li>Dump anyone's night: the dock's Dump works on your own; for others use the API with <code>{"user":"name"}</code> or ask the owner.</li>
         <li>New gamer: they self-serve on the <a class="l-clips" href="#">clips page</a>; you only need to watch Syncthing accept the device.</li>
+        <li><a class="l-stuck-note" href="#">Stuck recordings</a> (on the diagnostics host, with the dashboards) lists anything that finished but never published, with the reason. Retry puts it back in the queue; binning keeps the same 3-day undo as a dumped night.</li>
       </ul>
     </details>
   </section>
@@ -179,6 +182,8 @@ $('l-git').href = sub('git'); $('l-bot').href = sub('bot'); $('l-auth').href = s
 $('l-jfadmin').href = sub('watch') + '/web/#/dashboard'; $('l-gitadmin').href = sub('git') + '/-/admin';
 $('l-dock').href = sub('live') + '/dock/'; $('dockurl').textContent = sub('live') + '/dock/';
 $('l-files').href = sub('files') + '/'; $('l-finals').href = sub('files') + '/finals/'; $('l-trash').href = sub('files') + '/trash/'; $('l-sync').href = sub('sync') + '/';
+$('l-stuck').href = sub('diag') + '/streaming#stuck';
+document.querySelectorAll('a.l-stuck-note').forEach(a => a.href = sub('diag') + '/streaming#stuck');
 $('l-clips').href = sub('clips') + '/'; document.querySelectorAll('a.l-clips').forEach(a => a.href = sub('clips') + '/'); $('gitssh').textContent = `ssh://git@${'git.' + base}:2222/org/repo.git`;
 (async () => {
   try {

about/onboarding/index.html

@@ -0,0 +1,58 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Onboarding — Vanguard</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#5865F2;--ok:#2ecc71;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:17px/1.5 system-ui,sans-serif}
+  main{max-width:560px;margin:0 auto;padding:36px 16px 48px}
+  header{display:flex;align-items:center;gap:14px;margin-bottom:18px}
+  h1{font-size:26px;margin:0}
+  .right{margin-left:auto;display:flex;align-items:center;gap:12px;color:var(--mut);font-size:14px}
+  button{font:inherit;font-size:14px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:7px 14px;cursor:pointer}
+  button:hover{border-color:var(--acc)}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:8px 10px}
+  label{display:flex;gap:14px;align-items:center;padding:12px 10px;border-radius:10px;cursor:pointer;font-size:19px}
+  label:hover{background:#262626}
+  label input{width:22px;height:22px;margin:0;accent-color:var(--acc);flex:none}
+  label input:checked + span{color:var(--mut);text-decoration:line-through}
+  .done{display:none;color:var(--ok);margin-top:14px;font-size:15px}
+</style>
+</head>
+<body>
+<main>
+  <header>
+    <h1>Onboarding</h1>
+    <div class="right"><span id="count"></span><button id="clear">Clear</button></div>
+  </header>
+  <section>
+    <label><input type="checkbox" data-id="who"><span>Who we are</span></label>
+    <label><input type="checkbox" data-id="about"><span>About / verify</span></label>
+    <label><input type="checkbox" data-id="community"><span>Community</span></label>
+    <label><input type="checkbox" data-id="vanguard"><span>Who Vanguard is</span></label>
+    <label><input type="checkbox" data-id="spartan"><span>Who Spartan are</span></label>
+    <label><input type="checkbox" data-id="comms"><span>Comms</span></label>
+    <label><input type="checkbox" data-id="training"><span>Training</span></label>
+    <label><input type="checkbox" data-id="ptt"><span>Push to mute / talk</span></label>
+  </section>
+  <div class="done" id="done">All covered. Clear for the next one.</div>
+</main>
+<script>
+const KEY = 'vg.onboarding';
+const boxes = [...document.querySelectorAll('input[type=checkbox]')];
+const load = () => { try { return JSON.parse(localStorage.getItem(KEY) || '{}'); } catch { return {}; } };
+const save = s => { try { localStorage.setItem(KEY, JSON.stringify(s)); } catch {} };
+function render() {
+  const s = load(); let n = 0;
+  for (const b of boxes) { b.checked = !!s[b.dataset.id]; if (b.checked) n++; }
+  document.getElementById('count').textContent = `${n} / ${boxes.length}`;
+  document.getElementById('done').style.display = n === boxes.length ? 'block' : 'none';
+}
+for (const b of boxes) b.addEventListener('change', () => { const s = load(); s[b.dataset.id] = b.checked; save(s); render(); });
+document.getElementById('clear').onclick = () => { save({}); render(); };
+render();
+</script>
+</body>
+</html>

authentik/brand/favicon.svg

@@ -0,0 +1,4 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="64" height="64" viewBox="0 0 64 64">
+  <rect width="64" height="64" rx="14" fill="#5865F2"/>
+  <path d="M14 16h13l5 24 5-24h13L40 52H24z" fill="#141414"/>
+</svg>

authentik/brand/login-bg.svg

@@ -0,0 +1,16 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080" viewBox="0 0 1920 1080" preserveAspectRatio="xMidYMid slice">
+  <defs>
+    <radialGradient id="g" cx="30%" cy="20%" r="90%">
+      <stop offset="0" stop-color="#23264a"/>
+      <stop offset="0.55" stop-color="#141414"/>
+      <stop offset="1" stop-color="#0c0c0c"/>
+    </radialGradient>
+    <pattern id="grid" width="48" height="48" patternUnits="userSpaceOnUse">
+      <path d="M48 0H0v48" fill="none" stroke="#5865F2" stroke-opacity="0.08" stroke-width="1"/>
+    </pattern>
+  </defs>
+  <rect width="1920" height="1080" fill="url(#g)"/>
+  <rect width="1920" height="1080" fill="url(#grid)"/>
+  <circle cx="1500" cy="860" r="420" fill="#5865F2" fill-opacity="0.07"/>
+  <circle cx="300" cy="150" r="260" fill="#3d8bfd" fill-opacity="0.06"/>
+</svg>

authentik/brand/logo.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="400" height="96" viewBox="0 0 400 96" role="img" aria-label="Vanguard">
+  <rect x="4" y="8" width="80" height="80" rx="18" fill="#5865F2"/>
+  <path d="M22 26h16l6 30 6-30h16L54 72H34z" fill="#141414"/>
+  <text x="100" y="66" font-family="Inter, Segoe UI, Helvetica, Arial, sans-serif" font-weight="800" font-size="44" letter-spacing="3" fill="#e6e6e6">VANGUARD</text>
+</svg>

bin/dumpd.py

@@ -3,7 +3,10 @@
 
 POST /dump   identity from X-Authentik-Username / X-Authentik-Groups (set by Caddy after Authentik)
              body (optional JSON): {"night":"YYYY-MM-DD","user":"name"}
-GET  /dump   -> lists what a dump would move (dry run)
+GET  /dump   -> lists what a dump would move (dry run); ?user=all&night=YYYY-MM-DD select whose and when
+             Two doors, different reach: /dump on live. (the dock's Dump tonight) is ALWAYS the caller's own night,
+             admin or not — "user" is ignored there. /streaming/api/dump on diag. (the streaming page's Dump tab) is
+             where admins pick a streamer or "all"; non-admins still get only their own there.
 GET  /me/stream-token -> short-lived publish token for the caller (member of streamers/admins); the dock
              and the live portal set it as the OBS/WHIP credential. No stored passwords anywhere.
 GET  /me/clips  -> Syncthing self-service status for the caller: our device id, their folder/device state
@@ -21,6 +24,18 @@ POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a D
                    HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
                    May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
                    into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body.
+GET  /me/stream-health -> the caller's live streams with B-frame depth: the dock warns when WebRTC will refuse
+GET  /streaming/api/live  -> what is publishing, from where, loss on the way in, recorder restarts, transport history, blocks
+POST /streaming/api/live  -> {"action":"stop","path","kind","session","minutes"}: kick that publish session via MediaMTX's
+             API and block the account's publishing for N minutes (0 = kick only); {"action":"unblock","user"}. Admins only.
+             /mtx-auth denies a blocked account, so neither OBS's auto-reconnect nor a Go Live restart gets around it.
+GET  /streaming/api/stuck -> every unpublished recording and why it is stuck (any gated user)
+GET  /streaming/api/disk  -> disk use by streamer / clipper / trash by month, system dirs, Docker, and logs (ES indices
+             by month, journald, Docker json logs). Cached 10 min; ?refresh=1 rewalks.
+POST /streaming/api/stuck -> {"action":"retry"|"drop","src":...}: clear the blocker, or move it to trash with the
+             usual 3-day undo; {"action":"drop_under","mb":N} bins every stream fragment under N MB; {"action":"wipe"}
+             bins everything unprocessed (not being written). Admin groups only; every action runs bin/vgstuck.sh
+             so the web page and the CLI share one implementation.
 GET  /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live
              gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here.
 POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid
@@ -28,10 +43,12 @@ POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish i
 
 "Night" = everything the user recorded from 17:00 Eastern on <night> until 02:00 Eastern the next day.
 Default night: today if it is 17:00 or later, else yesterday (so 00:00-02:00 still counts as last night).
-Users may dump only their own prefix; members of the admins / media-admins groups may pass "user" to dump anyone, or "all".
+Users may dump only their own prefix; members of the admins / media-admins groups may pass "user" to dump anyone, or "all",
+but only through /streaming/api/dump (the diag triage page). On live.'s /dump everyone dumps their own night only.
 """
 import base64, hashlib, hmac, json, os, re, shutil, subprocess, sys, threading
 import time as _time
+import urllib.parse
 from datetime import datetime, timedelta, time
 from zoneinfo import ZoneInfo
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
@@ -47,7 +64,7 @@ PUBLISH_GROUPS = set(g.strip() for g in os.environ.get("PUBLISH_GROUPS", "stream
 AUTH_USER_HEADER = os.environ.get("AUTH_USER_HEADER", "X-Authentik-Username")
 AUTH_GROUPS_HEADER = os.environ.get("AUTH_GROUPS_HEADER", "X-Authentik-Groups")
 ADMIN_GROUPS = set(g.strip() for g in os.environ.get("DUMP_ADMIN_GROUPS", "admins,media-admins").split(",") if g.strip())
-LISTEN = ("127.0.0.1", 8080)                     # /dump, /me/* — only via Caddy (identity headers)
+LISTEN = ("127.0.0.1", 8080)                     # /dump, /streaming/api/*, /me/* — only via Caddy (identity headers)
 AUTH_LISTEN = (os.environ.get("MTX_AUTH_BIND", "0.0.0.0"), int(os.environ.get("MTX_AUTH_PORT", "8081")))   # /mtx-auth only — reachable from Docker's bridge (firewall), used by MediaMTX
 TZ = ZoneInfo(os.environ.get("DUMP_TZ", "America/New_York"))
 NIGHT_START = time(17, 0)   # 5 pm
@@ -425,6 +442,465 @@ def verify_token(token):
         return None
     return user.lower()
 
+
+
+# ---- stream health: B-frames ------------------------------------------------------------------------
+# MediaMTX refuses to serve H.264 with B-frames over WebRTC ("WebRTC doesn't support H264 streams with
+# B-frames"), so an OBS encoder left on its default of 2 B-frames is unwatchable on the portal. Nothing on the
+# publish path can see this: obs-websocket cannot set encoder settings in OBS's Advanced output mode, and
+# MediaMTX only notices when a viewer tries. The recording can: ffprobe reads the reorder depth off the first
+# fragments of the growing .live file in ~50 ms with a small probesize. Cached per path for 30 s.
+_health_cache = {}
+
+def stream_health(path):
+    """{'bframes', 'profile', 'res', 'fps', 'file'} for a live path, or None if nothing is on disk yet."""
+    now = _time.time()
+    hit = _health_cache.get(path)
+    if hit and now - hit["at"] < 30:
+        return hit["data"]
+    live_dir = ROOT / "streams" / path / ".live"
+    try:
+        newest = max((f for f in live_dir.glob("*.mp4")), key=lambda f: f.stat().st_mtime, default=None)
+    except OSError:
+        newest = None
+    data = None
+    if newest is not None:
+        # One probe, fields by name: csv column order follows ffprobe's internal field order, not the list.
+        done = subprocess.run(["ffprobe", "-v", "error", "-probesize", "2M", "-analyzeduration", "2M",
+                               "-select_streams", "v:0",
+                               "-show_entries", "stream=has_b_frames,profile,width,height,r_frame_rate",
+                               "-of", "default=noprint_wrappers=1", str(newest)],
+                              capture_output=True, text=True, timeout=10)
+        fields = dict(line.split("=", 1) for line in done.stdout.splitlines() if "=" in line)
+        if fields:
+            try:
+                num, den = (fields.get("r_frame_rate", "0/1").split("/") + ["1"])[:2]
+                fps = round(float(num) / float(den), 2) if float(den) else None
+            except (ValueError, ZeroDivisionError):
+                fps = None
+            data = {"bframes": int(fields.get("has_b_frames") or 0), "profile": fields.get("profile", ""),
+                    "res": f"{fields.get('width', '?')}x{fields.get('height', '?')}", "fps": fps, "file": newest.name}
+    _health_cache[path] = {"at": now, "data": data}
+    return data
+
+# ---- live streaming status ------------------------------------------------------------------------
+# GET /streaming/api/live: what is publishing, from where, how much it has lost on the way in, and why
+# MediaMTX has been opening recording files. Cached 5 s: the Live tab polls every 10 s and several admins
+# may have it open. Reads MediaMTX's API and metrics on loopback, and its container log for the counters.
+_live_cache = {"at": 0.0, "data": None}
+MTX_METRICS = os.environ.get("MTX_METRICS_URL", "http://127.0.0.1:9998/metrics")
+
+# WebRTC publish sessions and their transport. MediaMTX's API labels the selected ICE pair as
+# "<type>/<protocol>/<addr>/<port>" (host/udp/… or host/tcp/…), so the protocol a publisher is really on is
+# right there — that is how the Live tab tells a Go Live on TCP (the lossy-uplink switch) from one on UDP.
+# A session history is kept in MARK/webrtc-history.json so "when did he switch" survives the page being closed:
+# it is updated on every Live-tab poll AND every metrics scrape (Metricbeat, every few seconds), and a session
+# that is no longer listed is stamped ended. 24 h, 50 sessions per path.
+HISTORY_FILE = MARK / "webrtc-history.json"
+_history_lock = threading.Lock()
+
+def webrtc_sessions():
+    with urllib.request.urlopen(f"{MTX_API}/v3/webrtcsessions/list?itemsPerPage=1000", timeout=3) as response:
+        return json.load(response).get("items") or []
+
+def session_transport(item):
+    label = item.get("localCandidate") or item.get("remoteCandidate") or ""
+    parts = label.split("/")
+    return parts[1].lower() if len(parts) >= 2 and parts[1] else ""
+
+def track_sessions(items):
+    """Fold the current publish sessions into the history file; returns the history (path -> sessions)."""
+    now = datetime.now(TZ)
+    with _history_lock:
+        try:
+            hist = json.loads(HISTORY_FILE.read_text())
+        except (OSError, ValueError):
+            hist = {}
+        live_ids = set()
+        for it in items:
+            if str(it.get("state", "")).lower() != "publish":
+                continue
+            sid = it.get("id", ""); live_ids.add(sid)
+            rows = hist.setdefault(it.get("path", "?"), [])
+            row = next((r for r in rows if r["id"] == sid), None)
+            if row is None:
+                row = {"id": sid, "created": it.get("created"), "remote": it.get("remoteAddr"), "transport": "", "ended": None}
+                rows.append(row)
+            if session_transport(it): row["transport"] = session_transport(it)
+            row["received"] = it.get("rtpPacketsReceived", 0); row["lost"] = it.get("rtpPacketsLost", 0)
+            row["bytes"] = it.get("bytesReceived", 0); row["last_seen"] = now.isoformat(timespec="seconds")
+        cutoff = now - timedelta(hours=24)
+        for path in list(hist):
+            kept = []
+            for r in hist[path]:
+                if r["id"] not in live_ids and not r.get("ended"):
+                    r["ended"] = r.get("last_seen") or now.isoformat(timespec="seconds")
+                try:
+                    when = datetime.fromisoformat((r.get("ended") or r.get("last_seen") or "").replace("Z", "+00:00"))
+                    if when.tzinfo is None: when = when.replace(tzinfo=TZ)
+                except ValueError:
+                    when = now
+                if when >= cutoff: kept.append(r)
+            kept.sort(key=lambda r: r.get("created") or "")
+            hist[path] = kept[-50:]
+            if not hist[path]: del hist[path]
+        try:
+            MARK.mkdir(exist_ok=True)
+            tmp = HISTORY_FILE.with_suffix(".tmp"); tmp.write_text(json.dumps(hist)); os.replace(tmp, HISTORY_FILE)
+        except OSError as error:
+            log("history_write_failed", level="warn", error=str(error))
+        return hist
+
+def session_history():
+    """History as the page wants it: one list, newest first, with switch markers."""
+    try:
+        hist = json.loads(HISTORY_FILE.read_text())
+    except (OSError, ValueError):
+        return []
+    out = []
+    for path, rows in hist.items():
+        prev = None
+        for r in rows:
+            entry = {"path": path, **{k: r.get(k) for k in ("id", "created", "ended", "remote", "transport", "received", "lost", "bytes")}}
+            entry["id"] = (entry["id"] or "")[:8]
+            if prev is not None and prev.get("transport") and r.get("transport") and prev["transport"] != r["transport"]:
+                entry["switched_from"] = prev["transport"]
+            prev = r
+            out.append(entry)
+    out.sort(key=lambda e: e.get("created") or "", reverse=True)
+    return out
+
+# Turning a stream off from the Live tab. Two parts, because a kick alone is a hiccup: OBS reconnects on its
+# own within seconds and Go Live can be restarted with one click. So "stop" = kick the publish session through
+# MediaMTX's API AND put the streamer on a block list that /mtx-auth consults for every publish, for a chosen
+# number of minutes (0 = kick only). Blocks live in MARK/publish-blocks.json, survive a dumpd restart, expire on
+# their own, and the Live tab lists them with an unblock button.
+BLOCKS_FILE = MARK / "publish-blocks.json"
+_blocks_lock = threading.Lock()
+KICK_ENDPOINT = {"webrtc": "webrtcsessions", "rtmp": "rtmpconns", "srt": "srtconns", "rtsp": "rtspsessions"}
+
+def read_blocks():
+    try:
+        blocks = json.loads(BLOCKS_FILE.read_text())
+    except (OSError, ValueError):
+        blocks = {}
+    now = _time.time()
+    live = {u: b for u, b in blocks.items() if b.get("until", 0) > now}
+    if len(live) != len(blocks):
+        write_blocks(live)
+    return live
+
+def write_blocks(blocks):
+    try:
+        MARK.mkdir(exist_ok=True)
+        tmp = BLOCKS_FILE.with_suffix(".tmp"); tmp.write_text(json.dumps(blocks)); os.replace(tmp, BLOCKS_FILE)
+    except OSError as error:
+        log("blocks_write_failed", level="error", error=str(error))
+
+def block_user(user, minutes, by, reason=""):
+    with _blocks_lock:
+        blocks = read_blocks()
+        blocks[user] = {"until": _time.time() + minutes * 60, "by": by, "reason": reason,
+                        "since": datetime.now(TZ).isoformat(timespec="seconds")}
+        write_blocks(blocks)
+    log("publish_block", user=user, minutes=minutes, by=by)
+
+def unblock_user(user, by):
+    with _blocks_lock:
+        blocks = read_blocks()
+        if blocks.pop(user, None) is not None:
+            write_blocks(blocks); log("publish_unblock", user=user, by=by); return True
+    return False
+
+def blocked(user):
+    return read_blocks().get(user)
+
+def kick_publisher(kind, session_id):
+    endpoint = KICK_ENDPOINT.get(kind)
+    if not endpoint:
+        raise ValueError(f"cannot kick a {kind} publisher")
+    request = urllib.request.Request(f"{MTX_API}/v3/{endpoint}/kick/{urllib.parse.quote(session_id, safe='')}", method="POST")
+    with urllib.request.urlopen(request, timeout=5) as response:
+        return response.status
+
+def stop_stream(path, kind, session_id, minutes, who):
+    user = path.split("/", 1)[0].lower()
+    out = {"path": path, "user": user, "kicked": False, "blocked_minutes": minutes}
+    if minutes > 0:
+        block_user(user, minutes, who, reason=f"stopped from the Live tab ({path})")   # block first: no reconnect window
+    try:
+        kick_publisher(kind, session_id); out["kicked"] = True
+    except Exception as error:
+        out["error"] = f"kick failed: {error}"
+    # the Opus mirror of an OBS stream dies with its source (runOnReady exits), so nothing else to stop
+    _live_cache["data"] = None
+    log("stream_stop", level="info" if out["kicked"] else "error", stream=path, kind=kind, by=who, minutes=minutes,
+        result=out.get("error", "kicked"))
+    return out
+
+def live_status():
+    now = _time.time()
+    if _live_cache["data"] is not None and now - _live_cache["at"] < 5:
+        return _live_cache["data"]
+    out = {"paths": [], "publishers": [], "counts": {}, "log": [], "errors": [], "history": []}
+    sessions = {}
+    try:
+        items = webrtc_sessions()
+        track_sessions(items)
+        for it in items:
+            sessions[it.get("id", "")] = {"transport": session_transport(it), "since": it.get("created"),
+                                          "established": it.get("peerConnectionEstablished")}
+        out["history"] = session_history()
+    except Exception as error:
+        out["errors"].append(f"MediaMTX WebRTC sessions: {error}")
+    try:
+        with urllib.request.urlopen(f"{MTX_API}/v3/paths/list?itemsPerPage=1000", timeout=3) as response:
+            for item in json.load(response).get("items") or []:
+                if item.get("ready"):
+                    row = {"name": item["name"], "tracks": item.get("tracks") or [],
+                           "readers": len(item.get("readers") or []),
+                           "since": item.get("readyTime"), "bytes": item.get("bytesReceived", 0),
+                           "mirror": str(item["name"]).endswith(".webrtc")}
+                    if not row["mirror"]:
+                        try:
+                            row["health"] = stream_health(item["name"])
+                        except Exception as error:
+                            row["health_error"] = str(error)
+                    out["paths"].append(row)
+    except Exception as error:
+        out["errors"].append(f"MediaMTX API: {error}")
+    try:
+        with urllib.request.urlopen(MTX_METRICS, timeout=3) as response:
+            text = response.read().decode(errors="replace")
+        rows = {}
+        for match in re.finditer(r"^(webrtc_sessions|rtmp_conns|srt_conns)(_rtp_packets_received|_rtp_packets_lost|_bytes_received)?\{([^}]*)\} (\d+)$", text, re.M):
+            kind, field, labels_text, value = match.groups()
+            labels = dict(re.findall(r'(\w+)="([^"]*)"', labels_text))
+            if labels.get("state") != "publish":
+                continue
+            key = (labels.get("path", "?"), labels.get("id", ""), labels.get("remoteAddr", "?"))
+            row = rows.setdefault(key, {"kind": kind.split("_")[0]})
+            if field == "_rtp_packets_received": row["received"] = int(value)
+            elif field == "_rtp_packets_lost": row["lost"] = int(value)
+            elif field == "_bytes_received": row["bytes"] = int(value)
+        for (path, sid, addr), row in rows.items():
+            out["publishers"].append({"path": path, "id": sid[:8], "session": sid, "from": addr, **row, **sessions.get(sid, {})})
+        out["blocks"] = [{"user": u, **b, "until_iso": datetime.fromtimestamp(b["until"], TZ).isoformat(timespec="seconds")}
+                         for u, b in sorted(read_blocks().items())]
+    except Exception as error:
+        out["errors"].append(f"MediaMTX metrics: {error}")
+    try:
+        done = subprocess.run(["docker", "logs", "mediamtx", "--since", "10m"],
+                              capture_output=True, text=True, timeout=15)
+        log_text = done.stdout + done.stderr
+        out["counts"] = {
+            "publish_sessions": log_text.count("is publishing"),
+            "recorder_restarts": len(re.findall(r"recording \d+ tracks", log_text)),
+            "processing_errors": log_text.count("processing errors"),
+            "access_unit_errors": log_text.count("recorder] access unit"),
+            "window": "10m",
+        }
+        out["log"] = [l for l in log_text.splitlines() if re.search(r"recorder|is publishing|closed|ERR", l)][-14:]
+    except Exception as error:
+        out["errors"].append(f"MediaMTX log: {error}")
+    _live_cache.update(at=now, data=out)
+    return out
+
+# ---- disk report ------------------------------------------------------------------------------------
+# GET /streaming/api/disk: where the bytes are. Streamers (streams/<user>, month from the recording's own
+# "__YYYY-MM-DD" stamp, else mtime), clippers (clips/<user>, mtime), trash (its night folders), system (appdata
+# per app, /srv/live, /srv/bot, Docker's own accounting) and logs (Elasticsearch indices by the date in their
+# name, the ES data dir as the on-disk truth, journald, Docker's json logs). Walking a few TB of files takes
+# seconds, so the report is cached for ten minutes and ?refresh=1 forces it. Sizes are allocated blocks, i.e.
+# what the disk actually loses, not apparent length.
+_disk_cache = {"at": 0.0, "data": None}
+_disk_lock = threading.Lock()
+DISK_CACHE_S = 600
+APPDATA = Path("/srv/appdata")
+ES_NETRC = Path(os.environ.get("ES_NETRC_FILE", "/etc/vanguard/es-netrc"))
+ES_URL = os.environ.get("ES_URL", "http://127.0.0.1:9200")
+MONTH_IN_NAME = re.compile(r"__(\d{4})-(\d{2})-\d{2}_")
+MONTH_IN_INDEX = re.compile(r"(\d{4})\.(\d{2})\.\d{2}")
+
+def _blocks(st):
+    return st.st_blocks * 512 if hasattr(st, "st_blocks") else st.st_size
+
+def walk_sizes(root, month_of):
+    """{month: bytes} and total for everything under root; month_of(path, stat) -> 'YYYY-MM'."""
+    months, total = {}, 0
+    stack = [root]
+    while stack:
+        current = stack.pop()
+        try:
+            with os.scandir(current) as entries:
+                for entry in entries:
+                    try:
+                        if entry.is_symlink(): continue
+                        if entry.is_dir(follow_symlinks=False): stack.append(entry.path); continue
+                        st = entry.stat(follow_symlinks=False)
+                    except OSError:
+                        continue
+                    size = _blocks(st); total += size
+                    month = month_of(entry.path, st)
+                    months[month] = months.get(month, 0) + size
+        except OSError:
+            continue
+    return {"total": total, "months": months}
+
+def month_from_mtime(_path, st):
+    return datetime.fromtimestamp(st.st_mtime, TZ).strftime("%Y-%m")
+
+def month_from_name(path, st):
+    m = MONTH_IN_NAME.search(os.path.basename(path))
+    return f"{m.group(1)}-{m.group(2)}" if m else month_from_mtime(path, st)
+
+def dir_total(path):
+    return walk_sizes(path, lambda p, s: "all")["total"] if Path(path).is_dir() else 0
+
+def es_indices():
+    """[{index, bytes, month}] from _cat/indices, with the netrc the deadman also uses."""
+    import netrc as _netrc
+    request = urllib.request.Request(f"{ES_URL}/_cat/indices?bytes=b&format=json&h=index,store.size,creation.date.string&expand_wildcards=all")
+    try:
+        creds = _netrc.netrc(str(ES_NETRC)).authenticators(urllib.parse.urlparse(ES_URL).hostname or "127.0.0.1")
+    except (OSError, _netrc.NetrcParseError):
+        creds = None
+    if creds:
+        request.add_header("Authorization", "Basic " + base64.b64encode(f"{creds[0]}:{creds[2]}".encode()).decode())
+    with urllib.request.urlopen(request, timeout=10) as response:
+        rows = json.load(response)
+    out = []
+    for row in rows:
+        name = row.get("index", "")
+        m = MONTH_IN_INDEX.search(name)
+        month = f"{m.group(1)}-{m.group(2)}" if m else (row.get("creation.date.string") or "")[:7] or "other"
+        out.append({"index": name, "bytes": int(row.get("store.size") or 0), "month": month})
+    return out
+
+def docker_df():
+    done = subprocess.run(["docker", "system", "df", "--format", "{{json .}}"], capture_output=True, text=True, timeout=60)
+    rows = []
+    for line in done.stdout.splitlines():
+        try: rows.append(json.loads(line))
+        except ValueError: pass
+    return rows
+
+def journal_bytes():
+    done = subprocess.run(["journalctl", "--disk-usage"], capture_output=True, text=True, timeout=20)
+    m = re.search(r"([\d.]+)\s*([KMGT]?)B?", done.stdout.replace(",", ""))
+    if not m: return 0
+    return int(float(m.group(1)) * {"": 1, "K": 1e3, "M": 1e6, "G": 1e9, "T": 1e12}[m.group(2)])
+
+def disk_report(force=False):
+    now = _time.time()
+    with _disk_lock:
+        if not force and _disk_cache["data"] is not None and now - _disk_cache["at"] < DISK_CACHE_S:
+            return _disk_cache["data"]
+        t0 = _time.time()
+        out = {"generated": datetime.now(TZ).isoformat(timespec="seconds"), "fs": {}, "streamers": {}, "clippers": {},
+               "trash": {}, "system": {}, "logs": {}, "errors": []}
+        for label, path in (("media", ROOT), ("root", Path("/"))):
+            try:
+                u = shutil.disk_usage(path); out["fs"][label] = {"path": str(path), "total": u.total, "used": u.used, "free": u.free}
+            except OSError as error:
+                out["errors"].append(f"{path}: {error}")
+        same_fs = False
+        try: same_fs = os.stat(ROOT).st_dev == os.stat("/").st_dev
+        except OSError: pass
+        out["fs"]["same_filesystem"] = same_fs
+        for user_dir in sorted(p for p in (ROOT / "streams").glob("*") if p.is_dir()):
+            r = walk_sizes(user_dir, month_from_name)
+            r["raw"] = dir_total(user_dir / ".src"); r["unpublished"] = sum(dir_total(d) for d in user_dir.glob("*/.live"))
+            out["streamers"][user_dir.name] = r
+        for user_dir in sorted(p for p in (ROOT / "clips").glob("*") if p.is_dir()):
+            out["clippers"][user_dir.name] = walk_sizes(user_dir, month_from_mtime)
+        out["trash"] = walk_sizes(ROOT / "trash", lambda p, s: (re.search(r"/trash/(\d{4}-\d{2})-\d{2}/", p) or [None, month_from_mtime(p, s)])[1]) if (ROOT / "trash").is_dir() else {"total": 0, "months": {}}
+        other_media = 0
+        for entry in ROOT.iterdir():
+            if entry.name not in ("streams", "clips", "trash") and entry.is_dir() and not entry.is_symlink():
+                other_media += dir_total(entry)
+        out["system"]["media_other"] = other_media                       # .markers, .syncthing, .stversions, …
+        for app_dir in sorted(p for p in APPDATA.glob("*") if p.is_dir()) if APPDATA.is_dir() else []:
+            size = dir_total(app_dir)
+            if app_dir.name == "elasticsearch": out["logs"]["es_data_dir"] = size
+            else: out["system"][f"appdata/{app_dir.name}"] = size
+        for label, path in (("live portal", "/srv/live"), ("bot", "/srv/bot"), ("etc/vanguard", "/etc/vanguard")):
+            out["system"][label] = dir_total(path)
+        try:
+            for row in docker_df():
+                out["system"][f"docker {row.get('Type', '?').lower()}"] = row.get("Size", "?")   # docker's own figures (human strings)
+        except Exception as error:
+            out["errors"].append(f"docker system df: {error}")
+        try:
+            out["logs"]["indices"] = es_indices()
+            out["logs"]["es_total"] = sum(i["bytes"] for i in out["logs"]["indices"])
+            by_month = {}
+            for i in out["logs"]["indices"]: by_month[i["month"]] = by_month.get(i["month"], 0) + i["bytes"]
+            out["logs"]["months"] = by_month
+        except Exception as error:
+            out["errors"].append(f"Elasticsearch: {error}")
+        try: out["logs"]["journald"] = journal_bytes()
+        except Exception as error: out["errors"].append(f"journalctl: {error}")
+        out["logs"]["docker_json_logs"] = sum(_blocks(p.stat()) for p in Path("/var/lib/docker/containers").glob("*/*-json.log*")) if Path("/var/lib/docker/containers").is_dir() else 0
+        out["seconds"] = round(_time.time() - t0, 1)
+        _disk_cache.update(at=_time.time(), data=out)
+        return out
+
+# ---- stuck recordings -----------------------------------------------------------------------------
+# GET  /streaming/api/stuck   every unpublished recording and why, for anyone the gate let through
+# POST /streaming/api/stuck   {"action":"retry"|"drop","src":"<path>"} — admins only
+# The classification and the file moves both live in bin/vgstuck.sh, so the web page and the CLI can never
+# disagree about why something is stuck, and "delete" means the same three-day trash in both.
+STUCK_TOOL = os.environ.get("VGSTUCK_TOOL", "/usr/local/bin/vgstuck.sh")
+
+_stuck_cache = {"at": 0.0, "items": None}
+STUCK_CACHE_S = 20          # the page refreshes every 30 s and several admins may have it open
+
+def stuck_list():
+    """The survey, cached briefly. It runs ffprobe on every stuck file, so with many of them it is slow, and
+    without a cache every open tab re-ran it every 30 s. The cache is also invalidated by any action."""
+    now = _time.time()
+    if _stuck_cache["items"] is not None and now - _stuck_cache["at"] < STUCK_CACHE_S:
+        return _stuck_cache["items"]
+    try:
+        done = subprocess.run([STUCK_TOOL, "json"], capture_output=True, text=True, timeout=120)
+    except subprocess.TimeoutExpired:
+        log("stuck_list_timeout", level="error", seconds=120)
+        raise RuntimeError("the survey took over 120 s; too many files, or ffprobe is stuck") from None
+    if done.returncode != 0:
+        log("stuck_list_failed", level="error", rc=done.returncode, err=(done.stderr or "")[:300])
+        raise RuntimeError(f"vgstuck.sh exited {done.returncode}: {(done.stderr or '').strip()[-200:]}")
+    try:
+        items = json.loads(done.stdout or "[]")
+    except ValueError as error:
+        log("stuck_list_badjson", level="error", err=str(error), head=(done.stdout or "")[:120])
+        raise RuntimeError(f"vgstuck.sh did not return JSON: {error}") from None
+    _stuck_cache.update(at=now, items=items)
+    return items
+
+def stuck_target(src):
+    """Resolve what the browser asked us to act on, or raise. This is the only thing between a request body
+    and a file move, so it is deliberately strict: inside the streams tree, an unpublished .live recording,
+    and really there."""
+    path = Path(src).resolve()
+    root = (ROOT / "streams").resolve()
+    if root != path and root not in path.parents:
+        raise ValueError("that path is not inside the streams tree")
+    if f"{os.sep}.live{os.sep}" not in str(path) or path.suffix.lower() != ".mp4":
+        raise ValueError("that is not an unpublished recording")
+    if not path.is_file():
+        raise ValueError("no such recording (already published or already binned?)")
+    return path
+
+def stuck_action(action, src, who):
+    path = stuck_target(src)
+    done = subprocess.run([STUCK_TOOL, action, str(path)], capture_output=True, text=True, timeout=600)
+    _stuck_cache["items"] = None                    # the list just changed; the next read must re-survey
+    lines = [l for l in (done.stdout + done.stderr).splitlines() if l.strip()]
+    log(f"stuck_{action}", level="info" if done.returncode == 0 else "error",
+        file=str(path), by=who, rc=done.returncode)
+    return {"ok": done.returncode == 0, "rc": done.returncode, "output": lines[-8:]}
+
 # ---- GET /metrics -----------------------------------------------------------------------------------
 # Prometheus text for Metricbeat's prometheus module. Two sources are merged:
 #   1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/<component>.counters)
@@ -538,6 +1014,10 @@ def metrics_text():
     # 4. MediaMTX. The Prometheus scrape of :9998 already has per-path detail; this is the count the portal
     #    and the dock actually mean by "live", with the derived .webrtc audio mirrors separated out.
     paths = mtx_paths()
+    try:
+        track_sessions(webrtc_sessions())          # keeps the transport history current between page views
+    except Exception:
+        pass
     add("vanguard_mediamtx_up", "gauge", "did the MediaMTX API answer this scrape", {}, 0 if paths is None else 1)
     if paths is not None:
         ready = [p for p in paths if p.get("ready")]
@@ -580,6 +1060,11 @@ class H(BaseHTTPRequestHandler):
     def _parse(self):
         n = int(self.headers.get("Content-Length") or 0)
         body = json.loads(self.rfile.read(n) or b"{}") if n else {}
+        # ?user=all&night=YYYY-MM-DD also work (the Dump tab's dry run). GET never moves anything: a link
+        # unfurled by Discord must not be able to dump a night.
+        query = urllib.parse.parse_qs(urllib.parse.urlparse(self.path).query)
+        for key in ("night", "user"):
+            if key not in body and query.get(key): body[key] = query[key][0]
         night = body.get("night")
         if night:
             datetime.strptime(night, "%Y-%m-%d")  # validate
@@ -596,8 +1081,15 @@ class H(BaseHTTPRequestHandler):
             if pf:
                 return prev, pf
         return n, files
+    def _is_dump(self):
+        path = self.path.split("?", 1)[0]
+        return path in ("/dump", "/streaming/api/dump")       # live. dock button / diag. Dump tab
+    def _admin_door(self):
+        return self.path.split("?", 1)[0] == "/streaming/api/dump"
     def _target(self, auth_user, is_admin, req_user):
-        if is_admin:
+        # Dumping someone else's night is triage, so it is only offered where triage lives: the diag host's
+        # streaming page. The dock's Dump tonight (live. /dump) is personal for everyone, admins included.
+        if is_admin and self._admin_door():
             return req_user or auth_user          # admins default to their own night; pass "all" or a name for others
         return auth_user
     def do_GET(self):
@@ -619,6 +1111,51 @@ class H(BaseHTTPRequestHandler):
             return self._json(200, {"user": who, "groups": groups, "admin": is_admin,
                                     "name": self.headers.get("X-Authentik-Name") or who, "email": self.headers.get("X-Authentik-Email") or "",
                                     "discord_id": proxy_claim(self.headers, "discord_id")})
+        if self.path == "/me/stream-health":
+            # The dock polls this after Go Live. Only the caller's own paths, and only what is live now.
+            who, _ = identity(self.headers)
+            if not who: return self._json(401, {"error": "not authenticated"})
+            streams = []
+            try:
+                with urllib.request.urlopen(f"{MTX_API}/v3/paths/list?itemsPerPage=1000", timeout=3) as response:
+                    for item in json.load(response).get("items") or []:
+                        name = str(item.get("name", ""))
+                        if item.get("ready") and name.lower().startswith(who.lower() + "/") and not name.endswith(".webrtc"):
+                            health = stream_health(name)
+                            streams.append({"path": name, "tracks": item.get("tracks") or [], **(health or {}),
+                                            "webrtc_ok": bool(health) and health["bframes"] == 0})
+            except Exception as error:
+                return self._json(502, {"error": f"MediaMTX: {error}"})
+            # Go Live polls this while live: an empty list means the server no longer has the stream (kicked, or
+            # the session died), and "blocked" says why it must not simply reconnect.
+            hold = blocked(who.lower())
+            return self._json(200, {"user": who, "streams": streams,
+                                    "blocked": ({"until": datetime.fromtimestamp(hold["until"], TZ).isoformat(timespec="seconds"),
+                                                 "by": hold.get("by")} if hold else None)})
+        if self.path.split("?", 1)[0] == "/streaming/api/disk":
+            who, is_admin = identity(self.headers)
+            if not who: return self._json(401, {"error": "not authenticated"})
+            force = "refresh=1" in (self.path.split("?", 1) + [""])[1]
+            try:
+                return self._json(200, {"admin": is_admin, "user": who, **disk_report(force)})
+            except Exception as error:
+                return self._json(500, {"error": str(error)})
+        if self.path == "/streaming/api/live":
+            who, is_admin = identity(self.headers)
+            if not who: return self._json(401, {"error": "not authenticated"})
+            try:
+                return self._json(200, {"admin": is_admin, "user": who, **live_status()})
+            except Exception as error:
+                return self._json(500, {"error": str(error)})
+        if self.path == "/streaming/api/stuck":
+            who, is_admin = identity(self.headers)
+            if not who: return self._json(401, {"error": "not authenticated"})
+            try:
+                return self._json(200, {"admin": is_admin, "user": who, "items": stuck_list()})
+            except Exception as error:
+                # Always answer. A handler that raises sends nothing, Caddy turns that into an empty 502,
+                # and the page can only say "unexpected end of JSON" — which is exactly what happened.
+                return self._json(500, {"error": str(error)})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
             if not who: return self._json(401, {"error": "not authenticated"})
@@ -637,17 +1174,24 @@ class H(BaseHTTPRequestHandler):
                 return self._json(500, {"error": str(e)})
             # "pass" kept for the dock/portal: the token IS the password for MediaMTX
             return self._json(200, {"user": who, "pass": tok, "token": tok, "expires": exp, "ttl_hours": TOKEN_TTL_H})
-        if not self.path.startswith("/dump"):
+        if not self._is_dump():
             return self._json(404, {"error": "not found"})
         u, is_admin = identity(self.headers)
         if not u:
             return self._deny()
+        try:
             night, req_user = self._parse()
+        except Exception as e:
+            return self._json(400, {"error": str(e)})
         tgt = self._target(u, is_admin, req_user)
         night, files = self._resolve(night, tgt)
-        self._json(200, {"night": night, "user": tgt, "dry_run": True,
+        start, end = night_window(night)
-                         "files": [str(f.relative_to(ROOT)) for f in files],
+        entries = [{"path": str(f.relative_to(ROOT)), "bytes": f.stat().st_size} for f in files]
-                         "gb": round(sum(f.stat().st_size for f in files) / 1e9, 2)})
+        self._json(200, {"night": night, "user": tgt, "by": u, "admin": is_admin, "dry_run": True,
+                         "window": [start.isoformat(timespec="minutes"), end.isoformat(timespec="minutes")],
+                         "default_night": default_night(),
+                         "files": [e["path"] for e in entries], "entries": entries,
+                         "gb": round(sum(e["bytes"] for e in entries) / 1e9, 2)})
     def do_POST(self):
         if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/heartbeat"):
             # Alertmanager posts here every minute because the Watchdog alert always fires. Nothing is read from
@@ -731,7 +1275,7 @@ class H(BaseHTTPRequestHandler):
             path = (req.get("path") or "").lower()
             if path.endswith(".webrtc"):
                 # audio mirrors (<user>/<Title>.webrtc) are published by the hook inside the mediamtx container only
-                if req.get("ip") in ("127.0.0.1", "::1") and re.fullmatch(r"[a-z0-9_.-]+/[a-z0-9_.-]+\.webrtc", path):
+                if req.get("ip") in ("127.0.0.1", "::1") and re.fullmatch(r"[a-z0-9_.-]+/[a-z0-9_-]+\.webrtc", path):
                     return self._json(200, {"ok": True})
                 log("publish_deny", level="warn", stream=path, ip=req.get("ip"), reason="mirror paths are internal")
                 return self._json(401, {"error": ".webrtc paths are reserved"})
@@ -742,11 +1286,85 @@ class H(BaseHTTPRequestHandler):
             if not user:
                 log("publish_deny", level="warn", stream=path, ip=req.get("ip"), reason="bad or expired token")
                 return self._json(401, {"error": "invalid or expired stream token"})
-            if not re.fullmatch(r"[a-z0-9_.-]+/[a-z0-9_.-]+", path) or not path.startswith(user + "/"):
+            # title segment is dot-free (mirrors are <title>.webrtc and must never collide with a title)
+            if not re.fullmatch(r"[a-z0-9_.-]+/[a-z0-9_-]+", path) or not path.startswith(user + "/"):
                 log("publish_deny", level="warn", user=user, stream=path, reason="not their prefix")
                 return self._json(401, {"error": "path must be <your-username>/<Title>"})
+            hold = blocked(user.lower())
+            if hold:                                             # stopped from the Live tab: no reconnecting around it
+                log("publish_deny", level="warn", user=user, stream=path, ip=req.get("ip"), reason="blocked", by=hold.get("by"))
+                return self._json(401, {"error": "publishing is paused for this account by an admin"})
             log("publish_allow", user=user, stream=path, ip=req.get("ip"))
             return self._json(200, {"ok": True})
+        if self.path == "/streaming/api/live":
+            # {"action":"stop","path":..,"kind":..,"session":..,"minutes":N} or {"action":"unblock","user":..} — admins only
+            who, is_admin = identity(self.headers)
+            if not who: return self._deny()
+            if not is_admin:
+                return self._json(403, {"error": f"stopping streams is for {', '.join(sorted(ADMIN_GROUPS))}"})
+            try:
+                body = json.loads(self.rfile.read(int(self.headers.get("Content-Length") or 0)) or b"{}")
+            except ValueError as error:
+                return self._json(400, {"error": str(error)})
+            action = body.get("action")
+            if action == "unblock":
+                user = str(body.get("user", "")).lower()
+                if not re.fullmatch(r"[a-z0-9_.-]+", user): return self._json(400, {"error": "bad user"})
+                _live_cache["data"] = None
+                return self._json(200, {"ok": unblock_user(user, who), "user": user})
+            if action == "stop":
+                path, kind, session = str(body.get("path", "")), str(body.get("kind", "")), str(body.get("session", ""))
+                if not re.fullmatch(r"[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+", path) or kind not in KICK_ENDPOINT or not re.fullmatch(r"[A-Za-z0-9-]{8,64}", session):
+                    return self._json(400, {"error": "need path, kind and session as the live list reports them"})
+                try:
+                    minutes = max(0, min(int(body.get("minutes", 30)), 24 * 60))
+                except (TypeError, ValueError):
+                    minutes = 30
+                return self._json(200, stop_stream(path, kind, session, minutes, who))
+            return self._json(400, {"error": "action must be stop or unblock"})
+        if self.path == "/streaming/api/stuck":
+            who, is_admin = identity(self.headers)
+            if not who: return self._deny()
+            if not is_admin:
+                return self._json(403, {"error": f"retrying and binning recordings is for {', '.join(sorted(ADMIN_GROUPS))}"})
+            try:
+                body = json.loads(self.rfile.read(int(self.headers.get("Content-Length") or 0)) or b"{}")
+            except ValueError as error:
+                return self._json(400, {"error": str(error)})
+            action = body.get("action")
+            if action == "wipe":
+                # Everything unprocessed, deliberately (the Dump tab's button): stops the sweep, then every .live/
+                # file except one MediaMTX still has open, plus stale failure logs and .part leftovers. vgstuck.sh
+                # does the moving, into trash.
+                done = subprocess.run([STUCK_TOOL, "wipe"], capture_output=True, text=True, timeout=900)
+                _stuck_cache["items"] = None
+                lines = [l for l in (done.stdout + done.stderr).splitlines() if l.strip()]
+                log("stuck_wipe", level="info" if done.returncode == 0 else "error", by=who, rc=done.returncode,
+                    result=(lines[-1] if lines else ""))
+                return self._json(200, {"ok": done.returncode == 0, "rc": done.returncode, "output": lines[-6:]})
+            if action == "drop_under":
+                # Bulk: every unpublished STREAM recording smaller than N MB — reconnect stubs. Bounded so a
+                # typo cannot bin real recordings; clips never appear here because they never pass through
+                # .live/. Still-recording files are skipped by the tool itself.
+                try:
+                    mb = int(body.get("mb", 0))
+                except (TypeError, ValueError):
+                    mb = 0
+                if not 1 <= mb <= 50:
+                    return self._json(400, {"error": "mb must be between 1 and 50"})
+                done = subprocess.run([STUCK_TOOL, "drop", "--under", str(mb)], capture_output=True, text=True, timeout=900)
+                _stuck_cache["items"] = None
+                lines = [l for l in (done.stdout + done.stderr).splitlines() if l.strip()]
+                log("stuck_drop_under", level="info" if done.returncode == 0 else "error", mb=mb, by=who, rc=done.returncode)
+                return self._json(200, {"ok": done.returncode == 0, "rc": done.returncode, "output": lines[-6:]})
+            if action not in ("retry", "drop"):
+                return self._json(400, {"error": "action must be retry, drop, drop_under or wipe"})
+            try:
+                return self._json(200, stuck_action(action, str(body.get("src", "")), who))
+            except ValueError as error:
+                return self._json(400, {"error": str(error)})
+            except Exception as error:
+                return self._json(500, {"error": str(error)})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
             if not who: return self._json(401, {"error": "not authenticated"})
@@ -758,7 +1376,7 @@ class H(BaseHTTPRequestHandler):
                 return self._json(400, {"error": str(e)})
             except Exception as e:
                 return self._json(502, {"error": f"syncthing: {e}"})
-        if not self.path.startswith("/dump"):
+        if not self._is_dump():
             return self._json(404, {"error": "not found"})
         u, is_admin = identity(self.headers)
         if not u:

bin/install.list

@@ -0,0 +1,21 @@
+# What gets installed into /usr/local/bin, and with which mode. ONE list, read by install.sh,
+# configure.py --apply and deploy.sh, because three hand-kept copies drifted: a push-to-deploy was
+# silently not installing vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script.
+# check.sh fails if a file here is missing, or if something executable in bin/ is neither listed nor
+# excluded below.
+0755 bin/normalize-sweep.sh
+0755 bin/trash-purge.sh
+0755 bin/dumpd.py
+0755 bin/vgstrip.py
+0755 bin/vgstitch.py
+0755 bin/vghosts-check.py
+0755 bin/vgdeband.sh
+0755 bin/vgstuck.sh
+0755 bin/vanguard-deadman.sh
+0755 bin/vanguard-alerts.sh
+# sourced, never executed
+0644 bin/vgmetrics.sh
+0644 bin/vgnotify.sh
+# NOT installed on the host, on purpose:
+#   exclude bin/mtx-on-ready.sh        mounted into the mediamtx container (docker-compose.yml)
+#   exclude bin/vgbot-merge-sqlite.py  one-off migration, run from the checkout

bin/normalize-sweep.sh

@@ -16,6 +16,12 @@
 #   alone (zero copies); a non-compliant file is converted to name.mp4 (name.h264.mp4 if the raw is
 #   already .mp4), the raw's name is appended to .stlocal (an #include'd ignore list) so Syncthing
 #   drops it without flagging or re-pulling, then the raw is deleted. One copy on the server.
+# - reconnect fragments of ONE stream (same user, same title, fewer than STITCH_MAX_PARTS (500) files, gaps under
+#   STITCH_MAX_GAP_S) are first stitched back into one file by vgstitch.py (lossless concat; the arrival-time
+#   file names say where every piece sits), then published as one recording. Joins are recorded in the sidecar.
+#   Nothing in a stream's .live/ is touched until the stream is over (no open or recently written file).
+# - a LONE stream recording under MIN_LIVE_SECONDS (10) is a reconnect stub with nothing to join to: moved to
+#   trash, never published. Only .live/ files — clips are frequently a few seconds long and are never subject to this.
 # - failure: name.normalize-failed.log is written and the file is skipped until that log is deleted
 # Idempotent; safe every few minutes.
 set -uo pipefail
@@ -32,6 +38,18 @@ KEEP_RAW_STREAMS="${KEEP_RAW_STREAMS:-0}"
 # Encoder threads for every re-encode (clips and streams). 2 leaves the box responsive for Jellyfin/live viewers;
 # conversions take longer. systemd Environment=ENCODE_THREADS=... to change.
 ENCODE_THREADS="${ENCODE_THREADS:-2}"
+# STREAM recordings shorter than this are reconnect stubs, not recordings: every time a publisher drops and
+# comes back, MediaMTX opens a new file, and one bad night left 471 one- and two-second files in .live/.
+# They go to trash (3-day undo) instead of being published. Applies ONLY to .live/ — clips are often a few
+# seconds long on purpose and are never touched by this. 0 disables. systemd Environment=MIN_LIVE_SECONDS=…
+MIN_LIVE_SECONDS="${MIN_LIVE_SECONDS:-10}"
+# Stitching (vgstitch.py): fewer than this many fragments in one stream's .live/ are joined into one recording.
+# 500, not 20: on a lossy uplink MediaMTX restarts the recorder every ~10 s (a keyframe whose slices were lost
+# leaves an access unit with only SPS/PPS, which the recorder refuses), so a half-hour stream is ~180 same-
+# parameter files two seconds apart — exactly the case for stitching. The gap rule (STITCH_MAX_GAP_S seconds
+# between fragments = separate recordings) is the real guard. 0 parts disables stitching. systemd Environment=…
+STITCH_MAX_PARTS="${STITCH_MAX_PARTS:-500}"
+STITCH_MAX_GAP_S="${STITCH_MAX_GAP_S:-600}"
 MAX_W=2560; MAX_H=1440; MAX_FPS=60
 exec 9>"${LOCK_FILE:-/run/lock/normalize-sweep.lock}"
 flock -n 9 || exit 0
@@ -50,6 +68,11 @@ vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=sweep %s\n' "$
 log() { vglog info "msg=\"$*\""; }
 dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
 
+# A stream's fragments are one recording: join them before anything below looks at them one by one.
+if (( STITCH_MAX_PARTS > 0 )) && [[ -r "$(dirname "$0")/vgstitch.py" ]]; then        # installed side by side
+  python3 "$(dirname "$0")/vgstitch.py" "$ROOT" --max-parts "$STITCH_MAX_PARTS" --max-gap-s "$STITCH_MAX_GAP_S" --idle-min "$MINS_IDLE"
+fi
+
 # --- probe: prints "verdict reason" where verdict ∈ ok|remux|encode|skip ----------------------
 probe() {
   local f="$1" ext="${1##*.}"
@@ -63,7 +86,14 @@ probe() {
   afr=$(jq -r '[.streams[]|select(.codec_type=="video")][0].avg_frame_rate // "0/1"' <<<"$json")
   acodecs=$(jq -r '[.streams[]|select(.codec_type=="audio")|.codec_name]|unique|join(",")' <<<"$json")
   [[ $vcodec == none ]] && { echo "skip no-video"; return; }
-  local fps; fps=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$afr")
+  # Take the LOWER of r_frame_rate and avg_frame_rate. Either is unreliable alone: avg is garbage on a
+  # fragmented or looped recording (one reported 95 fps for a 30 fps source, which sent it down the fps-cap
+  # path and broke the encode), and r is a huge nominal value on genuine VFR. The lower of the two is the
+  # honest answer to "is this really above 60".
+  local fps; fps=$(awk -v rfr="$rfr" -v afr="$afr" 'BEGIN{
+      split(rfr, a, "/"); split(afr, b, "/")
+      r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
+      if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
   local reasons=()
   [[ $vcodec == h264 ]]            || reasons+=("codec=$vcodec")
   [[ $pix == yuv420p ]]            || reasons+=("pix=$pix")
@@ -108,11 +138,46 @@ while IFS= read -r -d '' src; do
     if [[ -d "$d/.stfolder" ]]; then synced=1; stroot="$d"; break; fi
     d="$(dirname "$d")"
   done
-  [[ -e "$base.normalize-failed.log" ]] && continue
+  if [[ -e "$base.normalize-failed.log" ]]; then
+    # Say so every run. Skipping silently meant a parked file produced no log line after its original
+    # failure, so Kibana could only ever show a count and never which recording. bin/vgstuck.sh triages it.
+    vglog info "event=parked file=\"$src\" stream=\"$stream\" log=\"$base.normalize-failed.log\""
+    continue
+  fi
   fuser -s "$src" 2>/dev/null && continue            # still being written
   [[ -e "$src" ]] || continue                          # dumped meanwhile
+  if (( live )); then
+    # The stream is still going while any sibling fragment is open or fresh: publishing this piece now would
+    # make it a separate recording that the stitcher can never join. Wait for the stream to end.
+    going=0
+    for sib in "$(dirname "$src")"/*.mp4; do
+      [[ -e $sib && $sib != "$src" ]] || continue
+      if fuser -s "$sib" 2>/dev/null || [[ -n $(find "$sib" -mmin -"$MINS_IDLE" 2>/dev/null) ]]; then going=1; break; fi
+    done
+    if (( going )); then vglog info "event=defer reason=\"stream still going\" file=\"$src\" stream=\"$stream\""; continue; fi
+  fi
+
+  if (( live && MIN_LIVE_SECONDS > 0 )); then
+    secs=$(dur "$src"); secs=${secs:-0}
+    if (( secs < MIN_LIVE_SECONDS )); then
+      night="$(date +%F)"; rel="${src#"$ROOT"/}"; dst="$ROOT/trash/$night/$rel"
+      mkdir -p "$(dirname "$dst")"
+      mv -f "$src" "$dst"
+      [[ -e "${src%.*}.json" ]] && mv -f "${src%.*}.json" "$(dirname "$dst")/"
+      rm -f "${src%.*}.stitch.json"
+      rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true
+      vgmet_add "jobs:discard:done:stream"
+      vglog info "event=discarded reason=too_short seconds=$secs min=$MIN_LIVE_SECONDS file=\"$src\" stream=\"$stream\" trash=\"$dst\""
+      continue
+    fi
+  fi
 
   read -r verdict reason <<<"$(probe "$src")"
+  # Keep what the PROBE said: the deband branch below overwrites $reason with its own text, which silently
+  # loses the "audio=<codec>" marker. That marker is the only thing telling the encode to convert the audio,
+  # so without this a Go Live stream with the timecode band AND Opus audio got -c:a copy and shipped Opus
+  # inside an MP4 — legal, but editors choke on it.
+  probe_reason="$reason"
   if (( live )); then
     [[ $verdict == skip ]] && continue
     [[ $verdict == ok ]] && verdict=remux          # must leave .live/ either way
@@ -173,12 +238,21 @@ PYEOF
     # scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC
     vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2"
     (( deband )) && vf="crop=iw:ih-${band}:0:0,$vf"          # drop the band rows first (band is even, so sizes stay even)
-    fpsargs=(); fpsnow=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$(ffprobe -v error -select_streams v:0 -show_entries stream=avg_frame_rate -of csv=p=0 "$src")")
+    fpsargs=()
+    read -r fps_r fps_a < <(ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,avg_frame_rate -of csv=p=0 "$src" | tr ',' ' ')
+    fpsnow=$(awk -v rfr="${fps_r:-0/1}" -v afr="${fps_a:-0/1}" 'BEGIN{
+        split(rfr, a, "/"); split(afr, b, "/")
+        r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
+        if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
     awk "BEGIN{exit !($fpsnow > $MAX_FPS + 0.01)}" && fpsargs=(-r "$MAX_FPS")
     # streams keep their exact frame timing (gaps stay gaps unless FILL_GAPS_MS asked for CFR) and their AAC audio as-is
     fpsmode=cfr; aargs=(-c:a aac -b:a 160k)
     if (( live )) && [[ $reason != fill-gaps* ]]; then fpsmode=passthrough; fi
-    if (( live )) && [[ $reason != *audio=* ]]; then aargs=(-c:a copy); fi
+    # Capping the rate means re-timing, which is by definition not passthrough. ffmpeg refuses the pair
+    # outright ("One of -r/-fpsmax was specified together a non-CFR -vsync/-fps_mode") and fails with
+    # "Error opening output files: Invalid argument", so the cap has to win.
+    (( ${#fpsargs[@]} )) && fpsmode=cfr
+    if (( live )) && [[ $probe_reason != *audio=* ]]; then aargs=(-c:a copy); fi
     nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
       -map 0:v:0 -map '0:a?' \
       -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "$ENCODE_THREADS" \
@@ -194,8 +268,19 @@ PYEOF
     (( deband )) && vgmet_add deband 1
     if (( live )); then
       mv -f "$tmp" "$out"; chown 1000:1000 "$out" 2>/dev/null
-      if [[ -n $tcjson ]]; then                        # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples
+      stitchjson="${src%.*}.stitch.json"               # vgstitch.py: parts, offsets and the time missing at each join
-        printf '%s\n' "$tcjson" > "${out%.*}.json"; chown 1000:1000 "${out%.*}.json" 2>/dev/null
+      if [[ -n $tcjson || -e $stitchjson ]]; then      # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples
+        python3 - "$tcjson" "$stitchjson" > "${out%.*}.json" <<'PYEOF'
+import json, sys, os
+tc, sj = sys.argv[1], sys.argv[2]
+d = json.loads(tc) if tc else {}
+if os.path.exists(sj):
+    d["stitched"] = json.load(open(sj))
+    # the strip decode's drift figure spans the joins too, so it is not clock drift on a stitched file
+    if "drift_ms" in d: d["drift_ms_note"] = "includes collapsed joins; see stitched.joins"
+print(json.dumps(d))
+PYEOF
+        chown 1000:1000 "${out%.*}.json" 2>/dev/null; rm -f "$stitchjson"
       fi
       if [[ $verdict == encode ]] && (( ! deband || KEEP_RAW_STREAMS )); then   # gap-filled re-encode (or asked to): keep the raw in the hidden .src/
         srcdir="$(dirname "$out")/.src"; mkdir -p "$srcdir"; chown 1000:1000 "$srcdir" 2>/dev/null

bin/vanguard-alerts.sh

@@ -18,6 +18,16 @@
 # delivered, so `systemctl status vanguard-alerts` and the journal both carry the failure.
 set -uo pipefail
 
+# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
+# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
+# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
+if [[ -r /etc/vanguard/alerts.env ]]; then
+  while IFS='=' read -r _k _v; do
+    [[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
+    [[ -n ${!_k:-} ]] || export "$_k=$_v"
+  done < /etc/vanguard/alerts.env
+fi
+
 ES="${ES_URL:-http://127.0.0.1:9200}"
 ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
 INDEX="${ALERT_INDEX:-vanguard-alerts}"
@@ -38,18 +48,27 @@ state_set() {
   mv -f "$tmp" "$STATE" 2>/dev/null || rm -f "$tmp"
 }
 
-webhook=""
+# Delivery (and the mention handling) is shared with the deadman. If the helper is missing — a half-finished
-[[ -r $WEBHOOK_FILE ]] && webhook=$(tr -d '[:space:]' < "$WEBHOOK_FILE")
+# deploy — fall back to posting without mentions rather than losing alerts entirely.
-# Same guard as the deadman: a typo that is not an https URL must not be posted to, and must not look like
+if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
-# delivery succeeded. Treated as "no webhook", which means journal-only and a non-zero exit.
+  . "$(dirname "$0")/vgnotify.sh"
-case "${webhook:-}" in
+else
-  "")       ;;
+  vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
-  https://*) ;;
+  vg_post() {
-  *)        vglog error 'event=bad_webhook msg="ALERT_WEBHOOK_URL is not an https URL; ignoring it"' >&2; webhook="" ;;
+    local url; url=$(vg_webhook) || return 1
-esac
+    curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
+      --data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
+      "$url"
+  }
+fi
+webhook=$(vg_webhook 2>/dev/null || true)
 
 now=$(date +%s)
 cursor=$(state_get cursor); cursor="${cursor:-0}"
+# Record every run, not just runs that did something. The deadman watches this file's age to tell whether
+# delivery is still happening at all, and almost every run is quiet: without this the file went stale within
+# minutes and the deadman reported the timer as dead, then recovered the moment an alert happened to flow.
+state_set last_run "$now"
 
 # Everything newer than the cursor that is not a heartbeat, oldest first so the cursor advances safely.
 query=$(printf '{"size":%d,"sort":[{"@timestamp":"asc"}],"query":{"bool":{"filter":[{"range":{"@timestamp":{"gt":%s,"format":"epoch_millis"}}}],"must_not":[{"term":{"kind":"heartbeat"}}]}}}' "$BATCH" "$cursor")
@@ -132,17 +151,12 @@ while IFS=$'\t' read -r millis rule level message; do
     state_set cursor "$millis"
     continue
   fi
-  payload=$(VG_TEXT="$text" python3 -c 'import json, os; print(json.dumps({"content": os.environ["VG_TEXT"][:1900]}))')
+  if vg_post "$level" "$text"; then
-  # curl already prints 000 on a connection failure; "|| echo 000" would concatenate and give 000000.
-  code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST -H 'Content-Type: application/json' \
-           -d "$payload" "$webhook" || true)
-  code="${code:-000}"
-  if [[ $code == 2* ]]; then
     sent=$((sent + 1)); state_set "sent:$rule" "$now"
-    vglog info "event=delivered rule=\"$rule\" level=$level"
+    vglog info "event=delivered rule=\"$rule\" level=$level mention=$([[ -n ${ALERT_MENTION:-} && ,${ALERT_MENTION_LEVELS:-error}, == *,$level,* ]] && echo yes || echo no)"
   else
     failed=$((failed + 1))
-    vglog error "event=delivery_failed rule=\"$rule\" http=$code" >&2
+    vglog error "event=delivery_failed rule=\"$rule\" level=$level" >&2
   fi
   state_set cursor "$millis"
 done <<<"$rows"

bin/vanguard-deadman.sh

@@ -24,6 +24,16 @@
 # `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
 set -uo pipefail
 
+# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
+# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
+# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
+if [[ -r /etc/vanguard/alerts.env ]]; then
+  while IFS='=' read -r _k _v; do
+    [[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
+    [[ -n ${!_k:-} ]] || export "$_k=$_v"
+  done < /etc/vanguard/alerts.env
+fi
+
 ES="${ES_URL:-http://127.0.0.1:9200}"
 # Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes
 # this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file
@@ -117,6 +127,11 @@ if [[ -f $ALERT_STATE ]]; then
   age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") ))
   (( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered"
 fi
+# Freshness above proves the timer fires; this proves the run finished. A unit that starts every minute and
+# dies immediately would otherwise look alive, because the file is written at the start of the run.
+if systemctl is-failed --quiet vanguard-alerts.service 2>/dev/null; then
+  note "vanguard-alerts.service is in a failed state — $(systemctl show -p Result --value vanguard-alerts.service 2>/dev/null); alerts are written but not delivered"
+fi
 
 # the disks, independently of any agent
 while read -r pct mount; do
@@ -130,26 +145,28 @@ was="$(cat "$STATE" 2>/dev/null || true)"
 printf '%s' "$now" > "$STATE"
 [[ $now == "$was" ]] && exit 0
 
-webhook=""
+# Delivery and mention handling are shared with vanguard-alerts.sh. A missing helper falls back to a plain
-if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi
+# post: losing the mention is acceptable, losing the alert is not.
-case "$webhook" in
+if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
-  https://*) ;;
+  . "$(dirname "$0")/vgnotify.sh"
-  "")        webhook="" ;;
+else
-  *)         echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;;
+  vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
-esac
+  vg_post() {
+    local url; url=$(vg_webhook) || return 1
+    curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
+      --data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
+      "$url"
+  }
+fi
+webhook=$(vg_webhook 2>/dev/null || true)
 
-post() {
+# A deadman alarm is the one that must wake somebody: it means no other alert can get through. Recovery is
-  [[ -n $webhook ]] || return 1
+# sent at info level so good news never pings.
-  curl -s -m 15 -H 'Content-Type: application/json' \
+post() { vg_post "${2:-error}" "$1"; }
-    --data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \
-    "$webhook" >/dev/null && return 0
-  echo "deadman: the webhook post failed" >&2
-  return 1
-}
 
 if [[ -z $now ]]; then
   echo "deadman: recovered — Elastic, Kibana, both Beats and the disks are healthy"
-  post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." || true
+  post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." info || true
   exit 0
 fi
 

bin/vghosts-check.py

@@ -0,0 +1,407 @@
+#!/usr/bin/env python3
+"""Validate a hostname change end to end: what is on disk, what is running, what the UIs were told, what DNS says.
+
+    sudo vghosts-check.py            on the server: everything
+    vghosts-check.py --repo          on a dev box: only the checks that need the repo (defaults, docs, pages)
+    sudo vghosts-check.py --json     machine-readable
+
+Source of truth is the Caddyfile (`# vanguard:<role>` markers), the same place configure.py reads the hostnames
+back from. Every other place a hostname lives is checked AGAINST it: a host that is not one of Caddy's is stale.
+"ovh1.<base>" is the box's ssh name and must appear nowhere a browser is sent.
+
+ORDER MATTERS, and the sections below run in the order a change must be made, because each depends on the one
+before it (HOSTNAME-CHANGE.md, section E):
+   1 DNS           nothing can be issued a certificate or reached until the name resolves to the box
+   2 Caddyfile     the roles and their names: what configure.py --full / --hosts wrote; certificates follow
+   3 .env          the same names as compose variables; a mismatch here means a half-done --full
+   4 mediamtx.yml  webrtcAdditionalHosts = main host (WebRTC ICE); needs a MediaMTX restart to take effect
+   5 containers    the env each running container was CREATED with (a .env edit without a recreate shows here)
+   6 HTTPS         every host answers with a valid certificate; gated hosts send browsers to the auth host
+   7 Jellyfin      SSO plugin OID endpoint, branding disclaimer, Vanguard plugin URLs (on disk) + the SSO start
+   8 Gitea         app.ini, the auth source (live redirect), the runner, the server clone's remote
+   9 Authentik     outpost host, proxy provider external host + cookie domain, Jellyfin/Gitea redirect URIs.
+                   LAST among the services, on purpose: an app changed before Authentik is merely unreachable on
+                   its new name until Authentik catches up; Authentik changed before the app locks you out of it
+  10 Discord       the OAuth2 redirect in the developer portal — cannot be read; the exact value is printed
+  11 repo          deployLive.sh / switchToClone.sh defaults, about page labels = Caddy roles, docs
+  12 streamers     OBS dock URL and clock source: cannot be read; listed so nobody forgets
+A failure in an early section usually explains failures in later ones: fix top-down, re-run. The sweep runs the
+Jellyfin and Gitea checks before Authentik, and the report lists them that way, because that is the order to make
+the change in.
+"""
+import argparse, json, os, re, socket, ssl, subprocess, sys, urllib.request, urllib.error
+from pathlib import Path
+
+ROLES = ("main", "auth", "watch", "git", "bot", "live", "files", "sync", "clips", "diag")
+ENV_KEY = {"main": "MAIN_HOST", "watch": "WATCH_HOST", "git": "GIT_HOST", "bot": "BOT_HOST", "live": "LIVE_HOST",
+           "files": "FILES_HOST", "sync": "SYNC_HOST", "clips": "CLIPS_HOST", "diag": "DIAG_HOST"}
+GATED = ("main", "files", "sync", "clips", "live", "bot", "diag")            # forward-auth: anonymous -> auth host
+REPO = Path(os.environ.get("VANGUARD_REPO", str(Path.home() / "vanguard-stream") if Path.home().joinpath("vanguard-stream").is_dir() else "."))
+APPDATA = Path("/srv/appdata")
+
+results = []            # (section, status, text)
+def rec(section, status, text): results.append((section, status, text))
+def ok(section, text): rec(section, "PASS", text)
+def bad(section, text): rec(section, "FAIL", text)
+def warn(section, text): rec(section, "WARN", text)
+def manual(section, text): rec(section, "MANUAL", text)
+
+def sh(cmd, timeout=30, **kw):
+    try:
+        return subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, **kw)
+    except (OSError, subprocess.TimeoutExpired) as error:
+        class R: returncode, stdout, stderr = 1, "", str(error)
+        return R()
+
+def read_hosts(caddyfile):
+    hosts, role = {}, None
+    for line in caddyfile.read_text().splitlines():
+        if line[:1].isspace(): continue
+        t = line.strip()
+        m = re.match(r"#\s*vanguard:(\w+)", t)
+        if m: role = m.group(1); continue
+        if t and not t.startswith(("#", "}", "{")) and t.endswith("{") and " " in t:
+            host = t.split()[0].rstrip(",")          # "a, b {": an alias; the first name is the role's
+            if "example.com" in host or role == "redirect": role = None; continue
+            hosts[role or ("main" if "main" not in hosts else host)] = host
+            role = None
+    return hosts
+
+def read_env(path):
+    env = {}
+    try:
+        for line in path.read_text().splitlines():
+            if "=" in line and not line.startswith("#"):
+                k, v = line.split("=", 1); env[k.strip()] = v.strip()
+    except OSError:
+        pass
+    return env
+
+URL_RE = re.compile(r"https?://([A-Za-z0-9.-]+\.[A-Za-z]{2,})(?::\d+)?")
+def hosts_in(text):
+    return sorted(set(h.lower() for h in URL_RE.findall(text)))
+
+def classify(section, where, found, allowed, expect=None, base=""):
+    """found: hostnames seen in `where`. allowed: every current host. expect: the one role's host, if specific."""
+    if not found:
+        warn(section, f"{where}: no hostname found"); return
+    for h in found:
+        if expect and h == expect: ok(section, f"{where}: {h}")
+        elif expect and h != expect and (h.endswith("." + base) or h.startswith("ovh1.")):
+            bad(section, f"{where}: {h} — expected {expect}")
+        elif h in allowed: ok(section, f"{where}: {h}")
+        elif base and (h.endswith("." + base) or h == base or h.startswith("ovh1.")):
+            bad(section, f"{where}: {h} — not a current host (stale)")
+        else:
+            pass    # third-party hostnames (discord.com, github.com, …) are not ours to judge
+
+class _NoRedirect(urllib.request.HTTPRedirectHandler):
+    def redirect_request(self, *a, **k): return None
+_opener = urllib.request.build_opener(_NoRedirect)
+
+def curl_head(url, timeout=12):
+    """(status, location, error) for a browser-like GET, without following redirects. GET, not HEAD: the
+    forward-auth gate answers HEAD with 200 and the SSO start endpoints are GET-only."""
+    req = urllib.request.Request(url, method="GET", headers={"User-Agent": "Mozilla/5.0 vghosts-check", "Accept": "text/html,*/*"})
+    try:
+        with _opener.open(req, timeout=timeout) as r:
+            r.read(2048)
+            return r.status, r.headers.get("Location", ""), ""
+    except urllib.error.HTTPError as e:
+        return e.code, e.headers.get("Location", ""), ""
+    except Exception as e:
+        return 0, "", str(e)
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--repo", action="store_true", help="repo-only checks (dev box)")
+    ap.add_argument("--json", action="store_true")
+    ap.add_argument("--caddyfile", default=None, help="path to the generated Caddyfile (default: <repo>/Caddyfile)")
+    args = ap.parse_args()
+    repo = REPO
+    caddyfile = Path(args.caddyfile) if args.caddyfile else repo / "Caddyfile"
+
+    # ---- 2 Caddyfile: the truth -------------------------------------------------------------------
+    S = "2 Caddyfile"
+    if not caddyfile.is_file():
+        if args.repo:
+            hosts = {}
+            warn(S, f"no Caddyfile at {caddyfile} (dev box): repo checks use the names in deployLive.sh / switchToClone.sh")
+        else:
+            bad(S, f"no Caddyfile at {caddyfile}: nothing to validate against — run configure.py --full first"); return report(args)
+    else:
+        hosts = read_hosts(caddyfile)
+    main_host = hosts.get("main", "")
+    base = main_host.split(".", 1)[1] if "." in main_host else ""
+    allowed = set(hosts.values())
+    for role in (ROLES if hosts else ()):
+        if role in hosts: ok(S, f"{role:6} {hosts[role]}")
+        else: warn(S, f"{role:6} no site block (configure.py --hosts to add one)")
+    if base and f"ovh1.{base}" in allowed: bad(S, f"ovh1.{base} still has a site block")
+
+    # ---- 1 DNS -------------------------------------------------------------------------------------
+    S = "1 DNS"
+    if hosts and not args.repo:
+        ips = {}
+        for role, h in hosts.items():
+            try: ips[h] = sorted(set(a[4][0] for a in socket.getaddrinfo(h, 443, proto=socket.IPPROTO_TCP)))
+            except OSError as e: bad(S, f"{h}: does not resolve ({e})")
+        role_ips = {h: v for r, h in hosts.items() for hh, v in ips.items() if hh == h and r in ROLES}
+        distinct = set(tuple(v) for v in role_ips.values())
+        for h, v in ips.items():
+            if h in role_ips: (ok if len(distinct) == 1 else bad)(S, f"{h} -> {', '.join(v)}")
+            else: warn(S, f"{h} -> {', '.join(v)} (apex/other block: a redirect may live anywhere, but Caddy must hold its certificate to answer)")
+        if len(distinct) > 1: bad(S, "the role hosts resolve to different addresses — a CNAME points elsewhere")
+        if base:
+            try:
+                old = sorted(set(a[4][0] for a in socket.getaddrinfo(f"ovh1.{base}", 22, proto=socket.IPPROTO_TCP)))
+                ok(S, f"ovh1.{base} -> {', '.join(old)} (ssh only; fine to keep)")
+            except OSError: warn(S, f"ovh1.{base} no longer resolves — ssh by IP or another name")
+
+    # ---- 3 .env --------------------------------------------------------------------------------------
+    S = "3 .env"
+    env = read_env(repo / ".env")
+    if not args.repo:
+        if not env: bad(S, f"{repo / '.env'} unreadable (run with sudo, from the checkout)")
+        for role, key in ENV_KEY.items():
+            if role not in hosts: continue
+            v = env.get(key, "")
+            if v == hosts[role]: ok(S, f"{key}={v}")
+            elif not v: bad(S, f"{key} unset — expected {hosts[role]}")
+            else: bad(S, f"{key}={v} — Caddyfile says {hosts[role]}")
+        apex, apex_to = env.get("APEX_HOST", ""), env.get("APEX_REDIRECT", "")
+        if apex:
+            found = hosts_in(apex_to); classify(S, f"APEX_REDIRECT ({apex} ->)", found, allowed, base=base)
+        else: ok(S, "APEX_HOST empty (bare domain not served)")
+
+    # ---- 4 mediamtx.yml ------------------------------------------------------------------------------
+    S = "4 mediamtx.yml"
+    if not args.repo:
+        mtx = repo / "mediamtx.yml"
+        if mtx.is_file():
+            m = re.search(r"^webrtcAdditionalHosts:\s*\[([^\]]*)\]", mtx.read_text(), re.M)
+            got = [x.strip().strip("'\"") for x in (m.group(1).split(",") if m else []) if x.strip()]
+            if main_host in got: ok(S, f"webrtcAdditionalHosts contains {main_host}")
+            else: bad(S, f"webrtcAdditionalHosts = {got or 'missing'} — expected {main_host} (then: docker compose restart mediamtx, between streams)")
+        else: bad(S, "mediamtx.yml missing from the checkout")
+
+    # ---- 5 running containers ------------------------------------------------------------------------
+    S = "5 containers"
+    if not args.repo:
+        want = {"jellyfin": ("JELLYFIN_PublishedServerUrl", "watch"), "gitea": ("GITEA__server__ROOT_URL", "git"),
+                "act_runner": ("GITEA_INSTANCE_URL", "git"), "discord-bot": ("WEB_PUBLIC_URL", "bot"),
+                "mediamtx": ("LIVE_HOST", "live"), "kibana": ("DIAG_HOST", "diag")}
+        for container, (var, role) in want.items():
+            if role not in hosts: continue
+            r = sh(["docker", "inspect", "-f", "{{range .Config.Env}}{{println .}}{{end}}", container])
+            if r.returncode != 0: warn(S, f"{container}: not running or no such container"); continue
+            val = next((l.split("=", 1)[1] for l in r.stdout.splitlines() if l.startswith(var + "=")), None)
+            if val is None: warn(S, f"{container}: {var} not in its environment"); continue
+            if hosts[role] in val: ok(S, f"{container}: {var}={val}")
+            else: bad(S, f"{container}: {var}={val} — expected {hosts[role]}; the container was created before .env changed: docker compose up -d {container}")
+        gd = sh(["docker", "inspect", "-f", "{{range .Config.Env}}{{println .}}{{end}}", "gitea"])
+        for var in ("GITEA__server__DOMAIN", "GITEA__server__SSH_DOMAIN"):
+            val = next((l.split("=", 1)[1] for l in gd.stdout.splitlines() if l.startswith(var + "=")), None)
+            if val and "git" in hosts: (ok if val == hosts["git"] else bad)(S, f"gitea: {var}={val}" + ("" if val == hosts["git"] else f" — expected {hosts['git']}"))
+
+    # ---- 6 HTTPS -------------------------------------------------------------------------------------
+    S = "6 HTTPS"
+    if hosts and not args.repo:
+        auth = hosts.get("auth", "")
+        for role, h in hosts.items():
+            status, loc, err = curl_head(f"https://{h}/")
+            if err:
+                if role in ROLES: bad(S, f"{h}: {err[:90]} (no certificate yet? Caddy block missing? DNS?)")
+                else: warn(S, f"{h}: {err[:90]} — the bare domain's A record does not reach this box, so its Caddy block cannot get a certificate; fix the record or clear APEX_HOST")
+                continue
+            loc_host = (hosts_in(loc) or [""])[0]
+            if role in GATED:
+                if status in (301, 302, 303, 307, 308) and loc_host == auth: ok(S, f"{h}: {status} -> {auth} (gate)")
+                elif status in (301, 302, 303, 307, 308): bad(S, f"{h}: {status} -> {loc} — gate sends to the wrong host")
+                else: warn(S, f"{h}: {status} without a redirect to {auth} — is the forward-auth gate on?")
+            else:
+                (ok if status < 400 else bad)(S, f"{h}: {status}{(' -> ' + loc) if loc else ''}")
+        # a move in progress: aliases must answer like the primary names; old names must redirect to the new ones
+        alt, red = env.get("ALT_BASE", "").strip().lower(), env.get("REDIRECT_BASE", "").strip().lower()
+        for role, h in hosts.items():
+            if role not in ROLES or not h.endswith("." + base): continue
+            if alt and alt != base:
+                a = h[: -len(base)] + alt
+                status, loc, err = curl_head(f"https://{a}/")
+                (bad if err else ok)(S, f"alias {a}: {err[:80] if err else status}")
+            if red and red != base:
+                o = h[: -len(base)] + red
+                status, loc, err = curl_head(f"https://{o}/")
+                if err: warn(S, f"old name {o}: {err[:80]} (its certificate may still be issuing)")
+                elif status == 308 and (hosts_in(loc) or [""])[0] == h: ok(S, f"old name {o}: 308 -> {h}")
+                else: bad(S, f"old name {o}: {status} -> {loc or '(no Location)'} — expected 308 to {h}")
+        if base:
+            status, loc, err = curl_head(f"https://ovh1.{base}/", timeout=8)
+            if err: ok(S, f"ovh1.{base}: no https (expected: ssh only)")
+            else: bad(S, f"ovh1.{base} still answers https ({status}) — a Caddy block or another server is serving the old name")
+
+    # ---- 7 Authentik ----------------------------------------------------------------------------------
+    S = "9 Authentik"
+    if not args.repo and hosts:
+        AK = r'''
+import json
+from urllib.parse import urlparse
+from authentik.outposts.models import Outpost
+from authentik.providers.proxy.models import ProxyProvider
+from authentik.providers.oauth2.models import OAuth2Provider
+from authentik.sources.oauth.models import OAuthSource
+from authentik.brands.models import Brand
+out = {"outposts": [], "proxy": [], "oauth2": [], "sources": [], "brands": []}
+for o in Outpost.objects.all():
+    out["outposts"].append({"name": o.name, "authentik_host": (o.config or {}).get("authentik_host", "") if isinstance(o.config, dict) else getattr(o.config, "authentik_host", "")})
+for p in ProxyProvider.objects.all():
+    out["proxy"].append({"name": p.name, "external_host": p.external_host, "cookie_domain": p.cookie_domain, "mode": str(p.mode)})
+for p in OAuth2Provider.objects.all():
+    uris = []
+    try:
+        uris = [u.url for u in p.redirect_uris]
+    except Exception:
+        try: uris = [u.get("url", "") for u in (p._redirect_uris or [])]
+        except Exception: uris = [str(getattr(p, "redirect_uris", ""))]
+    out["oauth2"].append({"name": p.name, "redirect_uris": uris})
+for s in OAuthSource.objects.all():
+    out["sources"].append({"name": s.name, "slug": s.slug, "provider_type": s.provider_type})
+for b in Brand.objects.all():
+    out["brands"].append({"domain": b.domain, "default": b.default, "logo": str(b.branding_logo), "default_app": str(b.default_application) if b.default_application_id else ""})
+print("VGHOSTS " + json.dumps(out))
+'''
+        r = sh(["docker", "exec", "-i", "authentik-server", "ak", "shell", "-c", "import sys; exec(sys.stdin.read())"], input=AK, timeout=120)
+        line = next((l for l in (r.stdout + r.stderr).splitlines() if l.startswith("VGHOSTS ")), None)
+        if not line:
+            bad(S, "could not read Authentik (" + (r.stderr.strip().splitlines() or ["no output"])[-1][:120] + ")")
+        else:
+            ak = json.loads(line[8:])
+            auth = hosts.get("auth", "")
+            for o in ak["outposts"]:
+                h = hosts_in(o["authentik_host"])
+                if h == [auth]: ok(S, f"outpost {o['name']!r}: authentik_host {o['authentik_host']}")
+                elif not o["authentik_host"]: warn(S, f"outpost {o['name']!r}: authentik_host unset (the embedded outpost then uses its own address; set it to https://{auth})")
+                else: bad(S, f"outpost {o['name']!r}: authentik_host {o['authentik_host']} — expected https://{auth}")
+            for p in ak["proxy"]:
+                h = hosts_in(p["external_host"])
+                classify(S, f"proxy provider {p['name']!r} external_host", h, allowed, base=base)
+                cd = p["cookie_domain"]
+                if "domain" in p["mode"].lower():
+                    (ok if cd == base else bad)(S, f"proxy provider {p['name']!r} cookie_domain={cd or '(unset)'}" + ("" if cd == base else f" — expected {base}; sessions will not span the hosts"))
+            for p in ak["oauth2"]:
+                expect = hosts.get("watch") if "jellyfin" in p["name"].lower() else hosts.get("git") if "gitea" in p["name"].lower() else None
+                for u in p["redirect_uris"]:
+                    classify(S, f"OAuth2 provider {p['name']!r} redirect", hosts_in(u), allowed, expect=expect, base=base)
+                if not p["redirect_uris"]: warn(S, f"OAuth2 provider {p['name']!r}: no redirect URIs")
+            for b in ak["brands"]:
+                if b["domain"] in ("", "authentik-default"): ok(S, f"brand domain {b['domain']!r} (default)")
+                elif b["domain"] == base or b["domain"].endswith(base): ok(S, f"brand domain {b['domain']}")
+                else: warn(S, f"brand domain {b['domain']} — not under {base}")
+            # ---- 8 Discord --------------------------------------------------------------------------
+            for s in ak["sources"]:
+                if s["provider_type"] == "discord":
+                    manual("10 Discord", f"developer portal → OAuth2 → Redirects must contain exactly: https://{auth}/source/oauth/callback/{s['slug']}/")
+            if not any(s["provider_type"] == "discord" for s in ak["sources"]):
+                bad("10 Discord", "no Discord OAuth source in Authentik")
+
+    # ---- 9 Jellyfin -----------------------------------------------------------------------------------
+    S = "7 Jellyfin"
+    if not args.repo and hosts:
+        jf = APPDATA / "jellyfin" / "config"
+        if jf.is_dir():
+            for xml in sorted(jf.rglob("*.xml")):
+                try: text = xml.read_text(errors="replace")
+                except OSError: continue
+                found = [h for h in hosts_in(text) if base and (h.endswith("." + base) or h == base or h.startswith("ovh1."))]
+                if not found: continue
+                name = xml.name.lower()
+                expect = hosts.get("auth") if "sso" in name else None
+                classify(S, f"{xml.relative_to(jf)}", found, allowed, expect=expect, base=base)
+        else: warn(S, f"{jf} not found")
+        if "watch" in hosts:
+            status, loc, err = curl_head(f"https://{hosts['watch']}/sso/OID/start/authentik")
+            lh = (hosts_in(loc) or [""])[0]
+            if err: bad(S, f"SSO start: {err[:80]}")
+            elif status in (301, 302, 303, 307) and lh == hosts.get("auth"): ok(S, f"SSO start -> {hosts['auth']} (plugin OID endpoint is right)")
+            else: bad(S, f"SSO start answered {status} -> {loc or '(no Location)'} — SSO-Auth plugin OID endpoint should be https://{hosts.get('auth')}/application/o/jellyfin/")
+
+    # ---- 10 Gitea ---------------------------------------------------------------------------------------
+    S = "8 Gitea"
+    if not args.repo and hosts and "git" in hosts:
+        ini = APPDATA / "gitea" / "gitea" / "conf" / "app.ini"
+        if ini.is_file():
+            text = ini.read_text(errors="replace")
+            for key in ("ROOT_URL", "DOMAIN", "SSH_DOMAIN"):
+                m = re.search(rf"^{key}\s*=\s*(\S+)", text, re.M)
+                v = m.group(1) if m else ""
+                (ok if hosts["git"] in v else bad)(S, f"app.ini {key}={v or '(missing)'}" + ("" if hosts["git"] in v else f" — expected {hosts['git']} (compose env sets it; recreate gitea)"))
+        else: warn(S, f"{ini} not found")
+        status, loc, err = curl_head(f"https://{hosts['git']}/user/oauth2/authentik")
+        lh = (hosts_in(loc) or [""])[0]
+        if err: bad(S, f"auth source: {err[:80]}")
+        elif lh == hosts.get("auth"): ok(S, f"auth source 'authentik' -> {hosts['auth']} (discovery URL is right)")
+        else: bad(S, f"auth source: {status} -> {loc or '(no Location)'} — Site administration → Authentication sources → authentik → discovery URL https://{hosts.get('auth')}/application/o/gitea/.well-known/openid-configuration")
+        runner = Path("/srv/appdata/act_runner/.runner")
+        if runner.is_file():
+            classify(S, "runner registration (.runner)", hosts_in(runner.read_text(errors="replace")), allowed, expect=hosts["git"], base=base)
+        else: warn(S, "runner: /srv/appdata/act_runner/.runner not found — ./registerGiteaRunner.sh")
+        r = sh(["git", "-C", str(repo), "remote", "get-url", "origin"])
+        if r.returncode == 0:
+            url = r.stdout.strip(); m = re.search(r"@([A-Za-z0-9.-]+)", url)
+            h = (m.group(1).lower() if m else "")
+            (ok if h == hosts["git"] else bad)(S, f"server clone remote: {url}" + ("" if h == hosts["git"] else f" — expected {hosts['git']}"))
+        else: warn(S, f"{repo}: not a git clone (deploy.sh pull needs one)")
+        for cfg in (Path.home() / ".ssh" / "config.vanguard", Path("/home/debian/.ssh/config.vanguard")):
+            if cfg.is_file():
+                classify(S, f"{cfg}", [h for h in re.findall(r"(?im)^\s*host(?:name)?\s+(\S+)", cfg.read_text()) if "." in h], allowed, expect=hosts["git"], base=base); break
+
+    # ---- 11 repo ----------------------------------------------------------------------------------------
+    S = "11 repo"
+    for script, var in (("deployLive.sh", "HOST"), ("switchToClone.sh", "REPO_SSH")):
+        p = repo / script
+        if not p.is_file(): warn(S, f"{script} missing"); continue
+        m = re.search(rf"^{var}=.*", p.read_text(), re.M)
+        found = [h for h in re.findall(r"@([A-Za-z0-9.-]+\.[a-z]{2,})", m.group(0))] if m else []
+        if hosts: classify(S, f"{script} {var}", [h.lower() for h in found], allowed, base=base)
+        else: ok(S, f"{script} {var}: {', '.join(found) or '(none)'}")
+    about = repo / "about" / "index.html"
+    if about.is_file():
+        labels = sorted(set(re.findall(r"sub\('([a-z]+)'\)", about.read_text())))
+        if hosts:
+            for label in labels:
+                (ok if label in hosts else bad)(S, f"about page label '{label}'" + ("" if label in hosts else " — no such Caddy role; the gateway link will 404/TLS-fail"))
+        else: ok(S, f"about page labels: {', '.join(labels)}")
+    for doc in ("README.md", "FRESH-INSTALL.md", "STATUS.md", "LOGGING.md"):
+        p = repo / doc
+        if not p.is_file(): continue
+        stale = [(n, l.strip()) for n, l in enumerate(p.read_text().splitlines(), 1)
+                 if re.search(r"https?://ovh1\.|ovh1\.[a-z.]+/", l) and not re.search(r"ssh|scp|ssh-only|ssh only", l)]
+        if stale: bad(S, f"{doc}: {len(stale)} line(s) send a browser to ovh1 — first: {stale[0][0]}: {stale[0][1][:80]}")
+        else: ok(S, f"{doc}: no browser links to ovh1")
+    plan = repo / "CommunityDiscordMedia.md"
+    if plan.is_file() and re.search(r"https?://ovh1\.", plan.read_text()):
+        warn(S, "CommunityDiscordMedia.md still describes ovh1 as a web host (historical plan; mark the lines as superseded or leave)")
+
+    # ---- 12 streamers -------------------------------------------------------------------------------
+    if hosts.get("live"):
+        manual("12 streamers", f"OBS → Docks → Custom Browser Docks: https://{hosts['live']}/dock/ ; dock Settings re-saved; Timecode band → \"Add band + clock source\" again; manual RTMP/SRT configs re-copied from the dock")
+        manual("12 streamers", f"Jellyfin apps on phones/TVs: server https://{hosts.get('watch','')} ; Discord pins: https://{main_host}/ (members), https://{hosts.get('clips','')}/ (gamers), https://{hosts.get('files','')}/ (editors)")
+    return report(args)
+
+def report(args):
+    order = lambda s: int(s.split()[0])
+    sections = sorted(set(s for s, _, _ in results), key=order)
+    if args.json:
+        print(json.dumps([{"section": s, "status": st, "text": t} for s, st, t in results], indent=1)); return 0
+    mark = {"PASS": "  ok  ", "FAIL": " FAIL ", "WARN": " warn ", "MANUAL": " todo "}
+    for s in sections:
+        print(f"\n== {s}")
+        for sec, st, t in results:
+            if sec == s: print(f" [{mark[st]}] {t}")
+    fails = sum(1 for _, st, _ in results if st == "FAIL"); warns = sum(1 for _, st, _ in results if st == "WARN")
+    todo = sum(1 for _, st, _ in results if st == "MANUAL")
+    print(f"\n{fails} failed, {warns} warnings, {todo} to check by hand. Fix top-down: an early failure explains the later ones.")
+    return 1 if fails else 0
+
+if __name__ == "__main__":
+    sys.exit(main())

bin/vgnotify.sh

@@ -0,0 +1,74 @@
+# Shared Discord delivery for bin/vanguard-alerts.sh and bin/vanguard-deadman.sh. Sourced, never executed.
+#
+#   . /usr/local/bin/vgnotify.sh
+#   vg_post error "disk is nearly full"     # 0 = delivered, 1 = not (no webhook, bad URL, or HTTP failure)
+#
+# WHY MENTIONS NEED allowed_mentions
+# Putting <@&ROLE_ID> in a webhook's content is not enough on its own: Discord will render it but will not
+# reliably ping, because a webhook has no permission to mention roles unless the request says which mentions
+# are allowed. So every post sends an explicit allowed_mentions. It also sets parse: [] always, which means
+# an @everyone appearing inside an alert's text (a log line, a filename) can never ping the whole server.
+#
+# ALERT_MENTION is written for humans rather than for Discord's syntax:
+#   ALERT_MENTION='role:1387247501387628585,role:1566623319027880086'   -> pings those roles
+#   ALERT_MENTION='user:123456789'                                 -> pings that person
+# Raw '<@&123>' / '<@123>' is accepted too and passed through unchanged.
+#
+# ALERT_MENTION_LEVELS decides which alarms are worth a ping; the default is errors only, so warnings still
+# arrive in the channel without waking anyone.
+
+VG_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
+VG_MENTION="${ALERT_MENTION:-}"
+VG_MENTION_LEVELS="${ALERT_MENTION_LEVELS:-error}"
+
+# The webhook URL, or nothing. A value that is not https is treated as absent and complained about once:
+# posting to it would fail anyway, and silently "succeeding" would be worse.
+vg_webhook() {
+  local url
+  [[ -r $VG_WEBHOOK_FILE ]] || return 1
+  url=$(tr -d '[:space:]' < "$VG_WEBHOOK_FILE" 2>/dev/null) || return 1
+  case "$url" in
+    https://*) printf '%s' "$url" ;;
+    "")        return 1 ;;
+    *)         echo "vgnotify: the configured webhook is not an https URL; ignoring it" >&2; return 1 ;;
+  esac
+}
+
+vg_post() {
+  local level="$1" text="$2" url payload code
+  url=$(vg_webhook) || return 1
+  payload=$(VG_LEVEL="$level" VG_TEXT="$text" VG_MENTION="$VG_MENTION" VG_LEVELS="$VG_MENTION_LEVELS" python3 - <<'PY'
+import json, os, re
+level   = os.environ.get("VG_LEVEL", "")
+text    = os.environ.get("VG_TEXT", "")
+raw     = os.environ.get("VG_MENTION", "").strip()
+levels  = [l.strip() for l in os.environ.get("VG_LEVELS", "error").split(",") if l.strip()]
+
+users, roles, parts = [], [], []
+for entry in [e.strip() for e in raw.replace(" ", ",").split(",") if e.strip()]:
+    if entry.startswith("role:"):
+        roles.append(entry[5:]); parts.append(f"<@&{entry[5:]}>")
+    elif entry.startswith("user:"):
+        users.append(entry[5:]); parts.append(f"<@{entry[5:]}>")
+    else:                                   # already Discord syntax
+        parts.append(entry)
+        roles += re.findall(r"<@&(\d+)>", entry)
+        users += re.findall(r"<@!?(\d+)>", entry)
+
+ping = bool(parts) and level in levels
+prefix = (" ".join(parts) + " ") if ping else ""
+# parse: [] unconditionally — an @everyone inside alert text must never ping the server.
+body = {"content": (prefix + text)[:1900],
+        "allowed_mentions": {"parse": [],
+                             "users": sorted(set(users)) if ping else [],
+                             "roles": sorted(set(roles)) if ping else []}}
+print(json.dumps(body))
+PY
+)
+  code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST \
+           -H 'Content-Type: application/json' -d "$payload" "$url" || true)
+  case "${code:-000}" in
+    2*) return 0 ;;
+    *)  echo "vgnotify: webhook post failed, HTTP ${code:-000}" >&2; return 1 ;;
+  esac
+}

bin/vgstitch.py

@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+"""Stitch reconnect fragments of one stream back into one recording (run by normalize-sweep.sh).
+
+Every time a publisher drops and comes back, MediaMTX opens a new file under streams/<user>/<Title>/.live/,
+so a night with a flaky uplink leaves one stream as a row of fragments: 20 min, 2 s, 14 min, 1 s, ... The
+fragments are the SAME stream (same user, same title) and the file names carry the server arrival time to
+the microsecond, so we know exactly where each piece sits on the clock. This joins them losslessly:
+
+  * only when the stream is over: nothing in .live/ open (fuser) or modified in the last IDLE_MIN minutes
+  * only fewer than MAX_PARTS fragments (default 500 — a lossy uplink splits a half-hour stream into ~180
+    files, see normalize-sweep.sh): beyond that is a broken night, left for vgstuck.sh / the streaming page
+  * a chain breaks where the gap between fragments is longer than MAX_GAP_S (default 10 min: a reconnect
+    is seconds, a second session the same evening is not), or where the video/audio parameters change
+    (resolution, codec, rate): those cannot be copy-concatenated, so each side is stitched on its own
+  * ffmpeg concat demuxer with -c copy — no re-encode, the sweep's own pass (timecode, band removal,
+    Opus->AAC) runs on the result as on any other recording
+  * the gaps between fragments are COLLAPSED (the picture jumps), and recorded in <stem>.stitch.json which
+    the sweep merges into the published sidecar: part files, each part's arrival time, its offset in the
+    stitched file, and the wall-clock time missing at every join
+  * the stitched file takes the FIRST fragment's name (so the arrival-time fallback for the timecode still
+    points at the real start), the fragments move to trash/<night>/ for the usual 3-day undo
+  * a fragment ffprobe cannot read is binned (reason=unreadable) rather than blocking its stream forever
+
+Prints logfmt lines (component=sweep) for Kibana. Exit 0 unless the arguments are wrong.
+    vgstitch.py <media root> [--max-parts N] [--max-gap-s S] [--idle-min M] [--dry-run]
+"""
+import argparse, json, os, re, subprocess, sys, time
+from datetime import datetime
+from pathlib import Path
+from zoneinfo import ZoneInfo
+
+TZ = ZoneInfo("America/New_York")
+NAME_RE = re.compile(r"(\d{4})-(\d{2})-(\d{2})_(\d{2})-(\d{2})-(\d{2})(?:-(\d{1,6}))?")
+
+def vglog(level, **kv):
+    parts = [f"ts={datetime.now().astimezone().isoformat(timespec='seconds')}", f"level={level}", "component=sweep"]
+    for k, v in kv.items():
+        v = str(v)
+        parts.append(f'{k}="{v}"' if (" " in v or "/" in v or v == "") else f"{k}={v}")
+    print(" ".join(parts), flush=True)
+
+def arrival_ms(stem):
+    m = NAME_RE.search(stem)
+    if not m: return None
+    y, mo, d, h, mi, s, us = m.groups()
+    us = int((us or "0").ljust(6, "0"))
+    return int(datetime(int(y), int(mo), int(d), int(h), int(mi), int(s), us, tzinfo=TZ).timestamp() * 1000)
+
+def ffprobe(path):
+    """(params tuple, duration_s) or None when unreadable."""
+    try:
+        out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
+                              "stream=codec_type,codec_name,width,height,pix_fmt,sample_rate,channels:format=duration",
+                              "-of", "json", str(path)], capture_output=True, text=True, timeout=120, check=True).stdout
+        j = json.loads(out)
+        v = [s for s in j["streams"] if s["codec_type"] == "video"]
+        a = [s for s in j["streams"] if s["codec_type"] == "audio"]
+        if not v: return None
+        dur = float(j["format"].get("duration") or 0)
+        if dur <= 0: return None
+        params = (v[0].get("codec_name"), v[0].get("width"), v[0].get("height"), v[0].get("pix_fmt"),
+                  tuple((s.get("codec_name"), s.get("sample_rate"), s.get("channels")) for s in a))
+        return params, dur
+    except Exception:
+        return None
+
+def busy(path):
+    return subprocess.run(["fuser", "-s", str(path)], capture_output=True).returncode == 0
+
+def to_trash(root, path, night):
+    rel = path.relative_to(root)
+    dst = root / "trash" / night / rel
+    dst.parent.mkdir(parents=True, exist_ok=True)
+    os.replace(path, dst)
+    return dst
+
+def stitch_dir(root, live, args):
+    titledir = live.parent
+    stream = f"{titledir.parent.name}/{titledir.name}"
+    files = sorted(p for p in live.iterdir() if p.is_file() and p.suffix.lower() == ".mp4" and not p.name.endswith(".part"))
+    if len(files) < 2: return
+    now = time.time()
+    for p in files:                                   # still streaming? leave the whole set alone
+        if busy(p) or now - p.stat().st_mtime < args.idle_min * 60:
+            vglog("info", event="stitch_wait", stream=stream, parts=len(files), reason="stream still going")
+            return
+    if len(files) >= args.max_parts:
+        vglog("warn", event="stitch_skipped", stream=stream, parts=len(files), reason="too_many", max=args.max_parts)
+        return
+    night = datetime.now().strftime("%Y-%m-%d")
+    parts = []
+    for p in files:
+        t = arrival_ms(p.stem)
+        info = ffprobe(p)
+        if t is None:
+            vglog("warn", event="stitch_skipped", stream=stream, file=p, reason="name has no timestamp"); return
+        if info is None:
+            if not args.dry_run: dst = to_trash(root, p, night)
+            else: dst = "(dry run)"
+            vglog("warn", event="discarded", reason="unreadable", file=p, stream=stream, trash=dst)
+            continue
+        parts.append({"file": p, "arrival_ms": t, "params": info[0], "duration_ms": int(info[1] * 1000)})
+    parts.sort(key=lambda x: x["arrival_ms"])
+    # chains: consecutive, same parameters, gap under the limit
+    chains, cur = [], []
+    for part in parts:
+        if cur:
+            prev = cur[-1]
+            gap = part["arrival_ms"] - (prev["arrival_ms"] + prev["duration_ms"])
+            if part["params"] != prev["params"]:
+                vglog("info", event="stitch_break", stream=stream, reason="params_changed", file=part["file"],
+                      before=f"{prev['params'][1]}x{prev['params'][2]}", after=f"{part['params'][1]}x{part['params'][2]}")
+                chains.append(cur); cur = []
+            elif gap > args.max_gap_s * 1000:
+                vglog("info", event="stitch_break", stream=stream, reason="gap", gap_s=gap // 1000, file=part["file"])
+                chains.append(cur); cur = []
+        cur.append(part)
+    if cur: chains.append(cur)
+    for chain in chains:
+        if len(chain) < 2: continue
+        first = chain[0]["file"]
+        offset, joins, plist = 0, [], []
+        for i, part in enumerate(chain):
+            if i:
+                prev = chain[i - 1]
+                missing = part["arrival_ms"] - (prev["arrival_ms"] + prev["duration_ms"])
+                joins.append({"at_s": round(offset / 1000, 3), "missing_ms": max(0, missing)})
+            plist.append({"file": part["file"].name, "arrival_ms": part["arrival_ms"],
+                          "duration_ms": part["duration_ms"], "offset_ms": offset})
+            offset += part["duration_ms"]
+        if args.dry_run:
+            vglog("info", event="stitched", stream=stream, parts=len(chain), joins=len(joins), dry_run=1,
+                  missing_ms=sum(j["missing_ms"] for j in joins), out=first); continue
+        listing = live / f"{first.stem}.concat.txt"
+        tmp = live / f"{first.stem}.stitch.part"
+        listing.write_text("".join("file '%s'\n" % str(p["file"]).replace("'", r"'\''") for p in chain))
+        run = subprocess.run(["nice", "-n", "15", "ionice", "-c", "3", "ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error", "-y",
+                              "-f", "concat", "-safe", "0", "-i", str(listing), "-map", "0", "-c", "copy",
+                              "-movflags", "+faststart", "-f", "mp4", str(tmp)], capture_output=True, text=True)
+        listing.unlink(missing_ok=True)
+        got = ffprobe(tmp) if run.returncode == 0 else None
+        want = offset / 1000
+        if got is None or abs(got[1] - want) > 2 + len(chain):
+            tmp.unlink(missing_ok=True)
+            why = (run.stderr.strip().splitlines() or ["no stderr"])[-1][:200]
+            vglog("error", event="stitch_failed", stream=stream, parts=len(chain), rc=run.returncode,
+                  want_s=int(want), got_s=int(got[1]) if got else "?", reason=why)
+            continue                                    # fragments untouched: the sweep publishes them one by one
+        last_mtime = max(p["file"].stat().st_mtime for p in chain)
+        for p in chain: to_trash(root, p["file"], night)
+        final = live / f"{first.stem}.mp4"
+        os.replace(tmp, final)
+        os.utime(final, (last_mtime, last_mtime))        # keep it "idle" so this very sweep publishes it
+        meta = {"parts": plist, "joins": joins, "missing_ms": sum(j["missing_ms"] for j in joins),
+                "trash": str(root / "trash" / night / live.relative_to(root)), "stitched_at": datetime.now(TZ).isoformat(timespec="seconds")}
+        (live / f"{first.stem}.stitch.json").write_text(json.dumps(meta))
+        vglog("info", event="stitched", stream=stream, parts=len(chain), joins=len(joins),
+              missing_ms=meta["missing_ms"], seconds=int(want), out=final, trash=meta["trash"])
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("root")
+    ap.add_argument("--max-parts", type=int, default=500)
+    ap.add_argument("--max-gap-s", type=int, default=600)
+    ap.add_argument("--idle-min", type=int, default=3)
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args()
+    root = Path(args.root)
+    streams = root / "streams"
+    if not streams.is_dir(): return 0
+    for live in sorted(streams.glob("*/*/.live")):
+        if live.is_dir():
+            try: stitch_dir(root, live, args)
+            except Exception as error:
+                vglog("error", event="stitch_failed", dir=live, reason=str(error)[:200])
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())

bin/vgstuck.sh

@@ -0,0 +1,207 @@
+#!/bin/bash
+# Triage recordings that never published: see why each one is stuck, then retry it or bin it.
+#
+#   vgstuck.sh                     list everything stuck, with the reason        (read-only, the default)
+#   vgstuck.sh list --durations    ...and measure each one's length (slow: fragmented MP4 is read end to end)
+#   vgstuck.sh retry <match>       clear what is blocking it so the sweep tries again
+#   vgstuck.sh retry --all         ...for everything that is parked
+#   vgstuck.sh drop  <match>       move the recording to trash: 3-day undo, same as "dump tonight"
+#   vgstuck.sh list  --under 5     only recordings smaller than 5 MB (reconnect fragments)
+#   vgstuck.sh drop  --under 5     bin all of those; still-recording files are never touched
+#   vgstuck.sh wipe                bin EVERYTHING unprocessed: stops the sweep (killing any conversion in flight),
+#                                  then moves every .live/ file — parked, waiting, published-collision, fresh — with
+#                                  its sidecars, failure logs and .part leftovers to trash, then drops the emptied
+#                                  folders. The ONLY thing left alone is a file MediaMTX has open, i.e. a stream
+#                                  being recorded this minute. The Dump tab's "Bin all unprocessed" button runs this.
+#   vgstuck.sh sweep               kick the sweep once, without waiting on it
+#   vgstuck.sh json                the same survey as JSON (what dumpd serves to the web page)
+#
+# <match> is any substring of the path (a streamer name, a title, a timestamp). It must match at least one
+# recording, and `drop` refuses --all so a stray pattern cannot bin everything by accident: the deliberate
+# version is spelled out as its own word, `wipe`.
+#
+# WHY A RECORDING STAYS IN .live/
+#   parked     a <Title>__<ts>.normalize-failed.log sits next to the intended output. The sweep skips the
+#              file entirely until that log is deleted, so it will never retry on its own.
+#   published  an output <Title>__<ts>.mp4 already exists, so the sweep leaves the source alone.
+#   waiting    nothing is blocking it; it is simply newer than MINS_IDLE or the sweep has not run yet.
+#   partial    a leftover <output>.mp4.part. This does NOT block anything (ffmpeg overwrites it with -y);
+#              it only wastes disk, and retry removes it.
+#
+# Nothing here deletes media. `drop` MOVES to /srv/media/trash/<today>/, which trash-purge clears after
+# three days, so a wrong call is recoverable for the same window as a dumped night.
+set -uo pipefail
+
+ROOT="${MEDIA_ROOT:-/srv/media}"
+TRASH="$ROOT/trash"
+MINS_IDLE="${MINS_IDLE:-3}"
+
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgstuck %s\n' "$(date -Is)" "$lvl" "$*"; }
+
+# Everything about one .live source, as: src|out|state|age_s|bytes|dur_s|reason
+survey() {
+  local src titledir user title stamp base out state reason dur
+  while IFS= read -r -d '' src; do
+    titledir="$(dirname "$(dirname "$src")")"
+    user="$(basename "$(dirname "$titledir")")"
+    title="$(basename "$titledir")"
+    stamp="$(basename "${src%.*}")"
+    base="$(dirname "$titledir")/${title}__${stamp}"
+    out="$base.mp4"
+    reason=""
+    if [[ -e "$base.normalize-failed.log" ]]; then
+      state=parked
+      reason="$(head -1 "$base.normalize-failed.log" 2>/dev/null)"
+      local why; why=$(grep -A2 '^--- ffmpeg stderr ---$' "$base.normalize-failed.log" 2>/dev/null | sed -n '2p')
+      [[ -n ${why:-} ]] && reason="$reason :: $why"
+    elif [[ -e "$out" ]]; then
+      state=published
+      reason="output already exists: $(basename "$out")"
+    elif [[ -n $(find "$src" -mmin "-$MINS_IDLE" -print -quit 2>/dev/null) ]]; then
+      state=writing
+      reason="touched in the last ${MINS_IDLE}m — still recording"
+    else
+      state=waiting
+      reason="nothing blocking it; the sweep has not got to it"
+    fi
+    [[ -e "$out.part" ]] && reason="$reason [leftover .part]"
+    # Duration is OPT-IN. Everything in .live/ is fragmented MP4 with no duration in its header, so ffprobe
+    # has to read the whole file to answer — gigabytes per recording. With many stuck files the survey took
+    # over three minutes, dumpd's request timed out, and the web page broke. Size and age come from stat.
+    dur=""
+    if [[ ${VGSTUCK_DURATIONS:-0} == 1 ]]; then
+      dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$src" 2>/dev/null | cut -d. -f1)
+    fi
+    printf '%s|%s|%s|%s|%s|%s|%s\n' "$src" "$out" "$state" \
+      "$(( $(date +%s) - $(stat -c %Y "$src") ))" "$(stat -c %s "$src")" "${dur:--}" "$reason"
+  done < <(find "$ROOT/streams" -path '*/.live/*' -name '*.mp4' -print0 2>/dev/null)
+}
+
+show() {
+  local any=0 total=0 src out state age bytes dur reason
+  printf '\n  %-9s %-9s %9s %8s  %s\n' state waiting length size recording
+  while IFS='|' read -r src out state age bytes dur reason; do
+    total=$(( total + bytes ))
+    any=1
+    if [[ $dur == - || -z $dur ]]; then length="-"; else
+      length=$(printf '%d:%02d:%02d' $(( dur / 3600 )) $(( (dur % 3600) / 60 )) $(( dur % 60 ))); fi
+    printf '  %-9s %-9s %9s %7.0fM  %s\n' "$state" \
+      "$(( age / 3600 ))h$(printf '%02d' $(( (age % 3600) / 60 )))m" \
+      "$length" \
+      "$(awk -v b="$bytes" 'BEGIN{print b/1000000}')" "${src#"$ROOT"/streams/}"
+    printf '            %s\n' "$reason"
+  done < <(selected "${1:---all}" "${2:-}" | sort -t'|' -k3,3 -k4,4nr)
+  (( any )) || printf '  nothing matches.\n'
+  (( any )) && printf '  %s item(s), %s GB\n' "$(selected "${1:---all}" "${2:-}" | wc -l)" "$(awk -v b="$total" 'BEGIN{printf "%.1f", b/1e9}')"
+  [[ ${VGSTUCK_DURATIONS:-0} == 1 ]] || printf '\n  lengths not measured (slow on fragmented MP4): vgstuck.sh list --durations\n'
+  printf '  retry one:  vgstuck.sh retry <substring>      bin one:  vgstuck.sh drop <substring>\n\n'
+}
+
+# rows matching the pattern, everything when --all, or everything smaller than N MB with --under N.
+# --under exists for reconnect fragments: every time a publisher drops and comes back MediaMTX opens a new
+# recording, so a bad night leaves hundreds of one- and two-second files. Still-recording files never match.
+selected() {
+  local pattern="$1" limit="${2:-}"
+  if [[ $pattern == --under ]]; then
+    [[ $limit =~ ^[0-9]+$ ]] || { echo "usage: --under <megabytes>" >&2; return 2; }
+    survey | awk -F'|' -v max="$(( limit * 1000000 ))" '$3 != "writing" && $5 < max'
+  elif [[ $pattern == --all ]]; then survey
+  else survey | grep -F -- "$pattern" || true
+  fi
+}
+
+do_retry() {
+  local pattern="$1" limit="${2:-}" n=0 src out state age bytes dur reason
+  while IFS='|' read -r src out state age bytes dur reason; do
+    [[ $state == writing ]] && { vglog info "event=skip reason=still_recording file=\"$src\""; continue; }
+    rm -f "$out.part"                                  # leftover partial output: harmless, but no reason to keep
+    rm -f "${out%.mp4}.normalize-failed.log"           # THIS is what makes the sweep skip the file forever
+    n=$((n + 1))
+    vglog info "event=retry file=\"$src\" was=$state"
+  done < <(selected "$pattern" "$limit")
+  (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; }
+  vglog info "event=retry_done count=$n msg=\"run vgstuck.sh sweep, or wait for the timer\""
+}
+
+# one row per line on stdin (survey format) -> trash. Prints the count it moved.
+drop_rows() {
+  local night="$1" n=0 src out state age bytes dur reason rel dst victim
+  while IFS='|' read -r src out state age bytes dur reason; do
+    # The survey labels a file by its blocker first (a parked file stays "parked" even while being written), so the
+    # safety check here does not trust the label: open by any process, or touched inside MINS_IDLE, means leave it.
+    # In wipe mode (WIPE=1) only "open" counts — the sweep has been stopped, so an open file is MediaMTX recording.
+    if fuser -s "$src" 2>/dev/null || { [[ ${WIPE:-0} != 1 ]] && { [[ $state == writing ]] || [[ -n $(find "$src" -mmin "-$MINS_IDLE" -print -quit 2>/dev/null) ]]; }; }; then
+      vglog info "event=skip reason=still_recording file=\"$src\"" >&2; continue
+    fi
+    for victim in "$src" "${src%.*}.json" "${src%.*}.stitch.json" "${out%.mp4}.normalize-failed.log" "$out.part"; do
+      [[ -e $victim ]] || continue
+      rel="${victim#"$ROOT"/}"
+      dst="$TRASH/$night/$rel"
+      mkdir -p "$(dirname "$dst")"
+      mv -f "$victim" "$dst" && vglog info "event=trashed file=\"$rel\" undo_days=3" >&2
+    done
+    rmdir "$(dirname "$src")" "$(dirname "$(dirname "$src")")" 2>/dev/null || true
+    n=$((n + 1))
+  done
+  echo "$n"
+}
+
+do_drop() {
+  local pattern="$1" limit="${2:-}" night n
+  [[ $pattern == --all ]] && { vglog error 'event=refused msg="drop --all is not allowed; name them, use --under <MB>, or wipe"' >&2; return 2; }
+  night="$(date +%F)"
+  n=$(selected "$pattern" "$limit" | drop_rows "$night")          # per-file lines go to stderr, the count to stdout
+  (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; }
+  vglog info "event=drop_done count=$n trash=\"$TRASH/$night\""
+}
+
+# Everything unprocessed, deliberately. First the sweep is stopped, which kills any ffmpeg conversion it has in
+# flight (the timer starts it again within five minutes, to an empty queue). Then every .live/ file goes, parked
+# or fresh, and so do failure logs whose recording is already gone (they would keep the Stuck tab listing ghosts)
+# and leftover .part outputs. The one exception: a file some process still holds open after the sweep is down,
+# which is MediaMTX recording a live stream — pulling that from under it would lose the stream in progress.
+do_wipe() {
+  local night n stale=0 log rel dst
+  night="$(date +%F)"
+  if systemctl is-active --quiet normalize-sweep.service; then
+    systemctl stop normalize-sweep.service && vglog info "event=cancelled unit=normalize-sweep.service msg=\"conversion in flight killed\""
+  fi
+  systemctl is-active --quiet vgdeband.service && systemctl stop vgdeband.service && vglog info "event=cancelled unit=vgdeband.service"
+  n=$(survey | WIPE=1 drop_rows "$night")
+  while IFS= read -r -d '' log; do
+    rel="${log#"$ROOT"/}"; dst="$TRASH/$night/$rel"; mkdir -p "$(dirname "$dst")"
+    mv -f "$log" "$dst" && stale=$((stale + 1))
+  done < <(find "$ROOT/streams" -maxdepth 2 \( -name '*.normalize-failed.log' -o -name '*.mp4.part' \) -print0 2>/dev/null)
+  find "$ROOT/streams" -mindepth 2 -type d -name .live -empty -delete 2>/dev/null
+  find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type d -empty -delete 2>/dev/null
+  vglog info "event=wipe_done recordings=${n:-0} stale_files=$stale trash=\"$TRASH/$night\" undo_days=3"
+}
+
+# Same survey, as JSON. bin/dumpd.py serves this to the web page, so the classification lives in exactly
+# one place and the browser and the CLI can never disagree about why something is stuck.
+emit_json() {
+  survey | python3 -c '
+import json, sys
+rows = []
+for line in sys.stdin:
+    parts = line.rstrip("\n").split("|", 6)
+    if len(parts) != 7:
+        continue
+    src, out, state, age, size, dur, reason = parts
+    rows.append({"src": src, "out": out, "state": state, "age_s": int(age or 0),
+                 "bytes": int(size or 0),
+                 "duration_s": int(dur) if dur not in ("", "-") else None, "reason": reason})
+print(json.dumps(rows))'
+}
+
+for a in "$@"; do [[ $a == --durations ]] && export VGSTUCK_DURATIONS=1; done
+
+case "${1:-list}" in
+  list|"")  show "${2:---all}" "${3:-}" ;;
+  json)     emit_json ;;
+  retry)    [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh retry <substring>|--all|--under <MB>" >&2; exit 1; }; do_retry "$2" "${3:-}" ;;
+  drop)     [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh drop <substring>|--under <MB>" >&2; exit 1; }; do_drop "$2" "${3:-}" ;;
+  wipe)     do_wipe ;;
+  sweep)    systemctl start --no-block normalize-sweep.service && vglog info "event=sweep_started" ;;
+  *)        sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 1 ;;
+esac

check.sh

@@ -9,16 +9,47 @@ for f in install.sh addCommunityMediaServer.sh installJellyfinPlugin.sh register
   [[ -f $f ]] || continue
   bash -n "$f" && say "$f" ok || { say "$f" "BASH SYNTAX"; fail=1; }
 done
-for f in configure.py bin/dumpd.py bin/vgstrip.py observability/kibana-setup.py observability/kibana-alarms.py; do
+for f in configure.py bin/dumpd.py bin/vgstrip.py bin/vgstitch.py bin/vghosts-check.py observability/kibana-setup.py observability/kibana-alarms.py; do
   python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
 done
 rm -rf __pycache__ bin/__pycache__
-python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
+# bin/install.list is the single source for what lands in /usr/local/bin. Fail if it names something that
-node -e "
+# does not exist, or if a file in bin/ is neither listed nor explicitly excluded — that drift is exactly how
+# deploy.sh ended up installing five of ten scripts.
+if python3 - <<'PYMAN' >/dev/null 2>&1
+import re, sys
+from pathlib import Path
+text = Path("bin/install.list").read_text()
+listed = dict((m.group(2), m.group(1)) for m in re.finditer(r"^(0\d{3}) (\S+)$", text, re.M))
+excluded = set(re.findall(r"^#\s+exclude (\S+)", text, re.M))
+present = set("bin/" + p.name for p in Path("bin").iterdir() if p.is_file() and p.name != "install.list")
+missing = set(listed) - present
+unclaimed = present - set(listed) - excluded
+assert not missing, f"install.list names files that do not exist: {sorted(missing)}"
+assert not unclaimed, f"bin/ files neither listed nor excluded in install.list: {sorted(unclaimed)}"
+PYMAN
+then say "bin/install.list" ok; else say "bin/install.list" "DRIFTED — see the assertion"; fail=1; fi
+# Only files that are actually here. The server checkout is partial (scp, not a full clone), and a tool that
+# reports "broken" when it means "absent" gets ignored — the same reason the deadman reports on change only.
+yamls=(); for f in docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml; do [[ -f $f ]] && yamls+=("$f"); done
+python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" "${yamls[@]}" \
+  && say "yaml (${#yamls[@]} files)" ok || { say "yaml" "INVALID"; fail=1; }
+if ! command -v node >/dev/null; then
+  say "page JS" "(skipped: node not installed here)"
+else
+  node -e "
 const fs=require('fs');
-for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
+for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html','streaming/index.html']) { if (!fs.existsSync(f)) continue; const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
-new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" && say "dock + clock JS" ok || { say "dock + clock JS" "SYNTAX ERROR"; fail=1; }
+if (fs.existsSync('dock/clock/vgclock.js')) new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" \
-diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null && say "vgclock.js in sync (dock/portal)" ok || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line"
+    && say "page JS" ok || { say "page JS" "SYNTAX ERROR"; fail=1; }
+fi
+if [[ -f live/src/vgclock.js && -f dock/clock/vgclock.js ]]; then
+  diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null \
+    && say "vgclock.js in sync (dock/portal)" ok \
+    || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line"
+else
+  say "vgclock.js in sync" "(skipped: only one copy in this checkout)"
+fi
 # generated Caddyfile / mediamtx / copyparty from a dry configure run must at least be produced + parse
 T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "
 import configure as c, yaml

configure.py

@@ -4,6 +4,9 @@
   ./configure.py --full     first-time setup: asks every question (current values as defaults)
   ./configure.py            manage: skips what is already set; add/remove/reset streamers & editors
   ./configure.py --show     print current streamers, editors, domain and the OBS settings
+  ./configure.py --set K=V  change ONE setting and apply only what it needs (Discord tokens and webhooks,
+                            retention, alert tuning). Run it with no arguments to list what is settable.
+                            Everything else regenerates a config or a container, so it needs --apply.
   ./configure.py --apply    regenerate everything from the values already on disk and apply (non-interactive;
                             used by deploy.sh). Fails if the main hostname was never set.
   ./configure.py --hosts AUTH WATCH [GIT [BOT [LIVE [FILES [SYNC [CLIPS [DIAG]]]]]]]
@@ -13,7 +16,7 @@
                             NEW host needs --full or this.
 
 Hosts: main = the gateway page at /, files = copyparty, clips = clip intake, sync = Syncthing UI,
-       live = portal + OBS dock (/dock/, /dock/clock/, /dump, /me, /time), diag = logs/metrics/alerts,
+       live = portal + OBS dock (/dock/, /dock/clock/, /dump, /me, /time), diag = logs/metrics/alerts + streaming triage (live, stuck, dump),
        auth, watch, git, bot.
 
 Reads  : Caddyfile (hostnames), mediamtx.yml (streamers), .env
@@ -24,7 +27,7 @@ Writes : Caddyfile, mediamtx.yml, copyparty.conf, stream-token.secret -> /etc/va
          Discord bot token/guild), folders, restarts
 Set VG_DRY=1 to write files only (no sudo / docker / folders).
 """
-import base64, getpass, hashlib, json, os, re, secrets, shutil, subprocess, sys, time
+import base64, getpass, hashlib, json, os, re, secrets, shutil, socket, subprocess, sys, time
 import urllib.error, urllib.request
 from pathlib import Path
 
@@ -45,8 +48,8 @@ def read_hosts():
             m = re.match(r"#\s*vanguard:(\w+)", t)
             if m: role = m.group(1); continue
             if t and not t.startswith(("#", "}", "{")) and t.endswith("{") and " " in t:
-                host = t.split()[0]
+                host = t.split()[0].rstrip(",")          # "a, b {" = an alias (decorate_caddy); the first name is the role's
-                if "example.com" in host: role = None; continue
+                if "example.com" in host or role == "redirect": role = None; continue   # redirect blocks are not roles
                 hosts[role or ("main" if "main" not in hosts else host)] = host
                 role = None
     except FileNotFoundError:
@@ -55,6 +58,115 @@ def read_hosts():
 
 def read_domain(): return read_hosts().get("main", "")
 
+# Every Discord credential and webhook, in one block, in this order, with these comments written into .env.
+# They used to be scattered through the file in the order they were added, so "where does the alarm webhook
+# go" had no obvious answer. read_env ignores any line starting with #, so the comments are free.
+DISCORD_KEYS = (
+    ("DISCORD_BOT_TOKEN",         "the bot's own token (Discord dev portal -> Bot). The bot's CI also sets this from Gitea"),
+    ("DISCORD_GUILD_ID",          "the server the bot and the Authentik membership policies work against"),
+    ("ALERT_WEBHOOK_URL",         "webhook: alarms (vanguard-alerts.timer) and the deadman"),
+    ("LIVE_WEBHOOK_URL",          "webhook: 'a stream went live' (MediaMTX runOnReady)"),
+    ("DEPLOY_NOTIFY_WEBHOOK_URL", "webhook: deploy results (deploy.sh)"),
+)
+DEPLOY_ENV_KEYS = ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL",
+                   "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")
+
+def render_env(env):
+    """.env text with the Discord block first and commented, everything else after it."""
+    named = dict(DISCORD_KEYS)
+    out = ["# ---- Discord: tokens and webhooks ----\n",
+           "# change any of these with:  ./configure.py --set KEY=VALUE\n"]
+    for key, why in DISCORD_KEYS:
+        out.append(f"# {why}\n{key}={env.get(key, '')}\n")
+    out.append("\n# ---- everything else ----\n")
+    out += [f"{k}={v}\n" for k, v in env.items() if k not in named]
+    return "".join(out)
+
+def write_deploy_env(env=None):
+    """/etc/vanguard/deploy.env — read by dumpd's /deploy webhook and by deploy.sh."""
+    env = env or read_env()
+    Path("deploy.env").write_text("".join(f"{k}={env.get(k, '')}\n" for k in DEPLOY_ENV_KEYS)
+                                  + f"DEPLOY_REPO_DIR={Path.cwd()}\n")
+    os.chmod("deploy.env", 0o600)
+    run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"])
+    Path("deploy.env").unlink(missing_ok=True)
+
+# Settings that can be changed on their own, with the one side effect each needs. Everything else has to go
+# through --apply or --full because it regenerates a config file or a container: DIAG_REQUIRE_GROUP needs the
+# Caddyfile rewritten and reloaded, DIAG_KIBANA_ROLES needs the Elasticsearch user changed and Kibana
+# restarted, hostnames need the whole pass. Keeping this list short is the point.
+SETTABLE = {
+    "DISCORD_BOT_TOKEN":   "the bot's token (restarts the bot container)",
+    "DISCORD_GUILD_ID":    "the Discord server id (restarts the bot container)",
+    "ALERT_WEBHOOK_URL":   "Discord webhook for alarms and the deadman (blank = journal only)",
+    "LIVE_WEBHOOK_URL":    "Discord webhook for 'went live' (needs a MediaMTX recreate — NOT done for you)",
+    "DEPLOY_NOTIFY_WEBHOOK_URL": "Discord webhook for deploy results",
+    "ALERT_RETENTION_DAYS": "how long delivered alerts stay queryable in vanguard-alerts",
+    "ALERT_SUPPRESS_S":    "per-rule repeat suppression in the delivery timer, seconds",
+    "ALERT_MENTION":       "who an alarm pings, e.g. role:123,role:456 or user:789 (blank = nobody)",
+    "ALERT_MENTION_LEVELS": "which levels ping, comma separated (default error; add warn to widen)",
+    "HEARTBEAT_MAX_S":     "how stale the alerting heartbeat may get before the deadman alarms, seconds",
+    "LOG_RETENTION_DAYS":  "ILM delete age for filebeat-*/metricbeat-* (re-applied immediately)",
+}
+
+def set_env(updates):
+    """Change individual .env keys in place and apply only what each one needs.
+
+    --full re-prompts for every hostname and secret, which is far too blunt for "point the alarms at a new
+    Discord channel" — mistyping a hostname there has broken TLS before. This writes the keys, rewrites the
+    files that derive from them, and touches nothing else.
+    """
+    env = read_env()
+    if not env:
+        print("no .env here — run ./configure.py --full first"); sys.exit(1)
+    for key, value in updates.items():
+        if key not in SETTABLE:
+            print(f"{key} is not settable on its own. Settable keys:")
+            for name, why in SETTABLE.items():
+                print(f"   {name:22} {why}")
+            print("Anything else regenerates a config or a container: use --apply (or --full for hostnames).")
+            sys.exit(1)
+        env[key] = value
+    Path(".env").write_text(render_env(env))
+    os.chmod(".env", 0o600)
+    for key, value in updates.items():
+        print(f"   {key}={value if key != 'ALERT_WEBHOOK_URL' else (value[:40] + '…' if value else '(cleared)')}")
+
+    # The webhook is a credential for that channel, so it stays in its own root-only file.
+    if "ALERT_WEBHOOK_URL" in updates:
+        Path("alert-webhook").write_text((env.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
+        run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"])
+        Path("alert-webhook").unlink(missing_ok=True)
+        print("   wrote /etc/vanguard/alert-webhook")
+    # Always, not only when one of its keys changed: it is two file writes, it is idempotent, and making it
+    # conditional means the file is missing on any system where the first --set happened to be the webhook.
+    # A systemd timer never reads the repo's .env, so this file is the only way those knobs reach the scripts.
+    write_alerts_env(env)
+    if "DEPLOY_NOTIFY_WEBHOOK_URL" in updates:
+        write_deploy_env(env); print("   wrote /etc/vanguard/deploy.env")
+    if "LOG_RETENTION_DAYS" in updates:
+        es_security(compose_cmd())          # re-injects the ILM delete phase at the new age
+    # The bot reads these from its container environment, so they only take effect on recreate. Its own CI
+    # does the same thing when it installs a bundle, so a restart here is nothing unusual for the bot.
+    if any(k in updates for k in ("DISCORD_BOT_TOKEN", "DISCORD_GUILD_ID")):
+        run(compose_cmd() + ["up", "-d", "discord-bot"]); print("   discord-bot recreated with the new settings")
+    # MediaMTX reads LIVE_WEBHOOK_URL from ITS environment too, but recreating MediaMTX cuts whatever is
+    # recording right now. That is never worth doing automatically, so it is left to a human and a quiet moment.
+    if "LIVE_WEBHOOK_URL" in updates:
+        print("   NOTE: MediaMTX still has the old value. It only reads this on start, and a restart cuts a")
+        print("         live recording, so run this yourself between streams:  docker compose up -d mediamtx")
+
+def write_alerts_env(env=None):
+    """/etc/vanguard/alerts.env — read by vanguard-alerts.service and vanguard-deadman.service."""
+    env = env or read_env()
+    keys = ("ALERT_INDEX", "ALERT_RETENTION_DAYS", "ALERT_SUPPRESS_S", "HEARTBEAT_MAX_S",
+            "ALERT_MENTION", "ALERT_MENTION_LEVELS")                                       # what the two scripts read
+    Path("alerts.env").write_text("".join(f"{k}={env[k]}\n" for k in keys if env.get(k)))
+    os.chmod("alerts.env", 0o644)
+    run(["sudo", "install", "-m", "0644", "-o", "root", "-g", "root", "alerts.env", "/etc/vanguard/alerts.env"])
+    Path("alerts.env").unlink(missing_ok=True)
+    print("   wrote /etc/vanguard/alerts.env")
+
 def read_env():
     env = {}
     try:
@@ -233,11 +345,30 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
 # vanguard:diag
 {diag_host} {{
 	encode zstd gzip
-{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))}	# Kibana IS this host: logs (Discover), dashboards and alerting in one app, served at the root so there
+{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))}	# Triage lives on this host, alongside the dashboards it belongs with. Both handles come BEFORE the
-	# is no path prefix to get wrong and no index page of links. Elasticsearch itself is never published.
+	# catch-all below, and the Kibana proxy is wrapped in its own handle so the three are evaluated in the
+	# order written rather than by Caddy's directive ordering.
+	handle /streaming/api/* {{
+		reverse_proxy 127.0.0.1:8080          # dumpd: live publishers + loss, and the stuck list / retry / bin
+	}}
+	handle_path /streaming* {{
+		root * /srv/streaming
+		try_files {{path}} /index.html
+		file_server                           # Live and Stuck tabs; this host is already media-admins only
+	}}
+	redir /stuck* /streaming#stuck 302        # it used to live here
+	handle_path /disk* {{
+		root * /srv/streaming
+		rewrite * /disk.html                  # the Disk page: same mount as the streaming page, its own address
+		file_server
+	}}
+	# Kibana IS the rest of this host: logs (Discover), dashboards and alerting in one app, served at the
+	# root so there is no path prefix to get wrong. Elasticsearch itself is never published.
+	handle {{
 		reverse_proxy 127.0.0.1:5601 {{
 			flush_interval -1
 		}}
+	}}
 }}
 """
     if live_host:
@@ -258,7 +389,7 @@ groups: {{http.request.header.X-Authentik-Groups}}"
 		reverse_proxy 127.0.0.1:8080          # server clock for the timecode strip (clock page + Go Live)
 	}}
 	handle /dump* {{
-		reverse_proxy 127.0.0.1:8080          # dump tonight (dock)
+		reverse_proxy 127.0.0.1:8080          # dump tonight (dock): the signed-in streamer's own night
 	}}
 	handle_path /dock* {{
 		root * /srv/dock
@@ -278,7 +409,14 @@ groups: {{http.request.header.X-Authentik-Groups}}"
 		reverse_proxy 127.0.0.1:8889
 	}}
 	handle_path /hls/* {{
-		reverse_proxy 127.0.0.1:8888
+		# handle_path strips /hls before proxying, so MediaMTX's own redirects come back root-relative and
+		# lose the prefix. It redirects every playlist request once, to ?cookieCheck=1, so without this the
+		# browser follows Location: /<user>/<Title>/index.m3u8 into the SPA catch-all below, receives
+		# index.html, and hls.js reports "manifestParsingError" — a web page where a manifest should be.
+		reverse_proxy 127.0.0.1:8888 {{
+			header_down Location ^/ /hls/
+			flush_interval -1          # low-latency HLS sends partial segments; do not buffer them
+		}}
 	}}
 	handle {{
 		root * /srv/live
@@ -343,7 +481,9 @@ name:   {{http.request.header.X-Authentik-Name}}"
 	redir /about* / 308                       # old bookmarks: the gateway page used to live at /about/
 	handle {{
 		root * /srv/about
-		try_files {{path}} /index.html
+		# {{path}}/index.html: Caddy's file matcher takes a slash-less path as "a file", so /onboarding would skip
+		# the onboarding/ folder and fall through to the gateway page. Sub-pages live in folders under about/.
+		try_files {{path}} {{path}}/index.html /index.html
 		file_server                           # the gateway page is the root of the main host
 	}}
 }}
@@ -408,25 +548,15 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b
     env.setdefault("ALERT_INDEX", "vanguard-alerts")         # the alert bus: Kibana writes, vanguard-alerts.sh delivers
     env.setdefault("LOG_RETENTION_DAYS", "30")               # ILM delete phase for filebeat-*/metricbeat-*
     env.setdefault("ALERT_RETENTION_DAYS", "90")             # how long delivered alerts stay queryable
+    env.setdefault("ALERT_MENTION", "")                      # role:<id>/user:<id> list an alarm pings
+    env.setdefault("ALERT_MENTION_LEVELS", "error")          # only errors ping; warnings arrive quietly
     env.setdefault("DISCORD_BOT_TOKEN", "")          # fill in: Discord dev portal -> the Authentik app -> Bot
     env.setdefault("DISCORD_GUILD_ID", "")
     env["WATCH_HOST"] = watch_host or "watch.example.com"
-    Path(".env").write_text("".join(f"{k}={v}\n" for k, v in env.items()))
+    Path(".env").write_text(render_env(env))
     os.chmod(".env", 0o600)
 
-def docker_gateway():
-    """Host IP as seen from the compose network: MediaMTX (bridge) calls dumpd's /mtx-auth there."""
-    if DRY: return "172.18.0.1"
-    for net in ("vanguard-stream_default", "vanguard_default"):
-        try:
-            out = subprocess.run(["docker", "network", "inspect", net, "-f", "{{(index .IPAM.Config 0).Gateway}}"], capture_output=True, text=True, timeout=10)
-            if out.returncode == 0 and out.stdout.strip(): return out.stdout.strip()
-        except Exception:
-            pass
-    return "172.18.0.1"
-
 def write_mediamtx(public_host=""):
-    auth_gw = docker_gateway()
     Path("mediamtx.yml").write_text(f"""# MediaMTX — ingest only. No playback endpoints exposed.  (generated by configure.py)
 logLevel: info
 
@@ -438,18 +568,18 @@ rtspTransports: [tcp]
 rtspEncryption: "no"
 moq: no
 metrics: yes                     # Prometheus text format, scraped by Metricbeat (LOGGING.md); loopback only
-metricsAddress: :9998
+metricsAddress: 127.0.0.1:9998   # host network: loopback only; Metricbeat scrapes it there
 pprof: no
 
 # live portal (served to the browser only via Caddy + Authentik on live.HOST; these ports are published on 127.0.0.1)
 api: yes
-apiAddress: :9997
+apiAddress: 127.0.0.1:9997       # host network: loopback only; Caddy proxies live.HOST/api/*
 hls: yes
-hlsAddress: :8888
+hlsAddress: 127.0.0.1:8888       # host network: loopback only; Caddy proxies live.HOST/hls/*
 hlsVariant: lowLatency
 hlsAlwaysRemux: no
 webrtc: yes
-webrtcAddress: :8889
+webrtcAddress: 127.0.0.1:8889    # WHIP/WHEP signalling: loopback only, via Caddy. Media is 8189 below
 webrtcLocalUDPAddress: :8189
 webrtcLocalTCPAddress: :8189
 webrtcIPsFromInterfaces: no
@@ -467,7 +597,7 @@ srtAddress: :8890
 # NOTE: on this setup (bind-mounted file) MediaMTX does not reload this file live: restart after any change here.
 # Day-to-day operations (streamers, tokens, titles) never touch this file.
 authMethod: http
-authHTTPAddress: http://{auth_gw}:8081/mtx-auth
+authHTTPAddress: http://127.0.0.1:8081/mtx-auth   # dumpd, on the same host network
 authHTTPExclude:
   - action: api
   - action: metrics
@@ -487,14 +617,51 @@ pathDefaults:
   runOnReadyRestart: no
 
 paths:
-  # audio mirrors (first rule wins): never recorded, no hook; only publishable from inside the container (dumpd)
+  # Audio mirrors: never recorded, no hook; only publishable from inside the container (dumpd).
-  ~^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+\\.webrtc$:
+  # The title segment of BOTH rules is dot-free, so a name matches exactly one of them and the order MediaMTX
+  # evaluates them in is irrelevant. They used to overlap (titles allowed dots) and whichever won was luck:
+  # a 36 GB mirror got recorded as a full duplicate of its source. Usernames keep dots: Discord allows them.
+  ~^[A-Za-z0-9_.-]+/[A-Za-z0-9_-]+\\.webrtc$:
     record: no
     runOnReady: ""
   # <username>/<Title>; dumpd decides who may publish where (token user == prefix)
-  ~^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$: {{}}
+  ~^[A-Za-z0-9_.-]+/[A-Za-z0-9_-]+$: {{}}
 """)
 
+# Two .env keys for a base-domain move with no outage (HOSTNAME-CHANGE.md §B):
+#   ALT_BASE=new.base       every role site ALSO answers on its name under that base ("media.old, media.new {"),
+#                           so certificates for the new names are issued and both sets work while nothing else
+#                           has moved yet. Set with --alias NEW; cleared by --base.
+#   REDIRECT_BASE=old.base  after the move: the old names get their own blocks that 308 to the primary names, so
+#                           bookmarks and pinned links keep working instead of failing TLS. Set by --base.
+# Both are applied by rewriting the Caddyfile that write_caddy just produced; read_hosts understands both forms.
+def decorate_caddy():
+    env = read_env()
+    alt, red = env.get("ALT_BASE", "").strip().lower(), env.get("REDIRECT_BASE", "").strip().lower()
+    if not alt and not red:
+        return
+    hosts = read_hosts()
+    main = hosts.get("main", "")
+    base = main.split(".", 1)[1] if "." in main else ""
+    if not base:
+        return
+    role_hosts = {h for r, h in hosts.items() if r != "redirect"}
+    out = []
+    for line in Path("Caddyfile").read_text().splitlines():
+        m = re.match(r"^([A-Za-z0-9.-]+) \{$", line)
+        if m and alt and alt != base and m.group(1) in role_hosts and m.group(1).endswith("." + base):
+            line = f"{m.group(1)}, {m.group(1)[: -len(base)]}{alt} {{"
+        out.append(line)
+    text = "\n".join(out) + "\n"
+    if red and red != base:
+        for role, h in hosts.items():
+            if role != "redirect" and h.endswith("." + base):
+                old = h[: -len(base)] + red
+                text += f"\n# vanguard:redirect\n{old} {{\n\tredir https://{h}{{uri}} 308         # old base {red}: bookmarks follow\n}}\n"
+    Path("Caddyfile").write_text(text)
+    if alt and alt != base: print(f"   Caddyfile: every site also answers under {alt}")
+    if red and red != base: print(f"   Caddyfile: names under {red} redirect (308) to the new ones")
+
 def write_copyparty():
     """copyparty behind Caddy + Authentik: identity and groups arrive as headers; permissions are per group."""
     key = read_env().get("COPYPARTY_IDP_KEY", "key")
@@ -555,12 +722,13 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
         if not yes("   continue anyway?", False): print("aborted; add streamers first ([s] in the menu)"); sys.exit(1)
     print("\n== writing configs")
     write_env(watch_host, discord_token, discord_guild, git_host, bot_host, live_host, domain, live_webhook, files_host, sync_host, clips_host, diag_host)
-    write_caddy(domain, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); write_mediamtx(domain); write_copyparty()
+    write_caddy(domain, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); decorate_caddy(); write_mediamtx(domain); write_copyparty()
     Path("stream-token.secret").write_text(read_env()["STREAM_TOKEN_SECRET"] + "\n"); os.chmod("stream-token.secret", 0o600)
     run(["sudo", "install", "-d", "-o", os.environ.get("SUDO_USER") or os.environ.get("USER") or "debian", "/srv/live"])
     print("== app data folders")
     run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/appdata/jellyfin/config", "/srv/appdata/jellyfin/cache",
          "/srv/appdata/authentik/media", "/srv/appdata/authentik/templates", "/srv/appdata/authentik/certs"])
+    install_brand_assets()       # authentik/brand/*.svg -> the login page's logo, favicon and background
     run(["sudo", "install", "-d", "/srv/appdata/authentik/db", "/srv/appdata/authentik/redis", "/srv/appdata/mariadb"])
     run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/appdata/discord-bot", "/srv/appdata/gitea", "/srv/appdata/act_runner"])
     print("== folders for legacy streamer entries (MediaMTX creates folders on first publish anyway)")
@@ -573,13 +741,13 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     run(["sudo", "install", "-m", "0600", "stream-token.secret", "/etc/vanguard/stream-token.secret"])
     run(["sudo", "install", "-m", "0644", "syncthing/stignore", "/etc/vanguard/stignore"])   # dumpd copies it into new clips folders
     e = read_env()
-    Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600)
+    write_deploy_env(e)                     # one copy of the key list, shared with --set
     Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
     # URL is a credential for that channel, so it is not world-readable.
     # 0600 root: the only readers are root timers (vanguard-deadman.sh, vanguard-alerts.sh). It used to be
     # group-readable by nobody(65534) for the Alertmanager container, which no longer exists.
     run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"])
-    run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"])           # read by dumpd (/deploy webhook) and deploy.sh
+    write_alerts_env(e)                     # the timers' own knobs; a systemd timer never reads .env
     if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")):
         run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"])
         run(["sudo", "install", "-m", "0644", "systemd/deploy-pending.service", "systemd/deploy-pending.timer", "/etc/systemd/system/"])
@@ -595,21 +763,15 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"])
     run(["sudo", "install", "-d", "/var/lib/vanguard"])                                                  # deadman state
     run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"])            # script counters -> dumpd /metrics
-    # The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in
+    # Everything bin/install.list names. Without this an --apply updated the containers but left STALE
-    # /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md).
+    # binaries in /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time.
-    media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py",
+    for mode, path in bin_manifest():
-                         "bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()]
+        run(["sudo", "install", "-m", mode, path, "/usr/local/bin/"])
-    if media:
-        run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"])
-    if Path("bin/vgmetrics.sh").exists():
-        run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"])                    # sourced by the sweep and vgdeband
     if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
-        run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
         run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"])
         run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"])
     # The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord.
     if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")):
-        run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"])
         run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"])
         run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"])
     run(["sudo", "systemctl", "restart", "dumpd"])
@@ -625,6 +787,26 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     print("            run  docker compose restart mediamtx  between streams (cuts a live recording). Adding/removing")
     print("            streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.")
 
+def bin_manifest():
+    """(mode, path) for everything that belongs in /usr/local/bin, from bin/install.list.
+
+    One list, shared with install.sh and deploy.sh. Three hand-kept copies had already drifted: a
+    push-to-deploy was installing five of the ten, so vgstuck.sh, both sourced helpers and both timer
+    scripts never reached the host unless somebody ran --apply by hand.
+    """
+    out = []
+    try:
+        for line in Path("bin/install.list").read_text().splitlines():
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            mode, path = line.split(None, 1)
+            if Path(path).exists():
+                out.append((mode, path))
+    except FileNotFoundError:
+        pass
+    return out
+
 def compose_cmd():
     """docker compose, with sudo only if this user cannot reach the daemon."""
     if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0:
@@ -752,11 +934,253 @@ def es_security(dc):
     run(dc + ["restart", "kibana", "filebeat", "metricbeat"])
     print("   elasticsearch: netrc written for the deadman; kibana + beats restarted")
 
+def ak_shell(dc, script, label):
+    """Run a short script inside authentik-server's Django shell and print only its own output.
+
+    `ak shell` is an interactive console: it reads stdin line by line and rejects indented blocks, so the
+    script goes in as a file and is executed by one unindented statement. Authentik also prints a wall of
+    JSON boot logs to the same stream, which is why the output is filtered.
+    """
+    if DRY:
+        print(f"   (dry) authentik: {label}"); return
+    path = Path("ak-fixup.py")
+    path.write_text(script)
+    try:
+        if subprocess.call(dc + ["cp", str(path), "authentik-server:/tmp/ak-fixup.py"],
+                           stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) != 0:
+            print(f"   authentik: cannot reach the container — {label} skipped"); return
+        result = subprocess.run(dc + ["exec", "-T", "authentik-server", "ak", "shell",
+                                      "-c", "exec(open('/tmp/ak-fixup.py').read())"],
+                                capture_output=True, text=True)
+        for line in (result.stdout + result.stderr).splitlines():
+            if line.startswith(("{", "#", ">>>", "...")) or not line.strip():
+                continue
+            print("  ", line.rstrip())
+    except Exception as error:
+        print(f"   authentik: {label} skipped ({error})")
+    finally:
+        path.unlink(missing_ok=True)
+
+# Proxy-provider state the Authentik UI can quietly put back. Enforced on every apply so a save in the UI
+# cannot leave logout half-working for weeks without anyone noticing.
+AK_LOGOUT_FIXUP = """
+from authentik.providers.proxy.models import ProxyProvider
+from authentik.flows.models import Flow
+target = Flow.objects.filter(slug="default-invalidation-flow").first()
+if target is None:
+    print("authentik: no default-invalidation-flow found; logout flow left alone")
+else:
+    for provider in ProxyProvider.objects.all():
+        before = getattr(provider.invalidation_flow, "slug", None)
+        if before == target.slug:
+            print(f"authentik: {provider.name!r} logout flow ok ({before})")
+        else:
+            provider.invalidation_flow = target
+            provider.save(update_fields=["invalidation_flow"])
+            print(f"authentik: {provider.name!r} logout flow {before} -> {target.slug}")
+"""
+
+# Login branding and "Discord forward". Everything the Authentik admin UI would want clicked for, done from the
+# ORM on every apply so it is reproducible and a UI save cannot quietly undo it:
+#   brand      title, logo, favicon, flow background, login-card CSS, root -> the media gateway (default application)
+#   identification stage   Discord source attached with its label (what akadmin sees behind ?admin=1)
+#   flow titles            the headings on the sign-in / signed-out / first-visit cards
+#   redirect stage         FIRST stage of default-authentication-flow, static target = the Discord source's login
+#                          URL, so a member never sees an Authentik page: protected link -> Discord consent -> back
+#   expression policy      on that binding: skip the redirect when the flow URL carries ?admin=1 — the executor
+#                          keeps the flow's query in the session under "authentik/flows/get", not in request.GET —
+#                          so https://auth.HOST/if/flow/default-authentication-flow/?admin=1 is the password card
+#                          for akadmin (linked from the diag landing dashboard, nowhere public)
+# The outpost's ?next= survives the detour: the source flow manager reads it back from the same session key.
+AK_BRANDING = r"""
+from urllib.parse import urlparse
+from authentik.brands.models import Brand
+from authentik.core.models import Application
+from authentik.flows.models import Flow, FlowStageBinding
+from authentik.policies.models import PolicyBinding
+from authentik.policies.expression.models import ExpressionPolicy
+from authentik.providers.proxy.models import ProxyProvider
+from authentik.sources.oauth.models import OAuthSource
+from authentik.stages.identification.models import IdentificationStage
+AUTH = "__AUTH_HOST__"; MAIN = "__MAIN_HOST__"; TITLE = "__TITLE__"
+ASSETS = "/media/public/brand"
+CSS = "\n".join([
+    ":root { --ak-accent: #5865F2; --pf-global--primary-color--100: #5865F2; --pf-global--primary-color--200: #4752c4; --pf-global--link--Color: #8ea1ff; --pf-global--link--Color--hover: #b4c0ff; }",
+    ".pf-c-login__main { background: rgba(20,20,20,.92); border: 1px solid #2e2e2e; border-radius: 14px; box-shadow: 0 20px 60px rgba(0,0,0,.6); }",
+    ".pf-c-login__main-header, .pf-c-login__main-body, .pf-c-login__main-footer, .pf-c-login__main-header .pf-c-title { color: #e6e6e6; }",
+    ".pf-c-button.pf-m-secondary, .pf-c-button.pf-m-primary { background: #5865F2; color: #fff; border: 0; border-radius: 10px; font-size: 1.05rem; padding: .85rem 1.2rem; }",
+    ".pf-c-button.pf-m-secondary:hover, .pf-c-button.pf-m-primary:hover { background: #4752c4; }",
+    ".pf-c-login__main-footer-band, .pf-c-login__footer { display: none; }",
+    ".pf-c-form-control { background: #1f1f1f; color: #e6e6e6; border-color: #2e2e2e; }",
+])
+
+def setattrs(obj, label, **kv):
+    changed = []
+    for key, value in kv.items():
+        if not hasattr(obj, key):
+            continue                                   # field absent in this authentik version: leave it
+        if getattr(obj, key) != value:
+            setattr(obj, key, value); changed.append(key)
+    if changed:
+        obj.save(); print(f"authentik: {label}: set {', '.join(changed)}")
+    else:
+        print(f"authentik: {label}: ok")
+
+# 1. brand
+brand = Brand.objects.filter(default=True).first() or Brand.objects.order_by("domain").first()
+if brand is None:
+    print("authentik: no brand found; branding skipped")
+else:
+    gateway = None
+    for pp in ProxyProvider.objects.all():
+        if urlparse(pp.external_host or "").hostname == MAIN:
+            gateway = Application.objects.filter(provider=pp).first()
+    extra = {"default_application": gateway} if gateway else {}
+    setattrs(brand, "brand", branding_title=TITLE, branding_logo=f"{ASSETS}/logo.svg", branding_favicon=f"{ASSETS}/favicon.svg",
+             branding_default_flow_background=f"{ASSETS}/login-bg.svg", flow_background=f"{ASSETS}/login-bg.svg",
+             branding_custom_css=CSS, **extra)
+    if gateway is None:
+        print(f"authentik: no proxy provider has external host {MAIN}; brand default application left alone")
+
+# 2. flow titles (and the per-flow background on versions that still keep it there)
+titles = {"default-authentication-flow": f"{TITLE} — sign in", "default-invalidation-flow": f"Signed out of {TITLE}",
+          "default-source-enrollment": f"Welcome to {TITLE}", "default-source-authentication": f"{TITLE} — signing you in",
+          "default-provider-invalidation-flow": f"Signed out of this app — still signed in to {TITLE}"}
+for slug, title in titles.items():
+    flow = Flow.objects.filter(slug=slug).first()
+    if flow is not None:
+        setattrs(flow, f"flow {slug}", title=title, background=f"{ASSETS}/login-bg.svg")
+
+# 3. Discord on the identification stage (the admin card still offers Discord too)
+discord = OAuthSource.objects.filter(provider_type="discord").first()
+if discord is None:
+    print("authentik: no Discord OAuth source; Discord forward skipped")
+else:
+    for stage in IdentificationStage.objects.filter(name="default-authentication-identification"):
+        if not stage.sources.filter(pk=discord.pk).exists():
+            stage.sources.add(discord); print("authentik: identification stage: Discord source attached")
+        setattrs(stage, "identification stage", show_source_labels=True)
+
+    # 4. the forward: redirect stage first in the authentication flow, skipped for ?admin=1
+    try:
+        from authentik.stages.redirect.models import RedirectStage
+    except ImportError:
+        RedirectStage = None
+        print("authentik: this version has no redirect stage; Discord forward skipped (branding applied)")
+    flow = Flow.objects.filter(slug="default-authentication-flow").first()
+    if RedirectStage is not None and flow is not None:
+        target = f"https://{AUTH}/source/oauth/login/{discord.slug}/"
+        stage, made = RedirectStage.objects.get_or_create(name="vanguard-discord-forward", defaults={"mode": "static", "target_static": target, "keep_context": True})
+        setattrs(stage, "redirect stage" + (" (created)" if made else ""), mode="static", target_static=target, keep_context=True)
+        expr = "\n".join([
+            "# Vanguard: members go straight to Discord. ?admin=1 on the flow URL shows the password card for akadmin.",
+            "# The executor stores the flow URL's query in the session, not in request.GET, so look in both.",
+            "http = request.http_request",
+            "if http is None:",
+            "    return True",
+            "if http.GET.get('admin') is not None:",
+            "    return False",
+            "flow_get = http.session.get('authentik/flows/get') or {}",
+            "return flow_get.get('admin') is None",
+            ""])
+        policy, made = ExpressionPolicy.objects.get_or_create(name="vanguard-discord-forward-unless-admin", defaults={"expression": expr})
+        setattrs(policy, "forward policy" + (" (created)" if made else ""), expression=expr, execution_logging=False)
+        others = list(FlowStageBinding.objects.filter(target=flow).exclude(stage=stage).values_list("order", flat=True))
+        order = min(others + [10]) - 10
+        binding, made = FlowStageBinding.objects.get_or_create(target=flow, stage=stage, defaults={"order": order})
+        setattrs(binding, "redirect binding" + (" (created)" if made else ""), order=order, evaluate_on_plan=True, re_evaluate_policies=False)
+        pb, made = PolicyBinding.objects.get_or_create(target=binding, policy=policy, defaults={"order": 0, "enabled": True})
+        setattrs(pb, "policy binding" + (" (created)" if made else ""), enabled=True, negate=False)
+        print(f"authentik: members -> {target} ; admins -> https://{AUTH}/if/flow/default-authentication-flow/?admin=1")
+"""
+
+def ak_branding(dc):
+    env, hosts = read_env(), read_hosts()            # the auth host lives only in the Caddyfile's vanguard:auth block
+    auth_host, main_host = hosts.get("auth", ""), hosts.get("main", "") or env.get("MAIN_HOST", "")
+    if not auth_host or not main_host:
+        print("   authentik: auth/main host not in the Caddyfile yet; branding skipped"); return
+    script = (AK_BRANDING.replace("__AUTH_HOST__", auth_host).replace("__MAIN_HOST__", main_host)
+                         .replace("__TITLE__", env.get("BRAND_TITLE", "Vanguard")))
+    ak_shell(dc, script, "login branding + Discord forward")
+
+def install_brand_assets():
+    # Brand assets for the Authentik login (authentik/brand/ in the repo). They go into Authentik's own media
+    # dir, served at https://auth.HOST/media/public/... with NO login — the login page is where they are shown,
+    # and every other host is behind the gate, so nothing else could serve them to a signed-out visitor.
+    run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/appdata/authentik/media/public/brand"])
+    for asset in sorted(Path("authentik/brand").glob("*.svg")):
+        run(["sudo", "install", "-m", "0644", "-o", "1000", "-g", "1000", str(asset), "/srv/appdata/authentik/media/public/brand/"])
+
+
+# Moving Authentik to a new base domain. NOT part of --apply, on purpose (HOSTNAME-CHANGE.md, "apps before
+# Authentik"): this is the step that decides whether the gated hosts are reachable, so it runs only when asked, after
+# Jellyfin and Gitea already answer on the new names. Every field that carries OLD is rewritten to NEW:
+#   outposts        config.authentik_host            (the forward-auth gate's login address)
+#   proxy providers external_host, cookie_domain     (domain-level: external_host IS the auth URL; cookie_domain = base)
+#   oauth2 providers redirect URIs                   (Jellyfin, Gitea callbacks)
+#   brand           domain, if it was the old base
+# Prints every change; a second run prints only "ok". --ak-hosts OLD NEW --dry-run shows without writing.
+AK_HOSTS = r"""
+import json, re
+OLD = "__OLD__"; NEW = "__NEW__"; DRY = __DRY__
+from authentik.outposts.models import Outpost
+from authentik.providers.proxy.models import ProxyProvider
+from authentik.providers.oauth2.models import OAuth2Provider
+from authentik.brands.models import Brand
+pat = re.compile(r"(?<![A-Za-z0-9-])" + re.escape(OLD) + r"(?![A-Za-z0-9-])")
+def swap(s): return pat.sub(NEW, s) if isinstance(s, str) else s
+def report(label, before, after):
+    if before == after: print(f"authentik: {label}: ok ({after})")
+    else: print(f"authentik: {label}: {before!r} -> {after!r}" + ("  [dry run]" if DRY else ""))
+for o in Outpost.objects.all():
+    cfg = o.config
+    before = getattr(cfg, "authentik_host", None)
+    if before is None:
+        continue
+    after = swap(before); report(f"outpost {o.name!r} authentik_host", before, after)
+    if after != before and not DRY:
+        cfg.authentik_host = after; o.config = cfg; o.save()
+for p in ProxyProvider.objects.all():
+    eh, cd = p.external_host, p.cookie_domain
+    eh2, cd2 = swap(eh), (NEW if cd == OLD else swap(cd))
+    report(f"proxy provider {p.name!r} external_host", eh, eh2); report(f"proxy provider {p.name!r} cookie_domain", cd, cd2)
+    if (eh2, cd2) != (eh, cd) and not DRY:
+        p.external_host, p.cookie_domain = eh2, cd2; p.save(update_fields=["external_host", "cookie_domain"])
+for p in OAuth2Provider.objects.all():
+    raw = list(p._redirect_uris or [])
+    new = [dict(u, url=swap(u.get("url", ""))) if isinstance(u, dict) else u for u in raw]
+    report(f"oauth2 provider {p.name!r} redirect_uris", [u.get("url") for u in raw if isinstance(u, dict)], [u.get("url") for u in new if isinstance(u, dict)])
+    if new != raw and not DRY:
+        p._redirect_uris = new; p.save(update_fields=["_redirect_uris"])
+for b in Brand.objects.all():
+    d2 = swap(b.domain); report(f"brand domain", b.domain, d2)
+    if d2 != b.domain and not DRY:
+        b.domain = d2; b.save(update_fields=["domain"])
+print("authentik: outposts re-read their config within a minute; the embedded outpost needs no restart")
+"""
+
+def ak_hosts(dc, old, new, dry=False):
+    script = AK_HOSTS.replace("__OLD__", old).replace("__NEW__", new).replace("__DRY__", "True" if dry else "False")
+    ak_shell(dc, script, f"authentik hosts {old} -> {new}" + (" (dry run)" if dry else ""))
+
+def rebase_hosts(hosts, old, new):
+    """Every role host with its label kept and the base swapped: media.old -> media.new."""
+    out = {}
+    for role, host in hosts.items():
+        if host == old: out[role] = new
+        elif host.endswith("." + old): out[role] = host[: -len(old)] + new
+        else: out[role] = host                       # a host on some other domain is left alone
+    return out
+
 def post_start_fixups(dc):
     """Idempotent first-run state that otherwise needs clicking: Syncthing GUI auth off, Gitea admin user, Jellyfin plugin."""
     env = read_env()
     print("== post-start fixups (idempotent)")
     es_security(dc)              # before anything else: Kibana and the Beats cannot authenticate until this runs
+    # /logout must end the whole Authentik session, not just the application's. The provider-scoped flow
+    # ("Logged out of application") leaves the SSO session alive, so logout looks like a page refresh.
+    ak_shell(dc, AK_LOGOUT_FIXUP, "proxy provider logout flow")
+    ak_branding(dc)              # login page on brand, members forwarded straight to Discord, ?admin=1 for akadmin
     # Syncthing: no GUI login (the Authentik gate is the login); needs insecureAdminAccess since Caddy is not "localhost" to it
     cfg = Path("/srv/media/.syncthing/config.xml")
     if not DRY and cfg.exists():
@@ -835,8 +1259,16 @@ def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
     print(f"== authentik: https://{auth_host}/   akadmin password = AUTHENTIK_BOOTSTRAP_PASSWORD in .env (set on first start; change it in the UI)" if auth_host else "== authentik: (not set — run --full)")
     print(f"== jellyfin : https://{watch_host}/" if watch_host else "== jellyfin : (not set — run --full)")
     env = read_env()
-    tok = env.get("DISCORD_BOT_TOKEN", "")
+    print("== discord  : (./configure.py --set KEY=VALUE)")
-    print(f"== discord  : guild {env.get('DISCORD_GUILD_ID') or '(not set)'}; bot token {'set (' + tok[:6] + '…)' if tok else '(not set)'}")
+    for key, why in DISCORD_KEYS:
+        value = env.get(key, "")
+        if not value:
+            shown = "(not set)"
+        elif key == "DISCORD_GUILD_ID":
+            shown = value
+        else:                                # tokens and webhook URLs are credentials: never print them whole
+            shown = value[:34] + "…" if len(value) > 34 else value[:8] + "…"
+        print(f"     {key:26} {shown:40} {why}")
     print(f"== mariadb  : {domain or 'HOST'}:3306 (public, root local-only)  root pass {env.get('MARIADB_ROOT_PASSWORD','(run apply)')}")
     print(f"     bot ci : Gitea -> VanguardBot -> Settings -> Actions -> Secrets: BOT_DEPLOY_SECRET={env.get('BOT_DEPLOY_SECRET','(run apply)')}")
     print(f"              pushes to main install themselves (BOT_DEPLOY_AUTO={env.get('BOT_DEPLOY_AUTO','1')}); by hand: ./deployBotBundle.sh <bundle>")
@@ -918,7 +1350,7 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
         Path(".env").write_text(f"APEX_HOST={apex}\nAPEX_REDIRECT={apex_to}\n"); os.chmod(".env", 0o600)
     if alert_webhook != env0.get("ALERT_WEBHOOK_URL", ""):
         env = read_env(); env["ALERT_WEBHOOK_URL"] = alert_webhook
-        Path(".env").write_text("".join(f"{k}={v}\n" for k, v in env.items())); os.chmod(".env", 0o600)
+        Path(".env").write_text(render_env(env)); os.chmod(".env", 0o600)
     return domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook, files_host, sync_host, clips_host, diag_host
 
 def ask_discord():
@@ -951,6 +1383,87 @@ def main():
     streamers, accounts = read_streamers(), read_accounts()
     admin_pw = accounts.pop("admin", "")
     editors = accounts
+    if mode == "--set":
+        pairs = [a for a in sys.argv[2:] if "=" in a]
+        if not pairs:
+            print("usage: configure.py --set KEY=VALUE [KEY=VALUE ...]\n\nSettable on their own:")
+            for name, why in SETTABLE.items():
+                print(f"   {name:22} {why}")
+            print('\nexample: ./configure.py --set ALERT_WEBHOOK_URL=https://discord.com/api/webhooks/…')
+            sys.exit(1)
+        set_env(dict(p.split("=", 1) for p in pairs)); return
+    if mode == "--alias":
+        # Step one of a no-outage base move: every site answers on BOTH bases. Only the Caddyfile changes
+        # (plus one .env key) and Caddy reloads; certificates for the new names are issued in the background.
+        # --alias "" takes the aliases away again. Needs the new names to resolve (certificates).
+        if len(sys.argv) != 3:
+            print('usage: configure.py --alias NEW-DOMAIN     (e.g. --alias vng.one ; --alias "" to remove)'); sys.exit(2)
+        if not domain: print("domain not set yet — run ./configure.py --full first"); sys.exit(1)
+        alt = sys.argv[2].strip().lower()
+        if alt:
+            for r, h in rebase_hosts(hosts, domain.split(".", 1)[1], alt).items():
+                if r in ("main", "auth", "watch", "git", "bot", "live", "files", "sync", "clips", "diag"):
+                    try: socket.getaddrinfo(h, 443)
+                    except OSError: print(f"   !! {h} does not resolve yet — add the DNS first"); sys.exit(1)
+        env0 = read_env(); env0["ALT_BASE"] = alt
+        Path(".env").write_text(render_env(env0)); os.chmod(".env", 0o600)
+        write_caddy(domain, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); decorate_caddy()
+        dc = compose_cmd()
+        if not DRY: run(dc + ["exec", "caddy", "caddy", "reload", "--config", "/etc/caddy/Caddyfile"])
+        print(f"   ALT_BASE={alt or '(cleared)'}; Caddy reloaded. Nothing else changed." + (f" Check: curl -sI https://auth.{alt}/ | head -1" if alt else ""))
+        return
+    if mode == "--base":
+        # Move every host to a new base domain in ONE apply, labels kept: --base vng.one turns media.old into
+        # media.vng.one, files.old into files.vng.one, … Needs the DNS to exist already (certificates). Does not
+        # touch the things that must come later (HOSTNAME-CHANGE.md §B): Jellyfin/Gitea UI settings, then
+        # --ak-hosts OLD NEW for Authentik, then the Discord portal.
+        if len(sys.argv) != 3 or "." not in sys.argv[2]:
+            print("usage: configure.py --base NEW-DOMAIN     (e.g. --base vng.one)"); sys.exit(2)
+        if not domain: print("domain not set yet — run ./configure.py --full first"); sys.exit(1)
+        old_base = domain.split(".", 1)[1] if "." in domain else domain
+        new_base = sys.argv[2].strip().lower()
+        moved = rebase_hosts(hosts, old_base, new_base)
+        unchanged = [f"{r}={h}" for r, h in hosts.items() if moved[r] == h]
+        print(f"== base domain {old_base} -> {new_base}")
+        for r in ("main", "auth", "watch", "git", "bot", "live", "files", "sync", "clips", "diag"):
+            if r in hosts: print(f"   {r:6} {hosts[r]:34} -> {moved[r]}")
+        if unchanged: print("   left alone (not under the old base):", ", ".join(unchanged))
+        env0 = read_env()
+        # the alias did its job; from now on the OLD names are redirects to the new ones (bookmarks keep working)
+        if env0.get("ALT_BASE", "").strip().lower() in (new_base, old_base): env0["ALT_BASE"] = ""
+        env0["REDIRECT_BASE"] = old_base
+        Path(".env").write_text(render_env(env0)); os.chmod(".env", 0o600)
+        print(f"   old names under {old_base} will 308 to the new ones (REDIRECT_BASE); --alias \"\" later removes nothing, edit REDIRECT_BASE= in .env to drop them")
+        apex = env0.get("APEX_HOST", "")
+        if apex and (apex == old_base or apex.endswith("." + old_base)):
+            # The bare old domain cannot stay: its Caddy block would keep asking for a certificate for a name that
+            # is no longer ours to serve. The bare NEW domain is not assumed either (it needs its own A record at
+            # the box, and today's does not point here): add it later with --full once that record exists.
+            print(f"   APEX_HOST={apex} -> cleared (bare domain of the old base; re-add the new one with --full once an A record points here)")
+            env0["APEX_HOST"] = ""; env0["APEX_REDIRECT"] = ""        # not a SETTABLE key: written directly, the apply below regenerates Caddy
+            Path(".env").write_text(render_env(env0)); os.chmod(".env", 0o600)
+        for r, h in moved.items():
+            if r in ("main", "auth", "watch", "git", "bot", "live", "files", "sync", "clips", "diag"):
+                try: socket.getaddrinfo(h, 443)
+                except OSError: print(f"   !! {h} does not resolve yet — Caddy cannot get its certificate; add the DNS first"); sys.exit(1)
+        if not yes(f"\napply now? Caddy, Jellyfin, Gitea, the runner, the bot and Kibana move to {new_base}; gated hosts stay unreachable until --ak-hosts"):
+            print("nothing written"); return
+        apply(moved["main"], admin_pw, streamers, editors, moved.get("auth", ""), moved.get("watch", ""), git_host=moved.get("git", ""), bot_host=moved.get("bot", ""),
+              live_host=moved.get("live", ""), files_host=moved.get("files", ""), sync_host=moved.get("sync", ""), clips_host=moved.get("clips", ""), diag_host=moved.get("diag", ""))
+        show(moved["main"], admin_pw, streamers, editors, moved.get("auth", ""), moved.get("watch", ""), moved.get("git", ""), moved.get("bot", ""), moved.get("live", ""), moved.get("files", ""), moved.get("sync", ""), moved.get("clips", ""), moved.get("diag", ""))
+        print(f"\nNEXT, in this order (HOSTNAME-CHANGE.md §B): docker compose restart mediamtx (between streams) → Jellyfin UI → Gitea UI + runner + clone remote → ./configure.py --ak-hosts {old_base} {new_base} → ./configure.py --ak-branding (brand default app follows the provider) → Discord portal → sudo vghosts-check.py")
+        return
+    if mode == "--ak-hosts":
+        # Authentik's hostnames, OLD -> NEW. Separate from --base and from --apply by design: it is the last
+        # service step of a domain move (apps before Authentik). --dry-run prints what would change.
+        rest = [a for a in sys.argv[2:] if not a.startswith("--")]
+        if len(rest) != 2:
+            print("usage: configure.py --ak-hosts OLD-DOMAIN NEW-DOMAIN [--dry-run]     (e.g. --ak-hosts dumpstervigilantes.rip vng.one)"); sys.exit(2)
+        ak_hosts(compose_cmd(), rest[0].strip().lower(), rest[1].strip().lower(), dry="--dry-run" in sys.argv); return
+    if mode == "--ak-branding":
+        # Just the Authentik login branding + Discord forward (also part of every --apply). On its own so a
+        # change to authentik/brand/*.svg or the CSS can be pushed without touching anything else.
+        install_brand_assets(); ak_branding(compose_cmd()); return
     if mode == "--es-security":
         # Just the Elasticsearch user bootstrap. Worth having on its own: if it is skipped or fails during an
         # --apply, Kibana reports "unavailable", Metricbeat exits on Kibana's 503 and Filebeat sits up but

deploy.sh

@@ -46,20 +46,31 @@ apply_changes() {
   if [[ -x check.sh ]]; then asowner bash check.sh >/tmp/vg-check.log 2>&1 || { log "check.sh FAILED — aborting"; tail -20 /tmp/vg-check.log; notify "❌ deploy $new: check.sh failed"; return 1; }; fi
 
   # 1. host scripts + units
-  if has '^bin/'; then run install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/; fi
+  # bin/install.list, not a second hand-kept list: this one had drifted to five of ten files, so a push
+  # never installed vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script.
+  if has '^bin/'; then
+    while read -r mode path; do
+      [[ -f $path ]] && run install -m "$mode" "$path" /usr/local/bin/
+    done < <(grep -vE '^[[:space:]]*(#|$)' bin/install.list)
+  fi
   if has '^systemd/'; then run install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/; run systemctl daemon-reload; fi
   if has '^deploy\.sh$'; then run install -m 0755 deploy.sh /usr/local/bin/vanguard-deploy; fi
   if has '^deployBotBundle\.sh$'; then run install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot; fi
   if has '^firewall/nftables\.conf$'; then run install -m 0644 firewall/nftables.conf /etc/nftables.conf; run nft -f /etc/nftables.conf; fi
 
   # 2. regenerate configs (Caddy reload, copyparty restart) — configure.py decides; MediaMTX is handled below
-  if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/'; then
+  if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/|^stuck/|^observability/kibana-setup\.py$'; then
     local before_mtx; before_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
     if [[ $DRY == 1 ]]; then log "\$ (as $OWNER) VG_DRY=1 python3 configure.py --apply"; else sudo -u "$OWNER" -H python3 configure.py --apply >/tmp/vg-configure.log 2>&1 || { log "configure --apply FAILED"; tail -20 /tmp/vg-configure.log; return 1; }; fi
     local after_mtx; after_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
     [[ $before_mtx == "$after_mtx" ]] || { log "mediamtx.yml changed"; restart_mediamtx_or_defer; }
   fi
   if has '^bin/dumpd\.py$|^systemd/dumpd\.service$'; then run systemctl restart dumpd; fi
+  # Beat and Kibana configs are bind-mounted, so a content change needs the container recreated. Nothing
+  # handled this before: a push that changed filebeat.yml silently did nothing.
+  if has '^observability/filebeat\.yml$'; then run docker compose up -d --force-recreate filebeat; fi
+  if has '^observability/metricbeat\.yml$'; then run docker compose up -d --force-recreate metricbeat; fi
+  if has '^observability/kibana\.yml$'; then run docker compose up -d --force-recreate kibana; fi
 
   # 3. compose: recreate changed services, but never MediaMTX while live
   if has '^docker-compose\.yml$|^bot/Dockerfile$|^bot/entrypoint\.sh$'; then

deployLive.sh

@@ -2,7 +2,7 @@
 # Build the live portal here and push the static files to /srv/live on the server. Caddy serves them as-is.
 set -euo pipefail
 cd "$(dirname "$0")/live"
-HOST="${1:-debian@media.dumpstervigilantes.rip}"   # any name of the box works for ssh
+HOST="${1:-debian@media.vng.one}"   # any name of the box works for ssh
 npm install --no-audit --no-fund --loglevel=error
 npm run build
 ssh "$HOST" 'sudo install -d -o "$(id -u)" -g "$(id -g)" /srv/live && rm -rf /srv/live/*'

dock/index.html

@@ -30,6 +30,12 @@
   .kv .k{color:var(--mut)}
   .tok{letter-spacing:1px}
   .who{color:var(--mut);font-size:12px}
+  /* post-Go-Live health check: what the server is actually receiving */
+  #health{margin-top:10px;padding:8px;border-radius:6px;font-size:13px;white-space:pre-wrap;word-break:break-word}
+  #health.hidden{display:none}
+  #health.bad{background:#43242a;color:#ffd0d0;border:1px solid var(--bad)}
+  #health.bad code{background:#1a1a1a;padding:1px 5px;border-radius:4px;color:var(--fg)}
+  #health.ok{color:var(--ok)}
 </style>
 </head>
 <body>
@@ -53,6 +59,11 @@
     <label>Ingest host</label><input id="host" placeholder="media.example.com">
     <label>OBS WebSocket password</label><input id="obspw" type="password">
     <label>OBS WebSocket port</label><input id="obsport" value="4455">
+    <label>OBS profile to stream from</label><input id="profile" value="Vanguard">
+    <label style="display:flex;gap:8px;align-items:center;margin-top:8px"><input type="checkbox" id="enforce" style="width:auto;margin:0"> switch OBS to this profile when the dock loads</label>
+    <div class="who">Encoder settings (B-frames, keyframe interval) live in an OBS profile and obs-websocket cannot change them,
+    but it can switch profiles. Create one named <b>Vanguard</b> once: Profile → New, then Output → Advanced → your encoder
+    with <b>B-frames 0</b> and keyframe interval 2 s. Go Live always switches to it; this box also does it on startup.</div>
     <label>Title prefix (optional, e.g. <code>HelmetCam-</code>)</label><input id="prefix" placeholder="">
     <label>Title suffix (optional; <code>{date}</code> becomes YYYY-MM-DD)</label><input id="suffix" placeholder="-{date}">
     <button id="save">Save settings</button>
@@ -69,6 +80,7 @@
     <button class="small" id="tc_remove">Remove band</button>
   </details>
 
+  <div id="health" class="hidden"></div>
   <details id="manual">
     <summary>Manual OBS setup (endpoints + token)</summary>
     <div class="kv"><span class="k">RTMP server  </span><span id="m_rtmp">—</span></div>
@@ -85,11 +97,12 @@
 
 <script>
 // ---- settings --------------------------------------------------------------
-const KEYS = ["host","obspw","obsport","prefix","suffix"];
+const KEYS = ["host","obspw","obsport","prefix","suffix","profile"];
+const BOOLS = ["enforce"];
 const $ = id => document.getElementById(id);
 const status = (msg, cls) => { $("status").textContent = msg; $("status").style.color = cls==="bad"?"#e74c3c":cls==="ok"?"#2ecc71":"#9a9a9a"; };
-function load(){ try{ KEYS.forEach(k => { const v = localStorage.getItem("vg."+k); if (v!==null) $(k).value = v; }); }catch(e){} if(!$("host").value) $("host").value = location.hostname; }
+function load(){ try{ KEYS.forEach(k => { const v = localStorage.getItem("vg."+k); if (v!==null) $(k).value = v; }); BOOLS.forEach(k => { $(k).checked = localStorage.getItem("vg."+k) === "1"; }); }catch(e){} if(!$("host").value) $("host").value = location.hostname; }
-function save(){ try{ KEYS.forEach(k => localStorage.setItem("vg."+k, $(k).value.trim())); status("settings saved"); }catch(e){ status("could not save settings (storage blocked)", "bad"); } renderManual(); }
+function save(){ try{ KEYS.forEach(k => localStorage.setItem("vg."+k, $(k).value.trim())); BOOLS.forEach(k => localStorage.setItem("vg."+k, $(k).checked ? "1" : "0")); status("settings saved"); }catch(e){ status("could not save settings (storage blocked)", "bad"); } renderManual(); }
 load(); $("save").onclick = save;
 const cfg = () => Object.fromEntries(KEYS.map(k => [k, $(k).value.trim()]));
 
@@ -99,7 +112,7 @@ function buildTitle(raw){
   const c = cfg();
   let t = (c.prefix||"") + raw.trim() + (c.suffix||"").replace("{date}", today());
   t = t.replace(/\s+/g, "-")              // spaces -> dashes
-       .replace(/[^A-Za-z0-9_.-]/g, "")   // allowed characters only
+       .replace(/[^A-Za-z0-9_-]/g, "")    // allowed characters only; no dots: <title>.webrtc is the audio mirror's name
        .replace(/_{2,}/g, "_")            // "__" is reserved: it separates title from timestamp in filenames
        .replace(/^[.-]+/, "")             // no leading dot (hidden) or dash
        .slice(0, 64);
@@ -108,7 +121,7 @@ function buildTitle(raw){
 function ruleText(raw){
   const r = [];
   if (/__/.test(raw)) r.push("double underscore is reserved (becomes single)");
-  if (/[^A-Za-z0-9_.\- ]/.test(raw)) r.push("only letters, digits, space, _ . - are kept");
+  if (/[^A-Za-z0-9_\- ]/.test(raw)) r.push("only letters, digits, space, _ - are kept");
   if (raw.trim().startsWith(".")) r.push("cannot start with a dot");
   if ((cfg().prefix + raw + cfg().suffix).length > 64) r.push("longer than 64 characters: trimmed");
   return r.join(" · ");
@@ -177,6 +190,28 @@ function obs(c){
   });
 }
 
+// ---- profile enforcement ----------------------------------------------------------------------------
+// Encoder settings (B-frames above all: MediaMTX refuses them over WebRTC) cannot be set through
+// obs-websocket in Advanced output mode — they live in the profile's streamEncoder.json, which the protocol
+// does not expose. Profiles can be switched, though, so the setting is enforced by making sure OBS is on a
+// profile that already has it. The post-Go-Live health check then verifies the actual encode.
+// Returns true when OBS is on the wanted profile, false (with a status line saying why) otherwise.
+async function ensureProfile(o, c) {
+  if (!c.profile) return true;
+  const { profiles, currentProfileName } = await o.call("GetProfileList");
+  if (!profiles.includes(c.profile)) {
+    status(`OBS has no profile named "${c.profile}". Create it once (Profile → New) with B-frames 0, or clear the name in settings. Streaming on "${currentProfileName}".`, "bad");
+    return false;
+  }
+  if (currentProfileName === c.profile) return true;
+  const st = await o.call("GetStreamStatus");
+  if (st.outputActive) { status(`OBS is on "${currentProfileName}" but streaming; not switching mid-stream`, "bad"); return false; }
+  await o.call("SetCurrentProfile", { profileName: c.profile });
+  await new Promise(r => setTimeout(r, 800));            // OBS reloads its outputs on a profile switch
+  status(`OBS switched to profile "${c.profile}"`, "ok");
+  return true;
+}
+
 // ---- timecode band (obs-websocket) -------------------------------------------------------------------
 const CLOCK_INPUT = "Vanguard Clock";
 const bandFor = w => { let b = Math.round(w / 160); return b + (b % 2); };
@@ -260,15 +295,66 @@ $("golive").onclick = async () => {
     const t = await mint();                                   // fresh token every Go Live; nothing stored
     status("connecting to OBS…");
     const o = await obs(c);
+    await ensureProfile(o, c);                                // the Vanguard profile carries B-frames 0; switch first
     await o.call("SetStreamServiceSettings", { streamServiceType: "rtmp_custom",
       streamServiceSettings: { server: `rtmp://${c.host}:1935/`, key: `${t.user}/${title}?user=${encodeURIComponent(t.user)}&pass=${encodeURIComponent(t.token)}`, use_auth: false } });
+    // B-frames make the stream unwatchable on the portal (MediaMTX refuses them over WebRTC). obs-websocket
+    // can only set encoder options in Simple output mode with x264, so do that much here; everything else is
+    // caught after start by checkHealth(), which reads the actual recording on the server.
+    try {
+      const mode = (await o.call("GetProfileParameter", { parameterCategory: "Output", parameterName: "Mode" })).parameterValue;
+      if (mode === "Simple") {
+        const enc = (await o.call("GetProfileParameter", { parameterCategory: "SimpleOutput", parameterName: "StreamEncoder" })).parameterValue || "";
+        if (/x264/i.test(enc)) {
+          const cur = (await o.call("GetProfileParameter", { parameterCategory: "SimpleOutput", parameterName: "x264Settings" })).parameterValue || "";
+          if (!/\b(bf|bframes)=0\b/.test(cur)) {
+            await o.call("SetProfileParameter", { parameterCategory: "SimpleOutput", parameterName: "x264Settings", parameterValue: (cur + " bf=0").trim() });
+            status("set x264 bf=0 in OBS (Simple output) so viewers can watch over WebRTC");
+          }
+        }
+      }
+    } catch (e) { /* older obs-websocket, or a locked profile: the health check will say if it matters */ }
     const st = await o.call("GetStreamStatus");
     if (st.outputActive) { await o.call("StopStream"); await new Promise(r => setTimeout(r, 1500)); }
     await o.call("StartStream"); o.close();
     status(`LIVE  →  ${t.user}/${title}`, "ok");
+    checkHealth(90);
   } catch (e) { status(e.message, "bad"); }
   $("golive").disabled = false;
 };
+// After Go Live, ask the server what it is actually receiving. The recording's first fragments reveal the
+// B-frame depth within seconds, and B-frames mean every WebRTC viewer is refused. Polls until the stream
+// shows up and is judged, or for `seconds`, whichever comes first.
+let healthTimer = null;
+async function checkHealth(seconds) {
+  clearInterval(healthTimer);
+  const until = Date.now() + seconds * 1000, box = $("health");
+  box.className = "who"; box.textContent = "checking what the server is receiving…";
+  healthTimer = setInterval(async () => {
+    try {
+      const r = await fetch("/me/stream-health", { credentials: "same-origin" });
+      if (!r.ok) throw new Error("HTTP " + r.status);
+      const d = await r.json();
+      const judged = (d.streams || []).filter(s => s.bframes != null);
+      if (judged.length) {
+        clearInterval(healthTimer);
+        const bad = judged.filter(s => !s.webrtc_ok);
+        if (bad.length) {
+          box.className = "bad";
+          box.innerHTML = `<b>Viewers cannot watch this stream over WebRTC.</b> Your encoder is sending B-frames `
+            + `(${bad.map(s => `${s.profile || "H.264"} ${s.res || ""}, B-frame depth ${s.bframes}`).join("; ")}). `
+            + `In OBS: Output → Advanced → <b>x264</b>: add <code>bf=0</code> to the options; <b>NVENC</b>: Max B-frames = 0; `
+            + `<b>AMF/QSV</b>: B-frames = 0. Then stop and start the stream. The recording itself is fine.`;
+        } else {
+          box.className = "ok";
+          box.textContent = `server receiving ${judged.map(s => `${s.res || ""} ${s.fps ? s.fps + " fps" : ""}`.trim()).join(", ")} with no B-frames — WebRTC viewers are good`;
+        }
+        return;
+      }
+    } catch (e) { /* not up yet, or the dock is not behind the proxy; keep trying until the deadline */ }
+    if (Date.now() > until) { clearInterval(healthTimer); box.className = "who"; box.textContent = "could not confirm what the server is receiving (is the stream up?)"; }
+  }, 6000);
+}
 $("stop").onclick = async () => { try { const o = await obs(cfg()); await o.call("StopStream"); o.close(); status("stream stopped"); } catch (e) { status(e.message, "bad"); } };
 
 let armed = 0;
@@ -290,6 +376,7 @@ $("dump").onclick = async () => {
   } catch (e) { status("dump failed: " + e.message, "bad"); }
 };
 
+
 $("signout").onclick = async e => {
   e.preventDefault(); status("signing out…");
   // Authentik's sign-out redirects to the site root (copyparty); do it in the background, then reload the dock,
@@ -304,6 +391,11 @@ $("signout").onclick = async e => {
   await whoami();
   try { await mint(); status("ready"); }
   catch (e) { status(e.message, "bad"); $("m_tok").textContent = "— " + e.message; }
+  // "switch OBS to this profile when the dock loads": inside OBS the dock loads with OBS, so this is startup
+  if ($("enforce").checked) {
+    try { const o = await obs(cfg()); await ensureProfile(o, cfg()); o.close(); }
+    catch (e) { status(`profile check on startup: ${e.message}`, "bad"); }
+  }
 })();
 renderPreview();
 </script>

docker-compose.yml

@@ -15,15 +15,13 @@ services:
       TZ: America/New_York          # recording filenames are stamped in the container's clock; must match the host
       LIVE_WEBHOOK_URL: ${LIVE_WEBHOOK_URL:-}
       LIVE_HOST: ${LIVE_HOST:-live.example.com}
-    ports:
+    # Host network, as MediaMTX recommends for WebRTC. On the bridge, every WebRTC media packet on 8189/udp
-      - "1935:1935"              # RTMP ingest
+    # was relayed by Docker's userland proxy (docker-proxy), which lost ~0.4% of them at a thousand packets a
-      - "8890:8890/udp"          # SRT ingest
+    # second; each loss corrupted an access unit, the recorder reopened its file, and one evening produced 471
-      - "8189:8189/udp"          # WebRTC media (viewers + Go Live)
+    # fragments. Publishers also all appeared as the bridge gateway (172.18.0.1). Now MediaMTX binds the host
-      - "8189:8189/tcp"          # WebRTC ICE-TCP fallback
+    # directly: ingest ports (1935, 8890/udp, 8189) are open in the nftables INPUT chain, and everything Caddy
-      - "127.0.0.1:8888:8888"    # HLS      -> Caddy live.HOST/hls/*
+    # or Metricbeat talks to is bound to 127.0.0.1 in mediamtx.yml. No ports: block, nothing to publish.
-      - "127.0.0.1:8889:8889"    # WebRTC   -> Caddy live.HOST/webrtc/*
+    network_mode: host
-      - "127.0.0.1:9997:9997"    # API      -> Caddy live.HOST/api/*
-      - "127.0.0.1:9998:9998"    # Prometheus metrics (per-stream bitrate, readers, drops) -> scraper on the host
     volumes:
       - ./mediamtx.yml:/mediamtx.yml:ro
       - ./bin/mtx-on-ready.sh:/vanguard/on-ready.sh:ro   # is-live notice + WebRTC audio mirror (runOnReady)
@@ -72,6 +70,7 @@ services:
       - ./dock:/srv/dock:ro
       - ./clips:/srv/clips:ro               # clip-intake self-service page
       - ./about:/srv/about:ro               # gateway page (/about)
+      - ./streaming:/srv/streaming:ro       # Live + Stuck tabs (diag host /streaming)
       - /srv/live:/srv/live:ro              # live portal static build (deployLive.sh)
       - caddy_data:/data
       - caddy_config:/config

firewall/nftables.conf

@@ -1,6 +1,7 @@
 #!/usr/sbin/nft -f
 # vanguard-stream host firewall. Own table so (re)applying never disturbs Docker's rules.
-# Input only: Docker-published ports (1935, 8890, 22000) arrive via Docker's forward chain and are
+# Input only. MediaMTX runs on the host network, so 1935, 8890/udp and 8189 arrive through THIS chain (the
+# accepts below). Docker-published ports (22000) arrive via Docker's forward chain and are
 # listed here for the host-network case (Caddy) and clarity. Nothing else should listen publicly.
 
 destroy table inet vanguard

install.sh

@@ -56,8 +56,11 @@ for d in /srv/media/clips/*/; do
 done
 
 echo "== host scripts + units"
-install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
+# bin/install.list is the single list, shared with configure.py --apply and deploy.sh. Three hand-kept
-install -m 0644 bin/vgmetrics.sh /usr/local/bin/      # sourced by the sweep/vgdeband; counters for dumpd's /metrics
+# copies had drifted, so a push-to-deploy was installing half of them.
+grep -vE '^\s*(#|$)' bin/install.list | while read -r mode path; do
+  install -m "$mode" "$path" /usr/local/bin/
+done
 install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot   # dumpd runs this for the bot CI's POST /deploy-bot
 install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/      # watches the observability stack from outside it
 install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/        # takes Kibana's alert documents to Discord

live/src/GoLive.jsx

@@ -1,5 +1,5 @@
 import React, { useEffect, useRef, useState } from 'react'
-import { captureWindow, publish, fetchStreamKey, defaultTitle, cleanTitle, withTimecodeBand, MAX_KBPS } from './broadcast.js'
+import { captureWindow, publish, fetchStreamKey, defaultTitle, cleanTitle, withTimecodeBand, MAX_KBPS, windowLoss, shouldSwitchToTcp } from './broadcast.js'
 
 // Broadcaster view: replaces the viewer grid while you're live.
 export default function GoLive({ onExit }) {
@@ -16,6 +16,81 @@ export default function GoLive({ onExit }) {
   const [hasAudio, setHasAudio] = useState(true)
   const [tcBand, setTcBand] = useState(() => { try { return localStorage.getItem('vg.tcband') !== '0' } catch { return true } })
   const bandRef = useRef(null)
+  const streamRef = useRef(null)                      // the tracks being published (survive a transport switch)
+  const samplesRef = useRef([])                       // recent {t, lost, sent} for the loss window
+  const switchingRef = useRef(false)
+  const [transport, setTransport] = useState('udp')
+  const [switched, setSwitched] = useState('')        // "switched to TCP after 0.6% loss" once it happens
+  // Remembered per browser. Starlink drops packets in bursts on every satellite handoff (its scheduler works in
+  // ~15 s cycles), so a dish user hits the auto-switch on every single stream; once it has fired, the next stream
+  // starts on TCP and skips the 20 s of corrupted recording and the extra file. Untick in setup to go back.
+  const [preferTcp, setPreferTcp] = useState(() => { try { return localStorage.getItem('vg.golive.tcp') === '1' } catch { return false } })
+  const [ended, setEnded] = useState('')               // why the server ended the stream, shown back on the setup card
+  const [surface, setSurface] = useState(() => { try { return localStorage.getItem('vg.golive.surface') === 'screen' ? 'screen' : 'window' } catch { return 'window' } })
+  const pickSurface = v => { setSurface(v); try { localStorage.setItem('vg.golive.surface', v) } catch {} }
+  const missingRef = useRef(0)                          // consecutive health polls that did not list our path
+  const disconnectRef = useRef(null)
+
+  // A kick is invisible to WebRTC for a long time: the browser sits in "disconnected" until ICE consent times
+  // out (~30 s) and only then goes "failed". So: treat failed/closed as the end immediately, give
+  // "disconnected" 8 s to recover, and ask the server every 5 s whether our path is still live — the kick
+  // path answers that in seconds and tells us whether we are blocked.
+  const endedByServer = async (why) => {
+    if (!sessionRef.current) return
+    const s = sessionRef.current; sessionRef.current = null
+    try { await s.stop() } catch {}
+    streamRef.current?.getTracks().forEach(t => t.stop()); streamRef.current = null
+    bandRef.current?.stop(); bandRef.current = null
+    if (videoRef.current) videoRef.current.srcObject = null
+    setPhase('setup'); setStarted(null); setStats(null); setConn(''); setTransport('udp'); setSwitched('')
+    setEnded(why)
+  }
+  const onConn = (state) => {
+    setConn(state)
+    if (switchingRef.current) return
+    if (state === 'failed' || state === 'closed') endedByServer('The server ended the stream.')
+    if (state === 'disconnected' && !disconnectRef.current) {
+      disconnectRef.current = setTimeout(() => { disconnectRef.current = null; if (sessionRef.current?.pc?.connectionState !== 'connected') endedByServer('Connection to the server was lost.') }, 8000)
+    }
+    if (state === 'connected' && disconnectRef.current) { clearTimeout(disconnectRef.current); disconnectRef.current = null }
+  }
+  useEffect(() => {
+    if (phase !== 'live') return
+    missingRef.current = 0
+    const t = setInterval(async () => {
+      const s = sessionRef.current; if (!s || switchingRef.current) return
+      try {
+        const r = await fetch('/me/stream-health', { credentials: 'same-origin' }); if (!r.ok) return
+        const d = await r.json()
+        const mine = (d.streams || []).some(x => x.path.toLowerCase() === s.path.toLowerCase())
+        missingRef.current = mine ? 0 : missingRef.current + 1
+        if (missingRef.current >= 2) {
+          endedByServer(d.blocked ? `An admin stopped your stream. Publishing is paused for your account until ${new Date(d.blocked.until).toLocaleTimeString()}.`
+                                  : 'The server ended the stream.')
+        }
+      } catch {}
+    }, 5000)
+    return () => clearInterval(t)
+  }, [phase])
+  const rememberTcp = on => { setPreferTcp(on); try { localStorage.setItem('vg.golive.tcp', on ? '1' : '0') } catch {} }
+
+  // Lossy uplink: re-publish the same tracks over TCP. One new recording file (the sweep stitches it), ~2 s
+  // of picture lost during the swap, and from then on no packet loss can reach the recorder. Never flips back:
+  // the latency cost of TCP is a nuisance, a corrupted recording is a loss.
+  const switchToTcp = async (loss) => {
+    const old = sessionRef.current; if (!old || switchingRef.current || old.transport === 'tcp') return
+    switchingRef.current = true
+    setConn(`loss ${loss.pct.toFixed(1)}% — switching to TCP…`)
+    try {
+      await old.release()
+      const next = await publish({ stream: streamRef.current, path: old.path, user: old.user, pass: old.pass, onState: onConn, transport: 'tcp' })
+      sessionRef.current = next; samplesRef.current = []
+      setTransport(next.transport)
+      if (next.transport === 'tcp') { rememberTcp(true); setSwitched(`switched to TCP after ${loss.pct.toFixed(1)}% loss — your next streams start on TCP`) }
+      else setSwitched('server offered no TCP candidate; still on UDP')
+    } catch (e) { setErr(`re-publish over TCP failed: ${e.message}`); setPhase('error') }
+    finally { switchingRef.current = false }
+  }
 
   useEffect(() => { fetchStreamKey().then(setKey).catch(e => { setErr(e.message); setPhase('error') }) }, [])
   useEffect(() => { if (!started) return; const t = setInterval(() => setElapsed(Math.floor((Date.now() - started) / 1000)), 1000); return () => clearInterval(t) }, [started])
@@ -25,11 +100,21 @@ export default function GoLive({ onExit }) {
     const t = setInterval(async () => {
       const pc = sessionRef.current?.pc; if (!pc) return
       const report = await pc.getStats()
-      let codec = ''
+      let codec = '', lost = 0, sent = 0
+      for (const s of report) if (s.type === 'remote-inbound-rtp' && s.kind === 'video') lost = s.packetsLost || 0   // what the server reports back
       for (const s of report) if (s.type === 'outbound-rtp' && s.kind === 'video') {
+        sent = s.packetsSent || 0
         const now = Date.now(); const kbps = Math.round((s.bytesSent - last.bytes) * 8 / ((now - last.t) || 1)); last = { bytes: s.bytesSent, t: now }
         const c = s.codecId && report.get(s.codecId); codec = c ? `${c.mimeType.replace('video/', '')}${/profile-level-id=64/i.test(c.sdpFmtpLine || '') ? ' High' : /profile-level-id=4d/i.test(c.sdpFmtpLine || '') ? ' Main' : /profile-level-id=42/i.test(c.sdpFmtpLine || '') ? ' Baseline' : ''}` : ''
-        setStats({ kbps, fps: Math.round(s.framesPerSecond || 0), w: s.frameWidth || 0, h: s.frameHeight || 0, q: s.qualityLimitationReason, enc: s.encoderImplementation || '', codec })
+        // nack = retransmit requests the server sent us; rtx = packets we re-sent. Both 0 under loss means the
+        // server is not asking for retransmits, and every lost packet is a corrupted frame in the recording.
+        setStats({ kbps, fps: Math.round(s.framesPerSecond || 0), w: s.frameWidth || 0, h: s.frameHeight || 0, q: s.qualityLimitationReason, enc: s.encoderImplementation || '', codec,
+                   lost, sent, nack: s.nackCount || 0, rtx: s.retransmittedPacketsSent || 0 })
+      }
+      if (sent && !switchingRef.current) {
+        const samples = samplesRef.current; samples.push({ t: Date.now(), lost, sent }); if (samples.length > 60) samples.shift()
+        const l = windowLoss(samples)
+        if (sessionRef.current?.transport === 'udp' && shouldSwitchToTcp(l)) switchToTcp(l)
       }
     }, 1000)
     return () => clearInterval(t)
@@ -41,19 +126,25 @@ export default function GoLive({ onExit }) {
     const t = cleanTitle(title); if (!t) return setErr('give it a title')
     if (!key) return setErr('no stream key')
     try {
+      // blocked by an admin? MediaMTX would answer the WHIP with a bare 401, which reads like a bad key — ask first
+      try { const h = await (await fetch('/me/stream-health', { credentials: 'same-origin' })).json()
+            if (h.blocked) return setErr(`Publishing is paused for your account by an admin until ${new Date(h.blocked.until).toLocaleTimeString()}.`) } catch {}
       setPhase('capturing')
-      const captured = await captureWindow()
+      const captured = await captureWindow(surface)
       setHasAudio(captured.getAudioTracks().length > 0)
       captured.getVideoTracks()[0].onended = () => stop()            // user clicked Chrome's "Stop sharing"
       let stream = captured
       if (tcBand) { bandRef.current = withTimecodeBand(captured); stream = bandRef.current.stream }
       if (videoRef.current) { videoRef.current.srcObject = stream; videoRef.current.muted = true; videoRef.current.play().catch(() => {}) }
-      const session = await publish({ stream, path: `${key.user}/${t}`, user: key.user, pass: key.pass, onState: setConn })
+      streamRef.current = stream; samplesRef.current = []; setSwitched('')
-      sessionRef.current = session
+      setEnded('')
+      const session = await publish({ stream, path: `${key.user}/${t}`, user: key.user, pass: key.pass, onState: onConn, transport: preferTcp ? 'tcp' : 'udp' })
+      sessionRef.current = session; setTransport(session.transport)
+      if (preferTcp && session.transport !== 'tcp') setSwitched('server offered no TCP candidate; on UDP')
       setStarted(Date.now()); setPhase('live')
     } catch (e) { setErr(e.message); setPhase(e.message === 'capture cancelled' ? 'setup' : 'error') }
   }
-  const stop = async () => { await sessionRef.current?.stop(); sessionRef.current = null; bandRef.current?.stop(); bandRef.current = null; if (videoRef.current) videoRef.current.srcObject = null; setPhase('setup'); setStarted(null); setStats(null); setConn('') }
+  const stop = async () => { await sessionRef.current?.stop(); sessionRef.current = null; streamRef.current?.getTracks().forEach(t => t.stop()); streamRef.current = null; bandRef.current?.stop(); bandRef.current = null; if (videoRef.current) videoRef.current.srcObject = null; setPhase('setup'); setStarted(null); setStats(null); setConn(''); setTransport('udp'); setSwitched('') }
 
   const mmss = s => `${String(Math.floor(s / 3600)).padStart(2, '0')}:${String(Math.floor(s / 60) % 60).padStart(2, '0')}:${String(s % 60).padStart(2, '0')}`
   return (
@@ -69,7 +160,7 @@ export default function GoLive({ onExit }) {
         <video ref={videoRef} autoPlay playsInline muted className={phase === 'live' || phase === 'capturing' ? 'show' : ''} />
         {phase !== 'live' && (
           <div className="golive-form">
-            <label>Stream title <small>(becomes the Jellyfin title; letters, digits, _ . - ; no double underscore)</small></label>
+            <label>Stream title <small>(becomes the Jellyfin title; letters, digits, _ - ; no double underscore)</small></label>
             <input value={title} onChange={e => setTitle(e.target.value)} disabled={phase === 'capturing'} />
             <div className="hint" style={{marginTop: '-4px'}}>stream name: <b>{key ? key.user : '<you>'}/{cleanTitle(title) || '—'}</b></div>
             <div className="hint">
@@ -78,11 +169,28 @@ export default function GoLive({ onExit }) {
               encoder. On Linux and Mac Chrome has no window audio and (on NVIDIA) no hardware encoder, so the stream is
               silent and visibly soft — use OBS with the dock there. Capped at {MAX_KBPS / 1000} Mbps, up to 1440p60, H.264.
             </div>
+            <div className="hint" style={{display:'flex',gap:16,alignItems:'center',flexWrap:'wrap'}}>
+              <span>Capture:</span>
+              <label style={{display:'flex',gap:6,alignItems:'center'}}><input type="radio" name="surface" checked={surface === 'window'} onChange={() => pickSurface('window')} /> the game's window</label>
+              <label style={{display:'flex',gap:6,alignItems:'center'}}><input type="radio" name="surface" checked={surface === 'screen'} onChange={() => pickSurface('screen')} /> the whole screen</label>
+            </div>
+            <div className="hint">
+              <b>Frozen picture that only updates when you alt-tab?</b> The game is in <i>exclusive</i> full-screen, which
+              window capture cannot see. Either set the game to <b>borderless / windowed full-screen</b> (best: keeps
+              window audio), or pick <b>the whole screen</b> above (always works; carries system audio, and shows
+              whatever else is on that monitor).
+            </div>
             <label className="hint" style={{display:'flex',gap:8,alignItems:'center'}}>
               <input type="checkbox" checked={tcBand} onChange={e => { setTcBand(e.target.checked); try { localStorage.setItem('vg.tcband', e.target.checked ? '1' : '0') } catch {} }} />
               <span>Timecode band: add a thin strip below the picture carrying the server clock, so editors get a
               frame-accurate timecode track. Viewers here never see it; Jellyfin shows a thin line at the bottom.</span>
             </label>
+            <label className="hint" style={{display:'flex',gap:8,alignItems:'center'}}>
+              <input type="checkbox" checked={preferTcp} onChange={e => rememberTcp(e.target.checked)} />
+              <span>Lossy uplink (Starlink, cellular, bad wifi): send over TCP from the start. Nothing gets lost on the
+              way to the recording; the cost is a second or two of extra delay for viewers. Ticks itself the first
+              time a stream has to switch.</span>
+            </label>
             <div className="warn-box">
               <b>⚠ This stream is recorded.</b> Everything the window shows and every sound it makes is saved to the
               server, visible to editors and members in Jellyfin, and kept until it is dumped or archived. Close
@@ -90,7 +198,8 @@ export default function GoLive({ onExit }) {
               should be thrown away (3-day undo).
             </div>
             {err && <div className="err">{err}</div>}
-            <button className="primary" disabled={!key || phase === 'capturing'} onClick={start}>{phase === 'capturing' ? 'Waiting for the window picker…' : 'Start'}</button>
+            {ended && <div className="warn-box"><b>Stream ended.</b> {ended} {/paused/.test(ended) ? 'Starting again before then will be refused.' : 'You can start again.'}</div>}
+            <button className="primary" disabled={!key || phase === 'capturing'} onClick={start}>{phase === 'capturing' ? 'Waiting for the picker…' : 'Start'}</button>
           </div>
         )}
         {phase === 'live' && (
@@ -99,8 +208,9 @@ export default function GoLive({ onExit }) {
             <span className="rec" title="this stream is being recorded to the server">● REC</span>
             <span>{mmss(elapsed)}</span>
             <span>{key.user}/{cleanTitle(title)}</span>
-            <span>{conn}</span>
+            <span>{conn}{transport === 'tcp' ? ' · via TCP' : ''}</span>
-            {stats && <span>{stats.w}×{stats.h} · {stats.fps} fps · {(stats.kbps / 1000).toFixed(1)} Mbps · {stats.codec}{stats.enc ? ` (${/openh264|libvpx|software|SimulcastEncoderAdapter/i.test(stats.enc) ? 'software' : 'hardware'}: ${stats.enc})` : ''}{stats.q && stats.q !== 'none' ? ` · limited by ${stats.q}` : ''}</span>}
+            {switched && <span className="warn" title="packet loss on your uplink was corrupting the recording; TCP retransmits everything, at the cost of some latency">{switched}</span>}
+            {stats && <span>{stats.w}×{stats.h} · {stats.fps} fps · {(stats.kbps / 1000).toFixed(1)} Mbps · {stats.codec}{stats.enc ? ` (${/openh264|libvpx|software|SimulcastEncoderAdapter/i.test(stats.enc) ? 'software' : 'hardware'}: ${stats.enc})` : ''}{stats.q && stats.q !== 'none' ? ` · limited by ${stats.q}` : ''}{stats.sent ? ` · loss ${(100 * stats.lost / stats.sent).toFixed(2)}% (${stats.nack} nack, ${stats.rtx} resent)` : ''}</span>}
             {!hasAudio && <span className="warn">no audio captured — restart and tick "Share audio"</span>}
           </div>
         )}

live/src/broadcast.js

@@ -10,18 +10,25 @@ export function defaultTitle(now = new Date()) {
   return `HelmetCam-${d}`
 }
 // Must match the dock and the server: the sweep publishes <Title>__<timestamp>.mp4, so "__" is reserved.
-export const cleanTitle = t => t.trim().replace(/\s+/g, '-').replace(/[^A-Za-z0-9_.-]/g, '').replace(/_{2,}/g, '_').replace(/^[.-]+/, '').slice(0, 64)
+// No dots: the server names the WebRTC audio mirror <title>.webrtc, so a dot in a title would let a title
+// collide with a mirror name and pick up the wrong MediaMTX rule (one such collision recorded 36 GB twice).
+export const cleanTitle = t => t.trim().replace(/\s+/g, '-').replace(/[^A-Za-z0-9_-]/g, '').replace(/_{2,}/g, '_').replace(/^[.-]+/, '').slice(0, 64)
 
-export async function captureWindow() {
+// surface: 'window' (default) or 'screen'. Window capture on Windows goes through the compositor, and a game in
-  // Window-only picker with that window's own audio. Chrome/Edge 141+ honour windowAudio: "window"
+// EXCLUSIVE full-screen bypasses the compositor: Chrome then gets a frame only when Windows composites one, i.e.
-  // (only the shared window's sound); older Chrome offers no audio for a window at all.
+// on alt-tab — Breezy's "one frame, then another when I alt-tab". Two ways out: the game in borderless/windowed
+// full-screen (compositor sees every frame), or capture the whole screen, which Chrome reads straight from the
+// GPU output (desktop duplication) and which exclusive full-screen cannot hide from. Screen capture carries
+// system audio instead of one window's audio.
+export async function captureWindow(surface = 'window') {
+  const screen = surface === 'screen'
   const constraints = {
-    video: { displaySurface: 'window', frameRate: { ideal: 60, max: 60 }, width: { max: 2560 }, height: { max: 1440 } },
+    video: { displaySurface: screen ? 'monitor' : 'window', frameRate: { ideal: 60, max: 60 }, width: { max: 2560 }, height: { max: 1440 } },
     audio: { echoCancellation: false, noiseSuppression: false, autoGainControl: false, sampleRate: 48000, channelCount: 2 },
     windowAudio: 'window',            // capture the picked window's audio (Chrome 141+)
-    systemAudio: 'exclude',
+    systemAudio: screen ? 'include' : 'exclude',
     selfBrowserSurface: 'exclude',
-    monitorTypeSurfaces: 'exclude',   // hide "Entire screen": this is a single-window helmet cam
+    monitorTypeSurfaces: screen ? 'include' : 'exclude',   // "Entire screen" only when asked for: a helmet cam is one window
     surfaceSwitching: 'include',
     preferCurrentTab: false,
   }
@@ -34,8 +41,18 @@ export async function captureWindow() {
   }
 }
 
-export const MIN_KBPS = 6000
+// A FLOOR, not a target. It was 6000, which told Chrome never to encode below 6 Mbps: on an uplink that
+// could not sustain that, Chrome kept sending anyway, the network discarded the excess, and MediaMTX logged
+// "invalid FU-A packet (non-starting)" while every viewer saw blocky corruption. A floor this low still
+// skips the painfully slow cold ramp without disabling congestion control.
+export const MIN_KBPS = 1200
 export const START_KBPS = 9000
+// On TCP nothing is lost, so the only thing Chrome's estimator sees is delay — and a Starlink dish swings its
+// delay on every handoff, which reads as congestion and parks the bitrate near the floor: stable but soft.
+// A higher floor is safe HERE because the failure mode a floor causes on UDP (sending into loss) cannot happen
+// on TCP; if the uplink truly cannot carry it, latency grows instead of frames corrupting, which is the trade
+// the TCP switch already made. Still well under a dish's typical 10-20 Mbps up.
+export const TCP_MIN_KBPS = 4000
 
 function preferCodecs(pc, kind = 'video') {
   const caps = RTCRtpSender.getCapabilities?.(kind)
@@ -54,11 +71,12 @@ function preferCodecs(pc, kind = 'video') {
   tx.setCodecPreferences([...high, ...base, ...vp9, ...rest])
 }
 
-// Chrome honours x-google-* fmtp hints: skip the slow bandwidth ramp and never sit below MIN_KBPS.
+// Chrome honours x-google-* fmtp hints. start-bitrate skips the slow cold ramp, max-bitrate is the ceiling,
-function mungeBitrates(sdp) {
+// and min-bitrate is a floor that must stay LOW: a high floor defeats congestion control entirely.
+function mungeBitrates(sdp, minKbps = MIN_KBPS) {
   return sdp.replace(/^a=fmtp:(\d+) (.*)$/gm, (line, pt, params) =>
     /packetization-mode|profile-id|VP9/i.test(params) || /H264|VP9/i.test(sdp.match(new RegExp(`a=rtpmap:${pt} .*`))?.[0] || '')
-      ? `a=fmtp:${pt} ${params};x-google-start-bitrate=${START_KBPS};x-google-min-bitrate=${MIN_KBPS};x-google-max-bitrate=${MAX_KBPS}`
+      ? `a=fmtp:${pt} ${params};x-google-start-bitrate=${START_KBPS};x-google-min-bitrate=${minKbps};x-google-max-bitrate=${MAX_KBPS}`
       : line)
 }
 
@@ -85,7 +103,23 @@ export function withTimecodeBand(stream) {
   return { stream: out, stop: () => { running = false; clearInterval(t); video.srcObject = null } }
 }
 
-export async function publish({ stream, path, user, pass, onState }) {
+// Transport. 'udp' is WebRTC as normal. 'tcp' keeps only MediaMTX's TCP candidates (webrtcLocalTCPAddress) in
+// the answer, so ICE can pair with nothing else and the media rides a TCP connection: no packet loss at all,
+// at the cost of latency spikes when the uplink hiccups. The recording cares about the former, not the latter.
+// MediaMTX requests retransmits (NACK, pion's ConfigureNack) either way, but a wifi burst that drops more
+// packets than its reorder window holds still reaches the recorder as a broken frame; TCP cannot.
+const LOSS_SWITCH_PCT = 0.3          // sustained loss above this (over LOSS_WINDOW_S) flips a publisher to TCP
+const LOSS_WINDOW_S = 20
+const LOSS_MIN_PACKETS = 1500        // don't judge a 2-second-old session
+
+function tcpOnly(sdp) {
+  const lines = sdp.split('\r\n')
+  const out = lines.filter(l => !(l.startsWith('a=candidate:') && / udp /i.test(l)))
+  if (!out.some(l => l.startsWith('a=candidate:'))) return null       // no TCP candidate offered: cannot do it
+  return out.join('\r\n')
+}
+
+export async function publish({ stream, path, user, pass, onState, transport = 'udp' }) {
   const pc = new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] })
   for (const track of stream.getTracks()) {
     if (track.kind === 'video') { try { track.contentHint = 'motion' } catch {} }   // game footage: favour frame rate + temporal quality
@@ -95,26 +129,36 @@ export async function publish({ stream, path, user, pass, onState }) {
   const vs = pc.getSenders().find(s => s.track?.kind === 'video')
   if (vs) {
     const p = vs.getParameters(); p.encodings = p.encodings?.length ? p.encodings : [{}]
-    p.encodings[0].maxBitrate = MAX_KBPS * 1000
+    p.encodings[0].maxBitrate = MAX_KBPS * 1000       // a ceiling; congestion control may go far below it
     p.encodings[0].maxFramerate = 60
-    p.encodings[0].scaleResolutionDownBy = 1
+    // Under pressure Chrome must shed FRAME RATE, never resolution. A resolution change is a new SPS, and
-    p.degradationPreference = 'balanced'
+    // MediaMTX opens a new recording file on every parameter change — on a lossy uplink that turned one
+    // stream into a row of files the sweep cannot copy-join (different sizes). Frame-rate drops cost
+    // nothing on the recording side. (No scaleResolutionDownBy pin: that would forbid shedding entirely.)
+    p.degradationPreference = 'maintain-resolution'
     await vs.setParameters(p).catch(() => {})
   }
   pc.onconnectionstatechange = () => onState?.(pc.connectionState)
   const offer = await pc.createOffer()
-  await pc.setLocalDescription({ type: 'offer', sdp: mungeBitrates(offer.sdp) })
+  await pc.setLocalDescription({ type: 'offer', sdp: mungeBitrates(offer.sdp, transport === 'tcp' ? TCP_MIN_KBPS : MIN_KBPS) })
   await new Promise(res => { if (pc.iceGatheringState === 'complete') return res(); const t = setTimeout(res, 2000); pc.onicegatheringstatechange = () => { if (pc.iceGatheringState === 'complete') { clearTimeout(t); res() } } })
   // MediaMTX takes WebRTC credentials only in an Authorization header ("Bearer user:pass"). We send them as
   // X-Stream-Auth and Caddy rewrites the header on the MediaMTX hop, so Authentik's gate never sees it.
   const auth = { 'X-Stream-Auth': `Bearer ${user}:${pass}` }
   const r = await fetch(`/webrtc/${path}/whip`, { method: 'POST', headers: { 'Content-Type': 'application/sdp', ...auth }, body: pc.localDescription.sdp, credentials: 'same-origin' })
   if (!r.ok) { pc.close(); throw new Error(r.status === 401 || r.status === 403 ? 'stream key rejected' : r.status === 400 && /busy|already/i.test(await r.text().catch(() => '')) ? 'that stream name is already live (OBS running?)' : `publish failed (${r.status})`) }
-  await pc.setRemoteDescription({ type: 'answer', sdp: await r.text() })
+  let answer = await r.text()
   let session = r.headers.get('Location')                       // MediaMTX answers relative to its own root
+  if (transport === 'tcp') {
+    const only = tcpOnly(answer)
+    if (!only) { transport = 'udp' } else answer = only          // server offered no TCP: stay on UDP, say so in stats
+  }
+  await pc.setRemoteDescription({ type: 'answer', sdp: answer })
   if (session && session.startsWith('/') && !session.startsWith('/webrtc/')) session = '/webrtc' + session
   return {
     pc,
+    transport,
+    path, user, pass,
     stats: async () => {
       let kbps = 0, fps = 0, w = 0, h = 0
       for (const s of await pc.getStats()) {
@@ -122,6 +166,11 @@ export async function publish({ stream, path, user, pass, onState }) {
       }
       return { kbps, fps, w, h }
     },
+    // release: end this WHIP session but keep capturing — used to re-publish the same tracks over TCP
+    release: async () => {
+      try { if (session) await fetch(session.startsWith('http') ? '/webrtc' + new URL(session).pathname : session, { method: 'DELETE', headers: auth, credentials: 'same-origin' }) } catch {}
+      pc.close()
+    },
     stop: async () => {
       try { if (session) await fetch(session.startsWith('http') ? '/webrtc' + new URL(session).pathname : session, { method: 'DELETE', headers: auth, credentials: 'same-origin' }) } catch {}
       pc.close()
@@ -130,6 +179,18 @@ export async function publish({ stream, path, user, pass, onState }) {
   }
 }
 
+// Loss over a sliding window from the server's receiver reports. samples: [{t, lost, sent}] newest last.
+export function windowLoss(samples) {
+  const now = Date.now()
+  const win = samples.filter(s => now - s.t <= LOSS_WINDOW_S * 1000)
+  if (win.length < 2) return null
+  const a = win[0], b = win[win.length - 1]
+  const sent = b.sent - a.sent, lost = b.lost - a.lost
+  if (sent < LOSS_MIN_PACKETS) return null
+  return { pct: 100 * Math.max(0, lost) / sent, sent, lost }
+}
+export const shouldSwitchToTcp = l => !!l && l.pct >= LOSS_SWITCH_PCT
+
 export async function fetchStreamKey() {
   // a fresh 12 h publish token for the logged-in Discord user (member of streamers); nothing is stored
   const r = await fetch('/me/stream-token', { credentials: 'same-origin' })

observability/kibana-setup.py

@@ -4,6 +4,9 @@
   connector  "Vanguard alert index"        — the .index connector, writing into the vanguard-alerts index
   rule       "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc
   data view  "Vanguard logs + metrics"     — filebeat-*,metricbeat-* so Discover and Lens have one source
+  landing    "Vanguard ops" dashboard      — our own links, and defaultRoute so diag.BASE opens on it
+
+Run one step on its own by naming it:  kibana-setup.py landing   (also: connector, rule, dataview, all)
 
 WHY AN INDEX AND NOT A WEBHOOK
 On a basic license the only connector types Kibana allows are .index and .server-log; every connector that
@@ -38,11 +41,25 @@ delivery script recognises the heartbeat and keeps it out of Discord. Use "On st
 for real alarms; only the heartbeat wants to fire every interval.
 """
 import json, os, time, urllib.error, urllib.request
+from pathlib import Path          # env_hosts() reads the stack's .env
+
+import sys
+
+STEPS = ("connector", "rule", "dataview", "landing")
+WANTED = sys.argv[1] if len(sys.argv) > 1 else "all"
+if WANTED not in STEPS + ("all",):
+    raise SystemExit(f"unknown step {WANTED!r}; pick one of: {', '.join(STEPS)}, all")
+def wanted(step):
+    return WANTED in ("all", step)
 
 KIBANA = "http://127.0.0.1:5601"
 ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts")
 HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
-           "Authorization": "ApiKey " + os.environ["KEY"]}
+           "Authorization": "ApiKey " + os.environ["KEY"],
+           # Kibana 9 ships with server.restrictInternalApis on. /api/kibana/settings (defaultRoute) is an
+           # internal API and refuses without this header: "exists but is not available with the current
+           # configuration". Sending it on every call is harmless; loosening the restriction is not.
+           "x-elastic-internal-origin": "Kibana"}
 
 def wait_for_kibana(seconds=300):
     """Kibana has to be *available*, not merely listening.
@@ -87,11 +104,15 @@ wait_for_kibana()
 
 # 1. the alert bus connector
 CONNECTOR_NAME = "Vanguard alert index"
-existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+connector_id = None
-if CONNECTOR_NAME in existing:
+if wanted("connector") or wanted("rule"):
+  existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+  if CONNECTOR_NAME in existing:
     connector_id = existing[CONNECTOR_NAME]
     print("connector already present:", connector_id)
-else:
+  elif not wanted("connector"):
+    raise SystemExit(f"connector {CONNECTOR_NAME!r} does not exist yet — run the connector step first")
+  else:
       connector_id = req("POST", "/api/actions/connector", {
         "name": CONNECTOR_NAME,
         "connector_type_id": ".index",
@@ -104,9 +125,11 @@ else:
 #    harmless and meant the data view was never created on any system where the rule already existed, which
 #    is every re-run.
 RULE_NAME = "Vanguard alerting heartbeat"
-existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
+existing_rule = None if not wanted("rule") else next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
                       if r["name"] == RULE_NAME), None)
-if existing_rule:
+if not wanted("rule"):
+    pass
+elif existing_rule:
     print("rule already present:", existing_rule["id"],
           "enabled" if existing_rule.get("enabled") else "DISABLED")
 else:
@@ -152,10 +175,118 @@ else:
 #    detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and
 #    metrics, which is the whole reason they share a cluster.
 VIEW_TITLE = "filebeat-*,metricbeat-*"
-views = req("GET", "/api/data_views").get("data_view", [])
+views = req("GET", "/api/data_views").get("data_view", []) if wanted("dataview") else []
-if any(v.get("title") == VIEW_TITLE for v in views):
+if not wanted("dataview"):
+    pass
+elif any(v.get("title") == VIEW_TITLE for v in views):
     print("data view already present:", VIEW_TITLE)
 else:
     view = req("POST", "/api/data_views/data_view", {
         "data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}})
     print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE)
+
+
+# 4. the landing page. Kibana stays at the root of the diag host; this just makes arriving there open OUR
+#    dashboard instead of Kibana's home. A markdown panel holds the links, and `defaultRoute` is the
+#    advanced setting that does the redirect. That is the whole trick, and it avoids moving Kibana under a
+#    /kibana prefix — which would have meant touching both Beats' setup.kibana, this script, the alarm
+#    script and the deadman's status check, with a silent crash-loop waiting for whichever one was missed.
+DASHBOARD_ID = "vanguard-ops"
+
+def env_hosts():
+    """Hostnames out of the stack's .env, so the links are real rather than examples."""
+    values = {}
+    for candidate in (Path(".env"), Path("/home/debian/vanguard-stream/.env")):
+        if candidate.is_file():
+            for line in candidate.read_text().splitlines():
+                if "=" in line and not line.startswith("#"):
+                    key, value = line.split("=", 1)
+                    values[key.strip()] = value.strip()
+            break
+    # .env carries no AUTH_HOST (it lives in the Caddyfile); every host here is <role>.<domain>, so derive it
+    if "AUTH_HOST" not in values and values.get("MAIN_HOST", "").count(".") >= 2:
+        values["AUTH_HOST"] = "auth." + values["MAIN_HOST"].split(".", 1)[1]
+    return values
+
+def landing_markdown(hosts):
+    def url(key, path="/"):
+        host = hosts.get(key)
+        return f"https://{host}{path}" if host else None
+    rows = [
+        ("Live streams", "/streaming#live", "who is publishing, loss on the way in, why files are splitting"),
+        ("Stuck recordings", "/streaming#stuck", "finished but never published — why, and retry or bin"),
+        ("Dump a night", "/streaming#dump", "a streamer's or everyone's night to trash; bin all unprocessed"),
+        ("Disk", "/disk", "space by streamer and clipper per month, trash, logs, system"),
+        ("Live portal", url("LIVE_HOST"), "watch every active stream"),
+        ("OBS dock", url("LIVE_HOST", "/dock/"), "go live, titles, dump tonight"),
+        ("Files", url("FILES_HOST"), "browse, download, upload finals"),
+        ("Jellyfin", url("WATCH_HOST"), "recordings by streamer and season"),
+        ("Gateway", url("MAIN_HOST"), "role-aware links and how-tos"),
+        ("Gitea", url("GIT_HOST"), "code and CI"),
+        ("Bot", url("BOT_HOST"), "the Discord bot's own pages"),
+        ("Syncthing", url("SYNC_HOST"), "clip intake, device pairing"),
+        ("Authentik", url("AUTH_HOST", "/if/admin/"), "users, groups, policies"),
+        ("Admin sign-in", url("AUTH_HOST", "/if/flow/default-authentication-flow/?admin=1"), "password card for akadmin — members are forwarded straight to Discord"),
+    ]
+    lines = ["### Vanguard", "", "| | |", "|---|---|"]
+    for name, link, why in rows:
+        if link:
+            lines.append(f"| **[{name}]({link})** | {why} |")
+    lines += [
+        "", "### In here", "",
+        "| | |", "|---|---|",
+        "| **[Discover](/app/discover)** | search every log line; filter on `service.name` or `vanguard.category` |",
+        "| **[Dashboards](/app/dashboards)** | the Metricbeat system and Docker dashboards |",
+        "| **[Stack Rules](/app/management/insightsAndAlerting/triggersActions/rules)** | the six alarms, and whether each is firing |",
+        "| **[Index Management](/app/management/data/index_management/indices)** | index sizes and ILM |",
+        "",
+        "_Alarms post to Discord through the `vanguard-alerts` index, not from Kibana directly: a basic "
+        "licence has no webhook connector. The host deadman watches that whole path from outside._",
+    ]
+    return "\n".join(lines)
+
+if wanted("landing"):
+    hosts = env_hosts()
+    panel = {
+        "type": "visualization",
+        "panelIndex": "1",
+        "gridData": {"x": 0, "y": 0, "w": 24, "h": 18, "i": "1"},
+        "embeddableConfig": {
+            "savedVis": {
+                "id": "", "title": "", "description": "", "type": "markdown",
+                "params": {"fontSize": 12, "openLinksInNewTab": False,
+                           "markdown": landing_markdown(hosts)},
+                "uiState": {},
+                "data": {"aggs": [], "searchSource": {"query": {"query": "", "language": "kuery"},
+                                                      "filter": []}},
+            },
+            "enhancements": {},
+        },
+    }
+    saved = req("POST", f"/api/saved_objects/dashboard/{DASHBOARD_ID}?overwrite=true", {
+        "attributes": {
+            "title": "Vanguard ops",
+            "description": "Landing page for the diagnostics host. Edit freely: add metric panels beside the links.",
+            "panelsJSON": json.dumps([panel]),
+            "optionsJSON": json.dumps({"hidePanelTitles": True, "useMargins": True,
+                                       "syncColors": False, "syncCursor": True, "syncTooltips": False}),
+            "timeRestore": False,
+            "kibanaSavedObjectMeta": {
+                "searchSourceJSON": json.dumps({"query": {"query": "", "language": "kuery"}, "filter": []})},
+        }})
+    # Prove it rather than assume it: read the dashboard back and count the panels Kibana actually stored.
+    stored = req("GET", f"/api/saved_objects/dashboard/{DASHBOARD_ID}")
+    panels = json.loads(stored.get("attributes", {}).get("panelsJSON", "[]"))
+    links = sum(1 for line in landing_markdown(hosts).splitlines() if line.startswith("| **["))
+    print(f"dashboard: {stored.get('id')} | title {stored.get('attributes', {}).get('title')!r} | "
+          f"{len(panels)} panel(s) stored | {links} links in the markdown")
+    if not panels:
+        raise SystemExit("  the dashboard saved but with no panels — Kibana rejected the panel shape")
+    print("open:", f"/app/dashboards#/view/{DASHBOARD_ID}")
+
+    route = f"/app/dashboards#/view/{DASHBOARD_ID}"
+    req("POST", "/api/kibana/settings", {"changes": {"defaultRoute": route}})
+    now = req("GET", "/api/kibana/settings").get("settings", {}).get("defaultRoute", {}).get("userValue")
+    print("defaultRoute:", now or "(not set!)")
+    if now != route:
+        raise SystemExit("  Kibana did not keep that defaultRoute — set it by hand in Advanced Settings")

streaming/disk.html

@@ -0,0 +1,132 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Disk — Vanguard</title>
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#3d8bfd;--ok:#2ecc71;--warn:#e3a008;--bad:#e05260;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
+  main{max-width:1100px;margin:0 auto;padding:24px 16px 48px}
+  header{display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;margin-bottom:10px}
+  h1{font-size:24px;margin:0}.who{color:var(--mut)}.who b{color:var(--fg)}
+  nav{display:flex;gap:6px;margin:6px 0 18px;border-bottom:1px solid var(--line)}
+  nav a{padding:8px 14px;color:var(--mut);text-decoration:none;border-bottom:2px solid transparent}
+  nav a.on{color:var(--fg);border-color:var(--acc)}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:14px 18px;margin-bottom:12px}
+  section h2{margin:0 0 8px;font-size:15px;color:var(--mut);font-weight:600;text-transform:uppercase;letter-spacing:.3px}
+  table{width:100%;border-collapse:collapse;font-size:14px}th,td{text-align:left;padding:6px 8px;border-top:1px solid var(--line);vertical-align:top}
+  th{color:var(--mut);font-weight:600;border-top:0}td.n{text-align:right;font-variant-numeric:tabular-nums;white-space:nowrap}
+  .mono{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .pill{display:inline-block;font-size:12px;border-radius:999px;padding:1px 9px;white-space:nowrap}
+  .ok{background:#1f3b2a;color:var(--ok)}.warn{background:#3a3220;color:var(--warn)}.bad{background:#43242a;color:var(--bad)}.info{background:#1e3147;color:var(--acc)}
+  .state{font-size:12px;border-radius:999px;padding:2px 10px;text-align:center;white-space:nowrap}
+  .parked{background:#43242a;color:var(--bad)}.published{background:#3a3220;color:var(--warn)}.waiting{background:#1e3147;color:var(--acc)}.writing{background:#1f3b2a;color:var(--ok)}
+  .row{display:grid;grid-template-columns:96px 1fr auto;gap:12px;align-items:start}
+  .name{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .meta{color:var(--mut);font-size:13px;margin-top:2px}
+  .why{color:var(--mut);font-size:13px;margin-top:6px;border-left:2px solid var(--line);padding-left:10px;white-space:pre-wrap}
+  .acts{display:flex;gap:8px;align-items:center}
+  button{font:inherit;font-size:13px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:6px 12px;cursor:pointer}
+  select.stopmins{font:inherit;font-size:12px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:5px 6px}
+  button:hover:not(:disabled){border-color:var(--acc)}button:disabled{opacity:.45;cursor:default}
+  button.danger:hover:not(:disabled){border-color:var(--bad);color:var(--bad)}
+  input[type=number]{font:inherit;background:#121212;color:var(--fg);border:1px solid #3a3a3a;border-radius:6px;padding:4px 6px}
+  .note{color:var(--mut);font-size:13px}.err{color:var(--bad)}.hidden{display:none}
+  pre{background:#121212;border-radius:8px;padding:10px;overflow:auto;font-size:12px;margin:8px 0 0;max-height:280px}
+  .verdict{margin-top:8px;font-size:14px}
+  footer{color:var(--mut);font-size:13px;margin-top:20px}
+</style>
+</head>
+<body>
+<main>
+  <header><h1>Disk</h1><div class="who" id="who"></div></header>
+  <nav><a href="/disk" class="on">Disk</a><a href="/" style="margin-left:auto">Dashboards ↗</a></nav>
+  <div id="pane-disk">
+    <section>
+      <div class="row" style="grid-template-columns:1fr auto;align-items:center">
+        <div id="fs" class="note">Loading…</div>
+        <div class="acts"><span class="note" id="diskgen"></span><button id="diskrefresh">Rewalk now</button></div>
+      </div>
+    </section>
+    <section><h2>Streamers — recordings by month</h2><div id="dstreamers" class="note">Loading…</div></section>
+    <section><h2>Clippers — clips by month</h2><div id="dclippers" class="note">Loading…</div></section>
+    <section><h2>Logs</h2><div id="dlogs" class="note">Loading…</div></section>
+    <section><h2>System</h2><div id="dsystem" class="note">Loading…</div></section>
+    <footer>Sizes are allocated blocks, what the disk actually loses. A recording's month is the date in its own name; clips and
+    everything else go by modification time. Cached for ten minutes; Rewalk now re-reads every file.</footer>
+  </div>
+</main>
+<script>
+const API = '/streaming/api';
+let admin = false;
+const esc = s => String(s ?? '').replace(/[&<>"]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
+const hms = s => { s = Math.max(0, s|0); const h = s/3600|0, m = (s%3600)/60|0; return h ? `${h}h ${m}m` : `${m}m`; };
+const dur = s => { s = Math.max(0, s|0); return `${s/3600|0}:${String((s%3600)/60|0).padStart(2,'0')}:${String(s%60).padStart(2,'0')}`; };
+const short = p => p.replace(/^.*\/streams\//, '').replace('/.live/', ' · ');
+const mb = b => (b/1e6).toFixed(b < 10e6 ? 1 : 0) + ' MB';
+
+async function getJSON(path) {
+  const r = await fetch(path, { credentials: 'same-origin' });
+  if (r.status === 401) throw new Error('Not signed in. Reload the page.');
+  const text = await r.text();
+  let data; try { data = JSON.parse(text); }
+  catch { throw new Error(`Server answered HTTP ${r.status} with no usable body${text ? ': ' + text.slice(0, 160) : ' (an empty 502 means dumpd did not reply)'}`); }
+  if (data.error) throw new Error(`HTTP ${r.status}: ${data.error}`);
+  return data;
+}
+function whoami(d) { admin = !!d.admin; document.getElementById('who').innerHTML = `signed in as <b>${esc(d.user || '?')}</b>${admin ? '' : ' · read only'}`; }
+
+const gb = b => b >= 1e12 ? (b / 1e12).toFixed(2) + ' TB' : b >= 1e9 ? (b / 1e9).toFixed(1) + ' GB' : b >= 1e6 ? (b / 1e6).toFixed(0) + ' MB' : b ? (b / 1e3).toFixed(0) + ' kB' : '';
+const monthName = m => { const [y, mo] = m.split('-'); return new Date(+y, +mo - 1, 1).toLocaleString(undefined, { month: 'short', year: '2-digit' }); };
+function monthTable(rows, months, extra) {
+  // rows: [{name, total, months:{}, note}] ; months: newest first, older ones folded into "older"
+  const shown = months.slice(0, 6), older = months.slice(6);
+  const cell = (r, m) => { const v = r.months[m] || 0; return `<td class="n">${v ? gb(v) : '<span class="note">·</span>'}</td>`; };
+  const oldSum = r => older.reduce((a, m) => a + (r.months[m] || 0), 0);
+  const tot = key => rows.reduce((a, r) => a + (key === 'total' ? r.total : key === 'older' ? oldSum(r) : (r.months[key] || 0)), 0);
+  return `<table>
+    <tr><th></th>${shown.map(m => `<th class="n">${monthName(m)}</th>`).join('')}${older.length ? '<th class="n">older</th>' : ''}<th class="n">total</th>${extra ? '<th></th>' : ''}</tr>
+    ${rows.map(r => `<tr><td><b>${esc(r.name)}</b></td>${shown.map(m => cell(r, m)).join('')}${older.length ? `<td class="n">${gb(oldSum(r)) || '<span class="note">·</span>'}</td>` : ''}<td class="n"><b>${gb(r.total)}</b></td>${extra ? `<td class="note">${r.note || ''}</td>` : ''}</tr>`).join('')}
+    ${rows.length > 1 ? `<tr><td class="note">all</td>${shown.map(m => `<td class="n">${gb(tot(m))}</td>`).join('')}${older.length ? `<td class="n">${gb(tot('older'))}</td>` : ''}<td class="n"><b>${gb(tot('total'))}</b></td>${extra ? '<td></td>' : ''}</tr>` : ''}
+  </table>`;
+}
+async function loadDisk(force) {
+  let d;
+  document.getElementById('diskgen').textContent = force ? 'walking every file…' : '';
+  try { d = await getJSON(API + '/disk' + (force ? '?refresh=1' : '')); }
+  catch (e) { document.getElementById('fs').innerHTML = `<span class="err">${esc(e.message)}</span>`; return; }
+  whoami(d);
+  document.getElementById('diskgen').textContent = `walked ${new Date(d.generated).toLocaleTimeString()} in ${d.seconds}s`;
+  const bar = (label, f) => { const pct = f.total ? 100 * f.used / f.total : 0; const cls = pct > 90 ? 'bad' : pct > 75 ? 'warn' : 'ok';
+    return `<div style="margin:4px 0"><b>${esc(label)}</b> <span class="mono note">${esc(f.path)}</span> — ${gb(f.used)} of ${gb(f.total)} used, <span class="pill ${cls}">${gb(f.free)} free</span>
+      <div style="height:8px;background:#262626;border-radius:4px;margin-top:4px"><div style="width:${pct.toFixed(1)}%;height:8px;border-radius:4px;background:var(--${cls === 'ok' ? 'ok' : cls === 'warn' ? 'warn' : 'bad'})"></div></div></div>`; };
+  const fs = d.fs || {};
+  document.getElementById('fs').innerHTML = (fs.media ? bar('media', fs.media) : '') + (fs.root && !fs.same_filesystem ? bar('system disk', fs.root) : '') + (fs.same_filesystem ? '<div class="note">media and the system share one filesystem</div>' : '');
+  const monthsOf = objs => [...new Set(objs.flatMap(o => Object.keys(o.months || {})))].sort().reverse();
+  const toRows = obj => Object.entries(obj).map(([name, v]) => ({ name, total: v.total, months: v.months || {},
+    note: [v.raw ? `${gb(v.raw)} raw in .src` : '', v.unpublished ? `${gb(v.unpublished)} unpublished in .live` : ''].filter(Boolean).join(' · ') })).sort((a, b) => b.total - a.total);
+  const sRows = toRows(d.streamers || {}); const trash = d.trash || { total: 0, months: {} };
+  if (trash.total) sRows.push({ name: 'trash (all users, 3-day undo)', total: trash.total, months: trash.months, note: 'dumped nights and binned recordings, purged after 3 days' });
+  document.getElementById('dstreamers').innerHTML = sRows.length ? monthTable(sRows, monthsOf(sRows), true) : '<span class="note">no recordings</span>';
+  const cRows = toRows(d.clippers || {});
+  document.getElementById('dclippers').innerHTML = cRows.length ? monthTable(cRows, monthsOf(cRows), false) : '<span class="note">no clips</span>';
+  const L = d.logs || {};
+  const lRows = [];
+  if (L.months) lRows.push({ name: 'Elasticsearch indices (logs + metrics)', total: L.es_total || 0, months: L.months, note: L.es_data_dir ? `${gb(L.es_data_dir)} on disk in appdata/elasticsearch` : '' });
+  document.getElementById('dlogs').innerHTML = (lRows.length ? monthTable(lRows, monthsOf(lRows), true) : '') + `<table style="margin-top:8px">
+    ${L.es_data_dir != null ? `<tr><td>Elasticsearch data dir</td><td class="n">${gb(L.es_data_dir)}</td><td class="note">the on-disk truth for the indices above</td></tr>` : ''}
+    <tr><td>journald</td><td class="n">${gb(L.journald || 0)}</td><td class="note">systemd journal (dumpd, sweep, timers)</td></tr>
+    <tr><td>Docker json logs</td><td class="n">${gb(L.docker_json_logs || 0)}</td><td class="note">per-container log files Filebeat reads</td></tr>
+  </table>` + (L.indices ? `<details style="margin-top:8px"><summary class="note">${L.indices.length} indices</summary><table>${L.indices.sort((a, b) => b.bytes - a.bytes).map(i => `<tr><td class="mono">${esc(i.index)}</td><td class="n">${gb(i.bytes)}</td><td class="note">${esc(i.month)}</td></tr>`).join('')}</table></details>` : '');
+  const S = d.system || {};
+  document.getElementById('dsystem').innerHTML = `<table>${Object.entries(S).sort((a, b) => (typeof b[1] === 'number' ? b[1] : 0) - (typeof a[1] === 'number' ? a[1] : 0))
+    .map(([k, v]) => `<tr><td>${esc(k)}</td><td class="n">${typeof v === 'number' ? gb(v) : esc(v)}</td><td class="note">${k.startsWith('docker') ? "Docker's own figure" : k === 'media_other' ? 'markers, Syncthing state, versions under /srv/media' : ''}</td></tr>`).join('')}</table>`
+    + (d.errors && d.errors.length ? `<div class="err" style="margin-top:8px">${d.errors.map(esc).join('<br>')}</div>` : '');
+}
+document.getElementById('diskrefresh').onclick = () => loadDisk(true);
+
+loadDisk(false);
+</script>
+</body>
+</html>

streaming/index.html

@@ -0,0 +1,416 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Streaming — Vanguard</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#3d8bfd;--ok:#2ecc71;--warn:#e3a008;--bad:#e05260;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
+  main{max-width:1100px;margin:0 auto;padding:24px 16px 48px}
+  header{display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;margin-bottom:10px}
+  h1{font-size:24px;margin:0}.who{color:var(--mut)}.who b{color:var(--fg)}
+  nav{display:flex;gap:6px;margin:6px 0 18px;border-bottom:1px solid var(--line)}
+  nav a{padding:8px 14px;color:var(--mut);text-decoration:none;border-bottom:2px solid transparent}
+  nav a.on{color:var(--fg);border-color:var(--acc)}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:14px 18px;margin-bottom:12px}
+  section h2{margin:0 0 8px;font-size:15px;color:var(--mut);font-weight:600;text-transform:uppercase;letter-spacing:.3px}
+  table{width:100%;border-collapse:collapse;font-size:14px}th,td{text-align:left;padding:6px 8px;border-top:1px solid var(--line);vertical-align:top}
+  th{color:var(--mut);font-weight:600;border-top:0}td.n{text-align:right;font-variant-numeric:tabular-nums;white-space:nowrap}
+  .mono{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .pill{display:inline-block;font-size:12px;border-radius:999px;padding:1px 9px;white-space:nowrap}
+  .ok{background:#1f3b2a;color:var(--ok)}.warn{background:#3a3220;color:var(--warn)}.bad{background:#43242a;color:var(--bad)}.info{background:#1e3147;color:var(--acc)}
+  .state{font-size:12px;border-radius:999px;padding:2px 10px;text-align:center;white-space:nowrap}
+  .parked{background:#43242a;color:var(--bad)}.published{background:#3a3220;color:var(--warn)}.waiting{background:#1e3147;color:var(--acc)}.writing{background:#1f3b2a;color:var(--ok)}
+  .row{display:grid;grid-template-columns:96px 1fr auto;gap:12px;align-items:start}
+  .name{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .meta{color:var(--mut);font-size:13px;margin-top:2px}
+  .why{color:var(--mut);font-size:13px;margin-top:6px;border-left:2px solid var(--line);padding-left:10px;white-space:pre-wrap}
+  .acts{display:flex;gap:8px;align-items:center}
+  button{font:inherit;font-size:13px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:6px 12px;cursor:pointer}
+  select.stopmins{font:inherit;font-size:12px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:5px 6px}
+  button:hover:not(:disabled){border-color:var(--acc)}button:disabled{opacity:.45;cursor:default}
+  button.danger:hover:not(:disabled){border-color:var(--bad);color:var(--bad)}
+  input[type=number]{font:inherit;background:#121212;color:var(--fg);border:1px solid #3a3a3a;border-radius:6px;padding:4px 6px}
+  .note{color:var(--mut);font-size:13px}.err{color:var(--bad)}.hidden{display:none}
+  pre{background:#121212;border-radius:8px;padding:10px;overflow:auto;font-size:12px;margin:8px 0 0;max-height:280px}
+  .verdict{margin-top:8px;font-size:14px}
+  footer{color:var(--mut);font-size:13px;margin-top:20px}
+</style>
+</head>
+<body>
+<main>
+  <header><h1>Streaming</h1><div class="who" id="who"></div></header>
+  <nav><a href="#live" id="tab-live">Live</a><a href="#stuck" id="tab-stuck">Stuck recordings</a><a href="#dump" id="tab-dump">Dump a night</a><a href="/disk" style="margin-left:auto">Disk ↗</a></nav>
+
+  <!-- ===================== LIVE ===================== -->
+  <div id="pane-live">
+    <section>
+      <h2>Publishing now</h2>
+      <div id="pubs" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Blocked from publishing</h2>
+      <div id="blocks" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Why MediaMTX opened recording files — last <span id="win">10m</span></h2>
+      <div id="counts" class="note">Loading…</div>
+      <div id="verdict" class="verdict"></div>
+    </section>
+    <section>
+      <h2>Live paths</h2>
+      <div id="paths" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Publish sessions — last 24 h</h2>
+      <div id="hist" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Recorder log</h2>
+      <pre id="log">Loading…</pre>
+    </section>
+    <footer>Refreshes every 10 seconds. Loss is counted on the <em>publish</em> session, i.e. on the way in: that is what corrupts recordings. Viewers losing packets only affects what they see.</footer>
+  </div>
+
+  <!-- ===================== STUCK ===================== -->
+  <div id="pane-stuck" class="hidden">
+    <p class="note">Recordings that finished but never reached Jellyfin or copyparty. Retry puts one back in the
+    queue; bin moves it to the trash, recoverable for three days just like a dumped night.</p>
+    <section id="bulk" class="hidden">
+      <div class="row" style="grid-template-columns:1fr auto">
+        <div>
+          <b>Reconnect fragments</b>
+          <div class="meta">Every time a publisher drops and comes back, MediaMTX opens a new recording. Stream
+          recordings this small are stubs, not recordings. Clips are not affected: they never pass through <code>.live/</code>.</div>
+        </div>
+        <div class="acts">
+          <label class="meta">under <input id="mb" type="number" min="1" max="50" value="5" style="width:4em"> MB</label>
+          <button id="bulkbtn" class="danger">Bin <span id="bulkcount">0</span> fragments</button>
+        </div>
+      </div>
+    </section>
+    <div id="list"><section class="note">Loading…</section></div>
+    <footer>Refreshes every 30 seconds. Trends and history live on the dashboards.</footer>
+  </div>
+
+  <!-- ===================== DUMP ===================== -->
+  <div id="pane-dump" class="hidden">
+    <p class="note">A bad night goes to the trash in one move: every stream recording and clip a streamer saved
+    between <b>17:00 and 02:00 Eastern</b> on the chosen night. Recoverable for three days, like the Stuck tab's
+    bin. Admins can dump anyone or everyone; everyone else sees and dumps only their own.</p>
+    <section>
+      <div class="row" style="grid-template-columns:auto 1fr auto;align-items:center">
+        <label class="meta">night <input id="night" type="date" style="font:inherit;background:#262626;color:var(--fg);border:1px solid #3a3a3a;border-radius:8px;padding:4px 8px"></label>
+        <div class="meta" id="dumpwin"></div>
+        <div class="acts"><button id="dumpall" class="danger hidden">Dump everyone</button></div>
+      </div>
+    </section>
+    <div id="dumplist"><section class="note">Loading…</section></div>
+    <section id="wipe" class="hidden">
+      <div class="row" style="grid-template-columns:1fr auto">
+        <div>
+          <b>Unprocessed and stuck recordings</b>
+          <div class="meta" id="wipemeta"></div>
+          <div class="meta">Everything still sitting in a <code>.live/</code> folder, whatever night it is from: stuck, parked, waiting, fresh, reconnect fragments, and the failure logs that go with them. Any conversion the sweep has in flight is cancelled first. The only thing left alone is a stream MediaMTX is recording this minute.</div>
+        </div>
+        <div class="acts"><button id="wipebtn" class="danger">Bin all unprocessed</button></div>
+      </div>
+      <pre id="wipeout" class="hidden"></pre>
+    </section>
+    <footer>Nothing here moves until you confirm twice. Streamers can also dump their own night from the OBS dock.</footer>
+  </div>
+
+</main>
+<script>
+const API = '/streaming/api';
+let admin = false;
+const esc = s => String(s ?? '').replace(/[&<>"]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
+const hms = s => { s = Math.max(0, s|0); const h = s/3600|0, m = (s%3600)/60|0; return h ? `${h}h ${m}m` : `${m}m`; };
+const dur = s => { s = Math.max(0, s|0); return `${s/3600|0}:${String((s%3600)/60|0).padStart(2,'0')}:${String(s%60).padStart(2,'0')}`; };
+const short = p => p.replace(/^.*\/streams\//, '').replace('/.live/', ' · ');
+const mb = b => (b/1e6).toFixed(b < 10e6 ? 1 : 0) + ' MB';
+
+async function getJSON(path) {
+  const r = await fetch(path, { credentials: 'same-origin' });
+  if (r.status === 401) throw new Error('Not signed in. Reload the page.');
+  const text = await r.text();
+  let data; try { data = JSON.parse(text); }
+  catch { throw new Error(`Server answered HTTP ${r.status} with no usable body${text ? ': ' + text.slice(0, 160) : ' (an empty 502 means dumpd did not reply)'}`); }
+  if (data.error) throw new Error(`HTTP ${r.status}: ${data.error}`);
+  return data;
+}
+function whoami(d) { admin = !!d.admin; document.getElementById('who').innerHTML = `signed in as <b>${esc(d.user || '?')}</b>${admin ? '' : ' · read only'}`; }
+
+// ---------------- Live ----------------
+async function loadLive() {
+  let d;
+  try { d = await getJSON(API + '/live'); }
+  catch (e) { document.getElementById('pubs').innerHTML = `<span class="err">${esc(e.message)}</span>`; return; }
+  whoami(d);
+
+  const pubs = d.publishers || [];
+  document.getElementById('pubs').innerHTML = pubs.length ? `<table>
+    <tr><th>path</th><th>via</th><th>from</th><th>since</th><th class="n">received</th><th class="n">lost</th><th>loss</th><th></th></tr>
+    ${pubs.map(p => {
+      const pct = p.received ? (100 * (p.lost || 0) / p.received) : 0;
+      const cls = p.kind !== 'webrtc' ? 'info' : pct === 0 ? 'ok' : pct < 0.05 ? 'warn' : 'bad';
+      const txt = p.kind !== 'webrtc' ? 'tcp' : `${pct.toFixed(3)}%`;
+      // webrtc: the selected ICE pair's protocol. tcp = the Go Live lossy-uplink switch (or the box ticked)
+      const via = p.kind === 'webrtc' ? `webrtc <span class="pill ${p.transport === 'tcp' ? 'info' : p.transport === 'udp' ? 'ok' : 'warn'}">${esc(p.transport || 'connecting')}</span>` : esc(p.kind);
+      return `<tr><td class="mono">${esc(p.path)}</td><td>${via}</td><td class="mono">${esc(p.from)}</td><td class="note">${p.since ? new Date(p.since).toLocaleTimeString() : ''}</td>
+        <td class="n">${p.received != null ? p.received.toLocaleString() : (p.bytes != null ? mb(p.bytes) : '')}</td>
+        <td class="n">${p.lost != null ? p.lost.toLocaleString() : ''}</td><td><span class="pill ${cls}">${txt}</span></td>
+        <td class="acts">${admin && p.session ? `<select class="stopmins" data-session="${esc(p.session)}" title="how long the account stays unable to publish after the kick">
+            <option value="0">kick only</option><option value="30" selected>block 30 min</option><option value="120">block 2 h</option><option value="480">block 8 h</option></select>
+          <button class="danger stopbtn" data-path="${esc(p.path)}" data-kind="${esc(p.kind)}" data-session="${esc(p.session)}">Stop stream</button>` : ''}</td></tr>`;
+    }).join('')}</table>` : '<span class="note">nobody is publishing</span>';
+  for (const b of document.querySelectorAll('#pubs .stopbtn')) b.onclick = () => stopStream(b);
+
+  // accounts currently blocked from publishing (set by Stop stream), with an unblock
+  const blocks = d.blocks || [];
+  document.getElementById('blocks').innerHTML = blocks.length ? `<table>
+    <tr><th>account</th><th>blocked</th><th>until</th><th>by</th><th>why</th><th></th></tr>
+    ${blocks.map(b => `<tr><td class="mono">${esc(b.user)}</td><td class="note">${b.since ? new Date(b.since).toLocaleTimeString() : ''}</td>
+      <td class="note">${new Date(b.until_iso).toLocaleTimeString()}</td><td>${esc(b.by || '')}</td><td class="note">${esc(b.reason || '')}</td>
+      <td class="acts">${admin ? `<button class="unblock" data-user="${esc(b.user)}">Allow again</button>` : ''}</td></tr>`).join('')}</table>`
+    : '<span class="note">nobody is blocked from publishing</span>';
+  for (const b of document.querySelectorAll('#blocks .unblock')) b.onclick = () => unblock(b.dataset.user);
+
+  const c = d.counts || {};
+  document.getElementById('win').textContent = c.window || '10m';
+  document.getElementById('counts').innerHTML = `<table>
+    <tr><td>new publish sessions</td><td class="n">${c.publish_sessions ?? '?'}</td><td class="note">a reconnect each</td></tr>
+    <tr><td>recorder (re)starts</td><td class="n">${c.recorder_restarts ?? '?'}</td><td class="note">a new .live file each</td></tr>
+    <tr><td>processing errors</td><td class="n">${c.processing_errors ?? '?'}</td><td class="note">lost packets corrupting frames</td></tr>
+    <tr><td>recorder access-unit errors</td><td class="n">${c.access_unit_errors ?? '?'}</td><td class="note">frames the recorder refused</td></tr>
+  </table>`;
+  let v = '';
+  if ((c.recorder_restarts|0) > 1) {
+    if ((c.processing_errors|0) > 0) v = '<span class="pill bad">loss</span> files are splitting because packets are being lost on the way in: check the publisher\'s uplink, and that MediaMTX is on the host network.';
+    else if ((c.publish_sessions|0) >= (c.recorder_restarts|0) - 1) v = '<span class="pill warn">reconnects</span> the publisher\'s session keeps ending: look at their browser console and the "closed" reasons below.';
+    else v = '<span class="pill warn">parameter changes</span> one stable session, no loss, yet files keep opening: the encoder is changing resolution (new SPS). Pin resolution in the publisher.';
+  } else if (pubs.length) v = '<span class="pill ok">steady</span> one file per stream, nothing lost.';
+  document.getElementById('verdict').innerHTML = v;
+
+  const paths = d.paths || [];
+  document.getElementById('paths').innerHTML = paths.length ? `<table>
+    <tr><th>path</th><th>tracks</th><th>encode</th><th class="n">viewers</th><th class="n">received</th><th>since</th></tr>
+    ${paths.map(p => {
+      const h = p.health;
+      // MediaMTX refuses WebRTC for H.264 with B-frames: that is the single most common "I can't watch it".
+      const enc = p.mirror ? '' : !h ? '<span class="note">probing…</span>'
+        : h.bframes > 0 ? `<span class="pill bad">B-frames (depth ${h.bframes}) · WebRTC refused</span> <span class="note">${esc(h.res || '')} ${h.fps ? h.fps + ' fps' : ''}</span>`
+        : `<span class="pill ok">no B-frames</span> <span class="note">${esc(h.res || '')} ${h.fps ? h.fps + ' fps' : ''}</span>`;
+      return `<tr><td class="mono">${esc(p.name)}${p.mirror ? ' <span class="pill info">audio mirror</span>' : ''}</td>
+      <td>${esc((p.tracks || []).join(', '))}</td><td>${enc}</td><td class="n">${p.readers}</td><td class="n">${mb(p.bytes || 0)}</td>
+      <td class="note">${p.since ? new Date(p.since).toLocaleTimeString() : ''}</td></tr>`;
+    }).join('')}</table>` : '<span class="note">nothing live</span>';
+
+  // Every WebRTC publish session MediaMTX has had in 24 h, with the transport it ran on. A session that starts
+  // on a different transport than the previous one of the same path is a switch: that is "when he swapped".
+  const hist = d.history || [];
+  const span = (a, b) => { if (!a) return ''; const s = Math.max(0, ((b ? new Date(b) : new Date()) - new Date(a)) / 1000 | 0); return hms(s) || '<1m'; };
+  document.getElementById('hist').innerHTML = hist.length ? `<table>
+    <tr><th>path</th><th>transport</th><th>started</th><th>ended</th><th>ran</th><th class="n">received</th><th class="n">lost</th><th>loss</th><th></th></tr>
+    ${hist.map(h => {
+      const pct = h.received ? (100 * (h.lost || 0) / h.received) : 0;
+      const cls = pct === 0 ? 'ok' : pct < 0.05 ? 'warn' : 'bad';
+      const sw = h.switched_from ? `<span class="pill info">switched from ${esc(h.switched_from)}</span>` : '';
+      return `<tr><td class="mono">${esc(h.path)}</td><td><span class="pill ${h.transport === 'tcp' ? 'info' : h.transport ? 'ok' : 'warn'}">${esc(h.transport || '?')}</span></td>
+        <td class="note">${h.created ? new Date(h.created).toLocaleTimeString() : ''}</td><td class="note">${h.ended ? new Date(h.ended).toLocaleTimeString() : '<b>live</b>'}</td>
+        <td class="note">${span(h.created, h.ended)}</td><td class="n">${(h.received || 0).toLocaleString()}</td><td class="n">${(h.lost || 0).toLocaleString()}</td>
+        <td><span class="pill ${cls}">${pct.toFixed(3)}%</span></td><td>${sw}</td></tr>`;
+    }).join('')}</table>` : '<span class="note">no WebRTC publish sessions in the last 24 h</span>';
+
+  document.getElementById('log').textContent = (d.log || []).join('\n') || '(quiet)';
+  if (d.errors && d.errors.length) document.getElementById('log').textContent += '\n\n' + d.errors.join('\n');
+}
+
+// Stop stream: MediaMTX kicks the publish session, and dumpd blocks the account's publishing for the chosen
+// time so OBS's auto-reconnect or a Go Live restart cannot bring it straight back. Kick only = a hiccup.
+async function stopStream(btn) {
+  const sel = document.querySelector(`#pubs .stopmins[data-session="${btn.dataset.session}"]`);
+  const minutes = parseInt(sel?.value || '30', 10);
+  const what = minutes ? `and block ${esc(btn.dataset.path.split('/')[0])} from publishing for ${minutes >= 60 ? (minutes / 60) + ' h' : minutes + ' min'}` : 'only (they can reconnect at once)';
+  if (!confirm(`Stop ${btn.dataset.path} now, ${what}?\n\nThe recording so far is kept.`)) return;
+  btn.disabled = true; btn.textContent = 'Stopping…';
+  try {
+    const r = await fetch(API + '/live', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ action: 'stop', path: btn.dataset.path, kind: btn.dataset.kind, session: btn.dataset.session, minutes }) });
+    const body = await r.json();
+    btn.textContent = body.error ? `Failed: ${body.error}` : (body.kicked ? 'Stopped' : 'Not stopped');
+  } catch (e) { btn.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadLive, 1500);
+}
+async function unblock(user) {
+  if (!confirm(`Let ${user} publish again now?`)) return;
+  try {
+    await fetch(API + '/live', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action: 'unblock', user }) });
+  } catch {}
+  loadLive();
+}
+
+// ---------------- Stuck ----------------
+async function loadStuck() {
+  const list = document.getElementById('list');
+  let data;
+  try { data = await getJSON(API + '/stuck'); }
+  catch (e) { list.innerHTML = `<section class="note err">${esc(e.message)}</section>`; return; }
+  whoami(data);
+  const items = (data.items || []).sort((a, b) => (a.state === b.state ? b.age_s - a.age_s : a.state.localeCompare(b.state)));
+  const bulk = document.getElementById('bulk');
+  bulk.classList.toggle('hidden', !admin || !items.length);
+  window.__items = items; updateBulkCount();
+  if (!items.length) { list.innerHTML = '<section class="note">Nothing is stuck. Everything recorded has published.</section>'; return; }
+  list.innerHTML = '';
+  for (const it of items) {
+    const el = document.createElement('section');
+    const actionable = admin && it.state !== 'writing';
+    el.innerHTML = `
+      <div class="row">
+        <div class="state ${esc(it.state)}">${esc(it.state)}</div>
+        <div>
+          <div class="name">${esc(short(it.src))}</div>
+          <div class="meta">waiting ${hms(it.age_s)}${it.duration_s == null ? '' : ' · ' + dur(it.duration_s) + ' long'} · ${mb(it.bytes)}</div>
+          <div class="why">${esc(it.reason || '')}</div>
+        </div>
+        <div class="acts">
+          <button data-act="retry" ${actionable ? '' : 'disabled'}>Retry</button>
+          <button data-act="drop" class="danger" ${actionable ? '' : 'disabled'}>Bin</button>
+        </div>
+      </div>`;
+    for (const b of el.querySelectorAll('button')) b.onclick = () => act(it, b.dataset.act, el);
+    list.appendChild(el);
+  }
+}
+async function act(item, action, el) {
+  if (action === 'drop' && !confirm(`Move this recording to the trash?\n\n${short(item.src)}\n\nRecoverable for three days.`)) return;
+  for (const b of el.querySelectorAll('button')) b.disabled = true;
+  const out = document.createElement('pre'); out.textContent = action === 'retry' ? 'Clearing the blocker…' : 'Moving to trash…'; el.appendChild(out);
+  try {
+    const r = await fetch(API + '/stuck', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action, src: item.src }) });
+    const body = await r.json();
+    out.textContent = body.error ? `Failed: ${body.error}` : (body.output || []).join('\n') || 'Done.';
+    if (!body.error) setTimeout(loadStuck, 1200);
+  } catch (e) { out.textContent = `Failed: ${e.message}`; }
+}
+function updateBulkCount() {
+  const mbv = Math.max(1, Math.min(50, parseInt(document.getElementById('mb').value || '5', 10)));
+  const n = (window.__items || []).filter(i => i.state !== 'writing' && i.bytes < mbv * 1e6).length;
+  document.getElementById('bulkcount').textContent = n;
+  document.getElementById('bulkbtn').disabled = n === 0;
+}
+document.getElementById('mb').addEventListener('input', updateBulkCount);
+document.getElementById('bulkbtn').onclick = async () => {
+  const mbv = parseInt(document.getElementById('mb').value || '5', 10);
+  const n = document.getElementById('bulkcount').textContent;
+  if (!confirm(`Move ${n} stream recordings smaller than ${mbv} MB to the trash?\n\nRecoverable for three days. Clips are not affected.`)) return;
+  const btn = document.getElementById('bulkbtn'); btn.disabled = true; btn.textContent = 'Binning…';
+  try {
+    const r = await fetch(API + '/stuck', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action: 'drop_under', mb: mbv }) });
+    const body = await r.json();
+    btn.textContent = body.error ? `Failed: ${body.error}` : 'Done';
+  } catch (e) { btn.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadStuck, 1500);
+};
+
+// ---------------- Dump ----------------
+// Same endpoint as the dock's "Dump tonight" button (dumpd), reached here as /streaming/api/dump so the
+// Authentik cookie for this host is enough. GET is always a dry run; only the POST moves anything.
+const nightEl = document.getElementById('night');
+let dumpNightTouched = false;
+nightEl.addEventListener('change', () => { dumpNightTouched = true; loadDump(); });
+async function loadDump() {
+  const list = document.getElementById('dumplist');
+  let d;
+  try {
+    const q = new URLSearchParams({ user: 'all' });                       // non-admins get their own back regardless
+    if (dumpNightTouched && nightEl.value) q.set('night', nightEl.value);
+    d = await getJSON(API + '/dump?' + q);
+  } catch (e) { list.innerHTML = `<section class="note err">${esc(e.message)}</section>`; return; }
+  whoami(d);
+  if (!dumpNightTouched) nightEl.value = d.night;
+  document.getElementById('dumpwin').textContent = d.window ? `${d.window[0].replace('T', ' ').slice(0, 16)} → ${d.window[1].replace('T', ' ').slice(0, 16)}` : '';
+  const byUser = {};
+  for (const e of d.entries || []) {
+    const m = e.path.match(/^(streams|clips)\/([^/]+)\//);
+    const u = m ? m[2] : '(other)'; const kind = m ? m[1] : 'other';
+    const row = byUser[u] ||= { user: u, streams: 0, clips: 0, bytes: 0 };
+    row[kind === 'streams' ? 'streams' : 'clips']++; row.bytes += e.bytes;
+  }
+  const rows = Object.values(byUser).sort((a, b) => b.bytes - a.bytes);
+  document.getElementById('dumpall').classList.toggle('hidden', !(admin && rows.length > 1));
+  if (!rows.length) { list.innerHTML = `<section class="note">Nothing recorded on the night of ${esc(d.night)}${admin ? '' : ' under your name'}.</section>`; return; }
+  list.innerHTML = `<section><table>
+    <tr><th>streamer</th><th class="n">stream recordings</th><th class="n">clips</th><th class="n">size</th><th></th></tr>
+    ${rows.map(r => `<tr data-user="${esc(r.user)}"><td><b>${esc(r.user)}</b></td><td class="n">${r.streams}</td><td class="n">${r.clips}</td><td class="n">${(r.bytes / 1e9).toFixed(2)} GB</td>
+      <td class="acts"><button class="danger" data-user="${esc(r.user)}">Dump ${esc(r.user)}'s night</button></td></tr>`).join('')}
+    <tr><td>total</td><td class="n">${rows.reduce((a, r) => a + r.streams, 0)}</td><td class="n">${rows.reduce((a, r) => a + r.clips, 0)}</td><td class="n"><b>${(d.gb || 0).toFixed(2)} GB</b></td><td></td></tr>
+  </table><pre id="dumpout" class="hidden"></pre></section>`;
+  for (const b of list.querySelectorAll('button[data-user]')) b.onclick = () => dumpNight(b.dataset.user, b);
+}
+async function dumpNight(user, btn) {
+  const night = nightEl.value;
+  const who = user === 'all' ? 'EVERYONE' : user;
+  if (!confirm(`Move ${who}'s recordings and clips from the night of ${night} to the trash?\n\nRecoverable for three days.`)) return;
+  if (btn.dataset.armed !== night) {                                   // second click, same night: the dock's two-step
+    btn.dataset.armed = night; const was = btn.textContent; btn.textContent = 'Click again to confirm';
+    setTimeout(() => { if (btn.isConnected) { btn.textContent = was; delete btn.dataset.armed; } }, 10000);
+    return;
+  }
+  btn.disabled = true; btn.textContent = 'Dumping…';
+  const out = document.getElementById('dumpout'); out.classList.remove('hidden'); out.textContent = `Moving ${who}'s night of ${night}…`;
+  try {
+    const r = await fetch(API + '/dump', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ user, night }) });
+    const body = await r.json();
+    out.textContent = body.error ? `Failed: ${body.error}` : `Dumped ${body.moved} file(s), ${body.gb} GB from ${body.user}'s night of ${body.night}. Recoverable until ${body.undo_until}.`;
+  } catch (e) { out.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadDump, 1200);
+}
+document.getElementById('dumpall').onclick = () => dumpNight('all', document.getElementById('dumpall'));
+
+// "Bin all unprocessed": the Stuck tab's survey gives the count and size; vgstuck.sh wipe does the moving.
+async function loadWipe() {
+  const box = document.getElementById('wipe');
+  let d;
+  try { d = await getJSON(API + '/stuck'); } catch (e) { box.classList.add('hidden'); return; }
+  const items = d.items || [];                         // "writing" is only a guess from mtime; the tool checks what is really open
+  box.classList.toggle('hidden', !admin || !items.length);
+  const gb = items.reduce((a, i) => a + i.bytes, 0) / 1e9;
+  document.getElementById('wipemeta').textContent = `${items.length} recording(s), ${gb.toFixed(2)} GB in .live/`;
+  const b = document.getElementById('wipebtn'); b.disabled = false; b.textContent = `Bin all ${items.length} unprocessed`; delete b.dataset.armed;
+}
+document.getElementById('wipebtn').onclick = async () => {
+  const btn = document.getElementById('wipebtn');
+  const meta = document.getElementById('wipemeta').textContent;
+  if (!confirm(`Move every unprocessed recording to the trash and cancel any conversion in progress?\n\n${meta}\n\nRecoverable for three days. Only a stream being recorded this minute is left alone.`)) return;
+  if (!btn.dataset.armed) { btn.dataset.armed = '1'; const was = btn.textContent; btn.textContent = 'Click again to confirm'; setTimeout(() => { if (btn.dataset.armed) { btn.textContent = was; delete btn.dataset.armed; } }, 10000); return; }
+  delete btn.dataset.armed; btn.disabled = true; btn.textContent = 'Binning…';
+  const out = document.getElementById('wipeout'); out.classList.remove('hidden'); out.textContent = 'Moving to trash…';
+  try {
+    const r = await fetch(API + '/stuck', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action: 'wipe' }) });
+    const body = await r.json();
+    out.textContent = body.error ? `Failed: ${body.error}` : (body.output || []).join('\n') || 'Done.';
+  } catch (e) { out.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadWipe, 1500);
+};
+
+// ---------------- tabs ----------------
+let timer = null;
+function show(tab) {
+  if (!['live', 'stuck', 'dump'].includes(tab)) tab = 'live';
+  for (const t of ['live', 'stuck', 'dump']) {
+    document.getElementById('pane-' + t).classList.toggle('hidden', t !== tab);
+    document.getElementById('tab-' + t).classList.toggle('on', t === tab);
+  }
+  clearInterval(timer);
+  if (tab === 'live') { loadLive(); timer = setInterval(loadLive, 10000); }
+  else if (tab === 'stuck') { loadStuck(); timer = setInterval(loadStuck, 30000); }
+  else { dumpNightTouched = false; loadDump(); loadWipe(); }
+}
+window.addEventListener('hashchange', () => show(location.hash.replace('#', '')));
+show(location.hash.replace('#', '') || 'live');
+</script>
+</body>
+</html>

switchToClone.sh

@@ -3,13 +3,13 @@
 #   ./switchToClone.sh key     -> creates a deploy key and prints the public half (add it in Gitea: repo → Settings → Deploy keys, read-only)
 #   ./switchToClone.sh clone   -> clones over ssh (port 2222), moves .env + generated files + plugin zips into it
 set -euo pipefail
-REPO_SSH="${REPO_SSH:-ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git}"
+REPO_SSH="${REPO_SSH:-ssh://git@git.vng.one:2222/techy/Media-Server-Setup.git}"
 DIR="$HOME/vanguard-stream"; KEY="$HOME/.ssh/vanguard-deploy"
 case "${1:-}" in
   key)
     [[ -f $KEY ]] || ssh-keygen -t ed25519 -N "" -C "vanguard-deploy@$(hostname)" -f "$KEY" >/dev/null
     cat > "$HOME/.ssh/config.vanguard" <<CFG
-Host git.dumpstervigilantes.rip
+Host git.vng.one
   Port 2222
   User git
   IdentityFile $KEY
@@ -19,7 +19,7 @@ CFG
     echo "Add this as a READ-ONLY deploy key on the repo (Gitea → repo → Settings → Deploy Keys):"; echo; cat "$KEY.pub" ;;
   clone)
     [[ -d $DIR/.git ]] && { echo "$DIR is already a clone"; exit 0; }
-    ssh -o StrictHostKeyChecking=accept-new -p 2222 git@git.dumpstervigilantes.rip 2>&1 | grep -qi "successfully authenticated\|Hi there" || { echo "deploy key not accepted yet — run './switchToClone.sh key' and add it in Gitea"; exit 1; }
+    ssh -o StrictHostKeyChecking=accept-new -p 2222 git@git.vng.one 2>&1 | grep -qi "successfully authenticated\|Hi there" || { echo "deploy key not accepted yet — run './switchToClone.sh key' and add it in Gitea"; exit 1; }
     mv "$DIR" "$DIR.old"
     git clone --quiet "$REPO_SSH" "$DIR"
     for f in .env Caddyfile mediamtx.yml copyparty.conf stream-token.secret deploy.env; do [[ -f $DIR.old/$f ]] && cp -a "$DIR.old/$f" "$DIR/$f"; done

systemd/normalize-sweep.service

@@ -9,4 +9,5 @@ ExecStart=/usr/local/bin/normalize-sweep.sh
 Nice=15
 IOSchedulingClass=idle
 Environment=ENCODE_THREADS=2   # x264 threads per conversion; keeps Jellyfin and the live portal responsive (default 2)
+# Environment=STITCH_MAX_PARTS=500  # join a stream's fragments when fewer than this (0 = never); STITCH_MAX_GAP_S=600 splits sessions
 # Environment=FILL_GAPS_MS=1000   # opt-in: CFR re-encode streams whose dropped-frame holes total >= 1 s (hours of CPU on this box)

systemd/vanguard-alerts.service

@@ -4,6 +4,9 @@ After=docker.service
 
 [Service]
 Type=oneshot
+# Written by configure.py from .env (ALERT_RETENTION_DAYS, ALERT_SUPPRESS_S, ALERT_INDEX). A timer never reads the repo's .env, so without
+# this the tunables were settable in name only. The leading - means "carry on if it is absent".
+EnvironmentFile=-/etc/vanguard/alerts.env
 ExecStart=/usr/local/bin/vanguard-alerts.sh
 # Kibana cannot post outside the stack on a basic license, so its rules write into the vanguard-alerts
 # index and this is the delivery side. See bin/vanguard-alerts.sh and LOGGING.md.

systemd/vanguard-deadman.service

@@ -4,4 +4,7 @@ After=docker.service
 
 [Service]
 Type=oneshot
+# Written by configure.py from .env (HEARTBEAT_MAX_S, ALERT_INDEX). A timer never reads the repo's .env, so without
+# this the tunables were settable in name only. The leading - means "carry on if it is absent".
+EnvironmentFile=-/etc/vanguard/alerts.env
 ExecStart=/usr/local/bin/vanguard-deadman.sh