bunch of shit

Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

.gitea/draft-workflows/ci.yml

@@ -0,0 +1,18 @@
+# DRAFT — inert while it lives in .gitea/draft-workflows/. Activate with:  git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml
+# Checks every push; builds + tests the Jellyfin plugin. Deploying is NOT done here (the runner must not touch
+# the host): the Gitea push webhook -> dumpd /deploy -> vanguard-deploy handles that, when DEPLOY_AUTO=1.
+name: ci
+on: [push, pull_request]
+jobs:
+  checks:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: sudo apt-get update -q && sudo apt-get install -y -q python3 python3-yaml nodejs npm ffmpeg jq
+      - run: bash check.sh
+  plugin:
+    runs-on: dotnet
+    steps:
+      - uses: actions/checkout@v4
+      - run: cd jellyfin-plugin-vanguard && dotnet test Jellyfin.Plugin.Vanguard.Tests --nologo -v q
+      - run: cd jellyfin-plugin-vanguard && dotnet build Jellyfin.Plugin.Vanguard -c Release --nologo -v q

.gitignore

@@ -9,3 +9,5 @@ mediamtx.yml
 copyparty.conf
 Caddyfile
 stream-token.secret
+deploy.env
+bot/app/

Caddyfile.example

@@ -1,29 +1,364 @@
+{
+	servers {
+		metrics                           # Prometheus metrics at http://127.0.0.1:2019/metrics (LOGGING.md)
+	}
+}
+
+# vanguard:main
 media.example.com {
 	encode zstd gzip
-
+	log {
-	# dump endpoint (python service on the host, does its own auth)
+		output stderr
-	handle /dump* {
+		format filter {
-		reverse_proxy 127.0.0.1:8080
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
 			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	# who does Authentik say I am? (debug: shows exactly the identity headers the apps receive)
+	handle /whoami {
+		header Content-Type text/plain
+		respond "user:   {http.request.header.X-Authentik-Username}
+groups: {http.request.header.X-Authentik-Groups}
+email:  {http.request.header.X-Authentik-Email}
+name:   {http.request.header.X-Authentik-Name}"
+	}
+	@me path /me /me/*
+	handle @me {
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity for the gateway page
+	}
+	redir /about* / 308                       # old bookmarks: the gateway page used to live at /about/
+	handle {
+		root * /srv/about
+		try_files {path} /index.html
+		file_server                           # the gateway page is the root of the main host
+	}
+}
 
-	# Syncthing admin UI (its own login; strip the prefix, present a local Host to pass its host check)
+# vanguard:apex
-	redir /sync /sync/ 308
+example.com {
-	handle_path /sync/* {
+	redir https://media.example.com/ 302
+}
+
+# vanguard:files
+files.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	# copyparty at the root. It honours X-Authentik-Username / -Groups only when the request also carries a header
+	# whose secret NAME only this proxy knows (copyparty idp-h-key). Clients cannot forge identity.
+	reverse_proxy 127.0.0.1:3923 {
+		header_up X-Vanguard-EXAMPLEKEY 1
+		flush_interval -1
+	}
+}
+
+# vanguard:sync
+sync.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@lacks_group not header_regexp X-Authentik-Groups (^|\|)media\-admins(\||$)
+	handle @lacks_group {
+		respond "This page is for members of the 'media-admins' group." 403
+	}
+	# Syncthing admin UI (its own GUI login is disabled; the gate + group is the login)
 	reverse_proxy 127.0.0.1:8384 {
 		header_up Host {upstream_hostport}
 	}
-	}
+}
 
-	# OBS browser dock (static)
+# vanguard:clips
-	handle_path /dock* {
+clips.example.com {
-		root * /srv/dock
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@me path /me /me/*
+	handle @me {
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity + /me/clips self-service
+	}
+	handle {
+		root * /srv/clips
+		try_files {path} /index.html
 		file_server
 	}
+}
 
-	# everything else is copyparty (web UI + WebDAV)
+# vanguard:diag
-	handle {
+diag.example.com {
-		reverse_proxy 127.0.0.1:3923 {
+	encode zstd gzip
-			flush_interval -1
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
 			}
 		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@lacks_group not header_regexp X-Authentik-Groups (^|\|)media\-admins(\||$)
+	handle @lacks_group {
+		respond "This page is for members of the 'media-admins' group." 403
+	}
+	# VictoriaLogs runs with -http.pathPrefix=/logs and emits /logs/... URLs itself, so the prefix is
+	# passed through unchanged. Stripping it would break the UI's own asset paths.
+	redir /logs /logs/select/vmui/ 302
+	redir /logs/ /logs/select/vmui/ 302
+	handle /logs* {
+		reverse_proxy 127.0.0.1:9428
+	}
+	handle /metrics-db* {
+		reverse_proxy 127.0.0.1:8428          # VictoriaMetrics UI + query API
+	}
+	handle /alerts* {
+		reverse_proxy 127.0.0.1:9093          # Alertmanager: what is firing, silences
+	}
+	handle /rules-metrics* {
+		reverse_proxy 127.0.0.1:8880          # vmalert over VictoriaMetrics
+	}
+	handle /rules-logs* {
+		reverse_proxy 127.0.0.1:8881          # vmalert over VictoriaLogs
+	}
+	handle {
+		root * /srv/diag
+		try_files {path} /index.html
+		file_server                           # diag/index.html: what each path above is
+	}
+}
+
+# vanguard:live
+live.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	handle /whoami {
+		header Content-Type text/plain
+		respond "user:   {http.request.header.X-Authentik-Username}
+groups: {http.request.header.X-Authentik-Groups}"
+	}
+	@me path /me /me/*
+	handle @me {
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity, stream tokens (Go Live + dock)
+	}
+	handle /time {
+		reverse_proxy 127.0.0.1:8080          # server clock for the timecode strip (clock page + Go Live)
+	}
+	handle /dump* {
+		reverse_proxy 127.0.0.1:8080          # dump tonight (dock)
+	}
+	handle_path /dock* {
+		root * /srv/dock
+		file_server                           # OBS browser dock + /dock/clock/
+	}
+	handle_path /api/* {
+		reverse_proxy 127.0.0.1:9997
+	}
+	handle_path /webrtc/* {
+		# Go Live sends MediaMTX credentials in X-Stream-Auth so Authentik's forward-auth never sees an
+		# Authorization header; Caddy turns it into Authorization only on the hop to MediaMTX.
+		@creds header X-Stream-Auth *
+		reverse_proxy @creds 127.0.0.1:8889 {
+			header_up Authorization {http.request.header.X-Stream-Auth}
+			header_up -X-Stream-Auth
+		}
+		reverse_proxy 127.0.0.1:8889
+	}
+	handle_path /hls/* {
+		reverse_proxy 127.0.0.1:8888
+	}
+	handle {
+		root * /srv/live
+		try_files {path} /index.html
+		file_server
+	}
+}
+
+# vanguard:bot
+bot.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@lacks_group not header_regexp X-Authentik-Groups (^|\|)bot\-users(\||$)
+	handle @lacks_group {
+		respond "This page is for members of the 'bot-users' group." 403
+	}
+	reverse_proxy 127.0.0.1:8000
+}
+
+# vanguard:git
+git.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	reverse_proxy 127.0.0.1:3000
+}
+
+# vanguard:auth
+auth.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	reverse_proxy 127.0.0.1:9000
+}
+
+# vanguard:watch
+watch.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	reverse_proxy 127.0.0.1:8096
 }

CommunityDiscordMedia.md

@@ -21,7 +21,7 @@ Bundle: `vanguard-stream/` in this repo; live copy at `~/vanguard-stream` on the
 | Authentik (Discord login, groups) at `auth.` | live |
 | Jellyfin at `watch.`, SSO via Authentik, direct play only | live |
 | Vanguard Metadata plugin v0.1 | installed, libraries being switched to Shows |
-| MariaDB (bot database) + `discord-bot` Python container (`bot/app`) | **in bundle, not yet applied** |
+| MariaDB (bot database) + `discord-bot` Python container (release installed in `/srv/bot/app`) | live; first release pending |
 | Gitea (`git.`, MariaDB, ssh 2222) + Actions runner | live (local admin account; Authentik OIDC → Phase 10) |
 
 ## Decisions already made
@@ -180,7 +180,7 @@ Checklist:
 - [ ] Medal local filename pattern → add to `ItemTimestamp` (need a sample filename)
 - [ ] later: serve `dist/` at `/plugins/` via Caddy as a plugin repo; theme CSS
 
-### Phase 5 — copyparty behind Authentik  (built; apply pending)
+### Phase 5 — copyparty behind Authentik  ✅ live
 - [ ] Authentik → Providers → Proxy provider, *forward auth (single application)*, external host
       `https://ovh1.HOST`; Application bound to it; add to the embedded outpost
 - [ ] Caddy: `forward_auth 127.0.0.1:9000 { uri /outpost.goauthentik.io/auth/caddy; copy_headers X-Authentik-Username X-Authentik-Groups … }`
@@ -194,13 +194,13 @@ Checklist:
 - [x] decided: no WebDAV (Discord login only); browser downloads
 - [ ] policy on the Application: only `editors` + `admins` + `streamers` may pass
 
-### Phase 6 — Syncthing UI + dump behind Authentik  (built together with phase 5; apply pending)
+### Phase 6 — Syncthing UI + dump behind Authentik  ✅ live
 - [ ] `/sync/`: forward-auth, policy `admins` only; Syncthing GUI auth can stay as a second layer
 - [ ] `dumpd.py`: read `X-Authentik-Username` / `X-Authentik-Groups` from Caddy instead of basic auth;
       admin = member of `admins`; drop `dump-users.txt`
 - [ ] dock: `Dump tonight` sends no credentials; session cookie carries identity
 
-### Phase 7 — dock polish + cleanup  (built with phase 11; apply pending)
+### Phase 7 — dock polish + cleanup  ✅ live (dock mints tokens; no stored credentials)
 - [ ] `dumpd.py` gains `GET /me` → `{user, groups, stream_key}` for the logged-in streamer;
       dock fills OBS settings from it — nobody types a stream password again
 - [ ] `configure.py` shrinks: hostnames + streamers only (editors and admin live in Authentik)
@@ -223,7 +223,8 @@ Checklist:
 
 ### Discord bot (separate dev) — runtime provided by the stack
 - `mariadb` container, db `vanguardbot` / user `bot`, loopback only (ssh tunnel for dev access)
-- `discord-bot` container: python 3.12 + ffmpeg + yt-dlp baseline; code bind-mounted from `bot/app/`;
+- `discord-bot` container: python 3.12 + ffmpeg + curl, and no bot dependencies — its CI builds a release
+  bundle (code + `vendor/` wheels) that dumpd installs into `/srv/bot/app`, bind-mounted at `/app`;
   env gives token, guild, DB creds, rw `/media/clips`, `/data` scratch; restarts on exit
 - web UI at `bot.` behind Caddy + Authentik forward-auth (first use of the forward-auth pattern that
   Phase 5 applies to copyparty); the bot trusts `X-Authentik-*` headers and implements no login
@@ -232,7 +233,7 @@ Checklist:
 - [ ] CNAME `bot` → main host; `configure.py --full` (asks the hostname, bot token + guild) → `docker compose up -d --build`
 - [ ] Authentik: domain-level Proxy Provider (covers live/bot/main); Caddy requires group `bot-users` on the bot host (`BOT_REQUIRE_GROUP`)
 - [ ] Discord roles → Authentik groups: `authentik/discord-roles-to-groups.py` on both source flows (scope `guilds.members.read`)
-- [ ] dev drops `main.py` into `bot/app/` → `docker compose restart discord-bot`
+- [x] the bot repo's CI builds a release bundle and a push to `main` installs it (signed, via dumpd `/deploy-bot`)
 
 ### Phase 9 — Gitea + Actions runner
 - `gitea` on MariaDB (db/user from `mariadb/init/10-gitea.sh` on first init, else `./createGiteaDb.sh`),
@@ -246,7 +247,7 @@ Checklist:
       publishes the plugin zip is the obvious first pipeline
 - risk: the runner's Docker socket is root on the host for anyone who can push a workflow
 
-### Phase 10 — Gitea login via Authentik (OIDC)  (later)
+### Phase 10 — Gitea login via Authentik (OIDC)  ✅ live, verified
 Same pattern as Jellyfin: Authentik is the identity, Gitea trusts it.
 - [ ] Authentik → Applications → Create with provider → OAuth2/OpenID: name `gitea`, redirect
       `https://git.HOST/user/oauth2/authentik/callback`, authorization flow implicit-consent, scopes
@@ -255,15 +256,14 @@ Same pattern as Jellyfin: Authentik is the identity, Gitea trusts it.
       provider OpenID Connect, name `authentik` (must match the callback path above),
       client ID/secret, discovery URL `https://auth.HOST/application/o/<slug>/.well-known/openid-configuration`,
       "Skip local 2FA", claim mapping: group claim `groups` → Gitea admin group `admins`
-- [ ] Gitea app.ini / env: `GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION=true`,
+- [x] Gitea env: `DISABLE_REGISTRATION=false` + `ALLOW_ONLY_EXTERNAL_REGISTRATION=true` (local self-signup stays
-      `GITEA__oauth2_client__ACCOUNT_LINKING=auto`, `GITEA__oauth2_client__USERNAME=nickname` so first
+      off), `ENABLE_AUTO_REGISTRATION`, `ACCOUNT_LINKING=auto`, `USERNAME=nickname`, `UPDATE_AVATAR`
-      Discord login creates the Gitea user with no form (registration stays closed for local accounts)
+- [ ] optional: hide the password form on the login page (custom template) so Discord is the only visible way in
-- [ ] optional: `GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION=true`, hide the password form on the
+- [ ] Authentik groups `git-users` and `git-admins` bound on the `Gitea` application; Gitea admin claim = `git-admins`;
-      login page (`GITEA__oauth2_client__OPENID_CONNECT_SCOPES`, custom template) so Discord is the only way in
+      nesting `authentik Admins` → `admins` → `git-admins` so admins inherit it
-- [ ] policy on the Authentik application: who may reach Gitea at all (e.g. `admins` + a `devs` group)
 - [ ] test with a non-admin Discord user; confirm admin mapping via `admins` group
 
-### Phase 11 — MediaMTX auth through Authentik (no stream keys anywhere)  (built; apply pending)
+### Phase 11 — MediaMTX auth through Authentik (no stream keys anywhere)  ✅ live
 Implemented variant: **short-lived HMAC tokens minted by dumpd** for Discord-authenticated members of
 `streamers` (no Authentik app passwords — "Discord only" decision). Dock and portal mint a token per Go Live;
 manual setups get it from the dock's accordion. MediaMTX `authMethod: http` → dumpd `/mtx-auth` on 8081
@@ -287,6 +287,76 @@ Discord roles); the publish path must start with the user's own name.
 - [ ] dock + README; remove streamers from configure.py; delete `streamers.json`
 - [ ] test: OBS with an app password; Go Live with no key; a non-streamer gets denied
 
+### Phase 12 — frame-accurate timecode for multicam via an overscan band  ✅ live, verified (OBS: strip, −3 ms drift, 0 gaps)
+Target: clips from several streamers align to the frame (16.7 ms @60) on a multicam timeline; the game
+image is untouched and finals carry nothing. Reference clock = the server; streamer PC clocks are never trusted.
+
+Facts that shape it: within a recording, frame timing is the encoder's capture clock (network lag only shifts
+*arrival*, which is measured once); but a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h), so a
+permanent reference is needed for long sessions, not just a start burst.
+
+- **Overscan band**: the published canvas is the game image plus 16 extra rows at the bottom. The band holds a
+  machine-readable strip (black/white cells: server time in ms + checksum + framing, cell size scaled to the
+  canvas) and a small human-readable server clock, **continuously**.
+  - Go Live: canvas composite = video (W×H) + band → W×(H+16); the `/clock/` logic runs in-page.
+  - OBS: the dock (obs-websocket) offers "add overscan band": SetVideoSettings canvas/output to H+16 while not
+    streaming, CreateInput `browser_source` → `/clock/` positioned in the band, toggle/remove. The browser
+    source shares the dock's cookies (Authentik). Opt-out = coarse timecode only.
+  - `/clock/` measures its offset to the server clock over HTTPS (±few ms) and shows **server** time.
+- **Sweep**: when publishing a stream: extract frames at the start, middle and end; locate + decode the strip;
+  frame-0 capture time = median(decoded − pts) at the start (frame-accurate) → QuickTime `tmcd` track
+  (time of day, Eastern, file fps) + `creation_time` (UTC) in the existing lossless remux; a `.tc.json`
+  sidecar with the samples records drift over the file. No strip → server arrival time (coarse), logged.
+  Strip present → the band is cropped off in a re-encode (decided 2026-11-01: finals = picture + audio + tmcd,
+  nothing else); no strip → lossless remux, never crop.
+- **Viewing**: live portal crops the band on the live picture (canvas/overflow fix pending); recordings are
+  already at game resolution.
+- MediaMTX `recordPath` with `%f` (microseconds) for the fallback; plugin `ItemTimestamp` prefers `creation_time`.
+- [x] `/clock/` page: offset measurement, strip encoder, readable clock, band layout
+- [x] dock: overscan band setup + browser source via obs-websocket; opt-out
+- [x] portal Go Live: band composite (toggle); portal viewer: CSS crop of the band
+- [x] sweep: `bin/vgstrip.py` (ffmpeg frames → decode → offset + drift), tmcd + creation_time, `<stem>.json` sidecar (read by the plugin)
+- [x] MediaMTX `%f`; plugin timestamp update; README editor workflow (multicam: timecode → audio cross-check;
+      export at 2560×1440)
+- rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
+
+### Phase 13 — `diag.` host: Kibana (logs + metrics + alerting)  (built; needs a webhook + two Kibana rules)
+
+**Decided 2026-11-03, after one wrong turn.** First build was VictoriaLogs + Vector + VictoriaMetrics + two
+vmalert instances + Alertmanager, eight containers counting the exporters, and it produced **no dashboards** —
+five vendor UIs behind one hostname and an index page of links. The requirement was one tool with views, and
+the honest answers to that were Netdata (one container, weakest log search) or Elastic (four containers, one
+UI, strongest log search). Elastic was chosen. Grafana was considered and rejected *for this purpose*: it is a
+UI only, so its stack is Loki + Prometheus + Alloy + Grafana, which is the same piece count with more parts.
+
+Four containers: `elasticsearch` (both logs and metrics) · `kibana` (the whole `diag.` host, at the root) ·
+`filebeat` · `metricbeat` (which also retires node_exporter and cAdvisor). Detail in README §5h and LOGGING.md.
+
+- [x] compose: the four, all loopback, published only through the gated `diag.` host
+- [x] pinned: 2 GB heap, `vm.max_map_count`, security off (loopback + Authentik), fixed Kibana encryption key
+- [x] both Beats emit the same correlation fields, so one dashboard spans logs and metrics
+- [x] `configure.py`: `diag` role → Kibana at the root; `--remove-orphans` on apply so the old eight go away
+- [x] `install.sh`: the ES data dir, the sysctl, the deadman
+- [x] deadman repointed at Elastic, including the two failures a process check misses: a Beat that runs without
+      shipping, and indices flipped read-only by the watermark (which never clears itself)
+- [ ] a Discord webhook → `ALERT_WEBHOOK_URL`
+- [ ] in Kibana: the retention delete phase; the always-firing heartbeat rule → dumpd `/heartbeat`; a webhook
+      connector for alerts; then rules for the conditions the old vmalert files listed (disk, load, restart
+      loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble)
+- [ ] later: ops controls on this host (restart buttons) — allowlisted dumpd endpoints, authorised from the
+      Authentik headers, every invocation logged as a logfmt event. Never a page with Docker access.
+
+### Phase 14 — push-to-deploy  (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
+- server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved
+- `vanguard-deploy pull|apply|rollback|pending|status` applies only the changed parts; `check.sh` gates it;
+  MediaMTX restarts are deferred while anything is live (`deploy-pending.timer`)
+- Gitea push webhook → dumpd `/deploy` (bridge port, HMAC) → `vanguard-deploy pull` when `DEPLOY_AUTO=1`
+- CI (draft): `check.sh` + plugin tests/build on the runner; the runner never touches the host
+- [ ] run `switchToClone.sh key` / add deploy key / `switchToClone.sh clone` on the server
+- [ ] `./configure.py --apply` once (installs vanguard-deploy, deploy.env, the pending timer)
+- [ ] add the Gitea webhook (secret from `.env`); watch `journalctl -u dumpd | grep deploy` for a few pushes
+- [ ] when trusted: `DEPLOY_AUTO=1` in `.env` + `./configure.py --apply`; move `ci.yml` into `.gitea/workflows/`
+
 ### Option — more clip sources (Medal, Twitch/YouTube/Kick clips, Discord uploads)
 
 Today the only clip intake is a local clipper folder synced by Syncthing. Members without that
@@ -325,10 +395,22 @@ Checklist if adopted:
 - [ ] per-user clips volume in `copyparty.conf` once forward-auth is live (Phase 5)
 
 ### Later — when it hurts
+
+- [ ] **Off-site gap filling** (instead of burning the SYS-1's CPU): streams whose sidecar reports
+      `gaps.total_missing_ms` ≥ threshold are re-encoded elsewhere with a hardware encoder.
+      - the sweep keeps the raw recording in `.src/` and the sidecar flags it (`needs_fill: true`)
+      - a nightly job on the home box (RTX 2080 Ti, NVENC) or a Spot GPU instance pulls flagged raws over
+        SFTP, re-encodes to CFR with duplicated frames (same editing standard, `tmcd` + `creation_time`
+        carried from the sidecar), pushes the result back over the published file, clears the flag
+      - Jellyfin keeps the same path → item identity and watched state survive; the plugin re-reads metadata
+      - until then: `FILL_GAPS_MS` stays off; players hold frames across gaps, editors see the gap list
+- [ ] **SYS-3 (E-2288G, Quick Sync)** — the 2×6 TB disk option is currently unavailable; re-check and swap when
+      it returns. Then: Jellyfin transcoding on, sweep re-encodes in minutes, `FILL_GAPS_MS` can simply be on.
 - [ ] **SYS-3 (E-2288G)**: Quick Sync → Jellyfin transcoding on, sweep re-encodes in minutes.
       Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
 - [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron
 
+
 ## Risks / gotchas
 
 - **Phase 5 lockout.** Keep one SSH session open and a copyparty local admin account until the
@@ -340,6 +422,12 @@ Checklist if adopted:
 - **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1.
 - **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
 - **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public.
+- **A new host needs `--full` or `--hosts`, never `--apply`.** `configure.py` is stateless and reads hostnames
+  back out of the Caddyfile, so apply silently omits a role that is not already there and the name ends up with
+  no certificate (HOSTNAME-CHANGE.md §D). Cost an hour when `diag.` was added.
+- **Observability path prefixes cascade.** Setting `-http.pathPrefix` (or Alertmanager's `--web.route-prefix`)
+  moves that service's ingestion, health and metrics paths too, so the Vector sink, the scrape config and the
+  deadman's probes all have to follow. LOGGING.md lists the four traps this produced.
 
 ## Monthly cost
 Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free.

FRESH-INSTALL.md

@@ -0,0 +1,373 @@
+# Fresh install — from an empty OVH box to a working stack with the deploy pipeline
+
+Everything in order. Scripted steps are commands; manual steps are click lists. `HOST` = your main hostname
+(ours: `media.dumpstervigilantes.rip`), `BASE` = the parent domain. Detailed background for each part is in
+README.md (section numbers given).
+
+## 0. Before touching the server
+- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `diag`, `auth`, `watch`, `git`, `bot` → `HOST`.
+  Optional: the bare domain `A` → server IP (it redirects to `https://HOST/`).
+- The Discord bot lives in its **own** repo (ours `VanguardOrg/VanguardBot`), not in this one. Nothing here
+  needs it to exist; section 7a wires it up afterwards.
+- Discord developer portal → New Application (one app serves Authentik login, the Jellyfin plugin's bot and the
+  Discord bot):
+  - OAuth2 → Redirects: `https://auth.BASE/source/oauth/callback/discord/` (the last segment is the Authentik
+    source slug — keep them identical). Copy Client ID + Client Secret.
+  - Bot → Reset Token (copy it); enable **Server Members** intent; OAuth2 URL generator: scope `bot`, permissions
+    View Channels + Send Messages + Embed Links + Attach Files → invite it to the server.
+  - Discord → User Settings → Advanced → Developer Mode on; right-click the server → Copy Server ID; Server
+    Settings → Roles → Copy Role ID for each role you will map to a group.
+- If this is a **rebuild**, have the backups ready: `.env`, `/srv/appdata`, `/srv/media`, `/etc/vanguard/clip-quotas.json`.
+  Restoring `.env` + `/srv/appdata` makes sections 5–7 unnecessary (that state lives there).
+
+## 1. OS (README §1)
+OVH manager → Reinstall → Debian 13, custom partitioning, distribution kernel. RAID 1:
+`/boot` 1 GB ext4 · `/` 250 GB ext4 · swap 8 GB per disk · `/srv` rest xfs. Add your SSH key. You get the `debian` user with sudo.
+
+## 2. Code onto the box
+Fresh box has no deploy key yet, so copy the repo in:
+
+    scp -r vanguard-stream debian@HOST:~/        # from a machine that has the repo (or git clone over https with a token)
+
+## 3. Host install (README §2, §4)
+
+    ssh debian@HOST
+    cd ~/vanguard-stream && sudo bash install.sh
+
+Packages, Docker, timezone (Eastern), uid 1000, `/srv` folders, host scripts and timers, sysctl, live-portal
+build, **firewall** (keep this session open; test a new ssh in a second terminal). Log out and back in once (docker group).
+
+## 4. Configure + start (README §3)
+
+    cd ~/vanguard-stream && ./configure.py --full
+
+Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`diag`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it
+redirects; Discord bot token + server ID (optional now, needed for the plugin feed); "is live" webhook (optional).
+Generates `.env` with every secret, the configs, starts all containers, waits, then runs the post-start fixups
+(Syncthing GUI auth off, Gitea admin user, Jellyfin plugin). `./configure.py --show` prints every credential
+you will need below (akadmin password, Gitea admin, MariaDB, bot DB).
+
+Check: `docker compose ps` → all `Up`; `https://auth.BASE/`, `https://watch.BASE/`, `https://git.BASE/`,
+`https://HOST/` answer with certificates (Caddy fetches them itself, give it a minute).
+
+## 5. Authentik (README §5c, §5g) — manual, ~20 minutes
+Log in at `https://auth.BASE/` as `akadmin` with `AUTHENTIK_BOOTSTRAP_PASSWORD` from `.env` (change it: user menu → settings).
+1. Directory → Federation & Social login → Create → **Discord OAuth Source**: name `Discord`, slug `discord`,
+   consumer key/secret = the Discord app's Client ID/Secret, scopes `guilds guilds.members.read`, user matching
+   "link on unique identifier", flows default-source-authentication / default-source-enrollment.
+2. Flows & Stages → Stages → `default-authentication-identification` → Sources: add Discord, tick "Show sources'
+   labels". (No Discord button appears until this is done.)
+3. Flows & Stages → Stages → `default-source-enrollment-write` → User type **Internal**, user path `users`.
+4. Directory → Groups: `streamers`, `editors`, `media-admins`, `admins`, `bot-users`, `git-users`, `git-admins`.
+   Nesting: `admins` parent `authentik Admins`; `authentik Admins` parent `git-admins` (admins inherit everything).
+   Inherited groups reach the apps only with the extra scope mapping `Vanguard profile (nested groups)` (README 5e),
+   ticked on the `Vanguard sites`, `Jellyfin` and `Gitea` providers next to the default profile mapping.
+5. Customization → Policies → Create Expression Policy ×2 from the repo:
+   `discord-guild-member` (`authentik/discord-guild-policy.py`, set your server ID) and
+   `discord-roles-to-groups` (`authentik/discord-roles-to-groups.py`, set server ID + role-id→group table).
+   Tick "Execution logging" on both.
+6. Flows & Stages → Flows → `default-source-enrollment` and `default-source-authentication`: bind both policies
+   (guild first), **Policy engine mode: all**.
+7. Applications → Outposts → edit "authentik Embedded Outpost" → config `authentik_host: https://auth.BASE`.
+8. Applications → Create with provider → `Vanguard sites`: **Proxy Provider**, implicit consent,
+   **Forward auth (domain level)**, authentication URL `https://auth.BASE`, cookie domain `BASE`,
+   invalidation flow `default-provider-invalidation-flow`. Edit the Embedded Outpost → add the application.
+9. Applications → Create with provider → `Jellyfin`: OAuth2/OpenID, implicit consent, Confidential, redirect
+   `https://watch.BASE/sso/OID/redirect/authentik`, default signing cert, scopes `openid email profile`. Keep ID + secret.
+10. Applications → Create with provider → `Gitea` (slug `gitea`): OAuth2/OpenID, same settings, redirect
+    `https://git.BASE/user/oauth2/authentik/callback`. Bind groups `git-users` + `git-admins` on the application.
+11. Test in a private window: `https://auth.BASE/` → Discord → your user exists and is Internal. Add yourself to
+    `admins` (and `streamers`, `media-admins` as needed) or give yourself the Discord roles; sign out and back in.
+    `https://HOST/whoami` shows your groups.
+
+## 6. Jellyfin (README §5c) — manual, ~10 minutes
+`https://watch.BASE/` → wizard: admin user; libraries `/media/streams` and `/media/clips` as **Shows** with
+metadata downloaders = only "Vanguard Metadata" (Series/Season/Episode), image fetchers Vanguard Metadata
+(Series) + Screen Grabber (Episode), real-time monitoring on; `/media/finals` as Home Videos, fetchers off.
+Dashboard → Playback → Transcoding: hardware acceleration none. Dashboard → Plugins: Vanguard Metadata is
+installed → its page: Discord bot token + server ID, feed channel ID, public URL `https://watch.BASE`.
+Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json`
+→ Catalog → SSO Authentication → install → `docker compose restart jellyfin`. Plugins → SSO-Auth → add OID provider
+`authentik`: endpoint `https://auth.BASE/application/o/jellyfin/`, client ID/secret, roles claim `groups`,
+admin roles `admins`, enabled folders = the 3 libraries, Scheme Override `https`, Port Override EMPTY, Save.
+Dashboard → General → Branding: paste `jellyfin/branding-disclaimer.html` (set the hostname) and `jellyfin/branding.css`.
+Test: `https://watch.BASE/sso/OID/start/authentik`. Scheduled Tasks → run "Vanguard: snapshot Discord events",
+"roll seasons", "enforce playback policy" once.
+
+## 7. Gitea (README §5e) — manual, ~10 minutes
+Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` from `.env`.
+1. Site Administration → Identity & Access → Authentication Sources → Add: OAuth2, name `authentik`, OpenID
+   Connect, client ID/secret from step 5.10, discovery URL
+   `https://auth.BASE/application/o/gitea/.well-known/openid-configuration`, Skip local 2FA, group claim
+   `groups`, administrator group `git-admins`.
+2. Create the repo `techy/Media-Server-Setup` (empty) and push this repo to it from your dev box:
+   `git remote add origin ssh://git@git.BASE:2222/techy/Media-Server-Setup.git && git push -u origin main`
+   (your SSH key under Settings → SSH keys).
+3. Runner: `./registerGiteaRunner.sh` on the server → Site Administration → Actions → Runners shows it idle.
+
+## 7a. Discord bot (README 5d)
+1. The bot is **not** part of this repo. Its code arrives as a release bundle built by its own Gitea Actions
+   workflow and installed by `deployBotBundle.sh`; the image here carries no bot dependencies.
+2. Create the org + repo in Gitea (ours: `VanguardOrg/VanguardBot`) and let the dev push to it.
+3. The runner needs the **`python312`** label (CPython 3.12 + node, for the JavaScript actions). It is in
+   `docker-compose.yml`; labels are fixed at registration, so `./registerGiteaRunner.sh` after any change.
+4. In Gitea → the bot repo → Settings → Actions: secrets `BOT_DEPLOY_SECRET` (from `./configure.py --show`)
+   and `DISCORD_BOT_TOKEN`; variable `DISCORD_GUILD_ID`. **The bot's secrets live here, not in `.env`.**
+5. Authentik: a Proxy Provider for `https://bot.BASE` on the embedded outpost (same shape as `live.`).
+6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the
+   container. By hand: `./deployBotBundle.sh <bundle.zip>`; back out with `--rollback`.
+
+## 7b. Diagnostics — Kibana (README §5h)
+1. The `diag` CNAME must exist first. Four containers come up from `configure.py`: Elasticsearch (logs AND
+   metrics), Kibana (the whole `diag.BASE` host, at the root), Filebeat and Metricbeat. Nothing is published
+   except through the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
+2. `install.sh` and every apply set `vm.max_map_count=262144`. Elasticsearch will not start without it.
+3. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it Kibana rules still evaluate and the deadman still
+   logs; only delivery is missing.
+4. Three first-run actions inside Kibana, all UI so they stay adjustable: the retention delete phase, the
+   always-firing heartbeat rule the deadman watches, and a webhook connector for alerts. README §5h lists them.
+5. Adding this host to a RUNNING server needs `--full` or `--hosts`, not `--apply` (HOSTNAME-CHANGE.md §D).
+
+## 8. Pipeline (README "Deployment")
+1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo
+   → `./switchToClone.sh clone` (keeps `.env`, generated configs, plugin zips) → `./configure.py --apply`.
+2. Gitea → repo → Settings → Webhooks → Add → Gitea: URL `http://<docker gateway>:8081/deploy`
+   (`docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}'`), secret =
+   `DEPLOY_WEBHOOK_SECRET` from `.env`, push events, branch `main`. Test delivery → 202 "auto-deploy disabled".
+3. Manual deploys from now on: push, then `sudo vanguard-deploy pull` on the server. When trusted:
+   `DEPLOY_AUTO=1` in `.env` → `./configure.py --apply`; CI: `git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml`.
+
+## 9. Streamers, gamers, editors (README §5, §5a, §6)
+- Streamers: OBS → WebSocket server on; Docks → Custom Browser Dock `https://live.BASE/dock/` → Discord login →
+  Settings: OBS WebSocket password; Timecode band → "Add band + clock source" (not streaming). Then title → Go Live.
+- Gamers: `https://clips.BASE/` → paste their Syncthing Device ID → accept device + folder on their PC (Send Only).
+  Syncthing UI (`https://sync.BASE/`, media-admins): Actions → Settings → Connections → incoming rate limit 12500 KiB/s.
+- Everyone: `https://HOST/` (the gateway page; editors browse at `https://files.BASE/`).
+
+## 10. Afterwards
+- Pin `AUTHENTIK_TAG` in `.env` to the running version. Rotate any secret that was ever pasted into chat.
+- Back up `.env`, `/srv/appdata`, `/srv/media` (home archive puller: README §7).
+- `STATUS.md` = what runs and what is unfinished; `check.sh` before any commit.
+
+---
+
+# Appendix — the runbook: every action, in order, with the exact commands
+
+Conventions: commands prefixed `dev$` run on your workstation (it has the repo at `~/claudevs/vanguard-stream`
+and your SSH key); `srv$` run on the server over ssh as `debian`. Replace `HOST` / `BASE` with your names
+(ours: `media.dumpstervigilantes.rip` / `dumpstervigilantes.rip`). Click paths are written `App → Menu → Item`.
+Every step ends with how you know it worked.
+
+## A. DNS and Discord (no server yet)
+
+A1. DNS at your registrar:
+    A      HOST            → <server IP>
+    CNAME  auth.BASE       → HOST
+    CNAME  watch.BASE      → HOST
+    CNAME  git.BASE        → HOST
+    CNAME  bot.BASE        → HOST
+    CNAME  live.BASE       → HOST
+    (optional) A  BASE     → <server IP>
+    Check from dev:  dev$ for h in HOST auth.BASE watch.BASE git.BASE bot.BASE live.BASE; do printf '%-28s ' $h; getent hosts $h | awk '{print $1}'; done
+    → every line shows the server IP.
+
+A2. Discord Developer Portal → New Application "Vanguard".
+    OAuth2 → Redirects → add  https://auth.BASE/source/oauth/callback/discord/  → Save. Copy Client ID and (Reset) Client Secret.
+    Bot → Reset Token → copy. Privileged Gateway Intents → Server Members Intent ON → Save.
+    OAuth2 → URL Generator → scopes: bot; permissions: View Channels, Send Messages, Embed Links, Attach Files → open the URL → invite the bot to your server.
+    → the bot appears in your server's member list.
+
+A3. Discord IDs: User Settings → Advanced → Developer Mode ON. Right-click your server → Copy Server ID.
+    Server Settings → Roles → right-click each role you'll map → Copy Role ID. Keep a note like:
+    guild=123…  streamers=…  editors=…  media-admins=…  admins=…  bot-users=…  git-users=…  git-admins=…
+
+## B. Operating system
+
+B1. OVH manager → the server → Reinstall → Debian 13 → custom partitioning, "use distribution kernel":
+    /boot 1 GiB ext4 (RAID1) · / 250 GiB ext4 (RAID1) · swap 8 GiB (per disk) · /srv remaining xfs (RAID1).
+    Add your SSH public key. Wait for the mail.
+B2. dev$ ssh debian@HOST 'lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT; df -h /srv'
+    → /srv is xfs, ~3.4 TiB, mounted.
+
+## C. Code onto the box
+
+C1. dev$ cd ~/claudevs && scp -r vanguard-stream debian@HOST:~/
+C2. dev$ ssh debian@HOST 'ls ~/vanguard-stream | head; test -f ~/vanguard-stream/install.sh && echo ok'
+    → "ok".
+
+## D. Host install
+
+D1. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && sudo bash install.sh'
+    (~5 min: packages, docker, timezone, folders, scripts, timers, portal build, firewall.) KEEP THIS TERMINAL OPEN.
+D2. In a SECOND terminal:  dev$ ssh debian@HOST 'echo still in'  → "still in" (firewall did not lock you out).
+D3. dev$ ssh debian@HOST 'timedatectl show -p Timezone --value; systemctl is-active docker dumpd normalize-sweep.timer trash-purge.timer nftables; docker --version; node --version; ls /srv/live | head -3'
+    → America/New_York; five "active"; versions; /srv/live has index.html.
+D4. Log out/in once for the docker group:  dev$ ssh debian@HOST 'docker ps >/dev/null && echo docker-ok'  → docker-ok.
+
+## E. Configure and start the stack
+
+E1. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./configure.py --full'
+    Answers: HOST; Enter ×8 for the sub-hostnames; bare domain (Enter = BASE, or blank to skip) and its redirect;
+    bot token (from A2) ; server ID (A3) ; "is live" webhook (optional); y to apply.
+    (Pulls ~2 GB of images, starts 13 containers, waits, runs post-start fixups.)
+E2. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}"'
+    → every container "Up" (authentik-server/worker take ~1 min; act_runner restarts until step G3 — fine).
+E3. dev$ for h in HOST auth.BASE watch.BASE git.BASE live.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done
+    → 200/302 for all (302 = redirected to login). If 000, wait a minute for certificates and retry.
+E4. dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./configure.py --show'
+    → copy out: akadmin password (AUTHENTIK_BOOTSTRAP_PASSWORD), Gitea admin user/password, MariaDB root + bot + gitea passwords.
+E5. dev$ ssh debian@HOST 'journalctl -u dumpd -n 2 --no-pager | tail -1; grep -c "authMethod: http" ~/vanguard-stream/mediamtx.yml; sudo nft list table inet vanguard | grep -c dport'
+    → dumpd line ending "token-secret=ok"; 1; a dozen rules.
+
+## F. Authentik (browser, logged in as akadmin at https://auth.BASE/ with the E4 password)
+
+F1. Change akadmin's password: top-right user → Settings → Change password.
+F2. Directory → Federation & Social login → Create → "Discord OAuth Source":
+    Name Discord · Slug discord · Enabled · User matching "Link users on unique identifier" ·
+    Consumer key = Client ID · Consumer secret = Client Secret · Scopes: guilds guilds.members.read ·
+    (Flow settings if shown: authentication default-source-authentication, enrollment default-source-enrollment) → Finish.
+F3. Flows & Stages → Stages → default-authentication-identification → Edit → Sources: select Discord; tick "Show sources' labels" → Update.
+    → the login page (private window) now shows a Discord button.
+F4. Flows & Stages → Stages → default-source-enrollment-write → Edit → User type: Internal · User path: users → Update.
+F5. Directory → Groups → Create, one each: streamers, editors, media-admins, admins, bot-users, git-users, git-admins.
+    Then nesting: Groups → admins → Edit → Parent: authentik Admins → Update. Groups → authentik Admins → Edit → Parent: git-admins → Update.
+F5b. Customization → Property Mappings → Create → Scope Mapping: Name `Vanguard profile (nested groups)`, Scope name `profile`, Expression:
+        direct = set(request.user.ak_groups.all())
+        return {"groups": sorted(g.name for g in request.user.all_groups() if g not in direct)}
+     → Finish. Later, on each provider created in F7–F9 (Vanguard sites, Jellyfin, Gitea) → Edit → Advanced protocol settings → Scopes: tick it
+     next to the default profile mapping (proxy providers re-add the default on save; that is fine, the two merge). Without it `/whoami` never shows git-admins.
+F6. Customization → Policies → Create → Expression Policy:
+    name discord-guild-member · expression = contents of authentik/discord-guild-policy.py with ACCEPTED_GUILD_ID + GUILD_NAME filled in · tick Execution logging → Create.
+    Again: name discord-roles-to-groups · expression = authentik/discord-roles-to-groups.py with GUILD_ID + ROLE_TO_GROUP filled from A3 · Execution logging → Create.
+F7. Flows & Stages → Flows → default-source-enrollment → Policy / Group / User Bindings → Bind existing policy → discord-guild-member (order 0) → Create;
+    again → discord-roles-to-groups (order 10) → Create. Then Edit the flow → Policy engine mode: all → Update.
+    Repeat all of F7 for default-source-authentication.
+F8. Applications → Outposts → "authentik Embedded Outpost" → Edit → in the configuration box set
+    authentik_host: https://auth.BASE  (leave authentik_host_browser empty) → Update.
+F9. Applications → Create with provider:
+    Application name "Vanguard sites" · Provider type Proxy Provider · Authorization flow default-provider-authorization-implicit-consent ·
+    Mode "Forward auth (domain level)" · Authentication URL https://auth.BASE · Cookie domain BASE ·
+    Invalidation flow default-provider-invalidation-flow → Finish.
+    Applications → Outposts → Embedded Outpost → Edit → Applications: add "Vanguard sites" → Update.
+F10. Applications → Create with provider: "Jellyfin" (slug jellyfin) · OAuth2/OpenID Provider · implicit-consent ·
+     Client type Confidential · Redirect URI (strict) https://watch.BASE/sso/OID/redirect/authentik ·
+     Signing key: the default certificate · Advanced → Scopes: openid, email, profile → Finish. Note Client ID + Secret.
+F11. Applications → Create with provider: "Gitea" (slug gitea) · same provider settings · Redirect URI https://git.BASE/user/oauth2/authentik/callback → Finish.
+     Note Client ID + Secret. Applications → Gitea → Policy / Group / User Bindings → Bind existing group → git-users → Create; again → git-admins.
+F12. Private window → https://auth.BASE/ → Discord → log in. Admin: Directory → Users → your user exists, type Internal.
+     Directory → Groups → admins → Users tab → Add existing user → you (and media-admins, streamers as you like) — or give yourself the Discord roles.
+F13. Private window → https://HOST/whoami → "user: <you>" and "groups: …" listing what you added. (If groups are missing: sign out at https://auth.BASE/if/flow/default-invalidation-flow/ and log in again.)
+     Events → Logs → filter policy_execution → entries for both policies with result pass.
+
+## G. Gitea (browser at https://git.BASE/, log in with GITEA_ADMIN_USER / GITEA_ADMIN_PASSWORD from E4)
+
+G1. Site Administration → Identity & Access → Authentication Sources → Add Authentication Source:
+    Type OAuth2 · Name authentik · Provider OpenID Connect · Client ID / Secret from F11 ·
+    OpenID Connect Auto Discovery URL https://auth.BASE/application/o/gitea/.well-known/openid-configuration ·
+    Skip local 2FA ✓ · Additional scopes (blank) · Claim name providing group names: groups · Group claim value for administrator users: git-admins → Add.
+G2. New repository (top-right +) → owner techy → name Media-Server-Setup → NOT initialized (no README) → Create.
+    Your SSH key: Settings → SSH / GPG Keys → Add Key (paste ~/.ssh/id_ed25519.pub from dev).
+G3. dev$ cd ~/claudevs/vanguard-stream && git remote add origin ssh://git@git.BASE:2222/techy/Media-Server-Setup.git && git push -u origin main
+    (accept the host key for port 2222) → "branch 'main' set up to track".
+G4. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./registerGiteaRunner.sh'
+    → log lines "runner: connect…" ; Site Administration → Actions → Runners lists vanguard-runner, Idle.
+G5. Private window → https://git.BASE/user/login → "Sign in with authentik" → Discord → logged in as your Discord username.
+    (Requires git-users or git-admins membership; admins inherit git-admins via F5.)
+
+## H. Jellyfin (browser at https://watch.BASE/)
+
+H1. Wizard: language → admin user + password → Add Media Library:
+    Content type Shows · Display name Streams · Folder /media/streams · Metadata downloaders (Series, Season, Episode): tick ONLY "Vanguard Metadata" ·
+    Image fetchers: Series: Vanguard Metadata; Episode: Screen Grabber · Enable real time monitoring ✓ → OK.
+    Add Media Library: Shows · Clips · /media/clips · same provider settings → OK.
+    Add Media Library: Home Videos and Photos · Finals · /media/finals · all fetchers unticked → OK.
+    Finish the wizard (metadata language, remote access defaults).
+H2. Dashboard → Playback → Transcoding → Hardware acceleration: None → Save.
+H3. Dashboard → Plugins → My Plugins → Vanguard Metadata → settings: Discord bot token (A2), Server ID (A3), feed channel ID
+    (right-click the Discord channel → Copy Channel ID), Public URL https://watch.BASE, Authentik URL https://auth.BASE (token optional) → Save.
+H4. Dashboard → Plugins → Repositories → + → Name SSO-Auth · URL https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json → Save.
+    Catalog → SSO Authentication → Install. Then:  dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose restart jellyfin'
+H5. Dashboard → Plugins → SSO-Auth → OpenID: Add provider: name authentik · OID Endpoint https://auth.BASE/application/o/jellyfin/ ·
+    Client ID / Secret from F10 · Enabled ✓ · Enable Authorization by Plugin ✓ · Roles claim: groups · Admin roles: admins ·
+    Enabled folders: Streams, Clips, Finals · Scheme Override: https · Port Override: EMPTY · (default: disable transcoding ✓) → Save (bottom of page).
+H6. Private window → https://watch.BASE/sso/OID/start/authentik → Discord → lands in Jellyfin logged in.
+H7. Dashboard → General → Branding → Login disclaimer: paste jellyfin/branding-disclaimer.html with watch.HOST replaced by watch.BASE;
+    Custom CSS: paste jellyfin/branding.css → Save. Login page now shows "Sign in with Discord" + a small "Admin login" link.
+H8. Dashboard → Scheduled Tasks → Vanguard: run "snapshot Discord events", then "roll seasons", then "enforce playback policy" (▶ each).
+    Dashboard → Users → your SSO user → Playback → "Allow video playback that requires transcoding" is unticked.
+
+## I. Pipeline
+
+I1. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh key'
+    → prints a public key. Gitea → repo → Settings → Deploy Keys → Add Deploy Key → paste → read-only → Add.
+I2. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh clone && cd ~/vanguard-stream && ./configure.py --apply && sudo vanguard-deploy status'
+    → "cloned. kept: …" ; status shows "main @ <commit>", DEPLOY_AUTO=0.
+I3. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker network inspect vanguard-stream_default -f "{{(index .IPAM.Config 0).Gateway}}"; grep ^DEPLOY_WEBHOOK_SECRET= .env'
+    → gateway IP (e.g. 172.18.0.1) and the secret.
+I4. Gitea → repo → Settings → Webhooks → Add Webhook → Gitea: Target URL http://<gateway>:8081/deploy · HTTP POST · Content type JSON ·
+    Secret = the value from I3 · Trigger on Push events · Branch filter main · Active → Add Webhook. Open it → Test Delivery.
+    dev$ ssh debian@HOST 'journalctl -u dumpd --since -5m --no-pager | grep "deploy webhook"'  → "… auto-deploy disabled (DEPLOY_AUTO=0)".
+I5. From now on, after a push:  dev$ ssh -t debian@HOST 'sudo vanguard-deploy pull'  (shows the changed files and every action).
+I6. (later, when trusted) dev$ ssh debian@HOST 'cd ~/vanguard-stream && sed -i "s/^DEPLOY_AUTO=.*/DEPLOY_AUTO=1/" .env && ./configure.py --apply >/dev/null && sudo vanguard-deploy status'
+    and  dev$ cd ~/claudevs/vanguard-stream && git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml && git commit -m "enable CI" && git push
+
+## J. People
+
+J1. Streamer (OBS): Tools → WebSocket Server Settings → Enable, set a password → OK.
+    Docks → Custom Browser Docks → name Vanguard, URL https://live.BASE/dock/ → Apply → log in with Discord inside the dock.
+    Dock → Settings: OBS WebSocket password → Save. Dock → Timecode band → "Add band + clock source" (while not streaming).
+    Dock: title → Go Live.  Check:  dev$ ssh debian@HOST 'journalctl -u dumpd --since -5m --no-pager | grep mtx-auth | tail -1'  → "allow publish user=…".
+J2. Gamer: https://clips.BASE/ → follow the three steps on the page (install Syncthing, paste Device ID, accept device + folder as Send Only).
+    Admin check: https://sync.BASE/ shows their device connected. Once: Syncthing → Actions → Settings → Connections → Incoming Rate Limit 12500 KiB/s.
+J3. Editor: https://files.BASE/ (copyparty) → browse, download; https://files.BASE/finals/ to upload. Needs the editors role.
+J4. Everyone: https://HOST/ (gateway page).
+
+## K. Verify the whole pipeline once
+
+K1. Stream 1 minute from OBS with the band, stop, wait ~8 min.
+K2. dev$ ssh debian@HOST 'f=$(ls -t /srv/media/streams/*/*.mp4 | head -1); echo "$f"; ffprobe -v error -show_entries stream_tags=timecode -of compact=nk=0:p=0 "$f" | head -1; python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print({k:d.get(k) for k in (\"source\",\"timecode\",\"drift_ms\")}, \"gaps:\", d.get(\"gaps\",{}).get(\"count\"))" "${f%.*}.json"'
+    → a flat file named <Title>__<timestamp>.mp4, tag:timecode set, source "strip", gaps 0.
+K3. https://watch.BASE/ → the recording is an episode under this weekday's season; Discord feed channel got a post with a frame grab.
+K4. https://live.BASE/ while someone streams → a window with the corner chip "webrtc".
+
+## L. Discord bot (first release)
+
+L1. Gitea → New Organization `VanguardOrg` → New Repository `VanguardBot` (private). The bot dev pushes to it.
+L2. dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./configure.py --show | grep -A2 "bot ci"'
+    → copy BOT_DEPLOY_SECRET and the list of keys CI may set.
+L3. Gitea → VanguardBot → Settings → Actions → Secrets → Add: `BOT_DEPLOY_SECRET` (L2), `DISCORD_BOT_TOKEN` (A2).
+    Variables → Add: `DISCORD_GUILD_ID` (A3). Optional `VANGUARD_DEPLOY_HOST` = the docker gateway from I3 if
+    the deploy job cannot detect it.
+L4. Check the runner advertises the python312 label:
+    dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./registerGiteaRunner.sh'   (only if it does not)
+    Site Administration → Actions → Runners → vanguard-runner → labels include python312.
+L5. Authentik → Applications → Create with provider → Proxy Provider, Forward auth (single application),
+    external host `https://bot.BASE` → add to the Embedded Outpost → bind group `bot-users`.
+L6. The dev pushes to `main`. The run should end with the install log, "settings written: DISCORD_BOT_TOKEN,
+    DISCORD_GUILD_ID" and HTTP 200.
+    dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./deployBotBundle.sh --status'
+    → version, commit, package count, container Up.
+L7. dev$ ssh debian@HOST 'docker logs discord-bot --tail 20'
+    → the bot's version line, then its MariaDB connection. First real MariaDB run: watch for a retry loop.
+L8. Private window → https://bot.BASE/ → Discord login → the bot's pages. Needs the `bot-users` group (admins
+    need it too, or set BOT_REQUIRE_GROUP= and let the bot's WEB_ALLOWED_GROUPS decide).
+
+## M. Diagnostics (browser at https://diag.BASE/)
+
+M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "elastic|kibana|beat"'
+    → four Up, elasticsearch healthy. Elasticsearch restarting is almost always vm.max_map_count or the heap.
+M2. https://diag.BASE/ → Discord login → Kibana. Discover shows logs within a minute of the Beats starting.
+M3. Stack Management → Data Views → create one over `filebeat-*,metricbeat-*`, so a single filter spans logs
+    and metrics. Metricbeat also installed its own system and Docker dashboards: start from those.
+M4. Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase (30 days is sensible).
+M5. Alerting → Connectors → Webhook → the URL from ALERT_WEBHOOK_URL. Then Rules → create the heartbeat rule:
+    always true, every minute, action = webhook POST to http://<docker gateway>:8081/heartbeat
+    (gateway: docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}').
+M6. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
+    → silent and exit 0 once M5 is done. Before that it correctly reports the heartbeat missing.
+
+## N. Afterwards
+
+N1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"'  → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply.
+N2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update).
+N3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7.

HOSTNAME-CHANGE.md

@@ -0,0 +1,238 @@
+# Changing hostnames safely
+
+Two cases: renaming one subdomain (e.g. `watch.BASE` → `video.BASE`), and moving the whole stack to a new base
+domain. Both are mostly `configure.py --full` plus a known list of places outside the repo that hold the name.
+Do it between streams: the main host is MediaMTX's WebRTC advertised host, so a main-host change needs a
+MediaMTX restart.
+
+Roles: `main` (gateway page at `/`, `/onboarding/`), `files` (copyparty), `clips`, `sync`, `live` (portal + Go Live,
+`/dock/`, `/dock/clock/`, `/dump`, `/me`, `/time`, `/whoami`), `auth`, `watch`, `git`, `bot`, `diag` (Kibana at `/`,
+`/streaming`, `/disk`, dumpd's `/streaming/api/*`), and optionally `apex` (the bare domain, a redirect to
+`APEX_REDIRECT`; it needs its own A record at the box or Caddy cannot get its certificate — as of 2026-11-06 the bare
+`dumpstervigilantes.rip` resolves elsewhere, so leave `APEX_HOST` empty or fix the record first). Ours: `media.`,
+`files.`, `clips.`, `sync.`, `live.`, `diag.`, … under `dumpstervigilantes.rip`. `ovh1.BASE` is ssh only.
+
+## The one rule: apps before Authentik
+
+Change the app, then Authentik, never the other way round. An app that already lives on its new name but whose
+Authentik provider still says the old one is simply unreachable until Authentik is updated; a provider pointed at
+a name the app does not answer on yet locks you out of the app, and if that app is the gate in front of the admin
+UI, out of everything. So in every procedure below Authentik comes after the servers, the containers, Jellyfin and
+Gitea — and the Discord developer portal after Authentik. The safety net at every step is akadmin's password
+card, `https://AUTH/if/flow/default-authentication-flow/?admin=1`, which bypasses the Discord forward.
+
+Note that `configure.py --apply` does touch Authentik on its own: the login redirect stage's target (the Discord
+source's login URL on the auth host) and the brand's default application. Both are Authentik-internal, neither
+decides whether an app is reachable, and the admin card ignores the redirect stage — so they cannot lock you out.
+The things that can — outpost host, proxy provider external host and cookie domain, Jellyfin/Gitea redirect
+URIs — are never changed by any script here.
+
+## Where hostnames live
+
+### Generated on the server by `configure.py` (regenerated by `--full`, nothing to edit by hand)
+| Place | Holds |
+|---|---|
+| `Caddyfile` | every site block — the source of truth `configure.py` reads back (`# vanguard:<role>` markers) |
+| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `DIAG_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT`; during a move `ALT_BASE` (every site also answers under it) and `REDIRECT_BASE` (old names 308 to the new) |
+| `mediamtx.yml` | `webrtcAdditionalHosts: [MAIN_HOST]` (needs a MediaMTX restart to take effect) |
+| `docker-compose.yml` → container env | Jellyfin `PublishedServerUrl` (WATCH), Gitea `DOMAIN/ROOT_URL/SSH_DOMAIN` (GIT), bot `WEB_PUBLIC_URL` (BOT), MediaMTX `LIVE_HOST` (text of the "is live" post), act_runner `GITEA_INSTANCE_URL` (GIT), Kibana `server.publicBaseUrl` via `DIAG_HOST` (DIAG). `compose up -d` in the apply recreates the affected containers (seconds each). MediaMTX is on the host network and is deliberately **not** recreated by the apply |
+| Authentik, by `configure.py --apply` (`--ak-branding` alone) | the login **redirect stage** target `https://AUTH/source/oauth/login/<discord-slug>/` (members never see an Authentik page), the brand's default application (the proxy provider whose external host is MAIN), logo/favicon/background paths (relative, no host). Re-run after an **auth** or **main** change |
+| Kibana landing dashboard, by `configure.py --apply` (`KEY=… kibana-setup.py landing` alone) | the Vanguard tile's links: LIVE, FILES, WATCH, MAIN, GIT, BOT, SYNC from `.env`, AUTH derived as `auth.<domain of MAIN>`; `/streaming`, `/disk` relative. Re-run after any change |
+| Caddy TLS | certificates for new names are fetched automatically; old ones linger harmlessly |
+
+### Pages in the repo that derive sibling hosts from the browser's location (label names hard-coded)
+| File | Assumes |
+|---|---|
+| `about/index.html` | `files.`, `clips.`, `sync.`, `live.`, `auth.`, `watch.`, `git.`, `bot.`, `diag.` + the parent of the current host (`sub('<label>')`); `/onboarding/` relative |
+| `dock/index.html` | `auth.` (sign-out), ingest host prefilled from the current host (`live.`; any name of the box works for RTMP/SRT); `/dump`, `/me/*`, `/time` relative |
+| `live/` portal, `streaming/index.html`, `streaming/disk.html`, `about/onboarding/` | nothing: every call is same-origin (`/webrtc/`, `/hls/`, `/me/*`, `/streaming/api/*`) |
+| `switchToClone.sh`, `deployLive.sh` | defaults for `git.dumpstervigilantes.rip` / `media.dumpstervigilantes.rip` (override by env/arg, or edit) |
+| `README.md`, `FRESH-INSTALL.md` | examples only |
+Renaming a *label* (e.g. `watch` → `video`) means editing the first two; changing only the domain does not.
+
+### Outside the repo — must be changed by hand
+| System | Setting | Which host |
+|---|---|---|
+| DNS | A record + CNAMEs | all |
+| Discord developer portal | OAuth2 redirect `https://AUTH/source/oauth/callback/discord/` | auth |
+| Authentik | Embedded Outpost `authentik_host` | auth |
+| Authentik | `Vanguard sites` proxy provider: authentication URL (auth) and **cookie domain** (base) | auth, base |
+| Authentik | `Jellyfin` provider redirect URI `https://WATCH/sso/OID/redirect/authentik` | watch |
+| Authentik | `Gitea` provider redirect URI `https://GIT/user/oauth2/authentik/callback` | git |
+| Jellyfin | SSO-Auth plugin: OID endpoint `https://AUTH/application/o/jellyfin/` | auth |
+| Jellyfin | Branding login disclaimer (the form action contains the watch host) | watch |
+| Jellyfin | Vanguard Metadata plugin: Public URL, Authentik URL | watch, auth |
+| Gitea | Authentication source `authentik`: discovery URL `https://AUTH/application/o/gitea/.well-known/openid-configuration` | auth |
+| Gitea runner | registered with the instance URL → `./registerGiteaRunner.sh` again | git |
+| Server clone | `git remote set-url origin ssh://git@GIT:2222/techy/Media-Server-Setup.git`; `~/.ssh/config.vanguard` Host line | git |
+| Dev box | `git remote set-url origin …` | git |
+| Gitea, the bot repo | nothing for a rename; `VANGUARD_DEPLOY_HOST` only if it was set to an address | — |
+| Streamers' OBS | dock URL (Custom Browser Docks), dock Settings → ingest host (re-save), clock browser source URL (dock → Timecode band → "Add band + clock source" again), any manual RTMP/SRT configs (copy from the dock) | live |
+| Jellyfin apps on phones/TVs | server address | watch |
+| Bookmarks, Discord pinned links | main (gateway), clips, files, watch, live | all |
+Not affected: Syncthing pairings (device IDs), MariaDB, stream tokens, the Gitea webhook (gateway IP), Authentik
+users/groups/policies, the alert pipeline (Discord webhook + loopback Elasticsearch), the deadman, the publish block
+list and the WebRTC session history (`.markers/`), MediaMTX's host-network ports.
+
+## A. Renaming one subdomain (example: `watch.BASE` → `video.BASE`)
+
+1. DNS: add `CNAME video.BASE → HOST`. Keep the old record until step 7.
+2. Server, between streams: `./configure.py --full` → Enter keeps every value; type the new name at the
+   "Jellyfin hostname" prompt; `y`. Caddy reloads (new block, old gone), Jellyfin is recreated with the new
+   published URL. `curl -sI https://video.BASE/ | head -1` → 200/302 once the certificate is issued.
+   (Only the *main* host touches `mediamtx.yml`; other hosts need no MediaMTX restart.)
+3. Jellyfin first: Dashboard → General → Branding → disclaimer form action → new host → Save.
+   Dashboard → Plugins → Vanguard Metadata → Public URL → new host → Save. (`https://video.BASE/` now loads and
+   fails only at the SSO step: expected until step 4.)
+4. Authentik last: Applications → Providers → `Jellyfin` → Edit → redirect URI → the new host → Update.
+5. If the *label* changed (`watch`→`video`), edit `about/index.html` (`sub('watch')` → `sub('video')`) — commit, push,
+   `sudo vanguard-deploy pull`. (`.env APEX_REDIRECT` too if the bare domain pointed at it: `./configure.py --full`.)
+6. Private window: `https://video.BASE/` → Discord → in. The gateway `https://HOST/` links to the new name.
+7. Remove the old DNS record. Tell members with Jellyfin apps to update the server address.
+
+Per-host extras when it is a different host:
+- **auth**, in this order: Jellyfin SSO plugin OID endpoint and Vanguard plugin Authentik URL; Gitea auth source
+  discovery URL; `dock/index.html` + `about/index.html` if the label changed; THEN Authentik's outpost
+  `authentik_host` and the proxy provider's authentication URL (log in there with the admin card); THEN the
+  Discord portal redirect. Everyone re-logs in.
+- **git**: Authentik Gitea provider redirect; `./registerGiteaRunner.sh`; server clone remote + `~/.ssh/config.vanguard`;
+  dev remote; `switchToClone.sh` default.
+- **live**, **bot**: only the compose env (automatic) and `about/index.html` if the label changed. Streamers who use
+  Go Live need nothing (same-origin); OBS users re-add the dock URL (the dock lives on `live`).
+- **diag**: compose env → Kibana's `publicBaseUrl` (automatic, Kibana recreated), the Kibana landing tile
+  (automatic in the apply), `DIAG_REQUIRE_GROUP` in `.env` if the gate group should differ, `about/index.html`
+  if the label changed. Alarm rules and the Discord alert bus carry no hostname.
+- **auth**, in addition to the list above: `configure.py --ak-branding` (the apply does it) so the Discord-forward
+  redirect stage points at the new auth host — until then every login lands on the old name; the admin card
+  moves to `https://NEW-AUTH/if/flow/default-authentication-flow/?admin=1` (the Kibana tile link follows).
+- **main (HOST)**: `mediamtx.yml` changes → `docker compose restart mediamtx` between streams; every streamer
+  re-adds the dock URL, re-saves Settings (ingest host), and re-runs "Add band + clock source"; manual RTMP/SRT
+  configs are re-copied from the dock; `deployLive.sh` default host.
+
+## B. Changing the base domain (everything at once)
+
+No maintenance window: every site answers on **both** bases first (`--alias`), so the old names keep working for
+members while the apps and Authentik are moved one by one; the flip (`--base`) is the last server step and turns the
+old names into redirects. Order follows the rule above: servers, then the apps, then Authentik, then Discord, then
+the flip. `OLD` = the current base, `NEW` = the new one. Members' sessions survive until step 6; from step 6 on
+they log in once on the new names.
+
+1. **DNS on NEW** (nothing else until this resolves): `A media.NEW` or CNAMEs for `media`, `files`, `clips`, `sync`,
+   `live`, `diag`, `auth`, `watch`, `git`, `bot` → the box. Leave every OLD record in place. Verify:
+   `for h in media files clips sync live diag auth watch git bot; do printf '%-6s ' $h; getent hosts $h.NEW | awk '{print $1}' | tail -1; done`
+   — ten identical addresses. No bare-domain record unless it really points at the box.
+2. **Both bases live:** `./configure.py --alias NEW`. Only the Caddyfile changes (`media.OLD, media.NEW {` …) and
+   Caddy reloads; certificates for the new names follow within a minute. Nothing else moves. Check:
+   `curl -sI https://auth.NEW/ | head -1` → 302, and the old names unchanged. `sudo vghosts-check.py` now also
+   probes every alias.
+3. **Discord developer portal** → OAuth2 → Redirects → **add** `https://auth.NEW/source/oauth/callback/discord/`
+   beside the old one.
+4. **Jellyfin** (any name, SSO still works): SSO-Auth `authentik` provider OID endpoint
+   `https://auth.NEW/application/o/jellyfin/` — works at once, Authentik answers on both names; Branding login
+   disclaimer `https://watch.NEW/sso/OID/start/authentik`; Vanguard Metadata Public URL `https://watch.NEW`,
+   Authentik URL `https://auth.NEW`.
+5. **Gitea**: Site administration → Authentication sources → `authentik` → discovery URL
+   `https://auth.NEW/application/o/gitea/.well-known/openid-configuration` (works at once, same reason).
+   Server clone remote and `~/.ssh/config.vanguard` → `git.NEW` (the host answers already). Dev box remote likewise.
+6. **Authentik, then the flip, back to back, between streams.** This is the only moment anything is unreachable,
+   and it is seconds:
+   `./configure.py --ak-hosts OLD NEW --dry-run` (read it) → `./configure.py --ak-hosts OLD NEW` → immediately
+   `./configure.py --base NEW` → `docker compose restart mediamtx` → `./configure.py --ak-branding` →
+   `./registerGiteaRunner.sh`.
+   `--ak-hosts` moves the outpost host, the proxy provider (external host + cookie domain `NEW`) and the
+   Jellyfin/Gitea redirect URIs: from this instant a login only sticks on the NEW names. `--base` makes NEW the
+   primary everywhere (`.env`, `mediamtx.yml`, Jellyfin/Gitea/runner/bot/Kibana recreated, Kibana tile), drops
+   the alias, and gives every OLD name a block that 308s to its NEW name, so bookmarks and pinned links keep
+   working. Members on an old name get redirected and log in once.
+7. **Test, private window:** `https://auth.NEW/` → Discord → in. `https://media.OLD/` → 308 → `https://media.NEW/`
+   → gate → in. `https://watch.NEW/` SSO round trip. `https://git.NEW/user/login` authentik sign-in.
+   `sudo vanguard-deploy pull`. `sudo vghosts-check.py` → 0 failed (it also checks every OLD name 308s).
+8. **Repo defaults:** `deployLive.sh` (HOST), `switchToClone.sh` (REPO_SSH) → NEW; commit, push, pull.
+9. **People:** OBS streamers re-add the dock at `https://live.NEW/dock/` (the old URL redirects, but OBS docks
+   want the real one), re-save Settings, "Add band + clock source" again; Go Live users just reload. Jellyfin apps
+   → new server address. Pins can stay, they redirect; re-pin at leisure. Remove the OLD Discord redirect entry
+   once everyone has logged in. The OLD DNS records stay as long as the NEW names are CNAMEs onto them.
+
+Rollback before step 6: `./configure.py --alias ""` and revert the two Jellyfin/Gitea screens. Rollback after step 6:
+`./configure.py --ak-hosts NEW OLD` then `./configure.py --base OLD` (which makes the NEW names the redirects).
+
+## C. Splitting the single main host into media / files / clips / sync (done 2026-11-01)
+
+Before: one main host (`ovh1.BASE`) served copyparty at `/`, `/about/`, `/clips/`, `/sync/`, `/dock/`, `/dump`, `/time`.
+After: `media.BASE` = gateway at `/`; `files.BASE` = copyparty; `clips.BASE`; `sync.BASE`; `live.BASE` gains `/dock/`,
+`/dock/clock/`, `/dump`, `/me`, `/time`, `/whoami`. `/time` is only on `live` because the two pages that fetch it
+(the clock source and Go Live) are same-origin there. Authentik needs nothing: the domain-level provider covers every
+name under BASE, the outpost host (`auth.`) is unchanged.
+
+1. DNS: CNAMEs `media`, `files`, `clips`, `sync` → the box (keep `ovh1` for ssh; it is no longer a web host).
+2. Dev box: pull/upload the repo (`configure.py`, `about/`, `addCommunityMediaServer.sh`, docs).
+3. Server, between streams: `./configure.py --full` → main hostname `media.BASE`; Enter for the sub-hostnames
+   (defaults `files.`, `clips.`, `sync.`, …); bare domain redirect → `https://media.BASE/`; `y`.
+   Then `docker compose restart mediamtx` (webrtcAdditionalHosts is now `media.`) and `./deployLive.sh` (portal
+   build picks up nothing new, but confirms `/srv/live` is served on `live.`).
+4. Streamers: OBS → Custom Browser Docks → replace the dock URL with `https://live.BASE/dock/`; dock Settings →
+   save; Timecode band → "Add band + clock source" again (the clock browser source URL moved).
+5. Discord: re-pin `https://media.BASE/` (members), `https://clips.BASE/` (gamers), `https://files.BASE/` (editors).
+6. Old `ovh1` bookmarks stop working (no Caddy block, TLS fails) — point people at the gateway.
+
+## D. Adding a NEW host
+
+**`./configure.py --apply` cannot do this, and will not complain.** It is stateless by design: it reads the
+hostnames back out of the `Caddyfile` (`# vanguard:<role>` markers), so a role that is not in there yet has no
+value to write and the site is simply left out. The containers come up, Caddy reloads, and the new name has no
+site block and therefore no certificate — which presents as an SSL handshake error, not as an error from the
+apply. This cost an hour on 2026-11-03 when `diag.` was added.
+
+Introduce a new host with **`--full`** (asks for every hostname, defaults from the current config) or with
+**`--hosts`**, which is the non-interactive form and takes them positionally:
+
+    ./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG     # "" leaves one unset
+
+After that, `--apply` maintains it like any other host, because the marker is now in the file.
+
+1. DNS: `CNAME` (or `A`) for the new name → the box; check it resolves before continuing.
+2. Add the role to `configure.py`: a `write_caddy` block, a `<ROLE>_REQUIRE_GROUP` default in `write_env`,
+   the prompt in `full()`, the positional in `--hosts`, and the line in `show()`.
+3. `./configure.py --hosts …` with every hostname including the new one.
+4. Verify the marker landed: `grep -c "vanguard:<role>" Caddyfile` → 1. Then `curl -sI https://NEW/` → 302.
+5. Add it to `addCommunityMediaServer.sh` so a rebuild asks for it too.
+
+## E. Validating a change: `sudo vghosts-check.py`
+
+One command on the server checks every place in the tables above against the Caddyfile, the same source of
+truth `configure.py` reads back, and prints `ok` / `FAIL` / `warn` per item plus a `todo` list for the things no
+script can read (the Discord developer portal, streamers' OBS). Exit code 1 if anything failed.
+
+    server$ cd ~/vanguard-stream && sudo vghosts-check.py          # everything
+    dev$    ./bin/vghosts-check.py --repo                           # repo defaults, about-page labels, docs only
+    server$ sudo vghosts-check.py --json                            # for a script
+
+**The order is the order of the change, and it matters.** The sections run top-down because each one depends
+on the one above it, so a failure early on is usually the *cause* of the failures below it; fix the first
+failure and re-run rather than chasing the later ones:
+
+| # | Section | Depends on | Why this position |
+|---|---|---|---|
+| 1 | DNS | — | nothing gets a certificate or a visitor until the name resolves to the box |
+| 2 | Caddyfile | 1 | the roles and names `--full`/`--hosts` wrote; Caddy fetches certificates from here |
+| 3 | `.env` | 2 | the same names as compose variables; a mismatch is a half-finished `--full` |
+| 4 | `mediamtx.yml` | 3 | `webrtcAdditionalHosts` = main host; only a MediaMTX restart (between streams) applies it |
+| 5 | containers | 3 | the env each container was *created* with: Jellyfin, Gitea, runner, bot, MediaMTX, Kibana |
+| 6 | HTTPS | 1, 2 | every name answers with a valid certificate; gated names bounce to the auth host |
+| 7 | Jellyfin | 5, 6 | SSO plugin OID endpoint, branding disclaimer, Vanguard plugin URLs; live SSO start |
+| 8 | Gitea | 5, 6 | `app.ini`, the auth source (live redirect), the runner's registration, the server clone's remote |
+| 9 | Authentik | 7, 8 | **last of the services**: outpost host, proxy provider external host + cookie domain, Jellyfin/Gitea redirect URIs — changing these before the apps is the one way to lock yourself out |
+| 10 | Discord portal | 9 | the OAuth2 redirect is printed exactly; Discord login is broken until it matches; akadmin's card still works |
+| 11 | repo | 2 | `deployLive.sh`/`switchToClone.sh` defaults, about-page labels = Caddy roles, no browser links to `ovh1` in the docs |
+| 12 | streamers | 2 | OBS dock URL, clock source, Jellyfin apps, Discord pins: listed, never checked |
+
+What it knows that a human forgets: a container keeps the environment it was created with, so step 5 catches a
+`.env` that changed without `docker compose up -d <service>`; Authentik's cookie domain decides whether one
+login spans every host; and `ovh1.BASE` must resolve (ssh) but must *not* answer https.
+
+By hand, for the round trips the script only probes:
+    https://HOST/whoami            → your identity (gate works on the new name)
+    https://watch.BASE/sso/OID/start/authentik   → Jellyfin SSO round trip
+    https://git.BASE/user/login    → "Sign in with authentik" round trip
+    dock → Go Live → journalctl -u dumpd | grep mtx-auth   → allow publish
+    live.BASE → a window appears for the test stream, chip "webrtc"

LOGGING.md

@@ -0,0 +1,356 @@
+# Logging framework
+
+Decided 2026-11-01. The store was chosen, built and replaced on 2026-11-03: a VictoriaLogs stack went in
+first and was torn out the same day for producing no dashboards, so the store is now **Elasticsearch with
+Kibana**, served at the root of the `diag.BASE` host (Phase 13 in `CommunityDiscordMedia.md`). Logs arrive via
+Filebeat, metrics via Metricbeat, into the same cluster so one dashboard can carry both. Everything upstream
+of the store was fixed first, so the swap was a collector config rather than a code change.
+
+## Where the lines come from (the only two taps a collector needs)
+
+| Tap | Holds | Read with |
+|---|---|---|
+| **journald** | our systemd units: `dumpd`, `normalize-sweep`, `trash-purge`, `deploy-pending`, `vgdeband` (transient), `deploy.sh` output | `journalctl -u <unit>`; collectors: Alloy `loki.source.journal`, Vector `journald`, VictoriaLogs journald ingestion, Netdata's journal plugin |
+| **Docker json-file logs** | every container (`docker-compose.yml`), rotated 5 × 50 MB each | `docker logs <name>`; collectors read `/var/lib/docker/containers/*/*-json.log` and get the container **labels** below for free |
+
+Filebeat reads both: a docker autodiscover template per container (filestream + the container parser) and a
+journald input for our units. Metrics are the same story: see the taps table below, all on loopback.
+
+## Metrics taps (Prometheus text format, loopback only — Metricbeat scrapes them)
+
+| endpoint | what | from |
+|---|---|---|
+| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets. `remoteAddr` labels are real client addresses now that MediaMTX runs on the host network; before, every publisher showed as the bridge gateway `172.18.0.1` | MediaMTX `metrics: yes` |
+| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
+| `http://127.0.0.1:8080/metrics` | ours: the normalise queue, encode time, live stream count | `bin/dumpd.py` |
+
+Host and per-container metrics (CPU, RAM, disk, network, block I/O, restarts) come from Metricbeat's `system`
+and `docker` modules, not from an exporter — node_exporter and cAdvisor were removed when the Elastic stack
+went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of
+them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher).
+
+### Stuck recordings: counts in Kibana, names in the logs, decisions on the host
+
+Three layers on purpose, because each answers a different question and none of them needs a new web page.
+
+- **How bad is it, and is it getting worse** — the gauges: `vanguard_media_pending`,
+  `vanguard_media_pending_oldest_seconds`, `vanguard_media_failed_files`. A Lens dashboard over
+  `metricbeat-*` shows the trend across a night, which is the thing a list never can.
+- **Stitching** — `event=stitched stream= parts= joins= missing_ms= seconds= out= trash=` when a stream's reconnect
+  fragments were joined; `event=stitch_wait` (stream still going), `event=stitch_skipped reason=too_many`
+  (500+ files: use vgstuck.sh), `event=stitch_break reason=gap|params_changed` (chain split there),
+  `event=stitch_failed` (fragments left alone, published one by one), `event=defer` (a fragment left alone
+  because a sibling is still being written).
+- **Stop stream** — dumpd `stream_stop stream= kind= by= minutes= result=kicked|kick failed…`, then `publish_block user= minutes= by=`
+  / `publish_unblock`, and every refused reconnect as `publish_deny reason=blocked by=`.
+- **Wipe** — `component=vgstuck event=wipe_done recordings= stale_files= trash=` when an admin pressed Bin all
+  unprocessed on the Dump tab (dumpd logs `stuck_wipe by=`), with one `event=trashed file=` line per file before it.
+- **Which recordings** — the sweep logs `event=parked file= stream= log=` once per run for every file a
+  failure log is holding back, plus `event=failed` with ffmpeg's own stderr when it first breaks. Both land
+  in `filebeat-*`, so a Discover search or a dashboard table names them and stays current. Names deliberately
+  do NOT come from the metrics: a label per filename would wreck index cardinality for no gain.
+- **What to do about it** — `bin/vgstuck.sh` on the host. It classifies every unpublished recording as
+  `parked` (a failure log is blocking it, with the reason), `published` (an output already exists),
+  `waiting` (nothing blocking it yet) or `writing` (still recording), then `retry <match>` clears the
+  blocker so the sweep tries again and `drop <match>` moves it to `trash/<today>/` for the same three-day
+  undo as a dumped night. `drop --all` is refused.
+
+A leftover `<output>.mp4.part` does **not** block anything — ffmpeg overwrites it with `-y`. It only wastes
+disk, and `retry` clears it.
+
+### Our own metrics, and how to add one
+
+MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising
+keeping up", because the sweep is a shell script on a timer. So the scripts keep counters and dumpd renders
+them.
+
+- `bin/vgmetrics.sh` is sourced by `normalize-sweep.sh` and `vgdeband.sh`. It accumulates `key value` lines in
+  `/srv/media/.markers/metrics/<component>.counters` and keeps a `<component>.pid` file for as long as the
+  script is executing. It renders nothing and serves nothing, so the scripts stay simple.
+- `bin/dumpd.py` turns those files into Prometheus text on `GET /metrics`, adds gauges it can compute at
+  scrape time, and asks the MediaMTX API for the live path list. `COUNTER_MAP` there is the single place that
+  knows the metric names.
+
+| metric | type | what it answers |
+|---|---|---|
+| `vanguard_media_pending{kind}` | gauge | finished recordings waiting to be normalised. **The capacity signal**: if this trends up across a night, CPU encoding is not keeping pace |
+| `vanguard_media_pending_oldest_seconds` | gauge | how long the oldest one has waited |
+| `vanguard_media_recording` | gauge | recordings MediaMTX is still writing (age under `MINS_IDLE`) |
+| `vanguard_media_encoding_in_flight` | gauge | ffmpeg output files (`*.part`) open right now |
+| `vanguard_media_running{component}` | gauge | 1 while the sweep or vgdeband is executing, by live pid, not a stale marker |
+| `vanguard_media_jobs_total{component,verdict,outcome,kind}` | counter | remuxes and encodes finished, split by success and by stream vs clip |
+| `vanguard_media_job_seconds_total{component,verdict,kind}` | counter | seconds spent in ffmpeg. Divide by the job count for mean encode time |
+| `vanguard_media_runs_total{component}` | counter | script starts. Flat means the timer stopped |
+| `vanguard_media_deband_total{component}` | counter | timecode bands cropped off |
+| `vanguard_media_last_finish_timestamp_seconds{component}` | gauge | when it last finished |
+| `vanguard_media_last_run_seconds{component}` | gauge | how long that run took |
+| `vanguard_media_failed_files` | gauge | files parked with a `normalize-failed.log` |
+| `vanguard_mediamtx_up` | gauge | did the MediaMTX API answer the scrape |
+| `vanguard_streams_active` | gauge | streams publishing now, derived `.webrtc` mirrors excluded |
+| `vanguard_streams_mirrors` | gauge | those mirrors, counted separately |
+| `vanguard_stream_readers` | gauge | viewers attached across all paths |
+
+To add a counter: pick a key of the form `name:label:label`, call `vgmet_add` with it, and add `name` to
+`COUNTER_MAP` with the label names in the same order. Nothing else changes, and a key dumpd does not
+recognise is ignored rather than breaking the scrape.
+
+Three rules this design exists to keep. Metrics must never break a sweep, so every write is atomic and every
+failure in the helper is silent, and a missing helper degrades to no-ops. A scrape must never disturb the
+stack, so dumpd never takes the sweep's flock and the filesystem globs are depth-limited rather than walking
+the media tree. And the counters live on disk rather than in the process, because the sweep exits between
+runs.
+
+Note for anyone editing `normalize-sweep.sh`: its file loop reads from a process substitution, not a pipe. A
+pipe would put the loop in a subshell and every counter set inside it would be discarded on exit.
+
+## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
+
+`vanguard.service` = the service name, `vanguard.category` = one of the categories below. A collector turns these
+into stream labels without parsing a single line.
+
+## Categories = the questions people ask
+
+| category | services / components | correlation key |
+|---|---|---|
+| `stream` | mediamtx (path ready/closed, recorder, WebRTC sessions, the on-ready hook's `event=mirror`), dumpd `publish_allow` / `publish_deny`, sweep `event=published` | `stream` = `user/Title` |
+| `auth` | caddy access logs (status, `X-Authentik-Username`), authentik-server/worker events, gitea OIDC | `user` |
+| `media` | sweep verdicts / timecode / deband / failed, vgdeband, jellyfin, copyparty | `file` |
+| `intake` | syncthing (`folder.id`), dumpd `clips_setup`, `quota_*` | `folder` = `clips-<user>` |
+| `infra` | mariadb, caddy TLS/ACME, container restarts, disk | host only |
+| `deploy` | deploy.sh, dumpd `deploy_*`, `bot_deploy_*` and `bot_env_*` (the bot's CI posting a release bundle and its settings), act_runner | `rev` |
+| `code`, `bot` | gitea, discord-bot | — |
+
+## Labels vs fields
+
+- **Labels** (low cardinality, what you filter by): `service`, `category`, `level`, `unit`. A dozen values each.
+- **Fields** (parsed at query time, what you search in): `stream`, `user`, `file`, `folder`, `session`, `status`, `ip`, `event`.
+  Thousands of values — fine as fields, ruinous as labels (Loki). VictoriaLogs indexes every word anyway.
+
+## Our own line format: logfmt, one event per line
+
+    ts=2026-11-01T19:52:14-06:00 level=info component=sweep event=published stream="techy2493/Vanguard_Training" file="/srv/media/streams/…" verdict=encode
+
+- `ts` ISO-8601 with offset · `level` info|warn|error · `component` dumpd|sweep|vgdeband|deploy|on-ready · `event` a verb/noun in snake_case · then `key=value`, values with spaces or quotes double-quoted.
+- Same key means the same thing everywhere: `stream` (user/Title), `user`, `file`, `folder`, `rev`, `ip`, `reason`.
+- Implemented by: `log()` in `bin/dumpd.py`, `vglog` in `bin/normalize-sweep.sh` and `bin/vgdeband.sh`, `log()` in `deploy.sh` (`msg=` only), `bin/mtx-on-ready.sh` (MediaMTX prefixes its own path/time).
+
+Events today: dumpd `publish_allow publish_deny dump clips_setup quota_paused quota_resumed quota_* deploy_* bot_deploy_start bot_deploy_exit bot_deploy_bad_signature bot_deploy_refused bot_env_applied bot_env_failed`;
+sweep `verdict timecode deband fill_gaps published done failed skip`; vgdeband `deband done failed skip`; hook `mirror`.
+A future alert script watches `level=error`, `event=failed`, `event=publish_deny` bursts, `event=quota_paused`, plus disk.
+
+## Third-party formats a collector must parse
+
+| source | format | stage |
+|---|---|---|
+| caddy | JSON (`logger`, `level`, `msg`; access logs: `request.*`, `status`, `request.headers.X-Authentik-Username`) | json |
+| authentik | JSON (`event`, `user`, `action`) | json |
+| syncthing | key=value after a `date time:` prefix (`folder.id=… log.pkg=model`) | logfmt |
+| mediamtx | `YYYY/MM/DD HH:MM:SS LVL [component] [session|path x] message` | one regex → component, id, msg |
+| jellyfin | `[HH:MM:SS.mmm] [LVL] [thread] Logger.Class: message` + stack-trace continuation lines | **multiline** (join lines not starting with `[`) then regex |
+| gitea, mariadb, copyparty | plain text | level regex only |
+
+## Caddy access logs
+
+On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie`, `Authorization` and
+`X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the
+"who downloaded what" audit trail for the files host.
+
+## Alerting: Kibana evaluates, we deliver
+
+A basic Elastic licence allows only two connector types, `.index` and `.server-log`. Everything that leaves
+the stack — webhook, Slack, email, PagerDuty — is Gold or above and refuses with
+`403 Action type .webhook is disabled because your basic license does not support it`. So Kibana cannot talk
+to Discord, and the pipeline is split:
+
+    Kibana rule  --.index connector-->  vanguard-alerts  --vanguard-alerts.timer-->  Discord webhook
+
+- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index"), the heartbeat rule,
+  the data view, and the "Vanguard ops" landing dashboard plus the `defaultRoute` that makes `diag.BASE`
+  open on it. Run it once, *after* Elasticsearch security is on. Name a step to run only that one:
+  `kibana-setup.py landing`.
+  Kibana deliberately stays at the **root** of that host. Moving it under `/kibana` would mean
+  `server.basePath` plus `rewriteBasePath`, a Caddy handle that does not strip the prefix, and appending the
+  prefix in both Beats' `setup.kibana`, this script, the alarm script and the deadman's status check — and a
+  miss in the Beats one crash-loops Metricbeat, because it exits when Kibana is unreachable. `defaultRoute`
+  buys the landing page without any of that.
+- **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind`
+  as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused
+  them, and replayable after the fact.
+- **`bin/vanguard-alerts.sh`** runs every minute, delivers anything newer than its cursor, and suppresses a
+  repeat of the same rule within 15 minutes as a safety net. No webhook configured means journal-only and a
+  non-zero exit, never a silent drop.
+- **`bin/vanguard-deadman.sh`** watches both halves: a `kind=heartbeat` document arriving (Kibana still
+  evaluating and writing) and `alerts.state` being fresh (something still delivering). Either half dying
+  means no alarm can arrive, and nothing inside the stack could tell you.
+
+### The rules themselves
+
+`observability/kibana-alarms.py` creates them, so they are reviewable and reproducible rather than clicked in.
+All of them repeat every **12 hours** while still true (`notify_when: onThrottleInterval`) and go silent when
+the condition clears.
+
+| Rule | Level | Fires when |
+|---|---|---|
+| Disk free below 15% | error | any real filesystem at ≥85% used. Pseudo filesystems are excluded by type, or every tmpfs would alarm forever |
+| Memory free below 15% | warn | `system.memory.actual.used.pct` ≥ 0.85 |
+| Discord bot not running | error | **no** metrics from the `discord-bot` container for 5 min (the one rule with a `<` comparator) |
+| Discord bot unhealthy | error | the container healthcheck reports `unhealthy`, i.e. the bot's own `/health` says Discord is disconnected or the database does not answer |
+| Recordings stuck waiting to be normalised | warn | `vanguard_media_pending_oldest_seconds` > 1 h |
+| Normalise failures parked | warn | any file has a `normalize-failed.log` |
+
+The two bot rules answer different questions and both are needed. "Not running" catches a stopped or
+hard-crash-looping container. "Unhealthy" catches the worse case: the process is up, the port is open, and the
+bot is doing nothing because its gateway or database is gone. That second one only works because the bot's
+`/health` is exempt from its own identity middleware and returns 503 rather than 200 when degraded — a
+healthcheck carries no Authentik headers, and a 200 with `discord_ready: false` would have made a dead bot
+look healthy.
+
+Elasticsearch, Kibana, Beat liveness and the disks are **not** here: that is the deadman's job, outside the
+stack. Nothing that depends on Kibana can report that Kibana is dead, so duplicating those as rules would go
+quiet in exactly the case that matters.
+
+To add an alarm: author the rule in Kibana, point its action at the "Vanguard alert index" connector, and give
+the document a `message` and a `level` of `warn` or `error`. Never set `kind: heartbeat` — that value is how
+the delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
+for real alarms; only the heartbeat should fire every interval.
+
+## Retention
+
+Set by `configure.py --apply`, so nothing here needs clicking and nothing grows without bound:
+
+| What | How long | Set by |
+|---|---|---|
+| `filebeat-*`, `metricbeat-*` | `LOG_RETENTION_DAYS`, default 30 | a **delete phase injected into the Beats' own ILM policies**. They ship a rollover but no delete, so indices otherwise grow until the disk watermark turns them read-only — which the deadman reports but cannot undo |
+| `vanguard-alerts` | `ALERT_RETENTION_DAYS`, default 90 | `vanguard-alerts.sh`, a `_delete_by_query` at most once every 6 h. A heartbeat document arrives every minute, so this index needs pruning more than it needs ILM |
+
+The delete phase is *injected* rather than written whole, so whatever rollover the Beats chose is preserved
+across upgrades. Keeping `auth` and `deploy` lines longer than the rest would mean separate indices and a
+second ILM policy; not worth it unless an audit asks.
+
+## As built (Phase 13)
+
+| container | port | job |
+|---|---|---|
+| `elasticsearch` | 9200 | one store for **both** logs and metrics. 2 GB heap, single node, security off (loopback only) |
+| `kibana` | 5601 | the single pane: Discover for logs, dashboards, alerting. Served at the root of `diag.BASE` |
+| `filebeat` | — | container logs (autodiscover) + our journal units |
+| `metricbeat` | — | host + Docker metrics (retires node_exporter and cAdvisor) + MediaMTX and Caddy Prometheus endpoints |
+
+**Logs and metrics correlate because both Beats write the same fields into the same store.** That is what makes
+one dashboard able to show a CPU spike next to the errors that caused it, rather than two dashboards side by
+side:
+
+| field | set by | use |
+|---|---|---|
+| `@timestamp` | both (ECS) | every panel shares the dashboard's time range |
+| `host.name` | both | one box today, but the field is there |
+| `container.name` | both | per-service metrics joined to that service's log lines |
+| `service.name` | both, from the `vanguard.service` label | the name you actually think in |
+| `vanguard.category` | both, from the label | filter a whole area at once: stream, media, auth, intake, deploy |
+
+To get a dashboard that filters across both, make **one data view** spanning `filebeat-*,metricbeat-*` in
+Kibana (Stack Management → Data Views). A single filter such as `service.name: mediamtx` then applies to the
+metric panels and the log panel together. Metricbeat also ships its own system and Docker dashboards, which are
+a good starting point to copy panels from.
+
+Retention is one edit in Kibana rather than config: Stack Management → Index Lifecycle Policies → the beats
+policy → add a delete phase. Doing it there rather than in a file keeps it adjustable without a redeploy.
+
+### Old notes
+
+
+(superseded 2026-11-03 — the Victoria/vmalert/Alertmanager shape, kept for the reasoning)
+
+| container | port | job |
+|---|---|---|
+| `victorialogs` | 9428 | store, 30 d, `-http.pathPrefix=/logs` so Caddy passes the prefix through unchanged |
+| `vector` | — | `docker_logs` + `journald` → VictoriaLogs (elasticsearch bulk). **Holds the Docker socket** |
+| `victoriametrics` | 8428 | 90 d, scrapes the taps above (`observability/scrape.yml`) |
+| `vmalert-metrics` | 8880 | `observability/alerts-metrics.yml` against VictoriaMetrics |
+| `vmalert-logs` | 8881 | `observability/alerts-logs.yml` (LogsQL) against VictoriaLogs' stats endpoints |
+| `alertmanager` | 9093 | groups, inhibits, posts to Discord (`webhook_url_file`, never in the config) |
+
+Published only through `diag.BASE`, gated on `DIAG_REQUIRE_GROUP` (`media-admins`): `/logs`, `/metrics-db`,
+`/alerts`, `/rules-metrics`, `/rules-logs`, and an index at `/`.
+
+**The deadman** (`bin/vanguard-deadman.sh`, host timer, 5 min) is deliberately outside all of it: curl only, it
+checks each component answers, that the always-firing `Watchdog` alert is still arriving through
+vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. It speaks on change, announces recovery once, and
+with no webhook configured still logs and exits non-zero so `systemctl is-failed vanguard-deadman.service` is
+a signal. Nothing inside the stack can report its own death; this can.
+
+`ALERT_WEBHOOK_URL` in `.env` → `/etc/vanguard/alert-webhook` (0640 root:65534). Empty means log-only, which is
+a working configuration, not an error: `configure.py` generates `observability/alertmanager.yml` with the
+Discord notifier omitted, so nothing is half-configured and nothing fails to load.
+
+## Gotchas this build produced (all fixed; here so they are not rediscovered)
+
+- **`-http.pathPrefix` applies to EVERY endpoint, not just the UI.** With VictoriaLogs behind `/logs`, the
+  ingestion path is `/logs/insert/elasticsearch/`, the health probe is `/logs/health` and the scrape is
+  `/logs/metrics`. It rejects a write to the unprefixed path with a message naming the missing flag.
+- **Every proxied UI needs its own prefix set**, or it loads at the sub-path and then fetches its assets from
+  the domain root and renders broken. VictoriaMetrics and vmalert use `-http.pathPrefix`; Alertmanager uses
+  `--web.route-prefix` plus `--web.external-url`. Setting those also moves their metrics and health paths.
+- **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped;
+  earlier entries are still on the host for `journalctl --boot=-1`.
+- **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a
+  file and mount it, the way the gateway and dock pages are served, instead of inlining it. (The diag host no
+  longer has such a page at all: Kibana occupies its root.)
+- **Filebeat 9 removed the `container` input.** It is not deprecated-but-working: the config is refused per
+  container with "Container input is deprecated. Use Filestream input with its container parser instead",
+  and Filebeat then runs happily with zero harvesters and creates no index at all. Container logs go through
+  `type: filestream` with `parsers: [container: ~]`, and every filestream input needs its own stable `id`.
+  This is exactly the failure the deadman's "up but not shipping" check exists to catch.
+- **`add_docker_metadata` should be told where the container id is.** Its path-sniffing default did not
+  apply to our filestream paths, so the autodiscover template stamps `container.id` and the processor reads
+  it via `match_fields: ["container.id"]`. Without the id there are no `container.labels.*`, and without
+  those the `service.name` / `vanguard.category` promotion silently produces nothing.
+- **A single node cannot place a replica**, so any beats index defaulting to one replica holds the cluster at
+  yellow permanently. Both beats set `setup.template.settings.index.number_of_replicas: 0` with
+  `setup.template.overwrite: true`; indices created before that need a one-time `_settings` call.
+- **Kibana's Observability apps that check privileges cannot work with `xpack.security.enabled: false`.** The
+  Infrastructure / Metrics app calls `api/infra/host/count` on every view and Elasticsearch answers
+  `no handler found for uri [/_security/user/_has_privileges]`, surfaced as a red "error while fetching
+  resource" banner. The fix is to turn security ON, not to avoid the app. Doing so normally adds a password
+  prompt, which this stack forbids, so Kibana's **anonymous provider** signs every visitor in as one native
+  user instead (`observability/kibana.yml`). Security being on is also what any future read-only tier needs.
+- **Moving a setting from a container environment variable into a config file can break it.** `SERVER_BASEPATH: ""`
+  was fine for years because the Kibana image drops empty env values, but `server.basePath: ""` in
+  `kibana.yml` is a validation error (`must start with a slash, don't end with one`) and Kibana refuses to
+  boot. When migrating env settings into a file, an empty value means "omit the key", not "set it to empty".
+- **Metricbeat exits when Kibana is unreachable**, because `setup.dashboards.enabled: true` runs at startup.
+  A Metricbeat restart loop is often a symptom of a Kibana problem, not a Metricbeat one.
+- **Alerting rules created while security was off hold no API key** and stop working once it is on, until each
+  rule is disabled and re-enabled. Create connectors and rules *after* security is on, not before.
+- **Kibana's Observability charts group by `event.dataset`.** Beats do not set it, so without a processor
+  every line lands in one "unknown" bucket and the Logs rate chart is useless. Filebeat now fills it from the
+  container's `vanguard.service` label, or from `systemd.unit` for journald lines. `copy_fields` never
+  overwrites an existing target, so listing several sources for one field means first match wins.
+- **Metricbeat field shapes differ per module on purpose.** The MediaMTX and Caddy scrapes are untyped, so a
+  value is at `prometheus.metrics.<name>` with labels at `prometheus.labels.*`. The dumpd scrape sets
+  `use_types` and `rate_counters`, so a gauge is at `prometheus.<name>.value` and a counter has both
+  `.counter` and `.rate`. Look for the right one before concluding a metric is missing.
+- **`-r` and a non-CFR `-fps_mode` are mutually exclusive in ffmpeg.** It refuses the pair with
+  "One of -r/-fpsmax was specified together a non-CFR -vsync/-fps_mode" and then fails with
+  `Error opening output files: Invalid argument` — which looks like a path or permission problem and is not.
+  Capping the frame rate means re-timing, so the cap forces CFR and passthrough has to lose.
+- **`avg_frame_rate` alone is not a frame rate.** A looped or fragmented recording reported 95 fps for a
+  30 fps source (`r_frame_rate=30/1`, `avg_frame_rate=3625/38`), which sent it down the fps-cap path and broke
+  the encode. `r_frame_rate` alone is no better: it is a huge nominal value on genuine VFR. The sweep takes
+  the **lower of the two**.
+- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
+
+## Wiring sketches (the options that were weighed, kept so the decision is not re-argued)
+
+- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
+  labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
+  Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
+- **VictoriaLogs**: single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
+  ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE`
+  (`media-admins`). VictoriaMetrics beside it would consume the four metric taps.
+- **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;
+  behind the Caddy gate. Weakest at log search.
+- **ELK** (chosen, and what is deployed): Elasticsearch with a pinned 2 GB heap, Kibana at the root of
+  `diag.BASE`, Filebeat and Metricbeat shipping into the same cluster so one dashboard can carry logs and
+  metrics together. The cost is the JVM heap and the index disk on a box that also runs x264; it was taken
+  because every lighter option needed three or four more pieces to reach a usable dashboard.

README.md

@@ -1,5 +1,29 @@
 # vanguard-stream — SYS-1 (Xeon E-2136, 32 GB ECC, 2×4 TB SATA soft RAID)
 
+> **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included.
+> **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely.
+> **LOGGING.md** = observability: what each service logs, the container labels, the logfmt event format, and the Elastic stack that consumes it (§5h).
+> **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app).
+
+**Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
+
+> Adding a host later needs `./configure.py --full` or `--hosts`, **not** `--apply`: apply reads the hostnames
+> back out of the Caddyfile, so a role that is not in there yet is silently left out and the new name ends up
+> with no certificate. HOSTNAME-CHANGE.md §D has the procedure.
+
+| Host | What | Who |
+|---|---|---|
+| `media.BASE` (main) | gateway page at `/` (role-aware links + how-tos), `/whoami`, `/logout` | everyone |
+| `files.BASE` | copyparty: browse, scrub, download, upload finals | editors / media-admins |
+| `clips.BASE` | self-service clip intake (pair your Syncthing) | any member |
+| `sync.BASE` | Syncthing admin UI | media-admins |
+| `live.BASE` | live portal, Go Live, OBS dock `/dock/`, clock source `/dock/clock/`, `/dump`, `/me`, `/time` | everyone (`LIVE_REQUIRE_GROUP`) |
+| `diag.BASE` | logs, metrics, alerts (§5h) | media-admins |
+| `watch.BASE` | Jellyfin | Discord via OIDC |
+| `git.BASE`, `bot.BASE`, `auth.BASE` | Gitea, bot web UI (`bot-users`), Authentik | — |
+
+`/time` (the server clock the timecode strip syncs to) lives only on `live` because browsers fetch it
+same-origin: both pages that use it, the clock source and Go Live, are on that host.
 > **STATUS.md** = what runs, who does what, what's unfinished (kept current). **CommunityDiscordMedia.md** = design + phases.
 
 Ingest from OBS, browse/scrub/download for editors, automatic normalization to one editing
@@ -45,6 +69,8 @@ works from ~/vanguard-stream without sudo.
     ./configure.py --full      # first time: asks everything, then applies
     ./configure.py             # later: add/remove/reset streamers and editors, then apply
     ./configure.py --show      # print current users and the OBS settings to hand out
+    ./configure.py --set K=V   # change one setting (Discord tokens/webhooks, retention, alarm mentions)
+                               # run it with no arguments to list what is settable
 
 Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates
 `Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates
@@ -80,8 +106,14 @@ Everything else is dropped and a trickle is logged (`journalctl -k | grep vg-dro
 Jellyfin (8096), copyparty, the Syncthing UI and dumpd bind to 127.0.0.1 only and are reached via Caddy
 or an ssh tunnel, so they are deliberately not opened. MariaDB runs on the host network (not a Docker
 published port) precisely so the firewall can govern it.
-Syncthing's admin UI is proxied at `https://HOST/sync/` behind Syncthing's own GUI login (set it in
+Syncthing's admin UI is proxied at `https://sync.BASE/` behind the Authentik gate + group `media-admins`; its
-Actions → Settings → GUI before relying on this). The ssh tunnel to 8384 still works as a fallback.
+own GUI login is disabled (no user/password, `insecureAdminAccess=true`) so Discord is the only login:
+
+    docker exec syncthing syncthing cli --home /var/syncthing/config config gui insecure-admin-access set true
+    docker exec syncthing syncthing cli --home /var/syncthing/config config gui user set ""
+    docker exec syncthing syncthing cli --home /var/syncthing/config config gui password set ""
+
+The ssh tunnel to 8384 still works as an admin fallback (no login at all — ssh is the gate there).
 
 To change rules: edit `/etc/nftables.conf`, then `sudo nft -f /etc/nftables.conf` or
 `sudo systemctl reload nftables`. **Never `systemctl stop|restart nftables`** — Debian's unit flushes
@@ -94,21 +126,70 @@ Being a streamer = being in the Authentik **streamers** group (a Discord role vi
 There are no stream passwords or keys to hand out: the dock mints a **12-hour publish token** for the
 logged-in Discord user each time Go Live is pressed and sets it as the OBS stream key itself.
 
+OBS → Settings → Output → Encoder: **B-frames = 0** (NVENC "Max B-frames: 0"; x264 add `bframes=0`). MediaMTX
+refuses WebRTC playback of H.264 with B-frames ("WebRTC doesn't support H264 streams with B-frames"), so the live
+page falls back to HLS. The dock checks the actual recording a few seconds after Go Live and shows a red banner
+if B-frames are present (and in Simple output mode with x264 it sets `bf=0` for you); the Live tab on `diag.`
+shows the same per publisher. Nothing can set it for you in Advanced output mode, so the live
+portal would fall back to HLS with seconds of delay. Keyframe interval 2 s. Recording quality is unaffected.
 OBS → Tools → WebSocket Server Settings → enable, set a password.
-OBS → Docks → Custom Browser Docks → `https://media.example.com/dock/` → log in with Discord once.
+OBS → Docks → Custom Browser Docks → `https://live.BASE/dock/` → log in with Discord once.
 In the dock's Settings: ingest host (prefilled), OBS WebSocket password/port, optional title prefix/suffix
 (`{date}` → YYYY-MM-DD, e.g. prefix `HelmetCam-` or suffix `-{date}`).
 Per stream: type a title, click **Go Live**. The dock shows the final stream name (`<you>/<Title>`) as you type
-and enforces the naming rules the metadata relies on: letters, digits, `_ . -` only, no double underscore
+and enforces the naming rules the metadata relies on: letters, digits, `_ -` only (no dots: `<title>.webrtc` is reserved for the audio mirror), no double underscore
 (it separates title and timestamp in filenames), no leading dot, max 64 chars.
-After a bad night: **Dump tonight** (dumps the night of the account you are logged in as; `admins`/`media-admins`
+After a bad night: **Dump tonight** in the dock dumps the night of the account you are logged in as, and only
-may dump anyone's).
+that, admin or not. Dumping someone else's night is triage: **Dump a night**, a tab of the streaming page on
+`diag.BASE/streaming#dump` (media-admins): the night's recordings and clips per streamer, a date picker, Dump per
+streamer or Dump everyone, two confirmations. Same endpoint, same 3-day undo; dumpd honours a `user` only on that
+route. The night is 17:00–02:00 Eastern.
 
 Manual OBS/other software: dock → **Manual OBS setup** shows the RTMP server, the stream key and the SRT URL
 with the token hidden (Show / Copy / Fresh token). Tokens expire after 12 h; a running stream is not cut when
 its token expires, but a reconnect needs a fresh one.
 
 Browser streaming without OBS: the live portal's **Go Live** (Windows Chrome 141+ for window audio + GPU encode).
+On a lossy uplink Go Live switches itself to TCP: when the server reports more than 0.3% packet loss over 20 s it
+re-publishes the same capture with only MediaMTX's TCP candidate, which costs latency and nothing else. The stats
+line shows loss, retransmit requests (nack) and `via TCP` once it has happened; the recording picks up one extra
+file at the switch, which the sweep stitches. The switch is remembered in that browser, so the next stream starts
+on TCP; the "Lossy uplink" checkbox in Go Live's setup shows and controls it. Starlink users: tick it before the
+first stream, the dish drops a burst on every satellite handoff (~15 s cycle) and that is what was splitting files.
+
+## 5a. Onboarding a gamer's clips (Syncthing) — self-service
+
+Gamers do it themselves at **https://clips.BASE/** (Discord login): the page shows our Syncthing Device ID and
+asks for theirs; on submit, dumpd creates `clips/<username>` with the video-only `.stignore`, adds their device
+to our Syncthing (REST API, key read from `/srv/media/.syncthing/config.xml`) and creates the receive-only
+folder `clips-<username>` shared with them (min free 300 GB). They accept the device + folder on their PC,
+point it at their clip directory, Send Only. The page shows pairing/connection/sync state. Idempotent.
+**Per-user quota (soft)**: dumpd measures every `clips/<user>` every 5 min and pauses that user's Syncthing
+folder when it exceeds the quota (default `CLIPS_QUOTA_GB=100` in `dumpd.service`), resuming automatically
+under 90 %. The clips page shows usage and the paused state. Overrides per user in `/etc/vanguard/clip-quotas.json`,
+e.g. `{"techy2493": 250, "newguy": 50}` (GB) — read live, no restart. A gamer frees space by deleting clips in
+their own folder (Syncthing mirrors the deletion). Hard byte-exact quotas would be XFS project quotas on `/srv`
+(needs a remount with `prjquota`); not set up.
+Manual fallback (admin, same result for the server folder):
+
+    ./addClipUser.sh <username>        # creates /srv/media/clips/<username> with the video-only .stignore
+
+Their side (Windows: Syncthing installer or SyncTrayzor; Mac/Linux: Syncthing): open the UI, **Actions → Show ID**,
+send you the Device ID. In their UI: **Add Remote Device** with the server's ID (ours: `sync.BASE` → Actions → Show ID),
+then **Add Folder** on their clip directory (ShadowPlay/Medal/Steam saves), Sharing tab → tick the server,
+Folder Type **Send Only**, Save.
+
+Our side at https://sync.BASE/: **Add Remote Device** → paste their ID, name it after them, Save. When their folder
+offer pops up: **Add** → Folder Path `/var/syncthing/clips/<username>`, Folder Type **Receive Only**, Advanced →
+Minimum Free Disk Space **300 GB**, Save. Clips start arriving; the sweep normalizes them; Jellyfin shows the
+folder as that person's show. Their own folder structure is mirrored; only video files are pulled.
+
+## 5a². Gateway page — https://media.BASE/ (the main host root; `/about/` redirects here)
+
+Role-aware landing page (Discord login): everyone sees Watch / Live / Send clips / Sign out with short how-tos;
+`streamers` get the dock + OBS setup steps; `editors` the browser + multicam/timecode notes; `media-admins`
+trash/Syncthing/whoami; `git-users` Gitea; `bot-users` the bot; `admins`/`authentik Admins` the admin consoles and
+where things live. Identity comes from `GET /me` (dumpd). Point people here first.
 
 ## 5b. Syncthing hardening (do once, in the server's Syncthing UI)
 
@@ -120,7 +201,7 @@ already `.mp4`), the raw's name is appended to `.stlocal` (an `#include`d ignore
 drops it without flagging or re-pulling), and the raw is deleted locally — it still exists on the
 gamer's PC. `configure.py` creates the folder + ignore file for each streamer.
 
-In the UI (https://HOST/sync/):
+In the UI (https://sync.BASE/):
 - each clips folder → path `/var/syncthing/clips/<user>`, **Folder Type: Receive Only**,
   Edit → Advanced → **Minimum Free Disk Space: 300 GB** (so clips can never fill the mirror and stall
   the recordings).
@@ -129,8 +210,17 @@ In the UI (https://HOST/sync/):
 
 ## 5c. Authentik (Discord login) and Jellyfin
 
-DNS: `auth` and `watch` as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
+**Login page branding and the Discord forward are not clicked in the UI.** `./configure.py --apply` (or
-for both hostnames, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
+`./configure.py --ak-branding` on its own) sets them from the ORM every time: brand title, logo, favicon and
+background from `authentik/brand/*.svg`, the login-card CSS, the media gateway as the brand's default application,
+flow titles, and a redirect stage at the top of `default-authentication-flow` that sends members straight to
+Discord. Nobody sees an Authentik page unless they open the admin card at
+`https://auth.BASE/if/flow/default-authentication-flow/?admin=1` (akadmin + password; the link is on the diag
+landing dashboard). Change the look by editing the SVGs or the `CSS` list in `configure.py`, then re-run.
+A UI edit to those objects is overwritten by the next apply, on purpose.
+
+DNS: `auth`, `watch` (and `files`, `clips`, `sync`, `live`, `git`, `bot`) as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
+for every hostname, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
 adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
 hostnames filled in. (`./configure.py --full` asks for the same hostnames if you prefer.) Then:
 
@@ -174,21 +264,82 @@ Phase 4 is the Vanguard Metadata Jellyfin plugin; Phase 5 puts copyparty, /sync
 ## 5d. MariaDB + Discord bot (for the bot dev)
 
 `mariadb` (public on 3306, host network, root local-only, 32-char generated passwords) with a database/user
-for the bot, and `discord-bot`, a Python 3.12 container that runs `bot/app/main.py` and idles until that file
+for the bot, and `discord-bot`, a Python 3.12 container that runs `/app/main.py` and idles until a release is
-exists. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
+installed. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
 or the `[d]` menu item; `./configure.py --show` prints them).
 
-The bot also serves a web UI on 127.0.0.1:8000, published as `https://bot.HOST/` behind **Authentik
+**Releases are self-contained bundles, and a push to `main` deploys itself.** The bot repo's Gitea Actions
-forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`. Create the matching
+workflow runs its tests and then builds one tar.gz holding `main.py`, `app/`, `website/`, `BUILD.json` **and
-Proxy Provider in Authentik (forward auth, single application, external host `https://bot.HOST`), add it to
+`vendor/`** — its own dependencies, as prebuilt wheels for this image's Python. `PYTHONPATH=/app/vendor` is how
-the embedded outpost, and bind a group policy. Until then the site returns an error, never an open page.
+the code finds them, so **a new dependency never needs an image rebuild** and the server never contacts a
+package index.
+
+The workflow's last job posts that bundle to **dumpd** at `POST /deploy-bot` on 8081 — the Docker-bridge-only
+port the firewall already restricts, so a CI job never needs the Docker socket or an ssh key. dumpd checks an
+HMAC-SHA256 signature over the body, then runs `/usr/local/bin/vanguard-deploy-bot` (this repo's
+`deployBotBundle.sh`), which verifies every file against `BUILD.json`, swaps the bundle into `/srv/bot/app`
+(bind-mounted at `/app`, previous release kept beside it) and restarts the container. The install log goes back
+in the HTTP response, so it appears in the CI job.
+
+**The bot's own secrets live in Gitea, not here.** The deploy job also sends the bot's settings, signed with
+the bundle, and dumpd writes them into this stack's `.env` — but only the keys `BOT_ENV_KEYS` lists. Anything
+else comes back to the CI job as `settings_refused` and is not written, so the database password and the
+Authentik keys this repo generates cannot be overwritten from CI. Rotating the Discord token is therefore a
+Gitea edit plus a re-run, with no server access.
+
+In Gitea, on the bot repo under Settings → Actions:
+
+| | |
+|---|---|
+| Secret `BOT_DEPLOY_SECRET` | must equal the server's; `./configure.py --show` prints it |
+| Secret `DISCORD_BOT_TOKEN` | the bot token |
+| Variable `DISCORD_GUILD_ID` | the server id |
+| Variable `VANGUARD_DEPLOY_HOST` | only if the job cannot detect the Docker gateway |
+
+On the server, generated into `.env` and copied to `/etc/vanguard/deploy.env` by `configure.py`:
+
+| | |
+|---|---|
+| `BOT_DEPLOY_SECRET` | 64 hex chars, the shared signing key |
+| `BOT_DEPLOY_AUTO` | `1` (default) accepts CI installs; `0` makes dumpd refuse them with 403 |
+| `BOT_ENV_KEYS` | the only `.env` keys CI may set, `DISCORD_BOT_TOKEN,DISCORD_GUILD_ID` by default. Extend it when the bot needs a new setting. |
+
+By hand, any time:
+
+    bin/vgbot-merge-sqlite.py --source /data/<old>.db  # merge an old SQLite database into the live MariaDB (report; --apply writes)
+    ./deployBotBundle.sh vanguard-bot-bundle.zip      # install a release (the artifact as Gitea hands it over, zip or tar.gz)
+    ./deployBotBundle.sh --latest                     # fetch the newest main build first (needs BOT_REPO_TOKEN in .env)
+    ./deployBotBundle.sh --status                     # version, commit, package list, container state
+    ./deployBotBundle.sh --rollback                   # back to the previous bundle
+    docker compose logs -f discord-bot                # watch it
+
+**Merging an old SQLite database in** (`bin/vgbot-merge-sqlite.py`): the bot rebuilds part of its state from
+Discord on every start, so after a backend switch some tables hold fresh rows while the old file holds months of
+history, and neither side is a superset. The bot's own importer refuses a non-empty destination by design, so this
+merges instead: log tables append on a natural key, and the handful of profile/state tables are resolved column by
+column with an explicit rule each (earliest `joined_at`, `first_seen_at` and rank `started_at` win, so a rebuild
+that stamped today's date is undone; latest activity and current rank/presence win). It runs in one transaction,
+is idempotent, and refuses rather than guessing if a column has no rule or a table is missing from the
+destination. Run it with the gateway stopped, report first, then `--apply`.
+
+**Identity, exactly:** the bot gets the caller's Discord snowflake from the `discord_id` claim the proxy puts in
+`X-Authentik-Jwt`, emitted by the scope mapping in `authentik/vanguard-profile-scope-mapping.py` (the same mapping
+that carries inherited groups). Authentik already stores that id on the account's source connection, so nothing
+has to be matched by name. Without the claim the bot falls back to matching the Discord **username** only, never a
+server nickname: nicknames are self-chosen and not unique, so matching one would let a member be mistaken for
+another or make another ambiguous. `discord-bot` also runs on **its own Docker network** (`botnet`), because it
+trusts the headers Caddy injects and nothing else should be able to reach `:8000` to forge them.
+
+The bot serves **both** its API and its web pages on 127.0.0.1:8000, published as `https://bot.HOST/` behind
+**Authentik forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`, and the
+bot authorises per request from those headers (`WEB_ALLOWED_GROUPS=bot-users,admins`; `WEB_SERVE_UI=true`
+turns the pages on). Create the matching Proxy Provider in Authentik (forward auth, single application,
+external host `https://bot.HOST`), add it to the embedded outpost, and bind a group policy. Until then the
+site returns an error, never an open page. Caddy's own gate is `BOT_REQUIRE_GROUP` (`bot-users` by default),
+so **admins need the `bot-users` role too** — or set `BOT_REQUIRE_GROUP=` and let the bot's group list decide.
 
 The build brief for whoever writes the bot is `bot/AGENT.md`; `bot/README.md` is the ops cheat-sheet.
 
-    docker compose logs -f discord-bot                     # watch it
-    docker compose restart discord-bot                     # after code changes
-    docker compose build discord-bot && docker compose up -d discord-bot   # after requirements.txt changes
-
 ## 5e. Gitea + Actions runner
 
 `gitea` (MariaDB-backed, `git.HOST` via Caddy, git-over-ssh on **2222**, registration closed) and
@@ -204,13 +355,51 @@ already existed run `./createGiteaDb.sh` once.
 Runner caveat: it holds the Docker socket, so anyone who can push a workflow can run as root on the box.
 Keep repo write access to trusted people.
 
+**Login via Authentik (Discord)** — same pattern as Jellyfin, done once:
+1. Authentik → Applications → Create with provider → name `Gitea`, slug `gitea`; **OAuth2/OpenID Provider**,
+   authorization flow implicit-consent, client type Confidential, redirect URI (strict)
+   `https://git.HOST/user/oauth2/authentik/callback`, signing key = the default certificate, scopes
+   `openid email profile` (the default profile mapping carries `preferred_username` and `groups`). Copy client ID + secret.
+2. Gitea → Site Administration → Identity & Access → Authentication Sources → Add: type **OAuth2**, name
+   **`authentik`** (must match the callback path), provider **OpenID Connect**, client ID/secret, OpenID Connect
+   Auto Discovery URL `https://auth.HOST/application/o/gitea/.well-known/openid-configuration`, tick "Skip local
+   2FA", claim name for groups `groups`, administrator group claim value **`git-admins`**. Save.
+   Authentik group nesting makes the admins inherit it: `authentik Admins` → parent `admins` → parent `git-admins`
+   (members of a child group count as members of the parent). **Inherited groups are not in the `groups` claim by
+   default**: the managed profile mapping uses `ak_groups` (direct memberships only), and proxy providers re-add
+   that mapping on every save, so it cannot be swapped out. Add a second scope mapping, Customization → Property
+   Mappings → Create → Scope Mapping, name `Vanguard profile (nested groups)`, scope `profile`, expression:
+
+       direct = set(request.user.ak_groups.all())
+       return {"groups": sorted(g.name for g in request.user.all_groups() if g not in direct)}
+
+   and tick it (next to the default one) on the `Vanguard sites`, `Jellyfin` and `Gitea` providers → Advanced
+   protocol settings → Scopes. Claims from several mappings merge, so the header becomes direct + inherited groups
+   without duplicates. Takes effect on the next login.
+3. compose (already set): registration only via external sources, auto-registration + account linking on,
+   username from `preferred_username`. A first Discord login creates the Gitea user with no form.
+4. Access = Authentik group **`git-users`**: Directory → Groups → create it (add a Discord role to it in the
+   roles→groups table); Applications → `Gitea` → Policy / Group / User Bindings → bind `git-users` and `git-admins`.
+   Non-members are refused by Authentik before Gitea sees them.
+5. Test in a private window: https://git.HOST/user/login → "Sign in with authentik" → Discord → logged in.
+   Members of `admins` become Gitea site admins on login. The local admin account stays as a fallback.
+Optional later: hide the password form (custom template) so Discord is the only visible way in.
+
 ## 5f. Live portal (`live.HOST`)
 
 React + dockview page (`live/`) behind Authentik forward-auth: every active stream is a dockable window,
 audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
 MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
-MediaMTX runs the `-ffmpeg` image so the optional "is live" Discord hook (`LIVE_WEBHOOK_URL` in `.env`,
+MediaMTX runs the `-ffmpeg` image for the on-ready hook `bin/mtx-on-ready.sh` (bind-mounted into the container):
-asked by `configure.py --full`) can post when a stream starts.
+the optional "is live" Discord post (`LIVE_WEBHOOK_URL`, one of the Discord settings grouped at the top of
+`.env`; set it with `./configure.py --set LIVE_WEBHOOK_URL=…`, which tells you MediaMTX needs a recreate) and the
+**WebRTC audio mirror**. Browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only) and
+OBS sends AAC, so for every stream whose audio is not already Opus the hook runs one ffmpeg that copies the video
+and transcodes the audio to Opus into the sibling path `<user>/<Title>.webrtc` (via MediaMTX's RTSP on the
+container loopback). The portal plays the mirror over WebRTC and hides it as a panel; recordings, HLS and the
+Discord post stay on the original. Mirrors are never recorded and can only be published from inside the container
+(dumpd checks the source IP). Browser Go Live already sends Opus and gets no mirror. Cost: a few % of a core per stream.
+Viewer requirement on the streamer side: **B-frames 0** (section 5), or WebRTC is refused and the portal falls back to HLS.
 
 1. CNAME `live` → main host; `./configure.py --full` asks the hostname (also `--hosts … LIVE`).
 2. Authentik: Proxy Provider (forward auth, single application, external host `https://live.HOST`) →
@@ -223,7 +412,7 @@ from `/me/stream-token` for the logged-in Discord user (member of `streamers`).
 Bandwidth: no transcoding, every viewer gets the source bitrate (~15 Mbps) → ~30 concurrent viewers on
 the 500 Mbps line. For a crowd, restream to Twitch from OBS.
 
-## 5g. Authentik forward-auth for the Caddy-gated hosts (main host, `live.`, `bot.`)
+## 5g. Authentik forward-auth for the Caddy-gated hosts (`media.`, `files.`, `clips.`, `sync.`, `live.`, `bot.`)
 
 Design: **Authentik authenticates once for the whole domain; Caddy authorizes per host** from the
 `X-Authentik-Groups` header the outpost returns; **Discord roles fill the Authentik groups** at login.
@@ -234,8 +423,8 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
    cookie domain = the parent domain (e.g. `dumpstervigilantes.rip`). Edit the Embedded Outpost →
    Applications → add it → Update. One login now covers every gated host.
 3. Per-host group requirement = `.env`: `BOT_REQUIRE_GROUP=bot-users` (default), `LIVE_REQUIRE_GROUP=`
-   (empty = any logged-in member), `SYNC_REQUIRE_GROUP=media-admins` (the `/sync` Syncthing UI on the main host).
+   (empty = any logged-in member), `SYNC_REQUIRE_GROUP=media-admins` (the Syncthing UI on `sync.`).
-   On the main host copyparty itself authorizes by group (`editors`, `media-admins`); the dump endpoint lets
+   On `files.` copyparty itself authorizes by group (`editors`, `media-admins`); the dump endpoint lets
    `admins`/`media-admins` dump anyone's night. Leave "Intercept header authentication" **off**: login is
    Discord only, no app passwords, no WebDAV. `./configure.py` apply regenerates the Caddy blocks; a user outside the
    group gets a 403 from Caddy before the app sees the request.
@@ -244,21 +433,110 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
    `discord-roles-to-groups` from `authentik/discord-roles-to-groups.py` (fill in the server ID and the
    role-id → group table). Bind it to `default-source-enrollment` **and** `default-source-authentication`
    (engine mode `all`, after `discord-guild-member`). From then on giving someone a Discord role gives them
-   the matching access on next login; removing it removes it.
+   the matching access on next login; removing it removes it. Memberships added by hand in Authentik are left
+   alone: the policy only revokes what it granted itself (tracked in the user attribute `discord_role_groups`).
    copyparty additionally requires a header with a secret name that only Caddy adds (`idp-h-key`,
    `COPYPARTY_IDP_KEY` in `.env`), so identity headers cannot be forged even if copyparty were reached directly.
-   Logout: copyparty's logout link and the live portal's "Sign out" go to `/outpost.goauthentik.io/sign_out`,
+   Logout: copyparty's logout link, the live portal's "Sign out" and the dock's "sign out" go through
-   which ends the Authentik session (a local app logout alone would be undone on the next request).
+   `/outpost.goauthentik.io/sign_out`; `https://<any gated host>/logout` does the same from anywhere
-5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them.
+   (e.g. from the Syncthing UI, which has no logout of its own). Set the provider's **Invalidation flow** to
+   `default-invalidation-flow` — the one named **"Logout"**, the same flow Authentik's own logout button uses.
+   Do NOT use `default-provider-invalidation-flow`: despite the name it is called "Logged out of application"
+   and ends the session for that one application only, leaving the SSO session alive. With it, `/logout` drops
+   the proxy cookie, forward-auth asks Authentik again, Authentik still knows you, and you are signed straight
+   back in — which looks exactly like the page merely refreshing. With `default-invalidation-flow` every
+   logout ends the whole session and forces a full Discord re-login on the next visit (and so picks up group
+   changes). Discord itself may still approve silently if you remain logged into Discord in that browser.
+5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them, and
+   `https://HOST/me` adds the `discord_id` claim decoded from `X-Authentik-Jwt` — the quickest check that the
+   `Vanguard profile` scope mapping is live. The raw token is deliberately never echoed: the browser does not
+   see that header today, and printing it would hand the user a bearer token they otherwise could not read.
    Group changes in Authentik apply on the next login — log out at `https://auth.HOST/` and back in.
 6. Test in a private window: `https://live.HOST/` → Discord login → portal; `https://bot.HOST/` → 403 unless
    you hold the bot role; `https://live.HOST/me/stream-key` shows your username.
 
 Jellyfin and Gitea use OIDC providers instead, not this.
 
+## 5h. Diagnostics: Kibana at `diag.HOST`
+
+Four containers, one store, one UI. Kibana is served at the **root** of `diag.HOST`, with no sub-path to get
+wrong. Arriving there opens a dashboard of ours, "Vanguard ops", via Kibana's `defaultRoute` setting: links to
+every host and to the triage page, plus shortcuts into Discover, Dashboards, Stack Rules and Index Management.
+It is an ordinary dashboard, so add metric panels beside the links (`vanguard_media_pending` first).
+
+Two paths on that host are ours rather than Kibana's:
+
+| path | what |
+|---|---|
+| `/streaming` (Stuck tab) | **stuck recordings**: everything that finished but never published, why each one is stuck, and Retry / Bin buttons. The list is readable by anyone the gate allows; the actions are enforced in dumpd for the admin groups, not hidden in the page. Binning moves the file to `trash/<today>/`, the same 3-day undo as a dumped night |
+| `/streaming/api/stuck` | dumpd serves it. Both the page and `vgstuck.sh` on the host run the same script, so they cannot disagree about why something is stuck |
+
+Kibana deliberately does **not** live under `/kibana`. That would need `server.basePath` plus
+`rewriteBasePath`, a Caddy handle that does not strip the prefix, and the prefix appended in both Beats'
+`setup.kibana`, `kibana-setup.py`, `kibana-alarms.py` and the deadman's status check — and missing the Beats
+one crash-loops Metricbeat, which exits when Kibana is unreachable. `defaultRoute` buys the landing page
+without any of it.
+
+After any hostname change, `sudo vghosts-check.py` validates every place a name lives, in the order the change
+must be made (HOSTNAME-CHANGE.md §E).
+`observability/kibana-setup.py` creates the first-run objects and takes a step name, so you can redo one:
+`connector`, `rule`, `dataview`, `landing`, or all of them when omitted. Disk use by streamer, clipper, month, logs
+and system is its own page, `https://diag.BASE/disk`. `observability/kibana-alarms.py`
+creates the six alarm rules.
+
+| container | job |
+|---|---|
+| `elasticsearch` | one store for **both** logs and metrics. 2 GB heap, single node, loopback only |
+| `kibana` | Discover, dashboards, alerting. Published as `https://diag.HOST/` behind the gate |
+| `filebeat` | every container's logs (autodiscover) plus our journal units |
+| `metricbeat` | host + Docker metrics, replacing node_exporter and cAdvisor, plus MediaMTX's and Caddy's Prometheus endpoints |
+
+Both Beats write the same correlation fields (`service.name`, `container.name`, `vanguard.category`,
+`@timestamp`), so one dashboard can show a CPU spike beside the log lines from that minute and one filter
+narrows both. `LOGGING.md` lists the fields and the single Kibana data view that makes it work.
+
+Four things are pinned on purpose, each because the default bites:
+
+| setting | why |
+|---|---|
+| `ES_JAVA_OPTS=-Xms2g -Xmx2g` | a self-sizing JVM takes a quarter of the box; the spare capacity belongs to x264 |
+| `vm.max_map_count=262144` | a host sysctl, set and persisted by `install.sh` and by every apply. ES will not start without it |
+| `xpack.security.enabled=false` | safe only because it listens on loopback behind Authentik; it is also why there is no second login |
+| `KIBANA_ENCRYPTION_KEY` | without a fixed key Kibana loses every saved object, dashboards included, on restart |
+
+**First-run steps in Kibana** (both are UI actions on purpose, so they stay adjustable without a redeploy):
+
+1. **Retention**: Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase.
+2. **The deadman's heartbeat**: a rule that always fires, every minute, with a webhook action posting to
+   `http://<docker gateway>:8081/heartbeat`. That proves rule evaluation *and* delivery work end to end, and the
+   deadman treats a stale heartbeat as the alarm. Without it the deadman reports the heartbeat missing, which is
+   correct rather than a bug.
+3. **Alerts to Discord**: a webhook connector pointing at `ALERT_WEBHOOK_URL`. Until one exists, rules still
+   evaluate and show in Kibana, they are simply not delivered.
+
+**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
+nothing but curl, because nothing inside Elastic can report that Elastic is the thing that broke:
+
+| it checks | because |
+|---|---|
+| Elasticsearch answers, cluster not red | no store, no anything |
+| no index is read-only | the disk watermark sets that and **does not unset it** when space is freed, so ingestion stays silently dead |
+| Kibana is available | it is the only UI |
+| each Beat wrote a document recently | "running" is not "shipping" |
+| the Kibana heartbeat is fresh | proves the whole rule → webhook → delivery path |
+| the disks | the likeliest death, and it must be reported even when the reporter is what broke |
+
+    systemctl list-timers vanguard-deadman.timer
+    journalctl -u vanguard-deadman -n 20
+    sudo /usr/local/bin/vanguard-deadman.sh          # run it now; exit 1 means it found something undeliverable
+
+Filebeat holds the Docker socket for autodiscover, which is root-equivalent on this host — the same exposure
+act_runner has. Accepted deliberately; the alternative is losing the container labels that make the
+correlation above work.
+
 ## 6. Editors
 
-Log in with Discord at https://media.example.com/ (copyparty behind Caddy + Authentik). Access is by Authentik
+Log in with Discord at https://files.BASE/ (copyparty behind Caddy + Authentik). Access is by Authentik
 group: **editors** read streams and clips and write `/finals`; **media-admins** have full rights everywhere
 including `/trash`. Scrub any `.mp4` in the browser, download what you want.
 
@@ -286,12 +564,158 @@ MediaMTX asks dumpd (`POST /mtx-auth` on 127.0.0.1:8081, reachable only from Doc
 connection. Reads/playback are allowed (viewers are gated by Caddy + Authentik). Publishing needs a token
 minted by `GET /me/stream-token` behind Authentik for a member of `streamers`/`admins`/`media-admins`; dumpd
 checks the HMAC signature (`/etc/vanguard/stream-token.secret`, generated into `.env` as `STREAM_TOKEN_SECRET`),
-the expiry, and that the path starts with that user's name. Changing this needs **no MediaMTX restart**:
+the expiry, and that the path starts with that user's name. MediaMTX does not pick up `mediamtx.yml` changes
-MediaMTX hot-reloads `mediamtx.yml`; `configure.py` reloads Caddy in place and only restarts copyparty.
+live on this setup: after any change to it, `docker compose restart mediamtx` between streams. Day-to-day
+operations (streamers, tokens, titles) never touch that file. `configure.py` reloads Caddy in place and
+restarts only copyparty.
+
+## Timecode for multicam editing (frame accurate)
+
+Every published stream carries a QuickTime **timecode track** (`tmcd`, time of day in Eastern, at the file's
+frame rate) and a UTC `creation_time`, written by the sweep when it publishes the file, plus a sidecar
+`<Title>__<ts>.json` with the exact capture time (the Jellyfin plugin reads it). Resolve and Premiere read the
+track natively: drop clips from several streamers on a multicam timeline, sync by timecode.
+
+Where the time comes from, best first:
+1. **Overscan band** — a thin strip of extra rows below the picture carrying the *server* clock, encoded
+   (black/white cells, format in `bin/vgstrip.py`) plus a small readable clock. The server decodes it
+   (`bin/vgstrip.py`) at the start, middle and end of each recording → the timecode is frame-accurate at the
+   start and the sidecar records drift over the file. **Then the band is cropped off**: the sweep re-encodes the
+   stream (libx264 CRF 18, frame timing preserved, audio copied) so the published file is the game image + audio +
+   the `tmcd` track and nothing else; the raw with the band is deleted (`KEEP_RAW_STREAMS=1` keeps it in `.src/`).
+   Streams without the band are remuxed losslessly. Cost: every re-encode (streams and clips) runs x264 on
+   `ENCODE_THREADS` threads (2, set in `systemd/normalize-sweep.service`) so Jellyfin and live viewers stay
+   responsive; a 3 h 1440p60 night therefore takes most of the next day to publish. OBS: dock → *Timecode band* →
+   "Add band + clock source" (canvas grows by W/160 rows, a locked browser source `Vanguard Clock` →
+   `https://live.BASE/dock/clock/` sits in the band; only while not streaming). Go Live: the "Timecode band" checkbox (default on).
+2. **Server arrival time** (no band) — microseconds in the recording filename. Within ~0.1–0.3 s per streamer
+   (encoder + network delay); coarse but always available.
+
+Viewing: the live portal crops the band on the live picture (the only place it still exists); recordings in
+Jellyfin, copyparty and the editors' timelines are already at game resolution. Recordings published before
+the crop existed: `sudo vgdeband.sh [FILE…]` re-encodes them the same way and marks the sidecar.
+Cross-check when it matters: everyone is in the same voice chat, so audio-waveform sync is sample-accurate.
+Why continuous and not a start burst: a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h).
+
+Dropped frames / gaps: OBS keeps stamping frames from its clock, so after a drop the file has a hole and every
+frame that exists is still at the right time — the timecode stays correct, and players (Jellyfin, the portal)
+hold the last frame across the hole with audio in sync. By default the sweep does **not** re-encode a stream to
+fill holes (hours of CPU on this box); the sidecar lists them (`gaps.count`, `total_missing_ms`, first 50
+positions). Premiere preserves timing on such files; in Resolve check the clip's frame-rate interpretation.
+**Gap filling (available, off by default).** The sweep can re-encode a stream whose holes total at least
+`FILL_GAPS_MS` milliseconds to constant frame rate with duplicated frames (raw kept in `.src/`, timecode carried
+over). It costs ~2× the stream length in CPU on the E-2136, so it is off until the box has a hardware encoder.
+Turn it on (threshold 1 s) / off, on the server:
+
+    sudo sed -i 's|^#* *Environment=FILL_GAPS_MS=.*|Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
+    sudo sed -i 's|^Environment=FILL_GAPS_MS=.*|# Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
+
+Check: `systemctl show normalize-sweep -p Environment`. Takes effect on the next sweep run; already-published
+files are not revisited. Clips are always converted to CFR regardless.
+Planned alternative: re-encode flagged streams off-site on a machine with a hardware encoder (home box NVENC or a
+Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
+
+## Deployment (push to deploy — built, OFF by default)
+
+What lands in `/usr/local/bin` is listed once, in `bin/install.list`, and read by `install.sh`,
+`configure.py --apply` and `deploy.sh`. It used to be three hand-kept lists and they drifted: a push was
+installing five of ten scripts, so `vgstuck.sh`, both sourced helpers and both timer scripts only reached the
+host if somebody ran `--apply` by hand. `check.sh` now fails if the list names a missing file, or if
+something in `bin/` is neither listed nor explicitly excluded (`mtx-on-ready.sh` is mounted into the MediaMTX
+container; `vgbot-merge-sqlite.py` is a one-off migration).
+
+The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea →
+`./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token
+secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout.
+
+`sudo vanguard-deploy pull` fetches `main` and applies **only what changed**, after `check.sh`:
+
+| changed | action | interruption |
+|---|---|---|
+| `configure.py`, `dock/`, `clips/`, `about/`, stignore | `configure.py --apply` → Caddy reload, copyparty restart | none / a second |
+| `bin/*`, `systemd/*` | install; dumpd restart when it changed; timers pick up next run | a second |
+| `firewall/nftables.conf` | `nft -f` | none |
+| `docker-compose.yml`, `bot/Dockerfile`, `bot/entrypoint.sh` | `compose up -d --build` for changed services | that service |
+| `deployBotBundle.sh` | reinstalled as `/usr/local/bin/vanguard-deploy-bot` (what dumpd runs for the bot's CI) | none |
+| anything that changes `mediamtx.yml` or the mediamtx service | **restart only when no stream is live**; otherwise deferred, `deploy-pending.timer` retries every 5 min | none while live |
+| `jellyfin-plugin-vanguard/dist/*.zip` (committed) | `installJellyfinPlugin.sh` | Jellyfin restart |
+| `live/` | `npm run build` on the server → `/srv/live` | none |
+
+`sudo vanguard-deploy rollback` checks out the previously deployed revision and applies the reverse diff.
+`sudo vanguard-deploy status` shows revisions, auto mode, pending restarts, live streams.
+
+**One-time rollout** (the last scp-style deploy; afterwards the server pulls). From the dev box:
+
+    cd ~/claudevs/vanguard-stream && git push
+    scp bin/dumpd.py debian@HOST:~/vanguard-stream/bin/
+    scp systemd/deploy-pending.service systemd/deploy-pending.timer debian@HOST:~/vanguard-stream/systemd/
+    scp docker-compose.yml configure.py deploy.sh switchToClone.sh check.sh install.sh README.md STATUS.md CommunityDiscordMedia.md debian@HOST:~/vanguard-stream/
+    ssh -t debian@HOST 'cd ~/vanguard-stream && sudo apt-get install -y -q git nodejs npm rsync python3-yaml >/dev/null && sudo install -m 0755 bin/dumpd.py /usr/local/bin/ && ./configure.py --apply && sudo systemctl restart dumpd && sudo vanguard-deploy status'
+
+The apply prints the post-start fixups (Syncthing, Gitea admin user, Jellyfin plugin) and ends with a status
+block: `DEPLOY_AUTO=0`, no pending restart, and "not a git checkout" until the next step.
+
+Then switch the server folder to a clone (two steps):
+
+    ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh key'       # prints a public key
+    # Gitea → repo → Settings → Deploy Keys → add it, read-only
+    ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh clone && cd ~/vanguard-stream && ./configure.py --apply >/dev/null && sudo vanguard-deploy status'
+
+`switchToClone.sh clone` keeps `.env`, the generated configs, the token secret and plugin zips, and
+leaves the old copy at `~/vanguard-stream.old` until you delete it. From here on a manual deploy is
+`sudo vanguard-deploy pull` on the server.
+
+**Automatic**: Gitea → repo → Settings → Webhooks → Gitea, URL `http://<docker gateway>:8081/deploy`
+(`docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}'`, usually `172.18.0.1`),
+secret = `DEPLOY_WEBHOOK_SECRET` from `.env`, push events, branch `main`. dumpd verifies the HMAC and runs
+`vanguard-deploy pull` **only when `DEPLOY_AUTO=1`** in `.env` (then `./configure.py --apply` to push the setting
+to `/etc/vanguard/deploy.env`). Until then every push is logged (`journalctl -u dumpd | grep deploy`) and ignored; Gitea's "test delivery"
+should get a 202 "auto-deploy disabled" back. Optional `DEPLOY_NOTIFY_WEBHOOK_URL` posts deploy results to Discord.
+Turning it on: `DEPLOY_AUTO=1` in `.env`, `./configure.py --apply`, and
+`git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml` for CI.
+
+CI: `.gitea/draft-workflows/ci.yml` (inert until moved to `.gitea/workflows/`) runs `check.sh` and the plugin
+tests/build on the Actions runner. The runner never deploys — it must not touch the host.
+
+## Rebuild from scratch (what is and isn't automated)
+
+Server side, scripted:
+
+    git clone ssh://git@git.HOST:2222/techy/Media-Server-Setup.git ~/vanguard-stream   # or scp the folder
+    cd ~/vanguard-stream && sudo bash install.sh        # packages, docker, folders, host scripts, timers, firewall, portal build
+    ./configure.py --full                               # hostnames, secrets, configs, containers, post-start fixups
+
+Restore first if rebuilding a known box: `.env` (all secrets — without it every password is regenerated and
+the databases won't match), `/srv/appdata` (Authentik DB, Jellyfin, Gitea, MariaDB, Syncthing config),
+`/srv/media` (the footage), `/etc/vanguard/clip-quotas.json` if used. With those restored, nothing below is
+needed except DNS.
+
+Post-start fixups `configure.py` performs on every apply (idempotent): Syncthing GUI auth off, Gitea admin
+user created from `.env` (`GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD`; the install page is locked), Jellyfin plugin
+installed from the committed zip. Authentik's `akadmin` password is set on first start from
+`AUTHENTIK_BOOTSTRAP_PASSWORD` in `.env` (no initial-setup page).
+
+Still manual on a truly fresh box (not in the repo; documented in §5c–5g):
+- Authentik: Discord source, identification-stage source, enrollment user type, groups, the two policies,
+  the domain-level proxy provider + outpost host, the Jellyfin and Gitea OIDC providers, application bindings
+- Jellyfin: first-run wizard, libraries (Shows + Vanguard provider), SSO plugin + provider, branding
+- Gitea: authentication source (authentik), deploy key, webhook; `./registerGiteaRunner.sh`
+- Discord bot: its repo lives outside this one, so nothing here restores it. Put `BOT_DEPLOY_SECRET`
+  (`./configure.py --show`), `DISCORD_BOT_TOKEN` and `DISCORD_GUILD_ID` into that repo's Gitea Actions
+  secrets/variables, then push to its `main` — or install a bundle by hand with `./deployBotBundle.sh`.
+  `/srv/bot/app` is created empty by `install.sh`; the container idles until a release lands there.
+- Discord developer portal: application, redirect URIs, bot; DNS records
+A `.env`-driven Authentik blueprint would remove the first line (planned).
 
 ## Layout on the server
 
+MediaMTX runs with `network_mode: host`, as its documentation recommends for WebRTC: the bridge put Docker's
+userland UDP proxy between browsers and port 8189 and it lost packets, which corrupted recordings and split
+them into fragments. Its ingest ports (1935, 8890/udp, 8189 tcp+udp) are accepted by the host firewall; its
+API, HLS, WHIP/WHEP signalling and metrics bind 127.0.0.1 and are reached only through Caddy and Metricbeat.
+
     /srv/media/streams/<user>/<Title>/.live/<ts>.mp4             recording in progress (hidden from Jellyfin/copyparty)
+    /srv/media/streams/<user>/<Title>__<ts>.json                 capture time (UTC ms), timecode, source (strip|arrival), drift
     /srv/media/streams/<user>/<Title>__<ts>.mp4                  finished stream, one flat MP4 per recording, directly in
                                                                  the streamer's folder (several files inside one title
                                                                  folder would be ONE Jellyfin episode with "versions")
@@ -299,6 +723,8 @@ MediaMTX hot-reloads `mediamtx.yml`; `configure.py` reloads Caddy in place and o
     /srv/media/clips/<user>/.stlocal                             Syncthing ignore list maintained by the sweep
     /srv/media/clips/<user>/.src/clip.mkv                        raw source, only for clips that did NOT arrive via Syncthing
     /srv/media/clips/<user>/clip.normalize-failed.log            only if normalization failed; delete to retry
+    /srv/bot/app/                                                the Discord bot's installed release (bind-mounted at /app)
+    /srv/bot/previous/                                           the release before it (deployBotBundle.sh --rollback)
     /srv/media/finals/                                           editors upload here
     /srv/media/trash/<night>/...                                 dumped, purged after 3 days
     /srv/media/.markers/                                         dump markers
@@ -313,7 +739,7 @@ upscaled; aspect kept).
 | input | action |
 |-------|--------|
 | already compliant | nothing |
-| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: lossless remux to `<user>/<Title>__<ts>.mp4`, `.live` source and empty title folder removed |
+| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: decode the timecode band (or use arrival time), then publish to `<user>/<Title>__<ts>.mp4` with a `tmcd` track + `creation_time` — band present: re-encode with the band cropped off; no band: lossless remux — write the `.json` sidecar, remove `.live` and the empty title folder |
 | in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
 | HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
 | ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |
@@ -345,4 +771,6 @@ evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
 |------|------|------|
 | normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
 | trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
-| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy at /dump |
+| vanguard-alerts.timer | every 1 min | delivers Kibana's alert documents to Discord, with a 15-min per-rule repeat guard |
+| vanguard-deadman.timer | every 5 min | watches the alerting stack from outside it, and the disks |
+| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time, on `clips.` at /me/clips and on `diag.` at /streaming/api/stuck |

STATUS.md

@@ -1,7 +1,7 @@
 # Vanguard stack — what runs, what people do, what's unfinished
 
-Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-10-31.
+Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03.
-Host: `ovh1.dumpstervigilantes.rip` (OVH SYS-1, Debian 13). Bundle: `~/vanguard-stream` on the box.
+Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
 Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).
 
 ## Runs on its own
@@ -11,12 +11,15 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | Continuous | OBS streams record into a hidden `streams/<user>/<Title>/.live/` folder, invisible to Jellyfin and copyparty until finished | MediaMTX |
 | Continuous | Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only) | Syncthing |
 | Continuous | New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab | Jellyfin plugin (needs bot token + channel in its config) |
-| Every 5 min | Finished files are checked. Streams idle ≥3 min: lossless remux out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4`, one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
+| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
 | Every 5 min | Discord scheduled events snapshotted into a history so recordings can be matched to their session | Jellyfin plugin |
 | Every 30 min + startup | Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box) | Jellyfin plugin |
 | Daily 04:10 ET | Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change | Jellyfin plugin |
+| Every 5 min | Each user's clips folder is measured against their quota (100 GB default, per-user overrides); over quota → their Syncthing folder is paused, under 90 % → resumed; the clips page shows usage | dumpd quota loop — **pending deploy** |
 | Daily 05:00 ET | Dumped nights older than 3 days deleted for good | trash purge (systemd timer) |
 | On boot | Everything restarts itself after the media mount is up; interrupted recordings stay playable | systemd + Docker |
+| Every 5 min | If a deploy deferred a MediaMTX restart, retry when no stream is live | `deploy-pending.timer` — **pending first apply** |
+| On push to `main` | (OFF: `DEPLOY_AUTO=0`) Gitea webhook → dumpd → `vanguard-deploy pull` applies only what changed | **built, disabled** |
 | Continuous | Active streams are watchable live (WebRTC, HLS fallback) by Discord-server members; optional "is live" Discord post when a stream starts | MediaMTX + live portal (page deployed; MediaMTX port fix pending apply) |
 | **Not yet running** | Pull of files older than 15 days to the home HDD | `home/rclone-archive.sh`, needs the home box |
 | **Not yet running** | Nightly backup of the Authentik database (Postgres) | planned |
@@ -25,16 +28,20 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 
 | Who | Activity | Where | Login |
 |---|---|---|---|
-| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://ovh1…/dock/` | Discord login in the dock (once); dock mints a 12 h token per Go Live — **no passwords** (pending apply) |
+| Streamer | One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode | OBS dock / Go Live | live |
-| Gamer | Nothing; clips save where they always did | their clipper + Syncthing, paired once | — |
+| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://live.…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** |
-| Editor | Browse by streamer/title, scrub in browser, download originals, upload finals | copyparty `https://ovh1…/` | Discord (Authentik); rights by group `editors` / `media-admins` — **pending apply** |
+| Gamer | Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://clips.…/` + Syncthing | Discord (any server member) |
+| Viewer | Live portal lays streams out as an even grid automatically when one joins or leaves ("Arrange" button redoes it after manual moves); audio follows the focused window; double-click a tab to rename a stream (kept in that browser), double-click the slot at the left of a tab bar to name the group (session); once the viewer has moved/resized/renamed anything the layout is left alone: new streams chime and wait as draggable chips in a tray, "Arrange" re-grids and returns to auto | `https://live.…/` | Discord |
+| Everyone | Landing page with role-aware links and instructions | `https://media.…/` | Discord |
+| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep) | Resolve / Premiere | — |
+| Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://files.…/` | Discord (Authentik); rights by group `editors` / `media-admins` |
 | Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin `https://watch.…/` | Discord (Authentik) |
-| Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal `https://live.…/` | Discord (Authentik, domain-level) — live; viewer side not yet exercised during a stream |
+| Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal `https://live.…/` | Discord (Authentik, domain-level) — **live, shows a real stream** |
 | Streamer (Windows) | **Go Live** from the browser: pick the WarDogs window (+audio on Windows Chrome 141+; Linux/Mac video only), streams as `<user>/HelmetCam-<date>`; no OBS needed | Live portal → Go Live | Discord; key fetched for the logged-in user — **verified end to end** (publish → `.live` → sweep → title folder) |
-| Devs | Host code, review, CI | Gitea `https://git.…/`, ssh 2222 | **local account** (Discord later) |
+| Devs | Host code, review, CI | Gitea `https://git.…/`, ssh 2222 | Discord (Authentik), group `git-users`; `git-admins` = site admin |
 | Bot dev | Push code, restart container, read logs; web UI at `https://bot.…/` | Gitea + compose; `bot/AGENT.md` | web UI behind Discord (Authentik), once its provider exists |
 | Admin | Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups | `./configure.py` | ssh |
-| Admin | Pair a gamer's device, approve their folder | Syncthing UI `https://ovh1…/sync/` | Discord + group `media-admins`, then Syncthing's GUI password |
+| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI `https://sync.…/` | Discord + group `media-admins` (Syncthing's own login disabled) |
 | Admin | Groups, who may log in, app access | Authentik `https://auth.…/` | Discord or akadmin |
 | Admin | Install/upgrade the Jellyfin plugin | `./installJellyfinPlugin.sh` | ssh |
 | Admin | Register the CI runner | `./registerGiteaRunner.sh` | ssh |
@@ -43,48 +50,83 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 
 | Service | Role | State |
 |---|---|---|
-| MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens) | live; auth switch **pending apply** (hot-reload, no restart) |
+| MediaMTX | **Host network since 2026-11-05** (MediaMTX's recommended deployment for WebRTC). On the bridge, every WebRTC media packet on 8189/udp was relayed by Docker's userland proxy, which lost ~0.4% of them: each loss corrupted an access unit, the recorder reopened its file, and one evening produced 471 fragments; every publisher also appeared as 172.18.0.1. Ingest (1935, 8890/udp, 8189) now arrives through the nftables INPUT chain; API, HLS, WHIP/WHEP signalling and metrics bind 127.0.0.1 and are reached only through Caddy and Metricbeat. RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`); on-ready hook posts "is live" and runs the WebRTC audio mirror (`<path>.webrtc`, audio → Opus, never recorded) | live, token auth verified |
-| copyparty | browse/scrub/download/WebDAV/finals upload | behind Authentik, group permissions — **built, pending apply** (currently local accounts) |
+| copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (`editors`, `media-admins`), Discord-only, no WebDAV | live |
-| dumpd + dock | dump-tonight endpoint, OBS dock page | identity from Authentik headers — **built, pending apply** |
+| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity (on `live.`), `/me/clips` Syncthing self-service (on `clips.`); OBS dock with timecode-band setup | identity from Authentik headers | live |
-| Syncthing | clip intake, UI at `/sync/` | UI behind Authentik + `media-admins` — **built, pending apply** |
+| `media.` gateway, `clips.` self-service | static pages behind the gate | — | live; **first real pairing 2026-11-01 (thrahg)** — exposed a missing `.stlocal` on fresh folders, fixed |
+| Timecode | `bin/vgstrip.py` decoder, `tmcd` + sidecar in the sweep, `/dock/clock/` strip page, Go Live band composite, portal crop | server clock is the reference | **live, verified** (OBS: frame-accurate, 0 gaps) |
+| Syncthing | clip intake, UI at `sync.` | UI behind Authentik + `media-admins`, own GUI auth off | live |
 | normalize sweep | editing standard, one copy per clip | live |
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
-| Caddy | TLS + routing for all hostnames | live |
+| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
+| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
+| diag landing | `diag.…` opens on a "Vanguard ops" dashboard: a markdown panel of links to every host plus the stuck page, and shortcuts to Discover, Dashboards, Stack Rules and Index Management. Kibana still owns the root; `defaultRoute` just redirects there. Created by `kibana-setup.py landing`, so a rebuild recreates it | **added 2026-11-04** |
+| Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
+| Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
+| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
+| Fragment stitching | A stream that dropped and came back is one recording in several `.live/` files. `bin/vgstitch.py` (run first by every sweep) joins them losslessly with the concat demuxer when there are fewer than `STITCH_MAX_PARTS` (500; was 20 until Kyle's lossy night showed a recorder restart every ~10 s is normal for 0.5% loss) and the gaps are under `STITCH_MAX_GAP_S` (10 min); the arrival-time file names place every piece. Joins are collapsed and recorded in the sidecar (`stitched.parts/joins`, `missing_ms`). Nothing in a stream's `.live/` is touched until no file there is open or fresh (`event=defer` / `event=stitch_wait`). A resolution/codec change breaks the chain (copy-concat is impossible across it), so Go Live now holds resolution and sheds frame rate instead. The fragments go to `trash/<night>/` for the 3-day undo; a failed stitch leaves them alone and the old per-file path publishes them | **added 2026-11-05** |
+| Reconnect fragments | Every publisher drop-and-reconnect opened a new MediaMTX recording; one bad night left **471** one- and two-second files in `.live/`, mostly from the 6 Mbps Go Live floor (fixed). Cleanup: `vgstuck.sh drop --under 5` or the page's bulk button, admin-only, confirm shows the count. Prevention: fragments are stitched first (row above); a **lone** stream recording under `MIN_LIVE_SECONDS` (10) goes to trash instead of being published. Clips are never subject to either: they are a few seconds long on purpose and never pass through `.live/` | **added 2026-11-04** |
+| Login branding + Discord forward | Done from the ORM by `configure.py` on every apply (`--ak-branding` runs just this): brand title/logo/favicon/background from `authentik/brand/*.svg` (installed into Authentik's own `/media/public/brand/`, the only unauthenticated place that can serve them), login-card CSS in the site palette with Discord blurple, brand default application = the media gateway (so `auth.` root never shows Authentik's app library), flow titles renamed, Discord attached to the identification stage. Then the forward: a **redirect stage first in `default-authentication-flow`** whose static target is the Discord source's login URL, bound to an expression policy that skips it when the flow URL carries `?admin=1` (the executor keeps the flow query in the session key `authentik/flows/get`, not `request.GET`). Members never see an Authentik page: protected link → Discord consent → back where they were (the outpost's `next` rides the same session key). akadmin's password card is `https://auth.BASE/if/flow/default-authentication-flow/?admin=1`, linked only from the diag landing dashboard | **added 2026-11-06** |
+| Frozen Go Live picture | Breezy: one frame, then one more per alt-tab. Window capture on Windows comes from the compositor and an **exclusive full-screen** game bypasses it, so Chrome only gets a frame when Windows composites one (alt-tab). Go Live now has a capture choice, window (default) or **whole screen** (`displaySurface: monitor`, desktop duplication: sees exclusive full-screen, carries system audio), remembered per browser, plus a hint naming the symptom and the better fix: the game in borderless/windowed full-screen keeps window audio and needs no change | **added 2026-11-06** |
+| Base-domain move, no outage | `configure.py --alias NEW` makes every Caddy site answer on both bases (`media.old, media.new {`), certificates issued, nothing else touched; apps and Authentik are then moved one by one while the old names keep working; `--ak-hosts OLD NEW` (dry-run first) moves Authentik last; `--base NEW` flips the primary, recreates the host-bearing containers, and turns the old names into 308 redirects (`REDIRECT_BASE`). `read_hosts` understands both Caddyfile forms; the sweep probes aliases and old-name redirects. Rule documented at the top of HOSTNAME-CHANGE.md: apps before Authentik, never the reverse | **added 2026-11-06** |
+| Hostname sweep | `bin/vghosts-check.py` (installed; `sudo vghosts-check.py` on the box, `--repo` on a dev box, `--json`): validates every place HOSTNAME-CHANGE.md lists against the Caddyfile — DNS, `.env`, `mediamtx.yml`, the env each running container was created with, HTTPS + gate redirects per host, Authentik (outpost host, proxy external host + cookie domain, Jellyfin/Gitea redirect URIs, the exact Discord callback to check by eye), Jellyfin config XML + live SSO start, Gitea `app.ini` + live auth-source redirect + runner + clone remote, repo defaults, about-page labels, docs with browser links to `ovh1`. Sections run in the order a change must be made (documented in HOSTNAME-CHANGE.md §E) because each depends on the previous; exit 1 on any failure | **added 2026-11-06** |
+| Onboarding checklist | `media.BASE/onboarding/` (`about/onboarding/index.html`, served by the gateway's file server, so behind the same Discord gate). For the person **doing** the onboarding: eight topics to tick as they are covered — who we are, about/verify, community, who Vanguard is, who Spartan are, comms, training, push to mute/talk — and a Clear button for the next recruit. Ticks are per browser. The gateway's `try_files` gained `{path}/index.html` so folder pages under `about/` resolve without a trailing slash | **added 2026-11-06** |
+| Disk page | `diag.BASE/disk`, its own page (`streaming/disk.html`, served from the same mount): filesystem bars, then recordings per **streamer by month** (month from the recording's own `__YYYY-MM-DD` stamp; raw `.src` and unpublished `.live` called out), **clips per clipper by month**, trash by night, **logs** (Elasticsearch indices by the date in their names, the ES data dir as the on-disk truth, journald, Docker json logs) and **system** (each `appdata/<app>`, live portal, bot, Docker's own image/volume figures). dumpd walks the tree with allocated-block sizes, caches ten minutes, `?refresh=1` / "Rewalk now" forces | **added 2026-11-06** |
+| Stop a stream (Live tab) | Each publisher row has **Stop stream** with a block length (kick only / 30 min / 2 h / 8 h, default 30 min). dumpd kicks the session through MediaMTX's API (`/v3/<webrtcsessions|rtmpconns|srtconns>/kick/<id>`) **and** writes the account into `.markers/publish-blocks.json`; `/mtx-auth` denies a blocked account on every publish, so OBS's auto-reconnect and a Go Live restart both get "publishing is paused for this account by an admin" until it expires or an admin presses Allow again in the new "Blocked from publishing" section. The block is written before the kick, so there is no reconnect window. Admin groups only; the recording so far is kept and the sweep publishes it as usual | **added 2026-11-06** |
+| Transport on the Live tab | MediaMTX's session API labels the selected ICE pair `host/udp/…` or `host/tcp/…`, so the publishers table now shows `webrtc udp` / `webrtc tcp` and the session start time, and a new "Publish sessions — last 24 h" table lists every WebRTC publish session with transport, start, end, duration, received/lost, and a "switched from udp" marker on the session that began the TCP leg: that is the when and the before/after loss of a lossy-uplink switch. dumpd keeps the history in `.markers/webrtc-history.json`, updated on every Live-tab poll and every metrics scrape, so it fills in while nobody is watching | **added 2026-11-06** |
+| Lossy uplinks (Go Live) | Kyle's night: one publish session, 16 recorder restarts, 165 processing errors, 0.5% loss. A loss burst that eats a keyframe's slices leaves MediaMTX's recorder an access unit with only SPS/PPS, it refuses it (`access unit doesn't contain an IDR or non-IDR NALU`) and the path restarts the recorder 2 s later = a new file every ~10 s. Three levers pulled: (1) retransmits — MediaMTX's WebRTC code calls pion `ConfigureNack` for every peer connection, so NACK is on; the Go Live stats line now shows `loss x% (n nack, m resent)` so a stream where NACKs stay at 0 under loss is visible at a glance; (2) the stitcher cap went 20 → 500 so such a night is one recording; (3) **automatic TCP**: Go Live samples the server's receiver reports every second and when loss over the last 20 s exceeds 0.3% (after at least 1500 packets) it ends the WHIP session and re-publishes the same tracks with only MediaMTX's TCP candidates (`webrtcLocalTCPAddress :8189`) left in the answer. One extra recording file (stitched), ~2 s of picture lost in the swap, then no loss can reach the recorder; latency is what TCP costs instead. It never flips back during a stream; the stats line says `via TCP` and why. Kyle is on **Starlink**, whose scheduler reshuffles every ~15 s and drops a burst at each handoff — that is the every-10-to-15-s restart cadence in his log, and no NACK can cover a burst that long. So the switch is also **remembered per browser** (`vg.golive.tcp`): once it has fired, the next stream starts on TCP, no 20 s of corrupted recording and no extra file; a "Lossy uplink" checkbox in Go Live's setup shows and controls it. First TCP night: stable, one file, zero recorder restarts — but soft: with no loss to react to, Chrome's estimator only sees the dish's delay swings and parks near the 1.2 Mbps floor. TCP publishes now use a **4 Mbps floor** (`TCP_MIN_KBPS`): safe there because the UDP failure mode of a floor (sending into loss) cannot happen on TCP, the worst case is added latency | **added 2026-11-06** |
+| B-frame enforcement | obs-websocket cannot set encoder options in Advanced output mode, but it can switch **profiles**, and encoder settings live in a profile. The dock has an "OBS profile to stream from" setting (default `Vanguard`) and switches OBS to it before every Go Live, plus a checkbox to do so when the dock loads, i.e. on OBS startup. A streamer sets the profile up once with B-frames 0; the post-Go-Live probe verifies the result. Missing profile = a red status line naming what to create, never a silent fallback | **added 2026-11-05** |
+| B-frame detection | MediaMTX refuses H.264 with B-frames over WebRTC, and nothing on the publish path can see it: obs-websocket cannot set encoder options in OBS's Advanced output mode, and MediaMTX only notices when a viewer connects. dumpd now reads the B-frame depth off the first fragments of the live recording (`ffprobe`, small probesize, ~50 ms, works on a growing file, cached 30 s). The **dock** polls `/me/stream-health` after Go Live and shows a red banner with per-encoder instructions; in Simple output mode with x264 it writes `bf=0` into the options itself first. The **Live tab** shows an encode column per publisher: "no B-frames" or "N B-frames · WebRTC refused" | **added 2026-11-05** |
+| Streaming page | `https://diag.…/streaming`, two tabs. **Live**: every publisher with its real address, packets received and lost on the way in, the last ten minutes of publish sessions / recorder restarts / processing errors, and a one-line verdict (loss, reconnects, or parameter changes) with the recorder log tail. **Stuck**: the triage below. Served by dumpd `/streaming/api/live` and `/streaming/api/stuck`; `/stuck` redirects | **added 2026-11-05** |
+| Dump a night (web) | Third tab of the streaming page on the triage host: the night's recordings and clips grouped by streamer with sizes, a date picker (defaults to dumpd's "tonight, else last night"), a Dump button per streamer and a Dump everyone button for admins. Two confirmations, then a POST to `/streaming/api/dump` on this host, so one cookie covers it. The dock's own Dump tonight button stays for a streamer's own night and is personal for everyone, admins included: dumpd honours a `user`/`all` only when the request came in on `/streaming/api/dump`, so the only place another streamer's night can be dumped is this tab. GET is a dry run; only the POST moves files, to `trash/<night>/` with the 3-day undo. Below the night table: **Bin all unprocessed**, which runs `vgstuck.sh wipe` — stops the sweep (killing any conversion in flight; the timer restarts it to an empty queue), then every `.live/` file from any night, parked or fresh, plus the failure logs and `.part` leftovers, then the emptied folders. The only thing it leaves is a file still held open after the sweep is down, i.e. a stream MediaMTX is recording this minute. `drop --all` stays refused so a pattern can never do this by accident | **added 2026-11-05** |
+| Stuck recordings | The Stuck tab of the page above for anyone past the gate, on the triage host (already `media-admins` only), linked from the gateway's Media admin card and from the diag landing dashboard: every unpublished recording, why it is stuck, and Retry / Bin buttons. dumpd serves `/streaming/api/stuck` and allows the actions only for the admin groups; both the page and the CLI run `bin/vgstuck.sh`, so they can never disagree. `bin/vgstuck.sh` triages anything that never published: it says why each one is stuck (`parked` by a failure log with ffmpeg's reason, `published` collision, `waiting`, or still `writing`), then `retry <match>` clears the blocker and `drop <match>` moves it to trash with the usual 3-day undo. `drop --all` is refused. The sweep also logs `event=parked` once per run so Kibana can name them, not just count them | **added 2026-11-04** |
+| Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
+| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
 | Jellyfin | watch page, SSO via Authentik, direct play only | live |
 | Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `<Title>__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |
 | MariaDB | databases for the bot and Gitea; public :3306, root local-only | live |
-| Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live, local admin |
+| Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live; Discord login via Authentik OIDC **verified**; access = `git-users`/`git-admins` groups; local admin as fallback |
 | act_runner | CI jobs in Docker | **not registered** |
-| discord-bot | Python runtime, idles until `bot/app/main.py` exists | live, no code yet |
+| `bot.` host + Discord bot | the bot's API **and** its web pages on one port (`WEB_SERVE_UI`), behind Authentik forward-auth; Caddy requires `bot-users`, the bot re-checks `bot-users,admins` per request | a push to `main` deploys itself: CI builds a **release bundle** (app + `vendor/` dependencies) and posts it signed to dumpd `POST /deploy-bot` (bridge-only port, no Docker socket in CI) together with the bot's own secrets, which are managed in Gitea; dumpd verifies everything, writes the keys `BOT_ENV_KEYS` allows into `.env` and swaps the bundle into `/srv/bot/app` → `/app`; image installs no bot dependencies | **reviewed + wired 2026-11-02, first install pending** (needs the Authentik app for `bot.`, and `BOT_DEPLOY_SECRET` + `DISCORD_BOT_TOKEN` + `DISCORD_GUILD_ID` set in Gitea) |
-| `bot.` host | bot web UI behind Authentik forward-auth; Caddy requires group `bot-users` (from a Discord role) | Caddy block live; **`bot` CNAME still points at the wrong IP; roles→groups policy pending** |
 | Live portal (`live/`, `live.` host) | React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd `/me/stream-key` | live; Go Live verified |
 
 ## Unfinished, in rough priority
 
-1. **MediaMTX container recreate** for `TZ=America/New_York` (filenames were UTC-stamped) and `moq: no` — `docker compose up -d --force-recreate mediamtx` after the compose upload.
+0a. **WebRTC audio** (found 2026-11-01 with the 5-stream test): AAC cannot cross WebRTC, so the portal was silent unless it fell back to HLS. Fix committed: audio mirror in `bin/mtx-on-ready.sh` + portal mapping. **Deploy pending** (MediaMTX recreate between streams; dumpd restart; portal push). Streamers must set B-frames 0 for WebRTC playback.
-1b. Authentik `discord-roles-to-groups` policy + `bot-users` group (`bot` CNAME now correct).
+0a. **Discord bot**: live on MariaDB since 2026-11-03 (139 members tracked, 36 commands synced, web UI on the `bot.` host).
-1c. Confirm plugin 0.1.2 installed and runner registered.
+   Old SQLite history merged in the same night with `bin/vgbot-merge-sqlite.py` (the bot's own importer refuses a
-1d. Watch a real stream from a second browser: corner chip should say `webrtc`; confirm WarDogs audio came through.
+   non-empty destination): Cool Points, training, invites and growth restored, original joined/first-seen/rank dates
-2. **Register the runner** (`./registerGiteaRunner.sh`).
+   recovered from the rebuild, live rank and activity kept. Pre-merge dump at `/srv/appdata/discord-bot/before-merge.sql`.
-3. **`bot` CNAME + Authentik proxy provider** — first use of forward-auth; proves the pattern for copyparty.
+   The tool is idempotent, so re-running it is safe.
-4. **Apply phase 5+6** (copyparty, dock, dump, /sync behind Authentik; group permissions). Authentik side: group `media-admins` with techy in it. Decision: Discord login only → no WebDAV; browser downloads (zip/tar) instead.
+0. **Band removal**: deployed 2026-11-01; `vgdeband.sh` catch-up ran on the Vanguard-Training recordings (check `journalctl -u vgdeband` for the done lines). Live portal: viewer now clips the band with a measured game-aspect frame (refits on every dockview resize/move) — committed 2026-11-01, deploy with `./deployLive.sh`.
-5. **Apply phase 7+11** (token publish auth): `configure.py` apply (MediaMTX hot-reloads; no stream interruption), new dumpd + unit, firewall rule for 8081 from Docker, dock upload, portal deploy. Then every streamer must be in the Authentik `streamers` group.
+
-6. **Gitea login via Authentik** (OIDC source, auto-register, `admins` → Gitea admin).
+1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (`/whoami` should show `bot-users`). Nested groups **verified 2026-11-01**: `git-admins` now reaches the header via the `Vanguard profile (nested groups)` scope mapping (README 5e). `akadmin` remains the recovery login if a policy misfires.
-7. **Housekeeping**: pin `AUTHENTIK_TAG` in `.env`, rotate the Discord client secret that was in a screenshot, nightly Postgres dump, Jellyfin login branding (Discord button first), check a non-admin Discord user's Jellyfin view.
+2. Confirm plugin 0.1.3 under Jellyfin → Plugins and the Gitea runner registered.
-8. **Medal filename pattern** in the plugin's timestamp parser — need one sample filename.
+3. Portal viewer during a real stream: corner chip `webrtc`; Windows Go Live with WarDogs audio (first Windows streamer).
-9. **Home archive puller** on the home box.
+5. Housekeeping: pin `AUTHENTIK_TAG`, rotate the Discord client secret seen in a screenshot, nightly Postgres dump, Caddyfile formatting warning.
-10. **Plugin repository** served by Caddy so Jellyfin updates the plugin from its catalog.
+6. Medal filename pattern in the plugin (need a sample filename).
-11. **Clip submissions** (Medal/Twitch/YouTube links, Discord uploads) — belongs to the bot.
+7. Home archive puller on the home box.
-13. **SYS-3** when transcoding or sweep speed hurts: Quick Sync, flip the plugin's policy toggle off.
+8. Plugin repository served by Caddy so Jellyfin updates from its catalog.
+9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run
+   (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served
+   from the bot itself now; an Authentik username is matched to one Discord member by name.
+9a. Kibana first-run: a data view over `filebeat-*,metricbeat-*`, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new `vanguard_media_*` metrics.
+9b. Elasticsearch security as its own phase, if `diag.` is ever to widen past `media-admins`: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected `Authorization`). Per-person identity is reachable but buys little, since audit logging is itself paid.
+10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
+11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.
 
 ## Verified behaviours worth remembering
 - **Never scp generated configs** (`mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt`, `.env`): a template copy wiped all streamers once. They are gitignored now; `*.example.*` are reference only.
 - Caddy runs in a container: anything it serves from disk must be a mount (`/srv/live` was missing → 404).
 - Go Live quality depends on the browser's encoder: Windows Chrome uses the GPU (fine); Linux Chromium on NVIDIA is software H.264 Baseline (soft at any bitrate, CPU-limited at 1440p60) and cannot be fixed by flags — Linux streamers use OBS. The stats strip shows encoder + profile.
 - Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as `<Title>__<ts>.mp4` in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
+- Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. **Gap filling is available** (`FILL_GAPS_MS` in `normalize-sweep.service`, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
 - The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
+- **An encode that fails with `Error opening output files: Invalid argument` is usually contradictory flags, not the path.** ffmpeg rejects `-r` together with a non-CFR `-fps_mode`; the real message is one line earlier in its stderr, which the sweep now captures into the failure log. Found 2026-11-03 on the 1 November test recordings, which also exposed that `avg_frame_rate` can be pure fiction (95 fps reported for a 30 fps source) — the sweep now takes the lower of r_frame_rate and avg_frame_rate.
 - A deploy block that is only half-run leaves old binaries in `/usr/local/bin`; `grep -c` the installed script before blaming logic.
+- MediaMTX does not reload `mediamtx.yml` live here (bind-mounted file): any change to it needs `docker compose restart mediamtx` between streams. Nothing day-to-day touches that file.
+- **Timecode verified live (2026-10-31)**: OBS + NVENC with the band → `source: strip`, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path.
 - MediaMTX stamps recording filenames in its container clock: without `TZ` it is UTC and the plugin reads names as Eastern (4 h off). `TZ=America/New_York` is set on the container.
 - MediaMTX 1.21 refuses to start without `webrtcAdditionalHosts` or `webrtcIPsFromInterfaces`; configure.py writes the public host in.
 - Jellyfin honours provider-supplied season numbers on a Shows library (day seasons appeared). Next check: Monday 04:10 ET should fold them into a week season.
@@ -92,3 +134,19 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
 - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
 - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
+- **"Enforce B-frames 0 through obs-websocket" is only possible in Simple output mode with x264.** Advanced-mode encoder settings live in `streamEncoder.json`, which the protocol does not expose. The enforceable thing is detection on the server (the recording reveals the reorder depth within seconds) with a loud dock banner, which covers every encoder. Found 2026-11-05 when an OBS stream was unwatchable with `has_b_frames=1`.
+- **A published UDP port on the Docker bridge goes through `docker-proxy`, a userland relay, and it drops packets under load.** The sign is every client appearing as the bridge gateway (`172.18.0.1`) in MediaMTX's metrics. For WebRTC media at ~1000 packets/s this produced steady ~0.4% loss that no browser-side setting could fix, because the loss was on our box. Anything doing real-time UDP belongs on `network_mode: host`, which is also what MediaMTX documents. Found 2026-11-05 after chasing it as a client bitrate problem first.
+- **Two MediaMTX path regexes that can both match a name is a coin toss, not "first rule wins".** The mirror rule (`…\.webrtc$`) and the generic rule both matched `Techy_multi_3.webrtc` because titles were allowed dots, and the generic one won: a **36 GB** audio mirror was recorded as a full duplicate of its source. Both rules now have a dot-free title segment, so a name satisfies exactly one, and titles no longer accept dots anywhere they are typed. Needs a MediaMTX recreate between streams. Found 2026-11-04.
+- **Never `ffprobe` fragmented MP4 in a request path.** `.live/` recordings have no duration in their header, so ffprobe reads the whole file to answer. The stuck-recordings survey did that per file, took over three minutes for 471 of them, dumpd's 180 s request ceiling killed the response, Caddy returned an empty 502 and the page said "unexpected end of JSON". Size and age come from `stat`; duration is opt-in (`vgstuck.sh list --durations`). Found 2026-11-04.
+- **A high `x-google-min-bitrate` disables WebRTC congestion control.** Go Live sent `min-bitrate=6000`, so Chrome never encoded below 6 Mbps; on an uplink that could not carry it, the excess was dropped, MediaMTX logged `invalid FU-A packet (non-starting)` and every viewer saw blocky corruption. The loss was on the PUBLISH session (4761 packets for one streamer) while the viewer sessions lost nothing, which is the fingerprint: the server relays damage it received. Floor lowered to 1200 and `scaleResolutionDownBy` unpinned so the encoder can shed resolution. Found 2026-11-04.
+- `tracks=['Opus','H264']` on a path means a browser (Go Live); `MPEG-4 Audio` means OBS. Quick way to tell who is publishing how.
+- **`handle_path` strips the prefix, so an upstream's own redirects lose it.** MediaMTX redirects every HLS playlist request once (to `?cookieCheck=1`) with a root-relative `Location`, so the browser followed `/<user>/<Title>/index.m3u8` into the portal's SPA catch-all, got `index.html`, and hls.js said `manifestParsingError` — a web page where a manifest should be. Fixed with `header_down Location ^/ /hls/`. `curl` against MediaMTX directly hides this completely, because there is no prefix to lose. Found 2026-11-04.
+- **Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked** in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a `tmcd` data stream is the fingerprint. First working Windows audio: 2026-11-03.
+- The sweep's band-removal branch **overwrote `$reason`**, which carried the probe's `audio=<codec>` marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got `-c:a copy` and shipped Opus inside an MP4. Fixed by keeping `probe_reason` separately; it only became reachable once Go Live audio worked.
+- **Authentik has two invalidation flows and the names are a trap.** `default-provider-invalidation-flow` is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so `/logout` drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. `default-invalidation-flow` ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03.
+- Prefer changing a proxy provider through the ORM (`ak shell`) over the UI: saving it in the UI re-adds the managed property mappings every time.
+- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
+- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
+- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.
+- `find ... | while read` runs the loop in a **subshell**, so anything it accumulates is lost at the end of the pipe. `normalize-sweep.sh` reads from a process substitution for exactly this reason.
+- Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With `xpack.security.enabled: false` every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on `diag.`. Anything in Kibana that asks Elasticsearch for privileges fails with `no handler found for uri [/_security/user/_has_privileges]`. The **Infrastructure / Metrics app does this on every page view**, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over `metricbeat-*` instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work.

UPCOMING.md

@@ -0,0 +1,95 @@
+# Upcoming — what a mil-sim community media stack still needs
+
+Written 2026-11-01 from a conversation between techy and Claude. Three parts: the gap list as first drafted, techy's
+take on it, and the agreed direction. Living plan lives in `CommunityDiscordMedia.md`; current state in `STATUS.md`.
+
+## 1. The gap list (first draft)
+
+Measured against what mil-sim groups actually do with footage, the stack covers capture, storage, access and live
+viewing. The gaps cluster around the debrief workflow, moments, comms audio, and keeping the box healthy.
+
+### Debrief and review (the core mil-sim use)
+- **Synced multi-POV playback.** The timecode track makes it possible, but nothing plays it yet. A VOD mode of the
+  live portal that opens every recording from one night and scrubs them in lockstep turns the archive into an
+  after-action-review tool. The single biggest missing piece.
+- **Ops as a first-class object.** Titles are free text today. An op/training event that groups every streamer's
+  recording, that night's clips, briefing files and finals → collections, per-op download bundle for editors, a
+  Discord thread per op; the dock offers tonight's event instead of a typed title.
+- **Markers.** A "mark this" button in the dock, a Discord command, or an emoji reaction during the live post stamps
+  the server clock into metadata → chapters, editor markers, the shortlist for the review session. Cheap: the clock
+  already exists.
+- **Twitch-style clips from the live recording.** Segments are on disk while someone streams, so a `!clip` of the
+  last 30–60 s is a server-side cut with no streamer action.
+
+### Audio
+- **Separate comms track.** Editors cutting to radio chatter want game audio and voice on separate tracks. RTMP
+  carries one; SRT can carry several and OBS can send multiple audio tracks over it.
+- **Voice server recording.** Some units record TeamSpeak/Mumble as the authoritative comms log. Consent question
+  before a technical one.
+
+### Standardisation
+- **OBS starter kit.** Downloadable scene collection + profile with the band, B-frames 0, bitrate caps, dock URL.
+- **Callsign tags.** The rename feature is per viewer; a server-side label ("1-1 Lead") on live panels and recordings.
+- **Non-video intake.** Briefing slides, map images, after-action notes have no drop point (clip intake is video only
+  by design).
+
+### Health and operations
+- **Disk monitoring and alerts.** Nothing warns when disks fill, when the sweep writes a failure log, or when a
+  stream drops. With hours of 1440p a night this is the most likely quiet death. Discord alert = small job.
+- **Archive tier.** The home archive puller is planned, not built; the 15-day hot window is the only window.
+- **Backups.** Authentik and Gitea databases are not dumped anywhere yet.
+
+### Audience
+- **Public showcase.** Everything is gated; recruitment needs a route for finals to go public (YouTube from finals).
+- **Mobile.** Jellyfin apps work; the live portal is desktop-only in practice.
+
+Suggested top three: alerts (protect everything else), markers (cheap, feed everything downstream), synced
+multi-POV playback (what the archive is actually for).
+
+## 2. techy's take
+
+- **Synced multi-POV playback — yes.** It means more metadata tracking, and we already have too many files: put the
+  metadata in a database (Postgres or MariaDB), not more sidecars.
+- **Ops come with the Discord bot.** It has roll calls; we are waiting on the bot, its website and its database.
+- **Markers — good.** Likely just in the live window for now.
+- **Live clips — nice idea, but** we may instead build something like the live page for *watching*: multi-POV playback
+  plus single-POV playback plus search, and replace Jellyfin entirely (it is only a half-baked fit). Decision for later.
+- **Separate comms is coming-ish.** We run mixed comms: once the bot exists it will listen to a voice channel after
+  roll call. The Go Live button already excludes system audio, so it will carry game audio only. We also use in-game
+  comms, so it will never be perfect.
+
+## 3. Agreed direction (Claude's reply)
+
+Everything lines up on one axis: **the server clock**. Recordings, markers, roll calls and comms audio are all stored
+as UTC instants and joined later, which is exactly what a synced player needs.
+
+**Database.** Use the MariaDB that already runs for the bot rather than adding Postgres: one database to back up, and
+the bot's ops/roll-call tables land beside ours, so "this op → these recordings" is a query, not a convention. Keep the
+per-file sidecar as the self-describing fallback; the sweep writes both; a reconcile command rebuilds the tables from
+sidecars if the database is ever lost. Rough shape:
+
+| table | holds | owner |
+|---|---|---|
+| `recordings` | path, streamer, title, start UTC (band or arrival), duration, fps, timecode source + drift, band removed, gaps, optional op id | sweep |
+| `markers` | UTC instant, author, stream path or null (= everyone), kind, note — stored by **time, not file offset**, so a marker placed live attaches to the recording when published and to every other POV of that moment for free | dumpd (portal), later the bot |
+| `clips` | file, owner, arrival time, converted or not | sweep / dumpd |
+| `ops`, `roll_call` | the bot's; we only read them | bot |
+
+**Markers in the live window.** A Mark button on the focused panel plus a hotkey, posting the current server time and
+the stream path to dumpd. Small, and it gives the future player its chapter list from day one. The Discord-command
+version comes with the bot and writes to the same table.
+
+**Replacing Jellyfin.** The database makes it a clean later decision. A watch app built on the portal's code does
+single POV with search by op, streamer, date and marker, and multi-POV with lockstep scrubbing driven by the timecode.
+Until then Jellyfin keeps reading sidecars and nothing breaks.
+
+**Comms.** Bot-captured Discord audio after roll call, stamped on receipt by the same box, lines up with everything
+else with no extra sync work. In-game comms stay in the game audio; fine for review.
+
+**Sequence.**
+1. Schema + sweep inserts + marker endpoint + Mark button in the live portal (small; everything else reads from it).
+2. Write the table contract into the bot developer's brief (`bot/AGENT.md`) so the bot is designed against it now.
+3. Watch app (single + multi-POV + search) once the bot's ops data exists and the Jellyfin decision is made.
+
+Not scheduled, still true: alerts (disk / sweep failures / dropped streams), backups of Authentik + Gitea databases,
+archive puller, OBS starter kit, callsign tags, non-video intake, public showcase, mobile.

about/index.html

@@ -0,0 +1,209 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Vanguard Media</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#3d8bfd;--ok:#2ecc71;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
+  main{max-width:860px;margin:0 auto;padding:28px 16px 48px}
+  header{display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;margin-bottom:6px}
+  h1{font-size:26px;margin:0}.who{color:var(--mut)}.who b{color:var(--fg)}
+  .pills{margin:6px 0 22px}.pill{display:inline-block;background:#2a2a2a;border-radius:999px;padding:2px 10px;font-size:12px;margin:0 6px 6px 0;color:#bbb}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:18px 20px;margin-bottom:16px}
+  section h2{margin:0 0 4px;font-size:17px}section .lead{color:var(--mut);margin:0 0 12px}
+  .links{display:grid;grid-template-columns:repeat(auto-fill,minmax(240px,1fr));gap:10px;margin:10px 0}
+  a.card{display:block;background:#262626;border:1px solid #333;border-radius:10px;padding:12px 14px;color:var(--fg);text-decoration:none}
+  a.card:hover{border-color:var(--acc)}a.card b{display:block;margin-bottom:2px}a.card span{color:var(--mut);font-size:13px}
+  details{margin-top:8px}summary{cursor:pointer;color:var(--mut);font-size:13px}
+  ol,ul{margin:8px 0 0;padding-left:20px}li{margin:4px 0}code{font-family:ui-monospace,monospace;background:#121212;padding:1px 6px;border-radius:5px}
+  .hidden{display:none}footer{color:var(--mut);font-size:13px;margin-top:20px}footer a{color:var(--mut)}
+</style>
+</head>
+<body>
+<main>
+  <header><h1>Vanguard Media</h1><div class="who">signed in as <b id="who">…</b></div></header>
+  <div class="pills" id="pills"></div>
+
+  <section id="s-everyone">
+    <h2>Watch</h2>
+    <p class="lead">Everything here uses your Discord login. One sign-in covers all of it.</p>
+    <div class="links">
+      <a class="card" id="l-watch" href="#"><b>Recordings &amp; clips</b><span>Jellyfin — by streamer, by day/week/month</span></a>
+      <a class="card" id="l-live" href="#"><b>Live now</b><span>every active stream in one page, audio follows the window you click</span></a>
+      <a class="card" href="/onboarding/"><b>Onboarding checklist</b><span>for whoever is onboarding someone: tick the topics as you cover them</span></a>
+      <a class="card" href="/logout"><b>Sign out</b><span>ends the Discord session for all Vanguard sites</span></a>
+    </div>
+    <details><summary>How watching works</summary>
+      <ul>
+        <li>Jellyfin: each streamer is a show. This week's recordings sit under weekday seasons, older ones roll into week, month and year. Nothing moves; it is only re-labelled.</li>
+        <li>Live: windows open as people go live and close when they stop. Click a window to hear it; the speaker button pins audio on; the slider is per window.</li>
+      </ul>
+    </details>
+    <details><summary>Live page: windows, groups, names</summary>
+      <ul>
+        <li><b>Layout.</b> Until you touch it, windows arrange themselves into an even grid whenever someone goes live or stops. Drag a tab to move a window, drag the bars between windows to resize, drop a tab onto another window to stack them as tabs.</li>
+        <li><b>Your layout stays yours.</b> The moment you move, resize or rename anything, the page stops rearranging. Streams that end just close. New streams <b>do not</b> barge in: a chime plays and a blinking chip appears in a thin tray under the top bar. Drag the chip onto any window edge, onto a tab bar to stack it, or to the page edge; or click it to add it as a new column on the right. <b>Arrange</b> (top bar) places everything in an even grid and switches automatic layout back on.</li>
+        <li><b>Audio.</b> Sound follows the window you click. The speaker button pins a window's audio on even when you click elsewhere; <b>Mute all</b> silences everything until you click a window; <b>Follow focus</b> clears pins.</li>
+        <li><b>Rename a stream.</b> Double-click its tab, type a name, Enter. Escape cancels; an empty name goes back to <code>user · title</code>. Names stay in your browser and come back the next time that streamer uses the same title.</li>
+        <li><b>Name a group.</b> Every tab bar starts with a <code>＋ name</code> slot. Double-click it, type, Enter. Group names last until you reload the page.</li>
+        <li><b>One window big.</b> The ⛶ button at the right of a tab bar maximises that window; click it again to restore. Arranging (a stream joining or leaving, or the Arrange button) restores it too.</li>
+        <li><b>Timecode band.</b> Streams with the timecode band on show it cropped off automatically; recordings never contain it.</li>
+      </ul>
+    </details>
+  </section>
+
+
+  <section id="s-streamers" class="hidden">
+    <h2>Contribute: stream</h2>
+    <p class="lead" data-group="streamers" data-yes="You are in <code>streamers</code>: you can publish under your own name. No passwords, no stream keys to keep." data-no="Admin view — you are <b>not</b> in <code>streamers</code>, so Go Live will refuse a token for you until you get the role."></p>
+    <div class="links">
+      <a class="card" id="l-dock" href="#"><b>OBS dock</b><span>title → Go Live; Dump tonight; manual setup</span></a>
+      <a class="card" id="l-golive" href="#"><b>Go Live from the browser</b><span>Windows Chrome: share the game window, no OBS</span></a>
+    </div>
+    <details><summary>OBS setup (once)</summary>
+      <ol>
+        <li>OBS → Tools → WebSocket Server Settings → enable, set a password.</li>
+        <li>OBS → Docks → Custom Browser Docks → add <code id="dockurl">…/dock/</code>; log in with Discord inside the dock.</li>
+        <li>Make an OBS <b>profile named Vanguard</b> once (Profile → New) and in it set Output → Advanced → your encoder with <b>B-frames 0</b> (NVENC "Max B-frames", x264 <code>bf=0</code>) and keyframe interval 2 s. The dock switches OBS to that profile on every Go Live, and on startup if you tick the box in its settings; after Go Live it checks what the server actually received and turns red if B-frames got through. With B-frames the live page cannot play you over WebRTC and falls back to a delayed HLS view.</li>
+        <li>Dock → Settings: OBS WebSocket password (the rest is prefilled). Optional title prefix/suffix (<code>{date}</code> = today).</li>
+        <li>Dock → Timecode band → <b>Add band + clock source</b> while not streaming (gives editors a frame-accurate timecode).</li>
+        <li>Per stream: type a title, <b>Go Live</b>. The dock mints a 12-hour token and sets your stream key itself.</li>
+      </ol>
+    </details>
+    <details><summary>Rules worth knowing</summary>
+      <ul>
+        <li>Everything streamed is recorded and visible to editors and members. <b>Dump tonight</b> throws a night away (3-day undo by an admin).</li>
+        <li>Titles: letters, digits, <code>_ -</code>; no dots (the server reserves <code>.webrtc</code>); no double underscore; the dock shows the final name as you type.</li>
+        <li>Browser Go Live needs Windows + Chrome/Edge 141+ for window audio and the GPU encoder; on Linux use OBS.</li>
+      </ul>
+    </details>
+  </section>
+
+  <section id="s-clips">
+    <h2>Contribute: your game clips</h2>
+    <p class="lead">Anyone in the server. Pair your gaming PC once; after that your clips upload themselves and appear in Jellyfin under your name.</p>
+    <div class="links">
+      <a class="card" id="l-clips" href="#"><b>Clip setup &amp; status</b><span>pair your PC, see if it is connected and syncing</span></a>
+    </div>
+    <details open><summary>Three steps</summary>
+      <ol>
+        <li>Install <a href="https://syncthing.net/downloads/" target="_blank" rel="noopener">Syncthing</a> on the PC your clips save to (Windows: Syncthing Tray works well). Open it → Actions → Show ID.</li>
+        <li>Paste that Device ID on the <a class="l-clips" href="#">clips page</a> and press Connect.</li>
+        <li>Back in your Syncthing: accept the new device <b>Vanguard</b>, then accept the folder it offers, point it at your clip directory (ShadowPlay / Medal / Steam), Folder Type <b>Send Only</b>.</li>
+      </ol>
+      <ul>
+        <li>Only video files are pulled; your folder layout is mirrored; odd formats are converted on the server and one copy is kept per clip.</li>
+        <li>Nothing is ever written back to your PC.</li>
+      </ul>
+    </details>
+  </section>
+
+  <section id="s-editors" class="hidden">
+    <h2>Edit</h2>
+    <p class="lead" data-group="editors" data-yes="You are in <code>editors</code>: read streams and clips, write finals." data-no="Admin view — you are not in <code>editors</code>; your copyparty rights come from <code>media-admins</code> instead."></p>
+    <div class="links">
+      <a class="card" id="l-files" href="#"><b>Browse &amp; download</b><span>copyparty — scrub in the browser, download originals, zip folders</span></a>
+      <a class="card" id="l-finals" href="#"><b>Upload finals</b><span>the only folder editors can write to</span></a>
+    </div>
+    <details><summary>Multicam &amp; timecode</summary>
+      <ul>
+        <li>Every stream file has a timecode track (time of day, Eastern) and a <code>.json</code> sidecar with the exact capture time, source (<code>strip</code> = frame-accurate from the overscan band, <code>arrival</code> = coarse), drift and gaps.</li>
+        <li>Multicam: sync by timecode, cross-check with voice audio. Export at the game resolution (2560×1440 / 1920×1080) — that drops the thin band at the bottom.</li>
+        <li>Files are H.264 MP4, constant frame rate, index up front. Streams are published a few minutes after they end.</li>
+        <li>No WebDAV (Discord-only login). Select files/folders and use copyparty's zip/tar download.</li>
+      </ul>
+    </details>
+  </section>
+
+  <section id="s-mediaadmins" class="hidden">
+    <h2>Media admin</h2>
+    <p class="lead" data-group="media-admins" data-yes="You are in <code>media-admins</code>: full rights on every folder, trash, Syncthing, dump anyone's night." data-no="Admin view — you are <b>not</b> in <code>media-admins</code>: copyparty will show you nothing and <code>/sync/</code> will 403 until you get the role."></p>
+    <div class="links">
+      <a class="card" id="l-stuck" href="#"><b>Streaming</b><span>who is live and losing packets; recordings that never published</span></a>
+      <a class="card" id="l-trash" href="#"><b>Trash</b><span>dumped nights, purged after 3 days</span></a>
+      <a class="card" id="l-sync" href="#"><b>Syncthing</b><span>device pairing, folder states</span></a>
+      <a class="card" href="/whoami"><b>Who am I</b><span>identity headers as the apps see them</span></a>
+    </div>
+    <details><summary>Admin notes</summary>
+      <ul>
+        <li>Group changes in Authentik apply on the next login: <a href="/logout">sign out</a> and back in.</li>
+        <li>Dump anyone's night: the dock's Dump works on your own; for others use the API with <code>{"user":"name"}</code> or ask the owner.</li>
+        <li>New gamer: they self-serve on the <a class="l-clips" href="#">clips page</a>; you only need to watch Syncthing accept the device.</li>
+        <li><a class="l-stuck-note" href="#">Stuck recordings</a> (on the diagnostics host, with the dashboards) lists anything that finished but never published, with the reason. Retry puts it back in the queue; binning keeps the same 3-day undo as a dumped night.</li>
+      </ul>
+    </details>
+  </section>
+
+  <section id="s-git" class="hidden">
+    <h2>Code</h2>
+    <p class="lead" data-group="git-users" data-alt="git-admins" data-yes="You are in <code>git-users</code>." data-no="Admin view — you are <b>not</b> in <code>git-users</code>/<code>git-admins</code>: Authentik will refuse the Gitea login until you get the role."></p>
+    <div class="links">
+      <a class="card" id="l-git" href="#"><b>Gitea</b><span>repos, issues, CI — sign in with authentik</span></a>
+    </div>
+    <details><summary>Notes</summary>
+      <ul><li>SSH remotes use port 2222: <code id="gitssh">ssh://git@…:2222/org/repo.git</code>. Add your key under Settings → SSH keys.</li>
+      <li>The stack's own repo is <code>techy/Media-Server-Setup</code>.</li></ul>
+    </details>
+  </section>
+
+  <section id="s-bot" class="hidden">
+    <h2>Bot</h2>
+    <p class="lead" data-group="bot-users" data-yes="You are in <code>bot-users</code>." data-no="Admin view — you are <b>not</b> in <code>bot-users</code>: the bot host will 403 you until you get the role."></p>
+    <div class="links"><a class="card" id="l-bot" href="#"><b>Bot web UI</b><span>Discord bot control panel</span></a></div>
+  </section>
+
+  <section id="s-admins" class="hidden">
+    <h2>Administration</h2>
+    <p class="lead">You are in <code>admins</code> or <code>authentik Admins</code>.</p>
+    <div class="links">
+      <a class="card" id="l-auth" href="#"><b>Authentik admin</b><span>users, groups, Discord source, policies</span></a>
+      <a class="card" id="l-jfadmin" href="#"><b>Jellyfin dashboard</b><span>libraries, plugin, scheduled tasks</span></a>
+      <a class="card" id="l-gitadmin" href="#"><b>Gitea site admin</b><span>auth sources, runners</span></a>
+    </div>
+    <details><summary>Where things live</summary>
+      <ul>
+        <li>Server: <code>~/vanguard-stream</code> on the box; <code>STATUS.md</code> = what runs / who does what / what's unfinished; <code>README.md</code> = how.</li>
+        <li>Change hostnames, bot token: <code>./configure.py</code>. Plugin updates: <code>./installJellyfinPlugin.sh</code>. Portal: <code>./deployLive.sh</code> (dev box).</li>
+        <li>Streamers = Authentik group <code>streamers</code>; editors = <code>editors</code>; everything follows Discord roles via the roles policy.</li>
+      </ul>
+    </details>
+  </section>
+
+  <footer>Vanguard Media · <a href="/">home</a> · <a href="/logout">sign out</a></footer>
+</main>
+<script>
+const $ = id => document.getElementById(id);
+const host = location.hostname, base = host.replace(/^[^.]+\./, '');
+const sub = s => `https://${s}.${base}`;
+$('l-watch').href = sub('watch'); $('l-live').href = sub('live'); $('l-golive').href = sub('live') + '/#golive';
+$('l-git').href = sub('git'); $('l-bot').href = sub('bot'); $('l-auth').href = sub('auth') + '/if/admin/';
+$('l-jfadmin').href = sub('watch') + '/web/#/dashboard'; $('l-gitadmin').href = sub('git') + '/-/admin';
+$('l-dock').href = sub('live') + '/dock/'; $('dockurl').textContent = sub('live') + '/dock/';
+$('l-files').href = sub('files') + '/'; $('l-finals').href = sub('files') + '/finals/'; $('l-trash').href = sub('files') + '/trash/'; $('l-sync').href = sub('sync') + '/';
+$('l-stuck').href = sub('diag') + '/streaming#stuck';
+document.querySelectorAll('a.l-stuck-note').forEach(a => a.href = sub('diag') + '/streaming#stuck');
+$('l-clips').href = sub('clips') + '/'; document.querySelectorAll('a.l-clips').forEach(a => a.href = sub('clips') + '/'); $('gitssh').textContent = `ssh://git@${'git.' + base}:2222/org/repo.git`;
+(async () => {
+  try {
+    const r = await fetch('/me', {credentials:'same-origin'}); const d = await r.json(); if (!r.ok) throw new Error(d.error || r.status);
+    $('who').textContent = d.name && d.name !== d.user ? `${d.name} (${d.user})` : d.user;
+    $('pills').innerHTML = d.groups.map(g => `<span class="pill">${g}</span>`).join('') || '<span class="pill">no groups yet — ask an admin for a role</span>';
+    if (d.discord_id) $('who').title = 'Discord ID ' + d.discord_id;   // apps identify you by this, not by your name
+    const has = g => d.groups.includes(g);
+    const show = (id, on) => $(id).classList.toggle('hidden', !on);
+    // sections appear only for what this account can actually do (the gates check these same groups)
+    const isAdmin = has('admins') || has('authentik Admins');
+    show('s-streamers', has('streamers'));
+    show('s-editors', has('editors') || has('media-admins'));
+    show('s-mediaadmins', has('media-admins'));
+    show('s-git', has('git-users') || has('git-admins'));
+    show('s-bot', has('bot-users'));
+    show('s-admins', isAdmin);
+    document.querySelectorAll('p.lead[data-group]').forEach(p => { p.innerHTML = p.dataset.yes; });
+  } catch (e) { $('who').textContent = 'unknown (' + e.message + ')'; }
+})();
+</script>
+</body>
+</html>

about/onboarding/index.html

@@ -0,0 +1,58 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Onboarding — Vanguard</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#5865F2;--ok:#2ecc71;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:17px/1.5 system-ui,sans-serif}
+  main{max-width:560px;margin:0 auto;padding:36px 16px 48px}
+  header{display:flex;align-items:center;gap:14px;margin-bottom:18px}
+  h1{font-size:26px;margin:0}
+  .right{margin-left:auto;display:flex;align-items:center;gap:12px;color:var(--mut);font-size:14px}
+  button{font:inherit;font-size:14px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:7px 14px;cursor:pointer}
+  button:hover{border-color:var(--acc)}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:8px 10px}
+  label{display:flex;gap:14px;align-items:center;padding:12px 10px;border-radius:10px;cursor:pointer;font-size:19px}
+  label:hover{background:#262626}
+  label input{width:22px;height:22px;margin:0;accent-color:var(--acc);flex:none}
+  label input:checked + span{color:var(--mut);text-decoration:line-through}
+  .done{display:none;color:var(--ok);margin-top:14px;font-size:15px}
+</style>
+</head>
+<body>
+<main>
+  <header>
+    <h1>Onboarding</h1>
+    <div class="right"><span id="count"></span><button id="clear">Clear</button></div>
+  </header>
+  <section>
+    <label><input type="checkbox" data-id="who"><span>Who we are</span></label>
+    <label><input type="checkbox" data-id="about"><span>About / verify</span></label>
+    <label><input type="checkbox" data-id="community"><span>Community</span></label>
+    <label><input type="checkbox" data-id="vanguard"><span>Who Vanguard is</span></label>
+    <label><input type="checkbox" data-id="spartan"><span>Who Spartan are</span></label>
+    <label><input type="checkbox" data-id="comms"><span>Comms</span></label>
+    <label><input type="checkbox" data-id="training"><span>Training</span></label>
+    <label><input type="checkbox" data-id="ptt"><span>Push to mute / talk</span></label>
+  </section>
+  <div class="done" id="done">All covered. Clear for the next one.</div>
+</main>
+<script>
+const KEY = 'vg.onboarding';
+const boxes = [...document.querySelectorAll('input[type=checkbox]')];
+const load = () => { try { return JSON.parse(localStorage.getItem(KEY) || '{}'); } catch { return {}; } };
+const save = s => { try { localStorage.setItem(KEY, JSON.stringify(s)); } catch {} };
+function render() {
+  const s = load(); let n = 0;
+  for (const b of boxes) { b.checked = !!s[b.dataset.id]; if (b.checked) n++; }
+  document.getElementById('count').textContent = `${n} / ${boxes.length}`;
+  document.getElementById('done').style.display = n === boxes.length ? 'block' : 'none';
+}
+for (const b of boxes) b.addEventListener('change', () => { const s = load(); s[b.dataset.id] = b.checked; save(s); render(); });
+document.getElementById('clear').onclick = () => { save({}); render(); };
+render();
+</script>
+</body>
+</html>

addClipUser.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+# Create the server-side clips folder for a Syncthing gamer. Name = their Authentik/Discord username (lowercase),
+# because the Jellyfin plugin treats the folder as the show and looks the person up by that name.
+set -euo pipefail
+cd "$(dirname "$0")"
+u="${1:-}"; [[ $u =~ ^[a-z0-9_.-]{2,32}$ ]] || { echo "usage: $0 <username>  (lowercase letters, digits, _ . -)"; exit 1; }
+sudo install -d -o 1000 -g 1000 "/srv/media/clips/$u"
+sudo install -m 0644 -o 1000 -g 1000 syncthing/stignore "/srv/media/clips/$u/.stignore"
+[[ -e "/srv/media/clips/$u/.stlocal" ]] || sudo install -m 0644 -o 1000 -g 1000 /dev/null "/srv/media/clips/$u/.stlocal"   # #include'd by .stignore
+echo "ready: /srv/media/clips/$u  (Syncthing path: /var/syncthing/clips/$u)"
+echo "next: the gamer pairs at the clips host (self-service), or you accept their folder in the Syncthing UI at that path, Receive Only, min free 300 GB"

addCommunityMediaServer.sh

@@ -24,7 +24,7 @@ read -r AUTH_CUR WATCH_CUR < <(python3 -c 'import configure as c; h=c.read_hosts
 BASE="${MAIN#*.}"; [[ $MAIN == *.*.* ]] || BASE="$MAIN"
 
 say "Community media server — Phase 1 (Authentik + Jellyfin) on $MAIN"
-echo "Needs two DNS names as CNAMEs to $MAIN. Create them now if you haven't."
+echo "Every hostname below is a CNAME to $MAIN. Create them now if you haven't."
 AUTH=$(ask "Authentik hostname" "${AUTH_CUR:-auth.$BASE}")
 WATCH=$(ask "Jellyfin hostname" "${WATCH_CUR:-watch.$BASE}")
 GIT_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("git",""))')
@@ -33,11 +33,19 @@ BOT_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("bot",""))
 BOT=$(ask "Bot web UI hostname" "${BOT_CUR:-bot.$BASE}")
 LIVE_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("live",""))')
 LIVE=$(ask "Live portal hostname" "${LIVE_CUR:-live.$BASE}")
+FILES_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("files",""))')
+FILES=$(ask "Files (copyparty) hostname" "${FILES_CUR:-files.$BASE}")
+SYNC_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("sync",""))')
+SYNC=$(ask "Syncthing UI hostname" "${SYNC_CUR:-sync.$BASE}")
+CLIPS_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("clips",""))')
+CLIPS=$(ask "Clips self-service hostname" "${CLIPS_CUR:-clips.$BASE}")
+DIAG_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("diag",""))')
+DIAG=$(ask "Diagnostics hostname: logs, metrics, alerts" "${DIAG_CUR:-diag.$BASE}")
 
 # ---- DNS sanity ----------------------------------------------------------------------------------
 say "DNS check"
 MAIN_IP=$(ip_of "$MAIN"); ok=1
-for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE"; do
+for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG"; do
   ip=$(ip_of "$h")
   if [[ -z $ip ]]; then echo "  $h: does not resolve yet"; ok=0
   elif [[ $ip != "$MAIN_IP" ]]; then echo "  $h -> $ip (main is $MAIN_IP) — points somewhere else"; ok=0
@@ -50,7 +58,7 @@ fi
 
 # ---- write configs, .env, start containers -----------------------------------------------------
 say "Applying (configure.py --hosts)"
-VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" | sed -n '/== writing/,$p' | grep -vE '^  (techy|diceofire|[a-z0-9_-]+)  password' || true
+VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG" | sed -n '/== writing/,$p' | grep -vE '^  (techy|diceofire|[a-z0-9_-]+)  password' || true
 [[ $DRY == 1 ]] && { echo "(dry run: skipping container wait)"; exit 0; }
 
 # ---- wait ---------------------------------------------------------------------------------------
@@ -150,21 +158,23 @@ PHASE 3 — Jellyfin (direct play only on this CPU)
           Custom CSS:       paste jellyfin/branding.css
         -> Save; reload https://$WATCH/
 
-AUTHENTIK FORWARD-AUTH (gates https://$LIVE/ and https://$BOT/; later the main host) — once:
+AUTHENTIK FORWARD-AUTH (gates every Vanguard host: $MAIN, $FILES, $CLIPS, $SYNC, $LIVE, $DIAG, $BOT) — once:
   1. Applications -> Outposts -> edit "authentik Embedded Outpost" -> config: authentik_host: https://$AUTH
   2. Applications -> Create with provider -> Application "Vanguard sites"; Proxy Provider, authorization
      flow implicit-consent, mode "Forward auth (domain level)", authentication URL https://$AUTH,
-     cookie domain ${BASE}   -> then edit the Embedded Outpost -> Applications: add it -> Update
+     cookie domain ${BASE}, Invalidation flow default-provider-invalidation-flow (logout = full re-login)
+     -> then edit the Embedded Outpost -> Applications: add it -> Update
   3. Per-host access is enforced by Caddy from X-Authentik-Groups: .env BOT_REQUIRE_GROUP=bot-users
-     (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member). ./configure.py apply regenerates.
+     (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member), DIAG_REQUIRE_GROUP=media-admins.
+   ./configure.py apply regenerates.
   4. Discord roles -> Authentik groups: Directory -> Groups: create bot-users (keep streamers/editors/admins);
      Discord source -> Scopes: "guilds guilds.members.read"; Customization -> Policies -> Expression Policy
      "discord-roles-to-groups" from authentik/discord-roles-to-groups.py (server id + role-id->group table);
      bind to default-source-enrollment AND default-source-authentication (engine mode all).
   5. Test: private window -> https://$LIVE/ -> Discord -> portal; https://$BOT/ -> 403 unless you hold the
-     bot role; https://$LIVE/me/stream-key shows your username.
+     bot role; https://$LIVE/me shows your username and groups.
 
-LIVE PORTAL (https://$LIVE/)
+LIVE PORTAL (https://$LIVE/) — also hosts the OBS dock (https://$LIVE/dock/) and the timecode clock source.
   Deploy the page from the dev box: ./deployLive.sh (builds live/ and copies dist/ to /srv/live).
   WebRTC viewers and Go Live need UDP/TCP 8189 open (firewall rule included). Optional "is live" Discord
   post: LIVE_WEBHOOK_URL in .env (asked by configure.py --full).

authentik/brand/favicon.svg

@@ -0,0 +1,4 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="64" height="64" viewBox="0 0 64 64">
+  <rect width="64" height="64" rx="14" fill="#5865F2"/>
+  <path d="M14 16h13l5 24 5-24h13L40 52H24z" fill="#141414"/>
+</svg>

authentik/brand/login-bg.svg

@@ -0,0 +1,16 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="1920" height="1080" viewBox="0 0 1920 1080" preserveAspectRatio="xMidYMid slice">
+  <defs>
+    <radialGradient id="g" cx="30%" cy="20%" r="90%">
+      <stop offset="0" stop-color="#23264a"/>
+      <stop offset="0.55" stop-color="#141414"/>
+      <stop offset="1" stop-color="#0c0c0c"/>
+    </radialGradient>
+    <pattern id="grid" width="48" height="48" patternUnits="userSpaceOnUse">
+      <path d="M48 0H0v48" fill="none" stroke="#5865F2" stroke-opacity="0.08" stroke-width="1"/>
+    </pattern>
+  </defs>
+  <rect width="1920" height="1080" fill="url(#g)"/>
+  <rect width="1920" height="1080" fill="url(#grid)"/>
+  <circle cx="1500" cy="860" r="420" fill="#5865F2" fill-opacity="0.07"/>
+  <circle cx="300" cy="150" r="260" fill="#3d8bfd" fill-opacity="0.06"/>
+</svg>

authentik/brand/logo.svg

@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="400" height="96" viewBox="0 0 400 96" role="img" aria-label="Vanguard">
+  <rect x="4" y="8" width="80" height="80" rx="18" fill="#5865F2"/>
+  <path d="M22 26h16l6 30 6-30h16L54 72H34z" fill="#141414"/>
+  <text x="100" y="66" font-family="Inter, Segoe UI, Helvetica, Arial, sans-serif" font-weight="800" font-size="44" letter-spacing="3" fill="#e6e6e6">VANGUARD</text>
+</svg>

authentik/discord-roles-to-groups.py

@@ -1,18 +1,27 @@
 # Authentik expression policy: sync a member's Discord ROLES into Authentik GROUPS at every login.
 # Paste into: Customization -> Policies -> Create -> Expression Policy  (name: discord-roles-to-groups)
 # Bind to BOTH flows (after discord-guild-member): default-source-enrollment and default-source-authentication.
+# Tick "Execution logging" on the policy: everything below shows up in Events -> Logs -> policy_execution
+# under result.log_messages, and in `docker logs authentik-server`.
 # Requires the "guilds.members.read" scope on the Discord source (source -> Protocol settings -> Scopes:
 #   guilds guilds.members.read ). Existing users re-consent once at Discord.
 # The Authentik groups must already exist (Directory -> Groups). Role IDs: Discord Developer Mode ->
 #   Server Settings -> Roles -> right-click a role -> Copy Role ID.
+# Manual memberships are safe: the policy only removes groups it granted itself (user attribute discord_role_groups).
 
 GUILD_ID = "1387203044084551690"                        # <- your Discord server ID
-ROLE_TO_GROUP = {                                       # <- Discord role id : Authentik group name
+
-    "1387239222070612110": "bot-users",
+# Discord role id : (label used in the log, [Authentik groups this role grants])
-    "1515901670943490129": "bot-users",
+# One role may grant several groups, and several roles may grant the same group. Do NOT repeat a role id:
-#    "222222222222222222": "streamers",
+# this is a dict, so a second entry for the same id silently replaces the first.
-#    "333333333333333333": "editors",
+ROLES = {
-#    "444444444444444444": "admins",
+    "1387239222070612110": ("Discord Admins", ["bot-users", "admins"]),
+    "1515901670943490129": ("Lead Dev",       ["bot-users"]),
+    "1566623319027880086": ("Dev",            ["bot-users"]),
+#   "222222222222222222": ("Streamer",        ["streamers"]),
+#   "333333333333333333": ("Editor",          ["editors"]),
+#   "444444444444444444": ("Media admin",     ["media-admins"]),
+#   "555555555555555555": ("Git user",        ["git-users"]),
 }
 
 from authentik.core.models import Group
@@ -23,6 +32,7 @@ if not source or getattr(source, "provider_type", "") != "discord":
 
 connection = context.get("goauthentik.io/sources/connection")
 if not connection:
+    ak_logger.info("discord-roles: no source connection in context; nothing to sync")
     return True
 
 resp = requests.get(
@@ -31,25 +41,67 @@ resp = requests.get(
     timeout=10,
 )
 if resp.status_code != 200:
+    ak_logger.info(f"discord-roles: member lookup failed, HTTP {resp.status_code}: {resp.text[:200]}")
     ak_message(f"Could not read your Discord roles ({resp.status_code}); groups unchanged.")
     return True                                          # don't block login on a Discord hiccup
 
 roles = set(resp.json().get("roles", []))
-wanted = {g for r, g in ROLE_TO_GROUP.items() if r in roles}
-managed = set(ROLE_TO_GROUP.values())
 
+# ---- what Discord says, what we map, what we ignore ------------------------------------------------
+matched = {rid: ROLES[rid] for rid in roles if rid in ROLES}
+unmapped = sorted(roles - set(ROLES))
+wanted = {group for _, groups in matched.values() for group in groups}
+managed = {group for _, groups in ROLES.values() for group in groups}
+
+ak_logger.info(f"discord-roles: discord returned {len(roles)} role(s): {sorted(roles)}")
+for rid, (label, groups) in sorted(matched.items(), key=lambda item: item[1][0]):
+    ak_logger.info(f"discord-roles: MAPPED   role {rid} ({label}) -> {groups}")
+if unmapped:
+    ak_logger.info(f"discord-roles: ignored  {len(unmapped)} role(s) not in the table: {unmapped}")
+if not matched:
+    ak_logger.info(f"discord-roles: no mapped roles; the table knows {sorted(ROLES)}")
+ak_logger.info(f"discord-roles: groups wanted {sorted(wanted)}; managed by this policy {sorted(managed)}; "
+               f"withheld {sorted(managed - wanted)}")
+
+# A group named in the table but absent from Authentik can never be granted: say so instead of failing quietly.
+existing = {group.name: group for group in Group.objects.filter(name__in=managed)}
+absent = sorted(managed - set(existing))
+if absent:
+    ak_logger.info(f"discord-roles: WARNING these groups do not exist in Authentik and were skipped: {absent}")
+
+# ---- apply ------------------------------------------------------------------------------------------
 # Authentication flow: the user exists -> edit memberships directly.
+# Only memberships THIS policy granted earlier are ever removed (tracked in user.attributes["discord_role_groups"]),
+# so a group someone was added to by hand in Authentik survives role changes and re-logins.
 user = context.get("pending_user") or (request.user if request.user.is_authenticated else None)
 if user and user.pk:
-    for group in Group.objects.filter(name__in=managed):
+    attrs = user.attributes or {}
-        if group.name in wanted:
+    granted = set(attrs.get("discord_role_groups", []))
-            group.users.add(user)
+    added, removed = [], []
-        else:
+    for name, group in existing.items():
-            group.users.remove(user)
+        if name in wanted and name not in granted:
+            group.users.add(user); granted.add(name); added.append(name)
+        elif name in wanted:
+            group.users.add(user)                 # idempotent: make sure the membership is really there
+        elif name in granted:
+            group.users.remove(user)                  # role gone -> take back what the role had given
+            granted.discard(name); removed.append(name)
+    attrs["discord_role_groups"] = sorted(granted)
+    user.attributes = attrs
+    user.save()
+    ak_logger.info(f"discord-roles: authentication flow, user {user.username} (pk {user.pk}): "
+                   f"added {added or '[]'}, removed {removed or '[]'}, role-granted now {sorted(granted)}")
     return True
 
-# Enrollment flow: the user is created later by the user-write stage; hand it the groups to attach.
+# Enrollment flow: the user is created later by the user-write stage; hand it the groups to attach and
+# record them as role-granted so a later role removal can revoke exactly these. The flow plan is reached
+# through context["flow_plan"] on some versions and IS context on others, so write to whichever is there.
+attach = [group for name, group in existing.items() if name in wanted]
 plan = context.get("flow_plan")
-if plan is not None:
+target = plan.context if plan is not None else context
-    plan.context["groups"] = list(Group.objects.filter(name__in=wanted))
+target["groups"] = attach
+prompt = target.setdefault("prompt_data", {})
+prompt.setdefault("attributes", {})["discord_role_groups"] = sorted(wanted)
+ak_logger.info(f"discord-roles: enrollment flow via {'flow_plan' if plan is not None else 'policy context'}: "
+               f"attaching {[group.name for group in attach]}, recording {sorted(wanted)}")
 return True

authentik/vanguard-profile-scope-mapping.py

@@ -0,0 +1,38 @@
+# Authentik SCOPE MAPPING (not a policy): adds two things the apps behind the proxy need and the built-in
+# mappings do not provide.
+#
+# Paste into: Customization -> Property Mappings -> Create -> Scope Mapping
+#   Name:        Vanguard profile (nested groups + discord id)
+#   Scope name:  profile
+# Then tick it, NEXT TO the managed "authentik default OAuth Mapping: OpenID 'profile'", on every provider the
+# community uses: the "Vanguard sites" proxy provider, Jellyfin and Gitea (Advanced protocol settings -> Scopes).
+# Claims from several mappings are merged, so both run and neither needs removing. Proxy providers re-add the
+# managed one on every save, which is why this adds to it rather than replacing it.
+#
+# 1. groups: the managed mapping emits ak_groups, i.e. DIRECT memberships only, so a group inherited through a
+#    parent never reaches the apps. This emits only the inherited ones; merged with the managed mapping's direct
+#    ones the apps see the full set, with no duplicates.
+#
+# 2. discord_id: the Discord snowflake, taken from the source connection Authentik already stores when it links
+#    an account. It reaches the apps in X-Authentik-Jwt. The Discord bot uses it to identify the caller exactly,
+#    instead of matching an Authentik username against Discord names — names are mutable, reusable, and server
+#    nicknames are not unique, so name matching can both lock the right person out and let the wrong one in.
+#
+# Takes effect on the next login.
+
+DISCORD_SOURCE_SLUG = "discord"
+
+direct = set(request.user.ak_groups.all())
+inherited = sorted(group.name for group in request.user.all_groups() if group not in direct)
+
+discord_id = None
+for attribute in ("usersourceconnection_set", "source_connections"):      # the related name differs by version
+    related = getattr(request.user, attribute, None)
+    if related is None:
+        continue
+    connection = (related.filter(source__slug=DISCORD_SOURCE_SLUG).first() or related.first())
+    if connection is not None:
+        discord_id = getattr(connection, "identifier", None) or None
+    break
+
+return {"groups": inherited, "discord_id": discord_id}

bin/install.list

@@ -0,0 +1,21 @@
+# What gets installed into /usr/local/bin, and with which mode. ONE list, read by install.sh,
+# configure.py --apply and deploy.sh, because three hand-kept copies drifted: a push-to-deploy was
+# silently not installing vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script.
+# check.sh fails if a file here is missing, or if something executable in bin/ is neither listed nor
+# excluded below.
+0755 bin/normalize-sweep.sh
+0755 bin/trash-purge.sh
+0755 bin/dumpd.py
+0755 bin/vgstrip.py
+0755 bin/vgstitch.py
+0755 bin/vghosts-check.py
+0755 bin/vgdeband.sh
+0755 bin/vgstuck.sh
+0755 bin/vanguard-deadman.sh
+0755 bin/vanguard-alerts.sh
+# sourced, never executed
+0644 bin/vgmetrics.sh
+0644 bin/vgnotify.sh
+# NOT installed on the host, on purpose:
+#   exclude bin/mtx-on-ready.sh        mounted into the mediamtx container (docker-compose.yml)
+#   exclude bin/vgbot-merge-sqlite.py  one-off migration, run from the checkout

bin/mtx-on-ready.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+# MediaMTX runOnReady hook (runs INSIDE the mediamtx container; busybox sh + wget + ffmpeg from the -ffmpeg image).
+# Env from MediaMTX: MTX_PATH (user/Title). From compose: LIVE_WEBHOOK_URL, LIVE_HOST.
+# 1. optional Discord "is live" notice
+# 2. WebRTC audio mirror: browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only), and
+#    OBS sends AAC. For every stream whose audio is not already Opus, copy the video and transcode the audio to Opus
+#    into the sibling path "<user>/<Title>.webrtc". The live portal plays that path over WebRTC; recordings, HLS and
+#    the Discord notice stay on the original. Mirrors are published from 127.0.0.1 inside this container only, which
+#    is the only publisher dumpd accepts for ".webrtc" paths. Audio-only transcode: a few % of one core per stream.
+# MediaMTX kills this process when the source path goes away.
+case "$MTX_PATH" in *.webrtc) exit 0 ;; esac
+u="${MTX_PATH%%/*}"; t="${MTX_PATH#*/}"
+if [ -n "$LIVE_WEBHOOK_URL" ]; then
+  wget -q -O /dev/null --header="Content-Type: application/json" \
+    --post-data="{\"content\":\"🔴 **$u** is live: $t — https://$LIVE_HOST/\"}" "$LIVE_WEBHOOK_URL" 2>/dev/null &
+fi
+# which audio does the source carry? (API on loopback; "Opus" needs no mirror, no audio needs no mirror)
+tracks=""
+for i in 1 2 3 4 5; do
+  tracks=$(wget -q -O - "http://127.0.0.1:9997/v3/paths/get/$MTX_PATH" 2>/dev/null) && [ -n "$tracks" ] && break
+  sleep 1
+done
+case "$tracks" in
+  *Opus*) exit 0 ;;
+  *"MPEG-4 Audio"*|*"MPEG-1 Audio"*|*G711*|*G722*|*LPCM*|*AC-3*) ;;
+  *) exit 0 ;;                                   # no (recognised) audio: nothing to mirror
+esac
+echo "event=mirror stream=$MTX_PATH target=$MTX_PATH.webrtc audio=opus"
+n=0
+while [ $n -lt 10 ]; do                          # retry: the source may not be readable for the first second
+  ffmpeg -nostdin -hide_banner -loglevel warning -fflags nobuffer -flags low_delay \
+    -rtsp_transport tcp -i "rtsp://127.0.0.1:8554/$MTX_PATH" \
+    -map 0:v:0 -map 0:a:0 -c:v copy -c:a libopus -b:a 128k -ar 48000 -ac 2 -application lowdelay -frame_duration 20 \
+    -f rtsp -rtsp_transport tcp "rtsp://127.0.0.1:8554/$MTX_PATH.webrtc"
+  n=$((n+1)); sleep 2
+done

bin/normalize-sweep.sh

@@ -6,26 +6,73 @@
 # - anything else (HEVC, 10-bit, VFR, MKV, >1440p, >60fps): full-res re-encode -> name.mp4,
 #   raw source kept in a hidden .src/ subfolder next to it (Jellyfin/copyparty hide dot-dirs)
 # - streams/<user>/<Title>/.live/<ts>.mp4 (MediaMTX writes here, hidden from Jellyfin/copyparty while live):
-#   when finished -> flat MP4 published as streams/<user>/<Title>__<ts>.mp4 (one file per recording directly
+#   when finished -> the timecode overscan band (if the streamer had it on) is decoded and then CROPPED OFF in a
+#   re-encode (libx264, frame timing preserved, audio copied) so the published file is the game image + audio +
+#   a tmcd timecode track, nothing else; streams without the band are remuxed losslessly. Published as flat
+#   MP4 streams/<user>/<Title>__<ts>.mp4 (one file per recording directly
 #   in the streamer's folder: Jellyfin would merge several files inside one title folder into ONE episode
 #   with "versions"), the .live source deleted, the empty title folder removed.
 # - Syncthing folders (marker dir .stfolder in the folder root, receive-only): compliant files are left
 #   alone (zero copies); a non-compliant file is converted to name.mp4 (name.h264.mp4 if the raw is
 #   already .mp4), the raw's name is appended to .stlocal (an #include'd ignore list) so Syncthing
 #   drops it without flagging or re-pulling, then the raw is deleted. One copy on the server.
+# - reconnect fragments of ONE stream (same user, same title, fewer than STITCH_MAX_PARTS (500) files, gaps under
+#   STITCH_MAX_GAP_S) are first stitched back into one file by vgstitch.py (lossless concat; the arrival-time
+#   file names say where every piece sits), then published as one recording. Joins are recorded in the sidecar.
+#   Nothing in a stream's .live/ is touched until the stream is over (no open or recently written file).
+# - a LONE stream recording under MIN_LIVE_SECONDS (10) is a reconnect stub with nothing to join to: moved to
+#   trash, never published. Only .live/ files — clips are frequently a few seconds long and are never subject to this.
 # - failure: name.normalize-failed.log is written and the file is skipped until that log is deleted
 # Idempotent; safe every few minutes.
 set -uo pipefail
 
 ROOT="${MEDIA_ROOT:-/srv/media}"
 MINS_IDLE=3
+# Gap filling for STREAMS: re-encode to constant frame rate (duplicating frames across dropped-frame holes) when
+# the total missing time is >= this many ms. 0 = never (default; a 3 h re-encode is ~6 h of CPU on the E-2136).
+# Set via systemd Environment=FILL_GAPS_MS=... ; viewers already see held frames either way.
+FILL_GAPS_MS="${FILL_GAPS_MS:-0}"
+# Band removal re-encodes every stream that carried the timecode band. The raw (with band) is deleted after a
+# successful encode; KEEP_RAW_STREAMS=1 keeps it in the hidden .src/ instead (doubles disk per night).
+KEEP_RAW_STREAMS="${KEEP_RAW_STREAMS:-0}"
+# Encoder threads for every re-encode (clips and streams). 2 leaves the box responsive for Jellyfin/live viewers;
+# conversions take longer. systemd Environment=ENCODE_THREADS=... to change.
+ENCODE_THREADS="${ENCODE_THREADS:-2}"
+# STREAM recordings shorter than this are reconnect stubs, not recordings: every time a publisher drops and
+# comes back, MediaMTX opens a new file, and one bad night left 471 one- and two-second files in .live/.
+# They go to trash (3-day undo) instead of being published. Applies ONLY to .live/ — clips are often a few
+# seconds long on purpose and are never touched by this. 0 disables. systemd Environment=MIN_LIVE_SECONDS=…
+MIN_LIVE_SECONDS="${MIN_LIVE_SECONDS:-10}"
+# Stitching (vgstitch.py): fewer than this many fragments in one stream's .live/ are joined into one recording.
+# 500, not 20: on a lossy uplink MediaMTX restarts the recorder every ~10 s (a keyframe whose slices were lost
+# leaves an access unit with only SPS/PPS, which the recorder refuses), so a half-hour stream is ~180 same-
+# parameter files two seconds apart — exactly the case for stitching. The gap rule (STITCH_MAX_GAP_S seconds
+# between fragments = separate recordings) is the real guard. 0 parts disables stitching. systemd Environment=…
+STITCH_MAX_PARTS="${STITCH_MAX_PARTS:-500}"
+STITCH_MAX_GAP_S="${STITCH_MAX_GAP_S:-600}"
 MAX_W=2560; MAX_H=1440; MAX_FPS=60
 exec 9>"${LOCK_FILE:-/run/lock/normalize-sweep.lock}"
 flock -n 9 || exit 0
 
-log() { echo "$(date '+%F %T') $*"; }
+# Metrics (LOGGING.md): counters for dumpd's /metrics. Optional on purpose — if the helper is not installed
+# yet, these become no-ops and normalising carries on exactly as before.
+if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then
+  VG_MET_COMPONENT=sweep
+  . "$(dirname "$0")/vgmetrics.sh" && vgmet_init
+else
+  vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; }
+fi
+
+# logfmt (LOGGING.md): vglog <level> event=<verb> key=value ...   values with spaces must arrive pre-quoted
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=sweep %s\n' "$(date -Is)" "$lvl" "$*"; }
+log() { vglog info "msg=\"$*\""; }
 dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
 
+# A stream's fragments are one recording: join them before anything below looks at them one by one.
+if (( STITCH_MAX_PARTS > 0 )) && [[ -r "$(dirname "$0")/vgstitch.py" ]]; then        # installed side by side
+  python3 "$(dirname "$0")/vgstitch.py" "$ROOT" --max-parts "$STITCH_MAX_PARTS" --max-gap-s "$STITCH_MAX_GAP_S" --idle-min "$MINS_IDLE"
+fi
+
 # --- probe: prints "verdict reason" where verdict ∈ ok|remux|encode|skip ----------------------
 probe() {
   local f="$1" ext="${1##*.}"
@@ -39,7 +86,14 @@ probe() {
   afr=$(jq -r '[.streams[]|select(.codec_type=="video")][0].avg_frame_rate // "0/1"' <<<"$json")
   acodecs=$(jq -r '[.streams[]|select(.codec_type=="audio")|.codec_name]|unique|join(",")' <<<"$json")
   [[ $vcodec == none ]] && { echo "skip no-video"; return; }
-  local fps; fps=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$afr")
+  # Take the LOWER of r_frame_rate and avg_frame_rate. Either is unreliable alone: avg is garbage on a
+  # fragmented or looped recording (one reported 95 fps for a 30 fps source, which sent it down the fps-cap
+  # path and broke the encode), and r is a huge nominal value on genuine VFR. The lower of the two is the
+  # honest answer to "is this really above 60".
+  local fps; fps=$(awk -v rfr="$rfr" -v afr="$afr" 'BEGIN{
+      split(rfr, a, "/"); split(afr, b, "/")
+      r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
+      if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
   local reasons=()
   [[ $vcodec == h264 ]]            || reasons+=("codec=$vcodec")
   [[ $pix == yuv420p ]]            || reasons+=("pix=$pix")
@@ -61,21 +115,21 @@ probe() {
                med=d[int(n/2)]; if (med<=0) { print "cfr"; exit }
                bad=0; for(i=0;i<n;i++){ x=d[i]-med; if(x<0)x=-x; if (x > med*0.25) bad++ }
                print (bad > n*0.02) ? "vfr" : "cfr" }')
-  [[ $vfr == vfr ]] && { echo "encode vfr"; return; }
+  # streams: gaps from dropped frames are legitimate and re-encoding a 3 h file is hours of CPU — never VFR-encode
+  # them (the timecode sidecar lists the gaps instead). Clips (short, often genuinely VFR) still get CFR.
+  if [[ $vfr == vfr && $f != "$ROOT/streams/"* ]]; then echo "encode vfr"; return; fi
   local moov mdat; moov=$(grep -o '[0-9]*:moov' <<<"$head" | cut -d: -f1 | head -1); mdat=$(grep -o '[0-9]*:mdat' <<<"$head" | cut -d: -f1 | head -1)
   if [[ -z $moov || ( -n $mdat && $mdat -lt $moov ) ]]; then echo "remux index-last"; return; fi
   echo "ok"
 }
 
-find "$ROOT/streams" "$ROOT/clips" -type f \
-     \( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \
-     ! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null |
 while IFS= read -r -d '' src; do
   base="${src%.*}"; ext="${src##*.}"
-  live=0
+  live=0; stream=""
   if [[ $(basename "$(dirname "$src")") == .live ]]; then
     live=1
     titledir="$(dirname "$(dirname "$src")")"                       # streams/<user>/<Title>
+    stream="$(basename "$(dirname "$titledir")")/$(basename "$titledir")"   # user/Title, the log correlation key
     base="$(dirname "$titledir")/$(basename "$titledir")__$(basename "${src%.*}")"   # streams/<user>/<Title>__<ts>
   fi
   synced=0; stroot=""
@@ -84,15 +138,50 @@ while IFS= read -r -d '' src; do
     if [[ -d "$d/.stfolder" ]]; then synced=1; stroot="$d"; break; fi
     d="$(dirname "$d")"
   done
-  [[ -e "$base.normalize-failed.log" ]] && continue
+  if [[ -e "$base.normalize-failed.log" ]]; then
+    # Say so every run. Skipping silently meant a parked file produced no log line after its original
+    # failure, so Kibana could only ever show a count and never which recording. bin/vgstuck.sh triages it.
+    vglog info "event=parked file=\"$src\" stream=\"$stream\" log=\"$base.normalize-failed.log\""
+    continue
+  fi
   fuser -s "$src" 2>/dev/null && continue            # still being written
   [[ -e "$src" ]] || continue                          # dumped meanwhile
+  if (( live )); then
+    # The stream is still going while any sibling fragment is open or fresh: publishing this piece now would
+    # make it a separate recording that the stitcher can never join. Wait for the stream to end.
+    going=0
+    for sib in "$(dirname "$src")"/*.mp4; do
+      [[ -e $sib && $sib != "$src" ]] || continue
+      if fuser -s "$sib" 2>/dev/null || [[ -n $(find "$sib" -mmin -"$MINS_IDLE" 2>/dev/null) ]]; then going=1; break; fi
+    done
+    if (( going )); then vglog info "event=defer reason=\"stream still going\" file=\"$src\" stream=\"$stream\""; continue; fi
+  fi
+
+  if (( live && MIN_LIVE_SECONDS > 0 )); then
+    secs=$(dur "$src"); secs=${secs:-0}
+    if (( secs < MIN_LIVE_SECONDS )); then
+      night="$(date +%F)"; rel="${src#"$ROOT"/}"; dst="$ROOT/trash/$night/$rel"
+      mkdir -p "$(dirname "$dst")"
+      mv -f "$src" "$dst"
+      [[ -e "${src%.*}.json" ]] && mv -f "${src%.*}.json" "$(dirname "$dst")/"
+      rm -f "${src%.*}.stitch.json"
+      rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true
+      vgmet_add "jobs:discard:done:stream"
+      vglog info "event=discarded reason=too_short seconds=$secs min=$MIN_LIVE_SECONDS file=\"$src\" stream=\"$stream\" trash=\"$dst\""
+      continue
+    fi
+  fi
 
   read -r verdict reason <<<"$(probe "$src")"
+  # Keep what the PROBE said: the deband branch below overwrites $reason with its own text, which silently
+  # loses the "audio=<codec>" marker. That marker is the only thing telling the encode to convert the audio,
+  # so without this a Go Live stream with the timecode band AND Opus audio got -c:a copy and shipped Opus
+  # inside an MP4 — legal, but editors choke on it.
+  probe_reason="$reason"
   if (( live )); then
     [[ $verdict == skip ]] && continue
     [[ $verdict == ok ]] && verdict=remux          # must leave .live/ either way
-    out="$base.mp4"; [[ -e "$out" ]] && { log "exists  $out (leaving $src)"; continue; }
+    out="$base.mp4"; [[ -e "$out" ]] && { vglog info "event=skip reason=exists file=\"$out\" src=\"$src\""; continue; }
   else
     case "$verdict" in ok|skip) continue ;; esac
     out="$base.mp4"
@@ -103,29 +192,105 @@ while IFS= read -r -d '' src; do
     [[ -e "$out" ]] && continue
   fi
   tmp="$out.part"
-  log "$verdict  $src  ($reason)"
+  kind=clip; (( live )) && kind=stream           # metric label: stream encodes are the expensive ones
+  vglog info "event=verdict verdict=$verdict file=\"$src\" reason=\"$reason\" stream=\"$stream\""
+  tcargs=(); tcjson=""; band=0; deband=0
+  if (( live )); then
+    # timecode: decode the overscan strip (server clock) or fall back to the server arrival time in the filename
+    stem="$(basename "${src%.*}")"
+    arrival_ms=$(python3 - "$stem" <<'PYEOF'
+import re, sys
+from datetime import datetime
+from zoneinfo import ZoneInfo
+m = re.search(r"(\d{4})-(\d{2})-(\d{2})_(\d{2})-(\d{2})-(\d{2})(?:-(\d{1,6}))?", sys.argv[1])
+if not m: sys.exit(0)
+y, mo, d, h, mi, s, us = m.groups()
+us = int((us or "0").ljust(6, "0"))
+t = datetime(int(y), int(mo), int(d), int(h), int(mi), int(s), us, tzinfo=ZoneInfo("America/New_York"))
+print(int(t.timestamp() * 1000))
+PYEOF
+)
+    eval "$(python3 /usr/local/bin/vgstrip.py plan "$src" "${arrival_ms:-}" 2>/dev/null || true)"
+    if [[ -n ${TC_TIMECODE:-} ]]; then
+      tcargs=(-timecode "$TC_TIMECODE" -metadata "creation_time=$TC_CREATION" -write_tmcd 1)
+      tcjson="${TC_JSON:-}"
+      vglog info "event=timecode file=\"$src\" stream=\"$stream\" source=$TC_SOURCE timecode=\"$TC_TIMECODE\""
+    fi
+    # the band has done its job once decoded: remove it from the picture (re-encode with a crop)
+    if [[ ${TC_SOURCE:-} == strip && ${TC_BAND:-0} -gt 0 ]]; then
+      band=$TC_BAND; deband=1; verdict=encode; reason="remove ${band}px timecode band"
+      vglog info "event=deband file=\"$src\" stream=\"$stream\" band_rows=$band"
+    fi
+    # opt-in gap filling: switch this stream from remux to a CFR re-encode when the holes add up to enough
+    if (( FILL_GAPS_MS > 0 )) && [[ $verdict == remux && -n $tcjson ]]; then
+      missing=$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get("gaps",{}).get("total_missing_ms",0))' "$tcjson" 2>/dev/null || echo 0)
+      if (( missing >= FILL_GAPS_MS )); then
+        verdict=encode; reason="fill-gaps ${missing}ms"; vglog info "event=fill_gaps file=\"$src\" stream=\"$stream\" missing_ms=$missing"
+      fi
+    fi
+  fi
+  job_t0=$(vgmet_time)
+  err=$(mktemp /tmp/vgsweep-err.XXXXXX)          # ffmpeg's own words; rc= alone is not diagnosable
   if [[ $verdict == remux ]]; then
     nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
-      -map 0 -c copy -movflags +faststart -f mp4 "$tmp"
+      -map 0 -c copy "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err"
   else
     # scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC
     vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2"
-    fpsargs=(); fpsnow=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$(ffprobe -v error -select_streams v:0 -show_entries stream=avg_frame_rate -of csv=p=0 "$src")")
+    (( deband )) && vf="crop=iw:ih-${band}:0:0,$vf"          # drop the band rows first (band is even, so sizes stay even)
+    fpsargs=()
+    read -r fps_r fps_a < <(ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate,avg_frame_rate -of csv=p=0 "$src" | tr ',' ' ')
+    fpsnow=$(awk -v rfr="${fps_r:-0/1}" -v afr="${fps_a:-0/1}" 'BEGIN{
+        split(rfr, a, "/"); split(afr, b, "/")
+        r = (a[2] > 0) ? a[1]/a[2] : 0; v = (b[2] > 0) ? b[1]/b[2] : 0
+        if (r > 0 && v > 0) printf "%.3f", (r < v) ? r : v; else printf "%.3f", (r > 0) ? r : v }')
     awk "BEGIN{exit !($fpsnow > $MAX_FPS + 0.01)}" && fpsargs=(-r "$MAX_FPS")
+    # streams keep their exact frame timing (gaps stay gaps unless FILL_GAPS_MS asked for CFR) and their AAC audio as-is
+    fpsmode=cfr; aargs=(-c:a aac -b:a 160k)
+    if (( live )) && [[ $reason != fill-gaps* ]]; then fpsmode=passthrough; fi
+    # Capping the rate means re-timing, which is by definition not passthrough. ffmpeg refuses the pair
+    # outright ("One of -r/-fpsmax was specified together a non-CFR -vsync/-fps_mode") and fails with
+    # "Error opening output files: Invalid argument", so the cap has to win.
+    (( ${#fpsargs[@]} )) && fpsmode=cfr
+    if (( live )) && [[ $probe_reason != *audio=* ]]; then aargs=(-c:a copy); fi
     nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
       -map 0:v:0 -map '0:a?' \
-      -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high \
+      -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "$ENCODE_THREADS" \
-      -vf "$vf" -fps_mode cfr "${fpsargs[@]}" \
+      -vf "$vf" -fps_mode "$fpsmode" "${fpsargs[@]}" \
-      -c:a aac -b:a 160k \
+      "${aargs[@]}" \
-      -movflags +faststart -f mp4 "$tmp"
+      "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err"
   fi
   rc=$?
+  job_secs=$(vgmet_since "$job_t0")
   d1=$(dur "$src"); d2=$(dur "$tmp")
   if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
+    vgmet_add "jobs:$verdict:done:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs"
+    (( deband )) && vgmet_add deband 1
     if (( live )); then
       mv -f "$tmp" "$out"; chown 1000:1000 "$out" 2>/dev/null
-      rm -f "$src"; rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true   # drop .live and the empty title folder
+      stitchjson="${src%.*}.stitch.json"               # vgstitch.py: parts, offsets and the time missing at each join
-      log "done   $out"; continue
+      if [[ -n $tcjson || -e $stitchjson ]]; then      # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples
+        python3 - "$tcjson" "$stitchjson" > "${out%.*}.json" <<'PYEOF'
+import json, sys, os
+tc, sj = sys.argv[1], sys.argv[2]
+d = json.loads(tc) if tc else {}
+if os.path.exists(sj):
+    d["stitched"] = json.load(open(sj))
+    # the strip decode's drift figure spans the joins too, so it is not clock drift on a stitched file
+    if "drift_ms" in d: d["drift_ms_note"] = "includes collapsed joins; see stitched.joins"
+print(json.dumps(d))
+PYEOF
+        chown 1000:1000 "${out%.*}.json" 2>/dev/null; rm -f "$stitchjson"
+      fi
+      if [[ $verdict == encode ]] && (( ! deband || KEEP_RAW_STREAMS )); then   # gap-filled re-encode (or asked to): keep the raw in the hidden .src/
+        srcdir="$(dirname "$out")/.src"; mkdir -p "$srcdir"; chown 1000:1000 "$srcdir" 2>/dev/null
+        mv -f "$src" "$srcdir/$(basename "$out" .mp4).raw.mp4"
+      else
+        rm -f "$src"                                   # band removal: the cropped file IS the recording; raw-with-band is not kept
+      fi
+      rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true   # drop .live and the empty title folder
+      rm -f "$err"
+      vglog info "event=published file=\"$out\" stream=\"$stream\" verdict=$verdict"; continue
     elif (( synced )); then
       rel_src="${src#"$stroot"/}"; rel_out="${out#"$stroot"/}"
       { echo "/$rel_out"; echo "/$rel_src"; } >> "$stroot/.stlocal"   # ignore our output and the raw
@@ -141,10 +306,16 @@ while IFS= read -r -d '' src; do
       mv -f "$tmp" "$out"
     fi
     chown 1000:1000 "$out" 2>/dev/null
-    log "done   $out"
+    vglog info "event=done file=\"$out\" verdict=$verdict"
   else
     rm -f "$tmp"
-    { echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)"; } > "$base.normalize-failed.log"
+    vgmet_add "jobs:$verdict:failed:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs"
-    log "FAILED $src (see $base.normalize-failed.log)" >&2
+    { echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)"
+      echo "--- ffmpeg stderr ---"; tail -40 "$err" 2>/dev/null; } > "$base.normalize-failed.log"
+    why=$(tail -1 "$err" 2>/dev/null | tr -d '"' | cut -c1-200)
+    vglog error "event=failed file=\"$src\" stream=\"$stream\" verdict=$verdict rc=$rc reason=\"${why:-no stderr}\" log=\"$base.normalize-failed.log\"" >&2
   fi
-done
+  rm -f "$err"
+done < <(find "$ROOT/streams" "$ROOT/clips" -type f \
+     \( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \
+     ! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null)

bin/vanguard-alerts.sh

@@ -0,0 +1,166 @@
+#!/bin/bash
+# Deliver Kibana's alerts to Discord. Runs from a systemd timer every minute; curl and python only.
+#
+# WHY THIS EXISTS
+# On a basic license Kibana's only connector types are .index and .server-log — every connector that leaves
+# the stack is Gold or above. So Kibana rules write their alerts into ONE Elasticsearch index (the alert bus,
+# created by observability/kibana-setup.py) and this script is the delivery side. Alerts are therefore
+# ordinary documents: queryable, replayable, and visible in Discover alongside the logs that caused them.
+#
+# Heartbeat documents (kind=heartbeat) are deliberately NOT delivered. They exist for bin/vanguard-deadman.sh
+# to notice their absence, which is the only way to catch the alerting pipeline itself dying.
+#
+# Repeat suppression is a safety net, not the primary throttle: a rule should use "On status change" or its
+# own throttle in Kibana. This stops a misconfigured rule turning into a message every minute, and it is per
+# rule name, so one noisy rule never silences another.
+#
+# Exit status: 0 when there was nothing to send or everything was sent; 1 when alerts existed and could not be
+# delivered, so `systemctl status vanguard-alerts` and the journal both carry the failure.
+set -uo pipefail
+
+# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
+# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
+# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
+if [[ -r /etc/vanguard/alerts.env ]]; then
+  while IFS='=' read -r _k _v; do
+    [[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
+    [[ -n ${!_k:-} ]] || export "$_k=$_v"
+  done < /etc/vanguard/alerts.env
+fi
+
+ES="${ES_URL:-http://127.0.0.1:9200}"
+ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
+INDEX="${ALERT_INDEX:-vanguard-alerts}"
+STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}"
+WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
+SUPPRESS_S="${ALERT_SUPPRESS_S:-900}"
+BATCH="${ALERT_BATCH:-50}"
+
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=alerts %s\n' "$(date -Is)" "$lvl" "$*"; }
+es_curl() { if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi; }
+
+state_get() { [[ -r $STATE ]] || return 0; grep -m1 "^$1=" "$STATE" 2>/dev/null | cut -d= -f2-; }
+state_set() {
+  local key="$1" value="$2" tmp
+  install -d "$(dirname "$STATE")" 2>/dev/null || return 0
+  tmp=$(mktemp "${STATE}.XXXXXX" 2>/dev/null) || return 0
+  { [[ -r $STATE ]] && grep -v "^${key}=" "$STATE"; printf '%s=%s\n' "$key" "$value"; } > "$tmp" 2>/dev/null
+  mv -f "$tmp" "$STATE" 2>/dev/null || rm -f "$tmp"
+}
+
+# Delivery (and the mention handling) is shared with the deadman. If the helper is missing — a half-finished
+# deploy — fall back to posting without mentions rather than losing alerts entirely.
+if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
+  . "$(dirname "$0")/vgnotify.sh"
+else
+  vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
+  vg_post() {
+    local url; url=$(vg_webhook) || return 1
+    curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
+      --data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
+      "$url"
+  }
+fi
+webhook=$(vg_webhook 2>/dev/null || true)
+
+now=$(date +%s)
+cursor=$(state_get cursor); cursor="${cursor:-0}"
+# Record every run, not just runs that did something. The deadman watches this file's age to tell whether
+# delivery is still happening at all, and almost every run is quiet: without this the file went stale within
+# minutes and the deadman reported the timer as dead, then recovered the moment an alert happened to flow.
+state_set last_run "$now"
+
+# Everything newer than the cursor that is not a heartbeat, oldest first so the cursor advances safely.
+query=$(printf '{"size":%d,"sort":[{"@timestamp":"asc"}],"query":{"bool":{"filter":[{"range":{"@timestamp":{"gt":%s,"format":"epoch_millis"}}}],"must_not":[{"term":{"kind":"heartbeat"}}]}}}' "$BATCH" "$cursor")
+response=$(es_curl -m 15 -H 'Content-Type: application/json' "$ES/$INDEX/_search" -d "$query")
+
+if [[ -z $response ]]; then
+  vglog error 'event=unreachable msg="Elasticsearch did not answer; cannot read the alert bus"' >&2
+  exit 1
+fi
+# A missing index just means no rule has fired yet. Not a fault, and not worth a line every minute.
+if grep -q 'index_not_found_exception' <<<"$response"; then
+  exit 0
+fi
+
+# Retention. A heartbeat document arrives every minute, so without this the alert bus grows without bound.
+# At most once every 6 h: a delete_by_query is cheap, but not worth doing sixty times an hour. Deliberately
+# before the "nothing new" exit below, because most runs have nothing to deliver and would skip it.
+retention_days="${ALERT_RETENTION_DAYS:-90}"
+last_purge=$(state_get purged); last_purge="${last_purge:-0}"
+if (( now - last_purge > 21600 )); then
+  purge=$(es_curl -m 120 -H 'Content-Type: application/json' \
+            -XPOST "$ES/$INDEX/_delete_by_query?conflicts=proceed" \
+            -d "{\"query\":{\"range\":{\"@timestamp\":{\"lt\":\"now-${retention_days}d\"}}}}")
+  gone=$(printf '%s' "${purge:-}" | python3 -c 'import json,sys
+try: print(json.load(sys.stdin).get("deleted", 0))
+except Exception: print("?")')
+  state_set purged "$now"
+  vglog info "event=purged older_than=${retention_days}d deleted=${gone:-?}"
+fi
+
+rows=$(printf '%s' "$response" | python3 -c '
+import json, sys
+from datetime import datetime
+try:
+    data = json.load(sys.stdin)
+except Exception:
+    sys.exit(0)
+for hit in data.get("hits", {}).get("hits", []):
+    source = hit.get("_source", {})
+    stamp = str(source.get("@timestamp", ""))
+    try:
+        when = datetime.fromisoformat(stamp.replace("Z", "+00:00"))
+        millis = int(when.timestamp() * 1000)
+    except Exception:
+        continue
+    def clean(value, limit):
+        return " ".join(str(value).split())[:limit] or "-"
+    print("\t".join((str(millis),
+                     clean(source.get("rule", "unnamed rule"), 120),
+                     clean(source.get("level", "warn"), 12),
+                     clean(source.get("message", json.dumps(source)[:400]), 1200))))
+')
+
+[[ -z $rows ]] && exit 0
+
+# One per line: "a=0 b=0 now" would make bash run `now` as a COMMAND with a and b as temporary
+# assignments, leaving every counter unset for the rest of the script.
+sent=0
+suppressed=0
+failed=0
+while IFS=$'\t' read -r millis rule level message; do
+  [[ -z ${millis:-} ]] && continue
+  last=$(state_get "sent:$rule"); last="${last:-0}"
+  if (( now - last < SUPPRESS_S )); then
+    suppressed=$((suppressed + 1))
+    vglog info "event=suppressed rule=\"$rule\" within=${SUPPRESS_S}s"
+    state_set cursor "$millis"
+    continue
+  fi
+  case "$level" in
+    error|critical) mark="[ERROR]" ;;
+    warn|warning)   mark="[WARN]" ;;
+    *)              mark="[$level]" ;;
+  esac
+  text="$mark $rule — $message"
+  if [[ -z $webhook ]]; then
+    vglog warn "event=undeliverable rule=\"$rule\" msg=\"no Discord webhook configured; alert only in the journal\"" >&2
+    vglog warn "event=alert rule=\"$rule\" level=$level msg=\"$message\"" >&2
+    failed=$((failed + 1))
+    state_set cursor "$millis"
+    continue
+  fi
+  if vg_post "$level" "$text"; then
+    sent=$((sent + 1)); state_set "sent:$rule" "$now"
+    vglog info "event=delivered rule=\"$rule\" level=$level mention=$([[ -n ${ALERT_MENTION:-} && ,${ALERT_MENTION_LEVELS:-error}, == *,$level,* ]] && echo yes || echo no)"
+  else
+    failed=$((failed + 1))
+    vglog error "event=delivery_failed rule=\"$rule\" level=$level" >&2
+  fi
+  state_set cursor "$millis"
+done <<<"$rows"
+
+vglog info "event=run sent=$sent suppressed=$suppressed failed=$failed"
+(( failed > 0 )) && exit 1
+exit 0

bin/vanguard-deadman.sh

@@ -0,0 +1,179 @@
+#!/bin/bash
+# The deadman: the one check that does not depend on the Elastic stack being alive.
+#
+# Kibana tells you when something on the box is wrong. Nothing inside Elasticsearch, Kibana or the Beats can
+# tell you that they are the thing that broke, and silence looks exactly like health. So this runs on the HOST
+# from a systemd timer, uses nothing but curl, and posts straight to a Discord webhook.
+#
+# It checks six things, in the order they go wrong in practice:
+#   1. Elasticsearch answers and the cluster is not red
+#   2. no index has been flipped to read-only by the disk watermark — the classic Elastic failure, because it
+#      does NOT clear itself when space is freed, so ingestion stays silently dead until someone resets it
+#   3. Kibana reports itself available (it is the only UI; if it is down you are blind)
+#   4. Filebeat has actually written a document recently — "running" is not the same as "shipping"
+#   5. Metricbeat likewise
+#   6. the heartbeat a Kibana rule posts to dumpd is fresh, which proves rule evaluation AND delivery work
+#   …and the disks, because that is the failure this box is most likely to die of, and it must be reported
+#   even when the thing that would normally report it is what broke.
+#
+# State in /var/lib/vanguard/deadman.state: it announces a change, not the same thing every five minutes,
+# and says so once when everything recovers.
+#
+# No webhook configured? The checks still run and still print to the journal, and the unit exits non-zero
+# when it found something it could not report — so `systemctl is-failed vanguard-deadman.service` and
+# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
+set -uo pipefail
+
+# systemd passes these in via EnvironmentFile. Load the same file when run by hand, or a manual run behaves
+# differently from the timer — which cost a confusing round of "the mention is configured but never appears".
+# Anything already in the environment wins, so `ALERT_SUPPRESS_S=0 vanguard-alerts.sh` still overrides.
+if [[ -r /etc/vanguard/alerts.env ]]; then
+  while IFS='=' read -r _k _v; do
+    [[ -z ${_k:-} || ${_k:0:1} == '#' ]] && continue
+    [[ -n ${!_k:-} ]] || export "$_k=$_v"
+  done < /etc/vanguard/alerts.env
+fi
+
+ES="${ES_URL:-http://127.0.0.1:9200}"
+# Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes
+# this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file
+# is missing, the calls below get 401 and the deadman reports Elasticsearch as unreachable — which is correct,
+# because from its point of view it is. Kibana needs nothing: its anonymous provider signs callers in.
+ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
+es_curl() {
+  if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi
+}
+KIBANA="${KIBANA_URL:-http://127.0.0.1:5601}"
+WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
+ALERT_INDEX="${ALERT_INDEX:-vanguard-alerts}"            # the alert bus Kibana's rules write into
+ALERT_STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}"  # touched by every vanguard-alerts.sh run
+ALERT_STATE_MAX_S="${ALERT_STATE_MAX_S:-900}"
+HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-900}"
+STALE_MAX_S="${BEAT_STALE_MAX_S:-900}"
+STATE="${DEADMAN_STATE:-/var/lib/vanguard/deadman.state}"
+DISK_PCT="${DEADMAN_DISK_PCT:-90}"
+
+problems=()
+note() { problems+=("$1"); }
+jq_get() { python3 -c 'import json,sys
+try: d=json.load(sys.stdin)
+except Exception: sys.exit(1)
+for k in sys.argv[1].split("."):
+    d = d.get(k) if isinstance(d, dict) else None
+    if d is None: sys.exit(1)
+print(d)' "$1" 2>/dev/null; }
+
+# 1. Elasticsearch
+health=$(es_curl -m 8 "$ES/_cluster/health" || true)
+if [[ -z $health ]]; then
+  note "Elasticsearch is not answering on ${ES#http://} — nothing is being stored"
+else
+  status=$(printf '%s' "$health" | jq_get status)
+  case "$status" in
+    green|yellow) ;;
+    red) note "Elasticsearch cluster health is RED: some index has no usable shard" ;;
+    *)   note "Elasticsearch health is unreadable (got: ${status:-nothing})" ;;
+  esac
+
+  # 2. read-only indices. Elastic sets this when the disk passes the flood watermark and never unsets it.
+  blocked=$(es_curl -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
+            | python3 -c 'import json,sys
+try: d=json.load(sys.stdin)
+except Exception: sys.exit(0)
+bad=[n for n,v in d.items() if str(v.get("settings",{}).get("index",{}).get("blocks",{}).get("read_only_allow_delete","")).lower()=="true"]
+print(len(bad))' 2>/dev/null || echo 0)
+  (( ${blocked:-0} > 0 )) && note "$blocked Elasticsearch index/indices are READ-ONLY from the disk watermark; ingestion is dead until this is reset (see README 5h)"
+
+  # 4 and 5. are the Beats actually writing?
+  for beat in filebeat metricbeat; do
+    count=$(es_curl -m 8 -H 'Content-Type: application/json' \
+              "$ES/${beat}-*/_count" \
+              -d "{\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-${STALE_MAX_S}s\"}}}}" | jq_get count)
+    if [[ -z ${count:-} ]]; then
+      note "cannot tell whether $beat is shipping (no index yet, or the query failed)"
+    elif (( count == 0 )); then
+      note "$beat has written nothing in the last $((STALE_MAX_S/60)) minutes — it may be up but not shipping"
+    fi
+  done
+fi
+
+# 3. Kibana
+level=$(curl -s -m 8 "$KIBANA/api/status" | jq_get status.overall.level)
+[[ ${level:-} == available ]] || note "Kibana is not available (${level:-no answer}) — the only UI is down"
+
+# 6. the heartbeat. A Kibana rule writes a kind=heartbeat document into the alert index every minute (see
+# observability/kibana-setup.py). Its absence is the ONLY thing that catches the alerting pipeline itself
+# dying: if Kibana, its rule engine or the index connector stops, no ordinary alert can report it, because
+# the thing that would report it is what broke. Checked in Elasticsearch, not as a file: a basic licence has
+# no webhook connector, so nothing posts to dumpd any more.
+if [[ -n $health ]]; then
+  beats=$(es_curl -m 8 -H 'Content-Type: application/json' "$ES/$ALERT_INDEX/_count" \
+            -d "{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"kind\":\"heartbeat\"}},{\"range\":{\"@timestamp\":{\"gte\":\"now-${HEARTBEAT_MAX_S}s\"}}}]}}}")
+  if grep -q 'index_not_found_exception' <<<"${beats:-}"; then
+    note "the alert index $ALERT_INDEX does not exist — the Kibana connector and heartbeat rule have not been created (observability/kibana-setup.py)"
+  else
+    hb=$(printf '%s' "${beats:-}" | jq_get count)
+    if [[ -z ${hb:-} ]]; then
+      note "cannot read the alert heartbeat (query against $ALERT_INDEX failed)"
+    elif (( hb == 0 )); then
+      note "no alert heartbeat in the last $((HEARTBEAT_MAX_S/60)) minutes — Kibana's rule engine or the index connector has stopped, so NO alert can reach Discord"
+    fi
+  fi
+fi
+
+# 7. is the delivery side running? The heartbeat proves Kibana still writes alerts; this proves something
+# still takes them to Discord. Both halves have to be alive for an alarm to arrive.
+if [[ -f $ALERT_STATE ]]; then
+  age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") ))
+  (( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered"
+fi
+# Freshness above proves the timer fires; this proves the run finished. A unit that starts every minute and
+# dies immediately would otherwise look alive, because the file is written at the start of the run.
+if systemctl is-failed --quiet vanguard-alerts.service 2>/dev/null; then
+  note "vanguard-alerts.service is in a failed state — $(systemctl show -p Result --value vanguard-alerts.service 2>/dev/null); alerts are written but not delivered"
+fi
+
+# the disks, independently of any agent
+while read -r pct mount; do
+  (( pct >= DISK_PCT )) && note "$mount is ${pct}% full"
+done < <(df --output=pcent,target /srv / 2>/dev/null | tail -n +2 | tr -d '%' | awk '{print $1, $2}')
+
+# ---- report only on change ---------------------------------------------------------------------------
+now="$(IFS='|'; echo "${problems[*]-}")"
+mkdir -p "$(dirname "$STATE")"
+was="$(cat "$STATE" 2>/dev/null || true)"
+printf '%s' "$now" > "$STATE"
+[[ $now == "$was" ]] && exit 0
+
+# Delivery and mention handling are shared with vanguard-alerts.sh. A missing helper falls back to a plain
+# post: losing the mention is acceptable, losing the alert is not.
+if [[ -r "$(dirname "$0")/vgnotify.sh" ]]; then
+  . "$(dirname "$0")/vgnotify.sh"
+else
+  vg_webhook() { [[ -r $WEBHOOK_FILE ]] && tr -d '[:space:]' < "$WEBHOOK_FILE" | grep '^https://'; }
+  vg_post() {
+    local url; url=$(vg_webhook) || return 1
+    curl -s -o /dev/null -m 15 -X POST -H 'Content-Type: application/json' \
+      --data "$(VG_T="$2" python3 -c 'import json,os; print(json.dumps({"content": os.environ["VG_T"][:1900]}))')" \
+      "$url"
+  }
+fi
+webhook=$(vg_webhook 2>/dev/null || true)
+
+# A deadman alarm is the one that must wake somebody: it means no other alert can get through. Recovery is
+# sent at info level so good news never pings.
+post() { vg_post "${2:-error}" "$1"; }
+
+if [[ -z $now ]]; then
+  echo "deadman: recovered — Elastic, Kibana, both Beats and the disks are healthy"
+  post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." info || true
+  exit 0
+fi
+
+printf 'deadman: %s\n' "${problems[@]}" >&2
+body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside Elastic):"
+for item in "${problems[@]}"; do body+=$'\n'"• $item"; done
+body+=$'\n'"Until this clears, assume no other alert can reach you."
+if post "$body"; then exit 0; fi
+[[ -z $webhook ]] && echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2
+exit 1

bin/vgbot-merge-sqlite.py

@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+"""Merge the Discord bot's old SQLite database into its live MariaDB WITHOUT losing either side.
+
+The bot rebuilds part of its state from Discord on every start, so after switching to MariaDB some tables
+hold fresh rows while the old SQLite file holds months of history. Neither is a superset. This merges them
+row by row and, where a row exists in both, column by column with an explicit rule per column.
+
+    vgbot-merge-sqlite.py --source /data/migration-source.db                 # report only, touches nothing
+    vgbot-merge-sqlite.py --source /data/migration-source.db --apply         # one transaction, rolls back on error
+
+Run it with the gateway stopped (docker compose stop discord-bot) so nothing writes underneath it.
+Idempotent: running it twice changes nothing the second time.
+
+Rules, and why:
+  min / max              ISO-8601 text, so lexicographic order is chronological. "min" keeps the earliest
+                         known date, which is what makes joined-at, rank-since and tracking-start survive a
+                         rebuild that reset them to today. "max" keeps the most recent activity.
+  dest                   the live row wins: it reflects the server as it is right now.
+  source                 the old row wins.
+  *_notnull              prefer that side unless it is NULL, then take the other.
+  source_unless_default  take the old value unless it is the schema's placeholder.
+Every non-key column must appear in the rules; an unclassified column stops the run rather than being guessed.
+"""
+import argparse, sqlite3, sys
+from pathlib import Path
+from sqlalchemy import MetaData, create_engine, inspect, select, func
+
+# ---- how a row present on BOTH sides is resolved ------------------------------------------------------
+ROW_RULE = {                      # whole row chosen by comparing one column (single-row marker tables)
+    "activity_monitor":    ("max", "last_check_at"),      # the later check wins
+    "membership_tracking": ("min", "started_at"),         # keep the original tracking start, not tonight's
+}
+COL_RULE = {
+    "member_activity": {"tracking_started_at": "min", "last_message_at": "max", "last_voice_at": "max"},
+    "member_profiles": {"first_seen_at": "min", "discord_joined_at": "min_notnull", "updated_at": "max",
+                        "username": "dest", "display_name": "dest", "current_rank": "dest", "is_present": "dest"},
+    "member_rank_tenure": {"started_at": "min", "joined_at": "min_notnull",
+                           "role_id": "dest", "active": "dest", "source": "dest"},
+    "membership_state": {"joined_at": "min", "observed_at": "max", "present": "dest"},
+    "recruits": {"recruit_started_at": "min", "discord_joined_at": "min_notnull",
+                 "recruited_by": "source_notnull", "invite_code": "source_notnull",
+                 "invite_source": ("source_unless_default", "unknown"),
+                 "activity_override": "source_notnull", "promoted_at": "dest_notnull",
+                 "username": "dest", "display_name": "dest", "status": "dest",
+                 "current_rank": "dest", "is_active": "dest"},
+    "voice_channel_departures": {"left_at": "max", "guild_id": "dest", "user_id": "dest", "user_name": "dest"},
+}
+# Append-only logs: the id is a surrogate, so rows are matched on a natural key and inserted with a new id.
+APPEND_KEY = {
+    "membership_events": ["user_id", "event_type", "joined_at"],   # also a UNIQUE constraint in the schema
+    "profile_history":   ["user_id", "event", "created_at", "details"],
+    "recruit_history":   ["user_id", "event", "recorded_at"],
+}
+SKIP = {"schema_migrations"}      # the bot owns this; never copied
+
+
+def pick(rule, source_value, dest_value):
+    if isinstance(rule, tuple) and rule[0] == "source_unless_default":
+        return dest_value if source_value in (None, rule[1]) else source_value
+    if rule == "dest": return dest_value
+    if rule == "source": return source_value
+    if rule == "source_notnull": return dest_value if source_value is None else source_value
+    if rule == "dest_notnull": return source_value if dest_value is None else dest_value
+    if rule in ("min", "max"):
+        if source_value is None: return dest_value
+        if dest_value is None: return source_value
+        return (min if rule == "min" else max)(source_value, dest_value)
+    if rule == "min_notnull":
+        if source_value is None: return dest_value
+        if dest_value is None: return source_value
+        return min(source_value, dest_value)
+    raise ValueError(f"unknown rule {rule!r}")
+
+
+def source_tables(connection):
+    names = [r[0] for r in connection.execute(
+        "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name")]
+    out = {}
+    for name in names:
+        if name in SKIP: continue
+        info = list(connection.execute(f'PRAGMA table_info("{name}")'))
+        out[name] = {"columns": [row[1] for row in info],
+                     "key": [row[1] for row in info if row[5]]}
+    return out
+
+
+def plan(source_path, url, apply=False):
+    source = sqlite3.connect(Path(source_path).resolve().as_uri() + "?mode=ro", uri=True)
+    source.row_factory = sqlite3.Row
+    tables = source_tables(source)
+    engine = create_engine(url)
+    metadata = MetaData()
+    metadata.reflect(bind=engine, only=[n for n in tables if n in set(inspect(engine).get_table_names())])
+    report, problems = [], []
+
+    with engine.begin() as target:                      # one transaction; an exception rolls everything back
+        for name, meta in tables.items():
+            if name not in metadata.tables:
+                problems.append(f"{name}: not in the destination schema"); continue
+            table = metadata.tables[name]
+            key = meta["key"] or APPEND_KEY.get(name)
+            rows = [dict(r) for r in source.execute(f'SELECT * FROM "{name}"')]
+            if not rows:
+                report.append((name, 0, 0, 0, "source empty")); continue
+
+            # unclassified columns are a stop, not a guess
+            if name in COL_RULE:
+                managed = set(COL_RULE[name]) | set(meta["key"])
+                missing = [c for c in meta["columns"] if c not in managed]
+                if missing: problems.append(f"{name}: no rule for {missing}"); continue
+
+            if name in APPEND_KEY:
+                natural = APPEND_KEY[name]
+                existing = {tuple(r) for r in target.execute(select(*[table.c[c] for c in natural]))}
+                fresh = [r for r in rows if tuple(r[c] for c in natural) not in existing]
+                if apply and fresh:
+                    target.execute(table.insert(), [{k: v for k, v in r.items() if k != "id"} for r in fresh])
+                report.append((name, len(fresh), 0, len(rows) - len(fresh), f"appended on {'+'.join(natural)}"))
+                continue
+
+            dest = {tuple(r._mapping[c] for c in key): dict(r._mapping)
+                    for r in target.execute(select(table))} if key else {}
+            inserted = updated = same = 0
+            for row in rows:
+                ident = tuple(row[c] for c in key)
+                current = dest.get(ident)
+                if current is None:
+                    if apply: target.execute(table.insert(), [row])
+                    inserted += 1
+                    continue
+                if name in ROW_RULE:
+                    how, column = ROW_RULE[name]
+                    winner = pick(how, row[column], current[column])
+                    merged = row if winner == row[column] and row[column] != current[column] else None
+                elif name in COL_RULE:
+                    merged = {c: pick(COL_RULE[name][c], row[c], current[c]) for c in COL_RULE[name]}
+                    merged = None if all(merged[c] == current[c] for c in merged) else merged
+                else:
+                    merged = None          # insert-only table and the row is already there: leave it alone
+                if merged is None:
+                    same += 1
+                    continue
+                values = {c: v for c, v in merged.items() if c not in key}
+                if apply:
+                    target.execute(table.update().where(
+                        *[table.c[c] == row[c] for c in key]).values(**values))
+                updated += 1
+            how = "merged per column" if name in COL_RULE else ("row rule" if name in ROW_RULE else "insert only")
+            report.append((name, inserted, updated, same, how))
+        if problems:
+            raise SystemExit("STOPPED, nothing changed:\n  " + "\n  ".join(problems))
+
+    source.close()
+    return report
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--source", required=True)
+    ap.add_argument("--url", help="SQLAlchemy URL; default: built from the bot's DB_* environment")
+    ap.add_argument("--apply", action="store_true", help="write; otherwise report only")
+    args = ap.parse_args()
+    url = args.url
+    if not url:
+        import os
+        url = (f"mysql+pymysql://{os.environ['DB_USER']}:{os.environ['DB_PASSWORD']}"
+               f"@{os.environ['DB_HOST']}:{os.environ.get('DB_PORT','3306')}/{os.environ['DB_NAME']}?charset=utf8mb4")
+    rows = plan(args.source, url, apply=args.apply)
+    print(f"\n{'table':28} {'insert':>7} {'update':>7} {'same':>6}  how")
+    for name, ins, upd, same, how in sorted(rows):
+        print(f"{name:28} {ins:>7} {upd:>7} {same:>6}  {how}")
+    totals = [sum(r[i] for r in rows) for i in (1, 2, 3)]
+    print(f"{'TOTAL':28} {totals[0]:>7} {totals[1]:>7} {totals[2]:>6}")
+    print("\n" + ("applied." if args.apply else "report only; add --apply to write."))

bin/vgdeband.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+# One-off / catch-up: remove the timecode overscan band from stream recordings that were published BEFORE the
+# sweep learned to crop it. Re-encodes (libx264, frame timing preserved, audio copied), re-applies the timecode
+# from the sidecar, replaces the file, marks the sidecar band_removed. Skips files whose sidecar already says so.
+#   vgdeband.sh                      all of /srv/media/streams/*/*.mp4 with a strip sidecar
+#   vgdeband.sh FILE.mp4 ...         just these
+# Long: a 3 h 1440p60 night is a few hours of CPU. Run detached: sudo systemd-run --unit vgdeband -p Nice=15 \
+#   -p IOSchedulingClass=idle /usr/local/bin/vgdeband.sh ; follow with journalctl -fu vgdeband
+set -uo pipefail
+ROOT="${MEDIA_ROOT:-/srv/media}"
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgdeband %s\n' "$(date -Is)" "$lvl" "$*"; }
+log() { vglog info "msg=\"$*\""; }
+dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
+# Metrics (LOGGING.md): the same counters the sweep keeps, under component=vgdeband. No-ops if not installed.
+if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then
+  VG_MET_COMPONENT=vgdeband
+  . "$(dirname "$0")/vgmetrics.sh" && vgmet_init
+else
+  vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; }
+fi
+if (( $# )); then files=("$@"); else mapfile -t files < <(find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type f -name '*.mp4' | sort); fi
+for f in "${files[@]}"; do
+  side="${f%.*}.json"
+  [[ -f $side ]] || { vglog info "event=skip reason=no_sidecar file=\"$f\""; continue; }
+  read -r src removed tc ctime < <(python3 -c '
+import json,sys; d=json.load(open(sys.argv[1]))
+print(d.get("source","none"), 1 if d.get("band_removed") else 0, d.get("timecode",""), d.get("timestamp",""))' "$side")
+  [[ $src == strip ]] || { vglog info "event=skip reason=source_$src file=\"$f\""; continue; }
+  (( removed )) && { vglog info "event=skip reason=already_removed file=\"$f\""; continue; }
+  fuser -s "$f" 2>/dev/null && { vglog info "event=skip reason=in_use file=\"$f\""; continue; }
+  w=$(ffprobe -v error -select_streams v:0 -show_entries stream=width -of csv=p=0 "$f")
+  band=$(python3 /usr/local/bin/vgstrip.py band "$w")
+  tmp="$f.part"
+  tcargs=(); [[ -n $tc ]] && tcargs=(-timecode "$tc" -write_tmcd 1); [[ -n $ctime ]] && tcargs+=(-metadata "creation_time=$ctime")
+  vglog info "event=deband file=\"$f\" width=$w band_rows=$band timecode=\"$tc\""
+  job_t0=$(vgmet_time)
+  nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$f" \
+    -map 0:v:0 -map '0:a?' \
+    -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "${ENCODE_THREADS:-2}" \
+    -vf "crop=iw:ih-${band}:0:0" -fps_mode passthrough -c:a copy \
+    "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
+  rc=$?; job_secs=$(vgmet_since "$job_t0"); d1=$(dur "$f"); d2=$(dur "$tmp")
+  vgmet_add "seconds:encode:stream" "$job_secs"
+  if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
+    vgmet_add "jobs:encode:done:stream"; vgmet_add deband 1
+    mv -f "$tmp" "$f"; chown 1000:1000 "$f" 2>/dev/null
+    python3 -c '
+import json,sys; p=sys.argv[1]; d=json.load(open(p)); d["band_rows"]=int(sys.argv[2]); d["band_removed"]=True
+json.dump(d, open(p,"w"))' "$side" "$band"; chown 1000:1000 "$side" 2>/dev/null
+    vglog info "event=done file=\"$f\""
+  else
+    rm -f "$tmp"; vgmet_add "jobs:encode:failed:stream"; vglog error "event=failed file=\"$f\" rc=$rc src_dur=${d1:-?} out_dur=${d2:-?}" >&2
+  fi
+done

bin/vghosts-check.py

@@ -0,0 +1,407 @@
+#!/usr/bin/env python3
+"""Validate a hostname change end to end: what is on disk, what is running, what the UIs were told, what DNS says.
+
+    sudo vghosts-check.py            on the server: everything
+    vghosts-check.py --repo          on a dev box: only the checks that need the repo (defaults, docs, pages)
+    sudo vghosts-check.py --json     machine-readable
+
+Source of truth is the Caddyfile (`# vanguard:<role>` markers), the same place configure.py reads the hostnames
+back from. Every other place a hostname lives is checked AGAINST it: a host that is not one of Caddy's is stale.
+"ovh1.<base>" is the box's ssh name and must appear nowhere a browser is sent.
+
+ORDER MATTERS, and the sections below run in the order a change must be made, because each depends on the one
+before it (HOSTNAME-CHANGE.md, section E):
+   1 DNS           nothing can be issued a certificate or reached until the name resolves to the box
+   2 Caddyfile     the roles and their names: what configure.py --full / --hosts wrote; certificates follow
+   3 .env          the same names as compose variables; a mismatch here means a half-done --full
+   4 mediamtx.yml  webrtcAdditionalHosts = main host (WebRTC ICE); needs a MediaMTX restart to take effect
+   5 containers    the env each running container was CREATED with (a .env edit without a recreate shows here)
+   6 HTTPS         every host answers with a valid certificate; gated hosts send browsers to the auth host
+   7 Jellyfin      SSO plugin OID endpoint, branding disclaimer, Vanguard plugin URLs (on disk) + the SSO start
+   8 Gitea         app.ini, the auth source (live redirect), the runner, the server clone's remote
+   9 Authentik     outpost host, proxy provider external host + cookie domain, Jellyfin/Gitea redirect URIs.
+                   LAST among the services, on purpose: an app changed before Authentik is merely unreachable on
+                   its new name until Authentik catches up; Authentik changed before the app locks you out of it
+  10 Discord       the OAuth2 redirect in the developer portal — cannot be read; the exact value is printed
+  11 repo          deployLive.sh / switchToClone.sh defaults, about page labels = Caddy roles, docs
+  12 streamers     OBS dock URL and clock source: cannot be read; listed so nobody forgets
+A failure in an early section usually explains failures in later ones: fix top-down, re-run. The sweep runs the
+Jellyfin and Gitea checks before Authentik, and the report lists them that way, because that is the order to make
+the change in.
+"""
+import argparse, json, os, re, socket, ssl, subprocess, sys, urllib.request, urllib.error
+from pathlib import Path
+
+ROLES = ("main", "auth", "watch", "git", "bot", "live", "files", "sync", "clips", "diag")
+ENV_KEY = {"main": "MAIN_HOST", "watch": "WATCH_HOST", "git": "GIT_HOST", "bot": "BOT_HOST", "live": "LIVE_HOST",
+           "files": "FILES_HOST", "sync": "SYNC_HOST", "clips": "CLIPS_HOST", "diag": "DIAG_HOST"}
+GATED = ("main", "files", "sync", "clips", "live", "bot", "diag")            # forward-auth: anonymous -> auth host
+REPO = Path(os.environ.get("VANGUARD_REPO", str(Path.home() / "vanguard-stream") if Path.home().joinpath("vanguard-stream").is_dir() else "."))
+APPDATA = Path("/srv/appdata")
+
+results = []            # (section, status, text)
+def rec(section, status, text): results.append((section, status, text))
+def ok(section, text): rec(section, "PASS", text)
+def bad(section, text): rec(section, "FAIL", text)
+def warn(section, text): rec(section, "WARN", text)
+def manual(section, text): rec(section, "MANUAL", text)
+
+def sh(cmd, timeout=30, **kw):
+    try:
+        return subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, **kw)
+    except (OSError, subprocess.TimeoutExpired) as error:
+        class R: returncode, stdout, stderr = 1, "", str(error)
+        return R()
+
+def read_hosts(caddyfile):
+    hosts, role = {}, None
+    for line in caddyfile.read_text().splitlines():
+        if line[:1].isspace(): continue
+        t = line.strip()
+        m = re.match(r"#\s*vanguard:(\w+)", t)
+        if m: role = m.group(1); continue
+        if t and not t.startswith(("#", "}", "{")) and t.endswith("{") and " " in t:
+            host = t.split()[0].rstrip(",")          # "a, b {": an alias; the first name is the role's
+            if "example.com" in host or role == "redirect": role = None; continue
+            hosts[role or ("main" if "main" not in hosts else host)] = host
+            role = None
+    return hosts
+
+def read_env(path):
+    env = {}
+    try:
+        for line in path.read_text().splitlines():
+            if "=" in line and not line.startswith("#"):
+                k, v = line.split("=", 1); env[k.strip()] = v.strip()
+    except OSError:
+        pass
+    return env
+
+URL_RE = re.compile(r"https?://([A-Za-z0-9.-]+\.[A-Za-z]{2,})(?::\d+)?")
+def hosts_in(text):
+    return sorted(set(h.lower() for h in URL_RE.findall(text)))
+
+def classify(section, where, found, allowed, expect=None, base=""):
+    """found: hostnames seen in `where`. allowed: every current host. expect: the one role's host, if specific."""
+    if not found:
+        warn(section, f"{where}: no hostname found"); return
+    for h in found:
+        if expect and h == expect: ok(section, f"{where}: {h}")
+        elif expect and h != expect and (h.endswith("." + base) or h.startswith("ovh1.")):
+            bad(section, f"{where}: {h} — expected {expect}")
+        elif h in allowed: ok(section, f"{where}: {h}")
+        elif base and (h.endswith("." + base) or h == base or h.startswith("ovh1.")):
+            bad(section, f"{where}: {h} — not a current host (stale)")
+        else:
+            pass    # third-party hostnames (discord.com, github.com, …) are not ours to judge
+
+class _NoRedirect(urllib.request.HTTPRedirectHandler):
+    def redirect_request(self, *a, **k): return None
+_opener = urllib.request.build_opener(_NoRedirect)
+
+def curl_head(url, timeout=12):
+    """(status, location, error) for a browser-like GET, without following redirects. GET, not HEAD: the
+    forward-auth gate answers HEAD with 200 and the SSO start endpoints are GET-only."""
+    req = urllib.request.Request(url, method="GET", headers={"User-Agent": "Mozilla/5.0 vghosts-check", "Accept": "text/html,*/*"})
+    try:
+        with _opener.open(req, timeout=timeout) as r:
+            r.read(2048)
+            return r.status, r.headers.get("Location", ""), ""
+    except urllib.error.HTTPError as e:
+        return e.code, e.headers.get("Location", ""), ""
+    except Exception as e:
+        return 0, "", str(e)
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--repo", action="store_true", help="repo-only checks (dev box)")
+    ap.add_argument("--json", action="store_true")
+    ap.add_argument("--caddyfile", default=None, help="path to the generated Caddyfile (default: <repo>/Caddyfile)")
+    args = ap.parse_args()
+    repo = REPO
+    caddyfile = Path(args.caddyfile) if args.caddyfile else repo / "Caddyfile"
+
+    # ---- 2 Caddyfile: the truth -------------------------------------------------------------------
+    S = "2 Caddyfile"
+    if not caddyfile.is_file():
+        if args.repo:
+            hosts = {}
+            warn(S, f"no Caddyfile at {caddyfile} (dev box): repo checks use the names in deployLive.sh / switchToClone.sh")
+        else:
+            bad(S, f"no Caddyfile at {caddyfile}: nothing to validate against — run configure.py --full first"); return report(args)
+    else:
+        hosts = read_hosts(caddyfile)
+    main_host = hosts.get("main", "")
+    base = main_host.split(".", 1)[1] if "." in main_host else ""
+    allowed = set(hosts.values())
+    for role in (ROLES if hosts else ()):
+        if role in hosts: ok(S, f"{role:6} {hosts[role]}")
+        else: warn(S, f"{role:6} no site block (configure.py --hosts to add one)")
+    if base and f"ovh1.{base}" in allowed: bad(S, f"ovh1.{base} still has a site block")
+
+    # ---- 1 DNS -------------------------------------------------------------------------------------
+    S = "1 DNS"
+    if hosts and not args.repo:
+        ips = {}
+        for role, h in hosts.items():
+            try: ips[h] = sorted(set(a[4][0] for a in socket.getaddrinfo(h, 443, proto=socket.IPPROTO_TCP)))
+            except OSError as e: bad(S, f"{h}: does not resolve ({e})")
+        role_ips = {h: v for r, h in hosts.items() for hh, v in ips.items() if hh == h and r in ROLES}
+        distinct = set(tuple(v) for v in role_ips.values())
+        for h, v in ips.items():
+            if h in role_ips: (ok if len(distinct) == 1 else bad)(S, f"{h} -> {', '.join(v)}")
+            else: warn(S, f"{h} -> {', '.join(v)} (apex/other block: a redirect may live anywhere, but Caddy must hold its certificate to answer)")
+        if len(distinct) > 1: bad(S, "the role hosts resolve to different addresses — a CNAME points elsewhere")
+        if base:
+            try:
+                old = sorted(set(a[4][0] for a in socket.getaddrinfo(f"ovh1.{base}", 22, proto=socket.IPPROTO_TCP)))
+                ok(S, f"ovh1.{base} -> {', '.join(old)} (ssh only; fine to keep)")
+            except OSError: warn(S, f"ovh1.{base} no longer resolves — ssh by IP or another name")
+
+    # ---- 3 .env --------------------------------------------------------------------------------------
+    S = "3 .env"
+    env = read_env(repo / ".env")
+    if not args.repo:
+        if not env: bad(S, f"{repo / '.env'} unreadable (run with sudo, from the checkout)")
+        for role, key in ENV_KEY.items():
+            if role not in hosts: continue
+            v = env.get(key, "")
+            if v == hosts[role]: ok(S, f"{key}={v}")
+            elif not v: bad(S, f"{key} unset — expected {hosts[role]}")
+            else: bad(S, f"{key}={v} — Caddyfile says {hosts[role]}")
+        apex, apex_to = env.get("APEX_HOST", ""), env.get("APEX_REDIRECT", "")
+        if apex:
+            found = hosts_in(apex_to); classify(S, f"APEX_REDIRECT ({apex} ->)", found, allowed, base=base)
+        else: ok(S, "APEX_HOST empty (bare domain not served)")
+
+    # ---- 4 mediamtx.yml ------------------------------------------------------------------------------
+    S = "4 mediamtx.yml"
+    if not args.repo:
+        mtx = repo / "mediamtx.yml"
+        if mtx.is_file():
+            m = re.search(r"^webrtcAdditionalHosts:\s*\[([^\]]*)\]", mtx.read_text(), re.M)
+            got = [x.strip().strip("'\"") for x in (m.group(1).split(",") if m else []) if x.strip()]
+            if main_host in got: ok(S, f"webrtcAdditionalHosts contains {main_host}")
+            else: bad(S, f"webrtcAdditionalHosts = {got or 'missing'} — expected {main_host} (then: docker compose restart mediamtx, between streams)")
+        else: bad(S, "mediamtx.yml missing from the checkout")
+
+    # ---- 5 running containers ------------------------------------------------------------------------
+    S = "5 containers"
+    if not args.repo:
+        want = {"jellyfin": ("JELLYFIN_PublishedServerUrl", "watch"), "gitea": ("GITEA__server__ROOT_URL", "git"),
+                "act_runner": ("GITEA_INSTANCE_URL", "git"), "discord-bot": ("WEB_PUBLIC_URL", "bot"),
+                "mediamtx": ("LIVE_HOST", "live"), "kibana": ("DIAG_HOST", "diag")}
+        for container, (var, role) in want.items():
+            if role not in hosts: continue
+            r = sh(["docker", "inspect", "-f", "{{range .Config.Env}}{{println .}}{{end}}", container])
+            if r.returncode != 0: warn(S, f"{container}: not running or no such container"); continue
+            val = next((l.split("=", 1)[1] for l in r.stdout.splitlines() if l.startswith(var + "=")), None)
+            if val is None: warn(S, f"{container}: {var} not in its environment"); continue
+            if hosts[role] in val: ok(S, f"{container}: {var}={val}")
+            else: bad(S, f"{container}: {var}={val} — expected {hosts[role]}; the container was created before .env changed: docker compose up -d {container}")
+        gd = sh(["docker", "inspect", "-f", "{{range .Config.Env}}{{println .}}{{end}}", "gitea"])
+        for var in ("GITEA__server__DOMAIN", "GITEA__server__SSH_DOMAIN"):
+            val = next((l.split("=", 1)[1] for l in gd.stdout.splitlines() if l.startswith(var + "=")), None)
+            if val and "git" in hosts: (ok if val == hosts["git"] else bad)(S, f"gitea: {var}={val}" + ("" if val == hosts["git"] else f" — expected {hosts['git']}"))
+
+    # ---- 6 HTTPS -------------------------------------------------------------------------------------
+    S = "6 HTTPS"
+    if hosts and not args.repo:
+        auth = hosts.get("auth", "")
+        for role, h in hosts.items():
+            status, loc, err = curl_head(f"https://{h}/")
+            if err:
+                if role in ROLES: bad(S, f"{h}: {err[:90]} (no certificate yet? Caddy block missing? DNS?)")
+                else: warn(S, f"{h}: {err[:90]} — the bare domain's A record does not reach this box, so its Caddy block cannot get a certificate; fix the record or clear APEX_HOST")
+                continue
+            loc_host = (hosts_in(loc) or [""])[0]
+            if role in GATED:
+                if status in (301, 302, 303, 307, 308) and loc_host == auth: ok(S, f"{h}: {status} -> {auth} (gate)")
+                elif status in (301, 302, 303, 307, 308): bad(S, f"{h}: {status} -> {loc} — gate sends to the wrong host")
+                else: warn(S, f"{h}: {status} without a redirect to {auth} — is the forward-auth gate on?")
+            else:
+                (ok if status < 400 else bad)(S, f"{h}: {status}{(' -> ' + loc) if loc else ''}")
+        # a move in progress: aliases must answer like the primary names; old names must redirect to the new ones
+        alt, red = env.get("ALT_BASE", "").strip().lower(), env.get("REDIRECT_BASE", "").strip().lower()
+        for role, h in hosts.items():
+            if role not in ROLES or not h.endswith("." + base): continue
+            if alt and alt != base:
+                a = h[: -len(base)] + alt
+                status, loc, err = curl_head(f"https://{a}/")
+                (bad if err else ok)(S, f"alias {a}: {err[:80] if err else status}")
+            if red and red != base:
+                o = h[: -len(base)] + red
+                status, loc, err = curl_head(f"https://{o}/")
+                if err: warn(S, f"old name {o}: {err[:80]} (its certificate may still be issuing)")
+                elif status == 308 and (hosts_in(loc) or [""])[0] == h: ok(S, f"old name {o}: 308 -> {h}")
+                else: bad(S, f"old name {o}: {status} -> {loc or '(no Location)'} — expected 308 to {h}")
+        if base:
+            status, loc, err = curl_head(f"https://ovh1.{base}/", timeout=8)
+            if err: ok(S, f"ovh1.{base}: no https (expected: ssh only)")
+            else: bad(S, f"ovh1.{base} still answers https ({status}) — a Caddy block or another server is serving the old name")
+
+    # ---- 7 Authentik ----------------------------------------------------------------------------------
+    S = "9 Authentik"
+    if not args.repo and hosts:
+        AK = r'''
+import json
+from urllib.parse import urlparse
+from authentik.outposts.models import Outpost
+from authentik.providers.proxy.models import ProxyProvider
+from authentik.providers.oauth2.models import OAuth2Provider
+from authentik.sources.oauth.models import OAuthSource
+from authentik.brands.models import Brand
+out = {"outposts": [], "proxy": [], "oauth2": [], "sources": [], "brands": []}
+for o in Outpost.objects.all():
+    out["outposts"].append({"name": o.name, "authentik_host": (o.config or {}).get("authentik_host", "") if isinstance(o.config, dict) else getattr(o.config, "authentik_host", "")})
+for p in ProxyProvider.objects.all():
+    out["proxy"].append({"name": p.name, "external_host": p.external_host, "cookie_domain": p.cookie_domain, "mode": str(p.mode)})
+for p in OAuth2Provider.objects.all():
+    uris = []
+    try:
+        uris = [u.url for u in p.redirect_uris]
+    except Exception:
+        try: uris = [u.get("url", "") for u in (p._redirect_uris or [])]
+        except Exception: uris = [str(getattr(p, "redirect_uris", ""))]
+    out["oauth2"].append({"name": p.name, "redirect_uris": uris})
+for s in OAuthSource.objects.all():
+    out["sources"].append({"name": s.name, "slug": s.slug, "provider_type": s.provider_type})
+for b in Brand.objects.all():
+    out["brands"].append({"domain": b.domain, "default": b.default, "logo": str(b.branding_logo), "default_app": str(b.default_application) if b.default_application_id else ""})
+print("VGHOSTS " + json.dumps(out))
+'''
+        r = sh(["docker", "exec", "-i", "authentik-server", "ak", "shell", "-c", "import sys; exec(sys.stdin.read())"], input=AK, timeout=120)
+        line = next((l for l in (r.stdout + r.stderr).splitlines() if l.startswith("VGHOSTS ")), None)
+        if not line:
+            bad(S, "could not read Authentik (" + (r.stderr.strip().splitlines() or ["no output"])[-1][:120] + ")")
+        else:
+            ak = json.loads(line[8:])
+            auth = hosts.get("auth", "")
+            for o in ak["outposts"]:
+                h = hosts_in(o["authentik_host"])
+                if h == [auth]: ok(S, f"outpost {o['name']!r}: authentik_host {o['authentik_host']}")
+                elif not o["authentik_host"]: warn(S, f"outpost {o['name']!r}: authentik_host unset (the embedded outpost then uses its own address; set it to https://{auth})")
+                else: bad(S, f"outpost {o['name']!r}: authentik_host {o['authentik_host']} — expected https://{auth}")
+            for p in ak["proxy"]:
+                h = hosts_in(p["external_host"])
+                classify(S, f"proxy provider {p['name']!r} external_host", h, allowed, base=base)
+                cd = p["cookie_domain"]
+                if "domain" in p["mode"].lower():
+                    (ok if cd == base else bad)(S, f"proxy provider {p['name']!r} cookie_domain={cd or '(unset)'}" + ("" if cd == base else f" — expected {base}; sessions will not span the hosts"))
+            for p in ak["oauth2"]:
+                expect = hosts.get("watch") if "jellyfin" in p["name"].lower() else hosts.get("git") if "gitea" in p["name"].lower() else None
+                for u in p["redirect_uris"]:
+                    classify(S, f"OAuth2 provider {p['name']!r} redirect", hosts_in(u), allowed, expect=expect, base=base)
+                if not p["redirect_uris"]: warn(S, f"OAuth2 provider {p['name']!r}: no redirect URIs")
+            for b in ak["brands"]:
+                if b["domain"] in ("", "authentik-default"): ok(S, f"brand domain {b['domain']!r} (default)")
+                elif b["domain"] == base or b["domain"].endswith(base): ok(S, f"brand domain {b['domain']}")
+                else: warn(S, f"brand domain {b['domain']} — not under {base}")
+            # ---- 8 Discord --------------------------------------------------------------------------
+            for s in ak["sources"]:
+                if s["provider_type"] == "discord":
+                    manual("10 Discord", f"developer portal → OAuth2 → Redirects must contain exactly: https://{auth}/source/oauth/callback/{s['slug']}/")
+            if not any(s["provider_type"] == "discord" for s in ak["sources"]):
+                bad("10 Discord", "no Discord OAuth source in Authentik")
+
+    # ---- 9 Jellyfin -----------------------------------------------------------------------------------
+    S = "7 Jellyfin"
+    if not args.repo and hosts:
+        jf = APPDATA / "jellyfin" / "config"
+        if jf.is_dir():
+            for xml in sorted(jf.rglob("*.xml")):
+                try: text = xml.read_text(errors="replace")
+                except OSError: continue
+                found = [h for h in hosts_in(text) if base and (h.endswith("." + base) or h == base or h.startswith("ovh1."))]
+                if not found: continue
+                name = xml.name.lower()
+                expect = hosts.get("auth") if "sso" in name else None
+                classify(S, f"{xml.relative_to(jf)}", found, allowed, expect=expect, base=base)
+        else: warn(S, f"{jf} not found")
+        if "watch" in hosts:
+            status, loc, err = curl_head(f"https://{hosts['watch']}/sso/OID/start/authentik")
+            lh = (hosts_in(loc) or [""])[0]
+            if err: bad(S, f"SSO start: {err[:80]}")
+            elif status in (301, 302, 303, 307) and lh == hosts.get("auth"): ok(S, f"SSO start -> {hosts['auth']} (plugin OID endpoint is right)")
+            else: bad(S, f"SSO start answered {status} -> {loc or '(no Location)'} — SSO-Auth plugin OID endpoint should be https://{hosts.get('auth')}/application/o/jellyfin/")
+
+    # ---- 10 Gitea ---------------------------------------------------------------------------------------
+    S = "8 Gitea"
+    if not args.repo and hosts and "git" in hosts:
+        ini = APPDATA / "gitea" / "gitea" / "conf" / "app.ini"
+        if ini.is_file():
+            text = ini.read_text(errors="replace")
+            for key in ("ROOT_URL", "DOMAIN", "SSH_DOMAIN"):
+                m = re.search(rf"^{key}\s*=\s*(\S+)", text, re.M)
+                v = m.group(1) if m else ""
+                (ok if hosts["git"] in v else bad)(S, f"app.ini {key}={v or '(missing)'}" + ("" if hosts["git"] in v else f" — expected {hosts['git']} (compose env sets it; recreate gitea)"))
+        else: warn(S, f"{ini} not found")
+        status, loc, err = curl_head(f"https://{hosts['git']}/user/oauth2/authentik")
+        lh = (hosts_in(loc) or [""])[0]
+        if err: bad(S, f"auth source: {err[:80]}")
+        elif lh == hosts.get("auth"): ok(S, f"auth source 'authentik' -> {hosts['auth']} (discovery URL is right)")
+        else: bad(S, f"auth source: {status} -> {loc or '(no Location)'} — Site administration → Authentication sources → authentik → discovery URL https://{hosts.get('auth')}/application/o/gitea/.well-known/openid-configuration")
+        runner = Path("/srv/appdata/act_runner/.runner")
+        if runner.is_file():
+            classify(S, "runner registration (.runner)", hosts_in(runner.read_text(errors="replace")), allowed, expect=hosts["git"], base=base)
+        else: warn(S, "runner: /srv/appdata/act_runner/.runner not found — ./registerGiteaRunner.sh")
+        r = sh(["git", "-C", str(repo), "remote", "get-url", "origin"])
+        if r.returncode == 0:
+            url = r.stdout.strip(); m = re.search(r"@([A-Za-z0-9.-]+)", url)
+            h = (m.group(1).lower() if m else "")
+            (ok if h == hosts["git"] else bad)(S, f"server clone remote: {url}" + ("" if h == hosts["git"] else f" — expected {hosts['git']}"))
+        else: warn(S, f"{repo}: not a git clone (deploy.sh pull needs one)")
+        for cfg in (Path.home() / ".ssh" / "config.vanguard", Path("/home/debian/.ssh/config.vanguard")):
+            if cfg.is_file():
+                classify(S, f"{cfg}", [h for h in re.findall(r"(?im)^\s*host(?:name)?\s+(\S+)", cfg.read_text()) if "." in h], allowed, expect=hosts["git"], base=base); break
+
+    # ---- 11 repo ----------------------------------------------------------------------------------------
+    S = "11 repo"
+    for script, var in (("deployLive.sh", "HOST"), ("switchToClone.sh", "REPO_SSH")):
+        p = repo / script
+        if not p.is_file(): warn(S, f"{script} missing"); continue
+        m = re.search(rf"^{var}=.*", p.read_text(), re.M)
+        found = [h for h in re.findall(r"@([A-Za-z0-9.-]+\.[a-z]{2,})", m.group(0))] if m else []
+        if hosts: classify(S, f"{script} {var}", [h.lower() for h in found], allowed, base=base)
+        else: ok(S, f"{script} {var}: {', '.join(found) or '(none)'}")
+    about = repo / "about" / "index.html"
+    if about.is_file():
+        labels = sorted(set(re.findall(r"sub\('([a-z]+)'\)", about.read_text())))
+        if hosts:
+            for label in labels:
+                (ok if label in hosts else bad)(S, f"about page label '{label}'" + ("" if label in hosts else " — no such Caddy role; the gateway link will 404/TLS-fail"))
+        else: ok(S, f"about page labels: {', '.join(labels)}")
+    for doc in ("README.md", "FRESH-INSTALL.md", "STATUS.md", "LOGGING.md"):
+        p = repo / doc
+        if not p.is_file(): continue
+        stale = [(n, l.strip()) for n, l in enumerate(p.read_text().splitlines(), 1)
+                 if re.search(r"https?://ovh1\.|ovh1\.[a-z.]+/", l) and not re.search(r"ssh|scp|ssh-only|ssh only", l)]
+        if stale: bad(S, f"{doc}: {len(stale)} line(s) send a browser to ovh1 — first: {stale[0][0]}: {stale[0][1][:80]}")
+        else: ok(S, f"{doc}: no browser links to ovh1")
+    plan = repo / "CommunityDiscordMedia.md"
+    if plan.is_file() and re.search(r"https?://ovh1\.", plan.read_text()):
+        warn(S, "CommunityDiscordMedia.md still describes ovh1 as a web host (historical plan; mark the lines as superseded or leave)")
+
+    # ---- 12 streamers -------------------------------------------------------------------------------
+    if hosts.get("live"):
+        manual("12 streamers", f"OBS → Docks → Custom Browser Docks: https://{hosts['live']}/dock/ ; dock Settings re-saved; Timecode band → \"Add band + clock source\" again; manual RTMP/SRT configs re-copied from the dock")
+        manual("12 streamers", f"Jellyfin apps on phones/TVs: server https://{hosts.get('watch','')} ; Discord pins: https://{main_host}/ (members), https://{hosts.get('clips','')}/ (gamers), https://{hosts.get('files','')}/ (editors)")
+    return report(args)
+
+def report(args):
+    order = lambda s: int(s.split()[0])
+    sections = sorted(set(s for s, _, _ in results), key=order)
+    if args.json:
+        print(json.dumps([{"section": s, "status": st, "text": t} for s, st, t in results], indent=1)); return 0
+    mark = {"PASS": "  ok  ", "FAIL": " FAIL ", "WARN": " warn ", "MANUAL": " todo "}
+    for s in sections:
+        print(f"\n== {s}")
+        for sec, st, t in results:
+            if sec == s: print(f" [{mark[st]}] {t}")
+    fails = sum(1 for _, st, _ in results if st == "FAIL"); warns = sum(1 for _, st, _ in results if st == "WARN")
+    todo = sum(1 for _, st, _ in results if st == "MANUAL")
+    print(f"\n{fails} failed, {warns} warnings, {todo} to check by hand. Fix top-down: an early failure explains the later ones.")
+    return 1 if fails else 0
+
+if __name__ == "__main__":
+    sys.exit(main())

bin/vgmetrics.sh

@@ -0,0 +1,63 @@
+# Shared metric counters for the media scripts. Sourced, never executed.
+#
+#   VG_MET_COMPONENT=sweep                      # a plain assignment, NOT a prefix on the next line:
+#   . /usr/local/bin/vgmetrics.sh && vgmet_init  # bash discards "VAR=x . file" assignments on return
+#   vgmet_add "jobs:encode:done:stream"      # +1
+#   vgmet_add "seconds:encode:stream" 41.2   # +41.2
+#
+# Counters accumulate across runs in MARK/metrics/<component>.counters as plain "key value" lines, and a
+# <component>.pid file exists only while the script is executing. dumpd turns both into Prometheus text on
+# its /metrics; nothing here renders or serves anything, so the scripts stay simple and there is one place
+# that knows the metric names (bin/dumpd.py COUNTER_MAP).
+#
+# Keys are "name:label:label..." — the name picks the metric, the rest fill its labels in order.
+# Writes are atomic (temp file + mv) and every failure is silent: metrics must never break a sweep.
+
+VG_MET_DIR="${VG_MET_DIR:-${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
+VG_MET_COMPONENT="${VG_MET_COMPONENT:-$(basename "${0%.sh}")}"
+VG_MET_START=0
+declare -A VGMET=()
+
+vgmet_init() {
+  : "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
+  : "${VG_MET_COMPONENT:=unknown}"
+  VG_MET_START=$(date +%s)
+  mkdir -p "$VG_MET_DIR" 2>/dev/null || return 0
+  local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" k v
+  if [[ -r $f ]]; then
+    while read -r k v; do [[ -n ${k:-} && -n ${v:-} ]] && VGMET["$k"]="$v"; done < "$f"
+  fi
+  vgmet_add runs 1
+  printf '%s\n' "$$" > "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
+  trap vgmet_flush EXIT
+}
+
+# vgmet_add <key> [amount]   — float-safe accumulate
+vgmet_add() {
+  local k="$1" v="${2:-1}"
+  VGMET["$k"]=$(awk -v a="${VGMET[$k]:-0}" -v b="$v" 'BEGIN{ printf "%.3f", a+b }')
+}
+
+vgmet_set() { VGMET["$1"]="$2"; }
+
+vgmet_flush() {
+  : "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
+  : "${VG_MET_COMPONENT:=unknown}"
+  local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" tmp now k
+  rm -f "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
+  [[ -d $VG_MET_DIR ]] || return 0
+  now=$(date +%s)
+  vgmet_set last_finish "$now"
+  vgmet_set last_duration "$(( now - VG_MET_START ))"
+  tmp=$(mktemp "$VG_MET_DIR/.$VG_MET_COMPONENT.XXXXXX" 2>/dev/null) || return 0
+  for k in "${!VGMET[@]}"; do printf '%s %s\n' "$k" "${VGMET[$k]}"; done > "$tmp" 2>/dev/null
+  chmod 0644 "$tmp" 2>/dev/null
+  mv -f "$tmp" "$f" 2>/dev/null || rm -f "$tmp" 2>/dev/null
+}
+
+# start/elapsed for one job
+vgmet_time()  { date +%s.%N; }
+vgmet_since() { awk -v s="$1" -v e="$(date +%s.%N)" 'BEGIN{ printf "%.3f", e-s }'; }
+
+# No-op fallbacks are defined by the callers if this file is missing, so a half-finished deploy cannot
+# break normalising. Keep every function above failure-tolerant.

bin/vgnotify.sh

@@ -0,0 +1,74 @@
+# Shared Discord delivery for bin/vanguard-alerts.sh and bin/vanguard-deadman.sh. Sourced, never executed.
+#
+#   . /usr/local/bin/vgnotify.sh
+#   vg_post error "disk is nearly full"     # 0 = delivered, 1 = not (no webhook, bad URL, or HTTP failure)
+#
+# WHY MENTIONS NEED allowed_mentions
+# Putting <@&ROLE_ID> in a webhook's content is not enough on its own: Discord will render it but will not
+# reliably ping, because a webhook has no permission to mention roles unless the request says which mentions
+# are allowed. So every post sends an explicit allowed_mentions. It also sets parse: [] always, which means
+# an @everyone appearing inside an alert's text (a log line, a filename) can never ping the whole server.
+#
+# ALERT_MENTION is written for humans rather than for Discord's syntax:
+#   ALERT_MENTION='role:1387247501387628585,role:1566623319027880086'   -> pings those roles
+#   ALERT_MENTION='user:123456789'                                 -> pings that person
+# Raw '<@&123>' / '<@123>' is accepted too and passed through unchanged.
+#
+# ALERT_MENTION_LEVELS decides which alarms are worth a ping; the default is errors only, so warnings still
+# arrive in the channel without waking anyone.
+
+VG_WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
+VG_MENTION="${ALERT_MENTION:-}"
+VG_MENTION_LEVELS="${ALERT_MENTION_LEVELS:-error}"
+
+# The webhook URL, or nothing. A value that is not https is treated as absent and complained about once:
+# posting to it would fail anyway, and silently "succeeding" would be worse.
+vg_webhook() {
+  local url
+  [[ -r $VG_WEBHOOK_FILE ]] || return 1
+  url=$(tr -d '[:space:]' < "$VG_WEBHOOK_FILE" 2>/dev/null) || return 1
+  case "$url" in
+    https://*) printf '%s' "$url" ;;
+    "")        return 1 ;;
+    *)         echo "vgnotify: the configured webhook is not an https URL; ignoring it" >&2; return 1 ;;
+  esac
+}
+
+vg_post() {
+  local level="$1" text="$2" url payload code
+  url=$(vg_webhook) || return 1
+  payload=$(VG_LEVEL="$level" VG_TEXT="$text" VG_MENTION="$VG_MENTION" VG_LEVELS="$VG_MENTION_LEVELS" python3 - <<'PY'
+import json, os, re
+level   = os.environ.get("VG_LEVEL", "")
+text    = os.environ.get("VG_TEXT", "")
+raw     = os.environ.get("VG_MENTION", "").strip()
+levels  = [l.strip() for l in os.environ.get("VG_LEVELS", "error").split(",") if l.strip()]
+
+users, roles, parts = [], [], []
+for entry in [e.strip() for e in raw.replace(" ", ",").split(",") if e.strip()]:
+    if entry.startswith("role:"):
+        roles.append(entry[5:]); parts.append(f"<@&{entry[5:]}>")
+    elif entry.startswith("user:"):
+        users.append(entry[5:]); parts.append(f"<@{entry[5:]}>")
+    else:                                   # already Discord syntax
+        parts.append(entry)
+        roles += re.findall(r"<@&(\d+)>", entry)
+        users += re.findall(r"<@!?(\d+)>", entry)
+
+ping = bool(parts) and level in levels
+prefix = (" ".join(parts) + " ") if ping else ""
+# parse: [] unconditionally — an @everyone inside alert text must never ping the server.
+body = {"content": (prefix + text)[:1900],
+        "allowed_mentions": {"parse": [],
+                             "users": sorted(set(users)) if ping else [],
+                             "roles": sorted(set(roles)) if ping else []}}
+print(json.dumps(body))
+PY
+)
+  code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST \
+           -H 'Content-Type: application/json' -d "$payload" "$url" || true)
+  case "${code:-000}" in
+    2*) return 0 ;;
+    *)  echo "vgnotify: webhook post failed, HTTP ${code:-000}" >&2; return 1 ;;
+  esac
+}

bin/vgstitch.py

@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+"""Stitch reconnect fragments of one stream back into one recording (run by normalize-sweep.sh).
+
+Every time a publisher drops and comes back, MediaMTX opens a new file under streams/<user>/<Title>/.live/,
+so a night with a flaky uplink leaves one stream as a row of fragments: 20 min, 2 s, 14 min, 1 s, ... The
+fragments are the SAME stream (same user, same title) and the file names carry the server arrival time to
+the microsecond, so we know exactly where each piece sits on the clock. This joins them losslessly:
+
+  * only when the stream is over: nothing in .live/ open (fuser) or modified in the last IDLE_MIN minutes
+  * only fewer than MAX_PARTS fragments (default 500 — a lossy uplink splits a half-hour stream into ~180
+    files, see normalize-sweep.sh): beyond that is a broken night, left for vgstuck.sh / the streaming page
+  * a chain breaks where the gap between fragments is longer than MAX_GAP_S (default 10 min: a reconnect
+    is seconds, a second session the same evening is not), or where the video/audio parameters change
+    (resolution, codec, rate): those cannot be copy-concatenated, so each side is stitched on its own
+  * ffmpeg concat demuxer with -c copy — no re-encode, the sweep's own pass (timecode, band removal,
+    Opus->AAC) runs on the result as on any other recording
+  * the gaps between fragments are COLLAPSED (the picture jumps), and recorded in <stem>.stitch.json which
+    the sweep merges into the published sidecar: part files, each part's arrival time, its offset in the
+    stitched file, and the wall-clock time missing at every join
+  * the stitched file takes the FIRST fragment's name (so the arrival-time fallback for the timecode still
+    points at the real start), the fragments move to trash/<night>/ for the usual 3-day undo
+  * a fragment ffprobe cannot read is binned (reason=unreadable) rather than blocking its stream forever
+
+Prints logfmt lines (component=sweep) for Kibana. Exit 0 unless the arguments are wrong.
+    vgstitch.py <media root> [--max-parts N] [--max-gap-s S] [--idle-min M] [--dry-run]
+"""
+import argparse, json, os, re, subprocess, sys, time
+from datetime import datetime
+from pathlib import Path
+from zoneinfo import ZoneInfo
+
+TZ = ZoneInfo("America/New_York")
+NAME_RE = re.compile(r"(\d{4})-(\d{2})-(\d{2})_(\d{2})-(\d{2})-(\d{2})(?:-(\d{1,6}))?")
+
+def vglog(level, **kv):
+    parts = [f"ts={datetime.now().astimezone().isoformat(timespec='seconds')}", f"level={level}", "component=sweep"]
+    for k, v in kv.items():
+        v = str(v)
+        parts.append(f'{k}="{v}"' if (" " in v or "/" in v or v == "") else f"{k}={v}")
+    print(" ".join(parts), flush=True)
+
+def arrival_ms(stem):
+    m = NAME_RE.search(stem)
+    if not m: return None
+    y, mo, d, h, mi, s, us = m.groups()
+    us = int((us or "0").ljust(6, "0"))
+    return int(datetime(int(y), int(mo), int(d), int(h), int(mi), int(s), us, tzinfo=TZ).timestamp() * 1000)
+
+def ffprobe(path):
+    """(params tuple, duration_s) or None when unreadable."""
+    try:
+        out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
+                              "stream=codec_type,codec_name,width,height,pix_fmt,sample_rate,channels:format=duration",
+                              "-of", "json", str(path)], capture_output=True, text=True, timeout=120, check=True).stdout
+        j = json.loads(out)
+        v = [s for s in j["streams"] if s["codec_type"] == "video"]
+        a = [s for s in j["streams"] if s["codec_type"] == "audio"]
+        if not v: return None
+        dur = float(j["format"].get("duration") or 0)
+        if dur <= 0: return None
+        params = (v[0].get("codec_name"), v[0].get("width"), v[0].get("height"), v[0].get("pix_fmt"),
+                  tuple((s.get("codec_name"), s.get("sample_rate"), s.get("channels")) for s in a))
+        return params, dur
+    except Exception:
+        return None
+
+def busy(path):
+    return subprocess.run(["fuser", "-s", str(path)], capture_output=True).returncode == 0
+
+def to_trash(root, path, night):
+    rel = path.relative_to(root)
+    dst = root / "trash" / night / rel
+    dst.parent.mkdir(parents=True, exist_ok=True)
+    os.replace(path, dst)
+    return dst
+
+def stitch_dir(root, live, args):
+    titledir = live.parent
+    stream = f"{titledir.parent.name}/{titledir.name}"
+    files = sorted(p for p in live.iterdir() if p.is_file() and p.suffix.lower() == ".mp4" and not p.name.endswith(".part"))
+    if len(files) < 2: return
+    now = time.time()
+    for p in files:                                   # still streaming? leave the whole set alone
+        if busy(p) or now - p.stat().st_mtime < args.idle_min * 60:
+            vglog("info", event="stitch_wait", stream=stream, parts=len(files), reason="stream still going")
+            return
+    if len(files) >= args.max_parts:
+        vglog("warn", event="stitch_skipped", stream=stream, parts=len(files), reason="too_many", max=args.max_parts)
+        return
+    night = datetime.now().strftime("%Y-%m-%d")
+    parts = []
+    for p in files:
+        t = arrival_ms(p.stem)
+        info = ffprobe(p)
+        if t is None:
+            vglog("warn", event="stitch_skipped", stream=stream, file=p, reason="name has no timestamp"); return
+        if info is None:
+            if not args.dry_run: dst = to_trash(root, p, night)
+            else: dst = "(dry run)"
+            vglog("warn", event="discarded", reason="unreadable", file=p, stream=stream, trash=dst)
+            continue
+        parts.append({"file": p, "arrival_ms": t, "params": info[0], "duration_ms": int(info[1] * 1000)})
+    parts.sort(key=lambda x: x["arrival_ms"])
+    # chains: consecutive, same parameters, gap under the limit
+    chains, cur = [], []
+    for part in parts:
+        if cur:
+            prev = cur[-1]
+            gap = part["arrival_ms"] - (prev["arrival_ms"] + prev["duration_ms"])
+            if part["params"] != prev["params"]:
+                vglog("info", event="stitch_break", stream=stream, reason="params_changed", file=part["file"],
+                      before=f"{prev['params'][1]}x{prev['params'][2]}", after=f"{part['params'][1]}x{part['params'][2]}")
+                chains.append(cur); cur = []
+            elif gap > args.max_gap_s * 1000:
+                vglog("info", event="stitch_break", stream=stream, reason="gap", gap_s=gap // 1000, file=part["file"])
+                chains.append(cur); cur = []
+        cur.append(part)
+    if cur: chains.append(cur)
+    for chain in chains:
+        if len(chain) < 2: continue
+        first = chain[0]["file"]
+        offset, joins, plist = 0, [], []
+        for i, part in enumerate(chain):
+            if i:
+                prev = chain[i - 1]
+                missing = part["arrival_ms"] - (prev["arrival_ms"] + prev["duration_ms"])
+                joins.append({"at_s": round(offset / 1000, 3), "missing_ms": max(0, missing)})
+            plist.append({"file": part["file"].name, "arrival_ms": part["arrival_ms"],
+                          "duration_ms": part["duration_ms"], "offset_ms": offset})
+            offset += part["duration_ms"]
+        if args.dry_run:
+            vglog("info", event="stitched", stream=stream, parts=len(chain), joins=len(joins), dry_run=1,
+                  missing_ms=sum(j["missing_ms"] for j in joins), out=first); continue
+        listing = live / f"{first.stem}.concat.txt"
+        tmp = live / f"{first.stem}.stitch.part"
+        listing.write_text("".join("file '%s'\n" % str(p["file"]).replace("'", r"'\''") for p in chain))
+        run = subprocess.run(["nice", "-n", "15", "ionice", "-c", "3", "ffmpeg", "-nostdin", "-hide_banner", "-loglevel", "error", "-y",
+                              "-f", "concat", "-safe", "0", "-i", str(listing), "-map", "0", "-c", "copy",
+                              "-movflags", "+faststart", "-f", "mp4", str(tmp)], capture_output=True, text=True)
+        listing.unlink(missing_ok=True)
+        got = ffprobe(tmp) if run.returncode == 0 else None
+        want = offset / 1000
+        if got is None or abs(got[1] - want) > 2 + len(chain):
+            tmp.unlink(missing_ok=True)
+            why = (run.stderr.strip().splitlines() or ["no stderr"])[-1][:200]
+            vglog("error", event="stitch_failed", stream=stream, parts=len(chain), rc=run.returncode,
+                  want_s=int(want), got_s=int(got[1]) if got else "?", reason=why)
+            continue                                    # fragments untouched: the sweep publishes them one by one
+        last_mtime = max(p["file"].stat().st_mtime for p in chain)
+        for p in chain: to_trash(root, p["file"], night)
+        final = live / f"{first.stem}.mp4"
+        os.replace(tmp, final)
+        os.utime(final, (last_mtime, last_mtime))        # keep it "idle" so this very sweep publishes it
+        meta = {"parts": plist, "joins": joins, "missing_ms": sum(j["missing_ms"] for j in joins),
+                "trash": str(root / "trash" / night / live.relative_to(root)), "stitched_at": datetime.now(TZ).isoformat(timespec="seconds")}
+        (live / f"{first.stem}.stitch.json").write_text(json.dumps(meta))
+        vglog("info", event="stitched", stream=stream, parts=len(chain), joins=len(joins),
+              missing_ms=meta["missing_ms"], seconds=int(want), out=final, trash=meta["trash"])
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("root")
+    ap.add_argument("--max-parts", type=int, default=500)
+    ap.add_argument("--max-gap-s", type=int, default=600)
+    ap.add_argument("--idle-min", type=int, default=3)
+    ap.add_argument("--dry-run", action="store_true")
+    args = ap.parse_args()
+    root = Path(args.root)
+    streams = root / "streams"
+    if not streams.is_dir(): return 0
+    for live in sorted(streams.glob("*/*/.live")):
+        if live.is_dir():
+            try: stitch_dir(root, live, args)
+            except Exception as error:
+                vglog("error", event="stitch_failed", dir=live, reason=str(error)[:200])
+    return 0
+
+if __name__ == "__main__":
+    sys.exit(main())

bin/vgstrip.py

@@ -0,0 +1,191 @@
+#!/usr/bin/env python3
+"""Vanguard timecode strip — shared format (encoder for tests, decoder for the sweep).
+
+The published canvas is the game image plus an "overscan band" of extra rows at the bottom. Everything is
+expressed relative to the canvas WIDTH so any uniform rescale keeps the geometry:
+
+    cell  = W / 160          (16 px at 2560, 12 px at 1920)
+    band  = round_even(cell) rows at the bottom of the frame
+    strip = 64 cells starting at x = cell, vertically centred in the band:
+            [1,0,1,0] framing · 48-bit server time in ms (MSB first) · 8-bit CRC-8 (poly 0x07) · [0,1,0,1] framing
+            1 = white, 0 = black
+    text  = human-readable server clock to the right of the strip (not decoded)
+
+Decode: sample the centre of each cell on the row y = H - band/2; threshold at the midpoint of the framing
+cells' brightness; verify framing + CRC. The decoded value is SERVER time at render; frame-0 capture time =
+median(decoded_k - pts_k) over several frames.
+"""
+import sys, json, struct, subprocess, statistics
+
+CELLS = 64
+FRAME_A = [1, 0, 1, 0]
+FRAME_B = [0, 1, 0, 1]
+
+def crc8(data: bytes, poly=0x07) -> int:
+    c = 0
+    for b in data:
+        c ^= b
+        for _ in range(8):
+            c = ((c << 1) ^ poly) & 0xFF if c & 0x80 else (c << 1) & 0xFF
+    return c
+
+def encode_bits(ms: int):
+    payload = ms.to_bytes(6, "big")
+    bits = [(payload[i // 8] >> (7 - i % 8)) & 1 for i in range(48)]
+    c = crc8(payload)
+    bits += [(c >> (7 - i)) & 1 for i in range(8)]
+    return FRAME_A + bits + FRAME_B
+
+def decode_bits(bits):
+    if bits[:4] != FRAME_A or bits[-4:] != FRAME_B: return None
+    body = bits[4:-4]
+    payload = bytes(int("".join(map(str, body[i:i + 8])), 2) for i in range(0, 48, 8))
+    c = int("".join(map(str, body[48:56])), 2)
+    if crc8(payload) != c: return None
+    return int.from_bytes(payload, "big")
+
+def geometry(w):
+    cell = w / 160.0
+    band = int(round(cell)); band += band % 2
+    return cell, band
+
+def cell_centres(w):
+    cell, band = geometry(w)
+    return [int(round(cell * (1 + i + 0.5))) for i in range(CELLS)], band
+
+# ------------------------------------------------------------------ decoder over a raw gray band
+def decode_band_row(row: bytes, w: int):
+    """row = one grayscale row (w bytes) taken from the middle of the band."""
+    xs, _ = cell_centres(w)
+    vals = []
+    for x in xs:
+        lo, hi = max(0, x - 1), min(w, x + 2)
+        vals.append(sum(row[lo:hi]) / max(1, hi - lo))
+    fr = vals[:4] + vals[-4:]
+    thr = (min(fr) + max(fr)) / 2.0
+    if max(fr) - min(fr) < 40: return None            # no contrast: strip absent
+    return decode_bits([1 if v > thr else 0 for v in vals])
+
+def probe(path):
+    out = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries",
+                          "stream=width,height,avg_frame_rate,r_frame_rate:format=duration", "-of", "json", path],
+                         capture_output=True, text=True, check=True).stdout
+    j = json.loads(out); st = j["streams"][0]
+    # nominal rate (r_frame_rate) — avg_frame_rate is skewed by dropped frames / gaps
+    def rate(x):
+        try: num, den = x.split("/"); return float(num) / float(den or 1)
+        except Exception: return 0.0
+    fps = rate(st.get("r_frame_rate", "")) or rate(st.get("avg_frame_rate", "")) or 60.0
+    if fps > 240: fps = rate(st.get("avg_frame_rate", "")) or 60.0      # bogus r_frame_rate on some files
+    return int(st["width"]), int(st["height"]), fps, float(j["format"]["duration"])
+
+def sample_window(path, w, h, fps, start_s, dur_s):
+    """Decode every frame in [start, start+dur). Returns [(pts_s, server_ms)]."""
+    import re
+    _, band = geometry(w)
+    y0 = h - band
+    # -copyts keeps the file's own timestamps; showinfo prints the exact pts of every emitted frame
+    cmd = ["ffmpeg", "-nostdin", "-v", "info", "-copyts", "-ss", f"{start_s:.3f}", "-t", f"{dur_s:.3f}", "-i", path,
+           "-an", "-vf", f"crop={w}:{band}:0:{y0},format=gray,showinfo", "-f", "rawvideo", "-"]
+    r = subprocess.run(cmd, capture_output=True, check=True)
+    raw = r.stdout
+    pts = [float(m) for m in re.findall(r"\bpts_time:\s*([0-9.]+)", r.stderr.decode(errors="replace"))]
+    frame_len = w * band
+    res = []
+    for k in range(min(len(raw) // frame_len, len(pts))):
+        row = raw[k * frame_len + (band // 2) * w: k * frame_len + (band // 2) * w + w]
+        ms = decode_band_row(row, w)
+        if ms is not None:
+            res.append((pts[k], ms))
+    return res
+
+def first_pts(path):
+    """pts of the first video frame (fMP4 from x264 with B-frames starts ~2 frames after 0)."""
+    out = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries", "frame=pts_time",
+                          "-read_intervals", "%+#1", "-of", "csv=p=0", path], capture_output=True, text=True).stdout.strip()
+    try: return float(out.splitlines()[0].strip(",").split(",")[0])
+    except Exception: return 0.0
+
+def analyse(path, win_s=4.0):
+    w, h, fps, dur = probe(path)
+    p0 = first_pts(path)                         # timecode start is the FIRST FRAME, not pts 0
+    windows = [0.0]
+    if dur > 60: windows.append(max(0.0, dur / 2 - win_s / 2))
+    if dur > 20: windows.append(max(0.0, dur - win_s - 0.5))
+    samples = []
+    for ws in windows:
+        pts = sample_window(path, w, h, fps, ws, win_s)
+        if pts:
+            offs = [ms - (pts_s - p0) * 1000.0 for pts_s, ms in pts]
+            samples.append({"at_s": round(ws, 3), "frames": len(pts), "t0_ms": int(statistics.median(offs))})
+    if not samples:
+        return {"source": "none", "fps": fps, "width": w, "height": h, "duration_s": dur}
+    t0 = samples[0]["t0_ms"]
+    drift = samples[-1]["t0_ms"] - t0 if len(samples) > 1 else 0
+    return {"source": "strip", "t0_ms": t0, "drift_ms": drift, "fps": fps, "width": w, "height": h,
+            "duration_s": dur, "samples": samples}
+
+def find_gaps(path, fps, max_list=50):
+    """Holes in the video timeline (dropped frames): deltas > 1.5 frame periods. Demux only, fast."""
+    out = subprocess.run(["ffprobe", "-v", "error", "-select_streams", "v:0", "-show_entries", "packet=pts_time",
+                          "-of", "csv=p=0", path], capture_output=True, text=True).stdout
+    pts = sorted(float(x.strip(",").split(",")[0]) for x in out.split() if x.strip(",") not in ("", "N/A"))
+    period = 1.0 / (fps or 60.0)
+    gaps, total = [], 0.0
+    for a, b in zip(pts, pts[1:]):
+        d = b - a
+        if d > period * 1.5:
+            total += d - period
+            if len(gaps) < max_list: gaps.append({"at_s": round(a, 3), "missing_ms": int(round((d - period) * 1000))})
+    return {"count": sum(1 for a, b in zip(pts, pts[1:]) if b - a > period * 1.5), "total_missing_ms": int(round(total * 1000)), "first": gaps}
+
+def timecode_string(t0_ms, fps, tz="America/New_York"):
+    """Time-of-day timecode for the tmcd track: HH:MM:SS:FF (drop-frame ';' for 29.97/59.94)."""
+    from datetime import datetime, timezone
+    from zoneinfo import ZoneInfo
+    local = datetime.fromtimestamp(t0_ms / 1000.0, timezone.utc).astimezone(ZoneInfo(tz))
+    frac = (t0_ms % 1000) / 1000.0
+    nominal = max(1, round(fps))
+    ff = min(nominal - 1, int(frac * fps))
+    drop = any(abs(fps - r) < 0.05 for r in (29.97, 59.94, 23.976))   # true NTSC rates only
+    sep = ";" if drop else ":"
+    return f"{local:%H:%M:%S}{sep}{ff:02d}", local
+
+def plan(path, arrival_ms=None, tz="America/New_York"):
+    """Decide the timecode for a recording: strip if present, else server arrival time. Prints shell vars."""
+    from datetime import datetime, timezone
+    r = analyse(path)
+    if r.get("source") == "strip":
+        t0 = r["t0_ms"]; src = "strip"
+    elif arrival_ms:
+        t0 = int(arrival_ms); src = "arrival"
+    else:
+        t0 = None; src = "none"
+    out = {"source": src, "fps": r.get("fps"), "drift_ms": r.get("drift_ms", 0), "samples": r.get("samples", [])}
+    if src == "strip":
+        # the band is removed (cropped off in a re-encode) by the sweep once it has been decoded; record it
+        _, band = geometry(r["width"]) if r.get("width") else geometry(probe(path)[0])
+        out.update({"band_rows": band, "band_removed": True})
+        print(f"TC_BAND={band}")
+    try:
+        out["gaps"] = find_gaps(path, r.get("fps") or 60.0)
+    except Exception:
+        pass
+    if t0 is not None:
+        tc, local = timecode_string(t0, r.get("fps") or 60.0, tz)
+        utc = datetime.fromtimestamp(t0 / 1000.0, timezone.utc)
+        out.update({"t0_ms": t0, "timestamp": utc.strftime("%Y-%m-%dT%H:%M:%S.") + f"{t0 % 1000:03d}Z",
+                    "timecode": tc, "timecode_tz": tz, "local_start": local.isoformat()})
+        print(f"TC_TIMECODE='{tc}'")              # quoted: drop-frame uses ';' which the shell would split on
+        print(f"TC_CREATION='{utc.strftime('%Y-%m-%dT%H:%M:%S')}.{t0 % 1000:03d}Z'")
+    print(f"TC_SOURCE='{src}'")
+    print("TC_JSON=" + json.dumps(json.dumps(out)))      # double-encoded: shell-safe single token
+    return out
+
+if __name__ == "__main__":
+    if len(sys.argv) < 2: print(__doc__); sys.exit(2)
+    if sys.argv[1] == "plan":
+        plan(sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 and sys.argv[3] else None); sys.exit(0)
+    if sys.argv[1] == "band":                 # band rows for a canvas width (vgdeband.sh)
+        print(geometry(int(sys.argv[2]))[1]); sys.exit(0)
+    print(json.dumps(analyse(sys.argv[1]), indent=2))

bin/vgstuck.sh

@@ -0,0 +1,207 @@
+#!/bin/bash
+# Triage recordings that never published: see why each one is stuck, then retry it or bin it.
+#
+#   vgstuck.sh                     list everything stuck, with the reason        (read-only, the default)
+#   vgstuck.sh list --durations    ...and measure each one's length (slow: fragmented MP4 is read end to end)
+#   vgstuck.sh retry <match>       clear what is blocking it so the sweep tries again
+#   vgstuck.sh retry --all         ...for everything that is parked
+#   vgstuck.sh drop  <match>       move the recording to trash: 3-day undo, same as "dump tonight"
+#   vgstuck.sh list  --under 5     only recordings smaller than 5 MB (reconnect fragments)
+#   vgstuck.sh drop  --under 5     bin all of those; still-recording files are never touched
+#   vgstuck.sh wipe                bin EVERYTHING unprocessed: stops the sweep (killing any conversion in flight),
+#                                  then moves every .live/ file — parked, waiting, published-collision, fresh — with
+#                                  its sidecars, failure logs and .part leftovers to trash, then drops the emptied
+#                                  folders. The ONLY thing left alone is a file MediaMTX has open, i.e. a stream
+#                                  being recorded this minute. The Dump tab's "Bin all unprocessed" button runs this.
+#   vgstuck.sh sweep               kick the sweep once, without waiting on it
+#   vgstuck.sh json                the same survey as JSON (what dumpd serves to the web page)
+#
+# <match> is any substring of the path (a streamer name, a title, a timestamp). It must match at least one
+# recording, and `drop` refuses --all so a stray pattern cannot bin everything by accident: the deliberate
+# version is spelled out as its own word, `wipe`.
+#
+# WHY A RECORDING STAYS IN .live/
+#   parked     a <Title>__<ts>.normalize-failed.log sits next to the intended output. The sweep skips the
+#              file entirely until that log is deleted, so it will never retry on its own.
+#   published  an output <Title>__<ts>.mp4 already exists, so the sweep leaves the source alone.
+#   waiting    nothing is blocking it; it is simply newer than MINS_IDLE or the sweep has not run yet.
+#   partial    a leftover <output>.mp4.part. This does NOT block anything (ffmpeg overwrites it with -y);
+#              it only wastes disk, and retry removes it.
+#
+# Nothing here deletes media. `drop` MOVES to /srv/media/trash/<today>/, which trash-purge clears after
+# three days, so a wrong call is recoverable for the same window as a dumped night.
+set -uo pipefail
+
+ROOT="${MEDIA_ROOT:-/srv/media}"
+TRASH="$ROOT/trash"
+MINS_IDLE="${MINS_IDLE:-3}"
+
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgstuck %s\n' "$(date -Is)" "$lvl" "$*"; }
+
+# Everything about one .live source, as: src|out|state|age_s|bytes|dur_s|reason
+survey() {
+  local src titledir user title stamp base out state reason dur
+  while IFS= read -r -d '' src; do
+    titledir="$(dirname "$(dirname "$src")")"
+    user="$(basename "$(dirname "$titledir")")"
+    title="$(basename "$titledir")"
+    stamp="$(basename "${src%.*}")"
+    base="$(dirname "$titledir")/${title}__${stamp}"
+    out="$base.mp4"
+    reason=""
+    if [[ -e "$base.normalize-failed.log" ]]; then
+      state=parked
+      reason="$(head -1 "$base.normalize-failed.log" 2>/dev/null)"
+      local why; why=$(grep -A2 '^--- ffmpeg stderr ---$' "$base.normalize-failed.log" 2>/dev/null | sed -n '2p')
+      [[ -n ${why:-} ]] && reason="$reason :: $why"
+    elif [[ -e "$out" ]]; then
+      state=published
+      reason="output already exists: $(basename "$out")"
+    elif [[ -n $(find "$src" -mmin "-$MINS_IDLE" -print -quit 2>/dev/null) ]]; then
+      state=writing
+      reason="touched in the last ${MINS_IDLE}m — still recording"
+    else
+      state=waiting
+      reason="nothing blocking it; the sweep has not got to it"
+    fi
+    [[ -e "$out.part" ]] && reason="$reason [leftover .part]"
+    # Duration is OPT-IN. Everything in .live/ is fragmented MP4 with no duration in its header, so ffprobe
+    # has to read the whole file to answer — gigabytes per recording. With many stuck files the survey took
+    # over three minutes, dumpd's request timed out, and the web page broke. Size and age come from stat.
+    dur=""
+    if [[ ${VGSTUCK_DURATIONS:-0} == 1 ]]; then
+      dur=$(ffprobe -v error -show_entries format=duration -of csv=p=0 "$src" 2>/dev/null | cut -d. -f1)
+    fi
+    printf '%s|%s|%s|%s|%s|%s|%s\n' "$src" "$out" "$state" \
+      "$(( $(date +%s) - $(stat -c %Y "$src") ))" "$(stat -c %s "$src")" "${dur:--}" "$reason"
+  done < <(find "$ROOT/streams" -path '*/.live/*' -name '*.mp4' -print0 2>/dev/null)
+}
+
+show() {
+  local any=0 total=0 src out state age bytes dur reason
+  printf '\n  %-9s %-9s %9s %8s  %s\n' state waiting length size recording
+  while IFS='|' read -r src out state age bytes dur reason; do
+    total=$(( total + bytes ))
+    any=1
+    if [[ $dur == - || -z $dur ]]; then length="-"; else
+      length=$(printf '%d:%02d:%02d' $(( dur / 3600 )) $(( (dur % 3600) / 60 )) $(( dur % 60 ))); fi
+    printf '  %-9s %-9s %9s %7.0fM  %s\n' "$state" \
+      "$(( age / 3600 ))h$(printf '%02d' $(( (age % 3600) / 60 )))m" \
+      "$length" \
+      "$(awk -v b="$bytes" 'BEGIN{print b/1000000}')" "${src#"$ROOT"/streams/}"
+    printf '            %s\n' "$reason"
+  done < <(selected "${1:---all}" "${2:-}" | sort -t'|' -k3,3 -k4,4nr)
+  (( any )) || printf '  nothing matches.\n'
+  (( any )) && printf '  %s item(s), %s GB\n' "$(selected "${1:---all}" "${2:-}" | wc -l)" "$(awk -v b="$total" 'BEGIN{printf "%.1f", b/1e9}')"
+  [[ ${VGSTUCK_DURATIONS:-0} == 1 ]] || printf '\n  lengths not measured (slow on fragmented MP4): vgstuck.sh list --durations\n'
+  printf '  retry one:  vgstuck.sh retry <substring>      bin one:  vgstuck.sh drop <substring>\n\n'
+}
+
+# rows matching the pattern, everything when --all, or everything smaller than N MB with --under N.
+# --under exists for reconnect fragments: every time a publisher drops and comes back MediaMTX opens a new
+# recording, so a bad night leaves hundreds of one- and two-second files. Still-recording files never match.
+selected() {
+  local pattern="$1" limit="${2:-}"
+  if [[ $pattern == --under ]]; then
+    [[ $limit =~ ^[0-9]+$ ]] || { echo "usage: --under <megabytes>" >&2; return 2; }
+    survey | awk -F'|' -v max="$(( limit * 1000000 ))" '$3 != "writing" && $5 < max'
+  elif [[ $pattern == --all ]]; then survey
+  else survey | grep -F -- "$pattern" || true
+  fi
+}
+
+do_retry() {
+  local pattern="$1" limit="${2:-}" n=0 src out state age bytes dur reason
+  while IFS='|' read -r src out state age bytes dur reason; do
+    [[ $state == writing ]] && { vglog info "event=skip reason=still_recording file=\"$src\""; continue; }
+    rm -f "$out.part"                                  # leftover partial output: harmless, but no reason to keep
+    rm -f "${out%.mp4}.normalize-failed.log"           # THIS is what makes the sweep skip the file forever
+    n=$((n + 1))
+    vglog info "event=retry file=\"$src\" was=$state"
+  done < <(selected "$pattern" "$limit")
+  (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; }
+  vglog info "event=retry_done count=$n msg=\"run vgstuck.sh sweep, or wait for the timer\""
+}
+
+# one row per line on stdin (survey format) -> trash. Prints the count it moved.
+drop_rows() {
+  local night="$1" n=0 src out state age bytes dur reason rel dst victim
+  while IFS='|' read -r src out state age bytes dur reason; do
+    # The survey labels a file by its blocker first (a parked file stays "parked" even while being written), so the
+    # safety check here does not trust the label: open by any process, or touched inside MINS_IDLE, means leave it.
+    # In wipe mode (WIPE=1) only "open" counts — the sweep has been stopped, so an open file is MediaMTX recording.
+    if fuser -s "$src" 2>/dev/null || { [[ ${WIPE:-0} != 1 ]] && { [[ $state == writing ]] || [[ -n $(find "$src" -mmin "-$MINS_IDLE" -print -quit 2>/dev/null) ]]; }; }; then
+      vglog info "event=skip reason=still_recording file=\"$src\"" >&2; continue
+    fi
+    for victim in "$src" "${src%.*}.json" "${src%.*}.stitch.json" "${out%.mp4}.normalize-failed.log" "$out.part"; do
+      [[ -e $victim ]] || continue
+      rel="${victim#"$ROOT"/}"
+      dst="$TRASH/$night/$rel"
+      mkdir -p "$(dirname "$dst")"
+      mv -f "$victim" "$dst" && vglog info "event=trashed file=\"$rel\" undo_days=3" >&2
+    done
+    rmdir "$(dirname "$src")" "$(dirname "$(dirname "$src")")" 2>/dev/null || true
+    n=$((n + 1))
+  done
+  echo "$n"
+}
+
+do_drop() {
+  local pattern="$1" limit="${2:-}" night n
+  [[ $pattern == --all ]] && { vglog error 'event=refused msg="drop --all is not allowed; name them, use --under <MB>, or wipe"' >&2; return 2; }
+  night="$(date +%F)"
+  n=$(selected "$pattern" "$limit" | drop_rows "$night")          # per-file lines go to stderr, the count to stdout
+  (( n )) || { vglog warn "event=nothing_matched pattern=\"$pattern\""; return 1; }
+  vglog info "event=drop_done count=$n trash=\"$TRASH/$night\""
+}
+
+# Everything unprocessed, deliberately. First the sweep is stopped, which kills any ffmpeg conversion it has in
+# flight (the timer starts it again within five minutes, to an empty queue). Then every .live/ file goes, parked
+# or fresh, and so do failure logs whose recording is already gone (they would keep the Stuck tab listing ghosts)
+# and leftover .part outputs. The one exception: a file some process still holds open after the sweep is down,
+# which is MediaMTX recording a live stream — pulling that from under it would lose the stream in progress.
+do_wipe() {
+  local night n stale=0 log rel dst
+  night="$(date +%F)"
+  if systemctl is-active --quiet normalize-sweep.service; then
+    systemctl stop normalize-sweep.service && vglog info "event=cancelled unit=normalize-sweep.service msg=\"conversion in flight killed\""
+  fi
+  systemctl is-active --quiet vgdeband.service && systemctl stop vgdeband.service && vglog info "event=cancelled unit=vgdeband.service"
+  n=$(survey | WIPE=1 drop_rows "$night")
+  while IFS= read -r -d '' log; do
+    rel="${log#"$ROOT"/}"; dst="$TRASH/$night/$rel"; mkdir -p "$(dirname "$dst")"
+    mv -f "$log" "$dst" && stale=$((stale + 1))
+  done < <(find "$ROOT/streams" -maxdepth 2 \( -name '*.normalize-failed.log' -o -name '*.mp4.part' \) -print0 2>/dev/null)
+  find "$ROOT/streams" -mindepth 2 -type d -name .live -empty -delete 2>/dev/null
+  find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type d -empty -delete 2>/dev/null
+  vglog info "event=wipe_done recordings=${n:-0} stale_files=$stale trash=\"$TRASH/$night\" undo_days=3"
+}
+
+# Same survey, as JSON. bin/dumpd.py serves this to the web page, so the classification lives in exactly
+# one place and the browser and the CLI can never disagree about why something is stuck.
+emit_json() {
+  survey | python3 -c '
+import json, sys
+rows = []
+for line in sys.stdin:
+    parts = line.rstrip("\n").split("|", 6)
+    if len(parts) != 7:
+        continue
+    src, out, state, age, size, dur, reason = parts
+    rows.append({"src": src, "out": out, "state": state, "age_s": int(age or 0),
+                 "bytes": int(size or 0),
+                 "duration_s": int(dur) if dur not in ("", "-") else None, "reason": reason})
+print(json.dumps(rows))'
+}
+
+for a in "$@"; do [[ $a == --durations ]] && export VGSTUCK_DURATIONS=1; done
+
+case "${1:-list}" in
+  list|"")  show "${2:---all}" "${3:-}" ;;
+  json)     emit_json ;;
+  retry)    [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh retry <substring>|--all|--under <MB>" >&2; exit 1; }; do_retry "$2" "${3:-}" ;;
+  drop)     [[ -n ${2:-} ]] || { echo "usage: vgstuck.sh drop <substring>|--under <MB>" >&2; exit 1; }; do_drop "$2" "${3:-}" ;;
+  wipe)     do_wipe ;;
+  sweep)    systemctl start --no-block normalize-sweep.service && vglog info "event=sweep_started" ;;
+  *)        sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'; exit 1 ;;
+esac

bot/.dockerignore

@@ -1,2 +1,4 @@
-app/__pycache__/
+# only entrypoint.sh is COPYed; keep everything else out of the build context
-app/*.pyc
+app/
+AGENT.md
+README.md

bot/AGENT.md

@@ -15,23 +15,54 @@ What you are given:
 
 | | |
 |---|---|
-| **Runtime** | `python:3.12-slim` + ffmpeg + curl. Dependencies from `app/requirements.txt`, installed at image build (`docker compose build discord-bot`). |
+| **Runtime** | `python:3.12-slim` + ffmpeg + curl, and nothing else. **Your dependencies ship with your code** (see §1a), imported from `/app/vendor` via `PYTHONPATH`. Adding one needs no image rebuild. |
-| **Your code** | bind-mounted at `/app` from `bot/app/`. Not baked into the image. Edit, then `docker compose restart discord-bot`. |
+| **Your code** | a release bundle from your CI, installed at `/srv/bot/app` and bind-mounted at `/app` (§1a). Treat `/app` as read-only; write to `/data`. |
-| **Entry point** | the container runs `python -u /app/main.py` and restarts it if it exits. With no `main.py` it idles. |
+| **Entry point** | the container runs `python -u /app/main.py` and restarts it if it exits. With no bundle installed it idles and logs a hint. |
 | **User** | uid 1000 |
 | **Database** | MariaDB, database `vanguardbot`, user `bot`, reachable at `DB_HOST:DB_PORT`. Yours alone. Other databases on the same server belong to other services; never touch them. |
 | **Persistent disk** | `/data`, yours |
-| **Web UI** | bind `WEB_HOST:WEB_PORT` (0.0.0.0:8000). The proxy publishes it as `https://bot.<domain>/` behind login (§3). |
+| **Web UI** | bind `WEB_HOST:WEB_PORT` (0.0.0.0:8000) — one port for your API *and* your pages. The proxy publishes it as `https://bot.<domain>/` behind login (§3). |
 | **Discord** | `DISCORD_BOT_TOKEN`, `DISCORD_GUILD_ID`. The bot user belongs to the community's existing Discord application. |
 | **Clock** | `TZ=America/New_York` |
 
 ### Environment variables
-`DISCORD_BOT_TOKEN`, `DISCORD_GUILD_ID`, `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`,
+`DISCORD_BOT_TOKEN`, `DISCORD_GUILD_ID`, `DB_BACKEND=mariadb`, `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`,
-`WEB_HOST`, `WEB_PORT`, `WEB_PUBLIC_URL`, `AUTH_HEADER_USER`, `AUTH_HEADER_GROUPS`, `AUTH_HEADER_EMAIL`,
+`DB_PASSWORD`, `DATA_DIR=/data`, `WEB_HOST`, `WEB_PORT`, `WEB_PUBLIC_URL`, `WEB_SERVE_UI=true`,
-`MEDIA_CLIPS`, `TZ`. Secrets arrive only through env. Never write them to the repo, `/data`, or logs.
+`WEB_ALLOWED_GROUPS=bot-users,admins`, `AUTH_HEADER_USER`, `AUTH_HEADER_GROUPS`, `AUTH_HEADER_EMAIL`,
+`MEDIA_CLIPS`, `PYTHONPATH=/app/vendor`, `TZ`. Secrets arrive only through env. Never write them to the repo,
+`/data`, or logs.
 
 SQLAlchemy URL: `mysql+pymysql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}` (async: `mysql+aiomysql`).
 
+## 1a. Releases: one bundle, dependencies included
+
+Your CI (`.gitea/workflows/ci.yml`, job `bundle`) runs after the tests and produces the artifact
+`vanguard-bot-bundle` — a single tar.gz whose contents **are** `/app`:
+
+    main.py  app/  website/  version.json  BUILD.json  vendor/
+
+`vendor/` is `pip install --only-binary=:all: --target vendor -r requirements.txt`, built on `python:3.12`
+so the wheels match the runtime image's ABI. `BUILD.json` records the version, commit, build time, the
+package list and a sha256 of every file.
+
+**A push to `main` deploys itself.** The workflow's last job posts the bundle to the server's deploy service,
+signed with the repository secret `BOT_DEPLOY_SECRET`; the server verifies the signature and every checksum,
+keeps the previous release for rollback, and restarts the container. The install log comes back in your CI
+job, so a failed deploy fails the run. Branches and pull requests test and build, and deploy nothing.
+
+**Your settings travel with it.** `DISCORD_BOT_TOKEN` (an Actions secret) and `DISCORD_GUILD_ID` (an Actions
+variable) are signed together with the bundle and written into the container's environment by the server, so
+this repo is where they are managed and rotated. The server accepts only the names its `BOT_ENV_KEYS` allows
+and reports the rest as `settings_refused`; it keeps ownership of what it generates, such as `DB_PASSWORD`.
+A new setting means asking the admin to add its name to that list (§6).
+
+Rules that follow from this:
+
+- Keep `requirements.txt` resolvable with **prebuilt wheels only**; a dependency that must compile will fail
+  the build. Ask the admin if you truly need one.
+- Don't import anything that is not in `requirements.txt` or the standard library.
+- Nothing is installed on the server at deploy time, and the server reaches no package index.
+
 ## 2. Process model
 
 - One process, one asyncio loop: the Discord gateway client and the web server run together
@@ -52,6 +83,10 @@ Every request to `https://bot.<domain>/` passes through Caddy, which sends the u
 | `X-Authentik-Groups` | group names, separated by `\|` or `,` — the community uses `admins`, `editors`, `streamers`; others may be added |
 | `X-Authentik-Email`, `X-Authentik-Name`, `X-Authentik-Uid` | extras |
 
+`WEB_SERVE_UI=true` is set on the server, so this process serves your HTML pages at `/` and their assets
+at `/assets/` as well as `/api/…` — same origin, one port, no second service and no proxy of your own.
+Locally the separate website launcher still works; it is off unless `WEB_SERVE_UI` says otherwise.
+
 Rules:
 1. **Implement no login page, no sessions, no passwords.** Identity is the headers, per request.
 2. Port 8000 is loopback-only on the host, so the headers can be trusted. Still fail closed: a request
@@ -59,8 +94,11 @@ Rules:
 3. Authorize by group, read from the header each time. Don't cache roles in your DB.
 4. Serve under `/`; the proxy does no path rewriting for this host.
 5. Build absolute links from `WEB_PUBLIC_URL`.
-6. The Authentik username may differ from the Discord username. Use the header value as the user key;
+6. **The Discord snowflake arrives in `X-Authentik-Jwt`** as the `discord_id` claim (a scope mapping on the
-   if you need the Discord snowflake, resolve it through your gateway client and cache it in your DB.
+   provider supplies it; Authentik takes it from the account's source connection). Decode the JWT payload and
+   use that as the user key. Do not match the Authentik username against Discord names: usernames are mutable
+   and reusable, and server nicknames are self-chosen and not unique, so name matching can mistake one member
+   for another. The signature needs no checking because the header can only come from the proxy.
 
 Until the admin creates the matching Authentik application, the hostname returns an error, never an
 open page. Test against `http://127.0.0.1:8000/` on the server with the headers set by hand.
@@ -99,10 +137,15 @@ message content; Message Content is a privileged intent, enable it only if you n
 
 ## 6. Working with the admin
 
-- **Source**: push to the community Gitea, `ssh://git@git.<domain>:2222/vanguard/discord-bot.git`.
+- **Source**: push to the community Gitea. CI builds the release bundle on the `python312` runner label
-  The admin deploys by pulling into `bot/app` and restarting. CI on the `python` runner label runs your tests.
+  (CPython 3.12 plus node) and, on `main`, installs it on the server. **The test jobs are commented out: run
+  the suites locally before you push** (`python -m pytest -q` and `node --test app/tests/*.test.js`). What CI
+  still enforces: the bundle must import cleanly with only its own `vendor/` on the path, every file must
+  match `BUILD.json`, and the bot must answer on its port within 45 s of the swap or the deploy fails and the
+  previous release stays rollback-ready.
 - **Need infra?** Open a Gitea issue titled `infra: …` stating the exact change: a new env var and why,
   a mount, a Caddy path that should skip login (e.g. a public webhook endpoint), a new database.
   Don't work around the proxy or the firewall.
-- **Dependencies changed?** Say so in the PR; the image must be rebuilt.
+- **Dependencies changed?** Just push: they travel inside the bundle. Only a new *system* package (beyond
+  ffmpeg and curl) needs the admin to rebuild the image.
 - Keep `README.md` updated with every command, env var, and table you add.

bot/Dockerfile

@@ -1,12 +1,11 @@
-# Python runtime for the Discord bot. Source is bind-mounted at /app from ./bot/app (not baked in),
+# Runtime for the Discord bot. The bot's CI builds a self-contained bundle (code + vendor/ dependencies)
-# so the dev edits code and restarts the container. Dependencies come from app/requirements.txt and
+# and deployBotBundle.sh installs it at BOT_APP_DIR, bind-mounted here at /app. Dependencies therefore live
-# are installed at BUILD time: after changing it run  docker compose build discord-bot && docker compose up -d discord-bot
+# WITH the code, not in this image: a new dependency needs no rebuild and no package index on the server.
+# Only the interpreter and the tools AGENT.md promises are baked in.
 FROM python:3.12-slim
-ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
+ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1 PYTHONDONTWRITEBYTECODE=1
 RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg ca-certificates curl \
  && rm -rf /var/lib/apt/lists/*
-COPY app/requirements.txt /tmp/requirements.txt
-RUN pip install -r /tmp/requirements.txt
 COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /entrypoint.sh
 WORKDIR /app

bot/README.md

@@ -2,42 +2,78 @@
 
 Start with **AGENT.md** in this folder: the stack, the rules, the layout, the web-UI auth contract.
 
-Your code goes in this folder (`bot/app/`). The container runs `python -u /app/main.py` and restarts
+## How code reaches the server
-it if it exits. No `main.py` yet → the container idles and logs a hint.
+
+Your repo's Gitea Actions workflow builds a **release bundle**: one tar.gz holding `main.py`, `app/`,
+`website/`, `version.json`, `BUILD.json` and `vendor/` (your dependencies as prebuilt wheels). **A push to
+`main` installs itself**: the workflow's last job posts the bundle to the server's deploy service, which
+checks its signature and every checksum, swaps it into `/srv/bot/app` (bind-mounted at `/app`, previous
+release kept for `--rollback`) and restarts the container. The install log comes back in the CI job.
+
+A branch or pull request builds the artifact and deploys nothing. The test jobs are commented out in the
+workflow — run `python -m pytest -q` and `node --test app/tests/*.test.js` locally before pushing. CI still
+refuses a bundle that cannot import from its own `vendor/`, or whose files do not match `BUILD.json`, and the
+deploy fails if the bot does not answer on its port after the swap.
+
+## Your secrets live in Gitea
+
+Set them on this repo under Settings → Actions. The deploy job sends them with the bundle, signed, and the
+server writes the ones it allows into the environment the container gets:
+
+| where | name | what |
+|---|---|---|
+| Secrets | `BOT_DEPLOY_SECRET` | signs the upload; must match the server (the admin has it) |
+| Secrets | `DISCORD_BOT_TOKEN` | the bot token |
+| Variables | `DISCORD_GUILD_ID` | the Discord server id |
+
+Rotating the token is an edit here plus a re-run of the workflow; nobody needs to touch the server. The
+server keeps what it generates, chiefly `DB_PASSWORD`, and accepts only the names its `BOT_ENV_KEYS` lists —
+anything else is reported in the job as `settings_refused`. Need a new setting? Ask the admin to add its name
+to that list, then add it here.
+
+The image contains **no** bot dependencies: the container imports yours from `/app/vendor`
+(`PYTHONPATH=/app/vendor`). Adding a dependency means editing `app/requirements.txt` and pushing — no image
+rebuild, no package index on the server. The image only changes if you need a new system package (ffmpeg and
+curl are already there); ask the admin for that.
 
 ## Day to day (on the server, in ~/vanguard-stream)
+    ./deployBotBundle.sh vanguard-bot-bundle.zip      # install a release by hand (zip or tar.gz)
+    ./deployBotBundle.sh --status                     # what is installed; container state
+    ./deployBotBundle.sh --rollback                   # previous release
     docker compose logs -f discord-bot                # live log
-    docker compose restart discord-bot            # after editing code
+    docker compose restart discord-bot                # restart without changing code
-    docker compose build discord-bot && docker compose up -d discord-bot   # after editing requirements.txt
+    docker compose exec discord-bot python            # a REPL inside the container (your vendor/ is on sys.path)
-    docker compose exec discord-bot python        # a REPL inside the container
-
-Edit files directly in `~/vanguard-stream/bot/app/` (they are bind-mounted) or scp them up.
 
 ## Environment the container gives you
 | var | value |
 |-----|-------|
-| `DISCORD_BOT_TOKEN` | from `.env` (bundle root) — the Bot added to the existing Authentik Discord application |
+| `DISCORD_BOT_TOKEN` | from `.env` (stack root) — the Bot added to the existing Authentik Discord application |
 | `DISCORD_GUILD_ID`  | the server |
+| `DB_BACKEND` | `mariadb` |
 | `DB_HOST` / `DB_PORT` | `host.docker.internal` / `3306` (MariaDB runs on the host network) |
 | `DB_NAME` / `DB_USER` / `DB_PASSWORD` | your own database + user (`vanguardbot` / `bot` / generated) |
+| `DATA_DIR` | `/data` — persistent, owned by you |
 | `MEDIA_CLIPS` | `/media/clips` — rw mount of `/srv/media/clips` (folder per user, for clip submissions) |
-| `/data` | persistent scratch dir owned by the bot |
+| `WEB_HOST` / `WEB_PORT` | `0.0.0.0` / `8000` — published as `https://bot.<domain>/` via Caddy + Authentik |
-| `WEB_HOST` / `WEB_PORT` | `0.0.0.0` / `8000` — your web UI; public as `https://bot.<domain>/` via Caddy + Authentik |
 | `WEB_PUBLIC_URL` | `https://bot.<domain>` for absolute links |
+| `WEB_SERVE_UI` | `true` — serve your pages at `/` on the same port as `/api` |
+| `WEB_ALLOWED_GROUPS` | `bot-users,admins` — the groups your own code must accept |
 | `AUTH_HEADER_USER` / `_GROUPS` / `_EMAIL` | `X-Authentik-Username` / `X-Authentik-Groups` / `X-Authentik-Email` — identity injected by Caddy, trust these, implement no login |
+| `PYTHONPATH` | `/app/vendor` |
 | `TZ` | America/New_York |
 
 SQLAlchemy URL: `mysql+pymysql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}`
 async: `mysql+aiomysql://...`
 
 ## Database from your own machine
-MariaDB is reachable directly: `ovh1.dumpstervigilantes.rip:3306`, user `bot`, database `vanguardbot`,
+MariaDB is reachable directly: `media.dumpstervigilantes.rip:3306`, user `bot`, database `vanguardbot`,
 password from the admin (`./configure.py --show` on the server). Root is local-only. New connections are
 rate limited at the firewall; if yours get dropped, ask the admin to add your IP to `db_allow`.
 
 ## Conventions
 - Run as uid 1000 (files you write under `/media/clips` are owned like everything else there).
+- `/app` holds a released bundle; treat it as read-only and write to `/data` instead.
 - If you write clips: `/media/clips/<discord-username>/<name>.mp4` plus `<name>.json` with
   `{"timestamp": "<UTC ISO-8601>", "source": "medal", "source_url": "...", "uploader": "...", "title": "..."}`
   — the Jellyfin plugin reads that sidecar for the exact recording time.
-- Log to stdout; the container captures it.
+- Log to stdout; the container captures it. One event per line, `key=value`, keeps it searchable (`LOGGING.md`).

bot/app/requirements.txt

@@ -1,8 +0,0 @@
-# Baseline for a discord.py bot with MariaDB. Add what you need; then:
-#   docker compose build discord-bot && docker compose up -d discord-bot
-discord.py>=2.4
-aiomysql>=0.2
-PyMySQL>=1.1
-SQLAlchemy>=2.0
-python-dotenv>=1.0
-yt-dlp

bot/entrypoint.sh

@@ -1,7 +1,13 @@
 #!/bin/sh
-# Run the bot if there is one; otherwise idle so the container stays up until code arrives.
+# /app is a bundle built by the bot's CI: main.py + app/ + website/ + vendor/ (its dependencies).
+# PYTHONPATH=/app/vendor is set by compose, so nothing is installed at run time.
 if [ -f /app/main.py ]; then
+  if [ ! -d /app/vendor ]; then
+    echo "discord-bot: /app has code but no vendor/ — this image installs no bot dependencies."
+    echo "             Install a CI bundle:  ./deployBotBundle.sh <vanguard-bot-bundle.tar.gz>"
+  fi
   exec python -u /app/main.py
 fi
-echo "discord-bot: no /app/main.py yet — drop the bot code into ./bot/app and 'docker compose restart discord-bot'"
+echo "discord-bot: /app is empty — install a bundle from the bot's CI:  ./deployBotBundle.sh <bundle.tar.gz>"
+echo "             (artifact 'vanguard-bot-bundle' on the VanguardBot repo's ci workflow)"
 exec sleep infinity

check.sh

@@ -0,0 +1,84 @@
+#!/bin/bash
+# Static checks for everything in the bundle. Run before committing / deploying.
+set -uo pipefail
+cd "$(dirname "$0")"
+fail=0
+say() { printf '%-28s %s\n' "$1" "$2"; }
+
+for f in install.sh addCommunityMediaServer.sh installJellyfinPlugin.sh registerGiteaRunner.sh createGiteaDb.sh deployLive.sh addClipUser.sh deploy.sh deployBotBundle.sh switchToClone.sh firewall/firewall.sh bin/*.sh home/*.sh bot/entrypoint.sh mariadb/init/*.sh; do
+  [[ -f $f ]] || continue
+  bash -n "$f" && say "$f" ok || { say "$f" "BASH SYNTAX"; fail=1; }
+done
+for f in configure.py bin/dumpd.py bin/vgstrip.py bin/vgstitch.py bin/vghosts-check.py observability/kibana-setup.py observability/kibana-alarms.py; do
+  python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
+done
+rm -rf __pycache__ bin/__pycache__
+# bin/install.list is the single source for what lands in /usr/local/bin. Fail if it names something that
+# does not exist, or if a file in bin/ is neither listed nor explicitly excluded — that drift is exactly how
+# deploy.sh ended up installing five of ten scripts.
+if python3 - <<'PYMAN' >/dev/null 2>&1
+import re, sys
+from pathlib import Path
+text = Path("bin/install.list").read_text()
+listed = dict((m.group(2), m.group(1)) for m in re.finditer(r"^(0\d{3}) (\S+)$", text, re.M))
+excluded = set(re.findall(r"^#\s+exclude (\S+)", text, re.M))
+present = set("bin/" + p.name for p in Path("bin").iterdir() if p.is_file() and p.name != "install.list")
+missing = set(listed) - present
+unclaimed = present - set(listed) - excluded
+assert not missing, f"install.list names files that do not exist: {sorted(missing)}"
+assert not unclaimed, f"bin/ files neither listed nor excluded in install.list: {sorted(unclaimed)}"
+PYMAN
+then say "bin/install.list" ok; else say "bin/install.list" "DRIFTED — see the assertion"; fail=1; fi
+# Only files that are actually here. The server checkout is partial (scp, not a full clone), and a tool that
+# reports "broken" when it means "absent" gets ignored — the same reason the deadman reports on change only.
+yamls=(); for f in docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml; do [[ -f $f ]] && yamls+=("$f"); done
+python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" "${yamls[@]}" \
+  && say "yaml (${#yamls[@]} files)" ok || { say "yaml" "INVALID"; fail=1; }
+if ! command -v node >/dev/null; then
+  say "page JS" "(skipped: node not installed here)"
+else
+  node -e "
+const fs=require('fs');
+for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html','streaming/index.html']) { if (!fs.existsSync(f)) continue; const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
+if (fs.existsSync('dock/clock/vgclock.js')) new Function(fs.readFileSync('dock/clock/vgclock.js','utf8'));" \
+    && say "page JS" ok || { say "page JS" "SYNTAX ERROR"; fail=1; }
+fi
+if [[ -f live/src/vgclock.js && -f dock/clock/vgclock.js ]]; then
+  diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') dock/clock/vgclock.js >/dev/null \
+    && say "vgclock.js in sync (dock/portal)" ok \
+    || say "vgclock.js in sync (dock/portal)" "DIFFERS — copy dock/clock/vgclock.js to live/src and re-add the export line"
+else
+  say "vgclock.js in sync" "(skipped: only one copy in this checkout)"
+fi
+# generated Caddyfile / mediamtx / copyparty from a dry configure run must at least be produced + parse
+T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "
+import configure as c, yaml
+c.write_caddy('media.example.com','auth.example.com','watch.example.com','git.example.com','bot.example.com','live.example.com','files.example.com','sync.example.com','clips.example.com')
+c.write_mediamtx('media.example.com'); yaml.safe_load(open('mediamtx.yml')); c.write_copyparty()
+assert 'authMethod: http' in open('mediamtx.yml').read(); assert 'idp-h-usr' in open('copyparty.conf').read()
+" ) && say "configure.py generators" ok || { say "configure.py generators" "FAILED"; fail=1; }
+rm -rf "$T"
+# dumpd GET /metrics: valid Prometheus text out of script counters + a fake media tree
+T=$(mktemp -d); mkdir -p "$T/.markers/metrics" "$T/streams/u/Title/.live"
+printf 'runs 2\njobs:encode:done:stream 1\nseconds:encode:stream 12.5\nlast_finish 1762210000\n' > "$T/.markers/metrics/sweep.counters"
+touch -d '2020-01-01' "$T/streams/u/Title/.live/old.mp4"
+if MEDIA_ROOT="$T" MTX_API_URL=http://127.0.0.1:1 python3 - <<'PYMET' >/dev/null 2>&1
+import re, sys
+sys.path.insert(0, "bin")
+import dumpd
+text = dumpd.metrics_text()
+for line in text.splitlines():
+    assert line.startswith("#") or re.fullmatch(r'[a-z_]+(\{[^}]*\})? -?[0-9.]+', line), line
+assert 'vanguard_media_jobs_total{component="sweep",kind="stream",outcome="done",verdict="encode"} 1' in text
+assert 'vanguard_media_last_finish_timestamp_seconds{component="sweep"} 1762210000' in text   # not 1.76e+09
+assert 'vanguard_media_pending{kind="stream"} 1' in text
+assert 'vanguard_mediamtx_up 0' in text and 'vanguard_streams_active' not in text             # no fake zeros
+PYMET
+then say "dumpd /metrics" ok; else say "dumpd /metrics" "FAILED"; fail=1; fi
+rm -rf "$T" bin/__pycache__
+if command -v caddy >/dev/null; then
+  T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "import configure as c; c.write_caddy('media.example.com','a.example.com','w.example.com','g.example.com','b.example.com','l.example.com','f.example.com','s.example.com','c.example.com')" && caddy validate --adapter caddyfile --config Caddyfile >/dev/null 2>&1 ) && say "Caddyfile (caddy validate)" ok || { say "Caddyfile (caddy validate)" "INVALID"; fail=1; }
+  rm -rf "$T"
+fi
+[[ -d live/node_modules ]] && ( cd live && npx --no-install vite build --logLevel error >/dev/null 2>&1 ) && say "live portal build" ok || say "live portal build" "(skipped or failed — run npm run build in live/)"
+exit $fail

clips/index.html

@@ -0,0 +1,97 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Vanguard Clips</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root{--bg:#1e1e1e;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#3d8bfd;--ok:#2ecc71;--bad:#e74c3c;--box:#2a2a2a}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
+  main{max-width:640px;margin:0 auto;padding:24px 16px}
+  h1{font-size:22px;margin:0 0 4px}.sub{color:var(--mut);margin-bottom:20px}
+  .card{background:var(--box);border:1px solid #3a3a3a;border-radius:12px;padding:18px;margin-bottom:14px}
+  .card h2{font-size:15px;margin:0 0 10px;color:var(--mut);text-transform:uppercase;letter-spacing:.06em}
+  ol{padding-left:20px;margin:8px 0}li{margin:6px 0}
+  code,.id{font-family:ui-monospace,monospace;background:#141414;padding:2px 6px;border-radius:5px;word-break:break-all}
+  .id{display:block;padding:10px;font-size:14px;margin:6px 0}
+  input{width:100%;box-sizing:border-box;padding:10px;border:1px solid #444;border-radius:8px;background:#141414;color:var(--fg);font:14px ui-monospace,monospace}
+  button{padding:10px 14px;border:0;border-radius:8px;background:var(--acc);color:#fff;font-weight:600;cursor:pointer;margin-top:10px}
+  button.small{background:#3a3a3a;font-weight:500;padding:6px 10px;margin:0 0 0 8px;font-size:13px}
+  .status{margin-top:10px;color:var(--mut);white-space:pre-wrap}.ok{color:var(--ok)}.bad{color:var(--bad)}
+  .pill{display:inline-block;padding:2px 8px;border-radius:999px;font-size:12px;background:#333;margin-right:6px}
+  .pill.on{background:#1e4620;color:#9f9}.pill.off{background:#4a1e1e;color:#f99}
+</style>
+</head>
+<body>
+<main>
+  <h1>Send your clips to Vanguard</h1>
+  <div class="sub">Logged in as <b id="who">…</b>. Your clips land in <code id="path">clips/…</code> and show up in Jellyfin under your name.</div>
+
+  <div class="card">
+    <h2>Status</h2>
+    <div id="pills"><span class="pill">loading…</span></div>
+    <div class="status" id="status"></div>
+  </div>
+
+  <div class="card">
+    <h2>1. Install Syncthing on your gaming PC</h2>
+    <ol>
+      <li>Windows: <a href="https://syncthing.net/downloads/" target="_blank" rel="noopener">Syncthing</a> or <a href="https://github.com/Martchus/syncthingtray/releases" target="_blank" rel="noopener">Syncthing Tray</a>. Mac/Linux: Syncthing.</li>
+      <li>Open its page (usually <code>http://localhost:8384</code>) → <b>Actions → Show ID</b>. Copy the Device ID.</li>
+    </ol>
+  </div>
+
+  <div class="card">
+    <h2>2. Paste your Device ID here</h2>
+    <input id="devid" placeholder="XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX-XXXXXXX" autocomplete="off" spellcheck="false">
+    <button id="go">Connect my device</button>
+    <div class="status" id="goStatus"></div>
+    <div class="sub" style="margin:10px 0 0">This creates your folder on the server, adds your device, and shares a receive-only folder named <code id="fid">clips-…</code> with you.</div>
+  </div>
+
+  <div class="card">
+    <h2>3. Accept on your PC</h2>
+    <ol>
+      <li>Your Syncthing shows <b>“New Device”</b> from <b>Vanguard</b> → Add Device. Our ID, for checking:
+        <span class="id" id="srvid">…</span><button class="small" id="copyid">copy</button></li>
+      <li>Then <b>“New Folder”</b> <code id="fid2">clips-…</code> → <b>Add</b>. Folder Path = the folder your clips save to
+        (ShadowPlay / Medal / Steam). Folder Type: <b>Send Only</b>. Save.</li>
+      <li>Done. Only video files are pulled; your folder layout is mirrored. The server converts odd formats and
+        keeps one copy per clip.</li>
+    </ol>
+  </div>
+</main>
+<script>
+const $ = id => document.getElementById(id);
+async function refresh(){
+  try {
+    const r = await fetch('/me/clips', {credentials:'same-origin'}); const d = await r.json();
+    if (!r.ok) throw new Error(d.error || r.status);
+    $('who').textContent = d.user; $('path').textContent = 'clips/' + d.user; $('fid').textContent = $('fid2').textContent = d.folder_id;
+    $('srvid').textContent = d.server_device_id || '(syncthing unreachable)';
+    const pills = [];
+    pills.push(`<span class="pill ${d.folder_exists?'on':'off'}">server folder ${d.folder_exists?'ready':'missing'}</span>`);
+    pills.push(`<span class="pill ${d.folder_configured?'on':'off'}">share ${d.folder_configured?'configured':'not yet'}</span>`);
+    if (d.device_id) pills.push(`<span class="pill ${d.device_connected?'on':'off'}">your device ${d.device_connected?'connected':'not connected'}</span>`);
+    if (d.files_here != null) pills.push(`<span class="pill on">${d.files_here} file(s) here</span>`);
+    if (d.quota_gb) pills.push(`<span class="pill ${d.paused_for_quota?'off':(d.used_gb > d.quota_gb*0.9?'':'on')}">${d.used_gb} / ${d.quota_gb} GB${d.paused_for_quota?' — PAUSED (over quota)':''}</span>`);
+    if (d.remote_completion != null && d.remote_completion < 100) pills.push(`<span class="pill">syncing ${Math.round(d.remote_completion)}%</span>`);
+    $('pills').innerHTML = pills.join('');
+    $('status').textContent = d.error ? d.error : (d.device_id ? `paired with ${d.device_name || 'your device'} (${d.device_id.slice(0,7)}…)` : 'not paired yet — do step 2');
+    if (d.paused_for_quota) $('status').textContent += `\nYour folder is over its ${d.quota_gb} GB quota, so syncing is paused. Delete older clips from your clip folder (they are removed here too) or ask an admin to raise the quota; syncing resumes automatically under 90 %.`;
+    if (d.device_id) $('devid').value = d.device_id;
+  } catch (e) { $('pills').innerHTML = `<span class="pill off">error</span>`; $('status').textContent = e.message; $('status').className = 'status bad'; }
+}
+$('go').onclick = async () => {
+  $('goStatus').textContent = 'working…'; $('goStatus').className = 'status';
+  try {
+    const r = await fetch('/me/clips', {method:'POST', credentials:'same-origin', headers:{'Content-Type':'application/json'}, body: JSON.stringify({device_id: $('devid').value})});
+    const d = await r.json(); if (!r.ok) throw new Error(d.error || r.status);
+    $('goStatus').textContent = 'done — now accept the device and the folder in your Syncthing (step 3)'; $('goStatus').className = 'status ok'; refresh();
+  } catch (e) { $('goStatus').textContent = e.message; $('goStatus').className = 'status bad'; }
+};
+$('copyid').onclick = async () => { try { await navigator.clipboard.writeText($('srvid').textContent); $('copyid').textContent = 'copied'; setTimeout(()=>$('copyid').textContent='copy', 1500); } catch {} };
+refresh(); setInterval(refresh, 10000);
+</script>
+</body>
+</html>

deploy.sh

@@ -0,0 +1,126 @@
+#!/bin/bash
+# Minimal-impact deployer for the server checkout. Run as root (sudo) — installed as /usr/local/bin/vanguard-deploy.
+#   vanguard-deploy pull            fetch + fast-forward DEPLOY_BRANCH, then apply only what changed
+#   vanguard-deploy apply [OLDREV]  apply changes between OLDREV (default: last deployed rev) and HEAD
+#   vanguard-deploy rollback        check out the previously deployed rev and apply the reverse diff
+#   vanguard-deploy pending         restart MediaMTX if a restart is pending and no stream is live (timer runs this)
+#   vanguard-deploy status
+# Env (from /etc/vanguard/deploy.env): DEPLOY_REPO_DIR, DEPLOY_BRANCH, DEPLOY_NOTIFY_WEBHOOK_URL. VG_DRY=1 prints only.
+set -uo pipefail
+ENVF=/etc/vanguard/deploy.env; [[ -f $ENVF ]] && set -a && . "$ENVF" && set +a
+REPO="${DEPLOY_REPO_DIR:-/home/debian/vanguard-stream}"; BRANCH="${DEPLOY_BRANCH:-main}"
+STATE=/var/lib/vanguard; mkdir -p "$STATE" 2>/dev/null || true
+LASTF="$STATE/deployed-rev"; PREVF="$STATE/previous-rev"; PENDING="$STATE/mediamtx-restart-pending"
+DRY="${VG_DRY:-0}"; OWNER=$(stat -c %U "$REPO" 2>/dev/null || echo debian)
+log() { printf 'ts=%s level=info component=deploy msg="%s"\n' "$(date -Is)" "${*//\"/\\\"}"; }
+run() { log "\$ $*"; [[ $DRY == 1 ]] || "$@"; }
+asowner() { log "\$ (as $OWNER) $*"; [[ $DRY == 1 ]] || sudo -u "$OWNER" -H "$@"; }
+notify() {
+  [[ -n ${DEPLOY_NOTIFY_WEBHOOK_URL:-} && $DRY != 1 ]] || return 0
+  curl -s -m 10 -H 'Content-Type: application/json' -d "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" "$DEPLOY_NOTIFY_WEBHOOK_URL" >/dev/null || true
+}
+cd "$REPO" || { log "repo dir $REPO missing"; exit 1; }
+# "pending" (the 5-min timer) needs no git; every other mode does
+[[ ${1:-} == pending ]] || git rev-parse --is-inside-work-tree >/dev/null 2>&1 || { log "$REPO is not a git checkout (run switchToClone.sh once)"; exit 1; }
+
+live_streams() { curl -s -m 3 'http://127.0.0.1:9997/v3/paths/list' 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(sum(1 for p in d.get("items",[]) if p.get("ready")))' 2>/dev/null || echo 0; }
+
+restart_mediamtx_or_defer() {
+  local n; n=$(live_streams)
+  if [[ $n == 0 ]]; then
+    run docker compose restart mediamtx; rm -f "$PENDING"; log "mediamtx restarted"
+  else
+    [[ $DRY == 1 ]] || touch "$PENDING"; log "mediamtx restart DEFERRED: $n live stream(s); deploy-pending.timer retries every 5 min"
+    notify "⏳ deploy: MediaMTX restart deferred ($n live stream(s)); will restart when idle"
+  fi
+}
+
+apply_changes() {
+  local old="$1" new="$2"
+  local changed; changed=$(git diff --name-only "$old" "$new" 2>/dev/null || git ls-files)
+  [[ -n $changed ]] || { log "nothing changed between $old and $new"; return 0; }
+  log "changed files:"; echo "$changed" | sed 's/^/    /'
+  has() { echo "$changed" | grep -qE "$1"; }
+
+  # 0. static checks on the new tree
+  if [[ -x check.sh ]]; then asowner bash check.sh >/tmp/vg-check.log 2>&1 || { log "check.sh FAILED — aborting"; tail -20 /tmp/vg-check.log; notify "❌ deploy $new: check.sh failed"; return 1; }; fi
+
+  # 1. host scripts + units
+  # bin/install.list, not a second hand-kept list: this one had drifted to five of ten files, so a push
+  # never installed vgstuck.sh, vgmetrics.sh, vgnotify.sh or either timer script.
+  if has '^bin/'; then
+    while read -r mode path; do
+      [[ -f $path ]] && run install -m "$mode" "$path" /usr/local/bin/
+    done < <(grep -vE '^[[:space:]]*(#|$)' bin/install.list)
+  fi
+  if has '^systemd/'; then run install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/; run systemctl daemon-reload; fi
+  if has '^deploy\.sh$'; then run install -m 0755 deploy.sh /usr/local/bin/vanguard-deploy; fi
+  if has '^deployBotBundle\.sh$'; then run install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot; fi
+  if has '^firewall/nftables\.conf$'; then run install -m 0644 firewall/nftables.conf /etc/nftables.conf; run nft -f /etc/nftables.conf; fi
+
+  # 2. regenerate configs (Caddy reload, copyparty restart) — configure.py decides; MediaMTX is handled below
+  if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/|^stuck/|^observability/kibana-setup\.py$'; then
+    local before_mtx; before_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
+    if [[ $DRY == 1 ]]; then log "\$ (as $OWNER) VG_DRY=1 python3 configure.py --apply"; else sudo -u "$OWNER" -H python3 configure.py --apply >/tmp/vg-configure.log 2>&1 || { log "configure --apply FAILED"; tail -20 /tmp/vg-configure.log; return 1; }; fi
+    local after_mtx; after_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
+    [[ $before_mtx == "$after_mtx" ]] || { log "mediamtx.yml changed"; restart_mediamtx_or_defer; }
+  fi
+  if has '^bin/dumpd\.py$|^systemd/dumpd\.service$'; then run systemctl restart dumpd; fi
+  # Beat and Kibana configs are bind-mounted, so a content change needs the container recreated. Nothing
+  # handled this before: a push that changed filebeat.yml silently did nothing.
+  if has '^observability/filebeat\.yml$'; then run docker compose up -d --force-recreate filebeat; fi
+  if has '^observability/metricbeat\.yml$'; then run docker compose up -d --force-recreate metricbeat; fi
+  if has '^observability/kibana\.yml$'; then run docker compose up -d --force-recreate kibana; fi
+
+  # 3. compose: recreate changed services, but never MediaMTX while live
+  if has '^docker-compose\.yml$|^bot/Dockerfile$|^bot/entrypoint\.sh$'; then
+    local mtx_changed=0
+    git diff "$old" "$new" -- docker-compose.yml | grep -qE '^[+-]' && git diff -U0 "$old" "$new" -- docker-compose.yml | grep -qiE '^[+-].*(mediamtx|8189|8890|1935)' && mtx_changed=1
+    if [[ $mtx_changed == 1 && $(live_streams) != 0 ]]; then
+      log "compose touches mediamtx and streams are live: recreating everything except mediamtx"
+      run docker compose up -d --build --no-deps $(docker compose config --services | grep -v '^mediamtx$' | tr '\n' ' ')
+      [[ $DRY == 1 ]] || touch "$PENDING"; notify "⏳ deploy: MediaMTX recreate deferred (live streams)"
+    else
+      run docker compose up -d --build
+    fi
+  fi
+
+  # 4. Jellyfin plugin (committed zip)
+  if has '^jellyfin-plugin-vanguard/dist/.*\.zip$'; then run bash installJellyfinPlugin.sh; fi
+
+  # 5. live portal
+  if has '^live/'; then
+    if command -v npm >/dev/null; then
+      asowner bash -c 'cd live && npm install --no-audit --no-fund --loglevel=error && npm run build' && { run install -d -o "$OWNER" /srv/live; run rsync -a --delete live/dist/ /srv/live/; }
+    else log "npm not installed on the server: live/ changed but not built (apt-get install nodejs npm)"; fi
+  fi
+  log "applied $old..$new"
+}
+
+case "${1:-}" in
+  pull)
+    old=$(git rev-parse HEAD)
+    asowner git fetch --quiet origin "$BRANCH" || { log "fetch failed"; exit 1; }
+    asowner git merge --ff-only "origin/$BRANCH" || { log "fast-forward failed (local changes?)"; exit 1; }
+    new=$(git rev-parse HEAD)
+    [[ $old == "$new" ]] && { log "already at $new"; exit 0; }
+    if apply_changes "$old" "$new"; then
+      [[ $DRY == 1 ]] || { cp -f "$LASTF" "$PREVF" 2>/dev/null; echo "$new" > "$LASTF"; }
+      notify "✅ deploy $(git log -1 --format='%h %s' "$new")"
+    else notify "❌ deploy $new failed — see journalctl -u dumpd"; exit 1; fi ;;
+  apply)
+    old="${2:-$(cat "$LASTF" 2>/dev/null || git rev-parse HEAD~1)}"; new=$(git rev-parse HEAD)
+    apply_changes "$old" "$new" && { [[ $DRY == 1 ]] || { cp -f "$LASTF" "$PREVF" 2>/dev/null; echo "$new" > "$LASTF"; }; } ;;
+  rollback)
+    prev=$(cat "$PREVF" 2>/dev/null) || { log "no previous rev recorded"; exit 1; }
+    cur=$(git rev-parse HEAD); log "rolling back $cur -> $prev"
+    asowner git checkout --quiet "$prev" && apply_changes "$cur" "$prev" && { [[ $DRY == 1 ]] || { echo "$cur" > "$PREVF"; echo "$prev" > "$LASTF"; }; notify "↩️ rolled back to $prev"; } ;;
+  pending)
+    [[ -f $PENDING ]] || exit 0
+    restart_mediamtx_or_defer ;;
+  status)
+    echo "repo:     $REPO ($(git rev-parse --abbrev-ref HEAD) @ $(git rev-parse --short HEAD))"
+    echo "deployed: $(cat "$LASTF" 2>/dev/null || echo '-')   previous: $(cat "$PREVF" 2>/dev/null || echo '-')"
+    echo "auto:     DEPLOY_AUTO=${DEPLOY_AUTO:-0}   branch: $BRANCH   pending mediamtx restart: $([[ -f $PENDING ]] && echo yes || echo no)   live streams: $(live_streams)" ;;
+  *) sed -n '2,9p' "$0"; exit 2 ;;
+esac

deployBotBundle.sh

@@ -0,0 +1,137 @@
+#!/bin/bash
+# Install a Vanguard Bot release bundle into the place the discord-bot container expects it.
+#
+# The bundle comes from the bot repo's CI (workflow "ci", job "bundle", artifact "vanguard-bot-bundle"):
+# one tar.gz holding main.py, app/, website/, version.json, BUILD.json and vendor/ — the bot's own
+# dependencies. Nothing is installed here and the server needs no package index.
+#
+#   ./deployBotBundle.sh vanguard-bot-bundle.tar.gz    install that file
+#   ./deployBotBundle.sh --latest                      download the newest main build from Gitea first
+#   ./deployBotBundle.sh --rollback                    put the previous bundle back
+#   ./deployBotBundle.sh --status                      what is installed right now
+#
+# --latest needs a Gitea token with read access to the repo, in .env as BOT_REPO_TOKEN (or $BOT_REPO_TOKEN).
+# Layout:  $BOT_DIR/app = live (mounted at /app)   $BOT_DIR/previous = the one before   $BOT_DIR/.incoming = staging
+set -euo pipefail
+# Runs from the bundle root or as /usr/local/bin/vanguard-deploy-bot (dumpd calls it that way): find the stack.
+STACK="${VANGUARD_DIR:-}"
+if [[ -z $STACK && -f /etc/vanguard/deploy.env ]]; then
+  STACK=$(grep -E '^DEPLOY_REPO_DIR=' /etc/vanguard/deploy.env | head -1 | cut -d= -f2- || true)
+fi
+if [[ -z $STACK || ! -f $STACK/docker-compose.yml ]]; then STACK="$(cd "$(dirname "$0")" && pwd)"; fi
+[[ -f $STACK/docker-compose.yml ]] || { echo "error: cannot find the stack (set VANGUARD_DIR)" >&2; exit 1; }
+cd "$STACK"
+BOT_DIR="${BOT_DIR:-/srv/bot}"
+REPO="${BOT_REPO:-VanguardOrg/VanguardBot}"
+BRANCH="${BOT_BRANCH:-main}"
+fromenv() { [[ -f .env ]] && grep -E "^$1=" .env | head -1 | cut -d= -f2- || true; }
+GIT_HOST="$(fromenv GIT_HOST)"
+BOT_HOST="$(fromenv BOT_HOST)"
+BOT_REPO_TOKEN="${BOT_REPO_TOKEN:-$(fromenv BOT_REPO_TOKEN)}"
+
+say() { printf '\n== %s\n' "$*"; }
+die() { echo "error: $*" >&2; exit 1; }
+
+unwrap() {   # unwrap <downloaded-file> <destination.tar.gz>  — Gitea hands out artifacts as a zip holding the tar.gz
+  if python3 -c "import zipfile,sys; sys.exit(0 if zipfile.is_zipfile(sys.argv[1]) else 1)" "$1"; then
+    python3 - "$1" "$2" <<'PYZIP'
+import sys,zipfile
+with zipfile.ZipFile(sys.argv[1]) as archive:
+    names=[n for n in archive.namelist() if n.endswith('.tar.gz')] or archive.namelist()
+    if not names: sys.exit('the zip holds no files')
+    open(sys.argv[2],'wb').write(archive.read(names[0]))
+PYZIP
+    echo "  (unpacked $(basename "$1"): a zip from Gitea)"
+  elif [[ $1 != "$2" ]]; then cp -f "$1" "$2"; fi
+}
+
+summary() {  # summary <bundle-root>
+  python3 - "$1" <<'PY'
+import json,sys,pathlib
+root=pathlib.Path(sys.argv[1]); build=root/'BUILD.json'
+if not build.is_file(): print('  (no BUILD.json — not a CI bundle)'); sys.exit(0)
+d=json.loads(build.read_text())
+print(f"  version {d.get('version')}  commit {(d.get('commit') or '?')[:8]}  built {d.get('built_at')}  {d.get('python')} ({d.get('abi')})")
+print(f"  {len(d.get('files',{}))} files, {len(d.get('packages',[]))} packages: {', '.join(d.get('packages',[])[:6])}…")
+PY
+}
+
+case "${1:-}" in
+  --status)
+    say "installed at $BOT_DIR/app"; summary "$BOT_DIR/app"
+    if [[ -d $BOT_DIR/previous ]]; then say "previous"; summary "$BOT_DIR/previous"; fi
+    say "container"; docker ps --filter name=discord-bot --format '  {{.Names}} {{.Status}}'
+    exit 0 ;;
+  --rollback)
+    [[ -d $BOT_DIR/previous ]] || die "no previous bundle in $BOT_DIR/previous"
+    say "rolling back"; summary "$BOT_DIR/previous"
+    docker stop discord-bot >/dev/null
+    rm -rf "$BOT_DIR/.rollback" && mv "$BOT_DIR/app" "$BOT_DIR/.rollback" && mv "$BOT_DIR/previous" "$BOT_DIR/app" && mv "$BOT_DIR/.rollback" "$BOT_DIR/previous"
+    docker start discord-bot >/dev/null; sleep 8; docker logs --tail 12 discord-bot; exit 0 ;;
+  --latest)
+    [[ -n ${BOT_REPO_TOKEN:-} ]] || die "set BOT_REPO_TOKEN in .env (Gitea token with read access to $REPO)"
+    [[ -n $GIT_HOST ]] || die "GIT_HOST not found in .env"
+    TARBALL=$(mktemp /tmp/vanguard-bot-bundle.XXXXXX.tar.gz)
+    say "newest '$BRANCH' bundle from https://$GIT_HOST/$REPO"
+    URL=$(curl -fsS -H "Authorization: token $BOT_REPO_TOKEN" \
+          "https://$GIT_HOST/api/v1/repos/$REPO/actions/artifacts?name=vanguard-bot-bundle" |
+          python3 -c "
+import json,sys
+items=json.load(sys.stdin).get('artifacts') or []
+ok=[a for a in items if not a.get('expired')]
+if not ok: sys.exit('no unexpired vanguard-bot-bundle artifact — check the ci workflow ran')
+print(sorted(ok,key=lambda a:a.get('created_at',''))[-1]['archive_download_url'])")
+    curl -fsSL -H "Authorization: token $BOT_REPO_TOKEN" -o "$TARBALL.zip" "$URL"
+    unwrap "$TARBALL.zip" "$TARBALL"; rm -f "$TARBALL.zip" ;;
+  "" | -h | --help) sed -n '2,20p' "$0"; exit 2 ;;
+  *) [[ -f $1 ]] || die "no such file: $1"
+     TARBALL=$(mktemp /tmp/vanguard-bot-bundle.XXXXXX.tar.gz); unwrap "$1" "$TARBALL" ;;
+esac
+
+say "staging $TARBALL"
+rm -rf "$BOT_DIR/.incoming"; mkdir -p "$BOT_DIR/.incoming"
+tar -xzf "$TARBALL" -C "$BOT_DIR/.incoming"
+for required in main.py app/main.py app/config.py website/static/home.html vendor; do
+  [[ -e "$BOT_DIR/.incoming/$required" ]] || die "bundle is missing $required — is this the CI artifact?"
+done
+summary "$BOT_DIR/.incoming"
+python3 - "$BOT_DIR/.incoming" <<'PY'
+import hashlib,json,pathlib,sys
+root=pathlib.Path(sys.argv[1]); d=json.loads((root/'BUILD.json').read_text())
+bad=[name for name,digest in d.get('files',{}).items()
+     if name!='BUILD.json' and hashlib.sha256((root/name).read_bytes()).hexdigest()!=digest]
+sys.exit(f"  {len(bad)} file(s) do not match BUILD.json: {bad[:3]}" if bad else 0)
+PY
+echo "  checksums match BUILD.json"
+
+say "swapping in and restarting discord-bot"
+docker stop discord-bot >/dev/null 2>&1 || true
+rm -rf "$BOT_DIR/previous"
+if [[ -d $BOT_DIR/app ]]; then mv "$BOT_DIR/app" "$BOT_DIR/previous"; fi
+mv "$BOT_DIR/.incoming" "$BOT_DIR/app"
+chown -R 1000:1000 "$BOT_DIR/app" 2>/dev/null || sudo chown -R 1000:1000 "$BOT_DIR/app"
+docker compose up -d discord-bot >/dev/null
+sleep 10
+
+say "result"
+# Wait for the bot to actually serve before calling this a success: a release that crash-loops must fail the
+# deploy (and the CI job), not report 200. BOT_HEALTH_WAIT=0 skips the wait, e.g. if WEB_ENABLED is ever off.
+wait_s="${BOT_HEALTH_WAIT:-45}"
+code=""; waited=0
+while (( waited < wait_s )); do
+  code=$(curl -s -m 3 -o /dev/null -w '%{http_code}' -H 'X-Authentik-Username: deploy-check' \
+           -H 'X-Authentik-Groups: admins' http://127.0.0.1:8000/health || true)
+  [[ $code =~ ^[2-4][0-9][0-9]$ ]] && break
+  sleep 3; waited=$((waited+3))
+done
+docker ps --filter name=discord-bot --format '  {{.Names}} {{.Status}}'
+echo "  /health -> ${code:-no answer} after ${waited}s   (200 = serving; 401/403 = serving but it refused the check's headers)"
+docker logs --tail 20 discord-bot 2>&1 | sed 's/^/  /'
+echo
+if [[ ! $code =~ ^[2-4][0-9][0-9]$ ]] && (( wait_s > 0 )); then
+  echo "  FAILED: the bot did not serve on 8000 within ${wait_s}s — the release is installed but not running."
+  echo "  previous release is still here: ./deployBotBundle.sh --rollback"
+  exit 1
+fi
+echo "  web UI: https://${BOT_HOST:-BOT_HOST-not-set}/   (Caddy + Authentik; needs the bot-users group)"
+echo "  rollback: ./deployBotBundle.sh --rollback"

deployLive.sh

@@ -2,7 +2,7 @@
 # Build the live portal here and push the static files to /srv/live on the server. Caddy serves them as-is.
 set -euo pipefail
 cd "$(dirname "$0")/live"
-HOST="${1:-debian@ovh1.dumpstervigilantes.rip}"
+HOST="${1:-debian@media.vng.one}"   # any name of the box works for ssh
 npm install --no-audit --no-fund --loglevel=error
 npm run build
 ssh "$HOST" 'sudo install -d -o "$(id -u)" -g "$(id -g)" /srv/live && rm -rf /srv/live/*'

dock/clock/index.html

@@ -0,0 +1,26 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Vanguard Clock</title>
+<style>html,body{margin:0;background:#000;overflow:hidden}canvas{display:block}</style>
+</head>
+<body>
+<canvas id="c"></canvas>
+<script src="vgclock.js"></script>
+<script>
+// OBS browser source for the overscan band: size it to the band (width = canvas width, height = band).
+// It draws the server-synced strip + readable clock at the display rate. ?text=0 hides the readable clock.
+(async () => {
+  const c = document.getElementById('c'), ctx = c.getContext('2d', { alpha: false });
+  const q = new URLSearchParams(location.search);
+  const showText = q.get('text') !== '0';
+  function fit() { c.width = Math.max(160, window.innerWidth); const { band } = VGClock.geometry(c.width); c.height = Math.max(band, window.innerHeight || band); }
+  fit(); window.addEventListener('resize', fit);
+  await VGClock.resync('/time');
+  setInterval(() => VGClock.resync('/time'), 60000);
+  (function frame() { VGClock.drawBand(ctx, c.width, 0, { text: showText }); requestAnimationFrame(frame); })();
+})();
+</script>
+</body>
+</html>

dock/clock/vgclock.js

@@ -0,0 +1,56 @@
+// Vanguard timecode strip — browser side. Shared by the OBS browser source (/dock/clock/) and the live portal.
+// Format: see bin/vgstrip.py. cell = W/160; band = round_even(cell); 64 cells from x = cell:
+//   [1,0,1,0] · 48-bit server ms (MSB first) · CRC-8 (poly 0x07) · [0,1,0,1]; 1 = white, 0 = black.
+(function (global) {
+  function crc8(bytes) { let c = 0; for (const b of bytes) { c ^= b; for (let i = 0; i < 8; i++) c = (c & 0x80) ? ((c << 1) ^ 0x07) & 0xff : (c << 1) & 0xff; } return c; }
+  function bitsFor(ms) {
+    const bytes = []; let v = BigInt(ms); for (let i = 5; i >= 0; i--) { bytes[i] = Number(v & 0xffn); v >>= 8n; }
+    const bits = [1, 0, 1, 0];
+    for (const b of bytes) for (let i = 7; i >= 0; i--) bits.push((b >> i) & 1);
+    const c = crc8(bytes); for (let i = 7; i >= 0; i--) bits.push((c >> i) & 1);
+    return bits.concat([0, 1, 0, 1]);
+  }
+  function geometry(w) { const cell = w / 160; let band = Math.round(cell); band += band % 2; return { cell, band }; }
+
+  // ---- server clock offset: offset = serverNow - localNow, from the lowest-RTT of several /time samples
+  const sync = { offset: 0, rtt: null, ok: false, samples: 0 };
+  async function sampleOnce(url) {
+    const t0 = performance.now(), wall0 = Date.now();
+    const r = await fetch(url, { cache: 'no-store', credentials: 'same-origin' }); const j = await r.json();
+    const t1 = performance.now(); const rtt = t1 - t0;
+    const localMid = wall0 + rtt / 2;
+    return { offset: j.now_ms - localMid, rtt };
+  }
+  async function resync(url, n = 7) {
+    let best = null;
+    for (let i = 0; i < n; i++) { try { const s = await sampleOnce(url); if (!best || s.rtt < best.rtt) best = s; } catch (e) {} }
+    if (best) { sync.offset = best.offset; sync.rtt = best.rtt; sync.ok = true; sync.samples++; }
+    return sync;
+  }
+  function serverNow() { return Date.now() + sync.offset; }
+
+  // ---- draw the band into a 2D context: strip + readable clock. (w = canvas width, y0 = band top)
+  function drawBand(ctx, w, y0, opts = {}) {
+    const { cell, band } = geometry(w);
+    const ms = Math.round(serverNow());
+    ctx.fillStyle = '#000'; ctx.fillRect(0, y0, w, band);
+    const bits = bitsFor(ms);
+    const pad = Math.max(1, Math.floor(cell * 0.12));
+    for (let i = 0; i < bits.length; i++) {
+      if (!bits[i]) continue;
+      const x0 = Math.round(cell * (1 + i)), x1 = Math.round(cell * (2 + i));
+      ctx.fillStyle = '#fff'; ctx.fillRect(x0 + pad, y0 + pad, x1 - x0 - 2 * pad, band - 2 * pad);
+    }
+    if (opts.text !== false) {
+      const d = new Date(ms);
+      const tz = opts.tz || 'America/New_York';
+      const hms = d.toLocaleTimeString('en-US', { hour12: false, timeZone: tz });
+      const txt = `${hms}.${String(ms % 1000).padStart(3, '0')} ${opts.label || 'SRV'}${sync.ok ? '' : ' (unsynced)'}`;
+      ctx.fillStyle = sync.ok ? '#9f9' : '#f66'; ctx.font = `${Math.max(8, band - 2 * pad)}px ui-monospace, monospace`; ctx.textBaseline = 'middle';
+      ctx.fillText(txt, Math.round(cell * (CELLS + 2.5)), y0 + band / 2);
+    }
+    return ms;
+  }
+  const CELLS = 64;
+  global.VGClock = { crc8, bitsFor, geometry, resync, serverNow, drawBand, sync, CELLS };
+})(window);

dock/index.html

@@ -30,6 +30,12 @@
   .kv .k{color:var(--mut)}
   .tok{letter-spacing:1px}
   .who{color:var(--mut);font-size:12px}
+  /* post-Go-Live health check: what the server is actually receiving */
+  #health{margin-top:10px;padding:8px;border-radius:6px;font-size:13px;white-space:pre-wrap;word-break:break-word}
+  #health.hidden{display:none}
+  #health.bad{background:#43242a;color:#ffd0d0;border:1px solid var(--bad)}
+  #health.bad code{background:#1a1a1a;padding:1px 5px;border-radius:4px;color:var(--fg)}
+  #health.ok{color:var(--ok)}
 </style>
 </head>
 <body>
@@ -53,11 +59,28 @@
     <label>Ingest host</label><input id="host" placeholder="media.example.com">
     <label>OBS WebSocket password</label><input id="obspw" type="password">
     <label>OBS WebSocket port</label><input id="obsport" value="4455">
+    <label>OBS profile to stream from</label><input id="profile" value="Vanguard">
+    <label style="display:flex;gap:8px;align-items:center;margin-top:8px"><input type="checkbox" id="enforce" style="width:auto;margin:0"> switch OBS to this profile when the dock loads</label>
+    <div class="who">Encoder settings (B-frames, keyframe interval) live in an OBS profile and obs-websocket cannot change them,
+    but it can switch profiles. Create one named <b>Vanguard</b> once: Profile → New, then Output → Advanced → your encoder
+    with <b>B-frames 0</b> and keyframe interval 2 s. Go Live always switches to it; this box also does it on startup.</div>
     <label>Title prefix (optional, e.g. <code>HelmetCam-</code>)</label><input id="prefix" placeholder="">
     <label>Title suffix (optional; <code>{date}</code> becomes YYYY-MM-DD)</label><input id="suffix" placeholder="-{date}">
     <button id="save">Save settings</button>
   </details>
 
+  <details id="tcband">
+    <summary>Timecode band (frame-accurate multicam sync)</summary>
+    <div class="who" style="margin-top:6px">Adds a thin "overscan" band below your canvas with the server clock encoded in it.
+      The server decodes it after each stream so editors get a frame-accurate timecode track; the live portal and
+      editors crop the band, Jellyfin shows it as a thin strip at the bottom. OBS must not be streaming or recording.</div>
+    <div class="kv" id="tc_state">canvas: — · band: —</div>
+    <button class="small" id="tc_add">Add band + clock source</button>
+    <button class="small" id="tc_scene">Add clock to current scene</button>
+    <button class="small" id="tc_remove">Remove band</button>
+  </details>
+
+  <div id="health" class="hidden"></div>
   <details id="manual">
     <summary>Manual OBS setup (endpoints + token)</summary>
     <div class="kv"><span class="k">RTMP server  </span><span id="m_rtmp">—</span></div>
@@ -74,11 +97,12 @@
 
 <script>
 // ---- settings --------------------------------------------------------------
-const KEYS = ["host","obspw","obsport","prefix","suffix"];
+const KEYS = ["host","obspw","obsport","prefix","suffix","profile"];
+const BOOLS = ["enforce"];
 const $ = id => document.getElementById(id);
 const status = (msg, cls) => { $("status").textContent = msg; $("status").style.color = cls==="bad"?"#e74c3c":cls==="ok"?"#2ecc71":"#9a9a9a"; };
-function load(){ try{ KEYS.forEach(k => { const v = localStorage.getItem("vg."+k); if (v!==null) $(k).value = v; }); }catch(e){} if(!$("host").value) $("host").value = location.hostname; }
+function load(){ try{ KEYS.forEach(k => { const v = localStorage.getItem("vg."+k); if (v!==null) $(k).value = v; }); BOOLS.forEach(k => { $(k).checked = localStorage.getItem("vg."+k) === "1"; }); }catch(e){} if(!$("host").value) $("host").value = location.hostname; }
-function save(){ try{ KEYS.forEach(k => localStorage.setItem("vg."+k, $(k).value.trim())); status("settings saved"); }catch(e){ status("could not save settings (storage blocked)", "bad"); } renderManual(); }
+function save(){ try{ KEYS.forEach(k => localStorage.setItem("vg."+k, $(k).value.trim())); BOOLS.forEach(k => localStorage.setItem("vg."+k, $(k).checked ? "1" : "0")); status("settings saved"); }catch(e){ status("could not save settings (storage blocked)", "bad"); } renderManual(); }
 load(); $("save").onclick = save;
 const cfg = () => Object.fromEntries(KEYS.map(k => [k, $(k).value.trim()]));
 
@@ -88,7 +112,7 @@ function buildTitle(raw){
   const c = cfg();
   let t = (c.prefix||"") + raw.trim() + (c.suffix||"").replace("{date}", today());
   t = t.replace(/\s+/g, "-")              // spaces -> dashes
-       .replace(/[^A-Za-z0-9_.-]/g, "")   // allowed characters only
+       .replace(/[^A-Za-z0-9_-]/g, "")    // allowed characters only; no dots: <title>.webrtc is the audio mirror's name
        .replace(/_{2,}/g, "_")            // "__" is reserved: it separates title from timestamp in filenames
        .replace(/^[.-]+/, "")             // no leading dot (hidden) or dash
        .slice(0, 64);
@@ -97,7 +121,7 @@ function buildTitle(raw){
 function ruleText(raw){
   const r = [];
   if (/__/.test(raw)) r.push("double underscore is reserved (becomes single)");
-  if (/[^A-Za-z0-9_.\- ]/.test(raw)) r.push("only letters, digits, space, _ . - are kept");
+  if (/[^A-Za-z0-9_\- ]/.test(raw)) r.push("only letters, digits, space, _ - are kept");
   if (raw.trim().startsWith(".")) r.push("cannot start with a dot");
   if ((cfg().prefix + raw + cfg().suffix).length > 64) r.push("longer than 64 characters: trimmed");
   return r.join(" · ");
@@ -127,7 +151,7 @@ async function mint(){
 }
 let showTok = false;
 function renderManual(t){
-  const c = cfg(); const title = t ?? buildTitle($("title").value) || "TITLE";
+  const c = cfg(); const title = (t ?? buildTitle($("title").value)) || "TITLE";
   const user = me ? me.user : "<you>", tok = me ? me.token : "<token>";
   $("m_rtmp").textContent = `rtmp://${c.host||location.hostname}:1935/`;
   $("m_key").textContent  = `${user}/${title}?user=${user}&pass=${showTok ? tok : "••••••••"}`;
@@ -166,6 +190,100 @@ function obs(c){
   });
 }
 
+// ---- profile enforcement ----------------------------------------------------------------------------
+// Encoder settings (B-frames above all: MediaMTX refuses them over WebRTC) cannot be set through
+// obs-websocket in Advanced output mode — they live in the profile's streamEncoder.json, which the protocol
+// does not expose. Profiles can be switched, though, so the setting is enforced by making sure OBS is on a
+// profile that already has it. The post-Go-Live health check then verifies the actual encode.
+// Returns true when OBS is on the wanted profile, false (with a status line saying why) otherwise.
+async function ensureProfile(o, c) {
+  if (!c.profile) return true;
+  const { profiles, currentProfileName } = await o.call("GetProfileList");
+  if (!profiles.includes(c.profile)) {
+    status(`OBS has no profile named "${c.profile}". Create it once (Profile → New) with B-frames 0, or clear the name in settings. Streaming on "${currentProfileName}".`, "bad");
+    return false;
+  }
+  if (currentProfileName === c.profile) return true;
+  const st = await o.call("GetStreamStatus");
+  if (st.outputActive) { status(`OBS is on "${currentProfileName}" but streaming; not switching mid-stream`, "bad"); return false; }
+  await o.call("SetCurrentProfile", { profileName: c.profile });
+  await new Promise(r => setTimeout(r, 800));            // OBS reloads its outputs on a profile switch
+  status(`OBS switched to profile "${c.profile}"`, "ok");
+  return true;
+}
+
+// ---- timecode band (obs-websocket) -------------------------------------------------------------------
+const CLOCK_INPUT = "Vanguard Clock";
+const bandFor = w => { let b = Math.round(w / 160); return b + (b % 2); };
+async function tcState() {
+  try {
+    const o = await obs(cfg());
+    const v = await o.call("GetVideoSettings");
+    const inputs = await o.call("GetInputList", { inputKind: "browser_source" });
+    const has = inputs.inputs.some(i => i.inputName === CLOCK_INPUT);
+    const band = bandFor(v.baseWidth);
+    const hasBand = (v.baseHeight % 2 === 0) && [720, 1080, 1440, 2160].some(h => v.baseHeight === h + bandFor(v.baseWidth));
+    o.close();
+    $("tc_state").textContent = `canvas: ${v.baseWidth}×${v.baseHeight} (output ${v.outputWidth}×${v.outputHeight}) · band: ${hasBand ? band + " px present" : "not set"} · clock source: ${has ? "present" : "absent"}`;
+    return { v, has, band, hasBand };
+  } catch (e) { $("tc_state").textContent = "OBS not reachable: " + e.message; return null; }
+}
+async function notStreaming(o) {
+  const s = await o.call("GetStreamStatus"), r = await o.call("GetRecordStatus");
+  if (s.outputActive || r.outputActive) throw new Error("stop streaming/recording first — the canvas size cannot change while an output is active");
+}
+async function ensureClockInput(o, width, band, scene) {
+  const url = `${location.origin}/dock/clock/`;
+  const inputs = await o.call("GetInputList", { inputKind: "browser_source" });
+  const settings = { url, width, height: band, fps_custom: true, fps: 60, shutdown: false, restart_when_active: false, reroute_audio: false };
+  if (!inputs.inputs.some(i => i.inputName === CLOCK_INPUT)) {
+    await o.call("CreateInput", { sceneName: scene, inputName: CLOCK_INPUT, inputKind: "browser_source", inputSettings: settings, sceneItemEnabled: true });
+  } else {
+    await o.call("SetInputSettings", { inputName: CLOCK_INPUT, inputSettings: settings, overlay: true });
+    const items = await o.call("GetSceneItemList", { sceneName: scene });
+    if (!items.sceneItems.some(i => i.sourceName === CLOCK_INPUT)) await o.call("CreateSceneItem", { sceneName: scene, sourceName: CLOCK_INPUT, sceneItemEnabled: true });
+  }
+  const items = await o.call("GetSceneItemList", { sceneName: scene });
+  const it = items.sceneItems.find(i => i.sourceName === CLOCK_INPUT);
+  if (it) {
+    const v = await o.call("GetVideoSettings");
+    await o.call("SetSceneItemTransform", { sceneName: scene, sceneItemId: it.sceneItemId,
+      sceneItemTransform: { positionX: 0, positionY: v.baseHeight - band, scaleX: 1, scaleY: 1, boundsType: "OBS_BOUNDS_NONE", cropLeft: 0, cropRight: 0, cropTop: 0, cropBottom: 0 } });
+    await o.call("SetSceneItemLocked", { sceneName: scene, sceneItemId: it.sceneItemId, sceneItemLocked: true });
+  }
+}
+$("tc_add").onclick = async () => {
+  try {
+    const o = await obs(cfg()); await notStreaming(o);
+    const v = await o.call("GetVideoSettings"); const band = bandFor(v.baseWidth);
+    const gameH = [720, 1080, 1440, 2160].find(h => v.baseHeight === h + band) ? v.baseHeight - band : v.baseHeight;
+    const scale = v.outputHeight / v.baseHeight;
+    await o.call("SetVideoSettings", { baseWidth: v.baseWidth, baseHeight: gameH + band, outputWidth: v.outputWidth, outputHeight: Math.round((gameH + band) * scale / 2) * 2 });
+    const scene = (await o.call("GetCurrentProgramScene")).currentProgramSceneName;
+    await ensureClockInput(o, v.baseWidth, band, scene); o.close();
+    status(`band added: canvas ${v.baseWidth}×${gameH + band}, clock source in "${scene}"`, "ok"); tcState();
+  } catch (e) { status(e.message, "bad"); }
+};
+$("tc_scene").onclick = async () => {
+  try {
+    const o = await obs(cfg()); const v = await o.call("GetVideoSettings"); const band = bandFor(v.baseWidth);
+    const scene = (await o.call("GetCurrentProgramScene")).currentProgramSceneName;
+    await ensureClockInput(o, v.baseWidth, band, scene); o.close(); status(`clock source placed in "${scene}"`, "ok"); tcState();
+  } catch (e) { status(e.message, "bad"); }
+};
+$("tc_remove").onclick = async () => {
+  try {
+    const o = await obs(cfg()); await notStreaming(o);
+    const v = await o.call("GetVideoSettings"); const band = bandFor(v.baseWidth);
+    const gameH = [720, 1080, 1440, 2160].find(h => v.baseHeight === h + band) ? v.baseHeight - band : v.baseHeight;
+    try { await o.call("RemoveInput", { inputName: CLOCK_INPUT }); } catch {}
+    const scale = v.outputHeight / v.baseHeight;
+    await o.call("SetVideoSettings", { baseWidth: v.baseWidth, baseHeight: gameH, outputWidth: v.outputWidth, outputHeight: Math.round(gameH * scale / 2) * 2 });
+    o.close(); status(`band removed: canvas ${v.baseWidth}×${gameH}`, "ok"); tcState();
+  } catch (e) { status(e.message, "bad"); }
+};
+$("tcband").addEventListener("toggle", () => { if ($("tcband").open) tcState(); });
+
 // ---- actions ---------------------------------------------------------------
 $("golive").onclick = async () => {
   const c = cfg(), title = buildTitle($("title").value);
@@ -177,15 +295,66 @@ $("golive").onclick = async () => {
     const t = await mint();                                   // fresh token every Go Live; nothing stored
     status("connecting to OBS…");
     const o = await obs(c);
+    await ensureProfile(o, c);                                // the Vanguard profile carries B-frames 0; switch first
     await o.call("SetStreamServiceSettings", { streamServiceType: "rtmp_custom",
       streamServiceSettings: { server: `rtmp://${c.host}:1935/`, key: `${t.user}/${title}?user=${encodeURIComponent(t.user)}&pass=${encodeURIComponent(t.token)}`, use_auth: false } });
+    // B-frames make the stream unwatchable on the portal (MediaMTX refuses them over WebRTC). obs-websocket
+    // can only set encoder options in Simple output mode with x264, so do that much here; everything else is
+    // caught after start by checkHealth(), which reads the actual recording on the server.
+    try {
+      const mode = (await o.call("GetProfileParameter", { parameterCategory: "Output", parameterName: "Mode" })).parameterValue;
+      if (mode === "Simple") {
+        const enc = (await o.call("GetProfileParameter", { parameterCategory: "SimpleOutput", parameterName: "StreamEncoder" })).parameterValue || "";
+        if (/x264/i.test(enc)) {
+          const cur = (await o.call("GetProfileParameter", { parameterCategory: "SimpleOutput", parameterName: "x264Settings" })).parameterValue || "";
+          if (!/\b(bf|bframes)=0\b/.test(cur)) {
+            await o.call("SetProfileParameter", { parameterCategory: "SimpleOutput", parameterName: "x264Settings", parameterValue: (cur + " bf=0").trim() });
+            status("set x264 bf=0 in OBS (Simple output) so viewers can watch over WebRTC");
+          }
+        }
+      }
+    } catch (e) { /* older obs-websocket, or a locked profile: the health check will say if it matters */ }
     const st = await o.call("GetStreamStatus");
     if (st.outputActive) { await o.call("StopStream"); await new Promise(r => setTimeout(r, 1500)); }
     await o.call("StartStream"); o.close();
     status(`LIVE  →  ${t.user}/${title}`, "ok");
+    checkHealth(90);
   } catch (e) { status(e.message, "bad"); }
   $("golive").disabled = false;
 };
+// After Go Live, ask the server what it is actually receiving. The recording's first fragments reveal the
+// B-frame depth within seconds, and B-frames mean every WebRTC viewer is refused. Polls until the stream
+// shows up and is judged, or for `seconds`, whichever comes first.
+let healthTimer = null;
+async function checkHealth(seconds) {
+  clearInterval(healthTimer);
+  const until = Date.now() + seconds * 1000, box = $("health");
+  box.className = "who"; box.textContent = "checking what the server is receiving…";
+  healthTimer = setInterval(async () => {
+    try {
+      const r = await fetch("/me/stream-health", { credentials: "same-origin" });
+      if (!r.ok) throw new Error("HTTP " + r.status);
+      const d = await r.json();
+      const judged = (d.streams || []).filter(s => s.bframes != null);
+      if (judged.length) {
+        clearInterval(healthTimer);
+        const bad = judged.filter(s => !s.webrtc_ok);
+        if (bad.length) {
+          box.className = "bad";
+          box.innerHTML = `<b>Viewers cannot watch this stream over WebRTC.</b> Your encoder is sending B-frames `
+            + `(${bad.map(s => `${s.profile || "H.264"} ${s.res || ""}, B-frame depth ${s.bframes}`).join("; ")}). `
+            + `In OBS: Output → Advanced → <b>x264</b>: add <code>bf=0</code> to the options; <b>NVENC</b>: Max B-frames = 0; `
+            + `<b>AMF/QSV</b>: B-frames = 0. Then stop and start the stream. The recording itself is fine.`;
+        } else {
+          box.className = "ok";
+          box.textContent = `server receiving ${judged.map(s => `${s.res || ""} ${s.fps ? s.fps + " fps" : ""}`.trim()).join(", ")} with no B-frames — WebRTC viewers are good`;
+        }
+        return;
+      }
+    } catch (e) { /* not up yet, or the dock is not behind the proxy; keep trying until the deadline */ }
+    if (Date.now() > until) { clearInterval(healthTimer); box.className = "who"; box.textContent = "could not confirm what the server is receiving (is the stream up?)"; }
+  }, 6000);
+}
 $("stop").onclick = async () => { try { const o = await obs(cfg()); await o.call("StopStream"); o.close(); status("stream stopped"); } catch (e) { status(e.message, "bad"); } };
 
 let armed = 0;
@@ -207,14 +376,28 @@ $("dump").onclick = async () => {
   } catch (e) { status("dump failed: " + e.message, "bad"); }
 };
 
+
 $("signout").onclick = async e => {
   e.preventDefault(); status("signing out…");
   // Authentik's sign-out redirects to the site root (copyparty); do it in the background, then reload the dock,
   // which sends us through the Discord login and back here.
   try { await fetch("/outpost.goauthentik.io/sign_out", {credentials:"same-origin", mode:"no-cors", redirect:"follow"}); } catch(e){}
-  setTimeout(() => location.replace("/dock/"), 500);
+  // also end the Authentik login itself (otherwise the next visit re-authorises silently); the flow returns to this host
+  const auth = "https://" + location.hostname.replace(/^[^.]+\./, "auth.") + "/if/flow/default-invalidation-flow/?next=" + encodeURIComponent(location.origin + "/dock/");
+  setTimeout(() => location.replace(auth), 400);
 };
-whoami(); renderPreview();
+// On load: who am I, and mint a token right away so the preview and the manual section show real values.
+(async () => {
+  await whoami();
+  try { await mint(); status("ready"); }
+  catch (e) { status(e.message, "bad"); $("m_tok").textContent = "— " + e.message; }
+  // "switch OBS to this profile when the dock loads": inside OBS the dock loads with OBS, so this is startup
+  if ($("enforce").checked) {
+    try { const o = await obs(cfg()); await ensureProfile(o, cfg()); o.close(); }
+    catch (e) { status(`profile check on startup: ${e.message}`, "bad"); }
+  }
+})();
+renderPreview();
 </script>
 </body>
 </html>

docker-compose.yml

@@ -1,5 +1,13 @@
+# Log framework (LOGGING.md): every container keeps rotated json-file logs and carries labels a collector can use as
+# low-cardinality stream labels without parsing: vanguard.service / vanguard.category.
+x-logging: &vglog
+  driver: json-file
+  options: { max-size: "50m", max-file: "5" }
+
 services:
   mediamtx:
+    logging: *vglog
+    labels: { vanguard.service: mediamtx, vanguard.category: stream }
     image: bluenviron/mediamtx:latest-ffmpeg     # alpine variant: has sh + wget for the is-live Discord hook
     container_name: mediamtx
     user: "1000:1000"             # recordings owned by the debian user, so the archive pull can delete them
@@ -7,20 +15,22 @@ services:
       TZ: America/New_York          # recording filenames are stamped in the container's clock; must match the host
       LIVE_WEBHOOK_URL: ${LIVE_WEBHOOK_URL:-}
       LIVE_HOST: ${LIVE_HOST:-live.example.com}
-    ports:
+    # Host network, as MediaMTX recommends for WebRTC. On the bridge, every WebRTC media packet on 8189/udp
-      - "1935:1935"              # RTMP ingest
+    # was relayed by Docker's userland proxy (docker-proxy), which lost ~0.4% of them at a thousand packets a
-      - "8890:8890/udp"          # SRT ingest
+    # second; each loss corrupted an access unit, the recorder reopened its file, and one evening produced 471
-      - "8189:8189/udp"          # WebRTC media (viewers + Go Live)
+    # fragments. Publishers also all appeared as the bridge gateway (172.18.0.1). Now MediaMTX binds the host
-      - "8189:8189/tcp"          # WebRTC ICE-TCP fallback
+    # directly: ingest ports (1935, 8890/udp, 8189) are open in the nftables INPUT chain, and everything Caddy
-      - "127.0.0.1:8888:8888"    # HLS      -> Caddy live.HOST/hls/*
+    # or Metricbeat talks to is bound to 127.0.0.1 in mediamtx.yml. No ports: block, nothing to publish.
-      - "127.0.0.1:8889:8889"    # WebRTC   -> Caddy live.HOST/webrtc/*
+    network_mode: host
-      - "127.0.0.1:9997:9997"    # API      -> Caddy live.HOST/api/*
     volumes:
       - ./mediamtx.yml:/mediamtx.yml:ro
+      - ./bin/mtx-on-ready.sh:/vanguard/on-ready.sh:ro   # is-live notice + WebRTC audio mirror (runOnReady)
       - /srv/media/streams:/recordings
     restart: unless-stopped
 
   copyparty:
+    logging: *vglog
+    labels: { vanguard.service: copyparty, vanguard.category: media }
     image: copyparty/ac:latest
     container_name: copyparty
     ports:
@@ -32,6 +42,8 @@ services:
     restart: unless-stopped
 
   syncthing:
+    logging: *vglog
+    labels: { vanguard.service: syncthing, vanguard.category: intake }
     image: syncthing/syncthing:latest
     container_name: syncthing
     environment:
@@ -47,6 +59,8 @@ services:
     restart: unless-stopped
 
   caddy:
+    logging: *vglog
+    labels: { vanguard.service: caddy, vanguard.category: auth }
     image: caddy:latest
     container_name: caddy
     depends_on: [copyparty]
@@ -54,6 +68,9 @@ services:
     volumes:
       - ./Caddyfile:/etc/caddy/Caddyfile:ro
       - ./dock:/srv/dock:ro
+      - ./clips:/srv/clips:ro               # clip-intake self-service page
+      - ./about:/srv/about:ro               # gateway page (/about)
+      - ./streaming:/srv/streaming:ro       # Live + Stuck tabs (diag host /streaming)
       - /srv/live:/srv/live:ro              # live portal static build (deployLive.sh)
       - caddy_data:/data
       - caddy_config:/config
@@ -61,6 +78,8 @@ services:
 
   # ---------------------------------------------------------------- identity (Authentik)
   authentik-db:
+    logging: *vglog
+    labels: { vanguard.service: authentik-db, vanguard.category: auth }
     image: docker.io/library/postgres:16-alpine
     container_name: authentik-db
     environment:
@@ -78,6 +97,8 @@ services:
     restart: unless-stopped
 
   authentik-redis:
+    logging: *vglog
+    labels: { vanguard.service: authentik-redis, vanguard.category: auth }
     image: docker.io/library/redis:alpine
     container_name: authentik-redis
     command: --save 60 1 --loglevel warning
@@ -91,11 +112,15 @@ services:
     restart: unless-stopped
 
   authentik-server:
+    logging: *vglog
+    labels: { vanguard.service: authentik-server, vanguard.category: auth }
     image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-2025.8}
     container_name: authentik-server
     command: server
     environment: &authentik_env
       AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
+      AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD:-}   # akadmin password on FIRST start (no initial-setup page)
+      AUTHENTIK_BOOTSTRAP_TOKEN: ${AUTHENTIK_BOOTSTRAP_TOKEN:-}         # API token for akadmin, created on first start (blueprints/automation)
       AUTHENTIK_REDIS__HOST: authentik-redis
       AUTHENTIK_POSTGRESQL__HOST: authentik-db
       AUTHENTIK_POSTGRESQL__USER: authentik
@@ -113,6 +138,8 @@ services:
     restart: unless-stopped
 
   authentik-worker:
+    logging: *vglog
+    labels: { vanguard.service: authentik-worker, vanguard.category: auth }
     image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-2025.8}
     container_name: authentik-worker
     command: worker
@@ -128,6 +155,8 @@ services:
 
   # ---------------------------------------------------------------- watching (Jellyfin, direct play only)
   jellyfin:
+    logging: *vglog
+    labels: { vanguard.service: jellyfin, vanguard.category: media }
     image: jellyfin/jellyfin:latest
     container_name: jellyfin
     user: "1000:1000"
@@ -145,6 +174,8 @@ services:
 
   # ---------------------------------------------------------------- database (for the Discord bot and friends)
   mariadb:
+    logging: *vglog
+    labels: { vanguard.service: mariadb, vanguard.category: infra }
     image: mariadb:11
     container_name: mariadb
     network_mode: host              # PUBLIC on 3306: host networking so the nftables input rules (db_allow / rate
@@ -167,8 +198,10 @@ services:
       start_period: 30s
     restart: unless-stopped
 
-  # ---------------------------------------------------------------- Discord bot (python; code lives in ./bot/app)
+  # ------------------------------------------- Discord bot (python; its release is installed in /srv/bot/app)
   discord-bot:
+    logging: *vglog
+    labels: { vanguard.service: discord-bot, vanguard.category: bot }
     build: ./bot
     container_name: discord-bot
     user: "1000:1000"
@@ -180,6 +213,11 @@ services:
       DB_NAME: ${BOT_DB_NAME:-vanguardbot}
       DB_USER: ${BOT_DB_USER:-bot}
       DB_PASSWORD: ${BOT_DB_PASSWORD}
+      DB_BACKEND: mariadb
+      DATA_DIR: /data
+      WEB_ALLOWED_GROUPS: ${BOT_WEB_ALLOWED_GROUPS:-bot-users,admins}   # the bot's own gate; Caddy already requires BOT_REQUIRE_GROUP
+      WEB_SERVE_UI: "true"                # serve the bot's own pages at / on this port (same origin as /api)
+      PYTHONPATH: /app/vendor             # the bundle's dependencies; the image installs none
       MEDIA_CLIPS: /media/clips
       WEB_HOST: 0.0.0.0
       WEB_PORT: "8000"
@@ -192,17 +230,35 @@ services:
     ports:
       - "127.0.0.1:8000:8000"             # web UI; public only via Caddy + Authentik forward-auth at https://BOT_HOST/
     volumes:
-      - ./bot/app:/app                    # the bot's source; edit and `docker compose restart discord-bot`
+      - ${BOT_APP_DIR:-/srv/bot/app}:/app  # CI bundle installed by ./deployBotBundle.sh (code + vendor/)
       - /srv/media/clips:/media/clips     # rw: for a clip-submissions feature (writes into clips/<user>/)
       - /srv/appdata/discord-bot:/data    # persistent scratch (sqlite, caches) if the bot wants files
+    # Asks the BOT about itself rather than just proving the port accepts a connection: /health answers 503
+    # unless the Discord gateway is connected AND the database responds, and -f turns that into an unhealthy
+    # container, which Metricbeat's docker healthcheck metricset turns into an alarm. Requires the bot release
+    # that made /health unauthenticated and status-bearing — on older code every check 401s and -f fails,
+    # so ship the bot first. start_period covers command sync and the first roster load.
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -m 5 http://127.0.0.1:8000/health >/dev/null || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 120s
     extra_hosts:
       - "host.docker.internal:host-gateway"
+    networks:
+      - botnet                          # its OWN network: the bot trusts the identity headers Caddy injects, so
+                                        # nothing else must be able to reach :8000 and forge them. Caddy is on the
+                                        # host network and uses the published 127.0.0.1:8000; MariaDB is reached
+                                        # through host.docker.internal. The bot needs no sibling container.
     depends_on:
       mariadb: { condition: service_healthy }
     restart: unless-stopped
 
   # ---------------------------------------------------------------- Gitea + Actions runner
   gitea:
+    logging: *vglog
+    labels: { vanguard.service: gitea, vanguard.category: code }
     image: docker.gitea.com/gitea:1.24
     container_name: gitea
     environment:
@@ -219,10 +275,20 @@ services:
       GITEA__server__SSH_DOMAIN: ${GIT_HOST:-git.example.com}
       GITEA__server__SSH_PORT: "2222"                       # what users put in their remotes
       GITEA__server__SSH_LISTEN_PORT: "22"                  # inside the container
-      GITEA__service__DISABLE_REGISTRATION: "true"          # accounts are created by an admin (or via Authentik OIDC later)
+      # accounts come from Authentik (Discord) via OIDC: external auto-registration on, local self-registration off
+      GITEA__service__DISABLE_REGISTRATION: "false"
+      GITEA__service__ALLOW_ONLY_EXTERNAL_REGISTRATION: "true"
+      GITEA__service__SHOW_REGISTRATION_BUTTON: "false"
+      GITEA__service__REQUIRE_SIGNIN_VIEW: "false"
+      GITEA__oauth2_client__ENABLE_AUTO_REGISTRATION: "true"
+      GITEA__oauth2_client__ACCOUNT_LINKING: auto
+      GITEA__oauth2_client__USERNAME: nickname                # Authentik preferred_username
+      GITEA__oauth2_client__UPDATE_AVATAR: "true"
+      GITEA__oauth2_client__OPENID_CONNECT_SCOPES: openid email profile
       GITEA__actions__ENABLED: "true"
       GITEA__actions__DEFAULT_ACTIONS_URL: github
       GITEA__webhook__ALLOWED_HOST_LIST: private,loopback,external
+      GITEA__security__INSTALL_LOCK: "true"                  # no install page: db/settings come from env; admin user via configure.py
       GITEA__time__DEFAULT_UI_LOCATION: America/New_York
     ports:
       - "127.0.0.1:3000:3000"
@@ -238,13 +304,18 @@ services:
     restart: unless-stopped
 
   act_runner:
+    logging: *vglog
+    labels: { vanguard.service: act_runner, vanguard.category: deploy }
     image: docker.gitea.com/act_runner:latest
     container_name: act_runner
     environment:
       GITEA_INSTANCE_URL: https://${GIT_HOST:-git.example.com}/
       GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_TOKEN:-}   # set by ./registerGiteaRunner.sh
       GITEA_RUNNER_NAME: vanguard-runner
-      GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,dotnet:docker://mcr.microsoft.com/dotnet/sdk:10.0,python:docker://python:3.12"
+      # JavaScript actions (checkout, upload-artifact) need node INSIDE the job image, so a label used with
+      # them must point at an image that has it. "python" is bare CPython for plain run: steps; "python312"
+      # adds node and is what the Discord bot's CI uses, so its wheels match the python:3.12-slim runtime.
+      GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,dotnet:docker://mcr.microsoft.com/dotnet/sdk:10.0,python:docker://python:3.12,python312:docker://nikolaik/python-nodejs:python3.12-nodejs22"
     volumes:
       - /srv/appdata/act_runner:/data
       - /var/run/docker.sock:/var/run/docker.sock   # jobs run as sibling containers; root-equivalent on the host
@@ -252,7 +323,127 @@ services:
       - gitea
     restart: unless-stopped
 
+
+
+  # ================================================================ observability — Elastic (Phase 13)
+  # One store, one UI, one collector pair. Replaces VictoriaLogs+Vector+VictoriaMetrics+2×vmalert+Alertmanager
+  # +node_exporter+cAdvisor: eight containers for four, and Kibana is the single pane for logs, dashboards and
+  # alerting. Everything binds loopback and is published only through diag.HOST behind the Authentik gate.
+  elasticsearch:
+    logging: *vglog
+    labels: { vanguard.service: elasticsearch, vanguard.category: infra }
+    image: docker.elastic.co/elasticsearch/elasticsearch:9.2.0
+    container_name: elasticsearch
+    environment:
+      # Pinned deliberately. A JVM that sizes its own heap on a 62 GB box would take a quarter of it, on a
+      # machine whose spare CPU and RAM belong to x264. 2 GB is ample for one host's logs and metrics.
+      ES_JAVA_OPTS: -Xms2g -Xmx2g
+      discovery.type: single-node
+      # ON. Kibana's Observability apps (Infrastructure/Metrics) ask Elasticsearch whether the current user
+      # holds a privilege; with security off that API does not exist and they fail on every view. Humans still
+      # never see a password: Kibana signs everyone in as one user (observability/kibana.yml). Caddy +
+      # Authentik + DIAG_REQUIRE_GROUP remains the only real gate.
+      xpack.security.enabled: "true"
+      xpack.security.http.ssl.enabled: "false"      # plain HTTP, published on loopback only
+      xpack.security.transport.ssl.enabled: "false" # single-node: no transport traffic, and the check is skipped
+      xpack.security.enrollment.enabled: "false"
+      xpack.security.autoconfiguration.enabled: "false"
+      # Bootstraps the elastic superuser on first start with security on. configure.py generates it and then
+      # sets up kibana_system and the service users; the error below means .env predates that.
+      ELASTIC_PASSWORD: "${ES_ELASTIC_PASSWORD:?run ./configure.py --apply first — it generates the Elastic passwords}"
+      xpack.ml.enabled: "false"             # not used, and it reserves memory
+      cluster.routing.allocation.disk.watermark.low: 85%
+      cluster.routing.allocation.disk.watermark.high: 90%
+      cluster.routing.allocation.disk.watermark.flood_stage: 95%
+      bootstrap.memory_lock: "true"
+    ulimits:
+      memlock: { soft: -1, hard: -1 }
+      nofile: { soft: 65536, hard: 65536 }
+    ports:
+      - "127.0.0.1:9200:9200"
+    volumes:
+      - /srv/appdata/elasticsearch:/usr/share/elasticsearch/data
+    healthcheck:
+      test: ["CMD-SHELL", "curl -sf -u elastic:$${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"]
+      interval: 20s
+      timeout: 10s
+      retries: 12
+      start_period: 90s
+    restart: unless-stopped
+
+  kibana:
+    logging: *vglog
+    labels: { vanguard.service: kibana, vanguard.category: infra }
+    image: docker.elastic.co/kibana/kibana:9.2.0
+    container_name: kibana
+    # Everything non-secret lives in observability/kibana.yml (which explains the auth model); only the
+    # generated secrets arrive as environment variables, which that file interpolates.
+    environment:
+      DIAG_HOST: ${DIAG_HOST:-diag.example.com}
+      KIBANA_ENCRYPTION_KEY: ${KIBANA_ENCRYPTION_KEY}
+      ES_KIBANA_PASSWORD: "${ES_KIBANA_PASSWORD:?run ./configure.py --apply first}"
+      VG_DIAG_USER: ${VG_DIAG_USER:-vanguard_diag}
+      VG_DIAG_PASSWORD: "${VG_DIAG_PASSWORD:?run ./configure.py --apply first}"
+    volumes:
+      - ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
+    ports:
+      - "127.0.0.1:5601:5601"
+    depends_on:
+      elasticsearch: { condition: service_healthy }
+    restart: unless-stopped
+
+  filebeat:
+    logging: *vglog
+    labels: { vanguard.service: filebeat, vanguard.category: infra }
+    image: docker.elastic.co/beats/filebeat:9.2.0
+    container_name: filebeat
+    environment:                            # Elasticsearch requires auth now; the config interpolates these
+      ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
+      ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
+    user: root
+    command: ["filebeat", "-e", "--strict.perms=false"]
+    volumes:
+      - ./observability/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - /var/run/docker.sock:/var/run/docker.sock     # autodiscover reads container names and our vanguard.* labels
+      - /var/log/journal:/var/log/journal:ro          # our own systemd units
+      - /run/log/journal:/run/log/journal:ro          # ...and the volatile journal, if this host has no persistent one
+      - /etc/machine-id:/etc/machine-id:ro
+      - filebeat_data:/usr/share/filebeat/data
+    depends_on:
+      elasticsearch: { condition: service_healthy }
+    restart: unless-stopped
+
+  metricbeat:
+    logging: *vglog
+    labels: { vanguard.service: metricbeat, vanguard.category: infra }
+    image: docker.elastic.co/beats/metricbeat:9.2.0
+    container_name: metricbeat
+    environment:                            # Elasticsearch requires auth now; the config interpolates these
+      ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
+      ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
+    user: root
+    command: ["metricbeat", "-e", "--strict.perms=false"]
+    # system + docker modules replace node_exporter and cAdvisor, and ship their own Kibana dashboards.
+    network_mode: host                      # real interfaces, and loopback reach to MediaMTX/Caddy/ES/Kibana
+    pid: host
+    volumes:
+      - ./observability/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /proc:/hostfs/proc:ro
+      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
+      - /:/hostfs:ro
+      - metricbeat_data:/usr/share/metricbeat/data
+    depends_on:
+      elasticsearch: { condition: service_healthy }
+    restart: unless-stopped
+
+networks:
+  botnet:                               # isolates discord-bot from the shared default network (see above)
+
 volumes:
   copyparty_hist:
+  filebeat_data:
+  metricbeat_data:
   caddy_data:
   caddy_config:

firewall/nftables.conf

@@ -1,6 +1,7 @@
 #!/usr/sbin/nft -f
 # vanguard-stream host firewall. Own table so (re)applying never disturbs Docker's rules.
-# Input only: Docker-published ports (1935, 8890, 22000) arrive via Docker's forward chain and are
+# Input only. MediaMTX runs on the host network, so 1935, 8890/udp and 8189 arrive through THIS chain (the
+# accepts below). Docker-published ports (22000) arrive via Docker's forward chain and are
 # listed here for the host-network case (Caddy) and clarity. Nothing else should listen publicly.
 
 destroy table inet vanguard

install.sh

@@ -7,7 +7,7 @@ OWNER="${SUDO_USER:-debian}"      # the login user; owns /srv/media and may run
 
 echo "== packages"
 apt-get update -q
-apt-get install -y -q ca-certificates curl gnupg ffmpeg jq psmisc rclone python3 xfsprogs
+apt-get install -y -q ca-certificates curl gnupg ffmpeg jq psmisc rclone python3 python3-yaml xfsprogs git nodejs npm rsync
 
 if ! command -v docker >/dev/null; then
   echo "== docker"
@@ -38,22 +38,43 @@ echo "== media user + folders"
 getent passwd 1000 >/dev/null || useradd -r -u 1000 -d /srv/media -s /usr/sbin/nologin media
 usermod -aG docker "$OWNER" 2>/dev/null || true
 install -d -o 1000 -g 1000 /srv/media/{streams,clips,finals,trash,.markers,.syncthing}
+install -d -o 1000 -g 1000 /srv/media/.markers/metrics                             # script counters -> dumpd /metrics
 install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/authentik/{media,templates,certs}
 install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
 install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
+install -d -o 1000 -g 1000 /srv/bot /srv/bot/app      # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md)
+install -d -o 1000 -g 1000 /srv/appdata/elasticsearch                              # the ES image runs as uid 1000
+install -d /var/lib/vanguard                                                       # deadman state
+# Elasticsearch refuses to start below this. Host setting, so set it now and persist it.
+sysctl -q -w vm.max_map_count=262144
+printf 'vm.max_map_count=262144\n' > /etc/sysctl.d/99-vanguard-elasticsearch.conf
 install -d -o "$OWNER" /srv/live                 # live portal static build (deployLive.sh)
 install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire}
 for d in /srv/media/clips/*/; do
   [[ -e "$d/.stignore" ]] || install -m 0644 -o 1000 -g 1000 syncthing/stignore "$d/.stignore"
+  [[ -e "$d/.stlocal" ]] || install -m 0644 -o 1000 -g 1000 /dev/null "$d/.stlocal"     # #include'd by .stignore
 done
 
 echo "== host scripts + units"
-install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py /usr/local/bin/
+# bin/install.list is the single list, shared with configure.py --apply and deploy.sh. Three hand-kept
+# copies had drifted, so a push-to-deploy was installing half of them.
+grep -vE '^\s*(#|$)' bin/install.list | while read -r mode path; do
+  install -m "$mode" "$path" /usr/local/bin/
+done
+install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot   # dumpd runs this for the bot CI's POST /deploy-bot
+install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/      # watches the observability stack from outside it
+install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/        # takes Kibana's alert documents to Discord
+# Read by the root timers vanguard-alerts.sh (delivery) and vanguard-deadman.sh. configure.py rewrites it
+# from ALERT_WEBHOOK_URL on every apply; an empty one means "journal only, and exit non-zero".
+if [[ ! -f /etc/vanguard/alert-webhook ]]; then
+  install -d -m 0750 /etc/vanguard
+  install -m 0600 -o root -g root /dev/null /etc/vanguard/alert-webhook   # root timers are the only readers
+fi
 install -d -m 0750 /etc/vanguard
 install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
 install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/
 systemctl daemon-reload
-systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service
+systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer vanguard-alerts.timer
 
 echo "== sysctl"
 cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'
@@ -67,6 +88,12 @@ echo "== compose images (the stack is started by ./configure.py --full, which ge
 chown -R "$OWNER:" .
 docker compose pull -q --ignore-buildable 2>/dev/null || docker compose pull -q || true
 
+echo "== live portal: build into /srv/live (node)"
+if [[ -d live && -f live/package.json ]]; then
+  ( cd live && sudo -u "$OWNER" -H npm install --no-audit --no-fund --loglevel=error && sudo -u "$OWNER" -H npm run build ) \
+    && install -d -o "$OWNER" /srv/live && rsync -a --delete live/dist/ /srv/live/ && echo "   built" || echo "   portal build failed (run ./deployLive.sh from a dev box later)"
+fi
+
 echo "== host firewall (nftables; own table, input only; keeps this SSH session alive)"
 if [[ -f firewall/firewall.sh ]]; then
   bash firewall/firewall.sh

jellyfin-plugin-vanguard/.gitignore

@@ -1,4 +1,5 @@
 bin/
 obj/
 *.user
-dist/
+dist/*
+!dist/VanguardMetadata_*.zip

live/src/App.jsx

@@ -1,19 +1,57 @@
 import React, { useCallback, useEffect, useRef, useState } from 'react'
-import { DockviewReact } from 'dockview-react'
+import { DockviewReact, positionToDirection } from 'dockview-react'
 import Player from './Player.jsx'
 import { fetchStreams } from './api.js'
 import { audio } from './useAudio.js'
 import GoLive from './GoLive.jsx'
+import Tab from './Tab.jsx'
+import GroupLabel from './GroupLabel.jsx'
+import GroupActions from './GroupActions.jsx'
+import { streamLabel } from './names.js'
+import { layout, ding } from './layout.js'
+import Tray, { STREAM_MIME } from './Tray.jsx'
 
 const components = { player: Player }
+const tabComponents = { player: Tab }
 const POLL_MS = 3000
 
+// Lay every panel out as a grid (cols = ceil(sqrt(n)), row-major in stream-start order) and equalize sizes.
+// Moves the existing panels (dockview keeps the React component and its <video> alive), so viewers do not
+// reconnect. Called when a stream joins or leaves, and from the "Arrange" button.
+function addStream(api, s, position) {
+  api.addPanel({ id: s.id, component: 'player', tabComponent: 'player', title: streamLabel(s.id, s), params: s, position })
+}
+
+let arranging = false
+export function arrange(api) {
+  arranging = true
+  try { arrangeInner(api) } finally { arranging = false }
+}
+function arrangeInner(api) {
+  const panels = [...api.panels].sort((a, b) => ((a.params?.since || '').localeCompare(b.params?.since || '')) || a.id.localeCompare(b.id))
+  const n = panels.length; if (n < 2) return
+  if (api.hasMaximizedGroup?.()) api.exitMaximizedGroup?.()     // a maximised window would hide the new grid
+  const cols = Math.ceil(Math.sqrt(n))
+  for (let i = 1; i < n; i++) {
+    const r = Math.floor(i / cols), c = i % cols
+    const ref = r === 0 ? panels[i - 1] : panels[(r - 1) * cols + c]      // right of the previous in row 0, below the one above otherwise
+    panels[i].api.moveTo({ group: ref.group, position: r === 0 ? 'right' : 'bottom', skipSetActive: true })
+  }
+  const W = api.width || 0, H = api.height || 0; if (!W || !H) return
+  panels.forEach((p, i) => {
+    const c = i % cols, rowsInCol = Math.ceil((n - c) / cols)
+    p.group.api.setSize({ width: Math.floor(W / cols), height: Math.floor(H / rowsInCol) })
+  })
+}
+
+window.__vg = { layout }   // debug/test hook
 export default function App() {
   const apiRef = useRef(null)
   const [streams, setStreams] = useState([])
   const [err, setErr] = useState(null)
   const [mode, setMode] = useState('watch')          // watch | golive
   const focused = audio.use(s => s.focused)
+  const custom = layout.use(s => s.custom)
 
   // poll MediaMTX; open/close panels to match
   useEffect(() => {
@@ -25,19 +63,22 @@ export default function App() {
         const api = apiRef.current; if (!api) return   // (in golive mode the dock is unmounted: no viewer connections)
         const have = new Set(api.panels.map(p => p.id))
         const want = new Set(list.map(s => s.id))
+        let changed = false
+        const custom = layout.get().custom
         for (const s of list) {
           if (!have.has(s.id)) {
-            const last = api.panels[api.panels.length - 1]
+            if (custom && api.panels.length) {                 // hands off a custom layout: park it in the tray and ding
-            api.addPanel({
+              if (!layout.get().pending.some(p => p.id === s.id)) { layout.addPending(s); ding() }
-              id: s.id, component: 'player', title: `${s.user} · ${s.title}`,
+            } else {
-              params: s,
+              addStream(api, s, api.panels.length ? { direction: 'right' } : undefined); changed = true
-              position: last ? { referencePanel: last.id, direction: api.panels.length % 2 ? 'below' : 'right' } : undefined,
+            }
-            })
           } else {
             api.getPanel(s.id)?.api.updateParameters(s)
           }
         }
-        for (const p of [...api.panels]) if (!want.has(p.id)) { api.removePanel(p); audio.forget(p.id) }
+        for (const p of [...api.panels]) if (!want.has(p.id)) { api.removePanel(p); audio.forget(p.id); changed = true }
+        layout.keepPending(want)
+        if (changed && !layout.get().custom) arrange(api)
       } catch (e) { if (alive) setErr(e.message) }
     }
     tick(); const t = setInterval(tick, POLL_MS)
@@ -45,9 +86,28 @@ export default function App() {
   }, [])
 
   const onReady = useCallback(e => {
-    apiRef.current = e.api
+    const api = e.api; apiRef.current = api
-    e.api.onDidActivePanelChange(p => audio.focus(p?.id ?? null))
+    api.onDidActivePanelChange(p => audio.focus(p?.id ?? null))
+    api.onDidMovePanel(() => { if (!arranging) layout.markCustom() })           // a drag by the viewer, not our arrange()
+    api.onUnhandledDragOver(ev => {                                             // chips from the tray: show dockview's drop targets
+      const dt = ev.nativeEvent?.dataTransfer; if (dt && Array.from(dt.types || []).includes(STREAM_MIME)) ev.accept()
+    })
+    api.onDidDrop(ev => {
+      const id = ev.nativeEvent?.dataTransfer?.getData(STREAM_MIME); if (!id) return
+      const dir = positionToDirection(ev.position)
+      place(id, ev.group ? { referenceGroup: ev.group, direction: dir } : { direction: dir === 'within' ? 'right' : dir })
+    })
   }, [])
+  // add a parked stream where the viewer dropped/clicked it (custom layout stays custom)
+  const place = (id, position) => {
+    const api = apiRef.current; const s = layout.get().pending.find(p => p.id === id); if (!api || !s) return
+    addStream(api, s, position); layout.dropPending(id)
+  }
+  const arrangeAll = () => {                                                     // Arrange: place everything, re-grid, back to auto
+    const api = apiRef.current; if (!api) return
+    for (const s of layout.get().pending) addStream(api, s, { direction: 'right' })
+    layout.reset(); arrange(api)
+  }
 
   return (
     <div className="app">
@@ -56,6 +116,8 @@ export default function App() {
         <span className="count">{streams.length ? `${streams.length} live` : 'nobody is live'}{err ? ` · ${err}` : ''}</span>
         <span className="spacer" />
         <span className="count">audio follows the focused window</span>
+        <span className="count">{custom ? 'layout: yours (new streams wait in the tray)' : 'layout: auto grid'}</span>
+        <button onClick={arrangeAll} title="lay every window out as an even grid and go back to automatic layout">Arrange</button>
         <button onClick={() => audio.muteAll()} title="mute every window until you click one">Mute all</button>
         <button onClick={() => audio.followFocus()} title="clear pinned audio; only the focused window plays sound">Follow focus</button>
         <button className="golive-btn" onClick={() => { audio.muteAll(); apiRef.current = null; setMode('golive') }} title="stop watching and broadcast a window from this browser">Go Live</button>
@@ -67,8 +129,11 @@ export default function App() {
         {streams.length === 0 && !err && (
           <div className="empty"><h2>Nobody is live right now</h2><span>This page updates itself when a stream starts.</span></div>
         )}
-        <div className="dock" style={{ display: streams.length ? 'block' : 'none' }}>
+        <Tray onPlace={place} />
-          <DockviewReact className="dockview-theme-vanguard" components={components} onReady={onReady} />
+        <div className="dock" style={{ display: streams.length ? 'block' : 'none' }}
+             onPointerDownCapture={e => { if (e.target?.closest?.('.dv-sash')) layout.markCustom() }}>
+          <DockviewReact className="dockview-theme-vanguard" components={components} tabComponents={tabComponents}
+                         leftHeaderActionsComponent={GroupLabel} rightHeaderActionsComponent={GroupActions} onReady={onReady} />
         </div>
       </>)}
     </div>

live/src/GoLive.jsx

@@ -1,5 +1,5 @@
 import React, { useEffect, useRef, useState } from 'react'
-import { captureWindow, publish, fetchStreamKey, defaultTitle, cleanTitle, MAX_KBPS } from './broadcast.js'
+import { captureWindow, publish, fetchStreamKey, defaultTitle, cleanTitle, withTimecodeBand, MAX_KBPS, windowLoss, shouldSwitchToTcp } from './broadcast.js'
 
 // Broadcaster view: replaces the viewer grid while you're live.
 export default function GoLive({ onExit }) {
@@ -14,6 +14,83 @@ export default function GoLive({ onExit }) {
   const [started, setStarted] = useState(null)
   const [elapsed, setElapsed] = useState(0)
   const [hasAudio, setHasAudio] = useState(true)
+  const [tcBand, setTcBand] = useState(() => { try { return localStorage.getItem('vg.tcband') !== '0' } catch { return true } })
+  const bandRef = useRef(null)
+  const streamRef = useRef(null)                      // the tracks being published (survive a transport switch)
+  const samplesRef = useRef([])                       // recent {t, lost, sent} for the loss window
+  const switchingRef = useRef(false)
+  const [transport, setTransport] = useState('udp')
+  const [switched, setSwitched] = useState('')        // "switched to TCP after 0.6% loss" once it happens
+  // Remembered per browser. Starlink drops packets in bursts on every satellite handoff (its scheduler works in
+  // ~15 s cycles), so a dish user hits the auto-switch on every single stream; once it has fired, the next stream
+  // starts on TCP and skips the 20 s of corrupted recording and the extra file. Untick in setup to go back.
+  const [preferTcp, setPreferTcp] = useState(() => { try { return localStorage.getItem('vg.golive.tcp') === '1' } catch { return false } })
+  const [ended, setEnded] = useState('')               // why the server ended the stream, shown back on the setup card
+  const [surface, setSurface] = useState(() => { try { return localStorage.getItem('vg.golive.surface') === 'screen' ? 'screen' : 'window' } catch { return 'window' } })
+  const pickSurface = v => { setSurface(v); try { localStorage.setItem('vg.golive.surface', v) } catch {} }
+  const missingRef = useRef(0)                          // consecutive health polls that did not list our path
+  const disconnectRef = useRef(null)
+
+  // A kick is invisible to WebRTC for a long time: the browser sits in "disconnected" until ICE consent times
+  // out (~30 s) and only then goes "failed". So: treat failed/closed as the end immediately, give
+  // "disconnected" 8 s to recover, and ask the server every 5 s whether our path is still live — the kick
+  // path answers that in seconds and tells us whether we are blocked.
+  const endedByServer = async (why) => {
+    if (!sessionRef.current) return
+    const s = sessionRef.current; sessionRef.current = null
+    try { await s.stop() } catch {}
+    streamRef.current?.getTracks().forEach(t => t.stop()); streamRef.current = null
+    bandRef.current?.stop(); bandRef.current = null
+    if (videoRef.current) videoRef.current.srcObject = null
+    setPhase('setup'); setStarted(null); setStats(null); setConn(''); setTransport('udp'); setSwitched('')
+    setEnded(why)
+  }
+  const onConn = (state) => {
+    setConn(state)
+    if (switchingRef.current) return
+    if (state === 'failed' || state === 'closed') endedByServer('The server ended the stream.')
+    if (state === 'disconnected' && !disconnectRef.current) {
+      disconnectRef.current = setTimeout(() => { disconnectRef.current = null; if (sessionRef.current?.pc?.connectionState !== 'connected') endedByServer('Connection to the server was lost.') }, 8000)
+    }
+    if (state === 'connected' && disconnectRef.current) { clearTimeout(disconnectRef.current); disconnectRef.current = null }
+  }
+  useEffect(() => {
+    if (phase !== 'live') return
+    missingRef.current = 0
+    const t = setInterval(async () => {
+      const s = sessionRef.current; if (!s || switchingRef.current) return
+      try {
+        const r = await fetch('/me/stream-health', { credentials: 'same-origin' }); if (!r.ok) return
+        const d = await r.json()
+        const mine = (d.streams || []).some(x => x.path.toLowerCase() === s.path.toLowerCase())
+        missingRef.current = mine ? 0 : missingRef.current + 1
+        if (missingRef.current >= 2) {
+          endedByServer(d.blocked ? `An admin stopped your stream. Publishing is paused for your account until ${new Date(d.blocked.until).toLocaleTimeString()}.`
+                                  : 'The server ended the stream.')
+        }
+      } catch {}
+    }, 5000)
+    return () => clearInterval(t)
+  }, [phase])
+  const rememberTcp = on => { setPreferTcp(on); try { localStorage.setItem('vg.golive.tcp', on ? '1' : '0') } catch {} }
+
+  // Lossy uplink: re-publish the same tracks over TCP. One new recording file (the sweep stitches it), ~2 s
+  // of picture lost during the swap, and from then on no packet loss can reach the recorder. Never flips back:
+  // the latency cost of TCP is a nuisance, a corrupted recording is a loss.
+  const switchToTcp = async (loss) => {
+    const old = sessionRef.current; if (!old || switchingRef.current || old.transport === 'tcp') return
+    switchingRef.current = true
+    setConn(`loss ${loss.pct.toFixed(1)}% — switching to TCP…`)
+    try {
+      await old.release()
+      const next = await publish({ stream: streamRef.current, path: old.path, user: old.user, pass: old.pass, onState: onConn, transport: 'tcp' })
+      sessionRef.current = next; samplesRef.current = []
+      setTransport(next.transport)
+      if (next.transport === 'tcp') { rememberTcp(true); setSwitched(`switched to TCP after ${loss.pct.toFixed(1)}% loss — your next streams start on TCP`) }
+      else setSwitched('server offered no TCP candidate; still on UDP')
+    } catch (e) { setErr(`re-publish over TCP failed: ${e.message}`); setPhase('error') }
+    finally { switchingRef.current = false }
+  }
 
   useEffect(() => { fetchStreamKey().then(setKey).catch(e => { setErr(e.message); setPhase('error') }) }, [])
   useEffect(() => { if (!started) return; const t = setInterval(() => setElapsed(Math.floor((Date.now() - started) / 1000)), 1000); return () => clearInterval(t) }, [started])
@@ -23,11 +100,21 @@ export default function GoLive({ onExit }) {
     const t = setInterval(async () => {
       const pc = sessionRef.current?.pc; if (!pc) return
       const report = await pc.getStats()
-      let codec = ''
+      let codec = '', lost = 0, sent = 0
+      for (const s of report) if (s.type === 'remote-inbound-rtp' && s.kind === 'video') lost = s.packetsLost || 0   // what the server reports back
       for (const s of report) if (s.type === 'outbound-rtp' && s.kind === 'video') {
+        sent = s.packetsSent || 0
         const now = Date.now(); const kbps = Math.round((s.bytesSent - last.bytes) * 8 / ((now - last.t) || 1)); last = { bytes: s.bytesSent, t: now }
         const c = s.codecId && report.get(s.codecId); codec = c ? `${c.mimeType.replace('video/', '')}${/profile-level-id=64/i.test(c.sdpFmtpLine || '') ? ' High' : /profile-level-id=4d/i.test(c.sdpFmtpLine || '') ? ' Main' : /profile-level-id=42/i.test(c.sdpFmtpLine || '') ? ' Baseline' : ''}` : ''
-        setStats({ kbps, fps: Math.round(s.framesPerSecond || 0), w: s.frameWidth || 0, h: s.frameHeight || 0, q: s.qualityLimitationReason, enc: s.encoderImplementation || '', codec })
+        // nack = retransmit requests the server sent us; rtx = packets we re-sent. Both 0 under loss means the
+        // server is not asking for retransmits, and every lost packet is a corrupted frame in the recording.
+        setStats({ kbps, fps: Math.round(s.framesPerSecond || 0), w: s.frameWidth || 0, h: s.frameHeight || 0, q: s.qualityLimitationReason, enc: s.encoderImplementation || '', codec,
+                   lost, sent, nack: s.nackCount || 0, rtx: s.retransmittedPacketsSent || 0 })
+      }
+      if (sent && !switchingRef.current) {
+        const samples = samplesRef.current; samples.push({ t: Date.now(), lost, sent }); if (samples.length > 60) samples.shift()
+        const l = windowLoss(samples)
+        if (sessionRef.current?.transport === 'udp' && shouldSwitchToTcp(l)) switchToTcp(l)
       }
     }, 1000)
     return () => clearInterval(t)
@@ -39,17 +126,25 @@ export default function GoLive({ onExit }) {
     const t = cleanTitle(title); if (!t) return setErr('give it a title')
     if (!key) return setErr('no stream key')
     try {
+      // blocked by an admin? MediaMTX would answer the WHIP with a bare 401, which reads like a bad key — ask first
+      try { const h = await (await fetch('/me/stream-health', { credentials: 'same-origin' })).json()
+            if (h.blocked) return setErr(`Publishing is paused for your account by an admin until ${new Date(h.blocked.until).toLocaleTimeString()}.`) } catch {}
       setPhase('capturing')
-      const stream = await captureWindow()
+      const captured = await captureWindow(surface)
-      setHasAudio(stream.getAudioTracks().length > 0)
+      setHasAudio(captured.getAudioTracks().length > 0)
+      captured.getVideoTracks()[0].onended = () => stop()            // user clicked Chrome's "Stop sharing"
+      let stream = captured
+      if (tcBand) { bandRef.current = withTimecodeBand(captured); stream = bandRef.current.stream }
       if (videoRef.current) { videoRef.current.srcObject = stream; videoRef.current.muted = true; videoRef.current.play().catch(() => {}) }
-      stream.getVideoTracks()[0].onended = () => stop()              // user clicked Chrome's "Stop sharing"
+      streamRef.current = stream; samplesRef.current = []; setSwitched('')
-      const session = await publish({ stream, path: `${key.user}/${t}`, user: key.user, pass: key.pass, onState: setConn })
+      setEnded('')
-      sessionRef.current = session
+      const session = await publish({ stream, path: `${key.user}/${t}`, user: key.user, pass: key.pass, onState: onConn, transport: preferTcp ? 'tcp' : 'udp' })
+      sessionRef.current = session; setTransport(session.transport)
+      if (preferTcp && session.transport !== 'tcp') setSwitched('server offered no TCP candidate; on UDP')
       setStarted(Date.now()); setPhase('live')
     } catch (e) { setErr(e.message); setPhase(e.message === 'capture cancelled' ? 'setup' : 'error') }
   }
-  const stop = async () => { await sessionRef.current?.stop(); sessionRef.current = null; if (videoRef.current) videoRef.current.srcObject = null; setPhase('setup'); setStarted(null); setStats(null); setConn('') }
+  const stop = async () => { await sessionRef.current?.stop(); sessionRef.current = null; streamRef.current?.getTracks().forEach(t => t.stop()); streamRef.current = null; bandRef.current?.stop(); bandRef.current = null; if (videoRef.current) videoRef.current.srcObject = null; setPhase('setup'); setStarted(null); setStats(null); setConn(''); setTransport('udp'); setSwitched('') }
 
   const mmss = s => `${String(Math.floor(s / 3600)).padStart(2, '0')}:${String(Math.floor(s / 60) % 60).padStart(2, '0')}:${String(s % 60).padStart(2, '0')}`
   return (
@@ -65,7 +160,7 @@ export default function GoLive({ onExit }) {
         <video ref={videoRef} autoPlay playsInline muted className={phase === 'live' || phase === 'capturing' ? 'show' : ''} />
         {phase !== 'live' && (
           <div className="golive-form">
-            <label>Stream title <small>(becomes the Jellyfin title; letters, digits, _ . - ; no double underscore)</small></label>
+            <label>Stream title <small>(becomes the Jellyfin title; letters, digits, _ - ; no double underscore)</small></label>
             <input value={title} onChange={e => setTitle(e.target.value)} disabled={phase === 'capturing'} />
             <div className="hint" style={{marginTop: '-4px'}}>stream name: <b>{key ? key.user : '<you>'}/{cleanTitle(title) || '—'}</b></div>
             <div className="hint">
@@ -74,6 +169,28 @@ export default function GoLive({ onExit }) {
               encoder. On Linux and Mac Chrome has no window audio and (on NVIDIA) no hardware encoder, so the stream is
               silent and visibly soft — use OBS with the dock there. Capped at {MAX_KBPS / 1000} Mbps, up to 1440p60, H.264.
             </div>
+            <div className="hint" style={{display:'flex',gap:16,alignItems:'center',flexWrap:'wrap'}}>
+              <span>Capture:</span>
+              <label style={{display:'flex',gap:6,alignItems:'center'}}><input type="radio" name="surface" checked={surface === 'window'} onChange={() => pickSurface('window')} /> the game's window</label>
+              <label style={{display:'flex',gap:6,alignItems:'center'}}><input type="radio" name="surface" checked={surface === 'screen'} onChange={() => pickSurface('screen')} /> the whole screen</label>
+            </div>
+            <div className="hint">
+              <b>Frozen picture that only updates when you alt-tab?</b> The game is in <i>exclusive</i> full-screen, which
+              window capture cannot see. Either set the game to <b>borderless / windowed full-screen</b> (best: keeps
+              window audio), or pick <b>the whole screen</b> above (always works; carries system audio, and shows
+              whatever else is on that monitor).
+            </div>
+            <label className="hint" style={{display:'flex',gap:8,alignItems:'center'}}>
+              <input type="checkbox" checked={tcBand} onChange={e => { setTcBand(e.target.checked); try { localStorage.setItem('vg.tcband', e.target.checked ? '1' : '0') } catch {} }} />
+              <span>Timecode band: add a thin strip below the picture carrying the server clock, so editors get a
+              frame-accurate timecode track. Viewers here never see it; Jellyfin shows a thin line at the bottom.</span>
+            </label>
+            <label className="hint" style={{display:'flex',gap:8,alignItems:'center'}}>
+              <input type="checkbox" checked={preferTcp} onChange={e => rememberTcp(e.target.checked)} />
+              <span>Lossy uplink (Starlink, cellular, bad wifi): send over TCP from the start. Nothing gets lost on the
+              way to the recording; the cost is a second or two of extra delay for viewers. Ticks itself the first
+              time a stream has to switch.</span>
+            </label>
             <div className="warn-box">
               <b>⚠ This stream is recorded.</b> Everything the window shows and every sound it makes is saved to the
               server, visible to editors and members in Jellyfin, and kept until it is dumped or archived. Close
@@ -81,7 +198,8 @@ export default function GoLive({ onExit }) {
               should be thrown away (3-day undo).
             </div>
             {err && <div className="err">{err}</div>}
-            <button className="primary" disabled={!key || phase === 'capturing'} onClick={start}>{phase === 'capturing' ? 'Waiting for the window picker…' : 'Start'}</button>
+            {ended && <div className="warn-box"><b>Stream ended.</b> {ended} {/paused/.test(ended) ? 'Starting again before then will be refused.' : 'You can start again.'}</div>}
+            <button className="primary" disabled={!key || phase === 'capturing'} onClick={start}>{phase === 'capturing' ? 'Waiting for the picker…' : 'Start'}</button>
           </div>
         )}
         {phase === 'live' && (
@@ -90,8 +208,9 @@ export default function GoLive({ onExit }) {
             <span className="rec" title="this stream is being recorded to the server">● REC</span>
             <span>{mmss(elapsed)}</span>
             <span>{key.user}/{cleanTitle(title)}</span>
-            <span>{conn}</span>
+            <span>{conn}{transport === 'tcp' ? ' · via TCP' : ''}</span>
-            {stats && <span>{stats.w}×{stats.h} · {stats.fps} fps · {(stats.kbps / 1000).toFixed(1)} Mbps · {stats.codec}{stats.enc ? ` (${/openh264|libvpx|software|SimulcastEncoderAdapter/i.test(stats.enc) ? 'software' : 'hardware'}: ${stats.enc})` : ''}{stats.q && stats.q !== 'none' ? ` · limited by ${stats.q}` : ''}</span>}
+            {switched && <span className="warn" title="packet loss on your uplink was corrupting the recording; TCP retransmits everything, at the cost of some latency">{switched}</span>}
+            {stats && <span>{stats.w}×{stats.h} · {stats.fps} fps · {(stats.kbps / 1000).toFixed(1)} Mbps · {stats.codec}{stats.enc ? ` (${/openh264|libvpx|software|SimulcastEncoderAdapter/i.test(stats.enc) ? 'software' : 'hardware'}: ${stats.enc})` : ''}{stats.q && stats.q !== 'none' ? ` · limited by ${stats.q}` : ''}{stats.sent ? ` · loss ${(100 * stats.lost / stats.sent).toFixed(2)}% (${stats.nack} nack, ${stats.rtx} resent)` : ''}</span>}
             {!hasAudio && <span className="warn">no audio captured — restart and tick "Share audio"</span>}
           </div>
         )}

live/src/GroupActions.jsx

@@ -0,0 +1,16 @@
+import React, { useEffect, useState } from 'react'
+
+// Right end of every tab bar: maximise / restore this group.
+export default function GroupActions({ api, containerApi }) {
+  const [max, setMax] = useState(() => api.isMaximized())
+  useEffect(() => {
+    const d = containerApi.onDidMaximizedGroupChange?.(() => setMax(api.isMaximized()))
+    return () => d?.dispose?.()
+  }, [api, containerApi])
+  const toggle = e => { e.stopPropagation(); if (api.isMaximized()) api.exitMaximized(); else api.maximize(); setMax(api.isMaximized()) }
+  return (
+    <div className="groupactions" onPointerDown={e => e.stopPropagation()}>
+      <button className="gbtn" onClick={toggle} title={max ? 'restore' : 'maximise this window'}>{max ? '🗗' : '⛶'}</button>
+    </div>
+  )
+}

live/src/GroupLabel.jsx

@@ -0,0 +1,23 @@
+import React, { useEffect, useRef, useState } from 'react'
+import { names } from './names.js'
+import { layout } from './layout.js'
+
+// Left end of every tab bar: an editable group name (double-click). Session-only.
+export default function GroupLabel({ group }) {
+  const id = group.id
+  const name = names.use(s => s.groups[id])
+  const [editing, setEditing] = useState(false)
+  const [draft, setDraft] = useState('')
+  const ref = useRef(null)
+  useEffect(() => { if (editing) { ref.current?.focus(); ref.current?.select() } }, [editing])
+  const start = e => { e.stopPropagation(); setDraft(name || ''); setEditing(true) }
+  const commit = () => { if (!editing) return; names.renameGroup(id, draft.trim()); layout.markCustom(); setEditing(false) }
+  return (
+    <div className="grouplabel" title="double-click to name this group" onDoubleClick={start} onPointerDown={e => e.stopPropagation()}>
+      {editing
+        ? <input ref={ref} value={draft} placeholder="group name" onChange={e => setDraft(e.target.value)} onBlur={commit}
+                 onKeyDown={e => { e.stopPropagation(); if (e.key === 'Enter') commit(); if (e.key === 'Escape') setEditing(false) }} />
+        : <span className={name ? 'gname' : 'hint'}>{name || '＋ name'}</span>}
+    </div>
+  )
+}

live/src/Player.jsx

@@ -2,13 +2,19 @@ import React, { useEffect, useRef, useState } from 'react'
 import Hls from 'hls.js'
 import { whepUrl, hlsUrl } from './api.js'
 import { audio, effectiveMuted } from './useAudio.js'
+import { names } from './names.js'
 
 // WebRTC (WHEP) first, HLS fallback. Mute/volume come from the shared audio store.
 export default function Player({ params, api }) {
   const { id, user, title } = params
+  const custom = names.use(s => s.streams[id])      // viewer-side rename (tab double-click)
+  const whep = params.whep || id                       // Opus audio mirror path when the source audio is AAC
   const videoRef = useRef(null)
   const [status, setStatus] = useState('connecting…')
   const [viewers, setViewers] = useState(params.viewers ?? 0)
+  const wrapRef = useRef(null)
+  const [dims, setDims] = useState(null)              // {w, h, gameH}: video size + game height (h minus the timecode band)
+  const [box, setBox] = useState(null)                // {W, H}: game-aspect frame fitted into the panel; video overflows it at the bottom
   const state = audio.use()
   const muted = effectiveMuted(state, id)
   const volume = state.streams[id]?.volume ?? 0.8
@@ -16,6 +22,26 @@ export default function Player({ params, api }) {
   const focused = state.focused === id
 
   useEffect(() => { audio.ensure(id) }, [id])
+  // size the frame: fit a box of the GAME aspect (video width / height-without-band) into the panel, letterboxed.
+  // The video is laid across the full box width, so its band rows fall below the box and are clipped.
+  useEffect(() => {
+    const el = wrapRef.current; if (!el) return
+    const fit = () => {
+      if (!dims) { setBox(null); return }
+      const cw = el.clientWidth, ch = el.clientHeight; if (!cw || !ch) return
+      const a = dims.w / dims.gameH
+      const W = cw / ch > a ? Math.floor(ch * a) : cw
+      setBox({ W, H: Math.floor(W / a) })
+    }
+    fit(); const ro = new ResizeObserver(fit); ro.observe(el); return () => ro.disconnect()
+  }, [dims])
+  const onSize = e => {
+    const v = e.currentTarget; const w = v.videoWidth, h = v.videoHeight; if (!w || !h) return
+    let band = Math.round(w / 160); band += band % 2
+    // only treat the extra rows as a band if what remains is a standard game height
+    const gameH = [720, 1080, 1440, 2160].includes(h - band) ? h - band : h
+    setDims(d => (d && d.w === w && d.h === h) ? d : { w, h, gameH })
+  }
   useEffect(() => { const d = api.onDidActiveChange?.(() => {}) ; return () => d?.dispose?.() }, [api])
   useEffect(() => { const t = setInterval(() => setViewers(params.viewers ?? 0), 2000); return () => clearInterval(t) }, [params.viewers])
 
@@ -55,7 +81,7 @@ export default function Player({ params, api }) {
         const offer = await pc.createOffer()
         await pc.setLocalDescription(offer)
         await new Promise(res => { if (pc.iceGatheringState === 'complete') return res(); const t = setTimeout(res, 1500); pc.onicegatheringstatechange = () => { if (pc.iceGatheringState === 'complete') { clearTimeout(t); res() } } })
-        const r = await fetch(whepUrl(id), { method: 'POST', headers: { 'Content-Type': 'application/sdp' }, body: pc.localDescription.sdp, credentials: 'same-origin' })
+        const r = await fetch(whepUrl(whep), { method: 'POST', headers: { 'Content-Type': 'application/sdp' }, body: pc.localDescription.sdp, credentials: 'same-origin' })
         if (!r.ok) throw new Error('whep ' + r.status)
         await pc.setRemoteDescription({ type: 'answer', sdp: await r.text() })
         setStatus('webrtc')
@@ -67,17 +93,21 @@ export default function Player({ params, api }) {
     }
     startWhep()
     return () => { cancelled = true; try { pc?.close() } catch {}; try { hls?.destroy() } catch {}; v.srcObject = null; v.removeAttribute('src') }
-  }, [id])
+  }, [id, whep])
 
   const icon = muted ? '🔇' : volume === 0 ? '🔈' : volume < 0.5 ? '🔉' : '🔊'
   return (
     <div className={'player' + (focused ? ' focused' : '')} onMouseDown={() => api.setActive?.()}>
-      <video ref={videoRef} autoPlay playsInline muted />
+      <div className="vwrap" ref={wrapRef}>
+        <div className="frame" style={box ? { width: box.W, height: box.H } : undefined}>
+          <video ref={videoRef} autoPlay playsInline muted onLoadedMetadata={onSize} onResize={onSize} />
+        </div>
+      </div>
       {status && <div className="status">{status}</div>}
       <div className="bar">
         <span className="live">LIVE</span>
         <span className="who">{user}</span>
-        <span className="title" title={title}>{title}</span>
+        <span className="title" title={title}>{custom || title}</span>
         <span className="viewers">👁 {viewers}</span>
         <button className={'mute' + (!muted ? ' on' : '') + (override === 'unmuted' && !focused ? ' pinned' : '')}
                 title={override ? (override === 'unmuted' ? 'audio pinned on (click to mute)' : 'muted (click to unmute)') : (focused ? 'focused: audio on (click to mute)' : 'muted: click to pin audio on')}

live/src/Tab.jsx

@@ -0,0 +1,25 @@
+import React, { useEffect, useRef, useState } from 'react'
+import { names } from './names.js'
+import { layout } from './layout.js'
+
+// Tab for a stream panel: double-click to rename (viewer-side label, kept in this browser). Empty = back to user · title.
+export default function Tab({ api, params }) {
+  const id = api.id
+  const custom = names.use(s => s.streams[id])
+  const base = `${params.user} · ${params.title}`
+  const [editing, setEditing] = useState(false)
+  const [draft, setDraft] = useState('')
+  const ref = useRef(null)
+  useEffect(() => { if (editing) { ref.current?.focus(); ref.current?.select() } }, [editing])
+  const start = e => { e.stopPropagation(); setDraft(custom || ''); setEditing(true) }
+  const commit = () => { if (!editing) return; const v = draft.trim(); names.rename(id, v); api.setTitle(v || base); layout.markCustom(); setEditing(false) }
+  return (
+    <div className="vtab" title={`${base}\ndouble-click to rename`} onDoubleClick={start}>
+      {editing
+        ? <input ref={ref} value={draft} placeholder={base} onChange={e => setDraft(e.target.value)}
+                 onPointerDown={e => e.stopPropagation()} onBlur={commit}
+                 onKeyDown={e => { e.stopPropagation(); if (e.key === 'Enter') commit(); if (e.key === 'Escape') setEditing(false) }} />
+        : <span className="vtab-text">{custom || base}</span>}
+    </div>
+  )
+}

live/src/Tray.jsx

@@ -0,0 +1,23 @@
+import React from 'react'
+import { layout } from './layout.js'
+import { streamLabel } from './names.js'
+
+export const STREAM_MIME = 'application/x-vanguard-stream'
+
+// Zero-height when empty. New streams in a custom layout wait here: drag a chip into the dock (any edge, any window,
+// or onto a tab bar to stack), or click it to add it as a new column on the right.
+export default function Tray({ onPlace }) {
+  const pending = layout.use(s => s.pending)
+  if (!pending.length) return null
+  return (
+    <div className="tray" title="new streams — drag into the layout, or click to add on the right">
+      {pending.map(s => (
+        <div key={s.id} className="chip" draggable
+             onDragStart={e => { e.dataTransfer.setData(STREAM_MIME, s.id); e.dataTransfer.setData('text/plain', s.id); e.dataTransfer.effectAllowed = 'move' }}
+             onClick={() => onPlace(s.id, { direction: 'right' })}>
+          <span className="blink">●</span> {streamLabel(s.id, s)}
+        </div>
+      ))}
+    </div>
+  )
+}

live/src/api.js

@@ -1,22 +1,37 @@
 // MediaMTX control API, proxied by Caddy at /api (-> 127.0.0.1:9997). Returns active streams.
+const VIDEO = new Set(['H264', 'H265', 'VP8', 'VP9', 'AV1', 'MJPEG', 'M-JPEG', 'MPEG-4 Video', 'MPEG-1 Video'])
+const firstSeen = new Map()                      // path -> first time we saw it ready (mirror grace period)
+const MIRROR_WAIT_MS = 15000
+
 export async function fetchStreams() {
   const r = await fetch('/api/v3/paths/list?itemsPerPage=200', { credentials: 'same-origin' })
   if (!r.ok) throw new Error(`api ${r.status}`)
   const data = await r.json()
-  return (data.items || [])
+  const ready = (data.items || []).filter(p => p.ready && (p.tracks || []).length > 0)
-    .filter(p => p.ready && (p.tracks || []).length > 0)
+  const byName = new Map(ready.map(p => [p.name, p]))
-    .map(p => {
+  const now = Date.now()
+  for (const p of ready) if (!firstSeen.has(p.name)) firstSeen.set(p.name, now)
+  for (const k of [...firstSeen.keys()]) if (!byName.has(k)) firstSeen.delete(k)
+  const out = []
+  for (const p of ready) {
+    if (p.name.endsWith('.webrtc')) continue       // audio mirror: played through its source's panel, never its own
+    const tracks = p.tracks || []
+    const audio = tracks.filter(t => !VIDEO.has(t))
+    const needsMirror = audio.length > 0 && !audio.includes('Opus')   // AAC etc. cannot cross WebRTC
+    const mirror = byName.get(p.name + '.webrtc')
+    if (needsMirror && !mirror && now - firstSeen.get(p.name) < MIRROR_WAIT_MS) continue   // give the mirror a moment to come up
     const [user, ...rest] = p.name.split('/')
-      return {
+    out.push({
       id: p.name,                                  // e.g. techy/VanguardTraining
       user,
       title: rest.join('/') || p.name,
-        viewers: (p.readers || []).length,
+      whep: mirror ? mirror.name : p.name,         // WebRTC plays the Opus mirror when there is one; HLS always the source
+      viewers: (p.readers || []).length + (mirror ? (mirror.readers || []).length - 1 : 0),   // the mirror's ffmpeg is one reader
       since: p.readyTime,
-        tracks: p.tracks || [],
+      tracks,
-      }
     })
-    .sort((a, b) => (a.since || '').localeCompare(b.since || ''))
+  }
+  return out.sort((a, b) => (a.since || '').localeCompare(b.since || ''))
 }
 
 export const whepUrl = id => `/webrtc/${id}/whep`

live/src/broadcast.js

@@ -1,6 +1,8 @@
 // Publish a captured window to MediaMTX over WHIP (WebRTC), through Caddy at /webrtc/<path>/whip.
 // Credentials go as query params so the Authentik forward-auth in front doesn't see an Authorization header.
 
+import VGClock from './vgclock.js'
+
 export const MAX_KBPS = 12000
 
 export function defaultTitle(now = new Date()) {
@@ -8,18 +10,25 @@ export function defaultTitle(now = new Date()) {
   return `HelmetCam-${d}`
 }
 // Must match the dock and the server: the sweep publishes <Title>__<timestamp>.mp4, so "__" is reserved.
-export const cleanTitle = t => t.trim().replace(/\s+/g, '-').replace(/[^A-Za-z0-9_.-]/g, '').replace(/_{2,}/g, '_').replace(/^[.-]+/, '').slice(0, 64)
+// No dots: the server names the WebRTC audio mirror <title>.webrtc, so a dot in a title would let a title
+// collide with a mirror name and pick up the wrong MediaMTX rule (one such collision recorded 36 GB twice).
+export const cleanTitle = t => t.trim().replace(/\s+/g, '-').replace(/[^A-Za-z0-9_-]/g, '').replace(/_{2,}/g, '_').replace(/^[.-]+/, '').slice(0, 64)
 
-export async function captureWindow() {
+// surface: 'window' (default) or 'screen'. Window capture on Windows goes through the compositor, and a game in
-  // Window-only picker with that window's own audio. Chrome/Edge 141+ honour windowAudio: "window"
+// EXCLUSIVE full-screen bypasses the compositor: Chrome then gets a frame only when Windows composites one, i.e.
-  // (only the shared window's sound); older Chrome offers no audio for a window at all.
+// on alt-tab — Breezy's "one frame, then another when I alt-tab". Two ways out: the game in borderless/windowed
+// full-screen (compositor sees every frame), or capture the whole screen, which Chrome reads straight from the
+// GPU output (desktop duplication) and which exclusive full-screen cannot hide from. Screen capture carries
+// system audio instead of one window's audio.
+export async function captureWindow(surface = 'window') {
+  const screen = surface === 'screen'
   const constraints = {
-    video: { displaySurface: 'window', frameRate: { ideal: 60, max: 60 }, width: { max: 2560 }, height: { max: 1440 } },
+    video: { displaySurface: screen ? 'monitor' : 'window', frameRate: { ideal: 60, max: 60 }, width: { max: 2560 }, height: { max: 1440 } },
     audio: { echoCancellation: false, noiseSuppression: false, autoGainControl: false, sampleRate: 48000, channelCount: 2 },
     windowAudio: 'window',            // capture the picked window's audio (Chrome 141+)
-    systemAudio: 'exclude',
+    systemAudio: screen ? 'include' : 'exclude',
     selfBrowserSurface: 'exclude',
-    monitorTypeSurfaces: 'exclude',   // hide "Entire screen": this is a single-window helmet cam
+    monitorTypeSurfaces: screen ? 'include' : 'exclude',   // "Entire screen" only when asked for: a helmet cam is one window
     surfaceSwitching: 'include',
     preferCurrentTab: false,
   }
@@ -32,8 +41,18 @@ export async function captureWindow() {
   }
 }
 
-export const MIN_KBPS = 6000
+// A FLOOR, not a target. It was 6000, which told Chrome never to encode below 6 Mbps: on an uplink that
+// could not sustain that, Chrome kept sending anyway, the network discarded the excess, and MediaMTX logged
+// "invalid FU-A packet (non-starting)" while every viewer saw blocky corruption. A floor this low still
+// skips the painfully slow cold ramp without disabling congestion control.
+export const MIN_KBPS = 1200
 export const START_KBPS = 9000
+// On TCP nothing is lost, so the only thing Chrome's estimator sees is delay — and a Starlink dish swings its
+// delay on every handoff, which reads as congestion and parks the bitrate near the floor: stable but soft.
+// A higher floor is safe HERE because the failure mode a floor causes on UDP (sending into loss) cannot happen
+// on TCP; if the uplink truly cannot carry it, latency grows instead of frames corrupting, which is the trade
+// the TCP switch already made. Still well under a dish's typical 10-20 Mbps up.
+export const TCP_MIN_KBPS = 4000
 
 function preferCodecs(pc, kind = 'video') {
   const caps = RTCRtpSender.getCapabilities?.(kind)
@@ -52,15 +71,55 @@ function preferCodecs(pc, kind = 'video') {
   tx.setCodecPreferences([...high, ...base, ...vp9, ...rest])
 }
 
-// Chrome honours x-google-* fmtp hints: skip the slow bandwidth ramp and never sit below MIN_KBPS.
+// Chrome honours x-google-* fmtp hints. start-bitrate skips the slow cold ramp, max-bitrate is the ceiling,
-function mungeBitrates(sdp) {
+// and min-bitrate is a floor that must stay LOW: a high floor defeats congestion control entirely.
+function mungeBitrates(sdp, minKbps = MIN_KBPS) {
   return sdp.replace(/^a=fmtp:(\d+) (.*)$/gm, (line, pt, params) =>
     /packetization-mode|profile-id|VP9/i.test(params) || /H264|VP9/i.test(sdp.match(new RegExp(`a=rtpmap:${pt} .*`))?.[0] || '')
-      ? `a=fmtp:${pt} ${params};x-google-start-bitrate=${START_KBPS};x-google-min-bitrate=${MIN_KBPS};x-google-max-bitrate=${MAX_KBPS}`
+      ? `a=fmtp:${pt} ${params};x-google-start-bitrate=${START_KBPS};x-google-min-bitrate=${minKbps};x-google-max-bitrate=${MAX_KBPS}`
       : line)
 }
 
-export async function publish({ stream, path, user, pass, onState }) {
+// Overscan band: draw the captured video onto a canvas that is `band` rows taller and paint the
+// server-synced timecode strip into the extra rows every frame. Returns a new MediaStream.
+export function withTimecodeBand(stream) {
+  const vt = stream.getVideoTracks()[0]; if (!vt) return { stream, stop: () => {} }
+  const st = vt.getSettings(); const W = st.width || 1920, H = st.height || 1080, fps = Math.min(60, st.frameRate || 60)
+  const { band } = VGClock.geometry(W)
+  const video = document.createElement('video'); video.muted = true; video.playsInline = true; video.srcObject = new MediaStream([vt]); video.play().catch(() => {})
+  const canvas = document.createElement('canvas'); canvas.width = W; canvas.height = H + band
+  const ctx = canvas.getContext('2d', { alpha: false, desynchronized: true })
+  let running = true
+  const draw = () => {
+    if (!running) return
+    if (video.readyState >= 2) ctx.drawImage(video, 0, 0, W, H)
+    VGClock.drawBand(ctx, W, H, { text: true })
+    if (video.requestVideoFrameCallback) video.requestVideoFrameCallback(draw); else requestAnimationFrame(draw)
+  }
+  draw()
+  VGClock.resync('/time'); const t = setInterval(() => VGClock.resync('/time'), 60000)
+  const out = canvas.captureStream(fps)
+  for (const a of stream.getAudioTracks()) out.addTrack(a)
+  return { stream: out, stop: () => { running = false; clearInterval(t); video.srcObject = null } }
+}
+
+// Transport. 'udp' is WebRTC as normal. 'tcp' keeps only MediaMTX's TCP candidates (webrtcLocalTCPAddress) in
+// the answer, so ICE can pair with nothing else and the media rides a TCP connection: no packet loss at all,
+// at the cost of latency spikes when the uplink hiccups. The recording cares about the former, not the latter.
+// MediaMTX requests retransmits (NACK, pion's ConfigureNack) either way, but a wifi burst that drops more
+// packets than its reorder window holds still reaches the recorder as a broken frame; TCP cannot.
+const LOSS_SWITCH_PCT = 0.3          // sustained loss above this (over LOSS_WINDOW_S) flips a publisher to TCP
+const LOSS_WINDOW_S = 20
+const LOSS_MIN_PACKETS = 1500        // don't judge a 2-second-old session
+
+function tcpOnly(sdp) {
+  const lines = sdp.split('\r\n')
+  const out = lines.filter(l => !(l.startsWith('a=candidate:') && / udp /i.test(l)))
+  if (!out.some(l => l.startsWith('a=candidate:'))) return null       // no TCP candidate offered: cannot do it
+  return out.join('\r\n')
+}
+
+export async function publish({ stream, path, user, pass, onState, transport = 'udp' }) {
   const pc = new RTCPeerConnection({ iceServers: [{ urls: 'stun:stun.l.google.com:19302' }] })
   for (const track of stream.getTracks()) {
     if (track.kind === 'video') { try { track.contentHint = 'motion' } catch {} }   // game footage: favour frame rate + temporal quality
@@ -70,26 +129,36 @@ export async function publish({ stream, path, user, pass, onState }) {
   const vs = pc.getSenders().find(s => s.track?.kind === 'video')
   if (vs) {
     const p = vs.getParameters(); p.encodings = p.encodings?.length ? p.encodings : [{}]
-    p.encodings[0].maxBitrate = MAX_KBPS * 1000
+    p.encodings[0].maxBitrate = MAX_KBPS * 1000       // a ceiling; congestion control may go far below it
     p.encodings[0].maxFramerate = 60
-    p.encodings[0].scaleResolutionDownBy = 1
+    // Under pressure Chrome must shed FRAME RATE, never resolution. A resolution change is a new SPS, and
-    p.degradationPreference = 'balanced'
+    // MediaMTX opens a new recording file on every parameter change — on a lossy uplink that turned one
+    // stream into a row of files the sweep cannot copy-join (different sizes). Frame-rate drops cost
+    // nothing on the recording side. (No scaleResolutionDownBy pin: that would forbid shedding entirely.)
+    p.degradationPreference = 'maintain-resolution'
     await vs.setParameters(p).catch(() => {})
   }
   pc.onconnectionstatechange = () => onState?.(pc.connectionState)
   const offer = await pc.createOffer()
-  await pc.setLocalDescription({ type: 'offer', sdp: mungeBitrates(offer.sdp) })
+  await pc.setLocalDescription({ type: 'offer', sdp: mungeBitrates(offer.sdp, transport === 'tcp' ? TCP_MIN_KBPS : MIN_KBPS) })
   await new Promise(res => { if (pc.iceGatheringState === 'complete') return res(); const t = setTimeout(res, 2000); pc.onicegatheringstatechange = () => { if (pc.iceGatheringState === 'complete') { clearTimeout(t); res() } } })
   // MediaMTX takes WebRTC credentials only in an Authorization header ("Bearer user:pass"). We send them as
   // X-Stream-Auth and Caddy rewrites the header on the MediaMTX hop, so Authentik's gate never sees it.
   const auth = { 'X-Stream-Auth': `Bearer ${user}:${pass}` }
   const r = await fetch(`/webrtc/${path}/whip`, { method: 'POST', headers: { 'Content-Type': 'application/sdp', ...auth }, body: pc.localDescription.sdp, credentials: 'same-origin' })
   if (!r.ok) { pc.close(); throw new Error(r.status === 401 || r.status === 403 ? 'stream key rejected' : r.status === 400 && /busy|already/i.test(await r.text().catch(() => '')) ? 'that stream name is already live (OBS running?)' : `publish failed (${r.status})`) }
-  await pc.setRemoteDescription({ type: 'answer', sdp: await r.text() })
+  let answer = await r.text()
   let session = r.headers.get('Location')                       // MediaMTX answers relative to its own root
+  if (transport === 'tcp') {
+    const only = tcpOnly(answer)
+    if (!only) { transport = 'udp' } else answer = only          // server offered no TCP: stay on UDP, say so in stats
+  }
+  await pc.setRemoteDescription({ type: 'answer', sdp: answer })
   if (session && session.startsWith('/') && !session.startsWith('/webrtc/')) session = '/webrtc' + session
   return {
     pc,
+    transport,
+    path, user, pass,
     stats: async () => {
       let kbps = 0, fps = 0, w = 0, h = 0
       for (const s of await pc.getStats()) {
@@ -97,6 +166,11 @@ export async function publish({ stream, path, user, pass, onState }) {
       }
       return { kbps, fps, w, h }
     },
+    // release: end this WHIP session but keep capturing — used to re-publish the same tracks over TCP
+    release: async () => {
+      try { if (session) await fetch(session.startsWith('http') ? '/webrtc' + new URL(session).pathname : session, { method: 'DELETE', headers: auth, credentials: 'same-origin' }) } catch {}
+      pc.close()
+    },
     stop: async () => {
       try { if (session) await fetch(session.startsWith('http') ? '/webrtc' + new URL(session).pathname : session, { method: 'DELETE', headers: auth, credentials: 'same-origin' }) } catch {}
       pc.close()
@@ -105,6 +179,18 @@ export async function publish({ stream, path, user, pass, onState }) {
   }
 }
 
+// Loss over a sliding window from the server's receiver reports. samples: [{t, lost, sent}] newest last.
+export function windowLoss(samples) {
+  const now = Date.now()
+  const win = samples.filter(s => now - s.t <= LOSS_WINDOW_S * 1000)
+  if (win.length < 2) return null
+  const a = win[0], b = win[win.length - 1]
+  const sent = b.sent - a.sent, lost = b.lost - a.lost
+  if (sent < LOSS_MIN_PACKETS) return null
+  return { pct: 100 * Math.max(0, lost) / sent, sent, lost }
+}
+export const shouldSwitchToTcp = l => !!l && l.pct >= LOSS_SWITCH_PCT
+
 export async function fetchStreamKey() {
   // a fresh 12 h publish token for the logged-in Discord user (member of streamers); nothing is stored
   const r = await fetch('/me/stream-token', { credentials: 'same-origin' })

live/src/layout.js

@@ -0,0 +1,30 @@
+import { create } from './tinyStore.js'
+
+// Layout mode. "auto" = untouched grid: streams joining/leaving re-grid everything. The first manual move, resize
+// or rename makes it "custom": the grid is left alone, leavers just close, joiners wait in the tray (with a ding)
+// until dragged in. "Arrange" re-grids everything and goes back to auto.
+export const layout = create({
+  custom: false,
+  pending: [],                               // streams live but not placed yet (custom mode)
+  markCustom() { return this.custom ? null : { custom: true } },
+  reset() { return { custom: false, pending: [] } },
+  addPending(s) { return this.pending.some(p => p.id === s.id) ? null : { pending: [...this.pending, s] } },
+  dropPending(id) { return this.pending.some(p => p.id === id) ? { pending: this.pending.filter(p => p.id !== id) } : null },
+  keepPending(ids) { const pending = this.pending.filter(p => ids.has(p.id)); return pending.length === this.pending.length ? null : { pending } },
+})
+
+// short two-note chime (WebAudio, no asset). Browsers need one prior click on the page to allow sound.
+let ctx
+export function ding() {
+  try {
+    ctx = ctx || new (window.AudioContext || window.webkitAudioContext)()
+    if (ctx.state === 'suspended') ctx.resume().catch(() => {})
+    const t0 = ctx.currentTime
+    for (const [i, f] of [[0, 880], [0.16, 1320]]) {
+      const o = ctx.createOscillator(), g = ctx.createGain()
+      o.type = 'sine'; o.frequency.value = f
+      g.gain.setValueAtTime(0, t0 + i); g.gain.linearRampToValueAtTime(0.25, t0 + i + 0.02); g.gain.exponentialRampToValueAtTime(0.001, t0 + i + 0.35)
+      o.connect(g).connect(ctx.destination); o.start(t0 + i); o.stop(t0 + i + 0.4)
+    }
+  } catch {}
+}

live/src/names.js

@@ -0,0 +1,25 @@
+import { create } from './tinyStore.js'
+
+// Viewer-side labels. Stream names persist in this browser (localStorage, keyed by the MediaMTX path);
+// group names live for the session (dockview group ids are regenerated on reload).
+const KEY = 'vg.names'
+let saved = {}
+try { saved = JSON.parse(localStorage.getItem(KEY) || '{}') || {} } catch {}
+
+export const names = create({
+  streams: saved,
+  groups: {},
+  rename(id, value) {
+    const streams = { ...this.streams }
+    if (value) streams[id] = value; else delete streams[id]
+    try { localStorage.setItem(KEY, JSON.stringify(streams)) } catch {}
+    return { streams }
+  },
+  renameGroup(id, value) {
+    const groups = { ...this.groups }
+    if (value) groups[id] = value; else delete groups[id]
+    return { groups }
+  },
+})
+
+export const streamLabel = (id, params) => names.get().streams[id] || `${params.user} · ${params.title}`

live/src/styles.css

@@ -16,7 +16,12 @@ html, body, #root { height:100%; margin:0; background:var(--bg); color:var(--fg)
   --dv-separator-border:#2a2a2a; --dv-group-view-background-color:var(--panel); --dv-drag-over-background-color:rgba(61,139,253,.2);
   --dv-drag-over-border-color:var(--acc); --dv-tabs-container-scrollbar-color:#444; --dv-icon-hover-background-color:#333; }
 .player { position:relative; width:100%; height:100%; background:#000; display:flex; flex-direction:column; }
-.player video { flex:1; min-height:0; width:100%; height:100%; object-fit:contain; background:#000; }
+.player .vwrap { flex:1; min-height:0; position:relative; display:flex; align-items:center; justify-content:center; background:#000; overflow:hidden; }
+/* .frame = game-aspect box (sized by Player.jsx); the video spans its width, so the timecode band rows overflow the bottom and are clipped */
+.player .frame { position:relative; width:100%; height:100%; overflow:hidden; background:#000; }
+.player .frame video { position:absolute; top:0; left:0; width:100%; height:auto; display:block; background:#000; }
+.player .frame[style] video { height:auto; }
+.player .frame:not([style]) video { position:static; width:100%; height:100%; object-fit:contain; }   /* before the size is known: plain contain */
 .player .bar { display:flex; align-items:center; gap:10px; padding:6px 10px; background:#181818; border-top:1px solid #2a2a2a; font-size:13px; }
 .player .live { background:var(--live); color:#fff; font-weight:700; font-size:11px; padding:2px 6px; border-radius:4px; letter-spacing:.5px; }
 .player .who { font-weight:600; }
@@ -58,3 +63,21 @@ html, body, #root { height:100%; margin:0; background:var(--bg); color:var(--fg)
 .golive-stats .rec { background:#f1c40f; color:#000; font-weight:700; font-size:11px; padding:2px 6px; border-radius:4px; }
 .topbar a.signout { color:var(--mut); font-size:13px; text-decoration:none; border:1px solid #3a3a3a; border-radius:6px; padding:5px 10px; }
 .topbar a.signout:hover { color:var(--fg); border-color:#666; }
+
+/* renameable tabs + group names */
+.vtab { display:flex; align-items:center; height:100%; padding:0 10px; min-width:0; }
+.vtab-text { overflow:hidden; text-overflow:ellipsis; white-space:nowrap; max-width:260px; }
+.vtab input, .grouplabel input { background:#111; color:var(--fg); border:1px solid var(--acc); border-radius:4px; font:inherit; font-size:12px; padding:1px 6px; width:170px; outline:none; }
+.grouplabel { display:flex; align-items:center; height:100%; padding:0 10px; font-size:12px; white-space:nowrap; border-right:1px solid #2a2a2a; cursor:default; }
+.grouplabel .gname { color:var(--acc); font-weight:700; }
+.grouplabel .hint { color:#555; }
+.groupactions { display:flex; align-items:center; height:100%; padding:0 6px; }
+.gbtn { background:transparent; color:var(--mut); border:0; cursor:pointer; font-size:14px; padding:0 6px; line-height:1; }
+.gbtn:hover { color:var(--fg); }
+
+/* tray: new streams waiting to be placed (custom layouts) */
+.tray { display:flex; gap:8px; align-items:center; padding:4px 10px; background:#1d1a10; border-bottom:1px solid #3a3320; font-size:12px; }
+.chip { display:flex; align-items:center; gap:6px; padding:3px 10px; border:1px solid #6b5a1e; border-radius:14px; background:#262012; color:var(--fg); cursor:grab; user-select:none; }
+.chip:active { cursor:grabbing; }
+.blink { color:#f1c40f; animation: vgblink 1s steps(2, start) infinite; }
+@keyframes vgblink { to { visibility:hidden; } }

live/src/vgclock.js

@@ -0,0 +1,57 @@
+// Vanguard timecode strip — browser side. Shared by the OBS browser source (/dock/clock/) and the live portal.
+// Format: see bin/vgstrip.py. cell = W/160; band = round_even(cell); 64 cells from x = cell:
+//   [1,0,1,0] · 48-bit server ms (MSB first) · CRC-8 (poly 0x07) · [0,1,0,1]; 1 = white, 0 = black.
+(function (global) {
+  function crc8(bytes) { let c = 0; for (const b of bytes) { c ^= b; for (let i = 0; i < 8; i++) c = (c & 0x80) ? ((c << 1) ^ 0x07) & 0xff : (c << 1) & 0xff; } return c; }
+  function bitsFor(ms) {
+    const bytes = []; let v = BigInt(ms); for (let i = 5; i >= 0; i--) { bytes[i] = Number(v & 0xffn); v >>= 8n; }
+    const bits = [1, 0, 1, 0];
+    for (const b of bytes) for (let i = 7; i >= 0; i--) bits.push((b >> i) & 1);
+    const c = crc8(bytes); for (let i = 7; i >= 0; i--) bits.push((c >> i) & 1);
+    return bits.concat([0, 1, 0, 1]);
+  }
+  function geometry(w) { const cell = w / 160; let band = Math.round(cell); band += band % 2; return { cell, band }; }
+
+  // ---- server clock offset: offset = serverNow - localNow, from the lowest-RTT of several /time samples
+  const sync = { offset: 0, rtt: null, ok: false, samples: 0 };
+  async function sampleOnce(url) {
+    const t0 = performance.now(), wall0 = Date.now();
+    const r = await fetch(url, { cache: 'no-store', credentials: 'same-origin' }); const j = await r.json();
+    const t1 = performance.now(); const rtt = t1 - t0;
+    const localMid = wall0 + rtt / 2;
+    return { offset: j.now_ms - localMid, rtt };
+  }
+  async function resync(url, n = 7) {
+    let best = null;
+    for (let i = 0; i < n; i++) { try { const s = await sampleOnce(url); if (!best || s.rtt < best.rtt) best = s; } catch (e) {} }
+    if (best) { sync.offset = best.offset; sync.rtt = best.rtt; sync.ok = true; sync.samples++; }
+    return sync;
+  }
+  function serverNow() { return Date.now() + sync.offset; }
+
+  // ---- draw the band into a 2D context: strip + readable clock. (w = canvas width, y0 = band top)
+  function drawBand(ctx, w, y0, opts = {}) {
+    const { cell, band } = geometry(w);
+    const ms = Math.round(serverNow());
+    ctx.fillStyle = '#000'; ctx.fillRect(0, y0, w, band);
+    const bits = bitsFor(ms);
+    const pad = Math.max(1, Math.floor(cell * 0.12));
+    for (let i = 0; i < bits.length; i++) {
+      if (!bits[i]) continue;
+      const x0 = Math.round(cell * (1 + i)), x1 = Math.round(cell * (2 + i));
+      ctx.fillStyle = '#fff'; ctx.fillRect(x0 + pad, y0 + pad, x1 - x0 - 2 * pad, band - 2 * pad);
+    }
+    if (opts.text !== false) {
+      const d = new Date(ms);
+      const tz = opts.tz || 'America/New_York';
+      const hms = d.toLocaleTimeString('en-US', { hour12: false, timeZone: tz });
+      const txt = `${hms}.${String(ms % 1000).padStart(3, '0')} ${opts.label || 'SRV'}${sync.ok ? '' : ' (unsynced)'}`;
+      ctx.fillStyle = sync.ok ? '#9f9' : '#f66'; ctx.font = `${Math.max(8, band - 2 * pad)}px ui-monospace, monospace`; ctx.textBaseline = 'middle';
+      ctx.fillText(txt, Math.round(cell * (CELLS + 2.5)), y0 + band / 2);
+    }
+    return ms;
+  }
+  const CELLS = 64;
+  global.VGClock = { crc8, bitsFor, geometry, resync, serverNow, drawBand, sync, CELLS };
+})(typeof window !== "undefined" ? window : globalThis);
+export default (typeof window !== "undefined" ? window : globalThis).VGClock;

mediamtx.example.yml

@@ -51,7 +51,7 @@ authInternalUsers:
 # ---- recording --------------------------------------------------------
 pathDefaults:
   record: yes
-  recordPath: /recordings/%path/.live/%Y-%m-%d_%H-%M-%S   # hidden while recording; the sweep publishes the finished file one level up
+  recordPath: /recordings/%path/.live/%Y-%m-%d_%H-%M-%S-%f   # hidden while recording; microseconds = server arrival time (timecode fallback)
   recordFormat: fmp4
   recordPartDuration: 1s
   recordSegmentDuration: 12h      # one file per stream unless it runs >12h

observability/filebeat.yml

@@ -0,0 +1,89 @@
+# Filebeat: container logs + our own systemd units -> Elasticsearch.
+# Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service /
+# vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana
+# needs no parsing. See LOGGING.md for what each service's lines look like.
+#
+# NOTE (9.x): the old "container" input is GONE, not merely deprecated — Filebeat 9 refuses the config and
+# logs "Container input is deprecated. Use Filestream input with its container parser instead", leaving the
+# process up with zero harvesters. Container logs are read by filestream + the container parser, and every
+# filestream input needs its own stable id.
+
+filebeat.autodiscover:
+  providers:
+    - type: docker
+      hints.enabled: false
+      templates:
+        - condition: ~                      # every container
+          config:
+            - type: filestream
+              id: container-${data.docker.container.id}
+              paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
+              parsers:
+                - container: ~
+              prospector.scanner.symlinks: true
+              # Stamp the id here rather than letting add_docker_metadata guess it from the path.
+              processors:
+                - add_fields:
+                    target: container
+                    fields:
+                      id: "${data.docker.container.id}"
+                      name: "${data.docker.container.name}"
+                      image.name: "${data.docker.container.image}"
+
+filebeat.inputs:
+  - type: journald
+    id: vanguard-units
+    include_matches:
+      match:
+        - "_SYSTEMD_UNIT=dumpd.service"
+        - "_SYSTEMD_UNIT=normalize-sweep.service"
+        - "_SYSTEMD_UNIT=trash-purge.service"
+        - "_SYSTEMD_UNIT=vgdeband.service"
+        - "_SYSTEMD_UNIT=vanguard-deadman.service"
+
+processors:
+  - add_docker_metadata:
+      match_fields: ["container.id"]        # the id the autodiscover template stamped above
+  - add_host_metadata: ~
+  # promote our labels to real fields so Kibana filters on service.name and vanguard.category
+  - copy_fields:
+      fields:
+        - from: container.labels.vanguard_service
+          to: service.name
+        - from: container.labels.vanguard_category
+          to: vanguard.category
+      fail_on_error: false
+      ignore_missing: true
+  # The Observability overview's "Logs rate" chart groups by event.dataset, so anything without one lands in
+  # a single "unknown" bucket and the chart says nothing. Give every line a dataset: the container's
+  # vanguard.service label, or the systemd unit for journald lines. copy_fields does not overwrite a target
+  # that already exists, so the first rule that matches wins and the other is skipped.
+  - copy_fields:
+      fields:
+        - from: container.labels.vanguard_service
+          to: event.dataset
+        - from: systemd.unit
+          to: event.dataset
+      fail_on_error: false
+      ignore_missing: true
+  - drop_fields:
+      fields: ["agent.ephemeral_id", "ecs.version", "log.offset"]
+      ignore_missing: true
+
+output.elasticsearch:
+  hosts: ["http://elasticsearch:9200"]
+  # Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
+  # come from the container environment, which compose fills from .env — no secret is committed here.
+  username: "${ES_BEATS_USER}"
+  password: "${ES_BEATS_PASSWORD}"
+
+setup.kibana:                               # no credentials: Kibana's anonymous provider signs these in
+  host: "http://kibana:5601"
+setup.ilm.enabled: true                     # retention: set the delete phase once in Kibana (README 5h)
+# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
+# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
+setup.template.overwrite: true
+setup.template.settings:
+  index.number_of_replicas: 0
+
+logging.level: info

observability/kibana-alarms.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""The real alarm rules. Idempotent: creates what is missing, leaves what exists.
+
+Every rule is an Elasticsearch query rule that counts matching documents in a window and writes one document
+into the alert bus through the "Vanguard alert index" connector, which bin/vanguard-alerts.sh delivers to
+Discord. Run observability/kibana-setup.py first — it creates that connector.
+
+FIRING CADENCE: notify_when=onThrottleInterval with throttle=12h, so an alarm that stays true writes again
+every 12 hours and stops when the condition clears. The delivery script's own 15-minute per-rule suppression
+sits well inside that and never interferes.
+
+WHAT IS DELIBERATELY NOT HERE
+Elasticsearch, Kibana and Beat liveness, and the disks, are bin/vanguard-deadman.sh's job. Nothing that
+depends on Kibana being alive can report that Kibana is dead, so those checks stay outside the stack and
+duplicating them here would be worse than useless: they would go quiet in exactly the case that matters.
+
+  pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
+  KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
+          -H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
+        | python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
+  KEY="$KEY" python3 observability/kibana-alarms.py
+
+To change a rule, edit it in Kibana or delete it there and re-run this. Thresholds here are the defaults:
+  FREE_PCT        15   alarm when a real filesystem or memory has less than this percent free
+  STUCK_SECONDS  3600  alarm when a finished recording has waited this long to be normalised
+  THROTTLE        12h  how often an unresolved alarm repeats
+"""
+import json, os, urllib.error, urllib.request
+
+KIBANA = "http://127.0.0.1:5601"
+HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
+           "Authorization": "ApiKey " + os.environ["KEY"]}
+CONNECTOR_NAME = "Vanguard alert index"
+
+FREE_PCT = float(os.environ.get("FREE_PCT", "15"))
+STUCK_SECONDS = int(os.environ.get("STUCK_SECONDS", "3600"))
+THROTTLE = os.environ.get("THROTTLE", "12h")
+USED_MAX = round(1 - FREE_PCT / 100, 4)          # 15% free -> used.pct >= 0.85
+
+# Filesystems that are not disks. Without this, every tmpfs and overlay mount reports ~100% used and the
+# disk alarm would fire permanently on something that cannot fill up.
+PSEUDO_FS = ["tmpfs", "devtmpfs", "overlay", "squashfs", "ramfs", "autofs", "nsfs", "tracefs", "debugfs",
+             "configfs", "mqueue", "hugetlbfs", "pstore", "bpf", "devpts", "proc", "sysfs", "cgroup",
+             "cgroup2", "fusectl", "securityfs", "efivarfs", "binfmt_misc", "fuse.lxcfs", "iso9660"]
+
+def rule(name, level, why, query, *, index="metricbeat-*", window=10, comparator=">", threshold=0):
+    return {"name": name, "level": level, "why": why, "query": query,
+            "index": index, "window": window, "comparator": comparator, "threshold": threshold}
+
+ALARMS = [
+    rule("Disk free below %g%%" % FREE_PCT, "error",
+         "A filesystem is nearly full. Elasticsearch turns its indices READ-ONLY at the flood watermark and "
+         "does not undo that by itself, and recordings stop being published.",
+         {"query": {"bool": {
+             "filter": [{"range": {"system.filesystem.used.pct": {"gte": USED_MAX}}}],
+             "must_not": [{"terms": {"system.filesystem.type": PSEUDO_FS}}]}}}),
+
+    rule("Memory free below %g%%" % FREE_PCT, "warn",
+         "Memory is nearly exhausted. Elasticsearch has a pinned 2 GB heap, so the pressure is usually "
+         "ffmpeg or Jellyfin, and the kernel will start killing processes rather than queueing.",
+         {"query": {"range": {"system.memory.actual.used.pct": {"gte": USED_MAX}}}}),
+
+    rule("Discord bot not running", "error",
+         "No metrics from the discord-bot container for 5 minutes: it is stopped, removed, or restarting too "
+         "fast to be sampled. Nothing it does is happening.",
+         {"query": {"term": {"container.name": "discord-bot"}}},
+         window=5, comparator="<", threshold=1),
+
+    rule("Discord bot unhealthy", "error",
+         "The bot is running but reports itself degraded: its /health says the Discord gateway is "
+         "disconnected or the database does not answer. It is up and doing nothing.",
+         {"query": {"bool": {"filter": [{"term": {"container.name": "discord-bot"}},
+                                        {"term": {"docker.healthcheck.status": "unhealthy"}}]}}},
+         window=5),
+
+    rule("Recordings stuck waiting to be normalised", "warn",
+         "A finished recording has waited over %d minutes in .live/. It is not in Jellyfin, not in copyparty "
+         "and not announced, so to everyone else it does not exist." % (STUCK_SECONDS // 60),
+         {"query": {"range": {"prometheus.vanguard_media_pending_oldest_seconds.value": {"gt": STUCK_SECONDS}}}}),
+
+    rule("Normalise failures parked", "warn",
+         "One or more files have a normalize-failed log, so the sweep skips them forever until it is deleted. "
+         "The log now contains ffmpeg's stderr, which says why.",
+         {"query": {"range": {"prometheus.vanguard_media_failed_files.value": {"gt": 0}}}}),
+]
+
+def req(method, path, body=None):
+    request = urllib.request.Request(KIBANA + path, method=method,
+                                     data=None if body is None else json.dumps(body).encode())
+    for key, value in HEADERS.items():
+        request.add_header(key, value)
+    try:
+        with urllib.request.urlopen(request, timeout=30) as response:
+            raw = response.read()
+            return json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as error:
+        print(f"  {method} {path} -> HTTP {error.code}")
+        print("  " + error.read().decode(errors="replace")[:600])
+        raise SystemExit(1)
+
+connectors = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+if CONNECTOR_NAME not in connectors:
+    raise SystemExit(f"connector {CONNECTOR_NAME!r} is missing — run observability/kibana-setup.py first")
+connector_id = connectors[CONNECTOR_NAME]
+
+present = {r["name"] for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])}
+created = skipped = 0
+for alarm in ALARMS:
+    if alarm["name"] in present:
+        print("already present:", alarm["name"])
+        skipped += 1
+        continue
+    made = req("POST", "/api/alerting/rule", {
+        "name": alarm["name"],
+        "rule_type_id": ".es-query",
+        "consumer": "stackAlerts",
+        "enabled": True,
+        "schedule": {"interval": "1m"},
+        "params": {
+            "searchType": "esQuery",
+            "index": [alarm["index"]],
+            "timeField": "@timestamp",
+            "esQuery": json.dumps(alarm["query"]),
+            "size": 1,
+            "aggType": "count",
+            "groupBy": "all",
+            "thresholdComparator": alarm["comparator"],
+            "threshold": [alarm["threshold"]],
+            "timeWindowSize": alarm["window"],
+            "timeWindowUnit": "m",
+            "excludeHitsFromPreviousRun": False,
+        },
+        "actions": [{
+            "group": "query matched",
+            "id": connector_id,
+            "params": {"documents": [{
+                "kind": "alarm",
+                "rule": "{{rule.name}}",
+                "level": alarm["level"],
+                "message": alarm["why"] + " [{{context.message}}]",
+            }]},
+            # every 12 h for as long as it stays true, then silence when it clears
+            "frequency": {"summary": False, "notify_when": "onThrottleInterval", "throttle": THROTTLE},
+        }],
+    })
+    print("created:", made["name"], "|", alarm["level"], "| repeats every", THROTTLE)
+    created += 1
+
+print(f"\n{created} created, {skipped} already there. Disk/memory threshold {FREE_PCT:g}% free; "
+      f"stuck after {STUCK_SECONDS}s.")

observability/kibana-setup.py

@@ -0,0 +1,292 @@
+#!/usr/bin/env python3
+"""Kibana first-run state. Idempotent; safe to re-run. configure.py --apply runs this for you.
+
+  connector  "Vanguard alert index"        — the .index connector, writing into the vanguard-alerts index
+  rule       "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc
+  data view  "Vanguard logs + metrics"     — filebeat-*,metricbeat-* so Discover and Lens have one source
+  landing    "Vanguard ops" dashboard      — our own links, and defaultRoute so diag.BASE opens on it
+
+Run one step on its own by naming it:  kibana-setup.py landing   (also: connector, rule, dataview, all)
+
+WHY AN INDEX AND NOT A WEBHOOK
+On a basic license the only connector types Kibana allows are .index and .server-log; every connector that
+leaves the stack (webhook, Slack, email, PagerDuty…) is Gold or above and returns
+    403 Action type .webhook is disabled because your basic license does not support it
+So Kibana cannot talk to Discord. Instead every rule writes a document into ONE index, which is the alert
+bus, and bin/vanguard-alerts.sh (a host timer) delivers from there to Discord and later to the bot. That is
+strictly simpler than the webhook route: no Docker-bridge hop, no dumpd endpoint, and alerts are queryable
+and replayable after the fact because they are just documents.
+
+THE HEARTBEAT
+bin/vanguard-deadman.sh alarms when no heartbeat document has arrived recently. It is the only check that
+catches the ALERTING pipeline itself dying — if Kibana, its rule engine or the connector stops, no ordinary
+alert can report it, because the thing that would report it is what broke. The rule queries metricbeat-*,
+which always has recent documents, so it doubles as a Metricbeat liveness check.
+
+Run it with an Elasticsearch API key: Basic is not an accepted scheme for this Kibana (the anonymous provider
+is the only one enabled, so only ApiKey and Bearer are taken).
+
+  pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
+  KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
+          -H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
+        | python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
+  KEY="$KEY" python3 observability/kibana-setup.py
+
+Order matters: run this AFTER security is on. A rule created while security was off holds no API key and
+stops working the moment it is enabled, until it is disabled and re-enabled.
+
+ADDING A REAL ALARM: author it in Kibana as usual, point its action at the "Vanguard alert index" connector,
+and give the document a message plus a level (warn|error). Do NOT set kind=heartbeat — that value is how the
+delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
+for real alarms; only the heartbeat wants to fire every interval.
+"""
+import json, os, time, urllib.error, urllib.request
+from pathlib import Path          # env_hosts() reads the stack's .env
+
+import sys
+
+STEPS = ("connector", "rule", "dataview", "landing")
+WANTED = sys.argv[1] if len(sys.argv) > 1 else "all"
+if WANTED not in STEPS + ("all",):
+    raise SystemExit(f"unknown step {WANTED!r}; pick one of: {', '.join(STEPS)}, all")
+def wanted(step):
+    return WANTED in ("all", step)
+
+KIBANA = "http://127.0.0.1:5601"
+ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts")
+HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
+           "Authorization": "ApiKey " + os.environ["KEY"],
+           # Kibana 9 ships with server.restrictInternalApis on. /api/kibana/settings (defaultRoute) is an
+           # internal API and refuses without this header: "exists but is not available with the current
+           # configuration". Sending it on every call is harmless; loosening the restriction is not.
+           "x-elastic-internal-origin": "Kibana"}
+
+def wait_for_kibana(seconds=300):
+    """Kibana has to be *available*, not merely listening.
+
+    configure.py --es-security restarts Kibana as its last step, so anything that calls the Kibana API
+    immediately afterwards races its boot and gets a connection reset. Kibana also answers /api/status while
+    still starting, so the status level is what has to be checked, not the port.
+    """
+    deadline = time.time() + seconds
+    announced = False
+    while time.time() < deadline:
+        try:
+            with urllib.request.urlopen(KIBANA + "/api/status", timeout=10) as response:
+                level = json.load(response).get("status", {}).get("overall", {}).get("level")
+            if level == "available":
+                print("kibana: available")
+                return
+            state = level or "no level"
+        except Exception as error:
+            state = type(error).__name__
+        if not announced:
+            print(f"kibana: waiting for it to become available ({state})…")
+            announced = True
+        time.sleep(5)
+    raise SystemExit(f"kibana did not become available within {seconds}s — nothing was created")
+
+def req(method, path, body=None):
+    request = urllib.request.Request(KIBANA + path, method=method,
+                                     data=None if body is None else json.dumps(body).encode())
+    for key, value in HEADERS.items():
+        request.add_header(key, value)
+    try:
+        with urllib.request.urlopen(request, timeout=30) as response:
+            raw = response.read()
+            return json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as error:
+        print(f"  {method} {path} -> HTTP {error.code}")
+        print("  " + error.read().decode(errors="replace")[:600])
+        raise SystemExit(1)
+
+wait_for_kibana()
+
+# 1. the alert bus connector
+CONNECTOR_NAME = "Vanguard alert index"
+connector_id = None
+if wanted("connector") or wanted("rule"):
+  existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+  if CONNECTOR_NAME in existing:
+    connector_id = existing[CONNECTOR_NAME]
+    print("connector already present:", connector_id)
+  elif not wanted("connector"):
+    raise SystemExit(f"connector {CONNECTOR_NAME!r} does not exist yet — run the connector step first")
+  else:
+      connector_id = req("POST", "/api/actions/connector", {
+        "name": CONNECTOR_NAME,
+        "connector_type_id": ".index",
+        # executionTimeField stamps every document, so a rule body never has to get the timestamp right.
+        "config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"},
+        "secrets": {}})["id"]
+      print("connector created:", connector_id, "->", ALERT_INDEX)
+
+# 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked
+#    harmless and meant the data view was never created on any system where the rule already existed, which
+#    is every re-run.
+RULE_NAME = "Vanguard alerting heartbeat"
+existing_rule = None if not wanted("rule") else next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
+                      if r["name"] == RULE_NAME), None)
+if not wanted("rule"):
+    pass
+elif existing_rule:
+    print("rule already present:", existing_rule["id"],
+          "enabled" if existing_rule.get("enabled") else "DISABLED")
+else:
+    rule = req("POST", "/api/alerting/rule", {
+        "name": RULE_NAME,
+        "rule_type_id": ".es-query",
+        "consumer": "stackAlerts",
+        "enabled": True,
+        "schedule": {"interval": "1m"},
+        "params": {
+            "searchType": "esQuery",
+            "index": ["metricbeat-*"],
+            "timeField": "@timestamp",
+            "esQuery": json.dumps({"query": {"match_all": {}}}),
+            "size": 1,
+            "aggType": "count",
+            "groupBy": "all",
+            "thresholdComparator": ">",
+            "threshold": [0],
+            "timeWindowSize": 5,
+            "timeWindowUnit": "m",
+            # re-evaluate the whole window every run, so the rule stays active instead of going quiet once
+            # it has already seen those documents
+            "excludeHitsFromPreviousRun": False,
+        },
+        # onActiveAlert writes on EVERY check. Kibana's default only acts when a rule CHANGES status, which
+        # would produce one heartbeat and then silence — the deadman would read that as a dead pipeline.
+        "actions": [{
+            "group": "query matched",
+            "id": connector_id,
+            "params": {"documents": [{
+                "kind": "heartbeat",             # the delivery script drops these; never reach Discord
+                "rule": "{{rule.name}}",
+                "level": "info",
+                "message": "kibana rule engine and the index connector are alive",
+            }]},
+            "frequency": {"summary": False, "notify_when": "onActiveAlert", "throttle": None},
+        }],
+    })
+    print("rule created:", rule["id"], "| enabled:", rule.get("enabled"))
+
+# 3. the data view. Without one, Discover and Lens have nothing to point at and every panel starts with a
+#    detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and
+#    metrics, which is the whole reason they share a cluster.
+VIEW_TITLE = "filebeat-*,metricbeat-*"
+views = req("GET", "/api/data_views").get("data_view", []) if wanted("dataview") else []
+if not wanted("dataview"):
+    pass
+elif any(v.get("title") == VIEW_TITLE for v in views):
+    print("data view already present:", VIEW_TITLE)
+else:
+    view = req("POST", "/api/data_views/data_view", {
+        "data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}})
+    print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE)
+
+
+# 4. the landing page. Kibana stays at the root of the diag host; this just makes arriving there open OUR
+#    dashboard instead of Kibana's home. A markdown panel holds the links, and `defaultRoute` is the
+#    advanced setting that does the redirect. That is the whole trick, and it avoids moving Kibana under a
+#    /kibana prefix — which would have meant touching both Beats' setup.kibana, this script, the alarm
+#    script and the deadman's status check, with a silent crash-loop waiting for whichever one was missed.
+DASHBOARD_ID = "vanguard-ops"
+
+def env_hosts():
+    """Hostnames out of the stack's .env, so the links are real rather than examples."""
+    values = {}
+    for candidate in (Path(".env"), Path("/home/debian/vanguard-stream/.env")):
+        if candidate.is_file():
+            for line in candidate.read_text().splitlines():
+                if "=" in line and not line.startswith("#"):
+                    key, value = line.split("=", 1)
+                    values[key.strip()] = value.strip()
+            break
+    # .env carries no AUTH_HOST (it lives in the Caddyfile); every host here is <role>.<domain>, so derive it
+    if "AUTH_HOST" not in values and values.get("MAIN_HOST", "").count(".") >= 2:
+        values["AUTH_HOST"] = "auth." + values["MAIN_HOST"].split(".", 1)[1]
+    return values
+
+def landing_markdown(hosts):
+    def url(key, path="/"):
+        host = hosts.get(key)
+        return f"https://{host}{path}" if host else None
+    rows = [
+        ("Live streams", "/streaming#live", "who is publishing, loss on the way in, why files are splitting"),
+        ("Stuck recordings", "/streaming#stuck", "finished but never published — why, and retry or bin"),
+        ("Dump a night", "/streaming#dump", "a streamer's or everyone's night to trash; bin all unprocessed"),
+        ("Disk", "/disk", "space by streamer and clipper per month, trash, logs, system"),
+        ("Live portal", url("LIVE_HOST"), "watch every active stream"),
+        ("OBS dock", url("LIVE_HOST", "/dock/"), "go live, titles, dump tonight"),
+        ("Files", url("FILES_HOST"), "browse, download, upload finals"),
+        ("Jellyfin", url("WATCH_HOST"), "recordings by streamer and season"),
+        ("Gateway", url("MAIN_HOST"), "role-aware links and how-tos"),
+        ("Gitea", url("GIT_HOST"), "code and CI"),
+        ("Bot", url("BOT_HOST"), "the Discord bot's own pages"),
+        ("Syncthing", url("SYNC_HOST"), "clip intake, device pairing"),
+        ("Authentik", url("AUTH_HOST", "/if/admin/"), "users, groups, policies"),
+        ("Admin sign-in", url("AUTH_HOST", "/if/flow/default-authentication-flow/?admin=1"), "password card for akadmin — members are forwarded straight to Discord"),
+    ]
+    lines = ["### Vanguard", "", "| | |", "|---|---|"]
+    for name, link, why in rows:
+        if link:
+            lines.append(f"| **[{name}]({link})** | {why} |")
+    lines += [
+        "", "### In here", "",
+        "| | |", "|---|---|",
+        "| **[Discover](/app/discover)** | search every log line; filter on `service.name` or `vanguard.category` |",
+        "| **[Dashboards](/app/dashboards)** | the Metricbeat system and Docker dashboards |",
+        "| **[Stack Rules](/app/management/insightsAndAlerting/triggersActions/rules)** | the six alarms, and whether each is firing |",
+        "| **[Index Management](/app/management/data/index_management/indices)** | index sizes and ILM |",
+        "",
+        "_Alarms post to Discord through the `vanguard-alerts` index, not from Kibana directly: a basic "
+        "licence has no webhook connector. The host deadman watches that whole path from outside._",
+    ]
+    return "\n".join(lines)
+
+if wanted("landing"):
+    hosts = env_hosts()
+    panel = {
+        "type": "visualization",
+        "panelIndex": "1",
+        "gridData": {"x": 0, "y": 0, "w": 24, "h": 18, "i": "1"},
+        "embeddableConfig": {
+            "savedVis": {
+                "id": "", "title": "", "description": "", "type": "markdown",
+                "params": {"fontSize": 12, "openLinksInNewTab": False,
+                           "markdown": landing_markdown(hosts)},
+                "uiState": {},
+                "data": {"aggs": [], "searchSource": {"query": {"query": "", "language": "kuery"},
+                                                      "filter": []}},
+            },
+            "enhancements": {},
+        },
+    }
+    saved = req("POST", f"/api/saved_objects/dashboard/{DASHBOARD_ID}?overwrite=true", {
+        "attributes": {
+            "title": "Vanguard ops",
+            "description": "Landing page for the diagnostics host. Edit freely: add metric panels beside the links.",
+            "panelsJSON": json.dumps([panel]),
+            "optionsJSON": json.dumps({"hidePanelTitles": True, "useMargins": True,
+                                       "syncColors": False, "syncCursor": True, "syncTooltips": False}),
+            "timeRestore": False,
+            "kibanaSavedObjectMeta": {
+                "searchSourceJSON": json.dumps({"query": {"query": "", "language": "kuery"}, "filter": []})},
+        }})
+    # Prove it rather than assume it: read the dashboard back and count the panels Kibana actually stored.
+    stored = req("GET", f"/api/saved_objects/dashboard/{DASHBOARD_ID}")
+    panels = json.loads(stored.get("attributes", {}).get("panelsJSON", "[]"))
+    links = sum(1 for line in landing_markdown(hosts).splitlines() if line.startswith("| **["))
+    print(f"dashboard: {stored.get('id')} | title {stored.get('attributes', {}).get('title')!r} | "
+          f"{len(panels)} panel(s) stored | {links} links in the markdown")
+    if not panels:
+        raise SystemExit("  the dashboard saved but with no panels — Kibana rejected the panel shape")
+    print("open:", f"/app/dashboards#/view/{DASHBOARD_ID}")
+
+    route = f"/app/dashboards#/view/{DASHBOARD_ID}"
+    req("POST", "/api/kibana/settings", {"changes": {"defaultRoute": route}})
+    now = req("GET", "/api/kibana/settings").get("settings", {}).get("defaultRoute", {}).get("userValue")
+    print("defaultRoute:", now or "(not set!)")
+    if now != route:
+        raise SystemExit("  Kibana did not keep that defaultRoute — set it by hand in Advanced Settings")

observability/kibana.yml

@@ -0,0 +1,41 @@
+# Kibana. Mounted read-only; every secret arrives as an environment variable that configure.py generates
+# into .env, so this file is committed and contains none.
+#
+# WHY SECURITY IS ON, AND WHY NOBODY SEES A LOGIN
+# Elasticsearch runs with xpack.security.enabled=true because Kibana's Observability apps ask Elasticsearch
+# whether the current user holds a privilege. With security off that API does not exist, and the
+# Infrastructure / Metrics app fails on every page view with
+#   no handler found for uri [/_security/user/_has_privileges]
+# Turning it on would normally put a password prompt in front of Kibana, which this stack does not allow:
+# Discord is the only login anywhere. So the anonymous provider below signs every visitor in automatically as
+# one Elasticsearch user. The real gate is unchanged and is still the only one: Caddy's Authentik
+# forward-auth plus the DIAG_REQUIRE_GROUP check on the diag host.
+#
+# This is not a permission tier. Everyone who clears that gate is the same Elasticsearch user with the same
+# rights, exactly as before. The difference is that the identity now exists, which is what the apps need and
+# what a read-only tier would later be built on (see STATUS.md 9b: add the http provider with the apikey
+# scheme and have Caddy inject a key chosen by Authentik group).
+
+server.host: "0.0.0.0"
+server.publicBaseUrl: "https://${DIAG_HOST}"
+# NO server.basePath. Kibana IS the root of the diag host, and the key must be ABSENT rather than empty:
+# an empty string fails validation with "must start with a slash, don't end with one" and Kibana refuses to
+# boot. It survived as an empty SERVER_BASEPATH env var only because the image drops empty values.
+telemetry.optIn: false
+
+elasticsearch.hosts: ["http://elasticsearch:9200"]
+elasticsearch.username: "kibana_system"     # Kibana's own internal account, NOT what visitors become
+elasticsearch.password: "${ES_KIBANA_PASSWORD}"
+
+# Fixed keys: without them Kibana re-generates them on restart and loses saved objects (dashboards, rules).
+xpack.encryptedSavedObjects.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+xpack.reporting.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+xpack.security.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+
+# The only provider, so there is no login page and no login selector: every request is already signed in.
+xpack.security.authc.providers:
+  anonymous.anonymous1:
+    order: 0
+    credentials:
+      username: "${VG_DIAG_USER}"
+      password: "${VG_DIAG_PASSWORD}"

observability/metricbeat.yml

@@ -0,0 +1,69 @@
+# Metricbeat: host and per-container metrics, replacing node_exporter and cAdvisor, plus the Prometheus
+# endpoints the stack already exposes. Runs on the host network so the system module sees the real interfaces
+# and so it can reach MediaMTX, Caddy, Elasticsearch and Kibana on loopback.
+
+metricbeat.modules:
+  - module: system
+    period: 30s
+    metricsets: [cpu, load, memory, network, process, process_summary, socket_summary, filesystem, fsstat, diskio, uptime]
+    processes: ['.*']
+    process.include_top_n:
+      by_cpu: 10
+      by_memory: 10
+    system.hostfs: /hostfs                  # the host's real filesystems, not the container's
+
+  - module: docker
+    period: 30s
+    hosts: ["unix:///var/run/docker.sock"]
+    metricsets: [container, cpu, diskio, healthcheck, info, memory, network]
+
+  - module: prometheus                      # MediaMTX: per path bytes/s, readers, publishers, dropped packets
+    period: 30s
+    metrics_path: /metrics
+    hosts: ["http://127.0.0.1:9998"]
+
+  - module: prometheus                      # Caddy: requests by host and status, durations, TLS
+    period: 60s
+    metrics_path: /metrics
+    hosts: ["http://127.0.0.1:2019"]
+
+  - module: prometheus                      # dumpd: OUR metrics — normalise queue, encode time, live streams
+    period: 30s
+    metrics_path: /metrics
+    hosts: ["http://127.0.0.1:8080"]        # dumpd's loopback listener; Caddy routes no host to /metrics
+    # These are real counters, so let them arrive typed: Kibana then gets .rate (per-second) as well as the
+    # raw total, which is what makes "encode seconds per hour" a chart rather than a staircase.
+    use_types: true
+    rate_counters: true
+
+processors:
+  - add_host_metadata: ~
+  # The same correlation fields Filebeat sets, so ONE dashboard filter spans logs and metrics. Without these a
+  # metric event carries container.name while a log event carries service.name, and nothing joins them.
+  - add_docker_metadata: ~
+  - copy_fields:
+      fields:
+        - from: container.labels.vanguard_service
+          to: service.name
+        - from: container.labels.vanguard_category
+          to: vanguard.category
+      fail_on_error: false
+      ignore_missing: true
+
+output.elasticsearch:
+  hosts: ["http://127.0.0.1:9200"]
+  # Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
+  # come from the container environment, which compose fills from .env — no secret is committed here.
+  username: "${ES_BEATS_USER}"
+  password: "${ES_BEATS_PASSWORD}"
+
+setup.kibana:                               # no credentials: Kibana's anonymous provider signs these in
+  host: "http://127.0.0.1:5601"
+setup.dashboards.enabled: true              # the system and docker dashboards, so there is something to look at
+# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
+# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
+setup.template.overwrite: true
+setup.template.settings:
+  index.number_of_replicas: 0
+
+logging.level: info

registerGiteaRunner.sh

@@ -11,3 +11,5 @@ sudo rm -rf /srv/appdata/act_runner/.runner 2>/dev/null || true     # forget any
 docker compose up -d --force-recreate act_runner
 sleep 6; docker logs act_runner --tail 8 2>&1 | sed 's/^/  /'
 echo "check: Gitea -> Site Administration -> Actions -> Runners (should list 'vanguard-runner' as idle)"
+echo "labels: $(grep -o 'GITEA_RUNNER_LABELS: \".*\"' docker-compose.yml | sed 's/.*\"\(.*\)\"/\1/' | tr ',' '\n' | cut -d: -f1 | tr '\n' ' ')"
+echo "  (labels are fixed at registration — re-run this script after changing GITEA_RUNNER_LABELS)"

streaming/disk.html

@@ -0,0 +1,132 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<title>Disk — Vanguard</title>
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#3d8bfd;--ok:#2ecc71;--warn:#e3a008;--bad:#e05260;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
+  main{max-width:1100px;margin:0 auto;padding:24px 16px 48px}
+  header{display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;margin-bottom:10px}
+  h1{font-size:24px;margin:0}.who{color:var(--mut)}.who b{color:var(--fg)}
+  nav{display:flex;gap:6px;margin:6px 0 18px;border-bottom:1px solid var(--line)}
+  nav a{padding:8px 14px;color:var(--mut);text-decoration:none;border-bottom:2px solid transparent}
+  nav a.on{color:var(--fg);border-color:var(--acc)}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:14px 18px;margin-bottom:12px}
+  section h2{margin:0 0 8px;font-size:15px;color:var(--mut);font-weight:600;text-transform:uppercase;letter-spacing:.3px}
+  table{width:100%;border-collapse:collapse;font-size:14px}th,td{text-align:left;padding:6px 8px;border-top:1px solid var(--line);vertical-align:top}
+  th{color:var(--mut);font-weight:600;border-top:0}td.n{text-align:right;font-variant-numeric:tabular-nums;white-space:nowrap}
+  .mono{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .pill{display:inline-block;font-size:12px;border-radius:999px;padding:1px 9px;white-space:nowrap}
+  .ok{background:#1f3b2a;color:var(--ok)}.warn{background:#3a3220;color:var(--warn)}.bad{background:#43242a;color:var(--bad)}.info{background:#1e3147;color:var(--acc)}
+  .state{font-size:12px;border-radius:999px;padding:2px 10px;text-align:center;white-space:nowrap}
+  .parked{background:#43242a;color:var(--bad)}.published{background:#3a3220;color:var(--warn)}.waiting{background:#1e3147;color:var(--acc)}.writing{background:#1f3b2a;color:var(--ok)}
+  .row{display:grid;grid-template-columns:96px 1fr auto;gap:12px;align-items:start}
+  .name{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .meta{color:var(--mut);font-size:13px;margin-top:2px}
+  .why{color:var(--mut);font-size:13px;margin-top:6px;border-left:2px solid var(--line);padding-left:10px;white-space:pre-wrap}
+  .acts{display:flex;gap:8px;align-items:center}
+  button{font:inherit;font-size:13px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:6px 12px;cursor:pointer}
+  select.stopmins{font:inherit;font-size:12px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:5px 6px}
+  button:hover:not(:disabled){border-color:var(--acc)}button:disabled{opacity:.45;cursor:default}
+  button.danger:hover:not(:disabled){border-color:var(--bad);color:var(--bad)}
+  input[type=number]{font:inherit;background:#121212;color:var(--fg);border:1px solid #3a3a3a;border-radius:6px;padding:4px 6px}
+  .note{color:var(--mut);font-size:13px}.err{color:var(--bad)}.hidden{display:none}
+  pre{background:#121212;border-radius:8px;padding:10px;overflow:auto;font-size:12px;margin:8px 0 0;max-height:280px}
+  .verdict{margin-top:8px;font-size:14px}
+  footer{color:var(--mut);font-size:13px;margin-top:20px}
+</style>
+</head>
+<body>
+<main>
+  <header><h1>Disk</h1><div class="who" id="who"></div></header>
+  <nav><a href="/disk" class="on">Disk</a><a href="/" style="margin-left:auto">Dashboards ↗</a></nav>
+  <div id="pane-disk">
+    <section>
+      <div class="row" style="grid-template-columns:1fr auto;align-items:center">
+        <div id="fs" class="note">Loading…</div>
+        <div class="acts"><span class="note" id="diskgen"></span><button id="diskrefresh">Rewalk now</button></div>
+      </div>
+    </section>
+    <section><h2>Streamers — recordings by month</h2><div id="dstreamers" class="note">Loading…</div></section>
+    <section><h2>Clippers — clips by month</h2><div id="dclippers" class="note">Loading…</div></section>
+    <section><h2>Logs</h2><div id="dlogs" class="note">Loading…</div></section>
+    <section><h2>System</h2><div id="dsystem" class="note">Loading…</div></section>
+    <footer>Sizes are allocated blocks, what the disk actually loses. A recording's month is the date in its own name; clips and
+    everything else go by modification time. Cached for ten minutes; Rewalk now re-reads every file.</footer>
+  </div>
+</main>
+<script>
+const API = '/streaming/api';
+let admin = false;
+const esc = s => String(s ?? '').replace(/[&<>"]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
+const hms = s => { s = Math.max(0, s|0); const h = s/3600|0, m = (s%3600)/60|0; return h ? `${h}h ${m}m` : `${m}m`; };
+const dur = s => { s = Math.max(0, s|0); return `${s/3600|0}:${String((s%3600)/60|0).padStart(2,'0')}:${String(s%60).padStart(2,'0')}`; };
+const short = p => p.replace(/^.*\/streams\//, '').replace('/.live/', ' · ');
+const mb = b => (b/1e6).toFixed(b < 10e6 ? 1 : 0) + ' MB';
+
+async function getJSON(path) {
+  const r = await fetch(path, { credentials: 'same-origin' });
+  if (r.status === 401) throw new Error('Not signed in. Reload the page.');
+  const text = await r.text();
+  let data; try { data = JSON.parse(text); }
+  catch { throw new Error(`Server answered HTTP ${r.status} with no usable body${text ? ': ' + text.slice(0, 160) : ' (an empty 502 means dumpd did not reply)'}`); }
+  if (data.error) throw new Error(`HTTP ${r.status}: ${data.error}`);
+  return data;
+}
+function whoami(d) { admin = !!d.admin; document.getElementById('who').innerHTML = `signed in as <b>${esc(d.user || '?')}</b>${admin ? '' : ' · read only'}`; }
+
+const gb = b => b >= 1e12 ? (b / 1e12).toFixed(2) + ' TB' : b >= 1e9 ? (b / 1e9).toFixed(1) + ' GB' : b >= 1e6 ? (b / 1e6).toFixed(0) + ' MB' : b ? (b / 1e3).toFixed(0) + ' kB' : '';
+const monthName = m => { const [y, mo] = m.split('-'); return new Date(+y, +mo - 1, 1).toLocaleString(undefined, { month: 'short', year: '2-digit' }); };
+function monthTable(rows, months, extra) {
+  // rows: [{name, total, months:{}, note}] ; months: newest first, older ones folded into "older"
+  const shown = months.slice(0, 6), older = months.slice(6);
+  const cell = (r, m) => { const v = r.months[m] || 0; return `<td class="n">${v ? gb(v) : '<span class="note">·</span>'}</td>`; };
+  const oldSum = r => older.reduce((a, m) => a + (r.months[m] || 0), 0);
+  const tot = key => rows.reduce((a, r) => a + (key === 'total' ? r.total : key === 'older' ? oldSum(r) : (r.months[key] || 0)), 0);
+  return `<table>
+    <tr><th></th>${shown.map(m => `<th class="n">${monthName(m)}</th>`).join('')}${older.length ? '<th class="n">older</th>' : ''}<th class="n">total</th>${extra ? '<th></th>' : ''}</tr>
+    ${rows.map(r => `<tr><td><b>${esc(r.name)}</b></td>${shown.map(m => cell(r, m)).join('')}${older.length ? `<td class="n">${gb(oldSum(r)) || '<span class="note">·</span>'}</td>` : ''}<td class="n"><b>${gb(r.total)}</b></td>${extra ? `<td class="note">${r.note || ''}</td>` : ''}</tr>`).join('')}
+    ${rows.length > 1 ? `<tr><td class="note">all</td>${shown.map(m => `<td class="n">${gb(tot(m))}</td>`).join('')}${older.length ? `<td class="n">${gb(tot('older'))}</td>` : ''}<td class="n"><b>${gb(tot('total'))}</b></td>${extra ? '<td></td>' : ''}</tr>` : ''}
+  </table>`;
+}
+async function loadDisk(force) {
+  let d;
+  document.getElementById('diskgen').textContent = force ? 'walking every file…' : '';
+  try { d = await getJSON(API + '/disk' + (force ? '?refresh=1' : '')); }
+  catch (e) { document.getElementById('fs').innerHTML = `<span class="err">${esc(e.message)}</span>`; return; }
+  whoami(d);
+  document.getElementById('diskgen').textContent = `walked ${new Date(d.generated).toLocaleTimeString()} in ${d.seconds}s`;
+  const bar = (label, f) => { const pct = f.total ? 100 * f.used / f.total : 0; const cls = pct > 90 ? 'bad' : pct > 75 ? 'warn' : 'ok';
+    return `<div style="margin:4px 0"><b>${esc(label)}</b> <span class="mono note">${esc(f.path)}</span> — ${gb(f.used)} of ${gb(f.total)} used, <span class="pill ${cls}">${gb(f.free)} free</span>
+      <div style="height:8px;background:#262626;border-radius:4px;margin-top:4px"><div style="width:${pct.toFixed(1)}%;height:8px;border-radius:4px;background:var(--${cls === 'ok' ? 'ok' : cls === 'warn' ? 'warn' : 'bad'})"></div></div></div>`; };
+  const fs = d.fs || {};
+  document.getElementById('fs').innerHTML = (fs.media ? bar('media', fs.media) : '') + (fs.root && !fs.same_filesystem ? bar('system disk', fs.root) : '') + (fs.same_filesystem ? '<div class="note">media and the system share one filesystem</div>' : '');
+  const monthsOf = objs => [...new Set(objs.flatMap(o => Object.keys(o.months || {})))].sort().reverse();
+  const toRows = obj => Object.entries(obj).map(([name, v]) => ({ name, total: v.total, months: v.months || {},
+    note: [v.raw ? `${gb(v.raw)} raw in .src` : '', v.unpublished ? `${gb(v.unpublished)} unpublished in .live` : ''].filter(Boolean).join(' · ') })).sort((a, b) => b.total - a.total);
+  const sRows = toRows(d.streamers || {}); const trash = d.trash || { total: 0, months: {} };
+  if (trash.total) sRows.push({ name: 'trash (all users, 3-day undo)', total: trash.total, months: trash.months, note: 'dumped nights and binned recordings, purged after 3 days' });
+  document.getElementById('dstreamers').innerHTML = sRows.length ? monthTable(sRows, monthsOf(sRows), true) : '<span class="note">no recordings</span>';
+  const cRows = toRows(d.clippers || {});
+  document.getElementById('dclippers').innerHTML = cRows.length ? monthTable(cRows, monthsOf(cRows), false) : '<span class="note">no clips</span>';
+  const L = d.logs || {};
+  const lRows = [];
+  if (L.months) lRows.push({ name: 'Elasticsearch indices (logs + metrics)', total: L.es_total || 0, months: L.months, note: L.es_data_dir ? `${gb(L.es_data_dir)} on disk in appdata/elasticsearch` : '' });
+  document.getElementById('dlogs').innerHTML = (lRows.length ? monthTable(lRows, monthsOf(lRows), true) : '') + `<table style="margin-top:8px">
+    ${L.es_data_dir != null ? `<tr><td>Elasticsearch data dir</td><td class="n">${gb(L.es_data_dir)}</td><td class="note">the on-disk truth for the indices above</td></tr>` : ''}
+    <tr><td>journald</td><td class="n">${gb(L.journald || 0)}</td><td class="note">systemd journal (dumpd, sweep, timers)</td></tr>
+    <tr><td>Docker json logs</td><td class="n">${gb(L.docker_json_logs || 0)}</td><td class="note">per-container log files Filebeat reads</td></tr>
+  </table>` + (L.indices ? `<details style="margin-top:8px"><summary class="note">${L.indices.length} indices</summary><table>${L.indices.sort((a, b) => b.bytes - a.bytes).map(i => `<tr><td class="mono">${esc(i.index)}</td><td class="n">${gb(i.bytes)}</td><td class="note">${esc(i.month)}</td></tr>`).join('')}</table></details>` : '');
+  const S = d.system || {};
+  document.getElementById('dsystem').innerHTML = `<table>${Object.entries(S).sort((a, b) => (typeof b[1] === 'number' ? b[1] : 0) - (typeof a[1] === 'number' ? a[1] : 0))
+    .map(([k, v]) => `<tr><td>${esc(k)}</td><td class="n">${typeof v === 'number' ? gb(v) : esc(v)}</td><td class="note">${k.startsWith('docker') ? "Docker's own figure" : k === 'media_other' ? 'markers, Syncthing state, versions under /srv/media' : ''}</td></tr>`).join('')}</table>`
+    + (d.errors && d.errors.length ? `<div class="err" style="margin-top:8px">${d.errors.map(esc).join('<br>')}</div>` : '');
+}
+document.getElementById('diskrefresh').onclick = () => loadDisk(true);
+
+loadDisk(false);
+</script>
+</body>
+</html>

streaming/index.html

@@ -0,0 +1,416 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Streaming — Vanguard</title>
+<meta name="viewport" content="width=device-width, initial-scale=1">
+<style>
+  :root{--bg:#141414;--fg:#e6e6e6;--mut:#9a9a9a;--acc:#3d8bfd;--ok:#2ecc71;--warn:#e3a008;--bad:#e05260;--box:#1f1f1f;--line:#2e2e2e}
+  html,body{margin:0;background:var(--bg);color:var(--fg);font:15px/1.5 system-ui,sans-serif}
+  main{max-width:1100px;margin:0 auto;padding:24px 16px 48px}
+  header{display:flex;align-items:baseline;gap:14px;flex-wrap:wrap;margin-bottom:10px}
+  h1{font-size:24px;margin:0}.who{color:var(--mut)}.who b{color:var(--fg)}
+  nav{display:flex;gap:6px;margin:6px 0 18px;border-bottom:1px solid var(--line)}
+  nav a{padding:8px 14px;color:var(--mut);text-decoration:none;border-bottom:2px solid transparent}
+  nav a.on{color:var(--fg);border-color:var(--acc)}
+  section{background:var(--box);border:1px solid var(--line);border-radius:14px;padding:14px 18px;margin-bottom:12px}
+  section h2{margin:0 0 8px;font-size:15px;color:var(--mut);font-weight:600;text-transform:uppercase;letter-spacing:.3px}
+  table{width:100%;border-collapse:collapse;font-size:14px}th,td{text-align:left;padding:6px 8px;border-top:1px solid var(--line);vertical-align:top}
+  th{color:var(--mut);font-weight:600;border-top:0}td.n{text-align:right;font-variant-numeric:tabular-nums;white-space:nowrap}
+  .mono{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .pill{display:inline-block;font-size:12px;border-radius:999px;padding:1px 9px;white-space:nowrap}
+  .ok{background:#1f3b2a;color:var(--ok)}.warn{background:#3a3220;color:var(--warn)}.bad{background:#43242a;color:var(--bad)}.info{background:#1e3147;color:var(--acc)}
+  .state{font-size:12px;border-radius:999px;padding:2px 10px;text-align:center;white-space:nowrap}
+  .parked{background:#43242a;color:var(--bad)}.published{background:#3a3220;color:var(--warn)}.waiting{background:#1e3147;color:var(--acc)}.writing{background:#1f3b2a;color:var(--ok)}
+  .row{display:grid;grid-template-columns:96px 1fr auto;gap:12px;align-items:start}
+  .name{font-family:ui-monospace,monospace;font-size:13px;word-break:break-all}
+  .meta{color:var(--mut);font-size:13px;margin-top:2px}
+  .why{color:var(--mut);font-size:13px;margin-top:6px;border-left:2px solid var(--line);padding-left:10px;white-space:pre-wrap}
+  .acts{display:flex;gap:8px;align-items:center}
+  button{font:inherit;font-size:13px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:6px 12px;cursor:pointer}
+  select.stopmins{font:inherit;font-size:12px;border-radius:8px;border:1px solid #3a3a3a;background:#262626;color:var(--fg);padding:5px 6px}
+  button:hover:not(:disabled){border-color:var(--acc)}button:disabled{opacity:.45;cursor:default}
+  button.danger:hover:not(:disabled){border-color:var(--bad);color:var(--bad)}
+  input[type=number]{font:inherit;background:#121212;color:var(--fg);border:1px solid #3a3a3a;border-radius:6px;padding:4px 6px}
+  .note{color:var(--mut);font-size:13px}.err{color:var(--bad)}.hidden{display:none}
+  pre{background:#121212;border-radius:8px;padding:10px;overflow:auto;font-size:12px;margin:8px 0 0;max-height:280px}
+  .verdict{margin-top:8px;font-size:14px}
+  footer{color:var(--mut);font-size:13px;margin-top:20px}
+</style>
+</head>
+<body>
+<main>
+  <header><h1>Streaming</h1><div class="who" id="who"></div></header>
+  <nav><a href="#live" id="tab-live">Live</a><a href="#stuck" id="tab-stuck">Stuck recordings</a><a href="#dump" id="tab-dump">Dump a night</a><a href="/disk" style="margin-left:auto">Disk ↗</a></nav>
+
+  <!-- ===================== LIVE ===================== -->
+  <div id="pane-live">
+    <section>
+      <h2>Publishing now</h2>
+      <div id="pubs" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Blocked from publishing</h2>
+      <div id="blocks" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Why MediaMTX opened recording files — last <span id="win">10m</span></h2>
+      <div id="counts" class="note">Loading…</div>
+      <div id="verdict" class="verdict"></div>
+    </section>
+    <section>
+      <h2>Live paths</h2>
+      <div id="paths" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Publish sessions — last 24 h</h2>
+      <div id="hist" class="note">Loading…</div>
+    </section>
+    <section>
+      <h2>Recorder log</h2>
+      <pre id="log">Loading…</pre>
+    </section>
+    <footer>Refreshes every 10 seconds. Loss is counted on the <em>publish</em> session, i.e. on the way in: that is what corrupts recordings. Viewers losing packets only affects what they see.</footer>
+  </div>
+
+  <!-- ===================== STUCK ===================== -->
+  <div id="pane-stuck" class="hidden">
+    <p class="note">Recordings that finished but never reached Jellyfin or copyparty. Retry puts one back in the
+    queue; bin moves it to the trash, recoverable for three days just like a dumped night.</p>
+    <section id="bulk" class="hidden">
+      <div class="row" style="grid-template-columns:1fr auto">
+        <div>
+          <b>Reconnect fragments</b>
+          <div class="meta">Every time a publisher drops and comes back, MediaMTX opens a new recording. Stream
+          recordings this small are stubs, not recordings. Clips are not affected: they never pass through <code>.live/</code>.</div>
+        </div>
+        <div class="acts">
+          <label class="meta">under <input id="mb" type="number" min="1" max="50" value="5" style="width:4em"> MB</label>
+          <button id="bulkbtn" class="danger">Bin <span id="bulkcount">0</span> fragments</button>
+        </div>
+      </div>
+    </section>
+    <div id="list"><section class="note">Loading…</section></div>
+    <footer>Refreshes every 30 seconds. Trends and history live on the dashboards.</footer>
+  </div>
+
+  <!-- ===================== DUMP ===================== -->
+  <div id="pane-dump" class="hidden">
+    <p class="note">A bad night goes to the trash in one move: every stream recording and clip a streamer saved
+    between <b>17:00 and 02:00 Eastern</b> on the chosen night. Recoverable for three days, like the Stuck tab's
+    bin. Admins can dump anyone or everyone; everyone else sees and dumps only their own.</p>
+    <section>
+      <div class="row" style="grid-template-columns:auto 1fr auto;align-items:center">
+        <label class="meta">night <input id="night" type="date" style="font:inherit;background:#262626;color:var(--fg);border:1px solid #3a3a3a;border-radius:8px;padding:4px 8px"></label>
+        <div class="meta" id="dumpwin"></div>
+        <div class="acts"><button id="dumpall" class="danger hidden">Dump everyone</button></div>
+      </div>
+    </section>
+    <div id="dumplist"><section class="note">Loading…</section></div>
+    <section id="wipe" class="hidden">
+      <div class="row" style="grid-template-columns:1fr auto">
+        <div>
+          <b>Unprocessed and stuck recordings</b>
+          <div class="meta" id="wipemeta"></div>
+          <div class="meta">Everything still sitting in a <code>.live/</code> folder, whatever night it is from: stuck, parked, waiting, fresh, reconnect fragments, and the failure logs that go with them. Any conversion the sweep has in flight is cancelled first. The only thing left alone is a stream MediaMTX is recording this minute.</div>
+        </div>
+        <div class="acts"><button id="wipebtn" class="danger">Bin all unprocessed</button></div>
+      </div>
+      <pre id="wipeout" class="hidden"></pre>
+    </section>
+    <footer>Nothing here moves until you confirm twice. Streamers can also dump their own night from the OBS dock.</footer>
+  </div>
+
+</main>
+<script>
+const API = '/streaming/api';
+let admin = false;
+const esc = s => String(s ?? '').replace(/[&<>"]/g, c => ({'&':'&amp;','<':'&lt;','>':'&gt;','"':'&quot;'}[c]));
+const hms = s => { s = Math.max(0, s|0); const h = s/3600|0, m = (s%3600)/60|0; return h ? `${h}h ${m}m` : `${m}m`; };
+const dur = s => { s = Math.max(0, s|0); return `${s/3600|0}:${String((s%3600)/60|0).padStart(2,'0')}:${String(s%60).padStart(2,'0')}`; };
+const short = p => p.replace(/^.*\/streams\//, '').replace('/.live/', ' · ');
+const mb = b => (b/1e6).toFixed(b < 10e6 ? 1 : 0) + ' MB';
+
+async function getJSON(path) {
+  const r = await fetch(path, { credentials: 'same-origin' });
+  if (r.status === 401) throw new Error('Not signed in. Reload the page.');
+  const text = await r.text();
+  let data; try { data = JSON.parse(text); }
+  catch { throw new Error(`Server answered HTTP ${r.status} with no usable body${text ? ': ' + text.slice(0, 160) : ' (an empty 502 means dumpd did not reply)'}`); }
+  if (data.error) throw new Error(`HTTP ${r.status}: ${data.error}`);
+  return data;
+}
+function whoami(d) { admin = !!d.admin; document.getElementById('who').innerHTML = `signed in as <b>${esc(d.user || '?')}</b>${admin ? '' : ' · read only'}`; }
+
+// ---------------- Live ----------------
+async function loadLive() {
+  let d;
+  try { d = await getJSON(API + '/live'); }
+  catch (e) { document.getElementById('pubs').innerHTML = `<span class="err">${esc(e.message)}</span>`; return; }
+  whoami(d);
+
+  const pubs = d.publishers || [];
+  document.getElementById('pubs').innerHTML = pubs.length ? `<table>
+    <tr><th>path</th><th>via</th><th>from</th><th>since</th><th class="n">received</th><th class="n">lost</th><th>loss</th><th></th></tr>
+    ${pubs.map(p => {
+      const pct = p.received ? (100 * (p.lost || 0) / p.received) : 0;
+      const cls = p.kind !== 'webrtc' ? 'info' : pct === 0 ? 'ok' : pct < 0.05 ? 'warn' : 'bad';
+      const txt = p.kind !== 'webrtc' ? 'tcp' : `${pct.toFixed(3)}%`;
+      // webrtc: the selected ICE pair's protocol. tcp = the Go Live lossy-uplink switch (or the box ticked)
+      const via = p.kind === 'webrtc' ? `webrtc <span class="pill ${p.transport === 'tcp' ? 'info' : p.transport === 'udp' ? 'ok' : 'warn'}">${esc(p.transport || 'connecting')}</span>` : esc(p.kind);
+      return `<tr><td class="mono">${esc(p.path)}</td><td>${via}</td><td class="mono">${esc(p.from)}</td><td class="note">${p.since ? new Date(p.since).toLocaleTimeString() : ''}</td>
+        <td class="n">${p.received != null ? p.received.toLocaleString() : (p.bytes != null ? mb(p.bytes) : '')}</td>
+        <td class="n">${p.lost != null ? p.lost.toLocaleString() : ''}</td><td><span class="pill ${cls}">${txt}</span></td>
+        <td class="acts">${admin && p.session ? `<select class="stopmins" data-session="${esc(p.session)}" title="how long the account stays unable to publish after the kick">
+            <option value="0">kick only</option><option value="30" selected>block 30 min</option><option value="120">block 2 h</option><option value="480">block 8 h</option></select>
+          <button class="danger stopbtn" data-path="${esc(p.path)}" data-kind="${esc(p.kind)}" data-session="${esc(p.session)}">Stop stream</button>` : ''}</td></tr>`;
+    }).join('')}</table>` : '<span class="note">nobody is publishing</span>';
+  for (const b of document.querySelectorAll('#pubs .stopbtn')) b.onclick = () => stopStream(b);
+
+  // accounts currently blocked from publishing (set by Stop stream), with an unblock
+  const blocks = d.blocks || [];
+  document.getElementById('blocks').innerHTML = blocks.length ? `<table>
+    <tr><th>account</th><th>blocked</th><th>until</th><th>by</th><th>why</th><th></th></tr>
+    ${blocks.map(b => `<tr><td class="mono">${esc(b.user)}</td><td class="note">${b.since ? new Date(b.since).toLocaleTimeString() : ''}</td>
+      <td class="note">${new Date(b.until_iso).toLocaleTimeString()}</td><td>${esc(b.by || '')}</td><td class="note">${esc(b.reason || '')}</td>
+      <td class="acts">${admin ? `<button class="unblock" data-user="${esc(b.user)}">Allow again</button>` : ''}</td></tr>`).join('')}</table>`
+    : '<span class="note">nobody is blocked from publishing</span>';
+  for (const b of document.querySelectorAll('#blocks .unblock')) b.onclick = () => unblock(b.dataset.user);
+
+  const c = d.counts || {};
+  document.getElementById('win').textContent = c.window || '10m';
+  document.getElementById('counts').innerHTML = `<table>
+    <tr><td>new publish sessions</td><td class="n">${c.publish_sessions ?? '?'}</td><td class="note">a reconnect each</td></tr>
+    <tr><td>recorder (re)starts</td><td class="n">${c.recorder_restarts ?? '?'}</td><td class="note">a new .live file each</td></tr>
+    <tr><td>processing errors</td><td class="n">${c.processing_errors ?? '?'}</td><td class="note">lost packets corrupting frames</td></tr>
+    <tr><td>recorder access-unit errors</td><td class="n">${c.access_unit_errors ?? '?'}</td><td class="note">frames the recorder refused</td></tr>
+  </table>`;
+  let v = '';
+  if ((c.recorder_restarts|0) > 1) {
+    if ((c.processing_errors|0) > 0) v = '<span class="pill bad">loss</span> files are splitting because packets are being lost on the way in: check the publisher\'s uplink, and that MediaMTX is on the host network.';
+    else if ((c.publish_sessions|0) >= (c.recorder_restarts|0) - 1) v = '<span class="pill warn">reconnects</span> the publisher\'s session keeps ending: look at their browser console and the "closed" reasons below.';
+    else v = '<span class="pill warn">parameter changes</span> one stable session, no loss, yet files keep opening: the encoder is changing resolution (new SPS). Pin resolution in the publisher.';
+  } else if (pubs.length) v = '<span class="pill ok">steady</span> one file per stream, nothing lost.';
+  document.getElementById('verdict').innerHTML = v;
+
+  const paths = d.paths || [];
+  document.getElementById('paths').innerHTML = paths.length ? `<table>
+    <tr><th>path</th><th>tracks</th><th>encode</th><th class="n">viewers</th><th class="n">received</th><th>since</th></tr>
+    ${paths.map(p => {
+      const h = p.health;
+      // MediaMTX refuses WebRTC for H.264 with B-frames: that is the single most common "I can't watch it".
+      const enc = p.mirror ? '' : !h ? '<span class="note">probing…</span>'
+        : h.bframes > 0 ? `<span class="pill bad">B-frames (depth ${h.bframes}) · WebRTC refused</span> <span class="note">${esc(h.res || '')} ${h.fps ? h.fps + ' fps' : ''}</span>`
+        : `<span class="pill ok">no B-frames</span> <span class="note">${esc(h.res || '')} ${h.fps ? h.fps + ' fps' : ''}</span>`;
+      return `<tr><td class="mono">${esc(p.name)}${p.mirror ? ' <span class="pill info">audio mirror</span>' : ''}</td>
+      <td>${esc((p.tracks || []).join(', '))}</td><td>${enc}</td><td class="n">${p.readers}</td><td class="n">${mb(p.bytes || 0)}</td>
+      <td class="note">${p.since ? new Date(p.since).toLocaleTimeString() : ''}</td></tr>`;
+    }).join('')}</table>` : '<span class="note">nothing live</span>';
+
+  // Every WebRTC publish session MediaMTX has had in 24 h, with the transport it ran on. A session that starts
+  // on a different transport than the previous one of the same path is a switch: that is "when he swapped".
+  const hist = d.history || [];
+  const span = (a, b) => { if (!a) return ''; const s = Math.max(0, ((b ? new Date(b) : new Date()) - new Date(a)) / 1000 | 0); return hms(s) || '<1m'; };
+  document.getElementById('hist').innerHTML = hist.length ? `<table>
+    <tr><th>path</th><th>transport</th><th>started</th><th>ended</th><th>ran</th><th class="n">received</th><th class="n">lost</th><th>loss</th><th></th></tr>
+    ${hist.map(h => {
+      const pct = h.received ? (100 * (h.lost || 0) / h.received) : 0;
+      const cls = pct === 0 ? 'ok' : pct < 0.05 ? 'warn' : 'bad';
+      const sw = h.switched_from ? `<span class="pill info">switched from ${esc(h.switched_from)}</span>` : '';
+      return `<tr><td class="mono">${esc(h.path)}</td><td><span class="pill ${h.transport === 'tcp' ? 'info' : h.transport ? 'ok' : 'warn'}">${esc(h.transport || '?')}</span></td>
+        <td class="note">${h.created ? new Date(h.created).toLocaleTimeString() : ''}</td><td class="note">${h.ended ? new Date(h.ended).toLocaleTimeString() : '<b>live</b>'}</td>
+        <td class="note">${span(h.created, h.ended)}</td><td class="n">${(h.received || 0).toLocaleString()}</td><td class="n">${(h.lost || 0).toLocaleString()}</td>
+        <td><span class="pill ${cls}">${pct.toFixed(3)}%</span></td><td>${sw}</td></tr>`;
+    }).join('')}</table>` : '<span class="note">no WebRTC publish sessions in the last 24 h</span>';
+
+  document.getElementById('log').textContent = (d.log || []).join('\n') || '(quiet)';
+  if (d.errors && d.errors.length) document.getElementById('log').textContent += '\n\n' + d.errors.join('\n');
+}
+
+// Stop stream: MediaMTX kicks the publish session, and dumpd blocks the account's publishing for the chosen
+// time so OBS's auto-reconnect or a Go Live restart cannot bring it straight back. Kick only = a hiccup.
+async function stopStream(btn) {
+  const sel = document.querySelector(`#pubs .stopmins[data-session="${btn.dataset.session}"]`);
+  const minutes = parseInt(sel?.value || '30', 10);
+  const what = minutes ? `and block ${esc(btn.dataset.path.split('/')[0])} from publishing for ${minutes >= 60 ? (minutes / 60) + ' h' : minutes + ' min'}` : 'only (they can reconnect at once)';
+  if (!confirm(`Stop ${btn.dataset.path} now, ${what}?\n\nThe recording so far is kept.`)) return;
+  btn.disabled = true; btn.textContent = 'Stopping…';
+  try {
+    const r = await fetch(API + '/live', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ action: 'stop', path: btn.dataset.path, kind: btn.dataset.kind, session: btn.dataset.session, minutes }) });
+    const body = await r.json();
+    btn.textContent = body.error ? `Failed: ${body.error}` : (body.kicked ? 'Stopped' : 'Not stopped');
+  } catch (e) { btn.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadLive, 1500);
+}
+async function unblock(user) {
+  if (!confirm(`Let ${user} publish again now?`)) return;
+  try {
+    await fetch(API + '/live', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action: 'unblock', user }) });
+  } catch {}
+  loadLive();
+}
+
+// ---------------- Stuck ----------------
+async function loadStuck() {
+  const list = document.getElementById('list');
+  let data;
+  try { data = await getJSON(API + '/stuck'); }
+  catch (e) { list.innerHTML = `<section class="note err">${esc(e.message)}</section>`; return; }
+  whoami(data);
+  const items = (data.items || []).sort((a, b) => (a.state === b.state ? b.age_s - a.age_s : a.state.localeCompare(b.state)));
+  const bulk = document.getElementById('bulk');
+  bulk.classList.toggle('hidden', !admin || !items.length);
+  window.__items = items; updateBulkCount();
+  if (!items.length) { list.innerHTML = '<section class="note">Nothing is stuck. Everything recorded has published.</section>'; return; }
+  list.innerHTML = '';
+  for (const it of items) {
+    const el = document.createElement('section');
+    const actionable = admin && it.state !== 'writing';
+    el.innerHTML = `
+      <div class="row">
+        <div class="state ${esc(it.state)}">${esc(it.state)}</div>
+        <div>
+          <div class="name">${esc(short(it.src))}</div>
+          <div class="meta">waiting ${hms(it.age_s)}${it.duration_s == null ? '' : ' · ' + dur(it.duration_s) + ' long'} · ${mb(it.bytes)}</div>
+          <div class="why">${esc(it.reason || '')}</div>
+        </div>
+        <div class="acts">
+          <button data-act="retry" ${actionable ? '' : 'disabled'}>Retry</button>
+          <button data-act="drop" class="danger" ${actionable ? '' : 'disabled'}>Bin</button>
+        </div>
+      </div>`;
+    for (const b of el.querySelectorAll('button')) b.onclick = () => act(it, b.dataset.act, el);
+    list.appendChild(el);
+  }
+}
+async function act(item, action, el) {
+  if (action === 'drop' && !confirm(`Move this recording to the trash?\n\n${short(item.src)}\n\nRecoverable for three days.`)) return;
+  for (const b of el.querySelectorAll('button')) b.disabled = true;
+  const out = document.createElement('pre'); out.textContent = action === 'retry' ? 'Clearing the blocker…' : 'Moving to trash…'; el.appendChild(out);
+  try {
+    const r = await fetch(API + '/stuck', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action, src: item.src }) });
+    const body = await r.json();
+    out.textContent = body.error ? `Failed: ${body.error}` : (body.output || []).join('\n') || 'Done.';
+    if (!body.error) setTimeout(loadStuck, 1200);
+  } catch (e) { out.textContent = `Failed: ${e.message}`; }
+}
+function updateBulkCount() {
+  const mbv = Math.max(1, Math.min(50, parseInt(document.getElementById('mb').value || '5', 10)));
+  const n = (window.__items || []).filter(i => i.state !== 'writing' && i.bytes < mbv * 1e6).length;
+  document.getElementById('bulkcount').textContent = n;
+  document.getElementById('bulkbtn').disabled = n === 0;
+}
+document.getElementById('mb').addEventListener('input', updateBulkCount);
+document.getElementById('bulkbtn').onclick = async () => {
+  const mbv = parseInt(document.getElementById('mb').value || '5', 10);
+  const n = document.getElementById('bulkcount').textContent;
+  if (!confirm(`Move ${n} stream recordings smaller than ${mbv} MB to the trash?\n\nRecoverable for three days. Clips are not affected.`)) return;
+  const btn = document.getElementById('bulkbtn'); btn.disabled = true; btn.textContent = 'Binning…';
+  try {
+    const r = await fetch(API + '/stuck', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action: 'drop_under', mb: mbv }) });
+    const body = await r.json();
+    btn.textContent = body.error ? `Failed: ${body.error}` : 'Done';
+  } catch (e) { btn.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadStuck, 1500);
+};
+
+// ---------------- Dump ----------------
+// Same endpoint as the dock's "Dump tonight" button (dumpd), reached here as /streaming/api/dump so the
+// Authentik cookie for this host is enough. GET is always a dry run; only the POST moves anything.
+const nightEl = document.getElementById('night');
+let dumpNightTouched = false;
+nightEl.addEventListener('change', () => { dumpNightTouched = true; loadDump(); });
+async function loadDump() {
+  const list = document.getElementById('dumplist');
+  let d;
+  try {
+    const q = new URLSearchParams({ user: 'all' });                       // non-admins get their own back regardless
+    if (dumpNightTouched && nightEl.value) q.set('night', nightEl.value);
+    d = await getJSON(API + '/dump?' + q);
+  } catch (e) { list.innerHTML = `<section class="note err">${esc(e.message)}</section>`; return; }
+  whoami(d);
+  if (!dumpNightTouched) nightEl.value = d.night;
+  document.getElementById('dumpwin').textContent = d.window ? `${d.window[0].replace('T', ' ').slice(0, 16)} → ${d.window[1].replace('T', ' ').slice(0, 16)}` : '';
+  const byUser = {};
+  for (const e of d.entries || []) {
+    const m = e.path.match(/^(streams|clips)\/([^/]+)\//);
+    const u = m ? m[2] : '(other)'; const kind = m ? m[1] : 'other';
+    const row = byUser[u] ||= { user: u, streams: 0, clips: 0, bytes: 0 };
+    row[kind === 'streams' ? 'streams' : 'clips']++; row.bytes += e.bytes;
+  }
+  const rows = Object.values(byUser).sort((a, b) => b.bytes - a.bytes);
+  document.getElementById('dumpall').classList.toggle('hidden', !(admin && rows.length > 1));
+  if (!rows.length) { list.innerHTML = `<section class="note">Nothing recorded on the night of ${esc(d.night)}${admin ? '' : ' under your name'}.</section>`; return; }
+  list.innerHTML = `<section><table>
+    <tr><th>streamer</th><th class="n">stream recordings</th><th class="n">clips</th><th class="n">size</th><th></th></tr>
+    ${rows.map(r => `<tr data-user="${esc(r.user)}"><td><b>${esc(r.user)}</b></td><td class="n">${r.streams}</td><td class="n">${r.clips}</td><td class="n">${(r.bytes / 1e9).toFixed(2)} GB</td>
+      <td class="acts"><button class="danger" data-user="${esc(r.user)}">Dump ${esc(r.user)}'s night</button></td></tr>`).join('')}
+    <tr><td>total</td><td class="n">${rows.reduce((a, r) => a + r.streams, 0)}</td><td class="n">${rows.reduce((a, r) => a + r.clips, 0)}</td><td class="n"><b>${(d.gb || 0).toFixed(2)} GB</b></td><td></td></tr>
+  </table><pre id="dumpout" class="hidden"></pre></section>`;
+  for (const b of list.querySelectorAll('button[data-user]')) b.onclick = () => dumpNight(b.dataset.user, b);
+}
+async function dumpNight(user, btn) {
+  const night = nightEl.value;
+  const who = user === 'all' ? 'EVERYONE' : user;
+  if (!confirm(`Move ${who}'s recordings and clips from the night of ${night} to the trash?\n\nRecoverable for three days.`)) return;
+  if (btn.dataset.armed !== night) {                                   // second click, same night: the dock's two-step
+    btn.dataset.armed = night; const was = btn.textContent; btn.textContent = 'Click again to confirm';
+    setTimeout(() => { if (btn.isConnected) { btn.textContent = was; delete btn.dataset.armed; } }, 10000);
+    return;
+  }
+  btn.disabled = true; btn.textContent = 'Dumping…';
+  const out = document.getElementById('dumpout'); out.classList.remove('hidden'); out.textContent = `Moving ${who}'s night of ${night}…`;
+  try {
+    const r = await fetch(API + '/dump', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ user, night }) });
+    const body = await r.json();
+    out.textContent = body.error ? `Failed: ${body.error}` : `Dumped ${body.moved} file(s), ${body.gb} GB from ${body.user}'s night of ${body.night}. Recoverable until ${body.undo_until}.`;
+  } catch (e) { out.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadDump, 1200);
+}
+document.getElementById('dumpall').onclick = () => dumpNight('all', document.getElementById('dumpall'));
+
+// "Bin all unprocessed": the Stuck tab's survey gives the count and size; vgstuck.sh wipe does the moving.
+async function loadWipe() {
+  const box = document.getElementById('wipe');
+  let d;
+  try { d = await getJSON(API + '/stuck'); } catch (e) { box.classList.add('hidden'); return; }
+  const items = d.items || [];                         // "writing" is only a guess from mtime; the tool checks what is really open
+  box.classList.toggle('hidden', !admin || !items.length);
+  const gb = items.reduce((a, i) => a + i.bytes, 0) / 1e9;
+  document.getElementById('wipemeta').textContent = `${items.length} recording(s), ${gb.toFixed(2)} GB in .live/`;
+  const b = document.getElementById('wipebtn'); b.disabled = false; b.textContent = `Bin all ${items.length} unprocessed`; delete b.dataset.armed;
+}
+document.getElementById('wipebtn').onclick = async () => {
+  const btn = document.getElementById('wipebtn');
+  const meta = document.getElementById('wipemeta').textContent;
+  if (!confirm(`Move every unprocessed recording to the trash and cancel any conversion in progress?\n\n${meta}\n\nRecoverable for three days. Only a stream being recorded this minute is left alone.`)) return;
+  if (!btn.dataset.armed) { btn.dataset.armed = '1'; const was = btn.textContent; btn.textContent = 'Click again to confirm'; setTimeout(() => { if (btn.dataset.armed) { btn.textContent = was; delete btn.dataset.armed; } }, 10000); return; }
+  delete btn.dataset.armed; btn.disabled = true; btn.textContent = 'Binning…';
+  const out = document.getElementById('wipeout'); out.classList.remove('hidden'); out.textContent = 'Moving to trash…';
+  try {
+    const r = await fetch(API + '/stuck', { method: 'POST', credentials: 'same-origin', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ action: 'wipe' }) });
+    const body = await r.json();
+    out.textContent = body.error ? `Failed: ${body.error}` : (body.output || []).join('\n') || 'Done.';
+  } catch (e) { out.textContent = `Failed: ${e.message}`; }
+  setTimeout(loadWipe, 1500);
+};
+
+// ---------------- tabs ----------------
+let timer = null;
+function show(tab) {
+  if (!['live', 'stuck', 'dump'].includes(tab)) tab = 'live';
+  for (const t of ['live', 'stuck', 'dump']) {
+    document.getElementById('pane-' + t).classList.toggle('hidden', t !== tab);
+    document.getElementById('tab-' + t).classList.toggle('on', t === tab);
+  }
+  clearInterval(timer);
+  if (tab === 'live') { loadLive(); timer = setInterval(loadLive, 10000); }
+  else if (tab === 'stuck') { loadStuck(); timer = setInterval(loadStuck, 30000); }
+  else { dumpNightTouched = false; loadDump(); loadWipe(); }
+}
+window.addEventListener('hashchange', () => show(location.hash.replace('#', '')));
+show(location.hash.replace('#', '') || 'live');
+</script>
+</body>
+</html>

switchToClone.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+# One-time: turn ~/vanguard-stream (an scp copy) into a git clone of the Gitea repo, keeping the generated files.
+#   ./switchToClone.sh key     -> creates a deploy key and prints the public half (add it in Gitea: repo → Settings → Deploy keys, read-only)
+#   ./switchToClone.sh clone   -> clones over ssh (port 2222), moves .env + generated files + plugin zips into it
+set -euo pipefail
+REPO_SSH="${REPO_SSH:-ssh://git@git.vng.one:2222/techy/Media-Server-Setup.git}"
+DIR="$HOME/vanguard-stream"; KEY="$HOME/.ssh/vanguard-deploy"
+case "${1:-}" in
+  key)
+    [[ -f $KEY ]] || ssh-keygen -t ed25519 -N "" -C "vanguard-deploy@$(hostname)" -f "$KEY" >/dev/null
+    cat > "$HOME/.ssh/config.vanguard" <<CFG
+Host git.vng.one
+  Port 2222
+  User git
+  IdentityFile $KEY
+  IdentitiesOnly yes
+CFG
+    grep -q 'config.vanguard' "$HOME/.ssh/config" 2>/dev/null || { echo "Include config.vanguard" | cat - "$HOME/.ssh/config" 2>/dev/null > "$HOME/.ssh/config.new" || echo "Include config.vanguard" > "$HOME/.ssh/config.new"; mv "$HOME/.ssh/config.new" "$HOME/.ssh/config"; chmod 600 "$HOME/.ssh/config"; }
+    echo "Add this as a READ-ONLY deploy key on the repo (Gitea → repo → Settings → Deploy Keys):"; echo; cat "$KEY.pub" ;;
+  clone)
+    [[ -d $DIR/.git ]] && { echo "$DIR is already a clone"; exit 0; }
+    ssh -o StrictHostKeyChecking=accept-new -p 2222 git@git.vng.one 2>&1 | grep -qi "successfully authenticated\|Hi there" || { echo "deploy key not accepted yet — run './switchToClone.sh key' and add it in Gitea"; exit 1; }
+    mv "$DIR" "$DIR.old"
+    git clone --quiet "$REPO_SSH" "$DIR"
+    for f in .env Caddyfile mediamtx.yml copyparty.conf stream-token.secret deploy.env; do [[ -f $DIR.old/$f ]] && cp -a "$DIR.old/$f" "$DIR/$f"; done
+    mkdir -p "$DIR/jellyfin-plugin-vanguard/dist"; cp -a "$DIR.old"/jellyfin-plugin-vanguard/dist/*.zip "$DIR/jellyfin-plugin-vanguard/dist/" 2>/dev/null || true
+    ( cd "$DIR" && sudo -n true 2>/dev/null && sudo install -m 0600 -o root -g root .env /dev/null >/dev/null 2>&1 || true )
+    echo "cloned. kept: .env, generated configs, plugin zips. Old copy at $DIR.old (delete when happy)."
+    echo "  (the Discord bot is not in this repo: its release lives in /srv/bot/app and is untouched by this)"
+    echo "next: cd $DIR && ./configure.py --apply   (records nothing new; just confirms generation works from the clone)"
+    echo "      then: sudo vanguard-deploy status" ;;
+  *) sed -n '2,5p' "$0"; exit 2 ;;
+esac

syncthing/stignore

@@ -1,5 +1,5 @@
 // Server-side ignore list for a clips receive folder (clips/<user>): pull video only.
-// .stlocal is appended to by the sweep: raws it converted (then deleted locally) and the standard
+// .stlocal must exist (created empty with the folder) and is appended to by the sweep: raws it converted (then deleted locally) and the standard
 // files it produced. Ignored = not tracked, not flagged, never re-pulled, never sent back.
 #include .stlocal
 // Ignored files are never downloaded from the sender. Subfolders are still traversed

systemd/deploy-pending.service

@@ -0,0 +1,7 @@
+[Unit]
+Description=Restart MediaMTX if a deploy deferred it and no stream is live
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/vanguard-deploy pending

systemd/deploy-pending.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Retry deferred MediaMTX restart every 5 minutes
+
+[Timer]
+OnBootSec=3min
+OnUnitActiveSec=5min
+
+[Install]
+WantedBy=timers.target

systemd/dumpd.service

@@ -9,6 +9,8 @@ Environment=DUMP_ADMIN_GROUPS=admins,media-admins
 Environment=PUBLISH_GROUPS=streamers,admins,media-admins
 Environment=STREAM_TOKEN_TTL_HOURS=12
 Environment=STREAM_TOKEN_SECRET_FILE=/etc/vanguard/stream-token.secret
+Environment=CLIPS_QUOTA_GB=100
+Environment=CLIPS_QUOTA_FILE=/etc/vanguard/clip-quotas.json
 Environment=DUMP_TZ=America/New_York
 Restart=always
 RestartSec=3

systemd/normalize-sweep.service

@@ -8,3 +8,6 @@ Type=oneshot
 ExecStart=/usr/local/bin/normalize-sweep.sh
 Nice=15
 IOSchedulingClass=idle
+Environment=ENCODE_THREADS=2   # x264 threads per conversion; keeps Jellyfin and the live portal responsive (default 2)
+# Environment=STITCH_MAX_PARTS=500  # join a stream's fragments when fewer than this (0 = never); STITCH_MAX_GAP_S=600 splits sessions
+# Environment=FILL_GAPS_MS=1000   # opt-in: CFR re-encode streams whose dropped-frame holes total >= 1 s (hours of CPU on this box)

systemd/vanguard-alerts.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Vanguard alerts: deliver Kibana's alert documents to Discord
+After=docker.service
+
+[Service]
+Type=oneshot
+# Written by configure.py from .env (ALERT_RETENTION_DAYS, ALERT_SUPPRESS_S, ALERT_INDEX). A timer never reads the repo's .env, so without
+# this the tunables were settable in name only. The leading - means "carry on if it is absent".
+EnvironmentFile=-/etc/vanguard/alerts.env
+ExecStart=/usr/local/bin/vanguard-alerts.sh
+# Kibana cannot post outside the stack on a basic license, so its rules write into the vanguard-alerts
+# index and this is the delivery side. See bin/vanguard-alerts.sh and LOGGING.md.

systemd/vanguard-alerts.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Deliver Vanguard alerts every minute
+
+[Timer]
+OnBootSec=3min
+OnUnitActiveSec=1min
+AccuracySec=15s
+
+[Install]
+WantedBy=timers.target

systemd/vanguard-deadman.service

@@ -0,0 +1,10 @@
+[Unit]
+Description=Vanguard deadman: is the alerting stack itself alive, and are the disks ok
+After=docker.service
+
+[Service]
+Type=oneshot
+# Written by configure.py from .env (HEARTBEAT_MAX_S, ALERT_INDEX). A timer never reads the repo's .env, so without
+# this the tunables were settable in name only. The leading - means "carry on if it is absent".
+EnvironmentFile=-/etc/vanguard/alerts.env
+ExecStart=/usr/local/bin/vanguard-deadman.sh

systemd/vanguard-deadman.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run the Vanguard deadman every 5 minutes
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min
+AccuracySec=30s
+
+[Install]
+WantedBy=timers.target