Elastic security, our own metrics, and an alert pipeline that can actually reach Discord

Observability was running but could not report anything. This closes that end to end and
makes every piece of it reproducible from the installers instead of from a shell history.

Logs
- Filebeat 9 REMOVED the container input; it refuses the config per container and then runs
  with zero harvesters and no index. Container logs now use filestream + the container parser
  with a stable id each, and add_docker_metadata is told where the id is.
- Every line gets event.dataset, or Kibana's Logs rate chart is one "unknown" bucket.
- Beats pin replicas to 0: a single node cannot place one, so the cluster sat yellow forever.

Security
- xpack.security.enabled=true, because Kibana's Observability apps ask Elasticsearch whether
  the user holds a privilege and that API does not exist when security is off (the
  Infrastructure page failed on every view). Nobody sees a login: Kibana's anonymous provider
  signs every visitor in as one native user, and Caddy + Authentik + DIAG_REQUIRE_GROUP stays
  the only gate. configure.py generates the service passwords, creates the accounts and writes
  a root-only netrc for the deadman; --es-security retries just that step.

Alerting
- A basic licence has no webhook connector (403, Gold+), so Kibana's rules write documents into
  a vanguard-alerts index through the free .index connector and vanguard-alerts.timer delivers
  them to Discord. Alerts are therefore searchable beside the logs that caused them.
- Six rules in observability/kibana-alarms.py: disk and memory below 15% free, bot not running,
  bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every
  12 h while unresolved.
- The deadman watches both halves of that pipeline, since nothing inside the stack can report
  that the stack's own alerting died.

Our own metrics
- bin/vgmetrics.sh counters from the sweep and vgdeband, plus live gauges on dumpd /metrics
  (loopback only, unrouted by Caddy), scraped by Metricbeat. vanguard_media_pending is the
  measured answer to whether normalising keeps up, which was previously only a benchmark guess.
- The sweep now captures ffmpeg's stderr: rc= alone was not diagnosable.
- Its file loop reads a process substitution, not a pipe, or the subshell discards every counter.

Housekeeping
- configure.py --apply installs the media scripts (it never did, so applies left stale binaries),
  creates the alert index, injects an ILM delete phase and runs Kibana's first-run setup.
- The discord-bot container asks the bot about itself via /health rather than probing a port.
- Removed the dead diag landing page: Kibana occupies the root of that host.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitea/draft-workflows/ci.yml

@@ -0,0 +1,18 @@
+# DRAFT — inert while it lives in .gitea/draft-workflows/. Activate with:  git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml
+# Checks every push; builds + tests the Jellyfin plugin. Deploying is NOT done here (the runner must not touch
+# the host): the Gitea push webhook -> dumpd /deploy -> vanguard-deploy handles that, when DEPLOY_AUTO=1.
+name: ci
+on: [push, pull_request]
+jobs:
+  checks:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - run: sudo apt-get update -q && sudo apt-get install -y -q python3 python3-yaml nodejs npm ffmpeg jq
+      - run: bash check.sh
+  plugin:
+    runs-on: dotnet
+    steps:
+      - uses: actions/checkout@v4
+      - run: cd jellyfin-plugin-vanguard && dotnet test Jellyfin.Plugin.Vanguard.Tests --nologo -v q
+      - run: cd jellyfin-plugin-vanguard && dotnet build Jellyfin.Plugin.Vanguard -c Release --nologo -v q

.gitignore

@@ -9,3 +9,5 @@ mediamtx.yml
 copyparty.conf
 Caddyfile
 stream-token.secret
+deploy.env
+bot/app/

Caddyfile.example

@@ -1,29 +1,364 @@
-media.example.com {
+{
-	encode zstd gzip
+	servers {
-
+		metrics                           # Prometheus metrics at http://127.0.0.1:2019/metrics (LOGGING.md)
-	# dump endpoint (python service on the host, does its own auth)
-	handle /dump* {
-		reverse_proxy 127.0.0.1:8080
-	}
-
-	# Syncthing admin UI (its own login; strip the prefix, present a local Host to pass its host check)
-	redir /sync /sync/ 308
-	handle_path /sync/* {
-		reverse_proxy 127.0.0.1:8384 {
-			header_up Host {upstream_hostport}
-		}
-	}
-
-	# OBS browser dock (static)
-	handle_path /dock* {
-		root * /srv/dock
-		file_server
-	}
-
-	# everything else is copyparty (web UI + WebDAV)
-	handle {
-		reverse_proxy 127.0.0.1:3923 {
-			flush_interval -1
-		}
 	}
 }
+
+# vanguard:main
+media.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	# who does Authentik say I am? (debug: shows exactly the identity headers the apps receive)
+	handle /whoami {
+		header Content-Type text/plain
+		respond "user:   {http.request.header.X-Authentik-Username}
+groups: {http.request.header.X-Authentik-Groups}
+email:  {http.request.header.X-Authentik-Email}
+name:   {http.request.header.X-Authentik-Name}"
+	}
+	@me path /me /me/*
+	handle @me {
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity for the gateway page
+	}
+	redir /about* / 308                       # old bookmarks: the gateway page used to live at /about/
+	handle {
+		root * /srv/about
+		try_files {path} /index.html
+		file_server                           # the gateway page is the root of the main host
+	}
+}
+
+# vanguard:apex
+example.com {
+	redir https://media.example.com/ 302
+}
+
+# vanguard:files
+files.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	# copyparty at the root. It honours X-Authentik-Username / -Groups only when the request also carries a header
+	# whose secret NAME only this proxy knows (copyparty idp-h-key). Clients cannot forge identity.
+	reverse_proxy 127.0.0.1:3923 {
+		header_up X-Vanguard-EXAMPLEKEY 1
+		flush_interval -1
+	}
+}
+
+# vanguard:sync
+sync.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@lacks_group not header_regexp X-Authentik-Groups (^|\|)media\-admins(\||$)
+	handle @lacks_group {
+		respond "This page is for members of the 'media-admins' group." 403
+	}
+	# Syncthing admin UI (its own GUI login is disabled; the gate + group is the login)
+	reverse_proxy 127.0.0.1:8384 {
+		header_up Host {upstream_hostport}
+	}
+}
+
+# vanguard:clips
+clips.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@me path /me /me/*
+	handle @me {
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity + /me/clips self-service
+	}
+	handle {
+		root * /srv/clips
+		try_files {path} /index.html
+		file_server
+	}
+}
+
+# vanguard:diag
+diag.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@lacks_group not header_regexp X-Authentik-Groups (^|\|)media\-admins(\||$)
+	handle @lacks_group {
+		respond "This page is for members of the 'media-admins' group." 403
+	}
+	# VictoriaLogs runs with -http.pathPrefix=/logs and emits /logs/... URLs itself, so the prefix is
+	# passed through unchanged. Stripping it would break the UI's own asset paths.
+	redir /logs /logs/select/vmui/ 302
+	redir /logs/ /logs/select/vmui/ 302
+	handle /logs* {
+		reverse_proxy 127.0.0.1:9428
+	}
+	handle /metrics-db* {
+		reverse_proxy 127.0.0.1:8428          # VictoriaMetrics UI + query API
+	}
+	handle /alerts* {
+		reverse_proxy 127.0.0.1:9093          # Alertmanager: what is firing, silences
+	}
+	handle /rules-metrics* {
+		reverse_proxy 127.0.0.1:8880          # vmalert over VictoriaMetrics
+	}
+	handle /rules-logs* {
+		reverse_proxy 127.0.0.1:8881          # vmalert over VictoriaLogs
+	}
+	handle {
+		root * /srv/diag
+		try_files {path} /index.html
+		file_server                           # diag/index.html: what each path above is
+	}
+}
+
+# vanguard:live
+live.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	handle /whoami {
+		header Content-Type text/plain
+		respond "user:   {http.request.header.X-Authentik-Username}
+groups: {http.request.header.X-Authentik-Groups}"
+	}
+	@me path /me /me/*
+	handle @me {
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity, stream tokens (Go Live + dock)
+	}
+	handle /time {
+		reverse_proxy 127.0.0.1:8080          # server clock for the timecode strip (clock page + Go Live)
+	}
+	handle /dump* {
+		reverse_proxy 127.0.0.1:8080          # dump tonight (dock)
+	}
+	handle_path /dock* {
+		root * /srv/dock
+		file_server                           # OBS browser dock + /dock/clock/
+	}
+	handle_path /api/* {
+		reverse_proxy 127.0.0.1:9997
+	}
+	handle_path /webrtc/* {
+		# Go Live sends MediaMTX credentials in X-Stream-Auth so Authentik's forward-auth never sees an
+		# Authorization header; Caddy turns it into Authorization only on the hop to MediaMTX.
+		@creds header X-Stream-Auth *
+		reverse_proxy @creds 127.0.0.1:8889 {
+			header_up Authorization {http.request.header.X-Stream-Auth}
+			header_up -X-Stream-Auth
+		}
+		reverse_proxy 127.0.0.1:8889
+	}
+	handle_path /hls/* {
+		reverse_proxy 127.0.0.1:8888
+	}
+	handle {
+		root * /srv/live
+		try_files {path} /index.html
+		file_server
+	}
+}
+
+# vanguard:bot
+bot.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	# /logout on every gated host = sign out of the Discord login (domain-wide session)
+	redir /logout /outpost.goauthentik.io/sign_out 302
+	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
+	# whole domain (recommended) or single-application for this host — assigned to the embedded outpost.
+	# Until then: 4xx, never open. Per-host group requirements are enforced below from X-Authentik-Groups.
+	reverse_proxy /outpost.goauthentik.io/* 127.0.0.1:9000
+	forward_auth 127.0.0.1:9000 {
+		uri /outpost.goauthentik.io/auth/caddy
+		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Entitlements X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
+		trusted_proxies private_ranges
+	}
+	@lacks_group not header_regexp X-Authentik-Groups (^|\|)bot\-users(\||$)
+	handle @lacks_group {
+		respond "This page is for members of the 'bot-users' group." 403
+	}
+	reverse_proxy 127.0.0.1:8000
+}
+
+# vanguard:git
+git.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	reverse_proxy 127.0.0.1:3000
+}
+
+# vanguard:auth
+auth.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	reverse_proxy 127.0.0.1:9000
+}
+
+# vanguard:watch
+watch.example.com {
+	encode zstd gzip
+	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+	reverse_proxy 127.0.0.1:8096
+}

CommunityDiscordMedia.md

@@ -21,7 +21,7 @@ Bundle: `vanguard-stream/` in this repo; live copy at `~/vanguard-stream` on the
 | Authentik (Discord login, groups) at `auth.` | live |
 | Jellyfin at `watch.`, SSO via Authentik, direct play only | live |
 | Vanguard Metadata plugin v0.1 | installed, libraries being switched to Shows |
-| MariaDB (bot database) + `discord-bot` Python container (`bot/app`) | **in bundle, not yet applied** |
+| MariaDB (bot database) + `discord-bot` Python container (release installed in `/srv/bot/app`) | live; first release pending |
 | Gitea (`git.`, MariaDB, ssh 2222) + Actions runner | live (local admin account; Authentik OIDC → Phase 10) |
 
 ## Decisions already made
@@ -223,7 +223,8 @@ Checklist:
 
 ### Discord bot (separate dev) — runtime provided by the stack
 - `mariadb` container, db `vanguardbot` / user `bot`, loopback only (ssh tunnel for dev access)
-- `discord-bot` container: python 3.12 + ffmpeg + yt-dlp baseline; code bind-mounted from `bot/app/`;
+- `discord-bot` container: python 3.12 + ffmpeg + curl, and no bot dependencies — its CI builds a release
+  bundle (code + `vendor/` wheels) that dumpd installs into `/srv/bot/app`, bind-mounted at `/app`;
   env gives token, guild, DB creds, rw `/media/clips`, `/data` scratch; restarts on exit
 - web UI at `bot.` behind Caddy + Authentik forward-auth (first use of the forward-auth pattern that
   Phase 5 applies to copyparty); the bot trusts `X-Authentik-*` headers and implements no login
@@ -232,7 +233,7 @@ Checklist:
 - [ ] CNAME `bot` → main host; `configure.py --full` (asks the hostname, bot token + guild) → `docker compose up -d --build`
 - [ ] Authentik: domain-level Proxy Provider (covers live/bot/main); Caddy requires group `bot-users` on the bot host (`BOT_REQUIRE_GROUP`)
 - [ ] Discord roles → Authentik groups: `authentik/discord-roles-to-groups.py` on both source flows (scope `guilds.members.read`)
-- [ ] dev drops `main.py` into `bot/app/` → `docker compose restart discord-bot`
+- [x] the bot repo's CI builds a release bundle and a push to `main` installs it (signed, via dumpd `/deploy-bot`)
 
 ### Phase 9 — Gitea + Actions runner
 - `gitea` on MariaDB (db/user from `mariadb/init/10-gitea.sh` on first init, else `./createGiteaDb.sh`),
@@ -306,9 +307,10 @@ permanent reference is needed for long sessions, not just a start burst.
   frame-0 capture time = median(decoded − pts) at the start (frame-accurate) → QuickTime `tmcd` track
   (time of day, Eastern, file fps) + `creation_time` (UTC) in the existing lossless remux; a `.tc.json`
   sidecar with the samples records drift over the file. No strip → server arrival time (coarse), logged.
-  Never re-encode, never crop.
+  Strip present → the band is cropped off in a re-encode (decided 2026-11-01: finals = picture + audio + tmcd,
-- **Viewing**: live portal crops the band with CSS; Jellyfin shows a thin strip at the bottom edge (VHS-style).
+  nothing else); no strip → lossless remux, never crop.
-  Editors export at the game resolution (2560×1440), which drops the band.
+- **Viewing**: live portal crops the band on the live picture (canvas/overflow fix pending); recordings are
+  already at game resolution.
 - MediaMTX `recordPath` with `%f` (microseconds) for the fallback; plugin `ItemTimestamp` prefers `creation_time`.
 - [x] `/clock/` page: offset measurement, strip encoder, readable clock, band layout
 - [x] dock: overscan band setup + browser source via obs-websocket; opt-out
@@ -318,6 +320,43 @@ permanent reference is needed for long sessions, not just a start burst.
       export at 2560×1440)
 - rejected: LTC audio (SRT multitrack + per-PC generator); clock inside the picture; cropping at remux (re-encode)
 
+### Phase 13 — `diag.` host: Kibana (logs + metrics + alerting)  (built; needs a webhook + two Kibana rules)
+
+**Decided 2026-11-03, after one wrong turn.** First build was VictoriaLogs + Vector + VictoriaMetrics + two
+vmalert instances + Alertmanager, eight containers counting the exporters, and it produced **no dashboards** —
+five vendor UIs behind one hostname and an index page of links. The requirement was one tool with views, and
+the honest answers to that were Netdata (one container, weakest log search) or Elastic (four containers, one
+UI, strongest log search). Elastic was chosen. Grafana was considered and rejected *for this purpose*: it is a
+UI only, so its stack is Loki + Prometheus + Alloy + Grafana, which is the same piece count with more parts.
+
+Four containers: `elasticsearch` (both logs and metrics) · `kibana` (the whole `diag.` host, at the root) ·
+`filebeat` · `metricbeat` (which also retires node_exporter and cAdvisor). Detail in README §5h and LOGGING.md.
+
+- [x] compose: the four, all loopback, published only through the gated `diag.` host
+- [x] pinned: 2 GB heap, `vm.max_map_count`, security off (loopback + Authentik), fixed Kibana encryption key
+- [x] both Beats emit the same correlation fields, so one dashboard spans logs and metrics
+- [x] `configure.py`: `diag` role → Kibana at the root; `--remove-orphans` on apply so the old eight go away
+- [x] `install.sh`: the ES data dir, the sysctl, the deadman
+- [x] deadman repointed at Elastic, including the two failures a process check misses: a Beat that runs without
+      shipping, and indices flipped read-only by the watermark (which never clears itself)
+- [ ] a Discord webhook → `ALERT_WEBHOOK_URL`
+- [ ] in Kibana: the retention delete phase; the always-firing heartbeat rule → dumpd `/heartbeat`; a webhook
+      connector for alerts; then rules for the conditions the old vmalert files listed (disk, load, restart
+      loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble)
+- [ ] later: ops controls on this host (restart buttons) — allowlisted dumpd endpoints, authorised from the
+      Authentik headers, every invocation logged as a logfmt event. Never a page with Docker access.
+
+### Phase 14 — push-to-deploy  (built; OFF — `DEPLOY_AUTO=0`, CI workflow in draft-workflows/)
+- server folder = git clone (`switchToClone.sh`); generated files gitignored and preserved
+- `vanguard-deploy pull|apply|rollback|pending|status` applies only the changed parts; `check.sh` gates it;
+  MediaMTX restarts are deferred while anything is live (`deploy-pending.timer`)
+- Gitea push webhook → dumpd `/deploy` (bridge port, HMAC) → `vanguard-deploy pull` when `DEPLOY_AUTO=1`
+- CI (draft): `check.sh` + plugin tests/build on the runner; the runner never touches the host
+- [ ] run `switchToClone.sh key` / add deploy key / `switchToClone.sh clone` on the server
+- [ ] `./configure.py --apply` once (installs vanguard-deploy, deploy.env, the pending timer)
+- [ ] add the Gitea webhook (secret from `.env`); watch `journalctl -u dumpd | grep deploy` for a few pushes
+- [ ] when trusted: `DEPLOY_AUTO=1` in `.env` + `./configure.py --apply`; move `ci.yml` into `.gitea/workflows/`
+
 ### Option — more clip sources (Medal, Twitch/YouTube/Kick clips, Discord uploads)
 
 Today the only clip intake is a local clipper folder synced by Syncthing. Members without that
@@ -371,6 +410,7 @@ Checklist if adopted:
       Swap, not redesign: same compose, same `/srv` layout; restore `/srv/appdata` + `/srv/media`.
 - [ ] home archive puller (`home/rclone-archive.sh`) on the home box, nightly cron
 
+
 ## Risks / gotchas
 
 - **Phase 5 lockout.** Keep one SSH session open and a copyparty local admin account until the
@@ -382,6 +422,12 @@ Checklist if adopted:
 - **RAM**: Authentik ~1.5 GB, Jellyfin ~0.5 GB idle. Fine on 32 GB; check `free -h` after phase 1.
 - **nftables**: never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
 - **Firewall unchanged**: Authentik (9000) and Jellyfin (8096) bind to 127.0.0.1; only Caddy is public.
+- **A new host needs `--full` or `--hosts`, never `--apply`.** `configure.py` is stateless and reads hostnames
+  back out of the Caddyfile, so apply silently omits a role that is not already there and the name ends up with
+  no certificate (HOSTNAME-CHANGE.md §D). Cost an hour when `diag.` was added.
+- **Observability path prefixes cascade.** Setting `-http.pathPrefix` (or Alertmanager's `--web.route-prefix`)
+  moves that service's ingestion, health and metrics paths too, so the Vector sink, the scrape config and the
+  deadman's probes all have to follow. LOGGING.md lists the four traps this produced.
 
 ## Monthly cost
 Unchanged: the SYS-1 (~$30 + disks). No new services cost money. Discord OAuth is free.

FRESH-INSTALL.md

@@ -0,0 +1,373 @@
+# Fresh install — from an empty OVH box to a working stack with the deploy pipeline
+
+Everything in order. Scripted steps are commands; manual steps are click lists. `HOST` = your main hostname
+(ours: `media.dumpstervigilantes.rip`), `BASE` = the parent domain. Detailed background for each part is in
+README.md (section numbers given).
+
+## 0. Before touching the server
+- DNS: `A` record for `HOST` → the server IP; `CNAME`s `files`, `clips`, `sync`, `live`, `diag`, `auth`, `watch`, `git`, `bot` → `HOST`.
+  Optional: the bare domain `A` → server IP (it redirects to `https://HOST/`).
+- The Discord bot lives in its **own** repo (ours `VanguardOrg/VanguardBot`), not in this one. Nothing here
+  needs it to exist; section 7a wires it up afterwards.
+- Discord developer portal → New Application (one app serves Authentik login, the Jellyfin plugin's bot and the
+  Discord bot):
+  - OAuth2 → Redirects: `https://auth.BASE/source/oauth/callback/discord/` (the last segment is the Authentik
+    source slug — keep them identical). Copy Client ID + Client Secret.
+  - Bot → Reset Token (copy it); enable **Server Members** intent; OAuth2 URL generator: scope `bot`, permissions
+    View Channels + Send Messages + Embed Links + Attach Files → invite it to the server.
+  - Discord → User Settings → Advanced → Developer Mode on; right-click the server → Copy Server ID; Server
+    Settings → Roles → Copy Role ID for each role you will map to a group.
+- If this is a **rebuild**, have the backups ready: `.env`, `/srv/appdata`, `/srv/media`, `/etc/vanguard/clip-quotas.json`.
+  Restoring `.env` + `/srv/appdata` makes sections 5–7 unnecessary (that state lives there).
+
+## 1. OS (README §1)
+OVH manager → Reinstall → Debian 13, custom partitioning, distribution kernel. RAID 1:
+`/boot` 1 GB ext4 · `/` 250 GB ext4 · swap 8 GB per disk · `/srv` rest xfs. Add your SSH key. You get the `debian` user with sudo.
+
+## 2. Code onto the box
+Fresh box has no deploy key yet, so copy the repo in:
+
+    scp -r vanguard-stream debian@HOST:~/        # from a machine that has the repo (or git clone over https with a token)
+
+## 3. Host install (README §2, §4)
+
+    ssh debian@HOST
+    cd ~/vanguard-stream && sudo bash install.sh
+
+Packages, Docker, timezone (Eastern), uid 1000, `/srv` folders, host scripts and timers, sysctl, live-portal
+build, **firewall** (keep this session open; test a new ssh in a second terminal). Log out and back in once (docker group).
+
+## 4. Configure + start (README §3)
+
+    cd ~/vanguard-stream && ./configure.py --full
+
+Asks: main hostname (the gateway page); `files`/`clips`/`sync`/`diag`/`auth`/`watch`/`git`/`bot`/`live` hostnames (defaults); bare domain (optional) and where it
+redirects; Discord bot token + server ID (optional now, needed for the plugin feed); "is live" webhook (optional).
+Generates `.env` with every secret, the configs, starts all containers, waits, then runs the post-start fixups
+(Syncthing GUI auth off, Gitea admin user, Jellyfin plugin). `./configure.py --show` prints every credential
+you will need below (akadmin password, Gitea admin, MariaDB, bot DB).
+
+Check: `docker compose ps` → all `Up`; `https://auth.BASE/`, `https://watch.BASE/`, `https://git.BASE/`,
+`https://HOST/` answer with certificates (Caddy fetches them itself, give it a minute).
+
+## 5. Authentik (README §5c, §5g) — manual, ~20 minutes
+Log in at `https://auth.BASE/` as `akadmin` with `AUTHENTIK_BOOTSTRAP_PASSWORD` from `.env` (change it: user menu → settings).
+1. Directory → Federation & Social login → Create → **Discord OAuth Source**: name `Discord`, slug `discord`,
+   consumer key/secret = the Discord app's Client ID/Secret, scopes `guilds guilds.members.read`, user matching
+   "link on unique identifier", flows default-source-authentication / default-source-enrollment.
+2. Flows & Stages → Stages → `default-authentication-identification` → Sources: add Discord, tick "Show sources'
+   labels". (No Discord button appears until this is done.)
+3. Flows & Stages → Stages → `default-source-enrollment-write` → User type **Internal**, user path `users`.
+4. Directory → Groups: `streamers`, `editors`, `media-admins`, `admins`, `bot-users`, `git-users`, `git-admins`.
+   Nesting: `admins` parent `authentik Admins`; `authentik Admins` parent `git-admins` (admins inherit everything).
+   Inherited groups reach the apps only with the extra scope mapping `Vanguard profile (nested groups)` (README 5e),
+   ticked on the `Vanguard sites`, `Jellyfin` and `Gitea` providers next to the default profile mapping.
+5. Customization → Policies → Create Expression Policy ×2 from the repo:
+   `discord-guild-member` (`authentik/discord-guild-policy.py`, set your server ID) and
+   `discord-roles-to-groups` (`authentik/discord-roles-to-groups.py`, set server ID + role-id→group table).
+   Tick "Execution logging" on both.
+6. Flows & Stages → Flows → `default-source-enrollment` and `default-source-authentication`: bind both policies
+   (guild first), **Policy engine mode: all**.
+7. Applications → Outposts → edit "authentik Embedded Outpost" → config `authentik_host: https://auth.BASE`.
+8. Applications → Create with provider → `Vanguard sites`: **Proxy Provider**, implicit consent,
+   **Forward auth (domain level)**, authentication URL `https://auth.BASE`, cookie domain `BASE`,
+   invalidation flow `default-provider-invalidation-flow`. Edit the Embedded Outpost → add the application.
+9. Applications → Create with provider → `Jellyfin`: OAuth2/OpenID, implicit consent, Confidential, redirect
+   `https://watch.BASE/sso/OID/redirect/authentik`, default signing cert, scopes `openid email profile`. Keep ID + secret.
+10. Applications → Create with provider → `Gitea` (slug `gitea`): OAuth2/OpenID, same settings, redirect
+    `https://git.BASE/user/oauth2/authentik/callback`. Bind groups `git-users` + `git-admins` on the application.
+11. Test in a private window: `https://auth.BASE/` → Discord → your user exists and is Internal. Add yourself to
+    `admins` (and `streamers`, `media-admins` as needed) or give yourself the Discord roles; sign out and back in.
+    `https://HOST/whoami` shows your groups.
+
+## 6. Jellyfin (README §5c) — manual, ~10 minutes
+`https://watch.BASE/` → wizard: admin user; libraries `/media/streams` and `/media/clips` as **Shows** with
+metadata downloaders = only "Vanguard Metadata" (Series/Season/Episode), image fetchers Vanguard Metadata
+(Series) + Screen Grabber (Episode), real-time monitoring on; `/media/finals` as Home Videos, fetchers off.
+Dashboard → Playback → Transcoding: hardware acceleration none. Dashboard → Plugins: Vanguard Metadata is
+installed → its page: Discord bot token + server ID, feed channel ID, public URL `https://watch.BASE`.
+Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json`
+→ Catalog → SSO Authentication → install → `docker compose restart jellyfin`. Plugins → SSO-Auth → add OID provider
+`authentik`: endpoint `https://auth.BASE/application/o/jellyfin/`, client ID/secret, roles claim `groups`,
+admin roles `admins`, enabled folders = the 3 libraries, Scheme Override `https`, Port Override EMPTY, Save.
+Dashboard → General → Branding: paste `jellyfin/branding-disclaimer.html` (set the hostname) and `jellyfin/branding.css`.
+Test: `https://watch.BASE/sso/OID/start/authentik`. Scheduled Tasks → run "Vanguard: snapshot Discord events",
+"roll seasons", "enforce playback policy" once.
+
+## 7. Gitea (README §5e) — manual, ~10 minutes
+Log in at `https://git.BASE/` as `GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD` from `.env`.
+1. Site Administration → Identity & Access → Authentication Sources → Add: OAuth2, name `authentik`, OpenID
+   Connect, client ID/secret from step 5.10, discovery URL
+   `https://auth.BASE/application/o/gitea/.well-known/openid-configuration`, Skip local 2FA, group claim
+   `groups`, administrator group `git-admins`.
+2. Create the repo `techy/Media-Server-Setup` (empty) and push this repo to it from your dev box:
+   `git remote add origin ssh://git@git.BASE:2222/techy/Media-Server-Setup.git && git push -u origin main`
+   (your SSH key under Settings → SSH keys).
+3. Runner: `./registerGiteaRunner.sh` on the server → Site Administration → Actions → Runners shows it idle.
+
+## 7a. Discord bot (README 5d)
+1. The bot is **not** part of this repo. Its code arrives as a release bundle built by its own Gitea Actions
+   workflow and installed by `deployBotBundle.sh`; the image here carries no bot dependencies.
+2. Create the org + repo in Gitea (ours: `VanguardOrg/VanguardBot`) and let the dev push to it.
+3. The runner needs the **`python312`** label (CPython 3.12 + node, for the JavaScript actions). It is in
+   `docker-compose.yml`; labels are fixed at registration, so `./registerGiteaRunner.sh` after any change.
+4. In Gitea → the bot repo → Settings → Actions: secrets `BOT_DEPLOY_SECRET` (from `./configure.py --show`)
+   and `DISCORD_BOT_TOKEN`; variable `DISCORD_GUILD_ID`. **The bot's secrets live here, not in `.env`.**
+5. Authentik: a Proxy Provider for `https://bot.BASE` on the embedded outpost (same shape as `live.`).
+6. Push to `main` → CI tests, builds the bundle, posts it signed to dumpd, which installs it and restarts the
+   container. By hand: `./deployBotBundle.sh <bundle.zip>`; back out with `--rollback`.
+
+## 7b. Diagnostics — Kibana (README §5h)
+1. The `diag` CNAME must exist first. Four containers come up from `configure.py`: Elasticsearch (logs AND
+   metrics), Kibana (the whole `diag.BASE` host, at the root), Filebeat and Metricbeat. Nothing is published
+   except through the gate (`DIAG_REQUIRE_GROUP`, `media-admins`).
+2. `install.sh` and every apply set `vm.max_map_count=262144`. Elasticsearch will not start without it.
+3. `ALERT_WEBHOOK_URL` in `.env` is optional. Without it Kibana rules still evaluate and the deadman still
+   logs; only delivery is missing.
+4. Three first-run actions inside Kibana, all UI so they stay adjustable: the retention delete phase, the
+   always-firing heartbeat rule the deadman watches, and a webhook connector for alerts. README §5h lists them.
+5. Adding this host to a RUNNING server needs `--full` or `--hosts`, not `--apply` (HOSTNAME-CHANGE.md §D).
+
+## 8. Pipeline (README "Deployment")
+1. On the server: `./switchToClone.sh key` → add the printed key as a read-only **Deploy Key** on the repo
+   → `./switchToClone.sh clone` (keeps `.env`, generated configs, plugin zips) → `./configure.py --apply`.
+2. Gitea → repo → Settings → Webhooks → Add → Gitea: URL `http://<docker gateway>:8081/deploy`
+   (`docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}'`), secret =
+   `DEPLOY_WEBHOOK_SECRET` from `.env`, push events, branch `main`. Test delivery → 202 "auto-deploy disabled".
+3. Manual deploys from now on: push, then `sudo vanguard-deploy pull` on the server. When trusted:
+   `DEPLOY_AUTO=1` in `.env` → `./configure.py --apply`; CI: `git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml`.
+
+## 9. Streamers, gamers, editors (README §5, §5a, §6)
+- Streamers: OBS → WebSocket server on; Docks → Custom Browser Dock `https://live.BASE/dock/` → Discord login →
+  Settings: OBS WebSocket password; Timecode band → "Add band + clock source" (not streaming). Then title → Go Live.
+- Gamers: `https://clips.BASE/` → paste their Syncthing Device ID → accept device + folder on their PC (Send Only).
+  Syncthing UI (`https://sync.BASE/`, media-admins): Actions → Settings → Connections → incoming rate limit 12500 KiB/s.
+- Everyone: `https://HOST/` (the gateway page; editors browse at `https://files.BASE/`).
+
+## 10. Afterwards
+- Pin `AUTHENTIK_TAG` in `.env` to the running version. Rotate any secret that was ever pasted into chat.
+- Back up `.env`, `/srv/appdata`, `/srv/media` (home archive puller: README §7).
+- `STATUS.md` = what runs and what is unfinished; `check.sh` before any commit.
+
+---
+
+# Appendix — the runbook: every action, in order, with the exact commands
+
+Conventions: commands prefixed `dev$` run on your workstation (it has the repo at `~/claudevs/vanguard-stream`
+and your SSH key); `srv$` run on the server over ssh as `debian`. Replace `HOST` / `BASE` with your names
+(ours: `media.dumpstervigilantes.rip` / `dumpstervigilantes.rip`). Click paths are written `App → Menu → Item`.
+Every step ends with how you know it worked.
+
+## A. DNS and Discord (no server yet)
+
+A1. DNS at your registrar:
+    A      HOST            → <server IP>
+    CNAME  auth.BASE       → HOST
+    CNAME  watch.BASE      → HOST
+    CNAME  git.BASE        → HOST
+    CNAME  bot.BASE        → HOST
+    CNAME  live.BASE       → HOST
+    (optional) A  BASE     → <server IP>
+    Check from dev:  dev$ for h in HOST auth.BASE watch.BASE git.BASE bot.BASE live.BASE; do printf '%-28s ' $h; getent hosts $h | awk '{print $1}'; done
+    → every line shows the server IP.
+
+A2. Discord Developer Portal → New Application "Vanguard".
+    OAuth2 → Redirects → add  https://auth.BASE/source/oauth/callback/discord/  → Save. Copy Client ID and (Reset) Client Secret.
+    Bot → Reset Token → copy. Privileged Gateway Intents → Server Members Intent ON → Save.
+    OAuth2 → URL Generator → scopes: bot; permissions: View Channels, Send Messages, Embed Links, Attach Files → open the URL → invite the bot to your server.
+    → the bot appears in your server's member list.
+
+A3. Discord IDs: User Settings → Advanced → Developer Mode ON. Right-click your server → Copy Server ID.
+    Server Settings → Roles → right-click each role you'll map → Copy Role ID. Keep a note like:
+    guild=123…  streamers=…  editors=…  media-admins=…  admins=…  bot-users=…  git-users=…  git-admins=…
+
+## B. Operating system
+
+B1. OVH manager → the server → Reinstall → Debian 13 → custom partitioning, "use distribution kernel":
+    /boot 1 GiB ext4 (RAID1) · / 250 GiB ext4 (RAID1) · swap 8 GiB (per disk) · /srv remaining xfs (RAID1).
+    Add your SSH public key. Wait for the mail.
+B2. dev$ ssh debian@HOST 'lsblk -o NAME,SIZE,FSTYPE,MOUNTPOINT; df -h /srv'
+    → /srv is xfs, ~3.4 TiB, mounted.
+
+## C. Code onto the box
+
+C1. dev$ cd ~/claudevs && scp -r vanguard-stream debian@HOST:~/
+C2. dev$ ssh debian@HOST 'ls ~/vanguard-stream | head; test -f ~/vanguard-stream/install.sh && echo ok'
+    → "ok".
+
+## D. Host install
+
+D1. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && sudo bash install.sh'
+    (~5 min: packages, docker, timezone, folders, scripts, timers, portal build, firewall.) KEEP THIS TERMINAL OPEN.
+D2. In a SECOND terminal:  dev$ ssh debian@HOST 'echo still in'  → "still in" (firewall did not lock you out).
+D3. dev$ ssh debian@HOST 'timedatectl show -p Timezone --value; systemctl is-active docker dumpd normalize-sweep.timer trash-purge.timer nftables; docker --version; node --version; ls /srv/live | head -3'
+    → America/New_York; five "active"; versions; /srv/live has index.html.
+D4. Log out/in once for the docker group:  dev$ ssh debian@HOST 'docker ps >/dev/null && echo docker-ok'  → docker-ok.
+
+## E. Configure and start the stack
+
+E1. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./configure.py --full'
+    Answers: HOST; Enter ×8 for the sub-hostnames; bare domain (Enter = BASE, or blank to skip) and its redirect;
+    bot token (from A2) ; server ID (A3) ; "is live" webhook (optional); y to apply.
+    (Pulls ~2 GB of images, starts 13 containers, waits, runs post-start fixups.)
+E2. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}"'
+    → every container "Up" (authentik-server/worker take ~1 min; act_runner restarts until step G3 — fine).
+E3. dev$ for h in HOST auth.BASE watch.BASE git.BASE live.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done
+    → 200/302 for all (302 = redirected to login). If 000, wait a minute for certificates and retry.
+E4. dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./configure.py --show'
+    → copy out: akadmin password (AUTHENTIK_BOOTSTRAP_PASSWORD), Gitea admin user/password, MariaDB root + bot + gitea passwords.
+E5. dev$ ssh debian@HOST 'journalctl -u dumpd -n 2 --no-pager | tail -1; grep -c "authMethod: http" ~/vanguard-stream/mediamtx.yml; sudo nft list table inet vanguard | grep -c dport'
+    → dumpd line ending "token-secret=ok"; 1; a dozen rules.
+
+## F. Authentik (browser, logged in as akadmin at https://auth.BASE/ with the E4 password)
+
+F1. Change akadmin's password: top-right user → Settings → Change password.
+F2. Directory → Federation & Social login → Create → "Discord OAuth Source":
+    Name Discord · Slug discord · Enabled · User matching "Link users on unique identifier" ·
+    Consumer key = Client ID · Consumer secret = Client Secret · Scopes: guilds guilds.members.read ·
+    (Flow settings if shown: authentication default-source-authentication, enrollment default-source-enrollment) → Finish.
+F3. Flows & Stages → Stages → default-authentication-identification → Edit → Sources: select Discord; tick "Show sources' labels" → Update.
+    → the login page (private window) now shows a Discord button.
+F4. Flows & Stages → Stages → default-source-enrollment-write → Edit → User type: Internal · User path: users → Update.
+F5. Directory → Groups → Create, one each: streamers, editors, media-admins, admins, bot-users, git-users, git-admins.
+    Then nesting: Groups → admins → Edit → Parent: authentik Admins → Update. Groups → authentik Admins → Edit → Parent: git-admins → Update.
+F5b. Customization → Property Mappings → Create → Scope Mapping: Name `Vanguard profile (nested groups)`, Scope name `profile`, Expression:
+        direct = set(request.user.ak_groups.all())
+        return {"groups": sorted(g.name for g in request.user.all_groups() if g not in direct)}
+     → Finish. Later, on each provider created in F7–F9 (Vanguard sites, Jellyfin, Gitea) → Edit → Advanced protocol settings → Scopes: tick it
+     next to the default profile mapping (proxy providers re-add the default on save; that is fine, the two merge). Without it `/whoami` never shows git-admins.
+F6. Customization → Policies → Create → Expression Policy:
+    name discord-guild-member · expression = contents of authentik/discord-guild-policy.py with ACCEPTED_GUILD_ID + GUILD_NAME filled in · tick Execution logging → Create.
+    Again: name discord-roles-to-groups · expression = authentik/discord-roles-to-groups.py with GUILD_ID + ROLE_TO_GROUP filled from A3 · Execution logging → Create.
+F7. Flows & Stages → Flows → default-source-enrollment → Policy / Group / User Bindings → Bind existing policy → discord-guild-member (order 0) → Create;
+    again → discord-roles-to-groups (order 10) → Create. Then Edit the flow → Policy engine mode: all → Update.
+    Repeat all of F7 for default-source-authentication.
+F8. Applications → Outposts → "authentik Embedded Outpost" → Edit → in the configuration box set
+    authentik_host: https://auth.BASE  (leave authentik_host_browser empty) → Update.
+F9. Applications → Create with provider:
+    Application name "Vanguard sites" · Provider type Proxy Provider · Authorization flow default-provider-authorization-implicit-consent ·
+    Mode "Forward auth (domain level)" · Authentication URL https://auth.BASE · Cookie domain BASE ·
+    Invalidation flow default-provider-invalidation-flow → Finish.
+    Applications → Outposts → Embedded Outpost → Edit → Applications: add "Vanguard sites" → Update.
+F10. Applications → Create with provider: "Jellyfin" (slug jellyfin) · OAuth2/OpenID Provider · implicit-consent ·
+     Client type Confidential · Redirect URI (strict) https://watch.BASE/sso/OID/redirect/authentik ·
+     Signing key: the default certificate · Advanced → Scopes: openid, email, profile → Finish. Note Client ID + Secret.
+F11. Applications → Create with provider: "Gitea" (slug gitea) · same provider settings · Redirect URI https://git.BASE/user/oauth2/authentik/callback → Finish.
+     Note Client ID + Secret. Applications → Gitea → Policy / Group / User Bindings → Bind existing group → git-users → Create; again → git-admins.
+F12. Private window → https://auth.BASE/ → Discord → log in. Admin: Directory → Users → your user exists, type Internal.
+     Directory → Groups → admins → Users tab → Add existing user → you (and media-admins, streamers as you like) — or give yourself the Discord roles.
+F13. Private window → https://HOST/whoami → "user: <you>" and "groups: …" listing what you added. (If groups are missing: sign out at https://auth.BASE/if/flow/default-invalidation-flow/ and log in again.)
+     Events → Logs → filter policy_execution → entries for both policies with result pass.
+
+## G. Gitea (browser at https://git.BASE/, log in with GITEA_ADMIN_USER / GITEA_ADMIN_PASSWORD from E4)
+
+G1. Site Administration → Identity & Access → Authentication Sources → Add Authentication Source:
+    Type OAuth2 · Name authentik · Provider OpenID Connect · Client ID / Secret from F11 ·
+    OpenID Connect Auto Discovery URL https://auth.BASE/application/o/gitea/.well-known/openid-configuration ·
+    Skip local 2FA ✓ · Additional scopes (blank) · Claim name providing group names: groups · Group claim value for administrator users: git-admins → Add.
+G2. New repository (top-right +) → owner techy → name Media-Server-Setup → NOT initialized (no README) → Create.
+    Your SSH key: Settings → SSH / GPG Keys → Add Key (paste ~/.ssh/id_ed25519.pub from dev).
+G3. dev$ cd ~/claudevs/vanguard-stream && git remote add origin ssh://git@git.BASE:2222/techy/Media-Server-Setup.git && git push -u origin main
+    (accept the host key for port 2222) → "branch 'main' set up to track".
+G4. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./registerGiteaRunner.sh'
+    → log lines "runner: connect…" ; Site Administration → Actions → Runners lists vanguard-runner, Idle.
+G5. Private window → https://git.BASE/user/login → "Sign in with authentik" → Discord → logged in as your Discord username.
+    (Requires git-users or git-admins membership; admins inherit git-admins via F5.)
+
+## H. Jellyfin (browser at https://watch.BASE/)
+
+H1. Wizard: language → admin user + password → Add Media Library:
+    Content type Shows · Display name Streams · Folder /media/streams · Metadata downloaders (Series, Season, Episode): tick ONLY "Vanguard Metadata" ·
+    Image fetchers: Series: Vanguard Metadata; Episode: Screen Grabber · Enable real time monitoring ✓ → OK.
+    Add Media Library: Shows · Clips · /media/clips · same provider settings → OK.
+    Add Media Library: Home Videos and Photos · Finals · /media/finals · all fetchers unticked → OK.
+    Finish the wizard (metadata language, remote access defaults).
+H2. Dashboard → Playback → Transcoding → Hardware acceleration: None → Save.
+H3. Dashboard → Plugins → My Plugins → Vanguard Metadata → settings: Discord bot token (A2), Server ID (A3), feed channel ID
+    (right-click the Discord channel → Copy Channel ID), Public URL https://watch.BASE, Authentik URL https://auth.BASE (token optional) → Save.
+H4. Dashboard → Plugins → Repositories → + → Name SSO-Auth · URL https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json → Save.
+    Catalog → SSO Authentication → Install. Then:  dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose restart jellyfin'
+H5. Dashboard → Plugins → SSO-Auth → OpenID: Add provider: name authentik · OID Endpoint https://auth.BASE/application/o/jellyfin/ ·
+    Client ID / Secret from F10 · Enabled ✓ · Enable Authorization by Plugin ✓ · Roles claim: groups · Admin roles: admins ·
+    Enabled folders: Streams, Clips, Finals · Scheme Override: https · Port Override: EMPTY · (default: disable transcoding ✓) → Save (bottom of page).
+H6. Private window → https://watch.BASE/sso/OID/start/authentik → Discord → lands in Jellyfin logged in.
+H7. Dashboard → General → Branding → Login disclaimer: paste jellyfin/branding-disclaimer.html with watch.HOST replaced by watch.BASE;
+    Custom CSS: paste jellyfin/branding.css → Save. Login page now shows "Sign in with Discord" + a small "Admin login" link.
+H8. Dashboard → Scheduled Tasks → Vanguard: run "snapshot Discord events", then "roll seasons", then "enforce playback policy" (▶ each).
+    Dashboard → Users → your SSO user → Playback → "Allow video playback that requires transcoding" is unticked.
+
+## I. Pipeline
+
+I1. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh key'
+    → prints a public key. Gitea → repo → Settings → Deploy Keys → Add Deploy Key → paste → read-only → Add.
+I2. dev$ ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh clone && cd ~/vanguard-stream && ./configure.py --apply && sudo vanguard-deploy status'
+    → "cloned. kept: …" ; status shows "main @ <commit>", DEPLOY_AUTO=0.
+I3. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker network inspect vanguard-stream_default -f "{{(index .IPAM.Config 0).Gateway}}"; grep ^DEPLOY_WEBHOOK_SECRET= .env'
+    → gateway IP (e.g. 172.18.0.1) and the secret.
+I4. Gitea → repo → Settings → Webhooks → Add Webhook → Gitea: Target URL http://<gateway>:8081/deploy · HTTP POST · Content type JSON ·
+    Secret = the value from I3 · Trigger on Push events · Branch filter main · Active → Add Webhook. Open it → Test Delivery.
+    dev$ ssh debian@HOST 'journalctl -u dumpd --since -5m --no-pager | grep "deploy webhook"'  → "… auto-deploy disabled (DEPLOY_AUTO=0)".
+I5. From now on, after a push:  dev$ ssh -t debian@HOST 'sudo vanguard-deploy pull'  (shows the changed files and every action).
+I6. (later, when trusted) dev$ ssh debian@HOST 'cd ~/vanguard-stream && sed -i "s/^DEPLOY_AUTO=.*/DEPLOY_AUTO=1/" .env && ./configure.py --apply >/dev/null && sudo vanguard-deploy status'
+    and  dev$ cd ~/claudevs/vanguard-stream && git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml && git commit -m "enable CI" && git push
+
+## J. People
+
+J1. Streamer (OBS): Tools → WebSocket Server Settings → Enable, set a password → OK.
+    Docks → Custom Browser Docks → name Vanguard, URL https://live.BASE/dock/ → Apply → log in with Discord inside the dock.
+    Dock → Settings: OBS WebSocket password → Save. Dock → Timecode band → "Add band + clock source" (while not streaming).
+    Dock: title → Go Live.  Check:  dev$ ssh debian@HOST 'journalctl -u dumpd --since -5m --no-pager | grep mtx-auth | tail -1'  → "allow publish user=…".
+J2. Gamer: https://clips.BASE/ → follow the three steps on the page (install Syncthing, paste Device ID, accept device + folder as Send Only).
+    Admin check: https://sync.BASE/ shows their device connected. Once: Syncthing → Actions → Settings → Connections → Incoming Rate Limit 12500 KiB/s.
+J3. Editor: https://files.BASE/ (copyparty) → browse, download; https://files.BASE/finals/ to upload. Needs the editors role.
+J4. Everyone: https://HOST/ (gateway page).
+
+## K. Verify the whole pipeline once
+
+K1. Stream 1 minute from OBS with the band, stop, wait ~8 min.
+K2. dev$ ssh debian@HOST 'f=$(ls -t /srv/media/streams/*/*.mp4 | head -1); echo "$f"; ffprobe -v error -show_entries stream_tags=timecode -of compact=nk=0:p=0 "$f" | head -1; python3 -c "import json,sys; d=json.load(open(sys.argv[1])); print({k:d.get(k) for k in (\"source\",\"timecode\",\"drift_ms\")}, \"gaps:\", d.get(\"gaps\",{}).get(\"count\"))" "${f%.*}.json"'
+    → a flat file named <Title>__<timestamp>.mp4, tag:timecode set, source "strip", gaps 0.
+K3. https://watch.BASE/ → the recording is an episode under this weekday's season; Discord feed channel got a post with a frame grab.
+K4. https://live.BASE/ while someone streams → a window with the corner chip "webrtc".
+
+## L. Discord bot (first release)
+
+L1. Gitea → New Organization `VanguardOrg` → New Repository `VanguardBot` (private). The bot dev pushes to it.
+L2. dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./configure.py --show | grep -A2 "bot ci"'
+    → copy BOT_DEPLOY_SECRET and the list of keys CI may set.
+L3. Gitea → VanguardBot → Settings → Actions → Secrets → Add: `BOT_DEPLOY_SECRET` (L2), `DISCORD_BOT_TOKEN` (A2).
+    Variables → Add: `DISCORD_GUILD_ID` (A3). Optional `VANGUARD_DEPLOY_HOST` = the docker gateway from I3 if
+    the deploy job cannot detect it.
+L4. Check the runner advertises the python312 label:
+    dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./registerGiteaRunner.sh'   (only if it does not)
+    Site Administration → Actions → Runners → vanguard-runner → labels include python312.
+L5. Authentik → Applications → Create with provider → Proxy Provider, Forward auth (single application),
+    external host `https://bot.BASE` → add to the Embedded Outpost → bind group `bot-users`.
+L6. The dev pushes to `main`. The run should end with the install log, "settings written: DISCORD_BOT_TOKEN,
+    DISCORD_GUILD_ID" and HTTP 200.
+    dev$ ssh debian@HOST 'cd ~/vanguard-stream && ./deployBotBundle.sh --status'
+    → version, commit, package count, container Up.
+L7. dev$ ssh debian@HOST 'docker logs discord-bot --tail 20'
+    → the bot's version line, then its MariaDB connection. First real MariaDB run: watch for a retry loop.
+L8. Private window → https://bot.BASE/ → Discord login → the bot's pages. Needs the `bot-users` group (admins
+    need it too, or set BOT_REQUIRE_GROUP= and let the bot's WEB_ALLOWED_GROUPS decide).
+
+## M. Diagnostics (browser at https://diag.BASE/)
+
+M1. dev$ ssh debian@HOST 'cd ~/vanguard-stream && docker compose ps --format "{{.Name}} {{.Status}}" | grep -E "elastic|kibana|beat"'
+    → four Up, elasticsearch healthy. Elasticsearch restarting is almost always vm.max_map_count or the heap.
+M2. https://diag.BASE/ → Discord login → Kibana. Discover shows logs within a minute of the Beats starting.
+M3. Stack Management → Data Views → create one over `filebeat-*,metricbeat-*`, so a single filter spans logs
+    and metrics. Metricbeat also installed its own system and Docker dashboards: start from those.
+M4. Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase (30 days is sensible).
+M5. Alerting → Connectors → Webhook → the URL from ALERT_WEBHOOK_URL. Then Rules → create the heartbeat rule:
+    always true, every minute, action = webhook POST to http://<docker gateway>:8081/heartbeat
+    (gateway: docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}').
+M6. dev$ ssh debian@HOST 'sudo /usr/local/bin/vanguard-deadman.sh; echo exit=$?'
+    → silent and exit 0 once M5 is done. Before that it correctly reports the heartbeat missing.
+
+## N. Afterwards
+
+N1. dev$ ssh debian@HOST 'docker inspect authentik-server --format "{{.Config.Image}}"'  → put that tag in .env as AUTHENTIK_TAG, then ./configure.py --apply.
+N2. Rotate anything pasted into a chat or screenshot (Discord client secret: portal → Reset → Authentik source → update).
+N3. Backups: .env, /srv/appdata, /srv/media, /etc/vanguard/clip-quotas.json. Home archive puller: README §7.

HOSTNAME-CHANGE.md

@@ -0,0 +1,153 @@
+# Changing hostnames safely
+
+Two cases: renaming one subdomain (e.g. `watch.BASE` → `video.BASE`), and moving the whole stack to a new base
+domain. Both are mostly `configure.py --full` plus a known list of places outside the repo that hold the name.
+Do it between streams: the main host is MediaMTX's WebRTC advertised host, so a main-host change needs a
+MediaMTX restart.
+
+Roles: `main` (gateway page at `/`), `files` (copyparty), `clips`, `sync`, `live` (portal + `/dock/`, `/dock/clock/`,
+`/dump`, `/me`, `/time`), `auth`, `watch`, `git`, `bot`. Ours: `media.`, `files.`, `clips.`, `sync.`, `live.`, … under
+`dumpstervigilantes.rip`.
+
+## Where hostnames live
+
+### Generated on the server by `configure.py` (regenerated by `--full`, nothing to edit by hand)
+| Place | Holds |
+|---|---|
+| `Caddyfile` | every site block — the source of truth `configure.py` reads back (`# vanguard:<role>` markers) |
+| `.env` | `MAIN_HOST`, `FILES_HOST`, `CLIPS_HOST`, `SYNC_HOST`, `DIAG_HOST`, `WATCH_HOST`, `GIT_HOST`, `BOT_HOST`, `LIVE_HOST`, `APEX_HOST`, `APEX_REDIRECT` |
+| `mediamtx.yml` | `webrtcAdditionalHosts: [MAIN_HOST]` (needs a MediaMTX restart to take effect) |
+| `docker-compose.yml` → container env | Jellyfin `PublishedServerUrl` (WATCH), Gitea `DOMAIN/ROOT_URL/SSH_DOMAIN` (GIT), bot `WEB_PUBLIC_URL` (BOT), MediaMTX `LIVE_HOST` (text of the "is live" post), act_runner `GITEA_INSTANCE_URL` (GIT). `compose up -d` in the apply recreates the affected containers (seconds each) |
+| Caddy TLS | certificates for new names are fetched automatically; old ones linger harmlessly |
+
+### Pages in the repo that derive sibling hosts from the browser's location (label names hard-coded)
+| File | Assumes |
+|---|---|
+| `about/index.html` | `files.`, `clips.`, `sync.`, `live.`, `auth.`, `watch.`, `git.`, `bot.` + the parent of the current host |
+| `dock/index.html` | `auth.` (sign-out), ingest host prefilled from the current host (`live.`; any name of the box works for RTMP/SRT) |
+| `switchToClone.sh`, `deployLive.sh` | defaults for `git.dumpstervigilantes.rip` / `media.dumpstervigilantes.rip` (override by env/arg, or edit) |
+| `README.md`, `FRESH-INSTALL.md` | examples only |
+Renaming a *label* (e.g. `watch` → `video`) means editing the first two; changing only the domain does not.
+
+### Outside the repo — must be changed by hand
+| System | Setting | Which host |
+|---|---|---|
+| DNS | A record + CNAMEs | all |
+| Discord developer portal | OAuth2 redirect `https://AUTH/source/oauth/callback/discord/` | auth |
+| Authentik | Embedded Outpost `authentik_host` | auth |
+| Authentik | `Vanguard sites` proxy provider: authentication URL (auth) and **cookie domain** (base) | auth, base |
+| Authentik | `Jellyfin` provider redirect URI `https://WATCH/sso/OID/redirect/authentik` | watch |
+| Authentik | `Gitea` provider redirect URI `https://GIT/user/oauth2/authentik/callback` | git |
+| Jellyfin | SSO-Auth plugin: OID endpoint `https://AUTH/application/o/jellyfin/` | auth |
+| Jellyfin | Branding login disclaimer (the form action contains the watch host) | watch |
+| Jellyfin | Vanguard Metadata plugin: Public URL, Authentik URL | watch, auth |
+| Gitea | Authentication source `authentik`: discovery URL `https://AUTH/application/o/gitea/.well-known/openid-configuration` | auth |
+| Gitea runner | registered with the instance URL → `./registerGiteaRunner.sh` again | git |
+| Server clone | `git remote set-url origin ssh://git@GIT:2222/techy/Media-Server-Setup.git`; `~/.ssh/config.vanguard` Host line | git |
+| Dev box | `git remote set-url origin …` | git |
+| Gitea, the bot repo | nothing for a rename; `VANGUARD_DEPLOY_HOST` only if it was set to an address | — |
+| Streamers' OBS | dock URL (Custom Browser Docks), dock Settings → ingest host (re-save), clock browser source URL (dock → Timecode band → "Add band + clock source" again), any manual RTMP/SRT configs (copy from the dock) | live |
+| Jellyfin apps on phones/TVs | server address | watch |
+| Bookmarks, Discord pinned links | main (gateway), clips, files, watch, live | all |
+Not affected: Syncthing pairings (device IDs), MariaDB, stream tokens, the Gitea webhook (gateway IP), Authentik users/groups/policies.
+
+## A. Renaming one subdomain (example: `watch.BASE` → `video.BASE`)
+
+1. DNS: add `CNAME video.BASE → HOST`. Keep the old record until step 7.
+2. Server, between streams: `./configure.py --full` → Enter keeps every value; type the new name at the
+   "Jellyfin hostname" prompt; `y`. Caddy reloads (new block, old gone), Jellyfin is recreated with the new
+   published URL. `curl -sI https://video.BASE/ | head -1` → 200/302 once the certificate is issued.
+   (Only the *main* host touches `mediamtx.yml`; other hosts need no MediaMTX restart.)
+3. Authentik: Applications → Providers → `Jellyfin` → Edit → redirect URI → the new host → Update.
+4. Jellyfin: Dashboard → General → Branding → disclaimer form action → new host → Save.
+   Dashboard → Plugins → Vanguard Metadata → Public URL → new host → Save.
+5. If the *label* changed (`watch`→`video`), edit `about/index.html` (`sub('watch')` → `sub('video')`) — commit, push,
+   `sudo vanguard-deploy pull`. (`.env APEX_REDIRECT` too if the bare domain pointed at it: `./configure.py --full`.)
+6. Private window: `https://video.BASE/` → Discord → in. The gateway `https://HOST/` links to the new name.
+7. Remove the old DNS record. Tell members with Jellyfin apps to update the server address.
+
+Per-host extras when it is a different host:
+- **auth**: Discord portal redirect; Authentik outpost `authentik_host` and the proxy provider's authentication URL;
+  Jellyfin SSO plugin OID endpoint; Vanguard plugin Authentik URL; Gitea auth source discovery URL;
+  `dock/index.html` + `about/index.html` if the label changed. Everyone re-logs in.
+- **git**: Authentik Gitea provider redirect; `./registerGiteaRunner.sh`; server clone remote + `~/.ssh/config.vanguard`;
+  dev remote; `switchToClone.sh` default.
+- **live**, **bot**: only the compose env (automatic) and `about/index.html` if the label changed.
+- **main (HOST)**: `mediamtx.yml` changes → `docker compose restart mediamtx` between streams; every streamer
+  re-adds the dock URL, re-saves Settings (ingest host), and re-runs "Add band + clock source"; manual RTMP/SRT
+  configs are re-copied from the dock; `deployLive.sh` default host.
+
+## B. Changing the base domain (everything at once)
+
+Plan a maintenance window: Caddy, Jellyfin, Gitea, the bot and MediaMTX all restart, and every member logs in again
+(the Authentik session cookie is bound to the base domain).
+
+1. DNS on the new domain: `A HOST2 → IP`, CNAMEs `files`, `clips`, `sync`, `live`, `auth`, `watch`, `git`, `bot` → `HOST2` (optional bare domain A).
+   Verify they resolve before continuing. Leave the old domain's records in place for now.
+2. Discord developer portal → OAuth2 → add redirect `https://auth.NEW/source/oauth/callback/discord/` (keep the old one until step 9).
+3. Server, between streams: `./configure.py --full` → new main hostname; the eight sub-hostnames default to the new
+   base (Enter); bare domain; `y`. Then `docker compose restart mediamtx` (webrtcAdditionalHosts changed).
+   All sites answer on the new names within a minute (certificates).
+4. Authentik (log in at the NEW auth host with akadmin — Discord login is broken until step 5 is done):
+   - Outposts → Embedded Outpost → `authentik_host: https://auth.NEW`
+   - Providers → `Vanguard sites` → authentication URL `https://auth.NEW`, **cookie domain `NEW`**
+   - Providers → `Jellyfin` → redirect `https://watch.NEW/sso/OID/redirect/authentik`
+   - Providers → `Gitea` → redirect `https://git.NEW/user/oauth2/authentik/callback`
+5. Private window: `https://auth.NEW/` → Discord login works (the redirect from step 2 is used).
+6. Jellyfin: SSO plugin OID endpoint `https://auth.NEW/application/o/jellyfin/`; Branding disclaimer form action
+   `https://watch.NEW/sso/OID/start/authentik`; Vanguard plugin Public URL + Authentik URL. Test the SSO start URL.
+7. Gitea: auth source discovery URL `https://auth.NEW/application/o/gitea/.well-known/openid-configuration`;
+   `./registerGiteaRunner.sh`; server: `cd ~/vanguard-stream && git remote set-url origin ssh://git@git.NEW:2222/techy/Media-Server-Setup.git`
+   and edit `~/.ssh/config.vanguard` (Host line); dev box: `git remote set-url origin …`. Test `sudo vanguard-deploy pull`.
+8. Repo defaults: `switchToClone.sh` (REPO_SSH), `deployLive.sh` (HOST) — commit, push, pull.
+9. Streamers: re-add the dock at `https://live.NEW/dock/`, re-save Settings, "Add band + clock source" again.
+   Members: `https://HOST2/`; Jellyfin apps → new server address. Remove the old Discord redirect and the old DNS.
+
+## C. Splitting the single main host into media / files / clips / sync (done 2026-11-01)
+
+Before: one main host (`ovh1.BASE`) served copyparty at `/`, `/about/`, `/clips/`, `/sync/`, `/dock/`, `/dump`, `/time`.
+After: `media.BASE` = gateway at `/`; `files.BASE` = copyparty; `clips.BASE`; `sync.BASE`; `live.BASE` gains `/dock/`,
+`/dock/clock/`, `/dump`, `/me`, `/time`, `/whoami`. `/time` is only on `live` because the two pages that fetch it
+(the clock source and Go Live) are same-origin there. Authentik needs nothing: the domain-level provider covers every
+name under BASE, the outpost host (`auth.`) is unchanged.
+
+1. DNS: CNAMEs `media`, `files`, `clips`, `sync` → the box (keep `ovh1` for ssh; it is no longer a web host).
+2. Dev box: pull/upload the repo (`configure.py`, `about/`, `addCommunityMediaServer.sh`, docs).
+3. Server, between streams: `./configure.py --full` → main hostname `media.BASE`; Enter for the sub-hostnames
+   (defaults `files.`, `clips.`, `sync.`, …); bare domain redirect → `https://media.BASE/`; `y`.
+   Then `docker compose restart mediamtx` (webrtcAdditionalHosts is now `media.`) and `./deployLive.sh` (portal
+   build picks up nothing new, but confirms `/srv/live` is served on `live.`).
+4. Streamers: OBS → Custom Browser Docks → replace the dock URL with `https://live.BASE/dock/`; dock Settings →
+   save; Timecode band → "Add band + clock source" again (the clock browser source URL moved).
+5. Discord: re-pin `https://media.BASE/` (members), `https://clips.BASE/` (gamers), `https://files.BASE/` (editors).
+6. Old `ovh1` bookmarks stop working (no Caddy block, TLS fails) — point people at the gateway.
+
+## D. Adding a NEW host
+
+**`./configure.py --apply` cannot do this, and will not complain.** It is stateless by design: it reads the
+hostnames back out of the `Caddyfile` (`# vanguard:<role>` markers), so a role that is not in there yet has no
+value to write and the site is simply left out. The containers come up, Caddy reloads, and the new name has no
+site block and therefore no certificate — which presents as an SSL handshake error, not as an error from the
+apply. This cost an hour on 2026-11-03 when `diag.` was added.
+
+Introduce a new host with **`--full`** (asks for every hostname, defaults from the current config) or with
+**`--hosts`**, which is the non-interactive form and takes them positionally:
+
+    ./configure.py --hosts AUTH WATCH GIT BOT LIVE FILES SYNC CLIPS DIAG     # "" leaves one unset
+
+After that, `--apply` maintains it like any other host, because the marker is now in the file.
+
+1. DNS: `CNAME` (or `A`) for the new name → the box; check it resolves before continuing.
+2. Add the role to `configure.py`: a `write_caddy` block, a `<ROLE>_REQUIRE_GROUP` default in `write_env`,
+   the prompt in `full()`, the positional in `--hosts`, and the line in `show()`.
+3. `./configure.py --hosts …` with every hostname including the new one.
+4. Verify the marker landed: `grep -c "vanguard:<role>" Caddyfile` → 1. Then `curl -sI https://NEW/` → 302.
+5. Add it to `addCommunityMediaServer.sh` so a rebuild asks for it too.
+
+## Checklist after any change
+    dev$ for h in HOST files.BASE clips.BASE sync.BASE live.BASE diag.BASE auth.BASE watch.BASE git.BASE bot.BASE; do printf '%-26s ' $h; curl -s -o /dev/null -m 10 -w '%{http_code}\n' https://$h/; done
+    https://HOST/whoami            → your identity (gate works on the new name)
+    https://watch.BASE/sso/OID/start/authentik   → Jellyfin SSO round trip
+    https://git.BASE/user/login    → "Sign in with authentik" round trip
+    dock → Go Live → journalctl -u dumpd | grep mtx-auth   → allow publish
+    live.BASE → a window appears for the test stream, chip "webrtc"

LOGGING.md

@@ -0,0 +1,312 @@
+# Logging framework
+
+Decided 2026-11-01. The store was chosen, built and replaced on 2026-11-03: a VictoriaLogs stack went in
+first and was torn out the same day for producing no dashboards, so the store is now **Elasticsearch with
+Kibana**, served at the root of the `diag.BASE` host (Phase 13 in `CommunityDiscordMedia.md`). Logs arrive via
+Filebeat, metrics via Metricbeat, into the same cluster so one dashboard can carry both. Everything upstream
+of the store was fixed first, so the swap was a collector config rather than a code change.
+
+## Where the lines come from (the only two taps a collector needs)
+
+| Tap | Holds | Read with |
+|---|---|---|
+| **journald** | our systemd units: `dumpd`, `normalize-sweep`, `trash-purge`, `deploy-pending`, `vgdeband` (transient), `deploy.sh` output | `journalctl -u <unit>`; collectors: Alloy `loki.source.journal`, Vector `journald`, VictoriaLogs journald ingestion, Netdata's journal plugin |
+| **Docker json-file logs** | every container (`docker-compose.yml`), rotated 5 × 50 MB each | `docker logs <name>`; collectors read `/var/lib/docker/containers/*/*-json.log` and get the container **labels** below for free |
+
+Filebeat reads both: a docker autodiscover template per container (filestream + the container parser) and a
+journald input for our units. Metrics are the same story: see the taps table below, all on loopback.
+
+## Metrics taps (Prometheus text format, loopback only — Metricbeat scrapes them)
+
+| endpoint | what | from |
+|---|---|---|
+| `http://127.0.0.1:9998/metrics` | MediaMTX: per path bytes/s, readers, publishers, WebRTC/HLS/RTMP/SRT sessions, dropped packets | MediaMTX `metrics: yes` |
+| `http://127.0.0.1:2019/metrics` | Caddy: requests by host/status, durations, TLS | Caddy admin API (`servers { metrics }`) |
+| `http://127.0.0.1:8080/metrics` | ours: the normalise queue, encode time, live stream count | `bin/dumpd.py` |
+
+Host and per-container metrics (CPU, RAM, disk, network, block I/O, restarts) come from Metricbeat's `system`
+and `docker` modules, not from an exporter — node_exporter and cAdvisor were removed when the Elastic stack
+went in. None of these endpoints is reachable from outside: each binds loopback, the firewall opens none of
+them, and Caddy routes no host to `/metrics` (every dumpd route there is an explicit path matcher).
+
+### Our own metrics, and how to add one
+
+MediaMTX answers "who is live" and Metricbeat answers "is the box busy". Neither can answer "is normalising
+keeping up", because the sweep is a shell script on a timer. So the scripts keep counters and dumpd renders
+them.
+
+- `bin/vgmetrics.sh` is sourced by `normalize-sweep.sh` and `vgdeband.sh`. It accumulates `key value` lines in
+  `/srv/media/.markers/metrics/<component>.counters` and keeps a `<component>.pid` file for as long as the
+  script is executing. It renders nothing and serves nothing, so the scripts stay simple.
+- `bin/dumpd.py` turns those files into Prometheus text on `GET /metrics`, adds gauges it can compute at
+  scrape time, and asks the MediaMTX API for the live path list. `COUNTER_MAP` there is the single place that
+  knows the metric names.
+
+| metric | type | what it answers |
+|---|---|---|
+| `vanguard_media_pending{kind}` | gauge | finished recordings waiting to be normalised. **The capacity signal**: if this trends up across a night, CPU encoding is not keeping pace |
+| `vanguard_media_pending_oldest_seconds` | gauge | how long the oldest one has waited |
+| `vanguard_media_recording` | gauge | recordings MediaMTX is still writing (age under `MINS_IDLE`) |
+| `vanguard_media_encoding_in_flight` | gauge | ffmpeg output files (`*.part`) open right now |
+| `vanguard_media_running{component}` | gauge | 1 while the sweep or vgdeband is executing, by live pid, not a stale marker |
+| `vanguard_media_jobs_total{component,verdict,outcome,kind}` | counter | remuxes and encodes finished, split by success and by stream vs clip |
+| `vanguard_media_job_seconds_total{component,verdict,kind}` | counter | seconds spent in ffmpeg. Divide by the job count for mean encode time |
+| `vanguard_media_runs_total{component}` | counter | script starts. Flat means the timer stopped |
+| `vanguard_media_deband_total{component}` | counter | timecode bands cropped off |
+| `vanguard_media_last_finish_timestamp_seconds{component}` | gauge | when it last finished |
+| `vanguard_media_last_run_seconds{component}` | gauge | how long that run took |
+| `vanguard_media_failed_files` | gauge | files parked with a `normalize-failed.log` |
+| `vanguard_mediamtx_up` | gauge | did the MediaMTX API answer the scrape |
+| `vanguard_streams_active` | gauge | streams publishing now, derived `.webrtc` mirrors excluded |
+| `vanguard_streams_mirrors` | gauge | those mirrors, counted separately |
+| `vanguard_stream_readers` | gauge | viewers attached across all paths |
+
+To add a counter: pick a key of the form `name:label:label`, call `vgmet_add` with it, and add `name` to
+`COUNTER_MAP` with the label names in the same order. Nothing else changes, and a key dumpd does not
+recognise is ignored rather than breaking the scrape.
+
+Three rules this design exists to keep. Metrics must never break a sweep, so every write is atomic and every
+failure in the helper is silent, and a missing helper degrades to no-ops. A scrape must never disturb the
+stack, so dumpd never takes the sweep's flock and the filesystem globs are depth-limited rather than walking
+the media tree. And the counters live on disk rather than in the process, because the sweep exits between
+runs.
+
+Note for anyone editing `normalize-sweep.sh`: its file loop reads from a process substitution, not a pipe. A
+pipe would put the loop in a subshell and every counter set inside it would be discarded on exit.
+
+## Container labels (set in `docker-compose.yml`, `x-logging` anchor + `labels:`)
+
+`vanguard.service` = the service name, `vanguard.category` = one of the categories below. A collector turns these
+into stream labels without parsing a single line.
+
+## Categories = the questions people ask
+
+| category | services / components | correlation key |
+|---|---|---|
+| `stream` | mediamtx (path ready/closed, recorder, WebRTC sessions, the on-ready hook's `event=mirror`), dumpd `publish_allow` / `publish_deny`, sweep `event=published` | `stream` = `user/Title` |
+| `auth` | caddy access logs (status, `X-Authentik-Username`), authentik-server/worker events, gitea OIDC | `user` |
+| `media` | sweep verdicts / timecode / deband / failed, vgdeband, jellyfin, copyparty | `file` |
+| `intake` | syncthing (`folder.id`), dumpd `clips_setup`, `quota_*` | `folder` = `clips-<user>` |
+| `infra` | mariadb, caddy TLS/ACME, container restarts, disk | host only |
+| `deploy` | deploy.sh, dumpd `deploy_*`, `bot_deploy_*` and `bot_env_*` (the bot's CI posting a release bundle and its settings), act_runner | `rev` |
+| `code`, `bot` | gitea, discord-bot | — |
+
+## Labels vs fields
+
+- **Labels** (low cardinality, what you filter by): `service`, `category`, `level`, `unit`. A dozen values each.
+- **Fields** (parsed at query time, what you search in): `stream`, `user`, `file`, `folder`, `session`, `status`, `ip`, `event`.
+  Thousands of values — fine as fields, ruinous as labels (Loki). VictoriaLogs indexes every word anyway.
+
+## Our own line format: logfmt, one event per line
+
+    ts=2026-11-01T19:52:14-06:00 level=info component=sweep event=published stream="techy2493/Vanguard_Training" file="/srv/media/streams/…" verdict=encode
+
+- `ts` ISO-8601 with offset · `level` info|warn|error · `component` dumpd|sweep|vgdeband|deploy|on-ready · `event` a verb/noun in snake_case · then `key=value`, values with spaces or quotes double-quoted.
+- Same key means the same thing everywhere: `stream` (user/Title), `user`, `file`, `folder`, `rev`, `ip`, `reason`.
+- Implemented by: `log()` in `bin/dumpd.py`, `vglog` in `bin/normalize-sweep.sh` and `bin/vgdeband.sh`, `log()` in `deploy.sh` (`msg=` only), `bin/mtx-on-ready.sh` (MediaMTX prefixes its own path/time).
+
+Events today: dumpd `publish_allow publish_deny dump clips_setup quota_paused quota_resumed quota_* deploy_* bot_deploy_start bot_deploy_exit bot_deploy_bad_signature bot_deploy_refused bot_env_applied bot_env_failed`;
+sweep `verdict timecode deband fill_gaps published done failed skip`; vgdeband `deband done failed skip`; hook `mirror`.
+A future alert script watches `level=error`, `event=failed`, `event=publish_deny` bursts, `event=quota_paused`, plus disk.
+
+## Third-party formats a collector must parse
+
+| source | format | stage |
+|---|---|---|
+| caddy | JSON (`logger`, `level`, `msg`; access logs: `request.*`, `status`, `request.headers.X-Authentik-Username`) | json |
+| authentik | JSON (`event`, `user`, `action`) | json |
+| syncthing | key=value after a `date time:` prefix (`folder.id=… log.pkg=model`) | logfmt |
+| mediamtx | `YYYY/MM/DD HH:MM:SS LVL [component] [session|path x] message` | one regex → component, id, msg |
+| jellyfin | `[HH:MM:SS.mmm] [LVL] [thread] Logger.Class: message` + stack-trace continuation lines | **multiline** (join lines not starting with `[`) then regex |
+| gitea, mariadb, copyparty | plain text | level regex only |
+
+## Caddy access logs
+
+On for every site block (`CADDY_LOG` in `configure.py`): JSON to stderr; `Cookie`, `Authorization` and
+`X-Stream-Auth` request headers are deleted before logging, the `X-Authentik-*` identity headers are kept — that is the
+"who downloaded what" audit trail for the files host.
+
+## Alerting: Kibana evaluates, we deliver
+
+A basic Elastic licence allows only two connector types, `.index` and `.server-log`. Everything that leaves
+the stack — webhook, Slack, email, PagerDuty — is Gold or above and refuses with
+`403 Action type .webhook is disabled because your basic license does not support it`. So Kibana cannot talk
+to Discord, and the pipeline is split:
+
+    Kibana rule  --.index connector-->  vanguard-alerts  --vanguard-alerts.timer-->  Discord webhook
+
+- **`observability/kibana-setup.py`** creates the connector ("Vanguard alert index") and the heartbeat
+  rule. Run it once, *after* Elasticsearch security is on.
+- **`vanguard-alerts` index** is the alert bus, pre-created by `configure.py` with `rule`, `level` and `kind`
+  as keywords. Alerts are ordinary documents, so they are searchable in Discover next to the logs that caused
+  them, and replayable after the fact.
+- **`bin/vanguard-alerts.sh`** runs every minute, delivers anything newer than its cursor, and suppresses a
+  repeat of the same rule within 15 minutes as a safety net. No webhook configured means journal-only and a
+  non-zero exit, never a silent drop.
+- **`bin/vanguard-deadman.sh`** watches both halves: a `kind=heartbeat` document arriving (Kibana still
+  evaluating and writing) and `alerts.state` being fresh (something still delivering). Either half dying
+  means no alarm can arrive, and nothing inside the stack could tell you.
+
+### The rules themselves
+
+`observability/kibana-alarms.py` creates them, so they are reviewable and reproducible rather than clicked in.
+All of them repeat every **12 hours** while still true (`notify_when: onThrottleInterval`) and go silent when
+the condition clears.
+
+| Rule | Level | Fires when |
+|---|---|---|
+| Disk free below 15% | error | any real filesystem at ≥85% used. Pseudo filesystems are excluded by type, or every tmpfs would alarm forever |
+| Memory free below 15% | warn | `system.memory.actual.used.pct` ≥ 0.85 |
+| Discord bot not running | error | **no** metrics from the `discord-bot` container for 5 min (the one rule with a `<` comparator) |
+| Discord bot unhealthy | error | the container healthcheck reports `unhealthy`, i.e. the bot's own `/health` says Discord is disconnected or the database does not answer |
+| Recordings stuck waiting to be normalised | warn | `vanguard_media_pending_oldest_seconds` > 1 h |
+| Normalise failures parked | warn | any file has a `normalize-failed.log` |
+
+The two bot rules answer different questions and both are needed. "Not running" catches a stopped or
+hard-crash-looping container. "Unhealthy" catches the worse case: the process is up, the port is open, and the
+bot is doing nothing because its gateway or database is gone. That second one only works because the bot's
+`/health` is exempt from its own identity middleware and returns 503 rather than 200 when degraded — a
+healthcheck carries no Authentik headers, and a 200 with `discord_ready: false` would have made a dead bot
+look healthy.
+
+Elasticsearch, Kibana, Beat liveness and the disks are **not** here: that is the deadman's job, outside the
+stack. Nothing that depends on Kibana can report that Kibana is dead, so duplicating those as rules would go
+quiet in exactly the case that matters.
+
+To add an alarm: author the rule in Kibana, point its action at the "Vanguard alert index" connector, and give
+the document a `message` and a `level` of `warn` or `error`. Never set `kind: heartbeat` — that value is how
+the delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
+for real alarms; only the heartbeat should fire every interval.
+
+## Retention
+
+Set by `configure.py --apply`, so nothing here needs clicking and nothing grows without bound:
+
+| What | How long | Set by |
+|---|---|---|
+| `filebeat-*`, `metricbeat-*` | `LOG_RETENTION_DAYS`, default 30 | a **delete phase injected into the Beats' own ILM policies**. They ship a rollover but no delete, so indices otherwise grow until the disk watermark turns them read-only — which the deadman reports but cannot undo |
+| `vanguard-alerts` | `ALERT_RETENTION_DAYS`, default 90 | `vanguard-alerts.sh`, a `_delete_by_query` at most once every 6 h. A heartbeat document arrives every minute, so this index needs pruning more than it needs ILM |
+
+The delete phase is *injected* rather than written whole, so whatever rollover the Beats chose is preserved
+across upgrades. Keeping `auth` and `deploy` lines longer than the rest would mean separate indices and a
+second ILM policy; not worth it unless an audit asks.
+
+## As built (Phase 13)
+
+| container | port | job |
+|---|---|---|
+| `elasticsearch` | 9200 | one store for **both** logs and metrics. 2 GB heap, single node, security off (loopback only) |
+| `kibana` | 5601 | the single pane: Discover for logs, dashboards, alerting. Served at the root of `diag.BASE` |
+| `filebeat` | — | container logs (autodiscover) + our journal units |
+| `metricbeat` | — | host + Docker metrics (retires node_exporter and cAdvisor) + MediaMTX and Caddy Prometheus endpoints |
+
+**Logs and metrics correlate because both Beats write the same fields into the same store.** That is what makes
+one dashboard able to show a CPU spike next to the errors that caused it, rather than two dashboards side by
+side:
+
+| field | set by | use |
+|---|---|---|
+| `@timestamp` | both (ECS) | every panel shares the dashboard's time range |
+| `host.name` | both | one box today, but the field is there |
+| `container.name` | both | per-service metrics joined to that service's log lines |
+| `service.name` | both, from the `vanguard.service` label | the name you actually think in |
+| `vanguard.category` | both, from the label | filter a whole area at once: stream, media, auth, intake, deploy |
+
+To get a dashboard that filters across both, make **one data view** spanning `filebeat-*,metricbeat-*` in
+Kibana (Stack Management → Data Views). A single filter such as `service.name: mediamtx` then applies to the
+metric panels and the log panel together. Metricbeat also ships its own system and Docker dashboards, which are
+a good starting point to copy panels from.
+
+Retention is one edit in Kibana rather than config: Stack Management → Index Lifecycle Policies → the beats
+policy → add a delete phase. Doing it there rather than in a file keeps it adjustable without a redeploy.
+
+### Old notes
+
+
+(superseded 2026-11-03 — the Victoria/vmalert/Alertmanager shape, kept for the reasoning)
+
+| container | port | job |
+|---|---|---|
+| `victorialogs` | 9428 | store, 30 d, `-http.pathPrefix=/logs` so Caddy passes the prefix through unchanged |
+| `vector` | — | `docker_logs` + `journald` → VictoriaLogs (elasticsearch bulk). **Holds the Docker socket** |
+| `victoriametrics` | 8428 | 90 d, scrapes the taps above (`observability/scrape.yml`) |
+| `vmalert-metrics` | 8880 | `observability/alerts-metrics.yml` against VictoriaMetrics |
+| `vmalert-logs` | 8881 | `observability/alerts-logs.yml` (LogsQL) against VictoriaLogs' stats endpoints |
+| `alertmanager` | 9093 | groups, inhibits, posts to Discord (`webhook_url_file`, never in the config) |
+
+Published only through `diag.BASE`, gated on `DIAG_REQUIRE_GROUP` (`media-admins`): `/logs`, `/metrics-db`,
+`/alerts`, `/rules-metrics`, `/rules-logs`, and an index at `/`.
+
+**The deadman** (`bin/vanguard-deadman.sh`, host timer, 5 min) is deliberately outside all of it: curl only, it
+checks each component answers, that the always-firing `Watchdog` alert is still arriving through
+vmalert → Alertmanager → dumpd `/heartbeat`, and the disks. It speaks on change, announces recovery once, and
+with no webhook configured still logs and exits non-zero so `systemctl is-failed vanguard-deadman.service` is
+a signal. Nothing inside the stack can report its own death; this can.
+
+`ALERT_WEBHOOK_URL` in `.env` → `/etc/vanguard/alert-webhook` (0640 root:65534). Empty means log-only, which is
+a working configuration, not an error: `configure.py` generates `observability/alertmanager.yml` with the
+Discord notifier omitted, so nothing is half-configured and nothing fails to load.
+
+## Gotchas this build produced (all fixed; here so they are not rediscovered)
+
+- **`-http.pathPrefix` applies to EVERY endpoint, not just the UI.** With VictoriaLogs behind `/logs`, the
+  ingestion path is `/logs/insert/elasticsearch/`, the health probe is `/logs/health` and the scrape is
+  `/logs/metrics`. It rejects a write to the unprefixed path with a message naming the missing flag.
+- **Every proxied UI needs its own prefix set**, or it loads at the sub-path and then fetches its assets from
+  the domain root and renders broken. VictoriaMetrics and vmalert use `-http.pathPrefix`; Alertmanager uses
+  `--web.route-prefix` plus `--web.external-url`. Setting those also moves their metrics and health paths.
+- **Vector's `current_boot_only: false` is rejected on systemd 250–257.** Only the current boot is shipped;
+  earlier entries are still on the host for `journalctl --boot=-1`.
+- **A Caddy `respond` body is placeholder-expanded**, so `{...}` in CSS or JSON is eaten. Serve a page as a
+  file and mount it, the way the gateway and dock pages are served, instead of inlining it. (The diag host no
+  longer has such a page at all: Kibana occupies its root.)
+- **Filebeat 9 removed the `container` input.** It is not deprecated-but-working: the config is refused per
+  container with "Container input is deprecated. Use Filestream input with its container parser instead",
+  and Filebeat then runs happily with zero harvesters and creates no index at all. Container logs go through
+  `type: filestream` with `parsers: [container: ~]`, and every filestream input needs its own stable `id`.
+  This is exactly the failure the deadman's "up but not shipping" check exists to catch.
+- **`add_docker_metadata` should be told where the container id is.** Its path-sniffing default did not
+  apply to our filestream paths, so the autodiscover template stamps `container.id` and the processor reads
+  it via `match_fields: ["container.id"]`. Without the id there are no `container.labels.*`, and without
+  those the `service.name` / `vanguard.category` promotion silently produces nothing.
+- **A single node cannot place a replica**, so any beats index defaulting to one replica holds the cluster at
+  yellow permanently. Both beats set `setup.template.settings.index.number_of_replicas: 0` with
+  `setup.template.overwrite: true`; indices created before that need a one-time `_settings` call.
+- **Kibana's Observability apps that check privileges cannot work with `xpack.security.enabled: false`.** The
+  Infrastructure / Metrics app calls `api/infra/host/count` on every view and Elasticsearch answers
+  `no handler found for uri [/_security/user/_has_privileges]`, surfaced as a red "error while fetching
+  resource" banner. The fix is to turn security ON, not to avoid the app. Doing so normally adds a password
+  prompt, which this stack forbids, so Kibana's **anonymous provider** signs every visitor in as one native
+  user instead (`observability/kibana.yml`). Security being on is also what any future read-only tier needs.
+- **Moving a setting from a container environment variable into a config file can break it.** `SERVER_BASEPATH: ""`
+  was fine for years because the Kibana image drops empty env values, but `server.basePath: ""` in
+  `kibana.yml` is a validation error (`must start with a slash, don't end with one`) and Kibana refuses to
+  boot. When migrating env settings into a file, an empty value means "omit the key", not "set it to empty".
+- **Metricbeat exits when Kibana is unreachable**, because `setup.dashboards.enabled: true` runs at startup.
+  A Metricbeat restart loop is often a symptom of a Kibana problem, not a Metricbeat one.
+- **Alerting rules created while security was off hold no API key** and stop working once it is on, until each
+  rule is disabled and re-enabled. Create connectors and rules *after* security is on, not before.
+- **Kibana's Observability charts group by `event.dataset`.** Beats do not set it, so without a processor
+  every line lands in one "unknown" bucket and the Logs rate chart is useless. Filebeat now fills it from the
+  container's `vanguard.service` label, or from `systemd.unit` for journald lines. `copy_fields` never
+  overwrites an existing target, so listing several sources for one field means first match wins.
+- **Metricbeat field shapes differ per module on purpose.** The MediaMTX and Caddy scrapes are untyped, so a
+  value is at `prometheus.metrics.<name>` with labels at `prometheus.labels.*`. The dumpd scrape sets
+  `use_types` and `rate_counters`, so a gauge is at `prometheus.<name>.value` and a counter has both
+  `.counter` and `.rate`. Look for the right one before concluding a metric is missing.
+- **`configure.py --apply` cannot introduce a new host** — see HOSTNAME-CHANGE.md §D.
+
+## Wiring sketches (the options that were weighed, kept so the decision is not re-argued)
+
+- **Grafana + Loki**: `alloy` container with `loki.source.journal` + `loki.source.docker` (labels from the container
+  labels above) → `loki` on loopback → `grafana` behind Authentik OIDC (third provider, like Jellyfin/Gitea).
+  Metrics: Alloy scrapes the four taps above → Prometheus or VictoriaMetrics, same Grafana.
+- **VictoriaLogs**: single container on `127.0.0.1:9428`, run with `-http.pathPrefix=/logs`; Vector
+  ships journald and the Docker logs. Its built-in UI sits behind the Caddy forward-auth gate on `diag.BASE`
+  (`media-admins`). VictoriaMetrics beside it would consume the four metric taps.
+- **Netdata**: single container, host + container metrics and alerts (Discord) out of the box, journald viewer;
+  behind the Caddy gate. Weakest at log search.
+- **ELK** (chosen, and what is deployed): Elasticsearch with a pinned 2 GB heap, Kibana at the root of
+  `diag.BASE`, Filebeat and Metricbeat shipping into the same cluster so one dashboard can carry logs and
+  metrics together. The cost is the JVM heap and the index disk on a box that also runs x264; it was taken
+  because every lighter option needed three or four more pieces to reach a usable dashboard.

README.md

@@ -1,5 +1,29 @@
 # vanguard-stream — SYS-1 (Xeon E-2136, 32 GB ECC, 2×4 TB SATA soft RAID)
 
+> **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included.
+> **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely.
+> **LOGGING.md** = observability: what each service logs, the container labels, the logfmt event format, and the Elastic stack that consumes it (§5h).
+> **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app).
+
+**Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
+
+> Adding a host later needs `./configure.py --full` or `--hosts`, **not** `--apply`: apply reads the hostnames
+> back out of the Caddyfile, so a role that is not in there yet is silently left out and the new name ends up
+> with no certificate. HOSTNAME-CHANGE.md §D has the procedure.
+
+| Host | What | Who |
+|---|---|---|
+| `media.BASE` (main) | gateway page at `/` (role-aware links + how-tos), `/whoami`, `/logout` | everyone |
+| `files.BASE` | copyparty: browse, scrub, download, upload finals | editors / media-admins |
+| `clips.BASE` | self-service clip intake (pair your Syncthing) | any member |
+| `sync.BASE` | Syncthing admin UI | media-admins |
+| `live.BASE` | live portal, Go Live, OBS dock `/dock/`, clock source `/dock/clock/`, `/dump`, `/me`, `/time` | everyone (`LIVE_REQUIRE_GROUP`) |
+| `diag.BASE` | logs, metrics, alerts (§5h) | media-admins |
+| `watch.BASE` | Jellyfin | Discord via OIDC |
+| `git.BASE`, `bot.BASE`, `auth.BASE` | Gitea, bot web UI (`bot-users`), Authentik | — |
+
+`/time` (the server clock the timecode strip syncs to) lives only on `live` because browsers fetch it
+same-origin: both pages that use it, the clock source and Go Live, are on that host.
 > **STATUS.md** = what runs, who does what, what's unfinished (kept current). **CommunityDiscordMedia.md** = design + phases.
 
 Ingest from OBS, browse/scrub/download for editors, automatic normalization to one editing
@@ -80,7 +104,7 @@ Everything else is dropped and a trickle is logged (`journalctl -k | grep vg-dro
 Jellyfin (8096), copyparty, the Syncthing UI and dumpd bind to 127.0.0.1 only and are reached via Caddy
 or an ssh tunnel, so they are deliberately not opened. MariaDB runs on the host network (not a Docker
 published port) precisely so the firewall can govern it.
-Syncthing's admin UI is proxied at `https://HOST/sync/` behind the Authentik gate + group `media-admins`; its
+Syncthing's admin UI is proxied at `https://sync.BASE/` behind the Authentik gate + group `media-admins`; its
 own GUI login is disabled (no user/password, `insecureAdminAccess=true`) so Discord is the only login:
 
     docker exec syncthing syncthing cli --home /var/syncthing/config config gui insecure-admin-access set true
@@ -100,8 +124,11 @@ Being a streamer = being in the Authentik **streamers** group (a Discord role vi
 There are no stream passwords or keys to hand out: the dock mints a **12-hour publish token** for the
 logged-in Discord user each time Go Live is pressed and sets it as the OBS stream key itself.
 
+OBS → Settings → Output → Encoder: **B-frames = 0** (NVENC "Max B-frames: 0"; x264 add `bframes=0`). MediaMTX
+refuses WebRTC playback of H.264 with B-frames ("WebRTC doesn't support H264 streams with B-frames"), so the live
+portal would fall back to HLS with seconds of delay. Keyframe interval 2 s. Recording quality is unaffected.
 OBS → Tools → WebSocket Server Settings → enable, set a password.
-OBS → Docks → Custom Browser Docks → `https://media.example.com/dock/` → log in with Discord once.
+OBS → Docks → Custom Browser Docks → `https://live.BASE/dock/` → log in with Discord once.
 In the dock's Settings: ingest host (prefilled), OBS WebSocket password/port, optional title prefix/suffix
 (`{date}` → YYYY-MM-DD, e.g. prefix `HelmetCam-` or suffix `-{date}`).
 Per stream: type a title, click **Go Live**. The dock shows the final stream name (`<you>/<Title>`) as you type
@@ -118,26 +145,32 @@ Browser streaming without OBS: the live portal's **Go Live** (Windows Chrome 141
 
 ## 5a. Onboarding a gamer's clips (Syncthing) — self-service
 
-Gamers do it themselves at **https://HOST/clips/** (Discord login): the page shows our Syncthing Device ID and
+Gamers do it themselves at **https://clips.BASE/** (Discord login): the page shows our Syncthing Device ID and
 asks for theirs; on submit, dumpd creates `clips/<username>` with the video-only `.stignore`, adds their device
 to our Syncthing (REST API, key read from `/srv/media/.syncthing/config.xml`) and creates the receive-only
 folder `clips-<username>` shared with them (min free 300 GB). They accept the device + folder on their PC,
 point it at their clip directory, Send Only. The page shows pairing/connection/sync state. Idempotent.
+**Per-user quota (soft)**: dumpd measures every `clips/<user>` every 5 min and pauses that user's Syncthing
+folder when it exceeds the quota (default `CLIPS_QUOTA_GB=100` in `dumpd.service`), resuming automatically
+under 90 %. The clips page shows usage and the paused state. Overrides per user in `/etc/vanguard/clip-quotas.json`,
+e.g. `{"techy2493": 250, "newguy": 50}` (GB) — read live, no restart. A gamer frees space by deleting clips in
+their own folder (Syncthing mirrors the deletion). Hard byte-exact quotas would be XFS project quotas on `/srv`
+(needs a remount with `prjquota`); not set up.
 Manual fallback (admin, same result for the server folder):
 
     ./addClipUser.sh <username>        # creates /srv/media/clips/<username> with the video-only .stignore
 
 Their side (Windows: Syncthing installer or SyncTrayzor; Mac/Linux: Syncthing): open the UI, **Actions → Show ID**,
-send you the Device ID. In their UI: **Add Remote Device** with the server's ID (ours: `/sync/` → Actions → Show ID),
+send you the Device ID. In their UI: **Add Remote Device** with the server's ID (ours: `sync.BASE` → Actions → Show ID),
 then **Add Folder** on their clip directory (ShadowPlay/Medal/Steam saves), Sharing tab → tick the server,
 Folder Type **Send Only**, Save.
 
-Our side at https://HOST/sync/: **Add Remote Device** → paste their ID, name it after them, Save. When their folder
+Our side at https://sync.BASE/: **Add Remote Device** → paste their ID, name it after them, Save. When their folder
 offer pops up: **Add** → Folder Path `/var/syncthing/clips/<username>`, Folder Type **Receive Only**, Advanced →
 Minimum Free Disk Space **300 GB**, Save. Clips start arriving; the sweep normalizes them; Jellyfin shows the
 folder as that person's show. Their own folder structure is mirrored; only video files are pulled.
 
-## 5a². Gateway page — https://HOST/about/
+## 5a². Gateway page — https://media.BASE/ (the main host root; `/about/` redirects here)
 
 Role-aware landing page (Discord login): everyone sees Watch / Live / Send clips / Sign out with short how-tos;
 `streamers` get the dock + OBS setup steps; `editors` the browser + multicam/timecode notes; `media-admins`
@@ -154,7 +187,7 @@ already `.mp4`), the raw's name is appended to `.stlocal` (an `#include`d ignore
 drops it without flagging or re-pulling), and the raw is deleted locally — it still exists on the
 gamer's PC. `configure.py` creates the folder + ignore file for each streamer.
 
-In the UI (https://HOST/sync/):
+In the UI (https://sync.BASE/):
 - each clips folder → path `/var/syncthing/clips/<user>`, **Folder Type: Receive Only**,
   Edit → Advanced → **Minimum Free Disk Space: 300 GB** (so clips can never fill the mirror and stall
   the recordings).
@@ -163,8 +196,8 @@ In the UI (https://HOST/sync/):
 
 ## 5c. Authentik (Discord login) and Jellyfin
 
-DNS: `auth` and `watch` as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
+DNS: `auth`, `watch` (and `files`, `clips`, `sync`, `live`, `git`, `bot`) as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
-for both hostnames, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
+for every hostname, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
 adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
 hostnames filled in. (`./configure.py --full` asks for the same hostnames if you prefer.) Then:
 
@@ -208,21 +241,82 @@ Phase 4 is the Vanguard Metadata Jellyfin plugin; Phase 5 puts copyparty, /sync
 ## 5d. MariaDB + Discord bot (for the bot dev)
 
 `mariadb` (public on 3306, host network, root local-only, 32-char generated passwords) with a database/user
-for the bot, and `discord-bot`, a Python 3.12 container that runs `bot/app/main.py` and idles until that file
+for the bot, and `discord-bot`, a Python 3.12 container that runs `/app/main.py` and idles until a release is
-exists. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
+installed. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
 or the `[d]` menu item; `./configure.py --show` prints them).
 
-The bot also serves a web UI on 127.0.0.1:8000, published as `https://bot.HOST/` behind **Authentik
+**Releases are self-contained bundles, and a push to `main` deploys itself.** The bot repo's Gitea Actions
-forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`. Create the matching
+workflow runs its tests and then builds one tar.gz holding `main.py`, `app/`, `website/`, `BUILD.json` **and
-Proxy Provider in Authentik (forward auth, single application, external host `https://bot.HOST`), add it to
+`vendor/`** — its own dependencies, as prebuilt wheels for this image's Python. `PYTHONPATH=/app/vendor` is how
-the embedded outpost, and bind a group policy. Until then the site returns an error, never an open page.
+the code finds them, so **a new dependency never needs an image rebuild** and the server never contacts a
+package index.
+
+The workflow's last job posts that bundle to **dumpd** at `POST /deploy-bot` on 8081 — the Docker-bridge-only
+port the firewall already restricts, so a CI job never needs the Docker socket or an ssh key. dumpd checks an
+HMAC-SHA256 signature over the body, then runs `/usr/local/bin/vanguard-deploy-bot` (this repo's
+`deployBotBundle.sh`), which verifies every file against `BUILD.json`, swaps the bundle into `/srv/bot/app`
+(bind-mounted at `/app`, previous release kept beside it) and restarts the container. The install log goes back
+in the HTTP response, so it appears in the CI job.
+
+**The bot's own secrets live in Gitea, not here.** The deploy job also sends the bot's settings, signed with
+the bundle, and dumpd writes them into this stack's `.env` — but only the keys `BOT_ENV_KEYS` lists. Anything
+else comes back to the CI job as `settings_refused` and is not written, so the database password and the
+Authentik keys this repo generates cannot be overwritten from CI. Rotating the Discord token is therefore a
+Gitea edit plus a re-run, with no server access.
+
+In Gitea, on the bot repo under Settings → Actions:
+
+| | |
+|---|---|
+| Secret `BOT_DEPLOY_SECRET` | must equal the server's; `./configure.py --show` prints it |
+| Secret `DISCORD_BOT_TOKEN` | the bot token |
+| Variable `DISCORD_GUILD_ID` | the server id |
+| Variable `VANGUARD_DEPLOY_HOST` | only if the job cannot detect the Docker gateway |
+
+On the server, generated into `.env` and copied to `/etc/vanguard/deploy.env` by `configure.py`:
+
+| | |
+|---|---|
+| `BOT_DEPLOY_SECRET` | 64 hex chars, the shared signing key |
+| `BOT_DEPLOY_AUTO` | `1` (default) accepts CI installs; `0` makes dumpd refuse them with 403 |
+| `BOT_ENV_KEYS` | the only `.env` keys CI may set, `DISCORD_BOT_TOKEN,DISCORD_GUILD_ID` by default. Extend it when the bot needs a new setting. |
+
+By hand, any time:
+
+    bin/vgbot-merge-sqlite.py --source /data/<old>.db  # merge an old SQLite database into the live MariaDB (report; --apply writes)
+    ./deployBotBundle.sh vanguard-bot-bundle.zip      # install a release (the artifact as Gitea hands it over, zip or tar.gz)
+    ./deployBotBundle.sh --latest                     # fetch the newest main build first (needs BOT_REPO_TOKEN in .env)
+    ./deployBotBundle.sh --status                     # version, commit, package list, container state
+    ./deployBotBundle.sh --rollback                   # back to the previous bundle
+    docker compose logs -f discord-bot                # watch it
+
+**Merging an old SQLite database in** (`bin/vgbot-merge-sqlite.py`): the bot rebuilds part of its state from
+Discord on every start, so after a backend switch some tables hold fresh rows while the old file holds months of
+history, and neither side is a superset. The bot's own importer refuses a non-empty destination by design, so this
+merges instead: log tables append on a natural key, and the handful of profile/state tables are resolved column by
+column with an explicit rule each (earliest `joined_at`, `first_seen_at` and rank `started_at` win, so a rebuild
+that stamped today's date is undone; latest activity and current rank/presence win). It runs in one transaction,
+is idempotent, and refuses rather than guessing if a column has no rule or a table is missing from the
+destination. Run it with the gateway stopped, report first, then `--apply`.
+
+**Identity, exactly:** the bot gets the caller's Discord snowflake from the `discord_id` claim the proxy puts in
+`X-Authentik-Jwt`, emitted by the scope mapping in `authentik/vanguard-profile-scope-mapping.py` (the same mapping
+that carries inherited groups). Authentik already stores that id on the account's source connection, so nothing
+has to be matched by name. Without the claim the bot falls back to matching the Discord **username** only, never a
+server nickname: nicknames are self-chosen and not unique, so matching one would let a member be mistaken for
+another or make another ambiguous. `discord-bot` also runs on **its own Docker network** (`botnet`), because it
+trusts the headers Caddy injects and nothing else should be able to reach `:8000` to forge them.
+
+The bot serves **both** its API and its web pages on 127.0.0.1:8000, published as `https://bot.HOST/` behind
+**Authentik forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`, and the
+bot authorises per request from those headers (`WEB_ALLOWED_GROUPS=bot-users,admins`; `WEB_SERVE_UI=true`
+turns the pages on). Create the matching Proxy Provider in Authentik (forward auth, single application,
+external host `https://bot.HOST`), add it to the embedded outpost, and bind a group policy. Until then the
+site returns an error, never an open page. Caddy's own gate is `BOT_REQUIRE_GROUP` (`bot-users` by default),
+so **admins need the `bot-users` role too** — or set `BOT_REQUIRE_GROUP=` and let the bot's group list decide.
 
 The build brief for whoever writes the bot is `bot/AGENT.md`; `bot/README.md` is the ops cheat-sheet.
 
-    docker compose logs -f discord-bot                     # watch it
-    docker compose restart discord-bot                     # after code changes
-    docker compose build discord-bot && docker compose up -d discord-bot   # after requirements.txt changes
-
 ## 5e. Gitea + Actions runner
 
 `gitea` (MariaDB-backed, `git.HOST` via Caddy, git-over-ssh on **2222**, registration closed) and
@@ -248,7 +342,17 @@ Keep repo write access to trusted people.
    Auto Discovery URL `https://auth.HOST/application/o/gitea/.well-known/openid-configuration`, tick "Skip local
    2FA", claim name for groups `groups`, administrator group claim value **`git-admins`**. Save.
    Authentik group nesting makes the admins inherit it: `authentik Admins` → parent `admins` → parent `git-admins`
-   (members of a child group count as members of the parent, in the `groups` claim and in Caddy's header checks).
+   (members of a child group count as members of the parent). **Inherited groups are not in the `groups` claim by
+   default**: the managed profile mapping uses `ak_groups` (direct memberships only), and proxy providers re-add
+   that mapping on every save, so it cannot be swapped out. Add a second scope mapping, Customization → Property
+   Mappings → Create → Scope Mapping, name `Vanguard profile (nested groups)`, scope `profile`, expression:
+
+       direct = set(request.user.ak_groups.all())
+       return {"groups": sorted(g.name for g in request.user.all_groups() if g not in direct)}
+
+   and tick it (next to the default one) on the `Vanguard sites`, `Jellyfin` and `Gitea` providers → Advanced
+   protocol settings → Scopes. Claims from several mappings merge, so the header becomes direct + inherited groups
+   without duplicates. Takes effect on the next login.
 3. compose (already set): registration only via external sources, auto-registration + account linking on,
    username from `preferred_username`. A first Discord login creates the Gitea user with no form.
 4. Access = Authentik group **`git-users`**: Directory → Groups → create it (add a Discord role to it in the
@@ -263,8 +367,15 @@ Optional later: hide the password form (custom template) so Discord is the only
 React + dockview page (`live/`) behind Authentik forward-auth: every active stream is a dockable window,
 audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
 MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
-MediaMTX runs the `-ffmpeg` image so the optional "is live" Discord hook (`LIVE_WEBHOOK_URL` in `.env`,
+MediaMTX runs the `-ffmpeg` image for the on-ready hook `bin/mtx-on-ready.sh` (bind-mounted into the container):
-asked by `configure.py --full`) can post when a stream starts.
+the optional "is live" Discord post (`LIVE_WEBHOOK_URL` in `.env`, asked by `configure.py --full`) and the
+**WebRTC audio mirror**. Browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only) and
+OBS sends AAC, so for every stream whose audio is not already Opus the hook runs one ffmpeg that copies the video
+and transcodes the audio to Opus into the sibling path `<user>/<Title>.webrtc` (via MediaMTX's RTSP on the
+container loopback). The portal plays the mirror over WebRTC and hides it as a panel; recordings, HLS and the
+Discord post stay on the original. Mirrors are never recorded and can only be published from inside the container
+(dumpd checks the source IP). Browser Go Live already sends Opus and gets no mirror. Cost: a few % of a core per stream.
+Viewer requirement on the streamer side: **B-frames 0** (section 5), or WebRTC is refused and the portal falls back to HLS.
 
 1. CNAME `live` → main host; `./configure.py --full` asks the hostname (also `--hosts … LIVE`).
 2. Authentik: Proxy Provider (forward auth, single application, external host `https://live.HOST`) →
@@ -277,7 +388,7 @@ from `/me/stream-token` for the logged-in Discord user (member of `streamers`).
 Bandwidth: no transcoding, every viewer gets the source bitrate (~15 Mbps) → ~30 concurrent viewers on
 the 500 Mbps line. For a crowd, restream to Twitch from OBS.
 
-## 5g. Authentik forward-auth for the Caddy-gated hosts (main host, `live.`, `bot.`)
+## 5g. Authentik forward-auth for the Caddy-gated hosts (`media.`, `files.`, `clips.`, `sync.`, `live.`, `bot.`)
 
 Design: **Authentik authenticates once for the whole domain; Caddy authorizes per host** from the
 `X-Authentik-Groups` header the outpost returns; **Discord roles fill the Authentik groups** at login.
@@ -288,8 +399,8 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
    cookie domain = the parent domain (e.g. `dumpstervigilantes.rip`). Edit the Embedded Outpost →
    Applications → add it → Update. One login now covers every gated host.
 3. Per-host group requirement = `.env`: `BOT_REQUIRE_GROUP=bot-users` (default), `LIVE_REQUIRE_GROUP=`
-   (empty = any logged-in member), `SYNC_REQUIRE_GROUP=media-admins` (the `/sync` Syncthing UI on the main host).
+   (empty = any logged-in member), `SYNC_REQUIRE_GROUP=media-admins` (the Syncthing UI on `sync.`).
-   On the main host copyparty itself authorizes by group (`editors`, `media-admins`); the dump endpoint lets
+   On `files.` copyparty itself authorizes by group (`editors`, `media-admins`); the dump endpoint lets
    `admins`/`media-admins` dump anyone's night. Leave "Intercept header authentication" **off**: login is
    Discord only, no app passwords, no WebDAV. `./configure.py` apply regenerates the Caddy blocks; a user outside the
    group gets a 403 from Caddy before the app sees the request.
@@ -308,16 +419,74 @@ Design: **Authentik authenticates once for the whole domain; Caddy authorizes pe
    `default-provider-invalidation-flow` so that also ends the Authentik login: every logout then forces a full
    Discord re-login on the next visit (and picks up group changes). Discord itself may approve silently if
    you are still logged into Discord in that browser.
-5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them.
+5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them, and
+   `https://HOST/me` adds the `discord_id` claim decoded from `X-Authentik-Jwt` — the quickest check that the
+   `Vanguard profile` scope mapping is live. The raw token is deliberately never echoed: the browser does not
+   see that header today, and printing it would hand the user a bearer token they otherwise could not read.
    Group changes in Authentik apply on the next login — log out at `https://auth.HOST/` and back in.
 6. Test in a private window: `https://live.HOST/` → Discord login → portal; `https://bot.HOST/` → 403 unless
    you hold the bot role; `https://live.HOST/me/stream-key` shows your username.
 
 Jellyfin and Gitea use OIDC providers instead, not this.
 
+## 5h. Diagnostics: Kibana at `diag.HOST`
+
+Four containers, one store, one UI. `diag.HOST` **is** Kibana, served at the root: logs in Discover,
+dashboards, and alerting, with no sub-paths and no index page of links.
+
+| container | job |
+|---|---|
+| `elasticsearch` | one store for **both** logs and metrics. 2 GB heap, single node, loopback only |
+| `kibana` | Discover, dashboards, alerting. Published as `https://diag.HOST/` behind the gate |
+| `filebeat` | every container's logs (autodiscover) plus our journal units |
+| `metricbeat` | host + Docker metrics, replacing node_exporter and cAdvisor, plus MediaMTX's and Caddy's Prometheus endpoints |
+
+Both Beats write the same correlation fields (`service.name`, `container.name`, `vanguard.category`,
+`@timestamp`), so one dashboard can show a CPU spike beside the log lines from that minute and one filter
+narrows both. `LOGGING.md` lists the fields and the single Kibana data view that makes it work.
+
+Four things are pinned on purpose, each because the default bites:
+
+| setting | why |
+|---|---|
+| `ES_JAVA_OPTS=-Xms2g -Xmx2g` | a self-sizing JVM takes a quarter of the box; the spare capacity belongs to x264 |
+| `vm.max_map_count=262144` | a host sysctl, set and persisted by `install.sh` and by every apply. ES will not start without it |
+| `xpack.security.enabled=false` | safe only because it listens on loopback behind Authentik; it is also why there is no second login |
+| `KIBANA_ENCRYPTION_KEY` | without a fixed key Kibana loses every saved object, dashboards included, on restart |
+
+**First-run steps in Kibana** (both are UI actions on purpose, so they stay adjustable without a redeploy):
+
+1. **Retention**: Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase.
+2. **The deadman's heartbeat**: a rule that always fires, every minute, with a webhook action posting to
+   `http://<docker gateway>:8081/heartbeat`. That proves rule evaluation *and* delivery work end to end, and the
+   deadman treats a stale heartbeat as the alarm. Without it the deadman reports the heartbeat missing, which is
+   correct rather than a bug.
+3. **Alerts to Discord**: a webhook connector pointing at `ALERT_WEBHOOK_URL`. Until one exists, rules still
+   evaluate and show in Kibana, they are simply not delivered.
+
+**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
+nothing but curl, because nothing inside Elastic can report that Elastic is the thing that broke:
+
+| it checks | because |
+|---|---|
+| Elasticsearch answers, cluster not red | no store, no anything |
+| no index is read-only | the disk watermark sets that and **does not unset it** when space is freed, so ingestion stays silently dead |
+| Kibana is available | it is the only UI |
+| each Beat wrote a document recently | "running" is not "shipping" |
+| the Kibana heartbeat is fresh | proves the whole rule → webhook → delivery path |
+| the disks | the likeliest death, and it must be reported even when the reporter is what broke |
+
+    systemctl list-timers vanguard-deadman.timer
+    journalctl -u vanguard-deadman -n 20
+    sudo /usr/local/bin/vanguard-deadman.sh          # run it now; exit 1 means it found something undeliverable
+
+Filebeat holds the Docker socket for autodiscover, which is root-equivalent on this host — the same exposure
+act_runner has. Accepted deliberately; the alternative is losing the container labels that make the
+correlation above work.
+
 ## 6. Editors
 
-Log in with Discord at https://media.example.com/ (copyparty behind Caddy + Authentik). Access is by Authentik
+Log in with Discord at https://files.BASE/ (copyparty behind Caddy + Authentik). Access is by Authentik
 group: **editors** read streams and clips and write `/finals`; **media-admins** have full rights everywhere
 including `/trash`. Scrub any `.mp4` in the browser, download what you want.
 
@@ -353,7 +522,7 @@ restarts only copyparty.
 ## Timecode for multicam editing (frame accurate)
 
 Every published stream carries a QuickTime **timecode track** (`tmcd`, time of day in Eastern, at the file's
-frame rate) and a UTC `creation_time`, written by the sweep during the lossless remux, plus a sidecar
+frame rate) and a UTC `creation_time`, written by the sweep when it publishes the file, plus a sidecar
 `<Title>__<ts>.json` with the exact capture time (the Jellyfin plugin reads it). Resolve and Premiere read the
 track natively: drop clips from several streamers on a multicam timeline, sync by timecode.
 
@@ -361,14 +530,20 @@ Where the time comes from, best first:
 1. **Overscan band** — a thin strip of extra rows below the picture carrying the *server* clock, encoded
    (black/white cells, format in `bin/vgstrip.py`) plus a small readable clock. The server decodes it
    (`bin/vgstrip.py`) at the start, middle and end of each recording → the timecode is frame-accurate at the
-   start and the sidecar records drift over the file. Game image untouched. OBS: dock → *Timecode band* →
+   start and the sidecar records drift over the file. **Then the band is cropped off**: the sweep re-encodes the
+   stream (libx264 CRF 18, frame timing preserved, audio copied) so the published file is the game image + audio +
+   the `tmcd` track and nothing else; the raw with the band is deleted (`KEEP_RAW_STREAMS=1` keeps it in `.src/`).
+   Streams without the band are remuxed losslessly. Cost: every re-encode (streams and clips) runs x264 on
+   `ENCODE_THREADS` threads (2, set in `systemd/normalize-sweep.service`) so Jellyfin and live viewers stay
+   responsive; a 3 h 1440p60 night therefore takes most of the next day to publish. OBS: dock → *Timecode band* →
    "Add band + clock source" (canvas grows by W/160 rows, a locked browser source `Vanguard Clock` →
-   `/dock/clock/` sits in the band; only while not streaming). Go Live: the "Timecode band" checkbox (default on).
+   `https://live.BASE/dock/clock/` sits in the band; only while not streaming). Go Live: the "Timecode band" checkbox (default on).
 2. **Server arrival time** (no band) — microseconds in the recording filename. Within ~0.1–0.3 s per streamer
    (encoder + network delay); coarse but always available.
 
-Viewing: the live portal crops the band; Jellyfin shows it as a thin strip at the bottom (VHS-style);
+Viewing: the live portal crops the band on the live picture (the only place it still exists); recordings in
-**editors export at the game resolution (e.g. 2560×1440) which drops the band** — finals carry nothing.
+Jellyfin, copyparty and the editors' timelines are already at game resolution. Recordings published before
+the crop existed: `sudo vgdeband.sh [FILE…]` re-encodes them the same way and marks the sidecar.
 Cross-check when it matters: everyone is in the same voice chat, so audio-waveform sync is sample-accurate.
 Why continuous and not a start burst: a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h).
 
@@ -390,6 +565,91 @@ files are not revisited. Clips are always converted to CFR regardless.
 Planned alternative: re-encode flagged streams off-site on a machine with a hardware encoder (home box NVENC or a
 Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
 
+## Deployment (push to deploy — built, OFF by default)
+
+The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea →
+`./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token
+secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout.
+
+`sudo vanguard-deploy pull` fetches `main` and applies **only what changed**, after `check.sh`:
+
+| changed | action | interruption |
+|---|---|---|
+| `configure.py`, `dock/`, `clips/`, `about/`, stignore | `configure.py --apply` → Caddy reload, copyparty restart | none / a second |
+| `bin/*`, `systemd/*` | install; dumpd restart when it changed; timers pick up next run | a second |
+| `firewall/nftables.conf` | `nft -f` | none |
+| `docker-compose.yml`, `bot/Dockerfile`, `bot/entrypoint.sh` | `compose up -d --build` for changed services | that service |
+| `deployBotBundle.sh` | reinstalled as `/usr/local/bin/vanguard-deploy-bot` (what dumpd runs for the bot's CI) | none |
+| anything that changes `mediamtx.yml` or the mediamtx service | **restart only when no stream is live**; otherwise deferred, `deploy-pending.timer` retries every 5 min | none while live |
+| `jellyfin-plugin-vanguard/dist/*.zip` (committed) | `installJellyfinPlugin.sh` | Jellyfin restart |
+| `live/` | `npm run build` on the server → `/srv/live` | none |
+
+`sudo vanguard-deploy rollback` checks out the previously deployed revision and applies the reverse diff.
+`sudo vanguard-deploy status` shows revisions, auto mode, pending restarts, live streams.
+
+**One-time rollout** (the last scp-style deploy; afterwards the server pulls). From the dev box:
+
+    cd ~/claudevs/vanguard-stream && git push
+    scp bin/dumpd.py debian@HOST:~/vanguard-stream/bin/
+    scp systemd/deploy-pending.service systemd/deploy-pending.timer debian@HOST:~/vanguard-stream/systemd/
+    scp docker-compose.yml configure.py deploy.sh switchToClone.sh check.sh install.sh README.md STATUS.md CommunityDiscordMedia.md debian@HOST:~/vanguard-stream/
+    ssh -t debian@HOST 'cd ~/vanguard-stream && sudo apt-get install -y -q git nodejs npm rsync python3-yaml >/dev/null && sudo install -m 0755 bin/dumpd.py /usr/local/bin/ && ./configure.py --apply && sudo systemctl restart dumpd && sudo vanguard-deploy status'
+
+The apply prints the post-start fixups (Syncthing, Gitea admin user, Jellyfin plugin) and ends with a status
+block: `DEPLOY_AUTO=0`, no pending restart, and "not a git checkout" until the next step.
+
+Then switch the server folder to a clone (two steps):
+
+    ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh key'       # prints a public key
+    # Gitea → repo → Settings → Deploy Keys → add it, read-only
+    ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh clone && cd ~/vanguard-stream && ./configure.py --apply >/dev/null && sudo vanguard-deploy status'
+
+`switchToClone.sh clone` keeps `.env`, the generated configs, the token secret and plugin zips, and
+leaves the old copy at `~/vanguard-stream.old` until you delete it. From here on a manual deploy is
+`sudo vanguard-deploy pull` on the server.
+
+**Automatic**: Gitea → repo → Settings → Webhooks → Gitea, URL `http://<docker gateway>:8081/deploy`
+(`docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}'`, usually `172.18.0.1`),
+secret = `DEPLOY_WEBHOOK_SECRET` from `.env`, push events, branch `main`. dumpd verifies the HMAC and runs
+`vanguard-deploy pull` **only when `DEPLOY_AUTO=1`** in `.env` (then `./configure.py --apply` to push the setting
+to `/etc/vanguard/deploy.env`). Until then every push is logged (`journalctl -u dumpd | grep deploy`) and ignored; Gitea's "test delivery"
+should get a 202 "auto-deploy disabled" back. Optional `DEPLOY_NOTIFY_WEBHOOK_URL` posts deploy results to Discord.
+Turning it on: `DEPLOY_AUTO=1` in `.env`, `./configure.py --apply`, and
+`git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml` for CI.
+
+CI: `.gitea/draft-workflows/ci.yml` (inert until moved to `.gitea/workflows/`) runs `check.sh` and the plugin
+tests/build on the Actions runner. The runner never deploys — it must not touch the host.
+
+## Rebuild from scratch (what is and isn't automated)
+
+Server side, scripted:
+
+    git clone ssh://git@git.HOST:2222/techy/Media-Server-Setup.git ~/vanguard-stream   # or scp the folder
+    cd ~/vanguard-stream && sudo bash install.sh        # packages, docker, folders, host scripts, timers, firewall, portal build
+    ./configure.py --full                               # hostnames, secrets, configs, containers, post-start fixups
+
+Restore first if rebuilding a known box: `.env` (all secrets — without it every password is regenerated and
+the databases won't match), `/srv/appdata` (Authentik DB, Jellyfin, Gitea, MariaDB, Syncthing config),
+`/srv/media` (the footage), `/etc/vanguard/clip-quotas.json` if used. With those restored, nothing below is
+needed except DNS.
+
+Post-start fixups `configure.py` performs on every apply (idempotent): Syncthing GUI auth off, Gitea admin
+user created from `.env` (`GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD`; the install page is locked), Jellyfin plugin
+installed from the committed zip. Authentik's `akadmin` password is set on first start from
+`AUTHENTIK_BOOTSTRAP_PASSWORD` in `.env` (no initial-setup page).
+
+Still manual on a truly fresh box (not in the repo; documented in §5c–5g):
+- Authentik: Discord source, identification-stage source, enrollment user type, groups, the two policies,
+  the domain-level proxy provider + outpost host, the Jellyfin and Gitea OIDC providers, application bindings
+- Jellyfin: first-run wizard, libraries (Shows + Vanguard provider), SSO plugin + provider, branding
+- Gitea: authentication source (authentik), deploy key, webhook; `./registerGiteaRunner.sh`
+- Discord bot: its repo lives outside this one, so nothing here restores it. Put `BOT_DEPLOY_SECRET`
+  (`./configure.py --show`), `DISCORD_BOT_TOKEN` and `DISCORD_GUILD_ID` into that repo's Gitea Actions
+  secrets/variables, then push to its `main` — or install a bundle by hand with `./deployBotBundle.sh`.
+  `/srv/bot/app` is created empty by `install.sh`; the container idles until a release lands there.
+- Discord developer portal: application, redirect URIs, bot; DNS records
+A `.env`-driven Authentik blueprint would remove the first line (planned).
+
 ## Layout on the server
 
     /srv/media/streams/<user>/<Title>/.live/<ts>.mp4             recording in progress (hidden from Jellyfin/copyparty)
@@ -401,6 +661,8 @@ Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
     /srv/media/clips/<user>/.stlocal                             Syncthing ignore list maintained by the sweep
     /srv/media/clips/<user>/.src/clip.mkv                        raw source, only for clips that did NOT arrive via Syncthing
     /srv/media/clips/<user>/clip.normalize-failed.log            only if normalization failed; delete to retry
+    /srv/bot/app/                                                the Discord bot's installed release (bind-mounted at /app)
+    /srv/bot/previous/                                           the release before it (deployBotBundle.sh --rollback)
     /srv/media/finals/                                           editors upload here
     /srv/media/trash/<night>/...                                 dumped, purged after 3 days
     /srv/media/.markers/                                         dump markers
@@ -415,7 +677,7 @@ upscaled; aspect kept).
 | input | action |
 |-------|--------|
 | already compliant | nothing |
-| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: decode the timecode band (or use arrival time), lossless remux to `<user>/<Title>__<ts>.mp4` with a `tmcd` track + `creation_time`, write the `.json` sidecar, remove `.live` and the empty title folder |
+| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: decode the timecode band (or use arrival time), then publish to `<user>/<Title>__<ts>.mp4` with a `tmcd` track + `creation_time` — band present: re-encode with the band cropped off; no band: lossless remux — write the `.json` sidecar, remove `.live` and the empty title folder |
 | in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
 | HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
 | ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |
@@ -447,4 +709,4 @@ evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
 |------|------|------|
 | normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
 | trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
-| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy at /dump |
+| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time and on `clips.` at /me/clips |

STATUS.md

@@ -1,7 +1,7 @@
 # Vanguard stack — what runs, what people do, what's unfinished
 
-Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-10-31 (evening).
+Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03.
-Host: `ovh1.dumpstervigilantes.rip` (OVH SYS-1, Debian 13). Bundle: `~/vanguard-stream` on the box.
+Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
 Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).
 
 ## Runs on its own
@@ -11,12 +11,15 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | Continuous | OBS streams record into a hidden `streams/<user>/<Title>/.live/` folder, invisible to Jellyfin and copyparty until finished | MediaMTX |
 | Continuous | Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only) | Syncthing |
 | Continuous | New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab | Jellyfin plugin (needs bot token + channel in its config) |
-| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, lossless remux out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar, one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
+| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
 | Every 5 min | Discord scheduled events snapshotted into a history so recordings can be matched to their session | Jellyfin plugin |
 | Every 30 min + startup | Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box) | Jellyfin plugin |
 | Daily 04:10 ET | Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change | Jellyfin plugin |
+| Every 5 min | Each user's clips folder is measured against their quota (100 GB default, per-user overrides); over quota → their Syncthing folder is paused, under 90 % → resumed; the clips page shows usage | dumpd quota loop — **pending deploy** |
 | Daily 05:00 ET | Dumped nights older than 3 days deleted for good | trash purge (systemd timer) |
 | On boot | Everything restarts itself after the media mount is up; interrupted recordings stay playable | systemd + Docker |
+| Every 5 min | If a deploy deferred a MediaMTX restart, retry when no stream is live | `deploy-pending.timer` — **pending first apply** |
+| On push to `main` | (OFF: `DEPLOY_AUTO=0`) Gitea webhook → dumpd → `vanguard-deploy pull` applies only what changed | **built, disabled** |
 | Continuous | Active streams are watchable live (WebRTC, HLS fallback) by Discord-server members; optional "is live" Discord post when a stream starts | MediaMTX + live portal (page deployed; MediaMTX port fix pending apply) |
 | **Not yet running** | Pull of files older than 15 days to the home HDD | `home/rclone-archive.sh`, needs the home box |
 | **Not yet running** | Nightly backup of the Authentik database (Postgres) | planned |
@@ -26,18 +29,19 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 | Who | Activity | Where | Login |
 |---|---|---|---|
 | Streamer | One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode | OBS dock / Go Live | live |
-| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://ovh1…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** |
+| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://live.…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** |
-| Gamer | Self-serve pairing at `/clips/` (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://ovh1…/clips/` + Syncthing | Discord (any server member) |
+| Gamer | Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://clips.…/` + Syncthing | Discord (any server member) |
-| Everyone | Landing page with role-aware links and instructions | `https://ovh1…/about/` | Discord |
+| Viewer | Live portal lays streams out as an even grid automatically when one joins or leaves ("Arrange" button redoes it after manual moves); audio follows the focused window; double-click a tab to rename a stream (kept in that browser), double-click the slot at the left of a tab bar to name the group (session); once the viewer has moved/resized/renamed anything the layout is left alone: new streams chime and wait as draggable chips in a tray, "Arrange" re-grids and returns to auto | `https://live.…/` | Discord |
-| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; export at the game resolution to drop the band | Resolve / Premiere | — |
+| Everyone | Landing page with role-aware links and instructions | `https://media.…/` | Discord |
-| Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://ovh1…/` | Discord (Authentik); rights by group `editors` / `media-admins` |
+| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep) | Resolve / Premiere | — |
+| Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://files.…/` | Discord (Authentik); rights by group `editors` / `media-admins` |
 | Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin `https://watch.…/` | Discord (Authentik) |
 | Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal `https://live.…/` | Discord (Authentik, domain-level) — **live, shows a real stream** |
 | Streamer (Windows) | **Go Live** from the browser: pick the WarDogs window (+audio on Windows Chrome 141+; Linux/Mac video only), streams as `<user>/HelmetCam-<date>`; no OBS needed | Live portal → Go Live | Discord; key fetched for the logged-in user — **verified end to end** (publish → `.live` → sweep → title folder) |
 | Devs | Host code, review, CI | Gitea `https://git.…/`, ssh 2222 | Discord (Authentik), group `git-users`; `git-admins` = site admin |
 | Bot dev | Push code, restart container, read logs; web UI at `https://bot.…/` | Gitea + compose; `bot/AGENT.md` | web UI behind Discord (Authentik), once its provider exists |
 | Admin | Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups | `./configure.py` | ssh |
-| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI `https://ovh1…/sync/` | Discord + group `media-admins` (Syncthing's own login disabled) |
+| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI `https://sync.…/` | Discord + group `media-admins` (Syncthing's own login disabled) |
 | Admin | Groups, who may log in, app access | Authentik `https://auth.…/` | Discord or akadmin |
 | Admin | Install/upgrade the Jellyfin plugin | `./installJellyfinPlugin.sh` | ssh |
 | Admin | Register the CI runner | `./registerGiteaRunner.sh` | ssh |
@@ -46,35 +50,52 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 
 | Service | Role | State |
 |---|---|---|
-| MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`) | live, token auth verified |
+| MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`); on-ready hook posts "is live" and runs the WebRTC audio mirror (`<path>.webrtc`, audio → Opus, never recorded) | live, token auth verified |
 | copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (`editors`, `media-admins`), Discord-only, no WebDAV | live |
-| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity, `/me/clips` Syncthing self-service; OBS dock with timecode-band setup | identity from Authentik headers | live |
+| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity (on `live.`), `/me/clips` Syncthing self-service (on `clips.`); OBS dock with timecode-band setup | identity from Authentik headers | live |
-| `/about/` gateway, `/clips/` self-service | static pages behind the gate | — | live (first real pairing pending) |
+| `media.` gateway, `clips.` self-service | static pages behind the gate | — | live; **first real pairing 2026-11-01 (thrahg)** — exposed a missing `.stlocal` on fresh folders, fixed |
 | Timecode | `bin/vgstrip.py` decoder, `tmcd` + sidecar in the sweep, `/dock/clock/` strip page, Go Live band composite, portal crop | server clock is the reference | **live, verified** (OBS: frame-accurate, 0 gaps) |
-| Syncthing | clip intake, UI at `/sync/` | UI behind Authentik + `media-admins`, own GUI auth off | live |
+| Syncthing | clip intake, UI at `sync.` | UI behind Authentik + `media-admins`, own GUI auth off | live |
 | normalize sweep | editing standard, one copy per clip | live |
 | nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
-| Caddy | TLS + routing for all hostnames | live |
+| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
+| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
+| Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
+| Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
+| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
+| Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
+| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
 | Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
 | Jellyfin | watch page, SSO via Authentik, direct play only | live |
 | Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `<Title>__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |
 | MariaDB | databases for the bot and Gitea; public :3306, root local-only | live |
 | Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live; Discord login via Authentik OIDC **verified**; access = `git-users`/`git-admins` groups; local admin as fallback |
 | act_runner | CI jobs in Docker | **not registered** |
-| discord-bot | Python runtime, idles until `bot/app/main.py` exists | live, no code yet |
+| `bot.` host + Discord bot | the bot's API **and** its web pages on one port (`WEB_SERVE_UI`), behind Authentik forward-auth; Caddy requires `bot-users`, the bot re-checks `bot-users,admins` per request | a push to `main` deploys itself: CI builds a **release bundle** (app + `vendor/` dependencies) and posts it signed to dumpd `POST /deploy-bot` (bridge-only port, no Docker socket in CI) together with the bot's own secrets, which are managed in Gitea; dumpd verifies everything, writes the keys `BOT_ENV_KEYS` allows into `.env` and swaps the bundle into `/srv/bot/app` → `/app`; image installs no bot dependencies | **reviewed + wired 2026-11-02, first install pending** (needs the Authentik app for `bot.`, and `BOT_DEPLOY_SECRET` + `DISCORD_BOT_TOKEN` + `DISCORD_GUILD_ID` set in Gitea) |
-| `bot.` host | bot web UI behind Authentik forward-auth; Caddy requires group `bot-users` (from a Discord role) | Caddy block live; **`bot` CNAME still points at the wrong IP; roles→groups policy pending** |
 | Live portal (`live/`, `live.` host) | React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd `/me/stream-key` | live; Go Live verified |
 
 ## Unfinished, in rough priority
 
-1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (`/whoami` should show `bot-users`, `git-admins`). `akadmin` remains the recovery login if a policy misfires.
+0a. **WebRTC audio** (found 2026-11-01 with the 5-stream test): AAC cannot cross WebRTC, so the portal was silent unless it fell back to HLS. Fix committed: audio mirror in `bin/mtx-on-ready.sh` + portal mapping. **Deploy pending** (MediaMTX recreate between streams; dumpd restart; portal push). Streamers must set B-frames 0 for WebRTC playback.
+0a. **Discord bot**: live on MariaDB since 2026-11-03 (139 members tracked, 36 commands synced, web UI on the `bot.` host).
+   Old SQLite history merged in the same night with `bin/vgbot-merge-sqlite.py` (the bot's own importer refuses a
+   non-empty destination): Cool Points, training, invites and growth restored, original joined/first-seen/rank dates
+   recovered from the rebuild, live rank and activity kept. Pre-merge dump at `/srv/appdata/discord-bot/before-merge.sql`.
+   The tool is idempotent, so re-running it is safe.
+0. **Band removal**: deployed 2026-11-01; `vgdeband.sh` catch-up ran on the Vanguard-Training recordings (check `journalctl -u vgdeband` for the done lines). Live portal: viewer now clips the band with a measured game-aspect frame (refits on every dockview resize/move) — committed 2026-11-01, deploy with `./deployLive.sh`.
+
+1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (`/whoami` should show `bot-users`). Nested groups **verified 2026-11-01**: `git-admins` now reaches the header via the `Vanguard profile (nested groups)` scope mapping (README 5e). `akadmin` remains the recovery login if a policy misfires.
 2. Confirm plugin 0.1.3 under Jellyfin → Plugins and the Gitea runner registered.
 3. Portal viewer during a real stream: corner chip `webrtc`; Windows Go Live with WarDogs audio (first Windows streamer).
 5. Housekeeping: pin `AUTHENTIK_TAG`, rotate the Discord client secret seen in a screenshot, nightly Postgres dump, Caddyfile formatting warning.
 6. Medal filename pattern in the plugin (need a sample filename).
 7. Home archive puller on the home box.
 8. Plugin repository served by Caddy so Jellyfin updates from its catalog.
-9. Clip submissions (belongs to the bot). Bot code from the other dev.
+9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run
+   (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served
+   from the bot itself now; an Authentik username is matched to one Discord member by name.
+9a. Kibana first-run: a data view over `filebeat-*,metricbeat-*`, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new `vanguard_media_*` metrics.
+9b. Elasticsearch security as its own phase, if `diag.` is ever to widen past `media-admins`: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected `Authorization`). Per-person identity is reachable but buys little, since audit logging is itself paid.
 10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
 11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.
 
@@ -95,3 +116,8 @@ Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea
 - Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
 - Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
 - Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
+- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
+- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
+- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.
+- `find ... | while read` runs the loop in a **subshell**, so anything it accumulates is lost at the end of the pipe. `normalize-sweep.sh` reads from a process substitution for exactly this reason.
+- Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With `xpack.security.enabled: false` every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on `diag.`. Anything in Kibana that asks Elasticsearch for privileges fails with `no handler found for uri [/_security/user/_has_privileges]`. The **Infrastructure / Metrics app does this on every page view**, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over `metricbeat-*` instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work.

UPCOMING.md

@@ -0,0 +1,95 @@
+# Upcoming — what a mil-sim community media stack still needs
+
+Written 2026-11-01 from a conversation between techy and Claude. Three parts: the gap list as first drafted, techy's
+take on it, and the agreed direction. Living plan lives in `CommunityDiscordMedia.md`; current state in `STATUS.md`.
+
+## 1. The gap list (first draft)
+
+Measured against what mil-sim groups actually do with footage, the stack covers capture, storage, access and live
+viewing. The gaps cluster around the debrief workflow, moments, comms audio, and keeping the box healthy.
+
+### Debrief and review (the core mil-sim use)
+- **Synced multi-POV playback.** The timecode track makes it possible, but nothing plays it yet. A VOD mode of the
+  live portal that opens every recording from one night and scrubs them in lockstep turns the archive into an
+  after-action-review tool. The single biggest missing piece.
+- **Ops as a first-class object.** Titles are free text today. An op/training event that groups every streamer's
+  recording, that night's clips, briefing files and finals → collections, per-op download bundle for editors, a
+  Discord thread per op; the dock offers tonight's event instead of a typed title.
+- **Markers.** A "mark this" button in the dock, a Discord command, or an emoji reaction during the live post stamps
+  the server clock into metadata → chapters, editor markers, the shortlist for the review session. Cheap: the clock
+  already exists.
+- **Twitch-style clips from the live recording.** Segments are on disk while someone streams, so a `!clip` of the
+  last 30–60 s is a server-side cut with no streamer action.
+
+### Audio
+- **Separate comms track.** Editors cutting to radio chatter want game audio and voice on separate tracks. RTMP
+  carries one; SRT can carry several and OBS can send multiple audio tracks over it.
+- **Voice server recording.** Some units record TeamSpeak/Mumble as the authoritative comms log. Consent question
+  before a technical one.
+
+### Standardisation
+- **OBS starter kit.** Downloadable scene collection + profile with the band, B-frames 0, bitrate caps, dock URL.
+- **Callsign tags.** The rename feature is per viewer; a server-side label ("1-1 Lead") on live panels and recordings.
+- **Non-video intake.** Briefing slides, map images, after-action notes have no drop point (clip intake is video only
+  by design).
+
+### Health and operations
+- **Disk monitoring and alerts.** Nothing warns when disks fill, when the sweep writes a failure log, or when a
+  stream drops. With hours of 1440p a night this is the most likely quiet death. Discord alert = small job.
+- **Archive tier.** The home archive puller is planned, not built; the 15-day hot window is the only window.
+- **Backups.** Authentik and Gitea databases are not dumped anywhere yet.
+
+### Audience
+- **Public showcase.** Everything is gated; recruitment needs a route for finals to go public (YouTube from finals).
+- **Mobile.** Jellyfin apps work; the live portal is desktop-only in practice.
+
+Suggested top three: alerts (protect everything else), markers (cheap, feed everything downstream), synced
+multi-POV playback (what the archive is actually for).
+
+## 2. techy's take
+
+- **Synced multi-POV playback — yes.** It means more metadata tracking, and we already have too many files: put the
+  metadata in a database (Postgres or MariaDB), not more sidecars.
+- **Ops come with the Discord bot.** It has roll calls; we are waiting on the bot, its website and its database.
+- **Markers — good.** Likely just in the live window for now.
+- **Live clips — nice idea, but** we may instead build something like the live page for *watching*: multi-POV playback
+  plus single-POV playback plus search, and replace Jellyfin entirely (it is only a half-baked fit). Decision for later.
+- **Separate comms is coming-ish.** We run mixed comms: once the bot exists it will listen to a voice channel after
+  roll call. The Go Live button already excludes system audio, so it will carry game audio only. We also use in-game
+  comms, so it will never be perfect.
+
+## 3. Agreed direction (Claude's reply)
+
+Everything lines up on one axis: **the server clock**. Recordings, markers, roll calls and comms audio are all stored
+as UTC instants and joined later, which is exactly what a synced player needs.
+
+**Database.** Use the MariaDB that already runs for the bot rather than adding Postgres: one database to back up, and
+the bot's ops/roll-call tables land beside ours, so "this op → these recordings" is a query, not a convention. Keep the
+per-file sidecar as the self-describing fallback; the sweep writes both; a reconcile command rebuilds the tables from
+sidecars if the database is ever lost. Rough shape:
+
+| table | holds | owner |
+|---|---|---|
+| `recordings` | path, streamer, title, start UTC (band or arrival), duration, fps, timecode source + drift, band removed, gaps, optional op id | sweep |
+| `markers` | UTC instant, author, stream path or null (= everyone), kind, note — stored by **time, not file offset**, so a marker placed live attaches to the recording when published and to every other POV of that moment for free | dumpd (portal), later the bot |
+| `clips` | file, owner, arrival time, converted or not | sweep / dumpd |
+| `ops`, `roll_call` | the bot's; we only read them | bot |
+
+**Markers in the live window.** A Mark button on the focused panel plus a hotkey, posting the current server time and
+the stream path to dumpd. Small, and it gives the future player its chapter list from day one. The Discord-command
+version comes with the bot and writes to the same table.
+
+**Replacing Jellyfin.** The database makes it a clean later decision. A watch app built on the portal's code does
+single POV with search by op, streamer, date and marker, and multi-POV with lockstep scrubbing driven by the timecode.
+Until then Jellyfin keeps reading sidecars and nothing breaks.
+
+**Comms.** Bot-captured Discord audio after roll call, stamped on receipt by the same box, lines up with everything
+else with no extra sync work. In-game comms stay in the game audio; fine for review.
+
+**Sequence.**
+1. Schema + sweep inserts + marker endpoint + Mark button in the live portal (small; everything else reads from it).
+2. Write the table contract into the bot developer's brief (`bot/AGENT.md`) so the bot is designed against it now.
+3. Watch app (single + multi-POV + search) once the bot's ops data exists and the Jellyfin decision is made.
+
+Not scheduled, still true: alerts (disk / sweep failures / dropped streams), backups of Authentik + Gitea databases,
+archive puller, OBS starter kit, callsign tags, non-video intake, public showcase, mobile.

about/index.html

@@ -40,6 +40,17 @@
         <li>Live: windows open as people go live and close when they stop. Click a window to hear it; the speaker button pins audio on; the slider is per window.</li>
       </ul>
     </details>
+    <details><summary>Live page: windows, groups, names</summary>
+      <ul>
+        <li><b>Layout.</b> Until you touch it, windows arrange themselves into an even grid whenever someone goes live or stops. Drag a tab to move a window, drag the bars between windows to resize, drop a tab onto another window to stack them as tabs.</li>
+        <li><b>Your layout stays yours.</b> The moment you move, resize or rename anything, the page stops rearranging. Streams that end just close. New streams <b>do not</b> barge in: a chime plays and a blinking chip appears in a thin tray under the top bar. Drag the chip onto any window edge, onto a tab bar to stack it, or to the page edge; or click it to add it as a new column on the right. <b>Arrange</b> (top bar) places everything in an even grid and switches automatic layout back on.</li>
+        <li><b>Audio.</b> Sound follows the window you click. The speaker button pins a window's audio on even when you click elsewhere; <b>Mute all</b> silences everything until you click a window; <b>Follow focus</b> clears pins.</li>
+        <li><b>Rename a stream.</b> Double-click its tab, type a name, Enter. Escape cancels; an empty name goes back to <code>user · title</code>. Names stay in your browser and come back the next time that streamer uses the same title.</li>
+        <li><b>Name a group.</b> Every tab bar starts with a <code>＋ name</code> slot. Double-click it, type, Enter. Group names last until you reload the page.</li>
+        <li><b>One window big.</b> The ⛶ button at the right of a tab bar maximises that window; click it again to restore. Arranging (a stream joining or leaving, or the Arrange button) restores it too.</li>
+        <li><b>Timecode band.</b> Streams with the timecode band on show it cropped off automatically; recordings never contain it.</li>
+      </ul>
+    </details>
   </section>
 
 
@@ -47,13 +58,14 @@
     <h2>Contribute: stream</h2>
     <p class="lead" data-group="streamers" data-yes="You are in <code>streamers</code>: you can publish under your own name. No passwords, no stream keys to keep." data-no="Admin view — you are <b>not</b> in <code>streamers</code>, so Go Live will refuse a token for you until you get the role."></p>
     <div class="links">
-      <a class="card" href="/dock/"><b>OBS dock</b><span>title → Go Live; Dump tonight; manual setup</span></a>
+      <a class="card" id="l-dock" href="#"><b>OBS dock</b><span>title → Go Live; Dump tonight; manual setup</span></a>
       <a class="card" id="l-golive" href="#"><b>Go Live from the browser</b><span>Windows Chrome: share the game window, no OBS</span></a>
     </div>
     <details><summary>OBS setup (once)</summary>
       <ol>
         <li>OBS → Tools → WebSocket Server Settings → enable, set a password.</li>
         <li>OBS → Docks → Custom Browser Docks → add <code id="dockurl">…/dock/</code>; log in with Discord inside the dock.</li>
+        <li>OBS → Settings → Output → encoder <b>B-frames: 0</b> (NVENC "Max B-frames"). With B-frames the live page cannot play you over WebRTC and falls back to a delayed HLS view.</li>
         <li>Dock → Settings: OBS WebSocket password (the rest is prefilled). Optional title prefix/suffix (<code>{date}</code> = today).</li>
         <li>Dock → Timecode band → <b>Add band + clock source</b> while not streaming (gives editors a frame-accurate timecode).</li>
         <li>Per stream: type a title, <b>Go Live</b>. The dock mints a 12-hour token and sets your stream key itself.</li>
@@ -72,12 +84,12 @@
     <h2>Contribute: your game clips</h2>
     <p class="lead">Anyone in the server. Pair your gaming PC once; after that your clips upload themselves and appear in Jellyfin under your name.</p>
     <div class="links">
-      <a class="card" href="/clips/"><b>Clip setup &amp; status</b><span>pair your PC, see if it is connected and syncing</span></a>
+      <a class="card" id="l-clips" href="#"><b>Clip setup &amp; status</b><span>pair your PC, see if it is connected and syncing</span></a>
     </div>
     <details open><summary>Three steps</summary>
       <ol>
         <li>Install <a href="https://syncthing.net/downloads/" target="_blank" rel="noopener">Syncthing</a> on the PC your clips save to (Windows: Syncthing Tray works well). Open it → Actions → Show ID.</li>
-        <li>Paste that Device ID on the <a href="/clips/">clips page</a> and press Connect.</li>
+        <li>Paste that Device ID on the <a class="l-clips" href="#">clips page</a> and press Connect.</li>
         <li>Back in your Syncthing: accept the new device <b>Vanguard</b>, then accept the folder it offers, point it at your clip directory (ShadowPlay / Medal / Steam), Folder Type <b>Send Only</b>.</li>
       </ol>
       <ul>
@@ -91,8 +103,8 @@
     <h2>Edit</h2>
     <p class="lead" data-group="editors" data-yes="You are in <code>editors</code>: read streams and clips, write finals." data-no="Admin view — you are not in <code>editors</code>; your copyparty rights come from <code>media-admins</code> instead."></p>
     <div class="links">
-      <a class="card" href="/"><b>Browse &amp; download</b><span>copyparty — scrub in the browser, download originals, zip folders</span></a>
+      <a class="card" id="l-files" href="#"><b>Browse &amp; download</b><span>copyparty — scrub in the browser, download originals, zip folders</span></a>
-      <a class="card" href="/finals/"><b>Upload finals</b><span>the only folder editors can write to</span></a>
+      <a class="card" id="l-finals" href="#"><b>Upload finals</b><span>the only folder editors can write to</span></a>
     </div>
     <details><summary>Multicam &amp; timecode</summary>
       <ul>
@@ -108,15 +120,15 @@
     <h2>Media admin</h2>
     <p class="lead" data-group="media-admins" data-yes="You are in <code>media-admins</code>: full rights on every folder, trash, Syncthing, dump anyone's night." data-no="Admin view — you are <b>not</b> in <code>media-admins</code>: copyparty will show you nothing and <code>/sync/</code> will 403 until you get the role."></p>
     <div class="links">
-      <a class="card" href="/trash/"><b>Trash</b><span>dumped nights, purged after 3 days</span></a>
+      <a class="card" id="l-trash" href="#"><b>Trash</b><span>dumped nights, purged after 3 days</span></a>
-      <a class="card" href="/sync/"><b>Syncthing</b><span>device pairing, folder states</span></a>
+      <a class="card" id="l-sync" href="#"><b>Syncthing</b><span>device pairing, folder states</span></a>
       <a class="card" href="/whoami"><b>Who am I</b><span>identity headers as the apps see them</span></a>
     </div>
     <details><summary>Admin notes</summary>
       <ul>
         <li>Group changes in Authentik apply on the next login: <a href="/logout">sign out</a> and back in.</li>
         <li>Dump anyone's night: the dock's Dump works on your own; for others use the API with <code>{"user":"name"}</code> or ask the owner.</li>
-        <li>New gamer: they self-serve on <a href="/clips/">/clips/</a>; you only need to watch Syncthing accept the device.</li>
+        <li>New gamer: they self-serve on the <a class="l-clips" href="#">clips page</a>; you only need to watch Syncthing accept the device.</li>
       </ul>
     </details>
   </section>
@@ -156,7 +168,7 @@
     </details>
   </section>
 
-  <footer>Vanguard Media · <a href="/about/">/about</a> · <a href="/logout">sign out</a></footer>
+  <footer>Vanguard Media · <a href="/">home</a> · <a href="/logout">sign out</a></footer>
 </main>
 <script>
 const $ = id => document.getElementById(id);
@@ -165,12 +177,15 @@ const sub = s => `https://${s}.${base}`;
 $('l-watch').href = sub('watch'); $('l-live').href = sub('live'); $('l-golive').href = sub('live') + '/#golive';
 $('l-git').href = sub('git'); $('l-bot').href = sub('bot'); $('l-auth').href = sub('auth') + '/if/admin/';
 $('l-jfadmin').href = sub('watch') + '/web/#/dashboard'; $('l-gitadmin').href = sub('git') + '/-/admin';
-$('dockurl').textContent = location.origin + '/dock/'; $('gitssh').textContent = `ssh://git@${'git.' + base}:2222/org/repo.git`;
+$('l-dock').href = sub('live') + '/dock/'; $('dockurl').textContent = sub('live') + '/dock/';
+$('l-files').href = sub('files') + '/'; $('l-finals').href = sub('files') + '/finals/'; $('l-trash').href = sub('files') + '/trash/'; $('l-sync').href = sub('sync') + '/';
+$('l-clips').href = sub('clips') + '/'; document.querySelectorAll('a.l-clips').forEach(a => a.href = sub('clips') + '/'); $('gitssh').textContent = `ssh://git@${'git.' + base}:2222/org/repo.git`;
 (async () => {
   try {
     const r = await fetch('/me', {credentials:'same-origin'}); const d = await r.json(); if (!r.ok) throw new Error(d.error || r.status);
     $('who').textContent = d.name && d.name !== d.user ? `${d.name} (${d.user})` : d.user;
     $('pills').innerHTML = d.groups.map(g => `<span class="pill">${g}</span>`).join('') || '<span class="pill">no groups yet — ask an admin for a role</span>';
+    if (d.discord_id) $('who').title = 'Discord ID ' + d.discord_id;   // apps identify you by this, not by your name
     const has = g => d.groups.includes(g);
     const show = (id, on) => $(id).classList.toggle('hidden', !on);
     // sections appear only for what this account can actually do (the gates check these same groups)

addClipUser.sh

@@ -6,5 +6,6 @@ cd "$(dirname "$0")"
 u="${1:-}"; [[ $u =~ ^[a-z0-9_.-]{2,32}$ ]] || { echo "usage: $0 <username>  (lowercase letters, digits, _ . -)"; exit 1; }
 sudo install -d -o 1000 -g 1000 "/srv/media/clips/$u"
 sudo install -m 0644 -o 1000 -g 1000 syncthing/stignore "/srv/media/clips/$u/.stignore"
+[[ -e "/srv/media/clips/$u/.stlocal" ]] || sudo install -m 0644 -o 1000 -g 1000 /dev/null "/srv/media/clips/$u/.stlocal"   # #include'd by .stignore
 echo "ready: /srv/media/clips/$u  (Syncthing path: /var/syncthing/clips/$u)"
-echo "next: pair their device in https://$(grep -E '^# vanguard:main' -A1 Caddyfile 2>/dev/null | tail -1 | awk '{print $1}')/sync/ and accept their folder at that path, Receive Only, min free 300 GB"
+echo "next: the gamer pairs at the clips host (self-service), or you accept their folder in the Syncthing UI at that path, Receive Only, min free 300 GB"

addCommunityMediaServer.sh

@@ -24,7 +24,7 @@ read -r AUTH_CUR WATCH_CUR < <(python3 -c 'import configure as c; h=c.read_hosts
 BASE="${MAIN#*.}"; [[ $MAIN == *.*.* ]] || BASE="$MAIN"
 
 say "Community media server — Phase 1 (Authentik + Jellyfin) on $MAIN"
-echo "Needs two DNS names as CNAMEs to $MAIN. Create them now if you haven't."
+echo "Every hostname below is a CNAME to $MAIN. Create them now if you haven't."
 AUTH=$(ask "Authentik hostname" "${AUTH_CUR:-auth.$BASE}")
 WATCH=$(ask "Jellyfin hostname" "${WATCH_CUR:-watch.$BASE}")
 GIT_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("git",""))')
@@ -33,11 +33,19 @@ BOT_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("bot",""))
 BOT=$(ask "Bot web UI hostname" "${BOT_CUR:-bot.$BASE}")
 LIVE_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("live",""))')
 LIVE=$(ask "Live portal hostname" "${LIVE_CUR:-live.$BASE}")
+FILES_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("files",""))')
+FILES=$(ask "Files (copyparty) hostname" "${FILES_CUR:-files.$BASE}")
+SYNC_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("sync",""))')
+SYNC=$(ask "Syncthing UI hostname" "${SYNC_CUR:-sync.$BASE}")
+CLIPS_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("clips",""))')
+CLIPS=$(ask "Clips self-service hostname" "${CLIPS_CUR:-clips.$BASE}")
+DIAG_CUR=$(python3 -c 'import configure as c; print(c.read_hosts().get("diag",""))')
+DIAG=$(ask "Diagnostics hostname: logs, metrics, alerts" "${DIAG_CUR:-diag.$BASE}")
 
 # ---- DNS sanity ----------------------------------------------------------------------------------
 say "DNS check"
 MAIN_IP=$(ip_of "$MAIN"); ok=1
-for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE"; do
+for h in "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG"; do
   ip=$(ip_of "$h")
   if [[ -z $ip ]]; then echo "  $h: does not resolve yet"; ok=0
   elif [[ $ip != "$MAIN_IP" ]]; then echo "  $h -> $ip (main is $MAIN_IP) — points somewhere else"; ok=0
@@ -50,7 +58,7 @@ fi
 
 # ---- write configs, .env, start containers -----------------------------------------------------
 say "Applying (configure.py --hosts)"
-VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" | sed -n '/== writing/,$p' | grep -vE '^  (techy|diceofire|[a-z0-9_-]+)  password' || true
+VG_DRY=$DRY python3 configure.py --hosts "$AUTH" "$WATCH" "$GIT" "$BOT" "$LIVE" "$FILES" "$SYNC" "$CLIPS" "$DIAG" | sed -n '/== writing/,$p' | grep -vE '^  (techy|diceofire|[a-z0-9_-]+)  password' || true
 [[ $DRY == 1 ]] && { echo "(dry run: skipping container wait)"; exit 0; }
 
 # ---- wait ---------------------------------------------------------------------------------------
@@ -150,22 +158,23 @@ PHASE 3 — Jellyfin (direct play only on this CPU)
           Custom CSS:       paste jellyfin/branding.css
         -> Save; reload https://$WATCH/
 
-AUTHENTIK FORWARD-AUTH (gates https://$LIVE/ and https://$BOT/; later the main host) — once:
+AUTHENTIK FORWARD-AUTH (gates every Vanguard host: $MAIN, $FILES, $CLIPS, $SYNC, $LIVE, $DIAG, $BOT) — once:
   1. Applications -> Outposts -> edit "authentik Embedded Outpost" -> config: authentik_host: https://$AUTH
   2. Applications -> Create with provider -> Application "Vanguard sites"; Proxy Provider, authorization
      flow implicit-consent, mode "Forward auth (domain level)", authentication URL https://$AUTH,
      cookie domain ${BASE}, Invalidation flow default-provider-invalidation-flow (logout = full re-login)
      -> then edit the Embedded Outpost -> Applications: add it -> Update
   3. Per-host access is enforced by Caddy from X-Authentik-Groups: .env BOT_REQUIRE_GROUP=bot-users
-     (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member). ./configure.py apply regenerates.
+     (default), LIVE_REQUIRE_GROUP= (empty = any logged-in member), DIAG_REQUIRE_GROUP=media-admins.
+   ./configure.py apply regenerates.
   4. Discord roles -> Authentik groups: Directory -> Groups: create bot-users (keep streamers/editors/admins);
      Discord source -> Scopes: "guilds guilds.members.read"; Customization -> Policies -> Expression Policy
      "discord-roles-to-groups" from authentik/discord-roles-to-groups.py (server id + role-id->group table);
      bind to default-source-enrollment AND default-source-authentication (engine mode all).
   5. Test: private window -> https://$LIVE/ -> Discord -> portal; https://$BOT/ -> 403 unless you hold the
-     bot role; https://$LIVE/me/stream-key shows your username.
+     bot role; https://$LIVE/me shows your username and groups.
 
-LIVE PORTAL (https://$LIVE/)
+LIVE PORTAL (https://$LIVE/) — also hosts the OBS dock (https://$LIVE/dock/) and the timecode clock source.
   Deploy the page from the dev box: ./deployLive.sh (builds live/ and copies dist/ to /srv/live).
   WebRTC viewers and Go Live need UDP/TCP 8189 open (firewall rule included). Optional "is live" Discord
   post: LIVE_WEBHOOK_URL in .env (asked by configure.py --full).

authentik/discord-roles-to-groups.py

@@ -1,20 +1,27 @@
 # Authentik expression policy: sync a member's Discord ROLES into Authentik GROUPS at every login.
 # Paste into: Customization -> Policies -> Create -> Expression Policy  (name: discord-roles-to-groups)
 # Bind to BOTH flows (after discord-guild-member): default-source-enrollment and default-source-authentication.
+# Tick "Execution logging" on the policy: everything below shows up in Events -> Logs -> policy_execution
+# under result.log_messages, and in `docker logs authentik-server`.
 # Requires the "guilds.members.read" scope on the Discord source (source -> Protocol settings -> Scopes:
 #   guilds guilds.members.read ). Existing users re-consent once at Discord.
 # The Authentik groups must already exist (Directory -> Groups). Role IDs: Discord Developer Mode ->
 #   Server Settings -> Roles -> right-click a role -> Copy Role ID.
 # Manual memberships are safe: the policy only removes groups it granted itself (user attribute discord_role_groups).
 
-GUILD_ID = "1387203044084551690"                         # <- your Discord server ID
+GUILD_ID = "1387203044084551690"                        # <- your Discord server ID
-ROLE_TO_GROUP = {                                       # <- Discord role id : Authentik group name
+
-    "1387239222070612110": "bot-users",
+# Discord role id : (label used in the log, [Authentik groups this role grants])
-    "1515901670943490129": "bot-users",
+# One role may grant several groups, and several roles may grant the same group. Do NOT repeat a role id:
-#    "222222222222222222": "streamers",
+# this is a dict, so a second entry for the same id silently replaces the first.
-#    "333333333333333333": "editors",
+ROLES = {
-#    "444444444444444444": "admins",
+    "1387239222070612110": ("Discord Admins", ["bot-users", "admins"]),
-    "555555555555555555": "git-users",
+    "1515901670943490129": ("Lead Dev",       ["bot-users"]),
+    "1566623319027880086": ("Dev",            ["bot-users"]),
+#   "222222222222222222": ("Streamer",        ["streamers"]),
+#   "333333333333333333": ("Editor",          ["editors"]),
+#   "444444444444444444": ("Media admin",     ["media-admins"]),
+#   "555555555555555555": ("Git user",        ["git-users"]),
 }
 
 from authentik.core.models import Group
@@ -25,6 +32,7 @@ if not source or getattr(source, "provider_type", "") != "discord":
 
 connection = context.get("goauthentik.io/sources/connection")
 if not connection:
+    ak_logger.info("discord-roles: no source connection in context; nothing to sync")
     return True
 
 resp = requests.get(
@@ -33,13 +41,35 @@ resp = requests.get(
     timeout=10,
 )
 if resp.status_code != 200:
+    ak_logger.info(f"discord-roles: member lookup failed, HTTP {resp.status_code}: {resp.text[:200]}")
     ak_message(f"Could not read your Discord roles ({resp.status_code}); groups unchanged.")
     return True                                          # don't block login on a Discord hiccup
 
 roles = set(resp.json().get("roles", []))
-wanted = {g for r, g in ROLE_TO_GROUP.items() if r in roles}
-managed = set(ROLE_TO_GROUP.values())
 
+# ---- what Discord says, what we map, what we ignore ------------------------------------------------
+matched = {rid: ROLES[rid] for rid in roles if rid in ROLES}
+unmapped = sorted(roles - set(ROLES))
+wanted = {group for _, groups in matched.values() for group in groups}
+managed = {group for _, groups in ROLES.values() for group in groups}
+
+ak_logger.info(f"discord-roles: discord returned {len(roles)} role(s): {sorted(roles)}")
+for rid, (label, groups) in sorted(matched.items(), key=lambda item: item[1][0]):
+    ak_logger.info(f"discord-roles: MAPPED   role {rid} ({label}) -> {groups}")
+if unmapped:
+    ak_logger.info(f"discord-roles: ignored  {len(unmapped)} role(s) not in the table: {unmapped}")
+if not matched:
+    ak_logger.info(f"discord-roles: no mapped roles; the table knows {sorted(ROLES)}")
+ak_logger.info(f"discord-roles: groups wanted {sorted(wanted)}; managed by this policy {sorted(managed)}; "
+               f"withheld {sorted(managed - wanted)}")
+
+# A group named in the table but absent from Authentik can never be granted: say so instead of failing quietly.
+existing = {group.name: group for group in Group.objects.filter(name__in=managed)}
+absent = sorted(managed - set(existing))
+if absent:
+    ak_logger.info(f"discord-roles: WARNING these groups do not exist in Authentik and were skipped: {absent}")
+
+# ---- apply ------------------------------------------------------------------------------------------
 # Authentication flow: the user exists -> edit memberships directly.
 # Only memberships THIS policy granted earlier are ever removed (tracked in user.attributes["discord_role_groups"]),
 # so a group someone was added to by hand in Authentik survives role changes and re-logins.
@@ -47,23 +77,31 @@ user = context.get("pending_user") or (request.user if request.user.is_authentic
 if user and user.pk:
     attrs = user.attributes or {}
     granted = set(attrs.get("discord_role_groups", []))
-    for group in Group.objects.filter(name__in=managed):
+    added, removed = [], []
-        if group.name in wanted:
+    for name, group in existing.items():
-            group.users.add(user)
+        if name in wanted and name not in granted:
-            granted.add(group.name)
+            group.users.add(user); granted.add(name); added.append(name)
-        elif group.name in granted:
+        elif name in wanted:
-            group.users.remove(user)          # role gone -> take back what the role had given
+            group.users.add(user)                 # idempotent: make sure the membership is really there
-            granted.discard(group.name)
+        elif name in granted:
+            group.users.remove(user)                  # role gone -> take back what the role had given
+            granted.discard(name); removed.append(name)
     attrs["discord_role_groups"] = sorted(granted)
     user.attributes = attrs
     user.save()
+    ak_logger.info(f"discord-roles: authentication flow, user {user.username} (pk {user.pk}): "
+                   f"added {added or '[]'}, removed {removed or '[]'}, role-granted now {sorted(granted)}")
     return True
 
 # Enrollment flow: the user is created later by the user-write stage; hand it the groups to attach and
-# record them as role-granted so a later role removal can revoke exactly these.
+# record them as role-granted so a later role removal can revoke exactly these. The flow plan is reached
+# through context["flow_plan"] on some versions and IS context on others, so write to whichever is there.
+attach = [group for name, group in existing.items() if name in wanted]
 plan = context.get("flow_plan")
-if plan is not None:
+target = plan.context if plan is not None else context
-    plan.context["groups"] = list(Group.objects.filter(name__in=wanted))
+target["groups"] = attach
-    prompt = plan.context.setdefault("prompt_data", {})
+prompt = target.setdefault("prompt_data", {})
-    prompt.setdefault("attributes", {})["discord_role_groups"] = sorted(wanted)
+prompt.setdefault("attributes", {})["discord_role_groups"] = sorted(wanted)
+ak_logger.info(f"discord-roles: enrollment flow via {'flow_plan' if plan is not None else 'policy context'}: "
+               f"attaching {[group.name for group in attach]}, recording {sorted(wanted)}")
 return True

authentik/vanguard-profile-scope-mapping.py

@@ -0,0 +1,38 @@
+# Authentik SCOPE MAPPING (not a policy): adds two things the apps behind the proxy need and the built-in
+# mappings do not provide.
+#
+# Paste into: Customization -> Property Mappings -> Create -> Scope Mapping
+#   Name:        Vanguard profile (nested groups + discord id)
+#   Scope name:  profile
+# Then tick it, NEXT TO the managed "authentik default OAuth Mapping: OpenID 'profile'", on every provider the
+# community uses: the "Vanguard sites" proxy provider, Jellyfin and Gitea (Advanced protocol settings -> Scopes).
+# Claims from several mappings are merged, so both run and neither needs removing. Proxy providers re-add the
+# managed one on every save, which is why this adds to it rather than replacing it.
+#
+# 1. groups: the managed mapping emits ak_groups, i.e. DIRECT memberships only, so a group inherited through a
+#    parent never reaches the apps. This emits only the inherited ones; merged with the managed mapping's direct
+#    ones the apps see the full set, with no duplicates.
+#
+# 2. discord_id: the Discord snowflake, taken from the source connection Authentik already stores when it links
+#    an account. It reaches the apps in X-Authentik-Jwt. The Discord bot uses it to identify the caller exactly,
+#    instead of matching an Authentik username against Discord names — names are mutable, reusable, and server
+#    nicknames are not unique, so name matching can both lock the right person out and let the wrong one in.
+#
+# Takes effect on the next login.
+
+DISCORD_SOURCE_SLUG = "discord"
+
+direct = set(request.user.ak_groups.all())
+inherited = sorted(group.name for group in request.user.all_groups() if group not in direct)
+
+discord_id = None
+for attribute in ("usersourceconnection_set", "source_connections"):      # the related name differs by version
+    related = getattr(request.user, attribute, None)
+    if related is None:
+        continue
+    connection = (related.filter(source__slug=DISCORD_SOURCE_SLUG).first() or related.first())
+    if connection is not None:
+        discord_id = getattr(connection, "identifier", None) or None
+    break
+
+return {"groups": inherited, "discord_id": discord_id}

bin/dumpd.py

@@ -9,6 +9,20 @@ GET  /me/stream-token -> short-lived publish token for the caller (member of str
 GET  /me/clips  -> Syncthing self-service status for the caller: our device id, their folder/device state
 POST /me/clips  -> {"device_id": "..."}: create clips/<user> (+ .stignore), add their device to our Syncthing,
              create the receive-only folder clips-<user> shared with them (min free 300 GB). Idempotent.
+POST /deploy    -> Gitea push webhook (bridge port only, HMAC-verified with DEPLOY_WEBHOOK_SECRET from
+             /etc/vanguard/deploy.env). Runs /usr/local/bin/vanguard-deploy pull ONLY when DEPLOY_AUTO=1;
+             otherwise logs the push and answers 202 "auto-deploy disabled".
+POST /heartbeat  -> LEGACY, nothing calls this now. It was the Watchdog route when Alertmanager, and then a
+                   Kibana webhook connector, posted here. A basic Elastic licence has no webhook connector, so
+                   Kibana's rules write into the vanguard-alerts index instead and the deadman reads the
+                   heartbeat from there. Kept because it costs nothing and is a ready-made target if an
+                   external monitor is ever added.
+POST /deploy-bot -> Gitea Actions (job container -> host gateway) hands over a Discord-bot release bundle,
+                   HMAC-SHA256 signed with BOT_DEPLOY_SECRET; installs it with vanguard-deploy-bot.
+                   May carry X-Vanguard-Env (base64 JSON): the bot's own settings, kept in Gitea, written
+                   into the stack .env for the keys BOT_ENV_KEYS allows. Signature covers header+body.
+GET  /metrics -> Prometheus text for Metricbeat: the media scripts' counters (bin/vgmetrics.sh) plus live
+             gauges for the normalise queue and MediaMTX. Loopback listener only; Caddy routes nothing here.
 POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish iff user/pass carries a valid
              token whose user owns the path prefix. Tokens are HMAC-signed with /etc/vanguard/stream-token.secret.
 
@@ -16,7 +30,7 @@ POST /mtx-auth -> MediaMTX authHTTP hook: allows read/playback; allows publish i
 Default night: today if it is 17:00 or later, else yesterday (so 00:00-02:00 still counts as last night).
 Users may dump only their own prefix; members of the admins / media-admins groups may pass "user" to dump anyone, or "all".
 """
-import base64, hashlib, hmac, json, os, re, shutil, sys
+import base64, hashlib, hmac, json, os, re, shutil, subprocess, sys, threading
 import time as _time
 from datetime import datetime, timedelta, time
 from zoneinfo import ZoneInfo
@@ -39,6 +53,27 @@ TZ = ZoneInfo(os.environ.get("DUMP_TZ", "America/New_York"))
 NIGHT_START = time(17, 0)   # 5 pm
 NIGHT_END   = time(2, 0)    # 2 am next day
 
+AUTH_JWT_HEADER = os.environ.get("AUTH_JWT_HEADER", "X-Authentik-Jwt")
+
+def proxy_claim(headers, name):
+    """One claim out of the token the proxy injects, or None.
+
+    Authentik's proxy puts its signed token in X-Authentik-Jwt; a scope mapping on the provider adds the claims
+    we care about (authentik/vanguard-profile-scope-mapping.py supplies discord_id). The payload is only read,
+    never trusted beyond what the sibling headers already assert: all of them come from the same proxy, which is
+    the only thing that can reach this port. Returned for display, never for an authorisation decision here.
+    """
+    token = (headers.get(AUTH_JWT_HEADER) or "").strip()
+    if token.count(".") != 2:
+        return None
+    body = token.split(".")[1]
+    try:
+        claims = json.loads(base64.urlsafe_b64decode(body + "=" * (-len(body) % 4)))
+    except (ValueError, TypeError):
+        return None
+    value = claims.get(name)
+    return str(value) if value not in (None, "") else None
+
 def identity(headers):
     """(username, is_admin) from the headers Caddy sets after Authentik; (None, False) if absent.
     dumpd binds to loopback only, so these headers can only come from Caddy."""
@@ -100,6 +135,62 @@ ST_CONFIG = Path(os.environ.get("SYNCTHING_CONFIG", "/srv/media/.syncthing/confi
 ST_CLIPS_PREFIX = os.environ.get("SYNCTHING_CLIPS_PREFIX", "/var/syncthing/clips")      # path INSIDE the syncthing container
 STIGNORE_SRC = Path(os.environ.get("STIGNORE_SRC", "/etc/vanguard/stignore"))
 MIN_FREE_GB = int(os.environ.get("CLIPS_MIN_FREE_GB", "300"))
+CLIPS_QUOTA_GB = float(os.environ.get("CLIPS_QUOTA_GB", "100"))          # default per-user quota for clips/<user>
+CLIPS_QUOTA_FILE = Path(os.environ.get("CLIPS_QUOTA_FILE", "/etc/vanguard/clip-quotas.json"))   # {"user": GB, ...} overrides
+QUOTA_CHECK_S = int(os.environ.get("CLIPS_QUOTA_CHECK_S", "300"))
+QUOTA_STATE = Path(os.environ.get("CLIPS_QUOTA_STATE", "/srv/media/.markers/clip-quota-paused.json"))
+
+def quota_for(user):
+    try:
+        q = json.loads(CLIPS_QUOTA_FILE.read_text()).get(user)
+        if q is not None: return float(q)
+    except (OSError, json.JSONDecodeError, ValueError):
+        pass
+    return CLIPS_QUOTA_GB
+
+def folder_bytes(path):
+    total = 0
+    for root, dirs, files in os.walk(path):
+        for f in files:
+            try: total += os.lstat(os.path.join(root, f)).st_size
+            except OSError: pass
+    return total
+
+def quota_state():
+    try: return json.loads(QUOTA_STATE.read_text())
+    except (OSError, json.JSONDecodeError): return {}
+
+def quota_enforce_once():
+    """Pause a user's Syncthing folder when clips/<user> exceeds their quota; resume when back under 90 %."""
+    paused = quota_state(); changed = False
+    try:
+        folders = st("GET", "/rest/config/folders")
+    except Exception as e:
+        log("quota_unreachable", level="warn", error=e); return
+    for f in folders:
+        if not f["id"].startswith("clips-"): continue
+        user = f["id"][len("clips-"):]
+        path = Path(ROOT) / "clips" / user
+        if not path.is_dir(): continue
+        used = folder_bytes(path); q = quota_for(user) * 1e9
+        over = used > q; under = used < q * 0.9
+        if over and not f.get("paused"):
+            try: st("POST", f"/rest/db/pause?folder={f['id']}"); paused[user] = {"at": datetime.now(TZ).isoformat(), "used_gb": round(used / 1e9, 1), "quota_gb": q / 1e9}; changed = True
+            except Exception as e: log("quota_pause_failed", level="error", user=user, error=e); continue
+            log("quota_paused", level="warn", user=user, folder=f"clips-{user}", used_gb=f"{used/1e9:.1f}", quota_gb=f"{q/1e9:.0f}")
+        elif under and f.get("paused") and user in paused:
+            try: st("POST", f"/rest/db/resume?folder={f['id']}"); paused.pop(user, None); changed = True
+            except Exception as e: log("quota_resume_failed", level="error", user=user, error=e); continue
+            log("quota_resumed", user=user, folder=f"clips-{user}", used_gb=f"{used/1e9:.1f}")
+    if changed:
+        try: QUOTA_STATE.parent.mkdir(exist_ok=True); QUOTA_STATE.write_text(json.dumps(paused, indent=2))
+        except OSError: pass
+
+def quota_loop():
+    while True:
+        try: quota_enforce_once()
+        except Exception as e: log("quota_loop_error", level="error", error=e)
+        _time.sleep(QUOTA_CHECK_S)
 
 def st_apikey():
     try:
@@ -146,6 +237,11 @@ def clips_status(user):
     except Exception as e:
         out["error"] = f"syncthing: {e}"
     out["folder_exists"] = Path(out["server_path"]).is_dir()
+    out["quota_gb"] = quota_for(user)
+    out["used_gb"] = round(folder_bytes(out["server_path"]) / 1e9, 2) if out["folder_exists"] else 0.0
+    ps = quota_state().get(user)
+    out["paused_for_quota"] = bool(ps)
+    if ps: out["paused_since"] = ps.get("at")
     return out
 
 def clips_setup(user, device_id, device_name=None):
@@ -158,6 +254,9 @@ def clips_setup(user, device_id, device_name=None):
     ign = d / ".stignore"
     if not ign.exists() and STIGNORE_SRC.exists():
         ign.write_text(STIGNORE_SRC.read_text()); shutil.chown(ign, 1000, 1000); ign.chmod(0o644)
+    loc = d / ".stlocal"                     # #include'd by .stignore; Syncthing refuses to scan while it is missing
+    if not loc.exists():
+        loc.write_text(""); shutil.chown(loc, 1000, 1000); loc.chmod(0o644)
     # 2. their device
     my_id = st("GET", "/rest/system/status")["myID"]
     if device_id == my_id: raise ValueError("that is the server's own Device ID — paste yours (your Syncthing → Actions → Show ID)")
@@ -180,9 +279,118 @@ def clips_setup(user, device_id, device_name=None):
             cur["devices"].append({"deviceID": device_id})
         cur["type"] = "receiveonly"; cur["minDiskFree"] = spec["minDiskFree"]
         st("PUT", f"/rest/config/folders/{fid}", cur)
-    print(f"{datetime.now(TZ):%F %T} clips setup user={user} device={device_id[:7]}… folder={fid}", flush=True)
+    log("clips_setup", user=user, device=device_id[:7], folder=fid)
     return clips_status(user)
 
+DEPLOY_ENV = Path(os.environ.get("DEPLOY_ENV_FILE", "/etc/vanguard/deploy.env"))
+def deploy_env():
+    d = {}
+    try:
+        for line in DEPLOY_ENV.read_text().splitlines():
+            if "=" in line and not line.startswith("#"): k, v = line.split("=", 1); d[k.strip()] = v.strip()
+    except OSError:
+        pass
+    return d
+
+BOT_ENV_DEFAULT_KEYS = "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID"
+BOT_ENV_MAX_HEADER = 8192
+_env_write_lock = threading.Lock()
+
+def stack_env_path():
+    """The stack's .env — docker compose interpolates ${...} from it, so the bot's settings live there."""
+    repo = deploy_env().get("DEPLOY_REPO_DIR", "")
+    return (Path(repo) / ".env") if repo else None
+
+def apply_bot_env(values):
+    """Merge allowed keys into the stack .env. Returns (applied, refused); never logs a value."""
+    allowed = {k.strip() for k in deploy_env().get("BOT_ENV_KEYS", BOT_ENV_DEFAULT_KEYS).split(",") if k.strip()}
+    refused = sorted(k for k in values if k not in allowed)
+    wanted = {k: str(v) for k, v in values.items() if k in allowed}
+    if not wanted:
+        return [], refused
+    path = stack_env_path()
+    if path is None or not path.is_file():
+        raise OSError("DEPLOY_REPO_DIR is not set or holds no .env")
+    bad = [k for k, v in wanted.items() if not re.fullmatch(r"[A-Z][A-Z0-9_]{1,48}", k) or "\n" in v or "\r" in v]
+    if bad:
+        raise ValueError(f"not a usable env assignment: {bad}")
+    with _env_write_lock:
+        lines = path.read_text().splitlines()
+        seen = set()
+        for i, line in enumerate(lines):
+            if "=" in line and not line.lstrip().startswith("#"):
+                key = line.split("=", 1)[0].strip()
+                if key in wanted:
+                    lines[i] = f"{key}={wanted[key]}"; seen.add(key)
+        for key in wanted:
+            if key not in seen: lines.append(f"{key}={wanted[key]}")
+        stat = path.stat()
+        tmp = path.with_suffix(".env.incoming")
+        tmp.write_text("\n".join(lines) + "\n")
+        os.chmod(tmp, stat.st_mode & 0o777); os.chown(tmp, stat.st_uid, stat.st_gid)
+        os.replace(tmp, path)
+    applied = sorted(wanted)
+    log("bot_env_applied", keys=",".join(applied), refused=",".join(refused) or None)
+    return applied, refused
+
+HEARTBEAT_FILE = Path(os.environ.get("ALERT_HEARTBEAT_FILE", "/srv/media/.markers/alert-heartbeat"))
+
+_bot_deploy_lock = threading.Lock()
+BOT_DEPLOY_MAX_BYTES = 300 * 1024 * 1024
+BOT_DEPLOY_INSTALLER = os.environ.get("BOT_DEPLOY_INSTALLER", "/usr/local/bin/vanguard-deploy-bot")
+
+def install_bot_bundle(raw, reason):
+    """Write the uploaded bundle to a temp file and let vanguard-deploy-bot verify, swap and restart it."""
+    import tempfile
+    if not _bot_deploy_lock.acquire(blocking=False):
+        return 409, {"error": "a bot deploy is already running"}
+    tmp = None
+    try:
+        with tempfile.NamedTemporaryFile(prefix="vanguard-bot-bundle.", suffix=".bin", delete=False) as handle:
+            handle.write(raw); tmp = handle.name
+        log("bot_deploy_start", bytes=len(raw), reason=reason)
+        done = subprocess.run([BOT_DEPLOY_INSTALLER, tmp], capture_output=True, text=True, timeout=600)
+        output = (done.stdout + done.stderr).strip().splitlines()
+        log("bot_deploy_exit", level=("info" if done.returncode == 0 else "error"), rc=done.returncode, reason=reason)
+        return (200 if done.returncode == 0 else 500), {"installed": done.returncode == 0, "rc": done.returncode,
+                                                        "output": output[-40:]}
+    except subprocess.TimeoutExpired:
+        log("bot_deploy_timeout", level="error", reason=reason)
+        return 504, {"error": "the installer did not finish within 10 minutes"}
+    except OSError as error:
+        log("bot_deploy_error", level="error", error=error)
+        return 500, {"error": str(error)}
+    finally:
+        if tmp:
+            try: os.unlink(tmp)
+            except OSError: pass
+        _bot_deploy_lock.release()
+
+_deploy_lock = {"running": False}
+def run_deploy(reason):
+    if _deploy_lock["running"]: return False
+    _deploy_lock["running"] = True
+    def go():
+        try:
+            log("deploy_start", reason=reason)
+            r = subprocess.run(["/usr/local/bin/vanguard-deploy", "pull"], capture_output=True, text=True, timeout=1800)
+            for line in (r.stdout + r.stderr).splitlines()[-40:]: print(f"deploy| {line}", flush=True)
+            log("deploy_exit", level=("info" if r.returncode == 0 else "error"), rc=r.returncode)
+        except Exception as e:
+            log("deploy_error", level="error", error=e)
+        finally:
+            _deploy_lock["running"] = False
+    threading.Thread(target=go, daemon=True).start()
+    return True
+
+def log(event, level="info", **kv):
+    """One logfmt line per event (see LOGGING.md): ts= level= component=dumpd event= key=value ..."""
+    def q(v):
+        v = str(v)
+        return v if v and not re.search(r'[\s"=]', v) else '"' + v.replace("\\", "\\\\").replace('"', '\\"') + '"'
+    print(f"ts={datetime.now(TZ).isoformat(timespec='seconds')} level={level} component=dumpd event={event} "
+          + " ".join(f"{k}={q(v)}" for k, v in kv.items() if v is not None), flush=True)
+
 def token_secret():
     try:
         sec = TOKEN_SECRET_FILE.read_bytes().strip()
@@ -217,6 +425,140 @@ def verify_token(token):
         return None
     return user.lower()
 
+# ---- GET /metrics -----------------------------------------------------------------------------------
+# Prometheus text for Metricbeat's prometheus module. Two sources are merged:
+#   1. counters the media scripts keep (bin/vgmetrics.sh writes MARK/metrics/<component>.counters)
+#   2. gauges computed here at scrape time: unpublished recordings, encodes in flight, failures, MediaMTX
+# Only the loopback listener serves it, and Caddy routes no host to /metrics (every dumpd route over there is
+# an explicit path matcher), so this is reachable from the host and from nowhere else. Nothing here is
+# authenticated and nothing here is allowed to fail: a scrape must never be able to disturb the stack.
+METRICS_DIR = Path(os.environ.get("VG_METRICS_DIR", str(MARK / "metrics")))
+MTX_API = os.environ.get("MTX_API_URL", "http://127.0.0.1:9997")
+IDLE_S = int(os.environ.get("MINS_IDLE", "3")) * 60      # the sweep's own "finished being written" rule
+
+# counters key head -> (metric, type, help, the labels the remaining key fields fill, in order)
+COUNTER_MAP = {
+    "jobs":          ("vanguard_media_jobs_total", "counter", "normalise/deband jobs that finished", ("verdict", "outcome", "kind")),
+    "seconds":       ("vanguard_media_job_seconds_total", "counter", "wall-clock seconds spent in ffmpeg", ("verdict", "kind")),
+    "runs":          ("vanguard_media_runs_total", "counter", "times the script has started work", ()),
+    "deband":        ("vanguard_media_deband_total", "counter", "timecode bands cropped off", ()),
+    "last_finish":   ("vanguard_media_last_finish_timestamp_seconds", "gauge", "when the script last finished", ()),
+    "last_duration": ("vanguard_media_last_run_seconds", "gauge", "how long the last finished run took", ()),
+}
+
+def _num(value):
+    """Prometheus wants a plain number: %g would turn a unix timestamp into 1.76e+09."""
+    return "%d" % value if float(value).is_integer() else "%.3f" % value
+
+def _glob(*patterns):
+    found = []
+    for pattern in patterns:
+        try:
+            found.extend(ROOT.glob(pattern))
+        except OSError:
+            pass
+    return found
+
+def mtx_paths():
+    """MediaMTX's own view of what is live. None if it did not answer."""
+    try:
+        with urllib.request.urlopen(f"{MTX_API}/v3/paths/list?itemsPerPage=1000", timeout=1.5) as response:
+            return json.load(response).get("items") or []
+    except Exception:
+        return None
+
+def metrics_text():
+    order, grouped = [], {}
+    def add(metric, typ, helptext, labels, value):
+        if metric not in grouped:
+            grouped[metric] = (typ, helptext, [])
+            order.append(metric)
+        grouped[metric][2].append((labels, value))
+
+    # 1. whatever the scripts have counted
+    for counters in sorted(METRICS_DIR.glob("*.counters")):
+        component = counters.name[: -len(".counters")]
+        try:
+            lines = counters.read_text().splitlines()
+        except OSError:
+            continue
+        for line in lines:
+            field = line.split()
+            if len(field) != 2:
+                continue
+            try:
+                value = float(field[1])
+            except ValueError:
+                continue
+            head, *rest = field[0].split(":")
+            if head not in COUNTER_MAP:
+                continue
+            metric, typ, helptext, names = COUNTER_MAP[head]
+            labels = {"component": component}
+            labels.update(dict(zip(names, rest)))
+            add(metric, typ, helptext, labels, value)
+
+    # 2. is each script executing right now? (its pid file exists only while it runs; a crash leaves a stale
+    #    one, so the pid is checked against /proc rather than trusted)
+    for pid_file in sorted(METRICS_DIR.glob("*.pid")):
+        try:
+            pid = int(pid_file.read_text().strip() or 0)
+        except (OSError, ValueError):
+            pid = 0
+        add("vanguard_media_running", "gauge", "1 while the script is executing",
+            {"component": pid_file.name[: -len(".pid")]}, 1 if pid and Path(f"/proc/{pid}").is_dir() else 0)
+
+    # 3. the queue. Depth-limited globs on purpose: never a walk of the whole media tree.
+    now = _time.time()
+    recording, pending, oldest = 0, 0, 0.0
+    for f in _glob("streams/*/*/.live/*"):
+        if f.name.startswith(".") or not f.is_file():
+            continue
+        try:
+            age = now - f.stat().st_mtime
+        except OSError:
+            continue
+        # the sweep ignores anything touched in the last MINS_IDLE, so younger files are still being written
+        if age <= IDLE_S:
+            recording += 1
+        else:
+            pending += 1
+            oldest = max(oldest, age)
+    add("vanguard_media_recording", "gauge", "recordings MediaMTX is still writing into .live/", {}, recording)
+    add("vanguard_media_pending", "gauge", "finished recordings waiting to be normalised", {"kind": "stream"}, pending)
+    add("vanguard_media_pending_oldest_seconds", "gauge", "age of the oldest waiting recording", {}, round(oldest))
+    add("vanguard_media_encoding_in_flight", "gauge", "ffmpeg output files open right now", {},
+        len(_glob("streams/*/*.part", "streams/*/*/*.part", "streams/*/*/.live/*.part",
+                  "clips/*/*.part", "clips/*/*/*.part", "clips/*/*/*/*.part")))
+    add("vanguard_media_failed_files", "gauge", "files parked with a normalize-failed log", {},
+        len(_glob("streams/*/*.normalize-failed.log", "streams/*/*/*.normalize-failed.log",
+                  "clips/*/*.normalize-failed.log", "clips/*/*/*.normalize-failed.log",
+                  "clips/*/*/*/*.normalize-failed.log")))
+
+    # 4. MediaMTX. The Prometheus scrape of :9998 already has per-path detail; this is the count the portal
+    #    and the dock actually mean by "live", with the derived .webrtc audio mirrors separated out.
+    paths = mtx_paths()
+    add("vanguard_mediamtx_up", "gauge", "did the MediaMTX API answer this scrape", {}, 0 if paths is None else 1)
+    if paths is not None:
+        ready = [p for p in paths if p.get("ready")]
+        add("vanguard_streams_active", "gauge", "streams publishing right now, mirrors excluded", {},
+            sum(1 for p in ready if not str(p.get("name", "")).endswith(".webrtc")))
+        add("vanguard_streams_mirrors", "gauge", "derived .webrtc audio mirrors running", {},
+            sum(1 for p in ready if str(p.get("name", "")).endswith(".webrtc")))
+        add("vanguard_stream_readers", "gauge", "viewers attached across all paths", {},
+            sum(len(p.get("readers") or []) for p in ready))
+
+    out = []
+    for metric in order:
+        typ, helptext, series = grouped[metric]
+        if helptext:
+            out.append(f"# HELP {metric} {helptext}")
+        out.append(f"# TYPE {metric} {typ}")
+        for labels, value in series:
+            tags = ",".join(f'{k}="{v}"' for k, v in sorted(labels.items()))
+            out.append(f"{metric}{{{tags}}} {_num(value)}" if tags else f"{metric} {_num(value)}")
+    return "\n".join(out) + "\n"
+
 class H(BaseHTTPRequestHandler):
     server_version = "dumpd/1"
     def _json(self, code, obj):
@@ -226,6 +568,13 @@ class H(BaseHTTPRequestHandler):
         self.send_header("Content-Length", str(len(body)))
         self.end_headers()
         self.wfile.write(body)
+    def _text(self, code, body):
+        raw = body.encode()
+        self.send_response(code)
+        self.send_header("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
+        self.send_header("Content-Length", str(len(raw)))
+        self.end_headers()
+        self.wfile.write(raw)
     def _deny(self):
         self._json(401, {"error": "not authenticated (no identity header from the proxy)"})
     def _parse(self):
@@ -254,6 +603,12 @@ class H(BaseHTTPRequestHandler):
     def do_GET(self):
         if self.server.server_port == AUTH_LISTEN[1]:
             return self._json(404, {"error": "not found"})          # the auth port serves POST /mtx-auth only
+        if self.path == "/metrics":
+            try:
+                return self._text(200, metrics_text())
+            except Exception as error:                  # a scrape must never take dumpd down
+                log("metrics_failed", level="error", error=str(error))
+                return self._text(500, f"# metrics unavailable: {error}\n")
         if self.path.startswith("/time"):
             # server clock for the timecode strip (the /clock/ page and Go Live measure their offset against this)
             return self._json(200, {"now_ms": int(_time.time() * 1000)})
@@ -262,7 +617,8 @@ class H(BaseHTTPRequestHandler):
             if not who: return self._json(401, {"error": "not authenticated"})
             groups = sorted({g.strip() for g in re.split(r"[|,]", self.headers.get(AUTH_GROUPS_HEADER) or "") if g.strip()})
             return self._json(200, {"user": who, "groups": groups, "admin": is_admin,
-                                    "name": self.headers.get("X-Authentik-Name") or who, "email": self.headers.get("X-Authentik-Email") or ""})
+                                    "name": self.headers.get("X-Authentik-Name") or who, "email": self.headers.get("X-Authentik-Email") or "",
+                                    "discord_id": proxy_claim(self.headers, "discord_id")})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
             if not who: return self._json(401, {"error": "not authenticated"})
@@ -293,6 +649,71 @@ class H(BaseHTTPRequestHandler):
                          "files": [str(f.relative_to(ROOT)) for f in files],
                          "gb": round(sum(f.stat().st_size for f in files) / 1e9, 2)})
     def do_POST(self):
+        if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/heartbeat"):
+            # Alertmanager posts here every minute because the Watchdog alert always fires. Nothing is read from
+            # the body and nothing is authenticated: the port only accepts the Docker bridge, and the only thing
+            # this grants is keeping a timestamp fresh. The deadman treats a STALE file as the alarm.
+            try:
+                self.rfile.read(int(self.headers.get("Content-Length") or 0))
+                HEARTBEAT_FILE.parent.mkdir(parents=True, exist_ok=True)
+                HEARTBEAT_FILE.write_text(f"{int(_time.time())}\n")
+            except OSError as error:
+                return self._json(500, {"error": str(error)})
+            return self._json(200, {"ok": True})
+        if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/deploy-bot"):
+            # The bot's CI posts its release bundle here (bridge-only port; the firewall allows docker0/br-* alone).
+            # The body is the artifact; the signature proves it came from a runner holding BOT_DEPLOY_SECRET.
+            env = deploy_env()
+            if env.get("BOT_DEPLOY_AUTO", "1") != "1":
+                log("bot_deploy_refused", level="warn", reason="BOT_DEPLOY_AUTO=0")
+                return self._json(403, {"error": "automatic bot deploys are turned off (BOT_DEPLOY_AUTO=0)"})
+            secret = env.get("BOT_DEPLOY_SECRET", "")
+            size = int(self.headers.get("Content-Length") or 0)
+            if not secret:
+                return self._json(503, {"error": "BOT_DEPLOY_SECRET is not set on the server"})
+            if not 0 < size <= BOT_DEPLOY_MAX_BYTES:
+                return self._json(413, {"error": f"body must be 1..{BOT_DEPLOY_MAX_BYTES} bytes, got {size}"})
+            raw = self.rfile.read(size)
+            env_b64 = (self.headers.get("X-Vanguard-Env") or "").strip()
+            if len(env_b64) > BOT_ENV_MAX_HEADER:
+                return self._json(413, {"error": "X-Vanguard-Env is too large"})
+            sig = (self.headers.get("X-Vanguard-Signature") or "").replace("sha256=", "").strip()
+            signed = env_b64.encode() + raw          # the settings are signed with the bundle, not beside it
+            if not sig or not hmac.compare_digest(hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest(), sig):
+                log("bot_deploy_bad_signature", level="warn", ip=self.client_address[0], bytes=len(raw))
+                return self._json(401, {"error": "bad signature"})
+            applied, refused = [], []
+            if env_b64:
+                try:
+                    values = json.loads(base64.b64decode(env_b64, validate=True))
+                    if not isinstance(values, dict): raise ValueError("X-Vanguard-Env must be a JSON object")
+                    applied, refused = apply_bot_env(values)
+                except (ValueError, OSError, TypeError) as error:
+                    log("bot_env_failed", level="error", error=error)
+                    return self._json(400, {"error": f"settings rejected: {error}"})
+            code, body = install_bot_bundle(raw, self.headers.get("X-Vanguard-Reason") or "ci")
+            body["settings_applied"] = applied
+            if refused: body["settings_refused"] = refused
+            return self._json(code, body)
+        if self.server.server_port == AUTH_LISTEN[1] and self.path.startswith("/deploy"):
+            # Gitea (container) -> host gateway -> here. Verify HMAC-SHA256 of the raw body with the webhook secret.
+            n = int(self.headers.get("Content-Length") or 0); raw = self.rfile.read(n)
+            env = deploy_env(); secret = env.get("DEPLOY_WEBHOOK_SECRET", "")
+            sig = self.headers.get("X-Gitea-Signature") or self.headers.get("X-Hub-Signature-256", "").replace("sha256=", "")
+            if not secret or not sig or not hmac.compare_digest(hmac.new(secret.encode(), raw, hashlib.sha256).hexdigest(), sig):
+                log("deploy_webhook_bad_signature", level="warn", ip=self.client_address[0])
+                return self._json(401, {"error": "bad signature"})
+            try: payload = json.loads(raw or b"{}")
+            except json.JSONDecodeError: return self._json(400, {"error": "bad json"})
+            ref = payload.get("ref", ""); branch = env.get("DEPLOY_BRANCH", "main")
+            after = (payload.get("after") or "")[:10]; pusher = (payload.get("pusher") or {}).get("username", "?")
+            if ref != f"refs/heads/{branch}":
+                return self._json(200, {"ignored": f"ref {ref} is not {branch}"})
+            if env.get("DEPLOY_AUTO", "0") != "1":
+                log("deploy_webhook_push", rev=after, pusher=pusher, branch=branch, auto="0")
+                return self._json(202, {"queued": False, "note": "auto-deploy disabled (DEPLOY_AUTO=0); run vanguard-deploy pull by hand"})
+            started = run_deploy(f"push {after} by {pusher}")
+            return self._json(202, {"queued": started, "note": "deploy started" if started else "a deploy is already running"})
         if self.server.server_port == AUTH_LISTEN[1] and not self.path.startswith("/mtx-auth"):
             return self._json(404, {"error": "not found"})
         if self.path.startswith("/mtx-auth"):
@@ -307,18 +728,24 @@ class H(BaseHTTPRequestHandler):
                 return self._json(200, {"ok": True})            # viewers are gated by Caddy + Authentik
             if action != "publish":
                 return self._json(401, {"error": f"action {action} not allowed"})
+            path = (req.get("path") or "").lower()
+            if path.endswith(".webrtc"):
+                # audio mirrors (<user>/<Title>.webrtc) are published by the hook inside the mediamtx container only
+                if req.get("ip") in ("127.0.0.1", "::1") and re.fullmatch(r"[a-z0-9_.-]+/[a-z0-9_.-]+\.webrtc", path):
+                    return self._json(200, {"ok": True})
+                log("publish_deny", level="warn", stream=path, ip=req.get("ip"), reason="mirror paths are internal")
+                return self._json(401, {"error": ".webrtc paths are reserved"})
             cred = req.get("password") or req.get("token") or ""
             if cred.lower().startswith("bearer "): cred = cred[7:]
             if ":" in cred and "." in cred.split(":", 1)[1]: cred = cred.split(":", 1)[1]   # "user:token" form
             user = verify_token(cred)
-            path = (req.get("path") or "").lower()
             if not user:
-                print(f"{datetime.now(TZ):%F %T} mtx-auth DENY publish path={path} ip={req.get('ip')} (bad/expired token)", flush=True)
+                log("publish_deny", level="warn", stream=path, ip=req.get("ip"), reason="bad or expired token")
                 return self._json(401, {"error": "invalid or expired stream token"})
             if not re.fullmatch(r"[a-z0-9_.-]+/[a-z0-9_.-]+", path) or not path.startswith(user + "/"):
-                print(f"{datetime.now(TZ):%F %T} mtx-auth DENY publish user={user} path={path} (not their prefix)", flush=True)
+                log("publish_deny", level="warn", user=user, stream=path, reason="not their prefix")
                 return self._json(401, {"error": "path must be <your-username>/<Title>"})
-            print(f"{datetime.now(TZ):%F %T} mtx-auth allow publish user={user} path={path} ip={req.get('ip')}", flush=True)
+            log("publish_allow", user=user, stream=path, ip=req.get("ip"))
             return self._json(200, {"ok": True})
         if self.path.startswith("/me/clips"):
             who, _ = identity(self.headers)
@@ -345,7 +772,7 @@ class H(BaseHTTPRequestHandler):
         moved, size = do_move(files, night)
         MARK.mkdir(exist_ok=True)
         (MARK / f"{night}-{tgt}.dumped").write_text(f"{datetime.now(TZ).isoformat()} by {u}\n")
-        print(f"{datetime.now(TZ):%F %T} dump night={night} user={tgt} by={u} files={moved} gb={size/1e9:.2f}", flush=True)
+        log("dump", night=night, user=tgt, by=u, files=moved, gb=f"{size/1e9:.2f}")
         self._json(200, {"night": night, "user": tgt, "moved": moved, "gb": round(size / 1e9, 2),
                          "undo_until": (datetime.now(TZ) + timedelta(days=3)).strftime("%F")})
     def log_message(self, *a):  # quiet
@@ -353,8 +780,8 @@ class H(BaseHTTPRequestHandler):
 
 if __name__ == "__main__":
     TRASH.mkdir(exist_ok=True); MARK.mkdir(exist_ok=True)
-    import threading
     print(f"dumpd listening on {LISTEN[0]}:{LISTEN[1]} (dump, me) and {AUTH_LISTEN[0]}:{AUTH_LISTEN[1]} (mtx-auth)  tz={TZ.key}  night={NIGHT_START:%H:%M}-{NIGHT_END:%H:%M}  token-secret={'ok' if token_secret() else 'MISSING'}", flush=True)
     auth_srv = ThreadingHTTPServer(AUTH_LISTEN, H)
     threading.Thread(target=auth_srv.serve_forever, daemon=True).start()
+    threading.Thread(target=quota_loop, daemon=True).start()        # per-user clip quotas (pause/resume Syncthing folders)
     ThreadingHTTPServer(LISTEN, H).serve_forever()

bin/mtx-on-ready.sh

@@ -0,0 +1,36 @@
+#!/bin/sh
+# MediaMTX runOnReady hook (runs INSIDE the mediamtx container; busybox sh + wget + ffmpeg from the -ffmpeg image).
+# Env from MediaMTX: MTX_PATH (user/Title). From compose: LIVE_WEBHOOK_URL, LIVE_HOST.
+# 1. optional Discord "is live" notice
+# 2. WebRTC audio mirror: browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only), and
+#    OBS sends AAC. For every stream whose audio is not already Opus, copy the video and transcode the audio to Opus
+#    into the sibling path "<user>/<Title>.webrtc". The live portal plays that path over WebRTC; recordings, HLS and
+#    the Discord notice stay on the original. Mirrors are published from 127.0.0.1 inside this container only, which
+#    is the only publisher dumpd accepts for ".webrtc" paths. Audio-only transcode: a few % of one core per stream.
+# MediaMTX kills this process when the source path goes away.
+case "$MTX_PATH" in *.webrtc) exit 0 ;; esac
+u="${MTX_PATH%%/*}"; t="${MTX_PATH#*/}"
+if [ -n "$LIVE_WEBHOOK_URL" ]; then
+  wget -q -O /dev/null --header="Content-Type: application/json" \
+    --post-data="{\"content\":\"🔴 **$u** is live: $t — https://$LIVE_HOST/\"}" "$LIVE_WEBHOOK_URL" 2>/dev/null &
+fi
+# which audio does the source carry? (API on loopback; "Opus" needs no mirror, no audio needs no mirror)
+tracks=""
+for i in 1 2 3 4 5; do
+  tracks=$(wget -q -O - "http://127.0.0.1:9997/v3/paths/get/$MTX_PATH" 2>/dev/null) && [ -n "$tracks" ] && break
+  sleep 1
+done
+case "$tracks" in
+  *Opus*) exit 0 ;;
+  *"MPEG-4 Audio"*|*"MPEG-1 Audio"*|*G711*|*G722*|*LPCM*|*AC-3*) ;;
+  *) exit 0 ;;                                   # no (recognised) audio: nothing to mirror
+esac
+echo "event=mirror stream=$MTX_PATH target=$MTX_PATH.webrtc audio=opus"
+n=0
+while [ $n -lt 10 ]; do                          # retry: the source may not be readable for the first second
+  ffmpeg -nostdin -hide_banner -loglevel warning -fflags nobuffer -flags low_delay \
+    -rtsp_transport tcp -i "rtsp://127.0.0.1:8554/$MTX_PATH" \
+    -map 0:v:0 -map 0:a:0 -c:v copy -c:a libopus -b:a 128k -ar 48000 -ac 2 -application lowdelay -frame_duration 20 \
+    -f rtsp -rtsp_transport tcp "rtsp://127.0.0.1:8554/$MTX_PATH.webrtc"
+  n=$((n+1)); sleep 2
+done

bin/normalize-sweep.sh

@@ -6,7 +6,10 @@
 # - anything else (HEVC, 10-bit, VFR, MKV, >1440p, >60fps): full-res re-encode -> name.mp4,
 #   raw source kept in a hidden .src/ subfolder next to it (Jellyfin/copyparty hide dot-dirs)
 # - streams/<user>/<Title>/.live/<ts>.mp4 (MediaMTX writes here, hidden from Jellyfin/copyparty while live):
-#   when finished -> flat MP4 published as streams/<user>/<Title>__<ts>.mp4 (one file per recording directly
+#   when finished -> the timecode overscan band (if the streamer had it on) is decoded and then CROPPED OFF in a
+#   re-encode (libx264, frame timing preserved, audio copied) so the published file is the game image + audio +
+#   a tmcd timecode track, nothing else; streams without the band are remuxed losslessly. Published as flat
+#   MP4 streams/<user>/<Title>__<ts>.mp4 (one file per recording directly
 #   in the streamer's folder: Jellyfin would merge several files inside one title folder into ONE episode
 #   with "versions"), the .live source deleted, the empty title folder removed.
 # - Syncthing folders (marker dir .stfolder in the folder root, receive-only): compliant files are left
@@ -23,11 +26,28 @@ MINS_IDLE=3
 # the total missing time is >= this many ms. 0 = never (default; a 3 h re-encode is ~6 h of CPU on the E-2136).
 # Set via systemd Environment=FILL_GAPS_MS=... ; viewers already see held frames either way.
 FILL_GAPS_MS="${FILL_GAPS_MS:-0}"
+# Band removal re-encodes every stream that carried the timecode band. The raw (with band) is deleted after a
+# successful encode; KEEP_RAW_STREAMS=1 keeps it in the hidden .src/ instead (doubles disk per night).
+KEEP_RAW_STREAMS="${KEEP_RAW_STREAMS:-0}"
+# Encoder threads for every re-encode (clips and streams). 2 leaves the box responsive for Jellyfin/live viewers;
+# conversions take longer. systemd Environment=ENCODE_THREADS=... to change.
+ENCODE_THREADS="${ENCODE_THREADS:-2}"
 MAX_W=2560; MAX_H=1440; MAX_FPS=60
 exec 9>"${LOCK_FILE:-/run/lock/normalize-sweep.lock}"
 flock -n 9 || exit 0
 
-log() { echo "$(date '+%F %T') $*"; }
+# Metrics (LOGGING.md): counters for dumpd's /metrics. Optional on purpose — if the helper is not installed
+# yet, these become no-ops and normalising carries on exactly as before.
+if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then
+  VG_MET_COMPONENT=sweep
+  . "$(dirname "$0")/vgmetrics.sh" && vgmet_init
+else
+  vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; }
+fi
+
+# logfmt (LOGGING.md): vglog <level> event=<verb> key=value ...   values with spaces must arrive pre-quoted
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=sweep %s\n' "$(date -Is)" "$lvl" "$*"; }
+log() { vglog info "msg=\"$*\""; }
 dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
 
 # --- probe: prints "verdict reason" where verdict ∈ ok|remux|encode|skip ----------------------
@@ -73,15 +93,13 @@ probe() {
   echo "ok"
 }
 
-find "$ROOT/streams" "$ROOT/clips" -type f \
-     \( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \
-     ! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null |
 while IFS= read -r -d '' src; do
   base="${src%.*}"; ext="${src##*.}"
-  live=0
+  live=0; stream=""
   if [[ $(basename "$(dirname "$src")") == .live ]]; then
     live=1
     titledir="$(dirname "$(dirname "$src")")"                       # streams/<user>/<Title>
+    stream="$(basename "$(dirname "$titledir")")/$(basename "$titledir")"   # user/Title, the log correlation key
     base="$(dirname "$titledir")/$(basename "$titledir")__$(basename "${src%.*}")"   # streams/<user>/<Title>__<ts>
   fi
   synced=0; stroot=""
@@ -98,7 +116,7 @@ while IFS= read -r -d '' src; do
   if (( live )); then
     [[ $verdict == skip ]] && continue
     [[ $verdict == ok ]] && verdict=remux          # must leave .live/ either way
-    out="$base.mp4"; [[ -e "$out" ]] && { log "exists  $out (leaving $src)"; continue; }
+    out="$base.mp4"; [[ -e "$out" ]] && { vglog info "event=skip reason=exists file=\"$out\" src=\"$src\""; continue; }
   else
     case "$verdict" in ok|skip) continue ;; esac
     out="$base.mp4"
@@ -109,8 +127,9 @@ while IFS= read -r -d '' src; do
     [[ -e "$out" ]] && continue
   fi
   tmp="$out.part"
-  log "$verdict  $src  ($reason)"
+  kind=clip; (( live )) && kind=stream           # metric label: stream encodes are the expensive ones
-  tcargs=(); tcjson=""
+  vglog info "event=verdict verdict=$verdict file=\"$src\" reason=\"$reason\" stream=\"$stream\""
+  tcargs=(); tcjson=""; band=0; deband=0
   if (( live )); then
     # timecode: decode the overscan strip (server clock) or fall back to the server arrival time in the filename
     stem="$(basename "${src%.*}")"
@@ -130,47 +149,63 @@ PYEOF
     if [[ -n ${TC_TIMECODE:-} ]]; then
       tcargs=(-timecode "$TC_TIMECODE" -metadata "creation_time=$TC_CREATION" -write_tmcd 1)
       tcjson="${TC_JSON:-}"
-      log "tc     $src  source=$TC_SOURCE timecode=$TC_TIMECODE"
+      vglog info "event=timecode file=\"$src\" stream=\"$stream\" source=$TC_SOURCE timecode=\"$TC_TIMECODE\""
+    fi
+    # the band has done its job once decoded: remove it from the picture (re-encode with a crop)
+    if [[ ${TC_SOURCE:-} == strip && ${TC_BAND:-0} -gt 0 ]]; then
+      band=$TC_BAND; deband=1; verdict=encode; reason="remove ${band}px timecode band"
+      vglog info "event=deband file=\"$src\" stream=\"$stream\" band_rows=$band"
     fi
     # opt-in gap filling: switch this stream from remux to a CFR re-encode when the holes add up to enough
     if (( FILL_GAPS_MS > 0 )) && [[ $verdict == remux && -n $tcjson ]]; then
       missing=$(python3 -c 'import json,sys; d=json.loads(sys.argv[1]); print(d.get("gaps",{}).get("total_missing_ms",0))' "$tcjson" 2>/dev/null || echo 0)
       if (( missing >= FILL_GAPS_MS )); then
-        verdict=encode; reason="fill-gaps ${missing}ms"; log "gaps   $src  ${missing} ms missing -> CFR re-encode (FILL_GAPS_MS=$FILL_GAPS_MS)"
+        verdict=encode; reason="fill-gaps ${missing}ms"; vglog info "event=fill_gaps file=\"$src\" stream=\"$stream\" missing_ms=$missing"
       fi
     fi
   fi
+  job_t0=$(vgmet_time)
+  err=$(mktemp /tmp/vgsweep-err.XXXXXX)          # ffmpeg's own words; rc= alone is not diagnosable
   if [[ $verdict == remux ]]; then
     nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
-      -map 0 -c copy "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
+      -map 0 -c copy "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err"
   else
     # scale down into the cap (never up), keep aspect, even dimensions; cap fps; CFR; all audio -> AAC
     vf="scale='min(iw,$MAX_W)':'min(ih,$MAX_H)':force_original_aspect_ratio=decrease:force_divisible_by=2"
+    (( deband )) && vf="crop=iw:ih-${band}:0:0,$vf"          # drop the band rows first (band is even, so sizes stay even)
     fpsargs=(); fpsnow=$(awk -F/ '{ if ($2>0) printf "%.3f", $1/$2; else print 0 }' <<<"$(ffprobe -v error -select_streams v:0 -show_entries stream=avg_frame_rate -of csv=p=0 "$src")")
     awk "BEGIN{exit !($fpsnow > $MAX_FPS + 0.01)}" && fpsargs=(-r "$MAX_FPS")
+    # streams keep their exact frame timing (gaps stay gaps unless FILL_GAPS_MS asked for CFR) and their AAC audio as-is
+    fpsmode=cfr; aargs=(-c:a aac -b:a 160k)
+    if (( live )) && [[ $reason != fill-gaps* ]]; then fpsmode=passthrough; fi
+    if (( live )) && [[ $reason != *audio=* ]]; then aargs=(-c:a copy); fi
     nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$src" \
       -map 0:v:0 -map '0:a?' \
-      -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high \
+      -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "$ENCODE_THREADS" \
-      -vf "$vf" -fps_mode cfr "${fpsargs[@]}" \
+      -vf "$vf" -fps_mode "$fpsmode" "${fpsargs[@]}" \
-      -c:a aac -b:a 160k \
+      "${aargs[@]}" \
-      "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
+      "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp" 2>"$err"
   fi
   rc=$?
+  job_secs=$(vgmet_since "$job_t0")
   d1=$(dur "$src"); d2=$(dur "$tmp")
   if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
+    vgmet_add "jobs:$verdict:done:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs"
+    (( deband )) && vgmet_add deband 1
     if (( live )); then
       mv -f "$tmp" "$out"; chown 1000:1000 "$out" 2>/dev/null
       if [[ -n $tcjson ]]; then                        # sidecar: exact capture time (read by the Jellyfin plugin) + drift samples
         printf '%s\n' "$tcjson" > "${out%.*}.json"; chown 1000:1000 "${out%.*}.json" 2>/dev/null
       fi
-      if [[ $verdict == encode ]]; then                # gap-filled re-encode: keep the raw recording in the hidden .src/
+      if [[ $verdict == encode ]] && (( ! deband || KEEP_RAW_STREAMS )); then   # gap-filled re-encode (or asked to): keep the raw in the hidden .src/
         srcdir="$(dirname "$out")/.src"; mkdir -p "$srcdir"; chown 1000:1000 "$srcdir" 2>/dev/null
         mv -f "$src" "$srcdir/$(basename "$out" .mp4).raw.mp4"
       else
-        rm -f "$src"
+        rm -f "$src"                                   # band removal: the cropped file IS the recording; raw-with-band is not kept
       fi
       rmdir "$(dirname "$src")" "$titledir" 2>/dev/null || true   # drop .live and the empty title folder
-      log "done   $out"; continue
+      rm -f "$err"
+      vglog info "event=published file=\"$out\" stream=\"$stream\" verdict=$verdict"; continue
     elif (( synced )); then
       rel_src="${src#"$stroot"/}"; rel_out="${out#"$stroot"/}"
       { echo "/$rel_out"; echo "/$rel_src"; } >> "$stroot/.stlocal"   # ignore our output and the raw
@@ -186,10 +221,16 @@ PYEOF
       mv -f "$tmp" "$out"
     fi
     chown 1000:1000 "$out" 2>/dev/null
-    log "done   $out"
+    vglog info "event=done file=\"$out\" verdict=$verdict"
   else
     rm -f "$tmp"
-    { echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)"; } > "$base.normalize-failed.log"
+    vgmet_add "jobs:$verdict:failed:$kind"; vgmet_add "seconds:$verdict:$kind" "$job_secs"
-    log "FAILED $src (see $base.normalize-failed.log)" >&2
+    { echo "$(date '+%F %T') $verdict failed rc=$rc src_dur=${d1:-?} out_dur=${d2:-?} ($reason)"
+      echo "--- ffmpeg stderr ---"; tail -40 "$err" 2>/dev/null; } > "$base.normalize-failed.log"
+    why=$(tail -1 "$err" 2>/dev/null | tr -d '"' | cut -c1-200)
+    vglog error "event=failed file=\"$src\" stream=\"$stream\" verdict=$verdict rc=$rc reason=\"${why:-no stderr}\" log=\"$base.normalize-failed.log\"" >&2
   fi
-done
+  rm -f "$err"
+done < <(find "$ROOT/streams" "$ROOT/clips" -type f \
+     \( -iname '*.mp4' -o -iname '*.mkv' -o -iname '*.mov' -o -iname '*.ts' -o -iname '*.m4v' -o -iname '*.webm' \) \
+     ! -path '*/.src/*' ! -path '*/.stfolder/*' ! -path '*/.stversions/*' ! -name '*.part' -mmin +"$MINS_IDLE" -print0 2>/dev/null)

bin/vanguard-alerts.sh

@@ -0,0 +1,152 @@
+#!/bin/bash
+# Deliver Kibana's alerts to Discord. Runs from a systemd timer every minute; curl and python only.
+#
+# WHY THIS EXISTS
+# On a basic license Kibana's only connector types are .index and .server-log — every connector that leaves
+# the stack is Gold or above. So Kibana rules write their alerts into ONE Elasticsearch index (the alert bus,
+# created by observability/kibana-setup.py) and this script is the delivery side. Alerts are therefore
+# ordinary documents: queryable, replayable, and visible in Discover alongside the logs that caused them.
+#
+# Heartbeat documents (kind=heartbeat) are deliberately NOT delivered. They exist for bin/vanguard-deadman.sh
+# to notice their absence, which is the only way to catch the alerting pipeline itself dying.
+#
+# Repeat suppression is a safety net, not the primary throttle: a rule should use "On status change" or its
+# own throttle in Kibana. This stops a misconfigured rule turning into a message every minute, and it is per
+# rule name, so one noisy rule never silences another.
+#
+# Exit status: 0 when there was nothing to send or everything was sent; 1 when alerts existed and could not be
+# delivered, so `systemctl status vanguard-alerts` and the journal both carry the failure.
+set -uo pipefail
+
+ES="${ES_URL:-http://127.0.0.1:9200}"
+ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
+INDEX="${ALERT_INDEX:-vanguard-alerts}"
+STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}"
+WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
+SUPPRESS_S="${ALERT_SUPPRESS_S:-900}"
+BATCH="${ALERT_BATCH:-50}"
+
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=alerts %s\n' "$(date -Is)" "$lvl" "$*"; }
+es_curl() { if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi; }
+
+state_get() { [[ -r $STATE ]] || return 0; grep -m1 "^$1=" "$STATE" 2>/dev/null | cut -d= -f2-; }
+state_set() {
+  local key="$1" value="$2" tmp
+  install -d "$(dirname "$STATE")" 2>/dev/null || return 0
+  tmp=$(mktemp "${STATE}.XXXXXX" 2>/dev/null) || return 0
+  { [[ -r $STATE ]] && grep -v "^${key}=" "$STATE"; printf '%s=%s\n' "$key" "$value"; } > "$tmp" 2>/dev/null
+  mv -f "$tmp" "$STATE" 2>/dev/null || rm -f "$tmp"
+}
+
+webhook=""
+[[ -r $WEBHOOK_FILE ]] && webhook=$(tr -d '[:space:]' < "$WEBHOOK_FILE")
+# Same guard as the deadman: a typo that is not an https URL must not be posted to, and must not look like
+# delivery succeeded. Treated as "no webhook", which means journal-only and a non-zero exit.
+case "${webhook:-}" in
+  "")       ;;
+  https://*) ;;
+  *)        vglog error 'event=bad_webhook msg="ALERT_WEBHOOK_URL is not an https URL; ignoring it"' >&2; webhook="" ;;
+esac
+
+now=$(date +%s)
+cursor=$(state_get cursor); cursor="${cursor:-0}"
+
+# Everything newer than the cursor that is not a heartbeat, oldest first so the cursor advances safely.
+query=$(printf '{"size":%d,"sort":[{"@timestamp":"asc"}],"query":{"bool":{"filter":[{"range":{"@timestamp":{"gt":%s,"format":"epoch_millis"}}}],"must_not":[{"term":{"kind":"heartbeat"}}]}}}' "$BATCH" "$cursor")
+response=$(es_curl -m 15 -H 'Content-Type: application/json' "$ES/$INDEX/_search" -d "$query")
+
+if [[ -z $response ]]; then
+  vglog error 'event=unreachable msg="Elasticsearch did not answer; cannot read the alert bus"' >&2
+  exit 1
+fi
+# A missing index just means no rule has fired yet. Not a fault, and not worth a line every minute.
+if grep -q 'index_not_found_exception' <<<"$response"; then
+  exit 0
+fi
+
+# Retention. A heartbeat document arrives every minute, so without this the alert bus grows without bound.
+# At most once every 6 h: a delete_by_query is cheap, but not worth doing sixty times an hour. Deliberately
+# before the "nothing new" exit below, because most runs have nothing to deliver and would skip it.
+retention_days="${ALERT_RETENTION_DAYS:-90}"
+last_purge=$(state_get purged); last_purge="${last_purge:-0}"
+if (( now - last_purge > 21600 )); then
+  purge=$(es_curl -m 120 -H 'Content-Type: application/json' \
+            -XPOST "$ES/$INDEX/_delete_by_query?conflicts=proceed" \
+            -d "{\"query\":{\"range\":{\"@timestamp\":{\"lt\":\"now-${retention_days}d\"}}}}")
+  gone=$(printf '%s' "${purge:-}" | python3 -c 'import json,sys
+try: print(json.load(sys.stdin).get("deleted", 0))
+except Exception: print("?")')
+  state_set purged "$now"
+  vglog info "event=purged older_than=${retention_days}d deleted=${gone:-?}"
+fi
+
+rows=$(printf '%s' "$response" | python3 -c '
+import json, sys
+from datetime import datetime
+try:
+    data = json.load(sys.stdin)
+except Exception:
+    sys.exit(0)
+for hit in data.get("hits", {}).get("hits", []):
+    source = hit.get("_source", {})
+    stamp = str(source.get("@timestamp", ""))
+    try:
+        when = datetime.fromisoformat(stamp.replace("Z", "+00:00"))
+        millis = int(when.timestamp() * 1000)
+    except Exception:
+        continue
+    def clean(value, limit):
+        return " ".join(str(value).split())[:limit] or "-"
+    print("\t".join((str(millis),
+                     clean(source.get("rule", "unnamed rule"), 120),
+                     clean(source.get("level", "warn"), 12),
+                     clean(source.get("message", json.dumps(source)[:400]), 1200))))
+')
+
+[[ -z $rows ]] && exit 0
+
+# One per line: "a=0 b=0 now" would make bash run `now` as a COMMAND with a and b as temporary
+# assignments, leaving every counter unset for the rest of the script.
+sent=0
+suppressed=0
+failed=0
+while IFS=$'\t' read -r millis rule level message; do
+  [[ -z ${millis:-} ]] && continue
+  last=$(state_get "sent:$rule"); last="${last:-0}"
+  if (( now - last < SUPPRESS_S )); then
+    suppressed=$((suppressed + 1))
+    vglog info "event=suppressed rule=\"$rule\" within=${SUPPRESS_S}s"
+    state_set cursor "$millis"
+    continue
+  fi
+  case "$level" in
+    error|critical) mark="[ERROR]" ;;
+    warn|warning)   mark="[WARN]" ;;
+    *)              mark="[$level]" ;;
+  esac
+  text="$mark $rule — $message"
+  if [[ -z $webhook ]]; then
+    vglog warn "event=undeliverable rule=\"$rule\" msg=\"no Discord webhook configured; alert only in the journal\"" >&2
+    vglog warn "event=alert rule=\"$rule\" level=$level msg=\"$message\"" >&2
+    failed=$((failed + 1))
+    state_set cursor "$millis"
+    continue
+  fi
+  payload=$(VG_TEXT="$text" python3 -c 'import json, os; print(json.dumps({"content": os.environ["VG_TEXT"][:1900]}))')
+  # curl already prints 000 on a connection failure; "|| echo 000" would concatenate and give 000000.
+  code=$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST -H 'Content-Type: application/json' \
+           -d "$payload" "$webhook" || true)
+  code="${code:-000}"
+  if [[ $code == 2* ]]; then
+    sent=$((sent + 1)); state_set "sent:$rule" "$now"
+    vglog info "event=delivered rule=\"$rule\" level=$level"
+  else
+    failed=$((failed + 1))
+    vglog error "event=delivery_failed rule=\"$rule\" http=$code" >&2
+  fi
+  state_set cursor "$millis"
+done <<<"$rows"
+
+vglog info "event=run sent=$sent suppressed=$suppressed failed=$failed"
+(( failed > 0 )) && exit 1
+exit 0

bin/vanguard-deadman.sh

@@ -0,0 +1,162 @@
+#!/bin/bash
+# The deadman: the one check that does not depend on the Elastic stack being alive.
+#
+# Kibana tells you when something on the box is wrong. Nothing inside Elasticsearch, Kibana or the Beats can
+# tell you that they are the thing that broke, and silence looks exactly like health. So this runs on the HOST
+# from a systemd timer, uses nothing but curl, and posts straight to a Discord webhook.
+#
+# It checks six things, in the order they go wrong in practice:
+#   1. Elasticsearch answers and the cluster is not red
+#   2. no index has been flipped to read-only by the disk watermark — the classic Elastic failure, because it
+#      does NOT clear itself when space is freed, so ingestion stays silently dead until someone resets it
+#   3. Kibana reports itself available (it is the only UI; if it is down you are blind)
+#   4. Filebeat has actually written a document recently — "running" is not the same as "shipping"
+#   5. Metricbeat likewise
+#   6. the heartbeat a Kibana rule posts to dumpd is fresh, which proves rule evaluation AND delivery work
+#   …and the disks, because that is the failure this box is most likely to die of, and it must be reported
+#   even when the thing that would normally report it is what broke.
+#
+# State in /var/lib/vanguard/deadman.state: it announces a change, not the same thing every five minutes,
+# and says so once when everything recovers.
+#
+# No webhook configured? The checks still run and still print to the journal, and the unit exits non-zero
+# when it found something it could not report — so `systemctl is-failed vanguard-deadman.service` and
+# `journalctl -u vanguard-deadman` remain a signal. It never fails because the webhook is absent alone.
+set -uo pipefail
+
+ES="${ES_URL:-http://127.0.0.1:9200}"
+# Elasticsearch requires credentials (security is on; observability/kibana.yml says why). configure.py writes
+# this netrc, root-only: netrc rather than -u so the password is never visible in a process list. If the file
+# is missing, the calls below get 401 and the deadman reports Elasticsearch as unreachable — which is correct,
+# because from its point of view it is. Kibana needs nothing: its anonymous provider signs callers in.
+ES_NETRC="${ES_NETRC_FILE:-/etc/vanguard/es-netrc}"
+es_curl() {
+  if [[ -r $ES_NETRC ]]; then curl -s --netrc-file "$ES_NETRC" "$@"; else curl -s "$@"; fi
+}
+KIBANA="${KIBANA_URL:-http://127.0.0.1:5601}"
+WEBHOOK_FILE="${ALERT_WEBHOOK_FILE:-/etc/vanguard/alert-webhook}"
+ALERT_INDEX="${ALERT_INDEX:-vanguard-alerts}"            # the alert bus Kibana's rules write into
+ALERT_STATE="${ALERT_STATE:-/var/lib/vanguard/alerts.state}"  # touched by every vanguard-alerts.sh run
+ALERT_STATE_MAX_S="${ALERT_STATE_MAX_S:-900}"
+HEARTBEAT_MAX_S="${HEARTBEAT_MAX_S:-900}"
+STALE_MAX_S="${BEAT_STALE_MAX_S:-900}"
+STATE="${DEADMAN_STATE:-/var/lib/vanguard/deadman.state}"
+DISK_PCT="${DEADMAN_DISK_PCT:-90}"
+
+problems=()
+note() { problems+=("$1"); }
+jq_get() { python3 -c 'import json,sys
+try: d=json.load(sys.stdin)
+except Exception: sys.exit(1)
+for k in sys.argv[1].split("."):
+    d = d.get(k) if isinstance(d, dict) else None
+    if d is None: sys.exit(1)
+print(d)' "$1" 2>/dev/null; }
+
+# 1. Elasticsearch
+health=$(es_curl -m 8 "$ES/_cluster/health" || true)
+if [[ -z $health ]]; then
+  note "Elasticsearch is not answering on ${ES#http://} — nothing is being stored"
+else
+  status=$(printf '%s' "$health" | jq_get status)
+  case "$status" in
+    green|yellow) ;;
+    red) note "Elasticsearch cluster health is RED: some index has no usable shard" ;;
+    *)   note "Elasticsearch health is unreadable (got: ${status:-nothing})" ;;
+  esac
+
+  # 2. read-only indices. Elastic sets this when the disk passes the flood watermark and never unsets it.
+  blocked=$(es_curl -m 8 "$ES/_all/_settings/index.blocks.read_only_allow_delete" \
+            | python3 -c 'import json,sys
+try: d=json.load(sys.stdin)
+except Exception: sys.exit(0)
+bad=[n for n,v in d.items() if str(v.get("settings",{}).get("index",{}).get("blocks",{}).get("read_only_allow_delete","")).lower()=="true"]
+print(len(bad))' 2>/dev/null || echo 0)
+  (( ${blocked:-0} > 0 )) && note "$blocked Elasticsearch index/indices are READ-ONLY from the disk watermark; ingestion is dead until this is reset (see README 5h)"
+
+  # 4 and 5. are the Beats actually writing?
+  for beat in filebeat metricbeat; do
+    count=$(es_curl -m 8 -H 'Content-Type: application/json' \
+              "$ES/${beat}-*/_count" \
+              -d "{\"query\":{\"range\":{\"@timestamp\":{\"gte\":\"now-${STALE_MAX_S}s\"}}}}" | jq_get count)
+    if [[ -z ${count:-} ]]; then
+      note "cannot tell whether $beat is shipping (no index yet, or the query failed)"
+    elif (( count == 0 )); then
+      note "$beat has written nothing in the last $((STALE_MAX_S/60)) minutes — it may be up but not shipping"
+    fi
+  done
+fi
+
+# 3. Kibana
+level=$(curl -s -m 8 "$KIBANA/api/status" | jq_get status.overall.level)
+[[ ${level:-} == available ]] || note "Kibana is not available (${level:-no answer}) — the only UI is down"
+
+# 6. the heartbeat. A Kibana rule writes a kind=heartbeat document into the alert index every minute (see
+# observability/kibana-setup.py). Its absence is the ONLY thing that catches the alerting pipeline itself
+# dying: if Kibana, its rule engine or the index connector stops, no ordinary alert can report it, because
+# the thing that would report it is what broke. Checked in Elasticsearch, not as a file: a basic licence has
+# no webhook connector, so nothing posts to dumpd any more.
+if [[ -n $health ]]; then
+  beats=$(es_curl -m 8 -H 'Content-Type: application/json' "$ES/$ALERT_INDEX/_count" \
+            -d "{\"query\":{\"bool\":{\"filter\":[{\"term\":{\"kind\":\"heartbeat\"}},{\"range\":{\"@timestamp\":{\"gte\":\"now-${HEARTBEAT_MAX_S}s\"}}}]}}}")
+  if grep -q 'index_not_found_exception' <<<"${beats:-}"; then
+    note "the alert index $ALERT_INDEX does not exist — the Kibana connector and heartbeat rule have not been created (observability/kibana-setup.py)"
+  else
+    hb=$(printf '%s' "${beats:-}" | jq_get count)
+    if [[ -z ${hb:-} ]]; then
+      note "cannot read the alert heartbeat (query against $ALERT_INDEX failed)"
+    elif (( hb == 0 )); then
+      note "no alert heartbeat in the last $((HEARTBEAT_MAX_S/60)) minutes — Kibana's rule engine or the index connector has stopped, so NO alert can reach Discord"
+    fi
+  fi
+fi
+
+# 7. is the delivery side running? The heartbeat proves Kibana still writes alerts; this proves something
+# still takes them to Discord. Both halves have to be alive for an alarm to arrive.
+if [[ -f $ALERT_STATE ]]; then
+  age=$(( $(date +%s) - $(stat -c %Y "$ALERT_STATE") ))
+  (( age <= ALERT_STATE_MAX_S )) || note "vanguard-alerts.timer has not run for ${age}s — alerts are being written but not delivered"
+fi
+
+# the disks, independently of any agent
+while read -r pct mount; do
+  (( pct >= DISK_PCT )) && note "$mount is ${pct}% full"
+done < <(df --output=pcent,target /srv / 2>/dev/null | tail -n +2 | tr -d '%' | awk '{print $1, $2}')
+
+# ---- report only on change ---------------------------------------------------------------------------
+now="$(IFS='|'; echo "${problems[*]-}")"
+mkdir -p "$(dirname "$STATE")"
+was="$(cat "$STATE" 2>/dev/null || true)"
+printf '%s' "$now" > "$STATE"
+[[ $now == "$was" ]] && exit 0
+
+webhook=""
+if [[ -r $WEBHOOK_FILE ]]; then webhook="$(tr -d '[:space:]' < "$WEBHOOK_FILE" 2>/dev/null || true)"; fi
+case "$webhook" in
+  https://*) ;;
+  "")        webhook="" ;;
+  *)         echo "deadman: ALERT_WEBHOOK_URL is not an https URL; ignoring it" >&2; webhook="" ;;
+esac
+
+post() {
+  [[ -n $webhook ]] || return 1
+  curl -s -m 15 -H 'Content-Type: application/json' \
+    --data "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" \
+    "$webhook" >/dev/null && return 0
+  echo "deadman: the webhook post failed" >&2
+  return 1
+}
+
+if [[ -z $now ]]; then
+  echo "deadman: recovered — Elastic, Kibana, both Beats and the disks are healthy"
+  post "✅ **Vanguard monitoring recovered** — Elasticsearch, Kibana and both Beats are healthy again." || true
+  exit 0
+fi
+
+printf 'deadman: %s\n' "${problems[@]}" >&2
+body="⚠️ **Vanguard monitoring is not healthy** (sent from the host, outside Elastic):"
+for item in "${problems[@]}"; do body+=$'\n'"• $item"; done
+body+=$'\n'"Until this clears, assume no other alert can reach you."
+if post "$body"; then exit 0; fi
+[[ -z $webhook ]] && echo "deadman: no Discord webhook configured (ALERT_WEBHOOK_URL in .env -> $WEBHOOK_FILE), so the above was not sent" >&2
+exit 1

bin/vgbot-merge-sqlite.py

@@ -0,0 +1,174 @@
+#!/usr/bin/env python3
+"""Merge the Discord bot's old SQLite database into its live MariaDB WITHOUT losing either side.
+
+The bot rebuilds part of its state from Discord on every start, so after switching to MariaDB some tables
+hold fresh rows while the old SQLite file holds months of history. Neither is a superset. This merges them
+row by row and, where a row exists in both, column by column with an explicit rule per column.
+
+    vgbot-merge-sqlite.py --source /data/migration-source.db                 # report only, touches nothing
+    vgbot-merge-sqlite.py --source /data/migration-source.db --apply         # one transaction, rolls back on error
+
+Run it with the gateway stopped (docker compose stop discord-bot) so nothing writes underneath it.
+Idempotent: running it twice changes nothing the second time.
+
+Rules, and why:
+  min / max              ISO-8601 text, so lexicographic order is chronological. "min" keeps the earliest
+                         known date, which is what makes joined-at, rank-since and tracking-start survive a
+                         rebuild that reset them to today. "max" keeps the most recent activity.
+  dest                   the live row wins: it reflects the server as it is right now.
+  source                 the old row wins.
+  *_notnull              prefer that side unless it is NULL, then take the other.
+  source_unless_default  take the old value unless it is the schema's placeholder.
+Every non-key column must appear in the rules; an unclassified column stops the run rather than being guessed.
+"""
+import argparse, sqlite3, sys
+from pathlib import Path
+from sqlalchemy import MetaData, create_engine, inspect, select, func
+
+# ---- how a row present on BOTH sides is resolved ------------------------------------------------------
+ROW_RULE = {                      # whole row chosen by comparing one column (single-row marker tables)
+    "activity_monitor":    ("max", "last_check_at"),      # the later check wins
+    "membership_tracking": ("min", "started_at"),         # keep the original tracking start, not tonight's
+}
+COL_RULE = {
+    "member_activity": {"tracking_started_at": "min", "last_message_at": "max", "last_voice_at": "max"},
+    "member_profiles": {"first_seen_at": "min", "discord_joined_at": "min_notnull", "updated_at": "max",
+                        "username": "dest", "display_name": "dest", "current_rank": "dest", "is_present": "dest"},
+    "member_rank_tenure": {"started_at": "min", "joined_at": "min_notnull",
+                           "role_id": "dest", "active": "dest", "source": "dest"},
+    "membership_state": {"joined_at": "min", "observed_at": "max", "present": "dest"},
+    "recruits": {"recruit_started_at": "min", "discord_joined_at": "min_notnull",
+                 "recruited_by": "source_notnull", "invite_code": "source_notnull",
+                 "invite_source": ("source_unless_default", "unknown"),
+                 "activity_override": "source_notnull", "promoted_at": "dest_notnull",
+                 "username": "dest", "display_name": "dest", "status": "dest",
+                 "current_rank": "dest", "is_active": "dest"},
+    "voice_channel_departures": {"left_at": "max", "guild_id": "dest", "user_id": "dest", "user_name": "dest"},
+}
+# Append-only logs: the id is a surrogate, so rows are matched on a natural key and inserted with a new id.
+APPEND_KEY = {
+    "membership_events": ["user_id", "event_type", "joined_at"],   # also a UNIQUE constraint in the schema
+    "profile_history":   ["user_id", "event", "created_at", "details"],
+    "recruit_history":   ["user_id", "event", "recorded_at"],
+}
+SKIP = {"schema_migrations"}      # the bot owns this; never copied
+
+
+def pick(rule, source_value, dest_value):
+    if isinstance(rule, tuple) and rule[0] == "source_unless_default":
+        return dest_value if source_value in (None, rule[1]) else source_value
+    if rule == "dest": return dest_value
+    if rule == "source": return source_value
+    if rule == "source_notnull": return dest_value if source_value is None else source_value
+    if rule == "dest_notnull": return source_value if dest_value is None else dest_value
+    if rule in ("min", "max"):
+        if source_value is None: return dest_value
+        if dest_value is None: return source_value
+        return (min if rule == "min" else max)(source_value, dest_value)
+    if rule == "min_notnull":
+        if source_value is None: return dest_value
+        if dest_value is None: return source_value
+        return min(source_value, dest_value)
+    raise ValueError(f"unknown rule {rule!r}")
+
+
+def source_tables(connection):
+    names = [r[0] for r in connection.execute(
+        "SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%' ORDER BY name")]
+    out = {}
+    for name in names:
+        if name in SKIP: continue
+        info = list(connection.execute(f'PRAGMA table_info("{name}")'))
+        out[name] = {"columns": [row[1] for row in info],
+                     "key": [row[1] for row in info if row[5]]}
+    return out
+
+
+def plan(source_path, url, apply=False):
+    source = sqlite3.connect(Path(source_path).resolve().as_uri() + "?mode=ro", uri=True)
+    source.row_factory = sqlite3.Row
+    tables = source_tables(source)
+    engine = create_engine(url)
+    metadata = MetaData()
+    metadata.reflect(bind=engine, only=[n for n in tables if n in set(inspect(engine).get_table_names())])
+    report, problems = [], []
+
+    with engine.begin() as target:                      # one transaction; an exception rolls everything back
+        for name, meta in tables.items():
+            if name not in metadata.tables:
+                problems.append(f"{name}: not in the destination schema"); continue
+            table = metadata.tables[name]
+            key = meta["key"] or APPEND_KEY.get(name)
+            rows = [dict(r) for r in source.execute(f'SELECT * FROM "{name}"')]
+            if not rows:
+                report.append((name, 0, 0, 0, "source empty")); continue
+
+            # unclassified columns are a stop, not a guess
+            if name in COL_RULE:
+                managed = set(COL_RULE[name]) | set(meta["key"])
+                missing = [c for c in meta["columns"] if c not in managed]
+                if missing: problems.append(f"{name}: no rule for {missing}"); continue
+
+            if name in APPEND_KEY:
+                natural = APPEND_KEY[name]
+                existing = {tuple(r) for r in target.execute(select(*[table.c[c] for c in natural]))}
+                fresh = [r for r in rows if tuple(r[c] for c in natural) not in existing]
+                if apply and fresh:
+                    target.execute(table.insert(), [{k: v for k, v in r.items() if k != "id"} for r in fresh])
+                report.append((name, len(fresh), 0, len(rows) - len(fresh), f"appended on {'+'.join(natural)}"))
+                continue
+
+            dest = {tuple(r._mapping[c] for c in key): dict(r._mapping)
+                    for r in target.execute(select(table))} if key else {}
+            inserted = updated = same = 0
+            for row in rows:
+                ident = tuple(row[c] for c in key)
+                current = dest.get(ident)
+                if current is None:
+                    if apply: target.execute(table.insert(), [row])
+                    inserted += 1
+                    continue
+                if name in ROW_RULE:
+                    how, column = ROW_RULE[name]
+                    winner = pick(how, row[column], current[column])
+                    merged = row if winner == row[column] and row[column] != current[column] else None
+                elif name in COL_RULE:
+                    merged = {c: pick(COL_RULE[name][c], row[c], current[c]) for c in COL_RULE[name]}
+                    merged = None if all(merged[c] == current[c] for c in merged) else merged
+                else:
+                    merged = None          # insert-only table and the row is already there: leave it alone
+                if merged is None:
+                    same += 1
+                    continue
+                values = {c: v for c, v in merged.items() if c not in key}
+                if apply:
+                    target.execute(table.update().where(
+                        *[table.c[c] == row[c] for c in key]).values(**values))
+                updated += 1
+            how = "merged per column" if name in COL_RULE else ("row rule" if name in ROW_RULE else "insert only")
+            report.append((name, inserted, updated, same, how))
+        if problems:
+            raise SystemExit("STOPPED, nothing changed:\n  " + "\n  ".join(problems))
+
+    source.close()
+    return report
+
+
+if __name__ == "__main__":
+    ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--source", required=True)
+    ap.add_argument("--url", help="SQLAlchemy URL; default: built from the bot's DB_* environment")
+    ap.add_argument("--apply", action="store_true", help="write; otherwise report only")
+    args = ap.parse_args()
+    url = args.url
+    if not url:
+        import os
+        url = (f"mysql+pymysql://{os.environ['DB_USER']}:{os.environ['DB_PASSWORD']}"
+               f"@{os.environ['DB_HOST']}:{os.environ.get('DB_PORT','3306')}/{os.environ['DB_NAME']}?charset=utf8mb4")
+    rows = plan(args.source, url, apply=args.apply)
+    print(f"\n{'table':28} {'insert':>7} {'update':>7} {'same':>6}  how")
+    for name, ins, upd, same, how in sorted(rows):
+        print(f"{name:28} {ins:>7} {upd:>7} {same:>6}  {how}")
+    totals = [sum(r[i] for r in rows) for i in (1, 2, 3)]
+    print(f"{'TOTAL':28} {totals[0]:>7} {totals[1]:>7} {totals[2]:>6}")
+    print("\n" + ("applied." if args.apply else "report only; add --apply to write."))

bin/vgdeband.sh

@@ -0,0 +1,54 @@
+#!/bin/bash
+# One-off / catch-up: remove the timecode overscan band from stream recordings that were published BEFORE the
+# sweep learned to crop it. Re-encodes (libx264, frame timing preserved, audio copied), re-applies the timecode
+# from the sidecar, replaces the file, marks the sidecar band_removed. Skips files whose sidecar already says so.
+#   vgdeband.sh                      all of /srv/media/streams/*/*.mp4 with a strip sidecar
+#   vgdeband.sh FILE.mp4 ...         just these
+# Long: a 3 h 1440p60 night is a few hours of CPU. Run detached: sudo systemd-run --unit vgdeband -p Nice=15 \
+#   -p IOSchedulingClass=idle /usr/local/bin/vgdeband.sh ; follow with journalctl -fu vgdeband
+set -uo pipefail
+ROOT="${MEDIA_ROOT:-/srv/media}"
+vglog() { local lvl="$1"; shift; printf 'ts=%s level=%s component=vgdeband %s\n' "$(date -Is)" "$lvl" "$*"; }
+log() { vglog info "msg=\"$*\""; }
+dur() { ffprobe -v error -show_entries format=duration -of csv=p=0 "$1" 2>/dev/null | cut -d. -f1; }
+# Metrics (LOGGING.md): the same counters the sweep keeps, under component=vgdeband. No-ops if not installed.
+if [[ -r "$(dirname "$0")/vgmetrics.sh" ]]; then
+  VG_MET_COMPONENT=vgdeband
+  . "$(dirname "$0")/vgmetrics.sh" && vgmet_init
+else
+  vgmet_add() { :; }; vgmet_set() { :; }; vgmet_time() { echo 0; }; vgmet_since() { echo 0; }
+fi
+if (( $# )); then files=("$@"); else mapfile -t files < <(find "$ROOT/streams" -mindepth 2 -maxdepth 2 -type f -name '*.mp4' | sort); fi
+for f in "${files[@]}"; do
+  side="${f%.*}.json"
+  [[ -f $side ]] || { vglog info "event=skip reason=no_sidecar file=\"$f\""; continue; }
+  read -r src removed tc ctime < <(python3 -c '
+import json,sys; d=json.load(open(sys.argv[1]))
+print(d.get("source","none"), 1 if d.get("band_removed") else 0, d.get("timecode",""), d.get("timestamp",""))' "$side")
+  [[ $src == strip ]] || { vglog info "event=skip reason=source_$src file=\"$f\""; continue; }
+  (( removed )) && { vglog info "event=skip reason=already_removed file=\"$f\""; continue; }
+  fuser -s "$f" 2>/dev/null && { vglog info "event=skip reason=in_use file=\"$f\""; continue; }
+  w=$(ffprobe -v error -select_streams v:0 -show_entries stream=width -of csv=p=0 "$f")
+  band=$(python3 /usr/local/bin/vgstrip.py band "$w")
+  tmp="$f.part"
+  tcargs=(); [[ -n $tc ]] && tcargs=(-timecode "$tc" -write_tmcd 1); [[ -n $ctime ]] && tcargs+=(-metadata "creation_time=$ctime")
+  vglog info "event=deband file=\"$f\" width=$w band_rows=$band timecode=\"$tc\""
+  job_t0=$(vgmet_time)
+  nice -n 15 ionice -c 3 ffmpeg -nostdin -hide_banner -loglevel error -y -i "$f" \
+    -map 0:v:0 -map '0:a?' \
+    -c:v libx264 -preset veryfast -crf 18 -maxrate 30M -bufsize 60M -pix_fmt yuv420p -profile:v high -threads "${ENCODE_THREADS:-2}" \
+    -vf "crop=iw:ih-${band}:0:0" -fps_mode passthrough -c:a copy \
+    "${tcargs[@]}" -movflags +faststart -f mp4 "$tmp"
+  rc=$?; job_secs=$(vgmet_since "$job_t0"); d1=$(dur "$f"); d2=$(dur "$tmp")
+  vgmet_add "seconds:encode:stream" "$job_secs"
+  if (( rc == 0 )) && [[ -n $d1 && -n $d2 ]] && (( d2 >= d1 - 2 && d2 <= d1 + 2 )); then
+    vgmet_add "jobs:encode:done:stream"; vgmet_add deband 1
+    mv -f "$tmp" "$f"; chown 1000:1000 "$f" 2>/dev/null
+    python3 -c '
+import json,sys; p=sys.argv[1]; d=json.load(open(p)); d["band_rows"]=int(sys.argv[2]); d["band_removed"]=True
+json.dump(d, open(p,"w"))' "$side" "$band"; chown 1000:1000 "$side" 2>/dev/null
+    vglog info "event=done file=\"$f\""
+  else
+    rm -f "$tmp"; vgmet_add "jobs:encode:failed:stream"; vglog error "event=failed file=\"$f\" rc=$rc src_dur=${d1:-?} out_dur=${d2:-?}" >&2
+  fi
+done

bin/vgmetrics.sh

@@ -0,0 +1,63 @@
+# Shared metric counters for the media scripts. Sourced, never executed.
+#
+#   VG_MET_COMPONENT=sweep                      # a plain assignment, NOT a prefix on the next line:
+#   . /usr/local/bin/vgmetrics.sh && vgmet_init  # bash discards "VAR=x . file" assignments on return
+#   vgmet_add "jobs:encode:done:stream"      # +1
+#   vgmet_add "seconds:encode:stream" 41.2   # +41.2
+#
+# Counters accumulate across runs in MARK/metrics/<component>.counters as plain "key value" lines, and a
+# <component>.pid file exists only while the script is executing. dumpd turns both into Prometheus text on
+# its /metrics; nothing here renders or serves anything, so the scripts stay simple and there is one place
+# that knows the metric names (bin/dumpd.py COUNTER_MAP).
+#
+# Keys are "name:label:label..." — the name picks the metric, the rest fill its labels in order.
+# Writes are atomic (temp file + mv) and every failure is silent: metrics must never break a sweep.
+
+VG_MET_DIR="${VG_MET_DIR:-${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
+VG_MET_COMPONENT="${VG_MET_COMPONENT:-$(basename "${0%.sh}")}"
+VG_MET_START=0
+declare -A VGMET=()
+
+vgmet_init() {
+  : "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
+  : "${VG_MET_COMPONENT:=unknown}"
+  VG_MET_START=$(date +%s)
+  mkdir -p "$VG_MET_DIR" 2>/dev/null || return 0
+  local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" k v
+  if [[ -r $f ]]; then
+    while read -r k v; do [[ -n ${k:-} && -n ${v:-} ]] && VGMET["$k"]="$v"; done < "$f"
+  fi
+  vgmet_add runs 1
+  printf '%s\n' "$$" > "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
+  trap vgmet_flush EXIT
+}
+
+# vgmet_add <key> [amount]   — float-safe accumulate
+vgmet_add() {
+  local k="$1" v="${2:-1}"
+  VGMET["$k"]=$(awk -v a="${VGMET[$k]:-0}" -v b="$v" 'BEGIN{ printf "%.3f", a+b }')
+}
+
+vgmet_set() { VGMET["$1"]="$2"; }
+
+vgmet_flush() {
+  : "${VG_MET_DIR:=${MEDIA_ROOT:-/srv/media}/.markers/metrics}"
+  : "${VG_MET_COMPONENT:=unknown}"
+  local f="$VG_MET_DIR/$VG_MET_COMPONENT.counters" tmp now k
+  rm -f "$VG_MET_DIR/$VG_MET_COMPONENT.pid" 2>/dev/null
+  [[ -d $VG_MET_DIR ]] || return 0
+  now=$(date +%s)
+  vgmet_set last_finish "$now"
+  vgmet_set last_duration "$(( now - VG_MET_START ))"
+  tmp=$(mktemp "$VG_MET_DIR/.$VG_MET_COMPONENT.XXXXXX" 2>/dev/null) || return 0
+  for k in "${!VGMET[@]}"; do printf '%s %s\n' "$k" "${VGMET[$k]}"; done > "$tmp" 2>/dev/null
+  chmod 0644 "$tmp" 2>/dev/null
+  mv -f "$tmp" "$f" 2>/dev/null || rm -f "$tmp" 2>/dev/null
+}
+
+# start/elapsed for one job
+vgmet_time()  { date +%s.%N; }
+vgmet_since() { awk -v s="$1" -v e="$(date +%s.%N)" 'BEGIN{ printf "%.3f", e-s }'; }
+
+# No-op fallbacks are defined by the callers if this file is missing, so a half-finished deploy cannot
+# break normalising. Keep every function above failure-tolerant.

bin/vgstrip.py

@@ -162,6 +162,11 @@ def plan(path, arrival_ms=None, tz="America/New_York"):
     else:
         t0 = None; src = "none"
     out = {"source": src, "fps": r.get("fps"), "drift_ms": r.get("drift_ms", 0), "samples": r.get("samples", [])}
+    if src == "strip":
+        # the band is removed (cropped off in a re-encode) by the sweep once it has been decoded; record it
+        _, band = geometry(r["width"]) if r.get("width") else geometry(probe(path)[0])
+        out.update({"band_rows": band, "band_removed": True})
+        print(f"TC_BAND={band}")
     try:
         out["gaps"] = find_gaps(path, r.get("fps") or 60.0)
     except Exception:
@@ -181,4 +186,6 @@ if __name__ == "__main__":
     if len(sys.argv) < 2: print(__doc__); sys.exit(2)
     if sys.argv[1] == "plan":
         plan(sys.argv[2], int(sys.argv[3]) if len(sys.argv) > 3 and sys.argv[3] else None); sys.exit(0)
+    if sys.argv[1] == "band":                 # band rows for a canvas width (vgdeband.sh)
+        print(geometry(int(sys.argv[2]))[1]); sys.exit(0)
     print(json.dumps(analyse(sys.argv[1]), indent=2))

bot/.dockerignore

@@ -1,2 +1,4 @@
-app/__pycache__/
+# only entrypoint.sh is COPYed; keep everything else out of the build context
-app/*.pyc
+app/
+AGENT.md
+README.md

bot/AGENT.md

@@ -15,23 +15,54 @@ What you are given:
 
 | | |
 |---|---|
-| **Runtime** | `python:3.12-slim` + ffmpeg + curl. Dependencies from `app/requirements.txt`, installed at image build (`docker compose build discord-bot`). |
+| **Runtime** | `python:3.12-slim` + ffmpeg + curl, and nothing else. **Your dependencies ship with your code** (see §1a), imported from `/app/vendor` via `PYTHONPATH`. Adding one needs no image rebuild. |
-| **Your code** | bind-mounted at `/app` from `bot/app/`. Not baked into the image. Edit, then `docker compose restart discord-bot`. |
+| **Your code** | a release bundle from your CI, installed at `/srv/bot/app` and bind-mounted at `/app` (§1a). Treat `/app` as read-only; write to `/data`. |
-| **Entry point** | the container runs `python -u /app/main.py` and restarts it if it exits. With no `main.py` it idles. |
+| **Entry point** | the container runs `python -u /app/main.py` and restarts it if it exits. With no bundle installed it idles and logs a hint. |
 | **User** | uid 1000 |
 | **Database** | MariaDB, database `vanguardbot`, user `bot`, reachable at `DB_HOST:DB_PORT`. Yours alone. Other databases on the same server belong to other services; never touch them. |
 | **Persistent disk** | `/data`, yours |
-| **Web UI** | bind `WEB_HOST:WEB_PORT` (0.0.0.0:8000). The proxy publishes it as `https://bot.<domain>/` behind login (§3). |
+| **Web UI** | bind `WEB_HOST:WEB_PORT` (0.0.0.0:8000) — one port for your API *and* your pages. The proxy publishes it as `https://bot.<domain>/` behind login (§3). |
 | **Discord** | `DISCORD_BOT_TOKEN`, `DISCORD_GUILD_ID`. The bot user belongs to the community's existing Discord application. |
 | **Clock** | `TZ=America/New_York` |
 
 ### Environment variables
-`DISCORD_BOT_TOKEN`, `DISCORD_GUILD_ID`, `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`, `DB_PASSWORD`,
+`DISCORD_BOT_TOKEN`, `DISCORD_GUILD_ID`, `DB_BACKEND=mariadb`, `DB_HOST`, `DB_PORT`, `DB_NAME`, `DB_USER`,
-`WEB_HOST`, `WEB_PORT`, `WEB_PUBLIC_URL`, `AUTH_HEADER_USER`, `AUTH_HEADER_GROUPS`, `AUTH_HEADER_EMAIL`,
+`DB_PASSWORD`, `DATA_DIR=/data`, `WEB_HOST`, `WEB_PORT`, `WEB_PUBLIC_URL`, `WEB_SERVE_UI=true`,
-`MEDIA_CLIPS`, `TZ`. Secrets arrive only through env. Never write them to the repo, `/data`, or logs.
+`WEB_ALLOWED_GROUPS=bot-users,admins`, `AUTH_HEADER_USER`, `AUTH_HEADER_GROUPS`, `AUTH_HEADER_EMAIL`,
+`MEDIA_CLIPS`, `PYTHONPATH=/app/vendor`, `TZ`. Secrets arrive only through env. Never write them to the repo,
+`/data`, or logs.
 
 SQLAlchemy URL: `mysql+pymysql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}` (async: `mysql+aiomysql`).
 
+## 1a. Releases: one bundle, dependencies included
+
+Your CI (`.gitea/workflows/ci.yml`, job `bundle`) runs after the tests and produces the artifact
+`vanguard-bot-bundle` — a single tar.gz whose contents **are** `/app`:
+
+    main.py  app/  website/  version.json  BUILD.json  vendor/
+
+`vendor/` is `pip install --only-binary=:all: --target vendor -r requirements.txt`, built on `python:3.12`
+so the wheels match the runtime image's ABI. `BUILD.json` records the version, commit, build time, the
+package list and a sha256 of every file.
+
+**A push to `main` deploys itself.** The workflow's last job posts the bundle to the server's deploy service,
+signed with the repository secret `BOT_DEPLOY_SECRET`; the server verifies the signature and every checksum,
+keeps the previous release for rollback, and restarts the container. The install log comes back in your CI
+job, so a failed deploy fails the run. Branches and pull requests test and build, and deploy nothing.
+
+**Your settings travel with it.** `DISCORD_BOT_TOKEN` (an Actions secret) and `DISCORD_GUILD_ID` (an Actions
+variable) are signed together with the bundle and written into the container's environment by the server, so
+this repo is where they are managed and rotated. The server accepts only the names its `BOT_ENV_KEYS` allows
+and reports the rest as `settings_refused`; it keeps ownership of what it generates, such as `DB_PASSWORD`.
+A new setting means asking the admin to add its name to that list (§6).
+
+Rules that follow from this:
+
+- Keep `requirements.txt` resolvable with **prebuilt wheels only**; a dependency that must compile will fail
+  the build. Ask the admin if you truly need one.
+- Don't import anything that is not in `requirements.txt` or the standard library.
+- Nothing is installed on the server at deploy time, and the server reaches no package index.
+
 ## 2. Process model
 
 - One process, one asyncio loop: the Discord gateway client and the web server run together
@@ -52,6 +83,10 @@ Every request to `https://bot.<domain>/` passes through Caddy, which sends the u
 | `X-Authentik-Groups` | group names, separated by `\|` or `,` — the community uses `admins`, `editors`, `streamers`; others may be added |
 | `X-Authentik-Email`, `X-Authentik-Name`, `X-Authentik-Uid` | extras |
 
+`WEB_SERVE_UI=true` is set on the server, so this process serves your HTML pages at `/` and their assets
+at `/assets/` as well as `/api/…` — same origin, one port, no second service and no proxy of your own.
+Locally the separate website launcher still works; it is off unless `WEB_SERVE_UI` says otherwise.
+
 Rules:
 1. **Implement no login page, no sessions, no passwords.** Identity is the headers, per request.
 2. Port 8000 is loopback-only on the host, so the headers can be trusted. Still fail closed: a request
@@ -59,8 +94,11 @@ Rules:
 3. Authorize by group, read from the header each time. Don't cache roles in your DB.
 4. Serve under `/`; the proxy does no path rewriting for this host.
 5. Build absolute links from `WEB_PUBLIC_URL`.
-6. The Authentik username may differ from the Discord username. Use the header value as the user key;
+6. **The Discord snowflake arrives in `X-Authentik-Jwt`** as the `discord_id` claim (a scope mapping on the
-   if you need the Discord snowflake, resolve it through your gateway client and cache it in your DB.
+   provider supplies it; Authentik takes it from the account's source connection). Decode the JWT payload and
+   use that as the user key. Do not match the Authentik username against Discord names: usernames are mutable
+   and reusable, and server nicknames are self-chosen and not unique, so name matching can mistake one member
+   for another. The signature needs no checking because the header can only come from the proxy.
 
 Until the admin creates the matching Authentik application, the hostname returns an error, never an
 open page. Test against `http://127.0.0.1:8000/` on the server with the headers set by hand.
@@ -99,10 +137,15 @@ message content; Message Content is a privileged intent, enable it only if you n
 
 ## 6. Working with the admin
 
-- **Source**: push to the community Gitea, `ssh://git@git.<domain>:2222/vanguard/discord-bot.git`.
+- **Source**: push to the community Gitea. CI builds the release bundle on the `python312` runner label
-  The admin deploys by pulling into `bot/app` and restarting. CI on the `python` runner label runs your tests.
+  (CPython 3.12 plus node) and, on `main`, installs it on the server. **The test jobs are commented out: run
+  the suites locally before you push** (`python -m pytest -q` and `node --test app/tests/*.test.js`). What CI
+  still enforces: the bundle must import cleanly with only its own `vendor/` on the path, every file must
+  match `BUILD.json`, and the bot must answer on its port within 45 s of the swap or the deploy fails and the
+  previous release stays rollback-ready.
 - **Need infra?** Open a Gitea issue titled `infra: …` stating the exact change: a new env var and why,
   a mount, a Caddy path that should skip login (e.g. a public webhook endpoint), a new database.
   Don't work around the proxy or the firewall.
-- **Dependencies changed?** Say so in the PR; the image must be rebuilt.
+- **Dependencies changed?** Just push: they travel inside the bundle. Only a new *system* package (beyond
+  ffmpeg and curl) needs the admin to rebuild the image.
 - Keep `README.md` updated with every command, env var, and table you add.

bot/Dockerfile

@@ -1,12 +1,11 @@
-# Python runtime for the Discord bot. Source is bind-mounted at /app from ./bot/app (not baked in),
+# Runtime for the Discord bot. The bot's CI builds a self-contained bundle (code + vendor/ dependencies)
-# so the dev edits code and restarts the container. Dependencies come from app/requirements.txt and
+# and deployBotBundle.sh installs it at BOT_APP_DIR, bind-mounted here at /app. Dependencies therefore live
-# are installed at BUILD time: after changing it run  docker compose build discord-bot && docker compose up -d discord-bot
+# WITH the code, not in this image: a new dependency needs no rebuild and no package index on the server.
+# Only the interpreter and the tools AGENT.md promises are baked in.
 FROM python:3.12-slim
-ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1
+ENV PIP_NO_CACHE_DIR=1 PIP_DISABLE_PIP_VERSION_CHECK=1 PYTHONDONTWRITEBYTECODE=1
 RUN apt-get update && apt-get install -y --no-install-recommends ffmpeg ca-certificates curl \
  && rm -rf /var/lib/apt/lists/*
-COPY app/requirements.txt /tmp/requirements.txt
-RUN pip install -r /tmp/requirements.txt
 COPY entrypoint.sh /entrypoint.sh
 RUN chmod +x /entrypoint.sh
 WORKDIR /app

bot/README.md

@@ -2,42 +2,78 @@
 
 Start with **AGENT.md** in this folder: the stack, the rules, the layout, the web-UI auth contract.
 
-Your code goes in this folder (`bot/app/`). The container runs `python -u /app/main.py` and restarts
+## How code reaches the server
-it if it exits. No `main.py` yet → the container idles and logs a hint.
+
+Your repo's Gitea Actions workflow builds a **release bundle**: one tar.gz holding `main.py`, `app/`,
+`website/`, `version.json`, `BUILD.json` and `vendor/` (your dependencies as prebuilt wheels). **A push to
+`main` installs itself**: the workflow's last job posts the bundle to the server's deploy service, which
+checks its signature and every checksum, swaps it into `/srv/bot/app` (bind-mounted at `/app`, previous
+release kept for `--rollback`) and restarts the container. The install log comes back in the CI job.
+
+A branch or pull request builds the artifact and deploys nothing. The test jobs are commented out in the
+workflow — run `python -m pytest -q` and `node --test app/tests/*.test.js` locally before pushing. CI still
+refuses a bundle that cannot import from its own `vendor/`, or whose files do not match `BUILD.json`, and the
+deploy fails if the bot does not answer on its port after the swap.
+
+## Your secrets live in Gitea
+
+Set them on this repo under Settings → Actions. The deploy job sends them with the bundle, signed, and the
+server writes the ones it allows into the environment the container gets:
+
+| where | name | what |
+|---|---|---|
+| Secrets | `BOT_DEPLOY_SECRET` | signs the upload; must match the server (the admin has it) |
+| Secrets | `DISCORD_BOT_TOKEN` | the bot token |
+| Variables | `DISCORD_GUILD_ID` | the Discord server id |
+
+Rotating the token is an edit here plus a re-run of the workflow; nobody needs to touch the server. The
+server keeps what it generates, chiefly `DB_PASSWORD`, and accepts only the names its `BOT_ENV_KEYS` lists —
+anything else is reported in the job as `settings_refused`. Need a new setting? Ask the admin to add its name
+to that list, then add it here.
+
+The image contains **no** bot dependencies: the container imports yours from `/app/vendor`
+(`PYTHONPATH=/app/vendor`). Adding a dependency means editing `app/requirements.txt` and pushing — no image
+rebuild, no package index on the server. The image only changes if you need a new system package (ffmpeg and
+curl are already there); ask the admin for that.
 
 ## Day to day (on the server, in ~/vanguard-stream)
-    docker compose logs -f discord-bot            # live log
+    ./deployBotBundle.sh vanguard-bot-bundle.zip      # install a release by hand (zip or tar.gz)
-    docker compose restart discord-bot            # after editing code
+    ./deployBotBundle.sh --status                     # what is installed; container state
-    docker compose build discord-bot && docker compose up -d discord-bot   # after editing requirements.txt
+    ./deployBotBundle.sh --rollback                   # previous release
-    docker compose exec discord-bot python        # a REPL inside the container
+    docker compose logs -f discord-bot                # live log
-
+    docker compose restart discord-bot                # restart without changing code
-Edit files directly in `~/vanguard-stream/bot/app/` (they are bind-mounted) or scp them up.
+    docker compose exec discord-bot python            # a REPL inside the container (your vendor/ is on sys.path)
 
 ## Environment the container gives you
 | var | value |
 |-----|-------|
-| `DISCORD_BOT_TOKEN` | from `.env` (bundle root) — the Bot added to the existing Authentik Discord application |
+| `DISCORD_BOT_TOKEN` | from `.env` (stack root) — the Bot added to the existing Authentik Discord application |
 | `DISCORD_GUILD_ID`  | the server |
+| `DB_BACKEND` | `mariadb` |
 | `DB_HOST` / `DB_PORT` | `host.docker.internal` / `3306` (MariaDB runs on the host network) |
 | `DB_NAME` / `DB_USER` / `DB_PASSWORD` | your own database + user (`vanguardbot` / `bot` / generated) |
+| `DATA_DIR` | `/data` — persistent, owned by you |
 | `MEDIA_CLIPS` | `/media/clips` — rw mount of `/srv/media/clips` (folder per user, for clip submissions) |
-| `/data` | persistent scratch dir owned by the bot |
+| `WEB_HOST` / `WEB_PORT` | `0.0.0.0` / `8000` — published as `https://bot.<domain>/` via Caddy + Authentik |
-| `WEB_HOST` / `WEB_PORT` | `0.0.0.0` / `8000` — your web UI; public as `https://bot.<domain>/` via Caddy + Authentik |
 | `WEB_PUBLIC_URL` | `https://bot.<domain>` for absolute links |
+| `WEB_SERVE_UI` | `true` — serve your pages at `/` on the same port as `/api` |
+| `WEB_ALLOWED_GROUPS` | `bot-users,admins` — the groups your own code must accept |
 | `AUTH_HEADER_USER` / `_GROUPS` / `_EMAIL` | `X-Authentik-Username` / `X-Authentik-Groups` / `X-Authentik-Email` — identity injected by Caddy, trust these, implement no login |
+| `PYTHONPATH` | `/app/vendor` |
 | `TZ` | America/New_York |
 
 SQLAlchemy URL: `mysql+pymysql://{DB_USER}:{DB_PASSWORD}@{DB_HOST}:{DB_PORT}/{DB_NAME}`
 async: `mysql+aiomysql://...`
 
 ## Database from your own machine
-MariaDB is reachable directly: `ovh1.dumpstervigilantes.rip:3306`, user `bot`, database `vanguardbot`,
+MariaDB is reachable directly: `media.dumpstervigilantes.rip:3306`, user `bot`, database `vanguardbot`,
 password from the admin (`./configure.py --show` on the server). Root is local-only. New connections are
 rate limited at the firewall; if yours get dropped, ask the admin to add your IP to `db_allow`.
 
 ## Conventions
 - Run as uid 1000 (files you write under `/media/clips` are owned like everything else there).
+- `/app` holds a released bundle; treat it as read-only and write to `/data` instead.
 - If you write clips: `/media/clips/<discord-username>/<name>.mp4` plus `<name>.json` with
   `{"timestamp": "<UTC ISO-8601>", "source": "medal", "source_url": "...", "uploader": "...", "title": "..."}`
   — the Jellyfin plugin reads that sidecar for the exact recording time.
-- Log to stdout; the container captures it.
+- Log to stdout; the container captures it. One event per line, `key=value`, keeps it searchable (`LOGGING.md`).

bot/app/requirements.txt

@@ -1,8 +0,0 @@
-# Baseline for a discord.py bot with MariaDB. Add what you need; then:
-#   docker compose build discord-bot && docker compose up -d discord-bot
-discord.py>=2.4
-aiomysql>=0.2
-PyMySQL>=1.1
-SQLAlchemy>=2.0
-python-dotenv>=1.0
-yt-dlp

bot/entrypoint.sh

@@ -1,7 +1,13 @@
 #!/bin/sh
-# Run the bot if there is one; otherwise idle so the container stays up until code arrives.
+# /app is a bundle built by the bot's CI: main.py + app/ + website/ + vendor/ (its dependencies).
+# PYTHONPATH=/app/vendor is set by compose, so nothing is installed at run time.
 if [ -f /app/main.py ]; then
+  if [ ! -d /app/vendor ]; then
+    echo "discord-bot: /app has code but no vendor/ — this image installs no bot dependencies."
+    echo "             Install a CI bundle:  ./deployBotBundle.sh <vanguard-bot-bundle.tar.gz>"
+  fi
   exec python -u /app/main.py
 fi
-echo "discord-bot: no /app/main.py yet — drop the bot code into ./bot/app and 'docker compose restart discord-bot'"
+echo "discord-bot: /app is empty — install a bundle from the bot's CI:  ./deployBotBundle.sh <bundle.tar.gz>"
+echo "             (artifact 'vanguard-bot-bundle' on the VanguardBot repo's ci workflow)"
 exec sleep infinity

check.sh

@@ -5,15 +5,15 @@ cd "$(dirname "$0")"
 fail=0
 say() { printf '%-28s %s\n' "$1" "$2"; }
 
-for f in install.sh addCommunityMediaServer.sh installJellyfinPlugin.sh registerGiteaRunner.sh createGiteaDb.sh deployLive.sh addClipUser.sh firewall/firewall.sh bin/*.sh home/*.sh bot/entrypoint.sh mariadb/init/*.sh; do
+for f in install.sh addCommunityMediaServer.sh installJellyfinPlugin.sh registerGiteaRunner.sh createGiteaDb.sh deployLive.sh addClipUser.sh deploy.sh deployBotBundle.sh switchToClone.sh firewall/firewall.sh bin/*.sh home/*.sh bot/entrypoint.sh mariadb/init/*.sh; do
   [[ -f $f ]] || continue
   bash -n "$f" && say "$f" ok || { say "$f" "BASH SYNTAX"; fail=1; }
 done
-for f in configure.py bin/dumpd.py bin/vgstrip.py; do
+for f in configure.py bin/dumpd.py bin/vgstrip.py observability/kibana-setup.py observability/kibana-alarms.py; do
   python3 -m py_compile "$f" && say "$f" ok || { say "$f" "PYTHON SYNTAX"; fail=1; }
 done
 rm -rf __pycache__ bin/__pycache__
-python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
+python3 -c "import yaml,sys; [yaml.safe_load(open(f)) for f in sys.argv[1:]]" docker-compose.yml mediamtx.example.yml observability/filebeat.yml observability/metricbeat.yml observability/kibana.yml && say "yaml" ok || { say "yaml" "INVALID"; fail=1; }
 node -e "
 const fs=require('fs');
 for (const f of ['dock/index.html','dock/clock/index.html','clips/index.html','about/index.html']) { const html=fs.readFileSync(f,'utf8'); for (const m of html.matchAll(/<script(?:[^>]*)>([\s\S]*?)<\/script>/g)) if (m[1].trim()) new Function(m[1]); }
@@ -22,13 +22,31 @@ diff <(sed '$d' live/src/vgclock.js | sed 's/})(typeof window.*/})(window);/') d
 # generated Caddyfile / mediamtx / copyparty from a dry configure run must at least be produced + parse
 T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "
 import configure as c, yaml
-c.write_caddy('ovh1.example.com','auth.example.com','watch.example.com','git.example.com','bot.example.com','live.example.com')
+c.write_caddy('media.example.com','auth.example.com','watch.example.com','git.example.com','bot.example.com','live.example.com','files.example.com','sync.example.com','clips.example.com')
-c.write_mediamtx('ovh1.example.com'); yaml.safe_load(open('mediamtx.yml')); c.write_copyparty()
+c.write_mediamtx('media.example.com'); yaml.safe_load(open('mediamtx.yml')); c.write_copyparty()
 assert 'authMethod: http' in open('mediamtx.yml').read(); assert 'idp-h-usr' in open('copyparty.conf').read()
 " ) && say "configure.py generators" ok || { say "configure.py generators" "FAILED"; fail=1; }
 rm -rf "$T"
+# dumpd GET /metrics: valid Prometheus text out of script counters + a fake media tree
+T=$(mktemp -d); mkdir -p "$T/.markers/metrics" "$T/streams/u/Title/.live"
+printf 'runs 2\njobs:encode:done:stream 1\nseconds:encode:stream 12.5\nlast_finish 1762210000\n' > "$T/.markers/metrics/sweep.counters"
+touch -d '2020-01-01' "$T/streams/u/Title/.live/old.mp4"
+if MEDIA_ROOT="$T" MTX_API_URL=http://127.0.0.1:1 python3 - <<'PYMET' >/dev/null 2>&1
+import re, sys
+sys.path.insert(0, "bin")
+import dumpd
+text = dumpd.metrics_text()
+for line in text.splitlines():
+    assert line.startswith("#") or re.fullmatch(r'[a-z_]+(\{[^}]*\})? -?[0-9.]+', line), line
+assert 'vanguard_media_jobs_total{component="sweep",kind="stream",outcome="done",verdict="encode"} 1' in text
+assert 'vanguard_media_last_finish_timestamp_seconds{component="sweep"} 1762210000' in text   # not 1.76e+09
+assert 'vanguard_media_pending{kind="stream"} 1' in text
+assert 'vanguard_mediamtx_up 0' in text and 'vanguard_streams_active' not in text             # no fake zeros
+PYMET
+then say "dumpd /metrics" ok; else say "dumpd /metrics" "FAILED"; fail=1; fi
+rm -rf "$T" bin/__pycache__
 if command -v caddy >/dev/null; then
-  T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "import configure as c; c.write_caddy('ovh1.example.com','a.example.com','w.example.com','g.example.com','b.example.com','l.example.com')" && caddy validate --adapter caddyfile --config Caddyfile >/dev/null 2>&1 ) && say "Caddyfile (caddy validate)" ok || { say "Caddyfile (caddy validate)" "INVALID"; fail=1; }
+  T=$(mktemp -d); cp configure.py "$T/"; ( cd "$T" && VG_DRY=1 python3 -c "import configure as c; c.write_caddy('media.example.com','a.example.com','w.example.com','g.example.com','b.example.com','l.example.com','f.example.com','s.example.com','c.example.com')" && caddy validate --adapter caddyfile --config Caddyfile >/dev/null 2>&1 ) && say "Caddyfile (caddy validate)" ok || { say "Caddyfile (caddy validate)" "INVALID"; fail=1; }
   rm -rf "$T"
 fi
 [[ -d live/node_modules ]] && ( cd live && npx --no-install vite build --logLevel error >/dev/null 2>&1 ) && say "live portal build" ok || say "live portal build" "(skipped or failed — run npm run build in live/)"

clips/index.html

@@ -74,9 +74,11 @@ async function refresh(){
     pills.push(`<span class="pill ${d.folder_configured?'on':'off'}">share ${d.folder_configured?'configured':'not yet'}</span>`);
     if (d.device_id) pills.push(`<span class="pill ${d.device_connected?'on':'off'}">your device ${d.device_connected?'connected':'not connected'}</span>`);
     if (d.files_here != null) pills.push(`<span class="pill on">${d.files_here} file(s) here</span>`);
+    if (d.quota_gb) pills.push(`<span class="pill ${d.paused_for_quota?'off':(d.used_gb > d.quota_gb*0.9?'':'on')}">${d.used_gb} / ${d.quota_gb} GB${d.paused_for_quota?' — PAUSED (over quota)':''}</span>`);
     if (d.remote_completion != null && d.remote_completion < 100) pills.push(`<span class="pill">syncing ${Math.round(d.remote_completion)}%</span>`);
     $('pills').innerHTML = pills.join('');
     $('status').textContent = d.error ? d.error : (d.device_id ? `paired with ${d.device_name || 'your device'} (${d.device_id.slice(0,7)}…)` : 'not paired yet — do step 2');
+    if (d.paused_for_quota) $('status').textContent += `\nYour folder is over its ${d.quota_gb} GB quota, so syncing is paused. Delete older clips from your clip folder (they are removed here too) or ask an admin to raise the quota; syncing resumes automatically under 90 %.`;
     if (d.device_id) $('devid').value = d.device_id;
   } catch (e) { $('pills').innerHTML = `<span class="pill off">error</span>`; $('status').textContent = e.message; $('status').className = 'status bad'; }
 }

configure.py

@@ -4,8 +4,17 @@
   ./configure.py --full     first-time setup: asks every question (current values as defaults)
   ./configure.py            manage: skips what is already set; add/remove/reset streamers & editors
   ./configure.py --show     print current streamers, editors, domain and the OBS settings
-  ./configure.py --hosts AUTH WATCH [GIT [BOT [LIVE]]]   set the Authentik/Jellyfin/Gitea/bot/live hostnames and apply (non-interactive;
+  ./configure.py --apply    regenerate everything from the values already on disk and apply (non-interactive;
-                                      used by addCommunityMediaServer.sh)
+                            used by deploy.sh). Fails if the main hostname was never set.
+  ./configure.py --hosts AUTH WATCH [GIT [BOT [LIVE [FILES [SYNC [CLIPS [DIAG]]]]]]]
+                            set the other hostnames and apply (non-interactive; used by addCommunityMediaServer.sh).
+                            Pass "" to leave one unset. NOTE: --apply reads the hostnames back out of the
+                            Caddyfile, so it can only regenerate hosts that are already there. Introducing a
+                            NEW host needs --full or this.
+
+Hosts: main = the gateway page at /, files = copyparty, clips = clip intake, sync = Syncthing UI,
+       live = portal + OBS dock (/dock/, /dock/clock/, /dump, /me, /time), diag = logs/metrics/alerts,
+       auth, watch, git, bot.
 
 Reads  : Caddyfile (hostnames), mediamtx.yml (streamers), .env
          (editors/admins are Authentik groups now: editors, media-admins, admins — nothing local)
@@ -15,7 +24,8 @@ Writes : Caddyfile, mediamtx.yml, copyparty.conf, stream-token.secret -> /etc/va
          Discord bot token/guild), folders, restarts
 Set VG_DRY=1 to write files only (no sudo / docker / folders).
 """
-import getpass, hashlib, os, re, secrets, shutil, subprocess, sys
+import base64, getpass, hashlib, json, os, re, secrets, shutil, subprocess, sys, time
+import urllib.error, urllib.request
 from pathlib import Path
 
 HERE = Path(__file__).resolve().parent
@@ -127,6 +137,20 @@ def ask_name(kind, taken):
         return n
 
 # ------------------------------------------------------------------ writers
+# Access log per site, JSON on stderr (docker logs). Identity headers Authentik injected are kept (who did what);
+# cookies and tokens are dropped. See LOGGING.md.
+CADDY_LOG = """	log {
+		output stderr
+		format filter {
+			wrap json
+			fields {
+				request>headers>Cookie delete
+				request>headers>Authorization delete
+				request>headers>X-Stream-Auth delete
+			}
+		}
+	}
+"""
 AUTHENTIK_FORWARD_AUTH = """	# /logout on every gated host = sign out of the Discord login (domain-wide session)
 	redir /logout /outpost.goauthentik.io/sign_out 302
 	# Authentik forward-auth (embedded outpost). Needs a Proxy Provider in Authentik — domain-level for the
@@ -142,7 +166,7 @@ AUTHENTIK_FORWARD_AUTH = """	# /logout on every gated host = sign out of the Dis
 
 def gate(required_group=""):
     """Caddy lines: Authentik forward-auth, then deny unless X-Authentik-Groups contains required_group."""
-    block = AUTHENTIK_FORWARD_AUTH
+    block = CADDY_LOG + AUTHENTIK_FORWARD_AUTH
     if required_group:
         g = re.escape(required_group)
         block += f"""	@lacks_group not header_regexp X-Authentik-Groups (^|\\|){g}(\\||$)
@@ -152,8 +176,9 @@ def gate(required_group=""):
 """
     return block
 
-def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", live_host=""):
+def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", live_host="", files_host="", sync_host="", clips_host="", diag_host=""):
     env = read_env()
+    gate_any = gate("")
     extra = ""
     apex, target = env.get("APEX_HOST", ""), env.get("APEX_REDIRECT", "")
     if apex and target:
@@ -162,18 +187,82 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
 {apex} {{
 	redir {target} 302
 }}
+"""
+    if files_host:
+        extra += f"""
+# vanguard:files
+{files_host} {{
+	encode zstd gzip
+{gate_any}	# copyparty at the root. It honours X-Authentik-Username / -Groups only when the request also carries a header
+	# whose secret NAME only this proxy knows (copyparty idp-h-key). Clients cannot forge identity.
+	reverse_proxy 127.0.0.1:3923 {{
+		header_up X-Vanguard-{env.get("COPYPARTY_IDP_KEY", "key")} 1
+		flush_interval -1
+	}}
+}}
+"""
+    if sync_host:
+        extra += f"""
+# vanguard:sync
+{sync_host} {{
+	encode zstd gzip
+{gate(env.get("SYNC_REQUIRE_GROUP", "media-admins"))}	# Syncthing admin UI (its own GUI login is disabled; the gate + group is the login)
+	reverse_proxy 127.0.0.1:8384 {{
+		header_up Host {{upstream_hostport}}
+	}}
+}}
+"""
+    if clips_host:
+        extra += f"""
+# vanguard:clips
+{clips_host} {{
+	encode zstd gzip
+{gate_any}	@me path /me /me/*
+	handle @me {{
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity + /me/clips self-service
+	}}
+	handle {{
+		root * /srv/clips
+		try_files {{path}} /index.html
+		file_server
+	}}
+}}
+"""
+    if diag_host:
+        extra += f"""
+# vanguard:diag
+{diag_host} {{
+	encode zstd gzip
+{gate(env.get("DIAG_REQUIRE_GROUP", "media-admins"))}	# Kibana IS this host: logs (Discover), dashboards and alerting in one app, served at the root so there
+	# is no path prefix to get wrong and no index page of links. Elasticsearch itself is never published.
+	reverse_proxy 127.0.0.1:5601 {{
+		flush_interval -1
+	}}
+}}
 """
     if live_host:
         extra += f"""
 # vanguard:live
 {live_host} {{
 	encode zstd gzip
-{gate(env.get("LIVE_REQUIRE_GROUP", ""))}	@me path /me /me/*
+{gate(env.get("LIVE_REQUIRE_GROUP", ""))}	handle /whoami {{
+		header Content-Type text/plain
+		respond "user:   {{http.request.header.X-Authentik-Username}}
+groups: {{http.request.header.X-Authentik-Groups}}"
+	}}
+	@me path /me /me/*
 	handle @me {{
-		reverse_proxy 127.0.0.1:8080          # dumpd: identity + /me/stream-token for the logged-in streamer (Go Live)
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity, stream tokens (Go Live + dock)
 	}}
 	handle /time {{
-		reverse_proxy 127.0.0.1:8080          # server clock for the timecode strip
+		reverse_proxy 127.0.0.1:8080          # server clock for the timecode strip (clock page + Go Live)
+	}}
+	handle /dump* {{
+		reverse_proxy 127.0.0.1:8080          # dump tonight (dock)
+	}}
+	handle_path /dock* {{
+		root * /srv/dock
+		file_server                           # OBS browser dock + /dock/clock/
 	}}
 	handle_path /api/* {{
 		reverse_proxy 127.0.0.1:9997
@@ -211,7 +300,7 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
 # vanguard:git
 {git_host} {{
 	encode zstd gzip
-	reverse_proxy 127.0.0.1:3000
+{CADDY_LOG}	reverse_proxy 127.0.0.1:3000
 }}
 """
     if auth_host:
@@ -219,7 +308,7 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
 # vanguard:auth
 {auth_host} {{
 	encode zstd gzip
-	reverse_proxy 127.0.0.1:9000
+{CADDY_LOG}	reverse_proxy 127.0.0.1:9000
 }}
 """
     if watch_host:
@@ -227,26 +316,19 @@ def write_caddy(domain, auth_host="", watch_host="", git_host="", bot_host="", l
 # vanguard:watch
 {watch_host} {{
 	encode zstd gzip
-	reverse_proxy 127.0.0.1:8096
+{CADDY_LOG}	reverse_proxy 127.0.0.1:8096
 }}
 """
-    sync_group = env.get("SYNC_REQUIRE_GROUP", "media-admins")
+    Path("Caddyfile").write_text(f"""{{
-    sync_gate = ""
+	servers {{
-    if sync_group:
+		metrics                           # Prometheus metrics at http://127.0.0.1:2019/metrics (LOGGING.md)
-        g = re.escape(sync_group)
-        sync_gate = f"""	@sync_lacks_group {{
-		path /sync /sync/*
-		not header_regexp X-Authentik-Groups (^|\\|){g}(\\||$)
 	}}
-	handle @sync_lacks_group {{
+}}
-		respond "Syncthing admin is for the '{sync_group}' group." 403
+
-	}}
+# vanguard:main
-"""
-    Path("Caddyfile").write_text(f"""# vanguard:main
 {domain} {{
 	encode zstd gzip
-{AUTHENTIK_FORWARD_AUTH}
+{gate_any}	# who does Authentik say I am? (debug: shows exactly the identity headers the apps receive)
-	# who does Authentik say I am? (debug: shows exactly the identity headers the apps receive)
 	handle /whoami {{
 		header Content-Type text/plain
 		respond "user:   {{http.request.header.X-Authentik-Username}}
@@ -254,60 +336,25 @@ groups: {{http.request.header.X-Authentik-Groups}}
 email:  {{http.request.header.X-Authentik-Email}}
 name:   {{http.request.header.X-Authentik-Name}}"
 	}}
-
-	# dump endpoint + stream tokens (python service on the host; identity from the X-Authentik-* headers Caddy just set)
-	handle /dump* {{
-		reverse_proxy 127.0.0.1:8080
-	}}
 	@me path /me /me/*
 	handle @me {{
-		reverse_proxy 127.0.0.1:8080          # dumpd: identity, stream tokens, clips self-service
+		reverse_proxy 127.0.0.1:8080          # dumpd: identity for the gateway page
 	}}
-	handle /time {{
+	redir /about* / 308                       # old bookmarks: the gateway page used to live at /about/
-		reverse_proxy 127.0.0.1:8080          # server clock for the timecode strip
-	}}
-
-	# Syncthing admin UI: Authentik + group {sync_group or 'any'} (Syncthing's own GUI login is disabled)
-{sync_gate}	redir /sync /sync/ 308
-	handle_path /sync/* {{
-		reverse_proxy 127.0.0.1:8384 {{
-			header_up Host {{upstream_hostport}}
-		}}
-	}}
-
-	# OBS browser dock (static)
-	handle_path /dock* {{
-		root * /srv/dock
-		file_server
-	}}
-
-	# gateway page: role-aware links + instructions (static; talks to /me)
-	redir /about /about/ 308
-	handle_path /about/* {{
-		root * /srv/about
-		file_server
-	}}
-
-	# clip intake self-service (static page; talks to /me/clips)
-	redir /clips /clips/ 308
-	handle_path /clips/* {{
-		root * /srv/clips
-		file_server
-	}}
-
-	# copyparty. It honours X-Authentik-Username / -Groups only when the request also carries a header whose
-	# secret NAME only this proxy knows (copyparty idp-h-key). Clients cannot forge identity.
 	handle {{
-		reverse_proxy 127.0.0.1:3923 {{
+		root * /srv/about
-			header_up X-Vanguard-{env.get("COPYPARTY_IDP_KEY", "key")} 1
+		try_files {{path}} /index.html
-			flush_interval -1
+		file_server                           # the gateway page is the root of the main host
-		}}
 	}}
 }}
 """ + extra)
 
-def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", bot_host="", live_host="", main_host="", live_webhook=None):
+def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", bot_host="", live_host="", main_host="", live_webhook=None, files_host="", sync_host="", clips_host="", diag_host=""):
     env = read_env()
+    if files_host: env["FILES_HOST"] = files_host
+    if sync_host: env["SYNC_HOST"] = sync_host
+    if clips_host: env["CLIPS_HOST"] = clips_host
+    if diag_host: env["DIAG_HOST"] = diag_host
     env["LIVE_HOST"] = live_host or env.get("LIVE_HOST", "live.example.com")
     if main_host: env["MAIN_HOST"] = main_host
     if live_webhook is not None: env["LIVE_WEBHOOK_URL"] = live_webhook
@@ -316,6 +363,14 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b
     env.setdefault("APEX_REDIRECT", "")      # where the bare domain sends people, e.g. https://watch.HOST/
     env.setdefault("SYNC_REQUIRE_GROUP", "media-admins")  # Caddy gate for /sync on the main host
     env.setdefault("STREAM_TOKEN_SECRET", secrets.token_urlsafe(48))   # signs publish tokens (dumpd)
+    env.setdefault("AUTHENTIK_BOOTSTRAP_PASSWORD", "".join(c for c in secrets.token_urlsafe(30) if c.isalnum())[:20])  # akadmin on first start
+    env.setdefault("AUTHENTIK_BOOTSTRAP_TOKEN", secrets.token_urlsafe(40))
+    env.setdefault("GITEA_ADMIN_USER", "vanguard")
+    env.setdefault("GITEA_ADMIN_PASSWORD", "".join(c for c in secrets.token_urlsafe(30) if c.isalnum())[:20])
+    env.setdefault("DEPLOY_AUTO", "0")                                  # 1 = Gitea push webhook runs deploy.sh (off until trusted)
+    env.setdefault("DEPLOY_WEBHOOK_SECRET", secrets.token_urlsafe(32))  # Gitea webhook secret (HMAC)
+    env.setdefault("DEPLOY_BRANCH", "main")
+    env.setdefault("DEPLOY_NOTIFY_WEBHOOK_URL", "")                    # optional Discord webhook for deploy notices
     env.setdefault("COPYPARTY_IDP_KEY", "".join(c for c in secrets.token_urlsafe(40) if c.isalnum())[:32])  # Caddy->copyparty idp-h-key
     env.setdefault("BOT_REQUIRE_GROUP", "bot-users")   # Caddy 403s anyone not in this Authentik group; empty = any logged-in member
     env.setdefault("LIVE_REQUIRE_GROUP", "")
@@ -332,6 +387,27 @@ def write_env(watch_host, discord_token=None, discord_guild=None, git_host="", b
     env.setdefault("BOT_DB_NAME", "vanguardbot")
     env.setdefault("BOT_DB_USER", "bot")
     env.setdefault("BOT_DB_PASSWORD", "".join(c for c in secrets.token_urlsafe(40) if c.isalnum())[:32])
+    env.setdefault("BOT_DEPLOY_SECRET", secrets.token_hex(32))   # Gitea repo secret BOT_DEPLOY_SECRET: signs the bot's release bundle
+    env.setdefault("BOT_DEPLOY_AUTO", "1")                       # 0 = refuse bundles posted by CI; install by hand instead
+    env.setdefault("BOT_ENV_KEYS", "DISCORD_BOT_TOKEN,DISCORD_GUILD_ID")   # the only .env keys the bot's CI may set
+    env.setdefault("DIAG_REQUIRE_GROUP", "media-admins")     # who may see logs, metrics and alerts
+    env.setdefault("ALERT_WEBHOOK_URL", "")                  # Discord webhook: vanguard-alerts.sh delivery AND the deadman
+    env.setdefault("KIBANA_ENCRYPTION_KEY", secrets.token_hex(32))   # without a fixed key Kibana loses saved objects on restart
+    # Elasticsearch security is ON (observability/kibana.yml explains why, and why nobody sees a login).
+    # These are service credentials, never typed by a person: Discord remains the only human login.
+    def pw(): return "".join(c for c in secrets.token_urlsafe(40) if c.isalnum())[:32]
+    env.setdefault("ES_ELASTIC_PASSWORD", pw())      # the elastic superuser; bootstrapped on first secured start
+    env.setdefault("ES_KIBANA_PASSWORD", pw())       # kibana_system: Kibana's own internal account
+    env.setdefault("VG_BEATS_USER", "vanguard_beats")
+    env.setdefault("VG_BEATS_PASSWORD", pw())        # what Filebeat and Metricbeat ship as
+    env.setdefault("VG_DIAG_USER", "vanguard_diag")  # who every Kibana visitor silently becomes
+    env.setdefault("VG_DIAG_PASSWORD", pw())
+    # What that visitor may do. superuser keeps today's behaviour exactly (everyone past the gate is an admin).
+    # Narrowing this to "editor" is step one of the read-only tier in STATUS.md 9b.
+    env.setdefault("DIAG_KIBANA_ROLES", "superuser")
+    env.setdefault("ALERT_INDEX", "vanguard-alerts")         # the alert bus: Kibana writes, vanguard-alerts.sh delivers
+    env.setdefault("LOG_RETENTION_DAYS", "30")               # ILM delete phase for filebeat-*/metricbeat-*
+    env.setdefault("ALERT_RETENTION_DAYS", "90")             # how long delivered alerts stay queryable
     env.setdefault("DISCORD_BOT_TOKEN", "")          # fill in: Discord dev portal -> the Authentik app -> Bot
     env.setdefault("DISCORD_GUILD_ID", "")
     env["WATCH_HOST"] = watch_host or "watch.example.com"
@@ -354,9 +430,15 @@ def write_mediamtx(public_host=""):
     Path("mediamtx.yml").write_text(f"""# MediaMTX — ingest only. No playback endpoints exposed.  (generated by configure.py)
 logLevel: info
 
-rtsp: no
+# RTSP only on the container loopback: the audio-mirror hook (bin/mtx-on-ready.sh) reads the source and publishes
+# "<path>.webrtc" through it. Not published by compose, unreachable from outside.
+rtsp: yes
+rtspAddress: 127.0.0.1:8554
+rtspTransports: [tcp]
+rtspEncryption: "no"
 moq: no
-metrics: no
+metrics: yes                     # Prometheus text format, scraped by Metricbeat (LOGGING.md); loopback only
+metricsAddress: :9998
 pprof: no
 
 # live portal (served to the browser only via Caddy + Authentik on live.HOST; these ports are published on 127.0.0.1)
@@ -399,15 +481,16 @@ pathDefaults:
   recordPartDuration: 1s
   recordSegmentDuration: 12h      # one file per stream unless it runs >12h
   recordDeleteAfter: 0s           # never auto-delete; the archive job handles it
-  # optional Discord "is live" notice (LIVE_WEBHOOK_URL set in .env). busybox wget in the -ffmpeg image.
+  # on ready: optional Discord "is live" notice (LIVE_WEBHOOK_URL in .env) + the WebRTC audio mirror
-  runOnReady: >
+  # (bin/mtx-on-ready.sh, bind-mounted by compose; AAC cannot cross WebRTC, so audio is transcoded to Opus into <path>.webrtc)
-    sh -c '[ -n "$LIVE_WEBHOOK_URL" ] || exit 0;
+  runOnReady: /vanguard/on-ready.sh
-    u="${{MTX_PATH%%/*}}"; t="${{MTX_PATH#*/}}";
-    wget -q -O /dev/null --header="Content-Type: application/json"
-    --post-data="{{\\"content\\":\\"🔴 **$u** is live: $t — https://$LIVE_HOST/\\"}}" "$LIVE_WEBHOOK_URL"'
   runOnReadyRestart: no
 
 paths:
+  # audio mirrors (first rule wins): never recorded, no hook; only publishable from inside the container (dumpd)
+  ~^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+\\.webrtc$:
+    record: no
+    runOnReady: ""
   # <username>/<Title>; dumpd decides who may publish where (token user == prefix)
   ~^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$: {{}}
 """)
@@ -464,15 +547,15 @@ def run(cmd, **kw):
     if DRY: return 0
     return subprocess.call(cmd, **kw)
 
-def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", discord_token=None, discord_guild=None, git_host="", bot_host="", live_host="", live_webhook=None):
+def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", discord_token=None, discord_guild=None, git_host="", bot_host="", live_host="", live_webhook=None, files_host="", sync_host="", clips_host="", diag_host=""):
     # admin_pw / editors are legacy no-ops: access is by Authentik group now
     if not streamers and Path("mediamtx.yml").exists() and "CHANGEME" in Path("mediamtx.yml").read_text():
         print("\n!! mediamtx.yml contains CHANGEME placeholders — it looks like the repo template was copied over the")
         print("   generated config. Applying now would leave NO streamers (OBS ingest and the dump button stop working).")
         if not yes("   continue anyway?", False): print("aborted; add streamers first ([s] in the menu)"); sys.exit(1)
     print("\n== writing configs")
-    write_env(watch_host, discord_token, discord_guild, git_host, bot_host, live_host, domain, live_webhook)
+    write_env(watch_host, discord_token, discord_guild, git_host, bot_host, live_host, domain, live_webhook, files_host, sync_host, clips_host, diag_host)
-    write_caddy(domain, auth_host, watch_host, git_host, bot_host, live_host); write_mediamtx(domain); write_copyparty()
+    write_caddy(domain, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); write_mediamtx(domain); write_copyparty()
     Path("stream-token.secret").write_text(read_env()["STREAM_TOKEN_SECRET"] + "\n"); os.chmod("stream-token.secret", 0o600)
     run(["sudo", "install", "-d", "-o", os.environ.get("SUDO_USER") or os.environ.get("USER") or "debian", "/srv/live"])
     print("== app data folders")
@@ -489,39 +572,284 @@ def apply(domain, admin_pw, streamers, editors, auth_host="", watch_host="", dis
     run(["sudo", "install", "-d", "-m", "0750", "/etc/vanguard"])
     run(["sudo", "install", "-m", "0600", "stream-token.secret", "/etc/vanguard/stream-token.secret"])
     run(["sudo", "install", "-m", "0644", "syncthing/stignore", "/etc/vanguard/stignore"])   # dumpd copies it into new clips folders
+    e = read_env()
+    Path("deploy.env").write_text("".join(f"{k}={e.get(k, '')}\n" for k in ("DEPLOY_AUTO", "DEPLOY_WEBHOOK_SECRET", "DEPLOY_BRANCH", "DEPLOY_NOTIFY_WEBHOOK_URL", "BOT_DEPLOY_AUTO", "BOT_DEPLOY_SECRET", "BOT_ENV_KEYS")) + f"DEPLOY_REPO_DIR={Path.cwd()}\n"); os.chmod("deploy.env", 0o600)
+    Path("alert-webhook").write_text((e.get("ALERT_WEBHOOK_URL") or "") + "\n"); os.chmod("alert-webhook", 0o600)
+    # URL is a credential for that channel, so it is not world-readable.
+    # 0600 root: the only readers are root timers (vanguard-deadman.sh, vanguard-alerts.sh). It used to be
+    # group-readable by nobody(65534) for the Alertmanager container, which no longer exists.
+    run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "alert-webhook", "/etc/vanguard/alert-webhook"])
+    run(["sudo", "install", "-m", "0600", "deploy.env", "/etc/vanguard/deploy.env"])           # read by dumpd (/deploy webhook) and deploy.sh
+    if all(Path(f).exists() for f in ("deploy.sh", "systemd/deploy-pending.service", "systemd/deploy-pending.timer")):
+        run(["sudo", "install", "-m", "0755", "deploy.sh", "/usr/local/bin/vanguard-deploy"])
+        run(["sudo", "install", "-m", "0644", "systemd/deploy-pending.service", "systemd/deploy-pending.timer", "/etc/systemd/system/"])
+        run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "deploy-pending.timer"])
+    else:
+        print("   (deploy pipeline files missing — deploy.sh / systemd/deploy-pending.* not uploaded; skipping vanguard-deploy install)")
+    run(["sudo", "install", "-m", "0755", "deployBotBundle.sh", "/usr/local/bin/vanguard-deploy-bot"])   # dumpd runs this for POST /deploy-bot
+    run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/bot", "/srv/bot/app"])               # the bot's released bundle -> /app
+    run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/appdata/elasticsearch"])           # the ES image runs as uid 1000
+    # Elasticsearch refuses to start below this; it is a host setting, not a container one.
+    run(["sudo", "sysctl", "-q", "-w", "vm.max_map_count=262144"])
+    Path("99-vanguard-elasticsearch.conf").write_text("vm.max_map_count=262144\n")
+    run(["sudo", "install", "-m", "0644", "99-vanguard-elasticsearch.conf", "/etc/sysctl.d/"])
+    run(["sudo", "install", "-d", "/var/lib/vanguard"])                                                  # deadman state
+    run(["sudo", "install", "-d", "-o", "1000", "-g", "1000", "/srv/media/.markers/metrics"])            # script counters -> dumpd /metrics
+    # The media scripts themselves. Without this an --apply updated the containers but left STALE binaries in
+    # /usr/local/bin while the repo said otherwise — a trap that has cost real debugging time (STATUS.md).
+    media = [f for f in ("bin/normalize-sweep.sh", "bin/trash-purge.sh", "bin/dumpd.py",
+                         "bin/vgstrip.py", "bin/vgdeband.sh") if Path(f).exists()]
+    if media:
+        run(["sudo", "install", "-m", "0755"] + media + ["/usr/local/bin/"])
+    if Path("bin/vgmetrics.sh").exists():
+        run(["sudo", "install", "-m", "0644", "bin/vgmetrics.sh", "/usr/local/bin/"])                    # sourced by the sweep and vgdeband
+    if all(Path(f).exists() for f in ("bin/vanguard-deadman.sh", "systemd/vanguard-deadman.timer")):
+        run(["sudo", "install", "-m", "0755", "bin/vanguard-deadman.sh", "/usr/local/bin/"])
+        run(["sudo", "install", "-m", "0644", "systemd/vanguard-deadman.service", "systemd/vanguard-deadman.timer", "/etc/systemd/system/"])
+        run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-deadman.timer"])
+    # The other half: Kibana writes alerts into Elasticsearch, this takes them to Discord.
+    if all(Path(f).exists() for f in ("bin/vanguard-alerts.sh", "systemd/vanguard-alerts.timer")):
+        run(["sudo", "install", "-m", "0755", "bin/vanguard-alerts.sh", "/usr/local/bin/"])
+        run(["sudo", "install", "-m", "0644", "systemd/vanguard-alerts.service", "systemd/vanguard-alerts.timer", "/etc/systemd/system/"])
+        run(["sudo", "systemctl", "daemon-reload"]); run(["sudo", "systemctl", "enable", "--now", "vanguard-alerts.timer"])
     run(["sudo", "systemctl", "restart", "dumpd"])
     print("== containers")
-    dc = ["docker", "compose"] if (DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0) else ["sudo", "docker", "compose"]
+    dc = compose_cmd()
-    run(dc + ["up", "-d", "--build"])                      # recreates only services whose compose definition changed
+    # --remove-orphans: a service deleted from the compose file should stop existing, not linger. This is how
+    # the eight Victoria/vmalert/Alertmanager/exporter containers go away when the file no longer lists them.
+    run(dc + ["up", "-d", "--build", "--remove-orphans"])   # recreates only services whose definition changed
     run(dc + ["restart", "copyparty"])                      # copyparty re-reads its config on start
     run(dc + ["exec", "caddy", "caddy", "reload", "--config", "/etc/caddy/Caddyfile"])   # zero-downtime
+    post_start_fixups(dc)
     print("  mediamtx: config changes are NOT picked up live on this setup (bind-mounted file). If mediamtx.yml changed,")
     print("            run  docker compose restart mediamtx  between streams (cuts a live recording). Adding/removing")
     print("            streamers or tokens never touches mediamtx.yml, so day-to-day nothing restarts.")
 
-def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_host="", bot_host="", live_host=""):
+def compose_cmd():
-    print(f"\n== domain   : {domain or '(not set)'}")
+    """docker compose, with sudo only if this user cannot reach the daemon."""
+    if DRY or subprocess.call(["docker", "info"], stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) == 0:
+        return ["docker", "compose"]
+    return ["sudo", "docker", "compose"]
+
+def es_security(dc):
+    """Create Elasticsearch's native users after the cluster is up. Idempotent; safe to re-run.
+
+    Security is on, so three accounts have to exist before Kibana or the Beats can work:
+      kibana_system    Kibana's internal account (built in; only its password needs setting)
+      VG_BEATS_USER    what Filebeat and Metricbeat authenticate as
+      VG_DIAG_USER     who every Kibana visitor silently becomes (kibana.yml's anonymous provider)
+    The elastic superuser itself is bootstrapped by the container from ELASTIC_PASSWORD, which only works
+    while the cluster has never had security on. If that did not take, this says exactly what to run rather
+    than guessing: a reset would desync the password the compose healthcheck uses.
+    """
+    env = read_env()
+    elastic_pw = env.get("ES_ELASTIC_PASSWORD", "")
+    if not elastic_pw:
+        print("   elasticsearch: no ES_ELASTIC_PASSWORD in .env (write_env not run?) — skipped"); return
+    if DRY:
+        print("   (dry) elasticsearch: set kibana_system + create "
+              f"{env.get('VG_BEATS_USER')} and {env.get('VG_DIAG_USER')}"); return
+
+    base = "http://127.0.0.1:9200"
+    def call(method, path, body=None):
+        request = urllib.request.Request(base + path, method=method,
+                                         data=None if body is None else json.dumps(body).encode())
+        request.add_header("Content-Type", "application/json")
+        request.add_header("Authorization", "Basic " + base64.b64encode(f"elastic:{elastic_pw}".encode()).decode())
+        with urllib.request.urlopen(request, timeout=20) as response:
+            raw = response.read()
+            return json.loads(raw) if raw else {}
+
+    status = None
+    for _ in range(45):                      # the cluster can take a while on a box busy with x264
+        try:
+            call("GET", "/_cluster/health"); status = "ok"; break
+        except urllib.error.HTTPError as error:
+            status = error.code
+            if error.code == 401: break      # answering, but our password is not the one it bootstrapped
+            time.sleep(4)
+        except Exception:
+            time.sleep(4)
+    if status == 401:
+        print("   elasticsearch: REFUSED the password in .env. The cluster already had security enabled, so")
+        print("                  ELASTIC_PASSWORD no longer bootstraps anything. Set it to the real one:")
+        print("                    docker compose exec elasticsearch bin/elasticsearch-reset-password -u elastic -b -s")
+        print("                  put that value in .env as ES_ELASTIC_PASSWORD, then re-run --apply.")
+        return
+    if status != "ok":
+        print("   elasticsearch: not answering yet — re-run ./configure.py --apply once it is up"); return
+
+    try:
+        # If security is somehow still off, every call below 400s with "no handler found for uri". Check once
+        # and say so plainly, rather than reporting a confusing failure on the first password write.
+        call("POST", "/_security/user/_has_privileges", {"cluster": ["monitor"]})
+    except urllib.error.HTTPError as error:
+        if error.code == 400:
+            print("   elasticsearch: the security API is absent — this cluster is still running with")
+            print("                  xpack.security.enabled=false. Recreate it:  docker compose up -d elasticsearch")
+            return
+    try:
+        call("POST", "/_security/user/kibana_system/_password", {"password": env["ES_KIBANA_PASSWORD"]})
+        print("   elasticsearch: kibana_system password set")
+        for user, password, roles in (
+            (env["VG_BEATS_USER"], env["VG_BEATS_PASSWORD"], ["superuser"]),
+            (env["VG_DIAG_USER"], env["VG_DIAG_PASSWORD"],
+             [r.strip() for r in env.get("DIAG_KIBANA_ROLES", "superuser").split(",") if r.strip()]),
+        ):
+            call("PUT", f"/_security/user/{user}", {"password": password, "roles": roles,
+                                                   "full_name": "Vanguard service account"})
+            print(f"   elasticsearch: user {user} -> {', '.join(roles)}")
+    except urllib.error.HTTPError as error:
+        print(f"   elasticsearch: user setup failed ({error.code}) — {error.read()[:200].decode(errors='replace')}")
+        return
+
+    # The alert bus. Kibana's rules write here through the .index connector (the only outward connector a
+    # basic licence allows), bin/vanguard-alerts.sh delivers from here to Discord, and the deadman watches
+    # for the heartbeat document. Pre-created with an explicit mapping so "level" and "rule" are keywords
+    # and therefore aggregatable, rather than whatever dynamic mapping the first document implies.
+    alert_index = env.get("ALERT_INDEX", "vanguard-alerts")
+    try:
+        call("PUT", f"/{alert_index}", {
+            "settings": {"number_of_replicas": 0},
+            "mappings": {"properties": {
+                "@timestamp": {"type": "date"},
+                "kind": {"type": "keyword"},
+                "rule": {"type": "keyword"},
+                "level": {"type": "keyword"},
+                "service": {"type": "keyword"},
+                "message": {"type": "text", "fields": {"keyword": {"type": "keyword", "ignore_above": 1024}}},
+            }}})
+        print(f"   elasticsearch: alert index {alert_index} created")
+    except urllib.error.HTTPError as error:
+        body = error.read().decode(errors="replace")
+        print(f"   elasticsearch: alert index {alert_index} already there"
+              if "resource_already_exists" in body else f"   elasticsearch: alert index failed — {body[:160]}")
+
+    # Retention. The Beats create their own ILM policies with a rollover but NO delete phase, so indices
+    # grow until the disk watermark turns them read-only — which the deadman reports but cannot undo. Inject
+    # a delete phase instead of writing a whole policy, so whatever the Beats chose for rollover is kept.
+    keep_days = env.get("LOG_RETENTION_DAYS", "30")
+    for policy in ("filebeat", "metricbeat"):
+        try:
+            current = call("GET", f"/_ilm/policy/{policy}")[policy]["policy"]
+        except (urllib.error.HTTPError, KeyError):
+            print(f"   elasticsearch: ILM policy {policy} not created yet — re-run --apply after the beat has started")
+            continue
+        phases = current.setdefault("phases", {})
+        if phases.get("delete", {}).get("min_age") == f"{keep_days}d":
+            print(f"   elasticsearch: {policy} already deletes after {keep_days}d")
+            continue
+        phases["delete"] = {"min_age": f"{keep_days}d", "actions": {"delete": {}}}
+        call("PUT", f"/_ilm/policy/{policy}", {"policy": current})
+        print(f"   elasticsearch: {policy} indices now deleted after {keep_days}d")
+
+    # The deadman runs on the host as root and must authenticate too. netrc, not -u, so the password never
+    # shows up in anyone's process list.
+    Path("es-netrc").write_text(f"machine 127.0.0.1 login elastic password {elastic_pw}\n")
+    run(["sudo", "install", "-m", "0600", "-o", "root", "-g", "root", "es-netrc", "/etc/vanguard/es-netrc"])
+    Path("es-netrc").unlink(missing_ok=True)
+    # Kibana and the Beats were started before those accounts existed, so they are sitting in a retry loop.
+    run(dc + ["restart", "kibana", "filebeat", "metricbeat"])
+    print("   elasticsearch: netrc written for the deadman; kibana + beats restarted")
+
+def post_start_fixups(dc):
+    """Idempotent first-run state that otherwise needs clicking: Syncthing GUI auth off, Gitea admin user, Jellyfin plugin."""
+    env = read_env()
+    print("== post-start fixups (idempotent)")
+    es_security(dc)              # before anything else: Kibana and the Beats cannot authenticate until this runs
+    # Syncthing: no GUI login (the Authentik gate is the login); needs insecureAdminAccess since Caddy is not "localhost" to it
+    cfg = Path("/srv/media/.syncthing/config.xml")
+    if not DRY and cfg.exists():
+        import time as _t
+        for _ in range(20):
+            if "<gui" in cfg.read_text(): break
+            _t.sleep(2)
+        x = cfg.read_text()
+        if "<gui" in x and ('insecureAdminAccess="true"' not in x or "<user>" in x or "<password>" in x):
+            x = re.sub(r"^\s*<user>.*</user>\s*$\n?", "", x, flags=re.M); x = re.sub(r"^\s*<password>.*</password>\s*$\n?", "", x, flags=re.M)
+            x = re.sub(r'<gui ([^>]*?)insecureAdminAccess="false"', r'<gui \1insecureAdminAccess="true"', x)
+            if 'insecureAdminAccess=' not in x: x = x.replace("<gui ", '<gui insecureAdminAccess="true" ', 1)
+            subprocess.run(["sudo", "tee", str(cfg)], input=x, text=True, capture_output=True)
+            run(dc + ["restart", "syncthing"]); print("   syncthing: GUI auth disabled, insecureAdminAccess on")
+        else: print("   syncthing: ok")
+    elif DRY: print("   (dry) syncthing config patch")
+    # Gitea: create the admin user once (INSTALL_LOCK skips the install page)
+    if not DRY:
+        import time as _t
+        for _ in range(30):
+            r = subprocess.run(dc + ["exec", "-T", "-u", "1000", "gitea", "gitea", "admin", "user", "list", "--config", "/data/gitea/conf/app.ini"], capture_output=True, text=True)
+            if r.returncode == 0: break
+            _t.sleep(3)
+        if r.returncode == 0 and env.get("GITEA_ADMIN_USER") and env["GITEA_ADMIN_USER"] not in r.stdout:
+            subprocess.run(dc + ["exec", "-T", "-u", "1000", "gitea", "gitea", "admin", "user", "create", "--config", "/data/gitea/conf/app.ini", "--admin",
+                            "--username", env["GITEA_ADMIN_USER"], "--password", env["GITEA_ADMIN_PASSWORD"], "--email", f"{env['GITEA_ADMIN_USER']}@{env.get('MAIN_HOST','localhost')}", "--must-change-password=false"], capture_output=True, text=True)
+            print(f"   gitea: admin user {env['GITEA_ADMIN_USER']} created (password in .env)")
+        else: print("   gitea: ok" if r.returncode == 0 else "   gitea: not ready (install page or db?) — run ./configure.py --apply again later")
+    else: print("   (dry) gitea admin user")
+    # Kibana first-run state: the alert connector, the heartbeat rule and the data view. Without this a
+    # fresh install has no way to deliver an alert and the deadman alarms about its own missing heartbeat.
+    # It needs an API key because Basic is not an accepted scheme for this Kibana (anonymous is the only
+    # enabled provider), and it waits for Kibana itself, which es_security restarted a moment ago.
+    setup = Path("observability/kibana-setup.py")
+    if setup.exists() and env.get("ES_ELASTIC_PASSWORD"):
+        if DRY:
+            print("   (dry) kibana: connector, heartbeat rule, data view")
+        else:
+            try:
+                request = urllib.request.Request(
+                    "http://127.0.0.1:9200/_security/api_key", method="POST",
+                    data=json.dumps({"name": "vanguard-setup", "expiration": "10m"}).encode())
+                request.add_header("Content-Type", "application/json")
+                request.add_header("Authorization", "Basic " + base64.b64encode(
+                    f"elastic:{env['ES_ELASTIC_PASSWORD']}".encode()).decode())
+                with urllib.request.urlopen(request, timeout=20) as response:
+                    api_key = json.load(response)["encoded"]
+                result = subprocess.run([sys.executable, str(setup)], text=True, capture_output=True,
+                                        env={**os.environ, "KEY": api_key,
+                                             "ALERT_INDEX": env.get("ALERT_INDEX", "vanguard-alerts")})
+                for line in (result.stdout + result.stderr).strip().splitlines():
+                    print("   kibana:", line)
+            except Exception as error:
+                print(f"   kibana: first-run setup skipped ({error}) — run it by hand, see its docstring")
+
+    # Jellyfin plugin from the committed zip
+    if Path("installJellyfinPlugin.sh").exists():
+        if DRY: print("   (dry) installJellyfinPlugin.sh")
+        else: subprocess.run(["bash", "installJellyfinPlugin.sh"], capture_output=True, text=True); print("   jellyfin plugin: ensured")
+
+def show(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_host="", bot_host="", live_host="", files_host="", sync_host="", clips_host="", diag_host=""):
+    print(f"\n== gateway  : https://{domain or 'HOST'}/   (/whoami, /logout)")
+    print(f"== files    : https://{files_host}/  (copyparty; editors / media-admins)" if files_host else "== files    : (not set — run --full)")
+    print(f"== clips    : https://{clips_host}/  (self-service clip intake, any member)" if clips_host else "== clips    : (not set — run --full)")
+    print(f"== sync     : https://{sync_host}/  (Syncthing UI, media-admins)" if sync_host else "== sync     : (not set — run --full)")
     _e = read_env()
     if _e.get("APEX_HOST") and _e.get("APEX_REDIRECT"): print(f"== apex     : https://{_e['APEX_HOST']}/ -> {_e['APEX_REDIRECT']}")
-    print(f"== live     : https://{live_host}/  (behind Authentik forward-auth; deploy live/dist -> /srv/live; WebRTC needs 8189 open)" if live_host else "== live     : (not set — run --full)")
+    print(f"== live     : https://{live_host}/  (portal; /dock/ for OBS; WebRTC needs 8189 open)" if live_host else "== live     : (not set — run --full)")
+    alert_hook = read_env().get("ALERT_WEBHOOK_URL", "")
+    print("== alerts   : " + (f"Discord webhook set ({alert_hook[:38]}…); vanguard-alerts.timer and the deadman post there" if alert_hook
+          else "NO WEBHOOK — Kibana still records alerts in the vanguard-alerts index, but nothing delivers them: "
+               "vanguard-alerts.sh and the deadman write to the journal and exit non-zero. Set ALERT_WEBHOOK_URL in .env."))
+    print(f"== diag     : https://{diag_host}/  (Kibana: logs, dashboards, alerting; group '{read_env().get('DIAG_REQUIRE_GROUP','media-admins')}')" if diag_host else "== diag     : (not set — run --full)")
     print(f"== bot web  : https://{bot_host}/  (Authentik forward-auth; Caddy requires group '{read_env().get('BOT_REQUIRE_GROUP','bot-users') or 'any'}')" if bot_host else "== bot web  : (not set — run --full)")
-    print(f"== gitea    : https://{git_host}/  ssh port 2222  (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea    : (not set — run --full)")
+    print(f"== gitea    : https://{git_host}/  ssh port 2222  admin {read_env().get('GITEA_ADMIN_USER','vanguard')} / {read_env().get('GITEA_ADMIN_PASSWORD','?')}  (runner: ./registerGiteaRunner.sh)" if git_host else "== gitea    : (not set — run --full)")
-    print(f"== authentik: https://{auth_host}/   (first run: https://{auth_host}/if/flow/initial-setup/)" if auth_host else "== authentik: (not set — run --full)")
+    print(f"== authentik: https://{auth_host}/   akadmin password = AUTHENTIK_BOOTSTRAP_PASSWORD in .env (set on first start; change it in the UI)" if auth_host else "== authentik: (not set — run --full)")
     print(f"== jellyfin : https://{watch_host}/" if watch_host else "== jellyfin : (not set — run --full)")
     env = read_env()
     tok = env.get("DISCORD_BOT_TOKEN", "")
     print(f"== discord  : guild {env.get('DISCORD_GUILD_ID') or '(not set)'}; bot token {'set (' + tok[:6] + '…)' if tok else '(not set)'}")
     print(f"== mariadb  : {domain or 'HOST'}:3306 (public, root local-only)  root pass {env.get('MARIADB_ROOT_PASSWORD','(run apply)')}")
+    print(f"     bot ci : Gitea -> VanguardBot -> Settings -> Actions -> Secrets: BOT_DEPLOY_SECRET={env.get('BOT_DEPLOY_SECRET','(run apply)')}")
+    print(f"              pushes to main install themselves (BOT_DEPLOY_AUTO={env.get('BOT_DEPLOY_AUTO','1')}); by hand: ./deployBotBundle.sh <bundle>")
+    print(f"              CI may set these .env keys (BOT_ENV_KEYS): {env.get('BOT_ENV_KEYS','DISCORD_BOT_TOKEN,DISCORD_GUILD_ID')}")
     print(f"     bot    : db {env.get('BOT_DB_NAME','vanguardbot')}  user {env.get('BOT_DB_USER','bot')}  pass {env.get('BOT_DB_PASSWORD','(run apply)')}")
     print(f"     gitea  : db gitea  user gitea  pass {env.get('GITEA_DB_PASSWORD','(run apply)')}   (created on MariaDB first init, else ./createGiteaDb.sh)")
     print("== access   : by Authentik group — editors (read streams/clips, write finals), media-admins (everything), admins (dump anyone)")
     print("              Discord login only — no WebDAV; downloads via the web UI")
     print("== streamers: anyone in the Authentik 'streamers' group (Discord role). No passwords, no keys to hand out.")
-    print(f"    OBS: add the dock https://{domain or 'HOST'}/dock/ in OBS, log in with Discord, Go Live — it mints a 12 h token")
+    print(f"    OBS: add the dock https://{live_host or 'LIVE'}/dock/ in OBS, log in with Discord, Go Live — it mints a 12 h token")
     print(f"    and sets the stream key itself. Manual setups: dock → 'Manual OBS setup' (rtmp://{domain or 'HOST'}:1935/ + key).")
-    print(f"\n  editors log in with Discord at https://{domain or 'HOST'}/")
+    print(f"\n  editors: https://{files_host or 'FILES'}/   everyone: https://{domain or 'HOST'}/")
-    print(f"  verify: curl -sI https://{domain or 'HOST'}/dock/ | head -1      (expect 200)")
+    print(f"  verify: curl -s -o /dev/null -w '%{{http_code}}\\n' https://{domain or 'HOST'}/   (expect 302 → Authentik)\n")
-    print(f"          curl -s -o /dev/null -w '%{{http_code}}\\n' https://{domain or 'HOST'}/dump   (expect 401)\n")
 
 # ------------------------------------------------------------------ modes
 def manage_users(kind, users):
@@ -541,12 +869,16 @@ def manage_users(kind, users):
             else:
                 users[n] = ask_pw(n, users[n])
 
-def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_host="", bot_host="", live_host=""):
+def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_host="", bot_host="", live_host="", files_host="", sync_host="", clips_host="", diag_host=""):
     print("== FULL SETUP (enter keeps the shown default)\n")
     while True:
-        domain = ask("public hostname (DNS A record must point here)", domain)
+        domain = ask("main hostname = the gateway page (DNS A record must point here)", domain)
         if domain and "." in domain and " " not in domain: break
         print("  need a hostname like media.example.com")
+    files_host = ask("Files (copyparty) hostname (CNAME to the above)", files_host or default_sub(domain, "files"))
+    clips_host = ask("Clips self-service hostname (CNAME to the above)", clips_host or default_sub(domain, "clips"))
+    sync_host  = ask("Syncthing UI hostname (CNAME to the above)",     sync_host or default_sub(domain, "sync"))
+    diag_host  = ask("Diagnostics hostname: logs, metrics, alerts (CNAME to the above)", diag_host or default_sub(domain, "diag"))
     auth_host  = ask("Authentik hostname (CNAME to the above)", auth_host or default_sub(domain, "auth"))
     watch_host = ask("Jellyfin hostname (CNAME to the above)",  watch_host or default_sub(domain, "watch"))
     git_host   = ask("Gitea hostname (CNAME to the above)",     git_host or default_sub(domain, "git"))
@@ -554,10 +886,13 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
     live_host  = ask("Live portal hostname (CNAME to the above)", live_host or default_sub(domain, "live"))
     env0 = read_env(); parent = default_sub(domain, "x")[2:] if domain else ""
     apex = ask("Bare domain to serve too (A record must point here; blank = skip)", env0.get("APEX_HOST", parent))
-    apex_to = ask("  bare domain redirects to", env0.get("APEX_REDIRECT") or (f"https://{watch_host}/" if watch_host else "")) if apex else ""
+    apex_to = ask("  bare domain redirects to", env0.get("APEX_REDIRECT") or f"https://{domain}/") if apex else ""
     print("\n-- Discord bot (optional; the Bot added to the Authentik Discord application). Enter keeps the current value.")
     discord_token, discord_guild = ask_discord()
     live_webhook = ask("  Discord webhook URL for 'is live' notices (optional)", read_env().get("LIVE_WEBHOOK_URL", ""))
+    env0 = read_env()
+    alert_webhook = ask("Discord webhook for ALERTS and the deadman (blank = log only, no Discord)",
+                        env0.get("ALERT_WEBHOOK_URL", ""))
     print("\n-- streamers (OBS publishers; each gets streams/<name> and clips/<name>)")
     for u in list(streamers):
         k = ask(f"  {u}: [k]eep  [r]emove  [p]assword reset", "k").lower()[:1]
@@ -581,7 +916,10 @@ def full(domain, admin_pw, streamers, editors, auth_host="", watch_host="", git_
     Path(".env").write_text("".join(f"{k}={v}\n" for k, v in e.items())) if Path(".env").exists() else None
     if not Path(".env").exists():
         Path(".env").write_text(f"APEX_HOST={apex}\nAPEX_REDIRECT={apex_to}\n"); os.chmod(".env", 0o600)
-    return domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook
+    if alert_webhook != env0.get("ALERT_WEBHOOK_URL", ""):
+        env = read_env(); env["ALERT_WEBHOOK_URL"] = alert_webhook
+        Path(".env").write_text("".join(f"{k}={v}\n" for k, v in env.items())); os.chmod(".env", 0o600)
+    return domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook, files_host, sync_host, clips_host, diag_host
 
 def ask_discord():
     env = read_env()
@@ -608,23 +946,40 @@ def main():
     mode = sys.argv[1] if len(sys.argv) > 1 else ""
     hosts = read_hosts()
     domain, auth_host, watch_host, git_host, bot_host, live_host = hosts.get("main", ""), hosts.get("auth", ""), hosts.get("watch", ""), hosts.get("git", ""), hosts.get("bot", ""), hosts.get("live", "")
+    files_host, sync_host, clips_host = hosts.get("files", ""), hosts.get("sync", ""), hosts.get("clips", "")
+    diag_host = hosts.get("diag", "")
     streamers, accounts = read_streamers(), read_accounts()
     admin_pw = accounts.pop("admin", "")
     editors = accounts
+    if mode == "--es-security":
+        # Just the Elasticsearch user bootstrap. Worth having on its own: if it is skipped or fails during an
+        # --apply, Kibana reports "unavailable", Metricbeat exits on Kibana's 503 and Filebeat sits up but
+        # publishes nothing — three confusing symptoms with one cause.
+        es_security(compose_cmd()); return
     if mode == "--show":
-        show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host); return
+        show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); return
-    if mode == "--hosts":
+    if mode == "--apply":
-        if len(sys.argv) not in (4, 5, 6, 7): print("usage: configure.py --hosts AUTH_HOST WATCH_HOST [GIT_HOST [BOT_HOST [LIVE_HOST]]]"); sys.exit(2)
         if not domain: print("domain not set yet — run ./configure.py --full first"); sys.exit(1)
-        auth_host, watch_host = sys.argv[2].strip(), sys.argv[3].strip()
+        apply(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host=git_host, bot_host=bot_host, live_host=live_host, files_host=files_host, sync_host=sync_host, clips_host=clips_host, diag_host=diag_host)
-        if len(sys.argv) >= 5: git_host = sys.argv[4].strip()
+        show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); return
-        if len(sys.argv) >= 6: bot_host = sys.argv[5].strip()
+    if mode == "--hosts":
-        if len(sys.argv) == 7: live_host = sys.argv[6].strip()
+        if not 4 <= len(sys.argv) <= 11:
-        apply(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host=git_host, bot_host=bot_host, live_host=live_host)
+            print("usage: configure.py --hosts AUTH WATCH [GIT [BOT [LIVE [FILES [SYNC [CLIPS [DIAG]]]]]]]")
-        show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host); return
+            print('  Pass "" to leave one unset. This is the ONLY non-interactive way to introduce a NEW host:')
+            print("  --apply reads the hostnames back from the Caddyfile, so a role that is not in there yet")
+            print("  cannot be written by it. --full asks for them all instead.")
+            sys.exit(2)
+        if not domain: print("domain not set yet — run ./configure.py --full first"); sys.exit(1)
+        a = [x.strip() for x in sys.argv[2:]] + [None] * 9
+        auth_host, watch_host = a[0], a[1]
+        git_host, bot_host, live_host = (a[2] if a[2] is not None else git_host), (a[3] if a[3] is not None else bot_host), (a[4] if a[4] is not None else live_host)
+        files_host, sync_host, clips_host = (a[5] if a[5] is not None else files_host), (a[6] if a[6] is not None else sync_host), (a[7] if a[7] is not None else clips_host)
+        diag_host = a[8] if a[8] is not None else diag_host
+        apply(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host=git_host, bot_host=bot_host, live_host=live_host, files_host=files_host, sync_host=sync_host, clips_host=clips_host, diag_host=diag_host)
+        show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host); return
     discord_token = discord_guild = live_webhook = None
     if mode == "--full":
-        domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook = full(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host)
+        domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook, files_host, sync_host, clips_host, diag_host = full(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host)
         if not yes("\napply now (write configs, restart services)?"): print("nothing written"); return
     elif mode == "":
         r = normal(domain, admin_pw, streamers, editors)
@@ -632,8 +987,8 @@ def main():
         discord_token, discord_guild = r
     else:
         print(__doc__); sys.exit(2)
-    apply(domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook)
+    apply(domain, admin_pw, streamers, editors, auth_host, watch_host, discord_token, discord_guild, git_host, bot_host, live_host, live_webhook, files_host, sync_host, clips_host, diag_host)
-    show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host)
+    show(domain, admin_pw, streamers, editors, auth_host, watch_host, git_host, bot_host, live_host, files_host, sync_host, clips_host, diag_host)
 
 if __name__ == "__main__":
     try: main()

deploy.sh

@@ -0,0 +1,115 @@
+#!/bin/bash
+# Minimal-impact deployer for the server checkout. Run as root (sudo) — installed as /usr/local/bin/vanguard-deploy.
+#   vanguard-deploy pull            fetch + fast-forward DEPLOY_BRANCH, then apply only what changed
+#   vanguard-deploy apply [OLDREV]  apply changes between OLDREV (default: last deployed rev) and HEAD
+#   vanguard-deploy rollback        check out the previously deployed rev and apply the reverse diff
+#   vanguard-deploy pending         restart MediaMTX if a restart is pending and no stream is live (timer runs this)
+#   vanguard-deploy status
+# Env (from /etc/vanguard/deploy.env): DEPLOY_REPO_DIR, DEPLOY_BRANCH, DEPLOY_NOTIFY_WEBHOOK_URL. VG_DRY=1 prints only.
+set -uo pipefail
+ENVF=/etc/vanguard/deploy.env; [[ -f $ENVF ]] && set -a && . "$ENVF" && set +a
+REPO="${DEPLOY_REPO_DIR:-/home/debian/vanguard-stream}"; BRANCH="${DEPLOY_BRANCH:-main}"
+STATE=/var/lib/vanguard; mkdir -p "$STATE" 2>/dev/null || true
+LASTF="$STATE/deployed-rev"; PREVF="$STATE/previous-rev"; PENDING="$STATE/mediamtx-restart-pending"
+DRY="${VG_DRY:-0}"; OWNER=$(stat -c %U "$REPO" 2>/dev/null || echo debian)
+log() { printf 'ts=%s level=info component=deploy msg="%s"\n' "$(date -Is)" "${*//\"/\\\"}"; }
+run() { log "\$ $*"; [[ $DRY == 1 ]] || "$@"; }
+asowner() { log "\$ (as $OWNER) $*"; [[ $DRY == 1 ]] || sudo -u "$OWNER" -H "$@"; }
+notify() {
+  [[ -n ${DEPLOY_NOTIFY_WEBHOOK_URL:-} && $DRY != 1 ]] || return 0
+  curl -s -m 10 -H 'Content-Type: application/json' -d "$(python3 -c 'import json,sys; print(json.dumps({"content": sys.argv[1][:1900]}))' "$1")" "$DEPLOY_NOTIFY_WEBHOOK_URL" >/dev/null || true
+}
+cd "$REPO" || { log "repo dir $REPO missing"; exit 1; }
+# "pending" (the 5-min timer) needs no git; every other mode does
+[[ ${1:-} == pending ]] || git rev-parse --is-inside-work-tree >/dev/null 2>&1 || { log "$REPO is not a git checkout (run switchToClone.sh once)"; exit 1; }
+
+live_streams() { curl -s -m 3 'http://127.0.0.1:9997/v3/paths/list' 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(sum(1 for p in d.get("items",[]) if p.get("ready")))' 2>/dev/null || echo 0; }
+
+restart_mediamtx_or_defer() {
+  local n; n=$(live_streams)
+  if [[ $n == 0 ]]; then
+    run docker compose restart mediamtx; rm -f "$PENDING"; log "mediamtx restarted"
+  else
+    [[ $DRY == 1 ]] || touch "$PENDING"; log "mediamtx restart DEFERRED: $n live stream(s); deploy-pending.timer retries every 5 min"
+    notify "⏳ deploy: MediaMTX restart deferred ($n live stream(s)); will restart when idle"
+  fi
+}
+
+apply_changes() {
+  local old="$1" new="$2"
+  local changed; changed=$(git diff --name-only "$old" "$new" 2>/dev/null || git ls-files)
+  [[ -n $changed ]] || { log "nothing changed between $old and $new"; return 0; }
+  log "changed files:"; echo "$changed" | sed 's/^/    /'
+  has() { echo "$changed" | grep -qE "$1"; }
+
+  # 0. static checks on the new tree
+  if [[ -x check.sh ]]; then asowner bash check.sh >/tmp/vg-check.log 2>&1 || { log "check.sh FAILED — aborting"; tail -20 /tmp/vg-check.log; notify "❌ deploy $new: check.sh failed"; return 1; }; fi
+
+  # 1. host scripts + units
+  if has '^bin/'; then run install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/; fi
+  if has '^systemd/'; then run install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/; run systemctl daemon-reload; fi
+  if has '^deploy\.sh$'; then run install -m 0755 deploy.sh /usr/local/bin/vanguard-deploy; fi
+  if has '^deployBotBundle\.sh$'; then run install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot; fi
+  if has '^firewall/nftables\.conf$'; then run install -m 0644 firewall/nftables.conf /etc/nftables.conf; run nft -f /etc/nftables.conf; fi
+
+  # 2. regenerate configs (Caddy reload, copyparty restart) — configure.py decides; MediaMTX is handled below
+  if has '^configure\.py$|^syncthing/stignore$|^dock/|^clips/|^about/'; then
+    local before_mtx; before_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
+    if [[ $DRY == 1 ]]; then log "\$ (as $OWNER) VG_DRY=1 python3 configure.py --apply"; else sudo -u "$OWNER" -H python3 configure.py --apply >/tmp/vg-configure.log 2>&1 || { log "configure --apply FAILED"; tail -20 /tmp/vg-configure.log; return 1; }; fi
+    local after_mtx; after_mtx=$(md5sum mediamtx.yml 2>/dev/null | cut -d' ' -f1)
+    [[ $before_mtx == "$after_mtx" ]] || { log "mediamtx.yml changed"; restart_mediamtx_or_defer; }
+  fi
+  if has '^bin/dumpd\.py$|^systemd/dumpd\.service$'; then run systemctl restart dumpd; fi
+
+  # 3. compose: recreate changed services, but never MediaMTX while live
+  if has '^docker-compose\.yml$|^bot/Dockerfile$|^bot/entrypoint\.sh$'; then
+    local mtx_changed=0
+    git diff "$old" "$new" -- docker-compose.yml | grep -qE '^[+-]' && git diff -U0 "$old" "$new" -- docker-compose.yml | grep -qiE '^[+-].*(mediamtx|8189|8890|1935)' && mtx_changed=1
+    if [[ $mtx_changed == 1 && $(live_streams) != 0 ]]; then
+      log "compose touches mediamtx and streams are live: recreating everything except mediamtx"
+      run docker compose up -d --build --no-deps $(docker compose config --services | grep -v '^mediamtx$' | tr '\n' ' ')
+      [[ $DRY == 1 ]] || touch "$PENDING"; notify "⏳ deploy: MediaMTX recreate deferred (live streams)"
+    else
+      run docker compose up -d --build
+    fi
+  fi
+
+  # 4. Jellyfin plugin (committed zip)
+  if has '^jellyfin-plugin-vanguard/dist/.*\.zip$'; then run bash installJellyfinPlugin.sh; fi
+
+  # 5. live portal
+  if has '^live/'; then
+    if command -v npm >/dev/null; then
+      asowner bash -c 'cd live && npm install --no-audit --no-fund --loglevel=error && npm run build' && { run install -d -o "$OWNER" /srv/live; run rsync -a --delete live/dist/ /srv/live/; }
+    else log "npm not installed on the server: live/ changed but not built (apt-get install nodejs npm)"; fi
+  fi
+  log "applied $old..$new"
+}
+
+case "${1:-}" in
+  pull)
+    old=$(git rev-parse HEAD)
+    asowner git fetch --quiet origin "$BRANCH" || { log "fetch failed"; exit 1; }
+    asowner git merge --ff-only "origin/$BRANCH" || { log "fast-forward failed (local changes?)"; exit 1; }
+    new=$(git rev-parse HEAD)
+    [[ $old == "$new" ]] && { log "already at $new"; exit 0; }
+    if apply_changes "$old" "$new"; then
+      [[ $DRY == 1 ]] || { cp -f "$LASTF" "$PREVF" 2>/dev/null; echo "$new" > "$LASTF"; }
+      notify "✅ deploy $(git log -1 --format='%h %s' "$new")"
+    else notify "❌ deploy $new failed — see journalctl -u dumpd"; exit 1; fi ;;
+  apply)
+    old="${2:-$(cat "$LASTF" 2>/dev/null || git rev-parse HEAD~1)}"; new=$(git rev-parse HEAD)
+    apply_changes "$old" "$new" && { [[ $DRY == 1 ]] || { cp -f "$LASTF" "$PREVF" 2>/dev/null; echo "$new" > "$LASTF"; }; } ;;
+  rollback)
+    prev=$(cat "$PREVF" 2>/dev/null) || { log "no previous rev recorded"; exit 1; }
+    cur=$(git rev-parse HEAD); log "rolling back $cur -> $prev"
+    asowner git checkout --quiet "$prev" && apply_changes "$cur" "$prev" && { [[ $DRY == 1 ]] || { echo "$cur" > "$PREVF"; echo "$prev" > "$LASTF"; }; notify "↩️ rolled back to $prev"; } ;;
+  pending)
+    [[ -f $PENDING ]] || exit 0
+    restart_mediamtx_or_defer ;;
+  status)
+    echo "repo:     $REPO ($(git rev-parse --abbrev-ref HEAD) @ $(git rev-parse --short HEAD))"
+    echo "deployed: $(cat "$LASTF" 2>/dev/null || echo '-')   previous: $(cat "$PREVF" 2>/dev/null || echo '-')"
+    echo "auto:     DEPLOY_AUTO=${DEPLOY_AUTO:-0}   branch: $BRANCH   pending mediamtx restart: $([[ -f $PENDING ]] && echo yes || echo no)   live streams: $(live_streams)" ;;
+  *) sed -n '2,9p' "$0"; exit 2 ;;
+esac

deployBotBundle.sh

@@ -0,0 +1,137 @@
+#!/bin/bash
+# Install a Vanguard Bot release bundle into the place the discord-bot container expects it.
+#
+# The bundle comes from the bot repo's CI (workflow "ci", job "bundle", artifact "vanguard-bot-bundle"):
+# one tar.gz holding main.py, app/, website/, version.json, BUILD.json and vendor/ — the bot's own
+# dependencies. Nothing is installed here and the server needs no package index.
+#
+#   ./deployBotBundle.sh vanguard-bot-bundle.tar.gz    install that file
+#   ./deployBotBundle.sh --latest                      download the newest main build from Gitea first
+#   ./deployBotBundle.sh --rollback                    put the previous bundle back
+#   ./deployBotBundle.sh --status                      what is installed right now
+#
+# --latest needs a Gitea token with read access to the repo, in .env as BOT_REPO_TOKEN (or $BOT_REPO_TOKEN).
+# Layout:  $BOT_DIR/app = live (mounted at /app)   $BOT_DIR/previous = the one before   $BOT_DIR/.incoming = staging
+set -euo pipefail
+# Runs from the bundle root or as /usr/local/bin/vanguard-deploy-bot (dumpd calls it that way): find the stack.
+STACK="${VANGUARD_DIR:-}"
+if [[ -z $STACK && -f /etc/vanguard/deploy.env ]]; then
+  STACK=$(grep -E '^DEPLOY_REPO_DIR=' /etc/vanguard/deploy.env | head -1 | cut -d= -f2- || true)
+fi
+if [[ -z $STACK || ! -f $STACK/docker-compose.yml ]]; then STACK="$(cd "$(dirname "$0")" && pwd)"; fi
+[[ -f $STACK/docker-compose.yml ]] || { echo "error: cannot find the stack (set VANGUARD_DIR)" >&2; exit 1; }
+cd "$STACK"
+BOT_DIR="${BOT_DIR:-/srv/bot}"
+REPO="${BOT_REPO:-VanguardOrg/VanguardBot}"
+BRANCH="${BOT_BRANCH:-main}"
+fromenv() { [[ -f .env ]] && grep -E "^$1=" .env | head -1 | cut -d= -f2- || true; }
+GIT_HOST="$(fromenv GIT_HOST)"
+BOT_HOST="$(fromenv BOT_HOST)"
+BOT_REPO_TOKEN="${BOT_REPO_TOKEN:-$(fromenv BOT_REPO_TOKEN)}"
+
+say() { printf '\n== %s\n' "$*"; }
+die() { echo "error: $*" >&2; exit 1; }
+
+unwrap() {   # unwrap <downloaded-file> <destination.tar.gz>  — Gitea hands out artifacts as a zip holding the tar.gz
+  if python3 -c "import zipfile,sys; sys.exit(0 if zipfile.is_zipfile(sys.argv[1]) else 1)" "$1"; then
+    python3 - "$1" "$2" <<'PYZIP'
+import sys,zipfile
+with zipfile.ZipFile(sys.argv[1]) as archive:
+    names=[n for n in archive.namelist() if n.endswith('.tar.gz')] or archive.namelist()
+    if not names: sys.exit('the zip holds no files')
+    open(sys.argv[2],'wb').write(archive.read(names[0]))
+PYZIP
+    echo "  (unpacked $(basename "$1"): a zip from Gitea)"
+  elif [[ $1 != "$2" ]]; then cp -f "$1" "$2"; fi
+}
+
+summary() {  # summary <bundle-root>
+  python3 - "$1" <<'PY'
+import json,sys,pathlib
+root=pathlib.Path(sys.argv[1]); build=root/'BUILD.json'
+if not build.is_file(): print('  (no BUILD.json — not a CI bundle)'); sys.exit(0)
+d=json.loads(build.read_text())
+print(f"  version {d.get('version')}  commit {(d.get('commit') or '?')[:8]}  built {d.get('built_at')}  {d.get('python')} ({d.get('abi')})")
+print(f"  {len(d.get('files',{}))} files, {len(d.get('packages',[]))} packages: {', '.join(d.get('packages',[])[:6])}…")
+PY
+}
+
+case "${1:-}" in
+  --status)
+    say "installed at $BOT_DIR/app"; summary "$BOT_DIR/app"
+    if [[ -d $BOT_DIR/previous ]]; then say "previous"; summary "$BOT_DIR/previous"; fi
+    say "container"; docker ps --filter name=discord-bot --format '  {{.Names}} {{.Status}}'
+    exit 0 ;;
+  --rollback)
+    [[ -d $BOT_DIR/previous ]] || die "no previous bundle in $BOT_DIR/previous"
+    say "rolling back"; summary "$BOT_DIR/previous"
+    docker stop discord-bot >/dev/null
+    rm -rf "$BOT_DIR/.rollback" && mv "$BOT_DIR/app" "$BOT_DIR/.rollback" && mv "$BOT_DIR/previous" "$BOT_DIR/app" && mv "$BOT_DIR/.rollback" "$BOT_DIR/previous"
+    docker start discord-bot >/dev/null; sleep 8; docker logs --tail 12 discord-bot; exit 0 ;;
+  --latest)
+    [[ -n ${BOT_REPO_TOKEN:-} ]] || die "set BOT_REPO_TOKEN in .env (Gitea token with read access to $REPO)"
+    [[ -n $GIT_HOST ]] || die "GIT_HOST not found in .env"
+    TARBALL=$(mktemp /tmp/vanguard-bot-bundle.XXXXXX.tar.gz)
+    say "newest '$BRANCH' bundle from https://$GIT_HOST/$REPO"
+    URL=$(curl -fsS -H "Authorization: token $BOT_REPO_TOKEN" \
+          "https://$GIT_HOST/api/v1/repos/$REPO/actions/artifacts?name=vanguard-bot-bundle" |
+          python3 -c "
+import json,sys
+items=json.load(sys.stdin).get('artifacts') or []
+ok=[a for a in items if not a.get('expired')]
+if not ok: sys.exit('no unexpired vanguard-bot-bundle artifact — check the ci workflow ran')
+print(sorted(ok,key=lambda a:a.get('created_at',''))[-1]['archive_download_url'])")
+    curl -fsSL -H "Authorization: token $BOT_REPO_TOKEN" -o "$TARBALL.zip" "$URL"
+    unwrap "$TARBALL.zip" "$TARBALL"; rm -f "$TARBALL.zip" ;;
+  "" | -h | --help) sed -n '2,20p' "$0"; exit 2 ;;
+  *) [[ -f $1 ]] || die "no such file: $1"
+     TARBALL=$(mktemp /tmp/vanguard-bot-bundle.XXXXXX.tar.gz); unwrap "$1" "$TARBALL" ;;
+esac
+
+say "staging $TARBALL"
+rm -rf "$BOT_DIR/.incoming"; mkdir -p "$BOT_DIR/.incoming"
+tar -xzf "$TARBALL" -C "$BOT_DIR/.incoming"
+for required in main.py app/main.py app/config.py website/static/home.html vendor; do
+  [[ -e "$BOT_DIR/.incoming/$required" ]] || die "bundle is missing $required — is this the CI artifact?"
+done
+summary "$BOT_DIR/.incoming"
+python3 - "$BOT_DIR/.incoming" <<'PY'
+import hashlib,json,pathlib,sys
+root=pathlib.Path(sys.argv[1]); d=json.loads((root/'BUILD.json').read_text())
+bad=[name for name,digest in d.get('files',{}).items()
+     if name!='BUILD.json' and hashlib.sha256((root/name).read_bytes()).hexdigest()!=digest]
+sys.exit(f"  {len(bad)} file(s) do not match BUILD.json: {bad[:3]}" if bad else 0)
+PY
+echo "  checksums match BUILD.json"
+
+say "swapping in and restarting discord-bot"
+docker stop discord-bot >/dev/null 2>&1 || true
+rm -rf "$BOT_DIR/previous"
+if [[ -d $BOT_DIR/app ]]; then mv "$BOT_DIR/app" "$BOT_DIR/previous"; fi
+mv "$BOT_DIR/.incoming" "$BOT_DIR/app"
+chown -R 1000:1000 "$BOT_DIR/app" 2>/dev/null || sudo chown -R 1000:1000 "$BOT_DIR/app"
+docker compose up -d discord-bot >/dev/null
+sleep 10
+
+say "result"
+# Wait for the bot to actually serve before calling this a success: a release that crash-loops must fail the
+# deploy (and the CI job), not report 200. BOT_HEALTH_WAIT=0 skips the wait, e.g. if WEB_ENABLED is ever off.
+wait_s="${BOT_HEALTH_WAIT:-45}"
+code=""; waited=0
+while (( waited < wait_s )); do
+  code=$(curl -s -m 3 -o /dev/null -w '%{http_code}' -H 'X-Authentik-Username: deploy-check' \
+           -H 'X-Authentik-Groups: admins' http://127.0.0.1:8000/health || true)
+  [[ $code =~ ^[2-4][0-9][0-9]$ ]] && break
+  sleep 3; waited=$((waited+3))
+done
+docker ps --filter name=discord-bot --format '  {{.Names}} {{.Status}}'
+echo "  /health -> ${code:-no answer} after ${waited}s   (200 = serving; 401/403 = serving but it refused the check's headers)"
+docker logs --tail 20 discord-bot 2>&1 | sed 's/^/  /'
+echo
+if [[ ! $code =~ ^[2-4][0-9][0-9]$ ]] && (( wait_s > 0 )); then
+  echo "  FAILED: the bot did not serve on 8000 within ${wait_s}s — the release is installed but not running."
+  echo "  previous release is still here: ./deployBotBundle.sh --rollback"
+  exit 1
+fi
+echo "  web UI: https://${BOT_HOST:-BOT_HOST-not-set}/   (Caddy + Authentik; needs the bot-users group)"
+echo "  rollback: ./deployBotBundle.sh --rollback"

deployLive.sh

@@ -2,7 +2,7 @@
 # Build the live portal here and push the static files to /srv/live on the server. Caddy serves them as-is.
 set -euo pipefail
 cd "$(dirname "$0")/live"
-HOST="${1:-debian@ovh1.dumpstervigilantes.rip}"
+HOST="${1:-debian@media.dumpstervigilantes.rip}"   # any name of the box works for ssh
 npm install --no-audit --no-fund --loglevel=error
 npm run build
 ssh "$HOST" 'sudo install -d -o "$(id -u)" -g "$(id -g)" /srv/live && rm -rf /srv/live/*'

docker-compose.yml

@@ -1,5 +1,13 @@
+# Log framework (LOGGING.md): every container keeps rotated json-file logs and carries labels a collector can use as
+# low-cardinality stream labels without parsing: vanguard.service / vanguard.category.
+x-logging: &vglog
+  driver: json-file
+  options: { max-size: "50m", max-file: "5" }
+
 services:
   mediamtx:
+    logging: *vglog
+    labels: { vanguard.service: mediamtx, vanguard.category: stream }
     image: bluenviron/mediamtx:latest-ffmpeg     # alpine variant: has sh + wget for the is-live Discord hook
     container_name: mediamtx
     user: "1000:1000"             # recordings owned by the debian user, so the archive pull can delete them
@@ -15,12 +23,16 @@ services:
       - "127.0.0.1:8888:8888"    # HLS      -> Caddy live.HOST/hls/*
       - "127.0.0.1:8889:8889"    # WebRTC   -> Caddy live.HOST/webrtc/*
       - "127.0.0.1:9997:9997"    # API      -> Caddy live.HOST/api/*
+      - "127.0.0.1:9998:9998"    # Prometheus metrics (per-stream bitrate, readers, drops) -> scraper on the host
     volumes:
       - ./mediamtx.yml:/mediamtx.yml:ro
+      - ./bin/mtx-on-ready.sh:/vanguard/on-ready.sh:ro   # is-live notice + WebRTC audio mirror (runOnReady)
       - /srv/media/streams:/recordings
     restart: unless-stopped
 
   copyparty:
+    logging: *vglog
+    labels: { vanguard.service: copyparty, vanguard.category: media }
     image: copyparty/ac:latest
     container_name: copyparty
     ports:
@@ -32,6 +44,8 @@ services:
     restart: unless-stopped
 
   syncthing:
+    logging: *vglog
+    labels: { vanguard.service: syncthing, vanguard.category: intake }
     image: syncthing/syncthing:latest
     container_name: syncthing
     environment:
@@ -47,6 +61,8 @@ services:
     restart: unless-stopped
 
   caddy:
+    logging: *vglog
+    labels: { vanguard.service: caddy, vanguard.category: auth }
     image: caddy:latest
     container_name: caddy
     depends_on: [copyparty]
@@ -63,6 +79,8 @@ services:
 
   # ---------------------------------------------------------------- identity (Authentik)
   authentik-db:
+    logging: *vglog
+    labels: { vanguard.service: authentik-db, vanguard.category: auth }
     image: docker.io/library/postgres:16-alpine
     container_name: authentik-db
     environment:
@@ -80,6 +98,8 @@ services:
     restart: unless-stopped
 
   authentik-redis:
+    logging: *vglog
+    labels: { vanguard.service: authentik-redis, vanguard.category: auth }
     image: docker.io/library/redis:alpine
     container_name: authentik-redis
     command: --save 60 1 --loglevel warning
@@ -93,11 +113,15 @@ services:
     restart: unless-stopped
 
   authentik-server:
+    logging: *vglog
+    labels: { vanguard.service: authentik-server, vanguard.category: auth }
     image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-2025.8}
     container_name: authentik-server
     command: server
     environment: &authentik_env
       AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
+      AUTHENTIK_BOOTSTRAP_PASSWORD: ${AUTHENTIK_BOOTSTRAP_PASSWORD:-}   # akadmin password on FIRST start (no initial-setup page)
+      AUTHENTIK_BOOTSTRAP_TOKEN: ${AUTHENTIK_BOOTSTRAP_TOKEN:-}         # API token for akadmin, created on first start (blueprints/automation)
       AUTHENTIK_REDIS__HOST: authentik-redis
       AUTHENTIK_POSTGRESQL__HOST: authentik-db
       AUTHENTIK_POSTGRESQL__USER: authentik
@@ -115,6 +139,8 @@ services:
     restart: unless-stopped
 
   authentik-worker:
+    logging: *vglog
+    labels: { vanguard.service: authentik-worker, vanguard.category: auth }
     image: ghcr.io/goauthentik/server:${AUTHENTIK_TAG:-2025.8}
     container_name: authentik-worker
     command: worker
@@ -130,6 +156,8 @@ services:
 
   # ---------------------------------------------------------------- watching (Jellyfin, direct play only)
   jellyfin:
+    logging: *vglog
+    labels: { vanguard.service: jellyfin, vanguard.category: media }
     image: jellyfin/jellyfin:latest
     container_name: jellyfin
     user: "1000:1000"
@@ -147,6 +175,8 @@ services:
 
   # ---------------------------------------------------------------- database (for the Discord bot and friends)
   mariadb:
+    logging: *vglog
+    labels: { vanguard.service: mariadb, vanguard.category: infra }
     image: mariadb:11
     container_name: mariadb
     network_mode: host              # PUBLIC on 3306: host networking so the nftables input rules (db_allow / rate
@@ -169,8 +199,10 @@ services:
       start_period: 30s
     restart: unless-stopped
 
-  # ---------------------------------------------------------------- Discord bot (python; code lives in ./bot/app)
+  # ------------------------------------------- Discord bot (python; its release is installed in /srv/bot/app)
   discord-bot:
+    logging: *vglog
+    labels: { vanguard.service: discord-bot, vanguard.category: bot }
     build: ./bot
     container_name: discord-bot
     user: "1000:1000"
@@ -182,6 +214,11 @@ services:
       DB_NAME: ${BOT_DB_NAME:-vanguardbot}
       DB_USER: ${BOT_DB_USER:-bot}
       DB_PASSWORD: ${BOT_DB_PASSWORD}
+      DB_BACKEND: mariadb
+      DATA_DIR: /data
+      WEB_ALLOWED_GROUPS: ${BOT_WEB_ALLOWED_GROUPS:-bot-users,admins}   # the bot's own gate; Caddy already requires BOT_REQUIRE_GROUP
+      WEB_SERVE_UI: "true"                # serve the bot's own pages at / on this port (same origin as /api)
+      PYTHONPATH: /app/vendor             # the bundle's dependencies; the image installs none
       MEDIA_CLIPS: /media/clips
       WEB_HOST: 0.0.0.0
       WEB_PORT: "8000"
@@ -194,17 +231,35 @@ services:
     ports:
       - "127.0.0.1:8000:8000"             # web UI; public only via Caddy + Authentik forward-auth at https://BOT_HOST/
     volumes:
-      - ./bot/app:/app                    # the bot's source; edit and `docker compose restart discord-bot`
+      - ${BOT_APP_DIR:-/srv/bot/app}:/app  # CI bundle installed by ./deployBotBundle.sh (code + vendor/)
       - /srv/media/clips:/media/clips     # rw: for a clip-submissions feature (writes into clips/<user>/)
       - /srv/appdata/discord-bot:/data    # persistent scratch (sqlite, caches) if the bot wants files
+    # Asks the BOT about itself rather than just proving the port accepts a connection: /health answers 503
+    # unless the Discord gateway is connected AND the database responds, and -f turns that into an unhealthy
+    # container, which Metricbeat's docker healthcheck metricset turns into an alarm. Requires the bot release
+    # that made /health unauthenticated and status-bearing — on older code every check 401s and -f fails,
+    # so ship the bot first. start_period covers command sync and the first roster load.
+    healthcheck:
+      test: ["CMD-SHELL", "curl -fsS -m 5 http://127.0.0.1:8000/health >/dev/null || exit 1"]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 120s
     extra_hosts:
       - "host.docker.internal:host-gateway"
+    networks:
+      - botnet                          # its OWN network: the bot trusts the identity headers Caddy injects, so
+                                        # nothing else must be able to reach :8000 and forge them. Caddy is on the
+                                        # host network and uses the published 127.0.0.1:8000; MariaDB is reached
+                                        # through host.docker.internal. The bot needs no sibling container.
     depends_on:
       mariadb: { condition: service_healthy }
     restart: unless-stopped
 
   # ---------------------------------------------------------------- Gitea + Actions runner
   gitea:
+    logging: *vglog
+    labels: { vanguard.service: gitea, vanguard.category: code }
     image: docker.gitea.com/gitea:1.24
     container_name: gitea
     environment:
@@ -234,6 +289,7 @@ services:
       GITEA__actions__ENABLED: "true"
       GITEA__actions__DEFAULT_ACTIONS_URL: github
       GITEA__webhook__ALLOWED_HOST_LIST: private,loopback,external
+      GITEA__security__INSTALL_LOCK: "true"                  # no install page: db/settings come from env; admin user via configure.py
       GITEA__time__DEFAULT_UI_LOCATION: America/New_York
     ports:
       - "127.0.0.1:3000:3000"
@@ -249,13 +305,18 @@ services:
     restart: unless-stopped
 
   act_runner:
+    logging: *vglog
+    labels: { vanguard.service: act_runner, vanguard.category: deploy }
     image: docker.gitea.com/act_runner:latest
     container_name: act_runner
     environment:
       GITEA_INSTANCE_URL: https://${GIT_HOST:-git.example.com}/
       GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_TOKEN:-}   # set by ./registerGiteaRunner.sh
       GITEA_RUNNER_NAME: vanguard-runner
-      GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,dotnet:docker://mcr.microsoft.com/dotnet/sdk:10.0,python:docker://python:3.12"
+      # JavaScript actions (checkout, upload-artifact) need node INSIDE the job image, so a label used with
+      # them must point at an image that has it. "python" is bare CPython for plain run: steps; "python312"
+      # adds node and is what the Discord bot's CI uses, so its wheels match the python:3.12-slim runtime.
+      GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,dotnet:docker://mcr.microsoft.com/dotnet/sdk:10.0,python:docker://python:3.12,python312:docker://nikolaik/python-nodejs:python3.12-nodejs22"
     volumes:
       - /srv/appdata/act_runner:/data
       - /var/run/docker.sock:/var/run/docker.sock   # jobs run as sibling containers; root-equivalent on the host
@@ -263,7 +324,127 @@ services:
       - gitea
     restart: unless-stopped
 
+
+
+  # ================================================================ observability — Elastic (Phase 13)
+  # One store, one UI, one collector pair. Replaces VictoriaLogs+Vector+VictoriaMetrics+2×vmalert+Alertmanager
+  # +node_exporter+cAdvisor: eight containers for four, and Kibana is the single pane for logs, dashboards and
+  # alerting. Everything binds loopback and is published only through diag.HOST behind the Authentik gate.
+  elasticsearch:
+    logging: *vglog
+    labels: { vanguard.service: elasticsearch, vanguard.category: infra }
+    image: docker.elastic.co/elasticsearch/elasticsearch:9.2.0
+    container_name: elasticsearch
+    environment:
+      # Pinned deliberately. A JVM that sizes its own heap on a 62 GB box would take a quarter of it, on a
+      # machine whose spare CPU and RAM belong to x264. 2 GB is ample for one host's logs and metrics.
+      ES_JAVA_OPTS: -Xms2g -Xmx2g
+      discovery.type: single-node
+      # ON. Kibana's Observability apps (Infrastructure/Metrics) ask Elasticsearch whether the current user
+      # holds a privilege; with security off that API does not exist and they fail on every view. Humans still
+      # never see a password: Kibana signs everyone in as one user (observability/kibana.yml). Caddy +
+      # Authentik + DIAG_REQUIRE_GROUP remains the only real gate.
+      xpack.security.enabled: "true"
+      xpack.security.http.ssl.enabled: "false"      # plain HTTP, published on loopback only
+      xpack.security.transport.ssl.enabled: "false" # single-node: no transport traffic, and the check is skipped
+      xpack.security.enrollment.enabled: "false"
+      xpack.security.autoconfiguration.enabled: "false"
+      # Bootstraps the elastic superuser on first start with security on. configure.py generates it and then
+      # sets up kibana_system and the service users; the error below means .env predates that.
+      ELASTIC_PASSWORD: "${ES_ELASTIC_PASSWORD:?run ./configure.py --apply first — it generates the Elastic passwords}"
+      xpack.ml.enabled: "false"             # not used, and it reserves memory
+      cluster.routing.allocation.disk.watermark.low: 85%
+      cluster.routing.allocation.disk.watermark.high: 90%
+      cluster.routing.allocation.disk.watermark.flood_stage: 95%
+      bootstrap.memory_lock: "true"
+    ulimits:
+      memlock: { soft: -1, hard: -1 }
+      nofile: { soft: 65536, hard: 65536 }
+    ports:
+      - "127.0.0.1:9200:9200"
+    volumes:
+      - /srv/appdata/elasticsearch:/usr/share/elasticsearch/data
+    healthcheck:
+      test: ["CMD-SHELL", "curl -sf -u elastic:$${ELASTIC_PASSWORD} http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=5s || exit 1"]
+      interval: 20s
+      timeout: 10s
+      retries: 12
+      start_period: 90s
+    restart: unless-stopped
+
+  kibana:
+    logging: *vglog
+    labels: { vanguard.service: kibana, vanguard.category: infra }
+    image: docker.elastic.co/kibana/kibana:9.2.0
+    container_name: kibana
+    # Everything non-secret lives in observability/kibana.yml (which explains the auth model); only the
+    # generated secrets arrive as environment variables, which that file interpolates.
+    environment:
+      DIAG_HOST: ${DIAG_HOST:-diag.example.com}
+      KIBANA_ENCRYPTION_KEY: ${KIBANA_ENCRYPTION_KEY}
+      ES_KIBANA_PASSWORD: "${ES_KIBANA_PASSWORD:?run ./configure.py --apply first}"
+      VG_DIAG_USER: ${VG_DIAG_USER:-vanguard_diag}
+      VG_DIAG_PASSWORD: "${VG_DIAG_PASSWORD:?run ./configure.py --apply first}"
+    volumes:
+      - ./observability/kibana.yml:/usr/share/kibana/config/kibana.yml:ro
+    ports:
+      - "127.0.0.1:5601:5601"
+    depends_on:
+      elasticsearch: { condition: service_healthy }
+    restart: unless-stopped
+
+  filebeat:
+    logging: *vglog
+    labels: { vanguard.service: filebeat, vanguard.category: infra }
+    image: docker.elastic.co/beats/filebeat:9.2.0
+    container_name: filebeat
+    environment:                            # Elasticsearch requires auth now; the config interpolates these
+      ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
+      ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
+    user: root
+    command: ["filebeat", "-e", "--strict.perms=false"]
+    volumes:
+      - ./observability/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
+      - /var/lib/docker/containers:/var/lib/docker/containers:ro
+      - /var/run/docker.sock:/var/run/docker.sock     # autodiscover reads container names and our vanguard.* labels
+      - /var/log/journal:/var/log/journal:ro          # our own systemd units
+      - /run/log/journal:/run/log/journal:ro          # ...and the volatile journal, if this host has no persistent one
+      - /etc/machine-id:/etc/machine-id:ro
+      - filebeat_data:/usr/share/filebeat/data
+    depends_on:
+      elasticsearch: { condition: service_healthy }
+    restart: unless-stopped
+
+  metricbeat:
+    logging: *vglog
+    labels: { vanguard.service: metricbeat, vanguard.category: infra }
+    image: docker.elastic.co/beats/metricbeat:9.2.0
+    container_name: metricbeat
+    environment:                            # Elasticsearch requires auth now; the config interpolates these
+      ES_BEATS_USER: ${VG_BEATS_USER:-vanguard_beats}
+      ES_BEATS_PASSWORD: "${VG_BEATS_PASSWORD:?run ./configure.py --apply first}"
+    user: root
+    command: ["metricbeat", "-e", "--strict.perms=false"]
+    # system + docker modules replace node_exporter and cAdvisor, and ship their own Kibana dashboards.
+    network_mode: host                      # real interfaces, and loopback reach to MediaMTX/Caddy/ES/Kibana
+    pid: host
+    volumes:
+      - ./observability/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /proc:/hostfs/proc:ro
+      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
+      - /:/hostfs:ro
+      - metricbeat_data:/usr/share/metricbeat/data
+    depends_on:
+      elasticsearch: { condition: service_healthy }
+    restart: unless-stopped
+
+networks:
+  botnet:                               # isolates discord-bot from the shared default network (see above)
+
 volumes:
   copyparty_hist:
+  filebeat_data:
+  metricbeat_data:
   caddy_data:
   caddy_config:

install.sh

@@ -7,7 +7,7 @@ OWNER="${SUDO_USER:-debian}"      # the login user; owns /srv/media and may run
 
 echo "== packages"
 apt-get update -q
-apt-get install -y -q ca-certificates curl gnupg ffmpeg jq psmisc rclone python3 xfsprogs
+apt-get install -y -q ca-certificates curl gnupg ffmpeg jq psmisc rclone python3 python3-yaml xfsprogs git nodejs npm rsync
 
 if ! command -v docker >/dev/null; then
   echo "== docker"
@@ -38,22 +38,40 @@ echo "== media user + folders"
 getent passwd 1000 >/dev/null || useradd -r -u 1000 -d /srv/media -s /usr/sbin/nologin media
 usermod -aG docker "$OWNER" 2>/dev/null || true
 install -d -o 1000 -g 1000 /srv/media/{streams,clips,finals,trash,.markers,.syncthing}
+install -d -o 1000 -g 1000 /srv/media/.markers/metrics                             # script counters -> dumpd /metrics
 install -d -o 1000 -g 1000 /srv/appdata/jellyfin/{config,cache} /srv/appdata/authentik/{media,templates,certs}
 install -d /srv/appdata/authentik/{db,redis} /srv/appdata/mariadb
 install -d -o 1000 -g 1000 /srv/appdata/discord-bot /srv/appdata/gitea /srv/appdata/act_runner
+install -d -o 1000 -g 1000 /srv/bot /srv/bot/app      # the Discord bot's released CI bundle (bind-mounted at /app; see bot/README.md)
+install -d -o 1000 -g 1000 /srv/appdata/elasticsearch                              # the ES image runs as uid 1000
+install -d /var/lib/vanguard                                                       # deadman state
+# Elasticsearch refuses to start below this. Host setting, so set it now and persist it.
+sysctl -q -w vm.max_map_count=262144
+printf 'vm.max_map_count=262144\n' > /etc/sysctl.d/99-vanguard-elasticsearch.conf
 install -d -o "$OWNER" /srv/live                 # live portal static build (deployLive.sh)
 install -d -o 1000 -g 1000 /srv/media/streams/{techy,diceofire} /srv/media/clips/{techy,diceofire}
 for d in /srv/media/clips/*/; do
   [[ -e "$d/.stignore" ]] || install -m 0644 -o 1000 -g 1000 syncthing/stignore "$d/.stignore"
+  [[ -e "$d/.stlocal" ]] || install -m 0644 -o 1000 -g 1000 /dev/null "$d/.stlocal"     # #include'd by .stignore
 done
 
 echo "== host scripts + units"
-install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py /usr/local/bin/
+install -m 0755 bin/normalize-sweep.sh bin/trash-purge.sh bin/dumpd.py bin/vgstrip.py bin/vgdeband.sh /usr/local/bin/
+install -m 0644 bin/vgmetrics.sh /usr/local/bin/      # sourced by the sweep/vgdeband; counters for dumpd's /metrics
+install -m 0755 deployBotBundle.sh /usr/local/bin/vanguard-deploy-bot   # dumpd runs this for the bot CI's POST /deploy-bot
+install -m 0755 bin/vanguard-deadman.sh /usr/local/bin/      # watches the observability stack from outside it
+install -m 0755 bin/vanguard-alerts.sh /usr/local/bin/        # takes Kibana's alert documents to Discord
+# Read by the root timers vanguard-alerts.sh (delivery) and vanguard-deadman.sh. configure.py rewrites it
+# from ALERT_WEBHOOK_URL on every apply; an empty one means "journal only, and exit non-zero".
+if [[ ! -f /etc/vanguard/alert-webhook ]]; then
+  install -d -m 0750 /etc/vanguard
+  install -m 0600 -o root -g root /dev/null /etc/vanguard/alert-webhook   # root timers are the only readers
+fi
 install -d -m 0750 /etc/vanguard
 install -m 0640 dump-users.txt /etc/vanguard/dump-users.txt
 install -m 0644 systemd/*.service systemd/*.timer /etc/systemd/system/
 systemctl daemon-reload
-systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service
+systemctl enable --now normalize-sweep.timer trash-purge.timer dumpd.service vanguard-deadman.timer vanguard-alerts.timer
 
 echo "== sysctl"
 cat > /etc/sysctl.d/90-vanguard.conf <<'SYS'
@@ -67,6 +85,12 @@ echo "== compose images (the stack is started by ./configure.py --full, which ge
 chown -R "$OWNER:" .
 docker compose pull -q --ignore-buildable 2>/dev/null || docker compose pull -q || true
 
+echo "== live portal: build into /srv/live (node)"
+if [[ -d live && -f live/package.json ]]; then
+  ( cd live && sudo -u "$OWNER" -H npm install --no-audit --no-fund --loglevel=error && sudo -u "$OWNER" -H npm run build ) \
+    && install -d -o "$OWNER" /srv/live && rsync -a --delete live/dist/ /srv/live/ && echo "   built" || echo "   portal build failed (run ./deployLive.sh from a dev box later)"
+fi
+
 echo "== host firewall (nftables; own table, input only; keeps this SSH session alive)"
 if [[ -f firewall/firewall.sh ]]; then
   bash firewall/firewall.sh

jellyfin-plugin-vanguard/.gitignore

@@ -1,4 +1,5 @@
 bin/
 obj/
 *.user
-dist/
+dist/*
+!dist/VanguardMetadata_*.zip

live/src/App.jsx

@@ -1,19 +1,57 @@
 import React, { useCallback, useEffect, useRef, useState } from 'react'
-import { DockviewReact } from 'dockview-react'
+import { DockviewReact, positionToDirection } from 'dockview-react'
 import Player from './Player.jsx'
 import { fetchStreams } from './api.js'
 import { audio } from './useAudio.js'
 import GoLive from './GoLive.jsx'
+import Tab from './Tab.jsx'
+import GroupLabel from './GroupLabel.jsx'
+import GroupActions from './GroupActions.jsx'
+import { streamLabel } from './names.js'
+import { layout, ding } from './layout.js'
+import Tray, { STREAM_MIME } from './Tray.jsx'
 
 const components = { player: Player }
+const tabComponents = { player: Tab }
 const POLL_MS = 3000
 
+// Lay every panel out as a grid (cols = ceil(sqrt(n)), row-major in stream-start order) and equalize sizes.
+// Moves the existing panels (dockview keeps the React component and its <video> alive), so viewers do not
+// reconnect. Called when a stream joins or leaves, and from the "Arrange" button.
+function addStream(api, s, position) {
+  api.addPanel({ id: s.id, component: 'player', tabComponent: 'player', title: streamLabel(s.id, s), params: s, position })
+}
+
+let arranging = false
+export function arrange(api) {
+  arranging = true
+  try { arrangeInner(api) } finally { arranging = false }
+}
+function arrangeInner(api) {
+  const panels = [...api.panels].sort((a, b) => ((a.params?.since || '').localeCompare(b.params?.since || '')) || a.id.localeCompare(b.id))
+  const n = panels.length; if (n < 2) return
+  if (api.hasMaximizedGroup?.()) api.exitMaximizedGroup?.()     // a maximised window would hide the new grid
+  const cols = Math.ceil(Math.sqrt(n))
+  for (let i = 1; i < n; i++) {
+    const r = Math.floor(i / cols), c = i % cols
+    const ref = r === 0 ? panels[i - 1] : panels[(r - 1) * cols + c]      // right of the previous in row 0, below the one above otherwise
+    panels[i].api.moveTo({ group: ref.group, position: r === 0 ? 'right' : 'bottom', skipSetActive: true })
+  }
+  const W = api.width || 0, H = api.height || 0; if (!W || !H) return
+  panels.forEach((p, i) => {
+    const c = i % cols, rowsInCol = Math.ceil((n - c) / cols)
+    p.group.api.setSize({ width: Math.floor(W / cols), height: Math.floor(H / rowsInCol) })
+  })
+}
+
+window.__vg = { layout }   // debug/test hook
 export default function App() {
   const apiRef = useRef(null)
   const [streams, setStreams] = useState([])
   const [err, setErr] = useState(null)
   const [mode, setMode] = useState('watch')          // watch | golive
   const focused = audio.use(s => s.focused)
+  const custom = layout.use(s => s.custom)
 
   // poll MediaMTX; open/close panels to match
   useEffect(() => {
@@ -25,19 +63,22 @@ export default function App() {
         const api = apiRef.current; if (!api) return   // (in golive mode the dock is unmounted: no viewer connections)
         const have = new Set(api.panels.map(p => p.id))
         const want = new Set(list.map(s => s.id))
+        let changed = false
+        const custom = layout.get().custom
         for (const s of list) {
           if (!have.has(s.id)) {
-            const last = api.panels[api.panels.length - 1]
+            if (custom && api.panels.length) {                 // hands off a custom layout: park it in the tray and ding
-            api.addPanel({
+              if (!layout.get().pending.some(p => p.id === s.id)) { layout.addPending(s); ding() }
-              id: s.id, component: 'player', title: `${s.user} · ${s.title}`,
+            } else {
-              params: s,
+              addStream(api, s, api.panels.length ? { direction: 'right' } : undefined); changed = true
-              position: last ? { referencePanel: last.id, direction: api.panels.length % 2 ? 'below' : 'right' } : undefined,
+            }
-            })
           } else {
             api.getPanel(s.id)?.api.updateParameters(s)
           }
         }
-        for (const p of [...api.panels]) if (!want.has(p.id)) { api.removePanel(p); audio.forget(p.id) }
+        for (const p of [...api.panels]) if (!want.has(p.id)) { api.removePanel(p); audio.forget(p.id); changed = true }
+        layout.keepPending(want)
+        if (changed && !layout.get().custom) arrange(api)
       } catch (e) { if (alive) setErr(e.message) }
     }
     tick(); const t = setInterval(tick, POLL_MS)
@@ -45,9 +86,28 @@ export default function App() {
   }, [])
 
   const onReady = useCallback(e => {
-    apiRef.current = e.api
+    const api = e.api; apiRef.current = api
-    e.api.onDidActivePanelChange(p => audio.focus(p?.id ?? null))
+    api.onDidActivePanelChange(p => audio.focus(p?.id ?? null))
+    api.onDidMovePanel(() => { if (!arranging) layout.markCustom() })           // a drag by the viewer, not our arrange()
+    api.onUnhandledDragOver(ev => {                                             // chips from the tray: show dockview's drop targets
+      const dt = ev.nativeEvent?.dataTransfer; if (dt && Array.from(dt.types || []).includes(STREAM_MIME)) ev.accept()
+    })
+    api.onDidDrop(ev => {
+      const id = ev.nativeEvent?.dataTransfer?.getData(STREAM_MIME); if (!id) return
+      const dir = positionToDirection(ev.position)
+      place(id, ev.group ? { referenceGroup: ev.group, direction: dir } : { direction: dir === 'within' ? 'right' : dir })
+    })
   }, [])
+  // add a parked stream where the viewer dropped/clicked it (custom layout stays custom)
+  const place = (id, position) => {
+    const api = apiRef.current; const s = layout.get().pending.find(p => p.id === id); if (!api || !s) return
+    addStream(api, s, position); layout.dropPending(id)
+  }
+  const arrangeAll = () => {                                                     // Arrange: place everything, re-grid, back to auto
+    const api = apiRef.current; if (!api) return
+    for (const s of layout.get().pending) addStream(api, s, { direction: 'right' })
+    layout.reset(); arrange(api)
+  }
 
   return (
     <div className="app">
@@ -56,6 +116,8 @@ export default function App() {
         <span className="count">{streams.length ? `${streams.length} live` : 'nobody is live'}{err ? ` · ${err}` : ''}</span>
         <span className="spacer" />
         <span className="count">audio follows the focused window</span>
+        <span className="count">{custom ? 'layout: yours (new streams wait in the tray)' : 'layout: auto grid'}</span>
+        <button onClick={arrangeAll} title="lay every window out as an even grid and go back to automatic layout">Arrange</button>
         <button onClick={() => audio.muteAll()} title="mute every window until you click one">Mute all</button>
         <button onClick={() => audio.followFocus()} title="clear pinned audio; only the focused window plays sound">Follow focus</button>
         <button className="golive-btn" onClick={() => { audio.muteAll(); apiRef.current = null; setMode('golive') }} title="stop watching and broadcast a window from this browser">Go Live</button>
@@ -67,8 +129,11 @@ export default function App() {
         {streams.length === 0 && !err && (
           <div className="empty"><h2>Nobody is live right now</h2><span>This page updates itself when a stream starts.</span></div>
         )}
-        <div className="dock" style={{ display: streams.length ? 'block' : 'none' }}>
+        <Tray onPlace={place} />
-          <DockviewReact className="dockview-theme-vanguard" components={components} onReady={onReady} />
+        <div className="dock" style={{ display: streams.length ? 'block' : 'none' }}
+             onPointerDownCapture={e => { if (e.target?.closest?.('.dv-sash')) layout.markCustom() }}>
+          <DockviewReact className="dockview-theme-vanguard" components={components} tabComponents={tabComponents}
+                         leftHeaderActionsComponent={GroupLabel} rightHeaderActionsComponent={GroupActions} onReady={onReady} />
         </div>
       </>)}
     </div>

live/src/GroupActions.jsx

@@ -0,0 +1,16 @@
+import React, { useEffect, useState } from 'react'
+
+// Right end of every tab bar: maximise / restore this group.
+export default function GroupActions({ api, containerApi }) {
+  const [max, setMax] = useState(() => api.isMaximized())
+  useEffect(() => {
+    const d = containerApi.onDidMaximizedGroupChange?.(() => setMax(api.isMaximized()))
+    return () => d?.dispose?.()
+  }, [api, containerApi])
+  const toggle = e => { e.stopPropagation(); if (api.isMaximized()) api.exitMaximized(); else api.maximize(); setMax(api.isMaximized()) }
+  return (
+    <div className="groupactions" onPointerDown={e => e.stopPropagation()}>
+      <button className="gbtn" onClick={toggle} title={max ? 'restore' : 'maximise this window'}>{max ? '🗗' : '⛶'}</button>
+    </div>
+  )
+}

live/src/GroupLabel.jsx

@@ -0,0 +1,23 @@
+import React, { useEffect, useRef, useState } from 'react'
+import { names } from './names.js'
+import { layout } from './layout.js'
+
+// Left end of every tab bar: an editable group name (double-click). Session-only.
+export default function GroupLabel({ group }) {
+  const id = group.id
+  const name = names.use(s => s.groups[id])
+  const [editing, setEditing] = useState(false)
+  const [draft, setDraft] = useState('')
+  const ref = useRef(null)
+  useEffect(() => { if (editing) { ref.current?.focus(); ref.current?.select() } }, [editing])
+  const start = e => { e.stopPropagation(); setDraft(name || ''); setEditing(true) }
+  const commit = () => { if (!editing) return; names.renameGroup(id, draft.trim()); layout.markCustom(); setEditing(false) }
+  return (
+    <div className="grouplabel" title="double-click to name this group" onDoubleClick={start} onPointerDown={e => e.stopPropagation()}>
+      {editing
+        ? <input ref={ref} value={draft} placeholder="group name" onChange={e => setDraft(e.target.value)} onBlur={commit}
+                 onKeyDown={e => { e.stopPropagation(); if (e.key === 'Enter') commit(); if (e.key === 'Escape') setEditing(false) }} />
+        : <span className={name ? 'gname' : 'hint'}>{name || '＋ name'}</span>}
+    </div>
+  )
+}

live/src/Player.jsx

@@ -2,14 +2,19 @@ import React, { useEffect, useRef, useState } from 'react'
 import Hls from 'hls.js'
 import { whepUrl, hlsUrl } from './api.js'
 import { audio, effectiveMuted } from './useAudio.js'
+import { names } from './names.js'
 
 // WebRTC (WHEP) first, HLS fallback. Mute/volume come from the shared audio store.
 export default function Player({ params, api }) {
   const { id, user, title } = params
+  const custom = names.use(s => s.streams[id])      // viewer-side rename (tab double-click)
+  const whep = params.whep || id                       // Opus audio mirror path when the source audio is AAC
   const videoRef = useRef(null)
   const [status, setStatus] = useState('connecting…')
   const [viewers, setViewers] = useState(params.viewers ?? 0)
-  const [aspect, setAspect] = useState(null)          // game aspect once we know the video size (crops the timecode band)
+  const wrapRef = useRef(null)
+  const [dims, setDims] = useState(null)              // {w, h, gameH}: video size + game height (h minus the timecode band)
+  const [box, setBox] = useState(null)                // {W, H}: game-aspect frame fitted into the panel; video overflows it at the bottom
   const state = audio.use()
   const muted = effectiveMuted(state, id)
   const volume = state.streams[id]?.volume ?? 0.8
@@ -17,6 +22,26 @@ export default function Player({ params, api }) {
   const focused = state.focused === id
 
   useEffect(() => { audio.ensure(id) }, [id])
+  // size the frame: fit a box of the GAME aspect (video width / height-without-band) into the panel, letterboxed.
+  // The video is laid across the full box width, so its band rows fall below the box and are clipped.
+  useEffect(() => {
+    const el = wrapRef.current; if (!el) return
+    const fit = () => {
+      if (!dims) { setBox(null); return }
+      const cw = el.clientWidth, ch = el.clientHeight; if (!cw || !ch) return
+      const a = dims.w / dims.gameH
+      const W = cw / ch > a ? Math.floor(ch * a) : cw
+      setBox({ W, H: Math.floor(W / a) })
+    }
+    fit(); const ro = new ResizeObserver(fit); ro.observe(el); return () => ro.disconnect()
+  }, [dims])
+  const onSize = e => {
+    const v = e.currentTarget; const w = v.videoWidth, h = v.videoHeight; if (!w || !h) return
+    let band = Math.round(w / 160); band += band % 2
+    // only treat the extra rows as a band if what remains is a standard game height
+    const gameH = [720, 1080, 1440, 2160].includes(h - band) ? h - band : h
+    setDims(d => (d && d.w === w && d.h === h) ? d : { w, h, gameH })
+  }
   useEffect(() => { const d = api.onDidActiveChange?.(() => {}) ; return () => d?.dispose?.() }, [api])
   useEffect(() => { const t = setInterval(() => setViewers(params.viewers ?? 0), 2000); return () => clearInterval(t) }, [params.viewers])
 
@@ -56,7 +81,7 @@ export default function Player({ params, api }) {
         const offer = await pc.createOffer()
         await pc.setLocalDescription(offer)
         await new Promise(res => { if (pc.iceGatheringState === 'complete') return res(); const t = setTimeout(res, 1500); pc.onicegatheringstatechange = () => { if (pc.iceGatheringState === 'complete') { clearTimeout(t); res() } } })
-        const r = await fetch(whepUrl(id), { method: 'POST', headers: { 'Content-Type': 'application/sdp' }, body: pc.localDescription.sdp, credentials: 'same-origin' })
+        const r = await fetch(whepUrl(whep), { method: 'POST', headers: { 'Content-Type': 'application/sdp' }, body: pc.localDescription.sdp, credentials: 'same-origin' })
         if (!r.ok) throw new Error('whep ' + r.status)
         await pc.setRemoteDescription({ type: 'answer', sdp: await r.text() })
         setStatus('webrtc')
@@ -68,23 +93,21 @@ export default function Player({ params, api }) {
     }
     startWhep()
     return () => { cancelled = true; try { pc?.close() } catch {}; try { hls?.destroy() } catch {}; v.srcObject = null; v.removeAttribute('src') }
-  }, [id])
+  }, [id, whep])
 
   const icon = muted ? '🔇' : volume === 0 ? '🔈' : volume < 0.5 ? '🔉' : '🔊'
   return (
     <div className={'player' + (focused ? ' focused' : '')} onMouseDown={() => api.setActive?.()}>
-      <div className="vwrap" style={aspect ? { aspectRatio: aspect } : undefined}>
+      <div className="vwrap" ref={wrapRef}>
-        <video ref={videoRef} autoPlay playsInline muted className={aspect ? 'cropband' : ''}
+        <div className="frame" style={box ? { width: box.W, height: box.H } : undefined}>
-               onLoadedMetadata={e => { const v = e.currentTarget; const w = v.videoWidth, h = v.videoHeight; if (!w || !h) return
+          <video ref={videoRef} autoPlay playsInline muted onLoadedMetadata={onSize} onResize={onSize} />
-                 let band = Math.round(w / 160); band += band % 2; const gameH = h - band
+        </div>
-                 // only treat it as a band if the remaining height is a standard game height
-                 setAspect([720, 1080, 1440, 2160].includes(gameH) ? `${w} / ${gameH}` : null) }} />
       </div>
       {status && <div className="status">{status}</div>}
       <div className="bar">
         <span className="live">LIVE</span>
         <span className="who">{user}</span>
-        <span className="title" title={title}>{title}</span>
+        <span className="title" title={title}>{custom || title}</span>
         <span className="viewers">👁 {viewers}</span>
         <button className={'mute' + (!muted ? ' on' : '') + (override === 'unmuted' && !focused ? ' pinned' : '')}
                 title={override ? (override === 'unmuted' ? 'audio pinned on (click to mute)' : 'muted (click to unmute)') : (focused ? 'focused: audio on (click to mute)' : 'muted: click to pin audio on')}

live/src/Tab.jsx

@@ -0,0 +1,25 @@
+import React, { useEffect, useRef, useState } from 'react'
+import { names } from './names.js'
+import { layout } from './layout.js'
+
+// Tab for a stream panel: double-click to rename (viewer-side label, kept in this browser). Empty = back to user · title.
+export default function Tab({ api, params }) {
+  const id = api.id
+  const custom = names.use(s => s.streams[id])
+  const base = `${params.user} · ${params.title}`
+  const [editing, setEditing] = useState(false)
+  const [draft, setDraft] = useState('')
+  const ref = useRef(null)
+  useEffect(() => { if (editing) { ref.current?.focus(); ref.current?.select() } }, [editing])
+  const start = e => { e.stopPropagation(); setDraft(custom || ''); setEditing(true) }
+  const commit = () => { if (!editing) return; const v = draft.trim(); names.rename(id, v); api.setTitle(v || base); layout.markCustom(); setEditing(false) }
+  return (
+    <div className="vtab" title={`${base}\ndouble-click to rename`} onDoubleClick={start}>
+      {editing
+        ? <input ref={ref} value={draft} placeholder={base} onChange={e => setDraft(e.target.value)}
+                 onPointerDown={e => e.stopPropagation()} onBlur={commit}
+                 onKeyDown={e => { e.stopPropagation(); if (e.key === 'Enter') commit(); if (e.key === 'Escape') setEditing(false) }} />
+        : <span className="vtab-text">{custom || base}</span>}
+    </div>
+  )
+}

live/src/Tray.jsx

@@ -0,0 +1,23 @@
+import React from 'react'
+import { layout } from './layout.js'
+import { streamLabel } from './names.js'
+
+export const STREAM_MIME = 'application/x-vanguard-stream'
+
+// Zero-height when empty. New streams in a custom layout wait here: drag a chip into the dock (any edge, any window,
+// or onto a tab bar to stack), or click it to add it as a new column on the right.
+export default function Tray({ onPlace }) {
+  const pending = layout.use(s => s.pending)
+  if (!pending.length) return null
+  return (
+    <div className="tray" title="new streams — drag into the layout, or click to add on the right">
+      {pending.map(s => (
+        <div key={s.id} className="chip" draggable
+             onDragStart={e => { e.dataTransfer.setData(STREAM_MIME, s.id); e.dataTransfer.setData('text/plain', s.id); e.dataTransfer.effectAllowed = 'move' }}
+             onClick={() => onPlace(s.id, { direction: 'right' })}>
+          <span className="blink">●</span> {streamLabel(s.id, s)}
+        </div>
+      ))}
+    </div>
+  )
+}

live/src/api.js

@@ -1,22 +1,37 @@
 // MediaMTX control API, proxied by Caddy at /api (-> 127.0.0.1:9997). Returns active streams.
+const VIDEO = new Set(['H264', 'H265', 'VP8', 'VP9', 'AV1', 'MJPEG', 'M-JPEG', 'MPEG-4 Video', 'MPEG-1 Video'])
+const firstSeen = new Map()                      // path -> first time we saw it ready (mirror grace period)
+const MIRROR_WAIT_MS = 15000
+
 export async function fetchStreams() {
   const r = await fetch('/api/v3/paths/list?itemsPerPage=200', { credentials: 'same-origin' })
   if (!r.ok) throw new Error(`api ${r.status}`)
   const data = await r.json()
-  return (data.items || [])
+  const ready = (data.items || []).filter(p => p.ready && (p.tracks || []).length > 0)
-    .filter(p => p.ready && (p.tracks || []).length > 0)
+  const byName = new Map(ready.map(p => [p.name, p]))
-    .map(p => {
+  const now = Date.now()
-      const [user, ...rest] = p.name.split('/')
+  for (const p of ready) if (!firstSeen.has(p.name)) firstSeen.set(p.name, now)
-      return {
+  for (const k of [...firstSeen.keys()]) if (!byName.has(k)) firstSeen.delete(k)
-        id: p.name,                                  // e.g. techy/VanguardTraining
+  const out = []
-        user,
+  for (const p of ready) {
-        title: rest.join('/') || p.name,
+    if (p.name.endsWith('.webrtc')) continue       // audio mirror: played through its source's panel, never its own
-        viewers: (p.readers || []).length,
+    const tracks = p.tracks || []
-        since: p.readyTime,
+    const audio = tracks.filter(t => !VIDEO.has(t))
-        tracks: p.tracks || [],
+    const needsMirror = audio.length > 0 && !audio.includes('Opus')   // AAC etc. cannot cross WebRTC
-      }
+    const mirror = byName.get(p.name + '.webrtc')
+    if (needsMirror && !mirror && now - firstSeen.get(p.name) < MIRROR_WAIT_MS) continue   // give the mirror a moment to come up
+    const [user, ...rest] = p.name.split('/')
+    out.push({
+      id: p.name,                                  // e.g. techy/VanguardTraining
+      user,
+      title: rest.join('/') || p.name,
+      whep: mirror ? mirror.name : p.name,         // WebRTC plays the Opus mirror when there is one; HLS always the source
+      viewers: (p.readers || []).length + (mirror ? (mirror.readers || []).length - 1 : 0),   // the mirror's ffmpeg is one reader
+      since: p.readyTime,
+      tracks,
     })
-    .sort((a, b) => (a.since || '').localeCompare(b.since || ''))
+  }
+  return out.sort((a, b) => (a.since || '').localeCompare(b.since || ''))
 }
 
 export const whepUrl = id => `/webrtc/${id}/whep`

live/src/layout.js

@@ -0,0 +1,30 @@
+import { create } from './tinyStore.js'
+
+// Layout mode. "auto" = untouched grid: streams joining/leaving re-grid everything. The first manual move, resize
+// or rename makes it "custom": the grid is left alone, leavers just close, joiners wait in the tray (with a ding)
+// until dragged in. "Arrange" re-grids everything and goes back to auto.
+export const layout = create({
+  custom: false,
+  pending: [],                               // streams live but not placed yet (custom mode)
+  markCustom() { return this.custom ? null : { custom: true } },
+  reset() { return { custom: false, pending: [] } },
+  addPending(s) { return this.pending.some(p => p.id === s.id) ? null : { pending: [...this.pending, s] } },
+  dropPending(id) { return this.pending.some(p => p.id === id) ? { pending: this.pending.filter(p => p.id !== id) } : null },
+  keepPending(ids) { const pending = this.pending.filter(p => ids.has(p.id)); return pending.length === this.pending.length ? null : { pending } },
+})
+
+// short two-note chime (WebAudio, no asset). Browsers need one prior click on the page to allow sound.
+let ctx
+export function ding() {
+  try {
+    ctx = ctx || new (window.AudioContext || window.webkitAudioContext)()
+    if (ctx.state === 'suspended') ctx.resume().catch(() => {})
+    const t0 = ctx.currentTime
+    for (const [i, f] of [[0, 880], [0.16, 1320]]) {
+      const o = ctx.createOscillator(), g = ctx.createGain()
+      o.type = 'sine'; o.frequency.value = f
+      g.gain.setValueAtTime(0, t0 + i); g.gain.linearRampToValueAtTime(0.25, t0 + i + 0.02); g.gain.exponentialRampToValueAtTime(0.001, t0 + i + 0.35)
+      o.connect(g).connect(ctx.destination); o.start(t0 + i); o.stop(t0 + i + 0.4)
+    }
+  } catch {}
+}

live/src/names.js

@@ -0,0 +1,25 @@
+import { create } from './tinyStore.js'
+
+// Viewer-side labels. Stream names persist in this browser (localStorage, keyed by the MediaMTX path);
+// group names live for the session (dockview group ids are regenerated on reload).
+const KEY = 'vg.names'
+let saved = {}
+try { saved = JSON.parse(localStorage.getItem(KEY) || '{}') || {} } catch {}
+
+export const names = create({
+  streams: saved,
+  groups: {},
+  rename(id, value) {
+    const streams = { ...this.streams }
+    if (value) streams[id] = value; else delete streams[id]
+    try { localStorage.setItem(KEY, JSON.stringify(streams)) } catch {}
+    return { streams }
+  },
+  renameGroup(id, value) {
+    const groups = { ...this.groups }
+    if (value) groups[id] = value; else delete groups[id]
+    return { groups }
+  },
+})
+
+export const streamLabel = (id, params) => names.get().streams[id] || `${params.user} · ${params.title}`

live/src/styles.css

@@ -16,10 +16,12 @@ html, body, #root { height:100%; margin:0; background:var(--bg); color:var(--fg)
   --dv-separator-border:#2a2a2a; --dv-group-view-background-color:var(--panel); --dv-drag-over-background-color:rgba(61,139,253,.2);
   --dv-drag-over-border-color:var(--acc); --dv-tabs-container-scrollbar-color:#444; --dv-icon-hover-background-color:#333; }
 .player { position:relative; width:100%; height:100%; background:#000; display:flex; flex-direction:column; }
-.player .vwrap { flex:1; min-height:0; display:flex; align-items:center; justify-content:center; background:#000; }
+.player .vwrap { flex:1; min-height:0; position:relative; display:flex; align-items:center; justify-content:center; background:#000; overflow:hidden; }
-.player .vwrap[style*="aspect-ratio"] { max-height:100%; max-width:100%; margin:auto; }
+/* .frame = game-aspect box (sized by Player.jsx); the video spans its width, so the timecode band rows overflow the bottom and are clipped */
-.player video { width:100%; height:100%; object-fit:contain; background:#000; }
+.player .frame { position:relative; width:100%; height:100%; overflow:hidden; background:#000; }
-.player video.cropband { object-fit:cover; object-position:50% 0; }   /* top-aligned cover inside a game-aspect box = band cropped */
+.player .frame video { position:absolute; top:0; left:0; width:100%; height:auto; display:block; background:#000; }
+.player .frame[style] video { height:auto; }
+.player .frame:not([style]) video { position:static; width:100%; height:100%; object-fit:contain; }   /* before the size is known: plain contain */
 .player .bar { display:flex; align-items:center; gap:10px; padding:6px 10px; background:#181818; border-top:1px solid #2a2a2a; font-size:13px; }
 .player .live { background:var(--live); color:#fff; font-weight:700; font-size:11px; padding:2px 6px; border-radius:4px; letter-spacing:.5px; }
 .player .who { font-weight:600; }
@@ -61,3 +63,21 @@ html, body, #root { height:100%; margin:0; background:var(--bg); color:var(--fg)
 .golive-stats .rec { background:#f1c40f; color:#000; font-weight:700; font-size:11px; padding:2px 6px; border-radius:4px; }
 .topbar a.signout { color:var(--mut); font-size:13px; text-decoration:none; border:1px solid #3a3a3a; border-radius:6px; padding:5px 10px; }
 .topbar a.signout:hover { color:var(--fg); border-color:#666; }
+
+/* renameable tabs + group names */
+.vtab { display:flex; align-items:center; height:100%; padding:0 10px; min-width:0; }
+.vtab-text { overflow:hidden; text-overflow:ellipsis; white-space:nowrap; max-width:260px; }
+.vtab input, .grouplabel input { background:#111; color:var(--fg); border:1px solid var(--acc); border-radius:4px; font:inherit; font-size:12px; padding:1px 6px; width:170px; outline:none; }
+.grouplabel { display:flex; align-items:center; height:100%; padding:0 10px; font-size:12px; white-space:nowrap; border-right:1px solid #2a2a2a; cursor:default; }
+.grouplabel .gname { color:var(--acc); font-weight:700; }
+.grouplabel .hint { color:#555; }
+.groupactions { display:flex; align-items:center; height:100%; padding:0 6px; }
+.gbtn { background:transparent; color:var(--mut); border:0; cursor:pointer; font-size:14px; padding:0 6px; line-height:1; }
+.gbtn:hover { color:var(--fg); }
+
+/* tray: new streams waiting to be placed (custom layouts) */
+.tray { display:flex; gap:8px; align-items:center; padding:4px 10px; background:#1d1a10; border-bottom:1px solid #3a3320; font-size:12px; }
+.chip { display:flex; align-items:center; gap:6px; padding:3px 10px; border:1px solid #6b5a1e; border-radius:14px; background:#262012; color:var(--fg); cursor:grab; user-select:none; }
+.chip:active { cursor:grabbing; }
+.blink { color:#f1c40f; animation: vgblink 1s steps(2, start) infinite; }
+@keyframes vgblink { to { visibility:hidden; } }

observability/filebeat.yml

@@ -0,0 +1,89 @@
+# Filebeat: container logs + our own systemd units -> Elasticsearch.
+# Autodiscover picks up every container, and add_docker_metadata brings the vanguard.service /
+# vanguard.category labels with it (Beats turn the dots into underscores), so filtering by category in Kibana
+# needs no parsing. See LOGGING.md for what each service's lines look like.
+#
+# NOTE (9.x): the old "container" input is GONE, not merely deprecated — Filebeat 9 refuses the config and
+# logs "Container input is deprecated. Use Filestream input with its container parser instead", leaving the
+# process up with zero harvesters. Container logs are read by filestream + the container parser, and every
+# filestream input needs its own stable id.
+
+filebeat.autodiscover:
+  providers:
+    - type: docker
+      hints.enabled: false
+      templates:
+        - condition: ~                      # every container
+          config:
+            - type: filestream
+              id: container-${data.docker.container.id}
+              paths: ["/var/lib/docker/containers/${data.docker.container.id}/*.log"]
+              parsers:
+                - container: ~
+              prospector.scanner.symlinks: true
+              # Stamp the id here rather than letting add_docker_metadata guess it from the path.
+              processors:
+                - add_fields:
+                    target: container
+                    fields:
+                      id: "${data.docker.container.id}"
+                      name: "${data.docker.container.name}"
+                      image.name: "${data.docker.container.image}"
+
+filebeat.inputs:
+  - type: journald
+    id: vanguard-units
+    include_matches:
+      match:
+        - "_SYSTEMD_UNIT=dumpd.service"
+        - "_SYSTEMD_UNIT=normalize-sweep.service"
+        - "_SYSTEMD_UNIT=trash-purge.service"
+        - "_SYSTEMD_UNIT=vgdeband.service"
+        - "_SYSTEMD_UNIT=vanguard-deadman.service"
+
+processors:
+  - add_docker_metadata:
+      match_fields: ["container.id"]        # the id the autodiscover template stamped above
+  - add_host_metadata: ~
+  # promote our labels to real fields so Kibana filters on service.name and vanguard.category
+  - copy_fields:
+      fields:
+        - from: container.labels.vanguard_service
+          to: service.name
+        - from: container.labels.vanguard_category
+          to: vanguard.category
+      fail_on_error: false
+      ignore_missing: true
+  # The Observability overview's "Logs rate" chart groups by event.dataset, so anything without one lands in
+  # a single "unknown" bucket and the chart says nothing. Give every line a dataset: the container's
+  # vanguard.service label, or the systemd unit for journald lines. copy_fields does not overwrite a target
+  # that already exists, so the first rule that matches wins and the other is skipped.
+  - copy_fields:
+      fields:
+        - from: container.labels.vanguard_service
+          to: event.dataset
+        - from: systemd.unit
+          to: event.dataset
+      fail_on_error: false
+      ignore_missing: true
+  - drop_fields:
+      fields: ["agent.ephemeral_id", "ecs.version", "log.offset"]
+      ignore_missing: true
+
+output.elasticsearch:
+  hosts: ["http://elasticsearch:9200"]
+  # Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
+  # come from the container environment, which compose fills from .env — no secret is committed here.
+  username: "${ES_BEATS_USER}"
+  password: "${ES_BEATS_PASSWORD}"
+
+setup.kibana:                               # no credentials: Kibana's anonymous provider signs these in
+  host: "http://kibana:5601"
+setup.ilm.enabled: true                     # retention: set the delete phase once in Kibana (README 5h)
+# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
+# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
+setup.template.overwrite: true
+setup.template.settings:
+  index.number_of_replicas: 0
+
+logging.level: info

observability/kibana-alarms.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python3
+"""The real alarm rules. Idempotent: creates what is missing, leaves what exists.
+
+Every rule is an Elasticsearch query rule that counts matching documents in a window and writes one document
+into the alert bus through the "Vanguard alert index" connector, which bin/vanguard-alerts.sh delivers to
+Discord. Run observability/kibana-setup.py first — it creates that connector.
+
+FIRING CADENCE: notify_when=onThrottleInterval with throttle=12h, so an alarm that stays true writes again
+every 12 hours and stops when the condition clears. The delivery script's own 15-minute per-rule suppression
+sits well inside that and never interferes.
+
+WHAT IS DELIBERATELY NOT HERE
+Elasticsearch, Kibana and Beat liveness, and the disks, are bin/vanguard-deadman.sh's job. Nothing that
+depends on Kibana being alive can report that Kibana is dead, so those checks stay outside the stack and
+duplicating them here would be worse than useless: they would go quiet in exactly the case that matters.
+
+  pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
+  KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
+          -H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
+        | python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
+  KEY="$KEY" python3 observability/kibana-alarms.py
+
+To change a rule, edit it in Kibana or delete it there and re-run this. Thresholds here are the defaults:
+  FREE_PCT        15   alarm when a real filesystem or memory has less than this percent free
+  STUCK_SECONDS  3600  alarm when a finished recording has waited this long to be normalised
+  THROTTLE        12h  how often an unresolved alarm repeats
+"""
+import json, os, urllib.error, urllib.request
+
+KIBANA = "http://127.0.0.1:5601"
+HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
+           "Authorization": "ApiKey " + os.environ["KEY"]}
+CONNECTOR_NAME = "Vanguard alert index"
+
+FREE_PCT = float(os.environ.get("FREE_PCT", "15"))
+STUCK_SECONDS = int(os.environ.get("STUCK_SECONDS", "3600"))
+THROTTLE = os.environ.get("THROTTLE", "12h")
+USED_MAX = round(1 - FREE_PCT / 100, 4)          # 15% free -> used.pct >= 0.85
+
+# Filesystems that are not disks. Without this, every tmpfs and overlay mount reports ~100% used and the
+# disk alarm would fire permanently on something that cannot fill up.
+PSEUDO_FS = ["tmpfs", "devtmpfs", "overlay", "squashfs", "ramfs", "autofs", "nsfs", "tracefs", "debugfs",
+             "configfs", "mqueue", "hugetlbfs", "pstore", "bpf", "devpts", "proc", "sysfs", "cgroup",
+             "cgroup2", "fusectl", "securityfs", "efivarfs", "binfmt_misc", "fuse.lxcfs", "iso9660"]
+
+def rule(name, level, why, query, *, index="metricbeat-*", window=10, comparator=">", threshold=0):
+    return {"name": name, "level": level, "why": why, "query": query,
+            "index": index, "window": window, "comparator": comparator, "threshold": threshold}
+
+ALARMS = [
+    rule("Disk free below %g%%" % FREE_PCT, "error",
+         "A filesystem is nearly full. Elasticsearch turns its indices READ-ONLY at the flood watermark and "
+         "does not undo that by itself, and recordings stop being published.",
+         {"query": {"bool": {
+             "filter": [{"range": {"system.filesystem.used.pct": {"gte": USED_MAX}}}],
+             "must_not": [{"terms": {"system.filesystem.type": PSEUDO_FS}}]}}}),
+
+    rule("Memory free below %g%%" % FREE_PCT, "warn",
+         "Memory is nearly exhausted. Elasticsearch has a pinned 2 GB heap, so the pressure is usually "
+         "ffmpeg or Jellyfin, and the kernel will start killing processes rather than queueing.",
+         {"query": {"range": {"system.memory.actual.used.pct": {"gte": USED_MAX}}}}),
+
+    rule("Discord bot not running", "error",
+         "No metrics from the discord-bot container for 5 minutes: it is stopped, removed, or restarting too "
+         "fast to be sampled. Nothing it does is happening.",
+         {"query": {"term": {"container.name": "discord-bot"}}},
+         window=5, comparator="<", threshold=1),
+
+    rule("Discord bot unhealthy", "error",
+         "The bot is running but reports itself degraded: its /health says the Discord gateway is "
+         "disconnected or the database does not answer. It is up and doing nothing.",
+         {"query": {"bool": {"filter": [{"term": {"container.name": "discord-bot"}},
+                                        {"term": {"docker.healthcheck.status": "unhealthy"}}]}}},
+         window=5),
+
+    rule("Recordings stuck waiting to be normalised", "warn",
+         "A finished recording has waited over %d minutes in .live/. It is not in Jellyfin, not in copyparty "
+         "and not announced, so to everyone else it does not exist." % (STUCK_SECONDS // 60),
+         {"query": {"range": {"prometheus.vanguard_media_pending_oldest_seconds.value": {"gt": STUCK_SECONDS}}}}),
+
+    rule("Normalise failures parked", "warn",
+         "One or more files have a normalize-failed log, so the sweep skips them forever until it is deleted. "
+         "The log now contains ffmpeg's stderr, which says why.",
+         {"query": {"range": {"prometheus.vanguard_media_failed_files.value": {"gt": 0}}}}),
+]
+
+def req(method, path, body=None):
+    request = urllib.request.Request(KIBANA + path, method=method,
+                                     data=None if body is None else json.dumps(body).encode())
+    for key, value in HEADERS.items():
+        request.add_header(key, value)
+    try:
+        with urllib.request.urlopen(request, timeout=30) as response:
+            raw = response.read()
+            return json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as error:
+        print(f"  {method} {path} -> HTTP {error.code}")
+        print("  " + error.read().decode(errors="replace")[:600])
+        raise SystemExit(1)
+
+connectors = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+if CONNECTOR_NAME not in connectors:
+    raise SystemExit(f"connector {CONNECTOR_NAME!r} is missing — run observability/kibana-setup.py first")
+connector_id = connectors[CONNECTOR_NAME]
+
+present = {r["name"] for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])}
+created = skipped = 0
+for alarm in ALARMS:
+    if alarm["name"] in present:
+        print("already present:", alarm["name"])
+        skipped += 1
+        continue
+    made = req("POST", "/api/alerting/rule", {
+        "name": alarm["name"],
+        "rule_type_id": ".es-query",
+        "consumer": "stackAlerts",
+        "enabled": True,
+        "schedule": {"interval": "1m"},
+        "params": {
+            "searchType": "esQuery",
+            "index": [alarm["index"]],
+            "timeField": "@timestamp",
+            "esQuery": json.dumps(alarm["query"]),
+            "size": 1,
+            "aggType": "count",
+            "groupBy": "all",
+            "thresholdComparator": alarm["comparator"],
+            "threshold": [alarm["threshold"]],
+            "timeWindowSize": alarm["window"],
+            "timeWindowUnit": "m",
+            "excludeHitsFromPreviousRun": False,
+        },
+        "actions": [{
+            "group": "query matched",
+            "id": connector_id,
+            "params": {"documents": [{
+                "kind": "alarm",
+                "rule": "{{rule.name}}",
+                "level": alarm["level"],
+                "message": alarm["why"] + " [{{context.message}}]",
+            }]},
+            # every 12 h for as long as it stays true, then silence when it clears
+            "frequency": {"summary": False, "notify_when": "onThrottleInterval", "throttle": THROTTLE},
+        }],
+    })
+    print("created:", made["name"], "|", alarm["level"], "| repeats every", THROTTLE)
+    created += 1
+
+print(f"\n{created} created, {skipped} already there. Disk/memory threshold {FREE_PCT:g}% free; "
+      f"stuck after {STUCK_SECONDS}s.")

observability/kibana-setup.py

@@ -0,0 +1,161 @@
+#!/usr/bin/env python3
+"""Kibana first-run state. Idempotent; safe to re-run. configure.py --apply runs this for you.
+
+  connector  "Vanguard alert index"        — the .index connector, writing into the vanguard-alerts index
+  rule       "Vanguard alerting heartbeat" — always-firing ES query rule, every minute, writing one doc
+  data view  "Vanguard logs + metrics"     — filebeat-*,metricbeat-* so Discover and Lens have one source
+
+WHY AN INDEX AND NOT A WEBHOOK
+On a basic license the only connector types Kibana allows are .index and .server-log; every connector that
+leaves the stack (webhook, Slack, email, PagerDuty…) is Gold or above and returns
+    403 Action type .webhook is disabled because your basic license does not support it
+So Kibana cannot talk to Discord. Instead every rule writes a document into ONE index, which is the alert
+bus, and bin/vanguard-alerts.sh (a host timer) delivers from there to Discord and later to the bot. That is
+strictly simpler than the webhook route: no Docker-bridge hop, no dumpd endpoint, and alerts are queryable
+and replayable after the fact because they are just documents.
+
+THE HEARTBEAT
+bin/vanguard-deadman.sh alarms when no heartbeat document has arrived recently. It is the only check that
+catches the ALERTING pipeline itself dying — if Kibana, its rule engine or the connector stops, no ordinary
+alert can report it, because the thing that would report it is what broke. The rule queries metricbeat-*,
+which always has recent documents, so it doubles as a Metricbeat liveness check.
+
+Run it with an Elasticsearch API key: Basic is not an accepted scheme for this Kibana (the anonymous provider
+is the only one enabled, so only ApiKey and Bearer are taken).
+
+  pw=$(grep -m1 '^ES_ELASTIC_PASSWORD=' .env | cut -d= -f2-)
+  KEY=$(curl -s -XPOST -u "elastic:$pw" localhost:9200/_security/api_key \
+          -H 'Content-Type: application/json' -d '{"name":"vanguard-cli"}' \
+        | python3 -c 'import json,sys; print(json.load(sys.stdin)["encoded"])')
+  KEY="$KEY" python3 observability/kibana-setup.py
+
+Order matters: run this AFTER security is on. A rule created while security was off holds no API key and
+stops working the moment it is enabled, until it is disabled and re-enabled.
+
+ADDING A REAL ALARM: author it in Kibana as usual, point its action at the "Vanguard alert index" connector,
+and give the document a message plus a level (warn|error). Do NOT set kind=heartbeat — that value is how the
+delivery script recognises the heartbeat and keeps it out of Discord. Use "On status change" or a throttle
+for real alarms; only the heartbeat wants to fire every interval.
+"""
+import json, os, time, urllib.error, urllib.request
+
+KIBANA = "http://127.0.0.1:5601"
+ALERT_INDEX = os.environ.get("ALERT_INDEX", "vanguard-alerts")
+HEADERS = {"Content-Type": "application/json", "kbn-xsrf": "1",
+           "Authorization": "ApiKey " + os.environ["KEY"]}
+
+def wait_for_kibana(seconds=300):
+    """Kibana has to be *available*, not merely listening.
+
+    configure.py --es-security restarts Kibana as its last step, so anything that calls the Kibana API
+    immediately afterwards races its boot and gets a connection reset. Kibana also answers /api/status while
+    still starting, so the status level is what has to be checked, not the port.
+    """
+    deadline = time.time() + seconds
+    announced = False
+    while time.time() < deadline:
+        try:
+            with urllib.request.urlopen(KIBANA + "/api/status", timeout=10) as response:
+                level = json.load(response).get("status", {}).get("overall", {}).get("level")
+            if level == "available":
+                print("kibana: available")
+                return
+            state = level or "no level"
+        except Exception as error:
+            state = type(error).__name__
+        if not announced:
+            print(f"kibana: waiting for it to become available ({state})…")
+            announced = True
+        time.sleep(5)
+    raise SystemExit(f"kibana did not become available within {seconds}s — nothing was created")
+
+def req(method, path, body=None):
+    request = urllib.request.Request(KIBANA + path, method=method,
+                                     data=None if body is None else json.dumps(body).encode())
+    for key, value in HEADERS.items():
+        request.add_header(key, value)
+    try:
+        with urllib.request.urlopen(request, timeout=30) as response:
+            raw = response.read()
+            return json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as error:
+        print(f"  {method} {path} -> HTTP {error.code}")
+        print("  " + error.read().decode(errors="replace")[:600])
+        raise SystemExit(1)
+
+wait_for_kibana()
+
+# 1. the alert bus connector
+CONNECTOR_NAME = "Vanguard alert index"
+existing = {c["name"]: c["id"] for c in req("GET", "/api/actions/connectors")}
+if CONNECTOR_NAME in existing:
+    connector_id = existing[CONNECTOR_NAME]
+    print("connector already present:", connector_id)
+else:
+    connector_id = req("POST", "/api/actions/connector", {
+        "name": CONNECTOR_NAME,
+        "connector_type_id": ".index",
+        # executionTimeField stamps every document, so a rule body never has to get the timestamp right.
+        "config": {"index": ALERT_INDEX, "refresh": True, "executionTimeField": "@timestamp"},
+        "secrets": {}})["id"]
+    print("connector created:", connector_id, "->", ALERT_INDEX)
+
+# 2. the heartbeat rule. Handled independently of the data view below: an early exit here once looked
+#    harmless and meant the data view was never created on any system where the rule already existed, which
+#    is every re-run.
+RULE_NAME = "Vanguard alerting heartbeat"
+existing_rule = next((r for r in req("GET", "/api/alerting/rules/_find?per_page=100").get("data", [])
+                      if r["name"] == RULE_NAME), None)
+if existing_rule:
+    print("rule already present:", existing_rule["id"],
+          "enabled" if existing_rule.get("enabled") else "DISABLED")
+else:
+    rule = req("POST", "/api/alerting/rule", {
+        "name": RULE_NAME,
+        "rule_type_id": ".es-query",
+        "consumer": "stackAlerts",
+        "enabled": True,
+        "schedule": {"interval": "1m"},
+        "params": {
+            "searchType": "esQuery",
+            "index": ["metricbeat-*"],
+            "timeField": "@timestamp",
+            "esQuery": json.dumps({"query": {"match_all": {}}}),
+            "size": 1,
+            "aggType": "count",
+            "groupBy": "all",
+            "thresholdComparator": ">",
+            "threshold": [0],
+            "timeWindowSize": 5,
+            "timeWindowUnit": "m",
+            # re-evaluate the whole window every run, so the rule stays active instead of going quiet once
+            # it has already seen those documents
+            "excludeHitsFromPreviousRun": False,
+        },
+        # onActiveAlert writes on EVERY check. Kibana's default only acts when a rule CHANGES status, which
+        # would produce one heartbeat and then silence — the deadman would read that as a dead pipeline.
+        "actions": [{
+            "group": "query matched",
+            "id": connector_id,
+            "params": {"documents": [{
+                "kind": "heartbeat",             # the delivery script drops these; never reach Discord
+                "rule": "{{rule.name}}",
+                "level": "info",
+                "message": "kibana rule engine and the index connector are alive",
+            }]},
+            "frequency": {"summary": False, "notify_when": "onActiveAlert", "throttle": None},
+        }],
+    })
+    print("rule created:", rule["id"], "| enabled:", rule.get("enabled"))
+
+# 3. the data view. Without one, Discover and Lens have nothing to point at and every panel starts with a
+#    detour through Stack Management. Both patterns in one view so a single dashboard can mix logs and
+#    metrics, which is the whole reason they share a cluster.
+VIEW_TITLE = "filebeat-*,metricbeat-*"
+views = req("GET", "/api/data_views").get("data_view", [])
+if any(v.get("title") == VIEW_TITLE for v in views):
+    print("data view already present:", VIEW_TITLE)
+else:
+    view = req("POST", "/api/data_views/data_view", {
+        "data_view": {"title": VIEW_TITLE, "name": "Vanguard logs + metrics", "timeFieldName": "@timestamp"}})
+    print("data view created:", view["data_view"]["id"], "->", VIEW_TITLE)

observability/kibana.yml

@@ -0,0 +1,41 @@
+# Kibana. Mounted read-only; every secret arrives as an environment variable that configure.py generates
+# into .env, so this file is committed and contains none.
+#
+# WHY SECURITY IS ON, AND WHY NOBODY SEES A LOGIN
+# Elasticsearch runs with xpack.security.enabled=true because Kibana's Observability apps ask Elasticsearch
+# whether the current user holds a privilege. With security off that API does not exist, and the
+# Infrastructure / Metrics app fails on every page view with
+#   no handler found for uri [/_security/user/_has_privileges]
+# Turning it on would normally put a password prompt in front of Kibana, which this stack does not allow:
+# Discord is the only login anywhere. So the anonymous provider below signs every visitor in automatically as
+# one Elasticsearch user. The real gate is unchanged and is still the only one: Caddy's Authentik
+# forward-auth plus the DIAG_REQUIRE_GROUP check on the diag host.
+#
+# This is not a permission tier. Everyone who clears that gate is the same Elasticsearch user with the same
+# rights, exactly as before. The difference is that the identity now exists, which is what the apps need and
+# what a read-only tier would later be built on (see STATUS.md 9b: add the http provider with the apikey
+# scheme and have Caddy inject a key chosen by Authentik group).
+
+server.host: "0.0.0.0"
+server.publicBaseUrl: "https://${DIAG_HOST}"
+# NO server.basePath. Kibana IS the root of the diag host, and the key must be ABSENT rather than empty:
+# an empty string fails validation with "must start with a slash, don't end with one" and Kibana refuses to
+# boot. It survived as an empty SERVER_BASEPATH env var only because the image drops empty values.
+telemetry.optIn: false
+
+elasticsearch.hosts: ["http://elasticsearch:9200"]
+elasticsearch.username: "kibana_system"     # Kibana's own internal account, NOT what visitors become
+elasticsearch.password: "${ES_KIBANA_PASSWORD}"
+
+# Fixed keys: without them Kibana re-generates them on restart and loses saved objects (dashboards, rules).
+xpack.encryptedSavedObjects.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+xpack.reporting.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+xpack.security.encryptionKey: "${KIBANA_ENCRYPTION_KEY}"
+
+# The only provider, so there is no login page and no login selector: every request is already signed in.
+xpack.security.authc.providers:
+  anonymous.anonymous1:
+    order: 0
+    credentials:
+      username: "${VG_DIAG_USER}"
+      password: "${VG_DIAG_PASSWORD}"

observability/metricbeat.yml

@@ -0,0 +1,69 @@
+# Metricbeat: host and per-container metrics, replacing node_exporter and cAdvisor, plus the Prometheus
+# endpoints the stack already exposes. Runs on the host network so the system module sees the real interfaces
+# and so it can reach MediaMTX, Caddy, Elasticsearch and Kibana on loopback.
+
+metricbeat.modules:
+  - module: system
+    period: 30s
+    metricsets: [cpu, load, memory, network, process, process_summary, socket_summary, filesystem, fsstat, diskio, uptime]
+    processes: ['.*']
+    process.include_top_n:
+      by_cpu: 10
+      by_memory: 10
+    system.hostfs: /hostfs                  # the host's real filesystems, not the container's
+
+  - module: docker
+    period: 30s
+    hosts: ["unix:///var/run/docker.sock"]
+    metricsets: [container, cpu, diskio, healthcheck, info, memory, network]
+
+  - module: prometheus                      # MediaMTX: per path bytes/s, readers, publishers, dropped packets
+    period: 30s
+    metrics_path: /metrics
+    hosts: ["http://127.0.0.1:9998"]
+
+  - module: prometheus                      # Caddy: requests by host and status, durations, TLS
+    period: 60s
+    metrics_path: /metrics
+    hosts: ["http://127.0.0.1:2019"]
+
+  - module: prometheus                      # dumpd: OUR metrics — normalise queue, encode time, live streams
+    period: 30s
+    metrics_path: /metrics
+    hosts: ["http://127.0.0.1:8080"]        # dumpd's loopback listener; Caddy routes no host to /metrics
+    # These are real counters, so let them arrive typed: Kibana then gets .rate (per-second) as well as the
+    # raw total, which is what makes "encode seconds per hour" a chart rather than a staircase.
+    use_types: true
+    rate_counters: true
+
+processors:
+  - add_host_metadata: ~
+  # The same correlation fields Filebeat sets, so ONE dashboard filter spans logs and metrics. Without these a
+  # metric event carries container.name while a log event carries service.name, and nothing joins them.
+  - add_docker_metadata: ~
+  - copy_fields:
+      fields:
+        - from: container.labels.vanguard_service
+          to: service.name
+        - from: container.labels.vanguard_category
+          to: vanguard.category
+      fail_on_error: false
+      ignore_missing: true
+
+output.elasticsearch:
+  hosts: ["http://127.0.0.1:9200"]
+  # Elasticsearch requires authentication now (see observability/kibana.yml for why security is on). These
+  # come from the container environment, which compose fills from .env — no secret is committed here.
+  username: "${ES_BEATS_USER}"
+  password: "${ES_BEATS_PASSWORD}"
+
+setup.kibana:                               # no credentials: Kibana's anonymous provider signs these in
+  host: "http://127.0.0.1:5601"
+setup.dashboards.enabled: true              # the system and docker dashboards, so there is something to look at
+# Single node: a replica has nowhere to go, so every beats index would sit unassigned and hold the
+# cluster at yellow forever. Zero replicas is the correct setting here, not a workaround.
+setup.template.overwrite: true
+setup.template.settings:
+  index.number_of_replicas: 0
+
+logging.level: info

registerGiteaRunner.sh

@@ -11,3 +11,5 @@ sudo rm -rf /srv/appdata/act_runner/.runner 2>/dev/null || true     # forget any
 docker compose up -d --force-recreate act_runner
 sleep 6; docker logs act_runner --tail 8 2>&1 | sed 's/^/  /'
 echo "check: Gitea -> Site Administration -> Actions -> Runners (should list 'vanguard-runner' as idle)"
+echo "labels: $(grep -o 'GITEA_RUNNER_LABELS: \".*\"' docker-compose.yml | sed 's/.*\"\(.*\)\"/\1/' | tr ',' '\n' | cut -d: -f1 | tr '\n' ' ')"
+echo "  (labels are fixed at registration — re-run this script after changing GITEA_RUNNER_LABELS)"

switchToClone.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+# One-time: turn ~/vanguard-stream (an scp copy) into a git clone of the Gitea repo, keeping the generated files.
+#   ./switchToClone.sh key     -> creates a deploy key and prints the public half (add it in Gitea: repo → Settings → Deploy keys, read-only)
+#   ./switchToClone.sh clone   -> clones over ssh (port 2222), moves .env + generated files + plugin zips into it
+set -euo pipefail
+REPO_SSH="${REPO_SSH:-ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git}"
+DIR="$HOME/vanguard-stream"; KEY="$HOME/.ssh/vanguard-deploy"
+case "${1:-}" in
+  key)
+    [[ -f $KEY ]] || ssh-keygen -t ed25519 -N "" -C "vanguard-deploy@$(hostname)" -f "$KEY" >/dev/null
+    cat > "$HOME/.ssh/config.vanguard" <<CFG
+Host git.dumpstervigilantes.rip
+  Port 2222
+  User git
+  IdentityFile $KEY
+  IdentitiesOnly yes
+CFG
+    grep -q 'config.vanguard' "$HOME/.ssh/config" 2>/dev/null || { echo "Include config.vanguard" | cat - "$HOME/.ssh/config" 2>/dev/null > "$HOME/.ssh/config.new" || echo "Include config.vanguard" > "$HOME/.ssh/config.new"; mv "$HOME/.ssh/config.new" "$HOME/.ssh/config"; chmod 600 "$HOME/.ssh/config"; }
+    echo "Add this as a READ-ONLY deploy key on the repo (Gitea → repo → Settings → Deploy Keys):"; echo; cat "$KEY.pub" ;;
+  clone)
+    [[ -d $DIR/.git ]] && { echo "$DIR is already a clone"; exit 0; }
+    ssh -o StrictHostKeyChecking=accept-new -p 2222 git@git.dumpstervigilantes.rip 2>&1 | grep -qi "successfully authenticated\|Hi there" || { echo "deploy key not accepted yet — run './switchToClone.sh key' and add it in Gitea"; exit 1; }
+    mv "$DIR" "$DIR.old"
+    git clone --quiet "$REPO_SSH" "$DIR"
+    for f in .env Caddyfile mediamtx.yml copyparty.conf stream-token.secret deploy.env; do [[ -f $DIR.old/$f ]] && cp -a "$DIR.old/$f" "$DIR/$f"; done
+    mkdir -p "$DIR/jellyfin-plugin-vanguard/dist"; cp -a "$DIR.old"/jellyfin-plugin-vanguard/dist/*.zip "$DIR/jellyfin-plugin-vanguard/dist/" 2>/dev/null || true
+    ( cd "$DIR" && sudo -n true 2>/dev/null && sudo install -m 0600 -o root -g root .env /dev/null >/dev/null 2>&1 || true )
+    echo "cloned. kept: .env, generated configs, plugin zips. Old copy at $DIR.old (delete when happy)."
+    echo "  (the Discord bot is not in this repo: its release lives in /srv/bot/app and is untouched by this)"
+    echo "next: cd $DIR && ./configure.py --apply   (records nothing new; just confirms generation works from the clone)"
+    echo "      then: sudo vanguard-deploy status" ;;
+  *) sed -n '2,5p' "$0"; exit 2 ;;
+esac

syncthing/stignore

@@ -1,5 +1,5 @@
 // Server-side ignore list for a clips receive folder (clips/<user>): pull video only.
-// .stlocal is appended to by the sweep: raws it converted (then deleted locally) and the standard
+// .stlocal must exist (created empty with the folder) and is appended to by the sweep: raws it converted (then deleted locally) and the standard
 // files it produced. Ignored = not tracked, not flagged, never re-pulled, never sent back.
 #include .stlocal
 // Ignored files are never downloaded from the sender. Subfolders are still traversed

systemd/deploy-pending.service

@@ -0,0 +1,7 @@
+[Unit]
+Description=Restart MediaMTX if a deploy deferred it and no stream is live
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/vanguard-deploy pending

systemd/deploy-pending.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Retry deferred MediaMTX restart every 5 minutes
+
+[Timer]
+OnBootSec=3min
+OnUnitActiveSec=5min
+
+[Install]
+WantedBy=timers.target

systemd/dumpd.service

@@ -9,6 +9,8 @@ Environment=DUMP_ADMIN_GROUPS=admins,media-admins
 Environment=PUBLISH_GROUPS=streamers,admins,media-admins
 Environment=STREAM_TOKEN_TTL_HOURS=12
 Environment=STREAM_TOKEN_SECRET_FILE=/etc/vanguard/stream-token.secret
+Environment=CLIPS_QUOTA_GB=100
+Environment=CLIPS_QUOTA_FILE=/etc/vanguard/clip-quotas.json
 Environment=DUMP_TZ=America/New_York
 Restart=always
 RestartSec=3

systemd/normalize-sweep.service

@@ -8,4 +8,5 @@ Type=oneshot
 ExecStart=/usr/local/bin/normalize-sweep.sh
 Nice=15
 IOSchedulingClass=idle
+Environment=ENCODE_THREADS=2   # x264 threads per conversion; keeps Jellyfin and the live portal responsive (default 2)
 # Environment=FILL_GAPS_MS=1000   # opt-in: CFR re-encode streams whose dropped-frame holes total >= 1 s (hours of CPU on this box)

systemd/vanguard-alerts.service

@@ -0,0 +1,9 @@
+[Unit]
+Description=Vanguard alerts: deliver Kibana's alert documents to Discord
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/vanguard-alerts.sh
+# Kibana cannot post outside the stack on a basic license, so its rules write into the vanguard-alerts
+# index and this is the delivery side. See bin/vanguard-alerts.sh and LOGGING.md.

systemd/vanguard-alerts.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Deliver Vanguard alerts every minute
+
+[Timer]
+OnBootSec=3min
+OnUnitActiveSec=1min
+AccuracySec=15s
+
+[Install]
+WantedBy=timers.target

systemd/vanguard-deadman.service

@@ -0,0 +1,7 @@
+[Unit]
+Description=Vanguard deadman: is the alerting stack itself alive, and are the disks ok
+After=docker.service
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/vanguard-deadman.sh

systemd/vanguard-deadman.timer

@@ -0,0 +1,10 @@
+[Unit]
+Description=Run the Vanguard deadman every 5 minutes
+
+[Timer]
+OnBootSec=2min
+OnUnitActiveSec=5min
+AccuracySec=30s
+
+[Install]
+WantedBy=timers.target