Media-Server-Setup/STATUS.md

# Vanguard stack — what runs, what people do, what's unfinished

Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-03.
Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `diag.` (logs/metrics/alerts), `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).

## Runs on its own

| Cadence | What happens | Done by |
|---|---|---|
| Continuous | OBS streams record into a hidden `streams/<user>/<Title>/.live/` folder, invisible to Jellyfin and copyparty until finished | MediaMTX |
| Continuous | Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only) | Syncthing |
| Continuous | New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab | Jellyfin plugin (needs bot token + channel in its config) |
| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
| Every 5 min | Discord scheduled events snapshotted into a history so recordings can be matched to their session | Jellyfin plugin |
| Every 30 min + startup | Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box) | Jellyfin plugin |
| Daily 04:10 ET | Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change | Jellyfin plugin |
| Every 5 min | Each user's clips folder is measured against their quota (100 GB default, per-user overrides); over quota → their Syncthing folder is paused, under 90 % → resumed; the clips page shows usage | dumpd quota loop — **pending deploy** |
| Daily 05:00 ET | Dumped nights older than 3 days deleted for good | trash purge (systemd timer) |
| On boot | Everything restarts itself after the media mount is up; interrupted recordings stay playable | systemd + Docker |
| Every 5 min | If a deploy deferred a MediaMTX restart, retry when no stream is live | `deploy-pending.timer` — **pending first apply** |
| On push to `main` | (OFF: `DEPLOY_AUTO=0`) Gitea webhook → dumpd → `vanguard-deploy pull` applies only what changed | **built, disabled** |
| Continuous | Active streams are watchable live (WebRTC, HLS fallback) by Discord-server members; optional "is live" Discord post when a stream starts | MediaMTX + live portal (page deployed; MediaMTX port fix pending apply) |
| **Not yet running** | Pull of files older than 15 days to the home HDD | `home/rclone-archive.sh`, needs the home box |
| **Not yet running** | Nightly backup of the Authentik database (Postgres) | planned |

## What people do

| Who | Activity | Where | Login |
|---|---|---|---|
| Streamer | One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode | OBS dock / Go Live | live |
| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://live.…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** |
| Gamer | Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://clips.…/` + Syncthing | Discord (any server member) |
| Viewer | Live portal lays streams out as an even grid automatically when one joins or leaves ("Arrange" button redoes it after manual moves); audio follows the focused window; double-click a tab to rename a stream (kept in that browser), double-click the slot at the left of a tab bar to name the group (session); once the viewer has moved/resized/renamed anything the layout is left alone: new streams chime and wait as draggable chips in a tray, "Arrange" re-grids and returns to auto | `https://live.…/` | Discord |
| Everyone | Landing page with role-aware links and instructions | `https://media.…/` | Discord |
| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep) | Resolve / Premiere | — |
| Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://files.…/` | Discord (Authentik); rights by group `editors` / `media-admins` |
| Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin `https://watch.…/` | Discord (Authentik) |
| Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal `https://live.…/` | Discord (Authentik, domain-level) — **live, shows a real stream** |
| Streamer (Windows) | **Go Live** from the browser: pick the WarDogs window (+audio on Windows Chrome 141+; Linux/Mac video only), streams as `<user>/HelmetCam-<date>`; no OBS needed | Live portal → Go Live | Discord; key fetched for the logged-in user — **verified end to end** (publish → `.live` → sweep → title folder) |
| Devs | Host code, review, CI | Gitea `https://git.…/`, ssh 2222 | Discord (Authentik), group `git-users`; `git-admins` = site admin |
| Bot dev | Push code, restart container, read logs; web UI at `https://bot.…/` | Gitea + compose; `bot/AGENT.md` | web UI behind Discord (Authentik), once its provider exists |
| Admin | Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups | `./configure.py` | ssh |
| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI `https://sync.…/` | Discord + group `media-admins` (Syncthing's own login disabled) |
| Admin | Groups, who may log in, app access | Authentik `https://auth.…/` | Discord or akadmin |
| Admin | Install/upgrade the Jellyfin plugin | `./installJellyfinPlugin.sh` | ssh |
| Admin | Register the CI runner | `./registerGiteaRunner.sh` | ssh |

## Services and their state

| Service | Role | State |
|---|---|---|
| MediaMTX | **Host network since 2026-11-05** (MediaMTX's recommended deployment for WebRTC). On the bridge, every WebRTC media packet on 8189/udp was relayed by Docker's userland proxy, which lost ~0.4% of them: each loss corrupted an access unit, the recorder reopened its file, and one evening produced 471 fragments; every publisher also appeared as 172.18.0.1. Ingest (1935, 8890/udp, 8189) now arrives through the nftables INPUT chain; API, HLS, WHIP/WHEP signalling and metrics bind 127.0.0.1 and are reached only through Caddy and Metricbeat. RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`); on-ready hook posts "is live" and runs the WebRTC audio mirror (`<path>.webrtc`, audio → Opus, never recorded) | live, token auth verified |
| copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (`editors`, `media-admins`), Discord-only, no WebDAV | live |
| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity (on `live.`), `/me/clips` Syncthing self-service (on `clips.`); OBS dock with timecode-band setup | identity from Authentik headers | live |
| `media.` gateway, `clips.` self-service | static pages behind the gate | — | live; **first real pairing 2026-11-01 (thrahg)** — exposed a missing `.stlocal` on fresh folders, fixed |
| Timecode | `bin/vgstrip.py` decoder, `tmcd` + sidecar in the sweep, `/dock/clock/` strip page, Go Live band composite, portal crop | server clock is the reference | **live, verified** (OBS: frame-accurate, 0 gaps) |
| Syncthing | clip intake, UI at `sync.` | UI behind Authentik + `media-admins`, own GUI auth off | live |
| normalize sweep | editing standard, one copy per clip | live |
| nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
| Observability (Phase 13) | **Elastic**: Elasticsearch (logs + metrics in one store), Kibana = the whole `diag.` host (Discover, dashboards, alerting), Filebeat, Metricbeat (also retires node_exporter + cAdvisor). Both Beats emit the same correlation fields so one dashboard spans logs and metrics | **deployed 2026-11-03.** The first build (VictoriaLogs/VictoriaMetrics/vmalert ×2/Alertmanager + 2 exporters = 8 containers) produced no dashboards, so it was replaced by 4 Elastic containers with Kibana as the single pane. Metricbeat shipped at once; Filebeat ran with zero harvesters because Filebeat 9 **removed** the `container` input (fixed: filestream + container parser). Cluster is green with replicas pinned to 0 (single node). Still to do: `ALERT_WEBHOOK_URL`, and three UI actions in Kibana (retention delete phase, the heartbeat rule the deadman watches, a webhook connector) |
| diag landing | `diag.…` opens on a "Vanguard ops" dashboard: a markdown panel of links to every host plus the stuck page, and shortcuts to Discover, Dashboards, Stack Rules and Index Management. Kibana still owns the root; `defaultRoute` just redirects there. Created by `kibana-setup.py landing`, so a rebuild recreates it | **added 2026-11-04** |
| Alerting | **Kibana evaluates, we deliver.** A basic licence has no webhook/Slack/email connector (403, Gold+), so rules write documents into the `vanguard-alerts` index through the free `.index` connector and `vanguard-alerts.timer` takes them to Discord every minute, with 15-minute per-rule repeat suppression. Alerts are therefore searchable in Discover beside the logs that caused them. Rules are still authored in Kibana's UI; `observability/kibana-setup.py` creates the connector and the heartbeat rule | **added 2026-11-03** |
| Alarm rules | Six, created by `observability/kibana-alarms.py`: disk and memory below 15% free, bot not running, bot unhealthy, recordings stuck over an hour, normalise failures parked. Each repeats every 12 h while unresolved and stops when it clears | **added 2026-11-03**; needs the bot release that fixes `/health` before the bot-unhealthy rule means anything |
| Deadman | `vanguard-deadman.timer` on the host, every 5 min, curl only: Elasticsearch answering and not red, no index read-only from the watermark (never self-clears), Kibana available, **each Beat actually writing** (running ≠ shipping), the Kibana heartbeat fresh, and the disks. Reports on change; journal-only and non-zero exit when no webhook is set | **live 2026-11-03** — and it earned itself on the first run by catching Filebeat up-but-not-shipping, which every process check called healthy |
| Fragment stitching | A stream that dropped and came back is one recording in several `.live/` files. `bin/vgstitch.py` (run first by every sweep) joins them losslessly with the concat demuxer when there are fewer than `STITCH_MAX_PARTS` (500; was 20 until Kyle's lossy night showed a recorder restart every ~10 s is normal for 0.5% loss) and the gaps are under `STITCH_MAX_GAP_S` (10 min); the arrival-time file names place every piece. Joins are collapsed and recorded in the sidecar (`stitched.parts/joins`, `missing_ms`). Nothing in a stream's `.live/` is touched until no file there is open or fresh (`event=defer` / `event=stitch_wait`). A resolution/codec change breaks the chain (copy-concat is impossible across it), so Go Live now holds resolution and sheds frame rate instead. The fragments go to `trash/<night>/` for the 3-day undo; a failed stitch leaves them alone and the old per-file path publishes them | **added 2026-11-05** |
| Reconnect fragments | Every publisher drop-and-reconnect opened a new MediaMTX recording; one bad night left **471** one- and two-second files in `.live/`, mostly from the 6 Mbps Go Live floor (fixed). Cleanup: `vgstuck.sh drop --under 5` or the page's bulk button, admin-only, confirm shows the count. Prevention: fragments are stitched first (row above); a **lone** stream recording under `MIN_LIVE_SECONDS` (10) goes to trash instead of being published. Clips are never subject to either: they are a few seconds long on purpose and never pass through `.live/` | **added 2026-11-04** |
| Login branding + Discord forward | Done from the ORM by `configure.py` on every apply (`--ak-branding` runs just this): brand title/logo/favicon/background from `authentik/brand/*.svg` (installed into Authentik's own `/media/public/brand/`, the only unauthenticated place that can serve them), login-card CSS in the site palette with Discord blurple, brand default application = the media gateway (so `auth.` root never shows Authentik's app library), flow titles renamed, Discord attached to the identification stage. Then the forward: a **redirect stage first in `default-authentication-flow`** whose static target is the Discord source's login URL, bound to an expression policy that skips it when the flow URL carries `?admin=1` (the executor keeps the flow query in the session key `authentik/flows/get`, not `request.GET`). Members never see an Authentik page: protected link → Discord consent → back where they were (the outpost's `next` rides the same session key). akadmin's password card is `https://auth.BASE/if/flow/default-authentication-flow/?admin=1`, linked only from the diag landing dashboard | **added 2026-11-06** |
| Frozen Go Live picture | Breezy: one frame, then one more per alt-tab. Window capture on Windows comes from the compositor and an **exclusive full-screen** game bypasses it, so Chrome only gets a frame when Windows composites one (alt-tab). Go Live now has a capture choice, window (default) or **whole screen** (`displaySurface: monitor`, desktop duplication: sees exclusive full-screen, carries system audio), remembered per browser, plus a hint naming the symptom and the better fix: the game in borderless/windowed full-screen keeps window audio and needs no change | **added 2026-11-06** |
| Base-domain move, no outage | `configure.py --alias NEW` makes every Caddy site answer on both bases (`media.old, media.new {`), certificates issued, nothing else touched; apps and Authentik are then moved one by one while the old names keep working; `--ak-hosts OLD NEW` (dry-run first) moves Authentik last; `--base NEW` flips the primary, recreates the host-bearing containers, and turns the old names into 308 redirects (`REDIRECT_BASE`). `read_hosts` understands both Caddyfile forms; the sweep probes aliases and old-name redirects. Rule documented at the top of HOSTNAME-CHANGE.md: apps before Authentik, never the reverse | **added 2026-11-06** |
| Hostname sweep | `bin/vghosts-check.py` (installed; `sudo vghosts-check.py` on the box, `--repo` on a dev box, `--json`): validates every place HOSTNAME-CHANGE.md lists against the Caddyfile — DNS, `.env`, `mediamtx.yml`, the env each running container was created with, HTTPS + gate redirects per host, Authentik (outpost host, proxy external host + cookie domain, Jellyfin/Gitea redirect URIs, the exact Discord callback to check by eye), Jellyfin config XML + live SSO start, Gitea `app.ini` + live auth-source redirect + runner + clone remote, repo defaults, about-page labels, docs with browser links to `ovh1`. Sections run in the order a change must be made (documented in HOSTNAME-CHANGE.md §E) because each depends on the previous; exit 1 on any failure | **added 2026-11-06** |
| Onboarding checklist | `media.BASE/onboarding/` (`about/onboarding/index.html`, served by the gateway's file server, so behind the same Discord gate). For the person **doing** the onboarding: eight topics to tick as they are covered — who we are, about/verify, community, who Vanguard is, who Spartan are, comms, training, push to mute/talk — and a Clear button for the next recruit. Ticks are per browser. The gateway's `try_files` gained `{path}/index.html` so folder pages under `about/` resolve without a trailing slash | **added 2026-11-06** |
| Disk page | `diag.BASE/disk`, its own page (`streaming/disk.html`, served from the same mount): filesystem bars, then recordings per **streamer by month** (month from the recording's own `__YYYY-MM-DD` stamp; raw `.src` and unpublished `.live` called out), **clips per clipper by month**, trash by night, **logs** (Elasticsearch indices by the date in their names, the ES data dir as the on-disk truth, journald, Docker json logs) and **system** (each `appdata/<app>`, live portal, bot, Docker's own image/volume figures). dumpd walks the tree with allocated-block sizes, caches ten minutes, `?refresh=1` / "Rewalk now" forces | **added 2026-11-06** |
| Stop a stream (Live tab) | Each publisher row has **Stop stream** with a block length (kick only / 30 min / 2 h / 8 h, default 30 min). dumpd kicks the session through MediaMTX's API (`/v3/<webrtcsessions|rtmpconns|srtconns>/kick/<id>`) **and** writes the account into `.markers/publish-blocks.json`; `/mtx-auth` denies a blocked account on every publish, so OBS's auto-reconnect and a Go Live restart both get "publishing is paused for this account by an admin" until it expires or an admin presses Allow again in the new "Blocked from publishing" section. The block is written before the kick, so there is no reconnect window. Admin groups only; the recording so far is kept and the sweep publishes it as usual | **added 2026-11-06** |
| Transport on the Live tab | MediaMTX's session API labels the selected ICE pair `host/udp/…` or `host/tcp/…`, so the publishers table now shows `webrtc udp` / `webrtc tcp` and the session start time, and a new "Publish sessions — last 24 h" table lists every WebRTC publish session with transport, start, end, duration, received/lost, and a "switched from udp" marker on the session that began the TCP leg: that is the when and the before/after loss of a lossy-uplink switch. dumpd keeps the history in `.markers/webrtc-history.json`, updated on every Live-tab poll and every metrics scrape, so it fills in while nobody is watching | **added 2026-11-06** |
| Lossy uplinks (Go Live) | Kyle's night: one publish session, 16 recorder restarts, 165 processing errors, 0.5% loss. A loss burst that eats a keyframe's slices leaves MediaMTX's recorder an access unit with only SPS/PPS, it refuses it (`access unit doesn't contain an IDR or non-IDR NALU`) and the path restarts the recorder 2 s later = a new file every ~10 s. Three levers pulled: (1) retransmits — MediaMTX's WebRTC code calls pion `ConfigureNack` for every peer connection, so NACK is on; the Go Live stats line now shows `loss x% (n nack, m resent)` so a stream where NACKs stay at 0 under loss is visible at a glance; (2) the stitcher cap went 20 → 500 so such a night is one recording; (3) **automatic TCP**: Go Live samples the server's receiver reports every second and when loss over the last 20 s exceeds 0.3% (after at least 1500 packets) it ends the WHIP session and re-publishes the same tracks with only MediaMTX's TCP candidates (`webrtcLocalTCPAddress :8189`) left in the answer. One extra recording file (stitched), ~2 s of picture lost in the swap, then no loss can reach the recorder; latency is what TCP costs instead. It never flips back during a stream; the stats line says `via TCP` and why. Kyle is on **Starlink**, whose scheduler reshuffles every ~15 s and drops a burst at each handoff — that is the every-10-to-15-s restart cadence in his log, and no NACK can cover a burst that long. So the switch is also **remembered per browser** (`vg.golive.tcp`): once it has fired, the next stream starts on TCP, no 20 s of corrupted recording and no extra file; a "Lossy uplink" checkbox in Go Live's setup shows and controls it. First TCP night: stable, one file, zero recorder restarts — but soft: with no loss to react to, Chrome's estimator only sees the dish's delay swings and parks near the 1.2 Mbps floor. TCP publishes now use a **4 Mbps floor** (`TCP_MIN_KBPS`): safe there because the UDP failure mode of a floor (sending into loss) cannot happen on TCP, the worst case is added latency | **added 2026-11-06** |
| B-frame enforcement | obs-websocket cannot set encoder options in Advanced output mode, but it can switch **profiles**, and encoder settings live in a profile. The dock has an "OBS profile to stream from" setting (default `Vanguard`) and switches OBS to it before every Go Live, plus a checkbox to do so when the dock loads, i.e. on OBS startup. A streamer sets the profile up once with B-frames 0; the post-Go-Live probe verifies the result. Missing profile = a red status line naming what to create, never a silent fallback | **added 2026-11-05** |
| B-frame detection | MediaMTX refuses H.264 with B-frames over WebRTC, and nothing on the publish path can see it: obs-websocket cannot set encoder options in OBS's Advanced output mode, and MediaMTX only notices when a viewer connects. dumpd now reads the B-frame depth off the first fragments of the live recording (`ffprobe`, small probesize, ~50 ms, works on a growing file, cached 30 s). The **dock** polls `/me/stream-health` after Go Live and shows a red banner with per-encoder instructions; in Simple output mode with x264 it writes `bf=0` into the options itself first. The **Live tab** shows an encode column per publisher: "no B-frames" or "N B-frames · WebRTC refused" | **added 2026-11-05** |
| Streaming page | `https://diag.…/streaming`, two tabs. **Live**: every publisher with its real address, packets received and lost on the way in, the last ten minutes of publish sessions / recorder restarts / processing errors, and a one-line verdict (loss, reconnects, or parameter changes) with the recorder log tail. **Stuck**: the triage below. Served by dumpd `/streaming/api/live` and `/streaming/api/stuck`; `/stuck` redirects | **added 2026-11-05** |
| Dump a night (web) | Third tab of the streaming page on the triage host: the night's recordings and clips grouped by streamer with sizes, a date picker (defaults to dumpd's "tonight, else last night"), a Dump button per streamer and a Dump everyone button for admins. Two confirmations, then a POST to `/streaming/api/dump` on this host, so one cookie covers it. The dock's own Dump tonight button stays for a streamer's own night and is personal for everyone, admins included: dumpd honours a `user`/`all` only when the request came in on `/streaming/api/dump`, so the only place another streamer's night can be dumped is this tab. GET is a dry run; only the POST moves files, to `trash/<night>/` with the 3-day undo. Below the night table: **Bin all unprocessed**, which runs `vgstuck.sh wipe` — stops the sweep (killing any conversion in flight; the timer restarts it to an empty queue), then every `.live/` file from any night, parked or fresh, plus the failure logs and `.part` leftovers, then the emptied folders. The only thing it leaves is a file still held open after the sweep is down, i.e. a stream MediaMTX is recording this minute. `drop --all` stays refused so a pattern can never do this by accident | **added 2026-11-05** |
| Stuck recordings | The Stuck tab of the page above for anyone past the gate, on the triage host (already `media-admins` only), linked from the gateway's Media admin card and from the diag landing dashboard: every unpublished recording, why it is stuck, and Retry / Bin buttons. dumpd serves `/streaming/api/stuck` and allows the actions only for the admin groups; both the page and the CLI run `bin/vgstuck.sh`, so they can never disagree. `bin/vgstuck.sh` triages anything that never published: it says why each one is stuck (`parked` by a failure log with ffmpeg's reason, `published` collision, `waiting`, or still `writing`), then `retry <match>` clears the blocker and `drop <match>` moves it to trash with the usual 3-day undo. `drop --all` is refused. The sweep also logs `event=parked` once per run so Kibana can name them, not just count them | **added 2026-11-04** |
| Our own metrics | `bin/vgmetrics.sh` counters from the sweep and vgdeband + live gauges from dumpd `GET /metrics` (loopback, unrouted by Caddy), scraped by Metricbeat. Answers what MediaMTX and the system module cannot: normalise queue depth and its oldest item, encodes in flight, ffmpeg seconds by verdict and by stream/clip, failures parked, live stream count with the `.webrtc` mirrors separated. `vanguard_media_pending` trending up across a night is the measured signal that CPU-only encoding is not keeping pace | **added 2026-11-03** |
| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`). Consumed by Filebeat into Elasticsearch | live (labels + logfmt); Elastic deploy pending |
| Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
| Jellyfin | watch page, SSO via Authentik, direct play only | live |
| Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `<Title>__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |
| MariaDB | databases for the bot and Gitea; public :3306, root local-only | live |
| Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live; Discord login via Authentik OIDC **verified**; access = `git-users`/`git-admins` groups; local admin as fallback |
| act_runner | CI jobs in Docker | **not registered** |
| `bot.` host + Discord bot | the bot's API **and** its web pages on one port (`WEB_SERVE_UI`), behind Authentik forward-auth; Caddy requires `bot-users`, the bot re-checks `bot-users,admins` per request | a push to `main` deploys itself: CI builds a **release bundle** (app + `vendor/` dependencies) and posts it signed to dumpd `POST /deploy-bot` (bridge-only port, no Docker socket in CI) together with the bot's own secrets, which are managed in Gitea; dumpd verifies everything, writes the keys `BOT_ENV_KEYS` allows into `.env` and swaps the bundle into `/srv/bot/app` → `/app`; image installs no bot dependencies | **reviewed + wired 2026-11-02, first install pending** (needs the Authentik app for `bot.`, and `BOT_DEPLOY_SECRET` + `DISCORD_BOT_TOKEN` + `DISCORD_GUILD_ID` set in Gitea) |
| Live portal (`live/`, `live.` host) | React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd `/me/stream-key` | live; Go Live verified |

## Unfinished, in rough priority

0a. **WebRTC audio** (found 2026-11-01 with the 5-stream test): AAC cannot cross WebRTC, so the portal was silent unless it fell back to HLS. Fix committed: audio mirror in `bin/mtx-on-ready.sh` + portal mapping. **Deploy pending** (MediaMTX recreate between streams; dumpd restart; portal push). Streamers must set B-frames 0 for WebRTC playback.
0a. **Discord bot**: live on MariaDB since 2026-11-03 (139 members tracked, 36 commands synced, web UI on the `bot.` host).
   Old SQLite history merged in the same night with `bin/vgbot-merge-sqlite.py` (the bot's own importer refuses a
   non-empty destination): Cool Points, training, invites and growth restored, original joined/first-seen/rank dates
   recovered from the rebuild, live rank and activity kept. Pre-merge dump at `/srv/appdata/discord-bot/before-merge.sql`.
   The tool is idempotent, so re-running it is safe.
0. **Band removal**: deployed 2026-11-01; `vgdeband.sh` catch-up ran on the Vanguard-Training recordings (check `journalctl -u vgdeband` for the done lines). Live portal: viewer now clips the band with a measured game-aspect frame (refits on every dockview resize/move) — committed 2026-11-01, deploy with `./deployLive.sh`.

1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (`/whoami` should show `bot-users`). Nested groups **verified 2026-11-01**: `git-admins` now reaches the header via the `Vanguard profile (nested groups)` scope mapping (README 5e). `akadmin` remains the recovery login if a policy misfires.
2. Confirm plugin 0.1.3 under Jellyfin → Plugins and the Gitea runner registered.
3. Portal viewer during a real stream: corner chip `webrtc`; Windows Go Live with WarDogs audio (first Windows streamer).
5. Housekeeping: pin `AUTHENTIK_TAG`, rotate the Discord client secret seen in a screenshot, nightly Postgres dump, Caddyfile formatting warning.
6. Medal filename pattern in the plugin (need a sample filename).
7. Home archive puller on the home box.
8. Plugin repository served by Caddy so Jellyfin updates from its catalog.
9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run
   (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served
   from the bot itself now; an Authentik username is matched to one Discord member by name.
9a. Kibana first-run: a data view over `filebeat-*,metricbeat-*`, the ILM delete phase, a webhook connector + the always-firing heartbeat rule, then the old alert conditions as rules (disk, load, restart loops, sweep failures, publish-deny bursts, failed bot releases, certificate trouble). Dashboard panels for the new `vanguard_media_*` metrics.
9b. Elasticsearch security as its own phase, if `diag.` is ever to widen past `media-admins`: turn security on, credit both Beats + Kibana + the deadman + dumpd, and have Caddy inject a read-only or admin API key chosen by Authentik group (Kibana's HTTP auth provider accepts a proxy-injected `Authorization`). Per-person identity is reachable but buys little, since audit logging is itself paid.
10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.

## Verified behaviours worth remembering
- **Never scp generated configs** (`mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt`, `.env`): a template copy wiped all streamers once. They are gitignored now; `*.example.*` are reference only.
- Caddy runs in a container: anything it serves from disk must be a mount (`/srv/live` was missing → 404).
- Go Live quality depends on the browser's encoder: Windows Chrome uses the GPU (fine); Linux Chromium on NVIDIA is software H.264 Baseline (soft at any bitrate, CPU-limited at 1440p60) and cannot be fixed by flags — Linux streamers use OBS. The stats strip shows encoder + profile.
- Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as `<Title>__<ts>.mp4` in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
- Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. **Gap filling is available** (`FILL_GAPS_MS` in `normalize-sweep.service`, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
- The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
- **An encode that fails with `Error opening output files: Invalid argument` is usually contradictory flags, not the path.** ffmpeg rejects `-r` together with a non-CFR `-fps_mode`; the real message is one line earlier in its stderr, which the sweep now captures into the failure log. Found 2026-11-03 on the 1 November test recordings, which also exposed that `avg_frame_rate` can be pure fiction (95 fps reported for a 30 fps source) — the sweep now takes the lower of r_frame_rate and avg_frame_rate.
- A deploy block that is only half-run leaves old binaries in `/usr/local/bin`; `grep -c` the installed script before blaming logic.
- MediaMTX does not reload `mediamtx.yml` live here (bind-mounted file): any change to it needs `docker compose restart mediamtx` between streams. Nothing day-to-day touches that file.
- **Timecode verified live (2026-10-31)**: OBS + NVENC with the band → `source: strip`, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path.
- MediaMTX stamps recording filenames in its container clock: without `TZ` it is UTC and the plugin reads names as Eastern (4 h off). `TZ=America/New_York` is set on the container.
- MediaMTX 1.21 refuses to start without `webrtcAdditionalHosts` or `webrtcIPsFromInterfaces`; configure.py writes the public host in.
- Jellyfin honours provider-supplied season numbers on a Shows library (day seasons appeared). Next check: Monday 04:10 ET should fold them into a week season.
- A stream indexed by Jellyfin while still recording has no duration and skips to the next episode on seek — fixed by recording into `.live/` so nothing sees a file until it is finished and flat.
- Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
- Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
- Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.
- **"Enforce B-frames 0 through obs-websocket" is only possible in Simple output mode with x264.** Advanced-mode encoder settings live in `streamEncoder.json`, which the protocol does not expose. The enforceable thing is detection on the server (the recording reveals the reorder depth within seconds) with a loud dock banner, which covers every encoder. Found 2026-11-05 when an OBS stream was unwatchable with `has_b_frames=1`.
- **A published UDP port on the Docker bridge goes through `docker-proxy`, a userland relay, and it drops packets under load.** The sign is every client appearing as the bridge gateway (`172.18.0.1`) in MediaMTX's metrics. For WebRTC media at ~1000 packets/s this produced steady ~0.4% loss that no browser-side setting could fix, because the loss was on our box. Anything doing real-time UDP belongs on `network_mode: host`, which is also what MediaMTX documents. Found 2026-11-05 after chasing it as a client bitrate problem first.
- **Two MediaMTX path regexes that can both match a name is a coin toss, not "first rule wins".** The mirror rule (`…\.webrtc$`) and the generic rule both matched `Techy_multi_3.webrtc` because titles were allowed dots, and the generic one won: a **36 GB** audio mirror was recorded as a full duplicate of its source. Both rules now have a dot-free title segment, so a name satisfies exactly one, and titles no longer accept dots anywhere they are typed. Needs a MediaMTX recreate between streams. Found 2026-11-04.
- **Never `ffprobe` fragmented MP4 in a request path.** `.live/` recordings have no duration in their header, so ffprobe reads the whole file to answer. The stuck-recordings survey did that per file, took over three minutes for 471 of them, dumpd's 180 s request ceiling killed the response, Caddy returned an empty 502 and the page said "unexpected end of JSON". Size and age come from `stat`; duration is opt-in (`vgstuck.sh list --durations`). Found 2026-11-04.
- **A high `x-google-min-bitrate` disables WebRTC congestion control.** Go Live sent `min-bitrate=6000`, so Chrome never encoded below 6 Mbps; on an uplink that could not carry it, the excess was dropped, MediaMTX logged `invalid FU-A packet (non-starting)` and every viewer saw blocky corruption. The loss was on the PUBLISH session (4761 packets for one streamer) while the viewer sessions lost nothing, which is the fingerprint: the server relays damage it received. Floor lowered to 1200 and `scaleResolutionDownBy` unpinned so the encoder can shed resolution. Found 2026-11-04.
- `tracks=['Opus','H264']` on a path means a browser (Go Live); `MPEG-4 Audio` means OBS. Quick way to tell who is publishing how.
- **`handle_path` strips the prefix, so an upstream's own redirects lose it.** MediaMTX redirects every HLS playlist request once (to `?cookieCheck=1`) with a root-relative `Location`, so the browser followed `/<user>/<Title>/index.m3u8` into the portal's SPA catch-all, got `index.html`, and hls.js said `manifestParsingError` — a web page where a manifest should be. Fixed with `header_down Location ^/ /hls/`. `curl` against MediaMTX directly hides this completely, because there is no prefix to lose. Found 2026-11-04.
- **Go Live audio only exists on Windows Chrome 141+, and only if the share-audio box is ticked** in the window picker. Linux Chromium offers no window audio at all, so Linux streamers use OBS. The portal shows "no audio captured" at the time, and a published recording with only a video and a `tmcd` data stream is the fingerprint. First working Windows audio: 2026-11-03.
- The sweep's band-removal branch **overwrote `$reason`**, which carried the probe's `audio=<codec>` marker — the only signal that the audio needs converting. A Go Live stream with both the band and Opus audio therefore got `-c:a copy` and shipped Opus inside an MP4. Fixed by keeping `probe_reason` separately; it only became reachable once Go Live audio worked.
- **Authentik has two invalidation flows and the names are a trap.** `default-provider-invalidation-flow` is called "Logged out of application" and ends the session for that one application only, leaving the SSO session alive — so `/logout` drops the proxy cookie, forward-auth asks again, Authentik still knows you, and you are silently signed back in. `default-invalidation-flow` ("Logout") is what Authentik's own logout button uses and is the one that actually ends the session. The proxy provider must point at the latter. Set 2026-11-03.
- Prefer changing a proxy provider through the ORM (`ak shell`) over the UI: saving it in the UI re-adds the managed property mappings every time.
- **Filebeat 9 removed the `container` input** (the message still says "deprecated"). It refuses the config per container and then runs happily with no harvesters and no index: up, healthy, shipping nothing. Container logs need `filestream` + `parsers: [container: ~]` and a stable `id` each. `add_docker_metadata` also needs `match_fields: ["container.id"]` here, or there are no `container.labels.*` and the `service.name`/`vanguard.category` promotion yields empty fields.
- A single-node cluster cannot place a replica, so beats indices default to one and hold it yellow forever. Both beats set `setup.template.settings.index.number_of_replicas: 0`.
- **`VAR=value . file` does not work in bash**: assignments prefixed to the `.` builtin are discarded when source returns, so the sourced file's own `${VAR:-default}` writes to a binding that vanishes. Assign on its own line. Caught by a test, not by reading.
- `find ... | while read` runs the loop in a **subshell**, so anything it accumulates is lost at the end of the pipe. `normalize-sweep.sh` reads from a process substitution for exactly this reason.
- Kibana cannot see the Authentik identity headers: Elasticsearch has no proxy-header realm, and the OIDC path that would reuse Authentik is a paid feature (Platinum, closed to new customers). With `xpack.security.enabled: false` every visitor is the same unnamed superuser, so the Caddy gate group IS the access model on `diag.`. Anything in Kibana that asks Elasticsearch for privileges fails with `no handler found for uri [/_security/user/_has_privileges]`. The **Infrastructure / Metrics app does this on every page view**, so it is unusable here and is hidden in the default space along with APM, Synthetics, ML, Fleet, the search and security products and cases. Metrics views are built as dashboards over `metricbeat-*` instead, which never touch that endpoint. Discover, Dashboard, Lens, Logs, Dev Tools, data views, alerting connectors and Stack Rules all work.