# Vanguard stack — what runs, what people do, what's unfinished

Living document. Updated whenever something is deployed or changes behaviour. Last update: 2026-11-01.
Box: OVH SYS-1, Debian 13, ssh `ovh1.dumpstervigilantes.rip`. Web hosts (one Discord login): `media.` gateway at `/`, `files.` copyparty, `clips.`, `sync.`, `live.` (portal + `/dock/`, `/dump`, `/time`), `watch.`, `git.`, `bot.`, `auth.` — all under `dumpstervigilantes.rip`. Restructure deployed 2026-11-01 (`ovh1` is ssh-only now). Bundle: `~/vanguard-stream` on the box.
Source of truth: this repo, `techy/Media-Server-Setup` on the stack's own Gitea (`ssh://git@git.dumpstervigilantes.rip:2222/techy/Media-Server-Setup.git`).

## Runs on its own

| Cadence | What happens | Done by |
|---|---|---|
| Continuous | OBS streams record into a hidden `streams/<user>/<Title>/.live/` folder, invisible to Jellyfin and copyparty until finished | MediaMTX |
| Continuous | Gamers' clip folders mirror onto the server (video files only, their structure untouched, receive-only) | Syncthing |
| Continuous | New recordings and clips are announced in a Discord channel with the streamer's name and a frame grab | Jellyfin plugin (needs bot token + channel in its config) |
| Every 5 min | Finished files are checked. Streams idle ≥3 min: timecode band decoded (server clock) or arrival time used, then published out of `<Title>/.live/` to `<user>/<Title>__<ts>.mp4` with a frame-accurate `tmcd` track + sidecar — band present: re-encode with the band cropped off (hours of CPU per night, niced); no band: lossless remux — one file per recording (this is when they appear in Jellyfin/copyparty and the Discord notice fires). Non-compliant clips: converted to the editing standard, raw removed, one copy per clip | normalize sweep (systemd timer) |
| Every 5 min | Discord scheduled events snapshotted into a history so recordings can be matched to their session | Jellyfin plugin |
| Every 30 min + startup | Every non-admin Jellyfin user set to direct-play-only (no hardware encoder on this box) | Jellyfin plugin |
| Daily 04:10 ET | Recordings roll between weekday → week → month → year seasons as the calendar moves; no files change | Jellyfin plugin |
| Every 5 min | Each user's clips folder is measured against their quota (100 GB default, per-user overrides); over quota → their Syncthing folder is paused, under 90 % → resumed; the clips page shows usage | dumpd quota loop — **pending deploy** |
| Daily 05:00 ET | Dumped nights older than 3 days deleted for good | trash purge (systemd timer) |
| On boot | Everything restarts itself after the media mount is up; interrupted recordings stay playable | systemd + Docker |
| Every 5 min | If a deploy deferred a MediaMTX restart, retry when no stream is live | `deploy-pending.timer` — **pending first apply** |
| On push to `main` | (OFF: `DEPLOY_AUTO=0`) Gitea webhook → dumpd → `vanguard-deploy pull` applies only what changed | **built, disabled** |
| Continuous | Active streams are watchable live (WebRTC, HLS fallback) by Discord-server members; optional "is live" Discord post when a stream starts | MediaMTX + live portal (page deployed; MediaMTX port fix pending apply) |
| **Not yet running** | Pull of files older than 15 days to the home HDD | `home/rclone-archive.sh`, needs the home box |
| **Not yet running** | Nightly backup of the Authentik database (Postgres) | planned |

## What people do

| Who | Activity | Where | Login |
|---|---|---|---|
| Streamer | One-time: dock → Timecode band → "Add band + clock source" (OBS) or leave the Go Live checkbox on — gives editors frame-accurate timecode | OBS dock / Go Live | live |
| Streamer | Type a title (live preview + naming rules), Go Live, Stop. "Dump tonight" if it was junk (3-day undo). Manual-setup accordion with endpoints and a hidden token | OBS browser dock `https://live.…/dock/` | Discord login in the dock; dock mints a 12 h token per Go Live — **no passwords, verified live** |
| Gamer | Self-serve pairing (paste Syncthing Device ID → folder + share created); then nothing, clips save where they always did | `https://clips.…/` + Syncthing | Discord (any server member) |
| Viewer | Live portal lays streams out as an even grid automatically when one joins or leaves ("Arrange" button redoes it after manual moves); audio follows the focused window; double-click a tab to rename a stream (kept in that browser), double-click the slot at the left of a tab bar to name the group (session); once the viewer has moved/resized/renamed anything the layout is left alone: new streams chime and wait as draggable chips in a tray, "Arrange" re-grids and returns to auto | `https://live.…/` | Discord |
| Everyone | Landing page with role-aware links and instructions | `https://media.…/` | Discord |
| Editor | Multicam: drop clips on a timeline, sync by the embedded timecode track (frame-accurate when the band was on), cross-check with voice audio; files are already at game resolution (band removed by the sweep) | Resolve / Premiere | — |
| Editor | Browse by streamer/title, scrub in browser, download originals (zip/tar for folders), upload finals | copyparty `https://files.…/` | Discord (Authentik); rights by group `editors` / `media-admins` |
| Anyone in the Discord server | Watch recordings and clips by streamer and season | Jellyfin `https://watch.…/` | Discord (Authentik) |
| Anyone in the Discord server | Watch live: all active streams in dockable windows, audio follows focus, per-window mute/volume | Live portal `https://live.…/` | Discord (Authentik, domain-level) — **live, shows a real stream** |
| Streamer (Windows) | **Go Live** from the browser: pick the WarDogs window (+audio on Windows Chrome 141+; Linux/Mac video only), streams as `<user>/HelmetCam-<date>`; no OBS needed | Live portal → Go Live | Discord; key fetched for the logged-in user — **verified end to end** (publish → `.live` → sweep → title folder) |
| Devs | Host code, review, CI | Gitea `https://git.…/`, ssh 2222 | Discord (Authentik), group `git-users`; `git-admins` = site admin |
| Bot dev | Push code, restart container, read logs; web UI at `https://bot.…/` | Gitea + compose; `bot/AGENT.md` | web UI behind Discord (Authentik), once its provider exists |
| Admin | Hostnames, bot token/guild, webhook; streamers/editors are Discord roles → Authentik groups | `./configure.py` | ssh |
| Admin | Watch pairings / folder states (gamers self-serve the setup) | Syncthing UI `https://sync.…/` | Discord + group `media-admins` (Syncthing's own login disabled) |
| Admin | Groups, who may log in, app access | Authentik `https://auth.…/` | Discord or akadmin |
| Admin | Install/upgrade the Jellyfin plugin | `./installJellyfinPlugin.sh` | ssh |
| Admin | Register the CI runner | `./registerGiteaRunner.sh` | ssh |

## Services and their state

| Service | Role | State |
|---|---|---|
| MediaMTX | RTMP/SRT ingest, records fMP4; publish auth delegated to dumpd (tokens via `/mtx-auth`); on-ready hook posts "is live" and runs the WebRTC audio mirror (`<path>.webrtc`, audio → Opus, never recorded) | live, token auth verified |
| copyparty | browse/scrub/download/finals upload | behind Authentik, group permissions (`editors`, `media-admins`), Discord-only, no WebDAV | live |
| dumpd + dock | dump, stream tokens, MediaMTX auth hook, `/time`, `/me` identity (on `live.`), `/me/clips` Syncthing self-service (on `clips.`); OBS dock with timecode-band setup | identity from Authentik headers | live |
| `media.` gateway, `clips.` self-service | static pages behind the gate | — | live; **first real pairing 2026-11-01 (thrahg)** — exposed a missing `.stlocal` on fresh folders, fixed |
| Timecode | `bin/vgstrip.py` decoder, `tmcd` + sidecar in the sweep, `/dock/clock/` strip page, Go Live band composite, portal crop | server clock is the reference | **live, verified** (OBS: frame-accurate, 0 gaps) |
| Syncthing | clip intake, UI at `sync.` | UI behind Authentik + `media-admins`, own GUI auth off | live |
| normalize sweep | editing standard, one copy per clip | live |
| nftables | host firewall (22, 80/443, 1935, 2222, 3306, 8890/udp, 22000) | live |
| Caddy | TLS + routing for all hostnames; JSON access logs per site (identity headers kept, cookies/tokens dropped) | live (access logs: deploy pending) |
| Metrics taps | node_exporter (host), cAdvisor (containers), MediaMTX and Caddy metrics — Prometheus format on loopback, no store yet | **committed 2026-11-01, deploy pending** |
| Logging framework | rotated Docker logs + `vanguard.service/category` labels on every container; dumpd/sweep/vgdeband/deploy emit logfmt events (`LOGGING.md`); store not chosen (Loki vs VictoriaLogs vs Netdata) | **committed 2026-11-01, deploy pending** (recreates every container) |
| Authentik (+Postgres, Redis) | Discord login, groups, guild-membership policy | live |
| Jellyfin | watch page, SSO via Authentik, direct play only | live |
| Vanguard Metadata plugin | seasons by tag, Discord names/avatars/events, feed, playback policy | **0.1.3 built** (flat `<Title>__<ts>` names, policy task, feed wording) — install pending; server has 0.1.0 or 0.1.2 |
| MariaDB | databases for the bot and Gitea; public :3306, root local-only | live |
| Gitea | git hosting on MariaDB, Actions enabled; holds this repo | live; Discord login via Authentik OIDC **verified**; access = `git-users`/`git-admins` groups; local admin as fallback |
| act_runner | CI jobs in Docker | **not registered** |
| discord-bot | Python runtime, idles until `bot/app/main.py` exists | live, no code yet |
| `bot.` host + Discord bot | the bot's API **and** its web pages on one port (`WEB_SERVE_UI`), behind Authentik forward-auth; Caddy requires `bot-users`, the bot re-checks `bot-users,admins` per request | a push to `main` deploys itself: CI builds a **release bundle** (app + `vendor/` dependencies) and posts it signed to dumpd `POST /deploy-bot` (bridge-only port, no Docker socket in CI), which verifies it and swaps it into `/srv/bot/app` → `/app`; image installs no bot dependencies | **reviewed + wired 2026-11-02, first install pending** (needs the Authentik app for `bot.`, the bot token in `.env`, and `BOT_DEPLOY_SECRET` as a Gitea repo secret) |
| Live portal (`live/`, `live.` host) | React + dockview multi-stream viewer + browser Go Live (WHIP); MediaMTX HLS/WebRTC/API on loopback; dumpd `/me/stream-key` | live; Go Live verified |

## Unfinished, in rough priority

0a. **WebRTC audio** (found 2026-11-01 with the 5-stream test): AAC cannot cross WebRTC, so the portal was silent unless it fell back to HLS. Fix committed: audio mirror in `bin/mtx-on-ready.sh` + portal mapping. **Deploy pending** (MediaMTX recreate between streams; dumpd restart; portal push). Streamers must set B-frames 0 for WebRTC playback.
0. **Band removal**: deployed 2026-11-01; `vgdeband.sh` catch-up ran on the Vanguard-Training recordings (check `journalctl -u vgdeband` for the done lines). Live portal: viewer now clips the band with a measured game-aspect frame (refits on every dockview resize/move) — committed 2026-11-01, deploy with `./deployLive.sh`.

1. Roles→groups policy: bound and updated (revokes only its own grants); will be proven by the next natural Discord logins (`/whoami` should show `bot-users`). Nested groups **verified 2026-11-01**: `git-admins` now reaches the header via the `Vanguard profile (nested groups)` scope mapping (README 5e). `akadmin` remains the recovery login if a policy misfires.
2. Confirm plugin 0.1.3 under Jellyfin → Plugins and the Gitea runner registered.
3. Portal viewer during a real stream: corner chip `webrtc`; Windows Go Live with WarDogs audio (first Windows streamer).
5. Housekeeping: pin `AUTHENTIK_TAG`, rotate the Discord client secret seen in a screenshot, nightly Postgres dump, Caddyfile formatting warning.
6. Medal filename pattern in the plugin (need a sample filename).
7. Home archive puller on the home box.
8. Plugin repository served by Caddy so Jellyfin updates from its catalog.
9. Discord bot: first bundle install (`./deployBotBundle.sh`), Authentik application for `bot.`, first real MariaDB run
   (the bot's own README flags it as never tested end to end), then clip submissions. Review notes: its pages are served
   from the bot itself now; an Authentik username is matched to one Discord member by name.
10. Authentik blueprint from `.env` (Discord source, groups, policies, provider) for reproducible installs.
11. SYS-3 when transcoding or sweep speed hurts (2×6 TB option currently unavailable; watching for it). Interim option documented: off-site gap filling on the home NVENC box.

## Verified behaviours worth remembering
- **Never scp generated configs** (`mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt`, `.env`): a template copy wiped all streamers once. They are gitignored now; `*.example.*` are reference only.
- Caddy runs in a container: anything it serves from disk must be a mount (`/srv/live` was missing → 404).
- Go Live quality depends on the browser's encoder: Windows Chrome uses the GPU (fine); Linux Chromium on NVIDIA is software H.264 Baseline (soft at any bitrate, CPU-limited at 1440p60) and cannot be fixed by flags — Linux streamers use OBS. The stats strip shows encoder + profile.
- Several videos inside one non-season folder = ONE Jellyfin episode with "versions". Streams are therefore published flat as `<Title>__<ts>.mp4` in the streamer's folder (plugin 0.1.3 parses the title from the name). Four or five helmet cams a night each get their own episode.
- Streams are not re-encoded for VFR/gaps by default (dropped frames are legitimate; a 3 h re-encode is hours of CPU); players hold the last frame across a gap. The timecode sidecar lists gaps. **Gap filling is available** (`FILL_GAPS_MS` in `normalize-sweep.service`, README "Timecode") — turn on when the box has a hardware encoder. Clips always get CFR.
- The sweep's VFR check used a gawk-only function; Debian's mawk made it pass everything silently. Portable now.
- A deploy block that is only half-run leaves old binaries in `/usr/local/bin`; `grep -c` the installed script before blaming logic.
- MediaMTX does not reload `mediamtx.yml` live here (bind-mounted file): any change to it needs `docker compose restart mediamtx` between streams. Nothing day-to-day touches that file.
- **Timecode verified live (2026-10-31)**: OBS + NVENC with the band → `source: strip`, drift −3 ms over 80 s, 0 gaps, 1920×1092 (band present); arrival time was 2.3 s late vs the first frame, which the strip corrected. Go Live from Linux Chromium (software encode) → strip decoded fine but 239 gaps and −85 ms: frame dropping, as expected on that path.
- MediaMTX stamps recording filenames in its container clock: without `TZ` it is UTC and the plugin reads names as Eastern (4 h off). `TZ=America/New_York` is set on the container.
- MediaMTX 1.21 refuses to start without `webrtcAdditionalHosts` or `webrtcIPsFromInterfaces`; configure.py writes the public host in.
- Jellyfin honours provider-supplied season numbers on a Shows library (day seasons appeared). Next check: Monday 04:10 ET should fold them into a week season.
- A stream indexed by Jellyfin while still recording has no duration and skips to the next episode on seek — fixed by recording into `.live/` so nothing sees a file until it is finished and flat.
- Sweep leaves compliant Syncthing files untouched and keeps exactly one copy of converted ones (tested).
- Firewall: Docker-published ports bypass the input chain; MariaDB uses host networking precisely so the rules apply.
- Never `systemctl stop|restart nftables` (flushes Docker's NAT). Reload only.