# vanguard-stream — SYS-1 (Xeon E-2136, 32 GB ECC, 2×4 TB SATA soft RAID)

> **STATUS.md** = what runs, who does what, what's unfinished (kept current). **CommunityDiscordMedia.md** = design + phases.

Ingest from OBS, browse/scrub/download for editors, automatic normalization to one editing
standard, dump button, 15-day archive pull to a home HDD. Everything is plain files under /srv/media.

## 1. Reinstall from the OVH manager

Server → Reinstall → Debian 13 (Trixie).
Tick **custom partitioning** and **use distribution kernel**. Layout (RAID 1 across both HDDs):

| mount | size | fs |
|-------|------|----|
| /boot | 1 GB | ext4 |
| /     | 250 GB | ext4 |
| swap  | 8 GB per disk | swap |
| /srv  | rest (~3.4 TiB) | xfs |

Add your SSH key. OVH's template creates a `debian` user with sudo; root SSH is disabled. Wait for the mail.

## 2. Install

    scp -r vanguard-stream debian@SERVER:~/
    ssh debian@SERVER
    cd ~/vanguard-stream && sudo bash install.sh

install.sh installs docker, ffmpeg, rclone; creates /srv/media/{streams,clips,finals,trash,.markers}
owned by uid 1000 (the `debian` user); copies the systemd units; adds `debian` to the docker group;
starts the compose stack. It prints what you still have to edit.

The folder stays where it is: compose bind-mounts the configs and the dock page from it.
After install, log out and back in once so the docker group applies, then plain `docker compose`
works from ~/vanguard-stream without sudo.

## 3. Configure (domain, admin, streamers, editors)

    cd ~/vanguard-stream
    ./configure.py --full      # first time: asks everything, then applies
    ./configure.py             # later: add/remove/reset streamers and editors, then apply
    ./configure.py --show      # print current users and the OBS settings to hand out

Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `copyparty.conf`,
regenerates those three plus `dump-users.txt`, installs the dump users to `/etc/vanguard`, creates
`streams/<name>` and `clips/<name>` (with `.stignore`) for each streamer, restarts dumpd and the
containers. Removing a streamer or editor revokes access only; their files stay on disk.
Passwords may not contain `: & ? # / \ quotes or spaces` (they ride inside the stream key).

## 4. Firewall (host nftables; OVH edge firewall is optional on top)

Applied automatically at the end of `install.sh`. To re-apply by hand:

    cd ~/vanguard-stream && sudo bash firewall/firewall.sh

Installs `firewall/nftables.conf` as `/etc/nftables.conf`, syntax-checks, applies, enables at boot.
It uses its own table (`inet vanguard`) so applying or reloading never disturbs Docker's rules,
and filters input only so container networking keeps working. **Keep your SSH session open and
test a new one before closing it.**

| port | proto | who |
|------|-------|-----|
| 22    | tcp | anywhere, rate limited (edit `ssh_allow` in /etc/nftables.conf to restrict to your IPs) |
| 80, 443 | tcp | world (Caddy) |
| 443   | udp | world (Caddy HTTP/3) |
| 1935  | tcp | world (RTMP ingest) |
| 8890  | udp | world (SRT ingest) |
| 22000 | tcp+udp | world (Syncthing) |
| 3306  | tcp | world, rate limited (MariaDB for the bot dev; restrict via `db_allow` in /etc/nftables.conf) |
| 2222  | tcp | world (Gitea ssh) |
| 8189  | tcp+udp | world (WebRTC media for the live portal) |

Everything else is dropped and a trickle is logged (`journalctl -k | grep vg-drop`). Authentik (9000),
Jellyfin (8096), copyparty, the Syncthing UI and dumpd bind to 127.0.0.1 only and are reached via Caddy
or an ssh tunnel, so they are deliberately not opened. MariaDB runs on the host network (not a Docker
published port) precisely so the firewall can govern it.
Syncthing's admin UI is proxied at `https://HOST/sync/` behind Syncthing's own GUI login (set it in
Actions → Settings → GUI before relying on this). The ssh tunnel to 8384 still works as a fallback.

To change rules: edit `/etc/nftables.conf`, then `sudo nft -f /etc/nftables.conf` or
`sudo systemctl reload nftables`. **Never `systemctl stop|restart nftables`** — Debian's unit flushes
the whole ruleset on stop, which also wipes Docker's NAT rules and takes the containers offline
until Docker is restarted.

## 5. Streamer setup (once)

OBS → Settings → Stream → Custom: Server `rtmp://media.example.com:1935/`, key anything (the dock sets it).
OBS → Tools → WebSocket Server Settings → enable, set a password.
OBS → Docks → Custom Browser Docks → `https://media.example.com/dock/`.
In the dock: fill host/user/pass/OBS-ws-password once (stored in the dock's localStorage).
Per stream: type a title, click **Go Live**. After a bad night: **Dump tonight**.

Stream key format the dock produces: `techy/Title?user=techy&pass=...`
SRT equivalent if preferred: `srt://media.example.com:8890?streamid=publish:techy/Title:techy:PASS`

## 5b. Syncthing hardening (do once, in the server's Syncthing UI)

Syncthing receives straight into `clips/<user>` (Syncthing UI path `/var/syncthing/clips/<user>`),
receive-only, with a `.stignore` (from `syncthing/stignore`) that pulls video files only. The gamer's
folder structure is mirrored as-is. **One copy per clip on the server**: compliant files are left
alone; a non-compliant file (HEVC/MKV/VFR) is converted to `name.mp4` (`name.h264.mp4` if the raw was
already `.mp4`), the raw's name is appended to `.stlocal` (an `#include`d ignore list, so Syncthing
drops it without flagging or re-pulling), and the raw is deleted locally — it still exists on the
gamer's PC. `configure.py` creates the folder + ignore file for each streamer.

In the UI (https://HOST/sync/):
- each clips folder → path `/var/syncthing/clips/<user>`, **Folder Type: Receive Only**,
  Edit → Advanced → **Minimum Free Disk Space: 300 GB** (so clips can never fill the mirror and stall
  the recordings).
- Actions → Settings → Connections → **Incoming Rate Limit: 12500 KiB/s** (~100 Mbps) so a gamer
  sharing a year of old clips cannot starve the ingest during stream hours.

## 5c. Authentik (Discord login) and Jellyfin

DNS: `auth` and `watch` as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
for both hostnames, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
hostnames filled in. (`./configure.py --full` asks for the same hostnames if you prefer.) Then:

**Authentik first run** — https://auth.HOST/if/flow/initial-setup/ → set the `akadmin` password.
1. Discord developer portal → New Application → OAuth2: add redirect
   `https://auth.HOST/source/oauth/callback/discord/` (last segment = the source slug below), copy Client ID + Secret.
2. Authentik → Directory → Federation & Social login → Create → Discord OAuth Source: slug `discord`,
   consumer key/secret = Client ID/Secret, user matching "link on unique identifier", scopes empty.
3. Flows & Stages → Stages → `default-authentication-identification` → Sources: select Discord,
   tick "Show sources' labels". Without this the Discord button never appears on the login page.
4. Flows & Stages → Stages → `default-source-enrollment-write` → User type: Internal. Otherwise
   Discord users are External and get "Interface can only be accessed by internal users".
5. Directory → Groups: create `streamers`, `editors`, `admins`. Add people after their first login.
6. (optional) Restrict login to members of your Discord server: add scope `guilds` to the source;
   Customization → Policies → Create Expression Policy from `authentik/discord-guild-policy.py` (fill in
   your server ID); bind it to both `default-source-enrollment` and `default-source-authentication`
   (flow → Policy / Group / User Bindings) and set each flow's Policy engine mode to `all`. Server ID: Discord Developer Mode → right-click server → Copy ID.

**Jellyfin first run** — https://watch.HOST/ → wizard: admin user, add libraries: `/media/streams` and
`/media/clips` as **Shows** with only "Vanguard Metadata" as metadata downloader (Series/Season/Episode)
and Screen Grabber + Vanguard Metadata as image fetchers; `/media/finals` as Home Videos with fetchers off.
The Vanguard Metadata plugin (`jellyfin-plugin-vanguard/`, built with its `package.sh`) is installed by
`addCommunityMediaServer.sh` / `installJellyfinPlugin.sh` from `jellyfin-plugin-vanguard/dist/`.
- Dashboard → Playback → Transcoding: **disable** hardware acceleration. Per-user "allow transcoding" is
  switched off automatically for everyone (SSO users included) by the Vanguard plugin's "enforce playback
  policy" task; turn that toggle off in the plugin config once the server has Quick Sync (SYS-3).
- Dashboard → Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json`, then Catalog → **SSO Authentication** → install → restart Jellyfin.
- Authentik → Applications → Create with provider → OAuth2/OpenID: redirect
  `https://watch.HOST/sso/OID/redirect/authentik`, note client ID/secret.
- Jellyfin → Plugins → SSO-Auth → add OID provider "authentik": endpoint `https://auth.HOST/application/o/<slug>/`,
  client ID/secret, enable, roles claim `groups`, admin roles `admins`, enabled folders = the three libraries,
  Scheme Override `https`, Port Override empty (a value appends `:443` and Authentik rejects the redirect). Save.
- The plugin adds no button. Test with `https://watch.HOST/sso/OID/start/authentik`, then Dashboard →
  General → Branding: Login disclaimer = `jellyfin/branding-disclaimer.html` (set the hostname),
  Custom CSS = `jellyfin/branding.css`. The page then shows only "Sign in with Discord"; a small
  "Admin login" link reveals the local username/password form.
  Jellyfin creates the account on first SSO login.

Phase 4 is the Vanguard Metadata Jellyfin plugin; Phase 5 puts copyparty, /sync and /dump behind Authentik forward-auth.

## 5d. MariaDB + Discord bot (for the bot dev)

`mariadb` (public on 3306, host network, root local-only, 32-char generated passwords) with a database/user
for the bot, and `discord-bot`, a Python 3.12 container that runs `bot/app/main.py` and idles until that file
exists. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
or the `[d]` menu item; `./configure.py --show` prints them).

The bot also serves a web UI on 127.0.0.1:8000, published as `https://bot.HOST/` behind **Authentik
forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`. Create the matching
Proxy Provider in Authentik (forward auth, single application, external host `https://bot.HOST`), add it to
the embedded outpost, and bind a group policy. Until then the site returns an error, never an open page.

The build brief for whoever writes the bot is `bot/AGENT.md`; `bot/README.md` is the ops cheat-sheet.

    docker compose logs -f discord-bot                     # watch it
    docker compose restart discord-bot                     # after code changes
    docker compose build discord-bot && docker compose up -d discord-bot   # after requirements.txt changes

## 5e. Gitea + Actions runner

`gitea` (MariaDB-backed, `git.HOST` via Caddy, git-over-ssh on **2222**, registration closed) and
`act_runner` (jobs run as Docker containers; labels `ubuntu-latest`, `ubuntu-22.04`, `dotnet`, `python`).
The gitea db/user are created by `mariadb/init/10-gitea.sh` on MariaDB's first start; for a MariaDB that
already existed run `./createGiteaDb.sh` once.

1. `./configure.py --full` asks the Gitea hostname (CNAME `git`); apply starts both containers.
2. https://git.HOST/ → install page (db pre-filled) → create the admin account → Install.
3. `./registerGiteaRunner.sh` → generates a registration token, stores it in `.env`, starts the runner.
4. Remotes: `ssh://git@git.HOST:2222/<org>/<repo>.git` or `https://git.HOST/<org>/<repo>.git`.

Runner caveat: it holds the Docker socket, so anyone who can push a workflow can run as root on the box.
Keep repo write access to trusted people.

## 5f. Live portal (`live.HOST`)

React + dockview page (`live/`) behind Authentik forward-auth: every active stream is a dockable window,
audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
MediaMTX runs the `-ffmpeg` image so the optional "is live" Discord hook (`LIVE_WEBHOOK_URL` in `.env`,
asked by `configure.py --full`) can post when a stream starts.

1. CNAME `live` → main host; `./configure.py --full` asks the hostname (also `--hosts … LIVE`).
2. Authentik: Proxy Provider (forward auth, single application, external host `https://live.HOST`) →
   embedded outpost → group policy (e.g. everyone in the server, or `streamers`+`editors`+`admins`).
3. Dev box: `./deployLive.sh` builds `live/` and copies `dist/` to `/srv/live` (Caddy serves it directly).

**Go Live** button: broadcast a window from the browser (no OBS) as `<user>/HelmetCam-<date>` — key comes
from `/me/stream-key` (dumpd reads `/etc/vanguard/streamers.json`, identity from the Authentik header), so a
streamer's Authentik username must equal their MediaMTX name.

Bandwidth: no transcoding, every viewer gets the source bitrate (~15 Mbps) → ~30 concurrent viewers on
the 500 Mbps line. For a crowd, restream to Twitch from OBS.

## 5g. Authentik forward-auth for the Caddy-gated hosts (`live.`, `bot.`; later the main host)

Design: **Authentik authenticates once for the whole domain; Caddy authorizes per host** from the
`X-Authentik-Groups` header the outpost returns; **Discord roles fill the Authentik groups** at login.

1. Applications → Outposts → edit "authentik Embedded Outpost" → config `authentik_host: https://auth.HOST`.
2. Applications → Create with provider → Application `Vanguard sites`; **Proxy Provider**, authorization flow
   implicit-consent, mode **Forward auth (domain level)**, authentication URL `https://auth.HOST`,
   cookie domain = the parent domain (e.g. `dumpstervigilantes.rip`). Edit the Embedded Outpost →
   Applications → add it → Update. One login now covers every gated host.
3. Per-host group requirement = `.env`: `BOT_REQUIRE_GROUP=bot-users` (default), `LIVE_REQUIRE_GROUP=`
   (empty = any logged-in member). `./configure.py` apply regenerates the Caddy blocks; a user outside the
   group gets a 403 from Caddy before the app sees the request.
4. Discord roles → groups: Directory → Groups: create `bot-users` (and keep `streamers`, `editors`, `admins`).
   Discord source → Scopes: `guilds guilds.members.read`. Customization → Policies → Expression Policy
   `discord-roles-to-groups` from `authentik/discord-roles-to-groups.py` (fill in the server ID and the
   role-id → group table). Bind it to `default-source-enrollment` **and** `default-source-authentication`
   (engine mode `all`, after `discord-guild-member`). From then on giving someone a Discord role gives them
   the matching access on next login; removing it removes it.
5. Test in a private window: `https://live.HOST/` → Discord login → portal; `https://bot.HOST/` → 403 unless
   you hold the bot role; `https://live.HOST/me/stream-key` shows your username.

Jellyfin and Gitea use OIDC providers instead, not this.

## 6. Editors

Browser: https://media.example.com/ → log in. Scrub any `.mp4` in the browser, download what you want.
WebDAV mount: same URL, same login. Upload finished work to `/finals`.

Every `.mp4` an editor sees is the same kind of file (see "Editing standard"). Raw clipper output that
needed converting is kept in a hidden `.src/` folder next to it (admins can show dotfiles in copyparty).

## 7. Home archive box

Copy `home/rclone-archive.sh` to the home machine, set the vars, run from cron nightly.
It connects over SFTP as `debian` (all media files are owned by uid 1000 so it can delete them).
Pulls everything older than 15 days off the server, including the hidden `.src/` raw files.

## Layout on the server

    /srv/media/streams/<user>/<Title>/.live/<ts>.mp4             recording in progress (hidden from Jellyfin/copyparty)
    /srv/media/streams/<user>/<Title>/2025-10-10_21-03-11.mp4   finished stream, flat MP4 published by the sweep
    /srv/media/clips/<user>/clip.mp4                             clip (synced as-is, or converted; raw removed)
    /srv/media/clips/<user>/.stlocal                             Syncthing ignore list maintained by the sweep
    /srv/media/clips/<user>/.src/clip.mkv                        raw source, only for clips that did NOT arrive via Syncthing
    /srv/media/clips/<user>/clip.normalize-failed.log            only if normalization failed; delete to retry
    /srv/media/finals/                                           editors upload here
    /srv/media/trash/<night>/...                                 dumped, purged after 3 days
    /srv/media/.markers/                                         dump markers
    /srv/media/.syncthing/                                       Syncthing config

## Editing standard (what normalize-sweep enforces)

MP4 · H.264 High, 8-bit 4:2:0 · constant frame rate · AAC audio (all tracks kept) · index at the
front (instant seek in browser and NLE) · at most 2560x1440 and 60 fps (downscaled to fit, never
upscaled; aspect kept).

| input | action |
|-------|--------|
| already compliant | nothing |
| MediaMTX recording in `.live/` (fragmented MP4) | once idle 3 min: lossless remux to a flat MP4 in the title folder, `.live` source removed |
| in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
| HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
| ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |

Runs every 5 min on files idle ≥3 min and not open. Software x264 on the E-2136 does ~0.5x realtime
at 1440p60, so a 2 h clip batch takes ~4 h; streams only remux (minutes). All at idle priority.

## Reboot behaviour

Everything comes back on its own, in this order:

1. mdadm assembles the mirror, systemd mounts /srv (local-fs). A degraded mirror still boots.
2. docker.service has `RequiresMountsFor=/srv/media` (drop-in installed by install.sh), so it never
   starts with the media tree missing. All containers are `restart: unless-stopped`.
3. dumpd.service starts; normalize-sweep.timer fires 2 min after boot and every 5 min after;
   trash-purge.timer is `Persistent=true` so a missed 05:00 run fires at boot.
4. A stream cut by a reboot is playable up to the last second (fMP4 parts are flushed every 1 s);
   OBS reconnects and MediaMTX opens a new file in the same title folder.
5. Half-written `*.part` files are overwritten by the next sweep. Dumps are idempotent.

Time: install.sh sets the server clock to America/New_York so journal, timers, file timestamps
and the dump window all read in Eastern. Dump "night" = 17:00 → 02:00 Eastern (pinned in dumpd via
DUMP_TZ, independent of the host zone). Between 02:00 and 17:00 the default night is the previous
evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.

## Timers

| unit | when | does |
|------|------|------|
| normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
| trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
| dumpd.service     | always | the dump endpoint on 127.0.0.1:8080, behind Caddy at /dump |