Brandyn / Techy
b2869c3740
A deploy copied mediamtx.yml (template) over the server's generated file and the next apply wiped all streamers. Generated files are now gitignored so scp -r cannot clobber them. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
308 lines
20 KiB
Markdown
308 lines
20 KiB
Markdown
# vanguard-stream — SYS-1 (Xeon E-2136, 32 GB ECC, 2×4 TB SATA soft RAID)
|
||
|
||
> **STATUS.md** = what runs, who does what, what's unfinished (kept current). **CommunityDiscordMedia.md** = design + phases.
|
||
|
||
Ingest from OBS, browse/scrub/download for editors, automatic normalization to one editing
|
||
standard, dump button, 15-day archive pull to a home HDD. Everything is plain files under /srv/media.
|
||
|
||
## 1. Reinstall from the OVH manager
|
||
|
||
Server → Reinstall → Debian 13 (Trixie).
|
||
Tick **custom partitioning** and **use distribution kernel**. Layout (RAID 1 across both HDDs):
|
||
|
||
| mount | size | fs |
|
||
|-------|------|----|
|
||
| /boot | 1 GB | ext4 |
|
||
| / | 250 GB | ext4 |
|
||
| swap | 8 GB per disk | swap |
|
||
| /srv | rest (~3.4 TiB) | xfs |
|
||
|
||
Add your SSH key. OVH's template creates a `debian` user with sudo; root SSH is disabled. Wait for the mail.
|
||
|
||
## 2. Install
|
||
|
||
scp -r vanguard-stream debian@SERVER:~/
|
||
ssh debian@SERVER
|
||
cd ~/vanguard-stream && sudo bash install.sh
|
||
|
||
install.sh installs docker, ffmpeg, rclone; creates /srv/media/{streams,clips,finals,trash,.markers}
|
||
owned by uid 1000 (the `debian` user); copies the systemd units; adds `debian` to the docker group;
|
||
starts the compose stack. It prints what you still have to edit.
|
||
|
||
The folder stays where it is: compose bind-mounts the configs and the dock page from it.
|
||
After install, log out and back in once so the docker group applies, then plain `docker compose`
|
||
works from ~/vanguard-stream without sudo.
|
||
|
||
## 3. Configure (domain, admin, streamers, editors)
|
||
|
||
> `mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt` and `.env` are **generated on the server**
|
||
> by `configure.py` and gitignored. The `*.example.*` files in the repo are reference only. **Never copy them
|
||
> to the server**: doing so once replaced the live streamer passwords with `CHANGEME` and the next apply
|
||
> wiped the streamers. To update the server, copy scripts (`configure.py`, `bin/`, `docker-compose.yml`, …)
|
||
> and re-run `./configure.py`.
|
||
|
||
cd ~/vanguard-stream
|
||
./configure.py --full # first time: asks everything, then applies
|
||
./configure.py # later: add/remove/reset streamers and editors, then apply
|
||
./configure.py --show # print current users and the OBS settings to hand out
|
||
|
||
Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `copyparty.conf`,
|
||
regenerates those three plus `dump-users.txt`, installs the dump users to `/etc/vanguard`, creates
|
||
`streams/<name>` and `clips/<name>` (with `.stignore`) for each streamer, restarts dumpd and the
|
||
containers. Removing a streamer or editor revokes access only; their files stay on disk.
|
||
Passwords may not contain `: & ? # / \ quotes or spaces` (they ride inside the stream key).
|
||
|
||
## 4. Firewall (host nftables; OVH edge firewall is optional on top)
|
||
|
||
Applied automatically at the end of `install.sh`. To re-apply by hand:
|
||
|
||
cd ~/vanguard-stream && sudo bash firewall/firewall.sh
|
||
|
||
Installs `firewall/nftables.conf` as `/etc/nftables.conf`, syntax-checks, applies, enables at boot.
|
||
It uses its own table (`inet vanguard`) so applying or reloading never disturbs Docker's rules,
|
||
and filters input only so container networking keeps working. **Keep your SSH session open and
|
||
test a new one before closing it.**
|
||
|
||
| port | proto | who |
|
||
|------|-------|-----|
|
||
| 22 | tcp | anywhere, rate limited (edit `ssh_allow` in /etc/nftables.conf to restrict to your IPs) |
|
||
| 80, 443 | tcp | world (Caddy) |
|
||
| 443 | udp | world (Caddy HTTP/3) |
|
||
| 1935 | tcp | world (RTMP ingest) |
|
||
| 8890 | udp | world (SRT ingest) |
|
||
| 22000 | tcp+udp | world (Syncthing) |
|
||
| 3306 | tcp | world, rate limited (MariaDB for the bot dev; restrict via `db_allow` in /etc/nftables.conf) |
|
||
| 2222 | tcp | world (Gitea ssh) |
|
||
| 8189 | tcp+udp | world (WebRTC media for the live portal) |
|
||
|
||
Everything else is dropped and a trickle is logged (`journalctl -k | grep vg-drop`). Authentik (9000),
|
||
Jellyfin (8096), copyparty, the Syncthing UI and dumpd bind to 127.0.0.1 only and are reached via Caddy
|
||
or an ssh tunnel, so they are deliberately not opened. MariaDB runs on the host network (not a Docker
|
||
published port) precisely so the firewall can govern it.
|
||
Syncthing's admin UI is proxied at `https://HOST/sync/` behind Syncthing's own GUI login (set it in
|
||
Actions → Settings → GUI before relying on this). The ssh tunnel to 8384 still works as a fallback.
|
||
|
||
To change rules: edit `/etc/nftables.conf`, then `sudo nft -f /etc/nftables.conf` or
|
||
`sudo systemctl reload nftables`. **Never `systemctl stop|restart nftables`** — Debian's unit flushes
|
||
the whole ruleset on stop, which also wipes Docker's NAT rules and takes the containers offline
|
||
until Docker is restarted.
|
||
|
||
## 5. Streamer setup (once)
|
||
|
||
OBS → Settings → Stream → Custom: Server `rtmp://media.example.com:1935/`, key anything (the dock sets it).
|
||
OBS → Tools → WebSocket Server Settings → enable, set a password.
|
||
OBS → Docks → Custom Browser Docks → `https://media.example.com/dock/`.
|
||
In the dock: fill host/user/pass/OBS-ws-password once (stored in the dock's localStorage).
|
||
Per stream: type a title, click **Go Live**. After a bad night: **Dump tonight**.
|
||
|
||
Stream key format the dock produces: `techy/Title?user=techy&pass=...`
|
||
SRT equivalent if preferred: `srt://media.example.com:8890?streamid=publish:techy/Title:techy:PASS`
|
||
|
||
## 5b. Syncthing hardening (do once, in the server's Syncthing UI)
|
||
|
||
Syncthing receives straight into `clips/<user>` (Syncthing UI path `/var/syncthing/clips/<user>`),
|
||
receive-only, with a `.stignore` (from `syncthing/stignore`) that pulls video files only. The gamer's
|
||
folder structure is mirrored as-is. **One copy per clip on the server**: compliant files are left
|
||
alone; a non-compliant file (HEVC/MKV/VFR) is converted to `name.mp4` (`name.h264.mp4` if the raw was
|
||
already `.mp4`), the raw's name is appended to `.stlocal` (an `#include`d ignore list, so Syncthing
|
||
drops it without flagging or re-pulling), and the raw is deleted locally — it still exists on the
|
||
gamer's PC. `configure.py` creates the folder + ignore file for each streamer.
|
||
|
||
In the UI (https://HOST/sync/):
|
||
- each clips folder → path `/var/syncthing/clips/<user>`, **Folder Type: Receive Only**,
|
||
Edit → Advanced → **Minimum Free Disk Space: 300 GB** (so clips can never fill the mirror and stall
|
||
the recordings).
|
||
- Actions → Settings → Connections → **Incoming Rate Limit: 12500 KiB/s** (~100 Mbps) so a gamer
|
||
sharing a year of old clips cannot starve the ingest during stream hours.
|
||
|
||
## 5c. Authentik (Discord login) and Jellyfin
|
||
|
||
DNS: `auth` and `watch` as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
|
||
for both hostnames, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
|
||
adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
|
||
hostnames filled in. (`./configure.py --full` asks for the same hostnames if you prefer.) Then:
|
||
|
||
**Authentik first run** — https://auth.HOST/if/flow/initial-setup/ → set the `akadmin` password.
|
||
1. Discord developer portal → New Application → OAuth2: add redirect
|
||
`https://auth.HOST/source/oauth/callback/discord/` (last segment = the source slug below), copy Client ID + Secret.
|
||
2. Authentik → Directory → Federation & Social login → Create → Discord OAuth Source: slug `discord`,
|
||
consumer key/secret = Client ID/Secret, user matching "link on unique identifier", scopes empty.
|
||
3. Flows & Stages → Stages → `default-authentication-identification` → Sources: select Discord,
|
||
tick "Show sources' labels". Without this the Discord button never appears on the login page.
|
||
4. Flows & Stages → Stages → `default-source-enrollment-write` → User type: Internal. Otherwise
|
||
Discord users are External and get "Interface can only be accessed by internal users".
|
||
5. Directory → Groups: create `streamers`, `editors`, `admins`. Add people after their first login.
|
||
6. (optional) Restrict login to members of your Discord server: add scope `guilds` to the source;
|
||
Customization → Policies → Create Expression Policy from `authentik/discord-guild-policy.py` (fill in
|
||
your server ID); bind it to both `default-source-enrollment` and `default-source-authentication`
|
||
(flow → Policy / Group / User Bindings) and set each flow's Policy engine mode to `all`. Server ID: Discord Developer Mode → right-click server → Copy ID.
|
||
|
||
**Jellyfin first run** — https://watch.HOST/ → wizard: admin user, add libraries: `/media/streams` and
|
||
`/media/clips` as **Shows** with only "Vanguard Metadata" as metadata downloader (Series/Season/Episode)
|
||
and Screen Grabber + Vanguard Metadata as image fetchers; `/media/finals` as Home Videos with fetchers off.
|
||
The Vanguard Metadata plugin (`jellyfin-plugin-vanguard/`, built with its `package.sh`) is installed by
|
||
`addCommunityMediaServer.sh` / `installJellyfinPlugin.sh` from `jellyfin-plugin-vanguard/dist/`.
|
||
- Dashboard → Playback → Transcoding: **disable** hardware acceleration. Per-user "allow transcoding" is
|
||
switched off automatically for everyone (SSO users included) by the Vanguard plugin's "enforce playback
|
||
policy" task; turn that toggle off in the plugin config once the server has Quick Sync (SYS-3).
|
||
- Dashboard → Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json`, then Catalog → **SSO Authentication** → install → restart Jellyfin.
|
||
- Authentik → Applications → Create with provider → OAuth2/OpenID: redirect
|
||
`https://watch.HOST/sso/OID/redirect/authentik`, note client ID/secret.
|
||
- Jellyfin → Plugins → SSO-Auth → add OID provider "authentik": endpoint `https://auth.HOST/application/o/<slug>/`,
|
||
client ID/secret, enable, roles claim `groups`, admin roles `admins`, enabled folders = the three libraries,
|
||
Scheme Override `https`, Port Override empty (a value appends `:443` and Authentik rejects the redirect). Save.
|
||
- The plugin adds no button. Test with `https://watch.HOST/sso/OID/start/authentik`, then Dashboard →
|
||
General → Branding: Login disclaimer = `jellyfin/branding-disclaimer.html` (set the hostname),
|
||
Custom CSS = `jellyfin/branding.css`. The page then shows only "Sign in with Discord"; a small
|
||
"Admin login" link reveals the local username/password form.
|
||
Jellyfin creates the account on first SSO login.
|
||
|
||
Phase 4 is the Vanguard Metadata Jellyfin plugin; Phase 5 puts copyparty, /sync and /dump behind Authentik forward-auth.
|
||
|
||
## 5d. MariaDB + Discord bot (for the bot dev)
|
||
|
||
`mariadb` (public on 3306, host network, root local-only, 32-char generated passwords) with a database/user
|
||
for the bot, and `discord-bot`, a Python 3.12 container that runs `bot/app/main.py` and idles until that file
|
||
exists. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
|
||
or the `[d]` menu item; `./configure.py --show` prints them).
|
||
|
||
The bot also serves a web UI on 127.0.0.1:8000, published as `https://bot.HOST/` behind **Authentik
|
||
forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`. Create the matching
|
||
Proxy Provider in Authentik (forward auth, single application, external host `https://bot.HOST`), add it to
|
||
the embedded outpost, and bind a group policy. Until then the site returns an error, never an open page.
|
||
|
||
The build brief for whoever writes the bot is `bot/AGENT.md`; `bot/README.md` is the ops cheat-sheet.
|
||
|
||
docker compose logs -f discord-bot # watch it
|
||
docker compose restart discord-bot # after code changes
|
||
docker compose build discord-bot && docker compose up -d discord-bot # after requirements.txt changes
|
||
|
||
## 5e. Gitea + Actions runner
|
||
|
||
`gitea` (MariaDB-backed, `git.HOST` via Caddy, git-over-ssh on **2222**, registration closed) and
|
||
`act_runner` (jobs run as Docker containers; labels `ubuntu-latest`, `ubuntu-22.04`, `dotnet`, `python`).
|
||
The gitea db/user are created by `mariadb/init/10-gitea.sh` on MariaDB's first start; for a MariaDB that
|
||
already existed run `./createGiteaDb.sh` once.
|
||
|
||
1. `./configure.py --full` asks the Gitea hostname (CNAME `git`); apply starts both containers.
|
||
2. https://git.HOST/ → install page (db pre-filled) → create the admin account → Install.
|
||
3. `./registerGiteaRunner.sh` → generates a registration token, stores it in `.env`, starts the runner.
|
||
4. Remotes: `ssh://git@git.HOST:2222/<org>/<repo>.git` or `https://git.HOST/<org>/<repo>.git`.
|
||
|
||
Runner caveat: it holds the Docker socket, so anyone who can push a workflow can run as root on the box.
|
||
Keep repo write access to trusted people.
|
||
|
||
## 5f. Live portal (`live.HOST`)
|
||
|
||
React + dockview page (`live/`) behind Authentik forward-auth: every active stream is a dockable window,
|
||
audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
|
||
MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
|
||
MediaMTX runs the `-ffmpeg` image so the optional "is live" Discord hook (`LIVE_WEBHOOK_URL` in `.env`,
|
||
asked by `configure.py --full`) can post when a stream starts.
|
||
|
||
1. CNAME `live` → main host; `./configure.py --full` asks the hostname (also `--hosts … LIVE`).
|
||
2. Authentik: Proxy Provider (forward auth, single application, external host `https://live.HOST`) →
|
||
embedded outpost → group policy (e.g. everyone in the server, or `streamers`+`editors`+`admins`).
|
||
3. Dev box: `./deployLive.sh` builds `live/` and copies `dist/` to `/srv/live` (Caddy serves it directly).
|
||
|
||
**Go Live** button: broadcast a window from the browser (no OBS) as `<user>/HelmetCam-<date>` — key comes
|
||
from `/me/stream-key` (dumpd reads `/etc/vanguard/streamers.json`, identity from the Authentik header), so a
|
||
streamer's Authentik username must equal their MediaMTX name.
|
||
|
||
Bandwidth: no transcoding, every viewer gets the source bitrate (~15 Mbps) → ~30 concurrent viewers on
|
||
the 500 Mbps line. For a crowd, restream to Twitch from OBS.
|
||
|
||
## 5g. Authentik forward-auth for the Caddy-gated hosts (`live.`, `bot.`; later the main host)
|
||
|
||
Design: **Authentik authenticates once for the whole domain; Caddy authorizes per host** from the
|
||
`X-Authentik-Groups` header the outpost returns; **Discord roles fill the Authentik groups** at login.
|
||
|
||
1. Applications → Outposts → edit "authentik Embedded Outpost" → config `authentik_host: https://auth.HOST`.
|
||
2. Applications → Create with provider → Application `Vanguard sites`; **Proxy Provider**, authorization flow
|
||
implicit-consent, mode **Forward auth (domain level)**, authentication URL `https://auth.HOST`,
|
||
cookie domain = the parent domain (e.g. `dumpstervigilantes.rip`). Edit the Embedded Outpost →
|
||
Applications → add it → Update. One login now covers every gated host.
|
||
3. Per-host group requirement = `.env`: `BOT_REQUIRE_GROUP=bot-users` (default), `LIVE_REQUIRE_GROUP=`
|
||
(empty = any logged-in member). `./configure.py` apply regenerates the Caddy blocks; a user outside the
|
||
group gets a 403 from Caddy before the app sees the request.
|
||
4. Discord roles → groups: Directory → Groups: create `bot-users` (and keep `streamers`, `editors`, `admins`).
|
||
Discord source → Scopes: `guilds guilds.members.read`. Customization → Policies → Expression Policy
|
||
`discord-roles-to-groups` from `authentik/discord-roles-to-groups.py` (fill in the server ID and the
|
||
role-id → group table). Bind it to `default-source-enrollment` **and** `default-source-authentication`
|
||
(engine mode `all`, after `discord-guild-member`). From then on giving someone a Discord role gives them
|
||
the matching access on next login; removing it removes it.
|
||
5. Test in a private window: `https://live.HOST/` → Discord login → portal; `https://bot.HOST/` → 403 unless
|
||
you hold the bot role; `https://live.HOST/me/stream-key` shows your username.
|
||
|
||
Jellyfin and Gitea use OIDC providers instead, not this.
|
||
|
||
## 6. Editors
|
||
|
||
Browser: https://media.example.com/ → log in. Scrub any `.mp4` in the browser, download what you want.
|
||
WebDAV mount: same URL, same login. Upload finished work to `/finals`.
|
||
|
||
Every `.mp4` an editor sees is the same kind of file (see "Editing standard"). Raw clipper output that
|
||
needed converting is kept in a hidden `.src/` folder next to it (admins can show dotfiles in copyparty).
|
||
|
||
## 7. Home archive box
|
||
|
||
Copy `home/rclone-archive.sh` to the home machine, set the vars, run from cron nightly.
|
||
It connects over SFTP as `debian` (all media files are owned by uid 1000 so it can delete them).
|
||
Pulls everything older than 15 days off the server, including the hidden `.src/` raw files.
|
||
|
||
## Layout on the server
|
||
|
||
/srv/media/streams/<user>/<Title>/.live/<ts>.mp4 recording in progress (hidden from Jellyfin/copyparty)
|
||
/srv/media/streams/<user>/<Title>/2025-10-10_21-03-11.mp4 finished stream, flat MP4 published by the sweep
|
||
/srv/media/clips/<user>/clip.mp4 clip (synced as-is, or converted; raw removed)
|
||
/srv/media/clips/<user>/.stlocal Syncthing ignore list maintained by the sweep
|
||
/srv/media/clips/<user>/.src/clip.mkv raw source, only for clips that did NOT arrive via Syncthing
|
||
/srv/media/clips/<user>/clip.normalize-failed.log only if normalization failed; delete to retry
|
||
/srv/media/finals/ editors upload here
|
||
/srv/media/trash/<night>/... dumped, purged after 3 days
|
||
/srv/media/.markers/ dump markers
|
||
/srv/media/.syncthing/ Syncthing config
|
||
|
||
## Editing standard (what normalize-sweep enforces)
|
||
|
||
MP4 · H.264 High, 8-bit 4:2:0 · constant frame rate · AAC audio (all tracks kept) · index at the
|
||
front (instant seek in browser and NLE) · at most 2560x1440 and 60 fps (downscaled to fit, never
|
||
upscaled; aspect kept).
|
||
|
||
| input | action |
|
||
|-------|--------|
|
||
| already compliant | nothing |
|
||
| MediaMTX recording in `.live/` (fragmented MP4) | once idle 3 min: lossless remux to a flat MP4 in the title folder, `.live` source removed |
|
||
| in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
|
||
| HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
|
||
| ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |
|
||
|
||
Runs every 5 min on files idle ≥3 min and not open. Software x264 on the E-2136 does ~0.5x realtime
|
||
at 1440p60, so a 2 h clip batch takes ~4 h; streams only remux (minutes). All at idle priority.
|
||
|
||
## Reboot behaviour
|
||
|
||
Everything comes back on its own, in this order:
|
||
|
||
1. mdadm assembles the mirror, systemd mounts /srv (local-fs). A degraded mirror still boots.
|
||
2. docker.service has `RequiresMountsFor=/srv/media` (drop-in installed by install.sh), so it never
|
||
starts with the media tree missing. All containers are `restart: unless-stopped`.
|
||
3. dumpd.service starts; normalize-sweep.timer fires 2 min after boot and every 5 min after;
|
||
trash-purge.timer is `Persistent=true` so a missed 05:00 run fires at boot.
|
||
4. A stream cut by a reboot is playable up to the last second (fMP4 parts are flushed every 1 s);
|
||
OBS reconnects and MediaMTX opens a new file in the same title folder.
|
||
5. Half-written `*.part` files are overwritten by the next sweep. Dumps are idempotent.
|
||
|
||
Time: install.sh sets the server clock to America/New_York so journal, timers, file timestamps
|
||
and the dump window all read in Eastern. Dump "night" = 17:00 → 02:00 Eastern (pinned in dumpd via
|
||
DUMP_TZ, independent of the host zone). Between 02:00 and 17:00 the default night is the previous
|
||
evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
|
||
|
||
## Timers
|
||
|
||
| unit | when | does |
|
||
|------|------|------|
|
||
| normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
|
||
| trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
|
||
| dumpd.service | always | the dump endpoint on 127.0.0.1:8080, behind Caddy at /dump |
|