Vanguard media stack: ingest, copyparty, Syncthing, normalize sweep, Authentik, Jellyfin + Vanguard Metadata plugin, MariaDB, Gitea, bot runtime
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
255
README.md
Normal file
255
README.md
Normal file
@@ -0,0 +1,255 @@
|
||||
# vanguard-stream — SYS-1 (Xeon E-2136, 32 GB ECC, 2×4 TB SATA soft RAID)
|
||||
|
||||
> **STATUS.md** = what runs, who does what, what's unfinished (kept current). **CommunityDiscordMedia.md** = design + phases.
|
||||
|
||||
Ingest from OBS, browse/scrub/download for editors, automatic normalization to one editing
|
||||
standard, dump button, 15-day archive pull to a home HDD. Everything is plain files under /srv/media.
|
||||
|
||||
## 1. Reinstall from the OVH manager
|
||||
|
||||
Server → Reinstall → Debian 13 (Trixie).
|
||||
Tick **custom partitioning** and **use distribution kernel**. Layout (RAID 1 across both HDDs):
|
||||
|
||||
| mount | size | fs |
|
||||
|-------|------|----|
|
||||
| /boot | 1 GB | ext4 |
|
||||
| / | 250 GB | ext4 |
|
||||
| swap | 8 GB per disk | swap |
|
||||
| /srv | rest (~3.4 TiB) | xfs |
|
||||
|
||||
Add your SSH key. OVH's template creates a `debian` user with sudo; root SSH is disabled. Wait for the mail.
|
||||
|
||||
## 2. Install
|
||||
|
||||
scp -r vanguard-stream debian@SERVER:~/
|
||||
ssh debian@SERVER
|
||||
cd ~/vanguard-stream && sudo bash install.sh
|
||||
|
||||
install.sh installs docker, ffmpeg, rclone; creates /srv/media/{streams,clips,finals,trash,.markers}
|
||||
owned by uid 1000 (the `debian` user); copies the systemd units; adds `debian` to the docker group;
|
||||
starts the compose stack. It prints what you still have to edit.
|
||||
|
||||
The folder stays where it is: compose bind-mounts the configs and the dock page from it.
|
||||
After install, log out and back in once so the docker group applies, then plain `docker compose`
|
||||
works from ~/vanguard-stream without sudo.
|
||||
|
||||
## 3. Configure (domain, admin, streamers, editors)
|
||||
|
||||
cd ~/vanguard-stream
|
||||
./configure.py --full # first time: asks everything, then applies
|
||||
./configure.py # later: add/remove/reset streamers and editors, then apply
|
||||
./configure.py --show # print current users and the OBS settings to hand out
|
||||
|
||||
Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `copyparty.conf`,
|
||||
regenerates those three plus `dump-users.txt`, installs the dump users to `/etc/vanguard`, creates
|
||||
`streams/<name>` and `clips/<name>` (with `.stignore`) for each streamer, restarts dumpd and the
|
||||
containers. Removing a streamer or editor revokes access only; their files stay on disk.
|
||||
Passwords may not contain `: & ? # / \ quotes or spaces` (they ride inside the stream key).
|
||||
|
||||
## 4. Firewall (host nftables; OVH edge firewall is optional on top)
|
||||
|
||||
Applied automatically at the end of `install.sh`. To re-apply by hand:
|
||||
|
||||
cd ~/vanguard-stream && sudo bash firewall/firewall.sh
|
||||
|
||||
Installs `firewall/nftables.conf` as `/etc/nftables.conf`, syntax-checks, applies, enables at boot.
|
||||
It uses its own table (`inet vanguard`) so applying or reloading never disturbs Docker's rules,
|
||||
and filters input only so container networking keeps working. **Keep your SSH session open and
|
||||
test a new one before closing it.**
|
||||
|
||||
| port | proto | who |
|
||||
|------|-------|-----|
|
||||
| 22 | tcp | anywhere, rate limited (edit `ssh_allow` in /etc/nftables.conf to restrict to your IPs) |
|
||||
| 80, 443 | tcp | world (Caddy) |
|
||||
| 443 | udp | world (Caddy HTTP/3) |
|
||||
| 1935 | tcp | world (RTMP ingest) |
|
||||
| 8890 | udp | world (SRT ingest) |
|
||||
| 22000 | tcp+udp | world (Syncthing) |
|
||||
| 3306 | tcp | world, rate limited (MariaDB for the bot dev; restrict via `db_allow` in /etc/nftables.conf) |
|
||||
| 2222 | tcp | world (Gitea ssh) |
|
||||
|
||||
Everything else is dropped and a trickle is logged (`journalctl -k | grep vg-drop`). Authentik (9000),
|
||||
Jellyfin (8096), copyparty, the Syncthing UI and dumpd bind to 127.0.0.1 only and are reached via Caddy
|
||||
or an ssh tunnel, so they are deliberately not opened. MariaDB runs on the host network (not a Docker
|
||||
published port) precisely so the firewall can govern it.
|
||||
Syncthing's admin UI is proxied at `https://HOST/sync/` behind Syncthing's own GUI login (set it in
|
||||
Actions → Settings → GUI before relying on this). The ssh tunnel to 8384 still works as a fallback.
|
||||
|
||||
To change rules: edit `/etc/nftables.conf`, then `sudo nft -f /etc/nftables.conf` or
|
||||
`sudo systemctl reload nftables`. **Never `systemctl stop|restart nftables`** — Debian's unit flushes
|
||||
the whole ruleset on stop, which also wipes Docker's NAT rules and takes the containers offline
|
||||
until Docker is restarted.
|
||||
|
||||
## 5. Streamer setup (once)
|
||||
|
||||
OBS → Settings → Stream → Custom: Server `rtmp://media.example.com:1935/`, key anything (the dock sets it).
|
||||
OBS → Tools → WebSocket Server Settings → enable, set a password.
|
||||
OBS → Docks → Custom Browser Docks → `https://media.example.com/dock/`.
|
||||
In the dock: fill host/user/pass/OBS-ws-password once (stored in the dock's localStorage).
|
||||
Per stream: type a title, click **Go Live**. After a bad night: **Dump tonight**.
|
||||
|
||||
Stream key format the dock produces: `techy/Title?user=techy&pass=...`
|
||||
SRT equivalent if preferred: `srt://media.example.com:8890?streamid=publish:techy/Title:techy:PASS`
|
||||
|
||||
## 5b. Syncthing hardening (do once, in the server's Syncthing UI)
|
||||
|
||||
Syncthing receives straight into `clips/<user>` (Syncthing UI path `/var/syncthing/clips/<user>`),
|
||||
receive-only, with a `.stignore` (from `syncthing/stignore`) that pulls video files only. The gamer's
|
||||
folder structure is mirrored as-is. **One copy per clip on the server**: compliant files are left
|
||||
alone; a non-compliant file (HEVC/MKV/VFR) is converted to `name.mp4` (`name.h264.mp4` if the raw was
|
||||
already `.mp4`), the raw's name is appended to `.stlocal` (an `#include`d ignore list, so Syncthing
|
||||
drops it without flagging or re-pulling), and the raw is deleted locally — it still exists on the
|
||||
gamer's PC. `configure.py` creates the folder + ignore file for each streamer.
|
||||
|
||||
In the UI (https://HOST/sync/):
|
||||
- each clips folder → path `/var/syncthing/clips/<user>`, **Folder Type: Receive Only**,
|
||||
Edit → Advanced → **Minimum Free Disk Space: 300 GB** (so clips can never fill the mirror and stall
|
||||
the recordings).
|
||||
- Actions → Settings → Connections → **Incoming Rate Limit: 12500 KiB/s** (~100 Mbps) so a gamer
|
||||
sharing a year of old clips cannot starve the ingest during stream hours.
|
||||
|
||||
## 5c. Authentik (Discord login) and Jellyfin
|
||||
|
||||
DNS: `auth` and `watch` as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
|
||||
for both hostnames, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
|
||||
adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
|
||||
hostnames filled in. (`./configure.py --full` asks for the same hostnames if you prefer.) Then:
|
||||
|
||||
**Authentik first run** — https://auth.HOST/if/flow/initial-setup/ → set the `akadmin` password.
|
||||
1. Discord developer portal → New Application → OAuth2: add redirect
|
||||
`https://auth.HOST/source/oauth/callback/discord/` (last segment = the source slug below), copy Client ID + Secret.
|
||||
2. Authentik → Directory → Federation & Social login → Create → Discord OAuth Source: slug `discord`,
|
||||
consumer key/secret = Client ID/Secret, user matching "link on unique identifier", scopes empty.
|
||||
3. Flows & Stages → Stages → `default-authentication-identification` → Sources: select Discord,
|
||||
tick "Show sources' labels". Without this the Discord button never appears on the login page.
|
||||
4. Flows & Stages → Stages → `default-source-enrollment-write` → User type: Internal. Otherwise
|
||||
Discord users are External and get "Interface can only be accessed by internal users".
|
||||
5. Directory → Groups: create `streamers`, `editors`, `admins`. Add people after their first login.
|
||||
6. (optional) Restrict login to members of your Discord server: add scope `guilds` to the source;
|
||||
Customization → Policies → Create Expression Policy from `authentik/discord-guild-policy.py` (fill in
|
||||
your server ID); bind it to both `default-source-enrollment` and `default-source-authentication`
|
||||
(flow → Policy / Group / User Bindings) and set each flow's Policy engine mode to `all`. Server ID: Discord Developer Mode → right-click server → Copy ID.
|
||||
|
||||
**Jellyfin first run** — https://watch.HOST/ → wizard: admin user, add libraries: `/media/streams` and
|
||||
`/media/clips` as **Shows** with only "Vanguard Metadata" as metadata downloader (Series/Season/Episode)
|
||||
and Screen Grabber + Vanguard Metadata as image fetchers; `/media/finals` as Home Videos with fetchers off.
|
||||
The Vanguard Metadata plugin (`jellyfin-plugin-vanguard/`, built with its `package.sh`) is installed by
|
||||
`addCommunityMediaServer.sh` / `installJellyfinPlugin.sh` from `jellyfin-plugin-vanguard/dist/`.
|
||||
- Dashboard → Playback → Transcoding: **disable** hardware acceleration. Per-user "allow transcoding" is
|
||||
switched off automatically for everyone (SSO users included) by the Vanguard plugin's "enforce playback
|
||||
policy" task; turn that toggle off in the plugin config once the server has Quick Sync (SYS-3).
|
||||
- Dashboard → Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json`, then Catalog → **SSO Authentication** → install → restart Jellyfin.
|
||||
- Authentik → Applications → Create with provider → OAuth2/OpenID: redirect
|
||||
`https://watch.HOST/sso/OID/redirect/authentik`, note client ID/secret.
|
||||
- Jellyfin → Plugins → SSO-Auth → add OID provider "authentik": endpoint `https://auth.HOST/application/o/<slug>/`,
|
||||
client ID/secret, enable, roles claim `groups`, admin roles `admins`, enabled folders = the three libraries,
|
||||
Scheme Override `https`, Port Override empty (a value appends `:443` and Authentik rejects the redirect). Save.
|
||||
- The plugin adds no button. Test with `https://watch.HOST/sso/OID/start/authentik`, then Dashboard →
|
||||
General → Branding: Login disclaimer = `jellyfin/branding-disclaimer.html` (set the hostname),
|
||||
Custom CSS = `jellyfin/branding.css`. The page then shows only "Sign in with Discord"; a small
|
||||
"Admin login" link reveals the local username/password form.
|
||||
Jellyfin creates the account on first SSO login.
|
||||
|
||||
Phase 4 is the Vanguard Metadata Jellyfin plugin; Phase 5 puts copyparty, /sync and /dump behind Authentik forward-auth.
|
||||
|
||||
## 5d. MariaDB + Discord bot (for the bot dev)
|
||||
|
||||
`mariadb` (public on 3306, host network, root local-only, 32-char generated passwords) with a database/user
|
||||
for the bot, and `discord-bot`, a Python 3.12 container that runs `bot/app/main.py` and idles until that file
|
||||
exists. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
|
||||
or the `[d]` menu item; `./configure.py --show` prints them).
|
||||
|
||||
The bot also serves a web UI on 127.0.0.1:8000, published as `https://bot.HOST/` behind **Authentik
|
||||
forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`. Create the matching
|
||||
Proxy Provider in Authentik (forward auth, single application, external host `https://bot.HOST`), add it to
|
||||
the embedded outpost, and bind a group policy. Until then the site returns an error, never an open page.
|
||||
|
||||
The build brief for whoever writes the bot is `bot/AGENT.md`; `bot/README.md` is the ops cheat-sheet.
|
||||
|
||||
docker compose logs -f discord-bot # watch it
|
||||
docker compose restart discord-bot # after code changes
|
||||
docker compose build discord-bot && docker compose up -d discord-bot # after requirements.txt changes
|
||||
|
||||
## 5e. Gitea + Actions runner
|
||||
|
||||
`gitea` (MariaDB-backed, `git.HOST` via Caddy, git-over-ssh on **2222**, registration closed) and
|
||||
`act_runner` (jobs run as Docker containers; labels `ubuntu-latest`, `ubuntu-22.04`, `dotnet`, `python`).
|
||||
The gitea db/user are created by `mariadb/init/10-gitea.sh` on MariaDB's first start; for a MariaDB that
|
||||
already existed run `./createGiteaDb.sh` once.
|
||||
|
||||
1. `./configure.py --full` asks the Gitea hostname (CNAME `git`); apply starts both containers.
|
||||
2. https://git.HOST/ → install page (db pre-filled) → create the admin account → Install.
|
||||
3. `./registerGiteaRunner.sh` → generates a registration token, stores it in `.env`, starts the runner.
|
||||
4. Remotes: `ssh://git@git.HOST:2222/<org>/<repo>.git` or `https://git.HOST/<org>/<repo>.git`.
|
||||
|
||||
Runner caveat: it holds the Docker socket, so anyone who can push a workflow can run as root on the box.
|
||||
Keep repo write access to trusted people.
|
||||
|
||||
## 6. Editors
|
||||
|
||||
Browser: https://media.example.com/ → log in. Scrub any `.mp4` in the browser, download what you want.
|
||||
WebDAV mount: same URL, same login. Upload finished work to `/finals`.
|
||||
|
||||
Every `.mp4` an editor sees is the same kind of file (see "Editing standard"). Raw clipper output that
|
||||
needed converting is kept in a hidden `.src/` folder next to it (admins can show dotfiles in copyparty).
|
||||
|
||||
## 7. Home archive box
|
||||
|
||||
Copy `home/rclone-archive.sh` to the home machine, set the vars, run from cron nightly.
|
||||
It connects over SFTP as `debian` (all media files are owned by uid 1000 so it can delete them).
|
||||
Pulls everything older than 15 days off the server, including the hidden `.src/` raw files.
|
||||
|
||||
## Layout on the server
|
||||
|
||||
/srv/media/streams/<user>/<Title>/2025-10-10_21-03-11.mp4 stream, remuxed flat (lossless) by the sweep
|
||||
/srv/media/clips/<user>/clip.mp4 clip (synced as-is, or converted; raw removed)
|
||||
/srv/media/clips/<user>/.stlocal Syncthing ignore list maintained by the sweep
|
||||
/srv/media/clips/<user>/.src/clip.mkv raw source, only for clips that did NOT arrive via Syncthing
|
||||
/srv/media/clips/<user>/clip.normalize-failed.log only if normalization failed; delete to retry
|
||||
/srv/media/finals/ editors upload here
|
||||
/srv/media/trash/<night>/... dumped, purged after 3 days
|
||||
/srv/media/.markers/ dump markers
|
||||
/srv/media/.syncthing/ Syncthing config
|
||||
|
||||
## Editing standard (what normalize-sweep enforces)
|
||||
|
||||
MP4 · H.264 High, 8-bit 4:2:0 · constant frame rate · AAC audio (all tracks kept) · index at the
|
||||
front (instant seek in browser and NLE) · at most 2560x1440 and 60 fps (downscaled to fit, never
|
||||
upscaled; aspect kept).
|
||||
|
||||
| input | action |
|
||||
|-------|--------|
|
||||
| already compliant | nothing |
|
||||
| MediaMTX recording (fragmented MP4, good codecs) | lossless remux to flat MP4, replaced in place |
|
||||
| in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
|
||||
| HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
|
||||
| ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |
|
||||
|
||||
Runs every 5 min on files idle ≥3 min and not open. Software x264 on the E-2136 does ~0.5x realtime
|
||||
at 1440p60, so a 2 h clip batch takes ~4 h; streams only remux (minutes). All at idle priority.
|
||||
|
||||
## Reboot behaviour
|
||||
|
||||
Everything comes back on its own, in this order:
|
||||
|
||||
1. mdadm assembles the mirror, systemd mounts /srv (local-fs). A degraded mirror still boots.
|
||||
2. docker.service has `RequiresMountsFor=/srv/media` (drop-in installed by install.sh), so it never
|
||||
starts with the media tree missing. All containers are `restart: unless-stopped`.
|
||||
3. dumpd.service starts; normalize-sweep.timer fires 2 min after boot and every 5 min after;
|
||||
trash-purge.timer is `Persistent=true` so a missed 05:00 run fires at boot.
|
||||
4. A stream cut by a reboot is playable up to the last second (fMP4 parts are flushed every 1 s);
|
||||
OBS reconnects and MediaMTX opens a new file in the same title folder.
|
||||
5. Half-written `*.part` files are overwritten by the next sweep. Dumps are idempotent.
|
||||
|
||||
Time: install.sh sets the server clock to America/New_York so journal, timers, file timestamps
|
||||
and the dump window all read in Eastern. Dump "night" = 17:00 → 02:00 Eastern (pinned in dumpd via
|
||||
DUMP_TZ, independent of the host zone). Between 02:00 and 17:00 the default night is the previous
|
||||
evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
|
||||
|
||||
## Timers
|
||||
|
||||
| unit | when | does |
|
||||
|------|------|------|
|
||||
| normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
|
||||
| trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
|
||||
| dumpd.service | always | the dump endpoint on 127.0.0.1:8080, behind Caddy at /dump |
|
||||
Reference in New Issue
Block a user