Brandyn / Techy
25e9078d9b
switchToClone no longer tries to carry bot/app across a clone, and the compose comment, STATUS row, README and the plan doc all describe the installed-release model instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
632 lines
46 KiB
Markdown
632 lines
46 KiB
Markdown
# vanguard-stream — SYS-1 (Xeon E-2136, 32 GB ECC, 2×4 TB SATA soft RAID)
|
||
|
||
> **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included.
|
||
> **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely.
|
||
> **LOGGING.md** = log framework: journald + rotated Docker logs, container labels, logfmt events, categories; store (Loki/VictoriaLogs/Netdata) chosen later.
|
||
> **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app).
|
||
|
||
**Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
|
||
|
||
| Host | What | Who |
|
||
|---|---|---|
|
||
| `media.BASE` (main) | gateway page at `/` (role-aware links + how-tos), `/whoami`, `/logout` | everyone |
|
||
| `files.BASE` | copyparty: browse, scrub, download, upload finals | editors / media-admins |
|
||
| `clips.BASE` | self-service clip intake (pair your Syncthing) | any member |
|
||
| `sync.BASE` | Syncthing admin UI | media-admins |
|
||
| `live.BASE` | live portal, Go Live, OBS dock `/dock/`, clock source `/dock/clock/`, `/dump`, `/me`, `/time` | everyone (`LIVE_REQUIRE_GROUP`) |
|
||
| `watch.BASE` | Jellyfin | Discord via OIDC |
|
||
| `git.BASE`, `bot.BASE`, `auth.BASE` | Gitea, bot web UI (`bot-users`), Authentik | — |
|
||
|
||
`/time` (the server clock the timecode strip syncs to) lives only on `live` because browsers fetch it
|
||
same-origin: both pages that use it, the clock source and Go Live, are on that host.
|
||
> **STATUS.md** = what runs, who does what, what's unfinished (kept current). **CommunityDiscordMedia.md** = design + phases.
|
||
|
||
Ingest from OBS, browse/scrub/download for editors, automatic normalization to one editing
|
||
standard, dump button, 15-day archive pull to a home HDD. Everything is plain files under /srv/media.
|
||
|
||
## 1. Reinstall from the OVH manager
|
||
|
||
Server → Reinstall → Debian 13 (Trixie).
|
||
Tick **custom partitioning** and **use distribution kernel**. Layout (RAID 1 across both HDDs):
|
||
|
||
| mount | size | fs |
|
||
|-------|------|----|
|
||
| /boot | 1 GB | ext4 |
|
||
| / | 250 GB | ext4 |
|
||
| swap | 8 GB per disk | swap |
|
||
| /srv | rest (~3.4 TiB) | xfs |
|
||
|
||
Add your SSH key. OVH's template creates a `debian` user with sudo; root SSH is disabled. Wait for the mail.
|
||
|
||
## 2. Install
|
||
|
||
scp -r vanguard-stream debian@SERVER:~/
|
||
ssh debian@SERVER
|
||
cd ~/vanguard-stream && sudo bash install.sh
|
||
|
||
install.sh installs docker, ffmpeg, rclone; creates /srv/media/{streams,clips,finals,trash,.markers}
|
||
owned by uid 1000 (the `debian` user); copies the systemd units; adds `debian` to the docker group;
|
||
starts the compose stack. It prints what you still have to edit.
|
||
|
||
The folder stays where it is: compose bind-mounts the configs and the dock page from it.
|
||
After install, log out and back in once so the docker group applies, then plain `docker compose`
|
||
works from ~/vanguard-stream without sudo.
|
||
|
||
## 3. Configure (domain, admin, streamers, editors)
|
||
|
||
> `mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt` and `.env` are **generated on the server**
|
||
> by `configure.py` and gitignored. The `*.example.*` files in the repo are reference only. **Never copy them
|
||
> to the server**: doing so once replaced the live streamer passwords with `CHANGEME` and the next apply
|
||
> wiped the streamers. To update the server, copy scripts (`configure.py`, `bin/`, `docker-compose.yml`, …)
|
||
> and re-run `./configure.py`.
|
||
|
||
cd ~/vanguard-stream
|
||
./configure.py --full # first time: asks everything, then applies
|
||
./configure.py # later: add/remove/reset streamers and editors, then apply
|
||
./configure.py --show # print current users and the OBS settings to hand out
|
||
|
||
Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates
|
||
`Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates
|
||
`streams/<name>` and `clips/<name>` (with `.stignore`) for each streamer, restarts dumpd and the
|
||
containers. Editors and admins are Authentik groups (`editors`, `media-admins`, `admins`), not local
|
||
accounts. Removing a streamer revokes publishing only; their files stay on disk.
|
||
Passwords may not contain `: & ? # / \ quotes or spaces` (they ride inside the stream key).
|
||
|
||
## 4. Firewall (host nftables; OVH edge firewall is optional on top)
|
||
|
||
Applied automatically at the end of `install.sh`. To re-apply by hand:
|
||
|
||
cd ~/vanguard-stream && sudo bash firewall/firewall.sh
|
||
|
||
Installs `firewall/nftables.conf` as `/etc/nftables.conf`, syntax-checks, applies, enables at boot.
|
||
It uses its own table (`inet vanguard`) so applying or reloading never disturbs Docker's rules,
|
||
and filters input only so container networking keeps working. **Keep your SSH session open and
|
||
test a new one before closing it.**
|
||
|
||
| port | proto | who |
|
||
|------|-------|-----|
|
||
| 22 | tcp | anywhere, rate limited (edit `ssh_allow` in /etc/nftables.conf to restrict to your IPs) |
|
||
| 80, 443 | tcp | world (Caddy) |
|
||
| 443 | udp | world (Caddy HTTP/3) |
|
||
| 1935 | tcp | world (RTMP ingest) |
|
||
| 8890 | udp | world (SRT ingest) |
|
||
| 22000 | tcp+udp | world (Syncthing) |
|
||
| 3306 | tcp | world, rate limited (MariaDB for the bot dev; restrict via `db_allow` in /etc/nftables.conf) |
|
||
| 2222 | tcp | world (Gitea ssh) |
|
||
| 8189 | tcp+udp | world (WebRTC media for the live portal) |
|
||
|
||
Everything else is dropped and a trickle is logged (`journalctl -k | grep vg-drop`). Authentik (9000),
|
||
Jellyfin (8096), copyparty, the Syncthing UI and dumpd bind to 127.0.0.1 only and are reached via Caddy
|
||
or an ssh tunnel, so they are deliberately not opened. MariaDB runs on the host network (not a Docker
|
||
published port) precisely so the firewall can govern it.
|
||
Syncthing's admin UI is proxied at `https://sync.BASE/` behind the Authentik gate + group `media-admins`; its
|
||
own GUI login is disabled (no user/password, `insecureAdminAccess=true`) so Discord is the only login:
|
||
|
||
docker exec syncthing syncthing cli --home /var/syncthing/config config gui insecure-admin-access set true
|
||
docker exec syncthing syncthing cli --home /var/syncthing/config config gui user set ""
|
||
docker exec syncthing syncthing cli --home /var/syncthing/config config gui password set ""
|
||
|
||
The ssh tunnel to 8384 still works as an admin fallback (no login at all — ssh is the gate there).
|
||
|
||
To change rules: edit `/etc/nftables.conf`, then `sudo nft -f /etc/nftables.conf` or
|
||
`sudo systemctl reload nftables`. **Never `systemctl stop|restart nftables`** — Debian's unit flushes
|
||
the whole ruleset on stop, which also wipes Docker's NAT rules and takes the containers offline
|
||
until Docker is restarted.
|
||
|
||
## 5. Streamer setup (once)
|
||
|
||
Being a streamer = being in the Authentik **streamers** group (a Discord role via the roles→groups policy).
|
||
There are no stream passwords or keys to hand out: the dock mints a **12-hour publish token** for the
|
||
logged-in Discord user each time Go Live is pressed and sets it as the OBS stream key itself.
|
||
|
||
OBS → Settings → Output → Encoder: **B-frames = 0** (NVENC "Max B-frames: 0"; x264 add `bframes=0`). MediaMTX
|
||
refuses WebRTC playback of H.264 with B-frames ("WebRTC doesn't support H264 streams with B-frames"), so the live
|
||
portal would fall back to HLS with seconds of delay. Keyframe interval 2 s. Recording quality is unaffected.
|
||
OBS → Tools → WebSocket Server Settings → enable, set a password.
|
||
OBS → Docks → Custom Browser Docks → `https://live.BASE/dock/` → log in with Discord once.
|
||
In the dock's Settings: ingest host (prefilled), OBS WebSocket password/port, optional title prefix/suffix
|
||
(`{date}` → YYYY-MM-DD, e.g. prefix `HelmetCam-` or suffix `-{date}`).
|
||
Per stream: type a title, click **Go Live**. The dock shows the final stream name (`<you>/<Title>`) as you type
|
||
and enforces the naming rules the metadata relies on: letters, digits, `_ . -` only, no double underscore
|
||
(it separates title and timestamp in filenames), no leading dot, max 64 chars.
|
||
After a bad night: **Dump tonight** (dumps the night of the account you are logged in as; `admins`/`media-admins`
|
||
may dump anyone's).
|
||
|
||
Manual OBS/other software: dock → **Manual OBS setup** shows the RTMP server, the stream key and the SRT URL
|
||
with the token hidden (Show / Copy / Fresh token). Tokens expire after 12 h; a running stream is not cut when
|
||
its token expires, but a reconnect needs a fresh one.
|
||
|
||
Browser streaming without OBS: the live portal's **Go Live** (Windows Chrome 141+ for window audio + GPU encode).
|
||
|
||
## 5a. Onboarding a gamer's clips (Syncthing) — self-service
|
||
|
||
Gamers do it themselves at **https://clips.BASE/** (Discord login): the page shows our Syncthing Device ID and
|
||
asks for theirs; on submit, dumpd creates `clips/<username>` with the video-only `.stignore`, adds their device
|
||
to our Syncthing (REST API, key read from `/srv/media/.syncthing/config.xml`) and creates the receive-only
|
||
folder `clips-<username>` shared with them (min free 300 GB). They accept the device + folder on their PC,
|
||
point it at their clip directory, Send Only. The page shows pairing/connection/sync state. Idempotent.
|
||
**Per-user quota (soft)**: dumpd measures every `clips/<user>` every 5 min and pauses that user's Syncthing
|
||
folder when it exceeds the quota (default `CLIPS_QUOTA_GB=100` in `dumpd.service`), resuming automatically
|
||
under 90 %. The clips page shows usage and the paused state. Overrides per user in `/etc/vanguard/clip-quotas.json`,
|
||
e.g. `{"techy2493": 250, "newguy": 50}` (GB) — read live, no restart. A gamer frees space by deleting clips in
|
||
their own folder (Syncthing mirrors the deletion). Hard byte-exact quotas would be XFS project quotas on `/srv`
|
||
(needs a remount with `prjquota`); not set up.
|
||
Manual fallback (admin, same result for the server folder):
|
||
|
||
./addClipUser.sh <username> # creates /srv/media/clips/<username> with the video-only .stignore
|
||
|
||
Their side (Windows: Syncthing installer or SyncTrayzor; Mac/Linux: Syncthing): open the UI, **Actions → Show ID**,
|
||
send you the Device ID. In their UI: **Add Remote Device** with the server's ID (ours: `sync.BASE` → Actions → Show ID),
|
||
then **Add Folder** on their clip directory (ShadowPlay/Medal/Steam saves), Sharing tab → tick the server,
|
||
Folder Type **Send Only**, Save.
|
||
|
||
Our side at https://sync.BASE/: **Add Remote Device** → paste their ID, name it after them, Save. When their folder
|
||
offer pops up: **Add** → Folder Path `/var/syncthing/clips/<username>`, Folder Type **Receive Only**, Advanced →
|
||
Minimum Free Disk Space **300 GB**, Save. Clips start arriving; the sweep normalizes them; Jellyfin shows the
|
||
folder as that person's show. Their own folder structure is mirrored; only video files are pulled.
|
||
|
||
## 5a². Gateway page — https://media.BASE/ (the main host root; `/about/` redirects here)
|
||
|
||
Role-aware landing page (Discord login): everyone sees Watch / Live / Send clips / Sign out with short how-tos;
|
||
`streamers` get the dock + OBS setup steps; `editors` the browser + multicam/timecode notes; `media-admins`
|
||
trash/Syncthing/whoami; `git-users` Gitea; `bot-users` the bot; `admins`/`authentik Admins` the admin consoles and
|
||
where things live. Identity comes from `GET /me` (dumpd). Point people here first.
|
||
|
||
## 5b. Syncthing hardening (do once, in the server's Syncthing UI)
|
||
|
||
Syncthing receives straight into `clips/<user>` (Syncthing UI path `/var/syncthing/clips/<user>`),
|
||
receive-only, with a `.stignore` (from `syncthing/stignore`) that pulls video files only. The gamer's
|
||
folder structure is mirrored as-is. **One copy per clip on the server**: compliant files are left
|
||
alone; a non-compliant file (HEVC/MKV/VFR) is converted to `name.mp4` (`name.h264.mp4` if the raw was
|
||
already `.mp4`), the raw's name is appended to `.stlocal` (an `#include`d ignore list, so Syncthing
|
||
drops it without flagging or re-pulling), and the raw is deleted locally — it still exists on the
|
||
gamer's PC. `configure.py` creates the folder + ignore file for each streamer.
|
||
|
||
In the UI (https://sync.BASE/):
|
||
- each clips folder → path `/var/syncthing/clips/<user>`, **Folder Type: Receive Only**,
|
||
Edit → Advanced → **Minimum Free Disk Space: 300 GB** (so clips can never fill the mirror and stall
|
||
the recordings).
|
||
- Actions → Settings → Connections → **Incoming Rate Limit: 12500 KiB/s** (~100 Mbps) so a gamer
|
||
sharing a year of old clips cannot starve the ingest during stream hours.
|
||
|
||
## 5c. Authentik (Discord login) and Jellyfin
|
||
|
||
DNS: `auth`, `watch` (and `files`, `clips`, `sync`, `live`, `git`, `bot`) as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
|
||
for every hostname, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
|
||
adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
|
||
hostnames filled in. (`./configure.py --full` asks for the same hostnames if you prefer.) Then:
|
||
|
||
**Authentik first run** — https://auth.HOST/if/flow/initial-setup/ → set the `akadmin` password.
|
||
1. Discord developer portal → New Application → OAuth2: add redirect
|
||
`https://auth.HOST/source/oauth/callback/discord/` (last segment = the source slug below), copy Client ID + Secret.
|
||
2. Authentik → Directory → Federation & Social login → Create → Discord OAuth Source: slug `discord`,
|
||
consumer key/secret = Client ID/Secret, user matching "link on unique identifier", scopes empty.
|
||
3. Flows & Stages → Stages → `default-authentication-identification` → Sources: select Discord,
|
||
tick "Show sources' labels". Without this the Discord button never appears on the login page.
|
||
4. Flows & Stages → Stages → `default-source-enrollment-write` → User type: Internal. Otherwise
|
||
Discord users are External and get "Interface can only be accessed by internal users".
|
||
5. Directory → Groups: create `streamers`, `editors`, `admins`. Add people after their first login.
|
||
6. (optional) Restrict login to members of your Discord server: add scope `guilds` to the source;
|
||
Customization → Policies → Create Expression Policy from `authentik/discord-guild-policy.py` (fill in
|
||
your server ID); bind it to both `default-source-enrollment` and `default-source-authentication`
|
||
(flow → Policy / Group / User Bindings) and set each flow's Policy engine mode to `all`. Server ID: Discord Developer Mode → right-click server → Copy ID.
|
||
|
||
**Jellyfin first run** — https://watch.HOST/ → wizard: admin user, add libraries: `/media/streams` and
|
||
`/media/clips` as **Shows** with only "Vanguard Metadata" as metadata downloader (Series/Season/Episode)
|
||
and Screen Grabber + Vanguard Metadata as image fetchers; `/media/finals` as Home Videos with fetchers off.
|
||
The Vanguard Metadata plugin (`jellyfin-plugin-vanguard/`, built with its `package.sh`) is installed by
|
||
`addCommunityMediaServer.sh` / `installJellyfinPlugin.sh` from `jellyfin-plugin-vanguard/dist/`.
|
||
- Dashboard → Playback → Transcoding: **disable** hardware acceleration. Per-user "allow transcoding" is
|
||
switched off automatically for everyone (SSO users included) by the Vanguard plugin's "enforce playback
|
||
policy" task; turn that toggle off in the plugin config once the server has Quick Sync (SYS-3).
|
||
- Dashboard → Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json`, then Catalog → **SSO Authentication** → install → restart Jellyfin.
|
||
- Authentik → Applications → Create with provider → OAuth2/OpenID: redirect
|
||
`https://watch.HOST/sso/OID/redirect/authentik`, note client ID/secret.
|
||
- Jellyfin → Plugins → SSO-Auth → add OID provider "authentik": endpoint `https://auth.HOST/application/o/<slug>/`,
|
||
client ID/secret, enable, roles claim `groups`, admin roles `admins`, enabled folders = the three libraries,
|
||
Scheme Override `https`, Port Override empty (a value appends `:443` and Authentik rejects the redirect). Save.
|
||
- The plugin adds no button. Test with `https://watch.HOST/sso/OID/start/authentik`, then Dashboard →
|
||
General → Branding: Login disclaimer = `jellyfin/branding-disclaimer.html` (set the hostname),
|
||
Custom CSS = `jellyfin/branding.css`. The page then shows only "Sign in with Discord"; a small
|
||
"Admin login" link reveals the local username/password form.
|
||
Jellyfin creates the account on first SSO login.
|
||
|
||
Phase 4 is the Vanguard Metadata Jellyfin plugin; Phase 5 puts copyparty, /sync and /dump behind Authentik forward-auth.
|
||
|
||
## 5d. MariaDB + Discord bot (for the bot dev)
|
||
|
||
`mariadb` (public on 3306, host network, root local-only, 32-char generated passwords) with a database/user
|
||
for the bot, and `discord-bot`, a Python 3.12 container that runs `/app/main.py` and idles until a release is
|
||
installed. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
|
||
or the `[d]` menu item; `./configure.py --show` prints them).
|
||
|
||
**Releases are self-contained bundles, and a push to `main` deploys itself.** The bot repo's Gitea Actions
|
||
workflow runs its tests and then builds one tar.gz holding `main.py`, `app/`, `website/`, `BUILD.json` **and
|
||
`vendor/`** — its own dependencies, as prebuilt wheels for this image's Python. `PYTHONPATH=/app/vendor` is how
|
||
the code finds them, so **a new dependency never needs an image rebuild** and the server never contacts a
|
||
package index.
|
||
|
||
The workflow's last job posts that bundle to **dumpd** at `POST /deploy-bot` on 8081 — the Docker-bridge-only
|
||
port the firewall already restricts, so a CI job never needs the Docker socket or an ssh key. dumpd checks an
|
||
HMAC-SHA256 signature over the body, then runs `/usr/local/bin/vanguard-deploy-bot` (this repo's
|
||
`deployBotBundle.sh`), which verifies every file against `BUILD.json`, swaps the bundle into `/srv/bot/app`
|
||
(bind-mounted at `/app`, previous release kept beside it) and restarts the container. The install log goes back
|
||
in the HTTP response, so it appears in the CI job.
|
||
|
||
**The bot's own secrets live in Gitea, not here.** The deploy job also sends the bot's settings, signed with
|
||
the bundle, and dumpd writes them into this stack's `.env` — but only the keys `BOT_ENV_KEYS` lists. Anything
|
||
else comes back to the CI job as `settings_refused` and is not written, so the database password and the
|
||
Authentik keys this repo generates cannot be overwritten from CI. Rotating the Discord token is therefore a
|
||
Gitea edit plus a re-run, with no server access.
|
||
|
||
In Gitea, on the bot repo under Settings → Actions:
|
||
|
||
| | |
|
||
|---|---|
|
||
| Secret `BOT_DEPLOY_SECRET` | must equal the server's; `./configure.py --show` prints it |
|
||
| Secret `DISCORD_BOT_TOKEN` | the bot token |
|
||
| Variable `DISCORD_GUILD_ID` | the server id |
|
||
| Variable `VANGUARD_DEPLOY_HOST` | only if the job cannot detect the Docker gateway |
|
||
|
||
On the server, generated into `.env` and copied to `/etc/vanguard/deploy.env` by `configure.py`:
|
||
|
||
| | |
|
||
|---|---|
|
||
| `BOT_DEPLOY_SECRET` | 64 hex chars, the shared signing key |
|
||
| `BOT_DEPLOY_AUTO` | `1` (default) accepts CI installs; `0` makes dumpd refuse them with 403 |
|
||
| `BOT_ENV_KEYS` | the only `.env` keys CI may set, `DISCORD_BOT_TOKEN,DISCORD_GUILD_ID` by default. Extend it when the bot needs a new setting. |
|
||
|
||
By hand, any time:
|
||
|
||
./deployBotBundle.sh vanguard-bot-bundle.zip # install a release (the artifact as Gitea hands it over, zip or tar.gz)
|
||
./deployBotBundle.sh --latest # fetch the newest main build first (needs BOT_REPO_TOKEN in .env)
|
||
./deployBotBundle.sh --status # version, commit, package list, container state
|
||
./deployBotBundle.sh --rollback # back to the previous bundle
|
||
docker compose logs -f discord-bot # watch it
|
||
|
||
The bot serves **both** its API and its web pages on 127.0.0.1:8000, published as `https://bot.HOST/` behind
|
||
**Authentik forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`, and the
|
||
bot authorises per request from those headers (`WEB_ALLOWED_GROUPS=bot-users,admins`; `WEB_SERVE_UI=true`
|
||
turns the pages on). Create the matching Proxy Provider in Authentik (forward auth, single application,
|
||
external host `https://bot.HOST`), add it to the embedded outpost, and bind a group policy. Until then the
|
||
site returns an error, never an open page. Caddy's own gate is `BOT_REQUIRE_GROUP` (`bot-users` by default),
|
||
so **admins need the `bot-users` role too** — or set `BOT_REQUIRE_GROUP=` and let the bot's group list decide.
|
||
|
||
The build brief for whoever writes the bot is `bot/AGENT.md`; `bot/README.md` is the ops cheat-sheet.
|
||
|
||
## 5e. Gitea + Actions runner
|
||
|
||
`gitea` (MariaDB-backed, `git.HOST` via Caddy, git-over-ssh on **2222**, registration closed) and
|
||
`act_runner` (jobs run as Docker containers; labels `ubuntu-latest`, `ubuntu-22.04`, `dotnet`, `python`).
|
||
The gitea db/user are created by `mariadb/init/10-gitea.sh` on MariaDB's first start; for a MariaDB that
|
||
already existed run `./createGiteaDb.sh` once.
|
||
|
||
1. `./configure.py --full` asks the Gitea hostname (CNAME `git`); apply starts both containers.
|
||
2. https://git.HOST/ → install page (db pre-filled) → create the admin account → Install.
|
||
3. `./registerGiteaRunner.sh` → generates a registration token, stores it in `.env`, starts the runner.
|
||
4. Remotes: `ssh://git@git.HOST:2222/<org>/<repo>.git` or `https://git.HOST/<org>/<repo>.git`.
|
||
|
||
Runner caveat: it holds the Docker socket, so anyone who can push a workflow can run as root on the box.
|
||
Keep repo write access to trusted people.
|
||
|
||
**Login via Authentik (Discord)** — same pattern as Jellyfin, done once:
|
||
1. Authentik → Applications → Create with provider → name `Gitea`, slug `gitea`; **OAuth2/OpenID Provider**,
|
||
authorization flow implicit-consent, client type Confidential, redirect URI (strict)
|
||
`https://git.HOST/user/oauth2/authentik/callback`, signing key = the default certificate, scopes
|
||
`openid email profile` (the default profile mapping carries `preferred_username` and `groups`). Copy client ID + secret.
|
||
2. Gitea → Site Administration → Identity & Access → Authentication Sources → Add: type **OAuth2**, name
|
||
**`authentik`** (must match the callback path), provider **OpenID Connect**, client ID/secret, OpenID Connect
|
||
Auto Discovery URL `https://auth.HOST/application/o/gitea/.well-known/openid-configuration`, tick "Skip local
|
||
2FA", claim name for groups `groups`, administrator group claim value **`git-admins`**. Save.
|
||
Authentik group nesting makes the admins inherit it: `authentik Admins` → parent `admins` → parent `git-admins`
|
||
(members of a child group count as members of the parent). **Inherited groups are not in the `groups` claim by
|
||
default**: the managed profile mapping uses `ak_groups` (direct memberships only), and proxy providers re-add
|
||
that mapping on every save, so it cannot be swapped out. Add a second scope mapping, Customization → Property
|
||
Mappings → Create → Scope Mapping, name `Vanguard profile (nested groups)`, scope `profile`, expression:
|
||
|
||
direct = set(request.user.ak_groups.all())
|
||
return {"groups": sorted(g.name for g in request.user.all_groups() if g not in direct)}
|
||
|
||
and tick it (next to the default one) on the `Vanguard sites`, `Jellyfin` and `Gitea` providers → Advanced
|
||
protocol settings → Scopes. Claims from several mappings merge, so the header becomes direct + inherited groups
|
||
without duplicates. Takes effect on the next login.
|
||
3. compose (already set): registration only via external sources, auto-registration + account linking on,
|
||
username from `preferred_username`. A first Discord login creates the Gitea user with no form.
|
||
4. Access = Authentik group **`git-users`**: Directory → Groups → create it (add a Discord role to it in the
|
||
roles→groups table); Applications → `Gitea` → Policy / Group / User Bindings → bind `git-users` and `git-admins`.
|
||
Non-members are refused by Authentik before Gitea sees them.
|
||
5. Test in a private window: https://git.HOST/user/login → "Sign in with authentik" → Discord → logged in.
|
||
Members of `admins` become Gitea site admins on login. The local admin account stays as a fallback.
|
||
Optional later: hide the password form (custom template) so Discord is the only visible way in.
|
||
|
||
## 5f. Live portal (`live.HOST`)
|
||
|
||
React + dockview page (`live/`) behind Authentik forward-auth: every active stream is a dockable window,
|
||
audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
|
||
MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
|
||
MediaMTX runs the `-ffmpeg` image for the on-ready hook `bin/mtx-on-ready.sh` (bind-mounted into the container):
|
||
the optional "is live" Discord post (`LIVE_WEBHOOK_URL` in `.env`, asked by `configure.py --full`) and the
|
||
**WebRTC audio mirror**. Browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only) and
|
||
OBS sends AAC, so for every stream whose audio is not already Opus the hook runs one ffmpeg that copies the video
|
||
and transcodes the audio to Opus into the sibling path `<user>/<Title>.webrtc` (via MediaMTX's RTSP on the
|
||
container loopback). The portal plays the mirror over WebRTC and hides it as a panel; recordings, HLS and the
|
||
Discord post stay on the original. Mirrors are never recorded and can only be published from inside the container
|
||
(dumpd checks the source IP). Browser Go Live already sends Opus and gets no mirror. Cost: a few % of a core per stream.
|
||
Viewer requirement on the streamer side: **B-frames 0** (section 5), or WebRTC is refused and the portal falls back to HLS.
|
||
|
||
1. CNAME `live` → main host; `./configure.py --full` asks the hostname (also `--hosts … LIVE`).
|
||
2. Authentik: Proxy Provider (forward auth, single application, external host `https://live.HOST`) →
|
||
embedded outpost → group policy (e.g. everyone in the server, or `streamers`+`editors`+`admins`).
|
||
3. Dev box: `./deployLive.sh` builds `live/` and copies `dist/` to `/srv/live` (Caddy serves it directly).
|
||
|
||
**Go Live** button (window audio: Windows + Chrome/Edge 141+; Linux/Mac get video only): broadcast a window from the browser (no OBS) as `<user>/HelmetCam-<date>` — a fresh publish token comes
|
||
from `/me/stream-token` for the logged-in Discord user (member of `streamers`).
|
||
|
||
Bandwidth: no transcoding, every viewer gets the source bitrate (~15 Mbps) → ~30 concurrent viewers on
|
||
the 500 Mbps line. For a crowd, restream to Twitch from OBS.
|
||
|
||
## 5g. Authentik forward-auth for the Caddy-gated hosts (`media.`, `files.`, `clips.`, `sync.`, `live.`, `bot.`)
|
||
|
||
Design: **Authentik authenticates once for the whole domain; Caddy authorizes per host** from the
|
||
`X-Authentik-Groups` header the outpost returns; **Discord roles fill the Authentik groups** at login.
|
||
|
||
1. Applications → Outposts → edit "authentik Embedded Outpost" → config `authentik_host: https://auth.HOST`.
|
||
2. Applications → Create with provider → Application `Vanguard sites`; **Proxy Provider**, authorization flow
|
||
implicit-consent, mode **Forward auth (domain level)**, authentication URL `https://auth.HOST`,
|
||
cookie domain = the parent domain (e.g. `dumpstervigilantes.rip`). Edit the Embedded Outpost →
|
||
Applications → add it → Update. One login now covers every gated host.
|
||
3. Per-host group requirement = `.env`: `BOT_REQUIRE_GROUP=bot-users` (default), `LIVE_REQUIRE_GROUP=`
|
||
(empty = any logged-in member), `SYNC_REQUIRE_GROUP=media-admins` (the Syncthing UI on `sync.`).
|
||
On `files.` copyparty itself authorizes by group (`editors`, `media-admins`); the dump endpoint lets
|
||
`admins`/`media-admins` dump anyone's night. Leave "Intercept header authentication" **off**: login is
|
||
Discord only, no app passwords, no WebDAV. `./configure.py` apply regenerates the Caddy blocks; a user outside the
|
||
group gets a 403 from Caddy before the app sees the request.
|
||
4. Discord roles → groups: Directory → Groups: create `bot-users` (and keep `streamers`, `editors`, `admins`).
|
||
Discord source → Scopes: `guilds guilds.members.read`. Customization → Policies → Expression Policy
|
||
`discord-roles-to-groups` from `authentik/discord-roles-to-groups.py` (fill in the server ID and the
|
||
role-id → group table). Bind it to `default-source-enrollment` **and** `default-source-authentication`
|
||
(engine mode `all`, after `discord-guild-member`). From then on giving someone a Discord role gives them
|
||
the matching access on next login; removing it removes it. Memberships added by hand in Authentik are left
|
||
alone: the policy only revokes what it granted itself (tracked in the user attribute `discord_role_groups`).
|
||
copyparty additionally requires a header with a secret name that only Caddy adds (`idp-h-key`,
|
||
`COPYPARTY_IDP_KEY` in `.env`), so identity headers cannot be forged even if copyparty were reached directly.
|
||
Logout: copyparty's logout link, the live portal's "Sign out" and the dock's "sign out" go through
|
||
`/outpost.goauthentik.io/sign_out`; `https://<any gated host>/logout` does the same from anywhere
|
||
(e.g. from the Syncthing UI, which has no logout of its own). Set the provider's **Invalidation flow** to
|
||
`default-provider-invalidation-flow` so that also ends the Authentik login: every logout then forces a full
|
||
Discord re-login on the next visit (and picks up group changes). Discord itself may approve silently if
|
||
you are still logged into Discord in that browser.
|
||
5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them.
|
||
Group changes in Authentik apply on the next login — log out at `https://auth.HOST/` and back in.
|
||
6. Test in a private window: `https://live.HOST/` → Discord login → portal; `https://bot.HOST/` → 403 unless
|
||
you hold the bot role; `https://live.HOST/me/stream-key` shows your username.
|
||
|
||
Jellyfin and Gitea use OIDC providers instead, not this.
|
||
|
||
## 6. Editors
|
||
|
||
Log in with Discord at https://files.BASE/ (copyparty behind Caddy + Authentik). Access is by Authentik
|
||
group: **editors** read streams and clips and write `/finals`; **media-admins** have full rights everywhere
|
||
including `/trash`. Scrub any `.mp4` in the browser, download what you want.
|
||
|
||
No WebDAV: login is Discord only (no passwords of any kind), and a mounted drive cannot do a browser login.
|
||
Download from the browser: single files, or select files/folders and use copyparty's zip/tar download.
|
||
|
||
> **Available but not set up.** WebDAV mounts can be enabled later without code changes if we decide we want
|
||
> them: in Authentik, edit the `Vanguard sites` proxy provider and tick **Intercept header authentication**;
|
||
> each editor then creates an *app password* under their Authentik user settings and mounts the same URL with
|
||
> their Authentik username + that app password. Caddy and copyparty already pass the identity through, so
|
||
> nothing else changes. It is off because it introduces a password-like credential alongside Discord login.
|
||
|
||
Every `.mp4` an editor sees is the same kind of file (see "Editing standard"). Raw clipper output that
|
||
needed converting is kept in a hidden `.src/` folder next to it (media-admins can show dotfiles in copyparty).
|
||
|
||
## 7. Home archive box
|
||
|
||
Copy `home/rclone-archive.sh` to the home machine, set the vars, run from cron nightly.
|
||
It connects over SFTP as `debian` (all media files are owned by uid 1000 so it can delete them).
|
||
Pulls everything older than 15 days off the server, including the hidden `.src/` raw files.
|
||
|
||
## Publish authentication (no passwords)
|
||
|
||
MediaMTX asks dumpd (`POST /mtx-auth` on 127.0.0.1:8081, reachable only from Docker's bridge) for every
|
||
connection. Reads/playback are allowed (viewers are gated by Caddy + Authentik). Publishing needs a token
|
||
minted by `GET /me/stream-token` behind Authentik for a member of `streamers`/`admins`/`media-admins`; dumpd
|
||
checks the HMAC signature (`/etc/vanguard/stream-token.secret`, generated into `.env` as `STREAM_TOKEN_SECRET`),
|
||
the expiry, and that the path starts with that user's name. MediaMTX does not pick up `mediamtx.yml` changes
|
||
live on this setup: after any change to it, `docker compose restart mediamtx` between streams. Day-to-day
|
||
operations (streamers, tokens, titles) never touch that file. `configure.py` reloads Caddy in place and
|
||
restarts only copyparty.
|
||
|
||
## Timecode for multicam editing (frame accurate)
|
||
|
||
Every published stream carries a QuickTime **timecode track** (`tmcd`, time of day in Eastern, at the file's
|
||
frame rate) and a UTC `creation_time`, written by the sweep when it publishes the file, plus a sidecar
|
||
`<Title>__<ts>.json` with the exact capture time (the Jellyfin plugin reads it). Resolve and Premiere read the
|
||
track natively: drop clips from several streamers on a multicam timeline, sync by timecode.
|
||
|
||
Where the time comes from, best first:
|
||
1. **Overscan band** — a thin strip of extra rows below the picture carrying the *server* clock, encoded
|
||
(black/white cells, format in `bin/vgstrip.py`) plus a small readable clock. The server decodes it
|
||
(`bin/vgstrip.py`) at the start, middle and end of each recording → the timecode is frame-accurate at the
|
||
start and the sidecar records drift over the file. **Then the band is cropped off**: the sweep re-encodes the
|
||
stream (libx264 CRF 18, frame timing preserved, audio copied) so the published file is the game image + audio +
|
||
the `tmcd` track and nothing else; the raw with the band is deleted (`KEEP_RAW_STREAMS=1` keeps it in `.src/`).
|
||
Streams without the band are remuxed losslessly. Cost: every re-encode (streams and clips) runs x264 on
|
||
`ENCODE_THREADS` threads (2, set in `systemd/normalize-sweep.service`) so Jellyfin and live viewers stay
|
||
responsive; a 3 h 1440p60 night therefore takes most of the next day to publish. OBS: dock → *Timecode band* →
|
||
"Add band + clock source" (canvas grows by W/160 rows, a locked browser source `Vanguard Clock` →
|
||
`https://live.BASE/dock/clock/` sits in the band; only while not streaming). Go Live: the "Timecode band" checkbox (default on).
|
||
2. **Server arrival time** (no band) — microseconds in the recording filename. Within ~0.1–0.3 s per streamer
|
||
(encoder + network delay); coarse but always available.
|
||
|
||
Viewing: the live portal crops the band on the live picture (the only place it still exists); recordings in
|
||
Jellyfin, copyparty and the editors' timelines are already at game resolution. Recordings published before
|
||
the crop existed: `sudo vgdeband.sh [FILE…]` re-encodes them the same way and marks the sidecar.
|
||
Cross-check when it matters: everyone is in the same voice chat, so audio-waveform sync is sample-accurate.
|
||
Why continuous and not a start burst: a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h).
|
||
|
||
Dropped frames / gaps: OBS keeps stamping frames from its clock, so after a drop the file has a hole and every
|
||
frame that exists is still at the right time — the timecode stays correct, and players (Jellyfin, the portal)
|
||
hold the last frame across the hole with audio in sync. By default the sweep does **not** re-encode a stream to
|
||
fill holes (hours of CPU on this box); the sidecar lists them (`gaps.count`, `total_missing_ms`, first 50
|
||
positions). Premiere preserves timing on such files; in Resolve check the clip's frame-rate interpretation.
|
||
**Gap filling (available, off by default).** The sweep can re-encode a stream whose holes total at least
|
||
`FILL_GAPS_MS` milliseconds to constant frame rate with duplicated frames (raw kept in `.src/`, timecode carried
|
||
over). It costs ~2× the stream length in CPU on the E-2136, so it is off until the box has a hardware encoder.
|
||
Turn it on (threshold 1 s) / off, on the server:
|
||
|
||
sudo sed -i 's|^#* *Environment=FILL_GAPS_MS=.*|Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
|
||
sudo sed -i 's|^Environment=FILL_GAPS_MS=.*|# Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
|
||
|
||
Check: `systemctl show normalize-sweep -p Environment`. Takes effect on the next sweep run; already-published
|
||
files are not revisited. Clips are always converted to CFR regardless.
|
||
Planned alternative: re-encode flagged streams off-site on a machine with a hardware encoder (home box NVENC or a
|
||
Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
|
||
|
||
## Deployment (push to deploy — built, OFF by default)
|
||
|
||
The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea →
|
||
`./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token
|
||
secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout.
|
||
|
||
`sudo vanguard-deploy pull` fetches `main` and applies **only what changed**, after `check.sh`:
|
||
|
||
| changed | action | interruption |
|
||
|---|---|---|
|
||
| `configure.py`, `dock/`, `clips/`, `about/`, stignore | `configure.py --apply` → Caddy reload, copyparty restart | none / a second |
|
||
| `bin/*`, `systemd/*` | install; dumpd restart when it changed; timers pick up next run | a second |
|
||
| `firewall/nftables.conf` | `nft -f` | none |
|
||
| `docker-compose.yml`, `bot/Dockerfile`, `bot/entrypoint.sh` | `compose up -d --build` for changed services | that service |
|
||
| `deployBotBundle.sh` | reinstalled as `/usr/local/bin/vanguard-deploy-bot` (what dumpd runs for the bot's CI) | none |
|
||
| anything that changes `mediamtx.yml` or the mediamtx service | **restart only when no stream is live**; otherwise deferred, `deploy-pending.timer` retries every 5 min | none while live |
|
||
| `jellyfin-plugin-vanguard/dist/*.zip` (committed) | `installJellyfinPlugin.sh` | Jellyfin restart |
|
||
| `live/` | `npm run build` on the server → `/srv/live` | none |
|
||
|
||
`sudo vanguard-deploy rollback` checks out the previously deployed revision and applies the reverse diff.
|
||
`sudo vanguard-deploy status` shows revisions, auto mode, pending restarts, live streams.
|
||
|
||
**One-time rollout** (the last scp-style deploy; afterwards the server pulls). From the dev box:
|
||
|
||
cd ~/claudevs/vanguard-stream && git push
|
||
scp bin/dumpd.py debian@HOST:~/vanguard-stream/bin/
|
||
scp systemd/deploy-pending.service systemd/deploy-pending.timer debian@HOST:~/vanguard-stream/systemd/
|
||
scp docker-compose.yml configure.py deploy.sh switchToClone.sh check.sh install.sh README.md STATUS.md CommunityDiscordMedia.md debian@HOST:~/vanguard-stream/
|
||
ssh -t debian@HOST 'cd ~/vanguard-stream && sudo apt-get install -y -q git nodejs npm rsync python3-yaml >/dev/null && sudo install -m 0755 bin/dumpd.py /usr/local/bin/ && ./configure.py --apply && sudo systemctl restart dumpd && sudo vanguard-deploy status'
|
||
|
||
The apply prints the post-start fixups (Syncthing, Gitea admin user, Jellyfin plugin) and ends with a status
|
||
block: `DEPLOY_AUTO=0`, no pending restart, and "not a git checkout" until the next step.
|
||
|
||
Then switch the server folder to a clone (two steps):
|
||
|
||
ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh key' # prints a public key
|
||
# Gitea → repo → Settings → Deploy Keys → add it, read-only
|
||
ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh clone && cd ~/vanguard-stream && ./configure.py --apply >/dev/null && sudo vanguard-deploy status'
|
||
|
||
`switchToClone.sh clone` keeps `.env`, the generated configs, the token secret and plugin zips, and
|
||
leaves the old copy at `~/vanguard-stream.old` until you delete it. From here on a manual deploy is
|
||
`sudo vanguard-deploy pull` on the server.
|
||
|
||
**Automatic**: Gitea → repo → Settings → Webhooks → Gitea, URL `http://<docker gateway>:8081/deploy`
|
||
(`docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}'`, usually `172.18.0.1`),
|
||
secret = `DEPLOY_WEBHOOK_SECRET` from `.env`, push events, branch `main`. dumpd verifies the HMAC and runs
|
||
`vanguard-deploy pull` **only when `DEPLOY_AUTO=1`** in `.env` (then `./configure.py --apply` to push the setting
|
||
to `/etc/vanguard/deploy.env`). Until then every push is logged (`journalctl -u dumpd | grep deploy`) and ignored; Gitea's "test delivery"
|
||
should get a 202 "auto-deploy disabled" back. Optional `DEPLOY_NOTIFY_WEBHOOK_URL` posts deploy results to Discord.
|
||
Turning it on: `DEPLOY_AUTO=1` in `.env`, `./configure.py --apply`, and
|
||
`git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml` for CI.
|
||
|
||
CI: `.gitea/draft-workflows/ci.yml` (inert until moved to `.gitea/workflows/`) runs `check.sh` and the plugin
|
||
tests/build on the Actions runner. The runner never deploys — it must not touch the host.
|
||
|
||
## Rebuild from scratch (what is and isn't automated)
|
||
|
||
Server side, scripted:
|
||
|
||
git clone ssh://git@git.HOST:2222/techy/Media-Server-Setup.git ~/vanguard-stream # or scp the folder
|
||
cd ~/vanguard-stream && sudo bash install.sh # packages, docker, folders, host scripts, timers, firewall, portal build
|
||
./configure.py --full # hostnames, secrets, configs, containers, post-start fixups
|
||
|
||
Restore first if rebuilding a known box: `.env` (all secrets — without it every password is regenerated and
|
||
the databases won't match), `/srv/appdata` (Authentik DB, Jellyfin, Gitea, MariaDB, Syncthing config),
|
||
`/srv/media` (the footage), `/etc/vanguard/clip-quotas.json` if used. With those restored, nothing below is
|
||
needed except DNS.
|
||
|
||
Post-start fixups `configure.py` performs on every apply (idempotent): Syncthing GUI auth off, Gitea admin
|
||
user created from `.env` (`GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD`; the install page is locked), Jellyfin plugin
|
||
installed from the committed zip. Authentik's `akadmin` password is set on first start from
|
||
`AUTHENTIK_BOOTSTRAP_PASSWORD` in `.env` (no initial-setup page).
|
||
|
||
Still manual on a truly fresh box (not in the repo; documented in §5c–5g):
|
||
- Authentik: Discord source, identification-stage source, enrollment user type, groups, the two policies,
|
||
the domain-level proxy provider + outpost host, the Jellyfin and Gitea OIDC providers, application bindings
|
||
- Jellyfin: first-run wizard, libraries (Shows + Vanguard provider), SSO plugin + provider, branding
|
||
- Gitea: authentication source (authentik), deploy key, webhook; `./registerGiteaRunner.sh`
|
||
- Discord bot: its repo lives outside this one, so nothing here restores it. Put `BOT_DEPLOY_SECRET`
|
||
(`./configure.py --show`), `DISCORD_BOT_TOKEN` and `DISCORD_GUILD_ID` into that repo's Gitea Actions
|
||
secrets/variables, then push to its `main` — or install a bundle by hand with `./deployBotBundle.sh`.
|
||
`/srv/bot/app` is created empty by `install.sh`; the container idles until a release lands there.
|
||
- Discord developer portal: application, redirect URIs, bot; DNS records
|
||
A `.env`-driven Authentik blueprint would remove the first line (planned).
|
||
|
||
## Layout on the server
|
||
|
||
/srv/media/streams/<user>/<Title>/.live/<ts>.mp4 recording in progress (hidden from Jellyfin/copyparty)
|
||
/srv/media/streams/<user>/<Title>__<ts>.json capture time (UTC ms), timecode, source (strip|arrival), drift
|
||
/srv/media/streams/<user>/<Title>__<ts>.mp4 finished stream, one flat MP4 per recording, directly in
|
||
the streamer's folder (several files inside one title
|
||
folder would be ONE Jellyfin episode with "versions")
|
||
/srv/media/clips/<user>/clip.mp4 clip (synced as-is, or converted; raw removed)
|
||
/srv/media/clips/<user>/.stlocal Syncthing ignore list maintained by the sweep
|
||
/srv/media/clips/<user>/.src/clip.mkv raw source, only for clips that did NOT arrive via Syncthing
|
||
/srv/media/clips/<user>/clip.normalize-failed.log only if normalization failed; delete to retry
|
||
/srv/bot/app/ the Discord bot's installed release (bind-mounted at /app)
|
||
/srv/bot/previous/ the release before it (deployBotBundle.sh --rollback)
|
||
/srv/media/finals/ editors upload here
|
||
/srv/media/trash/<night>/... dumped, purged after 3 days
|
||
/srv/media/.markers/ dump markers
|
||
/srv/media/.syncthing/ Syncthing config
|
||
|
||
## Editing standard (what normalize-sweep enforces)
|
||
|
||
MP4 · H.264 High, 8-bit 4:2:0 · constant frame rate · AAC audio (all tracks kept) · index at the
|
||
front (instant seek in browser and NLE) · at most 2560x1440 and 60 fps (downscaled to fit, never
|
||
upscaled; aspect kept).
|
||
|
||
| input | action |
|
||
|-------|--------|
|
||
| already compliant | nothing |
|
||
| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: decode the timecode band (or use arrival time), then publish to `<user>/<Title>__<ts>.mp4` with a `tmcd` track + `creation_time` — band present: re-encode with the band cropped off; no band: lossless remux — write the `.json` sidecar, remove `.live` and the empty title folder |
|
||
| in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
|
||
| HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
|
||
| ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |
|
||
|
||
Runs every 5 min on files idle ≥3 min and not open. Software x264 on the E-2136 does ~0.5x realtime
|
||
at 1440p60, so a 2 h clip batch takes ~4 h; streams only remux (minutes). All at idle priority.
|
||
|
||
## Reboot behaviour
|
||
|
||
Everything comes back on its own, in this order:
|
||
|
||
1. mdadm assembles the mirror, systemd mounts /srv (local-fs). A degraded mirror still boots.
|
||
2. docker.service has `RequiresMountsFor=/srv/media` (drop-in installed by install.sh), so it never
|
||
starts with the media tree missing. All containers are `restart: unless-stopped`.
|
||
3. dumpd.service starts; normalize-sweep.timer fires 2 min after boot and every 5 min after;
|
||
trash-purge.timer is `Persistent=true` so a missed 05:00 run fires at boot.
|
||
4. A stream cut by a reboot is playable up to the last second (fMP4 parts are flushed every 1 s);
|
||
OBS reconnects and MediaMTX opens a new file in the same title folder.
|
||
5. Half-written `*.part` files are overwritten by the next sweep. Dumps are idempotent.
|
||
|
||
Time: install.sh sets the server clock to America/New_York so journal, timers, file timestamps
|
||
and the dump window all read in Eastern. Dump "night" = 17:00 → 02:00 Eastern (pinned in dumpd via
|
||
DUMP_TZ, independent of the host zone). Between 02:00 and 17:00 the default night is the previous
|
||
evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
|
||
|
||
## Timers
|
||
|
||
| unit | when | does |
|
||
|------|------|------|
|
||
| normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
|
||
| trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
|
||
| dumpd.service | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time and on `clips.` at /me/clips |
|