techy/Media-Server-Setup
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ca01516d42bffc554769696bd2d13c0b67515492
Media-Server-Setup/README.md
Brandyn / Techy ca01516d42 Stuck-recording triage, a diag landing page, and the fixes behind Tuesday's symptoms 
Triage
- bin/vgstuck.sh classifies every unpublished recording (parked by a failure log, published
  collision, waiting, still writing), with ffmpeg's own reason for parked ones. retry clears the
  blocker; drop moves to trash/<today>/ with the same 3-day undo as a dumped night; drop --all is
  refused. A leftover .mp4.part blocks nothing (ffmpeg overwrites with -y) and is only cleaned up.
- The same tool backs a page at diag.HOST/stuck. dumpd serves /stuck/api, lets anyone past the gate
  read, and allows retry/bin only for the admin groups; the path validation is the single thing
  between a request body and a file move, so it accepts only an existing .mp4 under .live/ inside
  the streams tree.
- The sweep logs event=parked once per run so Kibana can name stuck files rather than only count
  them.

Landing
- diag.HOST opens on a "Vanguard ops" dashboard via Kibana's defaultRoute: a markdown panel of links
  to every host and the triage page, plus Discover, Dashboards, Stack Rules and Index Management.
  Kibana stays at the root; moving it under /kibana would have meant the prefix in both Beats'
  setup.kibana, two scripts and the deadman, with Metricbeat crash-looping on whichever was missed.
- /api/kibana/settings is an internal API behind server.restrictInternalApis, which Kibana 9 turns
  on by default, so the script sends x-elastic-internal-origin. kibana-setup.py takes a step name and
  verifies each step by reading it back.

Installers
- bin/install.list is the one list of what lands in /usr/local/bin, read by install.sh,
  configure.py and deploy.sh. deploy.sh's own copy had drifted to five of ten files. check.sh fails
  on drift. deploy.sh also recreates a Beat or Kibana when its bind-mounted config changes, which
  nothing did before.
- check.sh distinguishes absent from broken: the server checkout is partial, and a yaml file or node
  binary that is not there is a skip, not a failure.

Fixes
- Go Live pixelation: x-google-min-bitrate=6000 forbade Chrome from encoding below 6 Mbps, so on a
  weak uplink the excess was dropped and MediaMTX logged invalid FU-A packets. Loss was on the
  publish sessions only. Floor is 1200 and scaleResolutionDownBy is unpinned.
- HLS manifestParsingError: handle_path strips /hls, so MediaMTX's cookieCheck redirect came back
  root-relative and landed on the SPA catch-all. header_down rewrites Location.
- Sweep audio: the band-removal branch overwrote $reason and lost the probe's audio=opus marker, so a
  Go Live recording with the band shipped Opus inside an MP4. probe_reason is kept separately.
- Deadman false alarm: alerts.state was only written when something changed, so quiet runs let it go
  stale. Every run stamps last_run now, and the deadman also checks is-failed on the unit.
- Both timer scripts load /etc/vanguard/alerts.env themselves, so a run by hand matches the timer.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-11-04 21:25:43 -06:00

748 lines
55 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# vanguard-stream — SYS-1 (Xeon E-2136, 32 GB ECC, 2×4 TB SATA soft RAID)
> **FRESH-INSTALL.md** = the ordered walkthrough from an empty box to a working stack and pipeline, manual steps included.
> **HOSTNAME-CHANGE.md** = every place a hostname lives and how to rename a subdomain or move to a new base domain safely.
> **LOGGING.md** = observability: what each service logs, the container labels, the logfmt event format, and the Elastic stack that consumes it (§5h).
> **UPCOMING.md** = what is still missing for a mil-sim community (gap list, techy's take, agreed direction: metadata DB, markers, watch app).
**Hostnames** (all CNAMEs to the main host; one Discord login covers all of them):
> Adding a host later needs `./configure.py --full` or `--hosts`, **not** `--apply`: apply reads the hostnames
> back out of the Caddyfile, so a role that is not in there yet is silently left out and the new name ends up
> with no certificate. HOSTNAME-CHANGE.md §D has the procedure.
| Host | What | Who |
|---|---|---|
| `media.BASE` (main) | gateway page at `/` (role-aware links + how-tos), `/whoami`, `/logout` | everyone |
| `files.BASE` | copyparty: browse, scrub, download, upload finals | editors / media-admins |
| `clips.BASE` | self-service clip intake (pair your Syncthing) | any member |
| `sync.BASE` | Syncthing admin UI | media-admins |
| `live.BASE` | live portal, Go Live, OBS dock `/dock/`, clock source `/dock/clock/`, `/dump`, `/me`, `/time` | everyone (`LIVE_REQUIRE_GROUP`) |
| `diag.BASE` | logs, metrics, alerts (§5h) | media-admins |
| `watch.BASE` | Jellyfin | Discord via OIDC |
| `git.BASE`, `bot.BASE`, `auth.BASE` | Gitea, bot web UI (`bot-users`), Authentik | — |
`/time` (the server clock the timecode strip syncs to) lives only on `live` because browsers fetch it
same-origin: both pages that use it, the clock source and Go Live, are on that host.
> **STATUS.md** = what runs, who does what, what's unfinished (kept current). **CommunityDiscordMedia.md** = design + phases.
Ingest from OBS, browse/scrub/download for editors, automatic normalization to one editing
standard, dump button, 15-day archive pull to a home HDD. Everything is plain files under /srv/media.
## 1. Reinstall from the OVH manager
Server → Reinstall → Debian 13 (Trixie).
Tick **custom partitioning** and **use distribution kernel**. Layout (RAID 1 across both HDDs):
| mount | size | fs |
|-------|------|----|
| /boot | 1 GB | ext4 |
| / | 250 GB | ext4 |
| swap | 8 GB per disk | swap |
| /srv | rest (~3.4 TiB) | xfs |
Add your SSH key. OVH's template creates a `debian` user with sudo; root SSH is disabled. Wait for the mail.
## 2. Install
scp -r vanguard-stream debian@SERVER:~/
ssh debian@SERVER
cd ~/vanguard-stream && sudo bash install.sh
install.sh installs docker, ffmpeg, rclone; creates /srv/media/{streams,clips,finals,trash,.markers}
owned by uid 1000 (the `debian` user); copies the systemd units; adds `debian` to the docker group;
starts the compose stack. It prints what you still have to edit.
The folder stays where it is: compose bind-mounts the configs and the dock page from it.
After install, log out and back in once so the docker group applies, then plain `docker compose`
works from ~/vanguard-stream without sudo.
## 3. Configure (domain, admin, streamers, editors)
> `mediamtx.yml`, `copyparty.conf`, `Caddyfile`, `dump-users.txt` and `.env` are **generated on the server**
> by `configure.py` and gitignored. The `*.example.*` files in the repo are reference only. **Never copy them
> to the server**: doing so once replaced the live streamer passwords with `CHANGEME` and the next apply
> wiped the streamers. To update the server, copy scripts (`configure.py`, `bin/`, `docker-compose.yml`, …)
> and re-run `./configure.py`.
cd ~/vanguard-stream
./configure.py --full # first time: asks everything, then applies
./configure.py # later: add/remove/reset streamers and editors, then apply
./configure.py --show # print current users and the OBS settings to hand out
./configure.py --set K=V # change one setting (Discord tokens/webhooks, retention, alarm mentions)
# run it with no arguments to list what is settable
Stateless: it reads the current values out of `Caddyfile`, `mediamtx.yml` and `.env`, regenerates
`Caddyfile`, `mediamtx.yml`, `copyparty.conf` and `streamers.json` (→ `/etc/vanguard`), creates
`streams/<name>` and `clips/<name>` (with `.stignore`) for each streamer, restarts dumpd and the
containers. Editors and admins are Authentik groups (`editors`, `media-admins`, `admins`), not local
accounts. Removing a streamer revokes publishing only; their files stay on disk.
Passwords may not contain `: & ? # / \ quotes or spaces` (they ride inside the stream key).
## 4. Firewall (host nftables; OVH edge firewall is optional on top)
Applied automatically at the end of `install.sh`. To re-apply by hand:
cd ~/vanguard-stream && sudo bash firewall/firewall.sh
Installs `firewall/nftables.conf` as `/etc/nftables.conf`, syntax-checks, applies, enables at boot.
It uses its own table (`inet vanguard`) so applying or reloading never disturbs Docker's rules,
and filters input only so container networking keeps working. **Keep your SSH session open and
test a new one before closing it.**
| port | proto | who |
|------|-------|-----|
| 22 | tcp | anywhere, rate limited (edit `ssh_allow` in /etc/nftables.conf to restrict to your IPs) |
| 80, 443 | tcp | world (Caddy) |
| 443 | udp | world (Caddy HTTP/3) |
| 1935 | tcp | world (RTMP ingest) |
| 8890 | udp | world (SRT ingest) |
| 22000 | tcp+udp | world (Syncthing) |
| 3306 | tcp | world, rate limited (MariaDB for the bot dev; restrict via `db_allow` in /etc/nftables.conf) |
| 2222 | tcp | world (Gitea ssh) |
| 8189 | tcp+udp | world (WebRTC media for the live portal) |
Everything else is dropped and a trickle is logged (`journalctl -k | grep vg-drop`). Authentik (9000),
Jellyfin (8096), copyparty, the Syncthing UI and dumpd bind to 127.0.0.1 only and are reached via Caddy
or an ssh tunnel, so they are deliberately not opened. MariaDB runs on the host network (not a Docker
published port) precisely so the firewall can govern it.
Syncthing's admin UI is proxied at `https://sync.BASE/` behind the Authentik gate + group `media-admins`; its
own GUI login is disabled (no user/password, `insecureAdminAccess=true`) so Discord is the only login:
docker exec syncthing syncthing cli --home /var/syncthing/config config gui insecure-admin-access set true
docker exec syncthing syncthing cli --home /var/syncthing/config config gui user set ""
docker exec syncthing syncthing cli --home /var/syncthing/config config gui password set ""
The ssh tunnel to 8384 still works as an admin fallback (no login at all — ssh is the gate there).
To change rules: edit `/etc/nftables.conf`, then `sudo nft -f /etc/nftables.conf` or
`sudo systemctl reload nftables`. **Never `systemctl stop|restart nftables`** — Debian's unit flushes
the whole ruleset on stop, which also wipes Docker's NAT rules and takes the containers offline
until Docker is restarted.
## 5. Streamer setup (once)
Being a streamer = being in the Authentik **streamers** group (a Discord role via the roles→groups policy).
There are no stream passwords or keys to hand out: the dock mints a **12-hour publish token** for the
logged-in Discord user each time Go Live is pressed and sets it as the OBS stream key itself.
OBS → Settings → Output → Encoder: **B-frames = 0** (NVENC "Max B-frames: 0"; x264 add `bframes=0`). MediaMTX
refuses WebRTC playback of H.264 with B-frames ("WebRTC doesn't support H264 streams with B-frames"), so the live
portal would fall back to HLS with seconds of delay. Keyframe interval 2 s. Recording quality is unaffected.
OBS → Tools → WebSocket Server Settings → enable, set a password.
OBS → Docks → Custom Browser Docks → `https://live.BASE/dock/` → log in with Discord once.
In the dock's Settings: ingest host (prefilled), OBS WebSocket password/port, optional title prefix/suffix
(`{date}` → YYYY-MM-DD, e.g. prefix `HelmetCam-` or suffix `-{date}`).
Per stream: type a title, click **Go Live**. The dock shows the final stream name (`<you>/<Title>`) as you type
and enforces the naming rules the metadata relies on: letters, digits, `_ . -` only, no double underscore
(it separates title and timestamp in filenames), no leading dot, max 64 chars.
After a bad night: **Dump tonight** (dumps the night of the account you are logged in as; `admins`/`media-admins`
may dump anyone's).
Manual OBS/other software: dock → **Manual OBS setup** shows the RTMP server, the stream key and the SRT URL
with the token hidden (Show / Copy / Fresh token). Tokens expire after 12 h; a running stream is not cut when
its token expires, but a reconnect needs a fresh one.
Browser streaming without OBS: the live portal's **Go Live** (Windows Chrome 141+ for window audio + GPU encode).
## 5a. Onboarding a gamer's clips (Syncthing) — self-service
Gamers do it themselves at **https://clips.BASE/** (Discord login): the page shows our Syncthing Device ID and
asks for theirs; on submit, dumpd creates `clips/<username>` with the video-only `.stignore`, adds their device
to our Syncthing (REST API, key read from `/srv/media/.syncthing/config.xml`) and creates the receive-only
folder `clips-<username>` shared with them (min free 300 GB). They accept the device + folder on their PC,
point it at their clip directory, Send Only. The page shows pairing/connection/sync state. Idempotent.
**Per-user quota (soft)**: dumpd measures every `clips/<user>` every 5 min and pauses that user's Syncthing
folder when it exceeds the quota (default `CLIPS_QUOTA_GB=100` in `dumpd.service`), resuming automatically
under 90 %. The clips page shows usage and the paused state. Overrides per user in `/etc/vanguard/clip-quotas.json`,
e.g. `{"techy2493": 250, "newguy": 50}` (GB) — read live, no restart. A gamer frees space by deleting clips in
their own folder (Syncthing mirrors the deletion). Hard byte-exact quotas would be XFS project quotas on `/srv`
(needs a remount with `prjquota`); not set up.
Manual fallback (admin, same result for the server folder):
./addClipUser.sh <username> # creates /srv/media/clips/<username> with the video-only .stignore
Their side (Windows: Syncthing installer or SyncTrayzor; Mac/Linux: Syncthing): open the UI, **Actions → Show ID**,
send you the Device ID. In their UI: **Add Remote Device** with the server's ID (ours: `sync.BASE` → Actions → Show ID),
then **Add Folder** on their clip directory (ShadowPlay/Medal/Steam saves), Sharing tab → tick the server,
Folder Type **Send Only**, Save.
Our side at https://sync.BASE/: **Add Remote Device** → paste their ID, name it after them, Save. When their folder
offer pops up: **Add** → Folder Path `/var/syncthing/clips/<username>`, Folder Type **Receive Only**, Advanced →
Minimum Free Disk Space **300 GB**, Save. Clips start arriving; the sweep normalizes them; Jellyfin shows the
folder as that person's show. Their own folder structure is mirrored; only video files are pulled.
## 5a². Gateway page — https://media.BASE/ (the main host root; `/about/` redirects here)
Role-aware landing page (Discord login): everyone sees Watch / Live / Send clips / Sign out with short how-tos;
`streamers` get the dock + OBS setup steps; `editors` the browser + multicam/timecode notes; `media-admins`
trash/Syncthing/whoami; `git-users` Gitea; `bot-users` the bot; `admins`/`authentik Admins` the admin consoles and
where things live. Identity comes from `GET /me` (dumpd). Point people here first.
## 5b. Syncthing hardening (do once, in the server's Syncthing UI)
Syncthing receives straight into `clips/<user>` (Syncthing UI path `/var/syncthing/clips/<user>`),
receive-only, with a `.stignore` (from `syncthing/stignore`) that pulls video files only. The gamer's
folder structure is mirrored as-is. **One copy per clip on the server**: compliant files are left
alone; a non-compliant file (HEVC/MKV/VFR) is converted to `name.mp4` (`name.h264.mp4` if the raw was
already `.mp4`), the raw's name is appended to `.stlocal` (an `#include`d ignore list, so Syncthing
drops it without flagging or re-pulling), and the raw is deleted locally — it still exists on the
gamer's PC. `configure.py` creates the folder + ignore file for each streamer.
In the UI (https://sync.BASE/):
- each clips folder → path `/var/syncthing/clips/<user>`, **Folder Type: Receive Only**,
Edit → Advanced → **Minimum Free Disk Space: 300 GB** (so clips can never fill the mirror and stall
the recordings).
- Actions → Settings → Connections → **Incoming Rate Limit: 12500 KiB/s** (~100 Mbps) so a gamer
sharing a year of old clips cannot starve the ingest during stream hours.
## 5c. Authentik (Discord login) and Jellyfin
DNS: `auth`, `watch` (and `files`, `clips`, `sync`, `live`, `git`, `bot`) as CNAMEs to the main host. Then run `./addCommunityMediaServer.sh` — it asks
for every hostname, checks DNS, generates `.env` (Authentik secret + DB password; never commit it),
adds the Caddy site blocks, starts the containers, waits for them and prints the steps below with your
hostnames filled in. (`./configure.py --full` asks for the same hostnames if you prefer.) Then:
**Authentik first run** — https://auth.HOST/if/flow/initial-setup/ → set the `akadmin` password.
1. Discord developer portal → New Application → OAuth2: add redirect
`https://auth.HOST/source/oauth/callback/discord/` (last segment = the source slug below), copy Client ID + Secret.
2. Authentik → Directory → Federation & Social login → Create → Discord OAuth Source: slug `discord`,
consumer key/secret = Client ID/Secret, user matching "link on unique identifier", scopes empty.
3. Flows & Stages → Stages → `default-authentication-identification` → Sources: select Discord,
tick "Show sources' labels". Without this the Discord button never appears on the login page.
4. Flows & Stages → Stages → `default-source-enrollment-write` → User type: Internal. Otherwise
Discord users are External and get "Interface can only be accessed by internal users".
5. Directory → Groups: create `streamers`, `editors`, `admins`. Add people after their first login.
6. (optional) Restrict login to members of your Discord server: add scope `guilds` to the source;
Customization → Policies → Create Expression Policy from `authentik/discord-guild-policy.py` (fill in
your server ID); bind it to both `default-source-enrollment` and `default-source-authentication`
(flow → Policy / Group / User Bindings) and set each flow's Policy engine mode to `all`. Server ID: Discord Developer Mode → right-click server → Copy ID.
**Jellyfin first run** — https://watch.HOST/ → wizard: admin user, add libraries: `/media/streams` and
`/media/clips` as **Shows** with only "Vanguard Metadata" as metadata downloader (Series/Season/Episode)
and Screen Grabber + Vanguard Metadata as image fetchers; `/media/finals` as Home Videos with fetchers off.
The Vanguard Metadata plugin (`jellyfin-plugin-vanguard/`, built with its `package.sh`) is installed by
`addCommunityMediaServer.sh` / `installJellyfinPlugin.sh` from `jellyfin-plugin-vanguard/dist/`.
- Dashboard → Playback → Transcoding: **disable** hardware acceleration. Per-user "allow transcoding" is
switched off automatically for everyone (SSO users included) by the Vanguard plugin's "enforce playback
policy" task; turn that toggle off in the plugin config once the server has Quick Sync (SYS-3).
- Dashboard → Plugins → Repositories → add `https://raw.githubusercontent.com/9p4/jellyfin-plugin-sso/manifest-release/manifest.json`, then Catalog → **SSO Authentication** → install → restart Jellyfin.
- Authentik → Applications → Create with provider → OAuth2/OpenID: redirect
`https://watch.HOST/sso/OID/redirect/authentik`, note client ID/secret.
- Jellyfin → Plugins → SSO-Auth → add OID provider "authentik": endpoint `https://auth.HOST/application/o/<slug>/`,
client ID/secret, enable, roles claim `groups`, admin roles `admins`, enabled folders = the three libraries,
Scheme Override `https`, Port Override empty (a value appends `:443` and Authentik rejects the redirect). Save.
- The plugin adds no button. Test with `https://watch.HOST/sso/OID/start/authentik`, then Dashboard →
General → Branding: Login disclaimer = `jellyfin/branding-disclaimer.html` (set the hostname),
Custom CSS = `jellyfin/branding.css`. The page then shows only "Sign in with Discord"; a small
"Admin login" link reveals the local username/password form.
Jellyfin creates the account on first SSO login.
Phase 4 is the Vanguard Metadata Jellyfin plugin; Phase 5 puts copyparty, /sync and /dump behind Authentik forward-auth.
## 5d. MariaDB + Discord bot (for the bot dev)
`mariadb` (public on 3306, host network, root local-only, 32-char generated passwords) with a database/user
for the bot, and `discord-bot`, a Python 3.12 container that runs `/app/main.py` and idles until a release is
installed. Credentials live in `.env` (generated by `configure.py`; bot token + guild ID are asked in `--full`
or the `[d]` menu item; `./configure.py --show` prints them).
**Releases are self-contained bundles, and a push to `main` deploys itself.** The bot repo's Gitea Actions
workflow runs its tests and then builds one tar.gz holding `main.py`, `app/`, `website/`, `BUILD.json` **and
`vendor/`** — its own dependencies, as prebuilt wheels for this image's Python. `PYTHONPATH=/app/vendor` is how
the code finds them, so **a new dependency never needs an image rebuild** and the server never contacts a
package index.
The workflow's last job posts that bundle to **dumpd** at `POST /deploy-bot` on 8081 — the Docker-bridge-only
port the firewall already restricts, so a CI job never needs the Docker socket or an ssh key. dumpd checks an
HMAC-SHA256 signature over the body, then runs `/usr/local/bin/vanguard-deploy-bot` (this repo's
`deployBotBundle.sh`), which verifies every file against `BUILD.json`, swaps the bundle into `/srv/bot/app`
(bind-mounted at `/app`, previous release kept beside it) and restarts the container. The install log goes back
in the HTTP response, so it appears in the CI job.
**The bot's own secrets live in Gitea, not here.** The deploy job also sends the bot's settings, signed with
the bundle, and dumpd writes them into this stack's `.env` — but only the keys `BOT_ENV_KEYS` lists. Anything
else comes back to the CI job as `settings_refused` and is not written, so the database password and the
Authentik keys this repo generates cannot be overwritten from CI. Rotating the Discord token is therefore a
Gitea edit plus a re-run, with no server access.
In Gitea, on the bot repo under Settings → Actions:
| | |
|---|---|
| Secret `BOT_DEPLOY_SECRET` | must equal the server's; `./configure.py --show` prints it |
| Secret `DISCORD_BOT_TOKEN` | the bot token |
| Variable `DISCORD_GUILD_ID` | the server id |
| Variable `VANGUARD_DEPLOY_HOST` | only if the job cannot detect the Docker gateway |
On the server, generated into `.env` and copied to `/etc/vanguard/deploy.env` by `configure.py`:
| | |
|---|---|
| `BOT_DEPLOY_SECRET` | 64 hex chars, the shared signing key |
| `BOT_DEPLOY_AUTO` | `1` (default) accepts CI installs; `0` makes dumpd refuse them with 403 |
| `BOT_ENV_KEYS` | the only `.env` keys CI may set, `DISCORD_BOT_TOKEN,DISCORD_GUILD_ID` by default. Extend it when the bot needs a new setting. |
By hand, any time:
bin/vgbot-merge-sqlite.py --source /data/<old>.db # merge an old SQLite database into the live MariaDB (report; --apply writes)
./deployBotBundle.sh vanguard-bot-bundle.zip # install a release (the artifact as Gitea hands it over, zip or tar.gz)
./deployBotBundle.sh --latest # fetch the newest main build first (needs BOT_REPO_TOKEN in .env)
./deployBotBundle.sh --status # version, commit, package list, container state
./deployBotBundle.sh --rollback # back to the previous bundle
docker compose logs -f discord-bot # watch it
**Merging an old SQLite database in** (`bin/vgbot-merge-sqlite.py`): the bot rebuilds part of its state from
Discord on every start, so after a backend switch some tables hold fresh rows while the old file holds months of
history, and neither side is a superset. The bot's own importer refuses a non-empty destination by design, so this
merges instead: log tables append on a natural key, and the handful of profile/state tables are resolved column by
column with an explicit rule each (earliest `joined_at`, `first_seen_at` and rank `started_at` win, so a rebuild
that stamped today's date is undone; latest activity and current rank/presence win). It runs in one transaction,
is idempotent, and refuses rather than guessing if a column has no rule or a table is missing from the
destination. Run it with the gateway stopped, report first, then `--apply`.
**Identity, exactly:** the bot gets the caller's Discord snowflake from the `discord_id` claim the proxy puts in
`X-Authentik-Jwt`, emitted by the scope mapping in `authentik/vanguard-profile-scope-mapping.py` (the same mapping
that carries inherited groups). Authentik already stores that id on the account's source connection, so nothing
has to be matched by name. Without the claim the bot falls back to matching the Discord **username** only, never a
server nickname: nicknames are self-chosen and not unique, so matching one would let a member be mistaken for
another or make another ambiguous. `discord-bot` also runs on **its own Docker network** (`botnet`), because it
trusts the headers Caddy injects and nothing else should be able to reach `:8000` to forge them.
The bot serves **both** its API and its web pages on 127.0.0.1:8000, published as `https://bot.HOST/` behind
**Authentik forward-auth**: Caddy asks Authentik, then injects `X-Authentik-Username/-Groups/-Email`, and the
bot authorises per request from those headers (`WEB_ALLOWED_GROUPS=bot-users,admins`; `WEB_SERVE_UI=true`
turns the pages on). Create the matching Proxy Provider in Authentik (forward auth, single application,
external host `https://bot.HOST`), add it to the embedded outpost, and bind a group policy. Until then the
site returns an error, never an open page. Caddy's own gate is `BOT_REQUIRE_GROUP` (`bot-users` by default),
so **admins need the `bot-users` role too** — or set `BOT_REQUIRE_GROUP=` and let the bot's group list decide.
The build brief for whoever writes the bot is `bot/AGENT.md`; `bot/README.md` is the ops cheat-sheet.
## 5e. Gitea + Actions runner
`gitea` (MariaDB-backed, `git.HOST` via Caddy, git-over-ssh on **2222**, registration closed) and
`act_runner` (jobs run as Docker containers; labels `ubuntu-latest`, `ubuntu-22.04`, `dotnet`, `python`).
The gitea db/user are created by `mariadb/init/10-gitea.sh` on MariaDB's first start; for a MariaDB that
already existed run `./createGiteaDb.sh` once.
1. `./configure.py --full` asks the Gitea hostname (CNAME `git`); apply starts both containers.
2. https://git.HOST/ → install page (db pre-filled) → create the admin account → Install.
3. `./registerGiteaRunner.sh` → generates a registration token, stores it in `.env`, starts the runner.
4. Remotes: `ssh://git@git.HOST:2222/<org>/<repo>.git` or `https://git.HOST/<org>/<repo>.git`.
Runner caveat: it holds the Docker socket, so anyone who can push a workflow can run as root on the box.
Keep repo write access to trusted people.
**Login via Authentik (Discord)** — same pattern as Jellyfin, done once:
1. Authentik → Applications → Create with provider → name `Gitea`, slug `gitea`; **OAuth2/OpenID Provider**,
authorization flow implicit-consent, client type Confidential, redirect URI (strict)
`https://git.HOST/user/oauth2/authentik/callback`, signing key = the default certificate, scopes
`openid email profile` (the default profile mapping carries `preferred_username` and `groups`). Copy client ID + secret.
2. Gitea → Site Administration → Identity & Access → Authentication Sources → Add: type **OAuth2**, name
**`authentik`** (must match the callback path), provider **OpenID Connect**, client ID/secret, OpenID Connect
Auto Discovery URL `https://auth.HOST/application/o/gitea/.well-known/openid-configuration`, tick "Skip local
2FA", claim name for groups `groups`, administrator group claim value **`git-admins`**. Save.
Authentik group nesting makes the admins inherit it: `authentik Admins` → parent `admins` → parent `git-admins`
(members of a child group count as members of the parent). **Inherited groups are not in the `groups` claim by
default**: the managed profile mapping uses `ak_groups` (direct memberships only), and proxy providers re-add
that mapping on every save, so it cannot be swapped out. Add a second scope mapping, Customization → Property
Mappings → Create → Scope Mapping, name `Vanguard profile (nested groups)`, scope `profile`, expression:
direct = set(request.user.ak_groups.all())
return {"groups": sorted(g.name for g in request.user.all_groups() if g not in direct)}
and tick it (next to the default one) on the `Vanguard sites`, `Jellyfin` and `Gitea` providers → Advanced
protocol settings → Scopes. Claims from several mappings merge, so the header becomes direct + inherited groups
without duplicates. Takes effect on the next login.
3. compose (already set): registration only via external sources, auto-registration + account linking on,
username from `preferred_username`. A first Discord login creates the Gitea user with no form.
4. Access = Authentik group **`git-users`**: Directory → Groups → create it (add a Discord role to it in the
roles→groups table); Applications → `Gitea` → Policy / Group / User Bindings → bind `git-users` and `git-admins`.
Non-members are refused by Authentik before Gitea sees them.
5. Test in a private window: https://git.HOST/user/login → "Sign in with authentik" → Discord → logged in.
Members of `admins` become Gitea site admins on login. The local admin account stays as a fallback.
Optional later: hide the password form (custom template) so Discord is the only visible way in.
## 5f. Live portal (`live.HOST`)
React + dockview page (`live/`) behind Authentik forward-auth: every active stream is a dockable window,
audio follows the focused window, per-window mute + volume. WebRTC (WHEP) with HLS fallback, served by
MediaMTX (HLS 8888, WebRTC 8889, API 9997 on loopback; WebRTC media on **8189** udp/tcp, firewall opened).
MediaMTX runs the `-ffmpeg` image for the on-ready hook `bin/mtx-on-ready.sh` (bind-mounted into the container):
the optional "is live" Discord post (`LIVE_WEBHOOK_URL`, one of the Discord settings grouped at the top of
`.env`; set it with `./configure.py --set LIVE_WEBHOOK_URL=…`, which tells you MediaMTX needs a recreate) and the
**WebRTC audio mirror**. Browsers cannot receive AAC over WebRTC (MediaMTX forwards Opus/G711/G722/LPCM only) and
OBS sends AAC, so for every stream whose audio is not already Opus the hook runs one ffmpeg that copies the video
and transcodes the audio to Opus into the sibling path `<user>/<Title>.webrtc` (via MediaMTX's RTSP on the
container loopback). The portal plays the mirror over WebRTC and hides it as a panel; recordings, HLS and the
Discord post stay on the original. Mirrors are never recorded and can only be published from inside the container
(dumpd checks the source IP). Browser Go Live already sends Opus and gets no mirror. Cost: a few % of a core per stream.
Viewer requirement on the streamer side: **B-frames 0** (section 5), or WebRTC is refused and the portal falls back to HLS.
1. CNAME `live` → main host; `./configure.py --full` asks the hostname (also `--hosts … LIVE`).
2. Authentik: Proxy Provider (forward auth, single application, external host `https://live.HOST`) →
embedded outpost → group policy (e.g. everyone in the server, or `streamers`+`editors`+`admins`).
3. Dev box: `./deployLive.sh` builds `live/` and copies `dist/` to `/srv/live` (Caddy serves it directly).
**Go Live** button (window audio: Windows + Chrome/Edge 141+; Linux/Mac get video only): broadcast a window from the browser (no OBS) as `<user>/HelmetCam-<date>` — a fresh publish token comes
from `/me/stream-token` for the logged-in Discord user (member of `streamers`).
Bandwidth: no transcoding, every viewer gets the source bitrate (~15 Mbps) → ~30 concurrent viewers on
the 500 Mbps line. For a crowd, restream to Twitch from OBS.
## 5g. Authentik forward-auth for the Caddy-gated hosts (`media.`, `files.`, `clips.`, `sync.`, `live.`, `bot.`)
Design: **Authentik authenticates once for the whole domain; Caddy authorizes per host** from the
`X-Authentik-Groups` header the outpost returns; **Discord roles fill the Authentik groups** at login.
1. Applications → Outposts → edit "authentik Embedded Outpost" → config `authentik_host: https://auth.HOST`.
2. Applications → Create with provider → Application `Vanguard sites`; **Proxy Provider**, authorization flow
implicit-consent, mode **Forward auth (domain level)**, authentication URL `https://auth.HOST`,
cookie domain = the parent domain (e.g. `dumpstervigilantes.rip`). Edit the Embedded Outpost →
Applications → add it → Update. One login now covers every gated host.
3. Per-host group requirement = `.env`: `BOT_REQUIRE_GROUP=bot-users` (default), `LIVE_REQUIRE_GROUP=`
(empty = any logged-in member), `SYNC_REQUIRE_GROUP=media-admins` (the Syncthing UI on `sync.`).
On `files.` copyparty itself authorizes by group (`editors`, `media-admins`); the dump endpoint lets
`admins`/`media-admins` dump anyone's night. Leave "Intercept header authentication" **off**: login is
Discord only, no app passwords, no WebDAV. `./configure.py` apply regenerates the Caddy blocks; a user outside the
group gets a 403 from Caddy before the app sees the request.
4. Discord roles → groups: Directory → Groups: create `bot-users` (and keep `streamers`, `editors`, `admins`).
Discord source → Scopes: `guilds guilds.members.read`. Customization → Policies → Expression Policy
`discord-roles-to-groups` from `authentik/discord-roles-to-groups.py` (fill in the server ID and the
role-id → group table). Bind it to `default-source-enrollment` **and** `default-source-authentication`
(engine mode `all`, after `discord-guild-member`). From then on giving someone a Discord role gives them
the matching access on next login; removing it removes it. Memberships added by hand in Authentik are left
alone: the policy only revokes what it granted itself (tracked in the user attribute `discord_role_groups`).
copyparty additionally requires a header with a secret name that only Caddy adds (`idp-h-key`,
`COPYPARTY_IDP_KEY` in `.env`), so identity headers cannot be forged even if copyparty were reached directly.
Logout: copyparty's logout link, the live portal's "Sign out" and the dock's "sign out" go through
`/outpost.goauthentik.io/sign_out`; `https://<any gated host>/logout` does the same from anywhere
(e.g. from the Syncthing UI, which has no logout of its own). Set the provider's **Invalidation flow** to
`default-invalidation-flow` — the one named **"Logout"**, the same flow Authentik's own logout button uses.
Do NOT use `default-provider-invalidation-flow`: despite the name it is called "Logged out of application"
and ends the session for that one application only, leaving the SSO session alive. With it, `/logout` drops
the proxy cookie, forward-auth asks Authentik again, Authentik still knows you, and you are signed straight
back in — which looks exactly like the page merely refreshing. With `default-invalidation-flow` every
logout ends the whole session and forces a full Discord re-login on the next visit (and so picks up group
changes). Discord itself may still approve silently if you remain logged into Discord in that browser.
5. Debug: `https://HOST/whoami` prints the username/groups headers exactly as the apps receive them, and
`https://HOST/me` adds the `discord_id` claim decoded from `X-Authentik-Jwt` — the quickest check that the
`Vanguard profile` scope mapping is live. The raw token is deliberately never echoed: the browser does not
see that header today, and printing it would hand the user a bearer token they otherwise could not read.
Group changes in Authentik apply on the next login — log out at `https://auth.HOST/` and back in.
6. Test in a private window: `https://live.HOST/` → Discord login → portal; `https://bot.HOST/` → 403 unless
you hold the bot role; `https://live.HOST/me/stream-key` shows your username.
Jellyfin and Gitea use OIDC providers instead, not this.
## 5h. Diagnostics: Kibana at `diag.HOST`
Four containers, one store, one UI. Kibana is served at the **root** of `diag.HOST`, with no sub-path to get
wrong. Arriving there opens a dashboard of ours, "Vanguard ops", via Kibana's `defaultRoute` setting: links to
every host and to the triage page, plus shortcuts into Discover, Dashboards, Stack Rules and Index Management.
It is an ordinary dashboard, so add metric panels beside the links (`vanguard_media_pending` first).
Two paths on that host are ours rather than Kibana's:
| path | what |
|---|---|
| `/stuck` | **stuck recordings**: everything that finished but never published, why each one is stuck, and Retry / Bin buttons. The list is readable by anyone the gate allows; the actions are enforced in dumpd for the admin groups, not hidden in the page. Binning moves the file to `trash/<today>/`, the same 3-day undo as a dumped night |
| `/stuck/api` | dumpd serves it. Both the page and `vgstuck.sh` on the host run the same script, so they cannot disagree about why something is stuck |
Kibana deliberately does **not** live under `/kibana`. That would need `server.basePath` plus
`rewriteBasePath`, a Caddy handle that does not strip the prefix, and the prefix appended in both Beats'
`setup.kibana`, `kibana-setup.py`, `kibana-alarms.py` and the deadman's status check — and missing the Beats
one crash-loops Metricbeat, which exits when Kibana is unreachable. `defaultRoute` buys the landing page
without any of it.
`observability/kibana-setup.py` creates the first-run objects and takes a step name, so you can redo one:
`connector`, `rule`, `dataview`, `landing`, or all of them when omitted. `observability/kibana-alarms.py`
creates the six alarm rules.
| container | job |
|---|---|
| `elasticsearch` | one store for **both** logs and metrics. 2 GB heap, single node, loopback only |
| `kibana` | Discover, dashboards, alerting. Published as `https://diag.HOST/` behind the gate |
| `filebeat` | every container's logs (autodiscover) plus our journal units |
| `metricbeat` | host + Docker metrics, replacing node_exporter and cAdvisor, plus MediaMTX's and Caddy's Prometheus endpoints |
Both Beats write the same correlation fields (`service.name`, `container.name`, `vanguard.category`,
`@timestamp`), so one dashboard can show a CPU spike beside the log lines from that minute and one filter
narrows both. `LOGGING.md` lists the fields and the single Kibana data view that makes it work.
Four things are pinned on purpose, each because the default bites:
| setting | why |
|---|---|
| `ES_JAVA_OPTS=-Xms2g -Xmx2g` | a self-sizing JVM takes a quarter of the box; the spare capacity belongs to x264 |
| `vm.max_map_count=262144` | a host sysctl, set and persisted by `install.sh` and by every apply. ES will not start without it |
| `xpack.security.enabled=false` | safe only because it listens on loopback behind Authentik; it is also why there is no second login |
| `KIBANA_ENCRYPTION_KEY` | without a fixed key Kibana loses every saved object, dashboards included, on restart |
**First-run steps in Kibana** (both are UI actions on purpose, so they stay adjustable without a redeploy):
1. **Retention**: Stack Management → Index Lifecycle Policies → the beats policy → add a delete phase.
2. **The deadman's heartbeat**: a rule that always fires, every minute, with a webhook action posting to
`http://<docker gateway>:8081/heartbeat`. That proves rule evaluation *and* delivery work end to end, and the
deadman treats a stale heartbeat as the alarm. Without it the deadman reports the heartbeat missing, which is
correct rather than a bug.
3. **Alerts to Discord**: a webhook connector pointing at `ALERT_WEBHOOK_URL`. Until one exists, rules still
evaluate and show in Kibana, they are simply not delivered.
**The deadman is the part that matters most.** `vanguard-deadman.timer` runs on the host every 5 minutes with
nothing but curl, because nothing inside Elastic can report that Elastic is the thing that broke:
| it checks | because |
|---|---|
| Elasticsearch answers, cluster not red | no store, no anything |
| no index is read-only | the disk watermark sets that and **does not unset it** when space is freed, so ingestion stays silently dead |
| Kibana is available | it is the only UI |
| each Beat wrote a document recently | "running" is not "shipping" |
| the Kibana heartbeat is fresh | proves the whole rule → webhook → delivery path |
| the disks | the likeliest death, and it must be reported even when the reporter is what broke |
systemctl list-timers vanguard-deadman.timer
journalctl -u vanguard-deadman -n 20
sudo /usr/local/bin/vanguard-deadman.sh # run it now; exit 1 means it found something undeliverable
Filebeat holds the Docker socket for autodiscover, which is root-equivalent on this host — the same exposure
act_runner has. Accepted deliberately; the alternative is losing the container labels that make the
correlation above work.
## 6. Editors
Log in with Discord at https://files.BASE/ (copyparty behind Caddy + Authentik). Access is by Authentik
group: **editors** read streams and clips and write `/finals`; **media-admins** have full rights everywhere
including `/trash`. Scrub any `.mp4` in the browser, download what you want.
No WebDAV: login is Discord only (no passwords of any kind), and a mounted drive cannot do a browser login.
Download from the browser: single files, or select files/folders and use copyparty's zip/tar download.
> **Available but not set up.** WebDAV mounts can be enabled later without code changes if we decide we want
> them: in Authentik, edit the `Vanguard sites` proxy provider and tick **Intercept header authentication**;
> each editor then creates an *app password* under their Authentik user settings and mounts the same URL with
> their Authentik username + that app password. Caddy and copyparty already pass the identity through, so
> nothing else changes. It is off because it introduces a password-like credential alongside Discord login.
Every `.mp4` an editor sees is the same kind of file (see "Editing standard"). Raw clipper output that
needed converting is kept in a hidden `.src/` folder next to it (media-admins can show dotfiles in copyparty).
## 7. Home archive box
Copy `home/rclone-archive.sh` to the home machine, set the vars, run from cron nightly.
It connects over SFTP as `debian` (all media files are owned by uid 1000 so it can delete them).
Pulls everything older than 15 days off the server, including the hidden `.src/` raw files.
## Publish authentication (no passwords)
MediaMTX asks dumpd (`POST /mtx-auth` on 127.0.0.1:8081, reachable only from Docker's bridge) for every
connection. Reads/playback are allowed (viewers are gated by Caddy + Authentik). Publishing needs a token
minted by `GET /me/stream-token` behind Authentik for a member of `streamers`/`admins`/`media-admins`; dumpd
checks the HMAC signature (`/etc/vanguard/stream-token.secret`, generated into `.env` as `STREAM_TOKEN_SECRET`),
the expiry, and that the path starts with that user's name. MediaMTX does not pick up `mediamtx.yml` changes
live on this setup: after any change to it, `docker compose restart mediamtx` between streams. Day-to-day
operations (streamers, tokens, titles) never touch that file. `configure.py` reloads Caddy in place and
restarts only copyparty.
## Timecode for multicam editing (frame accurate)
Every published stream carries a QuickTime **timecode track** (`tmcd`, time of day in Eastern, at the file's
frame rate) and a UTC `creation_time`, written by the sweep when it publishes the file, plus a sidecar
`<Title>__<ts>.json` with the exact capture time (the Jellyfin plugin reads it). Resolve and Premiere read the
track natively: drop clips from several streamers on a multicam timeline, sync by timecode.
Where the time comes from, best first:
1. **Overscan band** — a thin strip of extra rows below the picture carrying the *server* clock, encoded
(black/white cells, format in `bin/vgstrip.py`) plus a small readable clock. The server decodes it
(`bin/vgstrip.py`) at the start, middle and end of each recording → the timecode is frame-accurate at the
start and the sidecar records drift over the file. **Then the band is cropped off**: the sweep re-encodes the
stream (libx264 CRF 18, frame timing preserved, audio copied) so the published file is the game image + audio +
the `tmcd` track and nothing else; the raw with the band is deleted (`KEEP_RAW_STREAMS=1` keeps it in `.src/`).
Streams without the band are remuxed losslessly. Cost: every re-encode (streams and clips) runs x264 on
`ENCODE_THREADS` threads (2, set in `systemd/normalize-sweep.service`) so Jellyfin and live viewers stay
responsive; a 3 h 1440p60 night therefore takes most of the next day to publish. OBS: dock → *Timecode band* →
"Add band + clock source" (canvas grows by W/160 rows, a locked browser source `Vanguard Clock` →
`https://live.BASE/dock/clock/` sits in the band; only while not streaming). Go Live: the "Timecode band" checkbox (default on).
2. **Server arrival time** (no band) — microseconds in the recording filename. Within ~0.1–0.3 s per streamer
(encoder + network delay); coarse but always available.
Viewing: the live portal crops the band on the live picture (the only place it still exists); recordings in
Jellyfin, copyparty and the editors' timelines are already at game resolution. Recordings published before
the crop existed: `sudo vgdeband.sh [FILE…]` re-encodes them the same way and marks the sidecar.
Cross-check when it matters: everyone is in the same voice chat, so audio-waveform sync is sample-accurate.
Why continuous and not a start burst: a PC clock drifts vs the server (10 ppm ≈ 4 frames over 2 h).
Dropped frames / gaps: OBS keeps stamping frames from its clock, so after a drop the file has a hole and every
frame that exists is still at the right time — the timecode stays correct, and players (Jellyfin, the portal)
hold the last frame across the hole with audio in sync. By default the sweep does **not** re-encode a stream to
fill holes (hours of CPU on this box); the sidecar lists them (`gaps.count`, `total_missing_ms`, first 50
positions). Premiere preserves timing on such files; in Resolve check the clip's frame-rate interpretation.
**Gap filling (available, off by default).** The sweep can re-encode a stream whose holes total at least
`FILL_GAPS_MS` milliseconds to constant frame rate with duplicated frames (raw kept in `.src/`, timecode carried
over). It costs ~2× the stream length in CPU on the E-2136, so it is off until the box has a hardware encoder.
Turn it on (threshold 1 s) / off, on the server:
sudo sed -i 's|^#* *Environment=FILL_GAPS_MS=.*|Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
sudo sed -i 's|^Environment=FILL_GAPS_MS=.*|# Environment=FILL_GAPS_MS=1000|' /etc/systemd/system/normalize-sweep.service && sudo systemctl daemon-reload
Check: `systemctl show normalize-sweep -p Environment`. Takes effect on the next sweep run; already-published
files are not revisited. Clips are always converted to CFR regardless.
Planned alternative: re-encode flagged streams off-site on a machine with a hardware encoder (home box NVENC or a
Spot GPU) and push the result back — see CommunityDiscordMedia.md → Later.
## Deployment (push to deploy — built, OFF by default)
What lands in `/usr/local/bin` is listed once, in `bin/install.list`, and read by `install.sh`,
`configure.py --apply` and `deploy.sh`. It used to be three hand-kept lists and they drifted: a push was
installing five of ten scripts, so `vgstuck.sh`, both sourced helpers and both timer scripts only reached the
host if somebody ran `--apply` by hand. `check.sh` now fails if the list names a missing file, or if
something in `bin/` is neither listed nor explicitly excluded (`mtx-on-ready.sh` is mounted into the MediaMTX
container; `vgbot-merge-sqlite.py` is a one-off migration).
The server folder is a git clone of this repo (`./switchToClone.sh key` → add the printed deploy key in Gitea →
`./switchToClone.sh clone`). Generated files (`.env`, `Caddyfile`, `mediamtx.yml`, `copyparty.conf`, the token
secret) are gitignored and survive pulls; static pages are bind-mounted from the checkout.
`sudo vanguard-deploy pull` fetches `main` and applies **only what changed**, after `check.sh`:
| changed | action | interruption |
|---|---|---|
| `configure.py`, `dock/`, `clips/`, `about/`, stignore | `configure.py --apply` → Caddy reload, copyparty restart | none / a second |
| `bin/*`, `systemd/*` | install; dumpd restart when it changed; timers pick up next run | a second |
| `firewall/nftables.conf` | `nft -f` | none |
| `docker-compose.yml`, `bot/Dockerfile`, `bot/entrypoint.sh` | `compose up -d --build` for changed services | that service |
| `deployBotBundle.sh` | reinstalled as `/usr/local/bin/vanguard-deploy-bot` (what dumpd runs for the bot's CI) | none |
| anything that changes `mediamtx.yml` or the mediamtx service | **restart only when no stream is live**; otherwise deferred, `deploy-pending.timer` retries every 5 min | none while live |
| `jellyfin-plugin-vanguard/dist/*.zip` (committed) | `installJellyfinPlugin.sh` | Jellyfin restart |
| `live/` | `npm run build` on the server → `/srv/live` | none |
`sudo vanguard-deploy rollback` checks out the previously deployed revision and applies the reverse diff.
`sudo vanguard-deploy status` shows revisions, auto mode, pending restarts, live streams.
**One-time rollout** (the last scp-style deploy; afterwards the server pulls). From the dev box:
cd ~/claudevs/vanguard-stream && git push
scp bin/dumpd.py debian@HOST:~/vanguard-stream/bin/
scp systemd/deploy-pending.service systemd/deploy-pending.timer debian@HOST:~/vanguard-stream/systemd/
scp docker-compose.yml configure.py deploy.sh switchToClone.sh check.sh install.sh README.md STATUS.md CommunityDiscordMedia.md debian@HOST:~/vanguard-stream/
ssh -t debian@HOST 'cd ~/vanguard-stream && sudo apt-get install -y -q git nodejs npm rsync python3-yaml >/dev/null && sudo install -m 0755 bin/dumpd.py /usr/local/bin/ && ./configure.py --apply && sudo systemctl restart dumpd && sudo vanguard-deploy status'
The apply prints the post-start fixups (Syncthing, Gitea admin user, Jellyfin plugin) and ends with a status
block: `DEPLOY_AUTO=0`, no pending restart, and "not a git checkout" until the next step.
Then switch the server folder to a clone (two steps):
ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh key' # prints a public key
# Gitea → repo → Settings → Deploy Keys → add it, read-only
ssh -t debian@HOST 'cd ~/vanguard-stream && ./switchToClone.sh clone && cd ~/vanguard-stream && ./configure.py --apply >/dev/null && sudo vanguard-deploy status'
`switchToClone.sh clone` keeps `.env`, the generated configs, the token secret and plugin zips, and
leaves the old copy at `~/vanguard-stream.old` until you delete it. From here on a manual deploy is
`sudo vanguard-deploy pull` on the server.
**Automatic**: Gitea → repo → Settings → Webhooks → Gitea, URL `http://<docker gateway>:8081/deploy`
(`docker network inspect vanguard-stream_default -f '{{(index .IPAM.Config 0).Gateway}}'`, usually `172.18.0.1`),
secret = `DEPLOY_WEBHOOK_SECRET` from `.env`, push events, branch `main`. dumpd verifies the HMAC and runs
`vanguard-deploy pull` **only when `DEPLOY_AUTO=1`** in `.env` (then `./configure.py --apply` to push the setting
to `/etc/vanguard/deploy.env`). Until then every push is logged (`journalctl -u dumpd | grep deploy`) and ignored; Gitea's "test delivery"
should get a 202 "auto-deploy disabled" back. Optional `DEPLOY_NOTIFY_WEBHOOK_URL` posts deploy results to Discord.
Turning it on: `DEPLOY_AUTO=1` in `.env`, `./configure.py --apply`, and
`git mv .gitea/draft-workflows/ci.yml .gitea/workflows/ci.yml` for CI.
CI: `.gitea/draft-workflows/ci.yml` (inert until moved to `.gitea/workflows/`) runs `check.sh` and the plugin
tests/build on the Actions runner. The runner never deploys — it must not touch the host.
## Rebuild from scratch (what is and isn't automated)
Server side, scripted:
git clone ssh://git@git.HOST:2222/techy/Media-Server-Setup.git ~/vanguard-stream # or scp the folder
cd ~/vanguard-stream && sudo bash install.sh # packages, docker, folders, host scripts, timers, firewall, portal build
./configure.py --full # hostnames, secrets, configs, containers, post-start fixups
Restore first if rebuilding a known box: `.env` (all secrets — without it every password is regenerated and
the databases won't match), `/srv/appdata` (Authentik DB, Jellyfin, Gitea, MariaDB, Syncthing config),
`/srv/media` (the footage), `/etc/vanguard/clip-quotas.json` if used. With those restored, nothing below is
needed except DNS.
Post-start fixups `configure.py` performs on every apply (idempotent): Syncthing GUI auth off, Gitea admin
user created from `.env` (`GITEA_ADMIN_USER` / `GITEA_ADMIN_PASSWORD`; the install page is locked), Jellyfin plugin
installed from the committed zip. Authentik's `akadmin` password is set on first start from
`AUTHENTIK_BOOTSTRAP_PASSWORD` in `.env` (no initial-setup page).
Still manual on a truly fresh box (not in the repo; documented in §5c–5g):
- Authentik: Discord source, identification-stage source, enrollment user type, groups, the two policies,
the domain-level proxy provider + outpost host, the Jellyfin and Gitea OIDC providers, application bindings
- Jellyfin: first-run wizard, libraries (Shows + Vanguard provider), SSO plugin + provider, branding
- Gitea: authentication source (authentik), deploy key, webhook; `./registerGiteaRunner.sh`
- Discord bot: its repo lives outside this one, so nothing here restores it. Put `BOT_DEPLOY_SECRET`
(`./configure.py --show`), `DISCORD_BOT_TOKEN` and `DISCORD_GUILD_ID` into that repo's Gitea Actions
secrets/variables, then push to its `main` — or install a bundle by hand with `./deployBotBundle.sh`.
`/srv/bot/app` is created empty by `install.sh`; the container idles until a release lands there.
- Discord developer portal: application, redirect URIs, bot; DNS records
A `.env`-driven Authentik blueprint would remove the first line (planned).
## Layout on the server
/srv/media/streams/<user>/<Title>/.live/<ts>.mp4 recording in progress (hidden from Jellyfin/copyparty)
/srv/media/streams/<user>/<Title>__<ts>.json capture time (UTC ms), timecode, source (strip|arrival), drift
/srv/media/streams/<user>/<Title>__<ts>.mp4 finished stream, one flat MP4 per recording, directly in
the streamer's folder (several files inside one title
folder would be ONE Jellyfin episode with "versions")
/srv/media/clips/<user>/clip.mp4 clip (synced as-is, or converted; raw removed)
/srv/media/clips/<user>/.stlocal Syncthing ignore list maintained by the sweep
/srv/media/clips/<user>/.src/clip.mkv raw source, only for clips that did NOT arrive via Syncthing
/srv/media/clips/<user>/clip.normalize-failed.log only if normalization failed; delete to retry
/srv/bot/app/ the Discord bot's installed release (bind-mounted at /app)
/srv/bot/previous/ the release before it (deployBotBundle.sh --rollback)
/srv/media/finals/ editors upload here
/srv/media/trash/<night>/... dumped, purged after 3 days
/srv/media/.markers/ dump markers
/srv/media/.syncthing/ Syncthing config
## Editing standard (what normalize-sweep enforces)
MP4 · H.264 High, 8-bit 4:2:0 · constant frame rate · AAC audio (all tracks kept) · index at the
front (instant seek in browser and NLE) · at most 2560x1440 and 60 fps (downscaled to fit, never
upscaled; aspect kept).
| input | action |
|-------|--------|
| already compliant | nothing |
| MediaMTX recording in `<Title>/.live/` (fragmented MP4) | once idle 3 min: decode the timecode band (or use arrival time), then publish to `<user>/<Title>__<ts>.mp4` with a `tmcd` track + `creation_time` — band present: re-encode with the band cropped off; no band: lossless remux — write the `.json` sidecar, remove `.live` and the empty title folder |
| in a Syncthing folder (`.stfolder` marker) | compliant / index-only: untouched; otherwise convert → `name.mp4`, add raw to `.stlocal`, delete raw (one copy) |
| HEVC / 10-bit / VFR / MKV / >1440p / >60fps / non-AAC | full-res re-encode (x264 veryfast CRF 18) → `name.mp4`; source kept in `.src/` |
| ffmpeg fails or duration mismatch | `name.normalize-failed.log` written, file skipped until that log is deleted |
Runs every 5 min on files idle ≥3 min and not open. Software x264 on the E-2136 does ~0.5x realtime
at 1440p60, so a 2 h clip batch takes ~4 h; streams only remux (minutes). All at idle priority.
## Reboot behaviour
Everything comes back on its own, in this order:
1. mdadm assembles the mirror, systemd mounts /srv (local-fs). A degraded mirror still boots.
2. docker.service has `RequiresMountsFor=/srv/media` (drop-in installed by install.sh), so it never
starts with the media tree missing. All containers are `restart: unless-stopped`.
3. dumpd.service starts; normalize-sweep.timer fires 2 min after boot and every 5 min after;
trash-purge.timer is `Persistent=true` so a missed 05:00 run fires at boot.
4. A stream cut by a reboot is playable up to the last second (fMP4 parts are flushed every 1 s);
OBS reconnects and MediaMTX opens a new file in the same title folder.
5. Half-written `*.part` files are overwritten by the next sweep. Dumps are idempotent.
Time: install.sh sets the server clock to America/New_York so journal, timers, file timestamps
and the dump window all read in Eastern. Dump "night" = 17:00 → 02:00 Eastern (pinned in dumpd via
DUMP_TZ, independent of the host zone). Between 02:00 and 17:00 the default night is the previous
evening; after 17:00 it is tonight. The 05:00 trash purge is 05:00 Eastern.
## Timers
| unit | when | does |
|------|------|------|
| normalize-sweep.timer | every 5 min | remux/re-encode finished files to the editing standard |
| trash-purge.timer | daily 05:00 Eastern | deletes trash older than 3 days |
| vanguard-alerts.timer | every 1 min | delivers Kibana's alert documents to Discord, with a 15-min per-rule repeat guard |
| vanguard-deadman.timer | every 5 min | watches the alerting stack from outside it, and the disks |
| dumpd.service | always | the dump endpoint on 127.0.0.1:8080, behind Caddy on `live.` at /dump, /me, /time, on `clips.` at /me/clips and on `diag.` at /stuck/api |
Reference in New Issue View Git Blame Copy Permalink